# CAREA: Cotraining Attribute and Relation Embeddings for Cross-Lingual Entity Alignment in Knowledge Graphs

Authors: Baiyang Chen; Xiaoliang Chen; Peng Lu; Yajun Du
Journal: Discrete Dynamics in Nature and Society
 (2020)

---

## Abstract

Knowledge graphs (KGs) are one of the most widely used techniques of knowledge organizations and have been extensively used in many application fields related to artificial intelligence, for example, web search and recommendations. Entity alignment provides a useful tool for how to integrate multilingual KGs automatically. However, most of the existing studies evaluated ignore the abundant information of entity attributes except for entity relationships. This paper sets out to investigate cross-lingual entity alignment and proposes an iterative cotraining approach (CAREA) to train a pair of independent models. The two models can extract the attribute and the relation features of multilingual KGs, respectively. In each iteration, the two models alternate to predict a new set of potentially aligned entity pairs. Besides, this method further filters through the dynamic threshold value to enhance the two models’ supervision. Experimental results on three real-world datasets demonstrate the effectiveness and superiority of the proposed method. The CAREA model improves the performance with at least an absolute increase of 3.9% across all experiment datasets. The code is available at https://github.com/ChenBaiyang/CAREA.


---

## Body

## 1. Introduction

Knowledge graphs (KGs) that possess machine-readable representations of factual knowledge are becoming the basis for many applications such as web search (Google and Bing), recommendations (Amazon and eBay), and social networks (Facebook and Linkedin). Multilingual KGs (e.g., DBpedia [1], YAGO [2], and ConceptNet [3]) are constructed in separate languages from various data sources and contain a wealth of complementary facts. The bridging of language gaps and the improvement of user experience from downstream cross-language applications benefit a lot from the entity equivalent in multilingual KGs. Hence, aligning the entities in multilingual KGs has recently attracted an increasing amount of research attention and is called the problem of cross-lingual entity alignment.Most existing entity alignment methods entirely rely on the graph structures, while the abundant attribute information in KGs remains unexplored. The attributes of an entity represented by different languages often share enormous semantic information, leading to a potentially valid view of the entities connected to multilanguage KGs. However, it is nontrivial to capture and make use of such information for cross-lingual entity alignment. First, attribute information can be quite diverse across different KGs. The most likely cause of the differences is that there exist distinct attribute concerns in the process of developing applications. Second, the semantic association of attributes cannot be modeled directly since the critical entity expression languages are different. Moreover, the simultaneous use of relationships and attributes across multilingual KGs is a near term challenge in the area of knowledge graphs.Cotraining is a popular machine learning method, where two complementary models utilize a large number of unlabeled examples to bootstrap the performance of each other iteratively [4, 5]. Cotraining can be readily applied to multilingual tasks since the data in these tasks have two or more views (i.e., a subset of features). It is also applicable to employ cotraining to the task of entity alignment across multilingual KGs, as the entity attributes and graph structure information naturally form two independent views of a KG. In the cotraining framework, each model is trained on one of the two views, under the assumption that either view is sufficient to make a prediction. In each iteration, the cotraining algorithm selects high-confidence samples ranked by each of the models to form new auto-labeled data samples and then uses both labeled data and additional auto-labeled data to update the other model.This paper introduces a cotraining based approach CAREA to learn embeddings from two independent views of knowledge (relationships and attributes) in multilingual KGs. CAREA iteratively trains two-component models that are called attribute-based modelfattr and structure-based model fstruc, respectively. fattr extracts the attribute features according to attribute occurrence frequencies and value data types, which also employs a Multilayer Perceptron (MLP) to transform both KGs into a unified vector space. On the other hand, fstruc adopts a graph attention mechanism to capture the multirelation characteristics of KGs. During each iteration of the cotraining process, both models alternately predict a set of new potential aligned entity pairs to strengthen the supervision of cross-lingual learning. Such collaborative predictions gradually improve the performance of each model. To improve the accuracy of aligned entity prediction, we further evaluate the predicted entity pairs through a dynamic threshold. Experimental results on three real-world datasets demonstrate the effectiveness and superiority of our proposed method CAREA.The rest of this paper is organized as follows. Section2 summarizes the related works. Section 3 formally defines the research question. Section 4 introduces the proposed approach. Section 5 presents the experimental results. Finally, we conclude this work in Section 6.


## 2. Related Work

### 2.1. KG Embedding

Embedding-based entity analysis approaches have demonstrated their effectiveness in modeling the semantic information of KGs, which aim to project entities into low-dimensional embedding spaces. The KG embedding model TransE [6] interprets a relation as the translation from an entity to another. Such KG embedding models using the translations have shown their feasibility and later been improved by several subsequent studies, such as TransH [7], TransR [8], and TransD [9].TransH and TransR update the modeling of multimapping relations of TransE from one to many. TransD uses a dynamic matrix to transfer entities and relations rather than a fixed one. R-GCN [10] is a similar model that incorporates relation type information by setting a transformation matrix for each relation. Some authors consider avoiding the use of translation approaches for KG embedding, including [11–15]. The perfect example is shown in the study of Nathani et al. [15], which extended the graph attention mechanism (GAT) to capture entity and relationship features in the multihop neighbors of a given entity. Some research below makes use of the additional information in KGs to improve embedding performance. For example, reverse triples and relational paths are combined in PTransE [16]. The categorical attributes such as gender and hobby are introduced in KR-EAR [17]. In addition, some works explore the type, local structure, and global patterns in KG embedding [10, 18–20].


### 2.2. Entity Alignment

Entity alignment aims to automatically determine whether an entity pair in different KGs refers to the same entity in reality. Traditional entity alignment methods take advantage of various features of KGs, such as the semantics of OWL properties [21], compatible neighbors and attribute values of entities [22], and the relation structures [23].Many recent studies have used embedding methods to deal with the alignment problem in KGs. MTransE [24] deploys three mechanisms, distance-based axis alignment, translation vectors, and linear transformations to learn multilanguage KG embeddings. An improved model IPTransE [25] combines the advantages of TransE and PTransE to embed a single KG. Then, an iterative and parameter sharing step is added in IPTransE for various KGs embedding. BootEA [26] improves on JAPE [27] by using the bootstrapping strategy, which provides an iterative data labeling method. Accordingly, the constructed training data for potential entity alignment can be used to learn entity alignment-oriented embedding. MuGNN [28] learns alignment-oriented KG embeddings by a multichannel mechanism that encodes KGs via KG completion and entity pruning. NAEA [29] merges neighborhood subgraph-level information of entities and designs a neighborhood-aware attention representation mechanism on cross-lingual KGs. RDGCN [30] proposes a relation-aware dual-graph convolutional network to leverage relations through attentive interactions between the KG and its dual relation counterpart. MRAEA [31] learns cross-lingual entity embeddings by attending over the entity’s neighbors and the meta semantics of its connecting relations.Some literature on cross-lingual entity alignments has highlighted the role of both KG structures and attributes. JAPE [27] embeds the structures of different KGs into a uniform hidden space and uses the attribute correlation of KGs to realize the refinement of entity embedding. However, attribute components can significantly degrade the performance of JAPE’s structural components when attributes are heterogeneous or have a confused association between the attributes. Graph convolutional networks (GCNs) [32] are also employed in the study [33] to learn embeddings from both the structure and attribute information of entities for cross-lingual alignment.


## 2.1. KG Embedding

Embedding-based entity analysis approaches have demonstrated their effectiveness in modeling the semantic information of KGs, which aim to project entities into low-dimensional embedding spaces. The KG embedding model TransE [6] interprets a relation as the translation from an entity to another. Such KG embedding models using the translations have shown their feasibility and later been improved by several subsequent studies, such as TransH [7], TransR [8], and TransD [9].TransH and TransR update the modeling of multimapping relations of TransE from one to many. TransD uses a dynamic matrix to transfer entities and relations rather than a fixed one. R-GCN [10] is a similar model that incorporates relation type information by setting a transformation matrix for each relation. Some authors consider avoiding the use of translation approaches for KG embedding, including [11–15]. The perfect example is shown in the study of Nathani et al. [15], which extended the graph attention mechanism (GAT) to capture entity and relationship features in the multihop neighbors of a given entity. Some research below makes use of the additional information in KGs to improve embedding performance. For example, reverse triples and relational paths are combined in PTransE [16]. The categorical attributes such as gender and hobby are introduced in KR-EAR [17]. In addition, some works explore the type, local structure, and global patterns in KG embedding [10, 18–20].


## 2.2. Entity Alignment

Entity alignment aims to automatically determine whether an entity pair in different KGs refers to the same entity in reality. Traditional entity alignment methods take advantage of various features of KGs, such as the semantics of OWL properties [21], compatible neighbors and attribute values of entities [22], and the relation structures [23].Many recent studies have used embedding methods to deal with the alignment problem in KGs. MTransE [24] deploys three mechanisms, distance-based axis alignment, translation vectors, and linear transformations to learn multilanguage KG embeddings. An improved model IPTransE [25] combines the advantages of TransE and PTransE to embed a single KG. Then, an iterative and parameter sharing step is added in IPTransE for various KGs embedding. BootEA [26] improves on JAPE [27] by using the bootstrapping strategy, which provides an iterative data labeling method. Accordingly, the constructed training data for potential entity alignment can be used to learn entity alignment-oriented embedding. MuGNN [28] learns alignment-oriented KG embeddings by a multichannel mechanism that encodes KGs via KG completion and entity pruning. NAEA [29] merges neighborhood subgraph-level information of entities and designs a neighborhood-aware attention representation mechanism on cross-lingual KGs. RDGCN [30] proposes a relation-aware dual-graph convolutional network to leverage relations through attentive interactions between the KG and its dual relation counterpart. MRAEA [31] learns cross-lingual entity embeddings by attending over the entity’s neighbors and the meta semantics of its connecting relations.Some literature on cross-lingual entity alignments has highlighted the role of both KG structures and attributes. JAPE [27] embeds the structures of different KGs into a uniform hidden space and uses the attribute correlation of KGs to realize the refinement of entity embedding. However, attribute components can significantly degrade the performance of JAPE’s structural components when attributes are heterogeneous or have a confused association between the attributes. Graph convolutional networks (GCNs) [32] are also employed in the study [33] to learn embeddings from both the structure and attribute information of entities for cross-lingual alignment.


## 3. Problem Definition

In a KG, facts are mainly stored in two types of triples<entity, attribute, value> and <entity, relation, entity>, which are called attribute triple and relation triple, respectively. This paper denotes a KG as G=E,R,A, where E=e1,e2,…,eN is the set of entities, R=ei,ej|ei,ej∈E is the set of relations, and A=aei|ei∈E represents the set of attributes in the KG. Each attribute of an entity consists of a set of key-value pairs.Definition 1.
Cross-lingual entity alignment: letG1=E1,R1,A1 and G2=E2,R2,A2 be two arbitrary KGs in different languages. The entity pairs that refer to the same real-world object are called prealigned entities, denoted as L=ei1,ej2|ei1∈E1,ej2∈E2. The task of cross-lingual entity alignment is to find hidden aligned entity pairs M=ei1,ej2|ei1∈E1,ej2∈E2,ei1,ej2∉L based on the prealigned pairs L.


## 4. Proposed Approach

### 4.1. Overview

The details of the proposed model CAREA taken in this section are based on the cotraining algorithm. Its framework is shown in Figure1.Figure 1
The overview of CAREA framework.We construct two independent models: attribute-based modelfattr and relation-based model fstruc. The advantage of the cotraining algorithm is that it reinforces the performance of the two models in the process of iterations. Both models are retrained with the prealigned entity pairs L and predict the new pairs of potential aligned entities at each iteration. Subsequently, dynamic thresholds are used to filter the anticipated results further. The method merges the remaining entity pairs into L for the next iteration until convergence.


### 4.2. Attribute-Based Model

In our scenario, the attributes of an entity consist of a number of key-value pairs, for example,<name:Michael>, where “name” is the attribute key, and “Michael” is the attribute value. For simplicity, a key-value pair is also called an attribute.


#### 4.2.1. Attribute Extension

A critical problem of attribute representation is that some actual attributes may not be observed since they are not explicitly built or captured by the crawlers. Therefore, we first extend the attributes of both KGs by using the prealigned entity pairs. Typically, given a couple of aligned entities, if one entity has an attribute in a KG, the other KG’s corresponding entity also has this attribute. Based on such an observation, we can add a key-value pair to one entity in a KG if its counterpart in the other KG has this key-value pair. Formally, the attributes of an entityei are denoted by aei=p1,p2,…,pj,…, where pj=<keyj:valuej> is a key-value pair. For each entity ei in KG 1, its attribute aei1 can be extended to a˜ei1 by(1)a˜ei1=aei1∪pk|pk∈ej2,pk∉ei1,ei1,ej2∈L.Similarly, its counterpartaej2 in KG 2 can be extended into a˜ej2.


#### 4.2.2. Attribute Feature Representation

In multilingual KGs, the attributes are in different languages and cannot be directly compared. However, our observation shows the following:(1)
The occurrence frequencies of equivalent attribute pairs, that is, attribute keys, in multilingual KGs are approximately similar. For example, an entity representing a person in different KGs often has some equivalent attributes such as name, date of birth, and nationality. Although the texts that describe these attributes are multilingual, their frequency in different KGs is similar to the ratio of person entities to all KG entities.(2)
The values of an equivalent attribute pair in different KGs has the same data type. For example, both the English word “Michael” and the Chinese word “Mai Ke” are strings, and both “3.14” and “3.14159” are floating numbers.Hence, this study represents the attributes of an entity by its attribute key frequencies and attribute value types. The description of entity attribute features can be illustrated briefly by a concrete example shown in Figure2.Figure 2
An example of entity attributes representation: (a) attribute triples in each KG are merged into a set of key-value pairs, (b) attribute keys and values are used to represent the frequency feature and type feature, respectively, (c) the frequency and type information are integrated into a sparse matrix, and (d) each attribute pair forms a feature vector.First, the attribute triples in each KG are merged into a set of key-value pairs, where the keys and values are then used to represent the frequency and the type features, respectively. The frequencyFpj of an attribute pj is a floating number ranging from 0 to 1, calculated as follows:(2)Fpj=CountpjE,where Countpj is the occurrence number of attribute pj in a KG and E is the total number of entities in the KG. In this example, the frequencies of entity “Michael”’s nationality and birthdate are 0.2162 and 0.3351, respectively.Second, we divide the frequency range (i.e., the interval of [0, 1]) into a sequence of small real intervals. Its frequency interval number can represent an attribute. In this paper, a proportional sequence is applied to split the frequency range. The interval for an attributepj can be computed by(3)Intervalpj=log1+qFpjFmini,where q is the proportionality constant, and we fix q=0.001 in this paper. Fmini is the least frequency of occurrence of an attribute in a KG. For example, “Nationality” and “Guo Ji” in Figure 2 are both in interval 2, although their frequencies are different. The parameter setting makes the interval more robust to small changes caused by noise, especially when more different frequency attributes are merged into one interval.The value type of an attribute is its data type. Following previous work [27], this study distinguishes four kinds of data types, that is, Integer, Double, DateTime, and String. We encode the value type by a one-hot vector with the same dimension to the number of data types. For example, The codes for the attribute value “America” and “1958-08-29” are 0,0,0,1 and 0,0,1,0, respectively.As explained in the above two steps, it is clear that the primary ideal of attribute feature representation is to integrate the representations of the frequency and the type of an attributepj. We combine the two representations into a sparse matrix as shown in Figure 2. Each row in the matrix denotes the value type, and the row index is the frequency interval number. On top of that, we reshape this matrix into a row feature vector pj⟶. In this way, the attributes aei for an entity ei can be formed by the sum of its every attribute vectors pj⟶ as(4)aei⟶=∑pj∈aeipj⟶.To reduce noise, we use an indicator functionI⋅ to transform the attribute vector of an entity into the following binary representation:(5)Ix=1,ifx≠0,0,otherwise.The binary representation is averaged by the entity’s neighbors as(6)aei′⟶=1ℕie∑ek∈ℕieIaek⟶,where ℕie denotes the neighboring entities of ei. Then, a three-layer MLP transforms the attribute vectors of the two KGs into a uniform vector space, making the equivalent entities in different KGs close to each other. The MLP output is considered as the embedding of an entity, which is represented as heiattr. We use ReLU as an activation function in this paper. Batch normalization and dropout are added to increase performance. The details of the objective function are introduced in Section 4.4.


### 4.3. Relationship-Based Model

In KGs, there are various types of relations describing the role of entity associations that are crucial to aligning entities across KGs. Many previous works represent a relation by a transformation of the relationship connected entities. However, these methods bring the relationship too close to the entity [31]. Therefore, it will be difficult to capture the features of multiple relationships. Hence, this paper represents the entities and relations separately. Their combinations are adopted as the inputs to a graphical attention network (GAT) [34]. As a result, the two KGs are embedded into a unified vector space so that the equivalent entities in different KGs are close to each other. This study treats relations as undirected; that is, ei,ej is equivalent to ej,ei.The idea of GAT is to calculate each entity’s hidden representations in two KGs by focusing on their entity neighbors. GAT follows a self-attention strategy in its learning process. First, the embeddinghei∈ℝd×1 of each entity ei and its connected relation hrj∈ℝd×1 are randomly initialized. This study sets the embedding dimension of entities and relations as the same d. Second, we average the entity ei with its neighbors. Then, the entity embedding and the averaged embedding of entity connections are concatenated as the input to the GAT network as(7)heiin=1ℕie+1∑ej∈ℕie∪eihej1ℕir∑rj∈ℕirhrj,where ℕie represents ei’s neighboring entities, ℕir represents the set of relations that are outward from ei, and notation ‖ represents the concatenate operation. The attention coefficients can be calculated by(8)eij=aTheiinhejin,where eij indicates the weighted importance of neighboring ej to ei and a∈ℝ2d×1 is the shared attention weight vector.Different from the original GAT, there is no weight matrixW for the input feature in equation (8). In this study, all adjacent entities are normalized using softmax function and LeakyReLU nonlinearity with negative input slope α=0.2. Such normalization makes the coefficients between different nodes easy to compare, which can be denoted by(9)αij=Softmaxeij=expLeakyReLUeij∑k∈ℕieexpLeakyReLUeik.Nonlinear ReLU is applied to the combination of participating neighbors. The operation yields the output features of each entity:(10)heiout=ReLU∑ej∈ℕieαijhejin.The stability of the training process is prepared by adopting a multihead mechanism. Specifically,K independent heads of attention execute the transformation of equation (10). Then, the averaged features result in the following output:(11)heiout=ReLU1K∑k=1k=K∑ej∈ℕieαijkhejin,where k is the indicator of heads and αijk represents the attention coefficient in the k-th head. This study also expands the attention mechanism to multihop neighboring level information by adding more layers, thus creating a more global-aware representation of the KG. Let heiout0,…,heioutl be the output features of ei from 0-th (input features) to l-th layer. We concatenate them together to obtain the final output features heistruc of entity ei as(12)heistruc=heiout0,…,heioutl.


### 4.4. Objective Function

As was mentioned in the previous section, the two models both provide embeddings of the entities for two KGs from different views. This section uses the same objective function to optimize both of them. Following the previous work [33], Manhattan Distance is employed to be the similarity measure. The similarity of ei∈G1 and ej∈G2 in the joint vector space can be calculated by(13)sei1,ej2=heiattr/struc−hejattr/struc.All similar entities inG2 should be calculated using the same method to find the entity ei’s counterpart. The nearest one is chosen as ei’s equivalent. On top of that, we adopt the following margin-based loss function since it ensures the balance between positive and negative samples and ensures the lower scores for positive ones; that is,(14)L=∑ei,ej∈Lsei,ej−sei−ej′+γ++sei,ej−sei′−ej+γ+,where ·+ represents max·,0 and γ is a hyperparameter of margin. ei′ and ej′ are the negative counterpart of ei and ej, respectively. In this work, the entities in G1 and G2 are randomly selected as negative counterparts. Adam [35] is adopted to minimize the loss function.


### 4.5. Cotraining Algorithm

In this study, the cotraining process of the attribute-based modelfattr and relation-based model fstruc is conducted iteratively. Two components alternately take turns to train and predict new potential aligned entity pairs at each iteration until either of the two parts no longer obtain new pairs. Such a prediction is based on the cosine similarity of entities in the united vector space. A new pair sourced from a KG is suggested by searching the nearest neighbor (NN) in the other KG. It is worth noting that, in most cases, the NNs are asymmetric. For example, although ei in G1 is the most similar entity to ej in G2, there may be another entity in G(2) closer to ei. Thus, the newly predicted entity pairs should be bidirectional nearest neighbors.


#### 4.5.1. Dynamic Similarity Threshold

We further evaluate the predicted potential aligned entity pairs by dynamically adjusting the threshold in each iteration. That is to say, only the entity pairs, whose cosine similarity falls within a certain thresholdτ, are populated into the aligned pair set L. As higher similarity threshold implies higher precision, we set higher thresholds for earlier iterations. However, it may also limit the capability of the model to propose a sufficient number of aligned entity pairs. Thus, lower thresholds are taken for later iterations. The design of the threshold function is alternative. In this paper, we design a linear threshold function as(15)τn=α−δn,where α∈0,1 is the initial threshold, n∈0,1,2,… is the iteration number, and δ∈0,1 is the coefficient controlling the changing rate of each iteration. In order to control the precision of each component model, we set different threshold parameters for different models in our experiments. The detailed cotraining procedure of CAREA is given in Algorithm 1.Algorithm 1: Cotraining of CAREA.
Input: Two KGs to be aligned G1 and G2, a set L of prealigned entity pairs, and the parameters α1,α2,δ1,δ2 of threshold function.Output: The parameters of fattr and fstruc.(1)
Initial iteration numbern=0;(2)
repeat(3)
Reinitializefattr and fstruc;(4)
Trainfstruc based on L;(5)
S←NNfstruc,G1,G2∩NNfstruc,G2,G1;(6)
τ1=α1−δ1∗n(7)
S′=ei,ej|similarityei,ej<τ1,ei,ej∈S;(8)
L←L∪S(9)
Trainfattr based on L;(10)
S←NNfattr,G1,G2∩NNfattr,G2,G1;(11)
τ2=α2−δ2∗n;(12)
S′=ei,ej|similarityei,ej<τ2,ei,ej∈S;(13)
L←L∪S(14)
n←n+1(15)
untilS′=ϕ;


## 4.1. Overview

The details of the proposed model CAREA taken in this section are based on the cotraining algorithm. Its framework is shown in Figure1.Figure 1
The overview of CAREA framework.We construct two independent models: attribute-based modelfattr and relation-based model fstruc. The advantage of the cotraining algorithm is that it reinforces the performance of the two models in the process of iterations. Both models are retrained with the prealigned entity pairs L and predict the new pairs of potential aligned entities at each iteration. Subsequently, dynamic thresholds are used to filter the anticipated results further. The method merges the remaining entity pairs into L for the next iteration until convergence.


## 4.2. Attribute-Based Model

In our scenario, the attributes of an entity consist of a number of key-value pairs, for example,<name:Michael>, where “name” is the attribute key, and “Michael” is the attribute value. For simplicity, a key-value pair is also called an attribute.


### 4.2.1. Attribute Extension

A critical problem of attribute representation is that some actual attributes may not be observed since they are not explicitly built or captured by the crawlers. Therefore, we first extend the attributes of both KGs by using the prealigned entity pairs. Typically, given a couple of aligned entities, if one entity has an attribute in a KG, the other KG’s corresponding entity also has this attribute. Based on such an observation, we can add a key-value pair to one entity in a KG if its counterpart in the other KG has this key-value pair. Formally, the attributes of an entityei are denoted by aei=p1,p2,…,pj,…, where pj=<keyj:valuej> is a key-value pair. For each entity ei in KG 1, its attribute aei1 can be extended to a˜ei1 by(1)a˜ei1=aei1∪pk|pk∈ej2,pk∉ei1,ei1,ej2∈L.Similarly, its counterpartaej2 in KG 2 can be extended into a˜ej2.


### 4.2.2. Attribute Feature Representation

In multilingual KGs, the attributes are in different languages and cannot be directly compared. However, our observation shows the following:(1)
The occurrence frequencies of equivalent attribute pairs, that is, attribute keys, in multilingual KGs are approximately similar. For example, an entity representing a person in different KGs often has some equivalent attributes such as name, date of birth, and nationality. Although the texts that describe these attributes are multilingual, their frequency in different KGs is similar to the ratio of person entities to all KG entities.(2)
The values of an equivalent attribute pair in different KGs has the same data type. For example, both the English word “Michael” and the Chinese word “Mai Ke” are strings, and both “3.14” and “3.14159” are floating numbers.Hence, this study represents the attributes of an entity by its attribute key frequencies and attribute value types. The description of entity attribute features can be illustrated briefly by a concrete example shown in Figure2.Figure 2
An example of entity attributes representation: (a) attribute triples in each KG are merged into a set of key-value pairs, (b) attribute keys and values are used to represent the frequency feature and type feature, respectively, (c) the frequency and type information are integrated into a sparse matrix, and (d) each attribute pair forms a feature vector.First, the attribute triples in each KG are merged into a set of key-value pairs, where the keys and values are then used to represent the frequency and the type features, respectively. The frequencyFpj of an attribute pj is a floating number ranging from 0 to 1, calculated as follows:(2)Fpj=CountpjE,where Countpj is the occurrence number of attribute pj in a KG and E is the total number of entities in the KG. In this example, the frequencies of entity “Michael”’s nationality and birthdate are 0.2162 and 0.3351, respectively.Second, we divide the frequency range (i.e., the interval of [0, 1]) into a sequence of small real intervals. Its frequency interval number can represent an attribute. In this paper, a proportional sequence is applied to split the frequency range. The interval for an attributepj can be computed by(3)Intervalpj=log1+qFpjFmini,where q is the proportionality constant, and we fix q=0.001 in this paper. Fmini is the least frequency of occurrence of an attribute in a KG. For example, “Nationality” and “Guo Ji” in Figure 2 are both in interval 2, although their frequencies are different. The parameter setting makes the interval more robust to small changes caused by noise, especially when more different frequency attributes are merged into one interval.The value type of an attribute is its data type. Following previous work [27], this study distinguishes four kinds of data types, that is, Integer, Double, DateTime, and String. We encode the value type by a one-hot vector with the same dimension to the number of data types. For example, The codes for the attribute value “America” and “1958-08-29” are 0,0,0,1 and 0,0,1,0, respectively.As explained in the above two steps, it is clear that the primary ideal of attribute feature representation is to integrate the representations of the frequency and the type of an attributepj. We combine the two representations into a sparse matrix as shown in Figure 2. Each row in the matrix denotes the value type, and the row index is the frequency interval number. On top of that, we reshape this matrix into a row feature vector pj⟶. In this way, the attributes aei for an entity ei can be formed by the sum of its every attribute vectors pj⟶ as(4)aei⟶=∑pj∈aeipj⟶.To reduce noise, we use an indicator functionI⋅ to transform the attribute vector of an entity into the following binary representation:(5)Ix=1,ifx≠0,0,otherwise.The binary representation is averaged by the entity’s neighbors as(6)aei′⟶=1ℕie∑ek∈ℕieIaek⟶,where ℕie denotes the neighboring entities of ei. Then, a three-layer MLP transforms the attribute vectors of the two KGs into a uniform vector space, making the equivalent entities in different KGs close to each other. The MLP output is considered as the embedding of an entity, which is represented as heiattr. We use ReLU as an activation function in this paper. Batch normalization and dropout are added to increase performance. The details of the objective function are introduced in Section 4.4.


## 4.2.1. Attribute Extension

A critical problem of attribute representation is that some actual attributes may not be observed since they are not explicitly built or captured by the crawlers. Therefore, we first extend the attributes of both KGs by using the prealigned entity pairs. Typically, given a couple of aligned entities, if one entity has an attribute in a KG, the other KG’s corresponding entity also has this attribute. Based on such an observation, we can add a key-value pair to one entity in a KG if its counterpart in the other KG has this key-value pair. Formally, the attributes of an entityei are denoted by aei=p1,p2,…,pj,…, where pj=<keyj:valuej> is a key-value pair. For each entity ei in KG 1, its attribute aei1 can be extended to a˜ei1 by(1)a˜ei1=aei1∪pk|pk∈ej2,pk∉ei1,ei1,ej2∈L.Similarly, its counterpartaej2 in KG 2 can be extended into a˜ej2.


## 4.2.2. Attribute Feature Representation

In multilingual KGs, the attributes are in different languages and cannot be directly compared. However, our observation shows the following:(1)
The occurrence frequencies of equivalent attribute pairs, that is, attribute keys, in multilingual KGs are approximately similar. For example, an entity representing a person in different KGs often has some equivalent attributes such as name, date of birth, and nationality. Although the texts that describe these attributes are multilingual, their frequency in different KGs is similar to the ratio of person entities to all KG entities.(2)
The values of an equivalent attribute pair in different KGs has the same data type. For example, both the English word “Michael” and the Chinese word “Mai Ke” are strings, and both “3.14” and “3.14159” are floating numbers.Hence, this study represents the attributes of an entity by its attribute key frequencies and attribute value types. The description of entity attribute features can be illustrated briefly by a concrete example shown in Figure2.Figure 2
An example of entity attributes representation: (a) attribute triples in each KG are merged into a set of key-value pairs, (b) attribute keys and values are used to represent the frequency feature and type feature, respectively, (c) the frequency and type information are integrated into a sparse matrix, and (d) each attribute pair forms a feature vector.First, the attribute triples in each KG are merged into a set of key-value pairs, where the keys and values are then used to represent the frequency and the type features, respectively. The frequencyFpj of an attribute pj is a floating number ranging from 0 to 1, calculated as follows:(2)Fpj=CountpjE,where Countpj is the occurrence number of attribute pj in a KG and E is the total number of entities in the KG. In this example, the frequencies of entity “Michael”’s nationality and birthdate are 0.2162 and 0.3351, respectively.Second, we divide the frequency range (i.e., the interval of [0, 1]) into a sequence of small real intervals. Its frequency interval number can represent an attribute. In this paper, a proportional sequence is applied to split the frequency range. The interval for an attributepj can be computed by(3)Intervalpj=log1+qFpjFmini,where q is the proportionality constant, and we fix q=0.001 in this paper. Fmini is the least frequency of occurrence of an attribute in a KG. For example, “Nationality” and “Guo Ji” in Figure 2 are both in interval 2, although their frequencies are different. The parameter setting makes the interval more robust to small changes caused by noise, especially when more different frequency attributes are merged into one interval.The value type of an attribute is its data type. Following previous work [27], this study distinguishes four kinds of data types, that is, Integer, Double, DateTime, and String. We encode the value type by a one-hot vector with the same dimension to the number of data types. For example, The codes for the attribute value “America” and “1958-08-29” are 0,0,0,1 and 0,0,1,0, respectively.As explained in the above two steps, it is clear that the primary ideal of attribute feature representation is to integrate the representations of the frequency and the type of an attributepj. We combine the two representations into a sparse matrix as shown in Figure 2. Each row in the matrix denotes the value type, and the row index is the frequency interval number. On top of that, we reshape this matrix into a row feature vector pj⟶. In this way, the attributes aei for an entity ei can be formed by the sum of its every attribute vectors pj⟶ as(4)aei⟶=∑pj∈aeipj⟶.To reduce noise, we use an indicator functionI⋅ to transform the attribute vector of an entity into the following binary representation:(5)Ix=1,ifx≠0,0,otherwise.The binary representation is averaged by the entity’s neighbors as(6)aei′⟶=1ℕie∑ek∈ℕieIaek⟶,where ℕie denotes the neighboring entities of ei. Then, a three-layer MLP transforms the attribute vectors of the two KGs into a uniform vector space, making the equivalent entities in different KGs close to each other. The MLP output is considered as the embedding of an entity, which is represented as heiattr. We use ReLU as an activation function in this paper. Batch normalization and dropout are added to increase performance. The details of the objective function are introduced in Section 4.4.


## 4.3. Relationship-Based Model

In KGs, there are various types of relations describing the role of entity associations that are crucial to aligning entities across KGs. Many previous works represent a relation by a transformation of the relationship connected entities. However, these methods bring the relationship too close to the entity [31]. Therefore, it will be difficult to capture the features of multiple relationships. Hence, this paper represents the entities and relations separately. Their combinations are adopted as the inputs to a graphical attention network (GAT) [34]. As a result, the two KGs are embedded into a unified vector space so that the equivalent entities in different KGs are close to each other. This study treats relations as undirected; that is, ei,ej is equivalent to ej,ei.The idea of GAT is to calculate each entity’s hidden representations in two KGs by focusing on their entity neighbors. GAT follows a self-attention strategy in its learning process. First, the embeddinghei∈ℝd×1 of each entity ei and its connected relation hrj∈ℝd×1 are randomly initialized. This study sets the embedding dimension of entities and relations as the same d. Second, we average the entity ei with its neighbors. Then, the entity embedding and the averaged embedding of entity connections are concatenated as the input to the GAT network as(7)heiin=1ℕie+1∑ej∈ℕie∪eihej1ℕir∑rj∈ℕirhrj,where ℕie represents ei’s neighboring entities, ℕir represents the set of relations that are outward from ei, and notation ‖ represents the concatenate operation. The attention coefficients can be calculated by(8)eij=aTheiinhejin,where eij indicates the weighted importance of neighboring ej to ei and a∈ℝ2d×1 is the shared attention weight vector.Different from the original GAT, there is no weight matrixW for the input feature in equation (8). In this study, all adjacent entities are normalized using softmax function and LeakyReLU nonlinearity with negative input slope α=0.2. Such normalization makes the coefficients between different nodes easy to compare, which can be denoted by(9)αij=Softmaxeij=expLeakyReLUeij∑k∈ℕieexpLeakyReLUeik.Nonlinear ReLU is applied to the combination of participating neighbors. The operation yields the output features of each entity:(10)heiout=ReLU∑ej∈ℕieαijhejin.The stability of the training process is prepared by adopting a multihead mechanism. Specifically,K independent heads of attention execute the transformation of equation (10). Then, the averaged features result in the following output:(11)heiout=ReLU1K∑k=1k=K∑ej∈ℕieαijkhejin,where k is the indicator of heads and αijk represents the attention coefficient in the k-th head. This study also expands the attention mechanism to multihop neighboring level information by adding more layers, thus creating a more global-aware representation of the KG. Let heiout0,…,heioutl be the output features of ei from 0-th (input features) to l-th layer. We concatenate them together to obtain the final output features heistruc of entity ei as(12)heistruc=heiout0,…,heioutl.


## 4.4. Objective Function

As was mentioned in the previous section, the two models both provide embeddings of the entities for two KGs from different views. This section uses the same objective function to optimize both of them. Following the previous work [33], Manhattan Distance is employed to be the similarity measure. The similarity of ei∈G1 and ej∈G2 in the joint vector space can be calculated by(13)sei1,ej2=heiattr/struc−hejattr/struc.All similar entities inG2 should be calculated using the same method to find the entity ei’s counterpart. The nearest one is chosen as ei’s equivalent. On top of that, we adopt the following margin-based loss function since it ensures the balance between positive and negative samples and ensures the lower scores for positive ones; that is,(14)L=∑ei,ej∈Lsei,ej−sei−ej′+γ++sei,ej−sei′−ej+γ+,where ·+ represents max·,0 and γ is a hyperparameter of margin. ei′ and ej′ are the negative counterpart of ei and ej, respectively. In this work, the entities in G1 and G2 are randomly selected as negative counterparts. Adam [35] is adopted to minimize the loss function.


## 4.5. Cotraining Algorithm

In this study, the cotraining process of the attribute-based modelfattr and relation-based model fstruc is conducted iteratively. Two components alternately take turns to train and predict new potential aligned entity pairs at each iteration until either of the two parts no longer obtain new pairs. Such a prediction is based on the cosine similarity of entities in the united vector space. A new pair sourced from a KG is suggested by searching the nearest neighbor (NN) in the other KG. It is worth noting that, in most cases, the NNs are asymmetric. For example, although ei in G1 is the most similar entity to ej in G2, there may be another entity in G(2) closer to ei. Thus, the newly predicted entity pairs should be bidirectional nearest neighbors.


### 4.5.1. Dynamic Similarity Threshold

We further evaluate the predicted potential aligned entity pairs by dynamically adjusting the threshold in each iteration. That is to say, only the entity pairs, whose cosine similarity falls within a certain thresholdτ, are populated into the aligned pair set L. As higher similarity threshold implies higher precision, we set higher thresholds for earlier iterations. However, it may also limit the capability of the model to propose a sufficient number of aligned entity pairs. Thus, lower thresholds are taken for later iterations. The design of the threshold function is alternative. In this paper, we design a linear threshold function as(15)τn=α−δn,where α∈0,1 is the initial threshold, n∈0,1,2,… is the iteration number, and δ∈0,1 is the coefficient controlling the changing rate of each iteration. In order to control the precision of each component model, we set different threshold parameters for different models in our experiments. The detailed cotraining procedure of CAREA is given in Algorithm 1.Algorithm 1: Cotraining of CAREA.
Input: Two KGs to be aligned G1 and G2, a set L of prealigned entity pairs, and the parameters α1,α2,δ1,δ2 of threshold function.Output: The parameters of fattr and fstruc.(1)
Initial iteration numbern=0;(2)
repeat(3)
Reinitializefattr and fstruc;(4)
Trainfstruc based on L;(5)
S←NNfstruc,G1,G2∩NNfstruc,G2,G1;(6)
τ1=α1−δ1∗n(7)
S′=ei,ej|similarityei,ej<τ1,ei,ej∈S;(8)
L←L∪S(9)
Trainfattr based on L;(10)
S←NNfattr,G1,G2∩NNfattr,G2,G1;(11)
τ2=α2−δ2∗n;(12)
S′=ei,ej|similarityei,ej<τ2,ei,ej∈S;(13)
L←L∪S(14)
n←n+1(15)
untilS′=ϕ;


## 4.5.1. Dynamic Similarity Threshold

We further evaluate the predicted potential aligned entity pairs by dynamically adjusting the threshold in each iteration. That is to say, only the entity pairs, whose cosine similarity falls within a certain thresholdτ, are populated into the aligned pair set L. As higher similarity threshold implies higher precision, we set higher thresholds for earlier iterations. However, it may also limit the capability of the model to propose a sufficient number of aligned entity pairs. Thus, lower thresholds are taken for later iterations. The design of the threshold function is alternative. In this paper, we design a linear threshold function as(15)τn=α−δn,where α∈0,1 is the initial threshold, n∈0,1,2,… is the iteration number, and δ∈0,1 is the coefficient controlling the changing rate of each iteration. In order to control the precision of each component model, we set different threshold parameters for different models in our experiments. The detailed cotraining procedure of CAREA is given in Algorithm 1.Algorithm 1: Cotraining of CAREA.
Input: Two KGs to be aligned G1 and G2, a set L of prealigned entity pairs, and the parameters α1,α2,δ1,δ2 of threshold function.Output: The parameters of fattr and fstruc.(1)
Initial iteration numbern=0;(2)
repeat(3)
Reinitializefattr and fstruc;(4)
Trainfstruc based on L;(5)
S←NNfstruc,G1,G2∩NNfstruc,G2,G1;(6)
τ1=α1−δ1∗n(7)
S′=ei,ej|similarityei,ej<τ1,ei,ej∈S;(8)
L←L∪S(9)
Trainfattr based on L;(10)
S←NNfattr,G1,G2∩NNfattr,G2,G1;(11)
τ2=α2−δ2∗n;(12)
S′=ei,ej|similarityei,ej<τ2,ei,ej∈S;(13)
L←L∪S(14)
n←n+1(15)
untilS′=ϕ;


## 5. Experiments

### 5.1. Datasets

This section applies a popular public dataset DBP-15K [27] for entity alignment to evaluate the performance of the approach CAREA. DBP-15 K contains three cross-lingual subsets built from DBpedia: DBPZH−EN, DBPJA−EN, and DBPFR−EN. Each of the three subsets contains two KGs in different languages, for example, DBPFR−EN for French and English. Their statistics are displayed in Table 1.Table 1
The statistics of the datasets used in the experiment.
Datasets#Entities#RelationsAttributesAttr. triplesRel. triplesDBPZH−ENChinese19,3882,8308,113153,929379,684English19,5722,3177,173237,674567,755DBPJA−ENJapanese19,8142,0435,882164,373354,619English19,7802,0966,066233,319497,230DBPFR−ENFrench19,6611,3794,547192,191528,665English19,9932,2096,422278,590576,543


### 5.2. Experiment Settings

Following the previous work [29], we adopt two evaluation metrics: (1) Hits@k: the proportion of correctly aligned entities ranked in the top k. (2) Mean Reciprocal Rank (MRR): the average of the reciprocal ranks of results. Higher Hits@k and MRR scores indicate better alignment performance. The two metrics can be calculated as follows:(16)scorex=1,if posx≤k,0,otherwise,Hitsx=1T∑x∈Tscorex,MRRx=1T∑x∈T1posx,where pos· is the position of tested entity pair in the returned list, T is the set of tested entity pairs, and Hits@1 and Hits@10 are adopted in our experiments.This research is compared with other baselines by the same evaluation metrics and the dataset’s splitting method. The experiment randomly splits 30% of the prealigned entity pairs as training data, while the rest 70% for testing. The average score of both alignment directions (e.g., ZH⟶EN vs. EN⟶ZH) is reported by considering the asymmetric of the nearest relation across KGs. Each experiment instance is run five times independently. Their average performances are considered as the final results. The same settings for experiment models are applied unless otherwise stated. The hidden dimensions for attributes, entities, and relations are the same: d=100. The margin parameter γ, dropout rate, and learning rate of Adam are 3, 0.3, and 0.005, respectively. For the relation-based model fstruc, we fix the number of attention heads K=2 and GAT layer’s depth l=2. For the cotraining process, we take the results of CAREA’s third iteration as its final performance. On the other hand, the threshold coefficients for both model components are the same as δ=0.05. The initial thresholds α for fattr and astruc are empirically set to 0.95 and 0.9, respectively.


### 5.3. Baselines

To demonstrate the advantage of our method, we compared it with the following baselines:(i)
MtransE [24]: MtransE is a structure-based model for multilingual KG embeddings, to provide a simple and automated solution. The model characterizes monolingual relations and deploys three different techniques to represent cross-lingual transitions, namely, axis calibration, translation vectors, and linear transformations.(ii)
JAPE [27]: JAPE is an attribute-preserving embedding model that incorporates the relation and attribute embeddings for entity alignment.(iii)
GCN-Align [33]: GCN-Align employs GCNs to learn embeddings from both the structure and attribute information of entities for cross-lingual KG alignment.(iv)
BootEA [26]: BootEA adopts a bootstrapping strategy, which iteratively labels potential entity alignments as training data and leverages it for learning alignment-oriented embeddings.(v)
MuGNN [28]: MuGNN learns alignment-oriented KG embeddings by robustly encoding two KGs via KG completion and entities pruning.(vi)
NAEA [29]: NAEA incorporates neighborhood subgraph-level information of entities and designs a neighborhood-aware attentional representation mechanism on multilingual KGs.The performances of the above baselines come from the reported results in their papers. We also evaluate the effectiveness of the component models of our approach, including the following:(i)
Attribute-based model: the model, denoted as CAREA-a, ignores structure embedding components to assess the effect of the attribute embedding. In other words, the attribute features are only used to align entities without a cotraining strategy.(ii)
Structure-based model: we also estimate the component’s performance of network structure embedding, which ignores the attribute features except for the structure ones to align entities. The model is similarly denoted as CAREA-s.


### 5.4. Experiment Results

#### 5.4.1. Overall Performance

All the comparison methods can be treated under two groups according to feature categories. One is purely based on KG structures, including MTransE, BootEA, MuGNN, NAEA, and CAREA-s. The other leverages both entity attributes and relations for entity alignment, including JAPE, GCN-Align, and CAREA. Table2 summarizes the overall results of all compared methods on the three datasets.Table 2
Comparison with the baseline methods.
MethodDBPZH−ENDBPJA−ENDBPFR−ENHits@1Hits@10MRRHits@1Hits@10MRRHits@1Hits@10MRRMTransE30.8%61.4%0.36427.9%57.5%0.34924.4%55.6%0.335JAPE41.2%74.5%0.49036.3%68.5%0.47632.4%66.7%0.430GCN-Align41.3%74.4%0.54939.9%74.5%0.54637.3%74.5%0.532MuGNN49.4%84.4%0.61150.1%85.7%0.62149.5%87.0%0.621BootEA62.9%84.8%0.70362.2%85.4%0.70165.3%87.4%0.731NAEA65.0%86.7%0.72064.1%87.3%0.71867.3%89.4%0.752CAREA-a22.1%51.8%0.32018.4%46.4%0.27716.9%45.4%0.264CAREA-s58.2%87.5%0.68558.5%87.8%0.68960.5%90.5%0.711CAREA69.8%90.6%0.77268.9%90.4%0.76671.3%92.5%0.789In the structure-based group, our model CAREA-s performs better than MTransE by at least 27.3% in terms ofHits@1 on three datasets, which also resulted in a greater score than MuGNN by at least 8.4%. The comparison results have demonstrated the effectiveness of our structure-based approach. Further tests in other group revealed that CAREA outperforms JAPE and GCN-Align by at least 28.6% across all datasets on Hits@1. The result demonstrates our approach’s superiority by leveraging both entity attributes and KG structures for entity alignment. Finally, CAREA ranked the best over all the competing approaches across all datasets. For example, the performance of CAREA is better than NAEA and BootEA by at least 3.9% and 6.0% in terms of Hits@1, respectively.Another proposed component model CAREA-a is excluded from the above comparisons since it is the only one that relies solely on attribute information to align entities. It achieves lower scores. For example, when CAREA-a were stimulated withHits@1 and Hits@10 on DBPZH−EN, lower scores of 22.1% and 51.8% were reported. This is mainly because of the heterogeneity across multilingual KGs or the KGs may not be explicitly constructed or captured by the crawlers. Although CAREA-a does not perform as well as that of structure-based approaches, it provides another view for entity alignment and improves the performance of our method in the KG alignment task.


#### 5.4.2. Effects of Cotraining Algorithm

This part confirms the achievements of the approach CAREA by presenting each iteration of the cotraining process. The result is shown in Figure3. Its trend reveals that there has been a gradually similar increase in terms of all evaluation metrics, which is verified on all the three datasets of our two-component models. The iterative cotraining algorithm significantly improves the performance, with at least an absolute increase of 10.5% Hits@1 across all experiment datasets. Both the attribute fattr model and the structure model fstruc get enhanced from each iteration. After 3 to 4 iterations, the component model performance becomes stable.Figure 3
Hits@1 performances of the two-component models in each iteration. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)


#### 5.4.3. Parameter Sensitivity Analysis

This part investigates the parameter sensitivity of the proposed CAREA on three primary parameters: (1) the proportion of the prealigned entity pairs, (2) the feature dimensiond, and (3) the margin parameter γ of the proposed objective function.Sensitivity to data proportions. We run CAREA by the training proportions from 10% to 50% with a step size of 10%. Figure4 illustrates the change of Hits@k concerning different proportions. The shown results on all the datasets become better with the proportion increase following our expectations. The amount of data is a significant factor that more training data can provide more extended information to overlay the cross-lingual KGs. Figure 4 shows that CAREA performed encouragingly when using only 10% of the aligned entities as training data. For example, Hits@1 and Hits@10 on DBPZH−ENare 56.2% and 82.4%, respectively. Therefore, CAREA is expected to be well adapted to annotate constrained scenarios.Sensitivity to the feature dimensiond: Figure 5 depicts the sensitivity of the model performance on different feature dimensions. The model performance of CAREA keeps stable with the data dimension increases in all datasets. It is proved that a high-dimensional feature space is helpful to preserve entity information and improve the entity alignment performance. A larger dimension necessarily consumes more computing resources. Hence, we chose d=100 to weigh efficiency against effectiveness.Sensitivity to the marginγ: the model performance produced by setting different margin parameter γ (from 1 to 4) of the objective function is shown in Figure 6. The performances become steady when γ≥2 with at most 2.5% range from all datasets. Therefore, CAREA can keep stable when γ varies within a reasonable range.Figure 4
Hits@k on entity alignment with respect to the proportion of training data. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)Figure 5
CAREA performance with respect to the feature dimensiond. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)Figure 6
CAREA performance with respect to the margin parameterγ. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)


## 5.1. Datasets

This section applies a popular public dataset DBP-15K [27] for entity alignment to evaluate the performance of the approach CAREA. DBP-15 K contains three cross-lingual subsets built from DBpedia: DBPZH−EN, DBPJA−EN, and DBPFR−EN. Each of the three subsets contains two KGs in different languages, for example, DBPFR−EN for French and English. Their statistics are displayed in Table 1.Table 1
The statistics of the datasets used in the experiment.
Datasets#Entities#RelationsAttributesAttr. triplesRel. triplesDBPZH−ENChinese19,3882,8308,113153,929379,684English19,5722,3177,173237,674567,755DBPJA−ENJapanese19,8142,0435,882164,373354,619English19,7802,0966,066233,319497,230DBPFR−ENFrench19,6611,3794,547192,191528,665English19,9932,2096,422278,590576,543


## 5.2. Experiment Settings

Following the previous work [29], we adopt two evaluation metrics: (1) Hits@k: the proportion of correctly aligned entities ranked in the top k. (2) Mean Reciprocal Rank (MRR): the average of the reciprocal ranks of results. Higher Hits@k and MRR scores indicate better alignment performance. The two metrics can be calculated as follows:(16)scorex=1,if posx≤k,0,otherwise,Hitsx=1T∑x∈Tscorex,MRRx=1T∑x∈T1posx,where pos· is the position of tested entity pair in the returned list, T is the set of tested entity pairs, and Hits@1 and Hits@10 are adopted in our experiments.This research is compared with other baselines by the same evaluation metrics and the dataset’s splitting method. The experiment randomly splits 30% of the prealigned entity pairs as training data, while the rest 70% for testing. The average score of both alignment directions (e.g., ZH⟶EN vs. EN⟶ZH) is reported by considering the asymmetric of the nearest relation across KGs. Each experiment instance is run five times independently. Their average performances are considered as the final results. The same settings for experiment models are applied unless otherwise stated. The hidden dimensions for attributes, entities, and relations are the same: d=100. The margin parameter γ, dropout rate, and learning rate of Adam are 3, 0.3, and 0.005, respectively. For the relation-based model fstruc, we fix the number of attention heads K=2 and GAT layer’s depth l=2. For the cotraining process, we take the results of CAREA’s third iteration as its final performance. On the other hand, the threshold coefficients for both model components are the same as δ=0.05. The initial thresholds α for fattr and astruc are empirically set to 0.95 and 0.9, respectively.


## 5.3. Baselines

To demonstrate the advantage of our method, we compared it with the following baselines:(i)
MtransE [24]: MtransE is a structure-based model for multilingual KG embeddings, to provide a simple and automated solution. The model characterizes monolingual relations and deploys three different techniques to represent cross-lingual transitions, namely, axis calibration, translation vectors, and linear transformations.(ii)
JAPE [27]: JAPE is an attribute-preserving embedding model that incorporates the relation and attribute embeddings for entity alignment.(iii)
GCN-Align [33]: GCN-Align employs GCNs to learn embeddings from both the structure and attribute information of entities for cross-lingual KG alignment.(iv)
BootEA [26]: BootEA adopts a bootstrapping strategy, which iteratively labels potential entity alignments as training data and leverages it for learning alignment-oriented embeddings.(v)
MuGNN [28]: MuGNN learns alignment-oriented KG embeddings by robustly encoding two KGs via KG completion and entities pruning.(vi)
NAEA [29]: NAEA incorporates neighborhood subgraph-level information of entities and designs a neighborhood-aware attentional representation mechanism on multilingual KGs.The performances of the above baselines come from the reported results in their papers. We also evaluate the effectiveness of the component models of our approach, including the following:(i)
Attribute-based model: the model, denoted as CAREA-a, ignores structure embedding components to assess the effect of the attribute embedding. In other words, the attribute features are only used to align entities without a cotraining strategy.(ii)
Structure-based model: we also estimate the component’s performance of network structure embedding, which ignores the attribute features except for the structure ones to align entities. The model is similarly denoted as CAREA-s.


## 5.4. Experiment Results

### 5.4.1. Overall Performance

All the comparison methods can be treated under two groups according to feature categories. One is purely based on KG structures, including MTransE, BootEA, MuGNN, NAEA, and CAREA-s. The other leverages both entity attributes and relations for entity alignment, including JAPE, GCN-Align, and CAREA. Table2 summarizes the overall results of all compared methods on the three datasets.Table 2
Comparison with the baseline methods.
MethodDBPZH−ENDBPJA−ENDBPFR−ENHits@1Hits@10MRRHits@1Hits@10MRRHits@1Hits@10MRRMTransE30.8%61.4%0.36427.9%57.5%0.34924.4%55.6%0.335JAPE41.2%74.5%0.49036.3%68.5%0.47632.4%66.7%0.430GCN-Align41.3%74.4%0.54939.9%74.5%0.54637.3%74.5%0.532MuGNN49.4%84.4%0.61150.1%85.7%0.62149.5%87.0%0.621BootEA62.9%84.8%0.70362.2%85.4%0.70165.3%87.4%0.731NAEA65.0%86.7%0.72064.1%87.3%0.71867.3%89.4%0.752CAREA-a22.1%51.8%0.32018.4%46.4%0.27716.9%45.4%0.264CAREA-s58.2%87.5%0.68558.5%87.8%0.68960.5%90.5%0.711CAREA69.8%90.6%0.77268.9%90.4%0.76671.3%92.5%0.789In the structure-based group, our model CAREA-s performs better than MTransE by at least 27.3% in terms ofHits@1 on three datasets, which also resulted in a greater score than MuGNN by at least 8.4%. The comparison results have demonstrated the effectiveness of our structure-based approach. Further tests in other group revealed that CAREA outperforms JAPE and GCN-Align by at least 28.6% across all datasets on Hits@1. The result demonstrates our approach’s superiority by leveraging both entity attributes and KG structures for entity alignment. Finally, CAREA ranked the best over all the competing approaches across all datasets. For example, the performance of CAREA is better than NAEA and BootEA by at least 3.9% and 6.0% in terms of Hits@1, respectively.Another proposed component model CAREA-a is excluded from the above comparisons since it is the only one that relies solely on attribute information to align entities. It achieves lower scores. For example, when CAREA-a were stimulated withHits@1 and Hits@10 on DBPZH−EN, lower scores of 22.1% and 51.8% were reported. This is mainly because of the heterogeneity across multilingual KGs or the KGs may not be explicitly constructed or captured by the crawlers. Although CAREA-a does not perform as well as that of structure-based approaches, it provides another view for entity alignment and improves the performance of our method in the KG alignment task.


### 5.4.2. Effects of Cotraining Algorithm

This part confirms the achievements of the approach CAREA by presenting each iteration of the cotraining process. The result is shown in Figure3. Its trend reveals that there has been a gradually similar increase in terms of all evaluation metrics, which is verified on all the three datasets of our two-component models. The iterative cotraining algorithm significantly improves the performance, with at least an absolute increase of 10.5% Hits@1 across all experiment datasets. Both the attribute fattr model and the structure model fstruc get enhanced from each iteration. After 3 to 4 iterations, the component model performance becomes stable.Figure 3
Hits@1 performances of the two-component models in each iteration. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)


### 5.4.3. Parameter Sensitivity Analysis

This part investigates the parameter sensitivity of the proposed CAREA on three primary parameters: (1) the proportion of the prealigned entity pairs, (2) the feature dimensiond, and (3) the margin parameter γ of the proposed objective function.Sensitivity to data proportions. We run CAREA by the training proportions from 10% to 50% with a step size of 10%. Figure4 illustrates the change of Hits@k concerning different proportions. The shown results on all the datasets become better with the proportion increase following our expectations. The amount of data is a significant factor that more training data can provide more extended information to overlay the cross-lingual KGs. Figure 4 shows that CAREA performed encouragingly when using only 10% of the aligned entities as training data. For example, Hits@1 and Hits@10 on DBPZH−ENare 56.2% and 82.4%, respectively. Therefore, CAREA is expected to be well adapted to annotate constrained scenarios.Sensitivity to the feature dimensiond: Figure 5 depicts the sensitivity of the model performance on different feature dimensions. The model performance of CAREA keeps stable with the data dimension increases in all datasets. It is proved that a high-dimensional feature space is helpful to preserve entity information and improve the entity alignment performance. A larger dimension necessarily consumes more computing resources. Hence, we chose d=100 to weigh efficiency against effectiveness.Sensitivity to the marginγ: the model performance produced by setting different margin parameter γ (from 1 to 4) of the objective function is shown in Figure 6. The performances become steady when γ≥2 with at most 2.5% range from all datasets. Therefore, CAREA can keep stable when γ varies within a reasonable range.Figure 4
Hits@k on entity alignment with respect to the proportion of training data. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)Figure 5
CAREA performance with respect to the feature dimensiond. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)Figure 6
CAREA performance with respect to the margin parameterγ. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)


## 5.4.1. Overall Performance

All the comparison methods can be treated under two groups according to feature categories. One is purely based on KG structures, including MTransE, BootEA, MuGNN, NAEA, and CAREA-s. The other leverages both entity attributes and relations for entity alignment, including JAPE, GCN-Align, and CAREA. Table2 summarizes the overall results of all compared methods on the three datasets.Table 2
Comparison with the baseline methods.
MethodDBPZH−ENDBPJA−ENDBPFR−ENHits@1Hits@10MRRHits@1Hits@10MRRHits@1Hits@10MRRMTransE30.8%61.4%0.36427.9%57.5%0.34924.4%55.6%0.335JAPE41.2%74.5%0.49036.3%68.5%0.47632.4%66.7%0.430GCN-Align41.3%74.4%0.54939.9%74.5%0.54637.3%74.5%0.532MuGNN49.4%84.4%0.61150.1%85.7%0.62149.5%87.0%0.621BootEA62.9%84.8%0.70362.2%85.4%0.70165.3%87.4%0.731NAEA65.0%86.7%0.72064.1%87.3%0.71867.3%89.4%0.752CAREA-a22.1%51.8%0.32018.4%46.4%0.27716.9%45.4%0.264CAREA-s58.2%87.5%0.68558.5%87.8%0.68960.5%90.5%0.711CAREA69.8%90.6%0.77268.9%90.4%0.76671.3%92.5%0.789In the structure-based group, our model CAREA-s performs better than MTransE by at least 27.3% in terms ofHits@1 on three datasets, which also resulted in a greater score than MuGNN by at least 8.4%. The comparison results have demonstrated the effectiveness of our structure-based approach. Further tests in other group revealed that CAREA outperforms JAPE and GCN-Align by at least 28.6% across all datasets on Hits@1. The result demonstrates our approach’s superiority by leveraging both entity attributes and KG structures for entity alignment. Finally, CAREA ranked the best over all the competing approaches across all datasets. For example, the performance of CAREA is better than NAEA and BootEA by at least 3.9% and 6.0% in terms of Hits@1, respectively.Another proposed component model CAREA-a is excluded from the above comparisons since it is the only one that relies solely on attribute information to align entities. It achieves lower scores. For example, when CAREA-a were stimulated withHits@1 and Hits@10 on DBPZH−EN, lower scores of 22.1% and 51.8% were reported. This is mainly because of the heterogeneity across multilingual KGs or the KGs may not be explicitly constructed or captured by the crawlers. Although CAREA-a does not perform as well as that of structure-based approaches, it provides another view for entity alignment and improves the performance of our method in the KG alignment task.


## 5.4.2. Effects of Cotraining Algorithm

This part confirms the achievements of the approach CAREA by presenting each iteration of the cotraining process. The result is shown in Figure3. Its trend reveals that there has been a gradually similar increase in terms of all evaluation metrics, which is verified on all the three datasets of our two-component models. The iterative cotraining algorithm significantly improves the performance, with at least an absolute increase of 10.5% Hits@1 across all experiment datasets. Both the attribute fattr model and the structure model fstruc get enhanced from each iteration. After 3 to 4 iterations, the component model performance becomes stable.Figure 3
Hits@1 performances of the two-component models in each iteration. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)


## 5.4.3. Parameter Sensitivity Analysis

This part investigates the parameter sensitivity of the proposed CAREA on three primary parameters: (1) the proportion of the prealigned entity pairs, (2) the feature dimensiond, and (3) the margin parameter γ of the proposed objective function.Sensitivity to data proportions. We run CAREA by the training proportions from 10% to 50% with a step size of 10%. Figure4 illustrates the change of Hits@k concerning different proportions. The shown results on all the datasets become better with the proportion increase following our expectations. The amount of data is a significant factor that more training data can provide more extended information to overlay the cross-lingual KGs. Figure 4 shows that CAREA performed encouragingly when using only 10% of the aligned entities as training data. For example, Hits@1 and Hits@10 on DBPZH−ENare 56.2% and 82.4%, respectively. Therefore, CAREA is expected to be well adapted to annotate constrained scenarios.Sensitivity to the feature dimensiond: Figure 5 depicts the sensitivity of the model performance on different feature dimensions. The model performance of CAREA keeps stable with the data dimension increases in all datasets. It is proved that a high-dimensional feature space is helpful to preserve entity information and improve the entity alignment performance. A larger dimension necessarily consumes more computing resources. Hence, we chose d=100 to weigh efficiency against effectiveness.Sensitivity to the marginγ: the model performance produced by setting different margin parameter γ (from 1 to 4) of the objective function is shown in Figure 6. The performances become steady when γ≥2 with at most 2.5% range from all datasets. Therefore, CAREA can keep stable when γ varies within a reasonable range.Figure 4
Hits@k on entity alignment with respect to the proportion of training data. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)Figure 5
CAREA performance with respect to the feature dimensiond. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)Figure 6
CAREA performance with respect to the margin parameterγ. (a) DBPZH-EN. (b) DBPJA-EN. (c) DBPFR-EN.
(a)(b)(c)


## 6. Conclusion and Future Work

The purpose of the present research was to investigate the cross-lingual entity alignment problem in KGs. This study has constructed a cotraining based approach CAREA to learn entity embeddings from two independent views of knowledge (relationships and attributes). CAREA is innovatively constructed as a two-component modelfattr and fstruc, which can extract the attribute and the relation information, respectively. In each iteration, both models alternately take turns of the train-and-predict process, which gradually improves each model’s performance. Experiments on three popular datasets confirm the effectiveness and superiority of CAREA on the entity alignment task. The insights of model construction gained from this study may be of assistance to complex multilanguage and cross-domain knowledge organization and analysis. Future work seeks to extend the CAREA method to other applications, such as link prediction, information extraction, and entity classification.



---

*Source: 6831603-2020-12-08.xml*