Towards Principled Representation Learning for Entity Alignment

Knowledge graph ( KG ) representation learning for entity alignment has recently received great attention . Compared with conventional methods , these embeddingbased ones are considered to be robuster for highly-heterogeneous and cross-lingual entity alignment scenarios as they do not rely on the quality of machine translation or feature extraction . Despite the significant improvement that has been made , there is little understanding of how the embedding-based entity alignment methods actually work . Most existing methods rest on the foundation that a small number of pre-aligned entities can serve as anchors to connect the embedding spaces of two KGs . But no one investigates the rationality of such foundation . In this paper , we define a typical paradigm abstracted from the existing methods , and analyze how the representation discrepancy between two potentially-aligned entities is implicitly bounded by a predefined margin in the scoring function for embedding learning . However , such a margin can not guarantee to be tight enough for alignment learning . We mitigate this problem by proposing a new approach that explicitly learns KG-invariant and principled entity representations , meanwhile preserves the original infrastructure of existing methods . In this sense , the model not only pursues the closeness of aligned entities on geometric distance , but also aligns the neural ontologies of two KGs to eliminate the discrepancy in feature distribution and underlying ontology knowledge . Our experiments demonstrate consistent and significant improvement in performance against the existing embedding-based entity alignment methods , including several state-of-the-art ones . 1 INTRODUCTION . Knowledge Graphs ( KGs ) , such as DBpedia ( Auer et al. , 2007 ) and Wikidata ( Vrandečić & Krötzsch , 2014 ) , have become crucial data resources for many AI applications . Although a large-scale KG offers structured knowledge derived from millions of facts in the real world , it is still incomplete by nature , and the downstream applications are always demanding for more knowledge . To resolve this issue , the task of entity alignment ( EA ) is proposed , which exploits the potentially-aligned entities among different KGs to facilitate knowledge fusion and exchange . Recently , embedding-based entity alignment ( EEA ) methods ( Chen et al. , 2017 ; Zhu et al. , 2017 ; Wang et al. , 2018 ; Guo et al. , 2019 ; Ye et al. , 2019 ; Wu et al. , 2019 ; Sun et al. , 2020a ; Fey et al. , 2020 ) have been prevailing in this area . Their common idea is to encode semantics into embeddings and estimate the similarities by embedding distance . During this process , a small number of aligned entity pairs ( a.k.a. , seed alignment ) are required as supervision data to align ( or merge ) the embedding spaces of KGs . These methods either learn an alignment function fa to minimize the difference between two entity embeddings in each seed ( Wang et al. , 2018 ) , or directly map aligned entities to one embedding vector ( Sun et al. , 2017 ) . Meanwhile , they also leverage a shared scoring function fs to encode semantics into representations , such that two underlying aligned entities that connect to respective sides of a seed shall have similar characteristics in their feature expression . Although the effectiveness of current EEA methods are empirically demonstrated ( Sun et al. , 2020b ) , little efforts have been made on the theoretical analysis . In this paper , we fill this gap by formally defining a paradigm leveraged by the current methods . We show that the representation discrepancy of an underlying aligned entity pair is bounded in an indirect way by a margin λ in the scoring function fs . Unfortunately , we further find that this margin-based bound can not be set as tight as expected , causing that little constrain can be put on the entities with few neighbors . To mitigate the above problem , we propose neural ontology driven entity alignment ( abbr. , NeoEA ) , in which the entity representations are optimized jointly with a neural ontology . An ontology ( Baader et al. , 2005 ) is usually comprised of axioms that define the legitimate relationships among entities and relations . Those axioms make a KG principled ( i.e. , constrained by rules ) . For example , an “ Object Property Domain ” axiom in OWL2 ( Baader et al. , 2005 ) claims the valid head entities for a specific relation ( e.g. , the head entities of relation “ birthPlace ” should be in class “ Person ” ) , and it thus determines the head entity distributions of this relation . The neural ontology in this paper , however , is reversely deduced from the entity distributions . We expect to align the high-level neural ontology to diminish the discrepancy of feature distributions , as well as ontology knowledge , between two KGs . The main contributions of this paper are threefold : • We define the paradigm of the current EEA methods , and demonstrate that the embedding discrepancy in each potential alignment pair is implicitly bounded by the margin in the scoring function . We show that this bound can not be as tight as we expect . • We propose NeoEA to learn KG-invariant as well as principled representations by aligning the neural axioms of two KGs . We prove that minimizing the difference can substantially align their corresponding ontology-level knowledge , without the assumption about the existence of real ontology data . • We conducted experiments to verify the effectiveness of NeoEA with several state-of-the-art methods as baselines . The results show that NeoEA can consistently and significantly improve the performance of the EEA methods . 2 EMBEDDING-BASED ENTITY ALIGNMENT . 2.1 METHODOLOGY . We first summarize the common paradigm employed by most existing EEA methods ( Chen et al. , 2017 ; Sun et al. , 2017 ; Zhu et al. , 2017 ; Sun et al. , 2018 ; Wang et al. , 2018 ; Pei et al. , 2019a ; Guo et al. , 2019 ; Wu et al. , 2019 ; Ye et al. , 2019 ; Sun et al. , 2020a ) : Definition 1 ( Embedding-based Entity Alignment ) . The input of EEA is two KGs G1 = ( E1 , R1 , T1 ) , G2 = ( E2 , R2 , T2 ) , and a small subset of aligned entity pairs S ⊂ E1 × E2 as seeds to connect G1 with G2 . An EEA model consists of two neural functions : an alignment function fa , which is used to regularize the embeddings of pairwise entities in S ; and a scoring function fs , which scores the representations based on the joint triple set T1∪T2 . EEA estimates the alignment score of an arbitrary entity pair ( e1i , e 2 j ) by their geometric distance d ( e 1 i , e 2 j ) , where e 1 i , e 2 j denote the embeddings of e 1 i , e2i respectively . It is worth noting that , the existing EEA methods have different settings in relation seed alignment . Some works ( Chen et al. , 2017 ; Zhu et al. , 2017 ) assume that all aligned relation pairs are known in advance . Others ( Sun et al. , 2017 ; 2018 ) suppose that the number of relations is much smaller than that of entities , i.e. , |R| |E| , which means that the training data for aligning relations is sufficient . In this paper , we do not explore the details of relation seed setting . We assume that the relation representations for a well-trained EEA model are aligned . The existing works have explored the diversity of fa . The pioneering work MTransE ( Chen et al. , 2017 ) proposed to learn a mapping matrix to cast an entity representation e1i to the feature space of G2 . SEA ( Pei et al. , 2019a ) and OTEA ( Pei et al. , 2019b ) extended this approach by leveraging adversarial training to learn the projection matrix . Recently , a simpler yet more efficient choice was widely-used , which directly maps ( e1i , e 2 i ) ∈ S to one embedding vector ei ( Sun et al. , 2017 ; Zhu et al. , 2017 ; Trsedya et al. , 2019 ; Guo et al. , 2019 ) . Also , researchers ( Wang et al. , 2018 ; Pei et al. , 2019a ; Wu et al. , 2019 ) started to leverage a softer way to incorporate seed information , in which the distance between entities in a positive pair ( i.e. , supervised data in S ) is minimized , while that referred to the negative one will be enlarged . As the most common choice , we consider fa as Euclidean distance between two embeddings , such that the corresponding alignment loss can be written as follows : La = ∑ ( e1i , e 2 i ) ∈S ||e1i − e2i ||+ ∑ ( e1 i′ , e 2 j′ ) ∈S − ReLU ( α− ||e1i′ − e2j′ || ) , ( 1 ) where S− denotes the sampled set of negative pairs . α is the minimal margin allowed between entities in each negative entity pair . On the other hand , the scoring function fs can be also designed diversely . Most methods ( Chen et al. , 2017 ; Sun et al. , 2017 ; Pei et al. , 2019a ) choose TransE as their scoring function , i.e. , fs ( ei , r , ej ) = ||ei + r− ej || , ( ei , r , ej ) ∈ T1 ∪ T2 . The corresponding loss is : Ls = ∑ τ∈T1∪T2 ReLU ( fs ( τ ) − λ ) + ∑ τ ′∈T −1 ∪T − 2 ReLU ( λ− fs ( τ ′ ) ) , ( 2 ) where T −1 and T − 2 are negative triple sets . Ls is a margin-based loss in which the distance d ( ei+r , ej ) in a positive triple should at least be smaller than λ ≥ 0 , while larger than λ for negative ones . Note that , the negative triples are usually generated by randomly replacing the head or tail entity of a positive triple . If we only look at the replaced entity , minimizing the above loss can be also understood as randomly pushing entities away from this entity . This phenomena has also been studied in Wang & Isola ( 2020 ) . Additionally , some graph neural network ( GNN ) based methods ( Wang et al. , 2018 ; Sun et al. , 2020a ; Wu et al. , 2019 ) do not directly optimize fs . They encode the semantic information inside graph convolution . Therefore , the output vectors of GNN will be regarded as entity embeddings to feed into fa . For example , Ye et al . ( 2019 ) ; Wu et al . ( 2019 ) ; Sun et al . ( 2020a ) leverage TransE as scoring function in the aggregation of relational neighbors . 2.2 UNDERSTANDING EEA . We illustarte how an EEA model works by an example . Let ( e1x , e 2 y ) ∈ G1 × G2 be a potentiallyaligned entity pair . Each entity in this pair has only one neighbor , connected by the same relation r1 = r2 . We assume that their neighbors are actually a pair of entities ( e1i , e 2 i ) ∈ S . Therefore , if an EEA model is well-trained and almost optimal , we should have e1i = e 2 i ( as La is minimized ) and r1 = r2 ( denoted by r for simplicity ) . According to Equation 2 , we have : ||fs ( e1x , r , e1i ) || ≈ ||fs ( e2y , r , e2i ) || ≤ λ . ( 3 ) Take the scoring function of TransE as fs , we then derive : ||e1x + r− e1i || ≤ λ , ||e2y + r− e2i || ≤ λ . ( 4 ) As e1i = e 2 i , we can conclude that : Proposition 1 ( Discrepancy Bound ) . The representation difference of two potentially-aligned entities is bound by , which is proportional to the hyper-parameter λ : ||e1x − e2y|| ≤ ∝ λ . ( 5 ) The above proposition suggests that decreasing the value of margin λ will tighten the feature discrepancy of entities in the underlying aligned entity pairs . However , we soon find that λ can not be set as small as we want . We consider a more complicated yet realistic example , where each entity in ( e1x , e 2 y ) has a considerable number of neighbors . We denote the corresponding triple sets of e1x , e 2 y as T 1ex , T 2 ey , respectively . In this setting , a well-trained EEA model should satisfy that : ∀τ ∈ T 1ex ∪ T 2 ey , ||fs ( τ ) || ≤ λ . ( 6 ) Evidently , TransE with a small margin is not sufficient to fully express the semantics contained in T 1ex ∪ T 2 ey , which has already been explored by previous works ( Trouillon et al. , 2016 ; Kazemi & Poole , 2018 ; Sun et al. , 2019 ) . Some empirical statistics ( Sun et al. , 2018 ) also illustrate such results . However , enlarging the margin λ will bring significant variance between e1x and e 2 y . On the other hand , if the scoring function does not belong to the TransE family , e.g. , it is neural-based like ConvE ( Dettmers et al. , 2018 ) or composition-based like ComplEx ( Trouillon et al. , 2016 ) , both of which are fully expressive ( Kazemi & Poole , 2018 ) . In this case , entities with a large number of neighbors can be correctly modeled , while those with only few neighbors are less constrained . Therefore , those models allow even more diversity between e1x and e 2 y . We believe this is why they performed badly in EA task ( Guo et al. , 2019 ; Sun et al. , 2020b ) . In short , most existing works adopt the above implicit strategy to learn cross-KG representations for EA , which makes them struggled in balancing between the bound and the expressiveness . In this paper , we explore a new direction to explicitly align the feature distributions of two KGs , which ensures the embeddings tight and expressive .

Summary: Entity alignment plays an important role in improving the quality of cross-lingual knowledge graphs. As one of the most important solutions, embedding-based methods aim at learning a semantic space where the unique entity cross knowledge graphs can have the closest distance. Most of research focus on entity-level granular, but discard the whole picture of embedding space of cross-lingual KGs. Besides the aligned entity pairs as the labelled data, this paper extended the labelled data with the conditional neural and basic axioms, which are actually sets of randomly selected entities or entities with the same relation type. Then the final objective is to align the cross-lingual knowledge graphs by both optimizing the distance of labelled entity pairs and neural axioms.