Towards Principled Representation Learning for Entity Alignment

Knowledge graph ( KG ) representation learning for entity alignment has recently received great attention . Compared with conventional methods , these embeddingbased ones are considered to be robuster for highly-heterogeneous and cross-lingual entity alignment scenarios as they do not rely on the quality of machine translation or feature extraction . Despite the significant improvement that has been made , there is little understanding of how the embedding-based entity alignment methods actually work . Most existing methods rest on the foundation that a small number of pre-aligned entities can serve as anchors to connect the embedding spaces of two KGs . But no one investigates the rationality of such foundation . In this paper , we define a typical paradigm abstracted from the existing methods , and analyze how the representation discrepancy between two potentially-aligned entities is implicitly bounded by a predefined margin in the scoring function for embedding learning . However , such a margin can not guarantee to be tight enough for alignment learning . We mitigate this problem by proposing a new approach that explicitly learns KG-invariant and principled entity representations , meanwhile preserves the original infrastructure of existing methods . In this sense , the model not only pursues the closeness of aligned entities on geometric distance , but also aligns the neural ontologies of two KGs to eliminate the discrepancy in feature distribution and underlying ontology knowledge . Our experiments demonstrate consistent and significant improvement in performance against the existing embedding-based entity alignment methods , including several state-of-the-art ones . 1 INTRODUCTION . Knowledge Graphs ( KGs ) , such as DBpedia ( Auer et al. , 2007 ) and Wikidata ( Vrandečić & Krötzsch , 2014 ) , have become crucial data resources for many AI applications . Although a large-scale KG offers structured knowledge derived from millions of facts in the real world , it is still incomplete by nature , and the downstream applications are always demanding for more knowledge . To resolve this issue , the task of entity alignment ( EA ) is proposed , which exploits the potentially-aligned entities among different KGs to facilitate knowledge fusion and exchange . Recently , embedding-based entity alignment ( EEA ) methods ( Chen et al. , 2017 ; Zhu et al. , 2017 ; Wang et al. , 2018 ; Guo et al. , 2019 ; Ye et al. , 2019 ; Wu et al. , 2019 ; Sun et al. , 2020a ; Fey et al. , 2020 ) have been prevailing in this area . Their common idea is to encode semantics into embeddings and estimate the similarities by embedding distance . During this process , a small number of aligned entity pairs ( a.k.a. , seed alignment ) are required as supervision data to align ( or merge ) the embedding spaces of KGs . These methods either learn an alignment function fa to minimize the difference between two entity embeddings in each seed ( Wang et al. , 2018 ) , or directly map aligned entities to one embedding vector ( Sun et al. , 2017 ) . Meanwhile , they also leverage a shared scoring function fs to encode semantics into representations , such that two underlying aligned entities that connect to respective sides of a seed shall have similar characteristics in their feature expression . Although the effectiveness of current EEA methods are empirically demonstrated ( Sun et al. , 2020b ) , little efforts have been made on the theoretical analysis . In this paper , we fill this gap by formally defining a paradigm leveraged by the current methods . We show that the representation discrepancy of an underlying aligned entity pair is bounded in an indirect way by a margin λ in the scoring function fs . Unfortunately , we further find that this margin-based bound can not be set as tight as expected , causing that little constrain can be put on the entities with few neighbors . To mitigate the above problem , we propose neural ontology driven entity alignment ( abbr. , NeoEA ) , in which the entity representations are optimized jointly with a neural ontology . An ontology ( Baader et al. , 2005 ) is usually comprised of axioms that define the legitimate relationships among entities and relations . Those axioms make a KG principled ( i.e. , constrained by rules ) . For example , an “ Object Property Domain ” axiom in OWL2 ( Baader et al. , 2005 ) claims the valid head entities for a specific relation ( e.g. , the head entities of relation “ birthPlace ” should be in class “ Person ” ) , and it thus determines the head entity distributions of this relation . The neural ontology in this paper , however , is reversely deduced from the entity distributions . We expect to align the high-level neural ontology to diminish the discrepancy of feature distributions , as well as ontology knowledge , between two KGs . The main contributions of this paper are threefold : • We define the paradigm of the current EEA methods , and demonstrate that the embedding discrepancy in each potential alignment pair is implicitly bounded by the margin in the scoring function . We show that this bound can not be as tight as we expect . • We propose NeoEA to learn KG-invariant as well as principled representations by aligning the neural axioms of two KGs . We prove that minimizing the difference can substantially align their corresponding ontology-level knowledge , without the assumption about the existence of real ontology data . • We conducted experiments to verify the effectiveness of NeoEA with several state-of-the-art methods as baselines . The results show that NeoEA can consistently and significantly improve the performance of the EEA methods . 2 EMBEDDING-BASED ENTITY ALIGNMENT . 2.1 METHODOLOGY . We first summarize the common paradigm employed by most existing EEA methods ( Chen et al. , 2017 ; Sun et al. , 2017 ; Zhu et al. , 2017 ; Sun et al. , 2018 ; Wang et al. , 2018 ; Pei et al. , 2019a ; Guo et al. , 2019 ; Wu et al. , 2019 ; Ye et al. , 2019 ; Sun et al. , 2020a ) : Definition 1 ( Embedding-based Entity Alignment ) . The input of EEA is two KGs G1 = ( E1 , R1 , T1 ) , G2 = ( E2 , R2 , T2 ) , and a small subset of aligned entity pairs S ⊂ E1 × E2 as seeds to connect G1 with G2 . An EEA model consists of two neural functions : an alignment function fa , which is used to regularize the embeddings of pairwise entities in S ; and a scoring function fs , which scores the representations based on the joint triple set T1∪T2 . EEA estimates the alignment score of an arbitrary entity pair ( e1i , e 2 j ) by their geometric distance d ( e 1 i , e 2 j ) , where e 1 i , e 2 j denote the embeddings of e 1 i , e2i respectively . It is worth noting that , the existing EEA methods have different settings in relation seed alignment . Some works ( Chen et al. , 2017 ; Zhu et al. , 2017 ) assume that all aligned relation pairs are known in advance . Others ( Sun et al. , 2017 ; 2018 ) suppose that the number of relations is much smaller than that of entities , i.e. , |R| |E| , which means that the training data for aligning relations is sufficient . In this paper , we do not explore the details of relation seed setting . We assume that the relation representations for a well-trained EEA model are aligned . The existing works have explored the diversity of fa . The pioneering work MTransE ( Chen et al. , 2017 ) proposed to learn a mapping matrix to cast an entity representation e1i to the feature space of G2 . SEA ( Pei et al. , 2019a ) and OTEA ( Pei et al. , 2019b ) extended this approach by leveraging adversarial training to learn the projection matrix . Recently , a simpler yet more efficient choice was widely-used , which directly maps ( e1i , e 2 i ) ∈ S to one embedding vector ei ( Sun et al. , 2017 ; Zhu et al. , 2017 ; Trsedya et al. , 2019 ; Guo et al. , 2019 ) . Also , researchers ( Wang et al. , 2018 ; Pei et al. , 2019a ; Wu et al. , 2019 ) started to leverage a softer way to incorporate seed information , in which the distance between entities in a positive pair ( i.e. , supervised data in S ) is minimized , while that referred to the negative one will be enlarged . As the most common choice , we consider fa as Euclidean distance between two embeddings , such that the corresponding alignment loss can be written as follows : La = ∑ ( e1i , e 2 i ) ∈S ||e1i − e2i ||+ ∑ ( e1 i′ , e 2 j′ ) ∈S − ReLU ( α− ||e1i′ − e2j′ || ) , ( 1 ) where S− denotes the sampled set of negative pairs . α is the minimal margin allowed between entities in each negative entity pair . On the other hand , the scoring function fs can be also designed diversely . Most methods ( Chen et al. , 2017 ; Sun et al. , 2017 ; Pei et al. , 2019a ) choose TransE as their scoring function , i.e. , fs ( ei , r , ej ) = ||ei + r− ej || , ( ei , r , ej ) ∈ T1 ∪ T2 . The corresponding loss is : Ls = ∑ τ∈T1∪T2 ReLU ( fs ( τ ) − λ ) + ∑ τ ′∈T −1 ∪T − 2 ReLU ( λ− fs ( τ ′ ) ) , ( 2 ) where T −1 and T − 2 are negative triple sets . Ls is a margin-based loss in which the distance d ( ei+r , ej ) in a positive triple should at least be smaller than λ ≥ 0 , while larger than λ for negative ones . Note that , the negative triples are usually generated by randomly replacing the head or tail entity of a positive triple . If we only look at the replaced entity , minimizing the above loss can be also understood as randomly pushing entities away from this entity . This phenomena has also been studied in Wang & Isola ( 2020 ) . Additionally , some graph neural network ( GNN ) based methods ( Wang et al. , 2018 ; Sun et al. , 2020a ; Wu et al. , 2019 ) do not directly optimize fs . They encode the semantic information inside graph convolution . Therefore , the output vectors of GNN will be regarded as entity embeddings to feed into fa . For example , Ye et al . ( 2019 ) ; Wu et al . ( 2019 ) ; Sun et al . ( 2020a ) leverage TransE as scoring function in the aggregation of relational neighbors . 2.2 UNDERSTANDING EEA . We illustarte how an EEA model works by an example . Let ( e1x , e 2 y ) ∈ G1 × G2 be a potentiallyaligned entity pair . Each entity in this pair has only one neighbor , connected by the same relation r1 = r2 . We assume that their neighbors are actually a pair of entities ( e1i , e 2 i ) ∈ S . Therefore , if an EEA model is well-trained and almost optimal , we should have e1i = e 2 i ( as La is minimized ) and r1 = r2 ( denoted by r for simplicity ) . According to Equation 2 , we have : ||fs ( e1x , r , e1i ) || ≈ ||fs ( e2y , r , e2i ) || ≤ λ . ( 3 ) Take the scoring function of TransE as fs , we then derive : ||e1x + r− e1i || ≤ λ , ||e2y + r− e2i || ≤ λ . ( 4 ) As e1i = e 2 i , we can conclude that : Proposition 1 ( Discrepancy Bound ) . The representation difference of two potentially-aligned entities is bound by , which is proportional to the hyper-parameter λ : ||e1x − e2y|| ≤ ∝ λ . ( 5 ) The above proposition suggests that decreasing the value of margin λ will tighten the feature discrepancy of entities in the underlying aligned entity pairs . However , we soon find that λ can not be set as small as we want . We consider a more complicated yet realistic example , where each entity in ( e1x , e 2 y ) has a considerable number of neighbors . We denote the corresponding triple sets of e1x , e 2 y as T 1ex , T 2 ey , respectively . In this setting , a well-trained EEA model should satisfy that : ∀τ ∈ T 1ex ∪ T 2 ey , ||fs ( τ ) || ≤ λ . ( 6 ) Evidently , TransE with a small margin is not sufficient to fully express the semantics contained in T 1ex ∪ T 2 ey , which has already been explored by previous works ( Trouillon et al. , 2016 ; Kazemi & Poole , 2018 ; Sun et al. , 2019 ) . Some empirical statistics ( Sun et al. , 2018 ) also illustrate such results . However , enlarging the margin λ will bring significant variance between e1x and e 2 y . On the other hand , if the scoring function does not belong to the TransE family , e.g. , it is neural-based like ConvE ( Dettmers et al. , 2018 ) or composition-based like ComplEx ( Trouillon et al. , 2016 ) , both of which are fully expressive ( Kazemi & Poole , 2018 ) . In this case , entities with a large number of neighbors can be correctly modeled , while those with only few neighbors are less constrained . Therefore , those models allow even more diversity between e1x and e 2 y . We believe this is why they performed badly in EA task ( Guo et al. , 2019 ; Sun et al. , 2020b ) . In short , most existing works adopt the above implicit strategy to learn cross-KG representations for EA , which makes them struggled in balancing between the bound and the expressiveness . In this paper , we explore a new direction to explicitly align the feature distributions of two KGs , which ensures the embeddings tight and expressive .

Summary: The paper proposes NeoEA, an approach that further constrains KG embedding with ontology knowledge. The paper first tries to summarize the existing embedding-based entity alignment methods, stating that most of the methods choose TransE as scoring functions. But their embedding features are not aligned well compared to the neural-based or composition-based loss function. The paper, therefore, solves this problem by developing a new NeoEA architecture which shows that adding a KG-invariant ontology knowledge can minimize such difference. The experiment shows the new constraints can improve state-of-the-art baselines.