Multilingual Knowledge Graph Completion With Joint Relation and Entity Alignment

We study Multilingual KGC for KGs that associate entities and relations with surface forms from different languages . In such a setting , an entity ( or relation ) may be mentioned as separate IDs in different KGs , necessitating entity alignment ( EA ) and relation alignment ( RA ) . In addition to EA and RA being important subtasks for Multilingual KGC , we posit that high confidence fact predictions may also , in turn , add valuable information for alignment tasks , and vice versa . In response , we present the novel task of jointly training multilingual KGC , EA and RA models . Our approach , ALIGNKGC , uses mBERT-based surface form overlap for EA , and combines it with a KGC approach , extended to the multiple KG setting via a loss term that incentivizes RA . On experiments with DBPedia in five languages , we find that ALIGNKGC achieves up to 17 % absolute MRR improvements in KGC compared to a strong completion model that combines known facts in all languages . It also outperforms an mBERT-only alignment baseline for EA , underscoring the value of joint training for these tasks . 1 . Introduction . A knowledge graph ( KG ) has nodes representing entities and directed edges representing relations between subject and object entities . An entity has a unique node ( ID ) . Relations also have canonical labels such as born-in or works-at , with associated IDs . A KG is usually associated with a single human language — an entity node ( or relation ID ) is associated with one or more surface forms in that language . E.g. , the ID for the country USA may have aliases like “ United States of America '' . KGs are usually very incomplete , as curators struggle to keep up with the real world . KG completion ( KGC ) is thus a strongly motivated problem and studies the prediction of true facts unknown to the KG . While the problem has been intensively researched over the last few years [ Bordes et al. , 2013 , Trouillon et al. , 2016 , Jain et al. , 2018 , Sun et al. , 2019 ] , most KGC research is applicable to only one KG in one language at a time . However , different language speakers would maintain separate KGs in their own languages . Independent completions of each KG may not be optimal , since information from one KG will likely help completion of the other . Each KG will give a different ID , and often also a different surface form , to the same entity , such as “ Estados Unidos de América ” for the USA entity in a Spanish KG . This leads to the problem of ID proliferation . Recent work on entity alignment ( EA ) across KGs attempts to assign a unique ID to all nodes representing the same entity [ Chen et al. , 2017 , Sun et al. , 2017 , 2018 , 2020 , Cao et al. , 2019 , Chen et al. , 2021 , Tang et al. , 2020 ] . A related but under-explored task is relation alignment ( RA ) — assigning synonymous relations in different KGs the same relation ID . We note that RA involves global evidence , because the decision to merge two relations in two languages can have far-reaching consequences to many facts in both KGs . Our key contribution is to recognize and exploit the synergy between multilingual KGC , EA and RA tasks . Entity and relation alignments expose a KGC algorithm to more facts , which can lead to better completion . Conversely , a high-confidence completion can give additional evidence to align its constituent entities and relation . ( See Appendix A for an extended example . ) In this paper , we present ALIGNKGC , a multi-task system that learns to optimize for KGC , EA and RA jointly . ALIGNKGC uses state-of-the-art mBERT-based models for EA and combines it with an extension of the state-of-the-art ComplEx model [ Trouillon et al. , 2016 ] for single KGC . ALIGNKGC uses a novel subject-object signature of each relation , which it represents as a bag of embeddings , and compares them for equivalence and implication between relations . ALIGNKGC uses this information as additional term in its loss function for better RA and multilingual KGC performance . We evaluate ALIGNKGC on slices of DBPedia in five languages . We compare it against two single-KGC systems , one trained over each monolingual KG separately , and the other trained over a union of all KG facts . Compared to these strong baselines , we find that ALIGNKGC achieves substantial accuracy boost due to better entity and relation alignment , obtaining up to 17 % absolute MRR improvements in KGC across languages , around 20 % absolute HITS @ 1 gain for RA for rare relations , around 26 % absolute HITS @ 1 gain for RA for more frequent relations , and around 61 % absolute HITS @ 1 gain for EA . ALIGNKGC also achieves a 5.3 % absolute HITS @ 1 gain over an even stronger EA baseline that uses mBERT embeddings on entity names . These results underscore the value of joint training for these tasks . 2 . Notation and preliminaries . A knowledge graph ( KG ) provides a graph-view to a knowledge-base ( KB ) . A KG consists of entities E and relations ( aka relation types ) R. A KG instance is a triple ( s , r , o ) where s , o ∈ E and r ∈ R. These are all canonical IDs . Each ID is associated with one or more textual aliases . KGC task : For any single KG , training data is provided as ( s , r , o ) triples . A test instance has the form ( s , r , ? ) or ( ? , r , o ) where the system has to predict o or s. Multiple correct values are possible . The evaluation protocol takes a system-ranked list of candidate objects or subjects , and then measures MRR or hits @ K. Many KG embedding and KGC algorithms have been proposed in the last few years . ComplEx [ Trouillon et al. , 2016 ] with all negative instances ( no sampling ) is a simple model that gives near state-of-the-art predictions [ Jain et al. , 2020a ] . It has also formed the basis for many KGC extensions ( e.g. , [ Jain et al. , 2020b ] ) . So we use it as our baseline KGC gadget . ComplEx defines a triple score as f ( s , r , o ) = < ( 〈s , r , o ? 〉 ) , where c ? is complex conjugate , 〈· · · 〉 is a 3-way elementwise inner product and < ( · ) is the real part of a complex number . Using f , ComplEx defines Pr ( o|s , r ) = ef ( s , r , o ) /∑ o′ e f ( s , r , o′ ) , Pr ( s|o , r ) = ef ( s , r , o ) /∑ s′ e f ( s′ , r , o ) ( 1 ) and the log-likelihood KGC loss as LKGC = ∑ ( s , r , o ) ∈KG− log Pr ( o|s , r ) − log Pr ( s|o , r ) . Alignment tasks : We consider a set L = { l1 , l2 , . . . } of languages . KGl represents the KG supported by the language l. An entity in this KG is called el . A relation in this KG is called rl . While trying to improve KGC for target language l′ , KGs in multiple source languages l may be combined . An equivalence between entities el and el′ in KGs of two languages l and l′ is specified as el ≡ el′ , also written as the triple ( el , ≡ , el′ ) . Similarly , an equivalence between relations rl and rl′ is specified as rl ≡ rl′ . Other types of alignment may be possible between relation pairs , such as rl ⇒ rl′ , which means , for all s , o such that ( s , rl , o ) holds , so does ( s , rl′ , o ) . During training , a set of entity equivalences { ( enl , ≡ , enl′ ) : n = 1 , . . . , N } and a set of relation equivalences { ( rml , ≡ , rml′ ) : m = 1 , . . . , M } are revealed to the system between each KGl and KGl′ . The goal of the system is to infer additional entity and relation equivalences . The system is usually called upon to produce a ranked list of equivalences , which is evaluated using HITS @ K or MRR . To achieve KGC enhanced with alignment , the system has to infer additional triples in each KG , making best use of the revealed equivalences , while also inferring additional alignment triples between entities or relations across KGs of different languages . 3 . Proposed methods . 3.1 KGC baselines . When ComplEx is applied to any one KG in isolation , we call this method KGCMONO . Perhaps the most straight-forward way to apply a KGC system across multiple KGs is to compute KGU =⋃ l∈L KGl such that for all revealed entity equivalences { ( el , ≡ , el′ ) } , collapse their node pairs , and rename all equivalent rl , rl′ relation IDs to a common new ID . A standard KGC system can work on the resulting KGU , and we call this method KGCUNION . 3.2 Joint alignment and completion . For both textual and structured inputs , the problem of inferring a predicate or relation as entailed by another has been well-studied [ Lin and Pantel , 2001 , Bhagat et al. , 2007 , Nakashole et al. , 2012 ] . With that work as our point of departure , we ask : How similar are two relations rl , rl′ in KGU ? Comparing their embedding vectors rl , rl′ , as computed by a KGC algorithm such as ComplEx , gives an estimate . However , when training these embeddings , we found that a simple union of facts ( KGCMONO ) did not yield the best results . Instead , guiding the training through an auxiliary loss , computed using the following explicit relation signatures helped the model learn better embeddings . Here s , o are interpreted as canonical IDs ( hence , ‘ hard ’ ) . This will be generalized shortly . Definition 1 ( Hard SO-signature ) . The ( ‘ hard ’ ) subject-object ( SO ) signature of a relation is defined as SO ( r ) = { ( s , o ) : ( s , r , o ) ∈ KGU } . 3.2.1 HARD SO-OVERLAP & JACCARD SIMILARITY . Definition 2 ( SO-overlap ) . The ( hard ) overlap between two relations rl , rl′ is |SO ( rl ) ∩ SO ( rl′ ) | . Jaccard similarity can then be used as a standard symmetric belief that two relations IDs in different languages are equivalent : bJ ( rl ⇔ rl′ ) = |SO ( rl ) ∩ SO ( rl′ ) | |SO ( rl ) ∪ SO ( rl′ ) | . ( 2 ) If bJ ( rl ⇔ rl′ ) exceeds a threshold θ ( tuned using dev set ) , we add ( rl , rl′ ) to set AJ of ‘ silver ’ alignments . We define an additional RA loss term encouraging such embeddings to be aligned : LRA-J = ∑ ( rl , rl′ ) ∈AJ bJ ( rl ⇔ rl′ ) ∥∥rl − rl′∥∥1 . ( 3 ) We call this scheme Jaccard .

Summary: This paper introduces AlignKGC, a neural model that performs knowledge graph completion (KGC) on multilingual knowledge graphs that have entity and relation overlap. AlignKGC makes use of some of this overlap during training to optimize its parameters for two auxiliary tasks, entity alignment (EA) and relation alignment (RA). The resulting model demonstrates strong empirical results on DBP5L for all three tasks, and the paper demonstrates through extensive experiments the efficacy of each model components introduced.