Towards Principled Representation Learning for Entity Alignment
Authors: Lingbing Guo
Zequn Sun
Mingyang Chen
Wei Hu
Huajun Chen
Venue:  (2021)
Keywords: Representation Learning
Knowledge Graph
Entity Alignment
Knowledge Graph Embedding

Abstract: Knowledge graph (KG) representation learning for entity alignment has recently received great attention. Compared with conventional methods, these embedding-based ones are considered to be robuster for highly-heterogeneous and cross-lingual entity alignment scenarios as they do not rely on the quality of  machine translation or feature extraction. Despite the significant improvement that has been made, there is little understanding of how the embedding-based entity alignment methods actually work. Most existing methods rest on the foundation that a small number of pre-aligned entities can serve as anchors to connect the embedding spaces of two KGs. But no one investigates the rationality of such foundation. In this paper, we define a typical paradigm abstracted from the existing methods, and analyze how the representation discrepancy between two potentially-aligned entities is implicitly bounded by a predefined margin in the scoring function for embedding learning. However, such a margin cannot guarantee to be tight enough for alignment learning. We mitigate this problem by proposing a new approach that explicitly learns KG-invariant and principled entity representations, meanwhile preserves the original infrastructure of existing methods. In this sense, the model not only pursues the closeness of aligned entities on geometric distance, but also aligns the neural ontologies of two KGs to eliminate the discrepancy in feature distribution and underlying ontology knowledge. Our experiments demonstrate consistent and significant improvement in performance against the existing embedding-based entity alignment methods, including several state-of-the-art ones.

Under review as a conference paper at ICLR 2021
TOWARDS PRINCIPLED REPRESENTATION LEARNING FOR ENTITY ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge graph (KG) representation learning for entity alignment has recently received great attention. Compared with conventional methods, these embedding- based ones are considered to be robuster for highly-heterogenenw entity alignment scenarios they do not rely on the quality of machine translation or feature extraction. Despite the significant improvement that has been made, entity alignment methods
and cross- loossllingual
there is little understanding of how the embedding actually work. Most existing methods rest on the foundation that small number of pre-aligned entities can serve as anchors to connect the embedding spaces of two KGs. But no one investigates the rationality of such foundation. In this paper, we define a typical paradigm abstracted from the existing methods, and analyze how the representation discrepancy between two potentially-alignn implicitly bounded by a predefined margin in the scoring function for embedding learning. However, such a margin cannot guarantee to be tight enough for aligmment learning. We mitigate this problem by proposing a new approach that explicitly learns KG-invariant and principled entity representations, meanwhile preserves the original infrastructure of existing methods. In this sense, the model not only pursues the closeness of aligned entities on geometric distance, but also aligns the neural ontologies of two KGs to eliminate the discrepancy in feature distribution and underlying ontology knowledge. Our experiments demonstrate consistent and significant improvement in performance against the existing embedding- -assd entity alignment methods, including several state-of-the -hethh-rt art ones.
entities is
INTRODUCTION
Knowledge Graphs (KGs), such as DBpedia (Auer et al. 2007) and Wikidata (Vrandečić & Krötzsch 2014), have become crucial data resources for many Al applications. Although a large-scale KG offers structured knowledge derived from millions of facts in the real world, it is still incomplete by nature, and the downstream applications are always demanding for more knowledge. To resolve this issue, the task of entity alignment (EA) is proposed, which exploits the potentially-alignes entities among different KGs to facilitate knowledge fusion and exchange.
Recently, embedding-
entity alignment (EEA) methods (Chen al. 2017 Zhu et al. 2017
Wang et al. 2018:: Guo et al. 2019 Ye et al. 2019 Wu et al. 2019 Sun et al. 2020a; Fey et al. 2020p have been prevailing in this area. Their common idea to encode semantics into embeddings and estimate the similarities by embedding distance. During this process, small number of aligned entity pairs (a.k.a., seed alignment) are required as supervision data to align (or merge) the embedding spaces of KGs. These methods either learn alignment function fa to minimize the difference between two entity embeddings in each seed (Wang al., (018)) or directly map aligned entities to one embedding vector (Sun et al. (017)). Mcanwhile, they also leverage shared scoring function f. to encode semantics into representations, such that two underlying aligned entities that connect to respective sides of seed shall have similar characteristies in their feature expression.
Although the effectiveness of current EEA methods are empirically demonstrated (Sun et al. 2020b), little efforts have been made on the theoretical analysis. In this paper, fill this gap by formally defining paradigm leveraged by the current methods. We show that the representation discrepancy of an underlying aligned entity pair is bounded in indirect way by margin À in the scoring

---

Under review as a conference paper at ICLR 2021
function fa. Unfortunately, we further find that this margin- nargin-based bound cannot be set tight as expected, causing that little constrain can be put on the entities with few neighbors.
To mitigate the above problem, propose neural ontology driven entity alignment (abbr., NcoEA). in which the entity representations are optimized jointly with a neural ontology. An ontology (Baader
et al. (005) is usually comprised of axioms that define the legitimate relationships among entities and relations. Those axioms make a KG principled (i.e., constrained by rules). For example, an "Object Property Domain" axiom in OWL2 (Baader al. 2005) claims the valid head entities for a specific relation (e.g.. the head entities of relation "birthPlace should be in class "Person"), )) and thus determines the head entity distributions of this relation. The neural ontology in this paper, however, is reversely deduced from the entity distributions. We expect to align the high-level neural ontology to diminish the discrepancy of feature distributions, as well as ontology knowledge, between two KGs.
The main contributions of this paper are threefold:
We define the paradigm of the current EEA methods, and demonstrate that the embedding discrepancy in each potential alignment pair is implicitly bounded by the margin in the scoring function. We show that this bound cannot be as tight as we expect.
We propose NeoEA to learn KG-in "G-invariant as well as principled representations by aligning the neural axioms of two KGs. We prove that minimizing the difference can substantially align their corresponding ontology-level knowledge, without the assumption about the existence of real ontology data.
We conducted experiments to verify the effectiveness of NeoEA with several state-of-the-art methods as baselines. The results show that NeoEA can consistently and significantly improve the performance of the EEA methods.
EMBEDDING-BASED ENTITY ALIGNMENT
2.1
METHOODLLOGY
We first summarize the common paradigm employed by most existing EEA methods
(Chen et al.
2017 Sun et al./ 2017 Zhu et al.] 2017 Sun et al. [2018] Wang et al. 2018 Pei et al. 2019a Guo ct al. 2019 Wu et al. 2019; Ye et al. 2019 Sun et (0.200))
Definition (Embedding-banl
Entity Alignment). The input of EEA is two KGs 91 = (E,, R1, Tr)
G2 = (Ez, R2, T2), and small subset of aligned entity pairs S &1 x Eg as seeds to connect G with Gz. An EEA model consists of two neural functions: an alignment function fa. which is used to regularize the embeddings of pairwise entities in S; and scoring function fs. which scores the representations based on the joint triple set TT. EEA estimates the aligument score of an arbitrary entity pair (e),e) ez)) by their geometric distance d(e), ,z), where e,, e² denote the embeddings of e e? respectively.
is worth noting that, the existing EEA methods have different settings in relation seed alignment. Some works (Chen et al., (017]] Zhu et al. 2017) assume that all aligned relation pairs are known in advance. Others (Sun et al. 2017 2018) suppose that the number of relations is much smaller than that of entities, i.e., [ << ET which means that the training data for aligning relations sufficient. In this paper, we do not explore the details of relation seed setting. We assume that the relation representations for a well-trained EEA model are aligned.
The existing works have explored the diversity of fa. The pioneering work MTransE (Chen et al. 2017) proposed to learn a mapping matrix cast an entity representation of to the feature space of G2. SEA (Pei et al.) 2019a) and OTEA (Pei et al. 2019b) extended this approach by leveraging adversarial training to learn the projection matrix. Recently, a simpler yet more efficient choice
was widely-used, which directly maps (e) e ) E S to one embedding vector e; SSun et al. 2017
Zhu et al. [2017] Trsedya et al. 2019 Guo et al. 2019). Also, researchers (Wang et al. 2018 Pei et al. 2019a; Wu et al. 2019) started leverage softer way to incorporate seed information, in which the distance between entities in a positive pair (i.e., supervised data in S) is minimized, while
that referred to the negative one will be enlarged. As the most common choice, we consider fa as
2

---

Under review as a conference paper at ICLR 2021
Euclidean distance between two embeddings, such that the corresponding alignment loss can be written as follows:
La=
Σ
||
efll
Σ
ReLU (a ||e/ e/.
where S- denotes the sampled set of negative pairs. is the minimal margin allowed between entities in each negative entity pair.
On the other hand, the scoring function fs can be also designed diversely. Most methods (Chen et al.
2017 Sun al. 2017 Pei et 2019a) choose TransE as their scoring function, i.c., fo (,,, r, [ +
e
(,, (e) E 7 U72. The corresponding loss is:
C.
Σ
(eLUUU,(T) -
E (aLU(A-S(TT))
TET.UT2
TET JT
where TI and 72 negative triple sets. C, is margin-based loss in which the distance d(e ej) positive triple should at least be smaller than À 2 O, while larger than À for negative ones. Note that, the negative triples are usually generated by randomly replacing the head or tail entity of positive triple. we only look at the replaced entity, minimizing the above loss can also understood randomly pushing entities away from this entity. This phenomena has also been studied in Wang & Isola (020).
Additionally, some graph neural network (GNN) based methods (Wang al. 2018 Sun al. 2020a Wu et al. 2019) do not directly optimize fa- They encode the semantic information inside graph convolution. Therefore, the output vectors of GNN will be regarded as entity embeddings feed into fa- For example, Ye et al. )211)); Wu et al. (2019); ; Sun et al. (2020a) leverage TransE as scoring function in the aggregation of relational neighbors.
2.2 UNDERSTANDING EEA
We illustarte how an EEA model works by example. Let (et, e E Gi X G2 be potentially- aligned entity pair. Each entity in this pair has only one neighbor, connected by the same relation actually a pair of entities (e), et S. Therefore, if an EEA model is well-trained and almost optimal, we should have e, e (as La is minimized) and r 2 (denoted by for simplicity). According to Equation 2
rd 2². We assume that their neighbors
have:
[e r, r.ee)) xs (l r,)))l A.
(3)
Take the scoring function of TransE fs. then derive:
|| + r e < X,
| + - ef?l < X.
(4)
As c/ = ef, we can conclude that:
Proposition (Discrepancy Bound). The representation difference of two aolentially-aligned entities bound by c, which is proportional to the hyper-parameter X:
||e - cx
(5)
The above proposition suggests that decreasing the value of margin À will tighten the feature discrepancy of entities in the underlying aligned entity pairs. However, we soon find that cannot be set as small
want.
We consider a more complicated yet realistic example, where each entity in (c), e2) has a considerable number of neighbors. We denote the corresponding triple sets of e Eg as Tr) T22 respectively. In this setting, a well-trained EEA model should satisfy that:
VT Tel U T2,, [[ A.
Evidently, TransE with small margin is not sufficient to fully express the semantics contained in T.. U p which has already been explored by previous works (Trouillon et al. 2016 Kazemi & Poole 2018 Sun et al.) 2019). Some empirical statistics (Sun et 2018) also illustrate such results. However, enlarging the margin À will bring significant variance between e and e
3

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
(d)
(e)
Figure 1: Example of different feature distributions. (a) Overall entity feature distributions of two KGs, iee, Ag. Blue points denote entities in G1 and orange ones are entities in G2- The two distributions nearly uniformly distributed and almost aligned (based the EEA model RDGCN (Wu et al.) 2019)). (b) The head entity feature distributions of relation "genre". The two distributions are only aligned partially. (c) Head entity feature distributions conditioned "genre", i.e., A lr. (based on NcoEA with RDGCN as EEA model, the same below). Two conditioned distributions are aligned as expected.
(d) The head entity distributions conditoned on three different relations: "genre" (colors: <blue,,range>) "writer" (colors: <purple.pink>), "brithPlace" (colors: <green,red>). The distributions corresponding to the first two relations are overlapped, while clear decision boundary between them and the last one is observed. (e) Triple feature distributions conditioned on relations "artist" (colors: <blue, orange>) and "musicalArtist" (colors: spurple.pink>) respectively. The distributions referred to sub-relation "rrsicalArtist" are covered by those corresponded to "artist".
On the other hand, the scoring function does not belong to the TransE family, e.g., it neural-based like ConvE (Dettmers et al.) 2018) or composition- basse like ComplEx (Trouillon et al., (2116), both of which are fully expressive (Kazemi Poole 2018). In this case, entities with a large number of neighbors can be correctly modeled, while those with only few neighbors are less constrained. Therefore, those models allow even more diversity between e and eyy We believe this is why they performed badly in EA task (Guo al. 2019 Sun et al. 2020b).
In short, most existing works adopt the above implicit strategy to learn cross-KG representations for EA, which makes them struggled in balancing between the bound and the expressiveness. In this paper, we explore a new direction to explicitly align the feature distributions two KGs, which ensures the embeddings tight and expressive.
NEURAL ONTOLOGY
3.1 NEURAL AXTOM AND NEURAL ONTOLOGY ALIGNMENT
In world KGs, entities and their relation triples conform with the axioms in ontologies (Baader al. 2005). Similarly, we call the feature distributions "neural axioms", as aligning them also allows us to regularize the entity embeddings at a high level.
We start by defining the basic neural axiom:
Definition (Basic Neural Axiom).
Ag ffeee
Aligning the basic neural axioms 4 and A of two KGs is trivial, as we can take the advantages of existing domain adaptation methods (Ben-David et al. 2010 Ganin & Lempitsky 2015a b Courty
4

---

Under review as a conference paper at ICLR 2021
al. 2017 Shen al. 2018). which aims to learn domain iittii-iivvnnnnnna representations for various tasks. We consider the adversarial learning based ones (Ganin & Lempitsky 2015a Shen et al. (2018). In this way, KG discriminator is leveraged to distinguish entity representations of 91 from those of O2 (or vice versa), while the embeddings will try to confuse the discriminator. Therefore, the same semantics in two KGs shall be encoded in the same way into the embeddings to fool the discriminator.
Specifically, we regard two KGs G1 92 as two different domains, and their embedding vectors "learnable features", can align the above axioms by an empirical Wasserstein distance based loss AArjovsyy el al. 2017: Shen et al. 2018):
(8) where fu the learnable domain critic that maps the embedding vector to a scalar value. As suggested in (Arjovsky et al., (017)) the empirical Wasserstein distance can be approximated by maximizing Lag. if the parameterized family of fu are all I-Lipschitz.
L = EAg [fu(e)] - Ba [fw
However, from Figure Ta we observe that the trained embeddings are nearly uniformly distributed in the feature space, which we can also derive from Equation and Equation 2 Recall that the
alignment loss Ca consists of two terms, The first is
| cf (c) (S
Σ
which aims to minimize the difference of embeddings for each positive pair. The cardinality of S is usually small. But this contrastive requires a large size of negative samples, which means that ||S| < ||.- Therefore, the model more focuses on the second term
Σ (ELS)EE
ReLU - -le/ ef.
(10)
of which the main target is randomly push the embeddings of different entities away from each other. Furthermore, C, is also a contrastive loss, and has a similar effect on maximizing the pairwise distance between each positive entity and its corresponding sampled negative ones. Therefore, conclude that:
Proposition 2 (Uniformity). The entity embeddings tend to uniformly distributed in feature space an EEA model is optimized.
The above proposition suggests that only aligning the basic axioms may be insufficient to facilitate EEA. Hence, we propose conditional neural axioms which are more specific and expressive.
3.2 CONDITIONAL NEURAL AXIOM
Conditional neural axioms describe the entity (or triple) feature distributions under specific semantic conditions.
Definition (Conditional Neural Axioms).
A.RIr. (el ri, e (elVe' (e, ee,)), TTH A {(cr. rs, (e), (() (ea, et) T
(II)
where A Ex.r: denotes the head entities feature distribution conditioned on the relation embedding r., the similar to A.,,rrr (we reduce (en, ri, ex |r; to (e,, ex)(r; for simplicity).
Numerous methods are proposed to process the neural conditioning operation, ranging from addition
and concatenation MMizaa & Osindero 2014 Wang et al. 2014; Yang al. 2015). to matrix multiplication (Lin et al. Ji et al. 2015 Nguyen et al. 2016). Comparing with claborating this operation, value more its common merit, which can be understood projecting the entities to a Dilation-specific subspace (Wang et al.] 2014) Lin et al.) 2013a] Nguyen et al.) 2016). Hence, the corresponding feature distributions conditioned different relation embeddings become discriminative, rather than almost uniformly distributed in original feature space (Lin et al.) 2015a).
Furthermore, conditional neural axioms capture high-level ontology knowledge.

---

Under review as a conference paper at ICLR 2021
Neural Ontology Alignment
Embedding
Entity Alignment
Coudland
Coddband
optiniiee the cubeddiiage
2G
minimize
difference
coadiieed edlly dett Andeas
Nakd
Wasserstein Distance Critis
triple adjacency matris as Input
Embedding Lookup
coadloed siple valty Auribeton
Bacc Nearal
Beer Sparel
XXXX BEED
KGI
XXXX FEEB
2019
XXXX FEED
raadvally sample triples
geterate
Sampled KG2
Sampled KG1
KG Sampling
Krabedriags
calaem
Figure The architecture of NcoEA. The solid lines denote the forward propagation, while the dotted lines represents the backward propagation. The architecture consists of three decoupled modules: (1) Entity alignemnt module, which the EEA model is unaware of the existence of other modules. (2) KG sampling module, in which we sample sub-KGs to replace the whole KGs for efficiency. (3) Neural ontology alignment module, in which the discrepancy between each pair of neural axioms is estimated by Wasserstein-distaace critic and minimized by gradient descent.
Theorem (Expressiveness). Aligning the conditional neural axioms minimizes the discrepancy of two KGs at ontology level.
Proof. See Appendix A for details. We take Argi:; as an example, which can summarize the empirical "Object Property Domain" axiom of rı in OWL2 (Baader al. 2005). Supposed there exists such an axiom that states the head entities of should belong to some specific class (e.g.. only head entities under class "Person" " have the relation "birthPlace"). We further suppose that there exists classifier fo(e) [0, such that fo(e,) otherwise. Then, with the knowledge the given axiom, one may derive the following rule:
if head entity e, belongs to class c, and 0
Ve E Velve', (e,r,,e Ti TUTT fe(e) = 11
(12)
which is equivalent to:
EA [S((e)] E,g Bale
[fe(e]] 1,
(13)
both of which means that all head entities of r; in either KG should be correctly classified to Then, we have:
E A [s((e]] - RAg, [[((e]] =
(14)
In fact, we do not have such knowledge about r; and class c, instead we can leverage neural function fe (elr;) to empirically estimate fe- In this way, A A,,, and Agg, are supposed to be aligned to minimize the loss correponding to the above rule. Therefore, we deduce this problem back to similar form Equation i.e.,
C
EA Bylrr
[fe
E A (c(r))
(15)
which suggests that aligning the above conditional neural axioms can minimize the discrepancy of potential "Object Property Domain" axioms between two KGs.

---

Under review as a conference paper at ICLR 2021
Example (OWL2 axiom: ObjectPropertyDomain As shown in Figure Tb and Figure Ic assume that the head entity of relation "genre" are under class "Work of Art (although it does not exist in the dataset), It is clear that the head entity feature distributions only partially algined in Figure Th while those in Figure are matched well.
In Figure ld illustrate more complicated example. The head entities of relations "genre" and "writer" mainly belong to "Work of Art", which show overlapped distributions (blue-orange, pink-purple) in the figure. By contrast, there exists clear decision boundary between them and the distributions conditioned on relation "birthPlace" (red-green), as the head entities of relation "birthPlace are under class "Person".
Example 2 (OWL2 axiom: SubObjectPropertyOn We consider two relations "musicalArtist" and "artist" as example, where the former one is the sub-relation of the later one. In Figure Tel the triple distributions conditioned "musicalArtist" (pink-purple) are covered by those conditioned on "artist" (orange-blues -luu).
3.3 ARCHITECTURE OF NEOEA
We illustrate the overall structure of NeoEA in Figure 2 can be divided into three modules:
Entity Alignment. This module aims encoding the semantics of KGs into embeddings. Almost all existing EEA models can used here, no matter what the input data looks like (triples or adjacency matrix).
KG Sampling, For each KG, we choose to sample a sub-KG to estimate the data distributions of neural axioms. Comparing with separately sampling candidates for each axiom, it is more efficient especially when the size of KGs get bigger, because we only sample once at each iteration.
Neural Ontology Alignment. As aforementioned. for each pair of representation distributions, align them by minimizing the empirical Wasserstein distance. Please see Appendix for the detailed implementation of neural ontology alignment.
EXPERIMENTS
In this section, empirically verify the effectiveness of NeoEA by series of experiments, with several -fate-oftthe-et methods as baselines.
4.1 IMPLEMENTATION
We illustrate the implementation of NeoEA in Algorithm [ The whole framework is based on the OpenEA project (Sun et al., 2020b), which includes the implementations of latest EEA methods. Specifically, we implemented neural ontology as an external module, based on which we modified only the initialization of the original project. In this sense,, the EEA methods unaware of the existence of neural ontologies. Furthermore, we kept the optimal hyper-parameter settings OpenEA ensure fair comparison.
We selected several best-performing and representative methods as our baselines:
BootEA (Sun et al.
2018), a TransE-based EEA model with only structure data.
SEA Pei et 2019a). a TransE-based model with both structure and attribute data.
2019) 9, an RNN-based EEA model with only structure data. RSN (uoo al.
RDGCN (Wu al. 2019). GCN-based model with both structure and attribute data.
The data distributions of some previous benchmarks such as JAPE (Sun et al.) 2017) and BootEA (Sun) ctt al. 2018) are clearly different from those of real-world KGs, which means that conducting experiments on those benchmarks cannot reflect the realistic performance of an EEA model (Guo et al. 2019: Sun et al. (020bb. Therefore, we consider the latest benchmark (Sun et al. 2020b). which consists of four sub-dataee dab--atasess, with two different density settings. Specifically, "D-W". "D-Y" denote
"DBpedia (Auer et al.) 2007)- WiikiDta (Vrandečić & Krötzsch] (2044), "DBpedia- YAGO (Fabian) al. 2007)". respectively. "EN-DE" and "EN-FR" denote cross-lingual datasets, both of which
7

---

Under review as a conference paper at ICLR 2021
sampled from DBpedia. "V1" denotes the sampled KGs having the similar distributions the original KGs, while "V2" denotes the sampled KGs with doubled density, For detail statistics, please refer to Sun al. 2020b).
Algorithm 1 NeoEA
Input: two KGs 91, 92, the alignment seed set S, the EEA model M(ss, fa), number of steps for NeoEA n;
2: Initialize all variables;
3: repeat
4: for to n do
5: Sample sub-KGs from respective KGs G1, 92;
6: Compute the Wasserstain distance based loss L for each pair of neural axioms; Optimize the Wasserstain distance critic fu by maximizing fww. end for
7:
8:
9:
Sample sub-KGs from respective KGs 91, 92;
10:
Compute Wasserstain distance based loss L.v for each pair of neural axioms;
Compute the losses C,, L, of the EEA model M,
12: Optimize the EEA model and embeddings by minimizing Lr, Co, Lwi
until the alignment loss on validation set converged.
4.2 EMPIRICAL COMPARISONS
The main results are shown in Table 1 from which we find that: (1) The performance of four baseline models varied from different datasets, but all of them gained improvement with NeoEA. SEA and RDGCN was more significant than that BootEA
(2) The performance improvement and RSN, as both BootEA and RSN are not typical EEA models. BootEA has a sophisticated bootstrapping procedure, which may be difficult to be injected with NeoEA. RSN tries to capture long-term dependencies among entities and relations. The complicated objective may be conflict with NcoEA more or less. However, some datasets (e.g., EN-DE, V1). still observe relatively significant improvement. Therefore, we believe the performance of these two models can be further refined through joint hyper-parameter turning with NeoEA, which leave future work.
Table I: Entity alignment results (5-fold cross-validation).
VI-Original
VI-NeoEA
V2-Original
V2-NcoEA
Models
H@1
5@5
MRR
HΦ1 1
H@5
MRR
H@1
H605
MRR
H
H@5
MRR
BootEA
.077
.718
.603
.221
.733
.617
.660
.850
.745
.665
.853
.749
EN-FR
SEA
280
.300
.397
.320
.584
.433
.360
.651
.444
.375
.666
.588
RSN
.393
.595
.487
.399
.597
.490
.579
.759
.662
.583
.760
.666
RDGCN
.755
.854
.800
.775
.688
.817
.847
.919
.800
.864
.933
.896
BootEA
.675
.820
.740
.676
.20
.740
.833
.912
.869
.834
.916
.870
D-W EN-DE
SEA
.300
.718
.617
.586
.766
.668
.606
.779
.687
.637
.800
.712
RSN
.587
.752
.662
.600
.759
.673
.791
.90
.837
.794
.892
.839
RDGCN
.830
889
.859
846
.008
874
.333
.891
.860
.849
.902
.874
BootEA
.744
.649
.579
.753
.658
.821
.822
.926
869
SEA
.572
.389
.6008
.490
.567
.770
.784
.677
RSN
.441
.6115
.521
.450
.624
.530
.723
.854
.729
.858
.787
RDGCN
.515
.6699
.584
.527
.522
.623
.757
.632
.760
.690
BootEA
.739
.849
.788
.756
.859
.797
.958
.984
.969
.958
.984
.969
SEA
.500
.706
.591
.549
.752
.638
.899
.950
.923
.917
.959
.936
RSN
.514
.655
.580
.522
.663
.588
.933
.974
.951
.935
.976
.953
RDGCN
.931
.969
.949
.941
.972
.955
.936
.966
.950
.940
.970
.953
The results improved most holdfaced.
4.3 ABLATION STUDY
We designed an ablation study which is expected to empirically prove some claims in Section 3 We choose the current state-of-the-ar model RDGCN baseline. As shown in Table "Full"
8

---

Under review as a conference paper at ICLR 2021
Table Results of ablation study based on the best-performing model RDGCN, on VI datasets.
EN-FR
EN-DE
D-W
D-Y
Models
H@1
H@5
MRR
H@5
MRR
H@1
H@5
MRR
H@]
H@5
MRR
Full
.775
.868
.817
.846
.988
.874
.527
.671
.592
.941
.922
.955
Partial
.771
.863
.813
.840
.000
.871
.523
.669
.590
.936
.971
.952
Basic
.755
.853
.799
.827
.895
.858
.512
.656
.578
.931
.969
.948
Original
.755
.854
.800
.300
.895
.859
.515
.669
.584
.931
.969
.949
NesEA Original
NeaEA Difgiual
Table 3: Average ranking improvement.
10
Datasets
Overall
Popular
Long tail
EN-FR
63.5
36.9
116.7
EN-DE
13.0
8.1
23.4
Figure 3: Normalized histgrams of alignment rankings on EN-FR, V1 (left, long-tail enti- ties; right, popular entities).
D-W
43.5
34.5
61.3
D-Y
119.3
59.2
214.2
denote NeoEA with full set neural axioms. "Partial" denotes NeoEA that removed the conditional triple axioms. We further removed the conditional entity axioms from "Partial" construct "Basic", and the last one "Original" denotes the original EEA model. From the results we observe that: (1) Aligning basic axioms less effective (Uniformity). (2) Aligning only a part of conditional axioms :r.::; that describe entity feature distributions conditioned on relation representations was significantly helpful for the model. (3) Additional improvement was observed on the model with the full conditional axioms. Note that, the improvement from "Partial" to "Full" was not significant as that from "Basic" "Partial". This is because that the conditional triple axioms mainly describe the axioms between relations (see Appendix [A). Due to the sampling strategy of the existing datasets, the number of relations relatively small. Few correlated relation pairs exist in the datasets, resulting in limited improvement with conditional triple neural axiom alignment.
harmful to the model, which verifies Proposition
4.4 FURTHER ANALYSIS ON THE BOUND
We have shown that the discrepancy between each underlying aligned pair is bounded by associated with X, in Section But we still expect to obverse empirical statistics to verify this point. To this end, manually split the entities into two group: (1) long-tail entities, which are disconnected to seeds and have at most two neighbors; (2) popular entities, the remaining. We draw the histgrams respective groups based on the EEA model SEA. From Figure 3
of alignment rankings can find that the proportion of the inexact alignments (i.e., ranking > 5) for long-tail entities is evidently larger than that of popular entities, especially for the bins [50, 100). This verified that the long-tail entities are less constrained compared to those popular entities. Furthermore, with NcoEA, the rankings of those long-tail entities were also improved more significantly compared with those of popular entities, which empirically proved that NeoEA tightened the representation discrepancy of those entities that were less restrained. We report the average ranking improvement on four datasets (V1) in Table 3 which shows consistent observations.
5 CONCLUSION
In this paper, we proposed a new approach to learn representations for entity alignment. We proved its expressiveness theoretically and demonstrated its efficiency by conducting experiments on the latest benchmarks. We observed that four state-of-the- EEA methods gained evident improvements with NeoEA. Finally, we showed that the proposed conditional neural axioms are the key to improve the performance of current EEA methods.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. CoRR, 2017.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. Dbpedia: A nucleus for a web of open data. In ISWC, 2007.
Franz Baader, Sebastian Brandt, and Carsten Lutz. Pushing the EL envelope. In IJCAI, 2005.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Mach. Learn., 2010.
Muhao Chen, Yingtao Tian, Mohan Yang, and Carlo Zaniolo. Multilingual knowledge graph embeddings for cross-lingual knowledge alignment. In IJCAI, 2017.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In NeurIPS, 2017.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2D knowledge graph embeddings. AAAI, 2018.
MS Fabian, Kasneci Gjergji, WEIKUM Gerhard, et al. Yago: A core of semantic knowledge unifying wordnet and wikipedia. In WWW, 2007.
Matthias Fey, Jan Eric Lenssen, Christopher Morris, Jonathan Masci, and Nils M. Kriege. Deep graph matching consensus. In ICLR, 2020.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML., 2015a.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In Francis R. Bach and David M. Blei (eds.), JCML, 2015b.
Lingbing Guo, Zequn Sun, and Wei Hu. Learning to exploit long-term relational dependencies in knowledge graphs. In ICML,, 2019.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. Knowledge graph embedding via dynamic mapping matrix. In ACL, 2015.
Seyed Mehran Kazemi and David Poole. Simple embedding for link prediction in knowledge graphs. In NeurlIPS, Montréal, Canada, 2018.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI, 2015a.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, 2014.
Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu, and Mark Johnson. STransE: novel embedding model of entities and relationships in knowledge bases. In NAACL, San Diego, USA, 2016.
Shichao Pei, Lu Yu, Robert Hoehndorf, and Xiangliang Zhang. Semi-supervised entity alignment via knowledge graph embedding with awareness of degree difference. In WWW, 2019a.
Shichao Pei, Lu Yu, and Xiangliang Zhang. Improving lross-lingual entity alignment via optimal
transport. In IJCAI, 2019b.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAI, 2018.
Zequn Sun, Wei Hu, and Chengkai Li. Cross-lingual entity alignment via joint attribute- attribute-preserving embedding. In ISWC, 2017.
Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. Bootstrapping entity alignment with knowledge graph embedding. In IJCAI, 2018.
10

---

Under review as a conference paper at ICLR 2021
Zequn Sun, Chengming Wang, Wei Hu, Muhao Chen, Jian Dai, Wei Zhang, and Yuzhong Qu Knowledge graph alignment network with gated multi-hop neighborhood aggregation. In AAAI, 2020a.
Zequn Sun, Qingheng Zhang, Wei Hu, Chengming Wang, Muhao Chen, Farahnaz Akrami, and Chengkai Li. A benchmarking study of embedding- based entity alignment for knowledge graphs.
PVLDB, 2020b.
Zhiqing Sun, Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR, 2019.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In ICML, 2016.
Bayu Distiawan Trsedya, Jianzhong Qi, and Rui Zhang. Entity alignment between knowledge graphs using attribute embeddings. In AAAI, 2019.
Denny Vrandečić and Markus Krötzsch. Wikidata: free collaborative knowledgebase. Communica- tions of the ACM. 57, 2014.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICMI., 2020.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI, 2014.
Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. Cross-lingual knowledge graph alignment via graph convolutional networks. In EMNLP, 2018.
Yuting Wu, Xiao Liu, Yansong Feng, Zheng Wang, Rui Yan, and Dongyan Zhao. Relation-aware entity alignment for heterogeneous knowledge graphs. In ICCAI, 2019.
Bishan Yang. Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In ICLR, 2015.
Rui Ye, Xin Li, Yujie Fang, Hongyu Zang, and Mingzhong Wang. A vectorized relational graph convolutional network for multi-relational network alignment. In IJCAI, 2019.
Hao Zhu, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Iterative entity alignment via joint knowledge embeddings. In IJCAI, 2017.

---

Under review as a conference paper at ICLR 2021
A PROOF FOR THEOREM I
Aligning the conditional neural axioms minimizes the discrepancies of two KGs at ontology level.
Proof. We split the proofs by different axioms referred OWL2 [Badder et al. 2005).
ObjectProperty Domain ObjectProperty Range. The proof for ObjecdPropertyDomale has been presented in Section 3 and that for ObjeedPropertyyRagee is similar.
ReflexiveOhjeecPPoperyy IrretlexivebbjeccPppert, If we say that relation r; is reflexive, it must satisfy
V(e, ri, (e) ET, (e,
(16)
which means each head entities of r; must be connected by to itself. The above rule suggests that we can align such axioms by minimizing the discrepancy between triple distributions conditioned on relation r,, i.c., aligning Eo..lr with A Er..,) The similar to IreelexiveChjecuPppeety axiom.
FuncttonalOOjeecPPoperty Inverse onceern We first introduce FunctionalOb- jectProperty axiom. It compels each head entity connected by relation r; to have exact one tail entity, implying the following rule:
V(e, (.)) E E, (c,rs, & T.
(17)
Evidently, the above rule is also related to the triple distribution conditioned on r;. The similar the tnvereeFunciioolOOheePPPPrrr axiom.
SymmetriiChjecdPoppetty/ AsymmecricOhjeetPPpert The first axiom state relation r; symmetric, that is,
v(e,Ti;e') ET, (é,,T,)) ET
(18)
It is also related to the triple distributions referred to rs, implying that directly aligning 4 4 en
with A lr lr; is sufficient to minimize the difference. The proof for AsymmetricOhjjecPPopetty
axiom is similar.
SuboojjecPPopeertyfrr EguivalentOOjeecPPppertss DisjointOhnjecttooprrte InverseOb- jectProperties. We show that these axioms also define rules related to triple distributions condionted on relations. We start from SubObjectPropenty which can state that relation r; is a subproperty of relation ry (e.g., "hasDog" is one of the subproperties of "hasPet"). We formulate it as:
Vee, ri, e') E TT, (e',r,,e) ET.
(19)
To algin the potential SubObjeccProperty0r axioms between two KGs, we can respectively align (A I Fo. (re A gln.)) and (A (a.olly AZ Funn ry such that the joint one (A Fuunllrryy 1 olu.,
A2
will also be aligned.
Similarly, if r; and ry are equivalent,
interpret the axiom as
e')) E T,, (e', TT; V(e,T,, ET, (e
(20)
If they are disjoint, the corresponding rule will be Veerri,e') E T, (e,ry,e)) $T.
(21)
If they are inverse to each other, the rule is
V(e,ri,ee) ET, (c),
(22)
TransttivOOhjeedPoperr, We show that this axiom are also related to triple distributions condi- tioned on rü- Supposed that a relation ri is transitive, then one can derive the following rule:
Vee, r., e') E T&(ee' ri, e²) TT, (e,r,,e)) ET.
(23)
which means we can align the potential Thanspit axioms via minimizing the distribu- tion discrepancy between A Funer and A
Ea.elre
12

---

Under review as a conference paper at ICLR 2021
B NEURAL ONTOLOGY ALIGNMENT
There exist several ways to implement the neural ontology alignment module. But it must satisfy two requirements: (I) Suitability, As mentioned in Section different EEA benchmarks have different assumptions about the relation alignments, which means that in some cases we cannot obtain all aligned relation pairs in advance. (2) Efficiency. As there may exist millions of triples, we should consider the efficiency of performing the conditioning operations.
Therefore, in our implementation, do not perform pair-wise sampling, either the pair- neural axiom alignment. We share the parameters of faaserstein-distanee critic only in each rype of neural axiom, which saves parameters and avoid the situation that some relations only have small number of corresponding triples. Furthermore, this also allows to perform fast mini- batch training by aligning the axioms belonging to the same type in one operation. Given the sample KGs Gj = (E R2 TT), G2 -E2, R2y, T2),, the corresponding batch loss is:
Leep = Lax +
Σ N'ER
EAR, ([)r)(
Σ MERR
BARgie [fn]-())r')))
(24)
+ Σ
EAg, A (fn,e (e er(rr)]
Σ M'RR
Eag Ir
M'R
where Lag. is the basic axiom loss under the sampled KGs. fhr and fhrdlr are the critic functions of the two types of neural axioms, respectively. The loss Cratch will approximate to that in a pair-wise calculation when the batch size is considerably greater than the number of relations. We take the second term of the above equation as example. For pair- pair-wise estimation, the corresponding loss should be:
Σ (mim))s
(Ea (c)r(( -Bah, [f(r((c(r2)))
(25)
Σ
BA (e)r))
Σ (8.
RAE (e)r2))
(M1)2)ESS
where S, C. R₁ X R2 denotes the set of all aligned relation pairs. The above equation suggests that the pair-wise loss is based the respective relation sets of two KGs, not constrained by each pair of aligned relations. Oftentimes, the number of relations is much smaller than the number of sampled triples in one batch, which means that Rí R2 Equation 24 can cover a large proportion of elements in the full relation sets R2, R2. Therefore, we used Ceep to approximate the pair-wise loss the implementation.
13


Reviews:
[{"review_id": "dLvFJt30T9V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper, the authors propose to minimize the discrepancy between pairs of (conditional) neural axioms to align the embedding spaces of different KGs. This method is justified by the authors' study of all kinds of OWL2 properties. The author also studied the influence of margin $\\lambda$ on less constrained/long-tail entities. The authors conducted experiments by adding the proposed model on top of the best models for entity alignment. The results are mixed, but the proposed model improves the SEA and RDGCN consistently. \n\nReasons to accept:\n1. This paper provides a theoretic point of view of the entity alignment task, which was mostly studied in empirical methods. The idea to align the axioms by minimizing Wasserstein distance is well-justified.\n2. The experiment results are in favor of the intuitions.\n2. The method described in this paper can be in principle adapted to any previous and future EEA scoring functions. \n\nReasons to reject:\nThe idea of using adversarial training to align spaces, especially cross-lingual spaces, is based on the assumption of the large overlap between KGs. For KGs that are on very different domains, this method may include errors, as two heterogeneous KGs do not naturally fit in one unified space. The influence of overlap on this method is not well-studied. All of KGs used in the experiments are general domain KGs. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_ve0kRP5X0r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper proposes NeoEA, an approach that further constrains KG embedding with ontology knowledge. The paper first tries to summarize the existing embedding-based entity alignment methods, stating that most of the methods choose TransE as scoring functions. But their embedding features are not aligned well compared to the neural-based or composition-based loss function. The paper, therefore, solves this problem by developing a new NeoEA architecture which shows that adding a KG-invariant ontology knowledge can minimize such difference. The experiment shows the new constraints can improve state-of-the-art baselines.\n\nStrengths:\n+ The idea of using ontology constraint as an additional loss is new. The paper shows significant improvements when combining the newly designed loss with state-of-the-art baselines.\n+ The overall paper is clear to understand. \n\nWeaknesses:\n- The paper doesn't have a related work section. Figure 1 is a little bit messy. Especially for figure 1d, it would be better to make the figure a little bit larger. Moreover, it's unclear which color corresponds to the first KG, making readers confused. Section 2 should be merged into the introduction section.\n- Some of the words are confusing such as neural axioms. The neural ontology alignment (which is stated in the appendix figure) is much more clear than the current Conditional Neural Axioms. Section 3 is not well-organized. The theorems and axioms should be propositions or some hypothesis. The usage of those words is a little bit wield here. The proof in the appendix is also not well defined.\n- The experiment section is too short and not very informative. It would be better to include more comprehensive analysis such as providing some visualization before and after the new loss etc.\n\n**Post-Rebuttal:\n- I appreciate that the authors have conduct revisions on the current version. However, I think the current paper is probably still not strong enough for ICLR. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "VE3TMz0FBs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Overall ---\n\nThis paper proposes an entity alignment framework that leverages the dependencies between entities and relations (reminiscent of TransR [Lin Y, et al. AAAI '15]) to further refine the results of conventional embedding-based alignment approach. \n\nMerits: The proposed framework is shown to be effective in improving the performance of baseline models.\n\nWeaknesses: (1) the methodological contribution is limited. (2) the theoretical explanation part is trivial and contributes little scientific knowledge.\n\n\n--- major comments ----\n\n1.\tThe bound in Eq.5 seems meaningless since the assumption (i.e., one relation and one neighbour) on which the bound basis is too idealistic to meet in practice. \n2.\tThe explanations about the behaviour of embedding-based entity alignment (in both section 2.2 and section 3.1) are straight-forward and trivial, thus contribute little knowledge.\n3.\tIn my point of view, section 2.1 and section 3.1 are too lengthy. It would be better to highlight the most important part i.e., loss function, while avoid emphasising too much on the detailed definitions and examples.\n\n\n--- minor comments---\nThere are some typos and grammar mistakes, need to be proof-read carefully (e.g., “e_x^2” -> “e_y^2” in the paragraph just above Eq.6; “take X for example”-> “take X as an example”).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Up7lIdhzkvV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall Comments:\nEntity alignment plays an important role in improving the quality of cross-lingual knowledge graphs. As one of the most important solutions, embedding-based methods aim at learning a semantic space where the unique entity cross knowledge graphs can have the closest distance. Most of research focus on entity-level granular, but discard the whole picture of embedding space of cross-lingual KGs. Besides the aligned entity pairs as the labelled data, this paper extended the labelled data with the conditional neural and basic axioms, which are actually sets of randomly selected entities or entities with the same relation type. Then the final objective is to align the cross-lingual knowledge graphs by both optimizing the distance of labelled entity pairs and neural axioms.\n\nClarity:\nThe presentation and organization of this paper is very difficult to follow.  Besides the grammar and type errors, there exists many concepts that not clear, which makes it difficult to understand the main idea of this work. For example, the concept of axiom and ontology are introduced before giving a formal definition. The claimed challenges, that have not been solved well by previous works ,are not convincing enough. In the 3rd paragraph, authors argued that previous research shows very good performance, but has not made on the theoretical analysis. After reading the whole draft, it's still a big question on the given theoretical analysis of this work. Taking the theorem 1 as example, it's more like a justification but not a theorem to show the connection between the proposed axiom and \"ontology\". Ontology provides an empirical structure to organize and classify the entities in the KGs. Its structure will be changed along with the KG in hand. From this paper, I can not find the connection between relation type alignment loss and the ontology. Please pay more attention to the writing and organization. It's an interesting work but not ready.\n\nQuestions for Rebuttal:\n1. How to build the relation seed in this work? Are they labelled manually? If so, it will have flexibility issue for dealing multiple KGs.\n\n2. Please compare to a recent proposed method [1] which also optimizes the distance between a group of entities from cross-lingual KGs. Different from this work, it's not condition on the relation type, but based on a randomly sampled group of entities.\n\nMinor Comments:\n\n1. In the paragraph around the Equation (6), the e_x^2 should be e_y^2.\n\n2. Figure 3 shows the overall architecture of the proposed method. It should appear in the main content.  \n\nReferences:\n\n[1] Pei, Shichao, Lu Yu, and Xiangliang Zhang. \"Improving cross-lingual entity alignment via optimal transport.\" International Joint Conferences on Artificial Intelligence Organization, 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study the problem of augmenting embedding-based entity alignment in knowledge graphs (KG) through the use of joint alignment with deduced neural ontologies (more specifically, alignment of the KG 'neural' axioms). Motivated by the observation that the representation between two potentially aligned entities must be bound by a minimal margin, which can be problematic when there are many potential alignments, they propose aligning neural axioms by Wasserstein distance-based loss between learned entity embeddings conditioned on the relation embeddings. Experiments are conducted on OpenEA against multiple strong baselines -- showing that adding the ontology alignment to these baselines improves the results.

== Pros ==
+ The addition of aligning (conditional) ontologies is ostensibly novel.
+ For KGs with sufficient entity/relation overlap, the proposed NeoEA method is applicable. 
+ NeoEA has been shown empirically to improve many SoTA methods.

== Cons == 
- While the theoretical justification is a welcome motivation, the reviewers did not find the theoretical arguments significant nor convincing.
- Overall, the narrative needs work to make the paper more self-contained and approachable for a broader range of readers. The reviewers (and myself) found many concepts and statements somewhat confusing and needing clearly context and contrast with existing works.

Evaluating along the requested dimensions:
- Quality: Conceptually, the core idea is interesting, well-motivated, original, and ostensibly effective. Empirically, NeoEA is shown able to improve upon several strong baseline (underlying) methods.  I believe that all of the reviewers find the work is interesting and promising. However, there were continuing concerns the strength/value of the described theory; it isn't clear if stronger theory isn't possible or if this just hasn't been fleshed out. 
- Clarity: Most of the reviewers (and myself) found the paper difficult to follow as a self-contained work in terms of concepts, clear definitions (e.g., \mathcal T isn't defined early on) and the actual applicability of the theory. The figures help, but even these need some work. A related work section (or more structured presentation of related work) might be clarifying along with running examples and a more unifying math presentation that captures existing and proposed work. After thinking about this more, it is actually a relative simple (in a good way) and clever idea. However, it took several readings and readings of related work to get there. Additionally, the fact that all of the reviewers were concerned about different limitations is concerning wrt clarity. Appendix B helps a bit and I believe can also be put into the main paper.
- Originality: As best as the reviewers and I can tell, we haven't seen this method applied to entity alignment despite this being a relatively mature subfield.
- Significance: The consensus seems to be that the approach could be a notable contribution to an important area. However, it also appears that most of the reviewers don't feel the paper is ready for publication at a top-tier venue yet.

As stated throughout this meta-review, there are several aspects to like about this work including the originality of the idea, strong motivation, and good empirical results. However, we all agreed that the paper isn't quite ready in its current form -- thus, I presently recommend reject for this submission.
Decision: Reject