SlowGuess/NAACL_2022_OCR huggingface dataset:
Juncheng Liu<sup>1</sup> Zequn Sun<sup>2</sup> Bryan Hooi<sup>1</sup> Yiwei Wang<sup>1</sup> Dayiheng Liu<sup>3</sup> Baosong Yang<sup>3</sup> Xiaokui Xiao<sup>1</sup> Muhao Chen<sup>4</sup>
$^{1}$ National University of Singapore  $^{2}$ Nanjing University
$^{3}$ Alibaba Group  $^{4}$ University of Southern California
juncheng.liu@u.nus.edu, zqsun.nju@gmail.com
muhaoche@usc.edu
We study dangling-aware entity alignment in knowledge graphs (KGs), which is an under-explored but important problem. As different KGs are naturally constructed by different sets of entities, a KG commonly contains some dangling entities that cannot find counterparts in other KGs. Therefore, dangling-aware entity alignment is more realistic than the conventional entity alignment where prior studies simply ignore dangling entities. We propose a framework using mixed high-order proximities on dangling-aware entity alignment. Our framework utilizes both the local high-order proximity in a nearest neighbor subgraph and the global high-order proximity in an embedding space for both dangling detection and entity alignment. Extensive experiments with two evaluation settings show that our framework more precisely detects dangling entities, and better aligns matchable entities. Further investigations demonstrate that our framework can mitigate the hubness problem on dangling-aware entity alignment.
Knowledge graphs (KGs) have become the backbone of many intelligent applications (Ji et al., 2021). In spite of their importance, many KGs are independently created without considering the interrelated and interchangeable nature of individually created knowledge (Chen et al., 2020). To allow complementary knowledge to be automatically combined and migrated across individual KGs, entity alignment seeks to identify equivalent entities in distinct KGs (Sun et al., 2020a). Recent literature has focused on learning embedding representations of multiple KGs where identical entities are aligned based on their embedding similarity (Chen et al., 2017; Cao et al., 2019; Fey et al., 2020; Sun et al., 2020a; Liu et al., 2021).
Aside from the surge of research effort on entity alignment (Zeng et al., 2021), an unresolved
(b)
(c)
Figure 1: Toy examples for mixed high-order proximities. (a) Nearest neighbor (NN) subgraph where entities connect to NNs in the other KG using embedding similarities. 4 and its nearest neighbor B have 0.85 similarity. B prefers 2 and 3 with higher similarities. 1 and A are mutual nearest neighbors. (b) Labeled alignments and dangling entities. (c) Aligning matchable source and target distributions rather than only labeled alignments.
but important challenge that existing methods face is the dangling entity problem. Dangling entities are those unique entities in a KG that cannot find counterparts in another KG. Considering that individually created KGs are unlikely to share the same set of entities, identifying dangling entities is undoubtedly an indispensable step of any practical solution to entity alignment. However, nearly all prior studies have neglected dangling entities and assume there must be one-to-one entity mapping from the source KG to the target one (Sun et al., 2020c). This assumption prevents prior methods from practically supporting the alignment between KGs in real-world scenarios. To fill the gap, Sun et al. (2021) formally define a more practical problem setting where a model needs to both determine whether each given source entity is a matchable one, as well as retrieve counterparts for the predicted matchable entities.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1172 - 1184
Although some preliminary attempts have been made to implement dangling-aware entity alignment (Sun et al., 2021), the attempted methods still suffer from a major drawback, i.e., they only consider the first-order proximity (namely, pairwise cosine similarity) between source and target entities. However, to effectively discover dangling entities as outliers in the embedding representation, we argue that high-order proximity measures should also be involved. Fig. 1a shows the intuition of using the high-order proximity for dangling entity detection. Despite a fairly high cosine similarity, the source entity 4 is not the nearest neighbor of target entity B, indicating that 4 is likely to be dangling. In contrast, 1 and A are mutual nearest neighbors even with a relatively low similarity, indicating that 1 is more likely to be matchable. Hence, in addition to the first-order proximity from source to target, the local high-order proximity (e.g., the second-order proximity<sup>1</sup> in the nearest neighbor subgraph) is also useful for detecting dangling entities. In alignment learning, the previous works neglect global information since they merely optimize the entity-level alignment loss on labeled alignments without considering entity embedding distributions as shown in Fig. 1b. In Fig. 1c, we show that a desirable dangling-aware model should align the global distributions of matchable source and target entities (i.e., global high-order proximity in an embedding distribution space), such that dangling entities in both KGs could appear as dissimilar outliers in both distributions.
Motivated by the above intuition, we propose a dangling-aware entity alignment framework based on mixed high-order proximities (MHP). MHP considers both local and global high-order proximities to foster both dangling entity detection and matchable entity alignment. We introduce the optimization of global high-order proximity measure as finding the Optimal Transport between matchable source entities and target entities. Through this optimization process, to facilitate dangling detection, MHP also encourages a large distance between the dangling entity distribution and matchable entity distribution. Additionally, to leverage the local high-order proximity, we propose a dangling entity classifier which takes into account the second-order proximity in the nearest neighbor subgraph. Furthermore, with the similar principle of local high
order proximity, we adopt an NCA (Neighborhood Component Analysis) loss (Goldberger et al., 2004; Liu et al., 2021) for alignment learning to mitigate the hubness problem $^{2}$  (Radovanovic et al., 2010), which is severe in dangling-aware entity alignment as observed in our experiments.
Our main technical contributions to the studied problem are two-fold. First, the local high-order proximity (i.e., the second-order proximity in the nearest neighbor subgraph) is modeled to facilitate both dangling detection and alignment learning. Second, we design the use of the global high-order proximity to align the distributions of matchable entities, therefore precisely separating the representations of dangling entities and matchable ones. In addition, the techniques are model-agnostic and can be incorporated with various alignment methods (e.g., MTransE (Chen et al., 2017) or AliNet (Sun et al., 2020a)) and dangling detection methods (e.g., the marginal or background ranking (Sun et al., 2021)). Extensive experiments on DBP2.0 demonstrate its effectiveness and adaptiveness.
In this section, we provide the problem definition of dangling-aware entity alignment and briefly introduce previous methods for this problem.
A KG is defined as  $\mathcal{G} = (\mathcal{E},\mathcal{R},\mathcal{T})$ , where  $\mathcal{E}$  denotes a set of entities;  $\mathcal{R}$  denotes a set of relations, and  $\mathcal{T} \subset \mathcal{E} \times \mathcal{R} \times \mathcal{E}$  is a set of triples. Following the convention (Chen et al., 2017), we consider entity alignment between a source KG  $\mathcal{G}_s = (\mathcal{E}_s,\mathcal{R}_s,\mathcal{T}_s)$  and a target KG  $\mathcal{G}_t = (\mathcal{E}_t,\mathcal{R}_t,\mathcal{T}_t)$ . Our study focuses on a more practical and challenging setting with dangling entities (Sun et al., 2021). In this setting, the training data contains a set of seed entity alignment  $\mathcal{A} = \{(x_s,x_t) \in \mathcal{E}_s \times \mathcal{E}_t \| x_s \equiv x_t\}$  and a set of source dangling entities  $\mathcal{D} \subset \mathcal{E}_s$  that has no counterparts in target KG. After training, the model is required to first identify dangling entities and then find alignment for predicted matchable entities. This definition breaks the one-to-one assumption used in previous studies on the conventional setting (Sun et al., 2020c) and causes their methods to not be directly usable in our setting.
<sup>1</sup>The second-order proximity of a source entity  $s$  is defined as aggregated cosine similarities between the nearest targets of  $s$  and the nearest sources of these nearest targets.
2The hubness problem is where some target entities dominantly appear as the nearest neighbors of many source entities.
2.2 Dangling-aware entity alignment
To the best of our knowledge, there is only one previous work (Sun et al., 2021) which has been attempted for dangling-aware entity alignment along with the proposing of this important problem. This work also incorporates an embedding-based entity alignment technique (i.e., MTransE (Chen et al., 2017) and AliNet (Sun et al., 2020a)) as the backbone, which learns alignment of KG embeddings based on the seed entity pairs. Taking MTransE as an example, for each pair  $(x_{s}, x_{t}) \in \mathcal{A}$ , MTransE uses the learned embedding  $\mathbf{x}_{s}$  and  $\mathbf{x}_{t}$  to optimize a linear transformation matrix  $\mathbf{M}$  by minimizing  $\|\mathbf{M}\mathbf{x}_{s} - \mathbf{x}_{t}\|$ . To detect dangling entities in the embedding space, a margin ranking (MR) loss and a background ranking (BR) loss are experimented with, both encouraging dangling entities to be isolated from others in the embedding space. MR sets a distance margin  $\lambda$  to separate the dangling entity  $x$  and its nearest neighbors by minimizing  $\max(0, \lambda - \|\mathbf{M}\mathbf{x} - \mathbf{x}_{\mathrm{mn}}\|)$ . In like manner, BR treats dangling entities as the background of embedding space and learns to separate dangling entities and randomly-sampled other entities.
In this section, we introduce the techniques in our framework which captures both local and global high-order proximities to collaboratively tackle dangling detection and entity alignment.
3.1 Global high-order proximity
To leverage the global high-order proximity in an embedding space, in MHP, we introduce a method based on Optimal Transport (OT) for globally aligning the distributions of matchable source and target entities. In addition, to facilitate dangling entity detection, the OT model encourages a large distribution distance between source- and target-KG dangling entities. Intuitively, this strategy treats dangling entities as dissimilar parts of two embedding distributions, therefore tending to put dangling entities as outliers in the embedding space.
Optimal transport. Let  $\mathbf{s}$  and  $\mathbf{t}$  be the distribution of transformed source-space embeddings  $\mathbf{M}\mathbf{x}_s$  and target space embeddings  $\mathbf{x}_t$ , respectively. Intuitively,  $\mathbf{s}$  should be similar with  $\mathbf{t}$  if they represent matchable entities. Meanwhile, to make dangling
entities distinguishable, the distribution of transformed dangling entity vectors  $\mathbf{M}\mathbf{x}_d$  should be different from t. The discrepancy between s and t can be represented as a Wasserstein distance which is one type of OT distance (Peyre et al., 2019):
$$
\mathcal {D} _ {c} (\mathbf {s}, \mathbf {t}) = \inf  _ {\gamma \in \Pi (\mathbf {s}, \mathbf {t})} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim \gamma} [ c (\mathbf {x}, \mathbf {y}) ], \tag {1}
$$
where  $\Pi (\mathbf{s},\mathbf{t})$  is the set of all possible joint distributions  $\gamma (\mathbf{s},\mathbf{t})$  and  $c(\mathbf{x},\mathbf{y})$  denotes the cost function describing the distance between  $\mathbf{x}$  and  $\mathbf{y}$ . Then the Wasserstein distance  $\mathcal{D}_c(\mathbf{s},\mathbf{t})$  denotes the cost of the optimal transport plan.
However, the infimum to calculate  $\mathcal{D}_c(\mathbf{s},\mathbf{t})$  is highly intractable (Arjovsky et al., 2017). To handle this, the Kantorovich-Rubinstein duality points out that Eq. (1) can be transformed to:
$$
\mathcal {L} _ {o t} (\mathbf {s}, \mathbf {t}) = \frac {1}{K} \sup  _ {\| f \| _ {L} \leq K} \mathbb {E} _ {x \sim \mathbf {t}} [ f (x) ] - \mathbb {E} _ {x \sim \mathbf {s}} [ f (x) ], \tag {2}
$$
where the supremum is over all possible K-Lipschitz functions  $f$ . As Arjovsky et al. (2017) point out that optimizing Wasserstein GAN (WGAN) can be used to solve this optimal transport problem, we utilize WGAN in our study. Specifically, we adopt a MLP to approximate the function  $f$  (called as critic  $D$ ) since neural networks are universal approximators (Hornik et al., 1989). The objective of the critic is defined as follows:
$$
\max  _ {D} \mathbb {E} _ {y \sim \mathbf {t}} [ f _ {D} (y) ] - \mathbb {E} _ {x \sim \mathbf {s}} [ f _ {D} (\mathbf {M} x) ]. \tag {3}
$$
Thus, the critic  $D$  aims to distinguish transformed source embeddings from target embeddings. In contrast, the transformation matrix  $\mathbf{M}$  tries to minimize the distance between the two sets of embeddings. The objective to optimize  $\mathbf{M}$  is defined as:
$$
\begin{array}{l} \min _ {\mathbf {M}} \mathbb {E} _ {y \sim \mathbf {t}} \left[ f _ {D} (y) \right] - \mathbb {E} _ {x \sim \mathbf {s}} \left[ f _ {D} \left(\mathbf {M} x\right) \right] \\ = \min  _ {\mathbf {M}} - \mathbb {E} _ {x \sim \mathbf {s}} \left[ f _ {D} (\mathbf {M} x) \right]. \tag {4} \\ \end{array}
$$
Therefore, conducting entity alignment with the consideration of whole embedding distributions is converted to the problem of optimizing a WGAN.
So far, only the distribution of matchable source entity embeddings and that of target entity embeddings are considered, whereas the distribution of dangling entity embeddings is neglected. Therefore, to tailor the optimization problem for dangling entities, we adopt an additional objective for the
Without loss of generality, we use a matrix  $\mathbf{M}$  to transform embeddings from source KG to target KG.
(a) Proximity distribution of nearest entities.
(b) Proximity distribution of the second nearest entities.
Figure 2: Second-order proximity distributions.
transformation matrix M:
$$
\begin{array}{l} \max  _ {\mathbf {M}} \mathbb {E} _ {y \sim \mathbf {t}} \left[ f _ {D} (y) \right] - \mathbb {E} _ {x \sim \mathbf {d}} \left[ f _ {D} (\mathbf {M} x) \right] \\ = \min  _ {\mathbf {M}} \mathbb {E} _ {x \sim \mathbf {d}} \left[ f _ {D} (\mathbf {M} x) \right], \tag {5} \\ \end{array}
$$
where  $\mathbf{d}$  denotes the distribution of dangling entity embeddings. Hence, the transformation matrix  $\mathbf{M}$  is enforced to maximize the difference between the distribution of transformed dangling embeddings and that of target entity embeddings, which can make dangling entities more distinguishable.
3.2 Local high-order proximity
In addition to the global proximity measure, MHP also captures local high-order proximity measures in the nearest neighbor subgraph. In contrast, the previous work (Sun et al., 2021) merely uses the first-order proximity between an individual source entity  $s$  and its nearest target entity to decide whether  $s$  should be dangling. However, apart from the first-order proximity, the second-order proximity measure could be informative as well for detecting dangling entities.
Furthermore, we verify the above hypothesis empirically using the previous work. From the nearest target entity  $t$  of a given source entity, we obtain the cosine similarities between  $t$  and its top 2 nearest source entities as the second-order proximity measures, and plot the proximity distributions in Fig. 2a and 2b for the first and second nearest entities, respectively. Fig. 2a shows that the second-order proximity between matchable entities and
their nearest neighbors appear as a very different distribution in comparison to that of the proximity between dangling entities and their nearest neighbors. Fig. 2b demonstrates a similar observation for the proximity distributions of the second nearest entities. Therefore, the second-order proximities are informative and should be used for distinguishing dangling entities, meanwhile combining proximity measures that consider multiple neighboring entities is more useful than a single similarity measure on only the nearest entity.
To this end, we design a dangling entity classifier using both first-order and second-order proximity measures as the input. From the perspective of a given source entity  $s$ , we conduct nearest neighbor search to obtain top  $k$  nearest target entities  $\{t_1,\dots,t_k\}$  and their proximity score vector  $\mathbf{d}_1 = [d_{st_1},\dots,d_{st_k}]\in \mathbb{R}^{1\times k}$ . The proximity  $d_{st}$  is measured by the cosine similarity between transformed source embedding  $\mathbf{Mx}_s$  and target entity embedding  $\mathbf{x}_t$ :
$$
d _ {s t _ {1}} = \left\langle \frac {\mathbf {M} \mathbf {x} _ {s}}{\| \mathbf {M} \mathbf {x} _ {s} \| _ {2}}, \frac {\mathbf {x} _ {t}}{\| \mathbf {x} _ {t} \| _ {2}} \right\rangle \tag {6}
$$
After getting the first-order proximity vector  $\mathbf{d}_1$  between the source and target, through the reverse direction (target KG to source KG), we can further obtain the second-order proximity vector. Specifically, we retrieve top  $m$  nearest source entities  $\{s_1,\dots,s_m\}$  of each target entity  $t$  in  $\{t_1,\dots,t_k\}$ . Accordingly, through the target entity  $t$ , the second-order proximity measures with regard to the  $m$  retrieved source entities are obtained as the vector  $\mathbf{d}_t = [d_{s_1t},\dots,d_{smt}]\in \mathbb{R}^{1\times m}$ . Subsequently, we can collect  $\{\mathbf{d}_{t_1}\dots \mathbf{d}_{t_k}\}$  and concatenate them as the whole second-order proximity vector  $\mathbf{d}_2 = [\mathbf{d}_{t_1}||\dots||\mathbf{d}_{t_k}]\in \mathbb{R}^{1\times km}$ .
To utilize both second-order and first-order information, the whole proximity distribution vector is constructed as  $\mathbf{d} = [\mathbf{d}_1||\mathbf{d}_2]\in \mathbb{R}^{1\times (k + 1)m}$ . In this way, we use the distribution as a profile of the neighborhood of a source entity  $s$ , then we adopt a simple feed-forward neural network (FNN) binary classifier to determine whether  $s$  is dangling. The probability of  $s$  being a dangling entity can be calculated as  $p(y = 1|s) = \mathrm{sigmoid(FNN(d))}$ . Define  $\mathcal{D}$  and  $\mathcal{A}$  to be the training set of dangling entities and that of matchable entities, respectively.
We minimize the binary cross-entropy loss:
$$
\begin{array}{l} \mathcal {L} _ {s} = - \frac {1}{| \mathcal {D} \cup \mathcal {A} |} \sum_ {s \in \mathcal {D} \cup \mathcal {A}} \left(y _ {s} \log (p (y = 1 | s)) \right. \tag {7} \\ + \left(1 - y _ {s}\right) \log (1 - p (y = 1 | s))) \\ \end{array}
$$
NCA loss. With the similar principle of local high-order proximity, MHP adopts an additional Neighbor Component Analysis (NCA) loss (Liu et al., 2021) to mitigate the hubness problem. The hubness problem can be more severe in dangling-aware entity alignment as dangling entities might be aligned to some certain hubs if they are not detected as dangling. The NCA loss measures importance of samples and punishes hard negative pairs based on the proximities. Given the set of seed entity alignments  $\{(x_s,x_t)\in \mathcal{E}_s\times \mathcal{E}_t\}$ , let  $\mathbf{S}$  be the cosine similarity matrix between source and target entity embeddings  $\mathbf{E}_1$  and  $\mathbf{E}_2$ . The NCA loss can be defined as follows:
$$
\begin{array}{l} \mathcal {L} _ {N C A} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\frac {1}{\alpha} \log \left(1 + \sum_ {m \neq i} e ^ {\alpha \mathbf {S} _ {i m}}\right) + \right. \tag {8} \\ \frac {1}{\alpha} \log (1 + \sum_ {n \neq i} e ^ {\alpha \mathbf {S} _ {n i}}) - \log (1 + \beta e ^ {\mathbf {S} _ {i i}})), \\ \end{array}
$$
where  $\mathbf{S}_{ii}$  denotes the proximity of the i-th positive pair (i.e., the i-th source entity and the i-th target entity);  $\alpha, \beta$  are temperature hyper-parameters; and  $N$  is the number of positive pairs in the mini-batch.
Note that our techniques are used to improve existing first-order methods. MHP optimizes the entity alignment component and the dangling detection component alternately. For entity alignment, besides an entity-level loss (e.g., MTransE), we first train WGAN for optimal transport and then optimize the NCA loss  $\mathcal{L}_{NCA}$ . For dangling detection, besides a first-order objective (e.g., a marginal ranking loss) used in Sun et al. (2021), we train our dangling classifier for detection. In the inference phase, for each source entity, the dangling entity classifier provides a probability score and uses a probability threshold to decide whether an entity is dangling, where the threshold is set as the average probability. After this dangling detection process, the predicted dangling entities are excluded from being aligned. Then, in the alignment process, MHP conducts nearest neighbor search to find the alignment in the target KG embedding space for each of the rest matchable source entities.
In this section, we report our experiments to show the effectiveness of MHP. We describe the evaluation settings in Sec. 4.1, and present the results in two alignment settings separately in Sec. 4.2 and 4.3. We conduct an ablation study and demonstrate that MHP can mitigate the hubness problem in Sec. 4.4, followed by a case study to show the importance of local high-order proximity in Sec. 4.5.
We use two evaluation settings as suggested by Sun et al. (2021). The first one is consolidated evaluation which requires a model to first detect and remove dangling entities, and then conduct alignment search for the rest of entities. The performance of dangling entity detection is also evaluated in this setting. Besides, a simplified relaxed evaluation setting seeks to test the performance of alignment alone without involving dangling source entities in the test set. In this setting, the effect of dangling detection on entity alignment can be evaluated.
Evaluation protocol. For the relaxed setting, the counterpart list is selected by the Nearest Neighbor (NN) search in the embedding space for each source entity. To assess the ranking list, we use mean reciprocal rank (MRR), Hits@1 and Hits@10 (hereinafter H@1 and H@10) as metrics. Higher values indicate better performance.
For the consolidated setting, we evaluate the performance of both dangling entity detection and entity alignment using precision, recall, and F1 score, following Sun et al. (2021). In this setting, only the source entities that are correctly predicted as matchable are sent to the NN search and the nearest counterpart is evaluated. Particularly, incorrect dangling detection (i.e., a matchable entity is wrongly predicated as dangling or a dangling entity is predicted as matchable) will propagate an error case to the alignment process. We refer to this practical entity alignment as two-step entity alignment.
Dataset. We use the cross-lingual dangling-aware entity alignment dataset DBP2.0 (Sun et al., 2021), which is constructed using multilingual DBpedia (Lehmann et al., 2015). There are three language pairs (ZH-EN, JA-EN, FR-EN) in DBP2.0 and two alignment directions are considered for each pair. We follow its data splits where  $30\%$  dangling
4Note that H@1, H@10 and MRR are not applicable to this entity alignment in the consolidated setting.
entities are for training,  $20\%$  for validation, and  $50\%$  for test. The dataset statistics are reported in Appx. A.