SlowGuess/ACL_2020_OCR huggingface dataset:
Yuting Wu $^{1}$ , Xiao Liu $^{1}$ , Yansong Feng $^{1,2*}$ , Zheng Wang $^{3}$  and Dongyan Zhao $^{1,2}$
$^{1}$ Wangxuan Institute of Computer Technology, Peking University, China
2The MOE Key Laboratory of Computational Linguistics, Peking University, China
$^{3}$ School of Computing, University of Leeds, U.K.
{wyting, lxlisa, fengyansong, zhaodongyan}@pku.edu.cn
z.wang5@leeds.ac.uk
Structural heterogeneity between knowledge graphs is an outstanding challenge for entity alignment. This paper presents Neighborhood Matching Network (NMN), a novel entity alignment framework for tackling the structural heterogeneity challenge. NMN estimates the similarities between entities to capture both the topological structure and the neighborhood difference. It provides two innovative components for better learning representations for entity alignment. It first uses a novel graph sampling method to distill a discriminative neighborhood for each entity. It then adopts a cross-graph neighborhood matching module to jointly encode the neighborhood difference for a given entity pair. Such strategies allow NMN to effectively construct matching-oriented entity representations while ignoring noisy neighbors that have a negative impact on the alignment task. Extensive experiments performed on three entity alignment datasets show that NMN can well estimate the neighborhood similarity in more tough cases and significantly outperforms 12 previous state-of-the-art methods.
By aligning entities from different knowledge graphs (KGs) to the same real-world identity, entity alignment is a powerful technique for knowledge integration. Unfortunately, entity alignment is nontrivial because real-life KGs are often incomplete and different KGs typically have heterogeneous schemas. Consequently, equivalent entities from two KGs could have distinct surface forms or dissimilar neighborhood structures.
In recent years, embedding-based methods have become the dominated approach for entity alignment (Zhu et al., 2017; Pei et al., 2019a; Cao et al., 2019; Xu et al., 2019; Li et al., 2019a; Sun et al.,
(b)
Figure 1: Illustrative examples: two tough cases for entity alignment. Dashed rectangles denote the common neighbors between different KGs.
2020). Such approaches have the advantage of not relying on manually constructed features or rules (Mahdisoltani et al., 2015). Using a set of seed alignments, an embedding-based method models the KG structures to automatically learn how to map the equivalent entities among different KGs into a unified vector space where entity alignment can be performed by measuring the distance between the embeddings of two entities.
The vast majority of prior works in this direction build upon an important assumption - entities and their counterparts from other KGs have similar neighborhood structures, and therefore, similar embeddings will be generated for equivalent entities. Unfortunately, the assumption does not always hold for real-life scenarios due to the incompleteness and heterogeneities of KGs. As an example, consider Figure 1 (a), which shows two equivalent entities from the Chinese and English versions of Wikipedia. Here, both central entities refer to the same real-world identity, Brooklyn, a borough of New York City. However, the two entities have different sizes of neighborhoods and
6477
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6477-6487 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
distinct topological structures. The problem of dissimilar neighborhoods between equivalent entities is ubiquitous. Sun et al. (2020) reports that the majority of equivalent entity pairs have different neighbors in the benchmark datasets DBP15K, and the proportions of such entity pairs are over  $86\%$  (up to  $90\%$ ) in different language versions of DBP15K. Particularly, we find that the alignment accuracy of existing embedding-based methods decreases significantly as the gap of equivalent entities' neighborhood sizes increases. For instance, RDGCN (Wu et al., 2019a), a state-of-the-art, delivers an accuracy of  $59\%$  on the Hits@1 score on entity pairs whose number of neighbors differs by no more than 10 on DBP15K $_{ZH-EN}$ . However, its performance drops to  $42\%$  when the difference for the number of neighbors increases to 20 and to  $35\%$  when the difference increases to be above 30. The disparity of the neighborhood size and topological structures pose a significant challenge for entity alignment methods.
Even if we were able to set aside the difference in the neighborhood size, we still have another issue. Since most of the common neighbors would be popular entities, they will be neighbors of many other entities. As a result, it is still challenging to align such entities. To elaborate on this point, let us now consider Figure 1 (b). Here, the two central entities (both indicate the city Liverpool) have similar sizes of neighborhoods and three common neighbors. However, the three common neighbors (indicate United Kingdom, England and Labour Party (UK), respectively) are not discriminative enough. This is because there are many city entities for England which also have the three entities in their neighborhoods – e.g., the entity Birmingham. For such entity pairs, in addition to common neighbors, other informative neighbors – like those closely contextually related to the central entities – must be considered. Because existing embedding-based methods are unable to choose the right neighbors, we need a better approach.
We present Neighborhood Matching Network (NMN), a novel sampling-based entity alignment framework. NMN aims to capture the most informative neighbors and accurately estimate the similarities of neighborhoods between entities in different KGs. NMN achieves these by leveraging the recent development in Graph Neural Networks (GNNs). It first utilizes the Graph Convolutional Networks (GCNs) (Kipf and Welling, 2017)
to model the topological connection information, and then selectively samples each entity's neighborhood, aiming at retaining the most informative neighbors towards entity alignment. One of the key challenges here is how to accurately estimate the similarity of any two entities' sampled neighborhood. NMN addresses this challenge by designing a discriminative neighbor matching module to jointly compute the neighbor differences between the sampled subgraph pairs through a cross-graph attention mechanism. Note that we mainly focus on the neighbor relevance in the neighborhood sampling and matching modules, while the neighbor connections are modeled by GCNs. We show that, by integrating the neighbor connection information and the neighbor relevance information, NMN can effectively align entities from real-world KGs with neighborhood heterogeneity.
We evaluate NMN by applying it to benchmark datasets DBP15K (Sun et al., 2017) and DWY100K (Sun et al., 2018), and a sparse variant of DBP15K. Experimental results show that NMN achieves the best and more robust performance over state-of-the-arts. This paper makes the following technical contributions. It is the first to:
- employ a new graph sampling strategy for identifying the most informative neighbors towards entity alignment (Sec. 3.3).
- exploit a cross-graph attention-based matching mechanism to jointly compare discriminative subgraphs of two entities for robust entity alignment (Sec. 3.4).
Embedding-based entity alignment. In recent years, embedding-based methods have emerged as viable means for entity alignment. Early works in the area utilize TransE (Bordes et al., 2013) to embed KG structures, including MTransE (Chen et al., 2017), JAPE (Sun et al., 2017), ITPransE (Zhu et al., 2017), BootEA (Sun et al., 2018), NAEA (Zhu et al., 2019) and OTEA (Pei et al., 2019b). Some more recent studies use GNNs to model the structures of KGs, including GCN-Align (Wang et al., 2018), GMNN (Xu et al., 2019), RDGCN (Wu et al., 2019a), AVR-GCN (Ye et al., 2019), and HGCN-JE (Wu et al., 2019b). Besides the structural information, some recent methods like KDCoE (Chen et al., 2018), AttrE (Trisedya et al., 2019), MultiKE (Zhang et al., 2019) and HMAN
6478
(Yang et al., 2019) also utilize additional information like Wikipedia entity descriptions and attributes to improve entity representations.
However, all the aforementioned methods ignore the neighborhood heterogeneity of KGs. MuGNN (Cao et al., 2019) and AliNet (Sun et al., 2020) are two most recent efforts for addressing this issue. While promising, both models still have drawbacks. MuGNN requires both pre-aligned entities and relations as training data, which can have expensive overhead for training data labeling. AliNet considers all one-hop neighbors of an entity to be equally important when aggregating information. However, not all one-hop neighbors contribute positively to characterizing the target entity. Thus, considering all of them without careful selection can introduce noise and degrade the performance. NMN avoids these pitfalls. With only a small set of pre-aligned entities as training data, NMN chooses the most informative neighbors for entity alignment.
Graph neural networks. GNNs have recently been employed for various NLP tasks like semantic role labeling (Marcheggiani and Titov, 2017) and machine translation (Bastings et al., 2017). GNNs learn node representations by recursively aggregating the representations of neighboring nodes. There are a range of GNN variants, including the Graph Convolutional Network (GCN) (Kipf and Welling, 2017), the Relational Graph Convolutional Network (Schlichtkrull et al., 2018), the Graph Attention Network (Velicković et al., 2018). Giving the powerful capability for modeling graph structures, we also leverage GNNs to encode the structural information of KGs (Sec. 3.2).
Graph matching. The similarity of two graphs can be measured by exact matching (graph isomorphism) (Yan et al., 2004) or through structural information like the graph editing distance (Raymond et al., 2002). Most recently, the Graph Matching Network (GMN) (Li et al., 2019b) computes a similarity score between two graphs by jointly reasoning on the graph pair through cross-graph attention-based matching. Inspired by GMN, we design a cross-graph neighborhood matching module (Sec. 3.4) to capture the neighbor differences between two entities' neighborhoods.
Graph sampling. This technique samples a subset of vertices or edges from the original graph. Some of the popular sampling approaches include vertex-, edge- and traversal-based sampling (Hu
and Lau, 2013). In our entity alignment framework, we propose a vertex sampling method to select informative neighbors and to construct a neighborhood subgraph for each entity.
Formally, we represent a KG as  $G = (E, R, T)$ , where  $E, R, T$  denote the sets of entities, relations and triples respectively. Without loss of generality, we consider the task of entity alignment between two KGs,  $G_1$  and  $G_2$ , based on a set of pre-aligned equivalent entities. The goal is to find pairs of equivalent entities between  $G_1$  and  $G_2$ .
3.1 Overview of NMN
As highlighted in Sec. 1, the neighborhood heterogeneity and noisy common neighbors of real-world KGs make it difficult to capture useful information for entity alignment. To tackle these challenges, NMN first leverages GCNs to model the neighborhood topology information. Next, it employs neighborhood sampling to select the more informative neighbors. Then, it utilizes a cross-graph matching module to capture neighbor differences.
As depicted in Figure 2, NMN takes as input two KGs,  $G_{1}$  and  $G_{2}$ , and produces embeddings for each candidate pair of entities,  $e_{1}$  and  $e_{2}$ , so that entity alignment can be performed by measuring the distance,  $d(e_{1},e_{2})$ , of the learned embeddings. It follows a four-stage processing pipeline: (1) KG structure embedding, (2) neighborhood sampling, (3) neighborhood matching, and (4) neighborhood aggregation for generating embeddings.
3.2 KG Structure Embedding
To learn the KG structure embeddings, NMN utilizes multi-layered GCNs to aggregate higher degree neighboring structural information for entities.
NMNs uses pre-trained word embeddings to initialize the GCN. This strategy is shown to be effective in encoding the semantic information of entity names in prior work (Xu et al., 2019; Wu et al., 2019a). Formally, let  $G_{1} = (E_{1}, R_{1}, T_{1})$  and  $G_{2} = (E_{2}, R_{2}, T_{2})$  be two KGs to be aligned, we put  $G_{1}$  and  $G_{2}$  together as one big input graph to NMN. Each GCN layer takes a set of node features as input and updates the node representations as:
$$
\boldsymbol {h} _ {i} ^ {(l)} = \operatorname {R e L U} \left(\sum_ {j \in N _ {i} \cup \{i \}} \frac {1}{\epsilon_ {i}} \mathbf {W} ^ {(l)} \boldsymbol {h} _ {j} ^ {(l - 1)}\right) \tag {1}
$$
6479
Figure 2: Overall architecture and processing pipeline of Neighborhood Matching Network (NMN).
where  $\{\pmb{h}_1^{(l)},\pmb{h}_2^{(l)},\dots,\pmb{h}_n^{(l)}|\pmb{h}_i^{(l)}\in \mathbb{R}^{d^{(l)}}\}$  is the output node (entity) features of  $l$  -th GCN layer,  $\epsilon_{i}$  is the normalization constant,  $N_{i}$  is the set of neighbor indices of entity  $i$  , and  $\mathbf{W}^{(l)}\in \mathbb{R}^{d^{(l)}\times d^{(l - 1)}}$  is a layer-specific trainable weight matrix.
To control the accumulated noise, we also introduce highway networks (Srivastava et al., 2015) to GCN layers, which can effectively control the noise propagation across GCN layers (Rahimi et al., 2018; Wu et al., 2019b).
3.3 Neighborhood Sampling
The one-hop neighbors of an entity are key to determine whether the entity should be aligned with other entities. However, as we have discussed in Sec. 1, not all one-hop neighbors contribute positively for entity alignment. To choose the right neighbors, we apply a down-sampling process to select the most informative entities towards the central target entity from its one-hop neighbors.
Recall that we use pre-trained word embeddings of entity names to initialize the input node features of GCNs. As a result, the entity embeddings learned by GCNs contain rich contextual information for both the neighboring structures and the entity semantics. NMN exploits such information to sample informative neighbors, i.e., neighbors that are more contextually related to the central entity are more likely to be sampled. Our key insight is that the more often a neighbor and the central (or target) entity appear in the same context, the more representative and informative the neighbor is towards the central entity. Since the contexts of two equivalent entities in real-world corpora are usually similar, the stronger a neighbor is contextually related to the target entity, the more alignment clues the neighbor is likely to offer. Experimental results in Sec. 5.3 confirm this observation.
Formally, given an entity  $e_i$ , the probability to
sample its one-hop neighbor  $e_{i-j}$  is determined by:
$$
\begin{array}{l} p \left(\boldsymbol {h} _ {i - j} \mid \boldsymbol {h} _ {i}\right) = \operatorname {s o f t m a x} \left(\boldsymbol {h} _ {i} \mathbf {W} _ {s} \boldsymbol {h} _ {i - j} ^ {T}\right) \\ = \frac {\exp \left(\boldsymbol {h} _ {i} \mathbf {W} _ {s} \boldsymbol {h} _ {i - j} ^ {T}\right)}{\sum_ {k \in N _ {i}} \exp \left(\boldsymbol {h} _ {i} \mathbf {W} _ {s} \boldsymbol {h} _ {i - k} ^ {T}\right)} \tag {2} \\ \end{array}
$$
where  $N_{i}$  is the one-hop neighbor index of central entity  $e_i$ ,  $\pmb{h}_i$  and  $\pmb{h}_{i-j}$  are learned embeddings for entities  $e_i$  and  $e_{i-j}$  respectively, and  $\mathbf{W}_s$  is a shared weight matrix.
By selectively sampling one-hop neighbors, NMN essentially constructs a discriminative subgraph of neighborhood for each entity, which can enable more accurate alignment through neighborhood matching.
3.4 Neighborhood Matching
The neighborhood subgraph, produced by the sampling process, determines which neighbors of the target entity should be considered in the later stages. In other words, later stages of the NMN processing pipeline will only operate on neighbors within the subgraph. In the neighborhood matching stage, we wish to find out, for each candidate entity in the counterpart KG, which neighbors of that entity are closely related to a neighboring node within the subgraph of the target entity. Such information is essential for deciding whether two entities (from two KGs) should be aligned.
As discussed in Sec. 3.3, equivalent entities tend to have similar contexts in real-world corpora; therefore, their neighborhoods sampled by NMN should be more likely to be similar. NMN exploits this observation to estimate the similarities of the sampled neighborhoods.
Candidate selection. Intuitively, for an entity  $e_i$  in  $E_1$ , we need to compare its sampled neighborhood subgraph with the subgraph of each candidate entity in  $E_2$  to select an optimal alignment entity. Exhaustively trying all possible entities of  $E_2$  would be prohibitively expensive for large
6480
real-world KGs. To reduce the matching overhead, NMN takes a low-cost approximate approach. To that end, NMN first samples an alignment candidate set  $\mathcal{C}_i = \{c_{i_1}, c_{i_2}, \dots, c_{it} | c_{i_k} \in E_2\}$  for  $e_i$  in  $E_1$ , and then calculates the subgraph similarities between  $e_i$  and these candidates. This is based on an observation that the entities in  $E_2$  which are closer to  $e_i$  in the embedding space are more likely to be aligned with  $e_i$ . Thus, for an entity  $e_j$  in  $E_2$ , the probability that it is sampled as a candidate for  $e_i$  can be calculated as:
$$
p (\boldsymbol {h} _ {j} | \boldsymbol {h} _ {i}) = \frac {\exp \left(\| \boldsymbol {h} _ {i} - \boldsymbol {h} _ {j} \| _ {L _ {1}}\right)}{\sum_ {k \in E _ {2}} \exp \left(\| \boldsymbol {h} _ {i} - \boldsymbol {h} _ {k} \| _ {L _ {1}}\right)} \tag {3}
$$
Cross-graph neighborhood matching. Inspired by recent works in graph matching (Li et al., 2019b), our neighbor matching module takes a pair of subgraphs as input, and computes a cross-graph matching vector for each neighbor, which measures how well this neighbor can be matched to any neighbor node in the counterpart. Formally, let  $(e_i, c_{i_k})$  be an entity pair to be measured, where  $e_i \in E_1$  and  $c_{i_k} \in E_2$  is one of the candidates of  $e_i$ ,  $p$  and  $q$  are two neighbors of  $e_i$  and  $c_{i_k}$ , respectively. The cross-graph matching vector for neighbor  $p$  can be computed as:
$$
a _ {p q} = \frac {\exp \left(\boldsymbol {h} _ {p} \cdot \boldsymbol {h} _ {q}\right)}{\sum_ {q ^ {\prime} \in N _ {i _ {k}} ^ {s}} \exp \left(\boldsymbol {h} _ {p} \cdot \boldsymbol {h} _ {q ^ {\prime}}\right)} \tag {4}
$$
$$
\boldsymbol {m} _ {p} = \sum_ {q \in N _ {i _ {k}} ^ {s}} a _ {p q} (\boldsymbol {h} _ {p} - \boldsymbol {h} _ {q}) \tag {5}
$$
where  $a_{pq}$  are the attention weights,  $m_p$  is the matching vector for  $p$ , and it measures the difference between  $h_p$  and its closest neighbor in the other subgraph,  $N_{i_k}^s$  is the sampled neighbor set of  $c_{i_k}$ ,  $h_p$  and  $h_q$  are the GCN-output embeddings for  $p$  and  $q$  respectively.
Then, we concatenate neighbor  $p$ 's GCN-output embeddings with weighted matching vector  $m_p$ :
$$
\hat {\boldsymbol {h}} _ {p} = \left[ \boldsymbol {h} _ {p} \| \beta * \boldsymbol {m} _ {p} \right] \tag {6}
$$
For each target neighbor in a neighborhood subgraph, the attention mechanism in the matching module can accurately detect which of the neighbors in the subgraph of another KG is most likely to match the target neighbor. Intuitively, the matching vector  $m_{p}$  captures the difference between the two closest neighbors. When the representations of the two neighbors are similar, the matching vector
tends to be a zero vector so that their representations stay similar. When the neighbor representations differ, the matching vector will be amplified through propagation. We find this matching strategy works well for our problem settings.
3.5 Neighborhood Aggregation
In the neighborhood aggregation stage, we combine the neighborhood connection information (learned at the KG structure embedding stage) as well as the output of the matching stage (Sec. 3.4) to generate the final embeddings used for alignment.
Specifically, for entity  $e_i$ , we first aggregate its sampled neighbor representations  $\{\hat{h}_p\}$ . Inspired by the aggregation method in (Li et al., 2016), we compute a neighborhood representation for  $e_i$  as:
$$
\boldsymbol {g} _ {i} = \left(\sum_ {p \in N _ {i} ^ {s}} \sigma \left(\hat {\boldsymbol {h}} _ {p} \mathbf {W} _ {\text {g a t e}}\right) \cdot \hat {\boldsymbol {h}} _ {p}\right) \mathbf {W} _ {N} \tag {7}
$$
Then, we concatenate the central entity  $e_i$ 's GCN-output representation  $h_i$  with its neighborhood representation to construct the matching oriented representation for  $e_i$ :
$$
\boldsymbol {h} _ {i} ^ {\text {m a t c h}} = \left[ \boldsymbol {g} _ {i} \| \boldsymbol {h} _ {i} \right] \tag {8}
$$
3.6 Entity Alignment and Training
Pre-training. As discussed in Sec. 3.3, our neighborhood sampling is based on the GCN-output entity embeddings. Therefore, we first pretrain the GCN-based KG embedding model to produce quality entity representations. Specifically, we measure the distance between two entities to determine whether they should be aligned:
$$
\tilde {d} \left(e _ {1}, e _ {2}\right) = \left\| \boldsymbol {h} _ {e _ {1}} - \boldsymbol {h} _ {e _ {2}} \right\| _ {L _ {1}} \tag {9}
$$
The objective of the pre-trained model is:
$$
\tilde {L} = \sum_ {(i, j) \in \mathbb {L}} \sum_ {\left(i ^ {\prime}, j ^ {\prime}\right) \in \mathbb {L} ^ {\prime}} \max  \{0, \tilde {d} (i, j) - \tilde {d} \left(i ^ {\prime}, j ^ {\prime}\right) + \gamma \} \tag {10}
$$
where  $\gamma > 0$  is a margin hyper-parameter;  $\mathbb{L}$  is our alignment seeds and  $\mathbb{L}'$  is the set of negative aligned entity pairs generated by nearest neighbor sampling (Kotnis and Nastase, 2017).
Overall training objective. The pre-training phase terminates once the entity alignment performance has converged to be stable. We find that after this stage, the entity representations given by the GCN are sufficient for supporting the neighborhood sampling and matching modules. Hence,
6481
Figure 3: Distribution of difference in the size of neighborhoods of aligned entity pairs on DBP15K $_{ZH-EN}$ .
we replace the loss function of NMN after the pretraining phase as:
$$
L = \sum_ {(r, t) \in \mathbb {L}} \sum_ {\left(r ^ {\prime}, t ^ {\prime}\right) \in \mathbb {C}} \max  \{0, d (r, t) - d \left(r ^ {\prime}, t ^ {\prime}\right) + \gamma \} \tag {11}
$$
$$
d (r, t) = \left\| \boldsymbol {h} _ {r} ^ {\text {m a t c h}} - \boldsymbol {h} _ {t} ^ {\text {m a t c h}} \right\| _ {L _ {1}} \tag {12}
$$
where the negative alignments set  $\mathbb{C} = \{(r', t') | (r' = r \wedge t' \in \mathcal{C}_r) \vee (t' = t \wedge r' \in \mathcal{C}_t)\}$  is made up of the alignment candidate sets of  $r$  and  $t$ ,  $\mathcal{C}_r$  and  $\mathcal{C}_t$  are generated in the candidate selection stage described in Sec. 3.4.
Note that our sampling process is non-differentiable, which corrupts the training of weight matrix  $\mathbf{W}_s$  in Eq. 2. To avoid this issue, when training  $\mathbf{W}_s$ , instead of direct sampling, we aggregate all the neighbor information by intuitive weighted summation:
$$
\boldsymbol {g} _ {i} ^ {w} = \left(\sum_ {p \in N _ {i}} \alpha_ {i p} \cdot \sigma \left(\hat {\boldsymbol {h}} _ {p} \mathbf {W} _ {\text {g a t e}}\right) \cdot \hat {\boldsymbol {h}} _ {p}\right) \mathbf {W} _ {N} \tag {13}
$$
where  $\alpha_{ip}$  is the aggregation weight for neighbor  $p$ , and is the sampling probability  $p(\boldsymbol{h}_p|\boldsymbol{h}_i)$  for  $p$  given by Eq. 2. Since the aim of training  $\mathbf{W}_s$  is to let the learned neighborhood representations of aligned entities to be as similar as possible, the objective is:
$$
L _ {w} = \sum_ {(r, t) \in \mathbb {L}} \| \boldsymbol {g} _ {r} ^ {w} - \boldsymbol {g} _ {t} ^ {w} \| _ {L _ {1}} \tag {14}
$$