SlowGuess/ACL_2020_OCR huggingface dataset:
In general, our model is trained end-to-end after pre-training. During training, we use Eq. 11 as the main objective function, and, every 50 epochs, we tune  $\mathbf{W}_s$  using Eq. 14 as the objective function.
Datasets. Follow the common practice of recent works (Sun et al., 2018; Cao et al., 2019; Sun et al., 2020), we evaluate our model on DBP15K (Sun et al., 2017) and DWY100K (Sun et al., 2018) datasets, and use the same split with previous works,  $30\%$  for training and  $70\%$  for testing. To
Datasets	Ent.	Rel.	Tri.	Tri. Remain in S.
ZH-EN	ZH	66,469	2,830	153,929	26%
EN	98,125	2,317	237,674	100%
JA-EN	JA	65,744	2,043	164,373	41%
EN	95,680	2,096	233,319	100%
FR-EN	FR	66,858	1,379	192,191	45%
EN	105,889	2,209	278,590	100%
Table 1: Summary of DBP15K and S-DBP15k.
Datasets	Ent.	Rel.	Tri.
DBP-WD	DBpedia	100,000	330	463,294
Wikidata	100,000	220	448,774
DBP-YG	DBpedia	100,000	302	428,952
YAGO3	100,000	31	502,563
Table 2: Summary of DWY100K.
evaluate the performance of NMN in a more challenging setting, we also build a sparse dataset S-DBP15K based on DBP15K. Specifically, we randomly remove a certain proportion of triples in the non-English KG to increase the difference in neighborhood size for entities in different KGs. Table 1 gives the detailed statistics of DBP15K and S-DBP15K, and the information of DWY100K is exhibited in Table 2. Figure 3 shows the distribution of difference in the size of one-hop neighborhoods of aligned entity pairs. Our source code and datasets are freely available online.
Comparison models. We compare NMN against 12 recently proposed embedding-based alignment methods: MTransE (Chen et al., 2017), JAPE (Sun et al., 2017), IPTransE (Zhu et al., 2017), GCNAlign (Wang et al., 2018), BootEA (Sun et al., 2018), SEA (Pei et al., 2019a), RSN (Guo et al., 2019), MuGNN (Cao et al., 2019), KECG (Li et al., 2019a), AliNet (Sun et al., 2020), GMNN (Xu et al., 2019) and RDGCN (Wu et al., 2019a). The last two models also utilize entity names for alignment.
Model variants. To evaluate different components of our model, we provide two implementation variants of NMN: (1) NMN (w/o nbr-m), where we replace the neighborhood matching part by taking the average of sampled neighbor representations as the neighborhood representation; and (2) NMN (w/o nbr-s), where we remove the sampling process and perform neighborhood matching on all one-hop neighbors.
Implementation details. The configuration we use in the DBP15K and DWY100k datasets is:  $\beta = 0.1$ ,  $\gamma = 1.0$ , and we sample 5 neigh
<sup>1</sup>https://github.com/StephanieWyt/NMN
6482
bors for each entity in the neighborhood sampling stage (Sec. 3.3). For S-DBP15K, we set  $\beta$  to 1. We sample 3 neighbors for each entity in S-DBP15K $_{ZH-EN}$  and S-DBP15K $_{JA-EN}$ , and 10 neighbors in S-DBP15K $_{FR-EN}$ . NMN uses a 2-layer GCN. The dimension of hidden representations in GCN layers described in Sec. 3.2 is 300, and the dimension of neighborhood representation  $g_i$  described in Sec. 3.5 is 50. The size of the candidate set in Sec. 3.4 is 20 for each entity. The learning rate is set to 0.001.
To initialize entity names, for the DBP15K datasets, we first use Google Translate to translate all non-English entity names into English, and use pre-trained English word vectors glove.  $840B.300d^{2}$  to construct the initial node features of KGs. For the DWY100K datasets, we directly use the pretrained word vectors to initialize the nodes.
Metrics. Following convention, we use Hits@1 and Hits@10 as our evaluation metrics. A Hits@k score is computed by measuring the proportion of correctly aligned entities ranked in the top  $k$  list. A higher Hits@k score indicates better performance.
5.1 Performance on DBP15K and DWY100K
Table 3 reports the entity alignment performance of all approaches on DBP15K and DWY100K datasets. It shows that the full implementation of NMN significantly outperforms all alternative approaches.
Structured-based methods. The top part of the table shows the performance of the state-of-the-art structure-based models which solely utilize structural information. Among them, BootEA delivers the best performance where it benefits from more training instances through a bootstrapping process. By considering the structural heterogeneity, MuGNN and AliNet outperform most of other structure-based counterparts, showing the importance of tackling structural heterogeneity.
Entity name initialization. The middle part of Table 3 gives the results of embedding-based models that use entity name information along with structural information. Using entity names to initialize node features, the GNN-based models, GMNN and RDGCN, show a clear improvement over structure-based models, suggesting that entity
names provide useful clues for entity alignment. In particular, GMNN achieves the highest Hits@10 on the DWY100K datasets, which are the only monolingual datasets (in English) in our experiments. We also note that, GMNN pre-screens a small candidate set for each entity based on the entity name similarity, and only traverses this candidate set during testing and calculating the Hits@k scores.
NMN vs. its variants. The bottom part of Table 3 shows the performance of NMN and its variants. Our full NMN implementation substantially outperforms all baselines across nearly all metrics and datasets by accurately modeling entity neighborhoods through neighborhood sampling and matching and using entity name information. Specifically, NMN achieves the best Hits@1 score on  $\mathrm{DBP15K_{ZH - EN}}$ , with a gain of  $2.5\%$  compared with RDGCN, and  $5.4\%$  over GMNN. Although RDGCN employs a dual relation graph to model the complex relation information, it does not address the issue of neighborhood heterogeneity. While GMNN collects all one-hop neighbors to construct a topic entity graph for each entity, its strategy might introduce noises since not all one-hop neighbors are favorable for entity alignment.
When comparing NMN and NMN (w/o nbr-m), we can observe around a  $2.5\%$  drop in Hits@1 and a  $0.6\%$  drop in Hits@10 on average, after removing the neighborhood matching module. Specifically, the Hits@1 scores between NMN and NMN (w/o nbr-m) differ by  $3.9\%$  on DBP15K $_{FR-EN}$ . These results confirm the effectiveness of our neighborhood matching module in identifying matching neighbors and estimating the neighborhood similarity.
Removing the neighbor sampling module from NMN, i.e., NMN (w/o nbr-s), leads to an average performance drop of  $0.3\%$  on Hits@1 and  $1\%$  on Hits@10 on all the datasets. This result shows the important role of our sampling module in filtering irrelevant neighbors.
When removing either the neighborhood matching module (NMN (w/o nbr-m)) or sampling module (NMN (w/o nbr-s)) from our main model, we see a substantially larger drop in both Hits@1 and Hits@10 on DBP15K than on DWY100K. One reason is that the heterogeneity problem in DBP15K is more severe than that in DWY100K. The average proportion of aligned entity pairs that have a different number of neighbors is  $89\%$  in DBP15K compared to  $84\%$  in DWY100K. These results show
<sup>2</sup> http://nlp.stanford.edu/projects/glove/
6483
Models	DBPZH-EN	DBPJA-EN	DBPFR-EN	DBP-WD	DBP-YG
Hits@1	Hits@10	Hits@1	Hits@10	Hits@1	Hits@10	Hits@1	Hits@10	Hits@1	Hits@10
MTransE (Chen et al., 2017)	30.8	61.4	27.9	57.5	24.4	55.6	28.1	52.0	25.2	49.3
JAPE (Sun et al., 2017)	41.2	74.5	36.3	68.5	32.4	66.7	31.8	58.9	23.6	48.4
IPTransE (Zhu et al., 2017)	40.6	73.5	36.7	69.3	33.3	68.5	34.9	63.8	29.7	55.8
GCN-Align (Wang et al., 2018)	41.3	74.4	39.9	74.5	37.3	74.5	50.6	77.2	59.7	83.8
SEA (Pei et al., 2019a)	42.4	79.6	38.5	78.3	40.0	79.7	51.8	80.2	51.6	73.6
RSN (Guo et al., 2019)	50.8	74.5	50.7	73.7	51.6	76.8	60.7	79.3	68.9	87.8
KECG (Li et al., 2019a)	47.8	83.5	49.0	84.4	48.6	85.1	63.2	90.0	72.8	91.5
MuGNN (Cao et al., 2019)	49.4	84.4	50.1	85.7	49.5	87.0	61.6	89.7	74.1	93.7
AliNet (Sun et al., 2020)	53.9	82.6	54.9	83.1	55.2	85.2	69.0	90.8	78.6	94.3
BootEA (Sun et al., 2018)	62.9	84.8	62.2	85.4	65.3	87.4	74.8	89.8	76.1	89.4
GMNN (Xu et al., 2019)	67.9	78.5	74.0	87.2	89.4	95.2	93.0	99.6	94.4	99.8
RDGCN (Wu et al., 2019a)	70.8	84.6	76.7	89.5	88.6	95.7	97.9	99.1	94.7	97.3
NMN	73.3	86.9	78.5	91.2	90.2	96.7	98.1	99.2	96.0	98.2
w/o nbr-m	71.1	86.7	75.4	90.4	86.3	95.8	96.0	98.4	95.0	97.8
w/o nbr-s	73.0	85.6	77.9	88.8	89.9	95.7	98.0	99.0	95.9	98.1
Table 3: Performance on DBP15K and DWY100K.
Models	ZH-EN	JA-EN	FR-EN
Hits@1	Hits@10	Hits@1	Hits@10	Hits@1	Hits@10
BootEA	12.2	27.5	27.8	52.6	32.7	53.2
GMNN	47.5	68.3	58.8	78.2	75.0	90.9
RDGCN	60.7	74.6	69.3	82.9	83.6	92.6
NMN	62.0	75.1	70.3	84.4	86.3	94.0
w/o nbr-m	52.0	71.1	62.1	82.7	80.0	92.0
w/o nbr-s	60.9	74.1	70.7	84.5	86.5	94.2
Table 4: Performance on S-DBP15K.
that our sampling and matching modules are particularly important, when the neighborhood sizes of equivalent entities greatly differ and especially there may be few common neighbors in their neighborhoods.
5.2 Performance on S-DBP15K
On the more sparse and challenging datasets S-DBP15K, we compare our NMN model with the strongest structure-based model, BootEA, and GNN-based models, GMNN and RDGCN, which also utilize the entity name initialization.
Baseline models. In Table 4, we can observe that all models suffer a performance drop, where BootEA endures the most significant drop. With the support of entity names, GMNN and RDGCN achieve better performances over BootEA. These results show when the alignment clues are sparse, structural information alone is not sufficient to support precise comparisons, and the entity name semantics are particularly useful for accurate alignment in such case.
NMN. Our NMN outperforms all three baselines on all sparse datasets, demonstrating the effectiveness and robustness of NMN. As discussed in Sec. 1, the performances of existing embedding-based methods decrease significantly as the gap of equivalent entities' neighborhood sizes increases.
Specifically, on DBP15K $_{ZH-EN}$ , our NMN outperforms RDGCN, the best-performing baseline, by a large margin, achieving  $65\%$ ,  $53\%$  and  $48\%$  on Hits@1 on the entity pairs whose number of neighbors differs by more than 10, 20 and 30, respectively.
Sampling and matching strategies. When we compare NMN and NMN (w/o nbr-m) on the S-DBP15K, we can see a larger average drop in Hits@1 than on the DBP15K (8.2% vs. 3.1%). The result indicates that our neighborhood matching module plays a more important role on the more sparse dataset. When the alignment clues are less obvious, our matching module can continuously amplify the neighborhood difference of an entity pair during the propagation process. In this way, the gap between the equivalent entity pair and the negative pairs becomes larger, leading to correct alignment.
Compared with NMN, removing sampling module does hurt NMN in both Hits@1 and Hits@10 on S-DBP15K $_{ZH-EN}$ . But, it is surprising that NMN (w/o nbr-s) delivers slightly better results than NMN on S-DBP15K $_{JA-EN}$  and S-DBP15K $_{FR-EN}$ . Since the average number of neighbors of entities in S-DBP15K is much less than that in the DBP15K datasets. When the number of neighbors is small, the role of sampling will be unstable. In addition, our sampling method is relatively simple. When the alignment clues are very sparse, our strategy may not be robust enough. We will explore more adaptive sampling method and scope in the future.
Impact of neighborhood sampling strategies. To explore the impact of neighborhood sampling
6484
Figure 4: Comparison between our neighborhood sampling strategy and random sampling on S-DBP15K.
Figure 5: Visualization of attention weights in the neighborhood matching module for the example of Paramount Pictures. The green and blue words are two pairs of equivalent neighbors.
strategies, we compare our NMN with a variant that uses random sampling strategy on S-DBP15K datasets. Figure 4 illustrates the Hits@1 of NMN using our designed graph sampling method (Sec. 3.3) and a random-sampling-based variant when sampling different number of neighbors. Our NMN consistently delivers better results compared to the variant, showing that our sampling strategy can effectively select more informative neighbors.
Impact of neighborhood sampling size. From Figure 4, for S-DBP15K $_{ZH-EN}$ , both models reach a performance plateau with a sampling size of 3, and using a bigger sampling size would lead to performance degradation. For S-DBP15K $_{JA-EN}$  and S-DBP15K $_{FR-EN}$ , we observe that our NMN performs similarly when sampling different number of neighbors. From Table 1, we can see that S-DBP15K $_{ZH-EN}$  is more sparse than S-DBP15K $_{JA-EN}$  and S-DBP15K $_{FR-EN}$ . All models deliver much lower performance on S-DBP15K $_{ZH-EN}$ . Therefore, the neighbor quality of this dataset might be poor, and a larger sampling size will introduce more noise. On the other hand, the neighbors in JA-EN and FR-EN datasets might be more informative. Thus, NMN is not sensitive to the sampling size on these two datasets.
How does the neighborhood matching module work? In an attempt to understand how our neighborhood matching strategy helps alignment, we visualize the attention weights in the neighborhood matching module. Considering an equivalent
entity pair in DBP15K $_{ZH-EN}$ , both of which indicate an American film studio Paramount Pictures. From Figure 5, we can see that the five neighbors sampled by our sampling module for each central entity are very informative ones for aligning the two central entities, such as the famous movies released by Paramount Pictures, the parent company and subsidiary of Paramount Pictures. This demonstrates the effectiveness of our sampling strategy again. Among the sampled neighbors, there are also two pairs of common neighbors (indicate Saving Private Ryan and Viacom). We observe that for each pair of equivalent neighbors, one neighbor can be particularly attended by its counterpart (the corresponding square has a darker color). This example clearly demonstrates that our neighborhood matching module can accurately estimate the neighborhood similarity by accurately detecting the similar neighbors.
We have presented NMN, a novel embedded-based framework for entity alignment. NMN tackles the ubiquitous neighborhood heterogeneity in KGs. We achieve this by using a new sampling-based approach to choose the most informative neighbors for each entity. As a departure from prior works, NMN simultaneously estimates the similarity of two entities, by considering both topological structure and neighborhood similarity. We perform extensive experiments on real-world datasets and compare NMN against 12 recent embedded-based methods. Experimental results show that NMN achieves the best and more robust performance, consistently outperforming competitive methods across datasets and evaluation metrics.
This work is supported in part by the National Hi-Tech R&D Program of China (No. 2018YFB1005100), the NSFC under grant agreements 61672057, 61672058 and 61872294, and a UK Royal Society International Collaboration Grant. For any correspondence, please contact Yansong Feng.
6485
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957-1967, Copenhagen, Denmark. Association for Computational Linguistics.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26, pages 2787-2795. Curran Associates, Inc.
Yixin Cao, Zhiyuan Liu, Chengjiang Li, Zhiyuan Liu, Juanzi Li, and Tat-Seng Chua. 2019. Multi-channel graph neural network for entity alignment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1452-1461, Florence, Italy. Association for Computational Linguistics.
Muhao Chen, Yingtao Tian, Kai-Wei Chang, Steven Skiena, and Carlo Zaniolo. 2018. Co-training embeddings of knowledge graphs and entity descriptions for cross-lingual entity alignment. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 3998-4004. ijcai.org.
Muhao Chen, Yingtao Tian, Mohan Yang, and Carlo Zaniolo. 2017. Multilingual knowledge graph embeddings for cross-lingual knowledge alignment. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 1511-1517. ijcai.org.
Lingbing Guo, Zequn Sun, and Wei Hu. 2019. Learning to exploit long-term relational dependencies in knowledge graphs. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2505-2514. PMLR.
Pili Hu and Wing Cheong Lau. 2013. A survey and taxonomy of graph sampling. arXiv preprint arXiv:1308.5865.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Bhushan Kotnis and Vivi Nastase. 2017. Analysis of the impact of negative sampling on link prediction in knowledge graphs. arXiv preprint arXiv:1708.06816.
Chengjiang Li, Yixin Cao, Lei Hou, Jiaxin Shi, Juanzi Li, and Tat-Seng Chua. 2019a. Semi-supervised entity alignment via joint knowledge embedding model and cross-graph model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2723-2732, Hong Kong, China. Association for Computational Linguistics.
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. 2019b. Graph matching networks for learning the similarity of graph structured objects. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3835-3845. PMLR.
Yujia Li, Richard Zemel, Marc Brockschmidt, and Daniel Tarlow. 2016. Gated graph sequence neural networks. In Proceedings of ICLR'16.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M. Suchanek. 2015. YAGO3: A knowledge base from multilingual wikipedias. In CIDR 2015, Seventh Biennial Conference on Innovative Data Systems Research, Asilomar, CA, USA, January 4-7, 2015, Online Proceedings. www.cidrdb.org.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506-1515, Copenhagen, Denmark. Association for Computational Linguistics.
Shichao Pei, Lu Yu, Robert Hoehndorf, and Xiangliang Zhang. 2019a. Semi-supervised entity alignment via knowledge graph embedding with awareness of degree difference. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 3130-3136. ACM.
Shichao Pei, Lu Yu, and Xiangliang Zhang. 2019b. Improving cross-lingual entity alignment via optimal transport. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3231-3237. ijcai.org.
Afshin Rahimi, Trevor Cohn, and Timothy Baldwin. 2018. Semi-supervised user geolocation via graph convolutional networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2009-2019, Melbourne, Australia. Association for Computational Linguistics.
John W. Raymond, Eleanor J. Gardiner, and Peter Willett. 2002. RASCAL: calculation of graph similarity using maximum common edge subgraphs. Comput. J., 45(6):631-644.
6486
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In *The Semantic Web - 15th International Conference, ESWC* 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, volume 10843 of Lecture Notes in Computer Science, pages 593-607. Springer.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint arXiv:1505.00387.
Zequn Sun, Wei Hu, and Chengkai Li. 2017. Cross-lingual entity alignment via joint attribute-preserving embedding. In *The Semantic Web - ISWC* 2017 - 16th International Semantic Web Conference, Vienna, Austria, October 21-25, 2017, Proceedings, Part I, volume 10587 of Lecture Notes in Computer Science, pages 628-644. Springer.
Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. 2018. Bootstrapping entity alignment with knowledge graph embedding. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4396-4402. ijcai.org.
Zequn Sun, Chengming Wang, Wei Hu, Muhao Chen, Jian Dai, Wei Zhang, and Yuzhong Qu. 2020. Knowledge graph alignment network with gated multi-hop neighborhood aggregation. In AAAI.
Bayu Distiawan Trisedya, Jianzhong Qi, and Rui Zhang. 2019. Entity alignment between knowledge graphs using attribute embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 297-304.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. In ICLR.
Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. 2018. Cross-lingual knowledge graph alignment via graph convolutional networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 349–357, Brussels, Belgium. Association for Computational Linguistics.
Yuting Wu, Xiao Liu, Yansong Feng, Zheng Wang, Rui Yan, and Dongyan Zhao. 2019a. Relation-aware entity alignment for heterogeneous knowledge graphs. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5278-5284. ijcai.org.
Yuting Wu, Xiao Liu, Yansong Feng, Zheng Wang, and Dongyan Zhao. 2019b. Jointly learning entity and relation representations for entity alignment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the