SlowGuess/NAACL_2022_OCR huggingface dataset:
Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In Proceedings of the 6th Conference on Natural Language Learning, CoNLL 2002, Held in cooperation with COLING 2002, Taipei, Taiwan, 2002. ACL.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 142-147. ACL.
Moemmur Shahzad, Ayesha Amin, Diego Esteves, and Axel-Cyrille Ngonga Ngomo. 2021. Inferner: an attentive model leveraging the sentence-level information for named entity recognition in microblogs. In Proceedings of the Thirty-Fourth International Florida Artificial Intelligence Research Society Conference, North Miami Beach, Florida, USA, May 17-19, 2021.
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2054-2064. Association for Computational Linguistics.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Antonio Toral and Rafael Muñoz. 2006. A proposal to automatically build and maintain gazetteers for named entity recognition by using Wikipedia. In Proceedings of the Workshop on NEW TEXT Wikis and blogs and other dynamic text sources.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: deep
contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6442-6454. Association for Computational Linguistics.
Ayah Zirikly. 2015. Cross-lingual transfer of named entity recognizers without parallel corpora. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 2: Short Papers, pages 390-396. The Association for Computer Linguistics.
A NER Datasets
OWNER. Table 6 shows detailed statistics about the OWNER dataset. LOWER is used as our main training dataset, and additionally we use it for cross-lingual transfer of NER models. LOWER is constructed in 7 different languages from the corresponding Wikipedia dumps, where we extract the articles, which were then parsed to remove markup and extract sentences with their interlinks (links to other articles). We then mapped the interlinks in each sentence to the Wikidata KB then resolved them to our NER taxonomy as shown in Table 6.
We filter sentences using two strategies. Taking advantage of Wikipedia's well-formed text, we applied a Regex-based NER method to identify sentences containing named entities that were not linked, and removed them. This removes long and high-context sentences that contain references to many entities. Additionally we also removed any sentence where the links could not be resolved to Wikidata entities. This step discards over  $90\%$  of the sentences. This process yields short and low-context sentences, which represents a realistic NER dataset for cross-domain transfer, especially for cases like Web search or Q&A.
Below are shown example sentences from the EN-OWNER training set. The different entities are marked in colors according to their entity type.
- anthology is a compilation album by new zealand singer songwriter and multi instrumentalist bie runga
- his recordings include several issues for hyperion records, including music of benjamin britten, emmanuel chabrier, maurice duru and henry purcell.
together with the nearby village revetical it has a population (statistics norway 2005) of 1,902.
MSQ. This dataset aims to reflect NER in the Q&A domain, and is based on the MS-MARCO dataset (Nguyen et al., 2016) which contains over a million questions. We first construct templates from the questions by applying an existing NER system (e.g., spaCy) to identify entities in the questions. We then use our gazetteer to map the entities to their NER types to create slotted templates, e.g., "when did [[iphone]] come out" becomes "when did <PROD> come out". The templates are then aggregated by frequency. This process results in 3,445 unique question templates in English language, which we automatically translated into the remaining languages (NL, ES, RU, TR, KO, FA). While the NER system
cannot correctly identify many entities, the most frequent templates are reliable. Finally, we generate MSQ-NER by slotting the templates that have a frequency of  $\geq 5$  with random entities from the Wikipedia KB with the same class. Each template is slotted with the same number of times it appeared in MS-MARCO in order to maintain the same relative distribution as the original data. This results in 17,868 questions e.g., "when did [[xbox]] come out", which we use as a test set. Table 6 shows the stats for the MSQ dataset in the different languages.
The examples below show MSQ test instances. The different entities are marked in different colors according to their entity type.
where was benjamin mwangata born
- where is trenton-robbinsville airport ca
- how old is rafi ibn harthama
- what county is downtown washington dc
- how much doesCHAT pay
Gazetteers. Table 5 shows the gazetteers extracted from the entity titles in Wikidata (instances of Wikidata types that correspond to the NER taxonomy). We use gazetteers to aid knowledge transfer for our approach.
lang.	#entries	PER	LOC	GRP	CW	CORP	PROD
ES	2.3M	0.61	0.21	0.05	0.10	0.01	0.01
NL	2.4M	0.55	0.31	0.03	0.09	0.01	0.01
RU	1.7M	0.57	0.29	0.04	0.09	0.01	0.01
TR	393k	0.44	0.36	0.05	0.11	0.02	0.02
KO	332K	0.47	0.23	0.07	0.18	0.03	0.03
FA	554k	0.41	0.42	0.03	0.11	0.02	0.02
Table 5: Per-language statistics for the Wikidata gazetteers.
B NER Approaches Setup
Here we describe technical details on how we trained both NER approaches in this work. Our approach and the baseline are implemented in PyTorch (Paszke et al., 2019). We train our models on 4 NVIDIA Tesla V100 GPUs, with approximately 8-10 mins per epoch. The code repository will be released upon paper publication.
- Baseline-XLMR: We fine tune XLM-RoBERTa (XLMR) (Conneau et al., 2020) baseline for the NER task using the AdamW optimizer (Loshchilov and Hutter, 2019), with a learning rate of  $lr = 1e - 5$  and weight decay of  $w = 0.01$ . For XLMR we make use of
dataset	lang	split	instances	PER	LOC	GRP	PROD	CW	CORP
LOWER	English (EN)	train	15200	0.229	0.203	0.152	0.124	0.159	0.132
dev	800	0.236	0.19	0.154	0.12	0.143	0.157
test	10000	0.225	0.206	0.155	0.126	0.155	0.133
Dutch (NL)	train	15200	0.197	0.247	0.148	0.132	0.15	0.126
dev	800	0.183	0.258	0.141	0.119	0.157	0.141
test	10000	0.192	0.245	0.151	0.134	0.153	0.126
Spanish (ES)	train	15200	0.209	0.219	0.144	0.135	0.164	0.129
dev	800	0.21	0.233	0.143	0.131	0.163	0.12
test	10000	0.209	0.216	0.144	0.132	0.169	0.131
Russian (RU)	train	15200	0.185	0.211	0.151	0.148	0.163	0.143
dev	800	0.184	0.212	0.145	0.145	0.161	0.153
test	10000	0.181	0.212	0.155	0.147	0.162	0.143
Turkish (TR)	train	15200	0.189	0.248	0.154	0.137	0.153	0.119
dev	800	0.186	0.282	0.134	0.127	0.153	0.119
test	10000	0.182	0.253	0.154	0.135	0.152	0.125
Korean (KO)	train	15200	0.184	0.254	0.144	0.125	0.158	0.135
dev	800	0.205	0.248	0.141	0.136	0.151	0.12
test	10000	0.183	0.26	0.144	0.126	0.153	0.134
Farsi (FA)	train	15200	0.188	0.248	0.141	0.13	0.162	0.131
dev	800	0.166	0.267	0.135	0.129	0.171	0.132
test	10000	0.191	0.245	0.14	0.127	0.163	0.134
MSQ	English (EN)	test	17868	0.240	0.554	0.032	0.025	0.115	0.036
Spanish (ES)	test	17937	0.226	0.582	0.030	0.024	0.105	0.032
Dutch (NL)	test	17387	0.242	0.555	0.032	0.024	0.114	0.034
Russian (RU)	test	17551	0.232	0.561	0.033	0.024	0.114	0.036
Turkish (TR)	test	17405	0.246	0.544	0.033	0.025	0.116	0.037
Korean (KO)	test	17874	0.245	0.545	0.033	0.025	0.115	0.036
Farsi (FA)	test	16960	0.238	0.560	0.032	0.022	0.112	0.036
Table 6: Detailed breakdown of the ratio of entities for the different NER classes for LOWER and MSQ datasets for the different evaluation languages. Note that a sentences contains one or more entities, which may be of different types.
the implementation provided by the Transformer framework (Wolf et al., 2019). We first perform a linear warmup stage, which is done for a certain number of steps that corresponds to  $10\%$  of the number of batches. XLMR model converge to their optimal performance around 10 epochs.
- Approach: We train our approach in two stages. This is mainly due to the fact that the text and gazetteer components having unaligned weights. XLMR has weights coming from a pre-trained model, whereas the gazetteer encoder has randomly initialized weights. We use the same optimizer as for the Baseline, namely AdamW. In
the first sage, we freeze the XLMR weights and use a more aggressive learning rate to train the LSTM gazetteer encoder with  $lr = 0.01$ . We run the first stage for 10 epochs, and then perform a joint optimization in the second stage with the same learning rate and weight decay parameters as for the XLMR baseline approach.