SlowGuess/NAACL_2022_OCR huggingface dataset:
For the CoNLL and mLOWER datasets, we trained separate models for both our approach and baselines. Table 2 shows the micro-averaged F1 scores across all NER classes. Table 2 shows that in the case of CoNLL, there is a saturation in terms of the improvement we achieve across the different languages. One explanation for this is that pretrained transformer models like XLMR, are already highly efficient on news corpora and can exploit the regularities on how named entities are mentioned in text. Hence, the difference when comparing the baseline and our approach on the CoNLL test set varies from  $1\%$  to  $2\%$ , for EN and NL, respectively.
Contrary to CoNLL, in the case of mLOWNER, which is a more diverse dataset and with sentences that do not follow the strict language style present in news corpora, we achieve significant gains over the baseline approach. The average gains are around  $10.6\%$  absolute improvement in terms of
<sup>6</sup>We experimented with this approach by replacing its GloVE and ELMo embeddings with XLMR contextual embeddings, however, the performance was suboptimal, and thus conclude that the these two embeddings in the respective languages are crucial for the model's performance.
7 Data was collected using the twitter streaming API on 12 July, 2021.
Figure 2: F1 class performance of our approach on mLOWNER test set. In brackets is shown the F1 absolute gains against the XLMR baseline.
micro averaged F1. Moreover, it is encouraging to note that for low-resource languages such as KO or FA, the gains are even higher. This shows that when pre-trained transformer models do not contain knowledge about a specific token, integrating external gazetteers through MoE, we can accurately predict the NER class of a token. The gains are not evenly distributed across the different NER classes. Figure 2 shows the absolute F1 gains over the XLMR baseline. The highest gains are achieved for the classes PROD, CW, CORP (with an average absolute increase of  $\mathrm{F}1 = +14.02$ ), which contain NEs that do not follow typical syntactic patterns as is the case for PER, LOC.
For the BaG baseline, we see a large gap. This is due to the inability to resolve ambiguous cases, which highlights the difficulty of the task, and that gazetteers alone lead to noisy NER.
Finally, comparing against SubTagger on the LOWER test set, our approach achieves an increase of  $\mathrm{F}1 = +5\%$ . Given that both models are trained on the same dataset, the improvements come mainly from the way we model our approach. Namely, the contributions can be attributed on the way how we incorporate gazetteer matches using the MoE, which allows the model to either weigh higher or downweight matches according to the token's NER tag accuracy.
Multi-stage training impact. We assessed the performance of our approach without the multi-stage training. The results are negligibly better than the baseline. Given that the GAZ encoder and MoE module are randomly initialized, the model relies solely on XLMR for NER. Furthermore, a low learning rate is not suitable for GAZ and MoE, while a higher  $lr$  is not suitable for XLMR, hence, the multi-stage training is appropriate.
MoE Impact. The combined representation computed via MoE is highly effective, especially for cross-domain transfer. Simply concatenating the text/gazetteer vectors, we note an average decrease of  $\mathrm{MD} = -22\%$  across all languages for MSQ. For in-domain evaluation, the difference is negligible. This is due to two reasons: (i) the model's representation for out-of-domain entity tokens are not fine-tuned for the task, and (ii) without MoE, spurious gazetteer matches cannot be discarded.
6.2 Cross-Domain Transfer Results
Cross-domain transfer for NER remains still challenging, due to the lack of domain specific data, privacy concerns in generating such data, or existing datasets having a narrow domain coverage.
Figure 3: Cross-domain mention detection scores for models trained separately on mLOWNER and CoNLL datasets and tested on mLOWNER and MSQ.
Figure 3 shows the cross-domain transfer results for models trained separately on CoNLL and mLOWER $^{8}$  and tested on the mLOWER and MSQ test sets. Since CoNLL has a different NER class taxonomy than MSQ and mLOWER, we report only MD performance.
Pre-trained CoNLL models: For MD performance of CoNLL pre-trained models we note two aspects. First, there is a high mismatch between the performance achieved on CoNLL and that of for mLOWNER and MSQ. It is evident that due to the narrow domain coverage of CoNLL (consisting of only news genre), the models have difficulties in detecting NE boundaries for out-of-domain datasets. Second, our approach consistently outperforms the XLMR baseline for both datasets. We obtain an absolute average improvement of  $\mathrm{MD} = +3\%$  and  $\mathrm{MD} = +24\%$  for mLOWNER and MSQ, respectively. This shows that when NER models are applied to a distant domain from their initial training data (e.g.
<sup>8</sup>CoNLL overlaps only in 3 languages with mLOWNER.
EN	NL	ES	RU	TR	KO	FA	Average
zero-shot setting
EN	0.888 (+13.33)	0.806 (+14.13)	0.795 (+15.19)	0.604 (+9.66)	0.729 (+21.01)	0.641 (+22.21)	0.681 (+20.58)	0.709 (+17.13)
NL	0.818 (+19.26)	0.875 (+8.33)	0.784 (+13.95)	0.628 (+17.04)	0.738 (+16.91)	0.640 (+18.95)	0.675 (+20.96)	0.714 (+17.84)
ES	0.803 (+18.18)	0.790 (+14.86)	0.847 (+10.12)	0.640 (+12.20)	0.715 (+17.18)	0.607 (+20.75)	0.699 (+20.92)	0.709 (+17.35)
RU	0.740 (+19.80)	0.630 (+6.69)	0.554 (-0.99)	0.782 (+8.97)	0.677 (+23.21)	0.592 (+17.54)	0.631 (+18.72)	0.637 (+14.16)
TR	0.733 (+16.18)	0.795 (+18.03)	0.694 (+10.49)	0.623 (+14.99)	0.859 (+10.67)	0.682 (+24.23)	0.690 (+21.97)	0.703 (+17.65)
KO	0.678 (+19.25)	0.724 (+23.64)	0.693 (+23.96)	0.610 (+14.12)	0.670 (+22.62)	0.854 (+12.78)	0.628 (+21.08)	0.667 (+20.78)
FA	0.744 (+19.61)	0.757 (+19.48)	0.720 (+15.68)	0.631 (+13.21)	0.711 (+20.15)	0.667 (+23.32)	0.820 (+11.94)	0.705 (+18.58)
few-shot (+500 instances), F1 = ▲8.0% increase compared to zero-shot
EN	0.888 (+13.33)	0.831 (+11.28)	0.804 (+13.42)	0.696 (+10.75)	0.806 (+15.71)	0.750 (+19.63)	0.734 (+14.21)	0.770 (+14.17)
NL	0.843 (+14.61)	0.875 (+8.33)	0.800 (+11.85)	0.690 (+8.73)	0.811 (+15.47)	0.761 (+18.41)	0.740 (+14.02)	0.774 (+13.85)
ES	0.830 (+14.43)	0.833 (+11.38)	0.847 (+10.12)	0.688 (+9.53)	0.806 (+15.08)	0.746 (+19.94)	0.752 (+14.53)	0.776 (+14.15)
RU	0.809 (+14.86)	0.815 (+13.48)	0.792 (+13.91)	0.782 (+8.97)	0.798 (+14.59)	0.739 (+16.76)	0.736 (+15.68)	0.781 (+14.88)
TR	0.805 (+14.02)	0.814 (+11.01)	0.801 (+15.49)	0.692 (+11.78)	0.859 (+10.67)	0.767 (+17.99)	0.754 (+16.02)	0.772 (+14.39)
KO	0.781 (+14.07)	0.801 (+13.45)	0.779 (+14.96)	0.682 (+10.42)	0.791 (+15.72)	0.854 (+12.78)	0.728 (+14.30)	0.760 (+13.82)
FA	0.794 (+13.42)	0.807 (+12.92)	0.779 (+13.48)	0.692 (+11.87)	0.798 (+15.70)	0.745 (+18.23)	0.820 (+11.94)	0.769 (+14.27)
Table 3: NER F1 scores for our approach trained on a source language (rows) and tested on a target language (columns), with absolute percentage improvements over the XLMR baseline shown in parenthesis. The rightmost column shows the average cross-lingual model performance across all languages. In the top table, blue values are the F1 scores for the mono-lingual models.
MSQ), the ability to inject explicit NE knowledge provides significant gains.
Pre-trained mOWNER models: On the MSQ dataset, our approach obtains an average of absolute improvement of  $\mathrm{MD} = +21.3\%$  over the baseline across all languages. This validates our hypothesis that gazetteer knowledge allows models to adapt on out-of-domain data. The gains for EN are  $11\%$ , whereas the highest are for TR with  $35\%$ . The gain ratios are highly correlated with the gazetteer coverage on MSQ with Pearson's correlation of  $\rho = 0.67$ . The coverage for MSQ EN is at  $85\%$ , and thus the lowest gains, while for the remaining languages the coverage is at  $98\%$ .
The results in Figure 3 validate the utility of the proposed dataset mOWNER. Similar architectures trained on mOWNER and CoNLL have highly diverging performance, with models trained on CoNLL showing limited cross-domain transfer. For example, when assessed for cross-domain transfer on the MSQ dataset, the pre-trained models on EN-CoNLL and EN-mOWNER achieve MD=0.64 and MD=0.73, respectively.
Finally, comparing cross-domain transfer in terms of micro F1 score for the XLMR and SubTagger baselines trained on LOWER, our approach achieves an average  $\mathrm{F1} = +33.2\%$  absolute points improvements against XLMR across all languages, whereas for SubTagger for EN-MSQ, we see an absolute points of improvement of  $\mathrm{F1} = +18.8\%$ .
Cross-Domain Transfer on WNUT: Since WNUT is available in EN only, we show the zero-shot and fine-tuning performance of mLOWNER pretrained models (since they use the same NER taxonomy). Our approach obtains a score of F1=0.293, contrary to the XLMR which achieves F1=0.220. Fine tuning the mLOWNER models on the WNUT train set, we achieve a new state of the art result (cf. (Shahzad et al., 2021)) in WNUT with F1=0.507 for our approach, which is 9.7% higher than the baseline.
Cross-Domain Transfer on Twitter Data: Apart from assessing our models on cross-domain transfer on the WNUT dataset, we additionally assess the performance of the baseline and our approach on the 10k random Twitter sample data. Table 4 shows the precision per NER class of the competing approaches. For each model, we randomly sample a set of 30 tweets per NER class, leading to a total 180 tweets per model. This results into a total of 360 tweets for both models, which we annotate to measure the accuracy of models in detecting named entities. We use the resulting annotations to measure the precision for each model in Table 4.
Our approach significantly outperforms the baseline approach on the cross-domain transfer on the Twitter data as well, with an absolute difference of  $26.74\%$  in terms of overall precision.[9]
<sup>9</sup>We are unable to report recall numbers, given that this would require us to manually annotate all 10k tweets in order to measure recall.
From the results we note that both approaches perform fairly well for the PER class, where the difference between the two models is only with  $7.47\%$ . This is intuitive given that person names are quite regular, and even in out of domain corpora, both models have little difficulty in identifying them.
On the contrary, for NER classes such as GRP or CORP, the gap between the models is very large, with  $53.33\%$  and  $43.70\%$ , respectively. Contrary to person names, corporation and group names do not follow very strict pattern, hence, the low performance of the baseline model.
Finally, for CW, we note that our approach has the lowest performance among all the other NER classes. The lower performance in this case can be explained by the fact that our gazetteer knowledge containing CW entries leads to false positive matches in the Twitter data. Given that Twitter contains tweets that are highly diverse and that CW entries can be quite complex phrases, this leads to spurious matches, which we do not have in more controlled domains like CoNLL or mLOWER.
XLMR	Ours
PER	0.759	0.833 (+7.47%)
CORP	0.296	0.733 (+43.70%)
LOC	0.724	0.900 (+17.59%)
CW	0.345	0.500 (+15.52%)
PROD	0.600	0.833 (+23.33%)
GRP	0.133	0.667 (+53.33%)
micro@P	0.477	0.744 (+26.74%)
Table 4: Zero shot cross-domain per class performance (measured in terms of precision) for the XLMR baseline and our approach, on a sample of 360 tweets.
6.3 Cross-Linguual Transfer
Applying pre-trained models on a source language to other target languages provides several advantages in reducing annotation costs, which for some low-resource languages may be difficult to obtain. Table 3 shows the NER results of our approach when trained on a source language (rows) and tested on a target language mLOWNER (columns) dataset. In brackets is shown the absolute improvement over the baseline in terms of micro F1 score.
Zero-Shot Evaluation. In this setting, we consistently outperform XLMR (except  $\langle \mathrm{RU},\mathrm{ES}\rangle$  where we note a negligible difference). When applying the EN model on low-resource languages our gains are highest, with an average absolute improvement of  $\mathrm{F1} = +17.13\%$  . The gains over the
baseline are particularly high, when the source (EN) and target languages are distant, e.g., TR, KO or FA. This is intuitive as pre-trained textual knowledge is scarce for such pairs, however, the integrated gazetteer information through MoE provides the missing NE token knowledge for NER. Finally, for similar languages like EN, NL, ES, the differences to the mono-lingual performance is within a  $5 - 8\%$  F1. Such cross-lingual transfer is very promising, considering the zero-shot setting and the fact that we simply swap the gazetteer data to the target language without any fine-tuning.
Few-Shot Evaluation. In this setting, we used 500 instances from a target language for fine-tuning. Similarly, here too, our approach consistently improves over the baseline. The gap between the baseline and our approach increases slightly from zero-shot to few-shot. Overall comparison to zero-shot, with 500 instances, the improvements across all language pairs are with  $\mathrm{F}1 = +8\%$  absolute points.
With few-shot learning, we close the gap to the monolingual models significantly. For instance, the fine-tuned EN model for the rest of the target languages has only  $4.7\%$ ,  $4.4\%$ ,  $5.3\%$  lower performance for ES, NL, and TR whereas for FA, RU and KO the difference is higher with  $8.6\%$ ,  $9\%$  and  $10.4\%$ . The results are encouraging, considering that for low-resource languages like FA or KO, obtaining annotations can be problematic.
We presented an approach to flexibly inject gazetteers into multilingual transformers for NER, showing its utility for cross-domain and cross-lingual transfer. Furthermore, we propose and publish large multi-lingual and multi-domain corpora for training and testing NER performance.
Thorough evaluations show that NER knowledge transfer can be guided and significantly improved through external gazetteers. On cross-domain transfer our approach achieves an improvement of over  $\mathrm{MD} = +21.3\%$  across all languages, whereas for cross-lingual transfer, with only 500 instances we reach the monolingual performance with only  $6\%$  difference in terms of F1 across all languages.
Finally, we showed that training data plays a significant role in NER model's ability to transfer knowledge across domains and languages, where pre-trained models on CoNLL fail to perform well on out-of-domain and multi-lingual datasets.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8440-8451. Association for Computational Linguistics.
Alessandro Cucchiarelli, Danilo Luzi, and Paola Velardi. 1998. Automatic semantic tagging of unknown proper names. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, COLING-ACL '98, August 10-14, 1998, Université de Montréal, Montréal, Quebec, Canada. Proceedings of the Conference, pages 286-292. Morgan Kaufmann Publishers / ACL.
James R. Curran and Stephen Clark. 2003. Language independent NER using a maximum entropy tagger. In Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 164-167. ACL.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 140-147. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu, Linlin Li, and Luo Si. 2019. A neural multi-digraph model for chinese NER with gazetteers. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1462-1467. Association for Computational Linguistics.
Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. Gazetteer enhanced named entity recognition for code-mixed web queries. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1677-1681. ACM.
Chen Jia, Liang Xiao, and Yue Zhang. 2019. Cross-domain NER using cross-domain language modeling. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2464-2474. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018, pages 66-71. Association for Computational Linguistics.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 282-289. Morgan Kaufmann.
Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Bin Dong, and Shanshan Jiang. 2019. Gazetteer-enhanced attentive neural networks for named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6231-6236. Association for Computational Linguistics.
Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019. Towards improving neural named entity recognition with gazetteers. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5301-5307. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. MultiCoNER: a Large-scale Multilingual dataset for Complex Named Entity Recognition.
Stephen Mayhew, Tatiana Tsygankova, and Dan Roth. 2019. Ner and pos when nothing is capitalized. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6255-6260. Association for Computational Linguistics.
Tao Meng, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. GEMNET: effective gated gazetteer representations for recognizing complex entities in low-context input. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1499-1512. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 collocated with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Crosslingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1946-1958. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Shruti Rijhwani, Shuyan Zhou, Graham Neubig, and Jaime G. Carbonell. 2020. Soft gazetteers for low-resource named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8118-8123. Association for Computational Linguistics.