SlowGuess/NAACL_2022_OCR huggingface dataset:
Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, Shervin Malmasi  
Amazon, USA
{besnikf, njfn, olegro, malmasi}@amazon.com
Named entity recognition (NER) in a real-world setting remains challenging and is impacted by factors like text genre, corpus quality, and data availability. NER models trained on CoNLL do not transfer well to other domains, even within the same language. This is especially the case for multi-lingual models when applied to low-resource languages, and is mainly due to missing entity information.
We propose an approach that with limited effort and data, addresses the NER knowledge gap across languages and domains. Our novel approach uses a token-level gating layer to augment pre-trained multilingual transformers with gazetteers containing named entities (NE) from a target language or domain. This approach provides the flexibility to jointly integrate both textual and gazetteer information dynamically: entity knowledge from gazetteers is used only when a token's textual representation is insufficient for the NER task.
Evaluation on several languages and domains demonstrates: (i) a high mismatch of reported NER performance on CoNLL vs. domain specific datasets, (ii) gazetteers significantly improve NER performance across languages and domains, and (iii) gazetteers can be flexibly incorporated to guide knowledge transfer. On cross-lingual transfer we achieve an improvement over the baseline with  $\mathrm{F1} = +17.6\%$  and with  $\mathrm{F1} = +21.3\%$  for cross-domain transfer.
Advances in pre-trained models have achieved state of the art results for NER (Conneau et al., 2020; Yamada et al., 2020). Models like XLM-RoBERTa (XLMR) (Conneau et al., 2020) offer advantages as they can be applied on several languages with little fine-tuning to obtain optimal NER performance, with an F1 score of 92.92 for English and an average of 89.43 across all languages in CoNLL (Sang and Meulder, 2003).
Table 1: Example snippets in multiple-languages and domains. NER needs to resolve equivalent NE surface forms across languages, e.g. "Presidency of the European Council" to "EU-Ratspresidäntschaft", or across domains where entity distribution change (second row, where entity types are marked in different colors).
While NER results obtained on CoNLL have reached remarkable levels, in real-world settings, NER faces many challenges, related to application domain, language, or data quality. For uses cases such as Web search queries or utterances coming from voice assistants, data quality and obtaining annotations are an issue. Such corpora usually have low context and no casing information, or contain syntactic errors. For instance, by just dropping the casing information on CoNLL test set the NER performance drastically drops to  $\mathrm{F}1 = 0.35$  (Mayhew et al., 2019). Moreover, such snippets often cover diverse domains with named entities that are not part of the training data.
Table 1 shows example sentences<sup>1</sup> in different languages and genres/domains. For NER knowledge transfer across languages, a typical challenge is the significant surface form variations of NEs, in terms of their compositional nature, ambiguity of surface forms, and as well script. Similarly, a challenge across domains are the diverging named entity distributions or ambiguities that surface forms resolve to different entity types. To date, there are no existing datasets that allow to probe NER systems for cross-lingual and cross-domain transfer (e.g. domains like Q&A or Web search).
${}^{1}$  NEs of different types are annotated with specific colors.
Human Language Technologies, pages 2777 - 2790
Considering the above challenges, our objective in this work is to propose approaches and training strategies that fulfill the following desiderata:
- Cross-Linguuality: Models trained on a source language should transfer with minimal effort to a target language. The challenges are the compositionality of NEs across languages and script (c.f. NEs in green for EN and DE in Table 1).
- Cross-Domain: Models should transfer across domains that have diverging NE distributions. Specifically, determine entity boundaries (e.g. generalize from Person to Creative Work, which are often complex noun or verb phrases).
We propose an NER approach that fulfills the two desiderata. First, we address multi-linguality by encoding sentences using the pre-trained XLMR model (Conneau et al., 2020). Second, to account for domain differences, we enhance XLMR with multi-lingual gazetteers that can be extracted from resources like Wikidata, or domain-specific resources (e.g. product catalogs). Gazetteers aid the NER knowledge transfer and provide the models with explicit signal about NEs from a target language/domain. Since the two modules provide complementary information, we combine them using the mixture of experts (MoE) (Shazeer et al., 2017), allowing the model to dynamically determine which portion of the information is used for NER. Finally, we construct multi-lingual and -domain NER datasets, addressing some of the deficiencies of existing datasets like WikiAnn (Pan et al., 2017), which consists of sentences with popular entities across all languages, limiting knowledge transfer for low-contextual and emerging domains.
Experiments on 7 languages and multiple domains confirm that our model can adapt across domains and languages using few-shot learning (with as much as 500 instances transfer from high to low resource languages). Gazetteer information combined through MoE, provides an advantage over baselines with an average improvement of  $\mathrm{MD} = +33.21\%$  in mention detection across domains and F1  $= +17.6\%$  across languages.
In this work, our contributions are threefold:
- gazetteer integration into NER models for cross-lingual and -domain NER knowledge transfer,
- novel means in integrating text and gazetteer representations through Mixture-of-Experts (MoE),
- mOWNER a low-contextual and multilingual, and MSQ a multilingual questions dataset.
The use of gazetteers is not new. It has been a core principle in doing NER using feature-based approaches (Curran and Clark, 2003; Toral and Muñoz, 2006; Cucchiarelli et al., 1998). However, with neural models and recent pre-trained transformer models achieving state of the art results (Vaswani et al., 2017; Conneau et al., 2020; Devlin et al., 2019), the utility of gazetteers on standard benchmarks has diminished. Our related work discussion is focused towards works that have utilized gazetteers for NER.
Liu et al. (2019) propose the use of gazetteers with neural NER models, utilizing them in the form of a sub-tagsger. For each token a matching score to the gazetteer entries needs to be pre-computed and then fed into the NER framework. The main utility of gazetteers is to provide flexibility and be easy to swap, allowing NER models to adapt on out-of-domain data. Contrary to Liu et al. (2019), we flexibly combine gazetteers with the textual information and depending on the context are weighted accordingly. Gazetteers can be swapped during the test-phase without any fine-tuning. We compare against this approach and show the advantages of our approach both in terms monolingual and cross-domain performance.
Shang et al. (2018) create dictionaries for a given corpus on which the NER task is performed. This avoids ambiguous matches of named entities across domains. The task is to determine whether the tokens in a span belong together or not, as part of an entity, otherwise they can be two different entities or not be entities at all. Finally, the type of those spans is predicted. We differ from (Shang et al., 2018) on three main points. First, the dictionary creation is tied to the corpus. Second, the model fits parameters to predict if a text span on a given corpus represents an entity. Finally, the dictionary information and model weights are ingrained into the model, which is not the case for our approach.
Ding et al. (2019) create a di-graph from a sentence and gazetteer matches. Adjacent nodes are connected via a directed edge, after which, edges between the matched characters to the gazetteer nodes are added. The di-graph is then fed to a graph neural network for training and resolve ambiguous matches. Contrary to our work, here the gazetteer matches are ingrained into the model, and changes in gazetteers induce changes in the graph structure and thus require complete retraining.
Rijhwani et al. (2020) integrate entity linking systems in matching the tokens or token spans to some target entity or candidate entities. For each match, different features are proposed, e.g. top scoring entity for a span, top-3 candidate scores, top-3 entities, type counts etc. While, using predefined feature sets (Zirikly, 2015; Rijhwani et al., 2020) has the advantages of interpretability, however, generalizing models to unseen languages or domains is challenging. A direct comparison between feature-based models and our approach is not possible. Our approach automatically integrates external gazetteers without having the need to run entity linking or any hand-crafted features.
Lin et al. (2019) propose to integrate gazetteers for NER by training a gazetteer network to predict whether a text snippet represents a name or not. There are two diverging points to our work. First, the gazetteer network weights of Lin et al. (2019) are tied to the training data, thus, for any new data the gazetteer network needs to be retrained to accurately predict if a snippet represents an entity. Second, our approach performs a soft-match w.r.t the gazetteer entries, where each match is represented as a binary vector w.r.t NER matching classes. This allows us to flexibly change at inference time the gazetteer data, since the model captures only the structural information present in tokens and sentences. Furthermore, using the mixture of experts module to combine both the textual and gazetteer token representations, we can flexibly determine which representation to use for NER classification.
Finally, Jia et al. (2019) propose the use of mixture of experts, where the experts correspond to separate classifiers per NER class. We differ in that we utilize MoE to compute a unified representation of text and gazetteers.
Models trained on CoNLL typically perform poorly when applied on out-of-domain data. Similarly, WikiAnn (Pan et al., 2017), which consists of contextually rich sentences, is not suitable for domain transfer where context is scarce (e.g. Web search).
We describe the process of constructing the multilingual and multi-domain datasets. We include the following languages: English-EN, Spanish-ES, Dutch-NL, Russian-RU, Turkish-TR, Korean-KO, Farsi-FA, a mix of high and low resource languages. The data is available for download.2
mOWNER. Which stands for multilingual lowcontext Wikipedia NER dataset (Malmasi et al., 2022), extracted from the different localized versions of Wikipedia. We extract low-context sentences that contain interlinked entities and resolve the entity types using Wikidata as reference, according to the NER class taxonomy from (Derczynski et al., 2017).
Ensuring that the extracted sentences and the interlinked entities therein are of high quality we follow two filtering strategies. First, we apply regular expression to identify and filter out sentences that contain named entities that are not interlinked. This step removes long and high-context sentences. Second, we filter out sentences, in which the links could not be resolved to Wikidata entities. Applying the two steps filter out over  $90\%$  of the sentences from the respective Wikipedia versions. The resulting dataset is diverse in domains and multilingual, including low-resource languages FA, KO, TR. For more details regarding the mLOWNER dataset, we refer to the reader to dataset paper (Malmasi et al., 2022), and additional details provided in the paper appendix.
Sentences in mLOWNER have on average 15 to 19 tokens. Based on a manual inspection of 400 sample sentences in EN, the quality of the NER gold-labels is with  $94\%$  accuracy.
- his playlist includes sonny sharrock, gza, country teasers and the notorious b.i.g.
- the atari 2600 hardware design experienced many makeovers during its 14 year production history.
MSQ. From the MS-MARCO Q&A corpus (Nguyen et al., 2016) we construct question templates, where the entities are replaced by their type following the same NER taxonomy as mLOWNER. We identify entities in a question using spaCy<sup>3</sup>. For example, from the template "who produced ⟨CW | PROD⟩", we generate multiple instances by varying entities of type CW or PROD.
MSQ is used only for testing and to assess cross-domain knowledge transfer of NER models. Since the questions are only in English language, we translate the extracted templates using Amazon Translate. The translation quality is good considering that the question templates are short. The number of questions per language is around 17.5k with an average number of tokens  $4.9 \pm 1.73$ .
4https://aws.amazon.com/translate/
$^{2}$ https://registry.opendra\.aws/multiconer/
Figure 1 shows an overview of our approach based our prior work (Meng et al., 2021; Fetahu et al., 2021), which we adopt for our cross-lingual and cross-domain application scenario. It consists of three main components: (i) multi-lingual sentence representation, (ii) external gazetteer knowledge integration, and (iii) dynamic combination of text and gazetteer information.
For a sentence  $s = \{w_1, \ldots, w_N\}$  with  $N$  tokens we compute token representation as follows.
4.1 Multi-Lingual Text Representation
Using XLMR (Conneau et al., 2020) as a text encoder, we are able to encode sentences from multiple languages, and compute the sentence representation  $\mathbf{h}_s = \{\mathbf{h}_1,\dots ,\mathbf{h}_N\}$ , where  $\mathbf{h}_{\mathbf{s}}\in \mathbb{R}^{N\times L}$  represents the sentence representation for  $N$  tokens with  $L$  output dimensions.
While XLMR has remarkable NER performance on the CoNLL dataset, textual representation alone is not sufficient for cross-domain transfer. Such limitations are even higher when consider cross-lingual transfer on distant languages. Depending on the pre-training resources for a language, XLMR tokenization (Kudo and Richardson, 2018) of infrequent tokens or tokens from low-resource languages, can be problematic, often leading to over-segmentation. This in turn, introduces ambiguity for the NER task, e.g., "wunderkind little amadeus"  $\rightarrow$  "wunder kind_little_amade us", is tokenized into sub-words with ambiguous meaning within and across languages, e.g. wunder, kind, us.
4.2 Gazetteer Representation
Gazetteers inject explicit information about target NEs (e.g. Products from an e-commerce site). This provides the flexibility to adapt on target domains and for entities with variable surface forms (e.g. Movies, Product names). Typically complex entities (e.g. movie titles) consist of complex noun or word phrases that are to capture (cf. Figure 1).
Overall, gazetteers are easy to obtain from open resources like Wikidata<sup>5</sup>. A gazetteer  $\mathcal{G}$  consists of entities and their type, e.g.  $\langle \text{"No Time to Die", CW} \rangle$ .
Gazetteer Matcher. For a token or sequence of tokens  $s'$  from  $s' \subseteq s$ , we extract the longest match
from entries in  $\mathcal{G}$ . The gazetteer  $\mathcal{G}$  consists of a trie built from all the named entity entries of interest.
The matcher yields a sparse encoding  $\mathbf{g}_{\mathbf{s}} \in \mathbb{N}^{N \times k}$ , where  $\mathbf{g}_{s} = \{\mathbf{x}_{1}, \ldots, \mathbf{x}_{N}\}$  and  $\mathbf{x}_{i} \in (0, 1)_{k}$  is a binary vector of length  $k$  ( $k$  is the number of target NE types in  $\mathcal{G}$  in IBO format). More specifically, if our sequence of tokens is  $s' = \{the, late, show, with, stephen, colbert\}$ , the resulting matcher would yield the following matrix  $\mathbf{g}_{s'}$ :
$$
\begin{array}{c c c c c c} & \text {B - C W} & \text {I - C W} & \text {B - P E R} & \text {I - P E R} & \dots \\ \text {t h e} & 1 & 0 & 0 & 0 & 0 \\ \text {l a t e} & 0 & 1 & 0 & 0 & 0 \\ \text {s h o w} & 0 & 1 & 0 & 0 & 0 \\ \text {w i t h} & 0 & 1 & 0 & 0 & 0 \\ \text {s t e p h e n} & 0 & 1 & 1 & 0 & 0 \\ \text {c o l b e r t} & 0 & 1 & 0 & 1 & 0 \end{array}
$$
The sparse vectors in  $\mathbf{g}_s$  are converted into a dense representation by projecting them through a dense layer  $\theta$ , which are encoded using a BiLSTM,  $\mathbf{G}_s = \left[\overrightarrow{\mathrm{LSTM}} (\theta [\mathbf{g}_s]);\overleftarrow{\mathrm{LSTM}} (\theta [\mathbf{g}_s])\right]\in \mathbb{R}^{N\times L}$ .
The final gazetteer representation  $\mathbf{G}_s$ , a BiLSTM encoder learns the context of sentence  $s$ , and using its context learns to resolve ambiguous matches a token may have in  $\mathcal{G}$ , e.g., "stephen colbert", matches to both CW and PER entries.
4.3 Combined Representation
The encoded text and gazetteer representations capture complementary information. Depending on  $s$ , not always both representations are deemed as useful. For instance, if  $\mathbf{h}_s$  captures the contextual information of  $s$  and the pre-trained knowledge of XLMR for the tokens therein is not ambiguous, then  $\mathbf{G}_s$  may not be necessary. Otherwise, when the model is applied to out-of-domain sentences or tokens are ambiguous and match to multiple named entity types, in such cases  $\mathbf{G}_s$ , which encodes explicit information from a target domain or languages provides the necessary context.
At token level we learn a function that combines dynamically both representations by computing an importance score for  $\mathbf{h}_s$  and  $\mathbf{G}_s$ . The importance  $\mathbf{w}_{moe}$  is computed based on the mixture of experts approach (MoE) (Shazeer et al., 2017). Since we have two representations only, we use a Sigmoid function to split the importance accordingly:
$$
\mathbf {w} _ {m o e} = \sigma (\Lambda [ \mathbf {h} _ {s}; \mathbf {G} _ {s} ] ^ {T}) \tag {1}
$$
$$
\mathbf {h} = \mathbf {w} _ {m o e} \cdot \mathbf {h} _ {s} + (1 - \mathbf {w} _ {m o e}) \cdot \mathbf {G} _ {s} \tag {2}
$$
where,  $\Lambda \in \mathbb{R}^{2L}$ . From  $\mathbf{h}$  using a conditional random field (CRF) layer (Lafferty et al., 2001) we predict the token NER tags.
<sup>5</sup>https://www.wikipidata.org/
Figure 1: Approach overview: (a) GAZ Matcher: matches tokens to gazetteer entries, e.g. "stephen" is matched to both I-CW and B-PER; (b) GAZ LSTM Encoder: computes a contextual representation of the gazetteer matches; (c): Word Rep.: computes the token XLMR representation; (d) Mixture of Experts: computes the weights of both representations in (b) and (c); and (e) CRF: the classification layer that outputs the NER tags in BIO format.
Lastly, by feeding the gazetteer matches as a binary matrix, which corresponds to the NER class matches of a given text span, and combining this information jointly with the  $\mathbf{h}_{\mathrm{s}}$  representation, we allow our model to abstract the gazetteer representation  $\mathbf{G}_s$  and learn structural NE properties for a given text span (i.e. in terms of NER classes the span may belong to), given that textual representation is provided by XLMR. This is a significant improvement over existing work, which compute gazetteer representations w.r.t tokens and thus require re-training, whenever the gazetteer information is updated (Liu et al., 2019).
4.4 Multi-Stage Training Strategy
Our approach consists of modules like XLMR, whose parameters contain pre-trained knowledge, and the randomly initialized gazetteer and MoE modules. To align the parameter spaces of these components, and avoid that the NER model is not biased towards the pre-trained knowledge of XLMR, we device a two-stage training strategy.
First Stage. XLMR's weights are frozen, while gazetteer encoder is trained, allowing it to learn how to resolve ambiguities tokens matches.
Second Stage. All components are jointly trained, further fine-tuning XLMR and MoE to weigh between  $\mathbf{h}_s$  and  $\mathbf{G}_s$  according to their impact on predicting the NER class.
Here we describe the NER approaches under comparison for knowledge transfer across domains and languages. Next, we introduce the multilingual training data, and the corresponding test sets for cross-lingual and cross-domain evaluation.
5.1 Baselines and Approach Setup
Baseline - XLMR: The XLMR transformer (Conneau et al., 2020) with a CRF layer trained for the NER task is considered as a baseline. We use the AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of  $lr = 1e - 5$  to minimize the negative log-likelihood loss (NLL), and use a batch size of 64. This represents an ablation of our model without gazetteers and the MoE mechanism.
Baseline - Gazetteer Lookup (BaG): To assess that gazetteers alone are insufficient, we consider a gazetteer lookup to the longest matching text span to the gazetteer entries. For a more favorable setting for BaG, ambiguous span matches are counted as correct if the NER class is in the set of classes assigned by the gazetteer.
SubTagger (Liu et al., 2019): We train the SubTagger's gazetteer matcher on EN gazetteer data and test its monolingual and cross-domain perfor
mance for English. This is due to the fact that GloVe (Pennington et al., 2014) and ELMo embeddings (Peters et al., 2018) are available only for English, and are key components in training the gazetteer and NER model. Evaluating SubTagger for cross-lingual transfer is not possible since it uses monolingual embeddings and for any target language, the model needs to be retrained from scratch using the gazetteer and the word representations from the target language.[6]
Approach (Ours): Our approach consists of three components that are trained using the introduced multi-stage training strategy. Training details are provided in the Appendix B.
Below are shown the datasets (without casing) used for training and testing NER models.
CoNLL: exists in 4 languages (EN, DE, ES, NL) with sentences (Sang, 2002; Sang and Meulder, 2003), and used for training only.
mOWNER: mOWNER (Malmasi et al., 2022) is used for training. mOWNER test set is used to assess cross-lingual transfer. For training and development, for each language, we use  $15.2\mathrm{k}$  and 800, respectively. For testing we limit the number of instances to  $10\mathrm{k}$  per language. Additional details are provided in Appendix A.
MSQ: This dataset is used only to test the cross-domain transfer of pre-trained NER models.
WNUT: WNUT17 (Derczynski et al., 2017) is another test set for cross-domain evaluation, consisting of social media posts in EN language.
Twitter Data: Additionally, we collected a random sample of 10k tweets in the English language,7 to assess the competing approaches (XLMR baseline and our approach) in a zero-shot setting.
Gazetteers: Entries are extracted from Wikidata entity titles (from types corresponding to the NER taxonomy). More details in Appendix A.
5.3 Cross-domain & Cross-lingual Scenarios
Cross-Domain: Pre-trained models on CoNLL and mLOWER are assessed for out-of-domain transfer on MSQ in terms of mention detection (MD). MD measures the ability to predict the entity boundaries, disregarding the actual entity type. We also consider cross-domain transfer from ENLOWER to WNUT and report NER micro F1.
Cross-Linguual: Models trained on an mLOWER source language are assessed on a target language under zero-shot and few-shot learning.
Here we assess the monolingual NER model performance and impact of the multi-stage training strategy and that of MoE. Finally, we assess their knowledge transfer across languages and domains.
6.1 Model Comparison
CoNLL	mOWNER
BaG	XLMR	Ours	BaG	XLMR	Ours
EN	0.178	0.850	0.860 (+1%)	0.148	0.755	0.888 (+13.3%)
ES	0.184	0.798	0.813 (+1.5%)	0.047	0.746	0.847 (+10.1%)
NL	0.194	0.807	0.826 (+1.9%)	0.220	0.803	0.867 (+6.4%)
RU	-	-	-	0.213	0.693	0.782 (+8.9%)
TR	-	-	-	0.106	0.752	0.859 (+10.7%)
KO	-	-	-	0.268	0.726	0.854 (+12.8%)
FA	-	-	-	0.614	0.700	0.820 (+12.0%)
Table 2: Micro F1 results across all types. Note that the NER type taxonomies differ between CoNLL and mLOWNER.