SlowGuess/NAACL_2021_OCR huggingface dataset:
Tao Meng* UCLA Los Angeles, USA
Anjie Fang  
Amazon.com, Inc.  
Seattle, WA, USA
Oleg Rokhlenko  
Amazon.com, Inc.  
Seattle, WA, USA
Shervin Malmasi  
Amazon.com, Inc.  
Seattle, WA, USA
tmeng@cs.ucla.edu {njfn,olegro,malmasi}  $@$  amazon.com
Named Entity Recognition (NER) remains difficult in real-world settings; current challenges include short texts (low context), emerging entities, and complex entities (e.g. movie names). Gazetteer features can help, but results have been mixed due to challenges with adding extra features, and a lack of realistic evaluation data. It has been shown that including gazetteer features can cause models to overuse or underuse them, leading to poor generalization. We propose GEMNET, a novel approach for gazetteer knowledge integration, including (1) a flexible Contextual Gazetteer Representation (CGR) encoder that can be fused with any word-level model; and (2) a Mixture-of-Experts gating network that overcomes the feature overuse issue by learning to conditionally combine the context and gazetteer features, instead of assigning them fixed weights. To comprehensively evaluate our approaches, we create 3 large NER datasets (24M tokens) reflecting current challenges. In an uncased setting, our methods show large gains (up to  $+49\%$  F1) in recognizing difficult entities compared to existing baselines. On standard benchmarks, we achieve a new uncased SOTA on CoNLL03 and WNUT17.
Identifying entities is a core NLP task. Named Entity Recognition (NER) is the task of finding entities and recognizing their type (e.g., person or location). Mention Detection (MD) is a simpler task of identifying entity spans, without the types.
Advances in neural NER have produced high scores on benchmark datasets like CoNLL03 and OntoNotes (Devlin et al., 2019). However, a number of challenges remain. As noted by Augenstein et al. (2017), these scores are driven by the use of well-formed news text, the presence of "easy" entities, and memorization effects due to entity overlap between train/test sets; these models perform significantly worse on unseen entities or noisy text.
1.1 Current NER Challenges
Beyond news text, many challenges remain in NER. Context information has been shown to be important for NER (Jayarao et al., 2018), and short texts like search queries are very challenging due to low context and a lack of surface features (Guo et al., 2009; Carmel et al., 2014). Unseen and emerging entities also pose a challenge (Bernier-Colborne and Langlais, 2020). Finally, some entities, like movie names are not simple noun phrases and are harder to recognize (Ashwini and Choi, 2014). Table 1 lists more details about these challenges, and how they can be evaluated.
Entity Knowledge is essential for overcoming these issues, and critical in the absence of casing. Even a human may not correctly parse "what is [[life is beautiful]]?" without knowing that a movie is being referenced. However, most models start with no knowledge of real world entities, learning them from the training data. Continuous data annotation can add new entities, but is expensive and often not feasible.
Consequently, methods for integrating external knowledge, e.g., Knowledge Bases (KBs) or gazetteers, into neural architectures have gained renewed attention. However, such studies have reported limited gains (Liu et al., 2019; Rijhwani et al., 2020). The mixed success of gazetteers stems from three main limitations in current work: gazetteer feature representation, their integration with contextual models, and a lack of data.
For the representation, one-hot binary encoding is often used to represent gazetteer features (Song et al., 2020). However, this does not capture contextual info or span boundaries. Alternatively, independent span taggers trained on gazetteers have been proposed to extract potential entities Liu et al. (2019), but such models can be difficult to train and may not provide reliable features.
*This research was done during an internship at Amazon.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1499-1512 June 6-11, 2021. ©2021 Association for Computational Linguistics
Challenge	Description
Short Texts For: voice, search	News texts have long sentences discussing many entities, but other use cases (search queries, questions) have shorter inputs. Datasets with minimal context are needed to assess performance of such use cases. Capitalization/punctuation features are large drivers of success in NER (Mayhew et al., 2019), but short inputs (ASR, user input) often lack these surface feature. An uncased evaluation setting is needed to understand model performance.
Long-tail Entities For: domains with many entities	In many domains entities have a large long-tail distribution, with millions of values (e.g., location names). This makes it hard to build representative training data, as it can only cover a portion of the potentially infinite entity space. A very large test set is required for effective evaluation.
Emerging Entities For: domains with growing entities	All entity types are open classes (new ones are added), but some groups have a faster growth rate, e.g., new books/songs/movies are released weekly. Assessing true generalization requires test sets with many unseen entities, to mimic an open-world setting.
Complex Entities For: voice, search	Not all entities are proper names: some types (e.g. creative works) can be linguistically complex. They can be complex noun phrases (Eternal Sunshine of the Spotless Mind), gerunds (Saving Private Ryan), infinitives (To Kill a Mockingbird), or full clauses (Mr.Smith Goes to Washington). Syntactic parsing of such nouns is hard, and most current parsers/NER systems fail to recognize them. The top system from WNUT 2017 achieved 8% recall for creative work entities (Aguilar et al., 2017). Effective evaluation requires corpora with many such entities.
Table 1: NER challenges not addressed by current work and datasets, and proposed solutions.
There are also limitations in the integration of gazetteer features. Existing studies often add extra features to a word-level model's Contextual Word Representations (CWRs), which typically contain no info about real world entities or their spans (Yamada et al., 2020). This concatenation approach is sub-optimal as it creates additional, and often highly correlated features. This has been shown to cause feature "under-training", where the model will learn to mostly rely on either context or gazetteer during training, and underuse the other (Yang et al., 2016). This can be problematic as the utility of the gazetteer is variable: it is valuable in low-context cases, but may not be useful when rich syntactic context (from the CwR) can identify entities. Conversely, a true entity may be missing from the gazetteer. However, when gazetteers are represented as an independent feature, the model assigns it a fixed weight, and its contribution to the prediction is static. To overcome this, external knowledge should dynamically be infused into relevant dimensions of the CwR, with the model learning to conditionally balance the contribution of the CwR and gazetteer to the prediction.
Finally, these issues are compounded by a lack of data reflecting the challenges from Table 1, which prevents the exploration of effective architectures for knowledge injection.
The key contributions of this paper are new data and methods to address the above challenges.
We propose GEMNET, a gazetteer expert mixture network for effectively integrating gazetteers into any word-level model. The model includes an
encoder for Contextual Gazetteer Representations (CGRs) as a way to incorporate any number of gazetteers into a single, span-aware, dense representation. We also propose a gated Mixture-of-Experts (MoE) method to fuse CGRs with Contextual Word Representations from any word-level model (e.g., BERT), something not explored in previous work. Our novel MoE approach allows the model to conditionally compute a joint CGR-CWR representation, training a gating network to learn how to balance the contribution of context and gazetteer features. Finally, we employ multi-stage training to drive further improvements by aligning the CGR/CWR vectors.
To evaluate our proposed approaches, we create 3 challenging NER datasets that represent short sentences, questions, and search queries. The created datasets have complex entities with low-context and represent the challenges in Table 1.
Extensive experiments in an uncased setting show that our MoE model outperforms other baselines, including concatenation, in all experiments. We achieve state-of-the-art (SOTA) results on CoNLL03/WNUT17, but its utility is more notable on our difficult low-context data. We show that short texts make NER much harder, but gazetteers yield huge gains of up to  $+49\%$  F1, specially in recognizing complex/unseen entities. We also show that gazetteer coverage during training is important.
Deep Learning for NER Neural approaches have greatly improved NER results in recent years. A shift to encoders e.g., BiLSTM-CRF models (Huang et al., 2015), using static word embed
dings eliminated the need for manual feature engineering (e.g., capitalization features). More recently, transformer-based Language Models (LMs), e.g., BERT (Devlin et al., 2019), achieved further improvements by using deep contextual word representations. Such models jointly learn syntactic cues and entity knowledge, and may fail to recognize unseen or syntactically ambiguous entities. Consequently, training data is augmented with gazetteers.
NER with Gazetteers Annotated NER data can only achieve coverage for a finite set of entities, but models face a potentially infinite entity space in the real world. To address this, researchers have integrated gazetteers into models (Bender et al., 2003; Malmasi and Dras, 2015). String matching is commonly used to extract gazetteer matches, which are then concatenated to word representations. Song et al. (2020) use gazetteers from the Wikidata KB to generate one-hot vectors that are concatenated to BERT representations, yielding minor improvements on CoNLL03. This concatenation approach has been shown to cause feature "under-training" (Yang et al., 2016), as discussed in §1. An alternative approach uses gazetteers to train a subtagger model to recognize entity spans. Liu et al. (2019) propose a hybrid semi-Markov CRF subtagger, reporting minor improvements. While a subtagger may learn regularities in entity names, a key limitation is that it needs retraining and evaluation on gazetteer updates. Recent work has considered directly integrating knowledge into transformers, e.g., KnowBert adds knowledge to BERT layers (Peters et al., 2019), and LUKE is pretrained to predict masked entities (Yamada et al., 2020). The drawbacks of such methods are that they are specific to Transformers, and the model's knowledge cannot be updated without retraining. We aim to overcome the limitations of previous work by designing a model-agnostic gazetteer representation that can be fused into any word-level model.
Mixture-of-Experts (MoE) Models MoE is an approach for conditionally computing a representation, given several expert inputs, which can be neural models with different architectures (Arnaud et al., 2020) or models using different knowledge sources (Jain et al., 2019). In MoE, a gating network is trained to dynamically weight experts per instance, according to the input. It has demonstrated to be useful in various applications like recommendation (Zhu et al., 2020), domain adaptation
for sentiment analysis, and POS tagging (Guo et al., 2018). For NER, Liu et al. (2020) proposed a Mixture of Entity Experts (MoEE) approach where they train an expert layer for each entity type, and then combine them using an MoE approach. Their approach does not include external gazetteers, and the experts provide an independent representation that is not combined with the word representation. In our work we treat word and external gazetteer representations as independent experts, applying MoE to learn a dynamically fused representation.
We experiment using three standard benchmarks: CoNLL03, OntoNotes, and WNUT17. However, these corpora do not capture the issues from Table 1; rich context and common entities (country names) allow a simple RNN model to achieve near-SOTA results. A key contribution of our paper is the creation of 3 new datasets that represent those challenges. They are difficult, as shown in §5.1.
NER Taxonomy: We adopt the WNUT 2017 (Derczynski et al., 2017) taxonomy entity types: PERSON (PER for short, names of people), LOCATION (LOC, locations/physical facilities), CORPORATION (CORP, corporations and businesses), GROUPS (GRP, all other groups), PRODUCT (PROD, consumer products), and CREATIVE-WORK (CW, movie/song/book/etc. titles).
Our datasets are described below.1 All data are uncased, and we make them publicly available.2 Their statistics, listed in Table 2, show that they reflect the challenges from  $\S 1$ : short inputs (low context), with many unseen entities in the test set.
LOWERN (Low-Context Wikipedia NER) To create our training set, we take advantage of the rich interlinks in Wikipedia. We parse the English Wikipedia dump and extract sentences from all articles. The sentences are parsed, and linked pages are resolved to their respective Wikidata entities to identify their type. To mimic search and voice settings, we minimize the context around the entities by dropping sentences with unlinked entities, identified using interlinks and a capitalization heuristic. The result is a corpus of 1.4 million low-context sentences with annotated entities, e.g., "A version for the [sega cd] was also announced."
1More details about their development are in Appendix A
2https://registry.opendra\.aws/ lowcontext-ner-gaz
Set	Dataset	Type	# Sentence	# Token	# Entity	Avg. Sent Len	Entity Type Distribution
PER	LOC	CORP	GRP	PROD	CW
1	OWNER	Train	13,424	206,772	13,555	15.40±6.35	5,029	3,791	631	1,941	424	1,805
2	OWNER	Dev	3,366	51,651	3,813	15.34±6.28	1,255	1,235	169	565	101	499
3	OWNER	Test	1,385,290	21,303,399	490,749	15.37±6.29	215,411	120,480	20,015	52,566	15,976	74,830
4	MSQ-NER	Test	17,868	98,117	18,993	5.49±1.86	4,586	10,468	679	610	469	2,187
5	ORCAS-NER	Test	471,746	1,958,020	368,250	4.15±1.75	68,000	162,652	28,738	23,058	18,114	71,461
MSQ-NER: MS-MARCO Question NER To represent NER in the QA domain, we create a set of natural language questions, based on the MSMARCO QnA corpus (V2.1) (Bajaj et al., 2016). Like Wu et al. (2020), we temptatize the questions by applying NER to extract item names, which are then mapped to our taxonomy. Entities are replaced with their types to create templates, e.g., “whosang  $< \mathrm{CW}>$ ” and “when did <PROD> come out”. Approx 3.5k Templates (appearing  $>= 5$  times) are chosen and slotted with entities from a knowledge base to generate 18k annotated questions e.g., “when did [xbox 360] come out”. There are a wide range of question shapes and entity types, please see Appendix A for examples.
ORCAS-NER: Search Query NER To represent the search query domain, we utilize 10 million Bing user queries from the ORCAS dataset (Craswell et al., 2020) and apply the same templateization procedure as MSQ-NER. This yields search templates e.g., “<PROD> price” and “<CORP> phone number”, which are used to create annotated queries, e.g., “[airpods pro] reviews”. A total of 472k queries are generated from 97k unique templates, please see examples in Appendix A.
3.1 Gazetteer Data
Our gazetteer is composed of 1.67 million entities from the English Wikidata KB. Instead of collecting entities from the web (Khashabi et al., 2018), we focused on entities that map to our taxonomy. Alternative names (aliases) for entities are included. Gazetteer statistics are listed in Appendix B.
4 The GEMNET Model
We propose GEMNET, a generic gazetteer fusion approach that can be integrated with any word-level model, e.g., RNNs and Transformers. We experiment with both BiLSTM-CRF and BERT-CRF models which produce (contextual) word representations, and complement these "word experts" with gazetteers. The overall architecture is shown in Figure 1, and the components are detailed below.
4.1 Contextual Gazetteer Representations
Our gazetteer representations is obtained in two steps: entry matching, and contextual encoding.
Table 2: Data statistics. Entity counts are unique values. LOWER has train/dev/test sets, the rest are test sets.
O	B-PROD	I-PROD	B-CORP	I-CORP
How	1	0	0	0	0
much	1	0	0	0	0
is	1	0	0	0	0
Apple	0	1	0	1	0
iPhone	0	1	1	0	0
12	0	0	1	0	0
Table 3: Example of our gazetteer representation.
Gazetteer Entry Matching A gazetteer  $\mathbf{g}$  is a list of entries that are associated with a category. For instance, a PERSON gazetteer contains a list of known people. The  $k$ -th entry  $\mathbf{g}^{(k)}$  is associated with a tokenized string ('John Carpenter') and  $\mathbf{t}^{(k)}$  holds the IOB2 tags ([B-PER, I-PER]). We use  $T$  to denote the tag set over all gazetteers, e.g.,  $T = \{\mathrm{B - PER},\mathrm{I - PER},\mathrm{B - LOC},\mathrm{I - LOC},0,\ldots \}$ .
We denote input sentences as  $(w_{1},w_{2},\ldots ,w_{L})$  where  $w_{i}$  is the  $i$ -th word, and  $L$  is the length. Full string matching is applied to inputs to identify matches across all gazetteers. Overlapping matches are resolved by preferring longer ones over shorter ones, and earlier matches over later ones. A match matrix,  $\mathbf{M}\in \{0,1\}^{L\times |T|}$ , represents the matching results. It is initialized with zeros, and successful matches  $(w_{i},w_{i + 1},\dots ,w_{i + m}) = \mathbf{g}^{(k)}$  will set
$$
\mathbf {M} _ {i + j, \mathbf {t} _ {j} ^ {(k)}} = 1, j = 0, 1, \dots , m,
$$
indicating that the word  $w_{i+j}$  is represented by a one-hot vector over the tag set  $T$ .
A key advantage of this representation is that it captures multiple matches for a given span in a sentence. As shown in Table 3, the word "apple" can be matched to product and organization types. Furthermore, it is span-aware due to the IOB2 encoding. Any number of types and gazetteers can be added as needed, allowing the model to learn from correlations, and identify ambiguous entities.
Figure 1: GEMNET model architecture. Input is passed to the word expert (e.g. BERT) and its match matrix to the Contextual Gazetteer Representation (CGR) encoder. Their outputs are dynamically combined via a Mixture-of-Experts (MoE) gating network and passed to a CRF layer for word prediction.
M is extracted by a gazetteer matcher, as a preprocessing step outside the network. This modular approach has an important advantage: it allows the gazetteer to be updated without model retraining. This is useful for efficiently recognizing emerging entities, and supporting personalized user-defined entities (e.g., contact lists).
Contextual Encoding M can be directly used as input features, but is sparse. We use a linear projection to obtain a dense representation per word:
$$
\mathbf {h} _ {g a z} ^ {i} = f (\mathbf {w} \cdot \mathbf {M} _ {i} + \mathbf {b})
$$
where  $\mathbf{w} \in \mathbb{R}^{D \times T}$  and  $\mathbf{b} \in \mathbb{R}^D$  are trainable parameters,  $D$  is the hidden dimension of gazetteer
representation and  $f$  is an activation function. This creates a dense representation that captures interactions between multiple matches. We then contextualize this representation by applying a BiLSTM:
$$
\mathbf {h} _ {f o r w a r d} ^ {i} = \mathrm {L S T M} (\mathbf {h} _ {f o r w a r d} ^ {i - 1}, \mathbf {h} _ {g a z} ^ {i})
$$
$$
\mathbf {h} _ {b a c k w a r d} ^ {i} = \mathrm {L S T M} (\mathbf {h} _ {b a c k w a r d} ^ {i + 1}, \mathbf {h} _ {g a z} ^ {i})
$$
$$
\mathbf {h} _ {C G R} ^ {i} = [ \mathbf {h} _ {f o r w a r d} ^ {i}, \mathbf {h} _ {b a c k w a r d} ^ {i} ]
$$
where  $[\cdot, \cdot]$  is the concatenation. A sample visualization of the embeddings is shown in Appendix D.
This dense contextualized gazetteer representation (CGR) can capture information about entity span boundaries (present in  $\mathbf{M}$ ), as well as interactions between entities in a sentence.
4.2 Gazetteer Knowledge (CGR) Integration
The CGR operates on IOB2 tags and cannot memorize specific patterns; it is designed to be integrated with a lexical model. We consider these representations to be orthogonal: CGRs can complement the model's knowledge and syntactic representation.
CGR Concatenation The simplest integration is to concatenate the dense CGR to the CWR, while jointly training the two representations.
Mixture-of-Experts (MoE) Model The word-level model and CGRs complement each other and may not always be in agreement. The word model may have low confidence about the span of an unseen entity, but the gazetteer may have knowledge of it. Conversely, the model's syntactic context may be confident about a span not in the gazetteer.
In fact, the two sources can be considered as independent experts and an effective model should learn to use their outputs dynamically. Inspired by the MoE architecture (Pavlitskaya et al., 2020), we apply conditional computation to combine our representations, allowing the model to learn the contexts where it should rely more on each expert.
We add a gating network to create a weighted linear combination of the word and gazetteer representations. For a sentence, the two models output their representations  $\mathbf{h}_{word}$  and  $\mathbf{h}_{gaz}$ , which are used to train the gating network:
$$
w _ {e} = \sigma (\theta [ \mathbf {h} _ {w o r d}, \mathbf {h} _ {C G R} ]),
$$
$$
\mathbf {h} = w _ {e} \cdot \mathbf {h} _ {\text {w o r d}} + (1 - w _ {e}) \cdot \mathbf {h} _ {C G R},
$$
where  $\theta$  are trainable parameters with size  $2L$ ,  $[\cdot, \cdot]$  is the concatenation and  $\sigma$  is the Sigmoid activation function. We learn gating weights,  $w_{e}$ , so that the model can learn to dynamically compute the hidden information  $\mathbf{h}$  for each word. The architecture of our model is shown in Figure 1. After obtaining  $\mathbf{h}$ , we feed it to a CRF layer to predict a tag.
Two-stage Training Our architecture jointly optimizes over both experts, but their initial states differ. The word expert often contains pretrained elements, either as word embeddings or transformers. The randomly-initialized CGR will have high initial loss, and its representation is not aligned with the word expert, preventing correct convergence. We tackle this problem through a two-stage training method to adapt the two experts to each other. In the first stage, we freeze the word ex
pert and only train the CGR encoder with the MoE and CRF layers, forcing the model to use gazetteer knowledge in order to minimize the loss. Importantly, this also adapts the CGR encoder to align its representation with that of the word expert, e.g., the dimensions with noun signals will be aligned with those of BERT, enabling the computation of their linear combination. In the second stage, the two experts are jointly fine-tuned to co-adapt them. This ensures that the CGR encoder starts with reasonable weights, and allows the MoE gating network to better learn how to balance the two experts.
Data: All experiments are uncased, using standard benchmarks (CoNLL03, OntoNotes, WNUT17) and the new datasets we create (see §3).
Models: We integrate GEMNET with both BERT and BiLSTM word encoders. For BERT, we use the pretrained  $\mathrm{BERT}_{\mathrm{BASE}}$  model. The last output layer is used, and for each word, we use the first wordpiece representation as its representation. The BiLSTM model has 3 inputs: GloVe embeddings (Pennington et al., 2014), ELMo embeddings (Peters et al., 2018) and CharCNN embeddings (Ma and Hovy, 2016).
Evaluation: We evaluate MD and NER, and report entity-level precision, recall and F1 scores.
5.1 MD Baselines
Our first experiment aims to measure the difficulty of our datasets (§3) relative to existing benchmarks. We train a BERT model on CoNLL03 and use it to measure MD performance on our data. Measuring NER performance is not possible as we use a different tag set (WNUT17 vs CoNLL03).
Dataset	P	R	F1
CoNLL03	96.9	95.7	96.3
LOWER	67.5	74.5	70.9
MSQ-NER	38.9	38.7	38.8
ORCAS-NER	56.8	51.6	54.1