Venue: NAACL 2021

GemNet: Effective Gated Gazetteer Representations for Recognizing Complex Entities in Low-context Input

 Tao Meng

UCLA

Los Angeles, USA &Anjie Fang

Amazon.com, Inc.

Seattle, WA, USA &Oleg Rokhlenko

Amazon.com, Inc.

Seattle, WA, USA tmeng@cs.ucla.edu {njfn,olegro,malmsi}@amazon.com

This research was done during an internship at Amazon.

###### Abstract

Named Entity Recognition (NER) remains difficult in real-world settings; current challenges include short texts (low context), emerging entities, and complex entities (e.g. movie names). Gazetteer features can help, but results have been mixed due to challenges with adding extra features, and a lack of realistic evaluation data. It has been shown that including gazetteer features can cause models to overuse or underuse them, leading to poor generalization. We propose GemNet, a novel approach for gazetteer knowledge integration, including (1) a flexible Contextual Gazetteer Representation (CGR) encoder that can be fused with any word-level model; and (2) a Mixture-of-Experts gating network that overcomes the feature overuse issue by learning to conditionally combine the context and gazetteer features, instead of assigning them fixed weights. To comprehensively evaluate our approaches, we create 3 large NER datasets (24M tokens) reflecting current challenges. In an uncased setting, our methods show large gains (up to \(+49\%\) F1) in recognizing difficult entities compared to existing baselines. On standard benchmarks, we achieve a new uncased SOTA on CoNLL03 and WNUT17.

## 1 Introduction

Identifying entities is a core NLP task. Named Entity Recognition (NER) is the task of finding entities and recognizing their type (e.g., person or location). Mention Detection (MD) is a simpler task of identifying entity spans, without the types.

Advances in neural NER have produced high scores on benchmark datasets like CoNLL03 and OntoNotes Devlin et al. (2019). However, a number of challenges remain. As noted by Augenstein et al. (2017), these scores are driven by the use of well-formed news text, the presence of "easy" entities, and memorization effects due to entity overlap between train/test sets; these models perform significantly worse on unseen entities or noisy text.

### Current NER Challenges

Beyond news text, many challenges remain in NER. Context information has been shown to be important for NER Jayarao et al. (2018), and short texts like search queries are very challenging due to low context and a lack of surface features Guo et al. (2009); Carmel et al. (2014). Unseen and emerging entities also pose a challenge Bernier-Colborne and Langlais (2020). Finally, some entities, like movie names are not simple noun phrases and are harder to recognize Ashwini and Choi (2014). Table 1 lists more details about these challenges, and how they can be evaluated.

Entity Knowledge is essential for overcoming these issues, and critical in the absence of casing. Even a human may not correctly parse "what is [life is beautiful]?" without knowing that a movie is being referenced. However, most models start with no knowledge of real world entities, learning them from the training data. Continuous data annotation can add new entities, but is expensive and often not feasible.

Consequently, methods for integrating external knowledge, e.g., Knowledge Bases (KBs) or gazetteers, into neural architectures have gained renewed attention. However, such studies have reported limited gains Liu et al. (2019); Rijhwani et al. (2020). The mixed success of gazetteers stems from three main limitations in current work: gazetteer feature representation, their integration with contextual models, and a lack of data.

For the representation, one-hot binary encoding is often used to represent gazetteer features Song et al. (2020). However, this does not capture contextual info or span boundaries. Alternatively, independent span taggers trained on gazetteers have been proposed to extract potential entities Liu et al. (2019), but such models can be difficult to train and may not provide reliable features.

 

There are also limitations in the integration of gazetteer features. Existing studies often add extra features to a word-level model's Contextual Word Representations (CWRs), which typically contain no info about real world entities or their spans Yamada et al. (2020). This concatenation approach is sub-optimal as it creates additional, and often highly correlated features. This has been shown to cause feature "under-training", where the model will learn to mostly rely on either context or gazetteer during training, and underuse the other Yang et al. (2016). This can be problematic as the utility of the gazetteer is variable: it is valuable in low-context cases, but may not be useful when rich syntactic context (from the CWR) can identify entities. Conversely, a true entity may be missing from the gazetteer. However, when gazetteers are represented as an independent feature, the model assigns it a fixed weight, and its contribution to the prediction is static. To overcome this, external knowledge should dynamically be infused into relevant dimensions of the CWR, with the model learning to conditionally balance the contribution of the CWR and gazetteer to the prediction.

Finally, these issues are compounded by a lack of data reflecting the challenges from Table 1, which prevents the exploration of effective architectures for knowledge injection.

### Our Contributions

The key contributions of this paper are new data and methods to address the above challenges.

We propose GemNet, a gazetteer expert mixture network for effectively integrating gazetteers into any word-level model. The model includes an encoder for **Con**textual **G**azeetteer **R**epresentations (CGRs) as a way to incorporate any number of gazetteers into a single, span-aware, dense representation. We also propose a gated Mixture-of-Experts (MoE) method to fuse CGRs with Contextual Word Representations from any word-level model (e.g., BERT), something not explored in previous work. Our novel MoE approach allows the model to conditionally compute a joint CGR-CWR representation, training a gating network to learn how to balance the contribution of context and gazetteer features. Finally, we employ multi-stage training to drive further improvements by aligning the CGR/CWR vectors.

To evaluate our proposed approaches, we create 3 challenging NER datasets that represent short sentences, questions, and search queries. The created datasets have complex entities with low-context and represent the challenges in Table 1.

Extensive experiments in an uncased setting show that our MoE model outperforms other baselines, including concatenation, in all experiments. We achieve state-of-the-art (SOTA) results on CoNLL03/WNUT17, but its utility is more notable on our difficult low-context data. We show that short texts make NER much harder, but gazetteers yield huge gains of up to \(+49\%\) F1, specially in recognizing complex/unseen entities. We also show that gazetteer coverage during training is important.

## 2 Related Work

Deep Learning for NERNeural approaches have greatly improved NER results in recent years. A shift to encoders e.g., BiLSTM-CRF models Huang et al. (2015), using static word embed

\begin{table}
\begin{tabular}{l|p{85.4pt}} \hline Challenge & Description \\ \hline
**Short Texts** & News texts have long sentences discussing many entities, but other use cases (search queries, questions) have shorter inputs. Datasets with minimal context are needed to assess performance of such us cases. Capitalization/punctuation features are large drivers of success in NER Mayhew et al. (2019), but short inputs (ASR, user input) often lack these surface feature. An uncased evaluation setting is needed to understand model performance. \\ \hline
**Long-tail Entities** & In many domains entities have a large long-tail distribution, with millions of values (e.g., location names). This makes it hard to build representative training data, as it can only cover a portion of the potentially infinite entity space. A very large test set is required for effective evaluation. \\ \hline
**Emerging Entities** & All entity types are open classes (new ones are added), but some groups have a faster growth rate, e.g., new books/songs/movies are released weekly. Assessing true generalization requires test sets with many unseen entities, to mimic an open-world setting. \\ \hline
**Complex Entities** & Not all entities are proper names: some types (e.g. creative works) can be linguistically complex. They can be complex noun phrases (Eternal Sunshine of the Spotless Mind), gerunds (Saving Private Ryan), infinitesives (To Kill à Mockingbird), or full clauses (Mr.Smith Goes to Washington). Syntactic parsing of such nouns is hard, and most current parsers/NER systems fail to recognize them. The top system from WNUT 2017 achieved 8\% recall for creative work entities Aguilar et al. (2017). Effective evaluation requires corpora with many such entities. \\ \hline \end{tabular}
\end{table}
Table 1: NER challenges not addressed by current work and datasets, and proposed solutions.

 

dings eliminated the need for manual feature engineering (e.g., capitalization features). More recently, transformer-based Language Models (LMs), e.g., BERT Devlin et al. (2019), achieved further improvements by using deep contextual word representations. Such models jointly learn syntactic cues and entity knowledge, and may fail to recognize unseen or syntactically ambiguous entities. Consequently, training data is augmented with gazetteers.

NER with GazetteersAnnotated NER data can only achieve coverage for a finite set of entities, but models face a potentially infinite entity space in the real world. To address this, researchers have integrated gazetteers into models Bender et al. (2003); Malmasi and Dras (2015). String matching is commonly used to extract gazetteer matches, which are then concatenated to word representations. Song et al. Song et al. (2020) use gazetteers from the Wikidata KB to generate one-hot vectors that are concatenated to BERT representations, yielding minor improvements on CoNLL03. This concatenation approach has been shown to cause feature "under-training" Yang et al. (2016), as discussed in SS1. An alternative approach uses gazetteers to train a subtagger model to recognize entity spans. Liu et al. Liu et al. (2019) propose a hybrid semi-Markov CRF subdaggerger, reporting minor improvements. While a subtagger may learn regularities in entity names, a key limitation is that it needs retraining and evaluation on gazetteer updates. Recent work has considered directly integrating knowledge into transformers, e.g., KnowBert adds knowledge to BERT layers Peters et al. (2019), and LUKE is pretrained to predict masked entities Yamada et al. (2020). The drawbacks of such methods are that they are specific to Transformers, and the model's knowledge cannot be updated without retraining. We aim to overcome the limitations of previous work by designing a model-agnostic gazeteer representation that can be fused into any word-level model.

Mixture-of-Experts (MoE) ModelsMoE is an approach for conditionally computing a representation, given several expert inputs, which can be neural models with different architectures Arnaud et al. (2020) or models using different knowledge sources Jain et al. (2019). In MoE, a gating network is trained to dynamically weight experts per instance, according to the input. It has demonstrated to be useful in various applications like recommendation Zhu et al. (2020), domain adaptation for sentiment analysis, and POS tagging Guo et al. (2018). For NER, Liu et al. Liu et al. (2020) proposed a Mixture of Entity Experts (MoEE) approach where they train an expert layer for each entity type, and then combine them using an MoE approach. Their approach does not include external gazetteers, and the experts provide an independent representation that is not combined with the word representation. In our work we treat word and external gazetteer representations as independent experts, applying MoE to learn a dynamically fused representation.

## 3 Datasets

We experiment using three standard benchmarks: CoNLL03, OntoNotes, and WNUT17. However, these corpora do not capture the issues from Table 1; rich context and common entities (country names) allow a simple RNN model to achieve near-SOTA results. A key contribution of our paper is the creation of 3 new datasets that represent those challenges. They are difficult, as shown in SS5.1.

NER Taxonomy:We adopt the WNUT 2017 Deczynski et al. (2017) taxonomy entity types: PERSON (PER for short, names of people), LOCATION (LOC, locations/physical facilities), CORPORATION (CORP, corporations and businesses), GROUPS (GRP, all other groups), PRODUCT (PROD, consumer products), and CREATIVE-WORK (CW, movie/song/book/etc. titles).

Our datasets are described below.1 All data are uncased, and we make them publicly available.2 Their statistics, listed in Table 2, show that they reflect the challenges from SS1: short inputs (low context), with many unseen entities in the test set.

Footnote 1: More details about their development are in Appendix A

Lowner (Low-Context Wikipedia NER)To create our training set, we take advantage of the rich interlinks in Wikipedia. We parse the English Wikipedia dump and extract sentences from all articles. The sentences are parsed, and linked pages are resolved to their respective Wikidata entities to identify their type. To mimic search and voice settings, we minimize the context around the entities by dropping sentences with unlinked entities, identified using interlinks and a capitalization heuristic. The result is a corpus of 1.4 million low-context sentences with annotated entities, e.g., "A version for the [sega cd] was also announced." 

Msq-Ner: MS-MARCO Question NERTo represent NER in the QA domain, we create a set of natural language questions, based on the MSMC0 QnA corpus (V2.1) Bajaj et al. (2016). Like Wu et al. (2020), we templatize the questions by applying NER to extract item names, which are then mapped to our taxonomy. Entities are replaced with their types to create templates, e.g., "who sang <CW>" and "when did <PROD> come out". Approx \(3.5{\rm k}\) Templates (appearing \(>=5\) times) are chosen and slotted with entities from a knowledge base to generate \(~{}18{\rm k}\) annotated questions e.g., "when did [xbox 360] come out". There are a wide range of question shapes and entity types, please see Appendix A for examples.

Orcas-Ner: Search Query NERTo represent the search query domain, we utilize \(10\) million Bing user queries from the ORCAS dataset Craswell et al. (2020) and apply the same templatization procedure as Msq-Ner. This yields search templates e.g., "<PROD> price" and "<CORP> phone number", which are used to create annotated queries, e.g., "[airpods pro] reviews". A total of \(472{\rm k}\) queries are generated from \(97{\rm k}\) unique templates, please see examples in Appendix A.

### Gazetteer Data

Our gazetteer is composed of \(1.67\) million entities from the English Wikidata KB. Instead of collecting entities from the web Khashabi et al. (2018), we focused on entities that map to our taxonomy. Alternative names (aliases) for entities are included. Gazetteer statistics are listed in Appendix B.

## 4 The GemNet Model

We propose GemNet, a generic gazetteer fusion approach that can be integrated with any word-level model, e.g., RNNs and Transformers. We experiment with both BiLSTM-CRF and BERT-CRF models which produce (contextual) word representations, and complement these "word experts" with gazetteers. The overall architecture is shown in Figure 1, and the components are detailed below.

### Contextual Gazetteer Representations

Our gazetteer representations is obtained in two steps: entry matching, and contextual encoding.

Gazetteer Entry MatchingAgazetteer **g** is a list of entries that are associated with a category. For instance, a PERSON gazetteer contains a list of known people. The \(k\)-th entry \({\bf g}^{(k)}\) is associated with a tokenized string ('John Carpenter') and \({\bf t}^{(k)}\) holds the IOB2 tags ([B-PER, I-PER]). We use \(T\) to denote the tag set over all gazetteers, e.g., \(T=\{\texttt{B-PER},\text{I-PER},\text{B-LOC},\text{I-LOC},\dots\}\).

We denote input sentences as \((w_{1},w_{2},\ldots,w_{L})\), where \(w_{i}\) is the \(i\)-th word, and \(L\) is the length. Full string matching is applied to inputs to identify matches across all gazetteers. Overlapping matches are resolved by preferring longer ones over shorter ones, and earlier matches over later ones. A match matrix, \({\mathbf{M}}\in\{0,1\}^{L\times|T|}\), represents the matching results. It is initialized with zeros, and successful matches \((w_{i},w_{i+1},\ldots,w_{i+m})={\bf g}^{(k)}\) will set

\[{\mathbf{M}}_{i+j,{\bf t}_{j}^{(k)}}=1,~{}j=0,1,\ldots,m,\]

indicating that the word \(w_{i+j}\) is represented by a one-hot vector over the tag set \(T\).

A key advantage of this representation is that it captures multiple matches for a given span in a sentence. As shown in Table 3, the word "apple" can be matched to product and organization types. Furthermore, it is span-aware due to the IOB2 encoding. Any number of types and gazetteers can be added as needed, allowing the model to learn from correlations, and identify ambiguous entities.

\begin{table}
\begin{tabular}{c|l|r|r r r r|r r r r r} \hline  & \multicolumn{1}{c|}{Dataset} & Type & \multicolumn{1}{c|}{\# Sentence} & \multicolumn{1}{c|}{\# Token} & \multicolumn{1}{c|}{\# Entity} & Avg. Sent Len & \multicolumn{1}{c}{Entity Type Distribution} \\  & & & & & & & & & & & \\  & & & & & & & & & & & \\ Set & Dataset & Type & \# Sentence & \# Token & \# Entity & Verg. Rep & LOC & CORP & GRP & PROD & CM \\ \hline
1 & Lower & Train & 13,424 & 206,772 & 13,555 & 15.40\(\pm\)6.35 & 5,029 & 3,791 & 631 & 1,941 & 424 & 1,805 \\
2 & Lowner & Dev & 3,366 & 51,651 & 3,813 & 15.34\(\pm\)6.28 & 1,255 & 1,235 & 169 & 565 & 101 & 499 \\
3 & Lowner & Test & 1,385,290 & 21,303,399 & 490,749 & 15.37\(\pm\)6.29 & 215,411 & 120,480 & 20,015 & 52,566 & 15,976 & 74,830 \\
4 & Msq-Ner & Test & 17,868 & 98,117 & 18,993 & 5.49\(\pm\)1.86 & 4,586 & 10,468 & 679 & 610 & 469 & 2,187 \\
5 & Orcas-Ner & Test & 471,746 & 1,958,020 & 368,250 & 4.15\(\pm\)1.75 & 68,000 & 162,652 & 28,738 & 23,058 & 18,114 & 71,461 \\ \hline \end{tabular}
\end{table}
Table 2: Data statistics. Entity counts are unique values. Löwner has train/dev/test sets, the rest are test sets.

 

\({\bf M}\) is extracted by a gazetteer matcher, as a preprocessing step outside the network. This modular approach has an important advantage: it allows the gazetteer to be updated without model retraining. This is useful for efficiently recognizing emerging entities, and supporting personalized user-defined entities (e.g., contact lists).

Contextual Encoding\({\bf M}\) can be directly used as input features, but is sparse. We use a linear projection to obtain a dense representation per word:

\[{\bf h}^{i}_{gaz}=f({\bf w}\cdot{\bf M}_{i}+{\bf b})\]

where \({\bf w}\in\mathbb{R}^{D\times T}\) and \({\bf b}\in\mathbb{R}^{D}\) are trainable parameters, \(D\) is the hidden dimension of gazetteer representation and \(f\) is an activation function. This creates a dense representation that captures interactions between multiple matches. We then contextualize this representation by applying a BiLSTM:

\[{\bf h}^{i}_{forward} ={\rm LSTM}({\bf h}^{i-1}_{forward},{\bf h}^{i}_{gaz})\] \[{\bf h}^{i}_{backward} ={\rm LSTM}({\bf h}^{i+1}_{backward},{\bf h}^{i}_{gaz})\] \[{\bf h}^{i}_{CGR} =[{\bf h}^{i}_{forward},{\bf h}^{i}_{backward}]\]

where \([\cdot,\cdot]\) is the concatenation. A sample visualization of the embeddings is shown in Appendix D.

This dense **contextualized gazetteer representation** (CGR) can capture information about entity span boundaries (present in \({\bf M}\)), as well as interactions between entities in a sentence.

Figure 1: GemNet model architecture. Input is passed to the word expert (e.g. BERT) and its match matrix to the Contextual Gazetteer Representation (CGR) encoder. Their outputs are dynamically combined via a Mixture-of-Experts (MoE) gating network and passed to a CRF layer for word prediction.

 



### Gazetteer Knowledge (CGR) Integration

The CGR operates on IOB2 tags and cannot memorize specific patterns; it is designed to be integrated with a lexical model. We consider these representations to be orthogonal: CGRs can complement the model's knowledge and syntactic representation.

CGR ConcatenationThe simplest integration is to concatenate the dense CGR to the CWR, while jointly training the two representations.

Mixture-of-Experts (MoE) ModelThe word-level model and CGRs complement each other and may not always be in agreement. The word model may have low confidence about the span of an unseen entity, but the gazetteer may have knowledge of it. Conversely, the model's syntactic context may be confident about a span not in the gazetteer.

In fact, the two sources can be considered as independent experts and an effective model should learn to use their outputs dynamically. Inspired by the MoE architecture Pavlitskaya et al. (2020), we apply conditional computation to combine our representations, allowing the model to learn the contexts where it should rely more on each expert.

We add a gating network to create a weighted linear combination of the word and gazetteer representations. For a sentence, the two models output3 their representations \({\bf h}_{word}\) and \({\bf h}_{gaz}\), which are used to train the gating network:

Footnote 3: Outputs sizes must be equal, e.g., CGR must match BERT.

\[w_{e} =\sigma(\theta[{\bf h}_{word},{\bf h}_{CGR}]),\] \[{\bf h} =w_{e}\cdot{\bf h}_{word}+(1-w_{e})\cdot{\bf h}_{CGR},\]

where \(\theta\) are trainable parameters with size \(2L\), \([\cdot,\cdot]\) is the concatenation and \(\sigma\) is the Sigmoid activation function. We learn gating weights, \(w_{e}\), so that the model can learn to dynamically compute the hidden information \({\bf h}\) for each word. The architecture of our model is shown in Figure 1. After obtaining \({\bf h}\), we feed it to a CRF layer to predict a tag.

Two-stage TrainingOur architecture jointly optimizes over both experts, but their initial states differ. The word expert often contains pretrained elements, either as word embeddings or transformers. The randomly-initialized CGR will have high initial loss, and its representation is not aligned with the word expert, preventing correct convergence. We tackle this problem through a two-stage training method to adapt the two experts to each other. In the first stage, we freeze the word expert and only train the CGR encoder with the MoE and CRF layers, forcing the model to use gazetteer knowledge in order to minimize the loss. Importantly, this also adapts the CGR encoder to align its representation with that of the word expert, e.g., the dimensions with noun signals will be aligned with those of BERT, enabling the computation of their linear combination. In the second stage, the two experts are jointly fine-tuned to co-adapt them. This ensures that the CGR encoder starts with reasonable weights, and allows the MoE gating network to better learn how to balance the two experts.

## 5 Experiments

DataAll experiments are uncased, using standard benchmarks CoNLL03, OntoNotes, WNUT17) and the new datasets we create (see SS3).

ModelsWe integrate GemNet with both BERT and BiLSTM word encoders.For BERT, we use the pretrained BERTBASE model. The last output layer is used, and for each word, we use the first wordpiece representation as its representation. The BiLSTM model has 3 inputs: GloVe embeddings Pennington et al. (2014), ELMo embeddings Peters et al. (2018) and CharCNN embeddings Ma and Hovy (2016).

Evaluation:We evaluate MD and NER, and report entity-level precision, recall and F1 scores.

### MD Baselines

Our first experiment aims to measure the difficulty of our datasets (SS3) relative to existing benchmarks. We train a BERT model on CoNLL03 and use it to measure MD performance on our data. Measuring NER performance is not possible as we use a different tag set (WNUT17 vs CoNLL03).

Results:Compared to the CoNLL03 results, the Lowner performance is worse. Although the evaluation on Lowner is a transfer setting, the large gap shows the existing model cannot generalize

\begin{table}
\begin{tabular}{l c c c} \hline Dataset & P & R & F1 \\ \hline \hline CoNLL03 & 96.9 & 95.7 & 96.3 \\ \hline Lowner & 67.5 & 74.5 & 70.9 \\ Msq-Ner & 38.9 & 38.7 & 38.8 \\ Orcas-Ner & 56.8 & 51.6 & 54.1 \\ \hline \end{tabular}
\end{table}
Table 4: Mention detection (MD) results for a BERT model trained on CoNLL03, tested on our data.



well to our datasets due to the hard entities. Results for Msq-Ner and Orcas-Ner, which are short texts, are even lower. Overall, we note the difficulty of our datasets due to low context and hard entities.

### NER Ablation Experiments

We explore all model architectures by training on Lowner (set 1 in Table 2) and evaluating MD and NER performance on all datasets (sets 3-5 in Table 2). See Appendix C for training details.

Models:The GemNet model is jointly trained and fused with BERT and BiLSTM word encoders, with and without two-stage training. To assess the impact of the MoE component, we also concatenate the CGR and CWR vectors, without MoE.

Baselines:We compare against three baselines: (1) no gazetteer baselines; (2) binary concatenation: we simply concatenate the binary match features (M) to the word representations, as is common in the literature; (3) the subtagger model of Liu et al. (2019). They are shown as "baselines" in table 5.

Results:MD and NER performance for all models is shown in Table 5. Overall we note the high effectiveness of the GemNet model. In particular, our BiLSTM-based GemNet approach improves F1 by up to \(49\%\) over the no gazetteer BiLSTM baseline in Orcas-Ner. Different aspects of the results are discussed below.

Word Encoder Performance:For Lowner, we note that BERT achieves the best results, which is to be expected since the data consists of full sentences. MD is easier than NER, and represents the upper bound for NER. Performance in all cases decreases with low context, with search queries (Orcas-Ner) being the hardest. BiLSTMs perform better on shorter inputs, e.g., Orcas-Ner.

Impact of Gazetteers:Results improve in all cases with external knowledge. While the subtagger and the binary concatenation baselines yield gains compared to the no gazetteer baselines, our CGR-based approach outperforms all of them in all NER tests. This indicates the high effectiveness of our CGR. For Lowner, using CGR+MoE, MD performance improves by \(2.4\%\), while NER increases \(4.7\%\) over the no gazetteer BERT baseline. Low-context data, Msq-Ner and Orcas-Ner, have much lower baseline performance, and benefit greatly from external knowledge. The best Msq-Ner NER model improves \(36\%\) over the no gazetteer BiLSTM baseline, while Orcas-Ner increases by \(49\%\). This clearly demonstrates the impact of gazetteer integration.

Effect of Integration Method:CGR outperforms baselines in all NER experiments, showing the effectiveness of a span-aware, contextual representation that is jointly trained with the word-level model. The MoE integration is superior to concatenation in all cases. This is more salient in low context settings, demonstrating that the MoE model can rely on the CGR feature when the syntactic context (CWR) is not discriminative. In some cases baselines actually degrade performance as the model can not effectively balance the experts.

Effect of Two-stage Training:We observe that two-stage training is crucial for BERT, including concatenation models and MoE models, but not for the BiLSTM model. This confirms our hypothesis that the CGR cannot be jointly trained with a large pretrained model. Freezing BERT and then jointly fine-tuning them provides great improvements.

Results on Benchmarks:We applied GemNet, i.e., BERT using CGR+MoE with two stage training, to the standard benchmarks. We experiment in an uncased setting, and and compare with the reported uncased SOTA Mayhew et al. (2019). The SOTA uses BERT-CRF, which are the same as our baseline architecture. For comparison, we also reproduce the BERT baseline using our implementation. Results are shown in Table 6. Our models achieve SOTA results in all uncased settings, demonstrating generalization across domains; we improve by 3.9% on WNUT17.

### Per-Class Performance & Error Analysis

We also look at performance across different entity classes to understand the source of our improvements. Table 7 shows relative gains per class, comparing the no gazetteer baseline performance against the best model. Detailed precision/recall values are in Appendix E (Table 16).

The smallest gains are on PER and LOC types, and the largest gains are on products and creative works (CW). This agrees with our hypothesis that these complex entities are the hardest to recognize.

Comparing datasets, increases are much larger on Msq-Ner and Orcas-Ner, confirming the challenges of short low-context inputs, and our models effectiveness in such cases 

We also conduct a qualitative error analysis to identify instances where the best non-gazetteer baseline fails, but our model provides correct output. Some examples are shown in Table 8. The baseline often lacks knowledge about complex and long-tail entities, either missing them (#1,6,8 show full or partial MD failure) or misclassifying them (#3-5 show NER errors). Another common trend we observe is baselines incorrectly predicting nested entities within complex entities (#2,10).

### Effect of Gazetteer Coverage

We consider the impact of gazetteer coverage4 on performance. We hypothesize that training coverage impacts how much the model learns to rely on the gazetteer. To verify this we examine two scenarios: (1) the gazeteer coverage for train and test match (i.e., both high or low); and (2) there is a coverage gap between train and test, e.g., train coverage is 90% but is 25% for test, or vice versa.

Footnote 4: The proportion of entities that are present in the gazetteer case can be seen as a function of the number of cases.

**Model and Data:** For each train/test set we create gazetteers that have \(p\%\) coverage of the set's gold entities, with \(p\in\{5,10,20,30,50,75,90,95\}\). This is achieved by randomly dropping entities. We then train models using each \(p\) and evaluate on test sets, using all values of \(p\). This experiment is done using Lowner and Msq-Ner.

\begin{table}
\begin{tabular}{l|c c c} \hline Class & Lowner & Msq-Ner & Orcas-Ner \\ \hline \hline PER & \(+1.9\) & \(+21.8\) & \(+40.1\) \\ LOC & \(+2.2\) & \(+37.5\) & \(+46.5\) \\ GRP & \(+8.5\) & \(+57.3\) & \(+57.2\) \\ CORP & \(+12.7\) & \(+57.7\) & \(+56.5\) \\ CW & \(+10.2\) & \(+58.8\) & \(+61.4\) \\ PROD & \(+10.7\) & \(+64.2\) & \(+62.0\) \\ \hline \end{tabular}
\end{table}
Table 6: Uncased NER Results (F1 score) on CoNLL03, WNUT17 and OntoNotes v5.0.

Figure 2: Coverage analysis. X-axis is the testing coverage and Y-axis is the training coverage. (a) shows results for the Löwner test set, and (b) shows the results for Msq-Ner.

 

Results:Results are plotted as heatmaps in Figure 2. Best results occur with high train and test coverage, while the worst results fall under high training coverage but low test coverage. When train coverage is low, test coverage has no impact as the model presumably ignores the gazetteer input. Across test coverage values, best results are generally around the diagonal, i.e., matching training coverage. These patterns are identical across datasets, indicating that a train/test coverage gap should be avoided. In practice, if test set coverage cannot be measured, or high coverage is not guaranteed, then using lower training coverage (e.g., \(50\%\)) prevents performance degradation in very low test coverage cases.

We also note that the gap between the best and worst result for Lowner is not huge, showing the impact of sentence context. This gap is much larger for Orcas-Ner, where the model cannot rely on the context. Finally, we note that an alternative dynamic dropout method5 achieved similar results.

Footnote 5: Gazeteter matches are randomly dropped during training (i.e., random entity dropout).

### Performance in a Low-Resource Setting

We also consider the impact of a low-resource setting (limited annotations) on performance, hypothesizing that gazetteers are more helpful in such settings. To verify this, we create random subsets of 5/10/20% of the training data and compare the NER performance of a baseline (BERT-base) vs our best model(BERT+CGR+MoE+2stage) when trained on this data. Results are shown in Table 9.

The results show that gazetteers are always more effective than baseline in low-resource scenarios. Specifically, they improve much faster with less data, achieving close to maximum performance with only 20% of the data.

## 6 Conclusion

We focused on integrating gazetteers into NER models. We proposed GemNet, a flexible architecture that includes a Contextual Gazetteer Representation encoder, combined with a novel Mixture-of-Expert gating network to conditionally utilize this information alongside any word-level model. GemNet supports external gazetteers, allowing the model's knowledge to be updated without retraining.

We also developed new datasets to represent the current challenges in NER. Experimental results demonstrated that our method can alleviate the feature weight under-training issue, achieving significant improvements on our data and a standard benchmark, WNUT17. The datasets we released can serve as benchmarks for evaluating the entity knowledge possessed by models in future work.

Future work involves investigating integration with different model architectures, partial gazetteer matching, and additional entity features.

## Acknowledgements

We would like to extend our gratitude to Eugene Agichtein, Alexandre Salle, and Besnik Fetahu for their valuable inputs and discussion during this project. We also thank the anonymous reviewers for their constructive remarks.

\begin{table}
\begin{tabular}{c|l|c|p{348.3pt}} \hline Dataset & Gold Sentence & Entities by Baseline & Entities by Best Model \\ \hline \hline  & Example 1: he worked for & – & linear technology\(|_{CORP}\) \\ Lower & \begin{tabular}{l} bilinear technology\(|_{CORP}\) and \(|\text{analog devices}|_{CORP}\) \\ Example 2: his signature piece was \\ orange blossom special\(|_{CW}\) \\ Example 3: it is the last of the \(||\text{heinlein juveniles}|_{CW}\) \\ \end{tabular} & 
\begin{tabular}{l} \(|\text{a}|\text{a}\text{o}\text{z}\text{b}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{ z}\text{o}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{ o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z}\text{o} \text{z}\text{o}\text{z}\text{o}\text{z}\text{o}\text{z 



## References

* Aguilar et al. (2017) Gustavo Aguilar, Suraj Maharjan, Adrian Pastor Lopez-Monroy, and Thamar Solorio. 2017. A multi-task approach for named entity recognition in social media data. In _Proceedings of the 3rd Workshop on Noisy User-generated Text_, pages 148-153.
* Arnaud et al. (2020) Estephe Arnaud, Arnaud Dapogny, and Kevin Bailly. 2020. Tree-gated deep mixture-of-experts for pose-robust face alignment. _IEEE Trans. Biom. Behav. Identity Sci._, 2(2):122-132.
* Ashwini and Choi (2014) Sandeep Ashwini and Jinho D. Choi. 2014. Targetable named entity recognition in social media. _CoRR_, abs/1408.0782.
* Augenstein et al. (2017) Isabelle Augenstein, Leon Deczzynski, and Kalina Bontcheva. 2017. Generalisation in named entity recognition: A quantitative analysis. _Computer Speech & Language_, 44:61-83.
* Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. _arXiv preprint arXiv:1611.09268_.
* Bender et al. (2003) Oliver Bender, Franz Josef Och, and Hermann Ney. 2003. Maximum entropy models for named entity recognition. In _CoNLL_, pages 148-151. ACL.
* Bernier-Colborne and Langlais (2020) Gabriel Bernier-Colborne and Philippe Langlais. 2020. Hardeval: Focusing on challenging tokens to assess robustness of NER. In _LREC_, pages 1704-1711. European Language Resources Association.
* Carmel et al. (2014) David Carmel, Ming-Wei Chang, Evgeniy Gabrilovich, Bo-June Paul Hsu, and Kuansan Wang. 2014. Erd'14: entity recognition and disambiguation challenge. In _SIGIR_, page 1292. ACM.
* Caswell et al. (2020) Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. 2020. Orcas: 18 million clicked query-document pairs for analyzing search. _arXiv preprint arXiv:2006.05324_.
* Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In _Proceedings of the 3rd Workshop on Noisy User-generated Text_, pages 140-147.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _NAACL-HLT (1)_. Association for Computational Linguistics.
* Guo et al. (2009) Jiafeng Guo, Gu Xu, Xueqi Cheng, and Hang Li. 2009. Named entity recognition in query. In _SIGIR_, pages 267-274. ACM.
* Guo et al. (2018) Jiang Guo, Darsh J. Shah, and Regina Barzilay. 2018. Multi-source domain adaptation with mixture of experts. In _EMNLP_, pages 4694-4703. Association for Computational Linguistics.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _arXiv preprint arXiv:1508.01991_.
* Jain et al. (2019) Abhinav Jain, Vishwanath P. Singh, and Shakti P. Rath. 2019. A multi-accent acoustic model using mixture of experts for speech recognition. In _INTERSPEECH_, pages 779-783. ISCA.
* Jayarao et al. (2018) Pratik Jayarao, Chirag Jain, and Aman Srivastava. 2018. Exploring the importance of context and embeddings in neural NER models for task-oriented dialogue systems. _CoRR_, abs/1812.02370.
* Khashabi et al. (2018) Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev-Are Ratinov, Guanneng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhili Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan, Qiang Ning, Shaoshi Ling, and Dan Roth. 2018. Cogcompnlnp: Your swiss army knife for NLP. In _LREC_. European Language Resources Association (ELRA).
* Liu et al. (2019) Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019. Towards improving neural named entity recognition with gazetteers. In _ACL (1)_, pages 5301-5307. Association for Computational Linguistics.
* Liu et al. (2020) Zihan Liu, Genta Indra Winata, and Pascale Fung. 2020. Zero-resource cross-domain named entity recognition. In _Proceedings of the 5th Workshop on Representation Learning for NLP_, pages 1-6, Online. Association for Computational Linguistics.
* Ma and Hovy (2016) Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _ACL (1)_. The Association for Computer Linguistics.
* Maiaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(Nov):2579-2605.
* Malmasi and Dras (2015) Shervin Malmasi and Mark Dras. 2015. Location mention detection in tweets and microblogs. In _Conference of the Pacific Association for Computational Linguistics_, pages 123-134. Springer.
* Mayhew et al. (2019) Stephen Mayhew, Tatiana Tsygankova, and Dan Roth. 2019. ner and pos when nothing is capitalized. In _EMNLP/IJCNLP (1)_, pages 6255-6260. Association for Computational Linguistics.
* Pavlitskaya et al. (2020) Svetlana Pavlitskaya, Christian Hubschneider, Michael Weber, Ruby Moritz, Fabian Huger, Peter Schlicht, and J. Marius Zollner. 2020. Using mixture of expert models to gain insights into semantic segmentation. In _CVPR Workshops_, pages 1399-1406. IEEE.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, pages 1532-1543. ACL

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _NAACL-HLT_, pages 2227-2237. Association for Computational Linguistics.
* Peters et al. (2019) Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 43-54.
* Rijhwani et al. (2020) Shruti Rijhwani, Shuyan Zhou, Graham Neubig, and Jaime G. Carbonell. 2020. Soft gazetteers for low-resource named entity recognition. In _ACL_, pages 8118-8123. Association for Computational Linguistics.
* Song et al. (2020) Chan Hee Song, Dawn Lawrie, Tim Finin, and James Mayfield. 2020. Improving neural named entity recognition with gazetteers. _arXiv preprint arXiv:2003.03072_.
* Wu et al. (2020) Tongshuang Wu, Kanit Wongsuphasawat, Donghao Ren, Kayur Patel, and Chris DuBois. 2020. Tempura: Query analysis with structural templates. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_, pages 1-12.
* Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: Deep contextualized entity representations with entity-aware self-attention. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6442-6454.
* Yang et al. (2016) Eun-Suk Yang, Young-Bum Kim, Ruhi Sarikaya, and Yu-Seop Kim. 2016. Drop-out conditional random fields for twitter with huge mined gazetteet. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 282-288.
* Zhu et al. (2020) Ziwei Zhu, Shahin Safati, Parsa Saadatpanah, and James Caverlee. 2020. Recommendation for new users and new items via randomized training and mixture-of-experts transformation. In _SIGIR_, pages 1121-1130. ACM.

 



## Appendix A Dataset Details

Lowner:This dataset is based on Wikipedia, and uses the links as span annotations.

The complete English Wikipedia dump from July 2020 was downloaded. We extracted the articles, which were then parsed to remove markup and extract sentences with their interlinks (links to other articles). This resulted in the extraction of approx. 180 million sentences. We then mapped the interlinks in each sentence to the Wikidata KB then resolved them to our NER taxonomy (in same manner as Appendix B).

Next, we filtered sentences using two strategies. Taking advantage of Wikipedia's well-formed text, we applied a Regex-based NER method to identify sentences containing named entities that were not linked, and removed them. This removes long and high-context sentences that contain references to many entities. Additionally we also removed any sentence where the links could not be resolved to Wikidata entities. This process discards over 90% of the sentences, resulting in approx. 14 million candidate sentences.

This process is very effective at yielding short, low-context sentences. Example sentences are shown in Table 10. The sentences contain some context, but they are much shorter than the average Wikipedia sentence, and usually only contain a single entity, making them more aligned with the challenges listed in Table 1.

We randomly sampled 1.4 million sentences, where entities were tagged using the taxonomy described in Section 3. This forms the complete Lowner dataset. We created the training, development, and test sets by having the training and dev sets match the CoNLL03 data in size.6 The remaining items were used to form a very large test set that contains millions of entities not present in the training set.

Footnote 6: The split ratio between train and dev is about \(4:1\).

Msq-Ner:This dataset aims to reflect NER in the QA domain, and is based on the MS-MARCO QnA dataset (v2.1) Bajaj et al. (2016) which contains over a million questions.

We first templatize the questions by applying an existing NER system (e.g., spaCy) to identify entities in the questions. We then use our gazetteer to map the entities to their NER types to create slotted templates, e.g., "when did[{ipnone}] come out" becomes "when did<PROD> come out". The templates are then aggregated by frequency. This process results in \(3,445\) unique question templates.

While the NER system cannot correctly identify many entities, the most frequent templates are reliable. Examples are listed in Table 11.

Finally, we generate Msq-Ner by slotting the templates that have a frequency of \(>=5\) with random entities from the Wikipedia KB with the same class. Each template is slotted with the same number of times it appeared in MS-MARCO in order to maintain the same relative distribution as the original data. This results in \(17,868\) questions, e.g., "when did[{xbox 360}] come out", which we use as a test set.

Orcas-Ner:To represent the search query domain, we utilize \(10\) million Bing user queries from the ORCAS dataset Craswell et al. (2020) and apply the same templatization procedure described above for Msq-Ner. This yields search templates e.g., "<PROD> price" and "<CORP> phone number", which are used to create annotated queries, e.g., "[{airpods pro}] reviews". This process creates \(97,324\) unique query templates. We slot these templates according to their frequency, yielding a final dataset of \(471,746\) queries. This is our largest, and most challenging, test set. Examples of our templates are listed in Table 12.



During training we set the gradient norm to be \(5.0\) to ensure smooth training. We also apply early stopping, halting the training process when we cannot improve performance on the development set during the last \(15\) epochs.

## Appendix D CGR Embedding Visualization

As mentioned in SS4, given a sentence, we use a gazetteer matcher to extract its representation M. M is passed to the CGR encoder (i.e., the green paprt in Figure 1) to generate the gazetteer representation, i.e., \({\bf h}_{CGR}\). We give all the sentences in Msq-Nerv as inputs to the CGR encoder and obtain the averaged embedding vectors of all the gazetteer tags, e.g., B-PER and B-CW. To visualize these gazetteer tags, we apply t-SNE Maaten and Hinton (2008) and generate their 2D visualization shown in Figure 3. It is clear that the tags in the same type, e.g., B-PROD and I-PROD, are close to each other. This indicates that our CGR encode can identify the semantic meaning of these tags and provide effective gazetteer representation.

## Appendix E Additional Results

Some additional detailed results are included in this section.

Table 16 shows detailed precision, recall and F1 scores for each entity class, comparing the no gazetteer baseline model and the best model for each dataset. We note that the worst performance is on products and creative works, as we hypothesized since the entities are much more linguistically complex. These classes achieve the largest increases with our models, which demonstrates that our methods successfully make up the models' weakness in the complex entities challenge.

Figure 3: 2D visualization of the CGR (gazetteer tag) representations.

\begin{table}
\begin{tabular}{l|c l} \hline Entity Type (Tag) & \#Entires & Examples \\ \hline \hline PERSON (PER) & 799,072 & Frank Gray, Steven Jobs \\ LOCATION (LOC) & 430,630 & Seattle, Beijing \\ CORPORATION(CORP) & 48,446 & Amazon, Sony \\ GROUPS(GRP) & 106,940 & Uni. of Cambridge \\ PRODUCT (PROD) & 31,139 & TV, Smartphone \\ CREATIVE-WORK(CW) & 256,912 & La La Land \\ \hline \end{tabular}
\end{table}
Table 13: The distribution of gazetteer entries mapped to each NER class.

\begin{table}
\begin{tabular}{l} \hline <CW> imdb \\ best hotels <LOC> \\ <PER> parents \\ <PROD> price \\ <GRP> website \\ <CORP> customer service \\ \hline \end{tabular}
\end{table}
Table 11: Sample questions from Msq-Nerv. Slots are in angle brackets.

\begin{table}
\begin{tabular}{l} \hline Activity Type (Tag) & \#Entires & Examples \\ \hline \hline PERSON (PER) & 799,072 & Frank Gray, Steven Jobs \\ LOCATION(LOC) & 430,630 & Seattle, Beijing \\ CORPORATION(CORP) & 48,446 & Amazon, Sony \\ GROUPS(GRP) & 106,940 & Uni. of Cambridge \\ PRODUCT(PROD) & 31,139 & TV, Smartphone \\ CREATIVE-WORK(CW) & 256,912 & La La Land \\ \hline \end{tabular}
\end{table}
Table 10: Sample sentences from Löwner. Gold entities are in brackets.

 



\begin{table}
\begin{tabular}{c c|c c|c c|c c|c c|c c|c c|c c|c c|c c} \hline \multirow{2}{*}{Dataset} & \multicolumn{3}{c|}{PER} & \multicolumn{3}{c|}{GRP} & \multicolumn{3}{c|}{LOC} & \multicolumn{3}{c|}{CORP} & \multicolumn{3}{c|}{CW} & \multicolumn{3}{c}{PROD} \\  & \multicolumn{1}{r}{p} & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\ \hline Lowner & Baseline & 93.2 & 96.3 & 94.8 & 80.3 & 83.1 & 81.6 & 91.4 & 90.4 & 90.9 & 82.0 & 67.7 & 74.2 & 74.1 & 70.9 & 72.5 & 54.0 & 49.7 & 51.8 \\  & Ours & 96.8 & 96.6 & 96.7 & 91.3 & 88.9 & 90.1 & 94.0 & 92.2 & 93.1 & 85.9 & 87.9 & 86.9 & 85.3 & 80.3 & 82.7 & 65.0 & 60.3 & 62.5 \\ \hline Msq-Ner & Baseline & 68.1 & 85.0 & 75.6 & 28.7 & 23.2 & 25.7 & 61.0 & 39.6 & 48.0 & 50.0 & 18.3 & 26.8 & 23.6 & 15.1 & 18.4 & 46.2 & 8.9 & 15.0 \\  & Ours & 97.6 & 97.3 & 97.4 & 82.8 & 83.2 & 83.0 & 92.5 & 79.5 & 85.5 & 85.5 & 83.6 & 84.5 & 87.0 & 69.3 & 77.2 & 92.1 & 69.4 & 79.2 \\ \cline{2-19}  & Baseline & 33.9 & 64.4 & 44.4 & 20.9 & 13.6 & 16.5 & 39.5 & 23.2 & 29.2 & 21.3 & 12.2 & 15.5 & 13.9 & 10.9 & 12.2 & 54.0 & 8.3 & 14.4 \\ orcas-Ner & Ours & 78.2 & 91.9 & 84.5 & 69.7 & 78.2 & 73.7 & 81.6 & 70.6 & 75.7 & 66.7 & 78.2 & 72.0 & 72.8 & 74.3 & 73.6 & 91.3 & 65.7 & 76.4 \\ \hline \end{tabular}
\end{table}
Table 16: Per-class performance across entity types. We show the optimal model (Ours) for each dataset, which is BERT+MoE+Two-stage for Löwner, and BiLSTM+MoE for Msq-Ner and Orcas-Ner. The baselines are BERT, BiLSTM and BiLSTM without gazetteer, respectively.

\begin{table}
\begin{tabular}{c|c|c} Parameter & Search Range & Lowner optimal \\ \hline \hline Feedforward #layers & [1,3] & 1 & 1 \\ Feedforward dimensions & - & 768 & 768 \\ Feedforward activation & \{linear,tanh,relu\} & linear \\ Matcher dimension & [20,100] & 50 & 50 \\ Matcher biLSTM hidden & - & 384 & 384 \\ Matcher biLSTM dropout & [0,0.5] & 0.1 \\ Match size & [16, 32] & \(32\) \\ \hline \end{tabular}
\end{table}
Table 15: Optimal hyperparameters for BERT-MoE models