SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 4: Mention detection (MD) results for a BERT model trained on CoNLL03, tested on our data.
Results: Compared to the CoNLL03 results, the LOWER performance is worse. Although the evaluation on LOWER is a transfer setting, the large gap shows the existing model cannot generalize
3Outputs sizes must be equal, e.g., CGR must match BERT.
well to our datasets due to the hard entities. Results for MSQ-NER and ORCAS-NER, which are short texts, are even lower. Overall, we note the difficulty of our datasets due to low context and hard entities.
5.2 NER Ablation Experiments
We explore all model architectures by training on LOWER (set 1 in Table 2) and evaluating MD and NER performance on all datasets (sets 3-5 in Table 2). See Appendix C for training details.
Models: The GEMNET model is jointly trained and fused with BERT and BiLSTM word encoders, with and without two-stage training. To assess the impact of the MoE component, we also concatenate the CGR and CWR vectors, without MoE.
Baselines: We compare against three baselines: (1) no gazetteer baselines; (2) binary concatenation: we simply concatenate the binary match features (M) to the word representations, as is common in the literature; (3) the subTAGger model of Liu et al. (2019). They are shown as "baselines" in table 5.
Results: MD and NER performance for all models is shown in Table 5. Overall we note the high effectiveness of the GEMNET model. In particular, our BiLSTM-based GEMNET approach improves F1 by up to  $49\%$  over the no gazetteer BiLSTM baseline in ORCAS-NER. Different aspects of the results are discussed below.
Word Encoder Performance: For LOWER, we note that BERT achieves the best results, which is to be expected since the data consists of full sentences. MD is easier than NER, and represents the upper bound for NER. Performance in all cases decreases with low context, with search queries (ORCAS-NER) being the hardest. BiLSTMs perform better on shorter inputs, e.g., ORCAS-NER.
Impact of Gazetteers: Results improve in all cases with external knowledge. While the subtagger and the binary concatenation baselines yield gains compared to the no gazetteer baselines, our CGR-based approach outperforms all of them in all NER tests. This indicates the high effectiveness of our CGR. For LOWER, using CGR+MoE, MD performance improves by  $2.4\%$ , while NER increases  $4.7\%$  over the no gazetteer BERT baseline. Low-context data, MSQ-NER and ORCAS-NER, have much lower baseline performance, and benefit greatly from external knowledge. The best MSQ-NER NER model improves  $36\%$  over the no
gazetteer BiLSTM baseline, while ORCAS-NER increases by  $49\%$ . This clearly demonstrates the impact of gazetteer integration.
Effect of Integration Method: CGR outperforms baselines in all NER experiments, showing the effectiveness of a span-aware, contextual representation that is jointly trained with the word-level model. The MoE integration is superior to concatenation in all cases. This is more salient in low context settings, demonstrating that the MoE model can rely on the CGR feature when the syntactic context (CWR) is not discriminative. In some cases baselines actually degrade performance as the model can not effectively balance the experts.
Effect of Two-stage Training: We observe that two-stage training is crucial for BERT, including concatenation models and MoE models, but not for the BiLSTM model. This confirms our hypothesis that the CGR cannot be jointly trained with a large pretrained model. Freezing BERT and then jointly fine-tuning them provides great improvements.
Results on Benchmarks: We applied GEMNET, i.e., BERT using CGR+MoE with two stage training, to the standard benchmarks. We experiment in an uncased setting, and and compare with the reported uncased SOTA (Mayhew et al., 2019). The SOTA uses BERT-CRF, which are the same as our baseline architecture. For comparison, we also reproduce the BERT baseline using our implementation. Results are shown in Table 6. Our models achieve SOTA results in all uncased settings, demonstrating generalization across domains; we improve by  $3.9\%$  on WNUT17.
5.3 Per-Class Performance & Error Analysis
We also look at performance across different entity classes to understand the source of our improvements. Table 7 shows relative gains per class, comparing the no gazetteer baseline performance against the best model. Detailed precision/recall values are in Appendix E (Table 16).
The smallest gains are on PER and LOC types, and the largest gains are on products and creative works (CW). This agrees with our hypothesis that these complex entities are the hardest to recognize.
Comparing datasets, increases are much larger on MSQ-NER and ORCAS-NER, confirming the challenges of short low-context inputs, and our models effectiveness in such cases.
Word Encoder	Gazetter Model	2-stage	LOWER	MSQ-NER	ORCAS-NER
MD	NER	MD	NER	MD	NER
Baseline	BiLSTM	No gazetteer	No	86.5	81.7	62.9	51.4	38.3	27.3
Subtagger (Liu et al., 2019)	No	91.0	86.1	71.1	62.7	56.7	43.8
Binary Concatenation	No	90.6	87.3	55.7	51.1	41.2	33.3
Ours	BiLSTM	CGR + Concatenation	No	90.8	89.1	84.8	83.2	75.5	73.6
CGR + Concatenation	Yes	90.7	88.9	85.6	84.1	76.6	74.9
GEMNET (CGR + MoE)	No	90.6	89.0	88.7	87.3	78.1	76.3
GEMNET (CGR + MoE)	Yes	90.9	89.3	86.7	85.6	76.4	75.0
Baseline	BERT	No gazetteer	No	90.5	87.0	65.4	57.3	50.0	37.2
Subtagger (Liu et al., 2019)	No	90.2	86.3	60.8	53.7	44.8	32.5
Binary Concatenation	No	87.7	84.2	70.7	60.2	48.7	38.8
Ours	BERT	CGR + Concatenation	No	90.8	87.4	59.8	52.7	46.3	35.7
CGR + Concatenation	Yes	92.9	91.4	78.4	76.1	63.7	59.0
GEMNET (CGR + MoE)	No	90.1	86.6	62.7	56.1	47.9	37.2
GEMNET (CGR + MoE)	Yes	92.9	91.7	83.2	81.9	72.2	70.2
Table 5: MD and NER results (F1 score) on all test sets for models trained on LOWER.
Method	CoNLL03	WNUT17	OntoNotes
Uncased SOTA	91.0	46.1	88.1
BERT Baseline	89.6	46.9	86.9
GEMNET (BERT)	91.3	50.2	88.0
Table 6: Uncased NER Results (F1 score) on CoNLL03, WNUT17 and OntoNotes v5.0.
Class	LOWER	MSQ-NER	ORCAS-NER
PER	+1.9	+21.8	+40.1
LOC	+2.2	+37.5	+46.5
GRP	+8.5	+57.3	+57.2
CORP	+12.7	+57.7	+56.5
CW	+10.2	+58.8	+61.4
PROD	+10.7	+64.2	+62.0
Table 7: Relative gains over no gazetteer baseline for each entity class (F1 score) for each dataset.
We also conduct a qualitative error analysis to identify instances where the best non-gazetteer baseline fails, but our model provides correct output. Some examples are shown in Table 8. The baseline often lacks knowledge about complex and long-tail entities, either missing them (#1,6,8 show full or partial MD failure) or misclassifying them (#3-5 show NER errors). Another common trend we observe is baselines incorrectly predicting nested entities within complex entities (#2,10).
5.4 Effect of Gazetteer Coverage
We consider the impact of gazetteer coverage<sup>4</sup> on performance. We hypothesize that training coverage impacts how much the model learns to rely on the gazetteer. To verify this we examine two
Figure 2: Coverage analysis. X-axis is the testing coverage and Y-axis is the training coverage. (a) shows results for the LOWER test set, and (b) shows the results for MSQ-NER.
scenarios: (1) the gazetteer coverage for train and test match (i.e., both high or low); and (2) there is a coverage gap between train and test, e.g., train coverage is  $90\%$  but is  $25\%$  for test, or vice versa.
Model and Data: For each train/test set we create gazetteers that have  $p\%$  coverage of the set's gold entities, with  $p \in \{5, 10, 20, 30, 50, 75, 90, 95\}$ . This is achieved by randomly dropping entities. We then train models using each  $p$  and evaluate on test sets, using all values of  $p$ . This experiment is done using LOWER and MSQ-NER.
4The proportion of entities that are present in the gazetteer
Dataset	Gold Sentence	Entities by Baseline	Entities by Best Model
LOWER	Example 1: he worked for |linear technology|CORP and |analog devices|CORP Example 2: his signature piece was |orange blossom special|CW Example 3: it is the last of the |heinlein juveniles|CW	- |orange blossom|PROD |heinlein juveniles|GRP	|linear technology|CORP |analog devices|CORP |orange blossom special|CW |heinlein juveniles|CW
MSQ-NER	Example 4: what is the zip code for |basarbovo|LOC Example 5: who is the director of |el reino|CW Example 6: when was the |nokia 2.2|PROD invented	|basarbovo|PER |el reino|GRP |nokia|CORP	|basarbovo|LOC |el reino|CW |nokia 2.2|PROD
ORCAS-NER	Example 7: |bee-line|CORP revenue Example 8: |lexus rc 350|PROD height Example 9: how old is |ingross|PER Example 10: cast of |dr. devil and mr. hare|CW	|bee-line revenue|CW |lexus rc|PROD |ingross|LOC |dr. devil|PER, |mr. hare|PER	|bee-line|CORP |lexus rc 350|PROD |ingross|PER |dr. devil and mr. hare|CW
Results: Results are plotted as heatmaps in Figure 2. Best results occur with high train and test coverage, while the worst results fall under high training coverage but low test coverage. When train coverage is low, test coverage has no impact as the model presumably ignores the gazetteer input. Across test coverage values, best results are generally around the diagonal, i.e., matching training coverage. These patterns are identical across datasets, indicating that a train/test coverage gap should be avoided. In practice, if test set coverage cannot be measured, or high coverage is not guaranteed, then using lower training coverage (e.g.,  $50\%$ ) prevents performance degradation in very low test coverage cases.
We also note that the gap between the best and worst result for LOWER is not huge, showing the impact of sentence context. This gap is much larger for ORCAS-NER, where the model cannot rely on the context. Finally, we note that an alternative dynamic dropout method achieved similar results.
5.5 Performance in a Low-Resource Setting
We also consider the impact of a low-resource setting (limited annotations) on performance, hypothesizing that gazetteers are more helpful in such settings. To verify this, we create random subsets of  $5 / 10 / 20\%$  of the training data and compare the NER performance of a baseline (BERT-base) vs our best model(BERT+CGR+MoE+2stage) when trained on this data. Results are shown in Table 9.
The results show that gazetteers are always more effective than baseline in low-resource scenarios. Specifically, they improve much faster with less data, achieving close to maximum performance with only  $20\%$  of the data.
Table 8: Error analysis examples where baselines fail, but our models provide the correct recognition.
Size	LOWERNER	MSQ-NER	ORCAS-NER
Baseline	Ours	Baseline	Ours	Baseline	Ours
5%	74.2	81.2	52.2	60.4	33.1	44.2
10%	78.1	86.7	55.3	74.4	33.7	49.1
20%	81.2	88.7	55.5	81.3	33.9	69.7
100%	87.0	91.7	57.3	81.9	37.2	70.2
Table 9: NER results on the full test set (F1) for comparing a baseline model (BERT, No gazetteer) and GEMNET (BERT + CGR + MoE + 2stage) in low-resource settings using small subsets of the training data.
We focused on integrating gazetteers into NER models. We proposed GEMNET, a flexible architecture that includes a Contextual Gazetteer Representation encoder, combined with a novel Mixture-of-Expert gating network to conditionally utilize this information alongside any word-level model. GEMNET supports external gazetteers, allowing the model's knowledge to be updated without retraining.
We also developed new datasets to represent the current challenges in NER. Experimental results demonstrated that our method can alleviate the feature weight under-training issue, achieving significant improvements on our data and a standard benchmark, WNUT17. The datasets we released can serve as benchmarks for evaluating the entity knowledge possessed by models in future work.
Future work involves investigating integration with different model architectures, partial gazetteer matching, and additional entity features.
We would like to extend our gratitude to Eugene Agichtein, Alexandre Salle, and Besnik Fetahu for their valuable inputs and discussion during this project. We also thank the anonymous reviewers for their constructive remarks.
Gazetteer matches are randomly dropped during training (i.e., random entity dropout).
Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López-Monroy, and Thamar Solorio. 2017. A multi-task approach for named entity recognition in social media data. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 148-153.
Estephe Arnaud, Arnaud Dapogny, and Kevin Bailly. 2020. Tree-gated deep mixture-of-experts for pose-robust face alignment. IEEE Trans. Biom. Behav. Identity Sci., 2(2):122-132.
Sandeep Ashwini and Jinho D. Choi. 2014. Targetable named entity recognition in social media. CoRR, abs/1408.0782.
Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017. Generalisation in named entity recognition: A quantitative analysis. Computer Speech & Language, 44:61-83.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
Oliver Bender, Franz Josef Och, and Hermann Ney. 2003. Maximum entropy models for named entity recognition. In CoNLL, pages 148-151. ACL.
Gabriel Bernier-Colborne and Philippe Langlais. 2020. Hardeval: Focusing on challenging tokens to assess robustness of NER. In LREC, pages 1704-1711. European Language Resources Association.
David Carmel, Ming-Wei Chang, Evgeniy Gabrilovich, Bo-June Paul Hsu, and Kuansan Wang. 2014. Erd'14: entity recognition and disambiguation challenge. In SIGIR, page 1292. ACM.
Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. 2020. Orcas: 18 million clicked query-document pairs for analyzing search. arXiv preprint arXiv:2006.05324.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1). Association for Computational Linguistics.
Jiafeng Guo, Gu Xu, Xueqi Cheng, and Hang Li. 2009. Named entity recognition in query. In SIGIR, pages 267-274. ACM.
Jiang Guo, Darsh J. Shah, and Regina Barzilay. 2018. Multi-source domain adaptation with mixture of experts. In EMNLP, pages 4694-4703. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Abhinav Jain, Vishwanath P. Singh, and Shakti P. Rath. 2019. A multi-accent acoustic model using mixture of experts for speech recognition. In INTER-SPEECH, pages 779-783. ISCA.
Pratik Jayarao, Chirag Jain, and Aman Srivastava. 2018. Exploring the importance of context and embeddings in neural NER models for task-oriented dialogue systems. CoRR, abs/1812.02370.
Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev-Arie Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhili Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan, Qiang Ning, Shaoshi Ling, and Dan Roth. 2018. Cogcompnlp: Your swiss army knife for NLP. In LREC. European Language Resources Association (ELRA).
Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019. Towards improving neural named entity recognition with gazetteers. In ACL (1), pages 5301-5307. Association for Computational Linguistics.
Zihan Liu, Genta Indra Winata, and Pascale Fung. 2020. Zero-resource cross-domain named entity recognition. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 1-6, Online. Association for Computational Linguistics.
Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In ACL (1). The Association for Computer Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Shervin Malmasi and Mark Dras. 2015. Location mention detection in tweets and microblogs. In Conference of the Pacific Association for Computational Linguistics, pages 123-134. Springer.
Stephen Mayhew, Tatiana Tsygankova, and Dan Roth. 2019. ner and pos when nothing is capitalized. In EMNLP/IJCNLP (1), pages 6255-6260. Association for Computational Linguistics.
Svetlana Pavlitskaya, Christian Hubschneider, Michael Weber, Ruby Moritz, Fabian Hüger, Peter Schlicht, and J. Marius Zöllner. 2020. Using mixture of expert models to gain insights into semantic segmentation. In CVPR Workshops, pages 1399-1406. IEEE.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543. ACL.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In *NAACL-HLT*, pages 2227-2237. Association for Computational Linguistics.
Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54.
Shruti Rijhwani, Shuyan Zhou, Graham Neubig, and Jaime G. Carbonell. 2020. Soft gazetteers for low-resource named entity recognition. In ACL, pages 8118-8123. Association for Computational Linguistics.
Chan Hee Song, Dawn Lawrie, Tim Finin, and James Mayfield. 2020. Improving neural named entity recognition with gazetteers. arXiv preprint arXiv:2003.03072.
Tongshuang Wu, Kanit Wongsuphasawat, Donghao Ren, Kayur Patel, and Chris DuBois. 2020. Tempura: Query analysis with structural templates. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-12.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442-6454.
Eun-Suk Yang, Young-Bum Kim, Ruhi Sarikaya, and Yu-Seop Kim. 2016. Drop-out conditional random fields for twitter with huge mined gazetteer. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 282-288.
Ziwei Zhu, Shahin Sefati, Parsa Saadatpanah, and James Caverlee. 2020. Recommendation for new users and new items via randomized training and mixture-of-experts transformation. In SIGIR, pages 1121-1130. ACM.
LOWERN: This dataset is based on Wikipedia, and uses the links as span annotations.
The complete English Wikipedia dump from July 2020 was downloaded. We extracted the articles, which were then parsed to remove markup and extract sentences with their interlinks (links to other articles). This resulted in the extraction of approx. 180 million sentences. We then mapped the interlinks in each sentence to the Wikidata KB then resolved them to our NER taxonomy (in same manner as Appendix B).
Next, we filtered sentences using two strategies. Taking advantage of Wikipedia's well-formed text, we applied a Regex-based NER method to identify sentences containing named entities that were not linked, and removed them. This removes long and high-context sentences that contain references to many entities. Additionally we also removed any sentence where the links could not be resolved to Wikidata entities. This process discards over  $90\%$  of the sentences, resulting in approx. 14 million candidate sentences.
This process is very effective at yielding short, low-context sentences. Example sentences are shown in Table 10. The sentences contain some context, but they are much shorter than the average Wikipedia sentence, and usually only contain a single entity, making them more aligned with the challenges listed in Table 1.
We randomly sampled 1.4 million sentences, where entities were tagged using the taxonomy described in Section 3. This forms the complete LOWER dataset. We created the training, development, and test sets by having the training and dev sets match the CoNLL03 data in size. The remaining items were used to form a very large test set that contains millions of entities not present in the training set.
MSQ-NER: This dataset aims to reflect NER in the QA domain, and is based on the MS-MARCO QnA dataset (v2.1) (Bajaj et al., 2016) which contains over a million questions.
We first template the questions by applying an existing NER system (e.g., spaCy) to identify entities in the questions. We then use our gazetteer to map the entities to their NER
types to create slotted templates, e.g., "when did [[iphone]] come out" becomes "when did <PROD> come out". The templates are then aggregated by frequency. This process results in 3,445 unique question templates.
While the NER system cannot correctly identify many entities, the most frequent templates are reliable. Examples are listed in Table 11.
Finally, we generate MSQ-NER by slotting the templates that have a frequency of  $> = 5$  with random entities from the Wikipedia KB with the same class. Each template is slotted with the same number of times it appeared in MS-MARCO in order to maintain the same relative distribution as the original data. This results in 17,868 questions, e.g., "when did [[xbox 360]] come out", which we use as a test set.
ORCAS-NER: To represent the search query domain, we utilize 10 million Bing user queries from the ORCAS dataset (Craswell et al., 2020) and apply the same templatization procedure described above for MSQ-NER. This yields search templates e.g., “<PROD> price” and “<CORP> phone number”, which are used to create annotated queries, e.g., “[[airpods pro]] reviews”. This process creates 97,324 unique query templates. We slot these templates according to their frequency, yielding a final dataset of 471,746 queries. This is our largest, and most challenging, test set. Examples of our templates are listed in Table 12.
B Gazetteer Details and Statistics
We parsed a Wikidata dump from July 2020 and mapped entities to our NER taxonomy (§3). This was done by traversing Wikidata's class and instance relations, and mapping them to our NER classes, e.g., Wikidata's human class maps to PER in our taxonomy, song to CW, and so on.
We extracted 1.67 million entities that were mapped to our classes. The distribution of these entities is shown in Table 13.
C Training Details & Hyperparameters
The hyperparameters searching range and the optimal ones we use in Section 5.2, including the results on our created datasets in Table 5 and benchmark results in Table 6, are shown in Table 14 (BiLSTM model) and Table 15 (BERT model). The parameter tuning is performed on the development sets of the respective datasets.
6The split ratio between train and dev is about  $4:1$
During training we set the gradient norm to be 5.0 to ensure smooth training. We also apply early stopping, halting the training process when we cannot improve performance on the development set during the last 15 epochs.
D CGR Embedding Visualization
As mentioned in §4, given a sentence, we use a gazetteer matcher to extract its representation M. M is passed to the CGR encoder (i.e., the green paprt in Figure 1) to generate the gazetteer representation, i.e.,  $\mathbf{h}_{CGR}$ . We give all the sentences
Entity Type (Tag)	#Entries	Examples
PERSON (PER)	799,072	Frank Gray, Steven Jobs
LOCATION (LOC)	430,630	Seattle, Beijing
CORPORATION (CORP)	48,446	Amazon, Sony
GROUPS (GRP)	106,940	Uni. of Cambridge
PRODUCT (PROD)	31,139	TV, Smartphone
CREATIVE-WORK (CW)	256,912	La La Land
Table 13: The distribution of gazetteer entries mapped to each NER class.
Figure 3: 2D visualization of the CGR (gazetteer tag) representations.