SlowGuess/ACL_2020_OCR huggingface dataset:
OntoNotes 4.0 (Pradhan et al., 2011) is a Chinese dataset and consists of text from news domain. OntoNotes 4.0 annotates 18 named entity types. In this paper, we take the same data split as Wu et al. (2019).
Table 2: Results for nested NER tasks.
English CoNLL 2003
Model	Precision	Recall	F1
BiLSTM-CRF (Ma and Hovy, 2016)	-	-	91.03
ELMo (Peters et al., 2018b)	-	-	92.22
CVT (Clark et al., 2018)	-	-	92.6
BERT-Tagger (Devlin et al., 2018)	-	-	92.8
BERT-MRC	92.33	94.61	93.04 (+0.24)
English OntoNotes 5.0
Model	Precision	Recall	F1
BiLSTM-CRF (Ma and Hovy, 2016)	86.04	86.53	86.28
Strubell et al. (2017)	-	-	86.84
CVT (Clark et al., 2018)	-	-	88.8
BERT-Tagger (Devlin et al., 2018)	90.01	88.35	89.16
BERT-MRC	92.98	89.95	91.11 (+1.95)
Chinese MSRA
Model	Precision	Recall	F1
Lattice-LSTM (Zhang and Yang, 2018)	93.57	92.79	93.18
BERT-Tagger (Devlin et al., 2018)	94.97	94.62	94.80
Glyce-BERT (Wu et al., 2019)	95.57	95.51	95.54
BERT-MRC	96.18	95.12	95.75 (+0.21)
Chinese OntoNotes 4.0
Model	Precision	Recall	F1
Lattice-LSTM (Zhang and Yang, 2018)	76.35	71.56	73.88
BERT-Tagger (Devlin et al., 2018)	78.01	80.35	79.16
Glyce-BERT (Wu et al., 2019)	81.87	81.40	81.63
BERT-MRC	82.98	81.25	82.11 (+0.48)
Table 3: Results for flat NER tasks.
4.2.2 Baselines
For English datasets, we use the following models as baselines.
BiLSTM-CRF from Ma and Hovy (2016).
- ELMo tagging model from Peters et al. (2018b).
- CVT from Clark et al. (2018), which uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder.
- Bert-Tagger from Devlin et al. (2018), which treats NER as a tagging task.
For Chinese datasets, we use the following models as baselines:
- Lattice-LSTM: Zhang and Yang (2018) constructs a word-character lattice.
- Bert-Tagger: Devlin et al. (2018) treats NER as a tagging task.
- Glyce-BERT: The current SOTA model in Chinese NER developed by Wu et al. (2019), which combines glyph information with BERT pretraining.
4.2.3 Results and Discussions
Table 3 presents comparisons between the proposed model and baseline models. For English CoNLL 2003, our model outperforms the finetuned BERT tagging model by  $+0.24\%$  in terms of F1, while for English OntoNotes 5.0, the pro
English OntoNotes 5.0
Model	F1
LSTM tagger (Strubell et al., 2017)	86.84
BiDAF (Seo et al., 2017)	87.39 (+0.55)
QAnet (Yu et al., 2018)	87.98 (+1.14)
BERT-Tagger	89.16
BERT-MRC	91.11 (+1.95)
posed model achieves a huge gain of  $+1.95\%$  improvement. The reason why greater performance boost is observed for OntoNotes is that OntoNotes contains more types of entities than CoNLL03 (18 vs 4), and some entity categories face the severe data sparsity problem. Since the query encodes significant prior knowledge for the entity type to extract, the MRC formulation is more immune to the tag sparsity issue, leading to more improvements on OntoNotes. The proposed method also achieves new state-of-the-art results on Chinese datasets. For Chinese MSRA, the proposed method outperforms the fine-tuned BERT tagging model by  $+0.95\%$  in terms of F1. We also improve the F1 from  $79.16\%$  to  $82.11\%$  on Chinese OntoNotes4.0.
5.1 Improvement from MRC or from BERT
For flat NER, it is not immediately clear which proportion is responsible for the improvement, the MRC formulation or BERT (Devlin et al., 2018). On one hand, the MRC formulation facilitates the entity extraction process by encoding prior knowledge in the query; on the other hand, the good performance might also come from the large-scale pre-training in BERT.
To separate the influence from large-scale BERT pretraining, we compare the LSTM-CRF tagging model (Strubell et al., 2017) with other MRC based models such as QAnet (Yu et al., 2018) and BiDAF (Seo et al., 2017), which do not rely on large-scale pretraining. Results on English Ontonotes are shown in Table 5. As can be seen, though underperforming BERT-Tagger, the MRC based approaches QAnet and BiDAF still significantly outperform tagging models based on LSTM+CRF. This validates the importance of MRC formulation. The MRC formulation's benefits are also verified when comparing BERT-tagger
Table 4: Results of different MRC models on English OntoNotes5.0.
English OntoNotes 5.0
Model	F1
BERT-Tagger	89.16
Position index of labels	88.29 (-0.87)
Keywords	89.74 (+0.58)
Wikipedia	89.66 (+0.59)
Rule-based template filling	89.30 (+0.14)
Synonyms	89.92 (+0.76)
Keywords+Synonyms	90.23 (+1.07)
Annotation guideline notes	91.11 (+1.95)
Table 5: Results of different types of queries.
with BERT-MRC: the latter outperforms the former by  $+1.95\%$ .
We plot the attention matrices output from the BiDAF model between the query and the context sentence in Figure 2. As can be seen, the semantic similarity between tagging classes and the contexts are able to be captured in the attention matrix. In the examples, Flevland matches geographical, cities and state.
5.2 How to Construct Queries
How to construct query has a significant influence on the final results. In this subsection, we explore different ways to construct queries and their influence, including:
- Position index of labels: a query is constructed using the index of a tag to, i.e., "one", "two", "three".
- Keyword: a query is the keyword describing the tag, e.g., the question query for tag ORG is "organization".
- Rule-based template filling: generates questions using templates. The query for tag ORG is "which organization is mentioned in the text".
- Wikipedia: a query is constructed using its wikipedia definition. The query for tag ORG is "an organization is an entity comprising multiple people, such as an institution or an association."
- Synonyms: are words or phrases that mean exactly or nearly the same as the original keyword extracted using the Oxford Dictionary. The query for tag ORG is "association".
- Keyword+Synonyms: the concatenation of a keyword and its synonym.
- Annotation guideline notes: is the method we use in this paper. The query for tag ORG is "find organizations including companies, agencies and institutions".
Table 5 shows the experimental results on En
5855
Figure 2: An example of attention matrices between the query and the input sentence.
Models	Train	Test	F1
BERT-tags	OntoNotes5.0	OntoNotes5.0	89.16
BERT-MRC	OntoNotes5.0	OntoNotes5.0	91.11
BERT-tags	CoNLL03	OntoNotes5.0	31.87
BERT-MRC	CoNLL03	OntoNotes5.0	72.34
Table 6: Zero-shot evaluation on OntoNotes5.0. BERT-MRC can achieve better zero-shot performances.
glish OntoNotes 5.0. The BERT-MRC outperforms BERT-Tagger in all settings except Position Index of Labels. The model trained with the Annotation Guideline Notes achieves the highest F1 score. Explanations are as follows: for Position Index Dataset, queries are constructed using tag indexes and thus do not contain any meaningful information, leading to inferior performances; Wikipedia underperforms Annotation Guideline Notes because definitions from Wikipedia are relatively general and may not precisely describe the categories in a way tailored to data annotations.
5.3 Zero-shot Evaluation on Unseen Labels
It would be interesting to test how well a model trained on one dataset is transferable to another, which is referred to as the zero-shot learning ability. We trained models on CoNLL 2003 and test them on OntoNotes5.0. OntoNotes5.0 contains 18 entity types, 3 shared with CoNLL03, and 15 unseen in CoNLL03. Table 6 presents the results. As can been seen, BERT-tags does not have zero-shot learning ability, only obtaining an accuracy of  $31.87\%$ . This is in line with our expectation since it cannot predict labels unseen from the training set. The question-answering formal
Figure 3: Effect of varying percentage of training samples on Chinese OntoNotes 4.0. BERT-MRC can achieve the same F1-score comparing to BERT-Tagger with fewer training samples.
ization in MRC framework, which predicts the answer to the given query, comes with more generalization capability and achieves acceptable results.
5.4 Size of Training Data
Since the natural language query encodes significant prior knowledge, we expect that the proposed framework works better with less training data. Figure 3 verifies this point: on the Chinese OntoNotes 4.0 training set, the query-based BERT-MRC approach achieves comparable performance to BERT-tags even with half amount of training data.
In this paper, we reformulate the NER task as a MRC question answering task. This formalization comes with two key advantages: (1) being capa
5856
ble of addressing overlapping or nested entities; (2) the query encodes significant prior knowledge about the entity category to extract. The proposed method obtains SOTA results on both nested and flat NER datasets, which indicates its effectiveness. In the future, we would like to explore variants of the model architecture.
We thank all anonymous reviewers, as well as Jiawei Wu and Wei Wu for their comments and suggestions. The work is supported by the National Natural Science Foundation of China (NSFC No. 61625107 and 61751209).
Beatrice Alex, Barry Haddow, and Claire Grover. 2007. Recognising nested named entities in biomedical text. In Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing, pages 65-72. Association for Computational Linguistics.
Kate Byrne. 2007. Nested named entity recognition in historical archive text. In International Conference on Semantic Computing (ICSC 2007), pages 589-596. IEEE.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051.
Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional LSTM-cnns. arXiv preprint arXiv:1511.08308.
Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357-370.
Julie Medero Christopher Walker, Stephanie Strassel and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia 57.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1914-1925.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493-2537.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
George R Doddington, Alexis Mitchell, Mark A Przybocki, Stephanie M Strassel Lance A Ramshaw, and Ralph M Weischedel. 2005. The automatic content extraction (ace) program-tasks, data, and evaluation. In LREC, 2:1.
Jenny Rose Finkel and Christopher D Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 141-150. Association for Computational Linguistics.
Joseph Fisher and Andreas Vlachos. 2019. Merge and label: A novel neural network architecture for nested NER. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5840-5850.
James Hammerton. 2003. Named entity recognition with long short-term memory. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 172-175. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Heng Ji, Xiaoman Pan, Boliang Zhang, Joel Nothman, James Mayfield, Paul McNamee, and Cash Costello. 2017. Overview of TAC-KBP2017 13 languages entity discovery and linking. In Proceedings of the 2017 Text Analysis Conference, TAC 2017, Gaithersburg, Maryland, USA, November 13-14, 2017.
Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1446-1459, New Orleans, Louisiana. Association for Computational Linguistics.
Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 861-871.
J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun'ichi Tsujii. 2003. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl_1):i180-i182.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
5857
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108-117, Sydney, Australia. Association for Computational Linguistics.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1340-1350.
Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2019a. Sequence-to-nuggets: Nested entity mention detection via anchor-region networks. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5182-5192.
Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2019b. Sequence-to-nuggets: Nested entity mention detection via anchor-region networks. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 5182-5192.
Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857-867.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 3036-3046.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-Crf. arXiv preprint arXiv:1603.01354.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between words: Recognizing overlapping mentions with mention separators. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2608-2618, Copenhagen, Denmark. Association for Computational Linguistics.
Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. 2002. The genia corpus: An annotated research abstract corpus in molecular biology domain. In Proceedings of the Second International Conference on Human Language Technology Research, HLT '02, pages 82-86, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018b. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Sameer Pradhan, Mitchell P. Marcus, Martha Palmer, Lance A. Ramshaw, Ralph M. Weischedel, and Nianwen Xue, editors. 2011. Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, CoNLL 2011, Portland, Oregon, USA, June 23-24, 2011. ACL.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 142-147.
5858
Cicero Nogueira dos Santos and Victor Guimaraes. 2015. Boosting named entity recognition with neural character embeddings. arXiv preprint arXiv:1505.05008.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1047-1055. ACM.
Takashi Shibuya and Eduard H. Hovy. 2019. Nested named entity recognition via second-best sequence learning and decoding. CoRR, abs/1909.02250.
Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2843-2849, Brussels, Belgium. Association for Computational Linguistics.
Jana Straková, Milan Straka, and Jan Hajic. 2019. Neural architectures for nested NER through linearization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5326-5331, Florence, Italy. Association for Computational Linguistics.
Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. arXiv preprint arXiv:1702.02098.
Charles Sutton, Andrew McCallum, and Khashayar Rohanimanesh. 2007. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. Journal of Machine Learning Research, 8(Mar):693-723.
Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. arXiv preprint arXiv:1810.01817.
Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905.
Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211.
Wei Wu, Yuxian Meng, Qinghong Han, Muyu Li, Xiaoya Li, Jie Mei, Ping Nie, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. arXiv preprint arXiv:1901.10125.
Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604.
Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dcn+: Mixed objective and deep residual coattention for question answering. arXiv preprint arXiv:1711.00106.
Mingbin Xu, Hui Jiang, and Seditawut Watcharawittayakul. 2017. A local detection approach for named entity recognition and mention detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1237-1247.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. *Qanet: Combining local convolution with global self-attention* for reading comprehension. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Yue Zhang and Jie Yang. 2018. Chinese ner using lattice lstm. arXiv preprint arXiv:1805.02023.
Changmeng Zheng, Yi Cai, Jingyun Xu, Ho-fung Leung, and Guandong Xu. 2019. A boundary-aware neural model for nested named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 357-366, Hong Kong, China. Association for Computational Linguistics.