SlowGuess/ACL_2020_OCR huggingface dataset:
Model	ACE-2004	ACE-2005	GENIA	NNE
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Finkel and Manning 2009	-	-	-	-	-	-	75.4	65.9	70.3	-	-	-
Lu and Roth 2015	70.0	56.9	62.8	66.3	59.2	62.5	74.2	66.7	70.3	-	-	-
Muis and Lu 2017	72.7	58.0	64.5	69.1	58.1	63.1	75.4	66.8	70.8	-	-	-
Xu et al. 2017	68.2	54.3	60.5	67.4	55.1	60.6	-	-	-	-	-	-
Katiyar and Cardie 2018	73.6	71.8	72.7	70.6	70.4	70.5	79.8	68.2	73.6	-	-	-
Ju et al. 2018	-	-	-	74.2	70.3	72.2	78.5	71.3	74.7	-	-	-
Wang et al. 2018	74.9	71.8	73.3	74.5	71.5	73.0	78.0	70.2	73.9	77.4	70.1	73.6
Wang and Lu 2018	78.0	72.4	75.1	76.8	72.3	74.5	77.0	73.3	75.1	91.8	91.0	91.4
Sohrab and Miwa 2018	-	-	-	-	-	-	93.2	64.0	77.1	-	-	-
Fisher and Vlachos 2019	-	-	-	75.1	74.1	74.6	-	-	-	-	-	-
Lin et al. 2019	-	-	-	76.2	73.6	74.9	75.8	73.9	74.8	-	-	-
Straková et al. 2019	-	-	77.1	-	-	75.4	-	-	76.4	-	-	-
Pyramid-Basic	80.83	78.86	79.83	79.27	79.37	79.32	77.91	77.20	77.55	93.37	93.91	93.64
Pyramid-Full	81.14	79.42	80.27	80.01	78.85	79.42	78.60	77.02	77.78	93.44	93.95	93.70
LM-based
Xia et al. 2019 [ELMO]	81.7	77.4	79.5	79.0	77.3	78.2	-	-	-	-	-	-
Fisher and Vlachos 2019 [ELMO]	-	-	-	79.7	78.0	78.9	-	-	-	-	-	-
Fisher and Vlachos 2019 [BERT]	-	-	-	82.7	82.1	82.4	-	-	-	-	-	-
Shibuya and Hovy 2019 [BERT]	-	-	-	83.0	82.4	82.7	76.3	74.7	75.5	-	-	-
Luan et al. 2019 [ELMO]	-	-	84.7	-	-	82.9	-	-	76.2	-	-	-
Straková et al. 2019 [BERT]	-	-	84.3	-	-	83.4	-	-	78.2	-	-	-
Straková et al. 2019 [BERT+Flair]	-	-	84.4	-	-	84.3	-	-	78.3	-	-	-
Pyramid-Basic [BERT]	86.08	86.48	86.28	83.95	85.39	84.66	79.45	78.94	79.19	93.97	94.79	94.37
Pyramid-Basic [BERT+Flair]	87.01	86.55	86.78	84.90	86.08	85.49	79.98	78.51	79.24	93.97	94.98	94.47
Pyramid-Basic [ALBERT]	86.54	87.44	86.99	85.20	86.56	85.87	80.07	77.60	78.82	94.11	94.91	94.51
Pyramid-Basic [ALBERT+Flair]	86.63	87.15	86.89	85.10	87.22	86.15	78.48	79.39	78.93	94.18	94.79	94.48
Pyramid-Basic [ALBERT+BERT]	87.65	87.74	87.70	85.24	87.32	86.27	80.12	77.82	78.95	94.28	94.99	94.63
Pyramid-Full [BERT+Flair]	-	-	-	-	-	-	80.31	78.33	79.31	-	-	-
Pyramid-Full [ALBERT+BERT]	87.71	87.78	87.74	85.30	87.40	86.34	-	-	-	94.30	95.07	94.68
Table 3: Results of nested NER. Ju et al. 2018 used different dataset split. Straková et al. 2019 introduces two methods, here we report the better one. Bold and underline indicate the best and the second best F1 respectively.
even compatible with some LM-based baselines. A close one is from Straková et al. 2019, which employs many extra features including input forms, lemmas and POS, whereas our method does not. Additionally, our method brings much higher recall values than the other methods.
With pre-trained language model embeddings, specifically with ALBERT+BERT for ACE-2004, ACE-2005, NNE and with BERT+Flair for GENIA, our model achieves state-of-the-art F1 scores: 87.74, 86.34, 79.31, and 94.68 respectively.
4.4 Tuning Number of Layers
We evaluate our method with different  $L$  on all datasets. Due to space limit, we only present the results of ACE-2005 in Table 4. The findings on the other datasets are similar.
Results From All Layers We report in Table 4 the detailed results for all entity lengths while tuning  $L$  on ACE-2005. Obviously 1-word and 2-word entities account for the majority of entities (77%), where we achieve competitive results. Longer entities see reductions in performance. However, due to our remedy strategy, entities longer than  $L$  are still recognized with acceptable performance. Note
$\mathrm{R}(\mathrm{N})$  is the recall of nested entities, i.e. for layer  $l$ , entities nested with other entities shorter than  $l$  are also counted in.
Inference Speed Table 4 also shows the inference speed with different  $L$  for the basic and full models. Although the basic model does not perform as good as the full model, it is significantly faster. Since the time complexity of our method is  $O(TL)$  with  $T$  being the number of tokens and  $L$  the number of stacked layers, we can further speed up the inference by using smaller  $L$  value (e.g.  $L = 8$  or 4), while achieving F1 scores higher than most baselines.
We conduct ablation study to verify the effectiveness of components of Pyramid. Likewise, we only present the results on ACE-2005 here.
Character Embeddings: Using character is a standard technique for NER to dynamically capture orthographic and morphological features. It provides some improvements.
Layer Normalization: LayerNorm eliminates the bias and scale difference of the inputs of each
Pyramid-Basic	L=32	L=16	L=8	L=4
len(e)	# entities	F1	R(N)	F1	R(N)	F1	R(N)	F1	R(N)
all	-	79.3	73.6	79.3	74.4	78.8	73.9	77.6	69.5
1	1706 (56%)	84.0	82.3	84.3	82.5	84.0	83.0	83.4	81.4
2	635 (21%)	79.3	77.5	79.7	78.6	78.8	77.7	78.6	76.2
3	248 (8%)	74.9	75.5	75.3	76.8	75.6	77.5	72.9	73.7
4	140 (5%)	72.1	73.1	71.8	75.0	72.0	73.3	65.7	61.1
5	90 (3%)	73.6	77.5	72.3	78.9	69.3	75.5	63.6	60.3
6-8	106 (3%)	57.9	59.3	56.2	59.3	53.4	56.7	47.7	45.9
9-16	81 (3%)	42.0	36.4	43.1	39.9	42.3	39.5	40.0	36.8
17-	25 (1%)	33.8	26.1	23.0	18.8	27.2	21.7	23.6	18.8
Inference Speed (Basic/Full, words per second) on GTX 1080 Ti
batch size = 1	708 / 445	842 / 545	1116 / 781	1494 / 1153
batch size = 4	1526 / 955	2085 / 1361	2987 / 2151	4230 / 3280
batch size = 16	2949 / 2084	4372 / 3282	6660 / 5169	8999 / 7852
Table 4: Details of tuning  $L$  on ACE-2005. len(e) is the length of entities. R(N) is the recall of nested entities. Numbers below the horizontal lines indicate the results where the remedy solution starts working
Pyramid-Basic	P	R	F1
CharEmbs
with	79.27	79.37	79.32
without	79.54	77.67	78.59 (-0.73)
LayerNorm
with	79.27	79.37	79.32
without	79.17	78.01	78.59 (-0.73)
LSTMdec
shared weights	79.27	79.37	79.32
independent	78.19	78.75	78.47 (-0.85)
ReduceLength
Conv1d	79.27	79.37	79.32
MeanPooling	79.18	77.77	78.47 (-0.85)
MaxPooling	79.69	77.47	78.56 (-0.76)
decoding layer and improve the F1 score. It also substantially accelerates the converging speed.
Sharing LSTM $^{dec}$ : The jobs of decoding layers are similar: inheriting information from previous layers and recognizing entity representations of length one. Therefore, sharing weights maximizes the use of training data and prevents overfitting.
Method of Reducing Length: We use CNN to reduce the sequence length at each decoding layer. As shown in Table 5, compared with average pooling and maximum pooling, CNN can effectively retain the original semantic information and capture the boundary information.
Pyramid Layers: Apart from the results shown in Table 5, we emphasize that the performance gain of Pyramid owes a lot to the pyramid layers (both normal and inverse ones). As shown in Table 4, reducing  $L$  to 4 leads to a drop of F1 (-1.7). It is clear that when  $L = 1$ , our method degrades to a flat entity recognizer, which cannot handle nested mentions any more.
Table 5: Ablation study on ACE-2005
Dataset	Statistics	train	dev	test
Sentences	# total	1599	400	600
# nested	1594	400	600
# overlap	230	54	75
Entities	# total	16202	3978	5989
# nested	14506	3597	5390
# overlap	511	115	164
Pyramid-Basic	P	R	F1
overall	87.5	86.9	87.2
nested	-	87.4	-
overlap	-	70.1	-
Table 6: Results of Pyramid-Basic with nested and overlapping entities. The dataset is based on part of NNE, with additional program-generated labels.
4.6 Overlapping Entity Recognition
Overlapping mentions usually occur along with the attributive clause in natural language. For example, sentence "The burial site of Sheikh Abbad, who died 500 years ago, is located." contains two overlapping mentions "The burial site of Sheikh Abbad" and "Sheikh Abbad, who died 500 years ago".
Due to lack of datasets for overlapping NER, we create a small dataset. For all sentences in NNE, we find 2599 which contain “, which” or “, who”. We use the ELMo-based constituency parser<sup>9</sup> to find attributive clauses together with their modified noun phrases (“Sheikh Abbad, who ...”), and then see if a bigger noun phrase (“the burial site of Sheikh Abbad”) contains the noun phrase. Next, while keeping the original annotations, we add these two mentions to form a new dataset where around  $14\%$  sentences have overlapping but non-nested entity mentions. This dataset is split randomly into training, dev, and test sets containing 1599, 400, and 600 sentences respectively. Note the additional annotations are not verified by human, meaning they might contain some errors. However, it is still useful for testing the performance of our model for overlapping NER.
The statistics of the data and the experimental results are shown in Table 6. It can be seen that our method can effectively handle overlapping NER.
This paper presented Pyramid, a novel layered neural model for nested entity recognition. Our model relies on a layer-wise bidirectional decoding process (with both normal and inverse pyramids),
$^{9}$ Stern et al. 2017 with ELMo: https://allennlp.s3.amazon.com/models/elmo-constituency-parser-2018.03.14.tar.gz, implemented by Gardner et al. 2018.
5925
allowing each decoding layer to take into account the global information from lower and upper layers. Pyramid does not suffer from layer disorientation or error propagation, and is applicable for the more general overlapping NER. The proposed method obtained state-of-the-art results on four different nested NER datasets, confirming its effectiveness.
This work was supported by the Natural Science Foundation of China (No. 61672455), the Key Research and Development Program of Zhejiang Province of China (No. 2020C01024), the Natural Science Foundation of Zhejiang Province of China (No. LY18F020005), and the National Research Foundation, Prime Minister's Office, Singapore under its Strategic Capability Research Centres Funding Initiative.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. Flair: An easy-to-use framework for state-of-the-art nlp. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54-59.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649.
Beatrice Alex, Barry Haddow, and Claire Grover. 2007. Recognising nested named entities in biomedical text. In Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing, pages 65-72. Association for Computational Linguistics.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Kate Byrne. 2007. Nested named entity recognition in historical archive text. In International Conference on Semantic Computing (ICSC 2007), pages 589-596. IEEE.
Billy Chiu, Gamal Crichton, Anna Korhonen, and Sampo Pyysalo. 2016. How to train good word embeddings for biomedical nlp. In Proceedings of the 15th workshop on biomedical natural language processing, pages 166-174.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In Lrec, volume 2, page 1. Lisbon.
Jenny Rose Finkel and Christopher D Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 141-150.
Joseph Fisher and Andreas Vlachos. 2019. Merge and label: A novel neural network architecture for nested ner. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5840-5850.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1446-1459.
Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 861-871.
J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun'ichi Tsujii. 2003. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl_1):i180-i182.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.
5926
2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2019. Sequence-to-nuggets: Nested entity mention detection via anchor-region networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5182-5192.
Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857-867.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3036-3046.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074.
Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between words: Recognizing overlapping mentions with mention separators. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2608-2618.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155.
Nicky Ringland, Xiang Dai, Ben Hachey, Sarvnaz Karimi, Cecile Paris, and James R Curran. 2019. Nne: A dataset for nested named entity recognition in english newswire. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5176-5181.
Takashi Shibuya and Eduard Hovy. 2019. Nested named entity recognition via second-best sequence learning and decoding. arXiv preprint arXiv:1909.02250.
Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2843-2849.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 818-827.
Jana Straková, Milan Straka, and Jan Hajic. 2019. Neural architectures for nested ner through linearization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5326-5331.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57.
Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 204-214.
Bailin Wang, Wei Lu, Yu Wang, and Hongxia Jin. 2018. A neural transition-based model for nested mention recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1011-1017.
Thomas Wolf, L Debut, V Sanh, J Chaumond, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtopicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Congying Xia, Chenwei Zhang, Tao Yang, Yaliang Li, Nan Du, Xian Wu, Wei Fan, Fenglong Ma, and S Yu Philip. 2019. Multi-grained named entity recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1430-1440.
Mingbin Xu, Hui Jiang, and Sedtawut Watcharawittayakul. 2017. A local detection approach for named entity recognition and mention detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1237-1247.
Changmeng Zheng, Yi Cai, Jingyun Xu, Ho-fung Leung, and Guandong Xu. 2019. A boundary-aware neural model for nested named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
5927
9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 357-366.
5928