SlowGuess/ACL_2020_OCR huggingface dataset:
Table 4 evaluates the performance of each module on ACE2005 and GENIA datasets. Our flat module performs well on both datasets for outermost entity recognition. However, the recall of the inner entities is low on GENIA dataset. According to the statistics in Table 1, only  $11\%$  of the entities on GENIA are located in inner layers, while on ACE2005 dataset, the proportion is  $24\%$ . It can be inferred that the sparsity of the entity distribution in inner layers has a great impact on the results. If these inner entities are identified at each layer, the sparsity may be even worse. We can enhance the impact of sparse entities by increasing the weight  $\lambda_{1}$  in Eq.(14), but this may hurt precision, we set  $\lambda_{1} = 1.5$  to have a better tradeoff between precision and recall.
Table 3: Our results on five categories compared to (Zheng et al., 2019) and (Ju et al., 2018) on GENIA dataset.
ACE2005	GENIA
P	R	F	P	R	F
Outermost	73.7	75.0	74.3	78.4	78.9	78.7
Inner	58.3	55.2	56.7	50.9	34.7	41.2
3.5 Analysis of Entity Length
We conduct additional experiments on ACE2005 dataset to detect the effect of the lengths of the outermost entities on the extraction of their inner entities as shown in Table 6. Our flat module can well predict outermost entities which account for a large proportion among all types of entities. In general, the performance of inner entities is affected by the extracting performance and length of their outermost entities. A shorter outermost entity is more likely to have its inner entities shared either the
Table 4: Performance of each module on ACE2005 and GENIA datasets.
ACE2005	GENIA	KBP2017
Flat → Grpah
no graph	73.4	74.4	74.0
adjacent graph	73.8	74.9	74.7
entity graph	74.8	75.5	75.2
both graphs	75.1	76.0	75.6
Graph → Flat
without	74.3	74.5	75.1
with	75.1	76.0	75.6
Table 5: Ablation study on the three benchmark datasets.
first token or the last token, making the constructed graph more instructive, thus its inner entities are easier to extract.
3.6 Ablation Study
In this paper, we use the interactions of flat module and graph module to respectively help better predict outermost and inner entities. We conduct ablation study to verify the effectiveness of the interactions. The first part is the information delivery from the flat module to the graph module. We conduct four experiments: (1) no graph: we skip Eq. (5)-(6) and let graph feature  $f = \mathrm{LINEAR}(x)$ . In this case, inner entities are independent of the outermost entities and only rely on the word representation (section 2.1) which carries contextualized information. (2) adjacent graph: we further utilize the sequential information of the text to help inner entity prediction. (3) entity graph: the boundary information of outer entities can be indicative for inner entities, we construct an entity graph based on the entities extracted by the flat module. (4) both graphs: when outer entities are not recognized by the flat module, their inner entities will fail to receive the boundary information, we use the sequential information of the text to make up for the deficiency of using only entity graph. Experimental
6414
length	outermost entities	inner entities
P	R	F	Num.	P	R	F	Num.
1	75.9	80.6	78.2	1,260	-	-	-	-
2	72.1	74.8	73.4	488	76.6	63.6	69.5	77
3	67.8	72.2	69.9	198	67.6	56.5	61.5	85
4	62.5	60.9	61.7	112	68.1	42.3	52.2	111
5	60.7	48.7	54.0	76	56.0	37.8	45.1	74
6	46.3	46.3	46.3	41	28.0	25.9	26.9	54
7	44.4	30.8	36.4	26	21.7	16.7	18.9	30
8	64.3	40.9	50.0	22	31.8	21.2	25.5	33
9	35.7	31.3	33.3	16	23.1	19.4	21.1	31
10	57.1	22.2	32.0	18	20.0	15.4	17.4	26
Table 6: Length-wise results on ACE2005 test dataset.
results show that entity graph carries more useful information than adjacent graph, which enhances the baseline by  $1.4 / 1.1 / 1.2F_{1}$  score, respectively. By combing these two graphs together, we get a larger gain of  $1.7 / 1.6 / 1.6F_{1}$  score. The second part is the information delivery from the graph module to the flat module, the new representation  $X^{new}$  learned from graph module is propagated back to the flat module.  $X^{new}$  is equipped with the dependencies of inner entities and shows useful, yielding an improvement of  $0.8 / 1.5 / 0.5F_{1}$  for the three benchmarks, respectively.
3.7 Inference Time
We examine the inference speed of our BiFlaG with (Zheng et al., 2019), (Sohrab and Miwa, 2018) and (Ju et al., 2018) in terms of the number of words decoded per second. For all the compared models, we use the re-implemented code released by (Zheng et al., 2019) and set the same batch size 10. Compared with (Zheng et al., 2019) and (Sohrab and Miwa, 2018), our BiFlaG does not need to compute region representation for each potential entity, thus we can take full advantage of GPU parallelism. Compared with (Ju et al., 2018), which requires CRF decoding for each layer, our model only needs to calculate two modules, by contrast, the cascaded CRF layers limit their inference speed.
4 Case Study
Table 7 shows a case study of each module in our model. In this example, entities my, my town, that and Krispy Kreme are nested in the entity the location in my town that was recently abandoned by Krispy Kreme. Our BiFlaG model successfully
Figure 3: The inference speed of our BiFlaG and compared models on GENIA test set. t/s indicates token per second.
extracts all these entities with exact boundaries and entity categorical labels. Without graph construction, nested entities my town, that and Krispy Kreme are not identified. Without interaction between the two modules, the outermost entity the location in my town that was recently abandoned by Krispy Kreme is mislabeled as LOC (location), which is actually a FAC (Facility) type, inner nested entities my, my town and Krispy Kreme are not propagated back to the flat module, which maybe helpful to correct the extracting of the outermost entity.
Recently, with the development of deep neural network in a wide range of NLP tasks (Bai and Zhao, 2018; Huang et al., 2018; Huang and Zhao, 2018; He et al., 2018, 2019; Li et al., 2018a,b, 2019; Zhou and Zhao, 2019; Xiao et al., 2019; Zhang and Zhao,
6415
Setence	Interesting aside: Starbucks is taking over the location in my town that was recently abandoned by Krispy Kreme.
Gold Label	ORG: {Starbucks, Krispy Kreme}; FAC: {the location in my town that was recently abandoned by Krispy Kreme; that}; GPE: {my town}; PER: {my}
No Graph	ORG: {Starbucks}; LOC: {the location in my town that was recently abandoned by Krispy Kreme}; PER: {my}
No interaction	ORG: {Starbucks, Krispy Kreme}; LOC: {the location in my town that was recently abandoned by Krispy Kreme}; GPE: {my town}; PER: {my}
BiFlaG	ORG: {Starbucks, Krispy Kreme}; FAC: {the location in my town that was recently abandoned by Krispy Kreme; that}; GPE: {my town}; PER: {my}
Table 7: An example of predicted results in ACE2005 test dataset.
2018; Zhang et al., 2019, 2020a,b,c), it is possible to build reliable NER systems without hand-crafted features. Nested named entity recognition requires to identify all the entities in texts that may be nested with each other. Though NER is a traditional NLP task, it is not until the very recent years that researches have been paid to this nested structure for named entities.
(Lu and Roth, 2015) introduce a novel hypergraph representation to handle overlapping mentions. (Muis and Lu, 2017) further develop a gap-based tagging schema that assigns tags to gaps between words to address the spurious structures issue, which can be modeled using conventional linear-chain CRFs. However, it suffers from the structural ambiguity issue during inference. (Wang and Lu, 2018) propose a novel segmental hypergraph representation to eliminate structural ambiguity. (Katiyar and Cardie, 2018) also propose a hypergraph-based approach based on the BILOU tag scheme that utilizes an LSTM network to learn the hypergraph representation in a greedy manner.
Stacking sequence labeling models to extract entities from inner to outer (or outside-to-inside) can also handle such nested structures. (Alex et al., 2007) propose several different modeling techniques (layering and cascading) to combine multiple CRFs for nested NER. However, their approach cannot handle nested entities of the same entity type. (Ju et al., 2018) dynamically stack flat NER layers, and recognize entities from innermost layer to outer ones. Their approach can deal with nested entities of the same type, but suffers from error propagation among layers.
Region-based approaches are also commonly used for nested NER by extracting the subsequences in sentences and classifying their types. (Sohrab and Miwa, 2018) introduce a neural ex
haustive model that considers all possible spans and classify their types. This work is further improved by (Zheng et al., 2019), which first apply a single-layer sequence labeling model to identify the boundaries of potential entities using context information, and then classify these boundary-aware regions into their entity type or non-entity. (Lin et al., 2019a) propose a sequence-to-nuggets approach named as Anchor-Region Networks (ARNs) to detect nested entity mentions. They first use an anchor detector to detect the anchor words of entity mentions and then apply a region recognizer to identify the mention boundaries centering at each anchor word. (Fisher and Vlachos, 2019) decompose nested NER into two stages. Tokens are merged into entities through real-valued decisions, and then the entity embeddings are used to label the entities identified.
This paper proposes a new bipartite flat-graph (BiFlaG) model for nested NER which consists of two interacting subgraph modules. Applying the divide-and-conquer policy, the flat module is in charge of outermost entities, while the graph module focuses on inner entities. Our BiFlaG model also facilitates a full bidirectional interaction between the two modules, which let the nested NE structures jointly learned at most degree. As a general model, our BiFlaG model can also handle non-nested structures by simply removing the graph module. In terms of the same strict setting, empirical results show that our model generally outperforms previous state-of-the-art models.
6416
Beatrice Alex, Barry Haddow, and Claire Grover. 2007. Recognising nested named entities in biomedical text. In Biological, translational, and clinical language processing, pages 65-72.
Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571-583.
Jenny Rose Finkel and Christopher D. Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 141-150.
Joseph Fisher and Andreas Vlachos. 2019. Merge and label: A novel neural network architecture for nested NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5840-5850.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019. GraphRel: Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1409-1418.
Shexia He, Zuchao Li, and Hai Zhao. 2019. Syntax-aware multilingual semantic role labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5350-5359.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2061-2071.
Yafang Huang, Zuchao Li, Zhuosheng Zhang, and Hai Zhao. 2018. Moon IME: Neural-based Chinese pinyin aided input method with customizable association. In Proceedings of ACL 2018, System Demonstrations, pages 140-145.
Yafang Huang and Hai Zhao. 2018. Chinese pinyin aided IME, input what you have not keystroked yet. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2923-2929.
Heng Ji, Xiaoman Pan, Boliang Zhang, Joel Nothman, James Mayfield, Paul McNamee, and Cash Costello. 2017. Overview of tac-kbp2017 13 languages entity discovery and linking. Theory and Applications of Categories.
Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan
guage Technologies, Volume 1 (Long Papers), pages 1446-1459.
Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 861-871.
J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun'ichi Tsujii. 2003. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, pages i180-i182.
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR).
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In International Conference on Machine Learning.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270.
Zuchao Li, Jiaxun Cai, Shexia He, and Hai Zhao. 2018a. Seq2seq dependency parsing. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3203-3214.
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018b. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2401-2411.
Zuchao Li, Shexia He, Hai Zhao, Yiqing Zhang, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019. Dependency or span, end-to-end uniform semantic role labeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6730-6737.
Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2019a. Sequence-to-nuggets: Nested entity mention detection via anchor-region networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5182-5192.
Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Bin Dong, and Shanshan Jiang. 2019b. Gazetteer-enhanced attentive neural networks for named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6232-6237.
6417
Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857-867.
Ying Luo, Fengshun Xiao, and Hai Zhao. 2020. Hierarchical contextualized representation for named entity recognition. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2020).
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506-1515.
Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between words: Recognizing overlapping mentions with mention separators. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2608-2618. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756-1765.
Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. 2019. GraphIE: A graph-based framework for information extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 751-761.
Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2843-2849.
Jana Straková, Milan Straka, and Jan Hajic. 2019. Neural architectures for nested NER through linearization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5326-5331.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57.
Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 204-214.
Fengshun Xiao, Jiangtong Li, Hai Zhao, Rui Wang, and Kehai Chen. 2019. Lattice-based transformer encoder for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3090-3097.
Yingwei Xin, Ethan Hart, Vibhuti Mahajan, and Jean-David Ruvini. 2018. Learning better internal structure of words for sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2584-2593.
Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In Proceedings of ACL 2018, System Demonstrations, pages 74-79.
Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2020a. DCMN+: Dual co-matching network for multi-choice reading comprehension. In the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2020).
Zhuosheng Zhang, Yafang Huang, and Hai Zhao. 2019. Open vocabulary learning for neural Chinese pinyin IME. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1584-1594.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuaijiang Zhang, Xi Zhou, and Xiang Zhou. 2020b. Semantics-aware BERT for language understanding. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2020).
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, and Hai Zhao. 2020c. SG-Net: Syntax-guided machine reading comprehension. In the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2020).
Zhuosheng Zhang and Hai Zhao. 2018. One-shot learning for question-answering in Gaokao history challenge. In Proceedings of the 27th International Conference on Computational Linguistics, pages 449–461.
Changmeng Zheng, Yi Cai, Jingyun Xu, Ho-fung Leung, and Guandong Xu. 2019. A boundary-aware neural model for nested named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 357-366.
Junru Zhou and Hai Zhao. 2019. Head-driven phrase structure grammar parsing on Penn treebank. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2396-2408.
6418