SlowGuess/ACL_2020_OCR huggingface dataset:
Xiaoya Li\*, Jingrong Feng\*, Yuxian Meng\*, Qinghong Han\*, Fei Wu\* and Jiwei Li\*\*
$\spadesuit$  Department of Computer Science and Technology, Zhejiang University
* Shannon.AI
{xiaoya_li, jingrong_feng, yuxian_meng, qinghong_han} @ shannonai.com
wufei@cs.zju.edu.cn, jiwei_li@shannonai.com
The task of named entity recognition (NER) is normally divided into nested NER and flat NER depending on whether named entities are nested or not. Models are usually separately developed for the two tasks, since sequence labeling models are only able to assign a single label to a particular token, which is unsuitable for nested NER where a token may be assigned several labels.
In this paper, we propose a unified framework that is capable of handling both flat and nested NER tasks. Instead of treating the task of NER as a sequence labeling problem, we propose to formulate it as a machine reading comprehension (MRC) task. For example, extracting entities with the PER(Person) label is formalized as extracting answer spans to the question "which person is mentioned in the text". This formulation naturally tackles the entity overlapping issue in nested NER: the extraction of two overlapping entities with different categories requires answering two independent questions. Additionally, since the query encodes informative prior knowledge, this strategy facilitates the process of entity extraction, leading to better performances for not only nested NER, but flat NER.
We conduct experiments on both nested and flat NER datasets. Experiment results demonstrate the effectiveness of the proposed formulation. We are able to achieve a vast amount of performance boost over current SOTA models on nested NER datasets, i.e., +1.28, +2.55, +5.44, +6.37, respectively on ACE04, ACE05, GENIA and KBP17, as well as flat NER datasets, i.e., +0.24, +1.95, +0.21, +1.49 respectively on English CoNLL 2003, English OntoNotes 5.0, Chinese MSRA and Chinese OntoNotes 4.0. The code and datasets can be found at https://github.com/ShannonAI/mrc-for-flat-nested-ner.
Alpha B2 proteins bound the PEBP2 site within the mouse GM-CSF promoter.
Last night, at the Chinese embassy in France, there was a holiday atmosphere.
Figure 1: Examples for nested entities from GENIA and ACE04 corpora.
Named Entity Recognition (NER) refers to the task of detecting the span and the semantic category of entities from a chunk of text. The task can be further divided into two sub-categories, nested NER and flat NER, depending on whether entities are nested or not. Nested NER refers to a phenomenon that the spans of entities (mentions) are nested, as shown in Figure 1. Entity overlapping is a fairly common phenomenon in natural languages.
The task of flat NER is commonly formalized as a sequence labeling task: a sequence labeling model (Chiu and Nichols, 2016; Ma and Hovy, 2016; Devlin et al., 2018) is trained to assign a single tagging class to each unit within a sequence of tokens. This formulation is unfortunately incapable of handling overlapping entities in nested NER (Huang et al., 2015; Chiu and Nichols, 2015), where multiple categories need to be assigned to a single token if the token participates in multiple entities. Many attempts have been made to reconcile sequence labeling models with nested NER (Alex et al., 2007; Byrne, 2007; Finkel and Manning, 2009; Lu and Roth, 2015; Katiyar and Cardie, 2018), mostly based on the pipelined systems. However, pipelined systems suffer from the disadvantages of error propagation, long running time and the intensiveness in developing hand-crafted features, etc.
Inspired by the current trend of formalizing
5849
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5859 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
NLP problems as question answering tasks (Levy et al., 2017; McCann et al., 2018; Li et al., 2019), we propose a new framework that is capable of handling both flat and nested NER. Instead of treating the task of NER as a sequence labeling problem, we propose to formulate it as a SQuAD-style (Rajpurkar et al., 2016, 2018) machine reading comprehension (MRC) task. Each entity type is characterized by a natural language query, and entities are extracted by answering these queries given the contexts. For example, the task of assigning the PER(Person) label to "[Washington] was born into slavery on the farm of James Burroughs" is formalized as answering the question "which person is mentioned in the text?" This strategy naturally tackles the entity overlapping issue in nested NER: the extraction of two entities with different categories that overlap requires answering two independent questions.
The MRC formulation also comes with another key advantage over the sequence labeling formulation. For the latter, golden NER categories are merely class indexes and lack for semantic prior information for entity categories. For example, the ORG(ORGANIZATION) class is treated as a one-hot vector in sequence labeling training. This lack of clarity on what to extract leads to inferior performances. On the contrary, for the MRC formulation, the query encodes significant prior information about the entity category to extract. For example, the query "find an organization such as company, agency and institution in the context" encourages the model to link the word "organization" in the query to location entities in the context. Additionally, by encoding comprehensive descriptions (e.g., "company, agency and institution") of tagging categories (e.g., ORG), the model has the potential to disambiguate similar tagging classes.
We conduct experiments on both nested and flat NER datasets to show the generality of our approach. Experimental results demonstrate its effectiveness. We are able to achieve a vast amount of performance boost over current SOTA models on nested NER datasets, i.e., +1.28, +2.55, +5.44, +6.37, respectively on ACE04, ACE05, GENIA and KBP17, as well as flat NER datasets, i.e., +0.24, +1.95, +0.21, +1.49 respectively on English CoNLL 2003, English OntoNotes 5.0, Chinese MSRA, Chinese OntoNotes 4.0. We wish that our work would inspire the introduction of
new paradigms for the entity recognition task.
2.1 Named Entity Recognition (NER)
Traditional sequence labeling models use CRFs (Lafferty et al., 2001; Sutton et al., 2007) as a backbone for NER. The first work using neural models for NER goes back to 2003, when Hammerton (2003) attempted to solve the problem using unidirectional LSTMs. Collobert et al. (2011) presented a CNN-CRF structure, augmented with character embeddings by Santos and Guimaraes (2015). Lample et al. (2016) explored neural structures for NER, in which the bidirectional LSTMs are combined with CRFs with features based on character-based word representations and unsupervised word representations. Ma and Hovy (2016) and Chiu and Nichols (2016) used a character CNN to extract features from characters. Recent large-scale language model pretraining methods such as BERT (Devlin et al., 2018) and ELMo (Peters et al., 2018a) further enhanced the performance of NER, yielding state-of-the-art performances.
2.2 Nested Named Entity Recognition
The overlapping between entities (mentions) was first noticed by Kim et al. (2003), who developed handcrafted rules to identify overlapping mentions. Alex et al. (2007) proposed two multi-layer CRF models for nested NER. The first model is the inside-out model, in which the first CRF identifies the innermost entities, and the successive layer CRF is built over words and the innermost entities extracted from the previous CRF to identify second-level entities, etc. The other is the outside-in model, in which the first CRF identifies outermost entities, and then successive CRFs would identify increasingly nested entities. Finkel and Manning (2009) built a model to extract nested entity mentions based on parse trees. They made the assumption that one mention is fully contained by the other when they overlap. Lu and Roth (2015) proposed to use mention hyper-graphs for recognizing overlapping mentions. Xu et al. (2017) utilized a local classifier that runs on every possible span to detect overlapping mentions and Katiyar and Cardie (2018) used neural models to learn the hyper-graph representations for nested entities. Ju et al. (2018) dynamically stacked flat NER layers in a hierarchical manner. Lin et al.
(2019a) proposed the Anchor-Region Networks (ARNs) architecture by modeling and leveraging the head-driven phrase structures of nested entity mentions. Luan et al. (2019) built a span enumeration approach by selecting the most confident entity spans and linking these nodes with confidence-weighted relation types and coreferences. Other works (Muis and Lu, 2017; Sohrab and Miwa, 2018; Zheng et al., 2019) also proposed various methods to tackle the nested NER problem.
Recently, nested NER models are enriched with pre-trained contextual embeddings such as BERT (Devlin et al., 2018) and ELMo (Peters et al., 2018b). Fisher and Vlachos (2019) introduced a BERT-based model that first merges tokens and/or entities into entities, and then assigned labeled to these entities. Shibuya and Hovy (2019) provided inference model that extracts entities iteratively from outermost ones to inner ones. Straková et al. (2019) viewed nested NER as a sequence-to-sequence generation problem, in which the input sequence is a list of tokens and the target sequence is a list of labels.
2.3 Machine Reading Comprehension (MRC)
MRC models (Seo et al., 2016; Wang et al., 2016; Wang and Jiang, 2016; Xiong et al., 2016, 2017; Wang et al., 2016; Shen et al., 2017; Chen et al., 2017) extract answer spans from a passage through a given question. The task can be formalized as two multi-class classification tasks, i.e., predicting the starting and ending positions of the answer spans.
Over the past one or two years, there has been a trend of transforming NLP tasks to MRC question answering. For example, Levy et al. (2017) transformed the task of relation extraction to a QA task: each relation type  $R(x,y)$  can be parameterized as a question  $q(x)$  whose answer is  $y$ . For example, the relation EDUCATED-AT can be mapped to "Where did x study?". Given a question  $q(x)$ , if a non-null answer  $y$  can be extracted from a sentence, it means the relation label for the current sentence is  $R$ . McCann et al. (2018) transformed NLP tasks such as summarization or sentiment analysis into question answering. For example, the task of summarization can be formalized as answering the question "What is the summary?" Our work is significantly inspired by Li et al. (2019), which formalized the task
of entity-relation extraction as a multi-turn question answering task. Different from this work, Li et al. (2019) focused on relation extraction rather than NER. Additionally, Li et al. (2019) utilized a template-based procedure for constructing queries to extract semantic relations between entities and their queries lack diversity. In this paper, more factual knowledge such as synonyms and examples are incorporated into queries, and we present an in-depth analysis of the impact of strategies of building queries.
3 NER as MRC
Given an input sequence  $X = \{x_{1}, x_{2}, \dots, x_{n}\}$ , where  $n$  denotes the length of the sequence, we need to find every entity in  $X$ , and then assign a label  $y \in Y$  to it, where  $Y$  is a predefined list of all possible tag types (e.g., PER, LOC, etc).
Dataset Construction Firstly we need to transform the tagging-style annotated NER dataset to a set of (QUESTION, ANSWER, CONTEXT) triples. For each tag type  $y \in Y$ , it is associated with a natural language question  $q_{y} = \{q_{1}, q_{2}, \ldots, q_{m}\}$ , where  $m$  denotes the length of the generated query. An annotated entity  $x_{\text{start, end}} = \{x_{\text{start}}, x_{\text{start+1}}, \dots, x_{\text{end-1}}, x_{\text{end}}\}$  is a substring of  $X$  satisfying start  $\leq$  end. Each entity is associated with a golden label  $y \in Y$ . By generating a natural language question  $q_{y}$  based on the label  $y$ , we can obtain the triple  $(q_{y}, x_{\text{start, end}}, X)$ , which is exactly the (QUESTION, ANSWER, CONTEXT) triple that we need. Note that we use the subscript "start, end" to denote the continuous tokens from index 'start' to 'end' in a sequence.
3.2 Query Generation
The question generation procedure is important since queries encode prior knowledge about labels and have a significant influence on the final results. Different ways have been proposed for question generation, e.g., Li et al. (2019) utilized a template-based procedure for constructing queries to extract semantic relations between entities. In this paper, we take annotation guideline notes as references to construct queries. Annotation guideline notes are the guidelines provided to the annotators of the dataset by the dataset builder. They are descriptions of tag categories, which are described as generic and precise as possible so that
5851
Entity	Natural Language Question
Location	Find locations in the text, including non-geographical locations, mountain ranges and bodies of water.
Facility	Find facilities in the text, including buildings, airports, highways and bridges.
Organization	Find organizations in the text, including companies, agencies and institutions.
Table 1: Examples for transforming different entity categories to question queries.
human annotators can annotate the concepts or mentions in any text without running into ambiguity. Examples are shown in Table 1.
3.3.1 Model Backbone
Given the question  $q_{y}$ , we need to extract the text span  $x_{\mathrm{start, end}}$  which is with type  $y$  from  $X$  under the MRC framework. We use BERT (Devlin et al., 2018) as the backbone. To be in line with BERT, the question  $q_{y}$  and the passage  $X$  are concatenated, forming the combined string  $\{[\mathrm{CLS}], q_{1}, q_{2}, \dots, q_{m}, [\mathrm{SEP}], x_{1}, x_{2}, \dots, x_{n}\}$ , where [CLS] and [SEP] are special tokens. Then BERT receives the combined string and outputs a context representation matrix  $E \in \mathbb{R}^{n \times d}$ , where  $d$  is the vector dimension of the last layer of BERT and we simply drop the query representations.
3.3.2 Span Selection
There are two strategies for span selection in MRC: the first strategy (Seo et al., 2016; Wang et al., 2016) is to have two  $n$ -class classifiers separately predict the start index and the end index, where  $n$  denotes the length of the context. Since the softmax function is put over all tokens in the context, this strategy has the disadvantage of only being able to output a single span given a query; the other strategy is to have two binary classifiers, one to predict whether each token is the start index or not, the other to predict whether each token is the end index or not. This strategy allows for outputting multiple start indexes and multiple end indexes for a given context and a specific query, and thus has the potentials to extract all related entities according to  $q_{y}$ . We adopt the second strategy and describe the details below.
Start Index Prediction Given the representation matrix  $E$  output from BERT, the model first predicts the probability of each token being a start
index as follows:
$$
P _ {\text {s t a r t}} = \operatorname {s o f t m a x} _ {\text {e a c h r o w}} (E \cdot T _ {\text {s t a r t}}) \in \mathbb {R} ^ {n \times 2} \tag {1}
$$
$T_{\mathrm{start}} \in \mathbb{R}^{d \times 2}$  is the weights to learn. Each row of  $P_{\mathrm{start}}$  presents the probability distribution of each index being the start position of an entity given the query.
End Index Prediction The end index prediction procedure is exactly the same, except that we have another matrix  $T_{\mathrm{end}}$  to obtain probability matrix  $P_{\mathrm{end}} \in \mathbb{R}^{n \times 2}$ .
Start-End Matching In the context  $X$ , there could be multiple entities of the same category. This means that multiple start indexes could be predicted from the start-index prediction model and multiple end indexes predicted from the end-index prediction model. The heuristic of matching the start index with its nearest end index does not work here since entities could overlap. We thus further need a method to match a predicted start index with its corresponding end index.
Specifically, by applying argmax to each row of  $P_{\mathrm{start}}$  and  $P_{\mathrm{end}}$ , we will get the predicted indexes that might be the starting or ending positions, i.e.,  $\hat{I}_{\mathrm{start}}$  and  $\hat{I}_{\mathrm{end}}$ :
$$
\begin{array}{l} \hat {I} _ {\text {s t a r t}} = \{i \mid \operatorname {a r g m a x} (P _ {\text {s t a r t}} ^ {(i)}) = 1, i = 1, \dots , n \} \\ \hat {I} _ {\text {e n d}} = \{j \mid \operatorname {a r g m a x} \left(P _ {\text {e n d}} ^ {(j)}\right) = 1, j = 1, \dots , n \} \tag {2} \\ \end{array}
$$
where the superscript  $(i)$  denotes the  $i$ -th row of a matrix. Given any start index  $i_{\mathrm{start}} \in \hat{I}_{\mathrm{start}}$  and end index  $i_{\mathrm{end}} \in \hat{I}_{\mathrm{end}}$ , a binary classification model is trained to predict the probability that they should be matched, given as follows:
$$
P _ {i _ {\text {s t a r t}}, j _ {\text {e n d}}} = \operatorname {s i g m o i d} (m \cdot \operatorname {c o n c a t} \left(E _ {i _ {\text {s t a r t}}}, E _ {j _ {\text {e n d}}}\right)) \tag {3}
$$
where  $m\in \mathbb{R}^{1\times 2d}$  is the weights to learn.
3.4 Train and Test
At training time,  $X$  is paired with two label sequences  $Y_{\mathrm{start}}$  and  $Y_{\mathrm{end}}$  of length  $n$  representing the ground-truth label of each token  $x_{i}$  being the start index or end index of any entity. We therefore have the following two losses for start and end index predictions:
$$
\begin{array}{l} \begin{array}{l} \mathcal {L} _ {\text {s t a r t}} = \mathrm {C E} \left(P _ {\text {s t a r t}}, Y _ {\text {s t a r t}}\right) \\ \mathcal {L} _ {\text {s t a r t}} = \mathrm {C E} \left(P _ {\text {s t a r t}}, Y _ {\text {s t a r t}}\right) \end{array} \tag {4} \\ \mathcal {L} _ {\text {e n d}} = \mathrm {C E} (P _ {\text {e n d}}, Y _ {\text {e n d}}) \\ \end{array}
$$
Let  $Y_{\text{start, end}}$  denote the golden labels for whether each start index should be matched with each end
5852
index. The start-end index matching loss is given as follows:
$$
\mathcal {L} _ {\text {s p a n}} = \mathrm {C E} \left(P _ {\text {s t a r t , e n d}}, Y _ {\text {s t a r t , e n d}}\right) \tag {5}
$$
The overall training objective to be minimized is as follows:
$$
\mathcal {L} = \alpha \mathcal {L} _ {\text {s t a r t}} + \beta \mathcal {L} _ {\text {e n d}} + \gamma \mathcal {L} _ {\text {s p a n}} \tag {6}
$$
$\alpha, \beta, \gamma \in [0,1]$  are hyper-parameters to control the contributions towards the overall training objective. The three losses are jointly trained in an end-to-end fashion, with parameters shared at the BERT layer. At test time, start and end indexes are first separately selected based on  $\hat{I}_{\mathrm{start}}$  and  $\hat{I}_{\mathrm{end}}$ . Then the index matching model is used to align the extracted start indexes with end indexes, leading to the final extracted answers.
4.1 Experiments on Nested NER
For nested NER, experiments are conducted on the widely-used ACE 2004, ACE 2005, GENIA and KBP2017 datasets, which respectively contain  $24\%$ ,  $22\%$ ,  $10\%$  and  $19\%$  nested mentions. Hyperparameters are tuned on their corresponding development sets. For evaluation, we use span-level micro-averaged precision, recall and F1.
ACE 2004 and ACE 2005 (Doddington et al., 2005; Christopher Walker and Maeda, 2006): The two datasets each contain 7 entity categories. For each entity type, there are annotations for both the entity mentions and mention heads. For fair comparison, we exactly follow the data preprocessing strategy in Katiyar and Cardie (2018) and Lin et al. (2019b) by keeping files from bn, nw and wl, and splitting these files into train, dev and test sets by 8:1:1, respectively.
GENIA (Ohta et al., 2002) For the GENIA dataset, we use GENIAcorpus3.02p. We follow the protocols in Katiyar and Cardie (2018).
KBP2017 We follow Katiyar and Cardie (2018) and evaluate our model on the 2017 English evaluation dataset (LDC2017D55). Training set consists of RichERE annotated datasets, which include LDC2015E29, LDC2015E68, LDC2016E31 and LDC2017E02. We follow the dataset split strategy in Lin et al. (2019b).
We use the following models as baselines:
- Hyper-Graph: Katiyar and Cardie (2018) proposes a hypergraph-based model based on LSTMs.
- Seg-Graph: Wang and Lu (2018) proposes a segmental hypergargh representation to model overlapping entity mentions.
- ARN: Lin et al. (2019a) proposes AnchorRegion Networks by modeling and levraging the head-driven phrase structures of entity mentions.
- KBP17-Best: Ji et al. (2017) gives an overview of the Entity Discovery task at the Knowledge Base Population (KBP) track at TAC2017 and also reports previous best results for the task of nested NER.
- Seq2Seq-BERT: Straková et al. (2019) views the nested NER as a sequence-to-sequence problem. Input to the model is word tokens and the output sequence consists of labels.
- Path-BERT: Shibuya and Hovy (2019) treats the tag sequence as the second best path within in the span of their parent entity based on BERT.
- Merge-BERT: Fisher and Vlachos (2019) proposes a merge and label method based on BERT.
- DYGIE: Luan et al. (2019) introduces a general framework that share span representations using dynamically constructed span graphs.
4.1.3 Results
Table 2 shows experimental results on nested NER datasets. We observe huge performance boosts on the nested NER datasets over previous state-of-the-art models, achieving F1 scores of  $85.98\%$ ,  $86.88\%$ ,  $83.75\%$  and  $80.97\%$  on ACE04, ACE05, GENIA and KBP-2017 datasets, which are  $+1.28\%$ ,  $+2.55\%$ ,  $+5.44\%$  and  $+6.37\%$  over previous SOTA performances, respectively.
4.2 Experiments on Flat NER
4.2.1 Datasets
For flat NER, experiments are conducted on both English datasets i.e. CoNLL2003 and OntoNotes 5.0 and Chinese datasets i.e. OntoNotes 4.0 and MSRA. Hyperparameters are tuned on their corresponding development sets. We report span-level
5853
English ACE 2004
Model	Precision	Rrecall	F1
Hyper-Graph (Katiyar and Cardie, 2018)	73.6	71.8	72.7
Seg-Graph (Wang and Lu, 2018)	78.0	72.4	75.1
Seq2seq-BERT (Straková et al., 2019)	-	-	84.40
Path-BERT (Shibuya and Hovy, 2019)	83.73	81.91	82.81
DYGIE (Luan et al., 2019)	-	-	84.7
BERT-MRC	85.05	86.32	85.98 (+1.28)
English ACE 2005
Model	Precision	Recall	F1
Hyper-Graph (Katiyar and Cardie, 2018)	70.6	70.4	70.5
Seg-Graph (Wang and Lu, 2018)	76.8	72.3	74.5
ARN (Lin et al., 2019a)	76.2	73.6	74.9
Path-BERT (Shibuya and Hovy, 2019)	82.98	82.42	82.70
Merge-BERT (Fisher and Vlachos, 2019)	82.7	82.1	82.4
DYGIE (Luan et al., 2019)	-	-	82.9
Seq2seq-BERT (Straková et al., 2019)	-	-	84.33
BERT-MRC	87.16	86.59	86.88 (+2.55)
English GENIA
Model	Precision	Recall	F1
Hyper-Graph (Katiyar and Cardie, 2018)	77.7	71.8	74.6
ARN (Lin et al., 2019a)	75.8	73.9	74.8
Path-BERT (Shibuya and Hovy, 2019)	78.07	76.45	77.25
DYGIE (Luan et al., 2019)	-	-	76.2
Seq2seq-BERT (Straková et al., 2019)	-	-	78.31
BERT-MRC	85.18	81.12	83.75 (+5.44)
English KBP 2017
Model	Precision	Recall	F1
KBP17-Best (Ji et al., 2017)	76.2	73.0	72.8
ARN (Lin et al., 2019a)	77.7	71.8	74.6
BERT-MRC	82.33	77.61	80.97 (+6.37)
micro-averaged precision, recall and F1 scores for evaluation.
CoNLL2003 (Sang and Meulder, 2003) is an English dataset with four types of named entities: Location, Organization, Person and Miscellaneous. We followed data processing protocols in Ma and Hovy (2016).
OntoNotes 5.0 (Pradhan et al., 2013) is an English dataset and consists of text from a wide variety of sources. The dataset includes 18 types of named entity, consisting of 11 types (Person, Organization, etc) and 7 values (Date, Percent, etc).
MSRA (Levow, 2006) is a Chinese dataset and performs as a benchmark dataset. Data in MSRA is collected from news domain and is used as shared task on SIGNAN backoff 2006. There are three types of named entities.