Title: MULTI-GRAINED ENTITY PROPOSAL NETWORK FOR NAMED ENTITY RECOGNITION

Abstract: In this paper, we focus on a new Named Entity Recognition (NER) task, i.e., the Multi-grained NER task. This task aims to simultaneously detect both finegrained and coarse-grained entities in sentences. Correspondingly, we develop a novel Multi-grained Entity Proposal Network (MGEPN). Different from traditional NER models which regard NER as a sequential labeling task, MGEPN provides a new method that proposes entity candidates in the Proposal Network and classifies entities into different categories in the Classification Network. All possible entity candidates including fine-grained ones and coarse-grained ones are proposed in the Proposal Network, which enables the MGEPN model to identify multi-grained entities. In order to better identify named entities and determine their categories, context information is utilized and transferred from the Proposal Network to the Classification Network during the learning process. A novel Entity-Context attention mechanism is also introduced to help the model focus on entity-related context information. Experiments show that our model can obtain state-of-the-art performance on two real-world datasets for both the Multi-grained NER task and the traditional NER task.

1. INTRODUCTION:

In order to really understand the semantic meanings of natural languages, the first step is to identify meaningful entities from the raw text effectively. Such a process is usually called Named Entity Recognition (NER), which is one of the fundamental tasks in natural language processing (NLP). A typical NER takes an utterance as the input and outputs identified entities, such as person names, locations, and organizations. The extracted named entities can benefit various subsequent NLP tasks, including syntactic parsing (Koo & Collins, 2010) , question answering (Krishnamurthy & Mitchell, 2015) and relation extraction (Lao & Cohen, 2010) . However, accurately recognizing representative entities remains challenging.

Recent work treats NER as a sequence labeling problem. For example, Peters et al. ( 2017) achieves the state-of-the-art performance on sequence labeling problem by incorporating deep recurrent neural networks (RNN) (Hochreiter & Schmidhuber, 1997) with conditional random field (CRF) (Lafferty et al., 2001) . However, a critical problem that arises by treating NER as a sequence labeling task is that it can only recognize non-overlapping entities. It fails to detect entities when they are over-lapping with each other: e.g. a fine-grained entity and a coarse-grained entity as shown in Figure 1 . Traditional NER approaches can only identify either "Belgain Grand Prix" or "Grand Prix", but not both of them unless two NER models with different granularities are trained seperately. Here, we defined the NER task that needs to detect both fine-grained entities and coarse-grained entities as the Multi-grained Named Entity Recognition (MGNER) problem. Figure 1 : An example from the CoNLL-2003 dataset.In this utterance, the coarse-grained named entity "Grand Prix" overlaps with the fine-grained named entity "Belgian Grand Prix".

To solve the MGNER problem, we propose a novel deep neural network named Multi-grained Entity Proposal Network (MGEPN). Essentially, the idea of MGEPN is to first propose entity candidates and then classify these candidates into different categories. The proposed MGEPN model is composed of two sub-modules: the Proposal Network that proposes entities and the Classification Network that classifies entities. In the Proposal Network, all possible entity candidates including fine-grained ones and coarse-grained ones will be proposed like (Lee et al., 2017b; He et al., 2018) . Consequently, our model is able to detect both fine-grained entities and coarse-grained entities without explicitly assuming that entities are non-overlapping or totally nested (Finkel & Manning, 2009) . To improve the model performance as well as to speed up the model learning process, context information learned in the Proposal Network is transferred into the Classification Network. In the Classification Network, a novel Entity-Context attention mechanism is proposed to help the model focus on entity-related context for classification.

In summary, the contributions of this work are:

• We study the Multi-grained Named Entity Recognition (MGNER) problem, which aims to detect both coarse-grained and fine-grained name entities in a single system. 

x k = [w k ; c k ].

The pre-trained word embedding w k with dimension D w is obtained from GloVe (Pennington et al., 2014) , and the character-level word information c k is obtained with a bidirectional LSTM (Hochreiter & Schmidhuber, 1997) layer to capture the morphological information.

The hidden size of this character LSTM is set as D cl . As shown in the bottom of Figure 2 , character embeddings are fed into the character LSTM. Those character embeddings are randomly initialized and learned within the Proposal Network. The final hidden states from the forward and backward character LSTM are concatenated as the character-level word information c k .

2.1.2. SENTENCE REPRESENTATION:

To learn the context information from each sentence, another bidirectional LSTM, named word LSTM, is applied to sequentially encode the utterance. For each token, the forward hidden states 

EQUATION

Besides, we also utilize the language model embeddings pre-trained in the Elmo (Peters et al., 2018) fashion which is an unsupervised way. The pre-trained Elmo embeddings are concatenated to the hidden states h k in the word LSTM. Hence, the concatenated hidden states h k for each token can be reformulated as:

EQUATION

where ELMo k is the ELMo embeddings for token t k . A three-layer hierarchy bi-LSTM neural network is trained as the language model. Since the lower-level LSTM hidden states have the ability to model syntax properties and higher-level LSTM hidden states can capture context information, Elmo computes the language model embeddings as a weighted combination of all the bidirectional LSTM hidden states:

EQUATION

where γ is a task-specified scale parameter which indicates the importance of the entire ELMo vector to the NER task. L is the number of layers used in the pre-trained language model, the vector u = [u 0 , • • • , u L ] represents softmax-normalized weights that combines different layers. h LM k,l are the language model hidden states of layer l at time step k.

A sentence bidirectional LSTM layer with hidden dimension D sl is employed on the top of the concatenated hidden states h k . The forward and backward hidden states in this sentence LSTM are concatenated for each token as the final representation f k ∈ R 2D sl .

2.1.3. ENTITY PROPOSAL:

Using the semantic meaningful features obtained in f k , we can identify representative entity proposals for each utterance. The strategy of finding entity candidates is to first generate all possible entity proposals and then estimate the probability of being an entity or not for each proposal.

Different types of entity proposals are generated surrounding each token position, and a sliding window is applied among the sequence to output all possible ones. In order to limit the number of generated proposals, we set the maximum length of an entity proposal to R. Thus, there will be at most R types of different entity proposals generated at each position. An example of how the proposed method generates different types of entity proposals for the token t 3 is illustrated in Figure 3 . Specifically, assuming that an input utterance consisting of a sequence of five tokens (t 1 , t 2 , t 3 , t 4 , t 5 ), and the Proposal Network takes a maximum length of proposals R of 5. Here at token t 3 , we employ the following five proposals: (t 3 ), (t 3 , t 4 ), (t 2 , t 3 , t 4 ), (t 2 , t 3 , t 4 , t 5 ), (t 1 , t 2 , t 3 , t 4 , t 5 ). For each sliding-window location (token), we simultaneously estimate the probability of a proposal being an entity or not, for all types of entity proposals. A fully connected layer with a two-class softmax function is used to determine the quality of entity proposals:

EQUATION

where W p ∈ R 2D sl ×2R and b p ∈ R 2R are the weights for the entity proposal layer; s k contains 2R scores including R scores for being an entity and R scores for not being an entity at position k.

In the Proposal Network, we employ the cross-entropy loss as follows:

EQUATION

where y r k is the label for proposal type r at position k and s r k is the probability of being an entity for proposal type r at position k. It is worth mentioning that, most entity proposals are negative proposals. Thus, to balance the influence of positive proposals and negative proposals in the loss function, we keep all positive proposals and use down-sampling for negative proposals when calculating the loss L p . For each batch, we fix the number of the total proposals, including all positive proposals and sampled negative proposals, used in the loss function as N b . In the inference procedure of the Proposal Network, an entity proposal will be recognized as an entity candidate if its score of being an entity is higher than score of not being an entity.

2.2. THE CLASSIFICATION NETWORK:

The Classification Network aims to classify entity candidates obtained from the Proposal Network into different categories. Here we use both sentence-level context information and the proposed Entity-Context attention to help the model focus on entity-related context tokens. The framework of the Classification Network is shown in the right part of Figure 2 . Basically, it consists of three modules: Word Representation, Entity Representation and Entity Classification.

2.2.1. WORD REPRESENTATION:

The network architecture used for learning word representations in the Classification Network is the same as that in the Proposal Network. It is a concatenation of the pre-trained word-level embedding from (Pennington et al., 2014) and a character-level word information learned from a bidirectional character LSTM (Hochreiter & Schmidhuber, 1997) .

2.2.2. ENTITY REPRESENTATION:

According to the proposal type illustrated in Figure 3 , we can obtain words contained in the entity candidates from the utterances. Similar to the sentence representation in the Proposal Network, we concatenate the word representation and the ELMo language model embeddings together as the entity features. A bidirectional LSTM with hidden size D el is applied to the entity feature to capture sequence information among the entity words. The last hidden states of the forward and backward Entity LSTMs are concatenated as the entity representation e ∈ R 2D el .

2.2.3. ENTITY CLASSIFICATION:

A same word in different context may have different semantic meanings.Thus, in our model, we take the context information into consideration when learning the semantic representations of entity candidates. We capture the context information from other words in the same utterance. Denote c as the context feature vector for these context words, and it can be extracted from the sentence representation in the Proposal Network. The sentence features trained in the Proposal Network is directly transferred to the Classification Network to improve the speed of model learning.

An easy way to model context words is to concatenate all the word representations or average them. However, this naive approach may fail when there exists a lot of unrelated context words. To select high-relevant context words and learn an accurate context representation, we propose an Entity-Context attention mechanism, where the goal is to simulate the relatedness between the context and the entity. The attention module takes the entity representation e and all the context features C = [c 1 , c 2 , ..., c N ] as the inputs and outputs a vector of attention weights a:

EQUATION

where W ∈ R 2D sl ×2D el is a weight matrix for the attention layer, and a is the Entity-Context attention weight on different context words. To help the model focus on entity-related context, the attentive vector C att is calculated as the attention-weighted context:

EQUATION

)

The lengths of the attentive context C att varies in different contexts. However, the goal of the Classification Network is to classify entity candidates into different categories, and thus it requires a fixed embedding size. We achieve that by adding another LSTM layer. An Attention LSTM with the hidden dimension D ml is used and the concatenation of the last hidden states in the forward and backward LSTM layer as the context representation m ∈ R 2D ml . Hence the shape of the context representation is aligned. We concatenate the context representation and the entity representation together as the feature to classify entity candidates: o = [m; e].

A two-layer fully connected neural network is used to classify candidates into pre-defined categories:

EQUATION

where Dt+1) , b c2 ∈ R Dt+1 are the weights for this fully connected neural network, and D t is the number of entity types. Actually, this classification function classifies entity candidates into (D t + 1) types since we add one type for the entity candidates which are not real entities. Finally, the hinge-ranking loss is adopted in the Classification Network:

W c1 ∈ R (2D ml +2D el )×D h , b c1 ∈ R D h , W c2 ∈ R Dc1×(

EQUATION

where p w is the probability for the wrong labels y w , p r is the probability for the right label y r , and ∆ is a margin. The hinge-rank loss urges the probability for the right label higher than the probability for the wrong labels and improves the classification performance.

3. EXPERIMENTS 3.1 DATASETS:

To evaluate the performance of the proposed MGEPN model for NER, we conduct the experiments on two different datasets: the CoNLL-2003 dataset (Tjong Kim Sang & De Meulder, 2003) and the OntoNotes 5.0 dataset (Hovy et al., 2006; Pradhan et al., 2013) . Specifically, there are four types of named entities in the CoNLL-2003 dataset like including locations, persons, and organizations. The OntoNotes 5.0 dataset is larger and more challenging compared with the CoNLL-2003 dataset. Essentially, it contains 18 types of entities, such as persons, time, money and languages. For the Multi-grained task, we generate two datasets which contained multi-grained entities, named MG CoNLL-2003 and MG OntoNotes 5.0, based on the CoNLL-2003 dataset and the OntoNotes 5.0 dataset, respectively. We label coarse-entities that are contained in the fine-grained entities and add them to the multi-grained datasets. An overview of these four datasets are illustrated in Table 1 . , 2014) with learning rate decay for all the experiments. The learning rate is set as 0.001 at the beginning and exponential decayed by multiply 0.9 after each epoch. The batch size is set to 20 for CoNLL-2003 and 10 for Ontonotes 5.0, respectively. To alleviate over-fitting, we add dropout regularizations after all LSTM layers in our proposed MGEPN model with a dropout rate 0.5. In addition, we employ the early stopping strategy when there is no performance improvement on the development dataset after three epochs. The pre-trained word embeddings are from GloVe (Pennington et al., 2014) , and the word embedding dimension D w is 300. Besides, the ELMo 5.5B data1 is utilized in the experiment for the language model embedding. Moreover, the character embedding size is 100, and the hidden size of the Character LSTM D cl is also 100. The hidden size of the Entity LSTM D el and the Attention LSTM D ml are 300, respectively. We set the maximum length of entity proposals R to 5. 

3.3. RESULTS:

Multi-grained NER Task. The advantage of the proposed MGEPN is to detect multi-grained named entities. In order to validate this advantage, we compare MGEPN with a classical sequence labeling model LSTM-LSTM-CRF for the Multi-grained NER task. The LSTM-LSTM-CRF baseline combines LSTM with CRF. It uses one bidirectional LSTM to capture character information and another bidirectional LSTM to capture sentence information.

Experiment results of the Multi-grained NER task on the MG CoNLL-2003 dataset and the MG OntoNotes 5.0 dataset are reported in Table ??. We can observe from Table ?? that, our proposed model MGEPN achieves significant improvements compared with the baseline approach. For both two datasets, our model outperforms the basic LSTM-LSTM-CRF model for more than 10% in terms of precision, recall, as well as the F1 score. For sentences with over-lapping entities, we generate a sequence label for each annotation and feed them to sequence labeling based methods during both training and testing.

The first step in MGEPN is to propose entity candidates in the Proposal Network, where the effectiveness of proposing correct entity candidates immediately affects the performance of the whole model. To this end, we provide the experiment results of proposing multi-grained named entities 4 that the proposed MGEPN model outperforms all the baselines on the CoNLL-2003 dataset and achieves comparable performance on the OntoNotoes 5.0 dataset. To study the contribution of different modules in MGEPN, we also report the performance of two reduced variations of the proposed MGEPN at the bottom of Table 4 . MGEPN w/o attention is a variation of MGEPN which reducing the Entity-Context attention mechanism, and MGEPN w/o context is another variation which reduces all the context information. By purely adding the context information, the F1 score on the CoNLL-2003 test set improves from 92.23 to 92.26, and by adding the attention mechanism, the F1 score improves to 92.28. Also, we show the performance for proposing entity candidates in the NER task in Table 5 . 

3.4. CASE STUDY:

To demonstrate the benefit of the proposed MGEPN, we conduct the following case study. We first select three utterances that contain multi-grained entities from the MG OntoNotes 5.0 dataset, then run the proposed MGEPN, and finally show the results in Figure 4 . In Figure 4 , fine-grained entities are marked with red boxes like "Academia Sinica 's institute of Biomedical Science", and coarse-grained entities are highlighted in yellow like "the Financial Times". The baseline model LSTM-LSTM-CRF only recognizes one coarse-grained entity at the selected area, but the proposed MGEPN model can successfully detect both entities in each example. 

4. RELATED WORK:

Existing approaches for recognizing named entities usually cast the NER task to the problem of sequence labeling, and thus several sequence labeling models are proposed, including linear statistical models such as Conditional Random Fields (CRF) (Ratinov & Roth, 2009) , and deep neural networks like recurrent neural networks or convolutional neural networks (CNN). (Hammerton, 2003) is the first work to use LSTM for NER. Collobert et al. (2011) 2016) employ a character LSTM instead. The attention mechanism is also introduced to NER to dynamically decide how much information to use from a word or character level component (Rei et al., 2016) .

External resources have been used to further improve the NER performance including Peters et al. ( 2017) and semi-supervised learning. (Peters et al., 2017) adds pre-trained context embeddings from bidirectional language models to NER. ELMo (Peters et al., 2018) learns a linear combination of internal hidden states stacked in a deep bidirectional language model to utilize both higher-level states which capture context-dependent aspects and lower-level state which model aspects of syntax.

Other directions for NER related work are transfer learning and multi-task learning. Transfer learning models for NER transfer information from a source task with plentiful annotations to improve the performance on a target task (Luan et al., 2018; Yang et al., 2017; Lee et al., 2017a; Zirikly & Hagiwara, 2015) . These models can be transferred between different datasets or even different languages. Multi-task learning models jointly learn the NER task with other tasks like intent prediction (Goo et al., 2018) to obtain better semantic frame results by the global optimization, or fine-grained named entity categorization (Aguilar et al., 2017) for more generalized feature representations.

Several works (Xu et al., 2017; Katiyar & Cardie, 2018; Ju et al., 2018; Wang & Lu, 2018) have been proposed to address the task of nested NER where entities may have overlaps. To tackle this issue, we propose MGEPN, which can accurately detect both fine-grained and coarse-grained entities by first proposing entity candidates with the designed Proposal Network and then classifying entities using the proposed Classification Network.

5. CONCLUSIONS:

In this work, we propose a novel Named Entity Recognition approach, called Multi-grained Entity Proposal Network (MGEPN), which consists of two modules: the Proposal Network and the Classification Network. The Proposal Network aims to output all correct entity candidates. Among these candidates, both fine-grained and coarse-grained entities are proposed, which enables the proposed MGEPN to identify multi-grained entities. For better understanding the entities, context information learned in the Proposal Network is then transferred to the Classification Network. In the Classification Network, We develop a novel Entity-Context attention mechanism to help the model focus on entity-related context information. Experiments on two real-world datasets show the effectiveness of the proposed approach for both the NER task and the Multi-grained NER task.

:

https://allennlp.org/elmo

Figures and Tables:

concatenated into the hidden states h k . The dimension of the hidden states of the word LSTM is set as D wl .

Figure 3: All possible entity proposals generated surrounding token t 3 when the maximum length of an entity proposal R is set as 5.

The hidden dimension of the classification layer D h is 50. The margin ∆ in the hinge-ranking loss for the entity category classification is set to 5. The hidden dimension of the Word LSTM layer D wl is 300 for CoNLL-2003 and 1024 for OntoNotes 5.0. The hidden dimension of the sentence LSTM layer D sl is 300 for CoNLL-2003 and 128 for OntoNotes 5.0. The ELMo scale parameter γ used in the Proposal Network is 3.35 for CoNLL-2003 and 3.05 for OntoNotes 5.0. The ELMo scale parameter γ used in the Classification Network is 3.05 for CoNLL-2003 and 2.95 for OntoNotes 5.0, respectively.

Dr. Konan Peck , an assistant research fellow at Academia Sinica 's Institute of Biomedical Sciences … …. insurance services for the overseas business of the Gansu Province Thermoelectric Company .James Drummond , correspondent for the Financial Times of London speaking to us from Cairo .

Figure 4: Examples from the Multi-grained OntoNotes 5.0 dataset. Fine fine-grained entities are marked with red boxes and coarse-grained entities are highlighted in yellow.

employ a CNN-CRF structure, which obtains competitive results to statistical models. Santos & Guimaraes (2015) introduce a character CNN to augment the CNN-CRF model. Most recent work leverages an LSTM-CRF architecture. Huang et al. (2015) use hand-crafted spelling features; Ma & Hovy (2016) and Chiu & Nichols (2016) utilize a character CNN to represent spelling characteristics; Lample et al. (

We propose a novel deep neural network, the Multi-grained Entity Proposal Network (MGEPN),

Table content: <table><tr><td colspan="2">Fully Connected</td></tr><tr><td colspan="3">Entity Representation</td></tr><tr><td colspan="2">Entity LSTM</td></tr><tr><td colspan="2">Hidden States</td><td>ELMo</td></tr><tr><td colspan="2">Word LSTM</td></tr><tr><td>Word level</td><td>Char level</td></tr><tr><td colspan="3">Character LSTM</td></tr><tr><td colspan="3">Character Embedding</td></tr></table>



Table content: <table><tr><td/><td/><td>Category Probabilities</td></tr><tr><td colspan="2">Fully Connected</td><td>Attentive Context</td></tr><tr><td/><td/><td>Entity-Context Attention</td></tr><tr><td colspan="2">Sentence Features</td><td>Context Features</td></tr><tr><td colspan="2">Sentence LSTM</td></tr><tr><td>Hidden States</td><td>ELMo</td></tr></table>



Table content: <table><tr><td colspan="2">Word LSTM</td></tr><tr><td>Word level</td><td>Char level</td></tr><tr><td colspan="2">Character LSTM</td></tr><tr><td colspan="2">Character Embedding</td></tr></table>

Sentence-level semantic features are first trained in the Proposal Network and then transferred into the Classification Network to introduce and utilize the context information. A novel attention mechanism is also involved in the proposed MGEPN model with a special focus on the Entity-Context relationship. It helps the model capture and utilize entityrelated context information. It is worthy to mention that in order to improve the learning speed as well as the model performance of MGEPN, the Proposal Network and the Classification Network are trained with a series of shared input features, including the pre-trained word embeddings and the pre-trained language model features.We provide the key building blocks and the properties of the , ..., t K ), each token t k (1 ≤ k ≤ K) is represented as x k using a concatenation of a word-level embedding w k and a character-level word information c k :

Table content: <table><tr><td>Proposal Network in Section 3.1 and the Classification Network in Section 3.2, respectively.</td></tr><tr><td>2.1 THE PROPOSAL NETWORK</td></tr><tr><td>The Proposal Network aims at generating possible entity proposals for each utterance. It takes an ut-</td></tr><tr><td>terance as input and outputs a set of entity candidates. Essentially, we use a semi-supervised neural</td></tr><tr><td>network inspired by Peters et al. (2017) to model this process. The architecture of the Proposal Net-</td></tr><tr><td>work is illustrated in the left part of Figure 2. Three major components are contained in the Proposal</td></tr><tr><td>Network: Word Representation, Sentence Representation and Entity Proposal. More specifically,</td></tr><tr><td>pre-trained word embeddings and character-level word information are used for generating semantic</td></tr><tr><td>meaningful word representations. Word representations and the language model embeddings, ELMo</td></tr><tr><td>(Peters et al., 2018), are concatenated together to produce context aware sentence representations.</td></tr><tr><td>For the entity proposal task, possible named entities are proposed, and sentence representations are</td></tr><tr><td>fed into a fully connected layer to output the probability of a proposal being a real entity.</td></tr><tr><td>2.1.1 WORD REPRESENTATION</td></tr><tr><td>Given an input utterance with K tokens (t 1</td></tr><tr><td>Proposal Probabilities</td></tr></table>

Dataset statistics. CoNLL-2003 and OntoNotes 5.0 datasets are used for the NER tasks. MG CoNLL-2003 and MG OntoNotes 5.0 datasets are used for the Multi-grained NER tasks.

Table content: <table><tr><td>Dataset</td><td colspan="4">CoNLL-2003 MG CoNLL-2003 OntoNotes 5.0 MG OntoNotes 5.0</td></tr><tr><td># Training Entities</td><td>23,499</td><td>25,006</td><td>81,828</td><td>128,738</td></tr><tr><td># Validation Entities</td><td>5,942</td><td>6,272</td><td>11,066</td><td>20833</td></tr><tr><td># Test Entities</td><td>5,648</td><td>5,986</td><td>11,257</td><td>12971</td></tr><tr><td>Word Vocab Size</td><td>22,216</td><td/><td>50,394</td><td/></tr><tr><td>Character Vocab Size</td><td>83</td><td/><td>117</td><td/></tr><tr><td>Entity Types</td><td>4</td><td/><td>18</td><td/></tr><tr><td colspan="2">3.2 IMPLEMENTATION DETAILS</td><td/><td/><td/></tr><tr><td colspan="3">We employ the Adam optimizer (Kingma &amp; Ba</td><td/><td/></tr></table>

Multi-grained NER performance on test set for MG CoNLL2003, MG OntoNotes 5.

Table content: <table><tr><td>0 and</td></tr></table>

As shown inTable 3, on the MG CoNLL-2003 dataset, MGEPN obtains pretty high F1 score for proposing multi-grained named entities on both the development set and the test set.

Table content: <table/>

Performance for the Proposal Network in the Multi-grained NER task.

Table content: <table><tr><td>Dataset</td><td>Pre.</td><td>Dev Rec.</td><td>F1</td><td>Pre.</td><td>Test Rec.</td><td>F1</td></tr><tr><td colspan="7">MG CoNLL-2003 95.66 96.60 96.13 93.75 96.04 94.88</td></tr><tr><td colspan="7">MG OntoNotes 5.0 81.04 90.10 85.33 79.85 88.37 83.89</td></tr><tr><td>ACE2005</td><td colspan="6">87.03 90.30 88.64 84.95 89.35 87.09</td></tr><tr><td colspan="7">Traditional NER Task. We also evaluate the proposed MGEPN model on traditional NER tasks</td></tr><tr><td colspan="7">comparing with a series of classical NER models to show the effectiveness of MGEPN. Specifically,</td></tr><tr><td colspan="7">those baselines include: 1) Lample et al. (2016), which adopts the LSTM-CRF structure; 2) Ma</td></tr><tr><td colspan="7">&amp; Hovy (2016), which uses a LSTM-CNNs-CRF architecture; 3) Chiu &amp; Nichols (2016), which</td></tr><tr><td colspan="7">proposes a CNN-LSTM-CRF model; 4) Peters et al. (2017), which adds semi-supervised language</td></tr><tr><td colspan="7">model embeddings; and 5) Peters et al. (2018), which utilizes the state-of-the-art ELMo language</td></tr><tr><td>model embeddings.</td><td/><td/><td/><td/><td/><td/></tr></table>

F1 scores for the English NER task on the CoNLL-2003 dataset and the OntoNotes 5.0 dataset. Mean and standard deviation across five runs are reported.Table4shows the F1 scores of different approaches for NER on two experimental datasets. It can be observed from Table

Table content: <table><tr><td>Model</td><td colspan="2">CoNLL-2003 Dev Test</td><td colspan="2">OntoNotes 5.0 Dev Test</td></tr><tr><td>Xu et al. (2017)</td><td>-</td><td>90.85</td><td>-</td><td>-</td></tr><tr><td>Lample et al. (2016)</td><td>-</td><td>90.94</td><td>-</td><td>-</td></tr><tr><td>Ma &amp; Hovy (2016)</td><td>94.74</td><td>91.21</td><td>-</td><td>-</td></tr><tr><td colspan="5">Chiu &amp; Nichols (2016) 94.03 ± 0.23 91.62 ± 0.33 84.57 ± 0.27 86.17 ± 0.22</td></tr><tr><td>Peters et al. (2017)</td><td>-</td><td>91.93 ± 0.19</td><td>-</td><td>-</td></tr><tr><td>Peters et al. (2018)</td><td>-</td><td>92.22 ± 0.10</td><td>-</td><td>-</td></tr><tr><td>MGEPN</td><td colspan="4">95.24 ± 0.13 92.28 ± 0.12 85.24 ± 0.20 82.37 ± 0.17</td></tr><tr><td>MGEPN w/o context</td><td colspan="4">95.21 ± 0.12 92.23 ± 0.06 85.06 ± 0.19 82.11 ± 0.24</td></tr><tr><td colspan="5">MGEPN w/o attention 95.23 ± 0.06 92.26 ± 0.09 85.13 ± 0.24 82.20 ± 0.25</td></tr></table>

Performance for the Proposal Network in the traditional NER task.

Table content: <table><tr><td>Dataset</td><td>Pre.</td><td>Dev Rec.</td><td>F1</td><td>Pre.</td><td>Test Rec.</td><td>F1</td></tr><tr><td colspan="7">CoNLL-2003 96.74 96.41 96.57 95.33 95.69 95.51</td></tr><tr><td colspan="7">OntoNotes 5.0 85.69 90.60 88.08 82.26 89.43 85.70</td></tr></table>

Peer review (iclr2019):
 This paper proposed a entity proposal network for named entity recognition which can be effectively detect overlapped spans. The model obtain good performance on both Multi-grained NER task and traditional NER task. The paper is in general well written, the idea of proposal network break the traditional framework of sequence tagging formulation in NER task and thus can be effectively applied to detect overlapped named entities. However, I still have many concerns regarding the notation, the novelty of the paper, and the comparison with related literature, especially on previous overlapped span detection NER papers. The detailed concerns and questions are as follows: The notations are very confusing. Many of the notations are not defined. For example, what does in below Eq. 4 indicates? What does scores means? I guess does not equal to number of entity types, but I’m not sure what exactly indicates. If is not number of entity types, why do you need R scores for being an entity and R scores for not being an entity? And what is in Eq 5? Is that entity type id or something else? I’m still confused how you select the entity spans from a large number of entity candidates. In Figure 5, if the max window length is 5, there may be more span candidates than the listed 5 examples, such as t_3 t_4 t_5. How do you prune it out? Table 5 is weird. There is not comparison with any baselines but just a report of the performance with this system. I don’t know what point this table is showing. This is not the first paper that enumerates all possible spans for NER task.The idea of enumerating possible spans for NER task has appeared in [1] and can also effectively detect overlapped span. I would like to see the performance comparison between the two systems. The enumerating span ideas has been applied in many other tasks as well such as coreference resolution [2]and SRL[3], none of which is mentioned in related work. I feel that most of the gain is from ELMo but not the model architecture itself, since in Table 4, the improvement from the ELMo is only 0.06. The LSTM-LSTM-CRF is without adding ELMo, which is not a fair comparison. The comparison of baselines is not adequate and is far from enough. The paper only compares with LSTM+CRF frameworks, which are not designed for detecting overlapped spans. There are many papers on detecting overlapping spans, such as [4], [5] and [6]. It’s important to compare with those paper since those methods are especially designed for overlapped span NER tasks. [1] Multi-Task Identification of Entities, Relations, and Coreferencefor Scientific Knowledge Graph Construction, EMNLP 2018 [2] End-to-end neural coreference resolution, EMNLP 2017 [3] Jointly predicting predicates and arguments in neural semantic role labeling, ACL 2018 [4] Nested Named Entity Recognition Revisited, NAACL 2018 [5] A Neural Layered Model for Nested Named Entity Recognition, NAACL 2018 [6] Neural Segmental Hypergraphs for Overlapping Mention Recognition, EMNLP 2018