Title: UNDERSTANDING AND IMPROVING SEQUENCE-LABELING NER WITH SELF-ATTENTIVE LSTMS

Abstract: This paper improves upon the line of research that formulates named entity recognition (NER) as a sequence-labeling problem. We use so-called black-box long short-term memory (LSTM) encoders to achieve state-ofthe-art results while providing insightful understanding of what the autoregressive model learns with a parallel self-attention mechanism. Specifically, we decouple the sequence-labeling problem of NER into entity chunking, e.g., Barack B Obama E was O elected O , and entity typing, e.g., Barack PERSON Obama PERSON was NONE elected NONE , and analyze how the model learns to, or has difficulties in, capturing text patterns for each of the subtasks. The insights we gain then lead us to explore a more sophisticated deep cross-Bi-LSTM encoder, which proves better at capturing global interactions given both empirical results and a theoretical justification.

1. INTRODUCTION:

Named entity recognition is an important task in information extraction in which we seek to locate entity chunks in text and classify their entity types. Originally a structured prediction task, NER has since been formulated as a task of sequential token labeling, much like text chunking and part-ofspeech tagging. With the ability to compute representations of past and future context respectively for each token, bidirectional LSTM (Bi-LSTM) has proved a robust building block for sequencelabeling NER (Huang et al., 2015; Ma & Hovy, 2016; Chiu & Nichols, 2016 ). However, it has been predominantly used as a black box; research directed to understanding how the model learns to tackle the task is minimal.

In this work, we decouple sequence-labeling NER into the entity chunking and entity typing subtasks, and seek insight into what patterns LSTM learns to capture or has difficulties capturing. We propose the use of a fast and effective parallel self-attention mechanism alongside Bi-LSTM. Unlike traditional attention mechanisms used for tasks such as machine translation (Luong et al., 2015) and sentence classification (Conneau et al., 2017; Lin et al., 2017) , our self-attentive Bi-LSTM uses the hidden state of each token as its own query vector and computes context vectors for all tokens in parallel. For both subtasks, we then find important global patterns that cross past and future context, and in particular discover the way multi-chunk entities are handled. Furthermore, we discover that the theoretical limitations of traditional Bi-LSTMs harms performance on the task, and hence propose using a cross construction of deep Bi-LSTMs. As a result, with these cross structures, both selfattentive Bi-LSTM and cross-Bi-LSTM achieve new state-of-the-art results on sequence-labeling NER.

In Section 3, the normal Bi-LSTM-CNN model is formulated. Section 4 details the computation of the parallel self-attention mechanism. Section 5 presents the empirical results and detailed analyses of the models, with a particular focus on patterns captured for {B, I, E} labels. Finally in Section 6, cross-Bi-LSTM-CNN is formulated and evaluated on a theoretical basis. Our contribution is threefold:

• We provide insightful understanding of how a sequence-labeling model tackles NER and the difficulties it faces;

• We propose a fast and effective self-attention mechanism for sequence labeling;

• We propose using cross-Bi-LSTM-CNN for sequence-labeling NER with theoreticallygrounded improvements.

2. RELATED WORK:

Many have attempted tackling the NER task with LSTM-based sequence encoders (Huang et al., 2015; Ma & Hovy, 2016; Chiu & Nichols, 2016; Lample et al., 2016) . Among these, the most similar to the proposed Bi-LSTM-CNN is the model proposed by Chiu & Nichols (2016) . In contrast to previous work, Chiu & Nichols (2016) stack multiple layers of LSTM cells per direction, and also use a CNN to compute character-level word vectors alongside pre-trained word vectors. We largely follow their work in constructing the Bi-LSTM-CNN, including the selection of raw features, the CNN, and the multi-layer Bi-LSTM. The subtle difference is that they send the output of each direction through separate affine-softmax classifiers and then sum their probabilities, effectively forming an ensemble of forward and backward LSTM-CNNs. Another difference is that they focus on proposing a new representation of external lexicon features, which we do not make use of in this work.

The modeling of global context for sequential-labeling NER has been accomplished using traditional models with intensive feature engineering and conditional random fields (CRF). Ratinov & Roth (2009) build the Illinois NER tagger with feature-based perceptrons. In their analysis, the usefulness of Viterbi decoding is minimal, as class transition patterns only occur in small chunks and greedy decoding can handle them comparatively well. On the other hand, recent research on LSTM or CNNbased encoders report empirical improvements brought by CRF (Huang et al., 2015; Ma & Hovy, 2016; Lample et al., 2016; Strubell et al., 2017) , as it discourages illegal predictions by explicitly modeling class transition probabilities. In contrast, the cross structures of self-attention and cross-Bi-LSTM studied in this work provide for the direct capture of global patterns and extraction of better features to improve class observation likelihoods.

Various attention mechanisms have been proposed and shown success in natural language tasks. They lighten the LSTM's burden of compressing all relevant information into a single hidden state by consulting past memory. For seq2seq models, attention has been used for current decoder hidden states (Luong et al., 2015) . For models computing sentence representations, trainable weights are used for self-attention (Conneau et al., 2017; Lin et al., 2017) . In this work, we propose using a token-level parallel self-attention mechanism for sequential token-labeling and show that it enables the model to capture cross interactions between past and future contexts.

3 BI-LSTM-CNN FOR SEQUENCE LABELING

3.1. CNN AND WORD FEATURES:

All models in our experiments use the same set of raw features: word embedding, word capitalization pattern type, character embedding, and character type.

For character embedding, 25d vectors are randomly initialized and trained end-to-end with the model. Appended to these are 4d one-hot character-type features indicating whether a character is uppercase, lowercase, digit, or punctuation (Chiu & Nichols, 2016) . In addition, an unknown character vector and a padding character vector are also trained. We unify the word token length to 20 by truncation and padding. The resulting 20-by-(25+4) feature map of each token are applied to a character-trigram CNN with 20 kernels per length 1 to 3 and max-over-time pooling to compute a 60d character-based word vector (Kim et al., 2016; Chiu & Nichols, 2016; Ma & Hovy, 2016) .

For word embedding, pre-trained 300d GloVe word vectors (Pennington et al., 2014) are used without further tuning. In addition, 4d one-hot word capitalization features indicate whether a word is uppercase, upper-initial, lowercase, or mixed-caps (Collobert et al., 2011; Chiu & Nichols, 2016) .

Throughout this paper, we use X to denote the n-by-d x matrix of raw sequence features, with n denoting the number of word tokens in a sentence and d x = 60 + 300 + 4.

3.2. DEEP BI-LSTM:

Given a sequence of input feature vectors x 1 , x 2 , . . . , x T ∈ R d1 , an LSTM cell computes a sequence of hidden feature vectors h 1 , h 2 , . . . , h T ∈ R d2 by

g t = tanh(W g x t + V g h t-1 + b g ) i t = σ(W i x t + V i h t-1 + b i ) f t = σ(W f x t + V f h t-1 + b f ) o t = σ(W o x t + V o h t-1 + b o ) c t = g t i t + c t-1 f t h t = tanh(c t ) o t , where h 0 , c 0 ∈ R d2 are zero vectors, W g , W i , W f , W o ∈ R d2×d1 , V g , V i , V f , V o ∈ R d2×d2 , b g , b i , b f , b o ∈ R d2

are trainable weight matrices and biases, tanh denotes hyperbolic tangent, σ denotes sigmoid function, and denotes element-wise multiplication.

Bidirectional LSTMs (Bi-LSTMs) are used to capture the future and the past for each time step. Following Chiu & Nichols (2016) , 4 distinct LSTM cells -two in each direction -are stacked to capture higher level representations:

- → H = ----→ LST M 2 ( ----→ LST M 1 (X)) ← - H = ←---- LST M 4 ( ←---- LST M 3 (X)) H = - → H || ← - H , where ----→ LST M i , ←---- LST M i denote applying LSTM cell i in forward, backward order, - → H , ← -

H denote the resulting feature matrices of the stacked application, and || denotes row-wise concatenation. In all our experiments, 100d LSTM cells are used, so H ∈ R n×d h and d h = 200.

3.3. AFFINE-SOFTMAX AND CHUNK LABELS:

Finally, suppose there are d p token classes, the probability of each of which is given by the composition of affine and softmax transformations:

s t = H t W p + b t p ti = e sti dp j=1 e stj

, where H t is the t th row of H, W p ∈ R d h ×dp , b ∈ R dp are a trainable weight matrix and bias, and s ti and s tj are the i-th and j-th elements of s t .

Following Chiu & Nichols (2016) , we use the 5 chunk labels O, S, B, I, and E to denote if a word token is {O}utside any entities, the {S}ole token of an entity, the {B}eginning token of a multitoken entity, {I}n the middle of a multi-token entity, or the {E}nding token of a multi-token entity.

Hence when there are P types of named entities, the actual number of token classes d p = P × 4 + 1 for sequence labeling NER.

4. SELF-ATTENTIVE BI-LSTM-CNN FOR SEQUENCE LABELING:



4.1. PARALLEL MULTI-HEAD SELF-ATTENTION:

We propose using a token-level self-attention mechanism (Figure 1 ) that is computed after the autoregressive Bi-LSTM in Section 3.2. This has two benefits over traditional auto-regressive attention, which wraps stacked LSTM cells to look at past tokens at each time step for each direction of Bi-LSTM. First, it allows each token to look at both past and future sequences simultaneously with one combined hidden state of past and future, thus capturing cross interactions between the two contexts. And secondly, since all time steps run in parallel with matrix computations, it introduces little computation time overhead. Specifically, given the hidden features H of a whole sequence, we project each hidden state to different subspaces, depending on whether it is used as the {q}uery vector to consult other hidden states for each word token, the {k}ey vector to compute its dot-similarities with incoming queries, or the {v}alue vector to be weighted and actually convey information to the querying token. Moreover, as different aspects of a task can call for different attention, multiple "attentions" running in parallel are used, i.e., multi-head attention (Vaswani et al., 2017) .

Formally, let m be the number of attention heads and d c be the subspace dimension. For each head i ∈ {1..m}, the attention weight matrix and context matrix are computed by

α i = σ HW qi (HW ki ) T √ d c C i = α i HW vi ,

where W qi , W ki , W vi ∈ R d h ×dc are trainable projection matrices and σ performs softmax along the second dimension. Each row of the resulting α 1 , α 2 , . . . , α m ∈ R n×n contains the attention weights of a token to its context, and each row of

C 1 , C 2 , . . . , C m ∈ R n×dc is its context vector. Since H = - → H || ← -

H , the computation of α i and C i models the cross interaction between past and future.

4.2. AFFINE-SOFTMAX WITH MULTI-HEAD CONTEXT:

Finally, for Bi-LSTM-CNN augmented with the attention mechanism, the hidden vector and context vectors of each token are considered together for classification:

s c t = (H t ||C 1 t ||C 2 t ||...||C m t )W c + b t p c ti = e s c ti dp j=1 e s c tj ,

where || denotes row-wise concatenation, C i t is the t-th row of C i , and W c ∈ R (d h +mdc)×dp are trainable weight matrices. In all our experiments, we use m = 5 and We conduct experiments on the challenging OntoNotes 5.0 English NER corpus (Hovy et al., 2006; Pradhan et al., 2013) . OntoNotes is an ambitious project that collects large corpora from diverse sources and provides multi-layer annotations for joint research on constituency parsing, semantic role labeling, coreference resolution, and NER. The data sources include newswires, web, broadcast news, broadcast conversations, magazines, and telephone conversations. Some are transcriptions of talk shows and some are translated from Chinese or Arabic. Such diversity and noisiness requires that models are robust and able to capture a multitude of linguistic patterns.

d c = d h 5 , so W c ∈ R 2d h ×dp .

Table 1 summarizes the dataset statistics. Following previous lines of research, we use the standard split provided by Pradhan et al. (2013) , excluding the New Testament corpus as it contains no entity annotations. Despite this million-token corpus with over 100K annotated entities, previous work has struggled to reach state-of-the-art NER results on the dataset. This is due partly to the fact that there are 18 types of entities to be classified. Eleven of these are classes of general names, with NORP including nationalities such as American, FAC including facilities such as The White House, and WORK OF ART including titles of books, songs, and so on. Moreover, various forms of values of the seven numerical classes must also be identified.

5.2. IMPLEMENTATION DETAILS:

The hyperparameters of our models were given in Sections 3 and 4. When training the models, we minimized per-token cross-entropy loss with the Nadam optimizer (Dozat, 2016) . In addition, we randomly dropped 35% hidden features (dropout) and upscaled the same amount during training. Following previous lines of work, we evaluated NER performance with the per-entity F1 score. The tokens for an entity were all to be classified correctly to count as a correct prediction; otherwise it was counted as either a false positive prediction or a false negative non-prediction. We stopped training when the validation F1 had not improved for 20 epochs. All models were initialized and trained 5 times; we report the mean precision, recall, and F1 scores (%) of the experiments. Validation scores are also reported for future research on this task.

5.3. OVERALL RESULTS:

Table 2 shows the overall results of our models against notable previous work. It can be seen that simple LSTM-based sequence encoders already beat the previous best results without using external lexicons (Chiu & Nichols, 2016) , document-level context (Strubell et al., 2017) , or constituency parsers (Li et al., 2017) . Furthermore, with the proposed parallel self-attention mechanism (ATT), we achieve a new state-of-the-art result (88.29 F1) with a clear margin over past systems. More importantly, the attention mechanism allows us to conduct insightful analyses in the following sections, yielding important understanding of how Bi-LSTM learns or has difficulty tackling the different sequence-labeling NER subtasks: entity chunking and entity typing. 

5.4. ENTITY CHUNKING:

We decouple the entity chunking task from sequence-labeling NER. Specifically, for a sentence such as {Barack Obama moves out of the White House .}, the task is to correctly label each token as

{Barack B Obama E moves O out O of O the B White I House E . O }.

5.4.1. QUANTITATIVE ANALYSIS:

Table 3 shows the performance of different setups on validation data. We take the pre-trained models from Table 2 Looking at the three columns on the left, the first thing we discover is that Bi-LSTM-CNN+ATT designates the task of predicting {I} to the attention mechanism. The model performance on tokens {I}n the middle of an entity significantly degrades (-28.18) in the absence of global context C all , when token hidden state H is left alone. On the other hand, without the information on the token itself, it is clear that the model strongly favors predicting I (-3.80) given its global context C all .

Taking this one step further and zeroing out all other vectors except for each attention head, the roles of context for entity chunking become even clearer. C 2 and C 3 send strong signals (-36.45,-39.19 ) on entity chunk {E}nding to the model, plus weak signals (-60.56,-50.19 ) on entity chunk {I}nside, while C 4 sends a strong signal (-12.21) on entity chunk {B}eginning plus weak signals (-57.19 ) on {I}nside. When all these heads fire simultaneously, the model produces a strong signal to {I}.

However, NativeH -Bi-LSTM-CNN trained without attention -underperforms in chunk labels {B} (-0.63), {I} (-0.41), {E} (-0.38) in comparison to HC all , the model trained with ATT. This suggests that entity chunking is indeed a crucial aspect in sequence-labeling NER, and that it is difficult for pure LSTM encoders to compress all necessary information in each hidden state to correctly label all the tokens of a multi-token entity.

5.4.2. QUALITATIVE ANALYSIS:

Aside from knowing that entity chunking is a crucial, challenging aspect in sequence-labeling NER for Bi-LSTM, one remaining question is how exactly the encoder is attempting to properly classify the {B}egin, {I}nside, and {E}nd of a multi-token entity. To shed light on this question, we visualize samples from validation data and discover consistent patterns in the attention weight heat maps across sentences and entities.

Figure 2 shows one of the samples, where the attention weights α 2 , α 3 , α 4 of a sentence containing the B White I house E are visualized. The full Bi-LSTM-CNN+ATT (HC all ) classifies the tokens correctly, but when in the absence of the context vectors (H), the predictions become the B White S house E . For Bi-LSTM-CNN trained without attention at all (NativeH ), the predictions are the O White S house O . Each row of the matrix shows the attention weight distribution for the diagonal token in bold font.

We observe that α 2 and especially α 3 have a tendency to focus on the previous tokens: the diagonal shifted left. In contrast, α 4 tends to look at the immediate following tokens: the diagonal shifted right. By looking for previous tokens that belong to the same entity chunk and finding some, an attention head, via its context vector, can signal to the model that the token spoken of might be the {E}nding token or {I}nside token. The same is true for an attention head looking at next tokens, but this time signaling for {B}egin and {I}nside. This also dictates that both signals need to be weaker for {I} but stronger when combined. This behavior can be observed throughout the heat maps of α 2 , α 3 , α 4 . In particular for the White house, C all predicts the B White I house O as Saturday is wrongly focused by α 4 for house. .09 From Table 3 , we already know that NativeH has some difficulties in handling multi-token entities, being more inclined to predict {S}ingle-token entities, and that HC all mitigates this problem by delegating work to C all , especially by relying on the latter to signal for {I}n tokens. The heat maps further tell the story of how the related labels {B, I, E} are handled collectively. In addition, this also suggests that modeling interactions between future and past contexts is crucial for sequencelabeling NER and motivates the use of a deep cross-Bi-LSTM encoder in Section 6.

HC all H C all C 1 C 2 C 3 C 4 C 5 N ativeH

5.5. ENTITY TYPING:

When the entity chunking task is decoupled from sequence-labeling NER, the remaining en- With attention, in Figure 3 , we see that α 1 attends to Dutch and English at the same time for the two tokens and attends to Chinese and English at the same time for the other two. On the other hand, α 5 focuses on all possible LAN tokens, including a small mis-attention to Taiwanese in the second sentence, which is actually a NORP in this case. These attention weights signify that the model learns a pattern of cross interaction between entities: when two ambiguous entities of NORP , LAN occur together in the same context, the model predicts both as LAN .

6 CROSS STRUCTURES FOR SEQUENCE LABELING 6.1 THEORETICAL LIMITATION OF BI-LSTM In Section 4.1, we briefly mentioned that the computation of attention weights α i and context features C i models the cross interaction between past and future. Mathematically, since H = -→ H || ← -H , the computation of attention scores can be rewritten as

HW qi (HW ki ) T = ( - → H || ← - H )(W qi W ki T )( - → H || ← - H ) T .

The un-shifted covariance matrix of the projected ( -→ H || ← -H ) thus computes the interaction between past context and future context for each token, capturing cross-context patterns that the deep Bi-LSTM-CNN specified in Section 3 cannot. The consequence of this inability has been empirically shown in Section 5. Here, we further consider the following four simple phrases that form an XOR: {Key and Peele} WOA ; {You and I } WOA ; {Key and I }; {You and Peele} where WOA stands for WORK OF ART . The first two phrases are respectively a show title and a song title. The other two are not entities, where the last one actually occurs in an interview with Keegan-Michael Key. Suppose the phrases themselves are the only available context for the classification of and . Then the Bi-LSTM-CNN cannot capture good enough features to classify and correctly simultaneously for the four cases, even if they are the training data, no matter how many LSTM cells are stacked. The key is that given the same half-context of past or future, and is sometimes {WOA : I } but sometimes {NONE : O}. It is only when patterns that cross past and future are captured that the model is able to decide the correct label. Note that when computing sentence embeddings for tasks such as sentence classification, both directions of a normal Bi-LSTM look at the whole sentence. However, when computing hidden node features for sequence labeling, each direction of a normal Bi-LSTM looks at only half of the sentence. Cross-Bi-LSTM remedies this problem by interleaving the hidden features between LSTM layers. The output of the first layers of both directions are sent to the second layers of both directions, allowing higher layers to capture interactions between past and future contexts for each token. Empirically, we experiment with cross construction 5 times and find it further improves the performance of Bi-LSTM-CNN from 87.56 (±0.07) to 88.09 (±0.16).

7. CONCLUSION:

In this paper, we have decoupled named entity recognition into entity chunking and entity typing and demonstrated how sequence-labeling models can learn to handle each of these two subtasks. By using a fast parallel self-attention mechanism, we have discovered how the beginning and ending of a multi-token entity is determined and how they are jointly correlated to locate the inside tokens. Further, through our quantitative and qualitative analyses for both chunking and typing, we have shown that it is crucial to capture global patterns that cross both sides of a token. We demonstrate the theoretical limitation of the conventional deep Bi-LSTM-CNN used in sequence labeling tasks.

In addition to the interpretability of the proposed parallel self-attention, it is shown that it constitutes a way to correlate past and future contexts. We have also provided deep cross-Bi-LSTM-CNN as another way to extract global context features. With their respective cross structures, both selfattentive Bi-LSTM and cross-Bi-LSTM achieve new state-of-the-art results on sequence-labeling NER.

:



Figures and Tables:

Figure 1: Computation of i-th attention head of 4th word token

Figure 2: Attention heat maps for "...a meeting at the White house Saturday..."

Figure 3: Attention heat maps of "...Dutch into English..." and "...Chinese and English..."

CROSS-BI-LSTM-CNN FOR SEQUENCE LABELING Motivated by the limitation of the conventional Bi-LSTM-CNN for sequence labeling, we propose the use of Cross-Bi-LSTM-CNN by changing the deep structure in Section 3

Dataset Statistics

Table content: <table><tr><td>Split</td><td colspan="2">#Tokens #Entities</td><td>Entity type: names</td><td>Entity type: values</td></tr><tr><td colspan="2">Train 1,088,503</td><td colspan="2">81,828 PERSON, NORP, FAC,</td><td>DATE, TIME,</td></tr><tr><td>Validate</td><td>147,724</td><td>11,066</td><td>ORG, GPE, LOC,</td><td>PERCENT, MONEY,</td></tr><tr><td>Test</td><td>152,728</td><td>11,257</td><td>PRODUCT, EVENT,</td><td>QUANTITY,</td></tr><tr><td colspan="2">Total 1,388,955</td><td>104,151</td><td>WORK OF ART, LAW, LANGUAGE</td><td>ORDINAL, CARDINAL</td></tr></table>

Overall Results

Table content: <table><tr><td>Model</td><td colspan="2">Validate Prec. Recall</td><td>F1</td><td colspan="3">Test Prec. Recall</td><td>F1</td></tr><tr><td>Finkel &amp; Manning (2009)</td><td>-</td><td>-</td><td>-</td><td colspan="2">84.04 80.86</td><td/><td>82.42</td></tr><tr><td>Ratinov &amp; Roth (2009)</td><td>-</td><td>-</td><td>-</td><td colspan="2">82.00 84.95</td><td/><td>83.45</td></tr><tr><td>Passos et al. (2014)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td/><td>82.24</td></tr><tr><td>Durrett &amp; Klein (2014)</td><td>-</td><td>-</td><td>-</td><td colspan="2">85.22 82.89</td><td/><td>84.04</td></tr><tr><td>Chiu &amp; Nichols (2016)</td><td>-</td><td>-</td><td>-</td><td colspan="4">86.04 86.53 86.28 (±0.26)</td></tr><tr><td>Strubell et al. (2017)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td colspan="2">86.84 (±0.19)</td></tr><tr><td>Li et al. (2017)</td><td>85.5</td><td>84.7</td><td>85.08</td><td>88.0</td><td>86.5</td><td/><td>87.21</td></tr><tr><td>Bi-LSTM-CNN</td><td colspan="7">86.64 86.06 86.35 88.00 87.12 87.56 (±0.07)</td></tr><tr><td>Bi-LSTM-CNN +ATT</td><td colspan="7">87.22 86.69 86.95 88.79 87.81 88.29 (±0.20)</td></tr><tr><td colspan="2">5 EVALUATION AND ANALYSIS</td><td/><td/><td/><td/><td/></tr><tr><td>5.1 DATASET</td><td/><td/><td/><td/><td/><td/></tr></table>

Chunking Results

Table content: <table><tr><td/><td>HC all</td><td>H</td><td>C all</td><td>C 1</td><td>C 2</td><td>C 3</td><td>C 4</td><td>C 5 NativeH</td></tr><tr><td>O</td><td>99.05</td><td>-1.68</td><td>0.75</td><td>0.95</td><td colspan="2">-1.67 -45.57</td><td colspan="2">-0.81 -35.46</td><td>-0.03</td></tr><tr><td>S</td><td>93.74</td><td colspan="7">2.69 -91.02 -90.56 -90.88 -25.61 -86.25 -84.32</td><td>0.13</td></tr><tr><td>B</td><td>90.99</td><td colspan="7">1.21 -52.26 -90.78 -88.08 -90.88 -12.21 -87.45</td><td>-0.63</td></tr><tr><td>I</td><td colspan="2">90.09 -28.18</td><td colspan="6">-3.80 -87.93 -60.56 -50.19 -57.19 -79.63</td><td>-0.41</td></tr><tr><td>E</td><td>93.23</td><td colspan="7">2.00 -71.50 -93.12 -36.45 -39.19 -91.90 -90.83</td><td>-0.38</td></tr></table>

without re-training for this subtask. {O, S, B, I, E} are the chunk classes. The column of HC all lists the performance of the full Bi-LSTM-CNN+ATT model on each chunk class, where C all stands for C 1 , . . . , C 5 . Other columns list the performance of other setups compared to the full model. Columns H to C 5 are when the full model is deprived of all other information by zeroing all other vectors for the affine-softmax classification layer in testing time, except for those specified by the column header. NativeH is the native Bi-LSTM-CNN trained without attention. The figures shown in the table are the per-token recalls for each chunk class, which tells if a part of the model is responsible for signaling the whole model to predict the class.

Table content: <table/>

Notable Typing Results

Table content: <table/>

tity typing task requires a model to label {Barack Obama moves out of the White House .} as {Barack PERSON Obama PERSON moves NONE out NONE of NONE the FAC White FAC House FAC . NONE }. Table4shows the entity classes for which HC all yields notably different performance (> 2%) from that of NativeH . Of particular interest is C 5 's strong signal (27.27) for LAN (language) in comparison to the NativeH 's struggles (-9.09) on this class without attention.Qualitatively, we study the two sentences shown in Figure3, containing Dutch LAN into NONE English LAN and Chinese LAN and NONE English LAN . HC all classifies the tokens correctly, but both H and N ativeH wrongly predict Dutch NORP into NONE English LAN and Chinese NORP and NONE English LAN . Here NORP stands for nationality, meaning that both models without attention wrongly judge that Dutch and Chinese here refer to people from these countries.

Table content: <table/>

Peer review (iclr2019):
 In general, the paper proposes a deep architecture combining bi-LSTM, CNN and self-attentions to better model the interactions among the contexts for sequence labeling. The model is then evaluated on a sequence-labeling problem (NER) with one dataset. Although effective, the method seems to be a bit trivial through a combination. Evaluating on only one NER dataset is also not too convincing for me. Detailed comments are the following: Pros: 1. The paper is clearly written with a good organization to fully explain their model. The motivation is clear and comparison with existing work is clearly illustrated. 2. It is interesting to decouple the problem to entity chunking and entity typing for detailed evaluation, although I do not see any decoupling methodology in terms of the model itself. It would be better to also make the decoupling idea integrated into sequence-labeling models, in this way the model could be more different and novel compared to existing works. 3. The experimental section provides detailed analysis of the proposed model on the performance of each component as well as some qualitative analysis on what different attention heads learn. Cons: 1. The methodology itself is a bit trivial to me, since the bi-LSTM-CNN and self-attention mechanism were already proposed previously. It seems to be an improved combination, as also stated in Related Work ("The subtle difference is that they..."). It also seems over-stated in the abstract about the functionality on providing insightful understanding of attentions and decoupling NER into two subproblems. To my understanding, the understanding is only qualitative by observing a few examples, and the decoupling does not refer to the method, but the evaluation. 2. More existing works should be added in Related Work including sequence-labeling models as well as attention models. 3. It*s still unclear if one wants to re-implement the model. Section 5.2 still lacks some important parameters, e.g., learning rate, batch-size, training method, how to parallel etc. More explanations on baseline models should be added to better understand the comparisons. 4. Section 6.2 is confusing. Does the cross construction used in your experiments? Why do you put this part at the end of the paper. If section 6.2 provides better construction, why not making it integrated in the main part of the paper? Moreover, the authors claim the model*s ability to exploit cross-interactions among input sequence, but I do not really get this point. The paper should illustrate this characteristics into more details for the readers to understand. Questions: 1. Since your model is quite similar to Chiu & Nichols (2016), can you illustrate what*s the architecture of their model and what*s the difference between their work and Bi-LSTM-CNN in Table 2? 2. Can you explain how do you parallel the computation for self-attentions? 3. The statistics in Table 3 is quite unexpected to me. Although you mentioned C^{all} is much better than H for classifying "I", it did much worse for all the other classes (except "O"). In this case, I suppose the attention is actually worse than simply computing LSTM hidden representations (H). Can you explain why? The last column for NativeH actually shows little degradation compared to the full model, may I know how*s that happen, and what*s the difference between NativeH and H? Section 5.4.1 mentions that Table 3 shows "per-token recalls", can you explain more what does this mean? How*s the f1 scores?