Context-aware Attention Model for Coreference Resolution

1 INTRODUCTION . Coreference is one of the most frequent phenomena in English and the other languages . Coreference resolution is a crucial task before artificial intelligent systems capable of fully understanding the human language . Supervised methods , especially the models using neural-network-generated word representations , achieve outstanding performances Clark & Manning ( 2016 ) ; Lee et al . ( 2017 ; 2018 ) . However , for similar or identical text units , the problem of wrongfully getting the same coreferences is still puzzling . For example , the following conversation has the sentences , A and B . A : Yeah , it ’ s not far.Through the S-bahn here.I mean it ’ s like twenty minutes . B : Or something.And so , if I do it , I ’ d love to have you join me.It ’ s a fancy wedding too . The pronoun “ it ’ s ” in the sentence A and the “ it ’ s ” in the sentence B are obviously referring to the different things . As they are likely to get similar or even the same expression , a false link between them is often predicted . A similar case in sentence B . Due to different lemma and lexeme , the model would not predicate that “ I ” in A and ” you ” in B are coreferential . On the opposite , a false coreference between “ I ” in A and “ I ” in B would be predicted . In coreference resolution tasks , words referring each other are called mentions , while a mention could be a common noun , a proper noun or a pronoun . Taking the above example , a coreference system partitions the mentions in a sentence into one coreference chain- ( ” it ’ s ” , ” it ’ s ” ) , and singleton : ” the S-bahn ” for speaker1 . One coreference chain- ( ” I ” , ” I ’ d ” , ” me ” ) , and singleton : ” It ’ s ” for speaker2 and one coreference chain- ( ” I ” , ” you ” ) between two speakers . In recent years , several supervised approaches have been proposed to coreference resolution . The work can be categorized into three classes . 1 ) mention-pair models : A mention pair model is a binary classifier that determines whether a pair of mentions is co-referring or not McCarthy & Lehnert ( 1995 ) . One of the common limitation of the mention-pair model is that it can not capture information beyond the mention pair . The information that can be obtained from the two markables to determine their coreferential status is very limited . ; 2 ) entity-mention models : The entity-mention model aims to classify whether an NP ( Noun phrase ) is coreferent with a preceding cluster Yangy et al . ( 2004 ) ; Culotta et al . ( 2007 ) ; Daumé III & Marcu ( 2005 ) . This strategy considers the candidates independently . It can not measure how likely a candidate is the antecedent for a given anaphor , relative to the other candidates ; 3 ) ranking models : Ranking models allows candidate antecedents of a mention to be ranked simultaneously Iida et al . ( 2003 ) ; Denis & Baldridge ( 2008 ) ; Durrett & Klein ( 2013 ) . In the above three classes , the ranking models recently obtained the state-of-the-art performance Wiseman et al . ( 2015 ; 2016 ) ; Clark & Manning ( 2016 ) . More recently , Lee et al . ( 2018 ) proposed the first state-of-the-art end-to-end neural coreference resolution system . They consider all spans as potential mentions and learn distributions over possible antecedents for each . In addition , they use a fully differentiable approximation to higher-order inference to iteratively refine span representations , and the model only uses a minimal set of hand-engineered features ( speaker ID , document genre , span distance , span width ) . This leads to the problem that identical mentions tend to get similar or even identical representations , and further misled coreference resolutions to make mistakes . In this paper , we demonstrate a novel method utilizing attention mechanism to adaptively exploit features to represent identical mentions with different contexts . Inspired by the recent success of attention mechanism , we focus on this issue and develop a general attention mechanism that learns the importance/weight of each feature based on mention ’ s contexts and then add this information to the end-to-end neural model . The entire model could be trained end-to-end with gradient descent . The proposed model is evaluated on the CoNLL 2012 Shared Task Pradhan et al . ( 2012 ) . The results show that the method outperforms the baselines . Meanwhile , we made a statistic of the different features ’ weights in the attention mechanism . The statistic shows that the feature attention algorithm does help to distinguish the features for identical mentions based on different contexts . 2 THE MODEL . 2.1 TASK . In an end-to-end coreference resolution , the input is a document D with T words along with metadata , and the output is a set of mention clusters . Let N be the number of possible text spans in D. We consider all possible spans up to a predefined maximum width . Denote the start and end indices of a span i in D respectively by START ( i ) and END ( i ) . For each span i the system needs to assign an antecedent ai∈ { ,1 , ... , i-1 } from all preceding spans or a dummy antecedent . The dummy antecedent represents two cases : ( 1 ) the span i is not an entity mention , or ( 2 ) the span i is an entity mention but not coreferential with any previous span . Finally , all spans , that are connected by a set of antecedent predictions , are grouped . 2.2 METHODOLOGY . The section elaborates the model . We adopt a similar span representation approach as it in Lee et al . ( 2018 ) using bidirectional LSTMs and a headfinding attention . In our approach , we investigate grammatical numbers and use a general attention mechanism reweighing span features based on mention ’ s contexts to generate the new feature representation . Thereafter , the scores of how likely spans could be entity mentions are generated by a feed forward network . We also propose an attention model of antecedent scoring to reweigh pair-wise features and produce the new feature vectors . Span Representation Illustrated in Figure 1 , we assume vector representation of a sentence with L words as { x1 , x2 , ... , xL } , while xt denotes the concatenation of fixed pretrained word embeddings and CNN character embeddings Santos & Zadrozny ( 2014 ) for t-th word . Then Bidirectional LSTMs Hochreiter & Schmidhuber ( 1997 ) are used to encode each xt : ←− ht = LSTMbackward ( ←−− ht+1 , xt ) ( 1 ) −→ ht = LSTMforward ( −−→ ht+1 , xt ) ( 2 ) x∗t = [ ←− ht , −→ ht ] ( 3 ) Then , the model learns a task-specific notion of headedness using the attention mechanism Bahdanau et al . ( 2014 ) over words in each span : at = wa · FFNN ( x∗t ) ( 4 ) ai , t = exp ( at ) ∑END ( i ) k=START ( i ) exp ( ak ) ( 5 ) x̂i = END ( i ) ∑ t=START ( i ) ai , t · xt ( 6 ) Where xi is a weighted sum of word vectors in span i. FFNN is a feed forward network . Then the final representation gi of span i was produced by : gi = [ x ∗ START ( i ) , xEND ( i ) ∗ , x̂i , ϕ ( i ) ] ( 7 ) Where in Lee et al . ( 2018 ) , ϕ ( i ) is a 20-dimensional feature vector encoding only span width information . In our model , we incorporate one more feature , grammatical numbers . The grammatical numbers are reinforced with the feature attention method by reweighing contextual features . A new feature vector ϕ∗ . is generated by the method . Feature Attention Shown in Figure 2 , we use a general attention mechanism that learns the importance/weight of each feature based on contexts . Suppose the initial feature vectors is ϕ = [ ϕ1 , ϕ2 , . . . , ϕV ] , where ϕi ∈ R20 indicates the i-th feature and xu is the contexts vectors generated by Bi-LSTM . Then the model learns the weight of each feature based on the contexts , and generate the new feature vectors ϕ∗ : a′j = wa · FFNN ( ϕjf ( xu ) ) ( 8 ) a′j , u = exp ( a′j ) ∑V v=1 exp ( a ′ v ) ( 9 ) ϕ∗ = ⊕Vj=1a′j , u · ϕj ( 10 ) Where ⊕ is the concatenate operation and i is a linear function to map xu to the same dimension with the feature vector . ϕ∗ is the new reweighed feature vectors and a ′ j , u is the weight of each feature based on the contexts . Mention Scoring The new span representation is encoded to a FFNN to compute the mention score measuring if it is an entity mention , where FA is the feature attention algorithm : ϕ∗ ( i ) = FA ( ϕ ( i ) , x∗i ) ( 11 ) gi = [ x ∗ START ( i ) , x ∗ END ( i ) , x̂i , ϕ ∗ ( i ) ] ( 12 ) sm ( i ) = wm · FFNNm ( gi ) ( 13 ) Antecedent Scoring For antecedent scoring , we use the same features ( speaker , genre , distance ) as Lee et al . ( 2018 ) . And incorporate our feature attention algorithm to the features ϕ ( i , j ) : ϕ∗ ( i , j ) = FA ( ϕ ( i , j ) , x∗i ⊕ x∗j ) ( 14 ) sa ( i , j ) = wa · FFNNa ( gi , gj , gi ◦ gj , ϕ∗ ( i , j ) ) ( 15 ) Coreference Score The final coreference score of span i and j indicates that ( 1 ) whether span i is a mention , ( 2 ) whether span j is a mention , and ( 3 ) whether j is an antecedent of i : s ( i , j ) = { 0 , j = ε sm ( i ) + smj + sa ( i , j ) , j 6= ε ( 16 ) Loss Regression we use the same loss regression as Lee et al . ( 2018 ) , where GOLD ( i ) is the set of spans in the gold cluster containing span i : L = −log N∏ i=1 ∑ y′∈Y ( i ) ∩GOLD ( i ) p ( y′ ) ( 17 ) 3 EXPERIMENTS AND RESULTS . 3.1 EXPERIMENTAL SETUP . Dataset and baseline . The experiments are conducted on the English subset of the CONLL2012 Shared Task data Pradhan et al . ( 2012 ) and evaluated on three standard metrics : MUC Wiseman et al . ( 2016 ) , B3 Bagga & Baldwin ( 1998 ) , and CEAFϕ4 Luo ( 2005 ) . We report precision , recall , F1 for each metric and the average F1 as the final CoNLL score . We use the baseline below : Björkelund & Kuhn 2014 : The system obtains significant improvements over the baseline by modifying LaSO to delay updates until the end of each instance . Wiseman et al . 2015 : The system learns distinct feature representations for anaphoricity detection and antecedent ranking , encouraged by pre-training on a pair of corresponding subtasks . Wiseman et al . 2016 : The system proposes to use recurrent neural networks ( RNNs ) to learn latent , global representations of entity clusters directly from their mentions . Clark & Manning 2016 : The system presents a neural network based coreference system that produces high-dimensional vector representations for pairs of coreference clusters to learn when combining clusters is desirable . Lee et al . 2017 : The first state-of-the-art end-to-end neural coreference resolution system and only uses a minimal set of hand-craft features . Lee et al . 2018 : The improved model for Lee et al . ( 2017 ) using the antecedent distribution from a span-ranking architecture as an attention mechanism to iteratively refine span representations . Hyperparameters . we follow the same hyperparameters as in Lee et al . ( 2018 ) : Using GloVe word embeddings Pennington et al . ( 2014 ) with a window size of 2 for the head word embeddings and a window size of 10 for the LSTM inputs and embedding representations from a language model Peters et al . ( 2018 ) as the input to the LSTMs ( ELMo in the results ) . The maximum span width was 30 words and only consider 50 antecedents . λ = 0.4 is used for span pruning . The model is trained up to 150 epoch with early stopping .

Summary: This paper extends the neural coreference resolution model in Lee et al. (2018) by 1) introducing an additional mention-level feature (grammatical numbers), and 2) letting the mention/pair scoring functions attend over multiple mention-level features. The proposed model achieves marginal improvement (0.2 avg. F1 points) over Lee et al., 2018, on the CoNLL 2012 English test set.