Venue: NAACL 2019

# A General Framework for Information Extraction

using Dynamic Span Graphs

 Yi Luan\({}^{\dagger}\) Dave Wadden\({}^{\dagger}\) Luheng He\({}^{\ddagger}\) Amy Shah\({}^{\dagger}\)

Mari Ostendorf\({}^{\dagger}\) Hannaneh Hajishirzi\({}^{\dagger*}\)

\({}^{\dagger}\)University of Washington \({}^{*}\)Allen Institute for Artificial Intelligence

\({}^{\ddagger}\)Google AI Language

{luanyi, dwadden, amyshah, ostendor, hannaneh}@uw.edu luheng@google.com

###### Abstract

We introduce a general framework for several information extraction tasks that share span representations using dynamically constructed span graphs. The graphs are constructed by selecting the most confident entity spans and linking these nodes with confidence-weighted relation types and coreferences. The dynamic span graph allows coreference and relation type confidences to propagate through the graph to iteratively refine the span representations. This is unlike previous multi-task frameworks for information extraction in which the only interaction between tasks is in the shared first-layer LSTM. Our framework significantly outperforms the state-of-the-art on multiple information extraction tasks across multiple datasets reflecting different domains. We further observe that the span enumeration approach is good at detecting nested span entities, with significant F1 score improvement on the ACE dataset.1

Footnote 1: Code and pre-trained models are publicly available at [https://github.com/luanyi/DyGIE](https://github.com/luanyi/DyGIE).

## 1 Introduction

Most Information Extraction (IE) tasks require identifying and categorizing phrase spans, some of which might be nested. For example, entity recognition involves assigning an entity label to a phrase span. Relation Extraction (RE) involves assigning a relation type between pairs of spans. Coreference resolution groups spans referring to the same entity into one cluster. Thus, we might expect that knowledge learned from one task might benefit another.

Most previous work in IE (e.g., Nadeau and Sekine (2007); Chan and Roth (2011)) employs a pipeline approach, first detecting entities and then using the detected entity spans for relation extraction and coreference resolution. To avoid cascading errors introduced by pipeline-style systems, recent work has focused on coupling different IE tasks as in joint modeling of entities and relations Miwa and Bansal (2016); Zhang et al. (2017), entities and coreferences Hajishirzi et al. (2013); Durrett and Klein (2014), joint inference Singh et al. (2013) or multi-task (entity/relation/coreference) learning Luan et al. (2018a). These models mostly rely on the first layer LSTM to share span representations between different tasks and are usually designed for specific domains.

In this paper, we introduce a general framework Dynamic Graph IE (DYGIE) for coupling multiple information extraction tasks through shared span representations which are refined leveraging contextualized information from relations and coreferences. Our framework is effective in several domains, demonstrating a benefit from incorporating broader context learned from relation and coreference annotations.

Figure 1 shows an example illustrating the potential benefits of entity, relation, and coreference contexts. It is impossible to predict the entity labels for _This thing_ and _it_ from within-sentence context alone. However, the antecedent _car_ strongly suggests that these two entities have a VEH type. Similarly, the fact that _Tom_ is located at _Starbucks_ and _Mike_ has a relation to _Tom_ provides support for

Figure 1: A text passage illustrating interactions between entities, relations and coreference links. Some relation and coreference links are omitted.



the fact that _Mike_ is located at _Starbucks_.

DyGIE uses multi-task learning to identify entities, relations, and coreferences through shared span representations using dynamically constructed span graphs. The nodes in the graph are dynamically selected from a beam of highly-confident mentions, and the edges are weighted according to the confidence scores of relation types or coreferences. Unlike the multi-task method that only shares span representations from the local context Luan et al. (2018), our framework leverages rich contextual span representations by propagating information through coreference and relation links. Unlike previous BIO-based entity recognition systems Collobert and Weston (2008); Lample et al. (2016); Ma and Hovy (2016) that assign a text span to at most one entity, our framework enumerates and represents all possible spans to recognize arbitrarily overlapping entities.

We evaluate DyGIE on several datasets spanning many domains (including news, scientific articles, and wet lab experimental protocols), achieving state-of-the-art performance across all tasks and domains and demonstrating the value of coupling related tasks to learn richer span representations. For example, DyGIE achieves relative improvements of 5.7% and 9.9% over state of the art on the ACE05 entity and relation extraction tasks, and an 11.3% relative improvement on the ACE05 overlapping entity extraction task.

The contributions of this paper are threefold. 1) We introduce the dynamic span graph framework as a method to propagate global contextual information, making the code publicly available. 2) We demonstrate that our framework significantly outperforms the state-of-the-art on joint entity and relation detection tasks across four datasets: ACE 2004, ACE 2005, SciERC and the Wet Lab Protocol Corpus. 3) We further show that our approach excels at detecting entities with overlapping spans, achieving an improvement of up to 8 F1 points on three benchmarks annotated with overlapped spans: ACE 2004, ACE 2005 and GENIA.

## 2 Related Work

Previous studies have explored joint modeling Miwa and Bansal (2016); Zhang et al. (2017); Singh et al. (2013); Yang and Mitchell (2016)) and multi-task learning Peng and Dredze (2015); Peng et al. (2017); Luan et al. (2018, 2017) as methods to share representational strength across related information extraction tasks. The most similar to ours is the work in Luan et al. (2018) that takes a multi-task learning approach to entity, relation, and coreference extraction. In this model, the different tasks share span representations that only incorporate broader context indirectly via the gradients passed back to the LSTM layer. In contrast, DyGIE uses dynamic graph propagation to explicitly incorporate rich contextual information into the span representations.

Entity recognition has commonly been cast as a sequence labeling problem, and has benefited substantially from the use of neural architectures Collobert et al. (2011); Lample et al. (2016); Ma and Hovy (2016); Luan et al. (2017, 2018). However, most systems based on sequence labeling suffer from an inability to extract entities with overlapping spans. Recently Katiyar and Cardie (2018) and Wang and Lu (2018) have presented methods enabling neural models to extract overlapping entities, applying hypergraph-based representations on top of sequence labeling systems. Our framework offers an alternative approach, forgoing sequence labeling entirely and simply considering all possible spans as candidate entities.

Neural graph-based models have achieved significant improvements over traditional feature-based approaches on several graph modeling tasks. Knowledge graph completion Yang et al. (2015); Bordes et al. (2013) is one prominent example. For relation extraction tasks, graphs have been used primarily as a means to incorporate pipelined features such as syntactic or discourse relations Peng et al. (2017); Song et al. (2018); Zhang et al. (2018). Christopoulou et al. (2018) models all possible paths between entities as a graph, and refines pair-wise embeddings by performing a walk on the graph structure. All these previous works assume that the nodes of the graph (i.e. the entity candidates to be considered during relation extraction) are predefined and fixed throughout the learning process. On the other hand, our framework does not require a fixed set of entity boundaries as an input for graph construction. Motivated by state-of-the-art span-based approaches to coreference resolution Lee et al. (2017, 2018) and semantic role labeling He et al. (2018), the model uses a beam pruning strategy to dynamically select high-quality spans, and constructs a graph using the selected spans as nodes.

Many state-of-the-art RE models rely upon 

domain-specific external syntactic tools to construct dependency paths between the entities in a sentence Li and Ji (2014); Xu et al. (2015); Miwa and Bansal (2016); Zhang et al. (2017). These systems suffer from cascading errors from these tools and are hard to generalize to different domains. To make the model more general, we combine the multitask learning framework with ELMo embeddings Peters et al. (2018) without relying on external syntactic tools and risking the cascading errors that accompany them, and improve the interaction between tasks through dynamic graph propagation. While the performance of DyGIE benefits from ELMo, it advances over some systems Luan et al. (2018); Sanh et al. (2019) that also incorporate ELMo. The analyses presented here give insights into the benefits of joint modeling.

## 3 Model

Problem DefinitionThe input is a document represented as a sequence of words \(D\), from which we derive \(S=\{s_{1},\ldots,s_{T}\}\), the set of all possible within-sentence word sequence spans (up to length \(L\)) in the document. The output contains three structures: the entity types \(E\) for all spans \(S\), the relations \(R\) for all span pairs \(S\times S\) within the same sentence, and the coreference links \(C\) for all spans in \(S\) across sentences. We consider two primary tasks. First, _Entity Recognition_ is the task of predicting the best entity type labels \(e_{i}\) for each span \(s_{i}\). Second, _Relation Extraction_ involves predicting the best relation type \(r_{ij}\) for all span pairs \((s_{i},s_{j})\). We provide additional supervision by also training our model to perform a third, auxiliary task: _Coreference resolution_. For this task we predict the best antecedent \(c_{i}\) for each span \(s_{i}\).

Our ModelWe develop a general information extraction framework (DyGIE) to identify and classify entities, relations, and coreference in a multi-task setup. DyGIE first enumerates all text spans in each sentence, and computes a locally-contextualized vector space representation of each span. The model then employs a _dynamic span graph_ to incorporate global information into its span representations, as follows. At each training step, the model identifies the text spans that are most likely to represent entities, and treats these spans as nodes in a graph structure. It constructs confidence-weighted arcs for each node according to its predicted coreference and relation links with the other nodes in the graph. Then, the span representations are refined using broader context from gated updates propagated from neighboring relation types and co-referred entities. These refined span representations are used in a multi-task framework to predict entity types, relation types, and coreference links.

### Model Architecture

In this section, we give an overview of the main components and layers of the DyGIE framework, as illustrated in Figure 2. Details of the graph construction and refinement process will be presented in the next section.

Token Representation LayerWe apply a bidirectional LSTM over the input tokens. The input for each token is a concatenation of the character reprensetation, GLoVe Pennington et al. (2014) word embeddings, and ELMo embeddings Peters et al. (2018). The output token representations are obtained by stacking the forward and backward LSTM hidden states.

Span Representation LayerFor each span \(s_{i}\), its initial vector representation \(\mathbf{g}^{0}_{i}\) is obtained by concatenating BiLSTM outputs at the left and right end points of \(s_{i}\), an attention-based soft "headword," and an embedded span width feature, following Lee et al. (2017).

Coreference Propagation LayerThe propagation process starts from the span representations \(\mathbf{g}^{0}_{i}\). At each iteration \(t\), we first compute an _update vector_\(\mathbf{u}^{t}_{C}\) for each span \(s_{i}\). Then we use \(\mathbf{u}^{t}_{C}\) to update the current representation \(\mathbf{g}^{t}_{i}\), producing the next span representation \(\mathbf{g}^{t+1}_{i}\). By repeating this process \(N\) times, the final span representations \(\mathbf{g}^{N}_{i}\) share contextual information across spans that are likely to be antecedents in the coreference graph, similar to the process in Lee et al. (2018).

Relation Propagation LayerThe outputs \(\mathbf{g}^{N}_{i}\) from the coreference propagation layer are passed as inputs to the relation propagation layer. Similar to the coreference propagation process, at each iteration \(t\), we first compute the update vectors \(\mathbf{u}^{t}_{R}\) for each span \(s_{i}\), then use it to compute \(\mathbf{g}^{t+1}_{i}\). Information can be integrated from multiple relation paths by repeating this process \(M\) times.

Final Prediction LayerWe use the outputs of the relation graph layer \(\mathbf{g}^{N+M}_{i}\) to predict the entity labels \(E\) and relation labels \(R\). For entities, we pass \(\mathbf{g}^{N+M}_{i}\) to a feed-forward network (FFNN) to 

produce per-class scores \({\bf P}_{E}(i)\) for span \(s_{i}\). For relations, we pass the concatenation of \({\bf g}^{N+M}_{i}\) and \({\bf g}^{N+M}_{j}\) to a FFNN to produce per-class relation scores \({\bf P}_{R}(i,j)\) between spans \(s_{i}\) and \(s_{j}\). Entity and relation scores are normalized across the label space, similar to Luan et al. (2018a). For coreference, the scores between span pairs (\(s_{i},s_{j}\)) are computed from the coreference graph layer outputs (\({\bf g}^{N}_{i},{\bf g}^{N}_{j}\)), and then normalized across all possible antecedents, similar to Lee et al. (2018).

### Dynamic Graph Construction and Span Refinement

The dynamic span graph facilitates propagating broader contexts through soft coreference and relation links to refine span representations. The nodes in the graph are spans \(s_{i}\) with vector representations \({\bf g}^{t}_{i}\in\mathbb{R}^{d}\) for the \(t\)-th iteration. The edges are weighted by the coreference and relation scores, which are trained according to the neural architecture explained in Section 3.1. In this section, we explain how coreference and relation links can update span representations.

Coreference PropagationSimilar to (Luan et al., 2018a), we define a beam \(B_{C}\) consisting of \(b_{c}\) spans that are most likely to be in a coreference chain. We consider \({\bf P}^{t}_{C}\) to be a matrix of real values that indicate coreference confidence scores between these spans at the \(t\)-th iteration. \({\bf P}^{t}_{C}\) is of size \(b_{c}\times K\), where \(K\) is the maximum number of antecedents considered. For the coreference graph, an edge in the graph is single directional, connecting the current span \(s_{i}\) with all its potential antecedents \(s_{j}\) in the coreference beam, where \(j<i\). The edge between \(s_{i}\) and \(s_{j}\) is weighted by coreference confidence score at the current iteration \(P^{t}_{C}(i,j)\). The span update vector \({\bf u}^{t}_{C}(i)\in\mathbb{R}^{d}\) is computed by aggregating the neighboring span representations \({\bf g}^{t}_{j}\), weighted by their coreference scores \(P^{t}_{C}(i,j)\):

\[{\bf u}^{t}_{C}(i)=\sum_{j\in B_{C}(i)}P^{t}_{C}(i,j){\bf g}^{t}_{j} \tag{1}\]

where \(B_{C}(i)\) is the set of \(K\) spans that are antecedents of \(s_{i}\),

\[P^{t}_{C}(i,j)=\frac{\exp(V^{t}_{C}(i,j))}{\sum_{j^{\prime}\in B_{C}(i)}\exp( V^{t}_{C}(i,j))} \tag{2}\]

\(V^{t}_{C}(i,j)\) is a scalar score computed by concatenating the span representations \([{\bf g}^{t}_{i},{\bf g}^{t}_{j},{\bf g}^{t}_{i}\odot{\bf g}^{t}_{j}]\), where \(\odot\) is element-wise multiplication. The concatenated vector is then fed as input to a FFNN, similar to (Lee et al., 2018).

Relation PropagationFor each sentence, we define a beam \(B_{R}\) consisting of \(b_{r}\) entity spans that are mostly likely to be involved in a relation. Unlike the coreference graph, the weights of relation edges capture different relation types. Therefore, for the \(t\)-th iteration, we use a tensor \({\bf V}^{t}_{R}\in\mathbb{R}^{b_{R}\times b_{R}\times L_{R}}\) to capture scores of each of the \(L_{R}\) relation types. In other words, each edge in the

Figure 2: Overview of our DvGIE model. Dotted arcs indicate confidence weighted graph edges. Solid lines indicate the final predictions.

 

relation graph connects two entity spans \(s_{i}\) and \(s_{j}\) in the relation beam \(B_{R}\). \(\mathbf{V}^{t}_{R}(i,j)\) is a \(L_{R}\)-length vector of relation scores, computed with a FFNN with \([\mathbf{g}^{t}_{i},\mathbf{g}^{t}_{j}]\) as the input. The relation update vector \(\mathbf{u}^{t}_{R}(i)\in\mathbb{R}^{d}\) is computed by aggregating neighboring span representations on the relation graph:

\[\mathbf{u}^{t}_{R}(i)=\sum_{j\in B_{\text{R}}}f(\mathbf{V}^{t}_{ R}(i,j))\mathbf{A}_{R}\odot\mathbf{g}^{t}_{j}, \tag{3}\]

where \(\mathbf{A}_{R}\in\mathbb{R}^{L_{R}\times d}\) is a trainable linear projection matrix, \(f\) is a non-linear function to select the most important relations. Because only a small number of entities in the relation beam are actually linked to the target span, propagation among all possible span pairs would introduce too much noise to the new representation. Therefore, we choose \(f\) to be the ReLU function to remove the effect of unlikely relations by setting the all negative relation scores to 0. Unlike coreference connections, two spans linked via a relation are not expected to have similar representations, so the matrix \(\mathbf{A}_{R}\) helps to transform the embedding \(\mathbf{g}^{t}_{j}\) according to each relation type.

**Updating Span Representations with Gating** To compute the span representations for the next iteration \(t\in\{1,\ldots,N+M\}\), we define a gating vector \(\mathbf{f}^{t}_{x}(i)\in\mathbb{R}^{d}\), where \(x\in\{C,R\}\), to determine whether to keep the previous span representation \(\mathbf{g}^{t}_{i}\) or to integrate new information from the coreference or relation update vectors \(\mathbf{u}^{t}_{x}(i)\). Formally,

\[\mathbf{f}^{t}_{x}(i) = g(\mathbf{W}^{t}_{x}[\mathbf{g}^{t}_{i},\mathbf{u}^{t}_{x}(i)]) \tag{4}\] \[\mathbf{g}^{t+1}_{i} = \mathbf{f}^{t}_{x}(i)\odot\mathbf{g}^{t}_{i}+(1-\mathbf{f}^{t}_{ x}(i))\odot\mathbf{u}^{t}_{x}(i),\]

where \(\mathbf{W}^{t}_{x}\in\mathbb{R}^{d\times 2d}\) are trainable parameters, and \(g\) is an element-wise sigmoid function.

### Training

The loss function is defined as a weighted sum of the log-likelihood of all three tasks:

\[\sum_{(D,R^{*},E^{*},C^{*})\in\mathcal{D}}\Big{\{}\lambda_{\text{ E}}\log P(E^{*}\mid C,R,D) \tag{5}\] \[+\lambda_{\text{R}}\log P(R^{*}\mid C,D)+\lambda_{\text{C}}\log P (C^{*}\mid D)\Big{\}}\]

where \(E^{*}\), \(R^{*}\) and \(C^{*}\) are gold structures of the entity types, relations and coreference, respectively. \(\mathcal{D}\) is the collection of all training documents \(D\). The task weights \(\lambda_{\text{E}}\), \(\lambda_{\text{R}}\), and \(\lambda_{\text{C}}\) are hyper-parameters to control the importance of each task.

We use a 1 layer BiLSTM with 200-dimensional hidden layers. All the feed-forward functions have 2 hidden layers of 150 dimensions each. We use 0.4 variational dropout Gal and Ghahramani (2016) for the LSTMs, 0.4 dropout for the FFNNs, and 0.5 dropout for the input embeddings. The hidden layer dimensions and dropout rates are chosen based on the development set performance in multiple domains. The task weights, learning rate, maximum span length, number of propagation iterations and beam size are tuned specifically for each dataset using development data.

## 4 Experiments

DyGIE is a general IE framework that can be applied to multiple tasks. We evaluate the performance of DyGIE against models from two lines of work: combined entity and relation extraction, and overlapping entity extraction.

### Entity and relation extraction

For the entity and relation extraction task, we test the performance of DyGIE on four different datasets: ACE2004, ACE2005, SciERC and the Wet Lab Protocol Corpus. We include the relation graph propagation layer in our models for all datasets. We include the coreference graph propagation layer on the data sets that have coreference annotations available.

DataAll four data sets are annotated with entity and relation labels. Only a small fraction of entities (\(<3\%\) of total) in these data sets have a text span that overlaps the span of another entity. Statistics on all four data sets are displayed in Table 1.

The **ACE2004** and **ACE2005** corpora provide entity and relation labels for a collection of documents from a variety of domains, such as newswire and online forums. We use the same entity and relation types, data splits, and preprocessing as Miwa and Bansal (2016) and Li and Ji (2014). Following the convention established in this line of work, an entity prediction is considered correct

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline  & Domain & Docs & Ent & Rel & Coref \\ \hline ACE04 & News & 348 & 7 & 7 & ✓ \\ ACE05 & News & 511 & 7 & 6 & ✗ \\ SciERC & AI & 500 & 6 & 7 & ✓ \\ WLP & Bio lab & 622 & 18 & 13 & ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Datasets for joint entity and relation extraction and their statistics. _Ent_: Number of entity categories. _Rel_: Number of relation categories.

 

if its type label and head region match those of a gold entity. We will refer to this version of the ACE2004 and ACE2005 data as ACE04 and ACE05. Since the domain and mention span annotations in the ACE datasets are very similar to those of OntoNotes Pradhan et al. (2012), and OntoNotes contains significantly more documents with coreference annotations, we use OntoNotes to train the parameters for the auxiliary coreference task. The OntoNotes corpus contains 3493 documents, averaging roughly 450 words in length.

The **SciERC** corpus Luan et al. (2018) provides entity, coreference and relation annotations for a collection of documents from 500 AI paper abstracts. The dataset defines scientific term types and relation types specially designed for AI domain knowledge graph construction. An entity prediction is considered correct if its label and span match with a gold entity.

The **Wet Lab Protocol Corpus (WLPC)** provides entity, relation, and event annotations for 622 wet lab protocols Kulkarni et al. (2018). A wet lab protocol is a series of instructions specifying how to perform a biological experiment. Following the procedure in Kulkarni et al. (2018), we perform entity recognition on the union of entity tags and event trigger tags, and relation extraction on the union of entity-entity relations and entity-trigger event roles. Coreference annotations are not available for this dataset.

BaselinesWe compare DyGIE with current state of the art methods in different datasets. Miwa and Bansal (2016) provide the current state of the art on ACE04. They construct a Tree LSTM using dependency parse information, and use the representations learned by the tree structure as features for relation classification. Bekoulis et al. (2018) use adversarial training as regularization for a neural model. Zhang et al. (2017) cast joint entity and relation extraction as a table filling problem and build a globally optimized neural model incorporating syntactic representations from a dependency parser. Similar to DryGIE, Sanh et al. (2019) and Luan et al. (2018) use a multi-task learning framework for extracting entity, relation and coreference labels. Sanh et al. (2019) improved the state of the art on ACE05 using multi-task, hierarchical supervised training with a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. Luan et al. (2018) previously achieved the state of the art on SciERC and use a span-based neural model like our DyGIE. Kulkarni et al. (2018) provide a baseline for the WLPC data set. They employ an LSTM-CRF for entity recognition, following Lample et al. (2016). For relation extraction, they assume the presence of gold entities and train a maximum-entropy classifier using features from the labeled entities.

ResultsTable 2 shows test set F1 on the joint entity and relation extraction task. We observe that DyGIE achieves substantial improvements on both entity recognition and relation extraction across the four data sets and three domains, all in the realistic setting where no "gold" entity labels are supplied at test time. DyGIE achieves 7.1% and 7.0% relative improvements over the state of the art on NER for ACE04 and ACE05, respectively. For the relation extraction task, DyGIE attains 25.8% relative improvement over SOTA on ACE04 and 13.7% relative improvement on ACE05. For ACE05, the best entity extraction performance is obtained by switching the order between CorefProp and RelProp (RelProp first then CorefProp).

On SciERC, DyGIE advances the state of the art by 5.9% and 1.9% for relation extraction and NER, respectively. The improvement of DyGIE over the previous SciERC model underscores the ability of coreference and relation propagation to construct rich contextualized representations.

The results from Kulkarni et al. (2018) establish a baseline for IE on the WLPC. In that work, relation extraction is performed using gold entity boundaries as input. Without using any gold entity information, DyGIE improves on the baselines by 16.8% for relation extraction and 2.2% for NER.

\begin{table}
\begin{tabular}{l l c c} \hline \hline Dataset & System & Entity & Relation \\ \hline \multirow{3}{*}{ACE04} & Bekoulis et al. (2018) & 81.6 & 47.5 \\  & Miwa and Bansal (2016) & 81.8 & 48.4 \\  & DYGIE & **87.4** & **59.7** \\ \hline \multirow{3}{*}{ACE05} & Miwa and Bansal (2016) & 83.4 & 55.6 \\  & Zhang et al. (2017) & 83.6 & 57.5 \\  & Sanh et al. (2019) & 87.5 & 62.7 \\  & DYGIE & **88.4** & **63.2** \\ \hline \multirow{3}{*}{SciERC} & Luan et al. (2018) & 64.2 & 39.3 \\  & DyGIE & **65.2** & **41.6** \\ \hline \multirow{3}{*}{WLPC} & Kulkarni et al. (2018) & 78.0 & *54.9 \\ \cline{1-1}  & DYGIE & **79.5** & **64.1** \\ \hline \hline \end{tabular}
\end{table}
Table 2: F1 scores on the joint entity and relation extraction task on each test set, compared against the previous best systems. * indicates relation extraction system that takes gold entity boundary as input.

 

On the OntoNotes data set used for the auxiliary coreference task with ACE05, our model achieves coreference test set performance of 70.4 F1, which is competitive with the state-of-the-art performance reported in Lee et al. (2017).

### Overlapping Entity Extraction

There are many applications where the correct identification of overlapping entities is crucial for correct document understanding. For instance, in the biomedical domain, a _BRCA1 mutation carrier_ could refer to a patient taking part in a clinical trial, while _BRCA1_ is the name of a gene.

We evaluate the performance of DyGIE on overlapping entity extraction in three datasets: ACE2004, ACE2005 and GENIA. Since relation annotations are not available for these datasets, we include the coreference propagation layer in our models but not the relation layer.2

Footnote 2: We use the pre-processed ACE dataset from previous work and relation annotation is not available.

DataStatistics on our three datasets are listed in Table 3. All three have a substantial number (\(>20\%\) of total) of overlapping entities, making them appropriate for this task.

As in the joint case, we evaluate our model on **ACE2004** and **ACE2005**, but here we follow the same data preprocessing and evaluation scheme as Wang and Lu (2018). We refer to these data sets as ACE04-O and ACE05-O. Unlike the joint entity and relation task in Sec. 4.1, where only the entity head span need be predicted, an entity prediction is considered correct in these experiments if both its entity label and its full text span match a gold prediction. This is a more stringent evaluation criterion than the one used in Section 4.1. As before, we use the OntoNotes annotations to train the parameters of the coreference layer.

The **GENIA** corpus Kim et al. (2003) provides entity tags and coreferences for 1999 abstracts from the biomedical research literature. We only use the IDENT label to extract coreference clusters. We use the same data set split and preprocessing procedure as Wang and Lu (2018) for overlapping entity recognition.

BaselinesThe current state-of-the-art approach on all three data sets is Wang and Lu (2018), which uses a segmental hypergraph coupled with neural networks for feature learning. Katiyar and Cardie (2018) also propose a hypergraph approach using a recurrent neural network as a feature extractor.

ResultsTable 4 presents the results of our overlapping entity extraction experiments on the different datsets. DyGIE improves 11.6% on the state of the art for ACE04-O and 11.3% for ACE05-O. DYGIE also advances the state of the art on GENIA, albeit by a more modest 1.5%. Together these results suggest that DyGIE can be utilized fruitfully for information extraction across different domains with overlapped entities, such as bio-medicine.

## 5 Analysis of Graph Propagation

We use the dev sets of ACE2005 and SciERC to analyze the effect of different model components.

### Coreference and Relation Graph Layers

Tables 5 and 6 show the effects of graph propagation on entity and relation prediction accuracy,

\begin{table}
\begin{tabular}{l l c} \hline \hline Dataset & System & Entity F1 \\ \hline  & Katiyar and Cardie (2018) & 72.7 \\ ACE04-O & Wang and Lu (2018) & 75.1 \\  & DYGIE & **84.7** \\ \hline  & Katiyar and Cardie (2018) & 70.5 \\ ACEO5-O & Wang and Lu (2018) & 74.5 \\  & DYGIE & **82.9** \\ \hline  & Katiyar and Cardie (2018) & 73.8 \\ GENIA & Wang and Lu (2018) & 75.1 \\  & DYGIE & **76.2** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance on the overlapping entity extraction task, compared to previous best systems. We report F1 of extracted entities on the test sets.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & Domain & Docs & Ent & Overlap & Coref \\ \hline ACE04-O & News & 443 & 7 & 42\% & ✓ \\ ACE05-O & News & 437 & 7 & 32\% & ✗ \\ GENIA & Biomed & 1999 & 5 & 24\% & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 3: Datasets for overlapping entity extraction and their statistics. _Ent_: Number of entity categories. _Overlap_: Percentage of sentences that contain overlapping entities.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{3}{c}{Entity} & \multicolumn{3}{c}{Relation} \\ \cline{2-7} Model & P & R & F1 & P & R & F1 \\ \hline DyGIE & 87.4 & 86.7 & **87.1** & 56.2 & 60.9 & 58.4 \\ –CorefProp & 86.2 & 85.2 & 85.7 & 64.3 & 56.7 & **60.2** \\ –RelProp & 87.0 & 86.7 & 86.9 & 60.4 & 55.8 & 58.0 \\ Base & 86.1 & 85.7 & 85.9 & 59.5 & 55.7 & 57.6 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablations on the ACE05 development set with different graph propagation setups. –CorefProp ablates the coreference propagation layers, while –RelProp ablates the relation propagation layers. Base is the system without any propagation.

 

where \(-\)CorefProp and \(-\)RelProp denote ablating the propagation process by setting \(N=0\) or \(M=0\), respectively. Base is the base model without any propagation. For ACE05, we observe that coreference propagation is mainly helpful for entities; it appears to hurt relation extraction. On SciIE, coreference propagation gives a small benefit on both tasks. Relation propagation significantly benefits both entity and relation extraction in both domains. In particular, there are a large portion of sentences with multiple relation instances across different entities in both ACE05 and SciERC, which is the scenario in which we expect relation propagation to help.

Since coreference propagation has more effect on entity extraction and relation propagation has more effect on relation extraction, we mainly focus on ablating the effect of coreference propagation on entity extraction and relation propagation on relation extraction in the following subsections.

### Coreference Propagation and Entities

A major challenge of ACE05 is to disambiguate the entity class for pronominal mentions, which requires reasoning with cross-sentence contexts. For example, in a sentence from ACE05 dataset, "One of **[them]\({}_{\text{PER}}\)**, from a very close friend of **[ours]\({}_{\text{ORG}}\)**." It is impossible to identity whether _them_ and _ours_ is a person (_PER_) or organization (_ORG_) unless we have read previous sentences. We hypothesize that this is a context where coreference propagation can help. Table 7 shows the effect of the coreference layer for entity categorization of pronouns.3 DyGIE has 6.6% improvement on pronoun performance, confirming our hypothesis.

Footnote 3: Pronouns included: anyone, everyone, it, itself, one, our, ours, their, theirs, them, themselves, they, us, we, who

Looking further, Table 8 shows the impact on all entity categories, giving the difference between the confusion matrix entries with and without CorefProp. The frequent confusions associated with pronouns (_GPE/PER_ and _PER/ORG_, where _GPE_ is a geopolitical entity) greatly improve, but the benefit of CorefProp extends to most categories.

Of course, there are a few instances where CorefProp causes errors in entity extraction. For example, in the sentence "[They]\({}^{\text{DRG}}_{\text{PER}}\) might have been using Northshore...", DyGIE predicted _They_ to be of _ORG_ type because the most confident antecedent is _those companies_ in the previous sentence: "The money was invested in _those companies_." However, _They_ is actually referring to _these fund managers_ earlier in the document, which belongs to _PER_ category.

In the SciERC dataset, the pronouns are uniformly assigned with a _Generic_ label, which explains why CorefProp does not have much effect on entity extraction performance.

The Figure 2(a) shows the effect of number of iterations for coreference propagation in the entity extraction task. The figure shows that coreference layer obtains the best performance on the second iteration (\(N=2\)).

### Relation Propagation Impact

Figure 4 shows relation scores as a function of number of entities in sentence for DyGIE and DyGIE without relation propagation on ACE05. The figure indicates that relation propagation achieves significant improvement in sentences with more entities, where one might expect that using broader context

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{Entity} & \multicolumn{2}{c}{Relation} \\ \cline{2-6} Model & P & R & F1 & P & R & F1 \\ \hline DyGIE & 68.6 & 67.8 & **68.2** & 46.2 & 38.5 & **42.0** \\ \(-\)CorefProp & 69.2 & 66.9 & 68.0 & 42.0 & 40.5 & 41.2 \\ \(-\)RelProp & 69.1 & 66.0 & 67.5 & 43.6 & 37.6 & 40.4 \\ Base & 70.0 & 66.3 & 68.1 & 45.4 & 34.9 & 39.5 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Ablations on the SciERC development set on different graph progation setups. CorefProp has a much smaller effect on entity F1 compared to ACE05.

\begin{table}
\begin{tabular}{l l l l} \hline \hline Entity Perf. on Pronouns & P & R & F1 \\ \hline DyGIE & 79.0 & 77.1 & **78.0** \\ DyGIE\(-\)CorefProp & 73.8 & 72.6 & 73.2 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Entity extraction performance on pronouns in ACE05. CorefProp significantly increases entity extraction F1 on hard-to-disambiguate pronouns by allowing the model to leverage cross-sentence contexts.

Figure 3: F1 score of each layer on ACE development set for different number of iterations. \(N=0\) or \(M=0\) indicates no propagation is made for the layer.

 

could have more impact.

Figure 2(b) shows the effect of number of iterations for relation propagation in the relation extraction task. Our model achieves the best performance on the second iteration (\(M=2\)).

## 6 Conclusion

We have introduced DyGIE as a general information extraction framework, and have demonstrated that our system achieves state-of-the art results on entity recognition and relation extraction tasks across a diverse range of domains. The key contribution of our model is the dynamic span graph approach, which enhance interaction across tasks that allows the model to learn useful information from broader context. Unlike many IE frameworks, our model does not require any preprocessing using syntactic tools, and has significant improvement across different IE tasks including entity, relation extraction and overlapping entity extraction. The addition of co-reference and relation propagation across sentences adds only a small computation cost to inference; the memory cost is controlled by beam search. These added costs are small relative to those of the baseline span-based model. We welcome the community to test our model on different information extraction tasks. Future directions include extending the framework to encompass more structural IE tasks such as event extraction.

#### Acknowledgments

This research was supported by the Office of Naval Research under the MURI grant N00014-18-1-2670, NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, Samsung GRO and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg. We also thank the anonymous reviewers and the UW-NLP group for their helpful comments.

## References

* Bekoulis et al. (2018) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_, pages 2830-2836.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In _Advances in neural information processing systems_.
* Chan and Roth (2011) Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semantic structures for relation extraction. In _Proc. Annu. Meeting Assoc. for Computational Linguistics (ACL)_.
* Christopoulou et al. (2018) Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2018. A walk-based model on entity graphs for relation extraction. In _Proc. Annu. Meeting Assoc. for Computational Linguistics (ACL)_, volume 2, pages 81-88.
* Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In _Proc. Int. Conf. Machine Learning (ICML)_, pages 160-167.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _J. Machine Learning Research_, 12(Aug):2493-2537.
* Durrett and Klein (2014) Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. _Trans. Assoc. for Computational Linguistics (TACL)_, 2:477-490.
* Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In _Proc. Annu. Conf. Neural Inform. Process. Syst. (NIPS)_.
* Hajishirzi et al. (2013) Hannaneh Hajishirzi, Leila Zilles, Daniel S Weld, and Luke Zettlemoyer. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_, pages 289-299.

Figure 4: Relation F1 broken down by number of entities in each sentence. The performance of relation extraction degrades on sentences containing more entities. Adding relation propagation alleviates this problem.

 

Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In _ACL_.
* Katiyar and Cardie (2018) Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In _Proc. Conf. North American Assoc. for Computational Linguistics (NAACL)_.
* a semantically annotated corpus for bio-textmining. _Bioinformatics_, 19 Suppl 1:i180-2.
* Kulkarni et al. (2018) Chaitanya Kulkarni, Wei Xu, Alan Ritter, and Raghu Machiraju. 2018. An annotated corpus for machine reading of instructions in wet lab protocols. In _NAACL-HLT_.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proc. Conf. North American Assoc. for Computational Linguistics (NAACL)_.
* Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. In _EMNLP_.
* Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In _NAACL_.
* Li and Ji (2014) Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In _Proc. Annu. Meeting Assoc. for Computational Linguistics (ACL)_, volume 1, pages 402-412.
* Luan et al. (2017a) Yi Luan, Chris Brockett, Bill Dolan, Jianfeng Gao, and Michel Galley. 2017a. Multi-task learning for speaker-role adaptation in neural conversation models. In _Proc. IJCNLP_.
* Luan et al. (2018a) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018a. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_.
* Luan et al. (2017b) Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017b. Scientific information extraction with semi-supervised neural tagging. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_.
* Luan et al. (2018b) Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2018b. The uwnlp system at semeval-2018 task 7: Neural relation extraction model with selectively incorporated concept embeddings. In _Proc. Int. Workshop on Semantic Evaluation (SemEval)_, pages 788-792.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In _Proc. Annu. Meeting Assoc. for Computational Linguistics (ACL)_.
* Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In _Proc. Annu. Meeting Assoc. for Computational Linguistics (ACL)_, pages 1105-1116.
* Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. _Lingvisticae Investigations_, 30(1):3-26.
* Peng and Dredze (2015) Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_, pages 548-554.
* Peng et al. (2017) Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. _Trans. Assoc. for Computational Linguistics (TACL)_, 5:101-115.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_, volume 14, pages 1532-1543.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _NAACL_.
* Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In _Joint Conference on EMNLP and CoNLL-Shared Task_, pages 1-40. Association for Computational Linguistics.
* Sanh et al. (2019) Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2019. A hierarchical multi-task approach for learning embeddings from semantic tasks. _AAAI_.
* Singh et al. (2013) Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, and Andrew McCallum. 2013. Joint inference of entities, relations, and coreference. In _Proc. of the 2013 workshop on Automated knowledge base construction_, pages 1-6. ACM.
* Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. N-ary relation extraction using graph-state lstm. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_, pages 2226-2235.
* Wang and Lu (2018) Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. In _EMNLP_.
* Xu et al. (2015) Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2015. Semantic relation classification via convolutional neural networks with simple negative sampling. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_, pages 536-540 

Bishan Yang and Tom M Mitchell. 2016. Joint extraction of events and entities within a document context. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 289-299.
* Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In _Proc. Int. Conf. Learning Representations (ICLR)_.
* Zhang et al. (2017) Meishan Zhang, Yue Zhang, and Guohong Fu. 2017. End-to-end neural relation extraction with global optimization. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_, pages 1730-1740.
* Zhang et al. (2018) Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In _Proc. Conf. Empirical Methods Natural Language Process. (EMNLP)_.