Venue: NAACL 2019

# GraphIE: A Graph-Based Framework for Information Extraction

 Yujie Qian\({}^{1}\), Enrico Santus\({}^{1}\), Zhijing Jin\({}^{2}\), Jiang Guo\({}^{1}\), and Regina Barzilay\({}^{1}\)

\({}^{1}\)Computer Science and Artificial Intelligence Laboratory, MIT

\({}^{2}\)Department of Computer Science, The University of Hong Kong

{yujieq, jiang_guo, regina}@csail.mit.edu, {esantus, zhijing}@mit.edu

###### Abstract

Most modern Information Extraction (IE) systems are implemented as sequential taggers and only model local dependencies. Non-local and non-sequential context is, however, a valuable source of information to improve predictions. In this paper, we introduce GraphIE, a framework that operates over a graph representing a broad set of dependencies between textual units (i.e. words or sentences). The algorithm propagates information between connected nodes through graph convolutions, generating a richer representation that can be exploited to improve word-level predictions. Evaluation on three different tasks -- namely textual, social media and visual information extraction -- shows that GraphIE consistently outperforms the state-of-the-art sequence tagging model by a significant margin.1

Footnote 1: Our code and data are available at [https://github.com/thomas0809/GraphIE](https://github.com/thomas0809/GraphIE).

## 1 Introduction

Most modern Information Extraction (IE) systems are implemented as sequential taggers. While such models effectively capture relations in the local context, they have limited capability of exploiting non-local and non-sequential dependencies. In many applications, however, such dependencies can greatly reduce tagging ambiguity, thereby improving overall extraction performance. For instance, when extracting entities from a document, various types of non-local contextual information such as co-references and identical mentions may provide valuable cues. See for example Figure 1, in which the non-local relations are crucial to discriminate the entity type of the second mention of _Washington_ (i.e. Person or Location).

Most of the prior work looking at the non-local dependencies incorporates them by constraining the output space in a structured prediction framework [35, 46, 7]. Such approaches, however, mostly overlook the richer set of structural relations in the input space. With reference to the example in Figure 1, the co-referent dependencies would not be readily exploited by simply constraining the output space, as they would not necessarily be labeled as entities (e.g. pronouns). In the attempt to capture non-local dependencies in the input space, alternative approaches define a graph that outlines the input structure and engineer features to describe it [31]. Designing effective features is however challenging, arbitrary and time consuming, especially when the underlying structure is complex. Moreover, these approaches have limited capacity of capturing node interactions informed by the graph structure.

In this paper, we propose GraphIE, a framework

Figure 1: Example of the entity extraction task with an ambiguous entity mention (i.e. “...for Washington’s request...”). Aside from the sentential forward and backward edges (green, solid) which aggregate local contextual information, non-local relations — such as the co-referent edges (red, dashed) and the identification edges (blue, dotted) — provide additional valuable information to reduce tagging ambiguity (i.e. Person or Location). Best viewed in color.



that improves predictions by automatically learning the interactions between local and non-local dependencies in the input space. Our approach integrates a graph module with the encoder-decoder architecture for sequence tagging. The algorithm operates over a graph, where nodes correspond to textual units (i.e. words or sentences) and edges describe their relations. At the core of our model, a recurrent neural network sequentially encodes local contextual representations and then the graph module iteratively propagates information between neighboring nodes using graph convolutions Kipf and Welling (2016). The learned representations are finally projected back to a recurrent decoder to support tagging at the word level.

We evaluate GraphIE on three IE tasks, namely textual, social media, and visual Aumann et al. (2006) information extraction. For each task, we provide in input a simple task-specific graph, which defines the data structure without access to any major processing or external resources. Our model is expected to learn from the relevant dependencies to identify and extract the appropriate information. Experimental results on multiple benchmark datasets show that GraphIE consistently outperforms a strong and commonly adopted sequential model (SeqIE, i.e. a bi-directional long-short term memory (BiLSTM) followed by a conditional random fields (CRF) module). Specifically, in the textual IE task, we obtain an improvement of \(0.5\%\) over SeqIE on the CoNLL03 dataset, and an improvement of \(1.4\%\) on the chemical entity extraction Krallinger et al. (2015). In the social media IE task, GraphIE improves over SeqIE by \(3.7\%\) in extracting the Education attribute from twitter users. In visual IE, finally, we outperform the baseline by \(1.2\%\).

## 2 Related Work

The problem of incorporating non-local and non-sequential context to improve information extraction has been extensively studied in the literature. The majority of methods have focused on enforcing constraints in the output space during inference, through various mechanisms such as posterior regularization or generalized expectations Finkel et al. (2005); Mann and McCallum (2010); Reichart and Barzilay (2012); Li et al. (2013); Hu et al. (2016).

Research capturing non-local dependencies in the input space have mostly relied on feature-based approaches. Roberts et al. (2008) and Swampillai and Stevenson (2011) have designed intra- and inter-sentential features based on discourse and syntactic dependencies (e.g., shortest paths) to improve relation extraction. Quirk and Poon (2017) used document graphs to flexibly represent multiple types of relations between words (e.g., syntactic, adjacency and discourse relations).

Graph-based representations can be also learned with neural networks. The most related work to ours is the graph convolutional network by Kipf and Welling (2016), which was developed to encode graph structures and perform node classification. In our framework, we adapt GCN as an intermediate module that learns non-local context, which -- instead of being used directly for classification -- is projected to the decoder to enrich local information and perform sequence tagging.

A handful of other information extraction approaches have used graph-based neural networks. Miwa and Bansal (2016) applied Tree LSTM Tai et al. (2015) to jointly represent sequences and dependency trees for entity and relation extraction. On the same line of work, Peng et al. (2017) and Song et al. (2018) introduced Graph LSTM, which extended the traditional LSTM to graphs by enabling a varied number of incoming edges at each memory cell. Zhang et al. (2018) exploited graph convolutions to pool information over pruned dependency trees, outperforming existing sequence and dependency-based neural models in a relation extraction task. These studies differ from ours in several respects. First, they can only model word-level graphs, whereas our framework can learn non-local context either from word- or sentence-level graphs, using it to reduce ambiguity during tagging at the word level. Second, all these studies achieved improvements only when using dependency trees. We extend the graph-based approach to validate the benefits of using other types of relations in a broader range of tasks, such as reference in named entity recognition, _followed-by_ link in social media, and layout structure in visual information extraction.

## 3 Problem Definition

We formalize information extraction as a sequence tagging problem. Rather than simply modeling inputs as sequences, we assume there exists a graph structure in the data that can be exploited to cap 

ture non-local and non-sequential dependencies between textual units, namely words or sentences.

We consider the input to be a set of sentences \(S=\{s_{1},\ldots,s_{N}\}\) and an auxiliary graph \(G=(V,E)\), where \(V=\{v_{1},\ldots,v_{M}\}\) is the node set and \(E\subset V\times V\) is the edge set. Each sentence is a sequence of words. We consider two different designs of the graph:

1. [label=(0)]
2. _sentence-level graph_, where each node is a sentence (i.e. \(M=N\)), and the edges encode sentence dependencies;
3. _word-level graph_, where each node is a word (i.e. \(M\) is the number of words in the input), and the edges connect pairs of words, such as co-referent tokens.

The edges \(e_{i,j}=(v_{i},v_{j})\) in the graph can be either directed or undirected. Multiple edge types can also be defined to capture different structural factors underlying the task-specific input data.

We use the BIO (Begin, Inside, Outside) tagging scheme in this paper. For each sentence \(s_{i}=(w^{(i)}_{1},w^{(i)}_{2},\dots,w^{(i)}_{k})\),2 we sequentially tag each word as \(\mathbf{y}_{i}=(y^{(i)}_{1},y^{(i)}_{2},\dots,y^{(i)}_{k})\).

Footnote 2: While sentences may have different lengths, for notation simplicity we use a single variable \(k\).

## 4 Method

GraphIE jointly learns local and non-local dependencies by iteratively propagating information between node representations. Our model has three components:

* an _encoder_, which generates local context-aware hidden representations for the textual unit (i.e. word or sentence, depending on the task) with a recurrent neural network;
* a _graph module_, which captures the graph structure, learning non-local and non-sequential dependencies between textual units;
* a _decoder_, which exploits the contextual information generated by the graph module to perform labelling at the word level.

Figure 2: GraphLE framework: (a) an overview of the framework; (b) architecture for _sentence-level graph_, where each sentence is encoded to a node vector and fed into the graph module, and the output of the graph module is used as the initial state of the decoder; (c) architecture for _word-level graph_, where the hidden state for each word of the encoder is taken as the input node vector of the graph module, and then the output is fed into the decoder.

 

Figure 2 illustrates the overview of GraphIE and the model architectures for both sentence- and word-level graphs. In the following sections, we first introduce the case of the sentence-level graph, and then we explain how to adapt the model for the word-level graph.

### Encoder

In GraphIE, we first use an encoder to generate text representations. Given a sentence \(s_{i}=(w^{(i)}_{1},w^{(i)}_{2},\ldots,w^{(i)}_{k})\) of length \(k\), each word \(w^{(i)}_{t}\) is represented by a vector \(\mathbf{x}^{(i)}_{t}\), which is the concatenation of its word embedding and a feature vector learned with a character-level convolutional neural network [35]. We encode the sentence with a recurrent neural network (RNN), defining it as

\[\mathbf{h}^{(i)}_{1:k}=\texttt{RNN}\left(\mathbf{x}^{(i)}_{1:k}~{};\mathbf{0}, \Theta_{\text{enc}}\right), \tag{1}\]

where \(\mathbf{x}^{(i)}_{1:k}\) denotes the input sequence \([\mathbf{x}^{(i)}_{1},\cdots,\mathbf{x}^{(i)}_{k}]\), \(\mathbf{h}^{(i)}_{1:k}\) denotes the hidden states \([\mathbf{h}^{(i)}_{1},\cdots,\mathbf{h}^{(i)}_{k}]\), \(\mathbf{0}\) indicates the initial hidden state is zero, and \(\Theta_{\text{enc}}\) represents the encoder parameters. We implement the RNN as a bi-directional LSTM [9], and encode each sentence independently.

We obtain the sentence representation for \(s_{i}\) by averaging the hidden states of its words, i.e. \(\texttt{Enc}(s_{i})=\frac{1}{k}\left(\sum_{t=1}^{k}\mathbf{h}^{(i)}_{t}\right)\). The sentence representations are then fed into the graph module.

### Graph Module

The graph module is designed to learn the non-local and non-sequential information from the graph. We adapt the graph convolutional network (GCN) to model the graph context for information extraction.

Given the sentence-level graph \(G=(V,E)\), where each node \(v_{i}\) (i.e. sentence \(s_{i}\)) has the encoding \(\texttt{Enc}(s_{i})\) capturing its local information, the graph module enriches such representation with neighbor information derived from the graph structure.

Our graph module is a GCN which takes as input the sentence representation, i.e. \(\textbf{g}^{(0)}_{i}=\texttt{Enc}(s_{i})\), and conducts graph convolution on every node, propagating information between its neighbors, and integrating such information into a new hidden representation. Specifically, each layer of GCN has two parts. The first gets the information of each node from the previous layer, i.e.

\[\mathbf{\alpha}^{(l)}_{i}=\mathbf{W}^{(l)}_{v}~{}\textbf{g}^{(l-1)}_{i}, \tag{2}\]

where \(\mathbf{W}^{(l)}_{v}\) is the weight to be learned. The second aggregates information from the neighbors of each node, i.e. for node \(v_{i}\), we have

\[\mathbf{\beta}^{(l)}_{i}=\frac{1}{d(v_{i})}\cdot\mathbf{W}^{(l)}_{e}\Bigg{(}\sum_ {e_{ij}\in E}\textbf{g}^{(l-1)}_{j}\Bigg{)}, \tag{3}\]

where \(d(v_{i})\) is the degree of node \(v_{i}\) (i.e. the number of edges connected to \(v_{i}\)) and is used to normalize \(\mathbf{\beta}^{(l)}_{i}\), ensuring that nodes with different degrees have representations of the same scale.3 In the simplest case, where the edges in the graph are undirected and have the same type, we use the same weight \(\mathbf{W}^{(l)}_{e}\) for all of them. In a more general case, where multiple edge types exist, we expect them to have different impacts on the aggregation. Thus, we model these edge types with different weights in Eq. 3, similar to the relational GCN proposed by Schlichtrull et al. (2018). When edges are directed, i.e. edge \(e_{i,j}\) is different from \(e_{j,i}\), the propagation mechanism should mirror such difference. In this case, we consider directed edges as two types of edges (forward and backward), and use different weights for them.

Footnote 3: We choose this simple normalization strategy instead of the two-sided normalization in Kipf and Welling (2016), as it performs better in the experiments. The same strategy is also adopted by Zhang et al. (2018).

Finally, \(\mathbf{\alpha}^{(l)}_{i}\) and \(\mathbf{\beta}^{(l)}_{i}\) are combined to obtain the representation at the \(l\)-th layer,

\[\textbf{g}^{(l)}_{i}=\sigma\left(\mathbf{\alpha}^{(l)}_{i}+\mathbf{\beta}^{(l)}_{i}+b^ {(l)}\right), \tag{4}\]

where \(\sigma(\cdot)\) is the non-linear activation function, and \(b^{(l)}\) is a bias parameter.

Because each layer only propagates information between directly connected nodes, we can stack multiple graph convolutional layers to get a larger receptive field, i.e. each node can be aware of more distant neighbors. After \(L\) layers, for each node \(v_{i}\) we obtain a contextual representation, \(\texttt{GCN}(s_{i})=\textbf{g}^{(L)}_{i}\), that captures both local and non-local information.

### Decoder

To support tagging, the learned representation is propagated to the decoder.

 

In our work, the decoder is instantiated as a BiLSTM+CRF tagger (Lample et al., 2016). The output representation of the graph module, \({\tt GCN}(s_{i})\), is split into two vectors of the same length, which are used as the initial hidden states for the forward and backward LSTMs, respectively. In this way, the graph contextual information is propagated to each word through the LSTM. Specifically, we have

\[\mathbf{z}^{(i)}_{1:k}={\tt RNN}\left(\mathbf{h}^{(i)}_{1:k}~{};{\tt GCN}(s_{ i}),\Theta_{\text{dec}}\right), \tag{5}\]

where \(\mathbf{h}^{(i)}_{1:k}\) are the output hidden states of the encoder, \({\tt GCN}(s_{i})\) represents the initial state, and \(\Theta_{\text{dec}}\) is the decoder parameters. A simpler way to incorporate the graph representation into the decoder is concatenating with its input, but the empirical performance is worse than using as the initial state.

Finally, we use a CRF layer (Lafferty et al., 2001) on top of the BiLSTM to perform tagging,

\[\mathbf{y}^{*}_{i}=\operatorname*{arg\,max}_{\mathbf{y}\in\mathbf{Y}_{k}}p \left(\mathbf{y}~{}|~{}\mathbf{z}^{(i)}_{1:k}~{};\Theta_{\text{crf}}\right), \tag{6}\]

where \(\mathbf{Y}_{k}\) is the set of all possible tag sequences of length \(k\), and \(\Theta_{\text{crf}}\) represents the CRF parameters, i.e. transition scores of tags. CRF combines the local predictions of BiLSTM and the transition scores to model the joint probability of the tag sequence.4

Footnote 4: In GraphIE, the graph module models the input space structure, i.e. the dependencies between textual units (i.e. sentences or words), and the final CRF layer models the sequential connections of the output tags. Even though loops may exist in the input graph, CRF operates sequentially, thus the inference is tractable.

### Adaptation to Word-level Graphs

GraphIE can be easily adapted to model word-level graphs. In such case, the nodes represent words in the input, i.e. the number of nodes \(M\) equals the total number of words in the \(N\) sentences. At this point, each word's hidden state in the encoder can be used as the input node vector \(\mathbf{g}^{(0)}_{i}\) of the graph module. GCN can then conduct graph convolution on the word-level graph and generate graph-contextualized representations for the words. Finally, the decoder directly operates on the GCN's outputs, i.e. we change the BiLSTM decoder to

\[\mathbf{z}^{(i)}_{1:k}={\tt RNN}\left(\begin{bmatrix}{\tt GCN}(w^{(i)}_{1}), \cdots,{\tt GCN}(w^{(i)}_{k})\end{bmatrix};\mathbf{0},\Theta_{\text{dec}} \right),\]

where \({\tt GCN}(w^{(i)}_{t})\) is the GCN output for word \(w^{(i)}_{t}\). In this case, the BiLSTM initial states are set to the default zero vectors. The CRF layer remains unchanged.

As it can be seen in Figure 2(c), the word-level graph module differs from the sentence-level one because it directly takes the word representations from the encoder and feeds its output to the decoder. In sentence-level graph, the GCN operates on sentence representations, which are then used as the initial states of the decoder BiLSTM.

## 5 Experimental Setup

We evaluate the model on three tasks, including two traditional IE tasks, namely textual information extraction and social media information extraction, and an under-explored task -- _visual information extraction_. For each of these tasks, we created a simple task-specific graph topology, designed to easily capture the underlying structure of the input data without any major processing. Table 1 summarizes the three tasks.

### Task 1: Textual Information Extraction

In this task, we focus on named entity recognition at discourse level (DiscNER). In contrast to traditional sentence-level NER (SentNER), where sentences are processed independently, in DiscNER, long-range dependencies and constraints across sentences have a crucial role in the tagging process. For instance, multiple mentions of the same entity are expected to be tagged consistently in the same discourse. Here we propose to use this (soft) constraint to improve entity extraction.

DatasetWe conduct experiments on two NER datasets: the CoNLL-2003 dataset (CoNLL03) (Tjong et al., 2003), and the Chemdner dataset for chemical entity extraction (Krallinger et al., 2015). We follow the standard split of each corpora. Statistics are shown in Table 2.

Graph ConstructionIn this task, we use a word-level graph where nodes represent words. We create two types of edges for each document:

* _Local edges_: forward and backward edges are created between neighboring words in each sentence, allowing local contextual information to be utilized.
* _Non-local edges_: re-occurrences of the same token other than stop words are connected, so 

that information can be propagated through, encouraging global consistency of tagging.5

Footnote 4: Note that other non-local relations such as co-references (cf. the example in Figure 1) may be used for further improvement. However, these relations require additional resources to obtain, and we leave them to future work.

### Task 2: Social Media Information Extraction

Social media information extraction refers to the task of extracting information from users' posts in online social networks Benson et al. (2011); Li et al. (2014). In this paper, we aim at extracting _education_ and _job_ information from users' tweets. Given a set of tweets posted by a user, the goal is to extract mentions of the organizations to which they belong. The fact that the tweets are short, highly contextualized and show special linguistic features makes this task particularly challenging.

DatasetWe construct two datasets, Education and Job, from the Twitter corpus released by li2014multi. The original corpus contains millions of tweets generated by \(\approx 10\) thousand users, where the _education_ and _job_ mentions are annotated using distant supervision Mintz et al. (2009). We sample the tweets from each user, maintaining the ratio between positive and negative posts.6 The obtained Education dataset consists of \(443,476\) tweets generated by \(7,208\) users, and the Job dataset contains \(176,043\) tweets generated by \(1,772\) users. Dataset statistics are reported in Table 3.

Footnote 6: Positive and negative refer here to whether or not the _education_ or _job_ mention is present in the tweet.

The datasets are both split in 60% for training, 20% for development, and 20% for testing. We perform 5 different random splits and report the average results.

Graph ConstructionWe construct the graph as _ego-networks_Leskovec and Mcauley (2012), i.e. when we extract information about one user, we consider the subgraph formed by the user and his/her direct neighbors. Each node corresponds to a Twitter user, who is represented by the set of posted tweets.7 Edges are defined by the _followedby_ link, under the assumption that connected users are more likely to come from the same university or company. An example of the social media graph is reported in the appendices.

Footnote 7: As each node is a set of tweets posted by the user, we encode every tweet with the encoder, and then average them to obtain the node representation. In the decoding phase, the graph module’s output is fed to the decoder for each tweet.

### Task 3: Visual Information Extraction

Visual information extraction refers to the extraction of attribute values from documents formatted in various layouts. Examples include invoices and forms, whose format can be exploited to infer valuable information to support extraction.

DatasetThe corpus consists of 25,200 Adverse Event Case Reports (AACR) recording drug-related side effects. Each case contains an average of 9 pages. Since these documents are produced by multiple organizations, they exhibit large variability in the layout and presentation styles (e.g.

\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Evaluation Task** & **Graph Type** & **Node** & **Edge** \\ \hline Textual IE & word-level & word & 1. non-local consistency (identical mentions) \\  & & 2. local sentential forward and backward \\ \hline Social Media IE & sentence-level & user’s tweets & _followed-by_ \\ \hline Visual IE & sentence-level & text box & spatial layout (horizontal and vertical) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparisons of graph structure in the three IE tasks used for evaluation.

\begin{table}
\begin{tabular}{c c r} \hline \hline Dataset & Train & Dev & Test \\ \hline CoNLL03 & \#doc & 946 & 216 & 231 \\  & \#sent & 14,987 & 3,466 & 3,684 \\ \hline Chemdner & \#doc & 3,500 & 3,500 & 3,000 \\  & \#sent & 30,739 & 30,796 & 26,399 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of the CoNLL03 and the Chemdner datasets (Task 1).

\begin{table}
\begin{tabular}{r c c} \hline \hline  & Education & Job \\ \hline Users & 7,208 & 1,772 \\ Edges & 11,167 & 3,498 \\ Positive Tweets & 49,793 & 3,694 \\ Negative Tweets & 393,683 & 172,349 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Statistics of the Education and Job datasets (Task 2).

 

text, table, etc.).8 The collection is provided with a separate human-extracted ground truth database that is used as a source of distant supervision.

Footnote 7: This dataset cannot be shared for patient privacy and proprietary issues.

Our goal is to extract eight attributes related to the patient, the event, the drug and the reporter (cf. Table 6 for the full list). Attribute types include dates, words and phrases -- which can be directly extracted from the document.

The dataset is split in 50% cases for training, 10% for development, and 40% for testing.

Graph ConstructionWe first turn the PDFs to text using PDFMiner,9 which provides words along with their positions in the page (i.e. bounding-box coordinates). Consecutive words are then geometrically joined into _text boxes_. Each text box is considered as a "sentence" in this task, and corresponds to a _node_ in the graph.

Footnote 9: [https://euske.github.io/pdfminer/](https://euske.github.io/pdfminer/)

Since the page layout is the major structural factor in these documents, we work on page-by-page basis, i.e. each page corresponds to a graph. The _edges_ are defined to horizontally or vertically connect _nodes_ (text boxes) that are close to each other (i.e. when the overlap of their bounding boxes, in either the vertical or horizontal direction, is over 50%). Four types of edge are considered: left-to-right, right-to-left, up-to-down, and down-to-up. When multiple nodes are aligned, only the closest ones are connected. An example of visual document graph is reported in the appendices.

### Baseline and Our Method

We implement a two-layer BiLSTM with a conditional random fields (CRF) tagger as the sequential baseline (SeqIE). This architecture and its variants have been extensively studied and demonstrated to be successful in previous work on information extraction Lample et al. (2016); Ma and Hovy (2016). In the textual IE task (Task 1), our baseline is shown to obtain competitive results with the state-of-the-art method in the CONLL03 dataset. In the visual IE task (Task 3), in order to further increase the competitiveness of the baseline, we sequentially concatenate the horizontally aligned text boxes, therefore fully modeling the horizontal edges of the graph.

Our baseline shares the same encoder and decoder architecture with GraphIE, but without the graph module. Both architectures have similar computational cost. In Task 1, we apply GraphIE with word-level graph module (cf. Figure 2(c)), and in Task 2 and Task 3, we apply GraphIE with sentence-level graph module (cf. Figure 2(b)).

### Implementation Details

The models are trained with AdamKingma and Ba (2014) to minimize the CRF objective. For regularization, we choose dropout with a ratio of 0.1 on both the input word representation and the hidden layer of the decoder. The learning rate is set to 0.001. We use the development set for early-stopping and the selection of the best performing hyperparameters. For CharCNN, we use 64-dimensional character embeddings and 64 filters of width 2 to 4 Kim et al. (2016). The 100-dimensional pretrained GloVe word embeddings Pennington et al. (2014) are used in Task 1 and 2, and 64-dimensional randomly initialized word embeddings are used in Task 3. We use a two-layer GCN in Task 1, and a one-layer GCN in Task 2 and Task 3. The encoder and decoder BiLSTMs have the same dimension as the graph convolution layer. In Task 3, we concatenate a positional encoding to each text box's representation by transforming its bounding box coordinates to a vector of length 32, and then applying a \(\tanh\) activation.

## 6 Results

### Task 1: Textual Information Extraction

Table 4 describes the NER accuracy on the CoNLL03 Tjong et al. (2003) and the ChemderrKramlinger et al. (2015) datasets.

For CoNLL03, we list the performance of existing approaches. Our baseline SeqIE obtains competitive scores compared to the best methods. The fact that GraphIE significantly outperforms

\begin{table}
\begin{tabular}{l l c} \hline \hline Dataset & Model & F1 \\ \hline \multirow{4}{*}{CoNLL03} & Lample et al. (2016) & 90.94 \\  & Ma and Hovy (2016) & 91.21 \\  & Ye and Ling (2018) & 91.38 \\  & SeqIE & 91.16 \\  & GraphHE & **91.74*** \\ \hline \multirow{4}{*}{Chemdner} & Krallinger et al. (2015) & 87.39 \\  & SeqIE & 88.28 \\ \cline{1-1}  & GraphHE & **89.71*** \\ \hline \hline \end{tabular}
\end{table}
Table 4: NER accuracy on the CoNLL03 and the Chemdner datasets (Task 1). Scores for our methods are the average of 5 runs. * indicates statistical significance of the improvement over SeqIE (\(p<0.01\)).

 

it, highlights once more the importance of modeling non-local and non-sequential dependencies and confirms that our approach is an appropriate method to achieve this goal.10

Footnote 10: We achieve the best reported performance among methods not using the recently introduced ELMo Peters et al. (2018) and BERT Devlin et al. (2018), which are pretrained on extra-large corpora and computationally demanding.

For Chemdner, we show the best performance reported in Krallinger et al. (2015), obtained with a feature-based method. Our baseline outperforms the feature-based method, and GraphIE further improves the performance by \(1.4\%\).

AnalysisTo understand the advantage of GraphIE, we first investigate the importance of graph structure to the model. As shown in Figure 3, using random connections clearly hurts the performance, bringing down the F1 score of GraphIE from 95.12% to 94.29%. It indicates that the task-specific graph structures introduce beneficial inductive bias. Trivial feature augmentation also does not work well, confirming the necessity of learning the graph embedding with GCN.

We further conduct error analysis on the test set to validate our motivation that GraphIE resolves tagging ambiguity by encouraging consistency among identical entity mentions (cf. Figure 1). Here we examine the word-level tagging accuracy. We define the words that have more than one possible tags in the dataset as _ambiguous_. We find that among the \(1.78\%\) tagging errors of SeqIE, \(1.16\%\) are _ambiguous_ and \(0.62\%\) are _unambiguous_. GraphIE reduces the error rate to \(1.67\%\), with \(1.06\%\) to be _ambiguous_ and \(0.61\%\)_ unambiguous_. We can see that most of the error reduction indeed attributes to the _ambiguous_ words.

### Task 2: Social Media Information Extraction

Table 5 shows the results for the social media information extraction task. We first report a simple dictionary-based method as a baseline. Neural IE models achieve much better performance, showing that meaningful patterns are learned by the models rather than simply remembering the entities in the training set. The proposed GraphIE outperforms SeqIE in both the Education and Job datasets, and the improvements are more significant for the Education dataset (\(3.7\%\) versus \(0.3\%\)). The reason for such difference is the variance in the affinity scores Mislove et al. (2010) between the two datasets. Li et al. (2014) underline that affinity value for Education is \(74.3\) while for Job it is only \(14.5\), which means that in the datasets neighbors are \(5\) times more likely to have studied in the same university than worked in the same company. We can therefore expect that a model like GraphIE, which exploits neighbors' information, obtains larger advantages in a dataset characterized by higher affinity.

### Task 3: Visual Information Extraction

Table 6 shows the results in the visual information extraction task. GraphIE outperforms the SeqIE baseline in most attributes, and achieves \(1.2\%\) improvement in the mirco average F1 score. It confirms that the benefits of using layout graph structure in visual information extraction.

The extraction performance varies across the attributes, ranging from \(61.4\%\) for _Drug Name_ to

Figure 3: Analysis on the CoNLL03 dataset. We compare with two alternative designs: (1) _random connection_, where we replace the constructed graph by a random graph with the same number of edges; (2) _feature augmentation_, where we use the average embedding of each node and its neighbors as the input to the decoder, instead of the GCN which has additional parameters. We report F1 scores on the development set.

 

\(95.8\%\) for _Patient Birthday_ (similar variations are visible in the baseline). Similarly, the gap between GraphIE and SeqIE varies in relation to the attributes, ranging between \(-0.5\%\) in _Patient Birthday_ and \(2.4\%\) in _Event_.

In the ablation test described in Table 7, we can see the contribution of: using separate weights for different edge types (\(+0.8\%\)), horizontal edges (\(+3.1\%\)), vertical edges (\(+5.4\%\)), and CRF (\(+5.7\%\)).

GeneralizationWe also assess GraphIE's capacity of dealing with unseen layouts through an extra analysis. From our dataset, we sample \(2,000\) reports containing the three most frequent templates, and train the models on this subset. Then we test all models in two settings: 1) _seen templates_, consisting of \(1,000\) additional reports in the same templates used for training; and 2) _unseen templates_, consisting of \(1,000\) reports in two new template types.

The performance of GraphIE and SeqIE is reported in Figure 4. Both models achieve good results on _seen templates_, with GraphIE still scoring \(2.8\%\) higher than SeqIE. The gap becomes even larger when our model and the sequential one are tested on _unseen templates_ (i.e. \(20.3\%\)), demonstrating that by explicitly modeling the richer structural relations, GraphIE achieves better generalizability.

## 7 Conclusions

We introduced GraphIE, an information extraction framework that learns local and non-local contextual representations from graph structures to improve predictions. The system operates over a task-specific graph topology describing the underlying structure of the input data. GraphIE jointly models the node (i.e. textual units, namely words or sentences) representations and their dependencies. Graph convolutions project information through neighboring nodes to finally support the decoder during tagging at the word level.

We evaluated our framework on three IE tasks, namely textual, social media and visual information extraction. Results show that it efficiently models non-local and non-sequential context, consistently enhancing accuracy and outperforming the competitive SeqIE baseline (i.e. BiLSTM+CRF).

Future work includes the exploration of automatically learning the underlying graphical structure of the input data.

## Acknowledgments

We thank the MIT NLP group and the reviewers for their helpful comments. This work is supported by MIT-IBM Watson AI Lab. Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the authors, and do not necessarily reflect the views of the funding organizations.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{3}{c}{SeqIE} & \multicolumn{3}{c}{GraphIE} \\ \cline{2-6} Attribute & P & R & F1 & P & R & F1 \\ \hline _P. Initiits_ & 93.5 & 92.4 & **92.9** & 93.6 & 91.9 & 92.8 \\ _P. Age_ & 94.0 & 91.6 & 92.8 & 94.8 & 91.1 & **92.9** \\ _P. Birthday_ & 96.6 & 96.0 & **96.3** & 96.9 & 94.7 & 95.8 \\ _Drug Name_ & 71.2 & 51.2 & 59.4 & 78.5 & 50.4 & **61.4** \\ _Event_ & 62.6 & 65.2 & 63.9 & 64.1 & 68.7 & **66.3** \\ _R. First Name_ & 78.3 & 95.7 & 86.1 & 79.5 & 95.9 & **86.9** \\ _R. Last Name_ & 84.5 & 68.4 & 75.6 & 85.6 & 68.2 & **75.9** \\ _R. City_ & 88.9 & 65.4 & 75.4 & 92.1 & 66.3 & **77.1** \\ \hline Avg. (macro) & 83.7 & 78.2 & 80.3 & 85.7 & 78.4 & **81.1\({}^{\dagger}\)** \\ Avg. (micro) & 78.5 & 73.8 & 76.1 & 80.3 & 74.6 & **77.3\({}^{\ddagger}\)** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Extraction accuracy on the AECH dataset (Task 3). Scores are the average of 5 runs. _P._ is the abbreviation for _Patient_, and _R._ for _Reporter_. \(\dagger\) indicates statistical significance of the improvement over SeqIE (\(p<0.05\)).

Figure 4: Micro average F1 scores tested on _seen_ and _unseen_ templates (Task 3).

 



## References

* Aumann et al. (2006) Yonatan Aumann, Ronen Feldman, Yair Liberzon, Benjamin Rosenfeld, and Jonathan Schler. 2006. Visual information extraction. _Knowl. Inf. Syst._, 10(1):1-15.
* Benson et al. (2011) Edward Benson, Aria Haghighi, and Regina Barzilay. 2011. Event discovery in social media feeds. In _Proceedings of ACL_, pages 389-398. ACL.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In _Proceedings of ACL_, pages 363-370. ACL.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hu et al. (2016) Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing deep neural networks with logic rules. In _Proceedings of ACL_, pages 2410-2420.
* Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In _Proceedings of AAAI_, pages 2741-2749. AAAI Press.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. _arXiv preprint arXiv:1609.02907_.
* Krallinger et al. (2015) Martin Krallinger, Florian Leitner, Obudulia Rabal, Miguel Vazquez, Julen Oyarzabal, and Alfonso Valencia. 2015. Chemdner: The drugs and chemical names extraction challenge. _Journal of cheminformatics_, 7(1):S1.
* Lafferty et al. (2001) John D Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of ICML_, pages 282-289.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of NAACL-HLT_, pages 260-270, San Diego, California. ACL.
* Leskovec and Mcauley (2012) Jure Leskovec and Julian J Mcauley. 2012. Learning to discover social circles in ego networks. In _NIPS_, pages 539-547.
* Li et al. (2014) Jiwei Li, Alan Ritter, and Eduard Hovy. 2014. Weakly supervised user profile extraction from twitter. In _Proceedings of ACL_, volume 1, pages 165-174.
* Li et al. (2013) Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In _Proceedings of ACL_, volume 1, pages 73-82.
* Ma and Hovy (2016)Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _Proceedings of ACL_, pages 1064-1074, Berlin, Germany. ACL.
* Mann and McCallum (2010) Gideon S Mann and Andrew McCallum. 2010. Generalized expectation criteria for semi-supervised learning with weakly labeled data. _Journal of Machine Learning Research_, 11(Feb):955-984.
* Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In _Proceedings of ACL_, pages 1003-1011. ACL.
* Mislove et al. (2010) Alan Mislove, Bilmal Viswanath, Krishna P Gummadi, and Peter Druschel. 2010. You are who you know: inferring user profiles in online social networks. In _Proceedings of the 3rd ACM International Conference on Web Search and Data Mining_, pages 251-260. ACM.
* Miwa and Bansal (2016) Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. _arXiv preprint arXiv:1601.00770_.
* Peng et al. (2017)Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph lstms. _TACL_, 5:101-115.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of EMNLP_, pages 1532-1543.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of NAACL-HLT_, volume 1, pages 2227-2237.
* Quirk and Poon (2017) Chris Quirk and Hoifung Poon. 2017. Distant supervision for relation extraction beyond the sentence boundary. In _Proceedings of ACL_, volume 1, pages 1171-1182.
* Reichart and Barzilay (2012) Roi Reichart and Regina Barzilay. 2012. Multi event extraction guided by global constraints. In _Proceedings of NAACL-HLT_, pages 70-79. ACL.
* Roberts et al. (2008) Angus Roberts, Robert Gaizauskas, and Mark Hepple. 2008. Extracting clinical relationships from patient narratives. In _Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing_, pages 10-18. ACL

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In _European Semantic Web Conference_, pages 593-607. Springer.
* Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. N-ary relation extraction using graph-state lstm. In _Proceedings of EMNLP_, pages 2226-2235.
* Swampillai and Stevenson (2011) Kumutha Swampillai and Mark Stevenson. 2011. Extracting relations within and across sentences. In _Proceedings of the International Conference Recent Advances in Natural Language Processing_, pages 25-32.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. _arXiv preprint arXiv:1503.00075_.
* Tjong et al. (2003) Kim Sang Tjong, F Erik, and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In _Proceedings of NAACL-HLT_, pages 142-147. ACL.
* Ye and Ling (2018) Zhixiu Ye and Zhen-hua Ling. 2018. Hybrid semi-markov crf for neural sequence labeling. In _Proceedings of ACL_, pages 235-240.
* Zhang et al. (2018) Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In _Proceedings of EMNLP_.

## Appendix A Appendices

We show some examples of the constructed graphs for different information extraction tasks.

### Social Media Information Extraction

### Visual Information Extraction

Figure 5: Mock-up example of Social Media Information Extraction (Task 2). Nodes are represented as users and edges are _follow-by_ relations.

Figure 6: Mock-up example of Visual Information Extraction (Task 3). The two forms have different layouts. Graphical dependencies are shown as green lines connecting text in blue bounding-boxes.