SlowGuess/NAACL_2021_OCR huggingface dataset:
Minh Van Nguyen, Viet Duc Lai and Thien Huu Nguyen
University of Oregon, Eugene, Oregon, USA
{minhnv, vietl, thien}@cs.uoregon.edu
Existing works on information extraction (IE) have mainly solved the four main tasks separately (entity mention recognition, relation extraction, event trigger detection, and argument extraction), thus failing to benefit from inter-dependencies between tasks. This paper presents a novel deep learning model to simultaneously solve the four tasks of IE in a single model (called FourIE). Compared to few prior work on jointly performing four IE tasks, FourIE features two novel contributions to capture inter-dependencies between tasks. First, at the representation level, we introduce an interaction graph between instances of the four tasks that is used to enrich the prediction representation for one instance with those from related instances of other tasks. Second, at the label level, we propose a dependency graph for the information types in the four IE tasks that captures the connections between the types expressed in an input sentence. A new regularization mechanism is introduced to enforce the consistency between the golden and predicted type dependency graphs to improve representation learning. We show that the proposed model achieves the state-of-the-art performance for joint IE on both monolingual and multilingual learning settings with three different languages.
Information Extraction (IE) is an important and challenging task in Natural Language Processing (NLP) that aims to extract structured information from unstructured texts. Following the terminology for IE in the popular ACE 2005 program (Walker et al., 2006), we focus on four major IE tasks in this work: entity mention extraction (EME), relation extraction (RE), event trigger detection (ETD), and event argument extraction (EAE).
Given an input sentence, a vast majority of prior work has solved the four tasks in IE independently at both instance and task levels (called independent
Figure 1: A sentence example with the annotations for the four IE tasks. Blue words correspond to entity mentions while red words are event triggers. Also, orange edges represent relations while green edges indicate argument roles.
prediction models). First, at the instance level, each IE task often requires predictions/classifications for multiple instances in a single input sentence. For instance, in RE, one often needs to predict relations for every pair of entity mentions (called relation instances) in the sentence while multiple word spans in the sentence can be viewed as multiple instances where event type predictions have to be made in ETD (trigger instances). As such, most prior work on IE has performed predictions for instances in a sentence separately by treating each instance as one example in the dataset (Zhou et al., 2005; Nguyen and Grishman, 2015a; Santos and Guimaraes, 2015; Chen et al., 2015; Nguyen and Grishman, 2015b; Lai et al., 2020). Second, at the task level, prior work on IE tends to perform the four tasks in a pipelined approach where outputs from one task are used as inputs for other tasks (e.g., EAE is followed by EME and ETD) (Li et al., 2013; Chen et al., 2015; Veyseh et al., 2020c).
Despite its popularity, the main issue of the independent prediction models is that they suffer from the error propagation between tasks and the failure to exploit the cross-task and cross-instance interdependencies within an input sentence to improve the performance for IE tasks. For instance, such systems are unable to benefit from the dependency that the Victim of a Die event has a high chance to
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 27-38 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: Overall architecture of our proposed model. At the representation level,  $\mathsf{GCN}^{inst}$  is used to enrich the representations for instances of the four tasks. At the label level,  $\mathsf{GCN}^{type}$  is responsible for capturing the connections between the types in the dependency graphs, thus helping the model learn the structural difference between the gold graph  $\mathbf{G}^{gold}$  and the predicted graph  $\mathbf{G}^{pred}$ .
also be the Victim of an Attack event in the same sentence (i.e., type or label dependencies). To address these issues, some prior work has explored joint inference models where multiple tasks of IE are performed simultaneously for all task instances in a sentence, using both feature-based models (Roth and Yih, 2004; Li et al., 2013; Miwa and Sasaki, 2014; Yang and Mitchell, 2016) and recent deep learning models (Miwa and Bansal, 2016; Zhang et al., 2019). However, such prior work has mostly considered joint models for a subset of the four IE tasks (e.g., EME+RE or ETD+EAE), thus still suffering from the error propagation issue (with the missing tasks) and failing to fully exploit potential inter-dependencies between the four tasks. To this end, this work aims to design a single model to simultaneously solve the four IE tasks for each input sentence (joint four-task IE) to address the aforementioned issues of prior joint IE work.
Few recent work has considered joint four-task IE, using deep learning to produce state-of-the-art (SOTA) performance for the tasks (Wadden et al., 2019; Lin et al., 2020). However, there are still two problems that hinder further improvement of such models. First, at the instance level, an important component of deep learning models for joint IE involves the representation vectors of the instances that are used to perform the corresponding prediction tasks for IE in an input sentence (called
predictive instance representations). For joint four-task IE, we argue that there are inter-dependencies between predictive representation vectors of related instances for the four tasks that should be modeled to improve the performance for IE. For instance, the entity type information encoded in the predictive representation vector for an entity mention can constrain the argument role that the representation vector for a related EAE instance (e.g., involving the same entity mention and some event trigger in the same sentence) should capture and vice versa. As such, prior work for joint four-task IE has only computed predictive representation vectors for instances of the tasks independently using shared hidden vectors from some deep learning layer (Wadden et al., 2019; Lin et al., 2020). Although this shared mechanism helps capture the interaction of predictive representation vectors to some extent, it fails to explicitly present the connections between related instances of different tasks and encode them into the representation learning process. Consequently, to overcome this issue, we propose a novel deep learning model for joint four-task IE (called FourIE) that creates a graph structure to explicitly capture the interactions between related instances of the four IE tasks in a sentence. This graph will then be consumed by a graph convolutional network (GCN) (Kipf and Welling, 2017; Nguyen and Grishman, 2018) to enrich the representation vector
for an instance with those from the related (neighboring) instances for IE.
Second, at the task level, existing joint four-task models for IE have only exploited the cross-task type dependencies in the decoding step to constrain predictions for the input sentence (by manually converting the type dependency graphs of the input sentence into global feature vectors for scoring the predictions in the beam search-based decoding) (Lin et al., 2020). The knowledge from cross-task type dependencies thus cannot contribute to the training process of the IE models. This is unfortunate as we expect that deeper integration of this knowledge into the training process could provide useful information to enhance representation learning for IE tasks. To this end, we propose to use the knowledge from cross-task type dependencies to obtain an additional training signal for each sentence to directly supervise our joint four-task IE model. In particular, our motivation is that the types expressed in a sentence for the four IE tasks can be organized into a dependency graph between the types (global type dependencies for the sentence). As such, in order for a joint model to perform well, the type dependency graph generated by its predictions for a sentence should be similar to the dependency graph obtained from the golden types (i.e., a global type constraint on the predictions in the training step). A novel regularization term is thus introduced into the training loss of our joint model to encode this constraint, employing another GCN to learn representation vectors for the predicted and golden dependency graphs to facilitate the graph similarity promotion. To our knowledge, this is the first work that employs global type dependencies to regularize joint models for IE.
Finally, our extensive experiments demonstrate the effectiveness of the proposed model on benchmark datasets in three different languages (e.g., English, Chinese, and Spanish), leading to state-of-the-art performance on different settings.
Problem Statement: The joint four-task IE problem in this work takes a sentence as the input and aims to jointly solve four tasks EAE, ETD, RE, and EAE using an unified model. As such, the goal of EME is to detect and classify entity mentions (names, nominals, pronouns) according to a set of predefined (semantic) entity types (e.g., Person). Similarly, ETD seeks to identify and classify event
triggers (verbs or normalization) that clearly evoke an event in some predefined set of event types (e.g., Attack). Note that event triggers can involve multiple words. For RE, its concern is to predict the semantic relationship between two entity mentions in the sentence. Here, the set of relations of interest is also predefined and includes a special type of None to indicate no-relation. Finally, in EAE, given an event trigger, the systems need to predict the roles (also in a predefined set with a special type None) that each entity mention plays in the corresponding event. Entity mentions are thus also called event argument candidates in this work. Figure 1 presents a sentence example where the expected outputs for each IE task are illustrated.
Graph Convolutional Networks (GCN): As GCNs are used extensively in our model, we present their computation process in this section to facilitate the discussion. Given a graph  $\mathbf{G} = (\mathbf{V},\mathbf{E})$  where  $\mathbf{V} = \{v_{1},\dots,v_{u}\}$  is the node set (with  $u$  nodes) and  $\mathbf{E}$  is the edge set. In GCN, the edges in  $\mathbf{G}$  are often captured via the adjacency matrix  $\mathbf{A}\in \mathbb{R}^{u\times u}$ . Also, each node  $v_{i}\in \mathbf{V}$  is associated with an initial hidden vector  $\mathbf{v}_i^0$ . As such, a GCN model involves multiple layers of abstraction in which the hidden vector  $\mathbf{v}_i^l$  for the node  $v_{i}\in \mathbf{V}$  at the  $l$ -th layer is computed by  $(l\geq 1)$ :
$$
\mathbf {v} _ {i} ^ {l} = \mathrm {R e L U} \left(\frac {\sum_ {j = 1} ^ {u} \mathbf {A} _ {i j} \mathbf {W} ^ {l} \mathbf {v} _ {j} ^ {l - 1} + \mathbf {b} ^ {l}}{\sum_ {j = 1} ^ {u} \mathbf {A} _ {i j}}\right)
$$
where  $\mathbf{W}^l$  and  $\mathbf{b}^l$  are trainable weight and bias at the  $l$ -th layer. Assuming  $N$  GCN layers, the hidden vectors for the nodes in  $\mathbf{V}$  at the last layer  $\mathbf{v}_1^N,\ldots ,\mathbf{v}_u^N$  would capture richer and more abstract information for the nodes, serving as the outputs of the GCN model. This process is denoted by:  $\mathbf{v}_1^N,\dots ,\mathbf{v}_u^N = \mathrm{GCN}(\mathbf{A};\mathbf{v}_1^0,\dots ,\mathbf{v}_u^0;N)$ .
Given an input sentence  $\mathbf{w} = [w_1, w_2, \ldots, w_n]$  (with  $n$  words), our model for joint four-task IE on  $\mathbf{w}$  involves three major components: (i) Span Detection, (ii) Instance Interaction, and (iii) Type Dependency-based Regularization.
3.1 Span Detection
This component aims to identify spans of entity mentions and event triggers in  $\mathbf{w}$  that would be used to form the nodes in the interaction graph between different instances of our four IE tasks for  $\mathbf{w}$ . As such, we formulate the span detection
problems as sequence labeling tasks where each word  $w_{i}$  in  $\mathbf{w}$  is associated with two BIO tags to capture the span information for entity mentions and event triggers in  $\mathbf{w}$ . Note that we do not predict entity types and event types at this step, leading to only three possible values (i.e., B, I, and O) for the tags of the words.
In particular, following (Lin et al., 2020), we first feed  $\mathbf{w}$  into the pre-trained BERT encoder (Devlin et al., 2019) to obtain a sequence of vectors  $\mathbf{X} = [\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_n]$  to represent  $\mathbf{w}$ . Here, each vector  $\mathbf{x}_i$  serves as the representation vector for the word  $w_{i}\in \mathbf{w}$  that is obtained by averaging the hidden vectors of the word-pieces of  $w_{i}$  returned by BERT. Afterward,  $\mathbf{X}$  is fed into two conditional random field (CRF) layers to determine the best BIO tag sequences for event mentions and event triggers for  $\mathbf{w}$ , following (Chiu and Nichols, 2016). As such, the Viterbi algorithm is used to decode the input sentence while the negative log-likelihood losses are employed as the training objectives for the span detection component of the model. For convenience, let  $L_{span}^{ent}$  and  $L_{span}^{trg}$  be the negative log-likelihoods of the gold tag sequences for entity mentions and event triggers (respectively) for  $\mathbf{w}$ . These terms will be included in the overall loss function of the model later.
3.2 Instance Interaction
Based on the tag sequences for  $\mathbf{w}$  from the previous component, we can obtain two separate span sets for the entity mentions and event triggers in  $\mathbf{w}$  (the golden spans are used in the training phase to avoid noise). For the next computation, we first compute a representation vector for each span  $(i,j)$ $(1\leq i\leq j\leq n)$  in these two sets by averaging the BERT-based representation vectors for the words in this span (i.e.,  $\mathbf{x}_i,\dots ,\mathbf{x}_j$ ). For convenience, let  $\mathbf{R}^{ent} = \{\mathbf{e}_1,\mathbf{e}_2,\ldots ,\mathbf{e}_{n_{ent}}\}$  ( $n_{ent} = |\mathbf{R}^{ent}|$ ) and  $\mathbf{R}^{trg} = \{\mathbf{t}_1,\mathbf{t}_2,\dots ,\mathbf{t}_{n_{trg}}\}$  ( $n_{trg} = |\mathbf{R}^{trg}|$ ) be the sets of span representation vectors for the entity mentions and event triggers in  $\mathbf{w}^1$ . The goal of this component is to leverage such span representation vectors to form instance representations and enrich them with instance interactions to perform necessary predictions in IE.
Instance Representation. Prediction instances in our model amount to the specific objects that we need to predict a type for one of the four IE
tasks. As such, the prediction instances for EME and ETD, called entity and trigger instances, correspond directly to the entity mentions and event triggers in  $\mathbf{R}^{ent}$  and  $\mathbf{R}^{trg}$  respectively (as we need to predict the entity types for  $\mathbf{e}_i \in \mathbf{R}^{ent}$  and the event types for  $\mathbf{t}_i \in \mathbf{R}^{trg}$  in this step). Thus, we also use  $\mathbf{R}^{ent}$  and  $\mathbf{R}^{trg}$  as the sets of initial representation vectors for the entity/event instances for EME and ETD in the following. Next, for RE, the prediction instances (called relation instances) involve pairs of entity mentions in  $\mathbf{R}^{ent}$ . To obtain the initial representation vector for a relation instance, we concatenate the representation vectors of the two corresponding entity mentions, leading to the set of representation vectors  $\mathbf{rel}_{ij}$  for relation instances:  $\mathbf{R}^{rel} = \{\mathbf{rel}_{ij} = [\mathbf{e}_i, \mathbf{e}_j] \mid \mathbf{e}_i, \mathbf{e}_j \in \mathbf{R}^{ent}, i < j\}$  ( $|\mathbf{R}^{rel}| = n_{ent}(n_{ent} - 1)/2$ ). Finally, for EAE, we form the prediction instances (called argument instances) by pairing each event trigger in  $\mathbf{R}^{trg}$  with each entity mention in  $\mathbf{R}^{ent}$  (for the argument role predictions of the entity mentions with respect to the event triggers/mentions). By concatenating the representation vectors of the paired entity mentions and event triggers, we generate the initial representation vectors  $\mathbf{arg}_{ij}$  for the corresponding argument instances:  $\mathbf{R}^{arg} = \{\mathbf{arg}_{ij} = [\mathbf{t}_i, \mathbf{e}_j] \mid \mathbf{t}_i \in \mathbf{R}^{trg}, \mathbf{e}_j \in \mathbf{R}^{ent}\}$  ( $|\mathbf{R}^{arg}| = n_{trg}n_{ent}$ ^2). We also use the prediction instances and their representation vectors interchangeably in this work.
Instance Interaction. The initial representation vectors for the instances so far do not explicitly consider beneficial interactions between related instances. To address this issue, we explicitly create an interaction graph between the prediction instances for the four IE tasks to connect related instances to each other. This graph will be consumed by a GCN model to enrich instance representations with interaction information afterward. In particular, the node set  $\mathbf{N}^{inst}$  in our instance interaction graph  $\mathbf{G}^{inst} = \{\mathbf{N}^{inst},\mathbf{E}^{inst}\}$  involves all prediction instances for the four IE tasks, i.e.,  $\mathbf{N}^{inst} = \mathbf{R}^{ent}\cup \mathbf{R}^{trg}\cup \mathbf{R}^{rel}\cup \mathbf{R}^{arg}$ . The edge set  $\mathbf{E}^{inst}$  then captures instance interactions by connecting the instance nodes in  $\mathbf{N}^{inst}$  that involve the same entity mentions or event triggers (i.e., two instances are related if they concern the same entity mention or event trigger). As such, the edges in  $\mathbf{E}^{inst}$  are created as follows:
<sup>1</sup>We will also refer to entity mentions and event triggers interchangeably with their span representations  $\mathbf{e}_i$  and  $\mathbf{t}_i$  in this work.
2In our implementation,  $\mathbf{R}^{rel}$  and  $\mathbf{R}^{arg}$  are transformed into vectors of the same size with those in  $\mathbf{R}^{ent}$  and  $\mathbf{R}^{trg}$  (using one-layer feed forward networks) for future computation.
(i) An entity instance node  $\mathbf{e}_i$  is connected to all relation instance nodes of the forms  $\mathbf{rel}_{ij} = [\mathbf{e}_i, \mathbf{e}_j]$  and  $\mathbf{rel}_{ki} = [\mathbf{e}_k, \mathbf{e}_i]$  (sharing entity mention  $\mathbf{e}_i$ ).
(ii) An entity instance node  $\mathbf{e}_j$  is connected to all argument instance nodes of the form  $\mathbf{arg}_{ij} = [\mathbf{t}_i, \mathbf{e}_j]$  (sharing entity mention  $\mathbf{e}_j$ ).
(iii) A trigger node  $\mathbf{t}_i$  is connected to all argument instance nodes of the form  $\mathbf{arg}_{ij} = [\mathbf{t}_i, \mathbf{e}_j]$  (i.e., sharing event trigger  $\mathbf{t}_i$ ).
GCN. To enrich the representation vector for an instance in  $\mathbf{N}^{inst}$  with the information from the related (neighboring) nodes, we feed  $\mathbf{G}^{inst}$  into a GCN model (called GCN $^{inst}$ ). For convenience, we rename the initial representation vectors of all the instance nodes in  $\mathbf{N}^{inst}$  by:  $\mathbf{N}^{inst} = \{\mathbf{r}_1,\dots ,\mathbf{r}_{n_i}\}$  ( $n_i = |\mathbf{N}^{inst}|$ ). Also, let  $\mathbf{A}^{inst} \in \{0,1\}^{n_i \times n_i}$  be the adjacency matrix of the interaction graph  $\mathbf{G}^{inst}$  where  $\mathbf{A}_{ij}^{inst} = 1$  if the instance nodes  $\mathbf{r}_i$  and  $\mathbf{r}_j$  are connected in  $\mathbf{G}^{inst}$  or  $i = j$  (for self-connections). The interaction-enriched representation vectors for the instances in  $\mathbf{N}^{inst}$  are then computed by the GCN $^{inst}$  model:  $\mathbf{r}_1^{inst},\ldots,\mathbf{r}_{n_i}^{inst} = \mathrm{GCN}^{inst}(\mathbf{A}^{inst};\mathbf{r}_1,\ldots,\mathbf{r}_{n_i};N_i)$  where  $N_i$  is the number of layers for the GCN $^{inst}$  model.
Type Embedding and Prediction. Finally, the enriched instance representation vectors  $\mathbf{r}_1^{inst},\ldots ,\mathbf{r}_{n_i}^{inst}$  will be used to perform the predictions for the four IE tasks. In particular, let  $t_k\in \{ent,trg,rel,arg\}$  be the corresponding task index and  $y_{k}$  be the ground-truth type (of the task  $t_k$  ) for the prediction instance  $\mathbf{r}_k$  in  $\mathbf{N}^{inst}$ . Also, let  $\mathcal{T} = \mathcal{T}^{ent}\cup \mathcal{T}^{trg}\cup \mathcal{T}^{rel}\cup \mathcal{T}^{arg}$  be the union of the possible entity types (in  $\mathcal{T}^{ent}$  for EME), event types (in  $\mathcal{T}^{trg}$  for ETD), relations (in  $\mathcal{T}^{rel}$  for RE), and argument roles (in  $\mathcal{T}^{arg}$  for EAE) in our problem  $(y_{k}\in \mathcal{T}^{t_{k}})$ . Note that  $\mathcal{T}^{rel}$  and  $\mathcal{T}^{arg}$  contain the special types None. To prepare for the type predictions and the type dependency modeling in the next steps, we associate each type in  $\mathcal{T}$  with an embedding vector (of the same size as  $\mathbf{e}_i$  and  $\mathbf{t}_i$ ) that is initialized randomly and updated during our training process. For convenience, let  $\mathcal{T} = [\bar{\mathbf{t}}_1,\dots ,\bar{\mathbf{t}}_{n_t}]$  where  $\bar{\mathbf{t}}_i$  is used interchangeably for both a type and its embedding vector in  $\mathcal{T}$  ( $n_t$  is the total number of types). As such, to perform the prediction for an instance  $\mathbf{r}_k$  in  $\mathbf{N}^{inst}$ , we compute the dot products between  $\mathbf{r}_k^{inst}$  and each type embedding vectors in  $\mathcal{T}^{t_k}\cap \mathcal{T}$  to estimate the possibilities that  $\mathbf{r}_k$  has a type in  $\mathcal{T}^{t_k}$ . Afterward, these scores are normalized by the softmax function to obtain the probability distribution  $\hat{\mathbf{y}}_k$  over the possible types in
$\mathcal{T}^{t_k}$  for  $\mathbf{r}_k$ :  $\hat{\mathbf{y}}_k = \text{softmax}(\mathbf{r}_k^{\text{inst}}\bar{\mathbf{t}}^T|\bar{\mathbf{t}} \in \mathcal{T}^{t_k} \cap \mathcal{T})$ . In the decoding phase, the predicted type  $\hat{y}_k$  for  $\mathbf{r}_k$  is obtained via the argmax function (greedy decoding):  $\hat{y}_k = \arg \max_{\hat{\mathbf{y}}_k}$  The negative log-likelihood over all the prediction instances is used to train the model:  $L_{type} = -\sum_{k=1}^{n_i} \log \hat{\mathbf{y}}_k[y_k]$ .
3.3 Type Dependency-based Regularization
In this section, we aim to obtain the type dependencies across tasks and use them to supervise the model in the training process (to improve the representation vectors for IE). As presented in the introduction, our motivation is to generate global dependency graphs between types of different IE tasks for each input sentence whose representations are leveraged to regularize the model during training. In particular, starting with the golden types  $\mathbf{y} = y_{1},y_{2},\ldots ,y_{n_{i}}$  and the predicted types  $\hat{\mathbf{y}} = \hat{y}_1,\hat{y}_2,\dots ,\hat{y}_{n_i}$  for the instance nodes in  $\mathbf{N}^{inst}$ , we build two dependency graphs  $\mathbf{G}^{gold}$  and  $\mathbf{G}^{pred}$  to capture the global type dependencies for the tasks (called the golden and predicted dependency graphs respectively). Afterward, to supervise the training process, we seek to constrain the model so the predicted dependency graph  $\mathbf{G}^{pred}$  is similar to the golden graph  $\mathbf{G}^{gold}$  (i.e., using the dependency graphs as the bridges to inject the global type dependency knowledge in  $\mathbf{G}^{gold}$  into the model).
Dependency Graph Construction. Both  $\mathbf{G}^{gold}$  and  $\mathbf{G}^{pred}$  involve the types of all the four IE tasks in  $\mathcal{T}$  as the nodes. To encode the type dependencies, the connections/edges in  $\mathbf{G}^{gold}$  are computed based on the golden types  $\mathbf{y} = y_1, y_2, \ldots, y_{n_i}$  for the instance nodes in  $\mathbf{N}^{inst}$  as follows: