SlowGuess/ACL_2020_OCR huggingface dataset:
Ying Lin $^{1}$ , Heng Ji $^{1}$ , Fei Huang $^{2}$ , Lingfei Wu $^{3}$
<sup>1</sup>University of Illinois at Urbana-Champaign
2Alibaba DAMO Academy 3IBM Research
{yinglin8,hengji}@illinois.edu,
f.huang@alibaba-inc.com, wuli@us.ibm.com
Most existing joint neural models for Information Extraction (IE) use local task-specific classifiers to predict labels for individual instances (e.g., trigger, relation) regardless of their interactions. For example, a VICTIM of a DIE event is likely to be a VICTIM of an ATTACK event in the same sentence. In order to capture such cross-subtask and cross-instance inter-dependencies, we propose a joint neural framework, ONEIE, that aims to extract the globally optimal IE result as a graph from an input sentence. ONEIE performs end-to-end IE in four stages: (1) Encoding a given sentence as contextualized word representations; (2) Identifying entity mentions and event triggers as nodes; (3) Computing label scores for all nodes and their pairwise links using local classifiers; (4) Searching for the globally optimal graph with a beam decoder. At the decoding stage, we incorporate global features to capture the cross-subtask and cross-instance interactions. Experiments show that adding global features improves the performance of our model and achieves new state-of-the-art on all subtasks. As ONEIE does not use any language-specific feature, we prove it can be easily applied to new languages or trained in a multilingual manner. Our code and models for English, Spanish and Chinese are publicly available for research purpose<sup>1</sup>.
Information Extraction (IE) aims to extract structured information from unstructured texts. It is a complex task comprised of a wide range of subtasks, such as named, nominal, and pronominal mention extraction, entity linking, entity coreference resolution, relation extraction, event extraction, and event coreference resolution. Early efforts typically perform IE in a pipelined fashion,
which leads to the error propagation problem and disallows interactions among components in the pipeline. As a solution, some researchers propose joint inference and joint modeling methods to improve local prediction (Roth and Yih, 2004; Ji and Grishman, 2005; Ji et al., 2005; Sil and Yates, 2013; Li et al., 2014; Durrett and Klein, 2014; Miwa and Sasaki, 2014; Lu and Roth, 2015; Yang and Mitchell, 2016; Kirschnick et al., 2016). Due to the success of deep learning, neural models have been widely applied to various IE subtasks (Collobert et al., 2011; Chiu and Nichols, 2016; Chen et al., 2015; Lin et al., 2016). Recently, some efforts (Wadden et al., 2019; Luan et al., 2019) revisit global inference approaches by designing neural networks with embedding features to jointly model multiple subtasks. However, these methods use separate local task-specific classifiers in the final layer and do not explicitly model the interdependencies among tasks and instances. Figure 1 shows a real example where the local argument role classifier predicts a redundant PERSON edge. The model should be able to avoid such mistakes if it is capable of learning and leveraging the fact that it is unusual for an ELECT event to have two PERSON arguments.
Example: Prime Minister Abdullah Gul resigned earlier Tuesday to make way for Erdogan, who won a parliamentary seat in by-elections Sunday.
Figure 1: A typical error made by local classifiers without global constraints.
To address this issue, we propose a joint neu
http://blender.cs.illinois.edu/software/oneie
7999
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999-8009 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: An illustration of our end-to-end joint information extraction framework ONEIE at the test stage. We do not show all pairwise links for simplicity purposes.
ral framework, ONEIE, to perform end-to-end IE with global constraints. As Figure 2 shows, instead of predicting separate knowledge elements using local classifiers, ONEIE aims to extract a globally optimal information network for the input sentence. When comparing candidate information networks during the decoding process, we not only consider individual label scores for each knowledge element, but evaluate cross-subtask and cross-instance interactions in the network. In this example, a graph with the INJURE-VICTIM-ORG (the VICTIM of an INJURE event is an ORG entity) structure is demoted. Experiments show that our framework achieves comparable or better results compared to the state-of-the-art end-to-end architecture (Wadden et al., 2019).
To the best of our knowledge, ONEIE is the first end-to-end neural IE framework that explicitly models cross-subtask and cross-instance interdependencies and predicts the result as a unified graph instead of isolated knowledge elements. Because ONEIE does not rely on language-specific features, it can be rapidly applied to new languages. Furthermore, global features in our framework are highly explainable and can be explicitly analyzed.
2 Task
Given a sentence, our ONEIE framework aims to extract an information network representation (Li et al., 2014), where entity mentions and event triggers are represented as nodes, and relations and event-argument links are represented as edges. In other words, we perform entity, relation, and event extraction within a unified framework. In this sec
tion, we will elaborate these tasks and involved terminologies.
Entity Extraction aims to identify entity mentions in text and classify them into pre-defined entity types. A mention can be a name, nominal, or pronoun. For example, "Kashmir region" should be recognized as a location (LOC) named entity mention in Figure 2.
Relation Extraction is the task of assigning a relation type to an ordered pair of entity mentions. For example, there is a PART-WHOLE relation between "Kashmir region" and "India".
Event Extraction entails identifying event triggers (the words or phrases that most clearly express event occurrences) and their arguments (the words or phrases for participants in those events) in unstructured texts and classifying these phrases, respectively, for their types and roles. An argument can be an entity, time expression, or value (e.g., MONEY, JOB-TITLE, CRIME). For example, in Figure 2, the word "injured" triggers an INJURE event and "300" is the VICTIM argument.
We formulate the task of extracting information networks as follows. Given an input sentence, our goal is to predict a graph  $G = (V,E)$ , where  $V$  and  $E$  are the node and edge sets respectively. Each node  $v_{i} = \langle a_{i},b_{i},l_{i}\rangle \in V$  represents an entity mention or event trigger, where  $a$  and  $b$  are the start and end word indices, and  $l$  is the node type label. Each edge  $e_{ij} = \langle i,j,l_{ij}\rangle \in E$  is represented similarly, whereas  $i$  and  $j$  denote the indices of involved nodes. For example, in Figure 2, the trigger "injured" is represented as  $\langle 7,7,\text{INJURE}\rangle$ , the entity mention "Kashmir region" is represented as  $\langle 10,$
11, LOC), and the event-argument edge between them is  $\langle 2,3,\mathrm{PLACE}\rangle$
As Figure 2 illustrates, our ONEIE framework extracts the information network from a given sentence in four steps: encoding, identification, classification, and decoding. We encode the input sentence using a pre-trained BERT encoder (Devlin et al., 2019) and identify entity mentions and event triggers in the sentence. After that, we compute the type label scores for all nodes and pairwise edges among them. During decoding, we explore possible information networks for the input sentence using beam search and return the one with the highest global score.
3.1 Encoding
Given an input sentence of  $L$  words, we obtain the contextualized representation  $x_{i}$  for each word using a pre-trained BERT encoder. If a word is split into multiple word pieces (e.g., Mondrian → Mon, ##dr, ##ian), we use the average of all piece vectors as its word representation. While previous methods typically use the output of the last layer of BERT, our preliminary study shows that enriching word representations using the output of the third last layer of BERT can substantially improve the performance on most subtasks.
3.2 Identification
At this stage, we identify entity mentions and event triggers in the sentence, which will act as nodes in the information network. We use a feedforward network FFN to compute a score vector  $\hat{\pmb{y}}_i = \mathrm{FFN}(\pmb{x}_i)$  for each word, where each value in  $\hat{\pmb{y}}_i$  represents the score for a tag in a target tag set $^2$ . After that, we use a conditional random fields (CRFs) layer to capture the dependencies between predicted tags (e.g., an I-PER tag should not follow a B-GPE tag). Similar to (Chiu and Nichols, 2016), we calculate the score of a tag path  $\hat{\pmb{z}} = \{\hat{\pmb{z}}_1,\dots,\hat{\pmb{z}}_L\}$  as
$$
s (\pmb {X}, \hat {\pmb {z}}) = \sum_ {i = 1} ^ {L} \hat {y} _ {i, \hat {z} _ {i}} + \sum_ {i = 1} ^ {L + 1} A _ {\hat {z} _ {i - 1}, \hat {z} _ {i}},
$$
where  $\mathbf{X} = \{\pmb{x}_1,\dots,\pmb{x}_L\}$  is the contextualized representations of the input sequence,  $\hat{y}_{i,\hat{z}_i}$  is the  $\hat{z}_i$  -th
component of the score vector  $\hat{\pmb{y}}_i$ , and  $A_{\hat{z}_{i-1},\hat{z}_i}$  is the  $(\hat{z}_{i-1},\hat{z}_i)$  entry in matrix  $\mathbf{A}$  that indicates the transition score from tag  $\hat{z}_{i-1}$  to  $\hat{z}_i$ . The weights in  $\mathbf{A}$  are learned during training. We append two special tags <start> and <end> to the tag path as  $\hat{z}_0$  and  $\hat{z}_{L+1}$  to denote the start and end of the sequence. At the training stage, we maximize the log-likelihood of the gold-standard tag path as
$$
\log p (\boldsymbol {z} | \boldsymbol {X}) = s (\boldsymbol {X}, \boldsymbol {z}) - \log \sum_ {\hat {\boldsymbol {z}} \in Z} e ^ {s (\boldsymbol {X}, \hat {\boldsymbol {z}})},
$$
where  $Z$  is the set of all possible tag paths for a given sentence. Thus, we define the identification loss as  $\mathcal{L}^{\mathrm{I}} = -\log p(\boldsymbol{z}|\boldsymbol{X})$ .
In our implementation, we use separate taggers to extract entity mentions and event triggers. Note that we do not use types predicted by the taggers. Instead, we make a joint decision for all knowledge elements at the decoding stage to prevent error propagation and utilize their interactions to improve the prediction of node type.
3.3 Classification
We represent each identified node as  $\pmb{v}_i$  by averaging its word representations. After that, we use separate task-specific feed-forward networks to calculate label scores for each node as  $\hat{\pmb{y}}_i^t = \mathrm{FFN}^t(\pmb{v}_i)$ , where  $t$  indicates a specific task. To obtain the label score vector for the edge between the  $i$ -th and  $j$ -th nodes, we concatenate their span representations and calculate the vector as  $\hat{\pmb{y}}_k^t = \mathrm{FFN}^t(\pmb{v}_i, \pmb{v}_j)$ .
For each task, the training objective is to minimize the following cross-entropy loss
$$
\mathcal {L} ^ {\mathrm {t}} = - \frac {1}{N ^ {t}} \sum_ {i = 1} ^ {N ^ {t}} \boldsymbol {y} _ {i} ^ {t} \log \hat {\boldsymbol {y}} _ {i} ^ {t},
$$
where  $\pmb{y}_i^t$  is the true label vector and  $N^t$  is the number of instances for task  $t$ .
If we ignore the inter-dependencies between nodes and edges, we can simply predict the label with the highest score for each knowledge element and thus generate the locally best graph  $\hat{G}$ . The score of  $\hat{G}$  can be calculated as
$$
s ^ {\prime} (\hat {G}) = \sum_ {t \in T} \sum_ {i = 1} ^ {N ^ {t}} \max \hat {\boldsymbol {y}} _ {i} ^ {t},
$$
where  $T$  is the set of tasks. We refer to  $s'(\hat{G})$  as the local score of  $\hat{G}$ .
2We use the BIO tag scheme, in which the prefix B- marks the beginning of a mention, and I- means inside of a mention. A token not belonging to any mention is tagged with O.
Category	Description
Role	1. The number of entities that act as <rolei> and <rolej> arguments at the same time.
2. The number of <event_typei> events with <number> <rolej> arguments.
3. The number of occurrences of <event_typei>, <rolej>, and <entity_typek> combination.
4. The number of events that have multiple <rolei> arguments.
5. The number of entities that act as a <rolei> argument of an <event_typej> event and a <rolek> argument of an <event_typei> event at the same time.
Relation	6. The number of occurrences of <entity_typei>, <entity_typej>, and <relation_typek> combination.
7. The number of occurrences of <entity_typei> and <relation_typej> combination.
8. The number of occurrences of a <relation_typei> relation between a <rolej> argument and a <rolek> argument of the same event.
9. The number of entities that have a <relation_typei> relation with multiple entities.
10. The number of entities involving in <relation_typei> and <relation_typej> relations simultaneously.
Trigger	11. Whether a graph contains more than one <event_typei> event.
Table 1: Global feature categories.
3.4 Global Features
A limitation of local classifiers is that they are incapable of capturing inter-dependencies between knowledge elements in an information network. We consider two types of inter-dependencies in our framework.
The first type of inter-dependency is Cross-subtask interactions between entities, relations, and events. Consider the following sentence. "A civilian aid worker from San Francisco was killed in an attack in Afghanistan." A local classifier may predict "San Francisco" as a VICTIM argument because an entity mention preceding "was killed" is usually the victim despite the fact that a GPE is unlikely to be a VICTIM. To impose such constraints, we design a global feature as shown in Figure 3(a) to evaluate whether the structure DIE-VICTIM-GPE exists in a candidate graph.
Another type of inter-dependency is Cross-instance interactions between multiple event and/or relation instances in the sentence. Take the following sentence as an example. "South Carolina boy, 9, dies during hunting trip after his father accidentally shot him on Thanksgiving Day." It can be challenging for a local classifier to predict "boy" as the VICTIM of the ATTACK event triggered by "shot" due to the long distance between these two words. However, as shown in Figure 3(b), if an entity is the VICTIM of a DIE event, it is also likely to be the VICTIM of an ATTACK event in the same sentence.
Motivated by these observations, we design a set of global feature templates (event schemas) as listed in Table 1 to capture cross-subtask and cross-instance interactions, while the model fills in all possible types to generate features and learns the
(a) Cross-subtask Interaction
(b) Cross-instance Interactions
Figure 3: Examples of inter-dependencies between elements in information networks. (a) A VICTIM edge is unlikely to exist between a GPE entity and a DIE event trigger. (b) The VICTIM of a DIE event is likely to be the VICTIM of an ATTACK event in the same sentence.
weight of each feature during training. Given a graph  $G$ , we represent its global feature vector as  $\pmb{f}_G = \{f_1(G), \dots, f_M(G)\}$ , where  $M$  is the number of global features and  $f_i(\cdot)$  is a function that evaluates a certain feature and returns a scalar. For example,
$$
f _ {i} (G) = \left\{ \begin{array}{l} 1, G \text {h a s m u l t i p l e A T T A C K e v e n t s} \\ 0, \text {o t h e r w i s e .} \end{array} \right.
$$
Next, ONEIE learns a weight vector  $\pmb{u} \in \mathbb{R}^{M}$  and calculates the global feature score of  $G$  as the dot product of  $\pmb{f}_{G}$  and  $\pmb{u}$ . We define the global score of  $G$  as the sum of its local score and global feature score, namely
$$
s (G) = s ^ {\prime} (G) + \pmb {u} \pmb {f} _ {G},
$$
We make the assumption that the gold-standard graph for a sentence should achieve the highest global score. Therefore, we minimize the following loss function
$$
\mathcal {L} ^ {\mathrm {G}} = s (\hat {G}) - s (G),
$$
8002
where  $\hat{G}$  is the graph predicted by local classifiers and  $G$  is the gold-standard graph.
Finally, we optimize the following joint objective function during training
$$
\mathcal {L} = \mathcal {L} ^ {I} + \sum_ {t \in T} \mathcal {L} ^ {t} + \mathcal {L} ^ {G}
$$
3.5 Decoding
As we have discussed, because local classifiers ignore interactions among elements in an information network, they may predict contradictory results or fail to predict difficult edges that require information from other elements. In order to address these issues, ONEIE makes a joint decision for all nodes and their pairwise edges to obtain the globally optimal graph. The basic idea is to calculate the global score for each candidate graph and select the one with the highest score. However, exhaustive search is infeasible in many cases as the size of search space grows exponentially with the number of nodes. Therefore, we design a beam search-based decoder as Figure 4 depicts.
Given a set of identified nodes  $V$  and the label scores for all nodes and their pairwise links, we perform decoding with an initial beam set  $B = \{K_0\}$ , where  $K_{0}$  is an order-zero graph. At each step  $i$ , we expand each candidate in  $B$  in node step and edge step as follows.
Node step: We select  $v_{i} \in V$  and define its candidate set as  $V_{i} = \{\langle a_{i}, b_{i}, l_{i}^{(k)} \rangle | 1 \leq k \leq \beta_{v}\}$ , where  $l_{i}^{(k)}$  denotes the label with the  $k$ -th highest local score for  $v_{i}$ , and  $\beta_{v}$  is a hyper-parameter that controls the number of candidate labels to consider. We update the beam set by
$$
B \leftarrow \{G + v | (G, v) \in B \times V _ {i} \},
$$
Edge step: We iteratively select a previous node  $v_{j} \in V, j < i$  and add possible edges between  $v_{j}$  and  $v_{i}$ . Note that if  $v_{i}$  is a trigger, we skip  $v_{j}$  if it is also a trigger. At each iteration, we construct a candidate edge set as  $E_{ij} = \{\langle j,i,l_{ij}^{(k)}\rangle |1 \leq k \leq \beta_e\}$ , where  $l_{ij}^{(k)}$  is the label with  $k$ -th highest score for  $e_{ij}$  and  $\beta_{e}$  is a threshold for the number of candidate labels. Next, we update the beam set by
$$
B \gets \{G + e | (G, e) \in B \times E _ {i j} \},
$$
At the end of each edge step, if  $|B|$  is larger than the beam width  $\theta$ , we rank all candidates by global score in descending order and keep the top  $\theta$  ones.
After the last step, we return the graph with the highest global score as the information network for the input sentence.
We perform our experiments on the Automatic Content Extraction (ACE) 2005 dataset<sup>3</sup>, which provides entity, value, time, relation, and event annotations for English, Chinese, and Arabic. Following Wadden et al. (2019)'s pre-processing<sup>4</sup>, we conduct experiments on two datasets, ACE05-R that includes named entity and relation annotations, and ACE05-E that includes entity, relation, and event annotations. We keep 7 entity types, 6 coarse-grained relation types, 33 event types, and 22 argument roles.
In order to reinstate some important elements absent from ACE05-R and ACE05-E, we create a new dataset,  $\mathrm{ACE05 - E^{+}}$ , by adding back the order of relation arguments, pronouns, and multi-token event triggers, which have been largely ignored in previous work. We also skip lines before the <text> tag (e.g., headline, datetime) as they are not annotated.
In addition to ACE, we derive another dataset, ERE-EN, from the Entities, Relations and Events (ERE) annotation task created under the Deep Exploration and Filtering of Test (DEFT) program because it covers more recent articles. Specifically, we extract 458 documents and 16,516 sentences from three ERE datasets, LDC2015E29, LDC2015E68, and LDC2015E78. For ERE-EN, we keep 7 entity types, 5 relation types, 38 event types, and 20 argument roles.
To evaluate the portability of our model, we also develop a Chinese dataset from ACE2005 and a Spanish dataset from ERE (LDC2015E107). We refer to these datasets as ACE05-CN and ERE-ES respectively.
We optimize our model with BertAdam for 80 epochs with a learning rate of 5e-5 and weight decay of 1e-5 for BERT, and a learning rate of 1e-3 and weight decay of 1e-3 for other parameters. We use the bert-base-multilingual-cased
<sup>3</sup>https://www.ldc.upenn.edu/collaborations/past-projects/ace
4https://github.com/dwadden/dygiepp
8003
Figure 4: An illustration of our decoding algorithm. At each step, we expand each candidate graph by adding a new node and possible edges between it and existing nodes. After that, we rank all expanded graphs and keep the top ones.
Dataset	Split	#Sents	#Entities	#Refs	#Events
ACE05-R	Train	10,051	26,473	4,788	-
Dev	2,424	6,362	1,131	-
Test	2,050	5,476	1,151	-
ACE05-E	Train	17,172	29,006	4,664	4,202
Dev	923	2,451	560	450
Test	832	3,017	636	403
ACE05-CN	Train	6,841	29,657	7,934	2,926
Dev	526	2,250	596	217
Test	547	2,388	672	190
ACE05-E+	Train	19,240	47,525	7,152	4,419
Dev	902	3,422	728	468
Test	676	3,673	802	424
ERE-EN	Train	14,219	38,864	5,045	6,419
Dev	1,162	3,320	424	552
Test	1,129	3,291	477	559
ERE-ES	Train	7,067	11,839	1,698	3,272
Dev	556	886	120	210
Test	546	811	108	269
Table 2: Dataset statistics.
model<sup>5</sup> for ACE05-CN and ERE-ES, and use the bert-large-cased model for other datasets. Following (Wadden et al., 2019), we use two-layer FFNs with a dropout rate of 0.4 for local classifiers. We use 150 hidden units for entity and relation extraction, and 600 hidden units for event extraction. For global features, we set  $\beta_v$  and  $\beta_e$  to 2 and set  $\theta$  to 10. In our experiments, we use random seeds and report averaged scores across runs. We use the same criteria as (Zhang et al., 2019; Wadden et al., 2019) for evaluation as follows.
- Entity: An entity mention is correct if its offsets and type match a reference entity.
- Relation: A relation is correct if its relation type
is correct and the offsets of the related entity mentions are correct.
- Trigger: A trigger is correctly identified (Trig-I) if its offsets match a reference trigger. It is correctly classified (Trig-C) if its event type also matches the reference trigger.
- Argument: An argument is correctly identified (Arg-I) if its offsets and event type match a reference argument mention. It is correctly classified (Arg-C) if its role label also matches the reference argument mention.
4.3 Overall Performance
In Table 3, we compare our results with two models: (1) DYGIE++ (Wadden et al., 2019), the state-of-the-art end-to-end IE model that utilizes multisentence BERT encodings and span graph propagation; (2) BASELINE that follows the architecture of ONEIE but only uses the output of the last layer of BERT and local classifiers. We can see that our model consistently outperforms DYGIE++ and BASELINE on ACE05-R and ACE05-E.
In (Wadden et al., 2019), the authors show that combining triggers predicted by a four-model ensemble optimized for trigger detection can improve the performance of event extraction. While we also report our results using a four-model ensemble in Table 4 for fair comparison, we hold the opinion that the single-model scores in Table 3 better reflect the actual performance of ONEIE and should be used for future comparison.
Table 5 shows the performance of ONEIE on two new datasets,  $\mathrm{ACE05 - E^{+}}$  and ERE-EN.
In Table 6 we list salient global features learned by the model. Take feature #9 as an example, if a
<https://huggingface.co/transformers/pretrained_models.html>
8004
Dataset	Task	DYGIE++	BASELINE	ONEIE
ACE05-R	Entity	88.6	-	88.8
Relation	63.4	-	67.5
ACE05-E	Entity	89.7	90.2	90.2
Trig-I	-	76.6	78.2
Trig-C	69.7	73.5	74.7
Arg-I	53.0	56.4	59.2
Arg-C	48.8	53.9	56.8
Table 3: Results on ACE2005 datasets (F-score, %).
Dataset	Task	DYGIE++*	ONEIE*
ACE05-E	Entity	90.7	90.3
Trig-I	76.5	78.6
Trig-C	73.6	75.2
Arg-I	55.4	60.7
Arg-C	52.5	58.6
Table 4: Experiment results on ACE05-E (F-score, %). DYGIE++* and ONEIE* use a four-model ensemble optimized for trigger detection.
Task	Entity	Trig-I	Trig-C	Arg-I	Arg-C	Relation
ACE05-E+	89.6	75.6	72.8	57.3	54.8	58.6
ERE-EN	87.0	68.4	57.0	50.1	46.5	53.2