SlowGuess/ACL_2020_OCR huggingface dataset:
Xinya Du and Claire Cardie  
Department of Computer Science  
Cornell University  
Ithaca, NY, USA  
{xdu, cardie}@cs.cornell.edu
Few works in the literature of event extraction have gone beyond individual sentences to make extraction decisions. This is problematic when the information needed to recognize an event argument is spread across multiple sentences. We argue that document-level event extraction is a difficult task since it requires a view of a larger context to determine which spans of text correspond to event role fillers. We first investigate how end-to-end neural sequence models (with pre-trained language model representations) perform on document-level role filler extraction, as well as how the length of context captured affects the models' performance. To dynamically aggregate information captured by neural representations learned at different levels of granularity (e.g., the sentence- and paragraph-level), we propose a novel multi-granularity reader. We evaluate our models on the MUC-4 event extraction dataset, and show that our best system performs substantially better than prior work. We also report findings on the relationship between context length and neural model performance on the task.
The goal of document-level event extraction<sup>1</sup> is to identify in an article events of a pre-specified type along with their event-specific role fillers, i.e., arguments. The complete document-level extraction problem generally requires role filler extraction, noun phrase coreference resolution and event tracking (i.e., determine which extracted role fillers belong to which event). In this work, we focus only on document-level role filler extraction. Figure 1 provides a representative example of this task. Given an article consisting of multiple paragraphs/sentences, and a fixed set of event types
[S1] ... by special urban troops, four terrorists have been arrested in soacha.
[52] They are responsible for the car bomb attack on the Newspaper El Expectador, to a series of bogotadynamite attacks, to the freeing of a group of paid assassins.
[53] The terrorists are also connected to the murder of Teofilo Forero Castro, ...
[54] General Ramon is the commander of the  $13^{\text{th}}$  infantry brigade.
[SS] He said that at least two of those arrested have fully confessed to having taken part in the accident of Luis Carlos Galan Sarmiento in soacha, Cundinamarca.
[56] ... triumph over organized crime, its accomplices and its protectors.
Machine reader reads through the document
Perpetrator Individual	four terrorists
Perpetrator Organization	-
Target	Newspaper El Espectador
Victim	Teofilo Forero Castro, Luis Carlos Galan Sarmiento
Weapon	car bomb, dynamite
Figure 1: The document-level event role fillers extraction task.
(e.g., terrorist events) and associated roles (e.g., PERPETRATOR INDIVIDUAL, VICTIM, WEAPON), we aim to identify those spans of text that denote the role fillers for each event described in the text. This generally requires both sentence-level understanding and accurate interpretation of the context beyond the sentence. Examples include identifying "Teofilo Forero Castro" (mentioned in S3) as a victim of the car bomb attack event (mentioned in S2), determining there's no role filler in S4 (both of which rely mainly on sentence-level understanding, and identifying "four terrorists" in S1 as a perpetrator individual (which requires coreference resolution across sentence boundaries). Generating the document-level extractions for events is essential in facilitating downstream applications such as information retrieval and article summarization (Yang and Mitchell, 2016), and for real-life applications such as trends analysis of world events (Sundheim, 1992).
Recent work in document-level event role filler extraction has employed a pipeline architecture with separate classifiers for each type of role and for
<sup>1</sup>The task is also referred to as template filling (MUC-4, 1992).
8010
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8010-8020 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
relevant context detection (Patwardhan and Riloff, 2009; Huang and Riloff, 2011). However these methods: (1) suffer from error propagation across different pipeline stages; and (2) require heavy feature engineering (e.g., lexico-syntactic pattern features for candidate role filler extraction; lexical bridge and discourse bridge features for detecting event-relevant sentences at the document level). Moreover, the features are manually designed for a particular domain, which requires linguistic intuition and domain expertise (Nguyen and Grishman, 2015).
Neural end-to-end models have been shown to excel at sentence-level information extraction tasks, such as named entity recognition (Lample et al., 2016; Chiu and Nichols, 2016) and ACE-type within-sentence event extraction (Chen et al., 2015; Nguyen et al., 2016; Wadden et al., 2019). However, to the best of our knowledge, no prior work has investigated the formulation of document-level event role filler extraction as an end-to-end neural sequence learning task. In contrast to extracting events and their role fillers from standalone sentences, document-level event extraction poses special challenges for neural sequence learning models. First, capturing long-term dependencies in long sequences remains a fundamental challenge for recurrent neural networks (Trinh et al., 2018). To model long sequences, most RNN-based approaches use backpropagation through time. But it's still difficult for the models to scale to very long sequences. We provide empirical evidence for this for event extraction in Section 4.3. Second, although pretrained bi-directional transformer models such as BERT (Devlin et al., 2019) better capture long-distance dependencies as compared to an RNN architecture, they still have a constraint on the maximum length of the sequence, which is below the length of many articles about events.
In the sections below, we study how to train and apply end-to-end neural models for event role filler extraction. We first formalize the problem as a sequence tagging task over the tokens in a set of contiguous sentences in the document. To address the aforementioned challenges for neural models applied to long sequences, (1) we investigate the effect of context length (i.e., maximum input segment length) on model performance, and find the most appropriate length; and (2) propose a multi-granularity reader that dynamically aggregates the information learned from the local con
text (e.g., sentence-level) and the broader context (e.g., paragraph-level). A quantitative evaluation and qualitative analysis of our approach on the MUC-4 dataset (MUC-4, 1992) both show that the multi-granularity reader achieves substantial improvements over the baseline models and prior work.
For replication purposes, our repository for the evaluation and preprocessing scripts will be available at https://github.com/xinyadu/doc_event-role.
Event extraction has been mainly studied under two paradigms: detecting the event trigger and extracting the arguments from an individual sentence (e.g., the ACE task (Doddington et al., 2004) $^2$ ), vs. at the document level (e.g., the MUC-4 template-filling task (Sundheim, 1992)).
Sentence-level Event Extraction The ACE event extraction task requires extraction of the event trigger and its arguments from a sentence. For example, in the sentence “... Iraqi soldiers were killed by U.S. artillery ...”, the goal is to identify the “die” event triggered by killed and the corresponding arguments (PLACE, VICTIM, INSTRUMENT, etc.). Many approaches have been proposed to improve performance on this specific task. Li et al. (2013, 2015) explore various hand-designed features; Nguyen and Grishman (2015); Nguyen et al. (2016); Chen et al. (2015); Liu et al. (2017, 2018) employ deep learning based models such as recurrent neural networks (RNNs) and convolutional neural network (CNN). Wadden et al. (2019) utilize pre-trained contextualized representations. The approaches generally focus on sentence-level context for extracting event triggers and arguments and rarely generalize to the document-event extraction setting (Figure 1).
Only a few models have gone beyond individual sentences to make decisions. Ji and Grishman (2008) enforce event role consistency across documents. Liao and Grishman (2010) explore event type co-occurrence patterns to propagate event classification decisions. Similarly, Yang and Mitchell (2016) propose jointly extracting events and entities within a document context. Also related to our work are Duan et al. (2017) and Zhao et al. (2018), which utilize document embeddings to aid
$^{2}$ https://catalog.ldc.upenn.edu/LDC2006T06
8011
event detection with recurrent neural networks. Although these approaches make decisions with cross-sentence information, their extractions are still at the sentence level.
Document-level Event Extraction has been studied mainly under the classic MUC paradigm (MUC-4, 1992). The full task involves the construction of answer key templates, one template per event (some documents in the dataset describe more than one events). Typically three steps are involved — role filler extraction, role filler mention coreference resolution and event tracking. In this work we focus on role filler extraction.
From the modeling perspective, recent work explores both the local and additional context to make the role filler extraction decisions. GLACIER (Patwardhan and Riloff, 2009) jointly considers cross-sentence and noun phrase evidence in a probabilistic framework to extract role fillers. TIER (Huang and Riloff, 2011) proposes to first determine the document genre with a classifier and then identify event-relevant sentences and role fillers in the document. Huang and Riloff (2012) propose a bottom-up approach that first aggressively identifies candidate role fillers (with lexico-syntactic pattern features), and then removes the candidates that are in spurious sentences (i.e., not event-related) via a cohesion classifier (with discourse features). Similar to Huang and Riloff (2012), we also incorporate both intra-sentence and cross-sentence features (paragraph-level features), but instead of using manually designed linguistic information, our models learn in an automatic way how to dynamically incorporate learned representations of the article. Also, in contrast to prior work that is pipeline-based, our approach tackles the task as an end-to-end sequence tagging problem.
There has also been work on unsupervised event schema induction (Chambers and Jurafsky, 2011; Chambers, 2013) and open-domain event extraction (Liu et al., 2019) from documents: the main idea is to group entities corresponding to the same role into an event template. Our models, on the other hand, are trained in supervised way and the event schemas are pre-defined.
Apart from event extraction, there has been increasing interest on cross-sentence relation extraction (Mintz et al., 2009; Peng et al., 2017; Jia et al., 2019). This work assumes that mentions are provided, and thus is more of a mention/entity-level classification problem. Our work instead focuses
on role filler/ span extraction using sequence tagging approaches; role filler type is determined during this process.
Capturing Long-term Dependencies for Neural Sequence Models For training neural sequence models such as RNNs, capturing long-term dependencies in sequences remains a fundamental challenge (Trinh et al., 2018). Most approaches use backpropagation through time (BPTT) but it is difficult to scale to very long sequences. Many variations of models have been proposed to mitigate the effect of long sequence length, such as Long Short Term Memory (LSTM) Networks (Hochreiter and Schmidhuber, 1997; Gers et al., 1999; Graves, 2013) and Gated Recurrent Unit Networks (Cho et al., 2014). Transformer based models (Vaswani et al., 2017; Devlin et al., 2019) have also shown improvements in modeling long text. In our work for document-level event role filler extraction, we also implement LSTM layers in the models as well as utilize the pre-trained representations provided by the bi-directional transformer model - BERT. From an application perspective, we investigate the suitable length of context to incorporate for the neural sequence tagging model in the document-level extraction setting. We also study how to mitigate problems associated with long sequences by dynamically incorporating both sentence-level and paragraph-level representations in the model (Figure 3).
In the following we describe (1) how we transform the document into paired token-tag sequences and formalize the task as a sequence tagging problem (Section 3.1); (2) the architectures of our base  $k$ -sentence reader (Section 3.2) and multi-granularity reader (Section 3.3).
3.1 Constructing Paired Token-tag Sequences from Documents and Gold Role Fillers
We formalize document-level event role filler extraction as an end-to-end sequence tagging problem. The Figure 2 illustrates the general idea. Given a document and the text spans associated with the gold-standard (i.e., correct) fillers for each role, we adopt the BIO (Beginning, Inside, Outside) tagging scheme to transform the document into paired token/BIO-tag sequences..
We construct example sequences of variant context lengths for training and testing our end-to
8012
Figure 2: An overview of our framework for training the sequence reader for event role filler extraction.
end  $k$ -sentence readers (i.e., the single-sentence, double-sentence, paragraph and chunk readers). By "chunk", we mean the chunk of contiguous sentences which is right within the sequence length constraint for BERT - 512 in this case. Specifically, we use a sentence splitter<sup>3</sup> to divide the document into sentences  $s_1$ ,  $s_2$ , ...,  $s_n$ . To construct the training set, starting from each sentence  $i$ , we concatenate the  $k$  contiguous sentences ( $s_i$  to  $s_{i+k-1}$ ) to form overlapping candidate sequences of length  $k$  - sequence 1 consists of  $\{s_1, \ldots, s_k\}$ , sequence 2 consists of  $\{s_2, \ldots, s_{k+1}\}$ , etc. To make the training set balanced, we sample the same number of positive and negative sequences from the candidate sequences, where "positive" sequence contains at least one event role filler, and "negative" sequences contain no event role fillers. To construct the dev/test set, where the reader is applied, we simply group the contiguous  $k$  sentences together in order, producing  $\frac{n}{k}$  sequences (i.e., sequence 1 consists of  $\{s_1, \ldots, s_k\}$ , sequence 2 consists of  $\{s_{k+1}, \ldots, s_{2k}\}$ , etc.) For the paragraph reader, we set  $k$  to average paragraph length for the training set, and to the real paragraph length for test set.
We denote the token in the sequence with  $x$ , the input for the  $k$ -sentence reader is  $\mathbf{X} = \{x_1^{(1)}, x_2^{(1)}, \ldots, x_{l_1}^{(1)}, \ldots, x_1^{(k)}, x_2^{(k)}, \ldots, x_{l_k}^{(k)}\}$ ; where  $x_i^{(k)}$  is the  $i$ -th token of the  $k$ -th sentence, and  $l_k$  is the length of the  $k$ -th sentence.
3.2  $k$ -sentence Reader
Since our general  $k$ -sentence reader does not recognize sentence boundaries, we simplify the notation for the input sequence as  $\{x_1, x_2, \ldots, x_m\}$  here.
Embedding Layer In the embedding layer, we represent each token  $x_{i}$  in the input sequence as the concatenation of its word embedding and contextual token representation:
- Word Embedding: We use the 100-dimensional GloVe pre-trained word embeddings (Pennington et al., 2014) trained from 6B Web crawl data. We keep the pre-trained word embeddings fixed. Given a token  $x_{i}$ , we have its word embedding:  $\mathbf{x}\mathbf{e}_i = \mathbf{E}(x_i)$ .
- Pre-trained LM representation: Contextualized embeddings produced by pre-trained language models (Peters et al., 2018; Devlin et al., 2019) have been proved to be capable of modeling context beyond the sentence boundary and improve performance on a variety of tasks. Here we employ the contextualized representations produced by BERT-base for our  $k$ -sentence labeling model, as well as the multi-granularity reader to be introduced next. Specifically, we use the average of all the 12 layers' representations and freeze the weights (Peters et al., 2019) during training
<sup>3</sup>https://spacy.io/
8013
after empirical trials<sup>4</sup>. Given the sequence  $\{x_{1}, x_{2}, \ldots, x_{m}\}$ , we have:
$$
\mathbf {x b} _ {1}, \mathbf {x b} _ {2}, \dots , \mathbf {x b} _ {m} = \operatorname {B E R T} \left(x _ {1}, x _ {2}, \dots , x _ {m}\right)
$$
We forward the concatenation of the two representations for each token to the upper layers:
$$
\mathbf {x} _ {i} = \operatorname {c o n c a t} \left(\mathbf {x e} _ {i}, \mathbf {x b} _ {i}\right)
$$
BiLSTM Layer To help the model better capture task-specific features between the sequence tokens. We use a multi-layer (3 layers) bi-directional LSTM encoder on top of the token representations, which we denote as BiLSTM:
$$
\begin{array}{l} \left\{\mathbf {p} _ {1}, \mathbf {p} _ {2}, \dots , \mathbf {p} _ {m} \right\} \\ = \operatorname {B i L S T M} \left(\left\{\mathbf {x} _ {1}, \mathbf {x} _ {2}, \dots , \mathbf {x} _ {m} \right\}\right) \\ \end{array}
$$
CRF Layer Drawing inspirations for sentence-level sequence tagging models on tasks like NER (Lample et al., 2016). Modeling the labeling decisions jointly rather than independently improves the models performance (e.g., the tag "I-Weapon" should not follow "B-Victim"). We model labeling decisions jointly using a conditional random field (Lafferty et al., 2001).
After passing  $\{\mathbf{p}_1,\mathbf{p}_2,\dots ,\mathbf{p}_m\}$  through a linear layer, we have  $\mathbf{P}$  of size  $m\times$  size of tag space, where  $\mathbf{P}_{i,j}$  is the score of the tag  $j$  of the  $i$ -th token in the sequence. For a tag sequence  $\mathbf{y} = \{y_1,\dots,y_m\}$ , we have the score for the sequence-tag pair as:
$$
s c o r e (\mathbf {X}, \mathbf {y}) = \sum_ {i = 0} ^ {m} \mathbf {A} _ {y _ {i}, y _ {i + 1}} + \sum_ {i = 1} ^ {m} \mathbf {P} _ {i, y _ {i}}
$$
$\mathbf{A}$  is the transition matrix of scores such that  $\mathbf{A}_{i,j}$  represents the score of a transition from the tag  $i$  to tag  $j$ . A softmax function is applied over scores for all possible tag sequences, which yield a probability for the gold sequence  $\mathbf{y}_{gold}$ . The log-probability of the gold tag sequence is maximized during training. During decoding, the model predicts the output sequence that obtains the maximum score.
3.3 Multi-Granularity Reader
To explore the effect of aggregating contextualized token representations from different granularities
sentence- and paragraph-level), we propose the multi-granularity reader (Figure 3).
Similar to the general  $k$ -sentence reader, we use the same embedding layer here to represent the tokens. But we apply the embedding layer to two granularities of the paragraph text (sentence- and paragraph-level). Although the word embeddings are the same for the embedding layers from different granularities, the contextualized representations are different for each token - when the token is encoded in the context of a sentence, or in the context of a paragraph.
Correspondingly, we build two BiLSTMs (BiLSTM sent. and BiLSTM para.) on top of the sentence-level contextualized token representations  $\{\tilde{\mathbf{x}}_1^{(1)},\dots,\tilde{\mathbf{x}}_{l_1}^{(1)},\dots,\tilde{\mathbf{x}}_{l_k}^{(k)},\dots,\tilde{\mathbf{x}}_{l_k}^{(k)}\}$ , and the paragraph-level contextualized token representations  $\{\hat{\mathbf{x}}_1^{(1)},\dots,\hat{\mathbf{x}}_{l_1}^{(1)},\dots,\hat{\mathbf{x}}_{l_k}^{(k)},\dots,\hat{\mathbf{x}}_{l_k}^{(k)}\}$ :
Sentence-Level BiLSTM The  $\mathrm{BiLSTM}_{\mathrm{sent}}$  is applied sequentially to each sentence in the paragraph:
$$
\begin{array}{l} \{\tilde {\mathbf {p}} _ {1} ^ {(1)}, \tilde {\mathbf {p}} _ {2} ^ {(1)}, \dots , \tilde {\mathbf {p}} _ {l _ {1}} ^ {(1)} \} \\ = \operatorname {B i L S T M} _ {s e n t.} (\{\tilde {\mathbf {x}} _ {1} ^ {(1)}, \tilde {\mathbf {x}} _ {2} ^ {(1)}, \dots , \tilde {\mathbf {x}} _ {l _ {1}} ^ {(1)} \}) \\ \dots \\ \{\tilde {\mathbf {p}} _ {1} ^ {(k)}, \tilde {\mathbf {p}} _ {2} ^ {(k)}, \dots , \tilde {\mathbf {p}} _ {l _ {k}} ^ {(k)} \} \\ = \mathrm {B i L S T M} _ {s e n t.} (\{\tilde {\mathbf {x}} _ {1} ^ {(k)}, \tilde {\mathbf {x}} _ {2} ^ {(k)}, \dots , \tilde {\mathbf {x}} _ {l _ {k}} ^ {(k)} \}) \\ \end{array}
$$
Then we have the sentence-level representations for each token in the paragraph as  $\{\tilde{\mathbf{p}}_1^{(1)},\dots,\tilde{\mathbf{p}}_{l_1}^{(1)},\dots,\tilde{\mathbf{p}}_1^{(k)},\dots,\tilde{\mathbf{p}}_{l_k}^{(k)}\}$
Paragraph-Level BiLSTM Another BiLSTM layer  $(\mathrm{BiLSTM}_{\text {para.}})$  is applied to the entire paragraph (as compared to  $\mathrm{BiLSTM}_{\text {sent. }}$ , which is applied to each sentence), to capture the dependency between tokens in the paragraph:
$$
\begin{array}{l} \{\hat {\mathbf {p}} _ {1} ^ {(1)}, \dots , \hat {\mathbf {p}} _ {l _ {1}} ^ {(1)}, \dots , \hat {\mathbf {p}} _ {1} ^ {(k)}, \dots , \hat {\mathbf {p}} _ {l _ {k}} ^ {(k)} \} \\ = \mathrm {B i L S T M} _ {p a r a.} (\{\hat {\mathbf {x}} _ {1} ^ {(1)}, \dots , \hat {\mathbf {x}} _ {l _ {1}} ^ {(1)},.., \hat {\mathbf {x}} _ {l _ {k}} ^ {(k)}, \dots , \hat {\mathbf {x}} _ {l _ {k}} ^ {(k)} \}) \\ \end{array}
$$
Fusion and Inference Layer For each token  $x_{i}^{(j)}$  (the  $i$ -th token in the  $j$ -th sentence), to fuse the representations learned at the sentence-level  $(\tilde{\mathbf{p}}_i^{(j)})$  and paragraph-level  $(\hat{\mathbf{p}}_i^{(j)})$ , we propose two options - the first uses a sum operation, and the second uses a gated fusion operation:
4 Using the representations of the last layer, or summing all the 12 layers' representations give consistently worse results.
8014
Figure 3: Overview for our multi-granularity reader. The dark blue  $\mathsf{BiLSTM}_{\mathsf{sent}}$  produces sentence-level representations for each token, the yellow  $\mathsf{BiLSTM}_{\mathsf{para}}$  produces paragraph-level representations for each token.
- Simple Sum Fusion:
$$
\mathbf {p} _ {i} ^ {(j)} = \tilde {\mathbf {p}} _ {i} ^ {(j)} + \hat {\mathbf {p}} _ {i} ^ {(j)}
$$
- Gated Fusion: The gated fusion compute the gate vector  $\mathbf{g}_i^{(j)}$  with its sentence-level token representation  $\tilde{\mathbf{p}}_i^{(j)}$  and paragraph-level token representation  $\hat{\mathbf{p}}_i^{(j)}$ , to control how much information should be incorporated from the two representations.
$$
\mathbf {g} _ {i} ^ {(j)} = s i g m o i d (\mathbf {W} _ {1} \tilde {\mathbf {p}} _ {i} ^ {(j)} + \mathbf {W} _ {2} \hat {\mathbf {p}} _ {i} ^ {(j)} + b)
$$
$$
\mathbf {p} _ {i} ^ {(j)} = \mathbf {g} _ {i} ^ {(j)} \odot \tilde {\mathbf {p}} _ {i} ^ {(j)} + (1 - \mathbf {g} _ {i} ^ {(j)}) \odot \hat {\mathbf {p}} _ {i} ^ {(j)}
$$
: element-wise product
Similarly to in the general  $k$ -sentence reader, we add the CRF layer (section 3.2) on top of the fused representations for each token in the paragraph  $\{\mathbf{p}_1^{(1)}, \dots, \mathbf{p}_{l_1}^{(1)}, \dots, \mathbf{p}_1^{(k)}, \dots, \mathbf{p}_{l_k}^{(k)}\}$ , to help jointly model the labeling decisions between tokens in the paragraph.
We evaluate our models' performance on the MUC-4 event extraction benchmark (MUC-4, 1992), and compare to prior work. We also report findings on the effect of context length on the end-to-end readers' performance on this document-level task.
MUC-4 Event Extraction Dataset The MUC-4 dataset consists of 1,700 documents with associated answer key (role filler) templates. To make
sure our results are comparable to the previously reported results on this dataset, we use the 1300 documents for training, 200 documents  $(\mathrm{TST1} + \mathrm{TST2})$  as the development set and the 200 documents  $(\mathrm{TST3} + \mathrm{TST4})$  as the test set.