SlowGuess/ACL_2020_OCR huggingface dataset:
Seth Ebner* Patrick Xia* Ryan Culkin Kyle Rawlins Benjamin Van Durme
{seth，paxia}@cs.jhu.edu
{rculkin, kgr, vandurme}@jhu.edu
We present a novel document-level model for finding argument spans that fill an event's roles, connecting related ideas in sentence-level semantic role labeling and coreference resolution. Because existing datasets for cross-sentence linking are small, development of our neural model is supported through the creation of a new resource, Roles Across Multiple Sentences (RAMS), which contains 9,124 annotated events across 139 types. We demonstrate strong performance of our model on RAMS and other event-related datasets.
Textual event descriptions may span multiple sentences, yet large-scale datasets predominately annotate for events and their arguments at the sentence level. This has driven researchers to focus on sentence-level tasks such as semantic role labeling (SRL), even though perfect performance at such tasks would still enable a less than complete understanding of an event at the document level.
In this work, we approach event understanding as a form of linking, more akin to coreference resolution than sentence-level SRL. An event trigger evokes a set of roles regarded as latent arguments, with these implicit arguments then potentially linked to explicit mentions in the text.
Consider the example in Figure 1: the Airstrike Missile Strike event (triggered by "bombarding") gives rise to a frame or set of type-level roles (attacker, target, instrument, place) with the referents ("Russians", "rebel outpost", "aircraft", "Syria"). Intuitively we recognize the possible existence of fillers for these roles, for example, the place of the particular Air
When Russian aircraft bombed a remote garrison in southeastern Syria last month, alarm bells sounded at the Pentagon and the Ministry of Defense in London.
Figure 1: A passage annotated for an event's type, trigger, and arguments. Each arc points from the trigger to the argument that fills the labeled role.
strikeMissileStrike event. These implicit arguments are linked to explicit arguments in the document (i.e., text spans). We refer to the task of finding explicit argument(s) to fill each role for an event as argument linking.
Prior annotation of cross-sentence argument links has produced small datasets, with a focus either on a small number of predicate types (Gerber and Chai, 2010, 2012; Feizabadi and Padó, 2014) or on a small number of documents (Ruppenhofer et al., 2010). To enable the development of a neural model for argument linking, we produce Roles Across Multiple Sentences (RAMS), a dataset of 9,124 annotated events from news based on an ontology of 139 event types and 65 roles. In a 5-sentence window around each event trigger, we annotate the closest argument span for each role.
Our model builds on recent ideas in span selection models (Lee et al., 2018; He et al., 2018; Ouchi et al., 2018), used in this work for the multisentence argument linking task for RAMS and for several other event-based datasets (Gerber and Chai, 2012; Pradhan et al., 2013; Pavlick et al., 2016, AIDA Phase 1). On RAMS our best model achieves  $68.3\mathrm{F}_1$ , and it achieves  $73.3\mathrm{F}_1$  when event types are also known, outperforming strong baselines. We also demonstrate effective use of RAMS as pre-training for a related dataset.
Data and code at http://nlp.jhu.edu/rams/.
${}^{2}\epsilon$  would indicate there is no explicit referent in the text.
8057
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8057-8077 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Our main contributions are a novel model for argument linking and a new large-scale dataset for the task. Our dataset is annotated for arguments across multiple sentences and has broader coverage of event types and more examples than similar work. Our experiments highlight our model's adaptability to multiple datasets. Together, these contributions further the automatic understanding of events at the document level.
2 Non-local Arguments
We are not the first to consider non-local event arguments; here we review prior work and refer to O'Gorman (2019) for further reading. Whereas local (sentence-level) event arguments are well-studied as semantic role labeling—utilizing large datasets such as OntoNotes 5.0 (Weischedel et al., 2013; Pradhan et al., 2013)—existing datasets annotated for non-local arguments are too small for training neural models.
Much of the effort on non-local arguments, sometimes called implicit SRL, has focused on two datasets: SemEval-2010 Task 10 (Ruppenhofer et al., 2010) and Beyond NomBank (henceforth BNB) (Gerber and Chai, 2010, 2012). These datasets are substantially smaller than RAMS: the SemEval Task 10 training set contains 1,370 frame instantiations over 438 sentences, while BNB contains 1,247 examples covering just 10 nominal predicate types. Multi-sentence AMR (MS-AMR) (O'Gorman et al., 2018; Knight et al., 2020) contains 293 documents annotated with a document-level adaptation of the Abstract Meaning Representation (AMR) formalism. O'Gorman (2019) notes that the relatively small size of the MS-AMR and SemEval datasets hinders supervised training. In contrast to these datasets, RAMS contains 9,124 annotated examples covering a wide range of nominal and verbal triggers.
Under the DARPA AIDA program, the Linguistic Data Consortium (LDC) has annotated document-level event arguments under a three-level hierarchical event ontology (see Figure 2) influenced by prior LDC-supported ontologies such as ERE and ACE. These have been packaged as the AIDA Phase 1 Practice<sup>3</sup> and Eval<sup>4</sup> releases (henceforth AIDA-1), currently made available to performers in the AIDA program and participants
Figure 2: Subset of the AIDA-1 ontology illustrating the three-level Type/Subtype/Sub-subtype event hierarchy. Dashed gray edges point to roles for two event nodes, which have one role in common (Place).
in related NIST evaluations. $^{5}$  AIDA-1 documents focus on recent geopolitical events relating to interactions between Russia and Ukraine. Unless otherwise noted, statistics about AIDA-1 pertain only to the Practice portion of the dataset.
For each document in LDC's collection, only AIDA-salient events are annotated. This protocol does not guarantee coverage over the event ontology: 1,559 event triggers are annotated in the text portion of the collection, accounting for only 88 of the 139 distinct event sub-subtypes in the ontology. Our dataset, RAMS, employs the same annotation ontology but is substantially larger and covers all 139 types in the ontology. Figure 3 (§3) compares the two datasets.
Across multiple datasets, a substantial number of event arguments are observed to be non-local. For example, Gerber and Chai (2012) found that their annotation of non-local arguments added  $71\%$  (relative) role coverage to NomBank annotations. Additionally,  $38.1\%$  of the annotated events in AIDA-1 have an argument outside the sentence containing the trigger. This phenomenon is not surprising in light of the analysis of zero anaphora and definite null complements by Fillmore (1986) and the distinction between "core" and "non-core" frame elements or roles in FrameNet (Baker et al., 1998) and PropBank (Palmer et al., 2005).
As previous datasets have been small, various approaches have been taken to handle scarcity. To obtain more training data, Silberer and Frank (2012) created artificial instances from data annotated jointly for coreference and semantic roles. Roth and Frank (2013) automatically induced implicit arguments from pairs of comparable texts, but recovered a proportionally small set of additional arguments. Feizabadi and Padó (2015)
5While rarely freely released, historically such collections are eventually made available under a license to anyone, under some timeline established within a program.
$^{3}$ LDC2019E04 (data); LDC2019E07 (annotations)
$^{4}$ LDC2019E42 (data); LDC2019E77 (annotations)
8058
combined existing corpora to increase and diversify sources of model supervision. Cheng and Erk (2018, 2019) approached the data scarcity problem by recasting implicit SRL as a cloze task and as a reading comprehension task, for which data can be generated automatically.
The TAC KBP event argument extraction task also seeks arguments from document contexts. However, in our work we are concerned with reified events (explicit mentions) and links between event mentions and argument mentions rather than entity-level arguments (coreference clusters).
3 RAMS
Motivated by the scarcity of data for training neural models to predict non-local arguments, we constructed Roles Across Multiple Sentences (RAMS), a crowd-sourced dataset with annotations for 9,124 events following the AIDA ontology. We employed the AIDA ontology in RAMS so-as to be most similar to an existing corpus already being investigated by various members of the community. Each example consists of a typed trigger span and 0 or more argument spans in an English document. A trigger span is a word or phrase that evokes a certain event type in context, while argument spans denote role-typed participants in the event (e.g., the Recipient). Trigger and argument spans are token-level [start, end] offsets into a tokenized document.
Typically, event and relation datasets annotate only the argument spans that are in the same sentence as the trigger, but we present annotators with a multi-sentence context window surrounding the trigger. Annotators may select argument spans in any sentence in the context window.
3.1 Dataset Description
Data Source We used Reddit, a popular internet forum, to filter a collection of news articles to be topically similar to AIDA-1. After applying a set of criteria based on keywords, time period, and popularity (listed in Appendix A.1) we identified approximately 12,000 news articles with an average length of approximately 40 sentences.
Annotation We manually constructed a mapping from each event ((sub-)sub)type to a list of lexical units (LUs) likely to evoke that type. This mapping was designed to give high precision and
Train	Dev	Test	Total
Docs	3,194	399	400	3,993
Examples	7,329	924	871	9,124
Event Types	139	131	-	139
Roles	65	62	-	65
Arguments	17,026	2,188	2,023	21,237
Table 1: Sizes and coverage of RAMS splits. RAMS covers all of the 139 event types and 65 roles types in the AIDA Phase 1 ontology.
low recall, in that for a given (Type, LUs) pair, the items in LUs are all likely to evoke the Type, although LUs can omit items that also evoke the Type. On average,  $|\mathrm{LUs}| = 3.9$ .
We performed a soft match between every LU and every word in our text collection to select candidate sentences for each event type. This matching procedure produced approximately 94,000 candidates, which we balanced by sampling the same number of sentences for each LU.
Candidate sentences were then vetted by crowdsourcing to ensure that they evoked their associated event type and had positive factuality. We collected judgments on approximately 17,500 candidate sentences, of which  $52\%$  were determined to satisfy these constraints, yielding 9,124 sentences containing a LU trigger. Using these sentences we then collected multi-sentence annotations, presenting annotators with a 5-sentence window containing two sentences of context before the sentence with the trigger and two sentences after. Annotators then selected in the context window a span to fill each of the event's roles.
A window size of five sentences was chosen based on internal pilots and supported by our finding that  $90\%$  of event arguments in AIDA-1 are recoverable in this window size. Similarly, Gerber and Chai (2010) found that in their data almost  $90\%$  of implicit arguments can be resolved in the two sentences preceding the trigger.[9] Arguments fall close to the trigger in RAMS as well:  $82\%$  of arguments occur in the same sentence as the trigger. On average, we collected 66 full annotations (trigger and arguments) per event type. Table 1 shows dataset size and coverage. All aspects of the protocol, including the annotation interface and instructions, are included in Appendix A.
<sup>7</sup>We stem all words and ignore case.
<sup>8</sup>If fewer than two sentences appeared before/after the trigger, annotators were shown as many sentences as were available.
<sup>9</sup>Arguments following the trigger were not annotated.
For example, Conflict/Attack/SetFire is evoked by inferno, blaze, and arson (and word forms).
8059
Inter-Annotator Agreement We randomly selected 93 tasks for redundant annotation in order to measure inter-annotator agreement, collecting five responses per task from distinct users.  $68.5\%$  of the time, all annotators mark the role as either absent or present. Less frequently  $(21.7\%)$ , four of the five annotators agree, and rarely  $(9.8\%)$  is there strong disagreement.
We compute pairwise agreement for span boundaries. For each annotated (event, role) combination, we compare pairs of spans for which both annotators believe the role is present.  $55.3\%$  of the pairs agree exactly. Allowing for a fuzzier match, such as to account for whether one includes a determiner, spans whose boundaries differ by one token have a much higher agreement of  $69.9\%$ . Fewer spans agree on the start boundary  $(59.8\%)$  than on the end  $(73.5\%)$ , while  $78.0\%$  match at least one of the two boundaries. We demonstrate data quality in §5.2 by showing its positive impact on a downstream task.
Figure 3: Comparison of frequency of event types in various datasets sorted by decreasing frequency in that dataset. RAMS has a heavier tail than AIDA-1 and BNB and broader coverage of events.
Comparisons to Related Datasets Comparisons of event type coverage among RAMS, AIDA-1, and BNB (Gerber and Chai, 2010, 2012) are given in Figure 3. RAMS provides larger and broader coverage of event types than do AIDA-1 and BNB. By design, BNB focuses on only a few predicate types, but we include its statistics for reference. More figures regarding type and role coverage are included in Appendix A.4.
Related Protocols Feizabadi and Padó (2014) also considered the case of crowdsourcing annotations for cross-sentence arguments. Like us, they provided annotators with a context window rather than the whole document, annotating two frames each with four roles over 384 predicates. Annota
tors in that work were shown the sentence containing the predicate and the three previous sentences, unlike ours which shows two preceding and two following sentences.
Rather than instructing annotators to highlight spans in the text ("marking"), Feizabadi and Padó (2014) directed annotators to fill in blanks in tem- platic sentences ("gap filling"). We in contrast require annotators to highlight mention spans directly in the text.
Our protocol of event type verification followed by argument finding is similar to the protocol supported by interfaces such as SALTO (Burchardt et al., 2006) and that of Fillmore et al. (2002).
We formulate argument linking as follows, similar to the formulation in Das et al. (2010). Assume a document  $\mathcal{D}$  contains a set of described events  $\mathcal{E}$ , each designated by a trigger—a text span in  $\mathcal{D}$ . The type of an event  $e$  determines the set of roles the event's arguments may take, denoted  $\mathcal{R}_e$ . For each  $e \in \mathcal{E}$ , the task is to link the event's roles with arguments—text spans in  $\mathcal{D}$ —if they are attested. Specifically, one must find for each  $e$  all  $(r, a)$  pairs such that  $r \in \mathcal{R}_e$  and  $a \in \mathcal{D}$ . This formulation does not restrict each role to be filled by only one argument, nor does it restrict each explicit argument to take at most one role.
Our model architecture is related to recent models for SRL (He et al., 2018; Ouchi et al., 2018). Contextualized text embeddings are used to form candidate argument span representations,  $\mathcal{A}$ . These are then pruned and scored alongside the trigger span and learned role embeddings to determine the best argument span (possibly none) for each event and role, i.e.,  $\operatorname{argmax}_{a\in \mathcal{A}}P(a\mid e,r)$  for each event  $e\in \mathcal{E}$  and role  $r\in \mathcal{R}_e$ .
Representations To represent text spans, we adopt the convention from Lee et al. (2017) that has been used for a broad suite of core NLP tasks (Swayamdipta et al., 2018; He et al., 2018; Tenney et al., 2019b). A bidirectional LSTM encodes each sentence's contextualized embeddings (Peters et al., 2018; Devlin et al., 2018). The hidden states at the start and end of the span are concatenated along with a feature vector for the size of the span and a soft head word vector produced by a learned attention mask over the word vectors
(GloVe embeddings (Pennington et al., 2014) and character-level convolutions) within the span.
We use this method to form representations of trigger spans,  $\mathbf{e}$ , and of candidate argument spans, a. We learn a separate embedding,  $\mathbf{r}$ , for each role in the ontology,  $r\in \mathcal{R}$ . Since our objective is to link candidate arguments to event-role pairs, we construct an event-role representation by applying a feed-forward neural network  $(\mathrm{F}_{\tilde{a}})$  to the event trigger span and role embedding:
$$
\tilde {\mathbf {a}} _ {e, r} = \mathrm {F} _ {\tilde {a}} ([ \mathbf {e}; \mathbf {r} ]) \tag {1}
$$
This method is similar to one for forming edge representations for cross-sentence relation extraction (Song et al., 2018), but contrasts with prior work which limits the interaction between  $r$  and  $e$  (He et al., 2018; Tenney et al., 2019b).
Pruning Given a document with  $n$  tokens, there are  $O(n^{2})$  candidate argument text spans, which leads to intractability for large documents. Following Lee et al. (2017) and He et al. (2018), we consider within-sentence spans up to a certain width (giving  $O(n)$  spans) and score each span,  $a$ , using a learned unary function of its representation:  $s_A(a) = \mathbf{w}_A^\top \mathbf{F}_A(\mathbf{a})$ . We keep the top  $\lambda_{A}n$  spans  $(\lambda_{A}$  is a hyperparameter) and refer to this set of high-scoring candidate argument spans as  $\mathcal{A}$ .
In an unpruned model, we need to create at least  $\sum_{e}|\mathcal{R}_{e}|$  event-role representations and evaluate  $\Omega (n\sum_{e}|\mathcal{R}_{e}|)$  combinations of events, roles, and arguments, which can become prohibitively large when there are numerous events and roles. Assuming the number of events is linear in document length, the number of combinations would be quadratic in document length (rather than quadratic in sentence length as in He et al. (2018)).
Lee et al. (2018) addressed this issue in coreference resolution, a different document-level task, by implementing a coarse pruner to limit the number of candidate spans that are subsequently scored. For our model, any role can potentially be filled (if the event type is not known). Thus, we do not wish to prematurely prune  $(e,r)$  pairs, so we must further prune  $\mathcal{A}$ . Rather than scoring  $a\in \mathcal{A}$  with every event-role pair  $(e,r)$ , we assign a score between  $a$  and every event  $e$ . This relaxation reflects a loose notion of how likely an
argument span is to participate in an event, which can be determined irrespective of a role:
$$
s _ {c} (e, a) = \mathbf {e} ^ {\top} \mathbf {W} _ {c} \mathbf {a} + s _ {A} (a) + s _ {E} (e) + \phi_ {c} (e, a)
$$
where  $\mathbf{W}_c$  is learned and  $\phi_c(e, a)$  are task-specific features. We use  $\mathcal{A}_e \subseteq \mathcal{A}$  to refer to the top- $k$ -scoring candidate argument spans in relation to  $e$ .
Scoring We introduce a link scoring function,  $l(a,\tilde{a}_{e,r})$ , between candidate spans  $a \in \mathcal{A}_e$  and event-role pairs  $\tilde{a}_{e,r} = (e,r) \in \mathcal{E} \times \mathcal{R}$ . The scoring function decomposes as:
$$
\begin{array}{l} l (a, \tilde {a} _ {e, r}) = s _ {E, R} (e, r) + s _ {A, R} (a, r) \\ + s _ {l} \left(a, \tilde {a} _ {e, r}\right) + s _ {c} (e, a), \quad a \neq \epsilon \tag {2} \\ \end{array}
$$
$$
s _ {E} (e) = \mathbf {w} _ {E} ^ {\top} \mathbf {F} _ {E} (\mathbf {e})
$$
$$
s _ {E, R} (e, r) = \mathbf {w} _ {E, R} ^ {\top} \mathbf {F} _ {E, R} ([ \mathbf {e}; \mathbf {r} ])
$$
$$
s _ {A, R} (a, r) = \mathbf {w} _ {A, R} ^ {\top} \mathbf {F} _ {A, R} ([ \mathbf {a}; \mathbf {r} ])
$$
$$
\begin{array}{l} s _ {l} (a, \tilde {a} _ {e, r}) = \mathbf {w} _ {l} ^ {\top} \mathrm {F} _ {l} ([ \mathbf {a}; \tilde {\mathbf {a}} _ {e, r}; \mathbf {a} \circ \tilde {\mathbf {a}} _ {e, r}; \\ \left. \phi_ {l} \left(a, \tilde {a} _ {e, r}\right) \right]) \tag {3} \\ \end{array}
$$
where  $\phi_l(a, \tilde{a}_{e,r})$  is a feature vector containing information such as the (bucketed) token distance between  $e$  and  $a$ .<sup>12</sup>  $\mathrm{F}_x$  are feed-forward neural networks, and  $\mathbf{w}_x$  are learned weights. The decomposition is inspired by Lee et al. (2017) and He et al. (2018), while the direct scoring of candidate arguments against event-role pairs,  $s_l(a, \tilde{a}_{e,r})$ , bears similarities to the approach taken by Schenk and Chiarcos (2016), which finds the candidate argument whose representation is most similar to the prototypical filler of a frame element (role).
Learning We denote "no explicit argument" by  $\epsilon$  and assign it link score  $l(\epsilon, \tilde{a}_{e,r}) \triangleq 0$ , which acts as a threshold for the link function. For every event-role-argument triple  $(e,r,a)$ , we maximize
$$
P (a \mid e, r) = \frac {\exp \{l (a , \tilde {a} _ {e , r}) \}}{\sum_ {a ^ {\prime} \in \mathcal {A} _ {e} \cup \{\epsilon \}} \exp \left\{l \left(a ^ {\prime} , \tilde {a} _ {e , r}\right) \right\}}.
$$
Decoding We experiment with three decoding strategies: argmax, greedy, and type-constrained. If we assume each role is satisfied by exactly one argument (potentially  $\epsilon$ ), we can perform argmax decoding independently for each role:
$$
\hat {a} = \operatorname {a r g m a x} _ {a \in \mathcal {A} _ {e} \cup \{\epsilon \}} P (a \mid e, r)
$$
<sup>10</sup> As a role for an event evokes an implicit discourse refer- erent, this can be regarded as an implicit discourse referent representation.
<sup>11</sup>If the type of  $e$  is known, then we could restrict  $r \in \mathcal{R}_e$ .
12Distance  $=$  max  $(e_{start} - a_{end},a_{start} - e_{end})$
8061
To instead predict multiple non-overlapping arguments per role, we could use  $P(\epsilon \mid e, r)$  as a threshold in greedy decoding (Ouchi et al., 2018).
We may know the gold event types and the mapping between events  $e$  and their permitted roles,  $\mathcal{R}_e$ . While this information can be used during training, we take a simpler approach of using it for type-constrained decoding (TCD). If an event type allows  $m_r$  arguments for role  $r$ , we keep only the top-scoring  $m_r$  arguments based on link scores.
4.2 Related Models
Our model is inspired by several recent span selection models (He et al., 2018; Lee et al., 2018; Ouchi et al., 2018), as well as the long line of neural event extraction models (Chen et al., 2015; Nguyen et al., 2016, inter alia). O'Gorman (2019) speculates a joint coreference and SRL model in which implicit discourse referents are generated for each event predicate and subsequently clustered with the discovered referent spans using a model for coreference, which is similar to the approach of Silberer and Frank (2012). O'Gorman (2019) further claims that span selection models would be difficult to scale to the document level, which is the regime we are most interested in. We focus on the implicit discourse referents (i.e., the event-role representations) for an event and link them to argument mentions, rather than cluster them using a coreference resolution system or aggregate event structures across multiple events and documents (Wolfe et al., 2015). Our approach is also similar to the one used by Das et al. (2010) for FrameNet parsing.