SlowGuess/NAACL_2021_OCR huggingface dataset:
Sha Li and Heng Ji and Jiawei Han
University of Illinois at Urbana-Champaign, IL, USA
{shal2, hengji, hanj} @illinois.edu
Event extraction has long been treated as a sentence-level task in the IE community. We argue that this setting does not match human information seeking behavior and leads to incomplete and uninformative extraction results. We propose a document-level neural event argument extraction model by formulating the task as conditional generation following event templates. We also compile a new document-level event extraction benchmark dataset WIKIEVENTS which includes complete event and coreference annotation. On the task of argument extraction, we achieve an absolute gain of  $7.6\%$  F1 and  $5.7\%$  F1 over the next best model on the RAMS and WIKIEVENTS datasets respectively. On the more challenging task of informative argument extraction, which requires implicit coreference reasoning, we achieve a  $9.3\%$  F1 gain over the best baseline. To demonstrate the portability of our model, we also create the first end-to-end zero-shot event extraction framework and achieve  $97\%$  of fully supervised model's trigger extraction performance and  $82\%$  of the argument extraction performance given only access to 10 out of the 33 types on ACE.
By converting a large amount of unstructured text into trigger-argument structures, event extraction models provide unique value in assisting us process volumes of documents to form insights. While real-world events are often described throughout a news document (or even span multiple documents), the scope of operation for existing event extraction models have long been limited to the sentence level.
Early work on event extraction originally posed the task as document level role filling (Grishman and Sundheim, 1996) on a set of narrow scenarios
Figure 1: Two examples of cross-sentence inference for argument extraction from our WIKI EVENTS dataset. The PaymentBarter argument of the Transaction.ExchangeBuySell event triggered by "reserved" in the first sentence can only be found in the next sentence. The Attack.ExplodeDetonate event triggered by "detonated" in the third sentence has an uninformative argument "he", which needs to be resolved to the name mention "McVeigh" in the previous sentences.
and evaluated on small datasets. The release of  $\mathrm{ACE}^2$ , a large scale dataset with complete event annotation, opened the possibility of applying powerful machine learning models which led to substantial improvement in event extraction. The success of such models and the widespread adoption of ACE as the training dataset established sentence-level event extraction as the mainstream task definition.
This formulation signifies a misalignment between the information seeking behavior in real life and the exhaustive annotation process in creating the datasets. An information seeking session (Mai, 2016) can be divided into 6 stages: task initiation, topic selection, pre-focus exploration, focus information, information collection and search closure (Kuhlthau, 1991). Given a target event ontology, we can safely assume that topic selection is complete and users start from skimming the documents before they discover events of interest, focus on such events and then aggregate all relevant information for the events. In both the "pre-focus
1The programs, data and resources are publicly available for research purpose at https://github.com/raspberryice/gen-arg.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 894-908 June 6-11, 2021. ©2021 Association for Computational Linguistics
exploration" and "information collection" stages, users naturally cross sentence boundaries.
Empirically, using sentence boundaries as event scopes conveniently simplifies the problem, but also introduces fundamental flaws: the resulting extractions are incomplete and uninformative. We show two examples of this phenomenon in Figure 1. The first example exemplifies the case of implicit arguments across sentences. The sentence that contains the PaymentBarter argument "$280.32" is not the sentence that contains the trigger "reserve" for the ExchangeBuySell event. Without a document-level model, such arguments would be missed and result in incomplete extraction. In the second example, the arguments are present in the same sentence, but written as pronouns. Such extraction would be uninformative to the reader without cross-sentence coreference resolution.
We propose a new end-to-end document-level event argument extraction model by framing the problem as conditional generation given a template. Conditioned on the unfilled template and a given context, the model is asked to generate a filled-in template with arguments as shown in Figure 2. Our model does not require entity recognition nor coreference resolution as a preprocessing step and can work with long contexts beyond single sentences. Since templates are usually provided as part of the event ontology definition, this requires no additional human effort. Compared to recent efforts (Du and Cardie, 2020; Feng et al., 2020; Chen et al., 2020) that retarget question answering (QA) models for event extraction, our generation-based model can easily handle the case of missing arguments and multiple arguments in the same role without the need of tuning thresholds and can extract all arguments in a single pass.
In order to evaluate the performance of document-level event extraction, we collect and annotate a new benchmark dataset WIKI EVENTS. This document-level evaluation also allows us to move beyond the nearest mention of the argument and instead seek the most informative mention in the entire document context. In particular, only  $34.5\%$  of the arguments detected in the same sentence as the trigger can be considered informative. We present this new task of document-level informative argument extraction and show that while this task requires much more cross-sentence infer
ence, our model can still perform reliably well.
Since we provide the ontology information (which roles are needed for the event) through the template as an external condition, our model has excellent portability to unseen event types. By pairing up our argument extraction model with a keyword-based zero-shot trigger extraction model, we enable zero-shot transfer for new event types.
1. We address the document-level argument extraction task with an end-to-end neural event argument extraction model by conditional text generation. Our model does not rely on entity extraction nor entity/event coreference resolution. Compared to QA-based approaches, it can easily handle missing arguments and multiple arguments in the same role.
2. We present the first document-level event extraction benchmark dataset with complete event and coreference annotation. We also introduce the new document-level informative argument extraction task, which evaluates the ability of models to learn entity-event relations over long ranges.
3. We release the first end-to-end zero-shot event extraction framework by combining our argument extraction model with a zero-shot event trigger classification model.
The event extraction task consists of two subtasks: trigger extraction and argument extraction. The set of possible event types and roles for each event type are given by the event ontology as part of the dataset. One template for each event type is usually pre-defined in the ontology.
We first introduce our document-level argument extraction model in Section 2.1 and then introduce our zero-shot keyword-based trigger extraction model in Section 2.2.
2.1 Argument Extraction Model
We use a conditional generation model for argument extraction, where the condition is an unfilled template and a context. The template is a sentence
4 ACE does not come with templates, but since the event types are subsets of the RAMS AIDA ontology and the KAIROS ontology, we reused templates from the these ontologies.
3We prefer name mentions over nominal mentions and only use pronoun mentions when no other mentions exist.
Figure 2: Our argument extraction model using conditional generation. On the left we show an example document, template and the desired output for the instance. Each example document may contain multiple event triggers and we use special  $\langle tgr\rangle$  tokens to markup the target event trigger for argument extraction (the highlighted word "reserved"). The input to the model is the concatenation of the template and the document. The decoded tokens are either from the template or the document. The color of the generated tokens indicate its copy source. After the filled template is generated, we extract the spans to produce the final output.
that describes the event with  $\langle arg\rangle$  placeholders. The generated output is a filled template where placeholders are replaced by concrete arguments. An example of the unfilled template from the ontology and the filled template for the event type Transaction.ExchangeBuySell  $^5$  can be seen in Figure 2. Notably, one template per event type is given in the ontology, and does not require further human curation as opposed to the question designing process in question answering (QA) models (Du and Cardie, 2020; Feng et al., 2020).
Our base model is an encoder-decoder language model (BART (Lewis et al., 2020), T5 (Raffel et al., 2020). The generation process models the conditional probability of selecting a new token given the previous tokens and the input to the encoder.
$$
p (x \mid c) = \prod_ {i = 1} ^ {| x |} p \left(x _ {i} \mid x _ {<   i}, c\right) \tag {1}
$$
In the encoder, bidirectional attention layers are used to enable interaction between every pair of tokens and produce the encoding for the context  $c$ . Each layer of the decoder performs cross-attention over the output of the encoder in addition to the attention over the previous decoded tokens.
To utilize the encoder-decoder LM for argument extraction, we construct an input sequence of  $\langle s\rangle$  template  $\langle s\rangle \langle /s\rangle$  document  $\langle /s\rangle$ . All argument names (arg1, arg2, etc.) in the template are replaced by a special placeholder token  $\langle \mathrm{arg}\rangle$ . The
ground truth sequence is the filled template where the placeholder token is replaced by the argument span whenever possible. In the case where there are multiple arguments for the same slot, we connect the arguments with the word "and".
The generation probability is computed by taking the dot product between the decoder output and the embeddings of tokens from the input.
$$
\mathrm {p} \left(\mathrm {x} _ {i} = w \mid x _ {<   i}, c, t\right) = \left\{ \begin{array}{l l} \text {S o f t m a x} \left(h _ {i} ^ {T} \operatorname {E m b} (w)\right) & w \in V _ {c} \\ 0 & w \notin V _ {c} \end{array} \right. \tag {2}
$$
To prevent the model from hallucinating arguments, we restrict the vocabulary of words to  $V_{c}$ : the set of tokens in the input.
The model is trained by minimizing the negative loglikelihood over all (content, template, output) instances in the dataset  $D$ :
$$
\mathcal {L} (D) = - \sum_ {i = 1} ^ {| D |} \log p _ {\theta} \left(x ^ {i} \mid c ^ {i}\right) \tag {3}
$$
The event ontology often imposes entity type constraints on the arguments. When using the template only, the model has no access to such constraints and can generate seemingly fluent and sensible responses with the wrong arguments. Inspired by (Shwartz et al., 2020), we use clarification statements to add back constraints without breaking the end-to-end property of the model.
In the example presented in Table 1, we can see that the greedy decoding selects "tax plan" as the second Participant argument for the PublicStatement event. Apart from the preposition "with",
This type is used for a transaction transferring or obtaining money, ownership, possession, or control of something, applicable to any type, nature, or method of acquisition including barter.
Context	Original	After
When outlining her tax reform policy, Clinton has made clear that she wants to tax the wealthy and make sure they “pay their fair share.” She has proposed (PublicStatement) a tax plan that would require millionaires and billionaires to pay more taxes than middle-class and lower-income individuals.	She communicated with tax plan about ⟨arg⟩ at ⟨arg⟩ place. She is a person/organization/country. tax plan is a person/organization/country.	She communicated with ⟨arg⟩ about tax plan at ⟨arg⟩ place. She is a person/organization/country.
Table 1: Example of adding type constraints through clarification. The Participant argument of PublicStatement event can only be a person, organization or geo-political entity. The Topic argument can be any type of entity.
there is nothing in the template indicating that this slot should be filled in with a person instead of a topic. To remedy this mistake, we append "clarifications" for its argument fillers in the form of type statements:  $\langle \arg \rangle$  is a  $\langle$  type  $\rangle$ . We then rerank the candidate outputs by the language modeling probability of the filled template and clarifications. When there are multiple valid types, we take the maximum probability of the valid type statements.
$$
\log p (x \mid c) = \sum_ {i} \log p \left(x _ {i} \mid x _ {<   i}, c\right) + \max  _ {e \in E _ {r}} \log p \left(z _ {e} \mid x, c\right) \tag {4}
$$
$E_{r}$  is the set of valid entity types for the role  $r$  according to the ontology and  $z_{e}$  is the type statement. Since "tax plan is a person." goes against commonsense, the probability of generating this sentence will be low. In this way, we can prune responses with conflicting entity types.
2.2 Keyword-Based Trigger Extraction Model
Our argument extraction model relies on detected event triggers (type and offset) as input. Any trigger extraction model could be used in practice, but here we describe a trigger extraction model designed to work with only keyword-level supervision. For example for the "StartPosition" event, we use 3 keywords "hire, employ and appoint" as initial supervision with no mention level annotation. This module allows quick transfer to new event types of interest.
We treat the trigger extraction task as sequence labeling and our model is an adaptation of TapNet (Yoon et al., 2019; Hou et al., 2020), which was designed for few-shot classification and later extended to Conditional Random Field (CRF) models. Compared with (Hou et al., 2020), we do not collapse the entries of the transition matrix, making it possible for our model to learn different probabilities for each event type. Since our model takes
class keywords as input, we refer to this model as TAPKEY.
For each event type, we first obtain a class representation vector  $c_{k}$  based on given keywords using the masked category prediction method in (Meng et al., 2020). This class representation vector is an average over the BERT vector representations of the keywords, with some filtering applied to remove ambiguous occurrences. Details of the filtering process are included in Appendix A.
Following the linear-chain CRF model, the probability of a tagged sequence is:
$$
\log p (y | h; \theta) \propto \sum_ {i} \varphi \left(y _ {i} \mid h _ {i}\right) + \sum_ {i} \psi \left(y _ {i} \mid y _ {i - 1}, h _ {i}\right) \tag {5}
$$
$h_i$  is the output of the embedding network (in our case, BERT-large) corresponding to  $x_i$ .
The label space for  $y_{i}$  is the set of IO tags. We choose to use this simplified tagging scheme because it has fewer parameters and the fact that consecutive triggers of the same event are very rare.
The feature function  $\varphi (\cdot)$  is defined as
$$
\varphi \left(y _ {i} = k \mid h _ {i}\right) = \operatorname {S o f t m a x} \left(M \left(h _ {i}\right) ^ {T} M \left(\phi_ {k}\right)\right) \tag {6}
$$
$\phi_{k}$  is a normalized reference vector for class  $k$  and  $M$  is a projection matrix, both of which are parameters of the model.  $M$  is not a learned parameter, but solved by taking the QR decomposition of a modified reference vector matrix. Specifically,  $M$  satisfies the following equation:
$$
M ^ {T} \left(c _ {k} - \lambda \hat {\phi} _ {k}\right) = 0 \tag {7}
$$
We refer to the TapNet (Yoon et al., 2019) paper for details and also provide a simplified derivation in Appendix A.
The transition score  $\psi (\cdot)$  between tags is parameterized using two diagonal matrices  $W$  and  $W_{o}$ :
$$
\psi \left(y _ {i} = k \mid y _ {i - 1} = l, h _ {i}\right) = \left\{ \begin{array}{l l} M \left(\phi_ {k}\right) W M \left(h _ {i}\right) & k = l \neq 0 \\ M \left(\phi_ {k}\right) W _ {o} M \left(h _ {i}\right) & k \text {o r} l = 0 \\ 0 & k \neq l \end{array} \right. \tag {8}
$$
In a fully supervised setting, it might be desirable to use a supervised trigger extraction model for optimal performance.
Train	Dev	Test
# Event types	49	35	34
# Arg types	57	32	44
# Docs	206	20	20
# Sentences	5262	378	492
# Events	3241	345	365
Table 2: Statistics for the WIKI EVENTS dataset.
Figure 3: Distribution of event types and argument types in the WIKI EVENTS dataset.
In the training stage, the model parameters  $\{\phi ,W,\theta_f\}$  are learned by minimizing the negative log probability of the sequences.
$$
\mathcal {L} = - \frac {1}{N} \sum \log p (y | h; \theta) + \alpha \| \Phi^ {T} \Phi - I \| ^ {2} \tag {9}
$$
The matrix  $\Phi$  of all reference vectors is initialized as a diagonal matrix and the second term regularizes the vectors to be close to orthonormal during training.  $\alpha$  is a hyperparameter.
In the zero-shot setting, we first train on pseudo labeled data before we apply the model. In the pseudo labeling stage, we directly use the cosine similarity between class vectors and the embeddings of the tokens from the language model to assign labels to text. We only use labels with high confident for both event I tags and O tags. The remainder of the tokens will be tagged as X for unknown. Then we train the model on the token classification task. Since none of the parameters in the model are class-specific, the model can be used in a zero-shot transfer setting.
3 Benchmark Dataset WIKIEVENTS
3.1 Evaluation Tasks
Our dataset evaluates two tasks: argument extraction and informative argument extraction.
For argument extraction, we use head word F1 (Head F1) and coreferential mention F1 (Coref F1) as metrics. We consider an argument span to be correctly identified if the offsets match the reference. If the argument role also matches, we consider the argument is correctly classified. Since annotators
Figure 4: Distribution of distance between event trigger and arguments. Distance is measured in number of words.
are asked to annotate the head word when possible, we refer to this metric as Head F1. For Coref F1, the model is given full credit if the extracted argument is coreferential with the gold-standard argument as used in (Ji and Grishman, 2008).
For downstream applications such as knowledge base construction and question answering, argument fillers that are pronouns will not be useful to the user. Running an additional coreference resolution model to resolve them will inevitably introduce propagation errors. Hence, we propose a new task: document-level informative argument extraction. We define name mentions to be more informative than nominal mentions, and pronouns to be the least informative. When the mention type is the same, we select the longest mention as the most informative one. Under this task, the model will only be given credit if the extracted argument is the most informative mention in the entire document.
3.2 Dataset Creation
We collect English Wikipedia articles that describe real world events and then follow the reference links to crawl related news articles. We first manually identify category pages such as https://en.wikipedia.org/wiki/Category:
Improvised_explosive_device_bombings_in_the_United_States and then for each event page (i.e. https://en.wikipedia.org/wiki/Boston_Marathon_bombing), we record all the links in its "Reference" section and use an article scraping tool<sup>7</sup> to extract the full text of the webpage.
We follow the recently established ontology from the KAIROS project for event annotation. This ontology defines 67 event types in a three level hierarchy. In comparison, the commonly used
$^{7}$ https://github.com/codelucas/newspaper  
 $^{8}$ https://www.ldc.upenn.edu/collaborations/current-projects
ACE ontology has 33 event types defined in two levels.
We hired graduate students as annotators and provided example sentences for uncommon event types. A total of 26 annotators were involved in the process. We used the BRAT<sup>9</sup> interface for online annotation.
The annotation process is divided into 2 stages: event mention (trigger and argument) annotation and event coreference annotation. In addition to coreferential mention clusters, we also provide the most informative mention for each cluster. Details about the data collection and annotation process can be found in Appendix B.
Overall statistics of the dataset are listed in Table 2. Compared to ACE, our WIKI EVENTS dataset has a much richer event ontology, especially for argument roles. The observed distributions of event types and argument roles are shown in Figure 3.
We further examine the distance between the event trigger and arguments in Figure 4. When considering the nearest argument mention, the distribution of the arguments is very concentrated towards 0, showing that this annotation standard favors local extractions. In the case of extracting informative mentions, we have a relatively flat long tail distribution with the average distance being 68.82 words (compared to 4.75 words for the nearest mention). In particular, only  $34.5\%$  of the arguments detected in the same sentence as the trigger can be considered informative. This confirms the need for document level inference in the search of informative argument fillers.
Our experiments fall under three settings: (1) document-level event argument extraction; (2) document-level informative argument extraction and (3) zero-shot event extraction.
For document-level event argument extraction we follow the conventional approach of regarding the argument mention with closest proximity to the trigger as the ground truth. In the second setting we consider the most informative mention of the argument as the ground truth.
The zero-shot setting examines the portability of the model to new event types. Under this setting we consider a portion of the event types to be
Split	Event Types	# Sents	# Events
Full Training	33	17172	4202
Freq	10	17172	3398
Ontology	8	17172	1311
Dev	-	923	450
Test	-	832	403
Table 3: Dataset statistics for ACE under multiple settings. In the Freq split, we keep the 10 most frequent event types. In the Ontology split, we keep 1 event subtype per general type in LIFE, MOVEMENT, TRANSACTION, BUSINESS, CONFLICT, CONTACT, PERSONNEL and JUSTICE. $^{13}$
known and only annotation for these event types will be seen. We used two settings for selecting known types: 10 most frequent events types and 8 event types, one from each parent type of the event ontology. The evaluation is done on the complete set of event types. We refer the reader to Appendix C for implementation details and hyperparameter settings.
In addition to our dataset WIKI EVENTS, we also report the performance on the Automatic Content Extraction (ACE) 2005 dataset<sup>10</sup> and the Roles Across Multiple Sentences (RAMS) dataset<sup>11</sup>.
We follow preprocessing from (Lin et al., 2020; Wadden et al., 2019) for the ACE dataset. Statistics of the ACE data splits can be found in Table 3. RAMS (Ebner et al., 2020) is a recently released dataset with cross-sentence argument annotation. A 5-sentence window is provided for each event trigger and the closest argument span is annotated for each role. We follow the official data splits from Version 1.0.
4.2 Document-Level Event Argument Extraction
Table 4 shows the performance for argument extraction on RAMS. On the RAMS dataset, we mainly compare with Two-step (Zhang et al., 2020), which is the current SOTA on this dataset. To handle long contexts, it breaks down the argument extraction into two steps: head detection and expansion.