SlowGuess/NAACL_2022_OCR huggingface dataset:
Qi Zeng*, Qiusi Zhan*, Heng Ji
{qizeng2, qiusiz2,hengji}@illinois.edu
Events are inter-related in documents. Motivated by the one-sense-per-discourse theory, we hypothesize that a participant tends to play consistent roles across multiple events in the same document. However recent work on document-level event argument extraction models each individual event in isolation and therefore causes inconsistency among extracted arguments across events, which will further cause discrepancy for downstream applications such as event knowledge base population, question answering, and hypothesis generation. In this work, we formulate event argument consistency as the constraints from event-event relations under the document-level setting. To improve consistency we introduce the Event-Aware Argument Extraction  $(\mathrm{EA}^2\mathrm{E})$  model with augmented context for training and inference. Experiment results on WIKIEVENTS and ACE2005 datasets demonstrate the effectiveness of  $\mathrm{EA}^2\mathrm{E}$  compared to baseline methods<sup>1</sup>.
Document-level Event Argument Extraction aims at identifying arguments and their roles for multiple events in a document. It is a practically more useful but more challenging task than sentence-level Argument Extraction (Nguyen et al., 2016; Wadden et al., 2019; Lin et al., 2020) because in a typical long input document events usually scatter across multiple sentences and are inherently connected.
Multiple events in one document are usually interconnected, and thus the arguments are under certain consistency constraints. In Figure 1, the roles of the shared argument Ahmad Khan Rahami in multiple events are constrained because the Attacker in the DetonateExplode event is likely to be the IdentifiedRole in IdentifyCategorize event, the
Figure 1: Examples of extracting arguments for multiple events in one document. The casual relation between the Arrest event and the Detonate event puts their arguments under consistency constraints: Ahmad Khan Rahami, the detainee in Event 3, is very likely to be the attacker in Event 1. Sentence-level models tend to miss the cross-sentence attacker argument in Event 1.
Detainee in the ArrestJail Detain event, as well as the Defendant in ChargeIndict event. Motivated by the one-sense-per-discourse theory (Gale et al., 1992) that mentions of an ambiguous word usually tend to share the same sense in a given discourse, we hypothesize that a participant tends to play consistent roles across multiple events in the same document. However, previous work such as (Li et al., 2021; Du et al., 2021; Yang et al., 2021) on document-level event argument extraction focuses on modeling each event independently and ignores the relation between events, and thus the extracted arguments of multiple events may violate the constraints from event-event relations. We call this inconsistency phenomenon.
Though received much attention in various areas like Abstractive Summarization (Nan et al., 2021; Zhu et al., 2021; Kryscinski et al., 2020) and Question Answering (Honovich et al., 2021; Ribeiro
$^{1}$ Our code is released at https://github.com/ZQS1943/DOCIE
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2649 - 2655
et al., 2019), the inconsistency phenomenon addressed in previous research focuses on factual consistency instead of self-contained consistency as in document-level argument extraction. We approach this problem with inspiration from human behavior: while reading, humans subconsciously infer the event-event relations and correctly identify the event arguments under the perceived constraints. Therefore, we refer consistent argument extraction to applying the underlying Event-Event Relations as constraints in multi-event argument extraction.
An intuitive solution to improve consistency is to incorporate explicit Event-Event Relations into the extraction process as additional input. However, the underlying event-event relations are hard to identify and classify due to the lack of reliable resources as supervision signals, especially when the arguments are unknown. In addition, precise event-event relations may not be necessary for argument extraction when the implicit connections can already well support argument extraction.
To avoid explicit modeling of event-event relations, we label the arguments of other events in the context as an implication of event-event relations. We propose an Event-Aware Argument Extraction  $(\mathbf{EA}^2\mathbf{E})$  model, which incorporates alignment-enhanced training and iterative inference. When extracting arguments, the context can be self-augmented by tagging the argument labels of other events. Alignment-enhanced training implicitly introduces event awareness by pulling close the argument representation distributions under regular context and augmented context, where ground-truth argument labels of neighboring events are labeled. Iterative inference explicitly encourages event awareness by augmenting the context with the extracted arguments from the last inference iteration. The advantage of this method is that no predefined Event-Event Relation is required, nor event schema. Our experiments on WIKIEVENTS and ACE2005 datasets show that our  $\mathrm{EA}^2\mathrm{E}$  model brings improvement against previous methods.
2 Event-Aware Argument Extraction
Motivated by the observation that introducing event-event relations benefits the consistency of event argument extraction, we propose to incorporate implicit event-event relations with an Event-Aware Argument Extraction  $(\mathrm{EA}^2\mathrm{E})$  model. As shown in Figure 2,  $\mathrm{EA}^2\mathrm{E}$  contains alignment-enhanced training and iterative inference with self
augmented context. When extracting the arguments for a target event, the context is augmented by labeling the arguments from neighboring events. During training, an auxiliary training loss pulls close the event argument representations under the regular context and self-augmented context. During inference, iterative inference encourages event awareness by using the extraction arguments from the last inference iteration as inputs.
2.1 Base Encoder-Decoder Model
Following (Li et al., 2021), we formulate event argument extraction as a conditional generation task under the assumption that there exists a pre-defined event ontology describing each event type with an unfilled template with argument placeholders. For example, the template for Attack events is  $\langle arg \rangle$  detonated or exploded  $\langle arg \rangle$  explosive device using  $\langle arg \rangle$  to attack  $\langle arg \rangle$  target at  $\langle arg \rangle$  place. Formally, given a document context  $c$  and an event trigger  $x$  with template  $t$ , the task is to extract a set of arguments  $y = \{a_1, a_2, \ldots, a_n\}$ , where each  $a_i$  corresponds to a role predefined in the ontology.
We base our model on BART (Lewis et al., 2020), an encoder-decoder pretrained model. The input sequence is the concatenation of the document context and an event template, constructed as  $\langle s \rangle$  template  $\langle /s \rangle$ $\langle /s \rangle$  context  $\langle /s \rangle$ . The output is a filled-in template, where the tokens are all selected from the input context or template.
The model parameter  $\theta$  is trained by minimizing the argument extraction loss, the conditional probability over all instances:
$$
\mathcal {L} _ {E} = - \sum \operatorname {l o g p} _ {\theta} (y | x, t, c) \tag {1}
$$
2.2 Self-augmented Context
We refer event awareness as the implication of event-event relations and reach this goal by labeling the arguments of other neighboring events in the context. Given the arguments of the neighboring events  $\{j\in \mathcal{N}_i\}$ , which have small token-wise distances to the target event  $i$ , we augment the regular context  $c$  by labeling them with  $<\text{tag}>$ :
$$
c _ {i} ^ {\prime} = \mu \left(c _ {i}, \left\{y _ {j} \right\}\right), \tag {2}
$$
where  $\mu$  is the tagging operation, and  $y_{j}$  is the arguments of event  $j$ .
For example in Figure 1, when extracting arguments for the target event bombing (Event 1), the augmented context is "The Saturday night's <trg>
Figure 2: Our proposed Event-aware Argument Extraction model with alignment-enhanced training and iterative inference. During training, an auxiliary training loss aligns the event argument representations under regular context and augmented context. During inference, the context is augmented with results from the last iteration.
bombing <trg> in New York City, wounded 29 people. <tag> Prosecutor <tag> Prosecutors have charged 28-year-old <tag> Defendant <tag> Ahmad Khan Rahami...", where the two tags highlight the arguments of Event 4.
2.3 Alignment-enhanced Training
An encoder is considered consistent when it is able to understand and encode the underlying relation between events into the text representations. Therefore, we propose to enhance the encoder with an auxiliary training loss  $\mathcal{L}_T$  that pulls close the argument representation distributions under regular context  $c$  and under augmented context  $c'$ . During training,  $c'$  is constructed by tagging the ground-truth arguments of neighboring events.
$$
\mathcal {L} _ {T} = \sum \| p (a | c), p (a | c ^ {\prime}) \| _ {2} \tag {3}
$$
The final training loss is a weighted sum of argument extraction losses  $(\mathcal{L}_E$  for regular context  $c$  and  $\mathcal{L}_{E^{\prime}}$  for augmented context  $c^{\prime}$  ) and alignment-enhanced loss  $(\mathcal{L}_T)$  with weights  $\alpha$  and  $\beta$  ..
$$
\mathcal {L} = \mathcal {L} _ {E} + \alpha \mathcal {L} _ {E ^ {\prime}} + \beta \mathcal {L} _ {T} \tag {4}
$$
2.4 Iterative Inference
Iterative inference explicitly introduces event awareness by utilizing extracted results in multiple inference iterations. In the first iteration, for each target event trigger  $i$ , the model obtains the predicted results  $y_{i}^{1}$  given the regular context  $c_{i}^{1}$ . For the  $k$ -th iteration of inference, for each event trigger  $i$  the context  $c_{i}^{k}$  is augmented by labeling the extracted arguments  $\{y_{j}^{k-1}\}$  of neighboring events  $\{j \in \mathcal{N}_{i}\}$  from the  $(k-1)$ -th iteration.
$$
c _ {i} ^ {k} = \mu \left(c, \left\{y _ {j} ^ {k - 1} \right\}\right) \tag {5}
$$
3.1 Datasets and Baselines
We evaluate our proposed method on WIKIEVENTS (Li et al., 2021) dataset and ACE 2005 dataset<sup>2</sup>. Following previous work (Li et al., 2013), we consider an argument span to be correctly identified when its offsets match any of the reference arguments of the current event (i.e., Argument Identification), and to be correctly classified when its role matches (i.e., Argument Classification). We report the argument extraction performance in terms of Head Word F1 and Coreferential Mention F1. For the latter, full credit will be given when the extracted argument is coreferential with the gold-standard argument.
We compare EA $^2$ E with document-level BART-Gen (Li et al., 2021), sentence-level ONEIE (Lin et al., 2020) and BERT-CRF (Shi and Lin, 2019).
We implement our models with Huggingface (Wolf et al., 2019). We train each model for 4 epochs with a batch size of 4 for baselines and 2 for  $\mathrm{EA}^2\mathrm{E}$ . The model is optimized with the Adam optimizer with a learning rate of  $3\mathrm{e - }5$ ,  $\alpha = 1$  and  $\beta = 0.5$ . We define event neighborhood as trigger distance less than 40 tokens. For inference, the maximum number of iterations is 3. For numerical consistency, all experiment results are averaged across three random runs. The hyper-parameters are selected by grid search based on model performance on development set. On average it takes approximately
$^{2}$ https://www.ldc.upenn.edu/collaborations/past-projects/ace
$^3\beta$  is chosen from  $\{0.1, 0.5, 1\}$ , the trigger distance is chosen from  $\{20, 40, 60, 80, 100\}$ , and the learning rate is chosen in  $\{3e - 5, 5e - 5\}$ .
Model	Argument Identification	Argument Classification
Head Match	Coref Match	Head Match	Coref Match
P	R	F1	P	R	F1	P	R	F1	P	R	F1
BERT-CRF	72.66	53.82	61.84	74.58	55.24	63.47	61.87	45.83	52.65	63.79	47.25	54.29
ONEIE	68.16	56.66	61.88	70.09	58.26	63.63	63.46	52.75	57.61	65.17	54.17	59.17
BART-Gen	70.43	71.94	71.18	71.83	73.36	72.58	65.39	66.79	66.08	66.78	68.21	67.49
EA2E	76.51	72.82	74.62	77.69	73.95	75.77	70.35	66.96	68.61	71.47	68.03	69.70
EA2EW/o AT	77.26	71.23	74.12	78.61	72.47	75.42	71.10	65.54	68.21	72.25	66.61	69.32
EA2EW/o II	75.96	72.29	74.07	77.13	73.42	75.22	69.61	66.25	67.89	70.72	67.32	68.97
Table 1: Performance (%) on WIKIEVENTS dataset. AT: Alignment-enhanced Training. II:Iterative Inference.
Model	Argument Identification	Argument Classification
Head Match	Coref Match	Head Match	Coref Match
P	R	F1	P	R	F1	P	R	F1	P	R	F1
BERT-CRF	65.77	51.04	57.48	67.11	52.08	58.65	56.82	44.10	49.66	57.72	44.79	50.44
ONEIE	63.33	61.46	62.38	65.12	63.19	64.14	58.50	56.77	57.62	60.11	58.33	59.21
BART-Gen	70.00	73.84	71.87	71.37	75.29	73.27	65.72	69.33	67.47	66.76	70.43	68.54
EA2E	74.54	74.88	74.71	75.81	76.16	75.98	71.83	72.16	72.00	72.98	73.32	73.15
EA2EW/o AT	73.95	74.25	74.10	75.28	75.58	75.43	70.78	71.06	70.92	72.05	72.34	72.19
EA2EW/o II	74.36	75.00	74.68	75.56	76.22	75.88	71.49	72.11	71.80	72.58	73.20	72.89
Table 2: Performance (%) on ACE2005 dataset.
one hours to train a model until converge with one Tesla P100 GPU with 16GB DRAM.
3.3 Results and Analysis
Table 1 and Table 2 show that our proposed EA $^2$ E consistently performs better than strong baseline methods across datasets and evaluation metrics. In general document-level methods have better performance, especially in terms of recall, because sentence-level methods are more likely to miss cross-sentence arguments.
Alignment-enhanced training brings a significant improvement over BART-Gen but comes with higher training costs since the inputs are doubled.
Iterative inference brings unstable improvement. Figure 3 shows that more iterations brings higher performance only to a certain range. Since the only differences among iterations are their inputs, we conclude that labeling the arguments of other events helps the model extract the arguments of the current event. The upper bound of this improvement is limited by the error propagation in the augmented context.
Qualitative Analysis Table 3 presents some representative examples. BART-Gen incorrectly assigns Tsarnaev to the Target role, and police to the attacker role in the first example. It also misses the killer brothers in the second example and the attacker Laden in the third example. The second example shows the advantage of the Alignment-enhanced Training component in  $\mathrm{EA}^2\mathrm{E}$ , which
Figure 3: F1 score  $(\%)$  of  $\mathrm{EA}^2\mathrm{E}$  for different iterations in a single run on WIKIEVENTS.
helps extract the killer argument. The third example shows how Iterative Inference works with the augmented input: The tagged attacker in the neighboring bombing event is also the attacker in the target attack event.
Remaining Challenges Though effective, iterative inference may propagate errors among iterations. In addition, the success of event awareness relies on the assumption that events in a neighborhood defined by trigger distance are inter-related to the target event. However, this assumption is not always held true in the case that distant events bring redundant information. It is not necessarily hurting the information but it brings noise by incorrectly implying the relations between the distant events and the target event.
Gold	BART-Gen	EA2E w/o II	EA2E
Input	Dzhokhar Tsarnaev visits Silva and borrows the Ruger pistol — the gun that was later used to kill MIT police officer Sean Collier and during the shootout with police in Watertown.
Output	Target: police	Target: Tsarnaev	Target: police	Target: police
Instrument: gun	Attacker: police	Attacker: Tsarnaev	Attacker: Tsarnaev
Place: Watertown	Place: Watertown	Place: Watertown
Input	The brothers allegedly set off two bombs alongside the Boston Marathon course, killing three people and injuring 264.
Output	Killer: brothers	Victim: people	Killer: brothers	Killer: brothers
Victim: people	Victim: people	Victim: people
Input	Osama bin Laden is charged with masterminding the 1998 bombings of two U. S. embassies in East Africa, believed to have had a role in the October 2000 attack on the USS Cole in the Yemeni port of Aden.
Augmented	<tag> Attacker </tag> Osama bin Laden is charged with masterminding the 1998 bombings of two U. S.<tag>
Input	Target </tag> embassies in <tag> Place </tag> East Africa, believed to have had a role in the October 2000 attack on the USS Cole in the Yemeni port of Aden.
Output	Target: Cole	Target: Cole	Target: Cole	Target: Cole
Target: port	Place: Yemeni	Place: Yemeni	Place: Yemeni
Attacker: Laden	Attacker: Laden
Table 3: Examples of extracted arguments from BART-Gen, EA²E w/o II, and EA²E. We label target event mention with bold, gold arguments with underlines, correct but not annotated arguments with waves, and incorrect arguments with red. In the third example we also present the augmented input for Iterative Inference, in which the arguments of the bombing event are tagged.
Sentence-level argument extraction approaches, where the event trigger and its arguments are usually located within a single sentence, cannot handle the cross-sentence trigger-argument distribution and the existence of multiple events within one document. Though recent attempts on document-level argument extraction have gone beyond sentence boundaries, they either focus on one-event-per-document setting, or model each event independently. The most related work to ours is (Li et al., 2021), which formulates the task as conditional generation following event templates and extracts arguments for each event independently, while our work focuses on the consistency among arguments for different events. The methods in (Du et al., 2021) and (Du and Cardie, 2020) are designed for Role-filler Entity Extraction (REE) task under the assumption that one generic template is produced for each document, while our work focuses on extracting arguments for multiple events for each document. (Yang et al., 2021) introduces Parallel Prediction Network that generates all possible events in parallel based on the document-aware representations, while we adopt a generative framework. (Xu et al., 2021) model the whole document as graphs and capture the interdependency among events by tracking the extracted events with a global memory, while we introduce event awareness for interdependency without external memory modules.
We introduce Event-Aware Argument Extraction  $(\mathrm{EA}^2\mathrm{E})$  model to improve self-contained consistency in document-level event argument extraction. We conclude that in iterative inference brings higher performance only to a certain range of iterations and alignment-enhanced training brings significant improvement with costs.
We thank the anonymous reviewers helpful suggestions. This research is based upon work supported by U.S. DARPA AIDA Program No. FA8750-18-2-0014, U.S. DARPA KAIROS Program No. FA8750-19-2-1004, and DARPA INCAS Program No. HR001121C0165. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Xinya Du and Claire Cardie. 2020. Document-level event role filler extraction using multi-granularity contextualized encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July
5-10, 2020, pages 8010-8020. Association for Computational Linguistics.
Xinya Du, Alexander M. Rush, and Claire Cardie. 2021. GRIT: generative role-filler transformers for document-level event entity extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 634-644. Association for Computational Linguistics.
William A. Gale, Kenneth Ward Church, and David Yarowsky. 1992. One sense per discourse. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, USA, February 23-26, 1992. Morgan Kaufmann.
Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $Q^2$ : Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. CoRR, abs/2104.08202.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9332-9346. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Association for Computational Linguistics.
Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pages 73-82. The Association for Computer Linguistics.
Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 894-908. Association for Computational Linguistics.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July