SlowGuess/NAACL_2021_OCR huggingface dataset:
In Table 5 we show the results for the WIKI EVENTS dataset. We compare with a pop
10https://www.ldc.upenn.edu/ collaborations/past-projects/ace
11http://nlp.jhu.edu/rams
<sup>12</sup>Note that our preprocessing procedure is slightly different from (Du and Cardie, 2020) as we kept pronouns as valid event triggers and arguments.
<sup>9</sup>https://brat.nlplab.org/
Model	Span F1	Head F1
Seq	40.5	48.0
Two-step	41.8	49.7
BART-Gen	48.64	57.32
Table 4: Supervised argument extraction results (%) on RAMS test set. Both the Seq and Two-step model use type-constrained decoding to improve performance. The official scorer was used to compute results.
Model	Arg Identification	Arg Classification
Head F1	Coref F1	Head F1	Coref F1
BERT-CRF	69.83	72.24	54.48	56.72
BERT-QA	61.05	64.59	56.16	59.36
BERT-QA-Doc	39.15	51.25	34.77	45.96
BART-Gen	71.75	72.29	64.57	65.11
ularly used BERT-CRF baseline (Shi and Lin, 2019) that performs trigger extraction on sentencelevel and BERT-QA (Du and Cardie, 2020) ran on sentence-level and document-level.
4.3 Document-Level Informative Argument Extraction
We test on WIKI EVENTS using the informative argument as the training data and also compare with the BERT-CRF and BERT-QA baselines. Results are shown in Table 6.
Comparing the results in Tables 5 and 6, we have the following findings:
1. Informative argument extraction is a much more difficult task compared to nearest argument extraction. This is exemplified by the large performance gap for all models.
2. While CRF models are good at identifying spans, the performance is hindered by classification. The arguments follow a long tail distribution and since CRF models learn each argument tag separately, it cannot leverage the similarity between argument roles to improve the performance on rarely seen roles.
3. QA models, on the other hand, suffer from poor argument identification. When the QA model produces multiple answers for the same role, these answer spans are often close to each other or overlap. We show a concrete example in the qualitative analysis.
4. Directly applying the BERT-QA model to document level does not work. The QA model
Table 5: Argument extraction results (%) on WIKIEVENTS test set.
Model	Arg-I	Arg-C
Head F1	Coref F1	Head F1	Coref F1
BERT-CRF	52.71	58.12	43.29	47.70
BERT-QA	46.88	49.89	43.44	46.45
BERT-QA-Doc	26.29	31.54	24.46	29.26
BART-Gen	56.10	62.48	51.03	57.04
Table 6: Informative argument extraction results on WIKIEVENTS test set.
gets easily distracted by the additional context and does not know which event to focus on. We think that this is not a fundamental limitation of the QA approach, but a sign that repurposing QA models for document-level event extraction needs more investigation.
4.4 Zero-Shot Event Extraction
We show results for the zero-shot transfer setting in Table 8. Since the baseline BERT-CRF model (Shi and Lin, 2019) cannot handle new labels directly, we exclude it from comparison. In addition to BERT-QA, we also replace our TAPKEY trigger extraction model with a Prototype Network(Snell et al., 2017) $^{14}$ . We replace the prototypes with the class vectors to enable zero-shot learning. Complete results for trigger extraction are included in Appendix D.
The performance of BERT-QA is greatly limited by the trigger identification step. Both the Prototype network and our model TAPKEY can leverage the keyword information to assist transfer. Remarkably, TAPKEY has only 3 points drop in F1 using only  $30\%$  of the training data compared to the full set. The argument extraction component is more sensitive to the reduction in training data, but still performs relatively well. We notice that when a template is completely new, the model might alter the template structure during generation.
We show a comparison of our model's extractions with baselines in Table 7 for the argument extraction task on WIKI EVENTS. Our model is able to effectively capture all arguments, while the CRF model struggles with rare event types and the QA model is hindered by over-generation.
An example of informative argument extraction from our model is displayed in Figure 6. Our model
14 When 0 event types are seen, we set the transformation function in the Prototype Network to be an identity function; in other cases, we use a two layer MLP.
Context	Role	Ours	BERT-CRF	BERT-QA
I have been in touch
(E1:Contact.Contact.Correspondence) with the NDS
official in the province and they told me that over 100
members of the NDS were killed(E2:Life.Die) in the big
explosion , " the former provincial official said . Sharif
Hotak , a member of the provincial council in Maidan
Wardak said he saw (E3:Cognitive.IdentifyCategorize)
bodies of 35 Afghan forces in the hospital."	E1: Participant	I
NDS official	I
official	NDS
I
official
NDS official
members
E2: Victim	members	members
E3: Identifier	he	N/A	N/A
E3: IdentifiedObject	bodies	N/A	N/A
E3: Place	hospital	N/A	N/A
Table 7: An example of document-level argument extraction task on WIKI EVENTS. This excerpt contains 3 events: Contact, Die and IdentifyCategorize. For the Contact event E1, BERT-QA over-generates answers for the participant span. For the Die event, all three models can correctly extract the Victim argument. For the IdentifyCategorize event which is relatively rare, only our model can successfully extract all arguments.
Figure 5: An example of our model's argument extraction output on the SM-KBP dataset. Arguments highlighted in orange are newly found by our model compared to the baseline system OneIE (Lin et al., 2020).
Figure 6: An example of our model's prediction of informative arguments. Only arguments for the bold-faced event triggers are shown. Notably, our model can correctly identify "Bilal Mohammed and Mieralli Yusufu" as the as the Defendant for all the ChargeIndict events and "Erawan Shrine" as the place of attack.
is able to choose the informative mentions of the Defendant of indictment and Place of attack even when the trigger is a few sentences away.
We also applied our model as part of a pipeline multimedia multilingual knowledge extraction system (Li et al., 2020) for the NIST streaming multimedia knowledge base population task (SMKBP2020) $^{15}$ . Our model was able to discover  $53\%$  new arguments compared to the original system, especially for those that were further away from the event trigger. The overall system achieved top
1 performance. We show some examples in Figure 5.
4.6 Remaining Challenges
Ontological Constraints Some of the roles are mutually exclusive, such as the Origin/Destination in the Transport event and the Recipient/Yielder in the Surrender event. In the following example, "Japan" was extracted as both part of the Recipient and the Yielder of the Surrender event: "If South Korea drifts into the orbit of the US and Japan, China's influence on the Korean peninsula could be badly compromised." At a military parade in Beijing to mark the 70th anniversary of the surrender of Japan last September, "...". Such constraints might be incorporated into the decoding process of the model.
Commonsense Knowledge In the following instance with implicit arguments: "Whether the U.S. extradites Gulen or not this will be a political decision," Bozdag said. "If he is not extradited, Turkey will have been sacrificed for a terrorist." A recent opinion poll showed two thirds of Turks agree with their president that Gulen was behind the coup plot. ", our model mistakenly labels "U.S." as the Destination of the extradition and "Turkey" as the
15https://tac.nist.gov/2020/KBP/SM-KBP/
# Seen Event Types	Model	TI F1	TC F1	AI Head F1	AC Head F1
0	Prototype	3.19	2.22	-	-
TAPKEY	55.19	52.10	-	-
10 most frequent	BERT-QA	57.06	53.83	39.37	38.27
Prototype + BART-Gen	69.48	66.06	46.08	41.93
TAPKEY + BART-Gen	72.31	69.23	48.18	44.19
1 per general type	BERT-QA	27.56	25.32	24.87	24.29
Prototype + BART-Gen	68.29	65.85	39.51	34.63
TAPKEY + BART-Gen	72.23	68.55	39.80	35.11
Full	TAPKEY + BART-Gen	74.36	71.13	55.22	53.71
Table 8: Zero-shot event extraction results (%) on ACE. "10 most frequent event types" corresponds to the Freq data split and "1 per general type" corresponds to the Ontology data split. Fully supervised results are provided for reference.
Source even though the Extraditer is correctly identified as "U.S." .Commonsense knowledge such as "The extraditer, if being a country, is usually is same as the source of extradition" would be helpful to fix this error.
5.1 Document-Level Event Extraction
Document-level event extraction can be traced back role filling tasks from the MUC conferences (Grishman and Sundheim, 1996) that required retrieving participating entities and attribute values for specific scenarios. The KBP slot filling challenge<sup>16</sup> is akin to this task, but centered upon entities.
In general, document-level argument extraction is an under-explored topic, mainly due to the lack of datasets. There have been a few datasets published specifically for implicit semantic role labeling, such as the SemEval 2010 Task 10 (Ruppenhofer et al., 2010), the Beyond NomBank dataset (Gerber and Chai, 2010) and ON5V (Moor et al., 2013). However, these datasets were small in size and only covered a small set of carefully selected predicates. Recently, (Ebner et al., 2020) published the RAMS dataset, which contains annotation for cross-sentence implicit arguments covering a wide range of event types. Albeit, this dataset only annotates one event per document, motivating us to create a new benchmark with complete event and coreference annotation.
The GRIT model (Du et al., 2021) is a generative model designed for the MUC task which can be seen as filling in predefined tables. In comparison, we treat the template (for example "<arg1> attacked <arg2> using <arg3> at <arg4> place") as
part of the model input along with the document context. This allows us to share model parameters across all event types and enables zero-shot transfer to new event types.
5.2 Zero-shot Event Extraction
Early attempts at zero-shot or few-shot event extraction rely on preprocessing such as semantic role labeling(SRL) (Peng et al., 2016) or abstract meaning representation (AMR) (Huang et al., 2018) to detect trigger mentions and argument mentions before performing classification on the detected spans.
Another line of work only examines the subtask of trigger detection, essentially reducing the task to few-shot classification. Both (Lai et al., 2020) and (Deng et al., 2020) extend upon the prototype network model (Snell et al., 2017) for classification.
Recent work on zero-shot event extraction has posed the problem as question answering (Chen et al., 2020; Du and Cardie, 2020; Feng et al., 2020) with different ways of designing the questions.
In this paper, we advocate document-level event extraction and propose the first document-level neural event argument extraction model. We also release the first document-level event extraction benchmark dataset WIKIEVENTS with complete event and coreference annotation. On both the conventional argument extraction task and the new informative argument extraction task, our proposed model surpasses CRF-based and QA-based baselines by a wide margin. Additionally, we demonstrate the portability of our model by applying it to the zero-shot setting. Going forward, we would like to incorporate more ontological knowledge to produce more accurate extractions.
$^{16}$ https://tac.nist.gov/2017/KBP/index.html
The authors would like to thank Tuan Lai and Zhenhailong Wang for their help in the data processing and annotation organization effort. We also thank all the annotators who have contributed to the annotations of our training data (in alphabetical order): Daniel Campos, Anthony Cuff, Yi R. Fung, Xiaodan Hu, Emma Bonnette Hamel, Samual Kriman, Meha Goyal Kumar, Manling Li, Tuan M. Lai, Ying Lin, Sarah Moeller, Ashley Nobi, Xiaoman Pan, Nikolaus Parulian, Adams Pollins, Rachel Rosset, Haoyu Wang, Qingyun Wang, Zhenhailong Wang, Spencer Whitehead, Lucia Yao, Pengfei Yu, Qi Zeng, Haoran Zhang, Hongming Zhang, Zixuan Zhang.
This research is based upon work supported by U.S. DARPA KAIROS Program No. FA8750-19-2-1004, U.S. DARPA AIDA Program No. FA8750-18-2-0014, National Science Foundation IIS-19-56151, IIS-17-41317, and IIS 17-04532, and Air Force No. FA8650-17-C-7715. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Yunmo Chen, Tongfei Chen, Seth Ebner, Aaron Steven White, and Benjamin Van Durme. 2020. Reading the manual: Event extraction as definition comprehension. In Proceedings of the Fourth Workshop on Structured Prediction for NLP, pages 74-83, Online. Association for Computational Linguistics.
Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang, Wei Zhang, and Huajun Chen. 2020. Meta-learning with dynamic-memory-based prototypical network for few-shot event detection. WSDM.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671-683, Online. Association for Computational Linguistics.
Xinya Du, Alexander M. Rush, and Claire Cardie. 2021. Grit: Generative role-filler transformers for document-level event entity extraction.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics, pages 8057-8077, Online. Association for Computational Linguistics.
Rui Feng, Jie Yuan, and Chao Zhang. 2020. Probing and fine-tuning reading comprehension models for few-shot event extraction. In *ArXiv*.
Matthew Gerber and Joyce Chai. 2010. Beyond NornBank: A study of implicit arguments for nominal predicates. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1583-1592, Uppsala, Sweden. Association for Computational Linguistics.
Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1381-1393, Online. Association for Computational Linguistics.
Lifu Huang, Heng Ji, Kyunghyun Cho, Ido Dagan, Sebastian Riedel, and Clare Voss. 2018. Zero-shot transfer learning for event extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2160-2170, Melbourne, Australia. Association for Computational Linguistics.
Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL-08: HLT, pages 254-262, Columbus, Ohio. Association for Computational Linguistics.
Carol C Kuhlthau. 1991. Inside the search process: Information seeking from the user's perspective. Journal of the American society for information science, 42(5):361-371.
Viet Dac Lai, Thien Huu Nguyen, and Franck Der-noncourt. 2020. Extensively matching for few-shot learning event detection. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 38-45, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Manling Li, Ying Lin, Tuan Manh Lai, Xiaoman Pan, Haoyang Wen, Sha Li, Zhenhailong Wang, Pengfei Yu, Lifu Huang, Di Lu, Qingyun Wang, Haoran Zhang, Qi Zeng, Chi Han, Zixuan Zhang, Yujia Qin, Xiaodan Hu, Nikolaus Parulian, Daniel Campos, Heng Ji, Brian Chen, Xudong Lin, Alireza Zareian, Amith Ananthram, Emily Allaway, Shih-Fu Chang, Kathleen McKeown, Yixiang Yao, Yifan Wang, Michael Spector, Mitchell DeHaven, Daniel Napierski, Marjorie Freedman, Pedro Szekely, Haidong Zhu, Ram Nevatia, Yang Bai, Yifan Wang, Ali Sadeghian, Haodi Ma, and Daisy Zhe Wang. 2020. Gaia at sm-kbp 2020 - a dockerized multi-media multi-lingual knowledge extraction, clustering, temporal tracking and hypothesis generation system. In Proc. Text Analysis Conference (TAC2020).
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999-8009, Online. Association for Computational Linguistics.
Jens-Erik Mai. 2016. Looking for information: A survey of research on information seeking, needs, and behavior. Emerald Group Publishing.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9006-9017, Online. Association for Computational Linguistics.
Tatjana Moor, Michael Roth, and Anette Frank. 2013. Predicate-specific annotations for implicit role binding: Corpus annotation, data analysis and evaluation experiments. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) - Short Papers, pages 369-375, Potsdam, Germany. Association for Computational Linguistics.
Haoruo Peng, Yangqiu Song, and Dan Roth. 2016. Event detection and co-reference with minimal supervision. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 392-402, Austin, Texas. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Josef Ruppenhofer, Caroline Sporleder, Roser Morante, Collin Baker, and Martha Palmer. 2010. SemEval-2010 task 10: Linking events and their participants in discourse. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 45-50, Uppsala, Sweden. Association for Computational Linguistics.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615-4629, Online. Association for Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. NeurIPS.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5784-5789, Hong Kong, China. Association for Computational Linguistics.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. 2019. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In ICML.
Zhisong Zhang, Xiang Kong, Zhengzhong Liu, Xuezhe Ma, and Eduard Hovy. 2020. A two-step approach for implicit event argument detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7479-7485, Online. Association for Computational Linguistics.
Document-Level Event Argument Extraction by Conditional Generation: Appendix
1 Trigger Extraction Model Details
1.1 Tagging Scheme
We use the IO tagging scheme, where I stands for "inside a span" and O stands for "outside any span". This simplified tagging scheme was selected to reduce parameters without much loss of modeling power since (1) triggers are often single words and I tags in the BIO (B stands for "beginning of a span") scheme are infrequent and (2) we rarely see two consecutive event triggers of the same type.
1.2 Class Vectors
For each of the event types, we provided 3 keywords as initial seeds. If the event type can be triggered by nominals, we additionally add keywords for the nominal form. Our chosen keywords will be provided along with the ontology file as supplementary materials.
For each event type, we search for its corresponding keywords' occurrence in the Gigaword corpus. To filter out ambiguous usages of the keywords, we apply BERT-large as a masked language model and predict words that can replace the current mention of the keyword. If another keyword for this event type appears among the top 50 candidates, we accept this example. The vector representation for this example is the average of the wordpiece tokens that consist the keyword.
The class vector is an average over all the examples for the event type.
2 Solving for  $M$
The following section is a simplified version of the derivation from TapNet (Yoon et al., 2019).
In order to correctly classify  $c_k$ , we would like to maximize the dot product with  $\phi_k$  and minimize the dot product with  $\phi_{l\neq k}$  in the subspace defined by  $M$ . A possible solution would be to find the
projection matrix  $M$  so that:
$$
M \left(c _ {k}\right) = \lambda M \left(\phi_ {k} - \frac {1}{m - 1} \sum_ {l \neq k} \phi_ {l}\right) \tag {1}
$$
$$
\mathrm {s . t .} \| \phi_ {i} \| = 1, \phi_ {i} ^ {T} \phi_ {j \neq i} = 0.
$$
$$
M \left(c _ {k}\right) ^ {T} M \left(\phi_ {k}\right) = \lambda \| M \| ^ {2}
$$
$$
M \left(c _ {k}\right) ^ {T} M (\phi_ {l}) = - \lambda \frac {1}{m - 1} \| M \| ^ {2} \tag {2}
$$
which is a reasonably good separation between the classes.
Let  $\hat{\phi}_k = \phi_k - \sum_{l\neq k}\phi_l$  , then we can rearrange the previous equation as:
$$
M ^ {T} \left(c _ {k} - \lambda \hat {\phi} _ {k}\right) = 0 \tag {3}
$$
Note that this holds for every  $k$ . If we define  $D \in \mathbf{R}^{d \times n}$  as the matrix with  $c_k - \lambda \hat{\phi}_k$  as its  $k$ th column, we have  $M^T D = \vec{0}$ , implying that the columns in  $M$  are in the null space of  $D^T$ . This null space of  $D^T$  can be obtained by QR decomposition.
$$
D ^ {T} = Q R = \left[ Q _ {1}, Q _ {2} \right] \left[ \begin{array}{c} R _ {1} \\ 0 \end{array} \right] \tag {4}
$$
Although the rank of  $D$  is unknown, it will not be larger than  $n$  (and with high probability close to  $n$ ), and thus we can take  $m$  columns starting from the  $n + 1$  column of  $Q$  for  $M \in \mathbf{R}^{d \times m}$ .
In order to account for the new types, we apply some leniency at training time and learn  $n' > n$  reference vectors instead of only  $n$  vectors for the  $n$  classes that appear in the training set. Then when we are asked to identify new types during inference, we update  $M$  based on the new class vectors  $c_{n}^{\prime}$ .
The complete algorithm is listed in Algorithm 1.
2.1 Pseudo Labeling
In the pseudo-labeling process, we compute the token-wise cosine similarity between class vectors and averaged sentence-piece embeddings from
BERT-Large. The event type token labels are accepted if the similarity is higher than 0.65 and the O label is assigned if none of the similarity scores are higher than 0.4. For cases in between, we assign an X label which means ignoring the token for loss computation.
3 Dataset Collection and Annotation Details
We removed documents that have less than 100 tokens, and off-topic documents such as excerpts from history books. In the annotation process, annotators can also flag documents as duplicates, or irrelevant. All documents are in English.