SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2: Answer types of event arguments in DocEE.
Answer Types	%	Examples
Single Answer	78.6	A masked man in a black hoodie showed a gun and was handed money before running east on Warren Street, according to the initial report.
Argument Type: Bank Robbery Argument: Weapon Used
Multiple Answers	21.4	At around 6:20 a.m. a lorry, driven by David Fairclough of Wesnesfield, rammed into the rear of a tanker, which then struck a car in front and exploded. The ensuing pile-up involved 160 vehicles on a 400-yard (370 m) stretch of the motorway.
Argument Type: Road Crash Argument: Number of Vehicles involved in the Crash
In order to be application-oriented, we design cross-domain setting to test the transfer capability of the SOTA models. We choose the event type under the subject of natural disasters as the target domain, including Floods, Droughts, Earthquakes, Insect Disaster, Famine, Tsunamis, Mudslides, Hurricanes, Fire and Volcano Eruption, and adopt the remaining 49 event types as source domains. The division reduces the overlap of argument types between the source domain and the target domain. In this setting, the models will first be pre-trained on the source domain, and then conduct 5-shot fine-tuning on the target domain. The detailed data split for each setting is shown in Table 3.
Method	Normal	Cross-Domain
Train	Dev	Test	Train	Dev	Test
#EvTyp.	59	59	59	59	10	10
#Doc.	22k	2.7k	2.7k	23.7k	1.6k	2.0k
#ArgInst.	141k	19k	19k	156k	11k	13k
Hyperparameters We use base version of pretrained model for all the transformer-based methods, and set the learning rate to 2e-5. The batch size is 128 and the maximum document length is 512. All baselines are implemented by HuggingFace with default parameters and all models can be fit into eight V100 GPUs with 16G memory. The training procedure lasts for about a few hours. For all the experiments, we report the average result of five runs as the final result. In human evaluation, we randomly select 1,000 document-level events and invite three students to label them. The final result is the average of their labeling accuracy.
5.1 Event Classification
Baselines We adopt a CNN-based method and various pre-trained transformer-based methods as
our baselines, including: 1) TextCNN (Kim, 2014) uses different sizes CNN kernels to extract key information in text for classification. 2) BERT (Devlin et al., 2018) exploits unsupervised objective functions, masking language model (MLM) and next sentence prediction for pre-training. 3) ALBERT (Lan et al., 2020) proposes a self-supervised loss to improve inter-sentence coherence in BERT. 4) DistilBert (Sanh et al., 2019) combines language modeling, knowledge distillation and cosine-distance losses to improve BERT. 5) RoBERTa (Liu et al., 2019) is built on BERT and trains with much larger mini-batches and learning rates. Following (Kowsari et al., 2019), we use Precision(P), Recall(R) and Macro-F1 score as the evaluation metrics.
Table 3: Statistics for two benchmark settings (Sec.5): normal and cross-domain.
Method	Normal	Cross-Domain
P	R	F	P	R	F
TextCNN	78.6	75.4	76.2	8.5	2.1	2.6
BERT	89.3	89.8	89.5	72.6	78.4	75.4
ALBERT	88.8	88.9	88.1	25.5	21.3	23.2
DistilBert	89.6	90.7	90.1	79.8	79.6	79.7
RoBERTa	90.0	91.1	90.5	86.0	86.7	86.3
Human	98.4	97.7	98.0	-	-	-
Table 4: Overall Performance on Event Classification.
Overall Performance Table 4 shows the experimental results under the normal and cross-domain settings, from which we have the following observations: 1) Compared with TextCNN, transformer based models (BERT, ALBERT, DistillBert, RoBERTa) perform better, which are pre-trained on a large-scale unsupervised corpus and have more background semantic knowledge to rely on. 2) Humans have achieved high scores on DocEE, verifying the high quality of our annotated data sets. 3) There is still a gap between the performance of the current SOTA models and human beings, which indicates that more technological advances are needed in future work. Humans can connect and merge key information to form a knowledge
<sup>9</sup>https://huggingface.co/models
Methods	Normal Setting	Cross-domain Setting
EM	HM	EM	HM
P	R	F	P	R	F	P	R	F	P	R	F
BERT_Seq(sent)	23.8	28.2	25.8	38.1	45.1	41.3	7.0	7.6	7.2	18.4	20.0	19.1
BERT_Seq(chunk)	27.9	30.8	29.2	43.9	48.4	46.0	19.2	22.9	20.9	34.9	41.7	38.0
BERT_Seq(doc)	35.3	35.9	35.6	52.6	53.5	53.1	20.3	25.0	22.4	35.5	42.2	38.6
MG-Reader	30.3	35.9	32.9	45.6	50.8	48.1	19.9	23.2	21.4	37.1	39.7	38.4
Doc2EDAG	37.1	36.1	36.6	54.2	53.7	53.9	21.6	26.3	23.7	36.2	45.7	40.4
BERT_QA	41.9	28.1	33.5	75.8	50.6	60.7	29.4	20.3	24.0	68.1	46.9	55.5
Ontology_QA	51.3	34.2	41.0	80.3	53.6	64.3	36.6	25.2	29.8	69.7	48.0	56.9
Human	87.8	84.2	85.9	80.9	87.2	89.0	-	-	-	-	-	-
Table 5: Overall Performance on Event argument Extraction(%)
network to help them understand the main event, while deep learning models typically fail in long text perception. 4) There is a performance degradation from the normal setting to the cross-domain setting, which shows that domain migration is still a huge challenge for current SOTA models. Among the pre-trained baselines, ALBERT's performance drops the most. The reason may be that the parameter scale in ALBERT is relatively small, and the reserved source domain knowledge is limited.
5.2 Event argument Extraction
Baselines We introduce the following mainstream baselines for evaluation: 1) BERT_Seq (one of the baseline in Du and Cardie (2020a)) uses the pre-trained BERT model to sequentially label words in the article. Given the input article  $A = \{w_{1}, w_{2}, \ldots, w_{n}\}$ , the output of Sequence Labeling Methods is  $O = \{r_{1}, r_{2}, \ldots, r_{n}\}$ , where  $r \in R$  and  $R$  is the set of the argument types. 2) MG-Reader (Du and Cardie, 2020a) improves document-level EE by proposing a novel multi-granularity reader to dynamically aggregate information in sentence and paragraph-level. 3) DocEDAG (Zheng et al., 2019) generates an entity-based directed acyclic graph for document-level EE. 4) BERT_QA(Du and Cardie, 2020c) uses the argument type as a question to query the article for answer. Given the input article  $A$ , the argument type  $r \in R$  as the question, the output is  $O = \{start_{r}, end_{r}\}$ . We give  $-1$  for these not mentioned event arguments. 5) Ontology_QA. Following Vargas-Vera and Motta (2004), we refine the initial query in BERT_QA with argument ontology knowledge obtained from Oxford dictionary (Dictionary, 1989).
Considering the length limitation of pre-trained models, we split the article in three different ways.
(Sent) means to split the article by sentence<sup>10</sup>. (Chunk) means to split the article by every 128 tokens (default). (Doc) means no splitting. We adopt Longformer (Beltagy et al., 2020) as encoder for the (doc) baseline, and BERT-base for the other baselines.
Following prior work (Du and Cardie, 2020b), we use Head noun phrase Match (HM) and Exact Match (EM) as two evaluation metrics. HM is a relatively relaxed metric. As long as the head noun of the predicted result is consistent with the golden label, it will be judged as correct. While EM requires that the prediction result is exactly the same as the gold label, which is relatively stricter. Overall Performance As shown in Table 5, there is a big gap between the performance of SOTA models and human performance (41.0% Vs 85.9% in F score), indicating that document-level event argument extraction remains a challenge task.
The failure of existing baselines may be due to two reasons. One possible reason is the catastrophic forgetting in neural networks. Compared to NER and sentence-level EE, document-level EE (our task) highlights the model's capability to process long texts: the model has to read the entire text before determining the argument type of a span. Although a few models have been proposed to improve the long text capabilities of pre-trained models (such as longformer), and have achieved good results, (the performance of long-former (BERT_Seq(doc)) is superior to BERT_Seq(sent) and BERT_Seq(chunk) as shown in Table 5), but these models still have a big performance gap compared with human beings.
Another reason is that existing baselines suffer from the inferior capability in semantic understanding, which is reflected in two aspects: 1) EE models
10 https://www.nltk.org/api/nltk_tokenize.html
fail to distinguish arguments of similar events. For instance, the article mainly describes the 2021 U.S. Alaska Peninsula earthquake, and also briefly mentions 2008 Wenchuan earthquake. When asking the Date of the main event, EE models are easy to confuse the correct answer 2021 with the wrong answer 2008. 2) EE models often mistake unrelated entities for event arguments. For example, when extracting the event argument Attack Target in the 911 terrorist attack on the Pentagon event, except to the correct answer the New York Pentagon, EE models often mistake other unrelated location entities in the article (such as Mount Sinai Hospital) as one of the answers.
We believe that the following research directions are worthy of attention: 1) Exploring pre-trained models with stronger long text processing capabilities. 2) Exploiting ontology and commonsense knowledge to improve the semantic understanding of EE models. In the future, we will focus on promote event extraction to a higher level, such as cross-document level.
In this paper, we present DocEE, a large-scale document-level EE dataset to promote event extraction from sentence-level to document-level. Comparing to existing datasets, DocEE greatly expands the data scale, with more than 27,000+ events and 180,000+ arguments, and contains more refined event arguments. Experiments show that DocEE remains an open issue.
This work is supported by the National Key Research and Development Program of China (2018YFB1005100) and (2018YFB1005101), National Engineering Laboratory for Cyberlearning and Intelligent Technology, Beijing Key Lab of Networked Multimedia, the Institute for Guo Qiang, Tsinghua University (2019GQB0003), the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant, and Beijing Academy of Artificial Intelligence.
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
Collin F. Baker. 2014. FrameNet: A knowledge base for natural language processing. In Proceedings of Frame Semantics in NLP: A Workshop in Honor of Chuck Fillmore (1929-2014), pages 1-5, Baltimore, MD, USA. Association for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Yubo Chen, Shulin Liu, Xiang Zhang, Kang Liu, and Jun Zhao. 2017. Automatically labeled data generation for large scale event extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 409-419, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Oxford English Dictionary. 1989. Oxford english dictionary. Simpson, Ja & Weiner, Esc.
Xiao Ding, Zhongyang Li, Ting Liu, and Kuo Liao. 2019. ELG: an event logic graph. CoRR, abs/1907.08015.
Zhendong Dong and Qiang Dong. 2003. Hownet - a hybrid language and knowledge resource. In International Conference on Natural Language Processing and Knowledge Engineering, 2003. Proceedings. 2003, pages 820-824.
Valentina Dragos. 2013. Developing a core ontology to improve military intelligence analysis. International Journal of Knowledge-based and Intelligent Engineering Systems, 17(1):29-36.
Xinya Du and Claire Cardie. 2020a. Document-level event role filler extraction using multi-granularity contextualized encoding. CoRR, abs/2005.06579.
Xinya Du and Claire Cardie. 2020b. Document-level event role filler extraction using multi-granularity contextualized encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8010-8020, Online. Association for Computational Linguistics.
Xinya Du and Claire Cardie. 2020c. Event extraction by answering (almost) natural questions. arXiv preprint arXiv:2004.13625.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence argument linking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8057-8077, Online. Association for Computational Linguistics.
Ronen Feldman and James Sanger. 2006. Information Extraction, page 94-130. Cambridge University Press.
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, and Ming Zhou. 2018. Eventwiki: A knowledge base of major events. In LREC 2018. LREC 2018.
Simon Gottschalk and Elena Demidova. 2018. Eventkg: A multilingual event-centric temporal knowledge graph. CoRR, abs/1804.04526.
Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
Felix Hamborg, Corinna Breitinger, and Bela Gipp. 2019. *Giveme5w1h: A universal system for extracting main events from news articles.* In *Proceedings of the 13th ACM Conference on Recommender Systems*, 7th International Workshop on News Recommendation and Analytics (INRA 2019).
Felix Hamborg, Soeren Lachnit, Moritz Schubotz, Thomas Hepp, and Bela Gipp. 2018. *Giveme5w: Main event retrieval from news articles by extraction of the five journalistic w questions*.
Daniel Hienert and Francesco Luciano. 2012. Extraction of historical events from wikipedia. CoRR, abs/1205.4138.
Andrew Hsi. 2018. Event Extraction for Document-Level Structured Summarization. Ph.D. thesis, Carnegie Mellon University.
Jin-Dong Kim, Yue Wang, Toshihisa Takagi, and Akinori Yonezawa. 2011. Overview of genia event task in bionlp shared task 2011. In Proceedings of BioNLP shared task 2011 workshop, pages 7-15.
Jin-Dong Kim, Yue Wang, and Yamamoto Yasunori. 2013. The genia event extraction shared task, 2013 edition-overview. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 8-15.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. CoRR, abs/1408.5882.
Kowsari, Jafari Meimandi, Heidarysafo, Mendu, Barnes, and Brown. 2019. Text classification algorithms: A survey. Information, 10(4):150.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations.
Sam N. Lehman-Wilzig and Michal Seletzky. 2010. Hard news, soft news, 'general' news: The necessity and utility of an intermediate classification. Journalism, 11(1):37-56.
Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. CoRR, abs/2104.05919.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282.
Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besançon. 2016. A dataset for open event extraction in English. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1939-1943, Portoroz, Slovenia. European Language Resources Association (ELRA).
T. Nguyen and R. Grishman. 2018. Graph convolutional networks with argument-aware pooling for event detection. In AAAI.
Tomoko Ohta, Sampo Pyysalo, Rafal Rak, Andrew Rowley, Hong-Woo Chun, Sung-Jae Jung, Sung-Pil Choi, Sophia Ananiadou, and Jun'ichi Tsujii. 2013. Overview of the pathway curation (pc) task of bionlp shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 67-75.
Tomoko Ohta, Sampo Pyysalo, and Jun'ichi Tsujii. 2011. Overview of the epigenetics and posttranslational modifications (epi) task of bionlp shared task 2011. In Proceedings of BioNLP Shared Task 2011 Workshop, pages 16-25.
Walker Orr, Prasad Tadepalli, and Xiaoli Fern. 2018. Event detection with neural networks: A rigorous empirical evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 999-1004, Brussels, Belgium. Association for Computational Linguistics.
Sampo Pyysalo, Tomoko Ohta, and Sophia Ananiadou. 2013. Overview of the cancer genetics (CG) task of BioNLP shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 58-66, Sofia, Bulgaria. Association for Computational Linguistics.
Carsten Reinemann, James Stanyer, Sebastian Scherr, and Guido Legnante. 2012. Hard and soft news: A review of concepts, operationalizations and key findings. Journalism, 6(2):221-239.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Meihan Tong, Bin Xu, Shuai Wang, Yixin Cao, Lei Hou, Juanzi Li, and Jun Xie. 2020. Improving event detection via open-domain trigger knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5887-5897, Online. Association for Computational Linguistics.
Gaye Tuchman. 1973. Making news by doing work: Routinizing the unexpected. American journal of Sociology, 79(1):110-131.
Sofie Van Landeghem, Jari Björne, Chih-Hsuan Wei, Kai Hakala, Sampo Pyysalo, Sophia Ananiadou, Hung-Yu Kao, Zhiyong Lu, Tapio Salakoski, Yves Van de Peer, et al. 2013. Large-scale event extraction from literature with multi-level gene normalization. PloS one, 8(4):e55814.
Maria Vargas-Vera and Enrico Motta. 2004. Aquantology-based question answering system. In Mexican International Conference on Artificial Intelligence, pages 468-477. Springer.
Peng Wang, Zhenkai Deng, and Ruilong Cui. 2021. Tdjee: A document-level joint model for financial event extraction. *Electronics*, 10(7):824.
Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A massive general domain event detection dataset. CoRR, abs/2004.13590.
Runxin Xu, Tianyu Liu, Lei Li, and Baobao Chang. 2021. Document-level event extraction via heterogeneous graph-based interaction model with a tracker. CoRR, abs/2105.14924.
Ge Zhan, Ming Wang, and Meiyi Zhan. 2020. Public opinion detection in an online lending forum: Sentiment analysis and data visualization. In 2020 IEEE 5th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), pages 211-213.
Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019. Doc2edag: An end-to-end document-level framework for chinese financial event extraction. arXiv preprint arXiv:1904.07535.
Figure 5: The online annotation page for Stage 1 event classification. The annotator needs to read the news title and the article to decide the event type of the main event. We provide examples for each event type to facilitate understanding.
Figure 6: The online annotation page for Stage 2 event role extraction. The annotator first draws the answer from text on the left, and then selects the appropriate event role label from the label column on the right. The final drawn result will be displayed on the right.
Table 6: Event Type Statistic
Event Type	Documents	Event Type	Documents
Armed Conflict	2200	Air crash	1735
Protest	1209	CommitCrime - Accuse	1127
Election	1112	Sports Competition	1050
Fire	926	CommitCrime - Sentence	893
Government Policy Changes	838	Earthquakes	756
Riot	720	Resign	667
Famous Person - Death	569	Disease Outbreaks	550
Floods	516	Appoint	508
Storm	506	Diplomatic Talks	502
Strike	471	CommitCrime - Arrest	439
Road Crash	418	Environment Pollution	385
Organization Closed	382	Gas Explosion	370
Bank Robbery	363	Break Historical Records	359
Sign Agreement	341	Awards Ceremony	331
Famous Person - Give a speech	327	Shipwreck	321
Mine Collapses	318	CommitCrime - Release	313
Military Exercise	307	Mass Poisoning	303
Financial Crisis	302	New Achievements in Aerospace	299
Train Collisions	298	Withdraw from an Organization	292
Diplomatic Visit	269	Organization Merge	265
Insect Disaster	265	Organization Fine	260
CommitCrime - Investigate	260	New Wonders in Nature	255
Volcano Eruption	247	Famine	247
Famous Person - Sick	244	New Archeological Discoveries	240
Tear Up Agreement	222	Famous Person - Marriage	194
Financial Aid	189	Organization Established	167
Famous Person - Divorce	152	Tsunamis	140
Droughts	129	Mudslides	123
Famous Person - Recovered	118	Join in an Organization	117
Regime Change	69
Table 7: Template for Collecting Candidate Data
Template
"List of"+event type
event type+"in"+year
"Category:"+event type+"in"+country
"Category:"+event type+"in"+year
"List of"+event type+"in"+country
"List of"+event type+"by"+property
"List of"+event type+"in"+year
"List of historical"+event type year+event type
location+event type