SlowGuess/NAACL_2022_OCR huggingface dataset:
Meihan Tong $^{1}$ , Bin Xu $^{1*}$ , Shuai Wang $^{2}$ , Meihuan Han $^{1}$ , Yixin Cao $^{3}$ , Jiangqi Zhu $^{1}$ , Siyu Chen $^{1}$ , Lei Hou $^{1}$ , Juanzi Li $^{1}$
$^{3}$ Singapore Management University, Singapore
Event extraction aims to identify an event and then extract the arguments participating in the event. Despite the great success in sentence-level event extraction, events are more naturally presented in the form of documents, with event arguments scattered in multiple sentences. However, a major barrier to promote document-level event extraction has been the lack of large-scale and practical training and evaluation datasets. In this paper, we present DocEE, a new document-level event extraction dataset including 27,000+ events, 180,000+ arguments. We highlight three features: largescale manual annotations, fine-grained argument types and application-oriented settings. Experiments show that there is still a big gap between state-of-the-art models and human beings (41% Vs 85% in F1 score), indicating that DocEE is an open issue. DocEE is now available at https://github.com/tongmeihan1995/DocEE.git.
Event Extraction (EE) aims to detect events from text, including event classification and event argument extraction. EE is one of the fundamental tasks in text mining (Feldman and Sanger, 2006) and has many applications. For instance, it can monitor political or military crises to generate real-time notifications and alerts (Dragos, 2013), and dig the links and connections (e.g., Who Met Whom and When) between dignitaries for portrait analysis (Zhan et al., 2020).
Most existing datasets (e.g., ACE2005<sup>1</sup> and KBP2017<sup>2</sup>) focus on sentence-level event extraction, while events are usually described at the document level, and event arguments are typically scattered across different sentences (Hamborg et al.,
2019). Figure 1 shows an Air Crash event. To extract argument Date, we need to read sentence [1], while to extract argument Cause of the Accident, we need to integrate information in sentences [6] and [7]. Clearly, this requires reasoning over multiple sentences and modeling long-distance dependency, intuitively beyond the reach of sentence-level EE. Therefore, it is necessary to move EE forward from sentence-level to document-level.
Only a few datasets are curated for document-level EE. MUC-4(Grishman and Sundheim, 1996) provides 1,700 news articles annotated with 4 event types and 5 argument types. The 5 arguments are shared among different event types without further refinement. WikiEvents(Li et al., 2021) consists of only 246 documents with very few (22% of total) cross-sentences argument annotations. RAMS(Ebner et al., 2020) limits the scope of the arguments in a 5-sentence window around its event trigger, which is not in line with the actual application, and the number of the argument types in RAMS is only 65, which is quite limited. Doc2EDAG, TDJEE and GIT (Zheng et al., 2019; Wang et al., 2021; Xu et al., 2021) contain only 5 event types and 35 argument types in financial domain. In summary, existing datasets for document-level EE fail in the following aspects: small scale of data, limited coverage of domain and insufficient refinement of argument types. Therefore, it is urgent to develop a manually labeled, large-scale dataset to accelerate the research in document-level EE.
In the paper, we present DocEE, a large-scale human-annotated document-level EE dataset. Figure 1 illustrates an example of DocEE. DocEE focuses on the extraction of the main event, that is one-event-per-document. We regard news headlines as the main event trigger and focus on main event arguments extraction throughout the article. We highlight the following three contributions of DocEE to this field: 1) Large-scale Manual Annotations. Do
1 https://catalog.ldc.upenn.edu/LDC2006T06
2 https://tac.nist.gov/2017/KBP/
Human Language Technologies, pages 3970 - 3982
NAF Plane Crash: Military Takes Over Site, AIB To Commence Investigation
[1]Soldiers have cordoned off the site where the Nigerian Airforce plane crashed on Friday evening at the Kaduna International Airport.
[2]The plane which was carrying the Chief of Army Staff, Lieutenant General Ibrahim Attahiru, and other Senior Army Officers crashed near the airport's active runway, killing all the eleven people on board.
[3]Speaking on the incident, the Manager of Kaduna airport, Amina Salami told Channels Television that the military authorities have taken over the crash site. [4]She added that they have prevented civil aviation officials from gaining access to the crash site pending the arrival of officials from the Accident Investigation Bureau (AIB). [5]She explained that the military plane was initially scheduled to land at the Nigerian Airforce Base in Mando area, but was later diverted to the Kaduna airport due to poor weather conditions. The service life of the aircraft is three years.
[6]The incident happened as the plane was trying to land in bad weather, the military said. [7]President Muhammadu Buhari said that the lack of strict pre-flight inspections was also the cause of this disaster. [8]It was part of the government's plan to boost the military's efficiency in fighting a more than decade-long jihadist insurgency.
Event Type: Air Crash
Event Roles
Date
Location
Casualties and Losses
Survivors
Investigator of the Accident
Cause of the Accident
Passengers
Crew
Aircraft Agency
Flight No.
Service years
O Taking-off Place
Scheduled Landing Place
Alternate Landing Place
Figure 1: An example from DocEE. Each document in DocEE is annotated with event type and involved event arguments. In the example, the document mainly describes a Air Crash event which contains the following arguments: Data, Location, Causality and Losses and etc. We use different colors to distinguish event arguments.
cEE contains 27,485 document-level events with 180,528 arguments, far exceeding the scale of existing document-level EE datasets. The large-scale annotations of DocEE can provide sufficient training and testing data, to fairly evaluate EE models. 2) Fine-grained argument types. DocEE has a total of 356 argument types, which is much more than the number of argument types in existing dataset (5 in MUC-5 and 65 in RAMS). Besides the general arguments, such as time and location, we design more personalized event arguments for each event type, such as Water Level for Flood event and Magnitude for Earthquake event. These fine-grained roles can bring more detailed semantics and pose a higher challenge to the semantic disambiguation ability of existing models. 3) Application-oriented settings. In the actual application, event extraction often face the problems of how to quickly adapt from the rich-resource domains to new domains. Therefore, we have added a cross-domain setting to better test the transfer capability of the EE models. In addition, DocEE removes the limitation that the arguments range should be within a certain window in RAMS, to better cope with realistic scenarios where the length of the article will be particularly long, and the argument of the event may appear in any corner of the article. With more scattered event arguments (see Table 1), DocEE poses a higher challenge to the long text processing capability of existing models.
To assess the challenges of DocEE, we implement 9 recent state-of-the-art EE models on DocEE along with human evaluation. Experi
ments demonstrate the high-quality of DocEE and show that even the performance of SOTA model is far lower than human performance, showing that the faintness of existing technology in processing document-level EE.
Sentence-level Event Extraction Dataset Automatic Content Extraction (ACE2005)<sup>1</sup> consists of 599 documents with 8 event types and 33 subtypes. Text Analysis Conference (TACKBP)<sup>2</sup> also releases three benchmarks: TACKBP 2015/2016/2017, with 9/8/8 event types and 38/18/18 event subtypes. RED<sup>3</sup> annotates events from 95 English newswires. Chinese Emergency Corpus (CEC) focuses on Chinese breaking news, with a total of 332 articles in 5 categories. MAVEN (Wang et al., 2020) and LSEE (Chen et al., 2017) only annotate event triggers, with 168/21 types of trigger instances in 11,832/72,611 sentences. Based on them, various superior models have been proposed to improve the sentence-level EE and have achieved great success (Orr et al., 2018; Nguyen and Grishman, 2018; Tong et al., 2020).
Document-level Event Extraction Dataset Most of the existing document-level event datasets only focus on event classification, but lack event argument labelings, such as 20news and THUC-News. There are a few datasets annotated with cross-sentences event arguments. MUC-4 (Nguyen
3 https://catalog.ldc.upenn.edu/LDC2016T23
4 https://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups
5 http://thuctc.thunlp.org
Figure 2: Five examples of event schema in DocEE.
et al., 2016) only contains 4 event types and 5 argument types, and the 4 event types are close to each other and limited to the terrorist attack topic<sup>6</sup>. WikiEvents (Li et al., 2021) and RAMS(Ebner et al., 2020) consist of 246/9,124 documents with only 59/65 argument types, and most of the arguments in the two datasets are shared among different event types without further refinement. Doc2EDAG, TDJEE and GIT (Zheng et al., 2019; Wang et al., 2021; Xu et al., 2021) only define 5 event types and 35 argument types in financial domain. Cancer Genetics, EPM, GENIA2011, GENIA2013, Pathway Curation and MLEE (Pyysalo et al., 2013; Ohta et al., 2011; Kim et al., 2011, 2013; Ohta et al., 2013; Van Landeghem et al., 2013) are limited to the biological domain. In summary, these datasets are either limited to specific domains, or have very limited data scale, or have not carefully refined event argument schema.
Open-domain Event Extraction Dataset To collect EE dataset in open domain, one way is to leverage semi-structured resources (Wikipedia) or existing knowledge bases (Freebase). The representative works are EventKG (Gottschalk and Demidova, 2018), Event Wiki (Ge et al., 2018) and Historical Wiki (Hienert and Luciano, 2012). The other way is to exploit open IE tools, such as dependency parsing, to extract events from unstructured text. The representative works are Event Logic Graph (Ding et al., 2019) and Giveme5W1H (Hamborg et al., 2019). The advantage of the open-domain EE dataset is its large scale, but the disadvantage is that it lacks manual review, and thus the quality cannot be guaranteed.
3 Constructing DocEE
Our main goal is to collect a large-scale dataset to promote the development of event extraction from sentence-level to document-level. In the following sections, we will first introduce how to construct the event schema, and then how to collect candidate data and how to label them through crowdsourcing.
3.1 Event Schema Construction
News is the first-hand source of hot events, so we focus on extracting events from news. Previous event schemas, such as FrameNet (Baker, 2014) and HowNet (Dong and Dong, 2003), pay more attention to trivial actions such as eating and sleeping, and thus is not suitable for document-level news event extraction.
To construct event schema, we gain insight from journalism. Journalism typically divides events into hard news and soft news (Reinemann et al., 2012; Tuchman, 1973). Hard news is a social emergency that must be reported immediately, such as earthquakes, road accidents and armed conflicts. Soft news refers to interesting incidents related to human life, such as celebrity deeds, sports events and other entertainment-centric reports. Based on the hard/soft news theory and the category framework in (Lehman-Wilzig and Seletzky, 2010), we define a total of 59 event types, with 31 hard news event types and 28 soft news event types. Detailed information is shown in Appendix Table 1. Our schema covers influential events of human concern, such as earthquakes, floods and diplomatic summits, which cannot be extracted at the sentence level and require multiple sentences to describe.
To construct argument schema, we leverage infobox in Wikipedia. As shown in Figure 3(a), the Wikipedia page describes an event, and the keys
6 https://www-nlpir.nist.gov/related_projects/muc/muc_data/muc_data_index.html
in the infobox, such as Date and Total fatalities, can be regarded as the prototype arguments of the event. Based on this observation, we manually collect 20 wiki pages for each event type, and use their shared keys in infobox as our basic set of argument types. After that, we further expand the basic set. Specifically, for event type  $e$ , we first collect 20 news stories from New York Times, and then invited 5 students (native English-speaking, major in journalism) to summarize the key facts the public would like to learn from the news of  $e$ . For instance, in Flood event news, Water Level is a key fact, because it is an important factual basis for flood cause analysis and disaster relief decision-making, and can arouse widespread concern. Finally, by merging the key facts of the 5 students, we complete the argument types expansion. To ensure the quality, we further invite the above 5 students to make a trial labeling on the collected news, and filter argument types that appear less frequently in the article.
In total, we define 356 event arguments types for 59 event types. On average, there are 6.0 event arguments per class. Figure 2 illiterates some examples of event arguments types we defined. The complete schema and corresponding examples can be found Event Schema.md in the supplementary materials.
3.2 Candidate Data Collection
In this section, we introduce how to collect candidate document-level events. We choose wiki as our annotation source. Wiki contains two kinds of events: historical events and timeline events (Hienert and Luciano, 2012). Historical events refer to the events that have their own wiki page, such as 1922 Picardie mid-air collision. Timeline events refer to the news events organized in chronological order, such as A heatwave strikes India and South Asia in wiki page Portal:Current_events/June_2010.7 Figure 3 shows examples of two events. We adopt both kinds of events as our candidate data, because only using historical events will lead to uneven data distribution under our event schema, and timeline events can be a good supplement.
For a historical event, we adopt its Wikipedia article as the document of the event arguments to be annotated. For a timeline event, we use the URL to download the original news article as the
document of the event arguments to be annotated. Because  $22\%$  of the timeline events do not have URLs (Wikipedia editors do not provide the URL when editing the entry), so we use Scale SERP8 to find news articles and manually confirm their authenticity. For historical event, we adopt templates+event type as the query key to retrieve candidate events. The templates include "List of" + event type, event type + "in" + year, "Category:" + event type + "in" + country, etc. More templates show in Appendix Table 7. For timeline event, we choose events between 1980 and 2021 as candidates, because there are very few events before 1980.
In order to balance the length of the article, we filtered out articles less than 5 sentences, and also truncated articles that were too long (more than 50 sentences). Finally, we select 44,000 candidate events from Wikipedia.
3.3 Crowdsourced Labeling
Given the candidate events and the predefined event schema, we now introduce how to annotate them through crowdsourcing. To ensure the quality of annotations, all annotators are either native English speakers or English-major students with TOEFL higher than 100 or IELTS higher than 7.5. The crowdsourced labeling process consists of two stages.
3.3.1 Stage 1: Event Classification
At this stage, annotators are required to classify candidate events into predefined event types. Following (Hamborg et al., 2018; Hsi, 2018), we focus on main event classification, so Stage 1 is a single-label classification task. Specifically, the main event refers to the event reflected in the title and mainly described in the article. Formally, given the candidate event  $e = < t, a >$ , where  $t$  represents the title and  $a$  represents the article, Stage 1 aims to obtain label  $y$  for each  $e$ , where  $y$  belongs to the 59 event types defined in subsection 3.1.
In total, we invite about 60 annotators to participate in Stage 1 annotation. The online annotation page is displayed in Appendix Figure 5. We first manually label 100 articles as standard answers to pre-test annotators, and weed out annotators with an accuracy rate of less than  $70\%$ , which left us 48 valid annotators. Then, we ask two independent annotators to annotate each candidate event. If the results of the two annotators are inconsistent
7 en.wikipedia.org/wiki/Portal:Current_events/June_2010
8 https://app.scalesrp.com/playground
Figure 3: Two sources of candidate events in DocEE. The left is a historical event, which has its own wiki page, and the right are two timeline events arranged in a wiki page by time unit. Each timeline event consists of a brief description and a URL pointing to original news.
(32.8% in this case), a third annotator will be the final judge. Due to the variety of event types in reality, a candidate event may not belong to any predefined class. We classify such event into the other class, which accounts for 23.6% of the total data.
3.3.2 Stage 2: Event argument Extraction
At this stage, annotators are required to extract event arguments from the whole article. Formally, given the candidate event  $e = < t, a >$ , its event type  $y$  and the predefined argument types  $R$  of  $y$ , Stage 2 aims to find all the arguments from the article  $a$ .
Due to the heavy workload in Stage 2, we invite more than 90 annotators. An example of the online annotation page is shown in Appendix Figure 6. We use a preliminary annotation - multiple rounds inspection method for labeling. In the preliminary annotation step, each article will be labeled by an annotator. We distribute no more than two event types to each annotator in this step to make the annotators more focused. Then, in the step of multiple rounds inspection, we first select high-precision annotators based on inter-annotator agreement to form a reviewer team (44.4% of the total), and then each article will go through three rounds of error correction by three independent annotators in the reviewer team. After each round, we will feed back annotation issues to the reviewers so that they can correct them in the next round of annotation. The accuracy rate has steadily increased from 56.24%, 76.83% to 85.96% after each round, which shows the effectiveness of our labeling method. We take the third round results as the final annotations.
We clarify some annotation details here. We do not include articles, prepositions in our annotations.
For instance, we select "damaged car" among "damaged car", "damaged car belonging to the victim" and "the damaged car". For event arguments with multiple mentions in the document, for example, Cause of the Accident in Figure 1 that has two mentions, we will label all mentions to ensure the completeness of the extraction. For repeated mentions that refer to the same entity, we only label once.
3.3.3 Annotation Quality & Remuneration
Following (Artstein and Poesio, 2008; McHugh, 2012), we use Cohen's kappa coefficient to measure the Inter-Annotator Agreement(IAA). The IAA scores are  $94\%$  and  $81\%$  for State 1 Event Classification and Stage 2 Event Argument Extraction respectively, which are relatively high. The annotators spend an average of 0.5 minutes labeling a piece of data in Stage 1, so we pay them  $0.1$  for each piece of data. It takes about 5 minutes to label a piece of data in Stage 2, so we pay  $0.8$  for each piece of data.
4 Data Analysis of DocEE
In the section, we analyze various aspects of DocEE to provide a deep understanding of the dataset and the task of document-level event extraction.
Overall Statistic In total, DocEE labels 27,485 valid document-level events and 180,528 event arguments. Each article is annotated with 6.6 event arguments on average. Event Famous Person - Divorce has the highest average number of event arguments per article (18.1), while event Regime Change has the lowest average number of event arguments per article (3.8). We compare DocEE to various representative event extraction datasets in Table 1, including sentence-level EE datasets
Datasets	#isDocEvent	#EvTyp.	#ArgTyp.	#Doc.	#Tok.	#Sent.	#ArgInst.	#ArgScat.
ACE2005	✘	33	35	599	290k	15,789	9,590	1
KBP2016	✘	18	20	169	94k	5,295	7,919	1
KBP2017	✘	18	20	167	86k	4,839	10,929	1
MUC-4	✓	4	5	1,700	495k	21,928	2,641	4.0
WikiEvents	✓	50	59	246	190k	8,544	5,536	2.2
RAMS	✓	139	65	9,124	957k	34,536	21,237	4.8
DocEE(ours)	✓	59	356	27,485	16,268k	749,568	180,528	10.2
Table 1: Statistics of EE datasets (isDocEvent: whether the event in the corpus at the document-level, EvTyp.: event type, ArgTyp.: event argument type, Doc.: document, Sent.: sentence, ArgInst.: event arguments, ArgScat.: the number of sentences in which event arguments of the same event are scattered)
Figure 4: Top 15 event types in DocEE.
ACE2005, KBP and document-level EE dataset MUC-4, Wikievents, RAMS. We find that DocEE is much larger than existing datasets in many aspects, including the documents number and argument instances number. Compared to MUC-4, DocEE has far more event arguments (180,528 compared to 2,641). The reason is that among the 1,700 documents in MUC-4,  $47.4\%$  of articles are not labeled with any event argument, while DocEE guarantees that each article contains at least three event argument labels in crowdsourcing process, which greatly solves the problem of data scarcity of the event arguments in document-level EE.
Event Type Statistic Figure 4 shows the distribution of the top 15 frequent event types that have the most number of instances in DocEE. DocEE covers a variety of event types, including Armed Conflict (8.0%), Air crash (6.3%), Protest (4.4%), CommitCrime - Accuse (4.1%), Election (4.0%), Sports Competition (3.8%), Fire (3.4%), etc. Our annotated data follows a long-tailed distribution, which is due to the uneven distribution of class in the real data. According to statistics, there are
30.5% of classes with more than 500 instances and 83.1% of classes with more than 200 instances. More detailed information is shown in Appendix Table 6.
Event Arguments Statistic We randomly sample 1000 articles from DocEE for manual analysis, which contains a total of 4962 event arguments instances. We first classify event arguments based on their mention numbers. As shown in Table 2,  $78.6\%$  of the event arguments have a unique mention, and  $21.4\%$  of the event arguments have multiple mentions, which poses a great challenge to the model's recall capability. Then, we classify event arguments based on their mention lengths.  $60.8\%$  of the event arguments have no more than 3 words, and most of them are named entities such as person and location. While  $30.8\%$  event arguments have less than 10 words and  $8.4\%$  event arguments are answered by more than 10 words, such event arguments mainly include Cause of the Accident, Investigation Results, etc.
5 Experiments on DocEE
In this section, we show the challenges of DocEE by conducting comprehensive experiments on various SOTA models. We first introduce two benchmark settings, and then we conduct experiments on both event classification task and event argument extraction task. Finally, we discuss possible future research directions for document-level event extraction.
Benchmark Settings We design two benchmark settings for evaluation: normal setting and cross-domain setting. In the normal setting, we hope the training set and test set to be identically distributed. Specifically, for each event type, we randomly select  $80\%$  of the data as the training set,  $10\%$  of the data as the validation set, and the remaining  $10\%$  of the data as the test set.