MAVEN-ARG: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation (2024)

Abstract: Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships. However, due to the annotation challenges brought by task complexity, a largescale dataset covering the full process of event understanding has long been absent. In this paper, we introduce MAVEN-ARG, which augments MAVEN datasets with event argument annotations, making the first all-in-one dataset supporting event detection, event argument extraction (EAE), and event relation extraction. As an EAE benchmark, MAVEN-ARG offers three main advantages: (1) a comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples; (2) a large data scale, containing 98, 591 events and 290, 613 arguments obtained with laborious human annotation; (3) the exhaustive annotation supporting all task variants of EAE, which annotates both entity and non-entity event arguments in document level. Experiments indicate that MAVENARG is quite challenging for both fine-tuned EAE models and proprietary large language models (LLMs). Furthermore, to demonstrate the benefits of an all-in-one dataset, we preliminarily explore a potential application, future event prediction, with LLMs. MAVENARG and codes can be obtained from https: //github.com/THU-KEG/MAVEN-Argument.

heading: 1 Introduction
text: Conveying information about events is a core function of human languages (Levelt, 1993; Pinker, 2013; Miller and Johnson-Laird, 2013), which highlights event understanding as a major objective for natural language understanding and a foundation for various downstream applications (Ding et al., 2015; Li et al., 2018a; Goldfarb-Tarrant et al., 2019;
∗ Corresponding author.

heading: Location Agent
text: Huang et al., 2019; Wang et al., 2021a). As illustrated in Figure 1, event understanding is typically organized as three information extraction tasks (Ma et al., 2021; Peng et al., 2023b): event detection (ED), which detects event occurrences by identifying event triggers and classifying event types; event argument extraction (EAE), which extracts event arguments and classifies their argument roles; event relation extraction (ERE), which analyzes the coreference, temporal, causal, and hierarchical relationships among events.
Despite the importance of event understanding, a large-scale dataset covering all the event understanding tasks has long been absent. Established sentence-level event extraction (ED and EAE) datasets like ACE 2005 (Walker et al., 2006) and TAC KBP (Ellis et al., 2015, 2016; Getman et al., 2017) do not involve event relation types besides the basic coreferences. RAMS (Ebner et al., 2020) and WikiEvents (Li et al., 2021) extend EAE to the document level but do not involve event relations. ERE datasets are mostly developed independently for coreference (Cybulska and Vossen, 2014), temporal (Chambers et al., 2014; Ning et al., 2018), causal (Mirza et al., 2014; Mostafazadeh et al., 2016b; Caselli and Vossen, 2017), and subevent (Hovy et al., 2013; Glavaš
4072
and Šnajder, 2014) relationships and do not cover event arguments. Given annotation challenges from task complexity, these datasets often cover only thousands of events. Due to the inconsistent event schemata and data, these datasets cannot be unified. This status quo hinders the development of endto-end event understanding methods and limits the potential for event-based downstream applications.
MAVEN (Wang et al., 2020) is the largest humanannotated ED dataset, with a high-coverage event schema for general-domain events. Based on it, Wang et al. (2022) further annotates the first unified ERE dataset MAVEN-ERE, which covers all four types of event relationships and has a massive scale with more than one million event relations. Building on the sustained efforts of these works over years, we complete the puzzle of an all-in-one event understanding dataset in this work. We construct MAVEN-ARG, which provides exhaustive event argument annotations based on MAVEN.
Beyond finishing an all-in-one event understanding dataset, three main advantages of MAVEN-ARG make it a valuable EAE benchmark. (1) Comprehensive Event Schema. The original MAVEN schema only defines event types but without argument roles. We engage experts to enhance MAVEN schema with argument roles and to write detailed definitions for them, which help annotators and can also serve as task instructions for prompting large language models. The resulting event schema contains 162 event types, 612 argument roles, and 14, 655 words of definitions, which well cover general-domain events. (2) Large Data Scale. MAVEN-ARG comprises 107, 507 event mentions, 290, 613 event arguments, and 129, 126 entity mentions, all of which are human annotated. To our knowledge, this makes it the largest EAE dataset currently available. (3) Exhaustive Annotation. The development of EAE has seen many variations in task settings, including annotating only the topic event (Ebner et al., 2020; Tong et al., 2022) of a document or all fine-grained events (Walker et al., 2006), annotating event arguments at the sentence level (Walker et al., 2006) or document level (Ebner et al., 2020; Li et al., 2021), and limiting event arguments to entities (Walker et al., 2006; Li et al., 2021) or including non-entity arguments (Grishman and Sundheim, 1996; Parekh et al., 2023). MAVEN-ARG adopts the most exhaustive annotation. We annotate event arguments for all finegrained events at the document level, covering both entity and non-entity arguments. This enhances the
dataset’s utility for benchmarking and developing a wide range of EAE methods.
In the experiments, we reproduce several recent state-of-the-art EAE models as baselines and also evaluate large language models with in-context learning. Experimental results show that they can only achieve at most 40% F1 scores, which is far from promising. It indicates that MAVEN-ARG is quite challenging and more research efforts are needed to develop practical EAE methods. Furthermore, to demonstrate the advantage of an all-in-one event understanding dataset for enabling sophisticated event-based applications, we conduct a preliminary exploration of future event prediction. We sample causally related event chains from MAVENARG and prompt LLMs to predict future events, including their types and arguments. Experiments show that while most of the predictions are reasonable, they seldom align with the actual future. We encourage future work to further explore this application and hope MAVEN-ARG can help improve EAE and develop diverse event-based applications.

heading: 2 Dataset Construction
text: In this section, we introduce the dataset construction methodology of MAVEN-ARG, which involves designing event schema (§ 2.1), annotating entities (§ 2.2) and event arguments (§ 2.3).

heading: 2.1 Event Schema Creation
text: The event schema of MAVEN (Wang et al., 2020) covers a broad range of general-domain events and has a well-defined hierarchical structure. To enable event argument annotation based on MAVEN, one author and two engaged linguistic experts devoted three years to manually designing argument roles for MAVEN schema. Each argument role is accompanied by informative text definitions that are easy to understand, and each event type is provided with detailed annotation examples. An example in shown in appendix A.1. This not only helps annotators understand their tasks but also can prompt LLMs to perform EAE via in-context learning. To ensure quality, the argument role design for each event type is reviewed by at least one expert.
Our event schema creation involves the following steps: (1) Initially, to reduce annotation difficulty, we invite ten ordinary annotators, who are without dedicated study on event semantics, to review the event type schema and a portion of the data. Based on their feedback, we deleted 6
event types that are similar to others and renamed 4 event types for clarity. (2) The basic schema is constructed from a simplification and modification of FrameNet (Baker et al., 1998). The frame elements in FrameNet are widely considered akin to argument roles (Aguilar et al., 2014; Parekh et al., 2023), but they are often too complex for ordinary annotators to comprehend since FrameNet is primarily constructed for linguistic experts (Aguilar et al., 2014). Therefore, for each event type, we manually select frame elements related to describing events and suitable for annotation as MAVENARG argument roles from their FrameNet equivalents, and we rewrite the definitions and examples. (3) Extending argument roles based on the 5W1H analysis (What, Where, When, Why, Who, How) for describing events (Karaman et al., 2017; Hamborg et al., 2019). Temporal and causal relations from event relation extraction describe When and Why, while the event type describes What. We primarily refer to Who (participants), Where (locations), and How (manners, instruments, etc.) to design argument roles. (4) Considering the hierarchical structure. When designing subordinate types, we inherit and refine the argument roles of their superordinate types. (5) Sampling data to check if any event argument is missing.
Schema Statistics After the schema design, the final MAVEN-ARG schema contains 162 event types, 612 unique argument roles, and 14, 655 words of definitions. Taking inspiration from semantic role labeling (Fillmore, 1976; Banarescu et al., 2013), we tend to let the argument roles sharing the same semantic role use the same name but distinguish them with different textual definitions. For instance, we do not use Killer for the Killing event type and use Attacker for the Attack event type. Instead, we use Agent to denote them both but write different definitions for
them. This is to encourage the knowledge transfer between EAE for different event types. Therefore, 612 is the number of argument roles with unique definitions, and there are 143 unique names for all the argument roles. Table 1 compares the event schema size of MAVEN-ARG with existing EAE datasets, including ACE 2005 (Walker et al., 2006), DocEE (Tong et al., 2022), WikiEvents (Li et al., 2021), RAMS (Ebner et al., 2020), MEE (Pouran Ben Veyseh et al., 2022), and GENEVA1 (Parekh et al., 2023). We can observe that MAVEN-ARG has the largest event schema, which more comprehensively covers the broad range of diverse events and will help develop more generalizable methods.

heading: 2.2 Entity Annotation
text: The mainstream task setting for EAE (Walker et al., 2006; Li et al., 2021) confines event arguments to entities, which reduces the task’s complexity to some extent and provides more definite and standardized extraction results. Hence, before annotating event arguments, we annotate entities for the 4, 480 MAVEN documents. We follow the task definition and guidelines of a recent named entity recognition benchmark Few-NERD (Ding et al., 2021), but we only annotate coarse-grained entity types, including Person, Organization, Location, Building, Product, Art, and MISC. To deliver more unambiguous EAE results and reduce the argument annotation difficulty, we also annotate entity coreference, which means judging whether multiple entity mentions refer to the same entity.
During entity annotation, we engage 47 annotators, including 8 senior annotators selected during the annotation training. Each document is annotated by three independent annotators and further checked by one senior annotator. The final annotation results are aggregated via majority voting. If the senior annotator judged the accuracy of a document’s annotation to be below 90%, the document will be returned to the three first-stage annotators for re-annotation. To check data quality, we calculate Fleiss’ kappa (Fleiss, 1971) to measure the inter-annotator agreements. The result for entity recognition is 73.2%, and for entity coreference is 78.4%, both indicating high consistency.

heading: 2.3 Event Argument Annotation
text: Based on the event detection annotations of MAVEN and event coreferences of MAVEN-ERE,
1GENEVA has a larger ontology without data. Here we compare with its schema actually used in the dataset.
we conduct event argument annotations. For multiple coreferent event mentions (triggers), only one of them is displayed during annotation to reduce annotation overhead. Once the annotator selects an event trigger, the corresponding argument roles for its event type are displayed on the annotation interface, along with definitions and examples. This ensures that annotators do not have to memorize the lengthy event schema or frequently refer to the annotation guidelines. To annotate an event argument, annotators can either choose an entity from the whole document or select a continuous textual span; once an entity mention is selected, all of its coreferent entity mentions are automatically selected. Annotators have the option to report errors in the event type annotation of a trigger, which allows for the discarding of that trigger. In the annotation process, approximately 4% of triggers are discarded.
We employ 202 annotators, including 71 senior annotators selected during annotation training and 33 experts with rich annotation experiences. The experts are recommended by the commercial data annotation companies we employed, and they have undergone at least ten data annotation projects and led at least one. The annotation is divided into three phases. Each document is first annotated by an ordinary annotator, and then modified by a senior annotator. Finally, an expert will check whether the annotation accuracy reaches 90%. If not, the document’s annotation will be returned to the second phase. To measure data quality, we randomly sample 100 documents and conduct the three-phrase annotation for them twice with different annotator groups. The Fleiss’ kappa is 68.6%, which indicates a satisfactory level of annotation agreement. More annotation details are shown in appendix A.

heading: 3 Data Analysis
text: To provide intuitive descriptions for MAVEN-ARG, we conduct data analyses in this section.

heading: 3.1 Data Statistics
text: Table 2 shows the main statistics of MAVEN-ARG compared with various existing EAE datasets. Appendix B.1 further shows the statistics of different splits. We can observe that MAVEN-ARG has two advantages: (1) MAVEN-ARG has the largest data scale, surpassing previous datasets by several times. This ensures that even for long-tail event types, MAVEN-ARG has sufficient data to fully train and stably evaluate EAE models. (2) The exhaustive annotation of MAVEN-ARG makes it the only dataset that covers all settings of EAE task. MAVEN-ARG includes complete annotations of entity and event coreference and annotates both entity and non-entity arguments for all fine-grained events at the document level. This allows MAVEN-ARG to support the evaluation of all variants of EAE methods and the development of comprehensive event understanding applications.

heading: 3.2 Data Distribution
text: We present the distributions of the annotated entity and event arguments of MAVEN-ARG in Figure 2. Argument roles with the same name across different event types are merged for presentation clarity. We observe that: (1) The distribution of entity types is generally similar to that of FewNERD (Ding et al., 2021), demonstrating sufficient diversity. (2) The three most frequent basic argument roles (Agent, Patient, and Location) account for over 60% of event arguments. This highlights their ubiquity and encourages knowledge
transfer among different event types in EAE methods. (3) Event arguments exhibit a highly longtailed distribution. The 136 argument roles counted as “Others”, each constituting less than 1.5%, collectively accounts for 27.8% of event arguments. The long-tailed distribution of MAVEN-ARG poses a significant challenge to model generalizability.

heading: 3.3 Trigger-argument Distance
text: We analyze the distribution of trigger-argument distances in Figure 3. For events with multiple coreferent triggers and entity arguments with multiple entity mentions, the distance is calculated between the nearest trigger-argument pairs. The overall average trigger-argument distance is 37.8. From Figure 3, we observe that while the majority of event arguments are located near their triggers, which is natural for human writing, a substantial number of arguments are situated far from their triggers, with the furthest exceeding 800 words. This data characteristic challenges the ability of EAE methods to capture long-distance dependencies.

heading: 4 Experiment

heading: 4.1 Experimental Setup
text: Models To assess the challenge of MAVEN-ARG, we evaluate multiple advanced methods. For finetuned EAE models, we implement several stateof-the-art models, including DMBERT (Wang et al., 2019), CLEVE (Wang et al., 2021b), BERT+CRF (Wang et al., 2020), EEQA (Li et al., 2020), Text2Event (Lu et al., 2021), and PAIE (Ma et al., 2022). These methods cover all the mainstream EAE modeling paradigms (Peng et al., 2023c). Their detailed descriptions and implementations are introduced in appendix C.1. Recent works have achieved improvements in lowresource event extraction settings (Liu et al., 2023a; Ma et al., 2023). Considering our experiments prioritize understanding the challenge of large-scale
MAVEN-ARG over conducting comprehensive evaluations for existing models, we choose not to include those low-resource methods.
We also evaluate large language models (LLMs) with in-context learning on MAVEN-ARG. Specifically, we select two advanced LLMs, GPT3.5 (OpenAI, 2022) and GPT-4 (OpenAI, 2023), and evaluate them with 2-shot in-context learning. Here 2-shot means using full annotations of two documents as demonstrations. Considering time and cost constraints, we sample 50 documents from the test set for experimentation. We employ the gold trigger evaluation approach (Peng et al., 2023c) to directly assess their EAE performance.
Evaluation Metric Considering that MAVENARG covers non-entity argument annotations, traditional evaluation metrics (Peng et al., 2023c) designed only for entity arguments are no longer applicable. By taking each argument role as a question to the document, we propose to view EAE as a multi-answer question answering task2 and adopt its evaluation metrics (Rajpurkar et al., 2016; Amouyal et al., 2022; Yao et al., 2023), including bag-of-words F1 and exact match (EM).
Conventional evaluation calculates the micro average over all the entity and event mentions, which we dub it as mention-level evaluation. Considering that real-world applications only require the accurate prediction for one of all the coreferent mentions, we propose to consider entity (Li et al., 2021) and event coreference in evaluation. Specifically, for entity coreference level evaluation, an entity argument is considered as predicted correctly if one of its mentions is predicted correctly. For event coreference level evaluation, an argument is considered as predicted correctly if it is predicted correctly for one of the coreferent triggers.
2A single role may correspond to multiple argument spans (answers).

heading: 4.2 Experiment Results of Fine-tuned Models
text: The results of fine-tuned EAE models are shown in Table 3, and we have the following observations:
(1) Existing state-of-the-art EAE models exhibit moderate performance on MAVEN-ARG, which is significantly worse than their results on existing datasets (Peng et al., 2023c). This indicates that MAVEN-ARG is challenging and there is a need for increased efforts in developing practical event understanding models. (2) The BERT+CRF and PAIE models exhibit the best performance, potentially attributable to their ability to model rich interactions between different event arguments. (3) The previous top-performing classification-based models (DMBERT and CLEVE) (Peng et al., 2023c) perform poorly on MAVEN-ARG, which is due to their inability to handle non-entity arguments. Therefore, future research necessitates more flexible approaches to tackle the complex and real-world scenario in MAVEN-ARG. (4) Text2Event notably underperforms. This is potentially due to the intensive annotations of MAVEN-ARG, i.e., a high volume of events and argument annotations within a single document, making generating all events and arguments at once difficult. It indicates that generating complex structured outputs remains a major challenge for generation models (Peng et al., 2023a), requiring further exploration.

heading: 4.3 Experiment Results of LLMs
text: The results of LLMs with in-context learning are presented in Table 4, revealing that while LLMs with in-context learning are competitive compared to some fine-tuned EAE models, they still fall significantly short of the state-of-the-art. This is consistent with previous findings, suggesting that existing LLMs with in-context learning perform notably worse on specification-heavy information extraction tasks (Peng et al., 2023a; Li et al., 2023; Han et al., 2023). The LLMs’ bag-of-words F1 scores are notably higher than their exact match scores,

heading: DMBERT CLEVE BERT+CRF
text: suggesting that the LLMs’ predictions tend to be free-format and do not strictly match human annotations (Han et al., 2023).
One possible reason for the suboptimal performance is that LLMs cannot easily understand the schema from their names. Therefore, we conduct experiments with more informative prompts by incorporating definitions for each used argument role into the prompt, which are high-quality instructions used for guiding human annotators during data annotation. The results of these enhanced prompts are also shown in Table 4 (w/ definition). There is an obvious but marginal improvement after adding definitions, possibly due to the LLMs’ limitations in understanding long contexts (Shaham et al., 2022; Peng et al., 2023a; Liu et al., 2023b).

heading: 4.4 Analysis on Trigger-Argument Distance
text: As shown in Figure 3, MAVEN-ARG provides document-level annotations, covering data with varying trigger-argument distances. We conduct an analytical experiment on the impact of triggerargument distance to model performance. Specifically, we break down the predictions and annota-
tions in the test set by their trigger-argument distances and evaluate how the performance changes along with different distances. The experimental results are shown in Figure 4, which demonstrate that models perform significantly worse on samples with longer trigger-argument distances. This aligns with previous findings in document-level relation extraction regarding the distance between entity pairs (Ru et al., 2021). It suggests that modeling long-distance dependencies between triggers and arguments remains a challenge for existing EAE models. Future research can leverage MAVENARG to explore advanced methods for handling long-distance trigger-argument instances.

heading: 4.5 Analysis on Entity and Non-Entity Arguments
text: MAVEN-ARG provides comprehensive annotations, including both entity and non-entity arguments. We analyze the performance breakdown of investigated EAE models on these two types of arguments. The results are presented in Table 5, which reveals that EAE models generally perform better on non-entity arguments. The possible reason may be that there are more non-entity arguments in MAVEN-ARG and non-entity arguments are often presented in a looser form, making it easier for the models to learn the patterns and extract them. An exception is ob-
served for the generation-based model Text2Event, which exhibits poorer performance on non-entity arguments. This may be because non-entity arguments are typically longer, which are harder to generate at once. It suggests that further exploration is needed to investigate how to effectively handle EAE with generation methods.

heading: 4.6 Error Analysis
text: Table 3 shows that all the baselines can only achieve moderate performance on MAVEN-ARG, which demonstrates the challenge of our new dataset. To better understand the challenge of MAVEN-ARG and provide insights for future improvements, we conduct an error analysis for the top-performing PAIE (Ma et al., 2022) model. One of the authors manually checked the erroneous predictions on the arguments of 50 randomly sampled event triggers and categorized the errors into four categories: (1) False Negative (52.2%), which means PAIE falsely ignores some event arguments; (2) False Positive (12.5%), which means the model falsely identify non-argument entities/spans as event arguments; (3) Span Error (35.3%), which means the model partially corrects predicts an event argument but fails to precisely predict its exact textual span; (4) Classification Error (0.02%), which means the model correctly predicts the span of an event argument but misclassifies its argument role. We can see that similar to MAVEN (Wang et al., 2020), the majority of errors are still identification errors, which means that identifying event arguments from numerous entities/spans is much more challenging than determining specific argument roles.

heading: 5 Future Event Prediction Demonstration
text: MAVEN-ARG, in conjunction with MAVEN and MAVEN-ERE, creates the first all-in-one event understanding benchmark, which covers the full process of ED, EAE, and ERE. Beyond serving as an evaluation benchmark for these tasks, an allin-one event dataset naturally enables a variety of

heading: Model Reasonable (%) Matched (%)
text: event-based applications, especially considering the recent advances brought by LLMs. Here we preliminarily explore an application case, future event prediction, as a demonstration.
Predicting future events based on causality can help decision-making, which is of self-evident importance. Therefore, since the early script learning (Schank and Abelson, 1975; Mooney and DeJong, 1985), future event prediction has continually attracted research interest (Chambers and Jurafsky, 2008; Jans et al., 2012; Granroth-Wilding and Clark, 2016; Hu et al., 2017; Chaturvedi et al., 2017; Li et al., 2018b; Lee and Goldwasser, 2019; Zhao, 2021). However, due to the lack of highquality event resources, the evaluation of future event prediction often compromises by merely predicting verbs and subjects (Chambers et al., 2014), predicting according to textual order (Jans et al., 2012), or selecting story endings (Mostafazadeh et al., 2016a; Chaturvedi et al., 2017). The MAVEN series of datasets, with annotations of complete event structures and rich causal relations, may aid in predicting future events in real-world scenarios.
Experiment Setup We sample 100 event chains, each consisting of 3-5 events, from the training and validation sets. In each chain, preceding events cause the subsequent ones. Events are described in a structured JSON format, containing event type, event trigger, and event arguments. For each event chain, we hold out the last event and input the remaining incomplete chain into two proprietary LLMs, GPT-3.5 and GPT-4 (OpenAI, 2023), requiring them to predict the next occurring event. These LLMs are prompted with detailed task instructions and 5 demonstration event chains. To minimize the influence of the demonstrations, predictions are made independently three times under different demonstrations. More experimental details are shown in appendix D. We employ manual evaluation, with two experts engaged to judge (1) whether
the prediction is reasonable, and (2) whether the prediction matches the actual future event.
Experimental Results Experimental results are shown in Table 6. From these, we can see that the powerful LLMs can produce highly reasonable event predictions. However, their predictions seldom align with the actual future, making them not directly helpful. These observations suggest that using LLMs for future event prediction is promising, but there remain topics to explore on how to build practical future event prediction systems with LLMs. For instance, using retrievalaugmented methods may help LLMs access more timely evidence when making future predictions. As a preliminary attempt, the experiments demonstrate how our all-in-one event understanding dataset can assist in conveniently building and evaluating event-based applications. We hope that future works can explore using the MAVEN series datasets to build diverse applications.

heading: 6 Related Work
text: Event Argument Extraction Datasets Since the early MUC datasets (Grishman and Sundheim, 1996), event argument extraction (EAE) as a part of event extraction has received widespread attention. To reduce task complexity and provide standardized extraction results, the ACE datasets (Doddington et al., 2004) are designed with a schema covering 33 event types, limiting event argument annotation to entities within the same sentence as the trigger. ACE 2005 (Walker et al., 2006) has been the most widely used dataset for a long time, and the practice of ACE has been broadly adopted. Rich ERE (Song et al., 2015) expands ACE schema to 38 event types and constructs the TAC KBP datasets (Ellis et al., 2014, 2015, 2016; Getman et al., 2017). MEE (Pouran Ben Veyseh et al., 2022) follows the ACE schema to build a multilingual dataset. With the advancement of NLP methods, some works break some of the constraints of ACE task definition to construct more practical datasets. RAMS (Ebner et al., 2020), WikiEvents (Li et al., 2021), and DocEE (Tong et al., 2022) extends the annotation scope to the whole documents. However, RAMS and DocEE only annotate one topic event per document, ignoring fine-grained events within documents. MAVEN (Wang et al., 2020) and GENEVA (Parekh et al., 2023) both construct high-coverage general event schemata with over 100 event types. MAVEN supports only event
detection. GENEVA extends event arguments to cover non-entity spans but focuses on testing the generalizability rather than developing practical EAE methods. Its data are repurposed from FrameNet (Baker et al., 1998) examples, which are individual sentences without document structure. MAVEN-ARG meticulously designs 612 unique argument roles for MAVEN schema and conducts large-scale exhaustive annotation, which annotates both entity and non-entity arguments for finegrained events at the document level.
Event Argument Extraction Methods Traditional EAE methods primarily involve (1) Classification-based methods (Chen et al., 2015a, 2017; Sha et al., 2018; Wadden et al., 2019; Wang et al., 2019; Lin et al., 2020; Wang et al., 2021b; Zhou and Mao, 2022): employing text encoders like CNN (Krizhevsky et al., 2012) and BERT (Devlin et al., 2019), followed by an information aggregator, such as dynamic multi-pooling mechanism (Chen et al., 2015a), to obtain role-specific representations for classification. (2) Sequence labeling methods (Nguyen et al., 2016; Yang and Mitchell, 2017; Nguyen et al., 2021; Peng et al., 2023c): mainly adopting the conditional random field (CRF) (Lafferty et al., 2001) as the output layer to model structured dependencies between different arguments. Recently, increasing attention has been paid to transforming EAE into a questionanswering task, transferring question-answering capabilities to boost EAE (Liu et al., 2020; Du and Cardie, 2020; Li et al., 2020; Ma et al., 2022; Lu et al., 2023; Nguyen et al., 2023). Additionally, some research focuses on using generation models to directly generate structured outputs containing events and their arguments (Lu et al., 2021; Li et al., 2021; Lu et al., 2022; Ren et al., 2023; You et al., 2022, 2023; Hsu et al., 2022, 2023; Zhang et al., 2023; Ren et al., 2023; Liu et al., 2023a; Ma et al., 2023), which has been becoming increasingly important with the advance of large language models.

heading: 7 Conclusion and Future Work
text: We introduce MAVEN-ARG, an event argument extraction dataset with comprehensive schema, large data scale, and exhaustive annotation. Experiments indicate that MAVEN-ARG is quite challenging for both fine-tuned EAE models and proprietary large language models. Together with MAVEN and MAVEN-ERE, MAVEN-ARG completes an all-inone dataset covering the entire process of event
understanding. An application case of future event prediction demonstrates how an all-in-one dataset can enable broad event-based applications. In the future, we will explore constructing multilingual resources under this framework and developing practical EAE methods with MAVEN-ARG.

heading: Acknowledgements
text: This work is supported by a grant from the Institute for Guo Qiang, Tsinghua University (2019GQB0003). We thank all the annotators for their efforts and the anonymous reviewers for their valuable comments.

heading: Limitations
text: (1) MAVEN-ARG currently includes only English corpus, which limits its potential applications and coverage for diverse linguistic phenomena. In future work, we will try to support more languages under our framework and we also encourage community efforts in developing multilingual event understanding benchmarks. (2) MAVEN-ARG, along with MAVEN (Wang et al., 2020) and MAVEN-ERE (Wang et al., 2022), exclusively supports mainstream event understanding tasks. However, these datasets do not cover more broad event-related tasks such as event factuality identification (Qian et al., 2019, 2022) and event salience identification (Liu et al., 2018). We encourage future explorations in building more challenging and diverse tasks and applications on top of MAVEN data. (3) While previous research has found that LLMs perform poorly on specification-heavy tasks (Peng et al., 2023c; Han et al., 2023; Li et al., 2023) including the EAE task, there is no in-depth exploration of effective LLM-based approaches addressing the EAE task in this paper. We leave the exploration of how to better leverage LLMs for EAE tasks in future work.

heading: Ethical Considerations
text: In this section, we discuss the ethical considerations of this work: (1) Intellectual property. The MAVEN dataset is released under the CC BY-SA 4.0 license3. The MAVEN-ERE is shared under GPLv34 license and the original Wikipedia corpus is shared under the CC BY-SA 3.0 license5.
3https://creativecommons.org/licenses/by-sa/ 4.0/
4https://www.gnu.org/licenses/gpl-3.0.html 5https://creativecommons.org/licenses/by-sa/
3.0/
The usage of these data in this work strictly adheres to the corresponding licenses and intended use. (2) Intended use. MAVEN-ARG is an event argument extraction dataset. Researchers and practitioners can utilize MAVEN-ARG to train and evaluate models for event argument extraction, thereby advancing the field of event understanding. (3) Potential risk control. MAVEN-ARG is constructed based on publicly available data. We believe that the underlying public data has been adequately desensitized and anonymized. The event argument annotation does not involve judgments about social issues and thus we believe MAVEN-ARG will not involve additional risks. To avoid unfair comparisons caused by mismatched evaluation implementations (Peng et al., 2023c) and potential cheating behaviors, the event argument annotations of MAVEN-ARG test set will not be publicly released. Instead, following previous works (Rajpurkar et al., 2016; Wang et al., 2020, 2022), we will maintain an online judgment system with a leaderboard, allowing users to submit predictions and obtain evaluation results. (4) Worker Treatments are discussed in appendix A.2.

heading: Appendices

heading: A Data Collection Details

heading: A.1 Annotation Instruction
text: As introduced in § 2.1, we create a detailed event schema for both defining the task and instructing the annotators. We present the annotation instructions for the event type Incident in Table 7, including its argument schema and annotation examples. The overall event schema is released along with the dataset. To support the highly customized annotation process designed for us, we developed a new online annotation platform. A screenshot for the annotation platform is shown in Figure 5 to help understand the annotation operations.

heading: A.2 Annotation Coordination
text: We employ annotators (including senior annotators and expert annotators) from multiple commercial data annotation companies. 61% of them are female and 39% of them are male. All annotators hold undergraduate degrees. Annotators for entity and event argument annotation have no overlap since we cooperated with different companies for the two annotation tasks. The experts involved in schema creation are invited by the authors through personal connections. All the workers are fairly paid with agreed salaries and workloads. All employment is under contract and in compliance with local regulations. The overall annotation cost, including annotating entities and event arguments as well as developing and maintaining annotation platforms, is about 85, 000 USD.

heading: B Additional Data Statistics

heading: B.1 Data Split Statistics
text: The detailed statistics of different data splits of MAVEN-ARG are shown in Table 8.

heading: B.2 Differences with Predecessors
text: MAVEN-ARG inherits the efforts of previous works MAVEN (Wang et al., 2020) and MAVENERE (Wang et al., 2022). MAVEN supports the event detection task by annotating event triggers and event types, along with a preliminary version of event coreferences. MAVEN-ERE supports the event relation extraction task by annotating event coreference, temporal, causal, and hierarchical relations. MAVEN-ARG completes the all-in-one event understanding dataset by adding the annotations
of event arguments, which supports the event argument extraction task. The construction of MAVENERE and MAVEN-ARG involves fixing or ignoring the erroneous and ambiguous annotations of event triggers and coreference clusters in MAVEN, which results in minor statistical differences shown in Table 9.

heading: C EAE Experimental Details

heading: C.1 Fine-tuning Implementation Details
text: Here we provide brief descriptions of the finetuning-based models involved in our experiments. (1) DMBERT (Wang et al., 2019) utilizes BERT (Devlin et al., 2019) as the text encoder and a dynamic multi-pooling mechanism (Chen et al., 2015b) on top of BERT to aggregate argumentspecific features and map them onto the distribution in the label space. (2) CLEVE (Wang et al., 2021b) is an event-oriented pre-trained language model, which is pre-trained using contrastive pre-training objectives on large-scale unsupervised data and their semantic structures. (3) BERT+CRF (Wang et al., 2020) is a sequence labeling model, which leverages BERT as the backbone and the conditional random field (CRF) (Lafferty et al., 2001) as the output layer to model the structural dependencies of predictions. (4) EEQA (Li et al., 2020) is a span prediction model, which formulates event extraction as a question-answering task and outputs start and end positions to indicate triggers and arguments. (5) Text2Event (Lu et al., 2021) is a conditional generation model, which proposes a sequence-to-structure paradigm and generates structured outputs containing triggers and corresponding arguments with constrained decoding. (6) PAIE (Ma et al., 2022) adopts prompt tuning (Lester et al., 2021) to train two span selectors for each argument role in the provided prompt and conduct joint optimization to find optimal rolespan assignments. We adopt the same backbones with their original papers for all EAE models in our experiments. We employ pipeline evaluation as suggested by Peng et al. (2023c). Specifically, for PAIE, we conduct EAE experiments based on the triggers predicted by CLEVE. For the other models, the EAE experiments are based on the triggers extracted by corresponding models.
We implement the EAE models using code from the official repositories of OmniEvent (Peng et al., 2023b), PAIE (Ma et al., 2022), and Text2Event (Lu et al., 2021). The numbers of pa-
[Incident] Accident, unfortunate event Event Arguments:
1. Participant : Entities involved in the accident (individuals, institutions, organizations, and even trains, ships, etc.). They can be the ones causing the accident or the ones affected by it. Similar to the combination of Agent and Patient in previous events, but due to the difficulty in distinguishing between Agent and Patient in accidents, they are uniformly labeled as Participants.

heading: Entity
text: rameters of the EAE models are shown in Table 3. All open-source models are downloaded from the HuggingFace Transformers community (Wolf et al., 2020). Each of our fine-tuning experiments is conducted only once, on Nvidia A100 GPUs, consuming approximately 800 GPU hours in total. The hyper-parameters of the model are set based on prior experience and references from previous papers (Lu et al., 2021; Ma et al., 2022; Peng et al., 2023b). All hyper-parameters are shown in Table 10.

heading: C.2 LLM Experimental Details
text: We access ChatGPT and GPT-4 through the official OpenAI interfaces, namely gpt-3.5-turbo and gpt-4, respectively. The API access period spans from October 1 to October 31, 2023. The decoding sampling temperature for both models is set to 0. An example of the prompt, input, output, and ground-truth of this experiment are presented in Table 11. Model outputs are automatically extracted and evaluated using the evaluation approach mentioned in § 4.1.

heading: D Event Prediction Experimental Details
text: The future event prediction experiments (§ 5) were conducted in October and November, 2023. We use OpenAI API endpoints gpt-3.5-turbo and gpt-4 for GPT-3.5 and GPT-4 experiments, specifically. To ensure the consistency among different runs, we set temperature=0.0. Detailed instructions and example input and output are shown in Table 12.

heading: E More Experimental Results
text: In this section, we present more experimental results of using different proportions of training data for training (appendix E.1) and results on entity and non-entity arguments (appendix E.2).

heading: E.1 Analysis on Data Size
text: The data volume of MAVEN-ARG significantly exceeds that of commonly used datasets. To examine
the benefits of increased data scale, we train models on training data of varying sizes and observe their performance on the original test set. The experimental results are shown in Figure 6, which demonstrates that more training data indeed enhances model performance and allows for a comprehensive comparison of different models. The extensive data of MAVEN-ARG make it feasible to train a large language model (LLM) for general event understanding, which we leave as future work. Table 13 shows the detailed experimental results, i.e., mention level, entity coreference level, and event coreference level.

heading: E.2 Entity and Non-Entity Arguments
text: Table 14 presents the overall results on entity and non-entity arguments of MAVEN-ARG. The nonentity arguments do not have coreferential relationship with each other and hence there is no entity coreference level evaluation for them.

heading: Model #Params Mention Level Entity Coref Level Event Coref LevelP R F1 EM P R F1 EM P R F1 EM Entity Argument

heading: Future Work
text: The extensive data of MAVEN-ARG make it feasible to train a large language model (LLM) for general event understanding, which we leave as future work. Table 13 shows the detailed experimental results, i. e. , mention level, entity coreference level, and event coreference level