Title: MAVEN-ARG: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation.
Benchmark: MAVEN-ARG
Summary: This paper introduces MAVEN-ARG, an augmentation of the MAVEN dataset with event argument annotations, creating the first large-scale, all-in-one resource for event detection, argument extraction (EAE), and relation extraction. MAVEN-ARG features a comprehensive schema (162 event types, 612 argument roles), substantial data scale (over 290k annotated arguments), and exhaustive annotations (document-level, entity & non-entity args). Experiments show MAVEN-ARG poses significant challenges for existing EAE models and LLMs.
Contribution: The primary contribution is the creation and release of MAVEN-ARG, the largest EAE dataset and the first dataset integrating ED, EAE, and ERE annotations. Other contributions include the development of a comprehensive event argument schema with detailed definitions, the exhaustive annotation methodology, benchmarking results showing the dataset's difficulty, and a demonstration of its utility for downstream tasks like future event prediction.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Event Argument Extraction (EAE); Event Understanding.
Phenomenon defined by authors: Yes
Phenomenon definition: Event Argument Extraction (EAE) is defined as the task of extracting event arguments (participants, attributes) for identified event occurrences (triggers) and classifying their specific semantic roles according to a predefined schema.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: A main motivation was to create a unified, large-scale dataset covering the full spectrum of event understanding (ED, EAE, ERE) to overcome limitations of previous fragmented datasets and enable end-to-end modeling and applications.
Task definition: Event Argument Extraction (EAE): For a given event trigger in a document, identify all text spans (both entity mentions and non-entity spans) that function as arguments for that event, and assign the correct argument role label to each identified span based on the event schema.
Task item definition: An event trigger (a word or phrase indicating an event) within a document, linked to its event type. Associated with this trigger are annotated arguments, each consisting of a text span within the document and an assigned argument role label. Entity arguments are linked via coreference IDs.
Task definition detail: The annotation scope is document-level (arguments can be anywhere in the document, not just the trigger's sentence), includes arguments for all fine-grained event mentions (not just a single topic event), and covers both entity and non-entity arguments.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task source detail: The dataset builds on the MAVEN dataset's Wikipedia text and event trigger/type annotations. The argument schema was manually created by experts, adapting concepts from FrameNet. Entity and argument annotations were collected through a three-phase human annotation process involving ordinary, senior, and expert annotators using a custom platform. 
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task focuses on extracting structured event information from Wikipedia articles, representing a common information extraction goal.
Dataset size: Test set: 857 documents, 18,112 events, 53,676 arguments.
Dataset size notes: Train set: 2,913 documents, 64,923 events, 190,479 arguments. Dev set: 710 documents, 15,556 events, 46,458 arguments.
Dataset metadata available: Yes
Dataset metadata detail: Event Type, Event Trigger Span, Argument Role, Argument Span, Entity Annotations (span, type, coreference cluster ID).
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: The standard output format involves identifying argument text spans and assigning a role label from the schema for each argument associated with an event trigger.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Bag-of-words F1 and Exact Match (EM) scores. These are calculated at three levels: Mention Level, Entity Coreference Level, and Event Coreference Level.
Metric aggregation: Simple Mean
Metric statistics: Precision, Recall, F1 score, Exact Match (EM)
Metric subscores: Yes
Metric subscores detail: Performance is analysed based on trigger-argument distance, separately for entity vs. non-entity arguments, and using varying proportions of training data.
Metric access: Outputs alone
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Schema developed over 3 years by experts with definitions and examples. Multi-phase annotation included checks by senior annotators and experts. Satisfactory inter-annotator agreement (Fleiss' kappa 68.6% for arguments) achieved. Dataset statistics confirm largest scale and comprehensive schema/annotation style compared to predecessors. Data analysis revealed diverse distributions and challenges like long-distance dependencies.
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction
Procedural notes: Dataset builds on MAVEN/MAVEN-ERE. Uses coarse-grained entity types from Few-NERD guidelines. Custom annotation platform developed. Test set annotations withheld for online leaderboard evaluation. Annotation cost ~85k USD. Detailed model hyperparameters and LLM prompts provided in appendices.
Other notes: MAVEN-ARG completes the MAVEN trilogy, enabling research on integrated event understanding. Its exhaustive annotation style (document-level, all events, entity/non-entity args) is a key differentiator. Error analysis pinpoints argument identification as the primary difficulty for models.