Title: MEQA: A Benchmark for Multi-hop Event-centric Question Answering with Explanations
Benchmark: MEQA (Multi-hop Event-centric Question Answering)
Summary: This paper introduces MEQA, the first benchmark for multi-hop event-centric question answering, designed to evaluate reasoning over both events and entities. Using a novel semi-automatic strategy based on composing event structures from information extraction datasets, it created 2,243 challenging questions. Each question is paired with a multi-step QA-format explanation. Experiments show that MEQA is challenging for state-of-the-art models, including LLMs, which struggle with both answer accuracy and generating faithful explanations.
Contribution: Key contributions include: (1) Creating MEQA, the first benchmark targeting multi-hop event-centric QA. (2) Proposing a novel semi-automatic question/explanation generation method leveraging existing IE datasets. (3) Providing explanations in a QA-pair format for each question. (4) Introducing explanation evaluation metrics: completeness and logical consistency. (5) Benchmarking SOTA models (including LLMs and fine-tuned models) and analysing their performance, revealing significant challenges.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multi-hop reasoning; Event-centric question answering; Explanation generation; Faithfulness of explanations.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to perform multi-step reasoning by integrating information about both entities and events (including their relationships) from a given context to answer a complex question, and the ability to generate a faithful step-by-step explanation (reasoning chain) of this process.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: The benchmark aims to fill the gap of event-centric reasoning in multi-hop QA datasets, providing a more challenging evaluation scenario than entity-focused benchmarks. It also introduces metrics specifically for evaluating the generated reasoning explanations.
Task definition: Given a document and a multi-hop event-centric question, generate the correct answer and a step-by-step explanation in QA-pair format that outlines the reasoning process.
Task item definition: An instance contains a source document (from WikiEvents), a multi-hop question focusing on events, the gold answer, and a gold explanation structured as a sequence of single-hop question-answer pairs representing the reasoning chain.
Task definition detail: Questions involve diverse reasoning patterns like event relations, entity bridging, listing/counting, and comparison, often requiring 2-4 hops. Explanations are structured as QA chains. The dataset creation process includes steps to mitigate reasoning shortcuts.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: Started with event structures from WikiEvents. Composed event reasoning chains based on shared entities or relations. Filtered chains to avoid shortcuts. Generated synthetic QA pairs/explanations from chains using templates and schema info. Human annotators then curated (rephrased, corrected) these synthetic outputs and verified/completed answers.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task simulates deriving answers to complex event-related questions by chaining simpler inferences based on document content.
Dataset size: Test set: 287 questions.
Dataset size notes: Train set: 1,674 questions. Dev set: 282 questions.
Dataset metadata available: Yes
Dataset metadata detail: Source Document ID, Question Strategy Type, Explanation (Sequence of QA pairs), Event Structure information (from WikiEvents: triggers, arguments, roles, event types).
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Response format detail: Models need to output the final short answer. They are also evaluated on generating an explanation, which can be a sequence of QA pairs (CoT-QA) or freeform text (CoT-Freeform).
Metric definition: Exact Match (accuracy, F1, precision, recall), Explanation Completeness P/R/F1, Explanation Logical Consistency %
Metric definition detail: Answer: Precision, Recall, F1 (using HotpotQA script). Explanation: Completeness (P/R/F1 vs gold steps using semantic matching) and Logical Consistency (% of steps deemed consistent by an LLM verifier).
Metric aggregation: Simple Mean
Metric statistics: Precision, Recall, F1 score, Completeness (P/R/F1), Logical Consistency (%).
Metric subscores: Yes
Metric subscores detail: Performance broken down by question strategy type. Comparison of models with/without additional structured information.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Semi-automatic generation leverages existing event annotations. The reasoning shortcut problem is specifically addressed during chain filtering. Human annotators curate outputs. Explanation metrics (Completeness and logical Consistency) were validated with a human correlation study (0.693 and 0.601, respectively).
Taxonomy: Reasoning / 
New taxonomy: General Capability / Compositional Reasoning
Procedural notes: Uses WikiEvents dataset as source. Primarily uses ChatGPT (GPT-3.5-turbo-1106) for experiments. Details explanation of evaluation metrics and calculation (including prompts). Details crowd-sourcing setup (student workers, qualification test, payment). Discusses potential data leakage. Provides annotation interface examples.
Other notes: This benchmark uniquely tackles event-centric multi-hop reasoning and explicitly evaluates the generated explanations using novel metrics (completeness, logical consistency). The semi-automatic generation process leveraging IE datasets is a notable methodological contribution.