Title: WikiWhy: Answering and Explaining Cause-and-Effect Questions
Benchmark: WikiWhy
Summary: This paper introduces WikiWhy, a question-answering dataset focused on evaluating LLM reasoning by requiring models to answer "why" questions and provide explicit natural language rationales explaining cause-and-effect relationships. Grounded in Wikipedia facts across 11 diverse topics, the dataset contains over 9,000 question-answer-rationale triples. Experiments with GPT-3 baselines show low correctness (38.7% human eval) for end-to-end answer and explanation generation, indicating significant room for improvement.
Contribution: Proposes the task of explaining cause-effect relations via natural language rationales as a benchmark for LLM reasoning. Creates WikiWhy, a large dataset (>9k examples) for this task, grounded in Wikipedia and spanning 11 topics. Establishes baseline results using GPT-2 and GPT-3, highlighting the task's difficulty. Introduces and validates automatic evaluation metrics for generated explanations using human correlation studies.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Reasoning capability; Explanation generation; Cause-and-effect understanding; Commonsense reasoning.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to bridge the gap between a stated cause and its effect by generating a coherent natural language rationale (a set or sequence of supporting statements) that demonstrates an understanding of the underlying mechanism, often relying on implicit commonsense knowledge.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Aims to evaluate implicit commonsense knowledge within LLMs, which is often needed to explain why a cause leads to an effect, moving beyond factoid retrieval. Employs a generative task format to test recall rather than recognition. Covers a broad range of topics (11) for generality.
Task definition: Given either a cause-effect pair (EO task) or a "why" question about an effect (A&E task), generate a natural language rationale (set or sequence of sentences) explaining how the cause leads to the effect. For the A&E task, also generate the answer (which is the cause).
Task item definition: Each entry includes: Cause text, Effect text, "Why" Question text, Answer text (same as Cause), Rationale (one or more sentences), Source Wikipedia passage, Source Article URL, Topic Category.
Task definition detail: Rationales average 1.5 steps/sentences, but can be longer (36% have 2+ steps). Data derived from Wikipedia "Good Articles". Questions/cause/effect intended to be understandable without the original passage. Two explanation structures noted: sequential chain and rationale set.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: Data originated from Wikipedia "Good Articles". Passages are filtered using causal keywords. MTurk workers performed cause-effect extraction and QA synthesis in Stage 1, and rationale generation in Stage 2. Multi-stage validation and quality control were applied.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task requires generating human-like explanations for causal links found in encyclopedia text, testing a fundamental reasoning skill.
Dataset size: Total: 9,406 examples. Test set: 1,005 examples.
Dataset size notes: Train set: 7,397 examples. Dev set: 1,004 examples. Total rationale elements: 14,238.
Dataset metadata available: Yes
Dataset metadata detail: Cause text, Effect text, Source Passage, Source Article URL, Topic Category (11 types).
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Response format detail: Models generate a natural language explanation (rationale) consisting of one or more sentences. In the A&E task, they also output the short answer (the cause).
Metric definition: n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), Unordered/Ordered BERT-F1 using DeBERTa-based BERTScore
Metric definition detail: Automatic: Unordered/Ordered BERT-F1 (using DeBERTa-xlarge-mnli with threshold 0.64), ROUGE-L F1. Human Evaluation: Binary ratings for Correctness, Concision, Fluency, Validity; Win/Tie/Lose comparison vs gold rationale.
Metric aggregation: Simple Mean
Metric statistics: BERT-F1, ROUGE-L F1, Human Judgement Proportions (%), Pearson Correlation (r) for metric validation.
Metric subscores: Yes
Metric subscores detail: Results were analysed separately for Task 2 (EO) and Task 3 (A&E, with single-model vs. pipeline variants). Comparison across models (GPT-2 vs GPT-3) and decoding temperatures. Performance reported per topic category.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Used Wikipedia "Good Articles". Multi-stage MTurk process with worker qualification, validation, and manual review by panellists. Analysed rationale length and structure. Validated automatic metrics via correlation with human judgments (r=0.82 between human similarity/correctness; r=0.35 between ordered F1/human similarity).
Taxonomy: Reasoning / Commonsense
New taxonomy: General Capability / Commonsense Reasoning
Procedural notes: Details crowdsourcing setup (MTurk, worker quals, pay rate, interfaces). Fine-tuning details for GPT-2 provided. GPT-3 experiments use DaVinci-002 via API. Details evaluation metrics including BERTScore setup [cite: 136-146, 157]. Human evaluation criteria detailed.
Other notes: Unique focus on generating natural language explanations for "why" questions about cause-effect pairs derived from text, aiming to probe commonsense reasoning. Fully generative task formulation chosen deliberately. The correlation between human ratings of similarity and correctness suggests reference-based metrics are meaningful proxies for explanation quality.