Title: CRAB: Assessing the Strength of Causal Relationships Between Real-World Events.
Benchmark: CRAB (Causal Reasoning Assessment Benchmark)
Summary: This paper introduces CRAB, a new benchmark to evaluate the causal reasoning abilities of language models on real-world events presented in news narratives. It contains approximately 2,700 event pairs derived from 20 news stories, annotated with fine-grained causality scores (0-100) based on context. Experiments using large language models reveal poor performance, particularly when reasoning about complex causal structures (like causal frames and chains) versus simple ones.
Contribution: The main contributions are: (1) The creation of the CRAB benchmark with fine-grained, contextual causality annotations for real-world event pairs. (2) A data construction pipeline leveraging causal principles and involving LLMs for event extraction followed by human annotation and expert validation. (3) Benchmarking state-of-the-art LLMs on causal reasoning tasks derived from CRAB. (4) Analysis of model performance based on causal structures (frames and chains) and context (in-document vs. cross-document).
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Causal reasoning between real-world events; Understanding actual causality in narratives.
Phenomenon defined by authors: Yes
Phenomenon definition: The paper focuses on assessing the understanding of 'actual causality' - the causal relationship between specific, real-world events as perceived by humans based on context. This is operationalized by collecting graded (0-100) human judgments about the causal strength between pairs of events extracted from news narratives.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: The benchmark aims to address limitations in existing causal reasoning datasets by focusing on real-world events, contextual dependence (including multi-document context), and graded (non-binary) causality judgments. It draws on principles from cognitive science and actual causality research.
Task definition: To assess the strength of the causal relationship between a pair of real-world events, given the context from news articles. This involves predicting a scalar score (0-100) or classifying the relationship into discrete levels (e.g., High/Medium/Low/No, or Binary Yes/No), potentially within specific structural contexts like causal frames or chains.
Task item definition: A pair of event descriptions, the source news document(s) providing context, and a human-annotated causality score (0-100) indicating the perceived causal strength from the first to the second event. Event pairs are also grouped into causal frames and chains.
Task definition detail: The benchmark includes event pairs where both events originate from the same document ('in-doc') and pairs where events come from different documents ('cross-doc'). It uses a continuous 0-100 score for annotation, often mapped to 4 classes for evaluation. Events are based on real news stories from the past decade.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: News articles related to 20 selected stories were scraped (Google News API). Events were extracted using GPT-3 prompts, followed by expert filtering and validation. Timelines were manually constructed. Pairwise causality scores were annotated by AMT workers (7 per pair) and validated/adjusted by experts for ambiguous cases.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The task uses real events reported in news media and requires reasoning about their causal connections based on the provided context, mirroring how humans interpret such narratives.
Dataset size: 2,730 event pairs in total. Test set size is not applicable in the main zero-shot evaluation setup.
Dataset metadata available: Yes
Dataset metadata detail: Event pair descriptions, Source document(s), Story identifier, Temporal order (implicit in timeline), Pairwise causality score (0-100), Causality class (derived), Causal frame structure type, Causal chain structure type.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Response format detail: Depending on the specific task setup, models output a scalar score (0-100), a class label (e.g., High, Medium, Low, No), a binary label (Yes/No), or a choice from multiple options (MCQ).
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Macro F1 score (for binary and 4-class classification), Exact Match (EM) score (for causal structure analysis).
Metric aggregation: Simple Mean
Metric statistics: Macro F1 score, Exact Match (EM)
Metric subscores: Yes
Metric subscores detail: Performance broken down by: in-document vs. cross-document pairs; pre- vs. post-Jan 2022 events (model knowledge cutoff); causal frame type; causal chain type; individual causality classes (High/Medium/Low/No).
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Dataset creation motivated by causal principles. Event extraction pipeline included expert validation. Causality annotations used multiple AMT workers plus expert review for ambiguous cases. Inter-rater agreement was measured (Krippendorff's alpha), showing reasonable agreement for extreme classes and among experts. Analysis based on theoretically grounded causal frames/chains.
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Document sources from Google News API via SerpApi. Event extraction used GPT-3 (text-davinci-003). Annotation via AMT with specific qualification/payment details. Detailed prompts provided in appendix. Discussion of limitations and ethics provided. Fine-tuning experiments detailed in appendix.
Other notes: A key aspect is the focus on graded causal strength (0-100 score) rather than just binary causality. The analysis highlighting poorer performance on complex causal structures (e.g., mixed frames, colliders) and cross-document pairs is significant. The study also attempts to disentangle reasoning ability from memorization by analyzing performance based on event dates relative to model training cutoffs.