Title: CAN LARGE LANGUAGE MODELS INFER  CAUSATION FROM CORRELATION?
Benchmark: CORR2CAUSE
Summary: Dataset looking at causal reasoning in LLMs. Produce synthetic "stories" about variables and how they correlate, ask an LLM to decide whether given variables are causally linked.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Causal inference, i.e., the ability to establish the correct causal relationships between variables or events, is fundamental to human intelligence.
Phenomenon defined by authors: Yes
Phenomenon definition: Fully defined mathematically in a full page of maths. Definition follows Directed graphical causal models -> terminology of confounders, colliders and midators, then introduce D-Seperation and Markov Property and Markov Eequivalence of graphs.  
Definition scope: Subset
Definition integrity: Authors' description is unclear
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: Given a set of N variables X = {X1, . . . , XN }, we have a statement s about all the correlations among the variables, and a hypothesis h describing the causal relation r between the pair of variables Xi and Xj. The task is to learn a function f : (s, h) 7→ v which maps the correlation statement s and the causal relation hypothesis h to their validity v ∈ {0, 1}, which takes the value 0 if this inference is invalid, and the value 1 if this inference is valid.

The statement is a natural language "story" about the variables.
Task item definition: Premise: Let’s consider three factors: eating junk food (A), obesity (C), and watching television (B). There is a correlation between eating junk food and obesity, and between watching television and obesity. However, eating junk food and watching television are independent from each other. Hypothesis: Eating junk food directly affects obesity.  Relation between the premise and hypothesis: The premise provides the necessary conditions for the hypothesis. It establishes the independent variables A (eating junk food) and B (watching television) and their correlations with obesity. Given that these are true, it supports the hypothesis that eating junk food directly affects obesity.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1162
Dataset size detail: Dataset is procedurally generated so whilst appears large lots of questions are structually very similar
Dataset size notes: train: 205734 validation: 1076
Dataset metadata available: Yes
Dataset metadata detail: The number of variables in the statement
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Broken down by number of variables in statement, also types of relationships between nodes.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning