Title: FOLIO: Natural Language Reasoning with First-Order Logic
Benchmark: FOLIO
Summary: Benchmark of logical deduction puzzles. Model is given a list of statements in natural language "The Turkey is not an Eastern Wild Turkey" then has to decide which hypothesises are true or false.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Complex logical reasoning
Phenomenon defined by authors: No
Phenomenon definition: They formally define the task, but do not define "complex logical reasoning". They discuss other logical reason benchmarks, and discuss how their benchmark is "more complex" than those.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Each natural language (NL) story S in FOLIO consists of n premises: P = {p1, p2, ..., pn} and m  conclusions: H = {h1, h2, ..., hm}. All NL stories are annotated with parallel FOL stories SF , which are sets of FOL formulas consisting of n premises P F = {pf1, pf2, ..., pfn} and m conclusions HF = {hf1, hf2, ..., hfm}. pfi and hfi are  logically and semantically similar to pi and hi, respectively. Given P and H, the goal is to determine the truth values of the conclusions: "True", "False" or "Unknown", based on FOL reasoning.
Task item definition: NL premises
There are six types of wild turkeys: Eastern wild turkey, Osceola wild turkey, Gould’s wild turkey, Merriam’s wild turkey, Rio Grande wild turkey, and the Ocellated wild turkey.
Tom is not an Eastern wild turkey.
Tom is not an Osceola wild turkey.
Tom is also not a Gould’s wild turkey.
Tom is neither a Merriam’s wild turkey, nor a Rio Grande wild turkey.
Tom is a wild turkey.

NL Conclusions → Labels
A. Tom is an Ocellated wild turkey. → True
B. Tom is an Eastern wild turkey. → False
C. Joey is a wild turkey. → Unknown
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1435
Dataset metadata available: Yes
Dataset metadata detail: First Order Logic translations of the questions.
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Many ablations: where the data was sourced from, how many predicates etc
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning