Title: MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
Benchmark: MuSR
Summary: The paper introduces MuSR, a long‑form natural‑language problems that require multistep “soft” reasoning combining commonsense, deductive and theory‑of‑mind inference. Generated through a neurosymbolic pipeline, these long-form problems reveal critical gaps in current models' reasoning capabilities despite being consistently solvable by humans.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: reasoning, commonsense knowledge, multi-step deductive
Phenomenon defined by authors: Yes
Phenomenon definition: “Multistep soft reasoning” = reasoning that integrates narrative facts with implicit commonsense rules over several inference steps to reach an answer. 

First, a number of prior benchmarks do not have natural text. Others do not
blend commonsense and multistep reasoning. Finally, we want a dataset that contains ground-truth intermediate structure and which is not solvable with rules. (section2)
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a narrative and a question (multiple‑choice), the model is expected to choose the correct answer by reconstructing hidden reasoning chains.
Task item definition: a narrative (~1 000 words), a question, and answer options 
Task definition detail: Each item links to an underlying reasoning tree of ~10 steps and 6‑9 implicit commonsense facts.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Instances produced by GPT‑4 via a neurosymbolic tree‑construction and “chaptering” pipeline.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Synthetic but linguistically natural expressions, not a real‑world workflow or scenarios.
Dataset size: 756
Dataset metadata available: Yes
Dataset metadata detail: logic depth, number of commonsense facts, ground-truth intermediate facts
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: mean + standard deviatin, significance test. proportion
Metric subscores: Yes
Metric subscores detail: domain types, prompt variants, model baselines, prompt conditions
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: human/rule baselines, ablations, dataset difficulties
Taxonomy: Reasoning / Commonsense
New taxonomy: General Capability / Commonsense Reasoning