Title: BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information
Benchmark: BoardgameQA
Summary: BoardgameQA is a synthetic dataset designed to test language models’ ability to reason with contradictory and incomplete information. It models defeasible reasoning scenarios where conflicts must be resolved via predefined source preferences, combining multi-hop reasoning and missing knowledge challenges.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Defeasible reasoning with conflicting and incomplete information
Phenomenon defined by authors: Yes
Phenomenon definition: "The ability to reason and resolve conflicts between contradictory information sources based on predefined preferences, while incorporating missing background knowledge."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a set of facts, rules (some potentially contradictory), and preferences, determine whether a specific conclusion is proved, disproved, or unknown.
Task item definition: Each item includes a set of natural language facts, rules, preferences over rules, and a question requiring proof, disproof, or indeterminate status.
Task definition detail: Tasks simulate boardgame scenarios using synthetic generation, backward proof construction, and partial knowledge hiding.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: All examples are generated with controlled difficulty, conflict type, and knowledge requirements via a backward-chaining generation strategy.
Task ecology: Artificial task
Dataset size: ~1,000 examples per depth (depending on variation)
Dataset size notes: Default set: ~1,000 train, 500 val, 1,000 test
Dataset metadata available: Yes
Dataset metadata detail: Metadata includes depth (number of reasoning hops), conflict level, conflict type (Type1 vs Type2), knowledge incompleteness level, distractor counts.
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning