Title: SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Benchmark: SPARTQA (includes SPARTQA-HUMAN and SPARTQA-AUTO)
Summary: This paper introduces SPARTQA, a textual QA benchmark designed to evaluate spatial reasoning in language models, addressing limitations of prior datasets like bAbI Task 17. It includes SPARTQA-HUMAN, a set annotated by humans with more natural language and complex scenes (based on NLVR images), and SPARTQA-AUTO, a larger, automatically generated dataset using novel context-free grammar and spatial reasoning rules. Experiments show LMs perform poorly on SPARTQA-HUMAN but improve significantly after further pretraining on SPARTQA-AUTO.
Contribution: Key contributions include: (1) SPARTQA-HUMAN, a human-annotated benchmark for textual spatial reasoning exceeding the complexity of bAbI. (2) A novel automatic data generation method combining CFGs and spatial logic rules to create the large SPARTQA-AUTO dataset for distant supervision. (3) Demonstrating that pretraining on SPARTQA-AUTO significantly boosts LM performance on SPARTQA-HUMAN and generalizes to improve performance on external datasets (bAbI, boolQ). (4) Providing diverse question types (FR, FB, CO, YN) for detailed spatial reasoning analysis.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Spatial reasoning on natural language text.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to construct mental representations of spatial scenes based on natural language descriptions (stories) and use these representations along with spatial logic rules (e.g., transitivity, symmetry, inclusion/exclusion) to infer relationships and answer questions about object locations and configurations.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: To create a more challenging and realistic textual benchmark for spatial reasoning than bAbI Task 17. To explore the use of automatically generated, visually grounded data for improving LM spatial reasoning via distant supervision.
Task definition: Textual Spatial Reasoning QA: Given a story describing objects in blocks and their spatial relationships, answer a question probing this spatial configuration. Questions fall into four types: Find Relation (FR), Find Blocks (FB), Choose Object (CO), or Yes/No (YN).
Task item definition: An instance includes a textual story (describing a scene based on an NLVR image), a question (one of four types: FR, FB, CO, YN), and the corresponding correct answer (selected from candidates or Yes/No/DK).
Task definition detail: Scenes have objects with attributes in blocks. Stories provide partial descriptions. Reasoning requires applying spatial rules (transitivity, symmetry etc.). Includes "Don't Know" (DK) answers for YN questions under open-world assumption.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: SPARTQA-HUMAN stories and QA pairs written by two student volunteers based on visually grounded scenes (from NLVR images, potentially rearranged). SPARTQA-AUTO stories generated using Context-Free Grammars (CFGs); QA pairs generated programmatically using spatial logic rules applied to the underlying scene graph derived from NLVR images.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Models understanding potentially incomplete textual descriptions of spatial layouts to answer inferential questions, relevant to real-world language understanding.
Dataset size: Test sets: SPARTQA-HUMAN = 510 QA pairs; SPARTQA-AUTO Seen Test = 15,074 QA pairs; SPARTQA-AUTO Unseen Test = 15,087 QA pairs.
Dataset size notes: SPARTQA-HUMAN Train = 616 QA pairs. SPARTQA-AUTO Train = 93,673 QA pairs; Dev = 15,023 QA pairs.
Dataset metadata available: Yes
Dataset metadata detail: Question Type (FR, FB, CO, YN), Underlying Scene Graph (for SPARTQA-AUTO), SpRL annotations (Trajector, Landmark, Spatial Indicator).
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Response format detail: Classification task. Models select the correct answer from a list of candidates (for FR, FB, CO) or predict one of three labels (Yes/No/DK for YN).
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Accuracy (percentage of correctly answered questions). F1 score reported for YN type analysis due to potential class imbalance.
Metric aggregation: Simple Mean
Metric statistics: Accuracy (%), F1 Score (%)
Metric subscores: Yes
Metric subscores detail: Performance reported per question type (FB, FR, CO, YN). Comparisons between Seen vs Unseen test sets (AUTO) and HUMAN vs AUTO datasets. Consistency and contrast set evaluations.
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Built on complex NLVR scenes. SPARTQA-HUMAN created by annotators focusing on natural language and reasoning. SPARTQA-AUTO generated programmatically ensuring spatial consistency via grounding and rules. Human performance benchmarks provided. Consistency/contrast sets test robustness. Extrinsic evaluation on bAbI/boolQ shows positive transfer.
Taxonomy: Reasoning / Spatial
New taxonomy: General Capability / Spatial Reasoning
Procedural notes: Provides details on CFG design and question generation modules/rules. Describes model architectures and training parameters. Includes SpRL annotations as supplementary data. 
Other notes: The paper demonstrates a successful approach to creating large-scale distant supervision data (SPARTQA-AUTO) for a complex reasoning task (spatial reasoning) by leveraging grounded visual information and symbolic rules. This strategy effectively improves language model performance on human-created spatial reasoning tests (SPARTQA-HUMAN).