Title: Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Benchmark: SpatialEval
Summary: This paper introduces SpatialEval, a multimodal spatial reasoning dataset with four subtasks. SpatialEval tasks include map reading, maze navigation, locating objects on a grid, and QA from captioned images. Ablation studies show that LVLMs primarily use text over visual cues when provided.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Spatial reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: "Spatial reasoning, in particular, is fundamental to everyday human activities such as navigating environments, understanding maps, and manipulating objects. It encompasses skills that are crucial for both survival and higher-order cognition, including the ability to navigate through space, recognize patterns, and deduce relationships from spatial configurations" (2)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: A spatial reasoning question (naming objects at coordinates, counting right turns in a maze, ...) is presented as a multiple choice question. The context comes in one of three modalities: image only, caption only, and image with caption.
Task item definition: An image, a textual description / caption of the image, a multiple choice question, and a correct answer. Example could be an artificial grid maze and the question "How many right turns are there on the provided path from S to E?"
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Unclear
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: No
Sampling method: Unknown
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: mean with "error bars from 3 runs at temperature 0.2" (unsure if this is a standard error or just the range in scores) (17)
Metric subscores: Yes
Metric subscores detail: Spatial reasoning subtask (spatial map, maze navigation, spatial grid)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Spatial
New taxonomy: General Capability / Spatial Reasoning