Title: An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models
Benchmark: Spatial-MM
Summary: This paper introduces the Spatial-MM benchmark to evaluate the spatial reasoning capabilities of LMMs. The authors demonstrate that incorporating structured information like bounding boxes and scene graphs improves performance, while also revealing limitations in human-perspective reasoning and the ineffectiveness of chain-of-thought prompting for complex spatial tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: spatial reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: (1) What spatial relations are missed by models, and why it happen? (2) How can additional symbolic visual information, such as bounding boxes or scene graphs, improve the performance of LMMs? Which of these symbolic information is more useful, and how can they be integrated in the reasoning process effectively? (3) How does the questions complexity affect LMMs in handling spatial relations? (4) How does the reasoning path of LMMs behave when they fail to answer a multihop question? Is the failure due to incorrect spatial reasoning or non-spatial reasoning?
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: Take an image and multiple-choice question, provide an answer
Task item definition: <image> Q: Where is the bicycle from the woman’s perspective? A. Front, B. Behind, C. Right, D. Left
Task source: Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Spatial-Obj Subset is annotated by human, Spatial-CoT is generated by GPT-4o
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 2310
Dataset metadata available: Yes
Dataset metadata detail: whether require spatial reasoning
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: They present results for two subsets: Spatial-Obj, Spatial-CoT
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They include annotators for dataset verification
Taxonomy: Reasoning / Spatial
New taxonomy: General Capability / Spatial Reasoning