Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Authors: Jiayu Wang
Yifei Ming
Zhenmei Shi
Vibhav Vineet
Xin Wang
Yixuan Li
Neel Joshi
Venue: NeurIPS 2024 poster (2024)
Keywords: Spatial Reasoning
Evaluation
Vision-language Models
Language Models
Primary area: evaluation

Abstract: Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning—a fundamental component of human cognition—remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence. Our code is available at https://github.com/jiayuww/SpatialEval.

Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Jiayu Wang' Yifei Ming""
Zhenmei Shi
Vibhav Vineet
Xin Wang Yixuan Li
Neel Joshi
University of Wisconsin-Madison Microsoft Research
"Salefforce Al Research
fmilawang, zhmeishi, sharonli]@cs. shin wttpa//iiissssseseeeeeeeemeeeeee wisc.
yifei sinel.mmnnddslessoreee.. com
fvibhav.vineet, vineet, wanxin, www.:neeeeemeoooon nem
com
Abstract
Large language models (LLMs) and vision-language models (VLMs) have demon- strated remarkable performance across wide range of tasks and domains. Despite this promise, spatial understanding and reasoning-a fundamental component of human ceennttion-rmmainn under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (I) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perfoor compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual informa- tion if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal mod- els to improve spatial intelligence and further close the gap with human intelligence. Our code is available at https: //g //gi thub. com/
1 Introduction
The recent breakthroughs in foundation models have had transformative effect on research and industry, and we have seen these models rapidly integrated into products and new businesses that are shaping people's lives for the better. This sea change was initially driven by large language models (LLMs), which have shown at times near unbelievable, human-level performance across a wide range of tasks. Over the past year, many of these models have been extended to handle images in addition to text, leading to a significant increase in vision- linguula models (VLMs), especially multimodal large language models (MLLMs), which demonstrate groundbreaking performance in image-related tasks as in the text domain.
However, the reality is not quite as rosy as advertised. While these models have been instrumental in advancing the state-of-the-are in complex reasoning tasks such as common sense reasoning, mathematical problem solving, and scientific question answering [17, 30, 41, 72, 75), they have not been as effective for a number of problem domains. In particular, as we will show in this paper, they have limited performance on tasks that require detailed visual understanding and reasoning of images.
"The work completed in part during Yifei Ming's internship Microsoft Research, well PhD thesis research at UW-Madison.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Visual understanding and reasoning an intrinsic part of human perception and cognitive abil- ity- have been largely under-explored when it comes to LLMs and VLMs. In fact, it often argued that the visual sense is the dominant sense in people, yet when it comes to current models, it seems quite secondary. Spatial reasoning, in particular, is fundamental to everyday human activities such as navigating environments, understanding maps, and manipulating objects. It encompasses skills that are crucial for both survival and higher-order cognition, including the ability to navigate through space, recognize patterns, and deduce relationships from spatial configurations.
In this paper, we propose SpatialEval, a novel benchmark containing four tasks (Spatial-Map. Maze-Nav. Spatial- and Spatial -Real, Section 2.1) to explore the performance of LLMs and VLMs on diverse aspects of spatial reasoning. including relationship. navigation, position understanding, and object counting. Humans excel at such tasks, making them essential capabilities for intelligent systems to emulate for safe and effective deployment in the real world.
Our dataset, however, is constructed with a key twist - eech problem in our benchmark has an image and a text representation that is sufficient for answering each spatial understanding question. We denote the use of these sources as VQA, which is the standard task of visual-question answering that consists of a vision-only input and a question, TQA, text-only input and question, and VTQA, combination of the previous with vision and text input.
We conduct systematic and comprehensive evaluation of a wide range of open-source and proprictary LLMs and VLMs. We perform in-depth analysis and unveil several novel and surprising results that challenge the current understanding of how these models process spatial information:
Spatial reasoning remains challenging: VLMs frequently struggle with spatial reasoning tasks, with some competitive models performing worse than random guessing.
Visual vs. textual inputs: Without detailed textual descriptions, multimodal models rarely surpass the performance of their LLM backbones when relying solely on visual inputs. This underscores the critical role of text in enhancing model performance on spatial reasoning.
Reduced reliance on visual information: When both textual and visual inputs are provided, multimodal language models tend to rely less on the visual component if sufficient textual clues are available.
Textual performance of VLMs: VLMs often outperform their LLM counterparts with text-only inputs, indicating that the language model backbones within VLMs benefit from multimodal training, despite a lack of similar benefits from the visual components.
We surmise the limitations in VLMs' spatial understanding stem from the overly simplistic handling of visual information in current architectures and training pipelines. We believe the contributions of this paper will drive changes in model design, accelerating improvements that could unlock more robust spatial reasoning capabilities, and help bridge the gap toward human- intelligence.
2 Dataset and Task Construction
2.1 Dataset Setup
To evaluate the spatial reasoning abilities of LLMs and VLMs, we construct four diverse tasks including spatial relationships, navigation, position understanding, and counting. To systematically study the impact of modality, design three types of input formats for each task: (1) TQA (Text- only): the input is purely textual and contains all necessary information for a person to answer the questions. (2) VQA (Vision-only): the input consists solely of an image, which provides sufficient details for person to easily answer, a format also referred to as Visual Question Answering (VQA) in the literature. (3) VTQA (Vision-text): the input includes both an image and its textual representation with detailed descriptions, rendering the information in both modalities redundant. We evaluate LLMs using Text-only inputs and VLMs using Text-only, Vision-only, and Vision-text inputs on the same set of questions (TTble 1). The synthetic tasks are curated based on the following key guidelines: (1) Avoidance of data leakage- since LLMs are pre-trained on web-scale data, it crucial to ensure that the test data have not been seen during training: (2) saafguurability-biine configurable allows for controlled experiments and extends easily to additional tasks; (3) scalability- the ability to scale the number of test samples enhances the statistical significance of experimental results.

---

TQA (Text-only)
Spatial-Map
Consider map
objects:
Whale's Watches is in the map. Brews Brothers Pub to the Southeast Whale's Watches Himalayan Hot Springs is to the Southeast of Whale's Watches, Gale Gifts to the Northeast of Unicorn Umbrellas Gale Gifts the thwest Himalayaan Hot Springs.
While's Watches
VQA (Vision-only)
<ing- figure represents with multiple objects Each object is associated with figure
VTQA (Vision-text)
Himsllyan Hoat Springs
<img- figure represents map
objects object associated
name as shown in the figure. The same figure can be described as follows:
Gale Gitss
Whale's Watches is the map. Brews Brothers Pub to the Southeast of Whale's Watches. Southeast
Unicorn Umbrellx
Hiimalayan Springs
Drayorfy
Southwest Himalayan Hot Springs.
which direction Whales Watches relative Dragonfly Drones?
Questions
Which object the Southwest Gala Gifts? Umbrellas
Draganfly Drones
Himalayan Hot Springs
Brews
Q How many objects are the Southwest of Himalayan Hot Springs? A.3
8.4
C.1
0.0
Figure I: Illustration of the Spatial-May task, which simulates map with multiple locations. To investigate the impact of modality, consider three input formats: Text-only, Vision-only, and Vision-text. We evaluate language models (w. TQA input) and vision-language models (w. VQA and VTQA inputs) the same set of questions.
Spatial-Map. Understanding the spatial relationships among objects on a map a fundamental aspect of human cognitive abilities. To simulate this environment, create map-like dataset termed Spatial-Map with K objects, where K configurable. Each object is associated with unique location name, such as Unicorn Umbrellas and Gale Gifts. To study the impact of modality, the textual representation each input consists of pairwise relations such as Brews Brothers Pub is to the Southeast of Whale's Watches. An example with K is shown in Figure 1, with Text-only, Vision-only, and Vision-text inputs. These questions include asking about the spatial relationships between locations and the number of objects that meet specific spatial criteria.
Maze-Nav. Navigation through complex spaces essential for intelligent systems. To evaluate such abilities, have developed maze-like like dataset named Maze Visually, each sample can be
TQA (Text-only)
Maze represented ASCII code (LHS) where the symbols have following meanings: represents wails
Maze-Nav
((space) represents navigable path within the maze, ... A right turn is defined as change movement direction that is degrees clackwise relative the provious direction. A laft turn dafined as change direction
impassable
degr anticlaciewise
direction.
VQA (Vision-only)
<img- The figure represents Maze, where the colored blocks have the following meanings: Black blocks represent walls that impassable barrirs White blocks represent navigable paths within maze, necessarily the path to
right defined as change in movement direction 90 degrees clackwise relative to the previous direction..
VTQA (Vision-text)
<img- The figure represents Maze, where the colored blocks have the following meanings: walls that iippssaalle barrers. Thhe Maze can be represented
Black blocks code(L right turn defined change movement direction degrees clackwise relative the previous direction A laft turn defined as change movement direction that 90 degres anticloowiss relative to the previous direction.
Q: How many right turns are there the provided path Sto
Questions
How total turns are there the provided path from
Q: is the exit directly below the starting point, with no horizontal displacement?
A. Yes
a No
Figure 2: Illustration of the Maze-Nav task, which evaluates the model's ability to navigate from the starting point (S) to the exit (E).
3

---

represented colored blocks where different colors signify distinct elements: a green block marks the starting point (S), red block indicates the exit (E), black blocks represent impassable walls, white blocks denote navigable paths, and blue blocks trace the path from S to E. The objective is to navigate from (up, down, left, right). Alternatively, each input can be depicted in textual format using ASCII code. An example is illustrated in Figure 2, featuring Text-only. Vision-only, and Vision-text inputs. We construct this task based on an open-sourced library [26]. The questions asked include counting the number turns from S to E and determining the spatial relationship between S and E. While such questions easy for humans, for modern multimodal language models.
following the blue path, with movement permitted in the four cardinal directions
will show in Section 3 that they still pose significant challenges
TQA (Text-only) elepannt cat
Spatial-Grid
columns) containing various animals, where eath Ixt cell considered block and each block contains en animal from I'aat, Vog'' elephant. 'giraffe', rrabbit]. The distribution of animals
Consider 5x5
girratee |elephant cat
rabbit dog
gigrafee cat
dog
rabbit
cat
lelephant rabbit
dog
Ielephant
giraffe
cat
cat
follows (LHS)
rabbit
dog
giraffe rabbic
raabbit
VQA
OVision-only)
<lmg- The figure represents 5u5 grid and columnd) containing square considered block black contains from ('rat, 'dog', eelephantt giraffe; rabbitt..
VTQA (Vision-text)
<lmg- The figure represents a 5xS grid (s rows and coluimns) containing various animals, where each lxt
elephant cat rabbii celephant
Igitaffe eleepant cat Idog
5
giraffe cat
considered block each black animal from l'cat, 'dog wlephant, 'giraffee, Yabbit]] The same figure can be represented in textuul form as follows (LH(5)
eelephant rabbit
dog
giraffe cat cat
rabbit
dog
giraffe rabbit rabbii
many blocks rabbit?
Questions
B.2
0.6
@ ihat
black top-left corner (first row, column) grid? elephant
A. rabbit
B. giraffe
C. cat
Q: What the animal of black located at the first row, second calumn of the grid? rabbit
c.
girafff
dog
Figure Illustration of the Spatial-Grid task, which evaluates the model's spatial reasoning ability in rigid grid structure.
Spatial-Grid. To investigate spatial understanding within structured environments, we introduce grid-like dataset named Spatial-Grid, contrasting with the Spatial where objects are positioned arbitrarily. Visually, each imput consists of grid of cells, each containing image (e.g.. rabbit). An example illustrated Figure Alternatively, this grid can also be represented in purely textual format; for instance, the first row might be described as: elephant cat giraffe elephant cat. The evaluations focus on tasks such as counting specific objects (e.g., rabbits) and identifying the object located at specific coordinate in the grid (e.g., first row, second column).
Spatial-Real. To extend the evaluation of spatial reasoning beyond synthetic environments, introduce Spatial-Real, a task built on the Densely Captioned Images (DCI) dataset [67]. where each image has detailed caption with more than 1,000 words average. As DCI does contain questions, we curate multiple-choie questions regarding spatial reasoning (object counting, relation, and position understanding) and annotate the answers. An example is shown in Figure We provide detailed analysis on Spatial Reea in Appendix E.
2.2 Models
We consider a wide range of competitive open-source language models with different scales, including
Phi2-2.7B [35], the LLaMA family (LLaMA--27B.. LLaMA-2-13B, and LLaMA-3-8B) [65], Mistral-
7B [27], the Vicuna family (Vicuna-7B-155 and Micuna-13B-11 [11], and Heos--ermes--2- Y--33B.. For multimodal language models, we consider the Bunny family (Bunny-Phi-2--5 Phi-22SigLLIP, Bunny- Phi-1.5-SigL.n Bunny-Phi-2-EVA, and Bunny-Phi-1.5-- 1.5-EA) [22], CogVLM [69]. CogAgent [23],
InstructBIIIP family (InstructBILPP LI-rvccB..P-Viccua--- and
[13], and LLaVA LLaVA- V--1.66 1-----...Vv-a-111- LLa VA-1.6 1.6 Vicuna-13P and LLaVA-1.6
family

---

Spatial-Rea Real
Textual Description
the image tall tree directly right the orange truck hed and ovar the tree over the back end of the brown truck bed. Open overcast sky over everything the ground. Right group of tree leaves ever the orange truck bed between
The following
are suspending it. Window area of the driier''s side door of the white truck the left the brown truck bed... crane holding up the orange truck bed to the right another crane..
Please answer the fallowing question the provided information.
Q: many tranes are lifting the orange truck bed? Available options: Two t. Thraa
D.ur
Illustration of the Spatial-Resd task, which is built on real images with long captions, featuring detailed descriptions averaging over 1,000 words per image.
Figure 4:
34B) [38]. We also evaluate the proprietary models: Open AI's GPT-4V, GPT-4o, GPT-4, Google Gemini Pro 1.0, and Anthropic Claude 3 Opus.
Evaluation. As each question contains four options, we use accuracy as the main evaluation metric. The same user prompt is appended the end of each question: First, provide concise answer in one sentence. Then, elaborate on the reasoning behind your answer in a detailed, step-by-step explanation. For each model, we adopt the default configurations and decoding strategies, e.g. argmax for deterministic decoding and top-p for non-deterministic decoding. For non-deterministic decoding, the results are averaged over three independent due to their limited availability and increased compute time and cost, we only perform one run. We summarize the terminologies regarding input modalities for LLMs and VLMs Table 1:
for open-source models. For proprietary models,
Model
Input Modality
Term
Description
Text-only input that includes all necessary infor- mation to answer questions without visual context.
LLM
Text-only
TQA (LLM)
Text-only input as TQA (LLM) but applied to VLMs (e.g., the LLaVA family).
VLM
Text-only
TQA (VLM)
Input only includes an image without correspond-
VLM
Vision-only
VQA
ing textual description. Input includes both an image and its textual de- scription.
VLM
Vision-text
VTQA
Table I: Terms regarding input modalities for LLMs and VLMs.
We describe the Text-only, Vision-only, and Vision-texx input modalities based on how we feed the image information to the models. Vision-only input means the image is fed directly to the models without textual deseription, while all questions are presented in text.
3 Main Results and Analysis
Spatial reasoning remains surprisingly challenging. The evaluation results on open-source models on Spatial-May Maze-Nav, and Spatial-Grid are shown in Figure 5. For each task, the reported models, we choose the Vision-only
accuracy is averaged over all questions. For vision-languse input format, commonly used Visual Question Answering (VQA). We use dashed red line in each figure to indicate the expected accuracy if answering by random guessing. Our findings reveal several notable insights: (1) Vision-only inputs: Despite the simplicity of these tasks for humans, most competitive multimodal models perform at levels similar to barely above random guessing.
5

---

Mpatial-Map
Maze-Nav
Spatial-Grid
L--MA-2-78
Mistral-7B
Nous-Hermes-2-133gg
Vicuna-138-1.5
LLaV4-1.6-34B
LLa LaaWe116-Vicuna-133
LLaVA-1. 1--ccc---
Junny-Phi--SSgLP
Qwen-VL
Bunny-Phi-1.. Phi-P.h-.55 1.5-A.-
Bunny Phi- 1.5--..-SigL.
Bunny PhV-2-EVA
CogAgent
CLgVLM
InstructtLIA..cuaa-7
0.1
0.3
0.1
0.2
0.2
0.4
0.6
Random Guessing
Language Model
Vision-Language Model
Figure 5: Performance overview spatial reasoning tasks. We report the accuracy averaged all questions. We consider the VQA (Vision-only) format for vision-language models. The dashed red line denotes the expected accuracy for random guessing. For Spati iali-lll-a Maataa and Maze-Nav tasks, only a few models outperform random guessing by notable margin.
(2) Text-only inputs: While the textual imput includes essential spatial information, it generally does not significantly enhance the spatial reasoning capabilities of competitive models. An exception occurs in the Spatial-Gris Mistral-7B-Insiee at 62. 12, both notably surpassing random guessing. Despite these successes, the performance of these models still lags significantly behind human levels. These results underscore the need for further development of techniques tailored to spatial understanding and reasoning.
task, where Llama-3 achieves accuracy of 71.9%. followed by
The impact of input modality. To investigate the impact of modality, we compare the performance of a large language model (LLM) and vision-language model (VLM) with the same language backbone. We consider the VQA (Vision-only) format for vision-language models. The results are shown Figure 6. Each vertex on the spider plot represents the average accuracy of (VLM, LLM) pair. We observe that on Spatial performance compared their LLM counterpart, despite having an additional visual encoder. For example, on Spatial- Shatial-Crid Mixtral-7B achieves average accuracy of 62. %, while LLaVA-v1.6 Mistral-7B only yields an accuracy of 47.1 % (15% 4). Detailed results can be seen in Appendix F.
and Spatial-Gris Grid, the majority of VLMs yield worse
LLatB-- Miltrai-T1
LLNVA-16-348
LLaWA-U6-348
LLAVA-1-6-543
0.15
LLaWk-lg- Viound-79
LLaVA 18
LL-K--15- Vicina-78
CopVLM
InstrustLIP. Vicura-T0
pnttuusBLPP. Vicuns-taa
InstrutiliP.
InabiieBLIPP Vicuns-Di
InatrectBLPP Vicuhac78
InttrietBLPP Vieus-Ga
Vicanin2
Avg Acc LLM (TQA)
-
Avg Acc VLM (VQA)
TQA VQA Spatial-Map
(b) TQA VQA on
(c) TQA VQA Spatial-Grid
Figure 6: TQA (LLM) vs. VQA on spatial reasoning tasks. Each vertex on the spider plot represents the Avg Acc of (VLM, LLM) pair with the same language backbone, i.e., LLM v.s. VLM further finetuned on that. VLMs are depicted in red, and LLMs blue. We can see that VLMs rarely enhance the performance compared to their LLM counterparts.

---

Delving Into Spatial Reasoning for Vision-Language Models
4.1 Seeing Without Understanding: The Blindness of Multimodal Language Models
To better understand how VLMs process visual information, we conduct series of controlled experiments in the VTQA (Vision-text input) setting. For each sample, we replace the original image input (that matches the textual descrip- tion) with either: (1) No Image: only keep the textual input without the image input, (2) Noise Image: a Gaussian noise image irrelevant to the task, and (3) Random Image: a random image from the dataset that does not match the textual description, as shown in Figure 7.
Original
Random Image
Noise Image
Figure 7: Illustration of Random Image and Noise
Image for the example in Figure 2.
Spatial-Grid
Maze-Nav
Bunny Phi-2 EVA
LLaVA-1.6-34B
140
5,778
5.403
LLaVA-1. 6-Mistral-78
5751
0.473
0.271
0.381
0.234
LLaVA-1.-Vicuna11
Qwen--V ML
1:340
2.425
0.327
0.382
0.2
0.3
0.4
0.5
0.6
0.7
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Original Image
Image
Figure 8: VTQA vs. TQA (VLM) on spatial reasoning tasks. VLMs exhibit improved performance in spatial reasoning tasks when visual input is absent.
VLMs exhibit improved performance when visual input is absent. We conducted experiments by entirely removing the Original Image and relying solely on the textual description. The results are shown in Figure 8. For each task, we report the accuracy averaged across all questions. Remarkably, the absence of visual input leads to better performance across a range of VLM architectures. For instance, the performance of L.L. 1-6344BBBBBBBBB on the Spatial-Grid task improved by 20. % when no image was presented compared to scenarios with the original image. This observation underscores that when textual information alone can address the questions, additional visual inputs do not necessarily enhance, and may even hinder the performance, a sharp contrast to human capabilities where visual cues significantly aid in understanding. The removal of visual input forces the models to utilize the textual information to solve spatial reasoning tasks.
Spatial-Grid
Maze: Nav
Bunny-Phi-Z-EWA
InstractBUP-.kunaa1386
0.482
0.42
instuctBLI--Vccuaa-7
2.283
0.309
3:35
2.435
LLaVA-1.6.38 6-348
0.378
0.615
3.253
LLaVA-1.6Vkcuna-333 138
0.47
0.511
0.231
0.316
Owen-VL
5:309
0.355
0.327
0.3
0.4
0.20
0.40
Original Image
Nobs Image
Figure 9: Original Image vs. Noise Image in VTQA. Replacing the original image with a Gaussian noise image improves the performance across diverse VLM architectures.
Noise image can improve the performance. We replace the Original Image with a Noise Image while retaining the original textual description. The results are shown in Figure 9, Consistent with
7

---

Spatial-Grid
Maze-Nat
Dunny-Phi--EEWA
0,301
0,40]
0.222
InstructBLIPVViinaaa 138
InstructuUP--icua-778
0.189
0.256
0.395
LLaVA-1.6-340
0.361
0,590
0.288
0.363
LLaVA-1, 6-Mistral-78
0.508
LLaWA-1.--Wcuua-1338
2.363
202
Qwen-VL
0.339
0.349
0.377
0.364
0.20
0.25
0.30
0.35
Original Image
Random Image
Figure 10: Original Image Random Image in VTQA. On Maze-Nav, replacing the original image with random image leads to performance improvement across diverse VLM architectures.
the findings in Original Image vs. No Image, using a noise image also improves the performance across various VLM architectures. For example, the accuracy of LLa 1...-iccnn-1331 increases by 6.5% on the -az task when the noise image is used as opposed to the original image. In contrast to the No Image setting, noise images provide limited visual cues. Nonetheless, the model tends to prioritize the textual information, especially when visual cues are not pertinent to the task.
To build on previous findings, we further investigate the effects of replacing the Original Image with a Random Image (illustrated in Figure 7). Unlike a noise image, a random image is task-related but may provide conflicting information compared to the textual description. Intuitively, one might expect that such random images would degrade VLM performance due to contradictory cues. However, as demonstrated in Figure 10, this expectation does not always hold true. For instance, Random Image in the Maze- Nav task leads to improved performance across various VLM architectures. This outcome implies that VLMs are not heavily reliant visual information, particularly when adequate textual clues provided.
Mismatched image-text does not necessarily hurt.
LL M5
LLaWk-1 Mistral78
3040
0.38
0.6
CogAgeric
0.30
0.35
CogAgent
0.30
CogAdenc
0.05
LLAt0-L-- Vicura-sa
Vicyra-88
LLaVe-----
B.2
Vicuria78
M
CogRLM
CogVUM
IntrurtiUPP- Viuraa-7
INPPCeBLIP- Meuns-BB
InstruccRLIIP
IngrocttBL IP.
InstrucgUP-
IntractBLPP MiunnBB
Vicuna-71
Micon-GBa
Vicuna-78.
LLaW--E-Nitann-12
LLa------iliir--111
Aug VQA (Vision-only Input)
(a) VQA VTQA Spatial-Map
(b) VQA VTQA Maze-Nav
(c) VQA VTQA on Spatial-Grid
Figure 11: VQA VTQA spatial reasoning tasks. Each vertex on the spider plot represents the Avg Ace of a (Vision-only Vision-text) pair with the same VLM model. We can see that having the additional textual input (VTQA) enhances the performance compared to only using images (VQA).
4.2 Leveraging Redundancy in Multimodal Inputs
Multimodal language models offer considerable versatility in handling multimodal inputs. While the visual input alone often provides sufficient details for humans to address spatial reasoning tasks with ease, we propose that VLMs significantly benefit from the inclusion of textual descriptions alongside visual data, even if this introduces substantial redundancy. We verify this hypothesis by comparing VQA (Vision-only input) and VTQA (Vision-text input) across diverse VLM architectures. The results are shown in Figure 11, where each vertex on the spider plot represents the average accuracy of a (Vision-only, Vision-text) pair based on the same VLM. For Spatial-Map and Spatial-Grs -rrid we can clearly see that having the additional textual input (VTQA) enhances the performance compared
8

---

LL.00-15:344
0.99
L- Viceerts
Daltl---- Vilan IB
LLAIA-EE- Vicune
1.16
0:23
9.3
Mire-ggiP
LLaV--LE Wicrnee
LUB Viodar
Danry
Vicno-88
91-22996
P11-2-96
Avg Text-only with LLM (TQA)
Act of Text-only Input with VLM (No (mg)
TQA img Spatial-Map
(b) TQA No img Maze-Nav
(c) TQA Img apatial-Grid
Figure 12: Comparison of Text-only input with LLM (TQA) vs. Text-only input with VLM (No Img). We consider VLMs that support text-only inputs. Each vertex on the spider plot represents the Avg Acc of (LLM, VLM) pair with the same language model backbone.
to only using images (VQA) across different VLM architectures. This suggests that textual inputs improve the accuracy of spatial reasoning VLMs. We further compare TQA and VTQA in Appendix D. Detailed results included in Appendix F.
Text-only input with LLM vs. Text-only input with VLM. Given the demonstrated efficacy of text- only imputs, we conducted an ablation study to compare LLMs and VLMs using text-only inputs. We consider VL.Ms that are capable of processing text without accompanying visual data. The results are illustrated in Figure 12. Except for CogVLM the majority of VLMs outperform their corresponding LLM backbones. This suggests that the language model backbones VLMs demonstrate enhanced spatial reasoning abilities through multimodal learning. Conversely, the addition of visual information does not necessarily provide further benefits.
Spatial-Map
Maze Nav
Spatiall Grid
0.5
D.4
0.3
0.1
ony
Vistom-onty
Figure 13: Results with proprietary models. Similar trends are observed as with open-source models.
4.3 Proprietary vs. Open-Source Models
As many recent benchmarks have shown that proprictary models generally outperform open-source models, it is important to understand if our observed trends hold with proprictary models. The performance of several top proprietary models (GPT-4, GPT-4V, GPT-4o, Gemini Pro 1.0, and Claude 3 Opus) are shown in Figure 13. We have the following salient observations: (1) A significant performance gap exists between SoTA open-source models and proprietary models, expected. Furthermore, with both Text-only and Vision-text formats, GPT-4V and GPT-4o significantly out- perform random guessing across all tasks. For instance, in the Vision-text format, GPT-4o achieves
Comparison
Results
Summary of Findings
TQA (LLM) VQA
Figure
VQA rarely enhances the performance compared TQA (LLM). VLMs exhibit improved performance spatial reasoning tasks when image input is absent.
VTQA vs. TQA (VLM)
Figure 8
Given same image input, additional textual description en- hances VLM'ss performance.
VQA vs. VTQA
Figure 11
TQA (VLM) vs. TQA (LLM)
Figure 12
Multimodal fine-tuning enhances LLM'' spatial reasoning ability.
TQA (LLM) vs. VTQA
Figure 16
No defimitive winner.
Table 2:
Summary of main findings.

---

an accuracy of 0.989 on Spatial-Grid (Table 7). (2) Yet, the trends we observed with open source models hold, VQA consistently under-performs compared to TQA and VTQA, for example. GPT-4V's performance improves by 25.6% on Spatial-Grie when switching from Vision-only to Vision-text input; and again no clear winner between TQA and VTQA (see Appendix D for further details), showing that proprietary models, even the new GPT-4o model, still do not appear to fully leverage visual inputs. We summarize the key findings of this work in Table 2.
5 Related Work
Large language models. Large language models (LLMs) have achieved outstanding performance
across a diverse array of fields, including finance [34], bioinformatics [63]. law [60], education [29], coding [24], and creative tasks [2, 36]. LLM architectures have gone through significant changes in recent years, with notable developments such as BERT [14], OPT [76], PaLM [12]. Gemma family [62], Mistral family [27], GPT family [2. 10], Claude family [5], and LLaMA family [3, 64, 66]. These models have demonstrated emergent abilities and revolutionized numerous domains, supporting capabilities such as in-context learning [46, 51, 59], compositional reasoning [16, 20, 70], and task- specific adaptation [57, 58, 71J. However, visual understanding and reasoning an intrinsic part of human cognitive maility-remanne largely under-explored for LLMs.
Vision-language models and multi-modal language models. The success of LLMS has pro- pelled the adoption of the Transformer architecture [68] within the computer vision community, such as ViT [15], Beit [9]. CLIP [54], MAE [21]. Swin [42, 43], and DřT [53]. Building on the capabilities of powerful LLMs, multi-modal language models (MLLMs) such as Flamingo [4],
LLaMA-Adapter [19, 74], LLava [37, 391, stable-diffusion [55], BLIP [32, 331, MiniGPT-4 [77], Qwen [7, 8]. Gemini [61], MM1 [45] have significantly expanded the range of problems that can be addressed with improved reliability [49]. These models adeptly handle inputs from diverse modalities and have demonstrated remarkable performance on diverse tasks such as mathematical reasoning [44],
image-text retrieval [47, 73], and visual reasoning [6, 18, 25, 28, 31, 40, 50].
Spatial understanding and reasoning. Spatial reasoning entails comprehending and manipulating spatial relationships, a task significantly more challenging than visual grounding [1. 28, 52]. Although progress in natural language processing, evaluations, and benchmarks on LLMs such as GPT-4 [2] and Claude3 [5] have predominantly focused on textual relational reasoning. This focus often overlooks the intricate nature of spatial reasoning tasks. Notably, recent studies [17, 30, 41, 48, 56, 72, 75] have conducted thorough evaluations across a diverse range of tasks. Yet, they demonstrate a scant exploration of spatial reasoning capabilities, highlighting a gap in assessing this complex cognitive skill in current benchmarks. In particular, Zhang er al. [75] suggest that MLLMs primarily leverage textual cues rather than visual diagrams to solve math problems while focusing only on math problems. Yamada et al. [72] design simple navigation tasks and finds L.MMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains, while our benchmarks are more complicated than their navigation tasks based on simple geometry maps. Fu et al. [17] propose a benchmark on science and games with limited exploration related to spatial reasoning, while we are dedicated diverse spatial reasoning tasks with in in-depth analysis.
Discussion and Conclusions
We explored the spatial understanding capabilities of VLMs and LLMs. Our experiments resulted in several surprising conclusions across SoTA open-source and proprietary models. (1) VLMS struggle with spatial reasoning tasks, (2) multimodal models rarely surpass L.LMs when relying on visual inputs. (3) when both textual and visual inputs are provided, multimodal language models rely less on the visual inputs, and (4) VLMs often outperform their LLM counterparts with text-only inputs. This challenges the belief that current VLMs are highly performant at a wide range of vision-text tasks. However, with further thought, perhaps this is to be expected after all. The currently known architectures for VLMs attempt to "translate" " the vision input into the language space and all reasoning is then performed in the language domain. It is logical that this automatic translation path worse than human provided translation to text, as in our text-only scenario. Thus our work shows the limits of the translation approach. Instead, to bridge the gap to human performance, future models require new architectures that treat vision input as a first-class source of information and reason in a joint vision-language space. It is our hope that our work informs development on this path.
10

---

Acknowledgement
The authors would like thank NeurIPS anonymous reviewers for their insightful feedback and helpful discussions. Yifei Ming and Yixuan Li are funded in part by the AFOSR Young Investigator Program under award number FA9550-23-1- 1-01-4444 National Science Foundation (NSF) Award No. IIS IIS-2337077 & IIS-2331699, and Office of Naval Research under grant number N00014-23-1-2643
References
[1] Mostafa Abdou, Artur Kulmizev, Daniel Hersheovich, Stella Frank, Ellie Pavlick, and Anders Sógaard. Can language models encode perceptual structure without grounding? case study in color. arXiv preprini arXiv:
20Xix:2200.00 109.06 06129, 29, 2021.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[3] Meta AI. Introducing meta Ilama 3: The most capable openly available Ilm to date, 2024.
https://ai. http:/aa.mmmm...eae/mmm/... hom/blog///eoooooooooooooooooolo wwwpollggggaaaaaeeeeeeeeae mwtmmmmaaaaaaaaaaaaaaammmmaaaaaa
[4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson. Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances neural information processing systems, 35:23716-2377 2022.
[5] Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. /Model
https:
//www- edn anthropic. wttp///aaalllaaaaaaaaaaaaaaaaaaa Card_ Caadd__laue____ 3.pff.
[6] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages 2425-2433, 2015.
[7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[8] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding. localization, text reading, and beyond. arXiv preprint 2r0: /2966, 2023.
[9] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprin! arXiv: 08254, 2021.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:187711901, 2020.
[11] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[12) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[13] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances Neural Information Processing Systems, 36, 2024.
[14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 181111100000 10.044855 2018.
11

---

[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[16] Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
[17] Deqing Fu, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomorphic representations. arXiv preprint arX4i:2404.000 2466001266 2024.
[18] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see 2024.
but not perceive. arXiv preprint arXiv:2404.12388
[19] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. adaater v2: Parameter-efficient visual instruction 2023.
model. arXiv preprint arXiv:2304.
[20] Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Tianyi Zhou. Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic. arXiv preprint arXiv:2402.099-65 2024.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pages 16000-16009, 2022.
[22] Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402. 1530, 2024.
[23] Wenyi Hong. Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2024.
[24] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review, 2024.
[25] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 670066700700 1700-6709,00 2019.
[26] Michael Igorevich Ivanitskiy, Rusheb Shah, Alex F. Spies, Tilman Räuker, Dan Valentine, Can Rager, Lucia Quirke, Chris Mathwin, Guillaume Corlouer, Cecilia Diniz Behn, and Samy Wu Fung. A configurable library for generating and manipulating maze datasets, 2023.
[27] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arX1v:2310.06825, 2023.
[28] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017.
[29] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
12

---

[30] Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312. 122411 2023.
[31] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[33] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[34] Yinheng Li, Shaofei Wang. Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the Fourth ACM International Conference on Al in Finance, pages 374-382, 2023.
[35] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint urXiv:2309.05463 2023.
[36] Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp. Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. arXiv preprint arXiv:2403.07183, 2024.
[37] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint 2rX1v:2310.03744, 2023.
[38] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oer, and world knowledge, January 2024.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[40] Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-We Chang, and Yansong Feng. Are Ilms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. arXiv preprint 2402 2024.
[41] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui Hc, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307. 06281, 2023.
[42] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12009-12019, 2022.
[43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[44] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wa Kei-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint 2rXiv:23100022:0 2023.
[45] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training, arXiv preprint arXiv:2403.0m 09611, 2024.
13

---

[46] Sewon Min, Xinxi Lyu. Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. /2837, 2022.
[47] Yifei Ming and Yixuan Li. Understanding etrieval-augmented task adaptation for vision- language models. In Proceedings of the 41 International Conference on Machine Learning, pages 35719-35743, 21-27 Jul 2024.
[48] Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. Spartqa:: textual question answering benchmark for spatial reasoning. arXiv preprint 1045:2200..05892, 2021.
[49] Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, Ziwei Liu, Toshihiko Yamasaki, and Kiyoharu Aizawa. General- ized out-of-distribution detection and beyond in vision language model era: A survey. arXiv preprint arXiv:2407.21794, 2024.
[50] Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang. Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of vision language models. arXiv preprint 2rXiv:2403.20331, 2024.
[51] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 1895, 2022.
[52] Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International conference on learning representations, 2021.
[53] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684- 10695, 2022.
[56] Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts. In Proceedings of the AAAI conference on artificial intelligence, pages 11321-11322 2022.
[57] Zhenmei Shi, Jiefeng Chen, Kunyang Li. Jayaram Raghuram, Xi Wu, Yingyu Liang, and Somesh Jha. The trade-off between universality and label efficiency of representations from contrastive learning. In The Eleventh International Conference on Learning Representations, 2022.
[58) Zhenmei Shi, Yifei Ming, Ying Fan, Frederic Sala, and Yingyu Liang. Domain generalization via nuclear norm regularization. In Conference Parsimony and Learning (Proceedings Track), 2023.
[59] Zhenmei Shi, Junyi Wei, Zhuoyan Xu, and Yingyu Liang. Why larger language models do in-context learning differently? In RO- of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
[60] Zhongxiang Sun. A short survey of viewing large language models in legal aspect. arXiv preprint arXiv:2303.09 13,366 2023.
[61] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Japt-Sae Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arX1v:2312.11805. 2023.
14

---

[62] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprini arXiv:2403.08295, 2024.
[63] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):13301194
2023.
[64] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (..Xiv2332.139777. 2023.
[65] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09220 2023.
[66] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (rXiv:2307.09288 2023.
[67] Jack Urbanek. Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Nomero-Soriano, A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26700-26709, 2024.
[68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[69] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.
[70] Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang. Do large language models have composi- tional ability? an investigation into limitations and scalability, In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
[71] Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang. Towards few-shot adaptation of foundation models via multitask finetuning. In The Twelfth International Conference on Learning Representations, 2023.
[72] Yutaro Yamada, Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and lIker Yildirim. Eval- uating spatial understanding of large language models. Transactions on Machine Learning Research, 2024.
[73] Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Li. Context-aware attention network for image-text retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3536-3545, 2020.
[74] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama- adapte: Efficient fine- fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.161199 2023.
[75] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal 1lm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024.
[76] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained 2022.
transformer language models. arXiv preprint arXiv:
[77] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.100 10592, 2023.
15

---

Appendix
A
How Does SpatialEval Expand Beyond Traditional VQA?
We highlight several key rationales for SpatialEval in comparison to existing VQA benchmarks:
Task scope and focus:
Existing benchmarks such as Visual Genome [31], GQA [25]. and BLINK [18] focus only VQA, where the image is required but the text description is often omitted or optional. In contrast, SpatialEval further explores spatial reasoning across different settings: TQA (LLM), TQA (VLM), and VTQA, where images texts can be optional, thereby broadening the scope of tasks.
Evaluation scheme: we primarily focus on generative LLMs and VLMs which can "elab- orate on the reasoning behind your answer in detailed, step-by-step explanation" (Sec- tion 2.2). The task in prior VQA benchmarks is often treated as discriminative with no explicit reasoning. Therefore, it remains unknown if previous observations can be naturally transferred to foundation models pre-trained on web-scale data.
The Textual Representation of Images:
The textual descriptions in prior VQA benchmarks are often brief directly imply the answers. While these datasets are valuable, they do not consistently offer the level of complexity we require, such as numerous objects containing dense visual information along with detailed natural language captions that fully convey the image content. In SpatialEval, we provide long and dense captions for each image. As result, answers cannot be easily inferred. We also aim isolate object detection capability from spatial reasoning ability by simplifying objects to symbols (e.g., Spatial-Map).
IQ test for VLMs and LLMs: Tasks in SpatialEval are designed to serve as cognitive tests that evaluate basic capabilities of synthetic visual content, humans can solve them with near near-perfect accuracy. This indicates that the tasks are within the realm of human cognitive capabilities.
foundation models. While three tasks feature
B Limitations and Societal Impact
Limitations. In this work, we introduce four novel tasks and conduct a comprehensive evaluation of diverse large language models (LLMs) and dision-language models (VLMs), presenting a controlled and thorough evaluation of spatial reasoning capabilities. While our analysis is detailed and extensive, it remains primarily empirical. We believe that embarking challenges. would significantly enrich our understanding of pre-trained multimodal language models. Moreover, our focus has been on in-depth analysis rather than on developing new training strategies or adaptation algorithms to enhance spatial reasoning. Recognizing these points, we identify them as valuable directions for future research.
formal theoretical study, despite its
Societal impact. Regarding the positive societal impact, our evaluations and observations could catalyze the development of new algorithms that enhance the spatial reasoning abilities of LLMMs and VLMs. Improved understanding of these models has the potential to significantly benefit sectors requiring robust spatial understanding and navigation. This could lead to more reliable and efficient systems that enhance safety and user experience. As for potential negative societal impacts, our work primarily involves empirical evaluations using synthetic datasets designed to probe spatial reasoning. Therefore, we do not anticipate any direct negative societal impacts arising from our current research.
C Experimental Details
C.1 Software and Hardware
Our experiments are conducted on NVIDIA A A100 GPUs. Our implementation is based on Python 3.10 and Py Porch 1.2.
16

---

C.2 Hyperparameters and Error Bars
The hyperparameters discussed this paper pertain the decoding strategies of each model. For deterministic decoding, we adhere to the default settings specified for each model. The models employing deterministic decoding (argmax) include Bunny-Phi Pain---ii----LIII CogAgent, CogVLM, InstructBI.IP- (PP-Vicuna-11 13B, and InstructBLII--icunle Vicuna--B. For non-deterministic decoding, we utilize the default hyperparameters provided by Hugging Face (for instance, Top-P set 0.9 and the temperature at 0.2 for LL.-6-1. Error bars for the main results (Figure 5) are obtained with three independent runs.
C.3 Model Checkpoints
For most open open-source models, we use the checkpoints provided by Hugging Face as shown in Table 3. For Bunny variants Coonn-PPhi-L.5-E" Monny-Phi-L.5-SgLL.. and Bunny-Phi-2-EV.. we use merged weights following instructions in https: /ttp//ggtttt.tt//e//e/ com/BAAT-DCAT/Bumyyy
Model Name
Link
LLaMA-2-13B
chat-hf
LLaMA-2-7B
ch
LLaMA-3-8B
Mistral-7B
Nous-Heeres.2-Yi-344B
Phi-2
Bccun--33B115
Vicuna-7B-1.5
LLaVA-1.6-34B
LLaVA--..-Miistal--7
LLa 1-3
L LLaVA--.6-Vicuma-78
Vicunan--
Bunny-Phi-2-SSe Phi-2-SigLIP
Qwen-VL VL
CogAgent
Cog VLM
BastuctBLI--vicua----3 13cu
InstructBLIP Vicuna-7B D
Table 3: Model checkpoints from Hugging Face.
C.4 Detailed Illustration of Datasets
In the main text, abbreviated the textual descriptions in Figure 1, Figure due space constraints. In this section, we provide the full descriptions for each task to facilitate better understanding of spatial reasoning tasks. The complete illustrations are displayed Figure for Spatial -Map and Figure 15 for Maze-Nav respectively. Three questions (termed QI to Q3) are associated with each -Map. sample tasks Spaitial -Map, Maze-Nat Nav, and Spatial-Gris
17

---

TQA (Text-only)
Spatial-Map
Consider a map with multiple objects: Watches
Whale's
Himalayan Hot Springs is to the Southeast Whale's Watches Himalayan Hot Springs is to the Seutheast of Brews Brothers Pub. Dragonfly Drones Southwest Himalayan Hot Springs. Dragonfly Drones
Southeast Brews Brothers Pub. Unicom Umbrellas is to the Southwest of Brews Brothers Pub. Unicom Umbrellas is to the Northwest of Dragonfly Gifts to the Southwest Himalayan Hot Springs.
Whale's
Unicorn Umbrellas Gale
Brows Brothers Pub
VQA (Vision-only)
Sprinngs
<img> The figure represents map with multiple objects. Each object is associated with shown in the figure.
A
Gale Gifts
VTQA (Vision-text)
<img The figure represents figure. The same figure can be described as follows: Whales Watches is in the map. Brews Brothers Himalayan Springs
with multiple objects. Each object associated with
Dragonfly
name shown
Southeast of Whale's Watches. Himalayan Hot Springs is Southeast Brews Brothers Pub. Dragunfly Drones the Southwest of Himalayan Hot Springs. Dragonfly Drones the Southeast Brews Brothers Unicorn Umbrellas
the Southeast Whale's
Southwest Brews Brothers Pub. Unicorn Umbrellaz the Northwest of Dragonfly Drones. Gale Gifts to the the Southwest Himalayan Springs.
Northeast Unicorn Umbrellas Gale
which diraction Whale's Watches relative Dragonfly Drones? Northwest
Questions
3. Southwest
C. Southeast
Northeast
Q Which object is the Southwest Gale Gifts?
Dragonfly Dranes
Unicorn Umbrellas
Himalayan Hot Springs
Brews Brothers
0
How many objects are the Southwest Himalayan Hot Springs?
A
B.4
C1
D.0
Figure 14: Illustration of the Spaitial -Map task with complete textual descriptions in TQA and VTQA.
18

---

TQA (Text-only)
Maze- Nav
Hore is Maze represented ASCII code (LHS), where the symbols have the following
meanings:
regresents
impassable barriers
(space) represents the navigable path within the maze, but not necessanil the correct exit.
S denotes the starting point the maze. E marks the endpoint or the exit of the
X marks spacific route you should follow to navigate from the starting point to endpoint E.
The objective allowed in any of the four cardinal directions (up, down, left, right) along the spaces, but solve the maze, follow the path designated by right
from
following
marked by X
defined change movemant direction degrees clockwise relative to the previous direction.
defined change relative to the previous direction.
direction that degrees anticlockwise
VQA (Vision-only)
<Img> The represents where colored blocks have the fallowing meanings:
Black blocks represent walls are impassable barriers.
White blocks represent navigable paths within the maze, but not necessarily the correct
Green block denotes the starting point (S) of the maze. Red block marks endpoint the
of the
The objective is to nav' navigate from S to by following the path marked by Blue Moverent is allowed in any of the four cardinal directions (up, down, left, right) along the White blocks, the maze, follow path designated A right turn is defined as change movement direction that is 90 degrees clockwise relative the previous direction.
A turn defined as change in movement direction that 90 degrees anticlockwise relative to the previous direction.
VTQA (Vision-text)
Img- The figure represents a Maze, where the colored blocks have the following meanings Black blocks represent walls that impassable barriers.
White blocks represent navigable paths within the maze, but not necessarily the correct path to the exit.
block denotes starting point
Red block marks the endpoint or the exit (E) the maze.
navigate from by follewing the path marked Blue. Movement allowed any the four cardinal directions (up, down, right) along the White blocks, but to solve the maze, fallow the path designated by Blue blacks. represented ASCII (LHS),, following meanings:
The objertive
symbois
represents walls that impassable barriers.
(space) represents the navigable path within the maze, but not necessarily the correct path to the exit.
you
navigate starting point
endpoint 'E.
- denotes the starting point of the maze.
marks endpeint
maze.
The objective to navigate from to by folloving the path marked by Movement is
allowed the cardinal directions down, right along spaces, salve the maze, follow the path des gnated by
Q: How many right turns are there in the provided path from too
Questions
7
How many
previded path
A.4
8.1
C.9
D.5
I directly the starting point, horizontal displacement?
A Yes
No
Figure 15: Illustration of the Maze Maze-Nav task with complete textual descriptions in TQA, VQA, and VTQA.
19

---

D Further Ablation Studies and Discussions
TQA vs. VTQA:
No definitive winner. In Section 3, we demonstrated the advantages of text-only input (TQA) based on LLMs over vision-only input (VQA) based on VLMs. Subsequently, given the same VLM, the comparison between VTQA and VQA in Section 4 reveals that VLMs rely less on visual information when sufficient textual clues are provided. Curious readers seeking a quantitative comparison between TQA and VTQA can refer to the results illustrated in Figure 16. Unlike the other comparisons, no definitive winner emerges between TQA and VTQA across all spatial reasoning tasks, model sizes, and architectures. For instance, the VTQA performance of InstructBL.IP Vicuna- 133 surpasses Vicuna- 13B in the faze-Nav task, whereas Vicuna-1 13B outperforms InstructBI.PP -i--vcua-l 13B in the Spatial -Map task. This indicates that text-only inputs (with language models) still hold an advantage in certain spatial reasoning tasks.
3.35
Cogágent
Lavk 1. 348
CogAgent
LB 348
Gogágent
LLAWA 348
LL Vicuns-78
LEaMA Vituna-1
0.05
6.3
Vicana-38
Coggti
CapVEM
InstrunigLIP
betructBLIP-
InstrucitiII
ELIP- LLaWA-B--
InstrarcALIP.
LLaVbo 1-w-----
Aug Acc of TQA (Text- only Input )
Aug Acc of VTQA (Vision-tex: Input)
(2) TQA VTQA on Spatial-Map
(b) TQA VTQA Maze-Nav
(c) TQA vs. VTQA on Spatial-Grid
Figure 16: TQA (LLM) vs. VTQA spatial reasoning tasks. Each vertex the spider plot represents the Avg Acc of (LLM, VLM) pair with the same language model backbone. For LLM, we use the Text-only input (i.e., TQA); for VLM, we Vision-text input (i.e., VTQA).
Impact of prompting techniques. In line with our approach to sampling strategies, our primary goal in choosing prompting techniques is to report the best model performance given the same question. The prompt technique we use, which asks for step-by-step explanation is the most effective query among others our initial studies. As concrete example, we compare the original prompting strategy, "First, provide concise answer in one sentence. Then, elaborate on the reasoning behind your answer in a detailed, step-by-step explanation" (step-by-step explanation), with a simpler prompt, Answer: (completion). Results in Table 4 show that the simpler completion prompt consistently underperforms compared to the step-by-step explanation prompt.
Input Modality
Model
Avg Acc (completion)
Avg Ace (step-by-step explanation)
Mistral-7B
0.61
0.62
Text-only
Vicuna-133B 13B-
0.25
0.46
Vicuna-7B-1.5
0.31
0.41
-LaVA-1.6-Miistme
0.47
0.47
Vision-only
LLaVA-1.6 VVicuuca--
0.25
0.40
LLaVA-1.6-Vicume
0.26
0.31
LLaVA-I.6-Mistrall777
0.59
0.59
Vision-text
IL-VA--.6-Vieuna-I
0.26
0.48
LLaVA-1.66 1.V
0.30
0.46
Table 4: Results for completion and step-by-step prompts.
Impact of temperature for decoding strategies. We conducted an ablation study to examine how different temperatures affect the model performance. A higher temperature allows for more diversity model responses. Most models consistently underperform when the temperature set to 1.0 compared to 0.2 (our default) as shown in Table 5.
20

---

Input Modality
Model
Avg Ace (temperature=1)
Avg Ace ((emperature=0.2)
Mistral-7B
0.62
0.62
Text-only
Vicuna-13B-1
0.37
0.46
Vicuna-7B-1.5
0.36
0.41
LLaVA-1.6
0.37
0.47
Vision-only
LLaVA-1.6- Vicuna- 13B
0.32
0.40
LLaVA-I.6-Vicuma:
0.26
0.31
LL
0).66
0.59
Vision-text
LLaVA--.--Viccuaa
0.37
0.48
LLaVA-1.6- Vicuna-7B
0.33
0.46
Table 5: Results by varying different temperatures for decoding strategies.
E
Results on Spatial-Rens task
Table 6 presents the performance on the Spatial-Real task. The same trends observed in synthetic tasks persist with real images (see VQA vs. VTQA, TQA (LLM) vs. VTQA, TQA (LLM) vs. VQA in Table 2). Notably, compared to synthetic tasks (Figure 5 and Figure 11), the overall accuracy on Spatial-Real increased across all three input types (TQA, VQA, and VTQA). However, modality gap (accuracy difference between VTQA and VQA) widens significantly, from 7.0% on synthetic tasks to 30.0% on Spatial Real. This indicates that the performance disparity is more pronounced on natural images.
Input Format
Model
Average Accuracy
Vicuma-13B-
0.845
Vicuna-7B- 7B-1.5
0,716
Text-only (TQA (LLM))
Mistral-7B
0,800
LLaMA-3-8B
0.884
CogVLM
0.419
Qwen-Vi
0.329
1-6-Mistrl--7B
0,368
Vision-only (VQA)
CogAgent
0,400
I.L -i-cuna--
0.432
LLaVA-1.6- Vicuna--
0,445
CogVLM
0.594
Qwen-Vi
0.729
LLaVA-1.6-MM 1---s---Mitt
0,761
Vision-text (VTQA)
CogAgent
0.471
0.832
LLaVA-1.6- LLaVA-L.6VVieun-77B
7i
0.806
Table 6: Performance on the Spatial-Reat task. The same trends still hold on real images: VQA
vs. VTQA, TQA (LLM) vs. VTQA, and TQA (LLM) vs. VQA.
F Detailed Experimental Results
Results for proprietary models are summarized in Table In Section and Section 4, to clearly illustrate the impact of modality, we present the results averaged all questions in Figure 6 for VQA vs. TQA and Figure 11 for VQA vs. VTQA. This section provides comprehensive breakdown of results for individual questions. Detailed comparative results for Spatial-Map. Maze-Nav. and Spatial-Grid d are shown in Table 8, Table 9, and Table 10, respectively. We compare LLM and VLM with the same language model backbone. For the VLM assessments, we consider inputs in both vision-only (VQA) and vision-text (VTQA) formats.
21

---

Input Format
Model Name
Spatial-Map Acc
Maze-Nav Ace
Spatial-Grid Acc
Claude Opus
0.619
0.277
0.957
Gemini Pro 1.0
0.453
0,223
0.620
Text-only
GPT-4o
0.616
0,353
0,986
GPT-4V
0.555
0.321
0.923
GPT4
0,504
0.353
0,939
Claude Opus
0.348
0.163
0.455
Gemini Pro
0.263
0.219
0,512
Vision-only
GPT-4o
0.617
0.410
0.833
GPT-4V
0.237
0.263
0.668
Claude Opus
0.606
0.332
0.888
Gemini Pro 1.0
0.456
0,203
0.687
Vision-text
GPT-4o
0.643
0,382
0.989
GPT-4V
0.490
0.333
0.924
Table Detailed results for proprietary models.
Input Format
Model Name
Q1 Acc
Q2 Ace
Q3 Ace
Avg Ace
Mistral-7B
0.40
0.27
0.34
0.34
Nous-
mous-Hemees-2Yi-333
0.45
0.22
0.30
0.32
Text-only
Vicuna-7B-1.5
0.39
0.40
0.07
0.29
Vicuna-13B4
0.44
0.52
0.21
0.39
1aVAA-.6-Miistall778
0.26
0.37
0.13
0.25
LLaVA-1.6-34B
0.30
0.28
0.30
0.29
LLaVA-1.6-Vinee Vicuna-7B
0.27
0.32
0.26
0,28
InstructBLIP- Vicuna- 13B
0.22
0.21
0.08
0.17
Vision-only
LLaVA-L..-Vicuna-1333
0.31
0.32
0.15
0.26
DnstructBLIP-Vicunde
0.25
0.18
0.06
0.16
CogVLM
0.24
0.39
0.13
0.25
CogAgent
0.29
0.34
0.10
0.25
LLaVA-1.6-Mistrall
0.43
0.42
0.15
0.33
LLaVA-116-34p
0.58
0.47
0.21
0.42
LLaVA- 1.VVA-1.--Vicun--1B
Vicuna-7B
0.35
0.25
0.12
0.24
InstructBLIP- Vieuna-
0.39
0.33
0.27
Vision-text
LLaVA-1.6- LLaVA-I.6-Vicua--13
0.49
0.49
0.14
0.38
InstructBLIP-
0.35
0.25
0.12
0.24
CogVLM
0.40
0.52
0.14
0.35
CogAgent
0.37
0.45
0.12
0.31
Table 8: Detailed results for the Spatial-Map task. We compare LLM and VLM with the same
language model backbone. For VLMs, we consider both vision-only and vision-text input format. The averaged results are summarized in Figure and Figure 11.
22

---

Input Format
Model Name
Q1 Acc
Q2 Acc
Q3 Ace
Avg Ace
Mistral-7B
0.43
0.23
0.33
0.33
Mous-Hermes-2-1
0.25
0.18
0.67
0.36
Text-only
Vicuma-7B-1.5
0.11
0.13
0.63
0.29
(icuna-13B-11
0.20
0.15
0.33
0.23
LLaVA-I.6-Mistral-78
0.25
0.10
0.34
0.23
LLaVA-1.6-34BB
0.27
0.23
0.67
0.39
LLaVA-1.6 Vicuna-7B
-LaVA-I.6-iccum--B
0.17
0.22
0.65
0,35
DistuctBLLIP-Vicuna--
0.15
0.09
0.67
0.31
Vision-only
LLaVA-I.6-Vicuma-1133
0.33
0.20
0.33
0.29
DnstrucIBLIP-Vicuna-T7
0.21
0.28
0.33
0.27
CogVLM
0.13
0.16
0.66
0.32
CogAgent
0.16
0.22
0.33
0.24
I.aVAAI..-Miixraa--7
0.27
0.10
0.36
0.24
I.I.VA-1.6-34B
0.28
0.18
0.68
0.38
LLaVA-I.6-Vicuna-1
0.16
0.23
0.63
0.34
InstructBLIP- Vicuna-13B
0.11
0.11
0.58
0.27
Vision-text
LLaVA-1.6- Vicunaa-13
0.31
0.18
0.33
0,27
CatructBLIP-Vicunae
0.16
0.23
0,63
0.34
CogVLM
0.14
0.20
0.60
0.31
CogAgent
0.14
0.14
0.34
0.21
Table 9: Detailed results for the Maze-Nav task. We compare LLM and VLM with the same language
model backbone. For VLMs, consider both vision-only and vision-text input format. The averaged results are summarized in Figure and Figure 11.
Input Format
Model Name
Q1 Acc
Q2 Ace
Q3 Ace
Avg Ace
Mistral-7B
0.47
0.73
0.66
0.62
Yous-Hermes-2-Yi--44
0.33
0.91
0.50
0.58
Text-only
Vicuna-7B-1.5
0.38
0.48
0.38
0.41
Vicuna-13B-11.5
0.34
0.65
0.39
0.46
IL2VA--..6Miistra--77
0.28
0.69
0.45
0.47
I.AVA-I.6634B
0.32
0.29
0.21
0.27
JLaVA-I.6-Vicuna-/7
0.30
0.38
0.26
0.31
InstructBI CirrucBBLII-VVE BFrruccB.IP-Vicuma--1 13B
0.24
0.24
0.21
0.23
Vision-only
LLaVA-1166 Vicuna- 13B
-LaVA-1.6-Vicuna-1
0.30
0.58
0.30
0.40
PirrcctBLIP-Viuuaa--
0.27
0.25
0.24
0.25
CogVLM
0.25
0.38
0.34
0.32
CogAgent
0.33
0.33
0.29
0.32
LLaVVA-L.-Miisral-775
0.32
0.79
0.65
0.59
LLaVA-1.6-34B
0.44
0.40
0.33
0.39
LLaVA--.6-Vicuna-77
0.36
0.68
0.34
0.46
InstructBLIPP-ieeuee Vicuna- 13B
0.33
0.59
0.39
0.44
Vision-text
-LaVA-1.6-Vicunn. 13B
0.29
0.78
0.36
0.48
InstructBLIP- Vicuna-7B
0.28
0.29
0.27
0.28
CogVLM
0.30
0.82
0.35
0.49
CogAgent
0.33
0.52
0.36
0.40
Table 10: Detailed results for the Spatial-Gris task. We compare LLM and VLM with the same
language model backbone. The averaged results are summarized in Figure 6 and Figure 11.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims well-supported by the comprehensive experiments and analysis Section 3, Section 4, and further results in the Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provided detailed limitations in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the experimental details needed in Appendix C to support the reproducibility of our experiments. Datasets and codebase with detailed instructions will also be released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Both datasets and our code with detailed instructions will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss httppsccccpbccccccceececcccccccc /wt.dee/ooodeeummiieeeee /Cooeeeooiissseeeeeee wwssssnoolicyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All experimental details are included Appendix C to facilitate understanding of the experiments in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the standard deviation in main experiments (see Figure 5).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide hardware and software details Appendix C.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : http///eeuripp.eee httpp/uwww.....uuu.eeepeeeeeeete http:public/thhcsccididdee.
Answer: [Yes]
Justification: The authors have read the NeurIPS Code Ethics and made sure the paper follows the NeurIPS Code of Ethics in every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provided detailed societal impacts in Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The datasets in this paper are generated to evaluate the spatial reasoning abilities models and do not pose risks for potential misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly cites the original paper sources whenever asset is used. URLs of all retained model checkpoints are included in Appendix C.3 and Table 3.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Both the synthetic datasets and the code will be released with well-documented instructions.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "XMvrpf7mv8", "reviewer": "Reviewer_kVxr", "summary": "This paper develops a novel benchmark to understand spatial reasoning ability of LLM and VLM. Using such a benchmark, authors conduct experiments to evaluate models' performance, and reveal several results.", "questions": "See weakness.", "limitations": "No negative societal impacts of the study.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and well-structured.\n\n2. This paper conducts a series of experiments and include very recent LLMs and VLMs.", "weaknesses": "1. The design rationales of the three benchmarks are not included or discussed. \"Spatial-Map\" and \"Maze-Nav\" look reasonable but \"Spatial-grid\" seems ill-posed. Would we really meet any similar application scenario in the real world?\n\n2. In the experiments of \"The impact of input modality\", LLMs and VLMs take very different inputs in the benchmarks, with no part overlapping. In this case, it is unsure whether such a comparison is fair or even valid.\n\n3. It is unclear why the results of \"Spatial Map\" are not provided in main paper or appendix. In addition, the performance of some VLMs are also missing. Given that it can be already observed that there exist some outliers, it is important to show full results of the study to ensure the validity of observations.\n\n4. LLaVA-1.6-34B is the most performant model in this model family. However, in Figure 7-9, it consistently under-performs compared to other LLaVA models on Spatial grid. I would like to see authors' comments on these finding. What might be the cause to the weakness of LLaVA-1.6-34B?\n\n5. This paper overall provides quantitative results of current VLMs without deeper analysis. It remains unclear what on earth causes the weakness of these VLMs. What are insights or takeaway to improve VLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments and questions, which we address in detail below.\n\n> *Q1: Design rationales of the three benchmarks. Real world application scenarios for \"Spatial-Grid\"?*\n\nOur benchmarks are designed to cover diverse aspects of spatial reasoning, such as spatial relationships, navigation, position understanding, and object counting (L34-42). We illustrate the design rationale of each benchmark in more detail: \n- Spatial-Map: This benchmark resembles a map scattered with numerous objects (e.g., hotels and stores), each represented by distinct symbols. It evaluates the model's ability to identify and understand the spatial relationships between these dispersed objects.\n- Maze-Nav: This benchmark simulates navigation scenarios. It evaluates the model's capability to understand and navigate through complex environments, akin to finding a path in a maze.\n- Spatial-Grid: This benchmark reflects scenarios with dense visual information in structured, grid-like environments where understanding the **layout** is crucial.\n\nWe highlight several real-world applications for Spatial-Grid:\n- Warehouse Manipulation: Autonomous robots in warehouses navigate through grid-like storage layouts where objects are densely arranged. These robots need precise spatial understanding to efficiently retrieve and restock items, making them highly dependent on reliable spatial reasoning capabilities similar to those tested in Spatial-Grid.\n- Traffic Monitoring: Systems that identify and count vehicles within structured scenes, such as busy intersections with vehicles arranged compactly in lanes and rows, rely on accurate localization and counting. Such capabilities are critical for the safe deployment of VLM-based systems in traffic management.\n- Diagram and Document Understanding: An emerging application for MLLMs is understanding documents that contain dense collections of visual elements arranged in structured layouts. As businesses and educational sectors increasingly rely on digital data, the ability to parse and understand complex documents becomes crucial.\n\n> *Q2: Fairness of the experiment on \"The impact of input modality\" as \"LLMs and VLMs take very different, non-overlapping inputs in the benchmarks\"*\n\nIt is indeed challenging to directly compare LLMs and VLMs due to their difference in input modality. This is exactly the purpose of our benchmarks: for each sample, we create semantic overlap between the textual description and the image, where the input in each modality has similar information sufficient to answer the question. \nInstead of comparing LLM vs. VLM, we compare TQA (LLM), TQA (VLM), VQA, and VTQA. We would like to clarify the terminologies and input modalities considered for different models, summarized in **Table G.1** in the General Response.\nTo investigate \"The impact of input modality\", we conducted multiple set of controlled experiments between TQA, VQA, and VTQA. As acknowledged by R2 (i8Xy), \"The TQA, VQA, VTQA test is a smart way to reveal the modality bias in VLMs and LLMs, clearly showing the blindness of VLMs.\"\nOur comparisons and references in the paper are summarized in **Table G.2** in the General Response.\n\n> *Q3: Missing results of Spatial-Map and some VLMs in the main paper*\n\nThanks for the question! The results of the \"Spatial-Map\" benchmark are provided in the main paper in Sec 4 (Fig 4 and 5), with further details available in the ablation studies in Sec 5.2 and 5.3 for both open-sourced and proprietary models (Fig 10, 11, and 12). Additional detailed results can be found in Appendix D.\nIn Sec 5.1, the ablation study comparing VTQA vs. TQA (VLM) does not include results for the InstructBLIP family because these models do not support using only their LLM backbone. We apologize for the oversight of not including the \"Spatial-Map\" results in Sec 5.1. This was an unintentional mistake. The full results have been included in the Appendix, and we will ensure they are added to the main paper in the revised manuscript.\n\n> *Q4: Why LLaVA-1.6-34B underperforms other LLaVA models on Spatial-Grid?*\n\nGreat point! Fig 4, 5, 10, and 11 show that while LLaVA-1.6-34B consistently outperforms other LLaVA models on Spatial-Map and Maze-Nav, it lags in Spatial-Grid.\nWe conducted an ablation study, where we added three more questions Q4-Q6 (object in bottom-right, top-right, bottom-left corners) besides Q1-Q3 in Table 5 (Appendix D).\n\nDetailed breakdowns can be found in **Table 2** in the additional one-page PDF. The results indicate that LLaVA-1.6-34B excels in counting (Q1) but struggles with layout and fine-grained visual interpretation (Q2-Q6), limiting its spatial reasoning in dense grid environments. However, we believe a deeper understanding of this phenomenon is beyond the scope of our work. We hope our benchmark serves as a springboard for further study on spatial reasoning and design of better reliable MLLMs.\n\n> *Q5: It remains unclear what on earth causes the weakness of these VLMs. Insights or takeaway to improve VLMs?*\n\nDue to the differences in model architecture, training pipelines, the scale and diversity of training data, pinpointing the precise causes of weaknesses in modern VLMs is fundamentally challenging and beyond the scope of our study. \nThe primary purposes of this paper include: (1) Introducing a pioneering benchmark that evaluates diverse aspects of spatial reasoning with multimodal inputs. (2) Conducting a timely and comprehensive evaluation of a wide range of both open-sourced and proprietary LLMs and VLMs.(3) Highlighting the current limitations of VLMs in spatial reasoning, thereby setting the stage for further investigations.\nIn Sec 6, we proposed several hypotheses to explain the observed discrepancies among VLMs. While we acknowledge understanding the 'why' behind performance differences is crucial, we believe first systematically documenting and decomposing the observed phenomena is equally valuable to the research community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZXO41cHtqi", "reviewer": "Reviewer_i8Xy", "summary": "The paper proposes a set of synthetic tests to compare the spatial understanding in VLMs and LLMs. The tests include Spatial-Map, Maze-Nav, and Spatial-Grid, which all include an image, paired with text that describe the image, and a question. Using the synthetic data, the VLMs and LLMs are studied using different modalities, including VQA, TQA, and VTQA, where the models are asked to answer questions based on only the image, only the text, and both image and text. Findings include that the current VLMs are strongly biased towards texts (blind); VLMs outperform LLMs in TQA. Both open models and proprietary models have been tested.", "questions": "1. How does the findings in this paper differ from previous literatures like [1,2,3]. Are the findings different, or are they discovered in a new way as proposed in this paper?\n2. Any discussions about the synthetic-real gap?\n3. In Spatial-Map, without text descriptions, it looks like the VLMs may describe spatial using “left/right/up/down” instead of “west/east/north/south” - will this happen, and how is it addressed?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The TQA, VQA, VTQA test is a smart way to reveal the modality bias in VLMs and LLMs, clearly showing the blindness of VLMs.\n2. Intensive results and analysis are provided.", "weaknesses": "1. All datasets contain only 2D synthetic images, which is substantially different from real images. At least the images can be made more realistic using computer graphics, like [4,5].\n2. The synthetic nature of the tasks introduces artifacts. Good spatial reasoning ability on this task may not generalize to real ones. The results may be highly dependent on whether the model has been trained on a similar task. For example, the performance of GPT-4O almost doubles that of Gemini in this paper, while the difference of these models on real images is not this significant.\n3. The observation that VL models are highly biased towards text, and are “blind” visually, has been studied long in the VQA community, even before LLMs [1,2,3, etc.]. The related literature should be discussed.\n[1] Eyes wide shut? exploring the visual shortcomings of multimodal llms\n[2] Explicit Bias Discovery in Visual Question Answering Models\n[3] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering\n[4] Clevr: A diagnostic dataset for compositional language and elementary visual reasoning\n[5] Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments! \n\n> *W1: All datasets contain only 2D synthetic images, which is substantially different from real images.*\n\nWe choose synthetic data due to its controllability, scalability, and the ability to create highly specific scenarios with flexible, long and detailed captions that are not adequately covered by existing VQA benchmarks. \n\nIn addition, this approach avoids potential data leakage and ensures that model performance is not merely the result of data memorization during web-scale pre-training. Moreover, our benchmarks shift the focus from object recognition to evaluating spatial reasoning abilities involving numerous objects. Similar paradigms have been explored in recent works evaluating visual diagram understanding [6].\n\n\n> *Q1: The observation that VLMs are highly biased towards text, and are “blind” visually, has been studied long in the VQA community, even before LLMs [1,2,3, etc.]. How does the findings in this paper differ from previous literatures like [1,2,3]?**\n\nThank you for the suggestion! We find these works intriguing and relevant, and have cited and discussed references [1-5] in our revised manuscript. We would like to highlight several key differences in our approach:\n\n1. **Task Scope and Focus**: Previous works [1-5] primarily focus on the VQA task where images are required but text descriptions are often omitted or optional. In contrast, our study further explores spatial reasoning across different settings: TQA (LLM), TQA (VLM), and VTQA, where images or texts can be optional, thereby broadening the scope and complexity of tasks. \n2. **Evaluation**: we primarily focus on multimodal language models (MLLM). We treat these models as generative models which are required to \"elaborate on the reasoning behind your answer in a detailed, step-by-step explanation\" (L154). In contrast, the evaluation strategy is different in prior works [2-5], where the task is often discriminative with no explicit reasoning. Therefore, it remains unknown if previous observations can be naturally transferred to foundation models pre-trained on web-scale data. Concurrent work [1] also evaluates MLLMs but focuses on visual representations by curating pairs of images adversarial for CLIP-based models. In contrast, our data is unrelated to CLIP features and our questions are not designed to be adversarial. Humans can solve our tasks with near-perfect accuracy. This indicates that the tasks are within the realm of human cognitive capabilities and are realistic for evaluating modern MLLMs.\n3. **The Textual Representation of Images**: the textual descriptions in prior VQA benchmarks are often too brief or directly imply the answers. In contrast, we provide dense or detailed captions for each image. Therefore, our benchmarks do not include questions where answers can be easily inferred from a short caption. We also try to isolate object detection capability from spatial reasoning ability by simplifying objects to symbols.\n4. **Different Interpretation of Visual Blindness**: While the visual blindness of VLMs has been long studied, the context differs in our benchmark. In typical VQA tasks, bias might stem from the questions themselves [2]. For example, the answer to \"What is the color of the grass?\" is usually \"Green\", allowing models to answer the question correctly without seeing the image. Multiple prior works have focused on addressing such biases with new benchmarks [3]. In contrast, none of the questions in our benchmark can be answered without looking at the image and the blindness is unrelated to the question. \n\n\n> *W2 & Q2: The synthetic nature of the tasks introduces artifacts. Good spatial reasoning ability on this task may not generalize to real ones. Any discussions about the synthetic-real gap?*\n\nThanks for your comments! It is valuable to extend the scope of our work and validate our results on real images. We found that a very recent work [7] released a Densely Captioned Images (DCI) dataset where each image has a detailed caption with more than 1000 words on average. However, this dataset does not include questions. We then carefully curated multiple-choice questions regarding spatial-reasoning (object counting, relation, and position understanding) and annotated the answers. We name this new dataset as Spatial-Real.\n\n\nThe evaluation results are shown in **Table 1**  in the additional PDF. The same trends still hold for real images (see VQA vs. VTQA, TQA (LLM) vs. VTQA, TQA (LLM) vs. VQA in **Table G.2**). In addition, compared to Fig 4 and 10 in the paper, the overall accuracy increases across all three input modalities (TQA, VQA, VTQA) in the Spatial-Real benchmark. However, the modality gap (accuracy difference between VTQA and VQA) grows from 7.0% on synthetic benchmarks (avg) to 30.0% (avg) on Spatial-Real.\n\nWe will open source this new dataset and the full results will be included in our revised manuscript.\n\n> *Q3: In Spatial-Map, will VLMs describe directions using “left/right/up/down”?*\n\nGreat point! We carefully examined the VLMs' responses and did not observe notable differences in the way they describe spatial directions. This assessment was done by manually checking all results.\n\nGiven that the options with directions (e.g., A. Northwest, B. Southwest, C. Southeast, D. Northeast) are included in the question prompt, we observed consistent adherence to these instructions across all models. Rather than using relative terms like \"left/right/up/down,\" the models consistently responded with the specified cardinal directions.\n\nExamples of typical responses include:\n- \"A. Northwest\"\n- \"Children's Choice Toys is located to the northeast of Yak Yarns.\"\n- \"The correct option is B. Northeast.\"\n\n\n[6] Zhang et al., MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?\n\n[7] Urbanek et al., A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions, CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAzs3VCEQA", "reviewer": "Reviewer_JHoY", "summary": "The authors make a benchmark to test the spatial reasoning of multimodal language models (MLMs) using synthetic data consisting mostly of diagrams, mazes, and charts. They benchmark a lot of existing MLMs. Their key findings are: 1) Spatial reasoning is limited in most MLMs. 2) MLMs rely heavily on textual information and not the visual, and 3) interestingly, the VLMs are better at utilizing spatial text information than LLMs suggesting multimodal training helps understand the visual natural of language even though VLMs tend to rely less on visual information during test time.", "questions": "See above concerns. \n\nOverall I like the direction of the paper. But some analysis justifying the need of such type of synthetic data (mazes and custom diagrams) instead of real images or actual graphs (useful for scientific figures) would be nice.", "limitations": "The authors discuss some limitations of models, but not the benchmark. What can this benchmark power, and what are some analyses that the benchmark currently cannot power that would be good to have?", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of stress testing MLMs with spatial and figure understanding has been a recent trend and is very useful to push these models to do real-world tasks. \nThe analysis done is interesting and provides some useful insights into how these models use visual or textual information.", "weaknesses": "- Some of the claims may be too broad to make based on results of just this dataset. For instance, the models using more textual information than visual. This could simply be because the visual inputs in this benchmark are of a starkly different domain than what the models are trained on (real images instead of synthetic graphs and mazes and charts), whereas the domain gap in language is much lesser since the language tokens remain the same. Hence, we can only make the statement that the model is using more textual cues for such kinds of data. It is unclear if that is also the case on real image data. \n\n- There are many ways to test spatial reasoning using real VQA style datasets like BLINK, MMBench etc. So , it is unclear why they came up with these complicated tasks. Is there any real-world relevance for such tasks? \n\n- The authors mention they opt for synthetic data due to controllability, but all the analysis they do can be simply done on real image QA datasets like GQA.  For instance, for checking vision vs textual reliance, just feed in scene graphs vs the image to an MLM from GQA. Or, for mismatched image and text, randomly pair image and QA. At least a discussion on why such a benchmark is needed is missing. What special analysis in the paper is enabled by this benchmark that couldn't be enabled by an already existing benchmark like Visual Genome, GQA, BLINK etc.\n\n- Some settings could be explained more clearly in the paper. For instance, for the vision-only input - I am guessing the authors mean that they do not describe the image in text, but the question is still in text, correct? \n\n- A nit pick, but some grammar could be improved. e.g., intro, the first line (line 18):\" had a transformation effect on research\" - a transformational effect not affect. Other grammatical errors are also occasionally present in the paper. \n\n-  Another nit pick, but this paper seems more fitting for the datasets and benchmarks track.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your feedback and comments! \n\n> *Q1: Some claims may be too broad; we can only state the model uses more textual cues for such data*\n\nThanks for the comments and the suggestion! We acknowledge the visual domain gap and have added remarks in our revised manuscript. We intentionally include such synthetic data in our benchmarks to avoids potential data leakage from popular visual benchmarks, ensuring performance isn't due to data memorization during pre-training. Additionally, our benchmarks shift the focus from object recognition to evaluating spatial reasoning abilities with numerous objects. Similar paradigms have been explored in recent works evaluating visual diagram understanding for mathematical reasoning such as [1].\n\n> *Q2: Why need such a benchmark is missing. What analysis does it enable that existing benchmarks like Visual Genome, GQA, and BLINK do not?*\n\nWe would like to highlight some key rationales of our benchmark: \n1. Existing benchmarks focus only on VQA, where the image is required but the text description is often omitted or optional. Our study further explores spatial reasoning across different settings: TQA (LLM), TQA (VLM), and VTQA, where images or texts can be optional, thereby broadening the scope of tasks.\n2. We utilize synthetic data due to its controllability, scalability, and the ability to create highly specific scenarios with flexible, long and detailed captions that are not adequately covered by existing benchmarks such as Visual Genome, GQA, and BLINK. While these datasets are valuable, they do not consistently offer the level of complexity we require, such as numerous objects containing dense visual information along with detailed natural language captions that fully convey the image content. \n3. The textual descriptions in these datasets are often too brief or directly imply the answers. In our work, we provide dense or detailed captions for each image. As a result, no answers can be easily inferred from a short caption. We also try to isolate object detection capability from spatial reasoning ability by simplifying objects to symbols.\n4. Although our tasks seem complicated, humans can still solve them with near-perfect accuracy. This indicates that the tasks are within the realm of human cognitive capabilities and are therefore realistic for evaluating advanced AI models.\n\nGiven the increasing use of VLMs, it is crucial to have a diverse suite of benchmarks to assess their abilities.\nWe also believe these awesome VQA datasets are relevant and have cited in our revised manuscript.\n\n> *Q3: Real-world applications/relevance for such synthetic tasks?*\n\nWe would like to highlight some of the real-world applications: \n- Diagram and Document Understanding: An increasingly relevant application for MLLMs is the interpretation of digital documents that include a dense array of visual elements. For businesses and educational sectors, the ability to understand symbols, figures, and diagrams within structured layouts is crucial.\n- Map Understanding and Navigation: Our Spatial-Map and Maze-Nav benchmarks simulate map-like environments scattered with numerous objects, such as hotels and stores, each represented by distinct symbols. These configurations are typical in map apps.\n- Warehouse Operations and Traffic Management: autonomous robots navigate through grid-like storage layouts densely packed with objects, requiring precise spatial understanding for efficient item retrieval and restocking. Similarly, in urban traffic management, systems must accurately identify and count vehicles within structured scenes, such as busy intersections where vehicles are compactly arranged in lanes and rows. These capabilities are critical for the safe deployment of MLLM-based systems.\n\n> Q4: Will the statements still hold on real image data? \n\nThanks for your comments and suggestions! Inspired by your suggestion to explore natural images, we found a very recent work [2] released a Densely Captioned Images (DCI) dataset, featuring detailed captions with over 1000 words per image. However, this dataset lacks questions. Therefore, we carefully curated multiple-choice questions regarding spatial-reasoning (object counting, relation, and position understanding) and annotated the answers. We name this new dataset as **Spatial-Real**. Due to the rebuttal period's time constraints, we have created 200 TQA-VQA-VTQA pairs. \n\nThe evaluation results, shown in **Table 1** in the one-page PDF, indicate that the same trends still hold for real images (see VQA vs. VTQA, TQA (LLM) vs. VTQA, TQA (LLM) vs. VQA in **Table G.2**). In addition, compared to Fig 4 and 10 in the paper, overall accuracy increases across all three input modalities (Text-only, Vision-only, Vision-text) in the Spatial-Real benchmark. However, the modality gap (accuracy difference between VTQA and VQA) grows from 7.0% on synthetic benchmarks (avg) to 30.0% (avg) on Spatial-Real. \n\nTo ensure full reproducibility, we will open source this new dataset and extend the number of sample pairs to match Spatial-Map, Spatial-Grid, and Maze-Nav. The full results will be included in our revised manuscript.\n\n\n> *W4: Some settings could be clearer (e.g., the meaning of vision-only input)?*\n\nThanks for the suggestion! We have clarified this in the revised manuscript. Your understanding is correct. We describe the Text-only, Vision-only, and Vision-text input modalities based on how we feed the image information to the models. Vision-only input means the image is fed directly to the models without textual description, while all questions are presented in text.\n\n> *W5: Grammars and typos*\n\nThanks for the catch! We have carefully examined and fixed grammatical errors and typos throughout.\n\n\n[1] Zhang et al., MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?\n\n[2] Urbanek et al., A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions, CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNYLcDWDVR", "reviewer": "Reviewer_WNLP", "summary": "This work presents 3 new synthetically generated VQA evaluation benchmarks and conducts a comprehensive evaluation of the limitations of current  VLMs in spatial reasoning tasks. These tasks include spatial relationships, navigation, position understanding, and counting. The authors demonstrate using their introduced benchmarks that VLMs have limited performance on tasks requiring detailed visual understanding and reasoning of images.", "questions": "please see above", "limitations": "Yes. It is discussed and properly addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Evaluating the spatial reasoning capabilities of SoTA VLMs is highly impactful and very relevant. I find this work novel, intriguing, and highly useful as it points to very important limitations in VLMs.  The experimental analysis is very comprehensive, incorporating both several open-source VLMs and proprietary models. Additionally, the paper is well-written and structured, contributing to its overall readability and clarity.", "weaknesses": "Spatial Understanding and Evaluation: The use of synthetic data in the evaluation, while valuable, may introduce confounding factors unrelated to the task of spatial understanding/reasoning. \n* For instance, a vision-only model with limited OCR capabilities may perform poorly on the spatial-map task, regardless of its actual spatial reasoning ability. It is challenging to disentangle the contributions of OCR performance from spatial understanding in such benchmark.\n* The synthetic maze-navigation and spatial-grid images might be out-of-distribution for some models, especially open-source ones. This could also explain why a noise image improves the accuracy of Llava-1.6 on maze-navigation tasks, while a similarly out-of-distribution maze image does not necessarily harm or consistently improve the  performance. It is important to consider the potential impact of out-of-distribution data on open-source models. By looking at fig 12 it looks the performance of the latest gpt4o on vision-only is similar to vision-and-text. That said, I still believe the work remains valuable and important.\n\nHuman Performance Comparison:\n* The claim on line 172 that \"the performance of these models still lags significantly behind human levels\" would benefit from concrete numbers to substantiate it. Providing specific human performance metrics on these tasks or citations here could would make the claim more robust. Alternatively, softening the statement to reflect the need for further comparison could be considered.\n\nSampling Strategies and Prompting Techniques:\n* Exploring different sampling strategies or reasoning/prompting techniques could yield valuable insights into the model's performance. It would be beneficial to include discussions on how these variations impact the results. To my understanding only a simple prompting technique is utilized \"step-by-step explanation\". Was there any particular reason to append the selected prompt vs others. \nIn line 157, the paper also mentioned \"For each model, we adopt the default configurations and decoding strategies, e.g., argmax for deterministic decoding and top-p for non-deterministic decoding. \" It would be very useful to be more specific, e.g. what top-p and temperature was used.  How \"deterministic\" was the deterministic decoding in API based models? \n\nVQA Evaluation Benchmark Details:\n* The paper would benefit from clearly specifying the size of each VQA evaluation benchmark in terms of the number of samples or data points. This information is crucial for understanding the scope and scale of the evaluations conducted. I might have missed it but I couldn't find it in the paper. I would encourage the authors to particular address this in rebuttals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your support of our work and insightful comments!\n\n\n> *Q1: The use of synthetic data in the evaluation, while valuable, may introduce confounding factors (e.g., (1) It is challenging to disentangle the contributions of OCR)*. (2) The synthetic data might be out-of-distribution for some models. It is important to consider the potential impact of out-of-distribution data...That said, I still believe the work remains valuable and important.\n\nThank you for your insightful feedback, and we appreciate your recognition of the value and importance of our work! We agree that disentangling OCR capabilities from spatial reasoning abilities in VQA (Vision-only) input format for VLMs presents a significant challenge. This challenge motivated our development of the VTQA (Vision-Text) input format. In VTQA, we aim to enhance VLMs' OCR capabilities by providing a detailed textual description of the objects. This approach helps to mitigate the impact of OCR abilities and emphasizes spatial reasoning. Additionally, our benchmark avoids questions with direct textual answers provided in the prompt, requiring the model to engage in genuine reasoning to arrive at the correct answer. \n\nIndeed, the synthetic datasets might be out-of-distribution for some models. We have intentionally include such synthetic examples in our benchmarks to test the robustness and generalization ability. By doing so, we ensure that good performance is not merely a result of data memorization during web-scale pre-training. We hope our work will provide a foundation for future advancements in VLM development.\n\nWe further curate a real-world task, Spatial-Real, based on real images with dense captions [1]. Detailed results are shown in **Table 1** in the one-page PDF. We find that the trends (summarized comparisons VQA vs. VTQA, TQA (LLM) vs. VTQA, TQA (LLM) vs. VQA in **Table G.2**) stated in the paper still holds. The modality gap (accuracy difference between VTQA and VQA) even grows from 7.0% on synthetic benchmarks to 30.0% on Spatial-Real on average. \n\n> *Q2: The claim on line 172 that \"the performance of these models still lags significantly behind human levels\" would benefit from concrete numbers to substantiate it.*\n\nThank you for your suggestion! We have conducted a human evaluation on a subset of 900 samples for the three tasks. This subset revealed an average human accuracy rate exceeding 96% (VQA). While these results suggest high human performance on the evaluated tasks, we recognize the limitation of not extending this comparison across all 13,500 data points due to budget constraint. We are committed to an extensive human evaluation on the full dataset and will include the results in the revised manuscript.\n \n> *Q3: The specifics of decoding configs and the impact of prompting techniques.*\n\nThanks for the suggestion! We provide details for each below:\n\n**Decoding configurations:** We have included detailed decoding configurations in Appendix B.2. Our primary approach is to use the default decoding strategies provided by each model to ensure fair comparisons and report the best performance for each model given the same input. Specifically, we use deterministic decoding (argmax) for the following models: Bunny-Phi-2-SigLIP, CogAgent, CogVLM, InstructBLIP-Vicuna-13B, and InstructBLIP-Vicuna-7B. For all other models, we employ non-deterministic decoding with Top-p set to 0.9 and temperature set to 0.2.\n\n**Impact of Temperature**: We conducted an ablation study to examine how different temperatures affect the model performance. A higher temperature allows for more diversity in model responses. Most models consistently underperform when the temperature is set to 1.0 compared to 0.2 (our default), as shown below. We have included the complete results of this study in Appendix E.\n\n|Input modality|Model|Avg Acc (temperature=1)|Avg Acc (temperature=0.2)|\n|-|-|-|-|\n|Text-only|Mistral-7B| 0.62|0.62|\n|Text-only|Vicuna-13B-1.5| 0.37|0.46|\n|Text-only|Vicuna-7B-1.5| 0.36|0.41|\n|Vision-only|LLaVA-1.6-Mistral-7B|0.37|0.47|\n|Vision-only|LLaVA-1.6-Vicuna-13B|0.32|0.40|\n|Vision-only|LLaVA-1.6-Vicuna-7B|0.26|0.31|\n|Vision-text|LLaVA-1.6-Mistral-7B|0.46|0.59|\n|Vision-text|LLaVA-1.6-Vicuna-13B|0.37|0.48|\n|Vision-text|LLaVA-1.6-Vicuna-7B|0.33|0.46|\n\n**Prompting techniques:** In line with our approach to sampling strategies, our primary goal in choosing prompting techniques is to report the best model performance given the same question. The prompt technique we use, which asks for step-by-step explanation is the most effective query among others in our initial studies.\n\nAs a concrete example, we compare the original prompting strategy, *\"First, provide a concise answer in one sentence. Then, elaborate on the reasoning behind your answer in a detailed, step-by-step explanation\"* (step-by-step explanation), with a simpler prompt, *Answer:* (completion).\n\nThe results are shown below. We can see that the simpler completion prompt consistently underperforms compared to the step-by-step explanation prompt.\n\n|Input Modality|Model|Avg Acc (completion)|Avg Acc (step-by-step explanation)|\n|-|-|-|-|\n|Text-only|Mistral-7B|0.61|0.62|\n|Text-only|Vicuna-13B-1.5|0.25|0.46|\n|Text-only|Vicuna-7B-1.5| 0.31| 0.41|\n|Vision-only|LLaVA-1.6-Mistral-7B|0.47|0.47|\n|Vision-only|LLaVA-1.6-Vicuna-13B|0.25|0.40|\n|Vision-only|LLaVA-1.6-Vicuna-7B| 0.26|0.31|\n|Vision-text|LLaVA-1.6-Mistral-7B|0.59|0.59|\n|Vision-text|LLaVA-1.6-Vicuna-13B|0.26|0.48|\n|Vision-text|LLaVA-1.6-Vicuna-7B|0.30|0.46|\n\n> *Q4: sample size of each VQA benchmark*\n\nThank you for pointing this out! In our benchmark, each task (Spatial-Map, Maze-Nav, Spatial-Grid) contains 4,500 samples, resulting in a total of 13,500 samples. Additionally, our benchmarks are designed to be easily scalable. We have included this information in the revised manuscript (Appendix B).\n\n[1] Urbanek et al., A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions, CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Review summary** We sincerely appreciate all reviewers for their time and effort in providing valuable feedback and suggestions on our work. We are glad that reviewers recognize our work to be _novel_ (R1, R4), _highly impactful_, and _intruguing_ (R4). Additionally, reviewers found our results and analysis _intensive_ (R2) and _interesting_ (R3), with useful insights (R3, R4). We appreciate the positive remarks on our manuscript being well-written and well-structured (R1, R4), and the appreciation for the direction of our research (R3).


We have addressed the comments and questions in individual responses to each reviewer. Below, we include two tables that are frequently referenced in our responses. The attached PDF contains an illustration of our new real-world benchmark **Spatial-Real**, evaluation results (Table 1), and an ablation study for the LLaVA family (Table 2, [R1]).

|Model|Input Modality|Term|Description|
|-|-|-|-|
|LLM| Text-only | TQA (LLM) | Input is purely textual and contains all necessary information of an image to answer the question.|
|VLM|Text-only| TQA (VLM) | Input is purely textual, but applied to VLMs (such as the LLaVA family). In the paper, this setting is called Text-only input with VLM (No Img) (Sec 5.1, Fig 7, Fig 11). We have name it as TQA (VLM) in the revised manuscript for easier reference.|
|VLM|Vision-only|VQA| Image-only input without an equivalent textual description |
|VLM|Vision-text|VTQA| Input includes both an image and its textual description|

#### Table G.1: Terminology and input modalities for LLMs and VLMs.

|Comparison|Results and Analysis|Summary of Findings|
|-|-|-|
|TQA (LLM) vs. VQA|  Sec.4 Figure 5| VLMs (with image-only input) rarely enhance the performance compared to their LLM counterparts (with text-only input).| 
|VTQA vs. TQA (VLM)|Sec.5.1 Figure 7|VLMs exhibit improved performance in spatial reasoning tasks when the image input is absent.| 
|VQA vs. VTQA|Sec.5.2 Figure 10|Given the same image input, additional textual description enhances VLM's performance.| 
|TQA (VLM) vs. TQA (LLM)|Sec.5.2 Figure 11 |Multimodal finetuning enhance LLM's spatial reasoning performance.|
|TQA (LLM) vs. VTQA |Appendix C Figure 15| No definitive winner. |

#### Table G.2: Summary of experiments on the impact of input modalities.

Meta-review: This is an interesting analysis paper. The paper develops benchmarks to evaluate spatial reasoning in large language models (LLMs) and vision-language models (VLMs), revealing that VLMs often underperform compared to LLMs, even when additional visual input is provided. An interesting finding is that VLMs without the visual input perform better than VLMs with the visual input and LLMs with text input alone, highlighting the complexity of spatial reasoning in multimodal tasks.

The reviewers raised concerns about the evaluation methods and comparisons, particularly regarding the limited improvement of VLMs when visual inputs are available. However, the authors provided detailed rebuttals, introducing new baselines and conducting additional experiments that effectively clarified the concerns. The discussion around how redundancy between visual and textual information can be leveraged to improve performance also provided valuable insights.

Overall, the AC believes this is a comprehensive study of spatial reasoning in multimodal models, though the conclusion of treating vision as first-class modality might seem obvious to many vision and language researchers. The AC would highly suggest that the authors add more discussion (and potentially some probing experiments) regarding how to improve the spatial reasoning abilities of VLMs, especially when both modalities are available. Exploring ways to optimize the use of visual input for spatial reasoning would significantly strengthen the paper’s contributions to the field.
Decision: Accept (poster)