Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Authors: jiayu  wang
yifei  ming
zhenmei  shi
vibhav  vineet
xin  wang
yixuan  li
neel  joshi
-  madison
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:59:45

Abstract: Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning—a fundamental component of human cognition—remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence. Our code is available at https://github.com/jiayuww/SpatialEval.

Main paper: 
 introduction: 
 The recent breakthroughs in foundation models have had a transformative effect on research and industry, and we have seen these models rapidly integrated into products and new businesses that are shaping people's lives for the better. This sea change was initially driven by large language models (LLMs), which have shown at times near unbelievable, human-level performance across a wide range of tasks. Over the past year, many of these models have been extended to handle images in addition to text, leading to a significant increase in vision-language models (VLMs), especially multimodal large language models (MLLMs), which demonstrate groundbreaking performance in image-related tasks as in the text domain.However, the reality is not quite as rosy as advertised. While these models have been instrumental in advancing the state-of-the-art in complex reasoning tasks such as common sense reasoning, mathematical problem solving, and scientific question answering [17, 30, 41, 72, 75] , they have not been as effective for a number of problem domains. In particular, as we will show in this paper, they have limited performance on tasks that require detailed visual understanding and reasoning of images.Visual understanding and reasoning-an intrinsic part of human perception and cognitive ability-have been largely under-explored when it comes to LLMs and VLMs. In fact, it is often argued that the visual sense is the dominant sense in people, yet when it comes to current models, it seems quite secondary. Spatial reasoning, in particular, is fundamental to everyday human activities such as navigating environments, understanding maps, and manipulating objects. It encompasses skills that are crucial for both survival and higher-order cognition, including the ability to navigate through space, recognize patterns, and deduce relationships from spatial configurations.In this paper, we propose SpatialEval, a novel benchmark containing four tasks (Spatial-Map, Maze-Nav, Spatial-Grid, and Spatial-Real, Section 2.1) to explore the performance of LLMs and VLMs on diverse aspects of spatial reasoning, including relationship, navigation, position understanding, and object counting. Humans excel at such tasks, making them essential capabilities for intelligent systems to emulate for safe and effective deployment in the real world.Our dataset, however, is constructed with a key twist -each problem in our benchmark has an image and a text representation that is sufficient for answering each spatial understanding question. We denote the use of these sources as VQA, which is the standard task of visual-question answering that consists of a vision-only input and a question, TQA, text-only input and a question, and VTQA, a combination of the previous with vision and text input.We conduct a systematic and comprehensive evaluation of a wide range of open-source and proprietary LLMs and VLMs. We perform in-depth analysis and unveil several novel and surprising results that challenge the current understanding of how these models process spatial information:• Spatial reasoning remains challenging: VLMs frequently struggle with spatial reasoning tasks, with some competitive models performing worse than random guessing.• Visual vs. textual inputs: Without detailed textual descriptions, multimodal models rarely surpass the performance of their LLM backbones when relying solely on visual inputs. This underscores the critical role of text in enhancing model performance on spatial reasoning.• Reduced reliance on visual information: When both textual and visual inputs are provided, multimodal language models tend to rely less on the visual component if sufficient textual clues are available.• Textual performance of VLMs: VLMs often outperform their LLM counterparts with text-only inputs, indicating that the language model backbones within VLMs benefit from multimodal training, despite a lack of similar benefits from the visual components.We surmise the limitations in VLMs' spatial understanding stem from the overly simplistic handling of visual information in current architectures and training pipelines. We believe the contributions of this paper will drive changes in model design, accelerating improvements that could unlock more robust spatial reasoning capabilities, and help bridge the gap toward human-like intelligence.2 Dataset and Task Construction 
dataset setup: 
 To evaluate the spatial reasoning abilities of LLMs and VLMs, we construct four diverse tasks including spatial relationships, navigation, position understanding, and counting. To systematically study the impact of modality, we design three types of input formats for each task: (1) TQA (Textonly): the input is purely textual and contains all necessary information for a person to answer the questions. ( 2) VQA (Vision-only): the input consists solely of an image, which provides sufficient details for a person to easily answer, a format also referred to as Visual Question Answering (VQA) in the literature. ( 3) VTQA (Vision-text): the input includes both an image and its textual representation with detailed descriptions, rendering the information in both modalities redundant. We evaluate LLMs using Text-only inputs and VLMs using Text-only, Vision-only, and Vision-text inputs on the same set of questions ( Vision-text. We evaluate language models (w. TQA input) and vision-language models (w. VQA and VTQA inputs) on the same set of questions.Spatial-Map. Understanding the spatial relationships among objects on a map is a fundamental aspect of human cognitive abilities. To simulate this environment, we create a map-like dataset termed Spatial-Map with K objects, where K is configurable. Each object is associated with a unique location name, such as Unicorn Umbrellas and Gale Gifts. To study the impact of modality, the textual representation of each input consists of pairwise relations such as Brews Brothers Pub is to the Southeast of Whale's Watches. An example with K = 6 is shown in Figure 1 , with Text-only, Vision-only, and Vision-text inputs. These questions include asking about the spatial relationships between two locations and the number of objects that meet specific spatial criteria.Maze-Nav. Navigation through complex spaces is essential for intelligent systems. To evaluate such abilities, we have developed a maze-like dataset named Maze-Nav. Visually, each sample can beHere is a Maze represented in ASCII code (LHS) where the symbols have the following meanings: # represents walls that are impassable barriers. " " (space) represents the navigable path within the maze, .... A right turn is defined as a change in movement direction that is 90 degrees clockwise relative to the previous direction. A left turn is defined as a change in movement direction that is 90 degrees anticlockwise relative to the previous direction. represented as colored blocks where different colors signify distinct elements: a green block marks the starting point (S), a red block indicates the exit (E), black blocks represent impassable walls, white blocks denote navigable paths, and blue blocks trace the path from S to E. The objective is to navigate from S to E following the blue path, with movement permitted in the four cardinal directions (up, down, left, right). Alternatively, each input can be depicted in a textual format using ASCII code. An example is illustrated in Figure 2 , featuring Text-only, Vision-only, and Vision-text inputs. We construct this task based on an open-sourced library [26] . The questions asked include counting the number of turns from S to E and determining the spatial relationship between S and E. While such questions are easy for humans, we will show in Section 3 that they still pose significant challenges for modern multimodal language models. Spatial-Grid. To investigate spatial understanding within structured environments, we introduce a grid-like dataset named Spatial-Grid, contrasting with the Spatial-Map where objects are positioned arbitrarily. Visually, each input consists of a grid of cells, each containing an image (e.g., a rabbit). An example is illustrated in Figure 3 . Alternatively, this grid can also be represented in a purely textual format; for instance, the first row might be described as: elephant | cat | giraffe | elephant | cat. The evaluations focus on tasks such as counting specific objects (e.g., rabbits) and identifying the object located at a specific coordinate in the grid (e.g., first row, second column).Spatial-Real. To extend the evaluation of spatial reasoning beyond synthetic environments, we introduce Spatial-Real, a task built on the Densely Captioned Images (DCI) dataset [67] , where each image has a detailed caption with more than 1,000 words on average. As DCI does not contain questions, we curate multiple-choice questions regarding spatial reasoning (object counting, relation, and position understanding) and annotate the answers. An example is shown in Figure 4 . We provide detailed analysis on Spatial-Real in Appendix E.We consider a wide range of competitive open-source language models with different scales, including Phi2-2.7B [35] , the LLaMA family (LLaMA-2-7B, LLaMA-2-13B, and LLaMA-3-8B) [65] , Mistral-7B [27] , the Vicuna family (Vicuna-7B-1.5 and Vicuna-13B-1.5) [11] , and Nous-Hermes-2-Yi-34B.For multimodal language models, we consider the Bunny family (Bunny-Phi-2-SigLIP, Bunny-Phi-1.5-SigLIP, Bunny-Phi-2-EVA, and Bunny-Phi-1.5-EVA) [22] , CogVLM [69] , CogAgent [23] , InstructBLIP family (InstructBLIP-Vicuna-7B and InstructBLIP-Vicuna-13B) [13] , and LLaVA family (LLaVA-1.6-Mistral-7B, LLaVA-1.6-Vicuna-7B, LLaVA-1.6-Vicuna-13B, and LLaVA-1.6-The following can be seen in the image: tall tree directly right of the orange truck bed and over the tree over the back end of the brown truck bed.Open overcast sky over everything on the ground. Right group of tree leaves over the orange truck bed and between the cranes that are suspending it. Window area of the driver's side door of the white truck on the left of the brown truck bed. 34B) [38] . We also evaluate the proprietary models: Open AI's GPT-4V, GPT-4o, GPT-4, Google Gemini Pro 1.0, and Anthropic Claude 3 Opus.Evaluation. As each question contains four options, we use accuracy as the main evaluation metric.The same user prompt is appended at the end of each question: First, provide a concise answer in one sentence. Then, elaborate on the reasoning behind your answer in a detailed, step-by-step explanation. For each model, we adopt the default configurations and decoding strategies, e.g., argmax for deterministic decoding and top-p for non-deterministic decoding. For non-deterministic decoding, the results are averaged over three independent runs for open-source models. For proprietary models, due to their limited availability and increased compute time and cost, we only perform one run. We summarize the terminologies regarding input modalities for LLMs and VLMs in Table 1 : We describe the Text-only, Vision-only, and Vision-text input modalities based on how we feed the image information to the models. Vision-only input means the image is fed directly to the models without textual description, while all questions are presented in text.Model Input 
main results and analysis: 
 Spatial reasoning remains surprisingly challenging. The evaluation results on open-source models on Spatial-Map, Maze-Nav, and Spatial-Grid are shown in Figure 5 . For each task, the reported accuracy is averaged over all questions. For vision-language models, we choose the Vision-only input format, commonly used in Visual Question Answering (VQA). We use a dashed red line in each figure to indicate the expected accuracy if answering by random guessing. Our findings reveal several notable insights: (1) Vision-only inputs: Despite the simplicity of these tasks for humans, most competitive multimodal models perform at levels similar to or barely above random guessing. Figure 5 : Performance overview on spatial reasoning tasks. We report the accuracy averaged over all questions. We consider the VQA (Vision-only) format for vision-language models. The dashed red line denotes the expected accuracy for random guessing. For Spatial-Map and Maze-Nav tasks, only a few models outperform random guessing by a notable margin.(2) Text-only inputs: While the textual input includes essential spatial information, it generally does not significantly enhance the spatial reasoning capabilities of competitive models. An exception occurs in the Spatial-Grid task, where Llama-3 achieves an accuracy of 71.9%, followed by Mistral-7B-Instruct at 62.1%, both notably surpassing random guessing. Despite these successes, the performance of these models still lags significantly behind human levels. These results underscore the need for further development of techniques tailored to spatial understanding and reasoning.The impact of input modality. To investigate the impact of modality, we compare the performance of a large language model (LLM) and a vision-language model (VLM) with the same language backbone. We consider the VQA (Vision-only) format for vision-language models. The results are shown in Figure 6 . Each vertex on the spider plot represents the average accuracy of a (VLM, LLM) pair. We observe that on Spatial-Map and Spatial-Grid, the majority of VLMs yield worse performance compared to their LLM counterpart, despite having an additional visual encoder. For example, on Spatial-Grid, Mixtral-7B achieves an average accuracy of 62.1%, while LLaVA-v1.6-Mistral-7B only yields an accuracy of 47.1% (15% ↓). Detailed results can be seen in Appendix F. Original Image No Image VLMs exhibit improved performance when visual input is absent. We conducted experiments by entirely removing the Original Image and relying solely on the textual description. The results are shown in Figure 8 . For each task, we report the accuracy averaged across all questions. Remarkably, the absence of visual input leads to better performance across a range of VLM architectures. For instance, the performance of LLaVA-1.6-34B on the Spatial-Grid task improved by 20.1% when no image was presented compared to scenarios with the original image. This observation underscores that when textual information alone can address the questions, additional visual inputs do not necessarily enhance, and may even hinder the performance, a sharp contrast to human capabilities where visual cues significantly aid in understanding. The removal of visual input forces the models to utilize the textual information to solve spatial reasoning tasks. Original Image Random Image Mismatched image-text does not necessarily hurt. To build on previous findings, we further investigate the effects of replacing the Original Image with a Random Image (illustrated in Figure 7 ). Unlike a noise image, a random image is task-related but may provide conflicting information compared to the textual description. Intuitively, one might expect that such random images would degrade VLM performance due to contradictory cues. However, as demonstrated in Figure 10 , this expectation does not always hold true. For instance, Random Image in the Maze-Nav task leads to improved performance across various VLM architectures. This outcome implies that VLMs are not heavily reliant on visual information, particularly when adequate textual clues are provided.  
leveraging redundancy in multimodal inputs: 
 Multimodal language models offer considerable versatility in handling multimodal inputs. While the visual input alone often provides sufficient details for humans to address spatial reasoning tasks with ease, we propose that VLMs significantly benefit from the inclusion of textual descriptions alongside visual data, even if this introduces substantial redundancy. As many recent benchmarks have shown that proprietary models generally outperform open-source models, it is important to understand if our observed trends hold with proprietary models. The performance of several top proprietary models (GPT-4, GPT-4V, GPT-4o, Gemini Pro 1.0, and Claude 3 Opus) are shown in Figure 13 . We have the following salient observations: (1) A significant performance gap exists between SoTA open-source models and proprietary models, as expected. Furthermore, with both Text-only and Vision-text formats, GPT-4V and GPT-4o significantly outperform random guessing across all tasks. For instance, in the Vision-text format, GPT-4o achievesSummary of Findings TQA (LLM) vs. VQA Figure 6 VQA rarely enhances the performance compared to TQA (LLM).VTQA vs. TQA (VLM) Figure 8 VLMs exhibit improved performance in spatial reasoning tasks when the image input is absent.VQA vs. VTQA Figure 11 Given the same image input, additional textual description enhances VLM's performance. TQA (VLM) vs. TQA (LLM) Figure 12 Multimodal fine-tuning enhances LLM's spatial reasoning ability. TQA (LLM) vs. VTQA Figure 16 No definitive winner. an accuracy of 0.989 on Spatial-Grid (Table 7 ). ( 2) Yet, the trends we observed with open source models hold, VQA consistently under-performs compared to TQA and VTQA, for example, GPT-4V's performance improves by 25.6% on Spatial-Grid when switching from Vision-only to Vision-text input; and again no clear winner between TQA and VTQA (see Appendix D for further details), showing that proprietary models, even the new GPT-4o model, still do not appear to fully leverage visual inputs. We summarize the key findings of this work in Table 2 . 
related work: 
 Large language models. Large language models (LLMs) have achieved outstanding performance across a diverse array of fields, including finance [34] , bioinformatics [63] , law [60] , education [29] , coding [24] , and creative tasks [2, 36] . LLM architectures have gone through significant changes in recent years, with notable developments such as BERT [14] , OPT [76] , PaLM [12] , Gemma family [62] , Mistral family [27] , GPT family [2, 10], Claude family [5] , and LLaMA family [3, 64, 66] . These models have demonstrated emergent abilities and revolutionized numerous domains, supporting capabilities such as in-context learning [46, 51, 59] , compositional reasoning [16, 20, 70] , and taskspecific adaptation [57, 58, 71] . However, visual understanding and reasoning-an intrinsic part of human cognitive ability-remains largely under-explored for LLMs.Vision-language models and multi-modal language models. The success of LLMs has propelled the adoption of the Transformer architecture [68] within the computer vision community, such as ViT [15] , Beit [9] , CLIP [54] , MAE [21] , Swin [42, 43] , and DiT [53] . Building on the capabilities of powerful LLMs, multi-modal language models (MLLMs) such as Flamingo [4] , LLaMA-Adapter [19, 74] , LLava [37, 39] , stable-diffusion [55] , BLIP [32, 33] , MiniGPT-4 [77] , Qwen [7, 8] , Gemini [61] , MM1 [45] have significantly expanded the range of problems that can be addressed with improved reliability [49] . These models adeptly handle inputs from diverse modalities and have demonstrated remarkable performance on diverse tasks such as mathematical reasoning [44] ,image-text retrieval [47, 73] , and visual reasoning [6, 18, 25, 28, 31, 40, 50] .Spatial understanding and reasoning. Spatial reasoning entails comprehending and manipulating spatial relationships, a task significantly more challenging than visual grounding [1, 28, 52] . Although progress in natural language processing, evaluations, and benchmarks on LLMs such as GPT-4 [2] and Claude3 [5] have predominantly focused on textual or relational reasoning. This focus often overlooks the intricate nature of spatial reasoning tasks. Notably, recent studies [17, 30, 41, 48, 56, 72, 75] have conducted thorough evaluations across a diverse range of tasks. Yet, they demonstrate a scant exploration of spatial reasoning capabilities, highlighting a gap in assessing this complex cognitive skill in current benchmarks. In particular, Zhang et al. [75] suggest that MLLMs primarily leverage textual cues rather than visual diagrams to solve math problems while focusing only on math problems. Yamada et al. [72] design simple navigation tasks and finds LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains, while our benchmarks are more complicated than their navigation tasks based on simple geometry maps. Fu et al. [17] propose a benchmark on science and games with limited exploration related to spatial reasoning, while we are dedicated diverse spatial reasoning tasks with in-depth analysis. 
discussion and conclusions: 
 We explored the spatial understanding capabilities of VLMs and LLMs. Our experiments resulted in several surprising conclusions across SoTA open-source and proprietary models. (1) VLMs struggle with spatial reasoning tasks, (2) multimodal models rarely surpass LLMs when relying on visual inputs, (3) when both textual and visual inputs are provided, multimodal language models rely less on the visual inputs, and (4) VLMs often outperform their LLM counterparts with text-only inputs. This challenges the belief that current VLMs are highly performant at a wide range of vision-text tasks. However, with further thought, perhaps this is to be expected after all. The currently known architectures for VLMs attempt to "translate" the vision input into the language space and all reasoning is then performed in the language domain. It is logical that this automatic translation path is worse than a human provided translation to text, as in our text-only scenario. Thus our work shows the limits of the translation approach. Instead, to bridge the gap to human performance, future models require new architectures that treat vision input as a first-class source of information and reason in a joint vision-language space. It is our hope that our work informs development on this path.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: All experimental details are included in Appendix C to facilitate understanding of the experiments in this paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report the standard deviation in main experiments (see Figure 5 ).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We provided detailed societal impacts in Appendix B.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The datasets in this paper are generated to evaluate the spatial reasoning abilities of models and do not pose risks for potential misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper properly cites the original paper or sources whenever an asset is used. URLs of all retained model checkpoints are included in Appendix C.3 and Table 3 .Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Both the synthetic datasets and the code will be released with well-documented instructions.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: A How Does SpatialEval Expand Beyond Traditional VQA?We highlight several key rationales for SpatialEval in comparison to existing VQA benchmarks:• Task scope and focus: Existing benchmarks such as Visual Genome [31] , GQA [25] , and BLINK [18] focus only on VQA, where the image is required but the text description is often omitted or optional. In contrast, SpatialEval further explores spatial reasoning across different settings: TQA (LLM), TQA (VLM), and VTQA, where images or texts can be optional, thereby broadening the scope of tasks.• Evaluation scheme: we primarily focus on generative LLMs and VLMs which can "elaborate on the reasoning behind your answer in a detailed, step-by-step explanation" (Section 2.2). The task in prior VQA benchmarks is often treated as discriminative with no explicit reasoning. Therefore, it remains unknown if previous observations can be naturally transferred to foundation models pre-trained on web-scale data.• The Textual Representation of Images: The textual descriptions in prior VQA benchmarks are often brief or directly imply the answers. While these datasets are valuable, they do not consistently offer the level of complexity we require, such as numerous objects containing dense visual information along with detailed natural language captions that fully convey the image content. In SpatialEval, we provide long and dense captions for each image. As a result, answers cannot be easily inferred. We also aim to isolate object detection capability from spatial reasoning ability by simplifying objects to symbols (e.g., Spatial-Map).• IQ test for VLMs and LLMs: Tasks in SpatialEval are designed to serve as cognitive tests that evaluate basic capabilities of multi-modal foundation models. While three tasks feature synthetic visual content, humans can solve them with near-perfect accuracy. This indicates that the tasks are within the realm of human cognitive capabilities.Limitations. In this work, we introduce four novel tasks and conduct a comprehensive evaluation of diverse large language models (LLMs) and vision-language models (VLMs), presenting a controlled and thorough evaluation of spatial reasoning capabilities. While our analysis is detailed and extensive, it remains primarily empirical. We believe that embarking on a formal theoretical study, despite its challenges, would significantly enrich our understanding of pre-trained multimodal language models. Moreover, our focus has been on in-depth analysis rather than on developing new training strategies or adaptation algorithms to enhance spatial reasoning. Recognizing these points, we identify them as valuable directions for future research.Societal impact. Regarding the positive societal impact, our evaluations and observations could catalyze the development of new algorithms that enhance the spatial reasoning abilities of LLMs and VLMs. Improved understanding of these models has the potential to significantly benefit sectors requiring robust spatial understanding and navigation. This could lead to more reliable and efficient systems that enhance safety and user experience. As for potential negative societal impacts, our work primarily involves empirical evaluations using synthetic datasets designed to probe spatial reasoning. Therefore, we do not anticipate any direct negative societal impacts arising from our current research.Our experiments are conducted on NVIDIA A100 GPUs. Our implementation is based on Python 3.10 and PyTorch 2.1.2.The hyperparameters discussed in this paper pertain to the decoding strategies of each model. For deterministic decoding, we adhere to the default settings specified for each model. The models employing deterministic decoding (argmax) include Bunny-Phi-2-SigLIP, CogAgent, CogVLM, InstructBLIP-Vicuna-13B, and InstructBLIP-Vicuna-7B. For non-deterministic decoding, we utilize the default hyperparameters provided by Hugging Face (for instance, Top-P is set at 0.9 and the temperature at 0.2 for LLaVA-1.6). Error bars for the main results (Figure 5 ) are obtained with three independent runs.For most open-source models, we use the checkpoints provided by Hugging Face as shown in Table 3 .For Bunny variants (Bunny-Phi-1.5-EVA, Bunny-Phi-1.5-SigLIP and Bunny-Phi-2-EVA), we use merged weights following instructions in https://github.com/BAAI-DCAI/Bunny/.Table 3 : Model checkpoints from Hugging Face.In the main text, we abbreviated the textual descriptions in Figure 1 , Figure 2 due to space constraints. In this section, we provide the full descriptions for each task to facilitate a better understanding of spatial reasoning tasks. The complete illustrations are displayed in Figure 14 for Spatial-Map and Figure 15 for Maze-Nav, respectively. Three questions (termed Q1 to Q3) are associated with each sample in tasks Spaitial-Map, Maze-Nav, and Spatial-Grid.Here is a Maze represented in ASCII code (LHS), where the symbols have the following meanings:-# represents walls that are impassable barriers.-" " (space) represents the navigable path within the maze, but not necessarily the correct path to the exit.-S denotes the starting point of the maze.-E marks the endpoint or the exit of the maze. X marks the specific you should follow to navigate from the starting point 'S' to the endpoint 'E'.-The objective is to navigate from S to E by following the path marked by X. Movement is allowed in any of the four cardinal directions (up, down, left, right) along the spaces, but solve the maze, follow the designated by X.-A right turn is defined as a change in movement direction that is 90 degrees clockwise relative the previous direction.-A left turn is defined as a change in movement direction that is 90 degrees anticlockwise relative to the previous direction.<Img> The figure Maze, where the colored blocks have the following meanings: -Black blocks represent walls that impassable barriers.White blocks represent navigable paths within the maze, but not necessarily the correct to the exit. -Green block denotes the starting point (S) of the maze.-Red block marks the endpoint or the exit (E) of the maze.-The objective is to navigate from S to E by following the path marked by Blue. Movement is allowed in any of the four cardinal directions (up, down, left, right) along the White blocks, but to solve the maze, follow the path designated by Blue blocks.-A right turn is defined as change in movement direction that 90 degrees clockwise relative to the previous direction.-A left turn is defined as movement direction that is 90 anticlockwise relative to the previous direction.<Img> The figure represents a Maze, where the colored blocks have the meanings: -Black blocks represent walls that are impassable barriers.-White blocks navigable paths within the maze, but not necessarily the correct path to the -Green block denotes the starting point (S) of maze.-Red block marks the endpoint or the exit (E) of the maze.-The objective is to navigate from S to E by following the path marked by Blue. Movement is allowed in any of the four cardinal directions (up, down, left, right) along the White blocks, but to solve the maze, follow the path designated by Blue blocks. The same Maze can be represented in ASCII code (LHS), where the symbols have the following meanings: -# walls impassable barriers.-" (space) represents the navigable path within the maze, but not necessarily the correct path to the -X marks the specific route you should follow to navigate from the starting point 'S' to the endpoint 'E'.-S denotes the starting point of the maze.-E marks the or the exit of the maze.-objective is to navigate S to E by following the path marked by X. Movement is allowed in any of the cardinal directions (up, down, left, right) along the but to solve the maze, follow the path designated by X. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: This is an interesting analysis paper. The paper develops benchmarks to evaluate spatial reasoning in large language models (LLMs) and vision-language models (VLMs), revealing that VLMs often underperform compared to LLMs, even when additional visual input is provided. An interesting finding is that VLMs without the visual input perform better than VLMs with the visual input and LLMs with text input alone, highlighting the complexity of spatial reasoning in multimodal tasks.

The reviewers raised concerns about the evaluation methods and comparisons, particularly regarding the limited improvement of VLMs when visual inputs are available. However, the authors provided detailed rebuttals, introducing new baselines and conducting additional experiments that effectively clarified the concerns. The discussion around how redundancy between visual and textual information can be leveraged to improve performance also provided valuable insights.

Overall, the AC believes this is a comprehensive study of spatial reasoning in multimodal models, though the conclusion of treating vision as first-class modality might seem obvious to many vision and language researchers. The AC would highly suggest that the authors add more discussion (and potentially some probing experiments) regarding how to improve the spatial reasoning abilities of VLMs, especially when both modalities are available. Exploring ways to optimize the use of visual input for spatial reasoning would significantly strengthen the paper’s contributions to the field.

Reviews: review: paper_summary: paper_summary: This paper develops a novel benchmark to understand spatial reasoning ability of LLM and VLM. Using such a benchmark, authors conduct experiments to evaluate models' performance, and reveal several results.
strength_weakness: strength_weakness: 1. The design rationales of the three benchmarks are not included or discussed. "Spatial-Map" and "Maze-Nav" look reasonable but "Spatial-grid" seems ill-posed. Would we really meet any similar application scenario in the real world?

2. In the experiments of "The impact of input modality", LLMs and VLMs take very different inputs in the benchmarks, with no part overlapping. In this case, it is unsure whether such a comparison is fair or even valid.

3. It is unclear why the results of "Spatial Map" are not provided in main paper or appendix. In addition, the performance of some VLMs are also missing. Given that it can be already observed that there exist some outliers, it is important to show full results of the study to ensure the validity of observations.

4. LLaVA-1.6-34B is the most performant model in this model family. However, in Figure 7-9, it consistently under-performs compared to other LLaVA models on Spatial grid. I would like to see authors' comments on these finding. What might be the cause to the weakness of LLaVA-1.6-34B?

5. This paper overall provides quantitative results of current VLMs without deeper analysis. It remains unclear what on earth causes the weakness of these VLMs. What are insights or takeaway to improve VLMs?
questions: questions: See weakness.
limitations: limitations: No negative societal impacts of the study.
paper_summary: paper_summary: The paper proposes a set of synthetic tests to compare the spatial understanding in VLMs and LLMs. The tests include Spatial-Map, Maze-Nav, and Spatial-Grid, which all include an image, paired with text that describe the image, and a question. Using the synthetic data, the VLMs and LLMs are studied using different modalities, including VQA, TQA, and VTQA, where the models are asked to answer questions based on only the image, only the text, and both image and text. Findings include that the current VLMs are strongly biased towards texts (blind); VLMs outperform LLMs in TQA. Both open models and proprietary models have been tested.
strength_weakness: strength_weakness: 1. All datasets contain only 2D synthetic images, which is substantially different from real images. At least the images can be made more realistic using computer graphics, like [4,5].
2. The synthetic nature of the tasks introduces artifacts. Good spatial reasoning ability on this task may not generalize to real ones. The results may be highly dependent on whether the model has been trained on a similar task. For example, the performance of GPT-4O almost doubles that of Gemini in this paper, while the difference of these models on real images is not this significant.
3. The observation that VL models are highly biased towards text, and are “blind” visually, has been studied long in the VQA community, even before LLMs [1,2,3, etc.]. The related literature should be discussed.
[1] Eyes wide shut? exploring the visual shortcomings of multimodal llms
[2] Explicit Bias Discovery in Visual Question Answering Models
[3] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
[4] Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
[5] Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning
questions: questions: 1. How does the findings in this paper differ from previous literatures like [1,2,3]. Are the findings different, or are they discovered in a new way as proposed in this paper?
2. Any discussions about the synthetic-real gap?
3. In Spatial-Map, without text descriptions, it looks like the VLMs may describe spatial using “left/right/up/down” instead of “west/east/north/south” - will this happen, and how is it addressed?
limitations: limitations: N/A
paper_summary: paper_summary: The authors make a benchmark to test the spatial reasoning of multimodal language models (MLMs) using synthetic data consisting mostly of diagrams, mazes, and charts. They benchmark a lot of existing MLMs. Their key findings are: 1) Spatial reasoning is limited in most MLMs. 2) MLMs rely heavily on textual information and not the visual, and 3) interestingly, the VLMs are better at utilizing spatial text information than LLMs suggesting multimodal training helps understand the visual natural of language even though VLMs tend to rely less on visual information during test time.
strength_weakness: strength_weakness: - Some of the claims may be too broad to make based on results of just this dataset. For instance, the models using more textual information than visual. This could simply be because the visual inputs in this benchmark are of a starkly different domain than what the models are trained on (real images instead of synthetic graphs and mazes and charts), whereas the domain gap in language is much lesser since the language tokens remain the same. Hence, we can only make the statement that the model is using more textual cues for such kinds of data. It is unclear if that is also the case on real image data. 

- There are many ways to test spatial reasoning using real VQA style datasets like BLINK, MMBench etc. So , it is unclear why they came up with these complicated tasks. Is there any real-world relevance for such tasks? 

- The authors mention they opt for synthetic data due to controllability, but all the analysis they do can be simply done on real image QA datasets like GQA.  For instance, for checking vision vs textual reliance, just feed in scene graphs vs the image to an MLM from GQA. Or, for mismatched image and text, randomly pair image and QA. At least a discussion on why such a benchmark is needed is missing. What special analysis in the paper is enabled by this benchmark that couldn't be enabled by an already existing benchmark like Visual Genome, GQA, BLINK etc.

- Some settings could be explained more clearly in the paper. For instance, for the vision-only input - I am guessing the authors mean that they do not describe the image in text, but the question is still in text, correct? 

- A nit pick, but some grammar could be improved. e.g., intro, the first line (line 18):" had a transformation effect on research" - a transformational effect not affect. Other grammatical errors are also occasionally present in the paper. 

-  Another nit pick, but this paper seems more fitting for the datasets and benchmarks track.
questions: questions: See above concerns. 

Overall I like the direction of the paper. But some analysis justifying the need of such type of synthetic data (mazes and custom diagrams) instead of real images or actual graphs (useful for scientific figures) would be nice.
limitations: limitations: The authors discuss some limitations of models, but not the benchmark. What can this benchmark power, and what are some analyses that the benchmark currently cannot power that would be good to have?
paper_summary: paper_summary: This work presents 3 new synthetically generated VQA evaluation benchmarks and conducts a comprehensive evaluation of the limitations of current  VLMs in spatial reasoning tasks. These tasks include spatial relationships, navigation, position understanding, and counting. The authors demonstrate using their introduced benchmarks that VLMs have limited performance on tasks requiring detailed visual understanding and reasoning of images.
strength_weakness: strength_weakness: Spatial Understanding and Evaluation: The use of synthetic data in the evaluation, while valuable, may introduce confounding factors unrelated to the task of spatial understanding/reasoning. 
* For instance, a vision-only model with limited OCR capabilities may perform poorly on the spatial-map task, regardless of its actual spatial reasoning ability. It is challenging to disentangle the contributions of OCR performance from spatial understanding in such benchmark.
* The synthetic maze-navigation and spatial-grid images might be out-of-distribution for some models, especially open-source ones. This could also explain why a noise image improves the accuracy of Llava-1.6 on maze-navigation tasks, while a similarly out-of-distribution maze image does not necessarily harm or consistently improve the  performance. It is important to consider the potential impact of out-of-distribution data on open-source models. By looking at fig 12 it looks the performance of the latest gpt4o on vision-only is similar to vision-and-text. That said, I still believe the work remains valuable and important.

Human Performance Comparison:
* The claim on line 172 that "the performance of these models still lags significantly behind human levels" would benefit from concrete numbers to substantiate it. Providing specific human performance metrics on these tasks or citations here could would make the claim more robust. Alternatively, softening the statement to reflect the need for further comparison could be considered.

Sampling Strategies and Prompting Techniques:
* Exploring different sampling strategies or reasoning/prompting techniques could yield valuable insights into the model's performance. It would be beneficial to include discussions on how these variations impact the results. To my understanding only a simple prompting technique is utilized "step-by-step explanation". Was there any particular reason to append the selected prompt vs others. 
In line 157, the paper also mentioned "For each model, we adopt the default configurations and decoding strategies, e.g., argmax for deterministic decoding and top-p for non-deterministic decoding. " It would be very useful to be more specific, e.g. what top-p and temperature was used.  How "deterministic" was the deterministic decoding in API based models? 

VQA Evaluation Benchmark Details:
* The paper would benefit from clearly specifying the size of each VQA evaluation benchmark in terms of the number of samples or data points. This information is crucial for understanding the scope and scale of the evaluations conducted. I might have missed it but I couldn't find it in the paper. I would encourage the authors to particular address this in rebuttals.
questions: questions: please see above
limitations: limitations: Yes. It is discussed and properly addressed.

Comments: title: Thanks for the response
Thanks for the response
Thanks for the response
comment: Dear Reviewer i8Xy,

Thank you for affirming our rebuttal, positive recommendation, and valuable suggestions! We acknowledge the importance of real datasets and are committed to expanding both the diversity and scale of Spatial-Real. While we recognize the value of natural data, we also believe that synthetic tasks serve an important role. As acknowledged by Reviewer JHoY, these tasks can serve as cognitive tests that evaluate basic capabilities, which are relevant to broader real world applications. This approach has precedence in fields like IQ testing, where evaluating foundational cognitive skills is valuable.

We will open source our benchmark to facilitate future research in this area.


Best,

Authors
Dear Reviewer JHoY,
 
Thank you again for your valuable feedback. We sincerely appreciate your support of our work.
 
Q: What is new in Spatial-Real compared to existing image-based datasets that also have QAs on spatial understanding and counting?

In Spatial-Real, the form of QAs on spatial reasoning (object counting, relation, and position understanding) is similar to those in existing VQA datasets. However, the novel difference is that our dataset construction allows for a comprehensive study of spatial reasoning QAs across different modality settings—TQA (LLM), TQA (VLM), VQA, and VTQA, where images or texts can be optionally provided. This is largely unexplored in Vision and NLP communities. We aim to bridge this gap by offering a unique benchmark that we hope will be valuable and inspire future research, as new multimodal foundation models emerge that can handle interleaved text and image inputs with longer context lengths.


Best,

Authors
I thank the authors for providing the rebuttal. The rebuttal discusses the differences with prior works. I also appreciate the effort for experimenting with real dataset - it is interesting to see that the modality gaps becomes even larger on real image.  

I am still concerned about the artificially created toy tasks in the paper. I think more experiments with real data, or semi-real data like rendered images using graphics (e.g. CLEVR) that are 3D, will make the paper much stronger. 

I will keep my score as 5.
These responses are helpful and the introduction of a split with real images also broadens the scope of the paper. 
While some of the tasks seem contrived, performance on such tasks can be seen as basic cognitive tests, and as the authors point out, they can have real downstream applications, such as navigation given maps, etc. However, evidence that improving on this benchmark will actually improve navigation performance is missing. 

Some of the concerns are that the Spatial real example in the rebuttal seems to go back to simple image-based QAs again, and it's unclear how it is "spatial reasoning". It is also unclear how it is different from the QAs in existing datasets that also have spatial, counting, etc QAs. 

However, I still lean towards recommending accepting the paper since it will help drive research into the cognitive abilities of multimodal foundation models.
Dear Reviewer WNLP,

Thank you for your time and thoughtful feedback on our paper. We greatly appreciate your positive recommendation!

Best,
Authors
I thank the authors for the very comprehensive  rebuttal and new experiments added to the papers.
I have also read and appreciate other reviewers comments and the authors response. I think this is an important paper, technically solid with potential moderate-to-high impact. I keep my score of 6, and recommend acceptance.
Dear Reviewers,

Thank you again for your valuable time and insightful comments! 

In response, we have provided detailed answers, additional ablation studies, and a new real-world benchmark with thorough experimental results. As we are halfway through the discussion stage, please let us know if your concerns have been addressed. We are more than happy to address any additional questions before the discussion period officially concludes.

Best,

Authors of Paper 8807