Title: Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Spatial Reasoning

Evaluation

Vision-language Models

Language Models
Abstract: Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning—a fundamental component of human cognition—remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence. Our code is available at https://github.com/jiayuww/SpatialEval.

Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Abstract

Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning-a fundamental component of human cognition-remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence. Our code is available at https://github.com/jiayuww/SpatialEval .

1 Introduction

The recent breakthroughs in foundation models have had a transformative effect on research and industry, and we have seen these models rapidly integrated into products and new businesses that are shaping people's lives for the better. This sea change was initially driven by large language models (LLMs), which have shown at times near unbelievable, human-level performance across a wide range of tasks. Over the past year, many of these models have been extended to handle images in addition to text, leading to a significant increase in vision-language models (VLMs), especially multimodal large language models (MLLMs), which demonstrate groundbreaking performance in image-related tasks as in the text domain.

However, the reality is not quite as rosy as advertised. While these models have been instrumental in advancing the state-of-the-art in complex reasoning tasks such as common sense reasoning, mathematical problem solving, and scientific question answering [17, 30, 41, 72, 75], they have not been as effective for a number of problem domains. In particular, as we will show in this paper, they have limited performance on tasks that require detailed visual understanding and reasoning of images.

Visual understanding and reasoning-an intrinsic part of human perception and cognitive ability-have been largely under-explored when it comes to LLMs and VLMs. In fact, it is often argued that the visual sense is the dominant sense in people, yet when it comes to current models, it seems quite secondary. Spatial reasoning, in particular, is fundamental to everyday human activities such as navigating environments, understanding maps, and manipulating objects. It encompasses skills that are crucial for both survival and higher-order cognition, including the ability to navigate through space, recognize patterns, and deduce relationships from spatial configurations.

In this paper, we propose SpatialEval , a novel benchmark containing four tasks ( Spatial-Map , Maze-Nav , Spatial-Grid , and Spatial-Real , Section 2.1) to explore the performance of LLMs and VLMs on diverse aspects of spatial reasoning, including relationship, navigation, position understanding, and object counting. Humans excel at such tasks, making them essential capabilities for intelligent systems to emulate for safe and effective deployment in the real world.

Our dataset, however, is constructed with a key twist - each problem in our benchmark has an image and a text representation that is sufficient for answering each spatial understanding question. We denote the use of these sources as VQA, which is the standard task of visual-question answering that consists of a vision-only input and a question, TQA, text-only input and a question, and VTQA, a combination of the previous with vision and text input.

Weconduct a systematic and comprehensive evaluation of a wide range of open-source and proprietary LLMs and VLMs. We perform in-depth analysis and unveil several novel and surprising results that challenge the current understanding of how these models process spatial information:

Spatial reasoning remains challenging : VLMs frequently struggle with spatial reasoning tasks, with some competitive models performing worse than random guessing.

Visual vs. textual inputs : Without detailed textual descriptions, multimodal models rarely surpass the performance of their LLM backbones when relying solely on visual inputs. This underscores the critical role of text in enhancing model performance on spatial reasoning.

Reduced reliance on visual information : When both textual and visual inputs are provided, multimodal language models tend to rely less on the visual component if sufficient textual clues are available.

Textual performance of VLMs : VLMs often outperform their LLM counterparts with text-only inputs, indicating that the language model backbones within VLMs benefit from multimodal training, despite a lack of similar benefits from the visual components.

We surmise the limitations in VLMs' spatial understanding stem from the overly simplistic handling of visual information in current architectures and training pipelines. We believe the contributions of this paper will drive changes in model design, accelerating improvements that could unlock more robust spatial reasoning capabilities, and help bridge the gap toward human-like intelligence.

2 Dataset and Task Construction

2.1 Dataset Setup

To evaluate the spatial reasoning abilities of LLMs and VLMs, we construct four diverse tasks including spatial relationships, navigation, position understanding, and counting. To systematically study the impact of modality, we design three types of input formats for each task: (1) TQA (Textonly): the input is purely textual and contains all necessary information for a person to answer the questions. (2) VQA (Vision-only): the input consists solely of an image, which provides sufficient details for a person to easily answer, a format also referred to as Visual Question Answering (VQA) in the literature. (3) VTQA (Vision-text): the input includes both an image and its textual representation with detailed descriptions, rendering the information in both modalities redundant. We evaluate LLMs using Text-only inputs and VLMs using Text-only, Vision-only, and Vision-text inputs on the same set of questions (Table 1). The synthetic tasks are curated based on the following key guidelines: (1) Avoidance of data leakage-since LLMs are pre-trained on web-scale data, it is crucial to ensure that the test data have not been seen during training; (2) configurability-being configurable allows for controlled experiments and extends easily to additional tasks; (3) scalability-the ability to scale the number of test samples enhances the statistical significance of experimental results.

TQA

Spatial-Map

(Text-only)

Consideramapwithmultipleobjects:

Whale's Watchesisin the map.Brews BrothersPubisto theSoutheast ofWhale's Watches.

Himalayan Hot Springs is to the Southeast of Whale's Watches..Gale Gifts is to the

Northeast of Unicorn Umbrellas. Gale Gifts is to the Southwest of Himalayan Hot Springs.

Whale's Watches

VQA

<img>The figure represents a map with multiple objects.Each object is associated with a

(Vision-only)

name as shown in the figure.

Brews Brothers Pub

口

VTQA

(Vision-text)

<img> The figure represents a map with multiple objects. Each object is associated with a

name asshowninthefigure.Thesamefigurecanbedescribed asfollows:

Whale's Watches is in the map. Brews Brothers Pub is to the Southeast of Whale's Watches.

Unicorn Umbrellas

HimalayanHotSprings isto theSoutheastof Whale'sWatches..Gale Giftsistothe

Dragonfly Drones

Southwest of Himalayan Hot Springs.

Q: In which direction isWhale's Watches relativeto Dragonfly Drones?

Questions

A.Northwest

B.Southwest

C.Southeast

D. Northeast

Q:Whichobject is intheSouthwestofGaleGifts?

A.DragonflyDrones

B.UnicornUmbrellas

C.Himalayan Hot Springs

D.BrewsBrothersPub

Q:How many objects are in theSouthwest of Himalayan Hot Springs?

A.3

B.4

C.1

D.0

Spatial-Map. Understanding the spatial relationships among objects on a map is a fundamental aspect of human cognitive abilities. To simulate this environment, we create a map-like dataset termed Spatial-Map with K objects, where K is configurable. Each object is associated with a unique location name, such as Unicorn Umbrellas and Gale Gifts. To study the impact of modality, the textual representation of each input consists of pairwise relations such as Brews Brothers Pub is to the Southeast of Whale's Watches. An example with K = 6 is shown in Figure 1, with Text-only, Vision-only, and Vision-text inputs. These questions include asking about the spatial relationships between two locations and the number of objects that meet specific spatial criteria.

Maze-Nav. Navigation through complex spaces is essential for intelligent systems. To evaluate such abilities, we have developed a maze-like dataset named Maze-Nav . Visually, each sample can be

TQA

Maze-Nav

(Text-only)

Here is a Maze represented in AsCll code (LHS) where the symbols have the following

#

pathwithinthe maze,..A right turn isdefinedasa changeinmovement direction that is90

degrees clockwise relative to the previous direction. A left turn is defined as a change in

movementdirectionthatis90degreesanticlockwiserelativetotheprevious direction.

VQA

<Img> The figure represents a Maze, where the colored blocks have the following meanings:

(Vision-only)

Black blocks represent walls that are impassable barriers.White blocks represent navigable

paths within the maze,but not necessarilythe correct pathto the exit..A right turn is defined

asa changein movement direction that is90 degrees clockwiserelative tothe previous

direction...

VTQA

(Vision-text)

<Img> The figure represents a Maze, where the colored blocks have the following meanings:

Blackblocksrepresentwallsthat areimpassablebarriers..ThesameMazecanberepresented

###

#####×#

#

movement direction thatis90 degrees anticlockwiserelative tothepreviousdirection.

degrees clockwise relative to the previous direction.A left turn is defined as a change in

Q: How many right turns are there in the provided path from S to E?

Questions

A.4

B.3

C.7

D.2

Q: How many total turns are there in the provided path from S to E?

A.4

B. 1

C.9

D.5

A.Yes

B.No

represented as colored blocks where different colors signify distinct elements: a green block marks the starting point (S), a red block indicates the exit (E), black blocks represent impassable walls, white blocks denote navigable paths, and blue blocks trace the path from S to E. The objective is to navigate from S to E following the blue path, with movement permitted in the four cardinal directions (up, down, left, right). Alternatively, each input can be depicted in a textual format using ASCII code. An example is illustrated in Figure 2, featuring Text-only, Vision-only, and Vision-text inputs. We construct this task based on an open-sourced library [26]. The questions asked include counting the number of turns from S to E and determining the spatial relationship between S and E. While such questions are easy for humans, we will show in Section 3 that they still pose significant challenges for modern multimodal language models.

TQA

Spatial-Grid

(Text-only)

Considera 5x5 grid (5 rows and 5 columns) containing

elephant | cat

I giraffe|elephant| cat

various animals,where each1x1cell is considered a

Sop

|rabbit

I giraffe | cat

Sop1

blockandeachblockcontainsananimalfrom['cat',

rabbit

Sopl

|cat

|elephant|rabbit

'dog,'elephant','giraffe','rabbit].The distribution of

Sop

|elephant|giraffe|cat

|cat

animals in the grid is asfollows(LHS)

rabbit

Sop1

I giraffe| rabbit

|rabbit

VQA

(Vision-only) <lmg> The figure represents a 5x5 grid (5 rows and 5 columns) containing various animals,

3

where each 1x1 square is considered a block and each block contains an animal from [cat,

'dog,'elephant','giraffe','rabbit'].

VTQA

5

(Vision-text)

<lmg> Thefigure represents a 5x5 grid(5 rows and 5

elephant | cat

I giraffe|elephant| cat

columns)containingvariousanimals,whereeach1x1

1

2

3

4

5

Sop

|rabbit

Igiraffe |cat

sopl

square is considered a block and each block contains an

Sop

rabbit

sopl

|elephant|giraffe|cat

|cat

|elephant |rabbit

|cat

animal from ['cat,'dog,'elephant','giraffe,'rabbit'].

The same figure canberepresented intextual form as

rabbit

sopl

I giraffe | rabbit

|rabbit

follows (LHS)

Q:Howmanyblocks contain rabbit?

Questions

A.8

B.2

C.7

D.6

Q: What is the animal of the block located at the top-left corner (first row, frst column) of the grid?

A.rabbit

B. giraffe

C.cat

D.elephant

Q:What is the animal of the block located at the first row, second column of the grid?

A.rabbit

B.cat

C. giraffe

sop'0

Spatial-Grid. To investigate spatial understanding within structured environments, we introduce a grid-like dataset named Spatial-Grid , contrasting with the Spatial-Map where objects are positioned arbitrarily. Visually, each input consists of a grid of cells, each containing an image ( e.g., a rabbit). An example is illustrated in Figure 3. Alternatively, this grid can also be represented in a purely textual format; for instance, the first row might be described as: elephant | cat | giraffe | elephant | cat. The evaluations focus on tasks such as counting specific objects ( e.g., rabbits) and identifying the object located at a specific coordinate in the grid ( e.g., first row, second column).

Spatial-Real. To extend the evaluation of spatial reasoning beyond synthetic environments, we introduce Spatial-Real , a task built on the Densely Captioned Images (DCI) dataset [67], where each image has a detailed caption with more than 1,000 words on average. As DCI does not contain questions, we curate multiple-choice questions regarding spatial reasoning (object counting, relation, and position understanding) and annotate the answers. An example is shown in Figure 4. We provide detailed analysis on Spatial-Real in Appendix E.

2.2 Models

Weconsider a wide range of competitive open-source language models with different scales, including Phi2-2.7B [35], the LLaMA family (LLaMA-2-7B, LLaMA-2-13B, and LLaMA-3-8B) [65], Mistral7B [27], the Vicuna family (Vicuna-7B-1.5 and Vicuna-13B-1.5) [11], and Nous-Hermes-2-Yi-34B. For multimodal language models, we consider the Bunny family (Bunny-Phi-2-SigLIP, BunnyPhi-1.5-SigLIP, Bunny-Phi-2-EVA, and Bunny-Phi-1.5-EVA) [22], CogVLM [69], CogAgent [23], InstructBLIP family (InstructBLIP-Vicuna-7B and InstructBLIP-Vicuna-13B) [13], and LLaVA family (LLaVA-1.6-Mistral-7B, LLaVA-1.6-Vicuna-7B, LLaVA-1.6-Vicuna-13B, and LLaVA-1.6-

A.Two

B.One

Spatial-Real

Textual Description

Thefollowingcanbeseenintheimage:talltree directlyrightoftheorangetruckbedandoverthe treeoverthebackendofthebrowntruckbed. Open overcastsky overeverythingontheground. Rightgroupoftreeleavesoverthe orangetruck bedandbetweenthecranesthataresuspending it.Window area ofthedriver'ssidedoorofthe whitetruckontheleftofthebrowntruckbed... craneholdinguptheorangetruckbedtothe rightofanothercrane...

Please answer the following question based on theprovided information.

OVOOO

Q: How many cranes are involved in lifting theorangetruckbed?Availableoptions:

A.Two

B.One

C.Three

D.Four

34B) [38]. We also evaluate the proprietary models: Open AI's GPT-4V, GPT-4o, GPT-4, Google Gemini Pro 1.0, and Anthropic Claude 3 Opus.

Evaluation. As each question contains four options, we use accuracy as the main evaluation metric. The same user prompt is appended at the end of each question: First, provide a concise answer in one sentence. Then, elaborate on the reasoning behind your answer in a detailed, step-by-step explanation . For each model, we adopt the default configurations and decoding strategies, e.g., argmax for deterministic decoding and topp for non-deterministic decoding. For non-deterministic decoding, the results are averaged over three independent runs for open-source models. For proprietary models, due to their limited availability and increased compute time and cost, we only perform one run. We summarize the terminologies regarding input modalities for LLMs and VLMs in Table 1:

We describe the Text-only, Vision-only, and Vision-text input modalities based on how we feed the image information to the models. Vision-only input means the image is fed directly to the models without textual description, while all questions are presented in text.

3 Main Results and Analysis

Spatial reasoning remains surprisingly challenging. The evaluation results on open-source models on Spatial-Map , Maze-Nav , and Spatial-Grid are shown in Figure 5. For each task, the reported accuracy is averaged over all questions. For vision-language models, we choose the Vision-only input format, commonly used in Visual Question Answering (VQA). We use a dashed red line in each figure to indicate the expected accuracy if answering by random guessing. Our findings reveal several notable insights: (1) Vision-only inputs: Despite the simplicity of these tasks for humans, most competitive multimodal models perform at levels similar to or barely above random guessing.

0.0

0.1

0.2

0.3

0.4

LLaMA-2-13B

LLaMA-2-7B

LLaMA-3-8B

Mistral-7B

Nous-Hermes-2-Yi-34B

Phi-2

Vicuna-13B-1.5

Vicuna-7B-1.5

LLaVA-1.6-34B

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-7B

Qwen-VL

Bunny-Phi-1.5-EVA

Bunny-Phi-1.5-SigLIP

CogAgent

CogVLM

InstructBLIP-Vicuna-13B

InstructBLIP-Vicuna-7B

Spatial-Map

0.0

0.1

0.2

0.3

0.4

Maze-Nav

0.0

0.2

0.4

0.6

Spatial-Grid

Vision-Language Model

(2) Text-only inputs: While the textual input includes essential spatial information, it generally does not significantly enhance the spatial reasoning capabilities of competitive models. An exception occurs in the Spatial-Grid task, where Llama-3 achieves an accuracy of 71.9%, followed by Mistral-7B-Instruct at 62.1%, both notably surpassing random guessing. Despite these successes, the performance of these models still lags significantly behind human levels. These results underscore the need for further development of techniques tailored to spatial understanding and reasoning.

The impact of input modality. To investigate the impact of modality, we compare the performance of a large language model (LLM) and a vision-language model (VLM) with the same language backbone. We consider the VQA (Vision-only) format for vision-language models. The results are shown in Figure 6. Each vertex on the spider plot represents the average accuracy of a (VLM, LLM) pair. We observe that on Spatial-Map and Spatial-Grid , the majority of VLMs yield worse performance compared to their LLM counterpart, despite having an additional visual encoder. For example, on Spatial-Grid , Mixtral-7B achieves an average accuracy of 62.1%, while LLaVA-v1.6Mistral-7B only yields an accuracy of 47.1% (15% ↓ ). Detailed results can be seen in Appendix F.

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Mistral-7B

0.40

CogAgent

0.30

0.35

LLaVA-1.6-34B

CogAgent

0.30

0.35

LLaVA-1.6-34B

CogAgent

0:6

LLaVA-1.6-34B

0.25

0.25

O.3

0.15

0.15

0.05

LLaVA-1.6-

Vicuna-7B

0.05

LLaVA-1.6-

Vicuna-7B

0.1

Vicuna-7B

LLaVA-1.6

CogVLM

CogVLM

CogVLM

InstructBLIP-

lnstructBLIP-

InstructBLIP-

InstructBLIP-

InstructBLIP-

InstructBLIP-

Vicuna-7B

Vicuna-13B

Vicuna-7B

Vicuna-13B

Vicuna-7B

Vicuna-13B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-13B

AvgAcCofLLM(TQA)

AvgAcCofVLM(VQA)

(b)TQAvs.VQAonMaze-Nav

4 Delving Into Spatial Reasoning for Vision-Language Models

4.1 Seeing Without Understanding: The Blindness of Multimodal Language Models

To better understand how VLMs process visual information, we conduct a series of controlled experiments in the VTQA (Vision-text input) setting. For each sample, we replace the original image input (that matches the textual description) with either: (1) No Image : only keep the textual input without the image input, (2) Noise Image : a Gaussian noise image irrelevant to the task, and (3) Random Image : description, as shown in Figure 7.

a random image from the dataset that does not match the textual

0.2

0.3

0.4

0.5

0.6

0.7

LLaVA-1.6-34B

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-7B

Qwen-VL

0.401

0.390

0.588

0.478

0.460

0.349

0.483

0.591

0.731

0.581

0.414

0.425

Spatial-Grid

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.222

0.378

0.242

0.271

0.339

0.327

0.209

0.403

0.317

0.294

0.351

0.380

Maze-Nav

Original Image

No Image

VLMs exhibit improved performance when visual input is absent. We conducted experiments by entirely removing the Original Image and relying solely on the textual description. The results are shown in Figure 8. For each task, we report the accuracy averaged across all questions. Remarkably, the absence of visual input leads to better performance across a range of VLM architectures. For instance, the performance of LLaVA-1.6-34B on the Spatial-Grid task improved by 20.1% when no image was presented compared to scenarios with the original image. This observation underscores that when textual information alone can address the questions, additional visual inputs do not necessarily enhance, and may even hinder the performance, a sharp contrast to human capabilities where visual cues significantly aid in understanding. The removal of visual input forces the models to utilize the textual information to solve spatial reasoning tasks.

0.2

0.3

0.4

0.5

0.6

InstructBLIP-Vicuna-13B

InstructBLIP-Vicuna-7B

LLaVA-1.6-34B

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Vicuna-13B

Qwen-VL

0.401

0.437

0.281

0.390

0.588

0.478

0.349

0.437

0.481

0.325

0.348

0.615

0.511

0.365

Spatial-Grid

0.20

0.25

0.30

0.35

0.40

0.222

0.267

0.389

0.378

0.242

0.271

0.327

0.248

0.241

0.416

0.381

0.253

0.336

0.378

Maze-Nav

Original Image

Noise image can improve the performance. We replace the Original Image with a Noise Image while retaining the original textual description. The results are shown in Figure 9. Consistent with

0.2

0.3

0.4

0.5

0.6

InstructBLIP-Vicuna-13B

InstructBLIP-Vicuna-7B

LLaVA-1.6-34B

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Vicuna-13B

Qwen-VL

0.401

0.437

0.281

0.390

0.588

0.478

0.349

0.301

0.465

0.286

0.361

0.416

0.363

0.339

Spatial-Grid

0.20

0.25

0.30

0.35

0.40

0.222

0.267

0.389

0.378

0.242

0.271

0.327

0.227

0.265

0.396

0.383

0.243

0.281

0.364

Maze-Nav

Original Image

the findings in Original Image vs. No Image, using a noise image also improves the performance across various VLM architectures. For example, the accuracy of LLaVA-1.6-Vicuna-13B increases by 6.5% on the Maze-Nav task when the noise image is used as opposed to the original image. In contrast to the No Image setting, noise images provide limited visual cues. Nonetheless, the model tends to prioritize the textual information, especially when visual cues are not pertinent to the task.

Mismatched image-text does not necessarily hurt. To build on previous findings, we further investigate the effects of replacing the Original Image with a Random Image (illustrated in Figure 7). Unlike a noise image, a random image is task-related but may provide conflicting information compared to the textual description. Intuitively, one might expect that such random images would degrade VLM performance due to contradictory cues. However, as demonstrated in Figure 10, this expectation does not always hold true. For instance, Random Image in the Maze-Nav task leads to improved performance across various VLM architectures. This outcome implies that VLMs are not heavily reliant on visual information, particularly when adequate textual clues are provided.

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Mistral-7B

0.6

0.40

0.35

CogAgent

0.30

0.35

LLaVA-1.6-34B

CogAgent

0.30

LLaVA-1.6-34B

CogAgent

0.5

LLaVA-1.6-34B

0.25

0.25

O.3

0.15

0.15

LLaVA-1.6-

LLaVA-1.6-

LLaVA-1.6-

0.05

Vicuna-7B

0.05

Vicuna-7B

0.1

Vicuna-7B

CogVLM

CogVLM

CogVLM

InstructBLIP-

InstructBLIP-

InstructBLIP-

InstructBLIP-

InstructBLIP-

InstructBLIP-

Vicuna-7B

Vicuna-13B

Vicuna-7B

Vicuna-13B

Vicuna-7B

Vicuna-13B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-13B

AvgAccofVQA（Vision-onlyInput）

→AvgAccof VTQA(Vision-textInput)

(a) VQA vs.VTQA on Spatial-Map

(b)VQAvs.VTQAonMaze-Nav

(c)VQAvs.VTQAonSpatial-Grid

4.2 Leveraging Redundancy in Multimodal Inputs

Multimodal language models offer considerable versatility in handling multimodal inputs. While the visual input alone often provides sufficient details for humans to address spatial reasoning tasks with ease, we propose that VLMs significantly benefit from the inclusion of textual descriptions alongside visual data, even if this introduces substantial redundancy. We verify this hypothesis by comparing VQA (Vision-only input) and VTQA (Vision-text input) across diverse VLM architectures. The results are shown in Figure 11, where each vertex on the spider plot represents the average accuracy of a (Vision-only, Vision-text) pair based on the same VLM. For Spatial-Map and Spatial-Grid , we can clearly see that having the additional textual input (VTQA) enhances the performance compared

LLaVA-1.6-34B

LLaVA-1.6-34B

LLaVA-1.6-34B

CogVLM

0.25

0.30

0.35

0.48

LLaVA-1.6-

Vicuna-7B

CogVLM

0.25

0.35

0.40

LLaVA-1.6-

Vicuna-7B

CogVLM

0.5

0.6

0.7

Vicuna-7B

LLaVA-1.6-

0.15

0.15

0.3

0.05

0.05

0.1

Bunny-

Bunny-

Bunny-

Phi-2-SigLIP

Phi-2-SigLIP

Phi-2-SigLI

VA-1.6-

LLaVA-1.6-

LLaVA-1.6-

una-7B

Vicuna-7B

Vicuna-7B

LLaVA-1.6-

Bunny-

LLaVA-1.6-

Bunny-

LLaVA-1.6-

Bunny-

Vicuna-13B

Phi-2-EVA

Vicuna-13B

Phi-2-EVA

Vicuna-13B

Phi-2-EVA

AvgAccofText-only InputwithLLM(TQA)

AvgAccofText-only InputwithVLM(NoImg)

(a)TQAvs.NoImgonSpatial-Map

(b)TQAvs.NoImgonMaze-Nav

(c) TQA vs. No Img on Spatial-Grid

to only using images (VQA) across different VLM architectures. This suggests that textual inputs improve the accuracy of spatial reasoning in VLMs. We further compare TQA and VTQA in Appendix D. Detailed results are included in Appendix F.

Text-only input with LLM vs. Text-only input with VLM. Given the demonstrated efficacy of textonly inputs, we conducted an ablation study to compare LLMs and VLMs using text-only inputs. We consider VLMs that are capable of processing text without accompanying visual data. The results are illustrated in Figure 12. Except for CogVLM, the majority of VLMs outperform their corresponding LLM backbones. This suggests that the language model backbones in VLMs demonstrate enhanced spatial reasoning abilities through multimodal learning. Conversely, the addition of visual information does not necessarily provide further benefits.

0

0.2

0.4

0.6

0.8

Gemini Pro

1.0

Claude 3

Opus

GPT-4O

GPT-4V

GPT4

Spatial-Map

Text only

Vision-only

Vision-text

0

0.1

0.2

0.3

0.4

0.5

Gemini Pro

1.0

Claude 3

Opus

GPT-4O

GPT-4V

GPT4

Maze-Nav

Text only

Vision-only

Vision-text

0

0.2

0.4

0.6

0.8

1

1.2

Gemini Pro

1.0

Claude 3

Opus

GPT-4O

GPT-4V

GPT4

Spatial-Grid

Text only

Vision-only

Vision-text

4.3 Proprietary vs. Open-Source Models

As many recent benchmarks have shown that proprietary models generally outperform open-source models, it is important to understand if our observed trends hold with proprietary models. The performance of several top proprietary models (GPT-4, GPT-4V, GPT-4o, Gemini Pro 1.0, and Claude 3 Opus) are shown in Figure 13. We have the following salient observations: (1) A significant performance gap exists between SoTA open-source models and proprietary models, as expected. Furthermore, with both Text-only and Vision-text formats, GPT-4V and GPT-4o significantly outperform random guessing across all tasks. For instance, in the Vision-text format, GPT-4o achieves

an accuracy of 0.989 on Spatial-Grid (Table 7). (2) Yet, the trends we observed with open source models hold, VQA consistently under-performs compared to TQA and VTQA, for example, GPT-4V's performance improves by 25.6% on Spatial-Grid when switching from Vision-only to Vision-text input; and again no clear winner between TQA and VTQA (see Appendix D for further details), showing that proprietary models, even the new GPT-4o model, still do not appear to fully leverage visual inputs. We summarize the key findings of this work in Table 2.

5 Related Work

Large language models. Large language models (LLMs) have achieved outstanding performance across a diverse array of fields, including finance [34], bioinformatics [63], law [60], education [29], coding [24], and creative tasks [2, 36]. LLM architectures have gone through significant changes in recent years, with notable developments such as BERT [14], OPT [76], PaLM [12], Gemma family [62], Mistral family [27], GPT family [2, 10], Claude family [5], and LLaMA family [3, 64, 66]. These models have demonstrated emergent abilities and revolutionized numerous domains, supporting capabilities such as in-context learning [46, 51, 59], compositional reasoning [16, 20, 70], and taskspecific adaptation [57, 58, 71]. However, visual understanding and reasoning-an intrinsic part of human cognitive ability-remains largely under-explored for LLMs.

Vision-language models and multi-modal language models. The success of LLMs has propelled the adoption of the Transformer architecture [68] within the computer vision community, such as ViT [15], Beit [9], CLIP [54], MAE [21], Swin [42, 43], and DiT [53]. Building on the capabilities of powerful LLMs, multi-modal language models (MLLMs) such as Flamingo [4], LLaMA-Adapter [19, 74], LLava [37, 39], stable-diffusion [55], BLIP [32, 33], MiniGPT-4 [77], Qwen [7, 8], Gemini [61], MM1 [45] have significantly expanded the range of problems that can be addressed with improved reliability [49]. These models adeptly handle inputs from diverse modalities and have demonstrated remarkable performance on diverse tasks such as mathematical reasoning [44], image-text retrieval [47, 73], and visual reasoning [6, 18, 25, 28, 31, 40, 50].

Spatial understanding and reasoning. Spatial reasoning entails comprehending and manipulating spatial relationships, a task significantly more challenging than visual grounding [1, 28, 52]. Although progress in natural language processing, evaluations, and benchmarks on LLMs such as GPT-4 [2] and Claude3 [5] have predominantly focused on textual or relational reasoning. This focus often overlooks the intricate nature of spatial reasoning tasks. Notably, recent studies [17, 30, 41, 48, 56, 72, 75] have conducted thorough evaluations across a diverse range of tasks. Yet, they demonstrate a scant exploration of spatial reasoning capabilities, highlighting a gap in assessing this complex cognitive skill in current benchmarks. In particular, Zhang et al. [75] suggest that MLLMs primarily leverage textual cues rather than visual diagrams to solve math problems while focusing only on math problems. Yamada et al. [72] design simple navigation tasks and finds LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains, while our benchmarks are more complicated than their navigation tasks based on simple geometry maps. Fu et al. [17] propose a benchmark on science and games with limited exploration related to spatial reasoning, while we are dedicated diverse spatial reasoning tasks with in-depth analysis.

6 Discussion and Conclusions

We explored the spatial understanding capabilities of VLMs and LLMs. Our experiments resulted in several surprising conclusions across SoTA open-source and proprietary models. (1) VLMs struggle with spatial reasoning tasks, (2) multimodal models rarely surpass LLMs when relying on visual inputs, (3) when both textual and visual inputs are provided, multimodal language models rely less on the visual inputs, and (4) VLMs often outperform their LLM counterparts with text-only inputs. This challenges the belief that current VLMs are highly performant at a wide range of vision-text tasks. However, with further thought, perhaps this is to be expected after all. The currently known architectures for VLMs attempt to 'translate" the vision input into the language space and all reasoning is then performed in the language domain. It is logical that this automatic translation path is worse than a human provided translation to text, as in our text-only scenario. Thus our work shows the limits of the translation approach. Instead, to bridge the gap to human performance, future models require new architectures that treat vision input as a first-class source of information and reason in a joint vision-language space. It is our hope that our work informs development on this path.

References

Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color. arXiv preprint arXiv:2109.06129 , 2021.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Meta AI. Introducing meta llama 3: The most capable openly available llm to date, 2024. https://ai.meta.com/blog/meta-llama-3/ .

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.

Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. https: //www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf .

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425-2433, 2015.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 , 2023.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1113, 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems , 36, 2024.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems , 36, 2024.

Deqing Fu, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomorphic representations. arXiv preprint arXiv:2404.01266 , 2024.

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390 , 2024.

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 , 2023.

Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song, and Tianyi Zhou. Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic. arXiv preprint arXiv:2402.09469 , 2024.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530 , 2024.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2024.

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review, 2024.

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6700-6709, 2019.

Michael Igorevich Ivanitskiy, Rusheb Shah, Alex F. Spies, Tilman Räuker, Dan Valentine, Can Rager, Lucia Quirke, Chris Mathwin, Guillaume Corlouer, Cecilia Diniz Behn, and Samy Wu Fung. A configurable library for generating and manipulating maze datasets, 2023.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2901-2910, 2017.

Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences , 103:102274, 2023.

Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241 , 2023.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123:32-73, 2017.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International conference on machine learning , pages 12888-12900. PMLR, 2022.

Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the Fourth ACM International Conference on AI in Finance , pages 374-382, 2023.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463 , 2023.

Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. arXiv preprint arXiv:2403.07183 , 2024.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 , 2023.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. arXiv preprint arXiv:2402.17644 , 2024.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281 , 2023.

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12009-12019, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 , 2023.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611 , 2024.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022.

Yifei Ming and Yixuan Li. Understanding retrieval-augmented task adaptation for visionlanguage models. In Proceedings of the 41st International Conference on Machine Learning , pages 35719-35743, 21-27 Jul 2024.

Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. Spartqa:: A textual question answering benchmark for spatial reasoning. arXiv preprint arXiv:2104.05832 , 2021.

Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Irie, Shafiq Joty, Yixuan Li, Hai Li, Ziwei Liu, Toshihiko Yamasaki, and Kiyoharu Aizawa. Generalized out-of-distribution detection and beyond in vision language model era: A survey. arXiv preprint arXiv:2407.21794 , 2024.

Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Evaluating trustworthiness of vision language models. arXiv preprint arXiv:2403.20331 , 2024.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 , 2022.

Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International conference on learning representations , 2021.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multihop spatial reasoning in texts. In Proceedings of the AAAI conference on artificial intelligence , pages 11321-11329, 2022.

Zhenmei Shi, Jiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, and Somesh Jha. The trade-off between universality and label efficiency of representations from contrastive learning. In The Eleventh International Conference on Learning Representations , 2022.

Zhenmei Shi, Yifei Ming, Ying Fan, Frederic Sala, and Yingyu Liang. Domain generalization via nuclear norm regularization. In Conference on Parsimony and Learning (Proceedings Track) , 2023.

Zhenmei Shi, Junyi Wei, Zhuoyan Xu, and Yingyu Liang. Why larger language models do in-context learning differently? In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models , 2023.

Zhongxiang Sun. A short survey of viewing large language models in legal aspect. arXiv preprint arXiv:2303.09136 , 2023.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 , 2024.

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine , 29(8):1930-1940, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26700-26709, 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023.

Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang. Do large language models have compositional ability? an investigation into limitations and scalability. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2024.

Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang. Towards few-shot adaptation of foundation models via multitask finetuning. In The Twelfth International Conference on Learning Representations , 2023.

Yutaro Yamada, Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and Ilker Yildirim. Evaluating spatial understanding of large language models. Transactions on Machine Learning Research , 2024.

Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z Li. Context-aware attention network for image-text retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3536-3545, 2020.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 , 2023.

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624 , 2024.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

Appendix

A How Does SpatialEval Expand Beyond Traditional VQA?

We highlight several key rationales for SpatialEval in comparison to existing VQA benchmarks:

Task scope and focus : Existing benchmarks such as Visual Genome [31], GQA [25], and BLINK [18] focus only on VQA, where the image is required but the text description is often omitted or optional. In contrast, SpatialEval further explores spatial reasoning across different settings: TQA (LLM), TQA (VLM), and VTQA, where images or texts can be optional, thereby broadening the scope of tasks.

Evaluation scheme : we primarily focus on generative LLMs and VLMs which can 'elaborate on the reasoning behind your answer in a detailed, step-by-step explanation' (Section 2.2). The task in prior VQA benchmarks is often treated as discriminative with no explicit reasoning. Therefore, it remains unknown if previous observations can be naturally transferred to foundation models pre-trained on web-scale data.

The Textual Representation of Images : The textual descriptions in prior VQA benchmarks are often brief or directly imply the answers. While these datasets are valuable, they do not consistently offer the level of complexity we require, such as numerous objects containing dense visual information along with detailed natural language captions that fully convey the image content. In SpatialEval, we provide long and dense captions for each image. As a result, answers cannot be easily inferred. We also aim to isolate object detection capability from spatial reasoning ability by simplifying objects to symbols ( e.g., Spatial-Map).

IQ test for VLMs and LLMs : Tasks in SpatialEval are designed to serve as cognitive tests that evaluate basic capabilities of multi-modal foundation models. While three tasks feature synthetic visual content, humans can solve them with near-perfect accuracy. This indicates that the tasks are within the realm of human cognitive capabilities.

B Limitations and Societal Impact

Limitations. In this work, we introduce four novel tasks and conduct a comprehensive evaluation of diverse large language models (LLMs) and vision-language models (VLMs), presenting a controlled and thorough evaluation of spatial reasoning capabilities. While our analysis is detailed and extensive, it remains primarily empirical. We believe that embarking on a formal theoretical study, despite its challenges, would significantly enrich our understanding of pre-trained multimodal language models. Moreover, our focus has been on in-depth analysis rather than on developing new training strategies or adaptation algorithms to enhance spatial reasoning. Recognizing these points, we identify them as valuable directions for future research.

Societal impact. Regarding the positive societal impact, our evaluations and observations could catalyze the development of new algorithms that enhance the spatial reasoning abilities of LLMs and VLMs. Improved understanding of these models has the potential to significantly benefit sectors requiring robust spatial understanding and navigation. This could lead to more reliable and efficient systems that enhance safety and user experience. As for potential negative societal impacts, our work primarily involves empirical evaluations using synthetic datasets designed to probe spatial reasoning. Therefore, we do not anticipate any direct negative societal impacts arising from our current research.

C Experimental Details

C.1 Software and Hardware

Our experiments are conducted on NVIDIA A100 GPUs. Our implementation is based on Python 3.10 and PyTorch 2.1.2.

C.2 Hyperparameters and Error Bars

The hyperparameters discussed in this paper pertain to the decoding strategies of each model. For deterministic decoding, we adhere to the default settings specified for each model. The models employing deterministic decoding (argmax) include Bunny-Phi-2-SigLIP, CogAgent, CogVLM, InstructBLIP-Vicuna-13B, and InstructBLIP-Vicuna-7B. For non-deterministic decoding, we utilize the default hyperparameters provided by Hugging Face (for instance, Top-P is set at 0 . 9 and the temperature at 0 . 2 for LLaVA-1.6). Error bars for the main results (Figure 5) are obtained with three independent runs.

C.3 Model Checkpoints

For most open-source models, we use the checkpoints provided by Hugging Face as shown in Table 3. For Bunny variants (Bunny-Phi-1.5-EVA, Bunny-Phi-1.5-SigLIP and Bunny-Phi-2-EVA), we use merged weights following instructions in https://github.com/BAAI-DCAI/Bunny/ .

Link

LLaMA-2-13B

https://huggingface.co/meta-llama/Llama-2-13b-chat-hf

LLaMA-2-7B

https://huggingface.co/meta-llama/Llama-2-7b-chat-hf

LLaMA-3-8B

https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

Mistral-7B

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2

Nous-Hermes-2-Yi-34B

https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B

Phi-2

https://huggingface.co/microsoft/phi-2

Vicuna-13B-1.5

https://huggingface.co/lmsys/vicuna-13b-v1.5

Vicuna-7B-1.5

https://huggingface.co/lmsys/vicuna-7b-v1.5

LLaVA-1.6-34B

https://huggingface.co/liuhaotian/llava-v1.6-34b

LLaVA-1.6-Mistral-7B

https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b

LLaVA-1.6-Vicuna-13B

https://huggingface.co/liuhaotian/llava-v1.6-vicuna-13b

LLaVA-1.6-Vicuna-7B

https://huggingface.co/BAAI/Bunny-v1_0-3B

Qwen-VL

https://huggingface.co/Qwen/Qwen-VL-Chat

CogAgent

CogVLM

https://huggingface.co/THUDM/cogvlm-chat-hf

InstructBLIP-Vicuna-13B

https://huggingface.co/Salesforce/instructblip-vicuna-13b

InstructBLIP-Vicuna-7B

C.4 Detailed Illustration of Datasets

In the main text, we abbreviated the textual descriptions in Figure 1, Figure 2 due to space constraints. In this section, we provide the full descriptions for each task to facilitate a better understanding of spatial reasoning tasks. The complete illustrations are displayed in Figure 14 for Spatial-Map and Figure 15 for Maze-Nav , respectively. Three questions (termed Q1 to Q3) are associated with each sample in tasks Spaitial-Map , Maze-Nav , and Spatial-Grid .

TQA

Spatial-Map

(Text-only)

Consider a map with multiple objects:

Whale's Watchesis in the map.BrewsBrothersPub isto the Southeast of Whale's Watches.

Himalayan Hot Springs isto the Southeast of Whale's Watches.Himalayan Hot Springs is to

the Southeast of Brews Brothers Pub. Dragonfly Drones is to the Southwest of Himalayan

Hot Springs.Dragonfly Drones is to the Southeast of Brews Brothers Pub. Unicorn

UmbrellasistotheSouthwestofBrewsBrothersPub.UnicornUmbrellasistothe

Whale's Watches

Northwest of DragonflyDrones.Gale Gifts isto the Northeast of Unicorn Umbrellas.Gale

Gifts is to the Southwest of Himalayan Hot Springs.

Brews Brothers Pub

■

VQA

(Vision-only)

<img>The figure represents a map with multiple objects.Each object is associated with a

name asshownin thefigure.

GaleGifts

VTQA

Unicorn Umbrellas

(Vision-text)

<img> The figure represents a map with multiple objects.Each object is associated with a

Dragonfly Drones

name as shown in the figure.

The same figure canbe described as follows: Whale's Watches is in the map. Brews Brothers

Pubisto theSoutheast ofWhale's Watches.HimalayanHotSprings isto theSoutheast of

Whale'sWatches.Himalayan HotSpringsis to theSoutheast ofBrews Brothers Pub.

Dragonfly Drones is to the Southwest of Himalayan Hot Springs.Dragonfly Drones is to the

Southeastof BrewsBrothersPub.Unicorn Umbrellasisto theSouthwest of BrewsBrothers

Pub. Unicorn Umbrellas is to the Northwest of Dragonfly Drones. Gale Gifts is to the

Northeast of Unicorn Umbrellas. Gale Gifts is to the Southwest of Himalayan Hot Springs.

Q: In which direction is Whale's Watches relative to Dragonfly Drones?

Questions

A.Northwest

B.Southwest

C.Southeast

D.Northeast

Q: Which object is in theSouthwest of Gale Gifts?

B.Unicorn Umbrellas

C.Himalayan Hot Springs

D.BrewsBrothersPub

5

Q: How many objects are in the Southwest of Himalayan Hot Springs?

A.3

B.4

C.1

D.0

TQA

Maze-Nav

(Text-only)

Here is a Maze represented in ASCll code (LHS), where the symbols have the following meanings:

-#represents walls that areimpassable barriers.

" " (space) represents the navigable path within the maze, but not necessarily the correct pathto the exit.

-S denotes the starting point of the maze.

-E marks the endpoint or the exit of the maze.

-X marks the specific route you should follow to navigate from the starting point'S'to the endpoint'E'.

-The objective is to navigate from S to E by following the path marked by X.Movement is allowed in any ofthe four cardinal directions(up,down,left,right) along the spaces,butto solve the maze, follow the path designated by X.

-A right turn is defined as a change in movement direction that is 90 degrees clockwise relative to the previous direction.

-A left turn is defined as a changein movement direction that is90 degrees anticlockwise relative to the previous direction.

VQA

<lmg> The figure represents a Maze, where the colored blocks have the following meanings:

White blocks represent navigable paths within the maze, but not necessarily the correct path to the exit.

Green block denotes the starting point (S) of the maze.

-Red blockmarkstheendpointortheexit(E)ofthemaze.

-The objective is to navigate from S to Eby following the path marked by Blue.Movement is allowed in any of the four cardinal directions (up, down, left, right) along the White blocks, but to solve the maze,follow the path designated by Blue blocks.

-A right turn is defined as a change inmovement direction that is 90 degrees clockwise relative to the previous direction.

-A left turn is defined asa change in movement direction that is90 degrees anticlockwise relative to the previous direction.

VTQA (Vision-text)

<Img> The figure represents a Maze, where the colored blocks have the following meanings: -Blackblocksrepresent walls that areimpassable barriers.

-White blocks represent navigable paths within the maze,but not necessarily the correct path to the exit.

艾艾营#

Green block denotes the starting point (S) of the maze.

-Redblockmarkstheendpointortheexit(E)ofthemaze.

-The objective is to navigate from S to E by following the path marked by Blue. Movement is allowed in any ofthefourcardinal directions(up,down,left, right) along the Whiteblocks, but to solve the maze,follow the path designated by Blue blocks.

The same Maze can be represented in AsCll code (LHS), where the symbols have the followingmeanings:

-# represents walls that are impassable barriers.

-" " (space) represents the navigable path within the maze, but not necessarilythe correct path to the exit.

-X marks the specific route you should follow to navigate from the starting point'S'to the endpoint 'E'.

-S denotes the starting point of the maze.

-E marks the endpoint or the exit of the maze.

-The objective is to navigate from Sto E by following the path marked by X. Movement is allowed in any of the four cardinal directions (up,down, left,right) along the spaces,butto solve the maze, follow the path designated by X.

Q: How many right turns are there in the provided path from Sto E?

Questions

A.4

B.3

C.7

D.2

Q: How many total turns are there in the provided path from S to E? A.4 B.1

D.5

?

Q: Is the exit directly belowthe startingpoint,with no horizontal displacement? A.Yes B.No

D Further Ablation Studies and Discussions

TQA vs. VTQA: No definitive winner. In Section 3, we demonstrated the advantages of text-only input (TQA) based on LLMs over vision-only input (VQA) based on VLMs. Subsequently, given the same VLM, the comparison between VTQA and VQA in Section 4 reveals that VLMs rely less on visual information when sufficient textual clues are provided. Curious readers seeking a quantitative comparison between TQA and VTQA can refer to the results illustrated in Figure 16. Unlike the other comparisons, no definitive winner emerges between TQA and VTQA across all spatial reasoning tasks, model sizes, and architectures. For instance, the VTQA performance of InstructBLIP-Vicuna-13B surpasses Vicuna-13B in the Maze-Nav task, whereas Vicuna-13B outperforms InstructBLIP-Vicuna13B in the Spatial-Map task. This indicates that text-only inputs (with language models) still hold an advantage in certain spatial reasoning tasks.

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Mistral-7B

LLaVA-1.6-Mistral-7B

0.40

0.35

0.6

CogAgent

0.30

0.35

LLaVA-1.6-34B

CogAgent

0.30

LLaVA-1.6-34B

CogAgent

0.5

LLaVA-1.6-34B

0.25

0.25

0.3

0.15

LLaVA-1.6-

0.15

LLaVA-1.6-

LLaVA-1.6

0.05

Vicuna-7B

0.05

Vicuna-7B

0.1

Vicuna-7

CogVLM

CogVLM

CogVLM

InstructBLiP-

InstructBLIP-

InstructBLIP-

InstructBLIP-InstructBLIP-

InstructBLIP-

Vicuna-7B

Vicuna-13B

Vicuna-7B

Vicuna-13B

Vicuna-7B

Vicuna-13B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-13B

LLaVA-1.6-Vicuna-13B

AvgAccofTQA(Text-onlyInput)

→→Avg Acc of VTQA (Vision-text Input)

(a) TQA vs. VTQA on Spatial-Map

(b)TQAvs.VTQA onMaze-Nav

(c)TQAvs.VTQAonSpatial-Grid

Impact of prompting techniques. In line with our approach to sampling strategies, our primary goal in choosing prompting techniques is to report the best model performance given the same question. The prompt technique we use, which asks for step-by-step explanation is the most effective query among others in our initial studies. As a concrete example, we compare the original prompting strategy, "First, provide a concise answer in one sentence. Then, elaborate on the reasoning behind your answer in a detailed, step-by-step explanation" (step-by-step explanation), with a simpler prompt, Answer: (completion). Results in Table 4 show that the simpler completion prompt consistently underperforms compared to the step-by-step explanation prompt.

Impact of temperature for decoding strategies. We conducted an ablation study to examine how different temperatures affect the model performance. A higher temperature allows for more diversity in model responses. Most models consistently underperform when the temperature is set to 1.0 compared to 0.2 (our default) as shown in Table 5.

E Results on Spatial-Real task

Table 6 presents the performance on the Spatial-Real task. The same trends observed in synthetic tasks persist with real images (see VQA vs. VTQA, TQA (LLM) vs. VTQA, TQA (LLM) vs. VQA in Table 2). Notably, compared to synthetic tasks (Figure 5 and Figure 11), the overall accuracy on Spatial-Real is increased across all three input types (TQA, VQA, and VTQA). However, the modality gap (accuracy difference between VTQA and VQA) widens significantly, from 7.0% on synthetic tasks to 30.0% on Spatial-Real . This indicates that the performance disparity is more pronounced on natural images.

F Detailed Experimental Results

Results for proprietary models are summarized in Table 7. In Section 3 and Section 4, to clearly illustrate the impact of modality, we present the results averaged over all questions in Figure 6 for VQA vs. TQA and Figure 11 for VQA vs. VTQA. This section provides a comprehensive breakdown of results for individual questions. Detailed comparative results for Spatial-Map , Maze-Nav , and Spatial-Grid are shown in Table 8, Table 9, and Table 10, respectively. We compare LLM and VLM with the same language model backbone. For the VLM assessments, we consider inputs in both vision-only (VQA) and vision-text (VTQA) formats.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims are well-supported by the comprehensive experiments and analysis in Section 3, Section 4, and further results in the Appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provided detailed limitations in Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the experimental details needed in Appendix C to support the reproducibility of our experiments. Datasets and codebase with detailed instructions will also be released.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Both datasets and our code with detailed instructions will be released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All experimental details are included in Appendix C to facilitate understanding of the experiments in this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the standard deviation in main experiments (see Figure 5).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide hardware and software details in Appendix C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have read the NeurIPS Code of Ethics and made sure the paper follows the NeurIPS Code of Ethics in every aspect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provided detailed societal impacts in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The datasets in this paper are generated to evaluate the spatial reasoning abilities of models and do not pose risks for potential misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper properly cites the original paper or sources whenever an asset is used. URLs of all retained model checkpoints are included in Appendix C.3 and Table 3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Both the synthetic datasets and the code will be released with well-documented instructions.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents 3 new synthetically generated VQA evaluation benchmarks and conducts a comprehensive evaluation of the limitations of current  VLMs in spatial reasoning tasks. These tasks include spatial relationships, navigation, position understanding, and counting. The authors demonstrate using their introduced benchmarks that VLMs have limited performance on tasks requiring detailed visual understanding and reasoning of images.\n\nSTRENGTHS:\nEvaluating the spatial reasoning capabilities of SoTA VLMs is highly impactful and very relevant. I find this work novel, intriguing, and highly useful as it points to very important limitations in VLMs.  The experimental analysis is very comprehensive, incorporating both several open-source VLMs and proprietary models. Additionally, the paper is well-written and structured, contributing to its overall readability and clarity.\n\nWEAKNESSES:\nSpatial Understanding and Evaluation: The use of synthetic data in the evaluation, while valuable, may introduce confounding factors unrelated to the task of spatial understanding/reasoning. \n* For instance, a vision-only model with limited OCR capabilities may perform poorly on the spatial-map task, regardless of its actual spatial reasoning ability. It is challenging to disentangle the contributions of OCR performance from spatial understanding in such benchmark.\n* The synthetic maze-navigation and spatial-grid images might be out-of-distribution for some models, especially open-source ones. This could also explain why a noise image improves the accuracy of Llava-1.6 on maze-navigation tasks, while a similarly out-of-distribution maze image does not necessarily harm or consistently improve the  performance. It is important to consider the potential impact of out-of-distribution data on open-source models. By looking at fig 12 it looks the performance of the latest gpt4o on vision-only is similar to vision-and-text. That said, I still believe the work remains valuable and important.\n\nHuman Performance Comparison:\n* The claim on line 172 that \"the performance of these models still lags significantly behind human levels\" would benefit from concrete numbers to substantiate it. Providing specific human performance metrics on these tasks or citations here could would make the claim more robust. Alternatively, softening the statement to reflect the need for further comparison could be considered.\n\nSampling Strategies and Prompting Techniques:\n* Exploring different sampling strategies or reasoning/prompting techniques could yield valuable insights into the model's performance. It would be beneficial to include discussions on how these variations impact the results. To my understanding only a simple prompting technique is utilized \"step-by-step explanation\". Was there any particular reason to append the selected prompt vs others. \nIn line 157, the paper also mentioned \"For each model, we adopt the default configurations and decoding strategies, e.g., argmax for deterministic decoding and top-p for non-deterministic decoding. \" It would be very useful to be more specific, e.g. what top-p and temperature was used.  How \"deterministic\" was the deterministic decoding in API based models? \n\nVQA Evaluation Benchmark Details:\n* The paper would benefit from clearly specifying the size of each VQA evaluation benchmark in terms of the number of samples or data points. This information is crucial for understanding the scope and scale of the evaluations conducted. I might have missed it but I couldn't find it in the paper. I would encourage the authors to particular address this in rebuttals.\n\nQUESTIONS:\nplease see above\n\nLIMITATIONS:\nYes. It is discussed and properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors make a benchmark to test the spatial reasoning of multimodal language models (MLMs) using synthetic data consisting mostly of diagrams, mazes, and charts. They benchmark a lot of existing MLMs. Their key findings are: 1) Spatial reasoning is limited in most MLMs. 2) MLMs rely heavily on textual information and not the visual, and 3) interestingly, the VLMs are better at utilizing spatial text information than LLMs suggesting multimodal training helps understand the visual natural of language even though VLMs tend to rely less on visual information during test time.\n\nSTRENGTHS:\nThe idea of stress testing MLMs with spatial and figure understanding has been a recent trend and is very useful to push these models to do real-world tasks. \nThe analysis done is interesting and provides some useful insights into how these models use visual or textual information.\n\nWEAKNESSES:\n- Some of the claims may be too broad to make based on results of just this dataset. For instance, the models using more textual information than visual. This could simply be because the visual inputs in this benchmark are of a starkly different domain than what the models are trained on (real images instead of synthetic graphs and mazes and charts), whereas the domain gap in language is much lesser since the language tokens remain the same. Hence, we can only make the statement that the model is using more textual cues for such kinds of data. It is unclear if that is also the case on real image data. \n\n- There are many ways to test spatial reasoning using real VQA style datasets like BLINK, MMBench etc. So , it is unclear why they came up with these complicated tasks. Is there any real-world relevance for such tasks? \n\n- The authors mention they opt for synthetic data due to controllability, but all the analysis they do can be simply done on real image QA datasets like GQA.  For instance, for checking vision vs textual reliance, just feed in scene graphs vs the image to an MLM from GQA. Or, for mismatched image and text, randomly pair image and QA. At least a discussion on why such a benchmark is needed is missing. What special analysis in the paper is enabled by this benchmark that couldn't be enabled by an already existing benchmark like Visual Genome, GQA, BLINK etc.\n\n- Some settings could be explained more clearly in the paper. For instance, for the vision-only input - I am guessing the authors mean that they do not describe the image in text, but the question is still in text, correct? \n\n- A nit pick, but some grammar could be improved. e.g., intro, the first line (line 18):\" had a transformation effect on research\" - a transformational effect not affect. Other grammatical errors are also occasionally present in the paper. \n\n-  Another nit pick, but this paper seems more fitting for the datasets and benchmarks track.\n\nQUESTIONS:\nSee above concerns. \n\nOverall I like the direction of the paper. But some analysis justifying the need of such type of synthetic data (mazes and custom diagrams) instead of real images or actual graphs (useful for scientific figures) would be nice.\n\nLIMITATIONS:\nThe authors discuss some limitations of models, but not the benchmark. What can this benchmark power, and what are some analyses that the benchmark currently cannot power that would be good to have?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a set of synthetic tests to compare the spatial understanding in VLMs and LLMs. The tests include Spatial-Map, Maze-Nav, and Spatial-Grid, which all include an image, paired with text that describe the image, and a question. Using the synthetic data, the VLMs and LLMs are studied using different modalities, including VQA, TQA, and VTQA, where the models are asked to answer questions based on only the image, only the text, and both image and text. Findings include that the current VLMs are strongly biased towards texts (blind); VLMs outperform LLMs in TQA. Both open models and proprietary models have been tested.\n\nSTRENGTHS:\n1. The TQA, VQA, VTQA test is a smart way to reveal the modality bias in VLMs and LLMs, clearly showing the blindness of VLMs.\n2. Intensive results and analysis are provided.\n\nWEAKNESSES:\n1. All datasets contain only 2D synthetic images, which is substantially different from real images. At least the images can be made more realistic using computer graphics, like [4,5].\n2. The synthetic nature of the tasks introduces artifacts. Good spatial reasoning ability on this task may not generalize to real ones. The results may be highly dependent on whether the model has been trained on a similar task. For example, the performance of GPT-4O almost doubles that of Gemini in this paper, while the difference of these models on real images is not this significant.\n3. The observation that VL models are highly biased towards text, and are “blind” visually, has been studied long in the VQA community, even before LLMs [1,2,3, etc.]. The related literature should be discussed.\n[1] Eyes wide shut? exploring the visual shortcomings of multimodal llms\n[2] Explicit Bias Discovery in Visual Question Answering Models\n[3] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering\n[4] Clevr: A diagnostic dataset for compositional language and elementary visual reasoning\n[5] Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning\n\nQUESTIONS:\n1. How does the findings in this paper differ from previous literatures like [1,2,3]. Are the findings different, or are they discovered in a new way as proposed in this paper?\n2. Any discussions about the synthetic-real gap?\n3. In Spatial-Map, without text descriptions, it looks like the VLMs may describe spatial using “left/right/up/down” instead of “west/east/north/south” - will this happen, and how is it addressed?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper develops a novel benchmark to understand spatial reasoning ability of LLM and VLM. Using such a benchmark, authors conduct experiments to evaluate models' performance, and reveal several results.

STRENGTHS:
1. This paper is well-written and well-structured.

2. This paper conducts a series of experiments and include very recent LLMs and VLMs.

WEAKNESSES:
1. The design rationales of the three benchmarks are not included or discussed. "Spatial-Map" and "Maze-Nav" look reasonable but "Spatial-grid" seems ill-posed. Would we really meet any similar application scenario in the real world?

2. In the experiments of "The impact of input modality", LLMs and VLMs take very different inputs in the benchmarks, with no part overlapping. In this case, it is unsure whether such a comparison is fair or even valid.

3. It is unclear why the results of "Spatial Map" are not provided in main paper or appendix. In addition, the performance of some VLMs are also missing. Given that it can be already observed that there exist some outliers, it is important to show full results of the study to ensure the validity of observations.

4. LLaVA-1.6-34B is the most performant model in this model family. However, in Figure 7-9, it consistently under-performs compared to other LLaVA models on Spatial grid. I would like to see authors' comments on these finding. What might be the cause to the weakness of LLaVA-1.6-34B?

5. This paper overall provides quantitative results of current VLMs without deeper analysis. It remains unclear what on earth causes the weakness of these VLMs. What are insights or takeaway to improve VLMs?

QUESTIONS:
See weakness.

LIMITATIONS:
No negative societal impacts of the study.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13