SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation
Authors: Wenyu Zhang, Wei En Ng, Ma Lixin, Yuwen Wang, Junqi Zhao, Allison Koenecke, Boyang Li, WANGLU
Accepted at: acl25

SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation

Wenyu Zhang1, Wei En Ng2, Lixin Ma3*, Yuwen Wang2*, Junqi Zhao4*, Allison Koenecke5, Boyang Li4, Lu Wang1
1Institute for Infocomm Research (I²R), Agency for Science, Technology and Research (A*STAR)
2National University of Singapore (NUS), 3Tongji University
4Nanyang Technological University (NTU), 5Cornell University

Abstract

Current vision-language models may grasp basic spatial cues and simple directions (e.g. left, right, front, back), but struggle with the multi-dimensional spatial reasoning necessary for human-like understanding and real-world applications. To address this gap, we develop SPHERE (Spatial Perception and Hierarchical Evaluation of REasoning), a hierarchical evaluation framework supported by a new human-annotated dataset. SPHERE systematically probes models across increasing levels of complexity, from fundamental skills to multi-skill integration and high-level reasoning that combines spatial, visual, and logical understanding. Benchmark evaluation of state-of-the-art models reveals significant deficiencies, especially in reasoning about distance and proximity, understanding both egocentric and allocentric perspectives, and applying spatial logic in physical contexts. These findings expose critical blind spots in existing models and underscore the need for more advanced spatial reasoning techniques, driving the development of vision-language models that align more closely with human spatial cognition. The SPHERE benchmark is available at this repository.

<table>
  <tr>
    <th>Model</th>
    <th>Single-skill</th>
    <th>Multi-skill</th>
    <th>Reasoning</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>Human baseline</td>
    <td>95.4</td>
    <td>92.5</td>
    <td>89.0</td>
    <td>93.0</td>
  </tr>
  <tr>
    <td>Random baseline*</td>
    <td>50.0</td>
    <td>44.3</td>
    <td>50.0</td>
    <td>49.1</td>
  </tr>
  <tr>
    <td>Best model score</td>
    <td>78.2</td>
    <td>58.6</td>
    <td>64.7</td>
    <td>67.9</td>
  </tr>
  <tr>
    <td><b>Rank 1</b></td>
    <td>Gemini 2.0<br>Flash</td>
    <td>GPT-4o</td>
    <td>GPT-4o</td>
    <td>GPT-4o</td>
  </tr>
  <tr>
    <td><b>Rank 2</b></td>
    <td>GPT-4o</td>
    <td>Qwen2.5-VL</td>
    <td>SpatialBot-LLaVA-OneVision</td>
    <td>Qwen2.5-VL</td>
  </tr>
  <tr>
    <td><b>Rank 3</b></td>
    <td>Qwen2.5-VL</td>
    <td>LLaVA-OneVision</td>
    <td>Llama-3.2-Vision</td>
    <td>Gemini 2.0 Flash</td>
  </tr>
</table>

Table 1: VLM rankings on SPHERE tasks. * Random baseline scores excludes open-ended counting tasks.

depth information, object bounding boxes, and simple spatial directions (e.g. left/right, inside/outside, front/back) into the models (Chen et al., 2024a; Zhao et al., 2023; AI, 2024a; Cai et al., 2024; Cheng et al., 2024). However, the enhancements often focus on isolated, simplistic spatial cues, and therefore limit the models’ ability to handle the complexities of real-world scenarios. Practical applications demand more sophisticated spatial perception and reasoning capabilities, akin to human-like understanding. Figure 1 demonstrates failure cases of GPT-4o. In question 2, GPT-4o has difficulty counting while interpreting object proximity. In question 3, GPT-4o understands that the laptop is at the edge of the desk, but fails to reason that the laptop cannot be moved further back.

In this work, we propose a new benchmarking framework called SPHERE (Spatial Perception and Hierarchical Evaluation of REasoning), to systematically disentangle and measure spatial skills, in order to pinpoint model strengths and weaknesses and guide future research. We are motivated by prior research findings (West et al., 2023; Tiong et al., 2024) which reveal fundamental differences between machine and human intelligence. While LLMs excel in generating coherent and contextually relevant responses for advanced topics, they often exhibit surprising errors in basic understanding (West et al., 2023). We cannot assume good performance in one task to always transfer to a
Q: Which computer screen is smaller? The one on the left or the one on the right?
GPT-4o: The computer screen on the right (the laptop screen) appears to be smaller compared to the one on the left (the external monitor).

Q: How many crows are on the railing farther from the viewer?
GPT-4o: The image shows two crows on the railing farther from the viewer.

Q: If you are sitting at the desk, in which direction can you move the silver laptop? Backward away from you or forward toward you?
GPT-4o: If you are sitting at the desk, you can move the silver laptop backward away from you, as it is already positioned close to the edge of the desk nearest to you.

Figure 1: State-of-the-art models such as GPT-4o still has difficulty on questions that require multiple spatial, visual and reasoning skills. GPT-4o itself bolded the words in the second and third examples.

<table>
  <tr>
    <th>Reasoning</th>
    <th>Object occlusion<br>[reasoning about objects that may be present but occluded in the scene]</th>
    <th>Object manipulation<br>[reasoning about possible positions objects can move or be moved to in the scene]</th>
  </tr>
  <tr>
    <th rowspan="2">Multi-skill</th>
    <td>Position + Counting<br>[number of specific objects positioned relative to another object from allocentric/egocentric view]</td>
    <td>Distance + Counting<br>[number of specific objects in proximity to another object]</td>
    <td>Distance + Size<br>[size comparison of objects at varying proximities to viewer]</td>
  </tr>
  <tr>
    <th colspan="3">Single-skill</th>
  </tr>
  <tr>
    <td>Position<br>[relative positions from allocentric/egocentric view]</td>
    <td>Counting<br>[number of specific objects]</td>
    <td>Distance<br>[proximity between objects]</td>
    <td>Size<br>[size comparison of objects equidistant from viewer]</td>
  </tr>
</table>

Figure 2: The proposed SPHERE framework evaluates vision-language models on a hierarchy of tasks, advancing from single-skill tasks to multi-skill tasks, and ultimately to complex reasoning tasks that require the integration of multiple spatial and visual cues with logical reasoning abilities.

seemingly correlated or even more basic task.

SPHERE, illustrated in Figure 2, is a hierarchical evaluation framework that covers a spectrum of basic and advanced tasks. We begin with single-skill tasks (i.e. object positions, distances, sizes and counts), then expand to multi-skill tasks that require multiple spatial and visual skills (e.g. counting objects at specific positions, counting objects at certain distances, etc.). We also design two novel reasoning tasks (i.e. reasoning about occluded objects and object manipulation) that require advanced understanding of a scene as a 3-dimensional environment and the objects as physical entities that can be interacted with. We measure reasoning ability as the ability to infer about scenes or objects not explicitly observed but physically plausible in the input images. These challenging tasks allow us to assess the alignment of VLMs with human-like spatial reasoning capabilities. Our contributions are summarized as follows:

• We design a hierarchical evaluation framework to study the spatial perception and reasoning capabilities of models, progressing from single-skill, multi-skill, to complex reasoning tasks. The reasoning tasks are further decomposed into simpler sub-tasks to disentangle spatial perception and reasoning.

• We manually annotate and curate an evaluation dataset using real-world images from MS COCO based on the designed tasks.

• We evaluate state-of-the-art VLMs and reveal significant shortcomings in their capabilities. These models especially lack the ability to understand distance and proximity, to reason from both allocentric and egocentric viewpoints, and to perform complex reasoning in a physical world context.

2 Related works

2.1 Spatial Capabilities in Multimodal LLMs

The majority of literature on multimodal LLMs focus on their conversational ability to respond flu-
ently to the queries of human users through natural language (Dai et al., 2023; Liu et al., 2023b, 2024; Lyu et al., 2023) or other modalities such as audio and image (Wu et al., 2023; Tang et al., 2024). More recently, works have started to explore the integration of spatial capabilities in LLMs to better understand the physical world. Zheng et al. (2024) developed BAT based on text and audio by synthesizing data with a spatial audio simulator, and Devnani et al. (2024) trained a spatially-aware audio and text embedding model. SpatialBot (Cai et al., 2024) directly includes pairs of RGB and depth images in existing datasets for training. A popular approach for vision-language models is to use existing algorithms and models (e.g. object detection, segmentation, depth estimation) to extract additional information from available data, and then incorporate these enhanced data for training or fine-tuning (Chen et al., 2024a; Zhao et al., 2023; AI, 2024a; Cai et al., 2024; Cheng et al., 2024). The resulting models have improved capabilities to infer spatial relationships between objects and to estimate object dimensions from natural images.

2.2 Benchmark Evaluations

Spatial reasoning is necessary for VLMs to understand and interact with the physical world. This has spurred the development of numerous benchmarks that vary in scope, tasks, and the types of spatial skills they focus on. Some benchmarks (Li et al., 2024b; Yu et al., 2024) involve spatial capabilities in only a small subset of tasks. EmbSpatial-Bench (Du et al., 2024) evaluates only a limited number of positional relationships, namely ‘above’, ‘below’, ‘left’, ‘right’, ‘far’ and ‘close’. Zeng et al. (2024) tests the effects of spatial relations on visual grounding tasks. VSR (Liu et al., 2023a) includes proximity and topological concepts. SpatialBench (Cai et al., 2024) consists of a small dataset of 120 images to evaluate spatial relationships, counting and size comparison between objects. Q-Spatial Bench (Liao et al., 2024) and SpatialRGPT-Bench (Cheng et al., 2024) focuses on model ability to recognize quantitative information such as metric distances and object sizes. These evaluations and datasets do not disentangle the effects of multiple spatial and/or visual skills. CLEVR (Johnson et al., 2016) tests compositional reasoning with multiple spatial and visual skills, but the dataset is synthetic and the objects comprise only simple 3D shapes such as spheres, cubes and cylinders. Kamath et al. (2023) isolates the effect of spatial attributes by varying the spatial relations of objects while fixing object identities, but the collected dataset focuses only on positional relationships. We study the interaction effect of a wider range of skills, and introduce a novel study of model ability to infer about scenes or objects not explicitly observed but physically plausible in the input images.

3 The SPHERE Benchmark

3.1 Overview

The SPHERE evaluation framework is designed to systematically disentangle essential spatial and visual capabilities and to assess VLM performance in tasks that can guide physical world actions. The hierarchy of tasks is illustrated in Figure 2. The first two levels emphasize image understanding, while the final level builds upon image understanding and focuses on analysis and reasoning.

SPHERE begins at the first level with 4 single-skill tasks: Position, Counting, Distance and Size. We choose these skills because they are foundational building blocks for higher-order tasks, and are of interest to the research community as demonstrated through existing efforts to endow VLMs with these skills (Cai et al., 2024; Chen et al., 2024a; Du et al., 2024). At the second level, the first level skills are integrated into 3 multi-skill tasks: Position + Counting, Distance + Counting, and Distance + Size. We choose to combine 2 skills at a time to minimize confounding factors. Tasks in this category are more challenging. Specifically, Distance + Size require models to understand the concept of size constancy, instead of simply measuring size by the number of pixels occupied in an image. At the third level, 2 reasoning tasks assess models on their ability to reason about object occlusion and object manipulation. These tasks reflect scenarios commonly encountered in the physical world, such as inferring the presence or position of hidden objects or determining how objects can be moved or rearranged. Logical reasoning together with multiple spatial and visual understanding abilities may be required to successfully answer the questions in this category.

3.2 Benchmark Dataset

We manually curate and annotate a question-answering dataset for SPHERE, as automatic generation with VLMs remain unreliable due to accuracy limitations, as demonstrated in Table 1 and GPT-4o failure cases in Figure 1. We use images from
Figure 3: Examples of open-ended questions for counting-related tasks and multiple-choice questions for other tasks in the annotated SPHERE dataset. Ground-truth answers are in green.

the test split of MS COCO-2017 (Lin et al., 2014) due to its rich diversity in scenes and object categories. Since COCO-2017 is a widely-used and publicly available dataset, its images are likely to be in-distribution with respect to training datasets used by VLMs, ensuring that the evaluation aligns with their learned representations. This alignment reduces potential biases stemming from dataset shifts.

Annotations for each task are conducted and cross-verified by at least two authors for consistency and accuracy. Questions that did not elicit the same responses are discarded and replaced. The general guidelines for annotation are:

• Ensure non-ambiguity and diversity of question and answers.
• Omit images that may be irrelevant or ambiguous for the task.
• Counting-related tasks require open-ended numerical responses, while other tasks use multiple-choice questions (MCQs) with two options by default and three options when additional clarity is needed.

We use diverse types of options for MCQs, including boolean (e.g. yes vs. no), position descriptor (e.g. left vs. right), and name of object. Figure 3 shows examples of annotated data.

Figure 4: Examples of position-related questions asked from allocentric and egocentric viewpoints.

<table>
  <tr>
    <th>Task level</th>
    <th>Task type</th>
    <th>Sample size</th>
  </tr>
  <tr>
    <td rowspan="4">Single-skill</td>
    <td>Position</td>
    <td>Ego.: 172<br>Allo.: 185</td>
  </tr>
  <tr>
    <td>Counting</td>
    <td>201</td>
  </tr>
  <tr>
    <td>Distance</td>
    <td>202</td>
  </tr>
  <tr>
    <td>Size</td>
    <td>198</td>
  </tr>
  <tr>
    <td rowspan="3">Multi-skill</td>
    <td>Position + Counting</td>
    <td>Ego.: 64<br>Allo.: 105</td>
  </tr>
  <tr>
    <td>Distance + Counting</td>
    <td>158</td>
  </tr>
  <tr>
    <td>Distance + Size</td>
    <td>199</td>
  </tr>
  <tr>
    <td rowspan="2">Reasoning</td>
    <td>Object occlusion</td>
    <td>Intermed.: 202<br>Final: 200</td>
  </tr>
  <tr>
    <td>Object manipulation</td>
    <td>Intermed.: 199<br>Final: 200</td>
  </tr>
</table>

Table 2: The annotated dataset has a total of 2,285 question-answer pairs. For position-related tasks, we annotate for both egocentric (ego.) and allocentric (allo.) viewpoints. For reasoning tasks, we additionally annotate for intermediate (intermed.) questions.

3.2.1 Dataset Design and Statistics

We annotate a total of 2,285 question-answer pairs. The sample size for each task is delineated in Table 2. Detailed dataset statistics and analysis of dataset diversity are in Appendix A.

Single-skill tasks. We annotate for 4 single-tasks. Basic object recognition capabilities are assumed.

• Position: We pose questions on relative positional relationships (e.g. left vs. right, in front vs. behind, on top vs. below) between objects in the scene. For entities with clearly defined front and back orientations (e.g. humans, animals, vehicles), we include questions for both egocentric and allocentric viewpoints. Egocentric viewpoint questions assess the model’s ability to understand spatial relationships from a specified entity’s perspective, such as identifying objects to the entity’s left or right relative to its orientation. Allocentric viewpoint questions evaluate the model’s ability to describe spatial relationships from the external
Figure 5: Distribution of ground-truth answers for the counting task.

Figure 6: Distribution of base spatial and visual skills required for questions in reasoning tasks.

point of view of the camera or observer. Example questions are shown in Figure 4.
• Counting: These questions require the model to count the number of specified objects visible in the image. To evaluate the model’s susceptibility to object hallucinations, we included ‘trick’ questions where the correct answer is zero. Figure 5 illustrates the distribution of ground-truth answers.
• Distance: We ask questions on the relative proximity (e.g. closer vs. farther) between objects, and between objects and the viewer.
• Size: We ask questions on the relative size (e.g. smaller vs. bigger, shorter vs. taller) of objects. We focus on objects positioned approximately equidistant from the camera, where object size can be inferred directly from the number of pixels occupied by the object on the image.

Multi-skill tasks: We integrate the first level skills into 3 multi-skill tasks.
• Position + Counting: Questions require the model to count the number of specified objects that have a specified spatial relationship with another object in the image. We include both egocentric and allocentric viewpoints. Example questions are shown in Figure 4.
• Distance + Counting: Questions require the model to count the number of specified objects that are at a specified proximity to another object in the image or the observer. An example of a question in Figure 3 is “How many crows are on the railing farther from the viewer?”
• Distance + Size: We pose questions about the relative size of objects located at different distances from the camera. To succeed at this task, the model must demonstrate an understanding of size constancy, or the ability to recognize that an object’s actual size remains consistent despite changes in its apparent size due to distance or perspective.

Reasoning tasks: We design 2 tasks to evaluate the model’s ability to logically reason about the 3D physical world based on its understanding of the input 2D image. In addition to the final reasoning-based questions, we include understanding-based questions as an intermediate step to check the model’s comprehension of the 2D image. In our design, “intermediate” questions focus on assessing the perception and understanding of visible objects and scenes. These tasks align with cognitive theories like Piaget’s stages of cognitive development, which suggest that perceptual skills are crucial for building more abstract reasoning abilities assessed through the “final” reasoning questions. The “final” reasoning questions require the ability to infer the existence, position, or state of partially or non-visible objects and scenes that are physically plausible. These inferences rely on understanding spatial relationships and object permanence, as the missing information could only be revealed by changing viewpoints or interacting with the scene, such as moving objects to uncover hidden ones.
• Object occlusion: This task evaluates a model’s ability to infer the presence or properties of objects that may be fully occluded from the perspective of observer. From Figure 3, an example of an intermediate understanding-based question is “Which object is thicker? Fire hydrant or tree trunk?” The final reasoning-based question is “Where can a child be hiding? Behind the fire hydrant or behind the tree?” To answer this question correctly, the model needs to reason that a child is more likely to be obscured by the thicker tree.
• Object manipulation: This task tests a model’s ability to reason about potential or hypothetical positions that objects can move to
or be moved to within a scene, based on physical constraints. From Figure 3, an example of an intermediate understanding-based question is “What is behind the silver laptop? Empty space or window bars?” The final reasoning-based question is “Sitting at the desk, how can you move the silver laptop? Away from you or toward you?” To answer this question correctly, the model needs to reason that a person at the desk cannot move the laptop further back as the laptop is already positioned against the window bars.

Successfully solving reasoning tasks requires multiple spatial and visual skills. We categorize these foundational skills as counting, distance, position, shape, size and other visual skills, such as perceiving color and transparency. We annotate these foundational skills and illustrate the distribution of their occurrences in Figure 6.

4 Evaluations

4.1 Setup

We describe the experimental setup used to evaluate the performance of state-of-the-art VLMs on SPHERE tasks. All experiments were performed on NVIDIA GeForce RTX 3090 GPUs.

Models: Our evaluation encompasses two model categories: general-purpose VLMs and VLMs with enhanced spatial capabilities. The general-purpose VLMs evaluated are Phi-3.5-Vision (Abdin et al., 2024), LLaVA-NeXT (Liu et al., 2024), LLaVA-OneVision (Li et al., 2024a), InstructBLIP (Dai et al., 2023), Idefics2 (HuggingFaceM4, 2024), InternVL2.5 (Chen et al., 2024b), Janus-Pro (Chen et al., 2025), Qwen-VL (Bai et al., 2023), Qwen2-VL (Wang et al., 2024), Qwen2.5-VL (Bai et al., 2025), Llama-3.2-Vision (Meta, 2024), Gemini 2.0 Flash (Google, 2024), and GPT-4o (OpenAI, 2024). The VLMs specifically trained for spatial understanding and reasoning are SpatialBot (Cai et al., 2024), SpaceMantis (AI, 2024b), and SpatialRGPT (Cheng et al., 2024). For SpatialRGPT, we use the RGB-only version as the model requires highly accurate mask proposals to make use of depth maps. All model implementations follow their released versions on Hugging Face. We use the default prompt templates and hyperparameters in the demos of the respective models.

The text instructions given take the form:

“<question> Answer the question directly.”

For example, for the first image in Figure 4, the text instruction given is “From the perspective of the viewer, where is the girl relative to the man? Left or Right? Answer the question directly.” We added the instruction “Answer the question directly” to reduce the refusal-to-answer rate and to encourage concise answers that align with the options provided.

Metrics: We computed both the validity and accuracy of the VLM responses. Since the VLM generative responses are open-ended, we deem a response as valid if it is a relevant answer to the question, that is, a number is given for a counting question and the response contains one of the options for a multiple-choice question (MCQ). Accuracy measures the correctness of the model responses, and invalid responses are treated as incorrect. For MCQs, we set the baseline accuracy for each question as 1/#options, since this is the probability of selecting the correct answer purely by random guessing.

To account for the probabilistic nature of VLM generations, we evaluate each question 5 times using different random seeds and average the validity and accuracy scores across these runs. For MCQs, we introduce additional randomness by shuffling the order of the answer options at each seed. This prevents the models from exploiting positional biases in the option sequence and ensures that their performance reflects true understanding rather than reliance on specific patterns. Through these considerations, we increase the robustness of our evaluation, providing a more reliable measure of the models’ generalization and consistency.

4.2 Results

Table 3 shows model accuracy on each of the SPHERE tasks. The results highlight significant room for improvement, especially for multi-skill and reasoning tasks.

The proprietary model GPT-4o outperforms the open-weights models evaluated in almost all tasks and has an overall accuracy of 67.9%. Qwen2.5-VL (72B) has the second highest overall accuracy of 64.3%. Interestingly, amongst open-weights models, we observe that the smaller model LLaVA-OneVision (7B) can outperform larger models in terms of overall accuracy, and performs on par with Gemini 2.0 Flash.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Single-skill</th>
    <th colspan="5">Multi-skill</th>
    <th colspan="3">Reasoning</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>Pos.</th>
    <th>Count.</th>
    <th>Dist.</th>
    <th>Size</th>
    <th>Avg</th>
    <th>P + C</th>
    <th>D + C</th>
    <th>D + S</th>
    <th>Avg</th>
    <th>Occl.</th>
    <th>Manip.</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Human baseline</td>
    <td>90.4</td>
    <td>98.0</td>
    <td>98.0</td>
    <td>95.0</td>
    <td>95.4</td>
    <td>95.6</td>
    <td>91.0</td>
    <td>91.0</td>
    <td>92.5</td>
    <td>87.0</td>
    <td>91.0</td>
    <td>89.0</td>
    <td>93.0</td>
  </tr>
  <tr>
    <td>Random baseline*</td>
    <td>50.0</td>
    <td>-</td>
    <td>50.0</td>
    <td>50.0</td>
    <td>50.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>44.3</td>
    <td>44.3</td>
    <td>50.0</td>
    <td>50.0</td>
    <td>49.1</td>
  </tr>
  <tr>
    <th colspan="15">General purpose VLMs</th>
  </tr>
  <tr>
    <td>Phi-3.5-Vision (4B)</td>
    <td>61.8</td>
    <td>58.7</td>
    <td>56.7</td>
    <td>76.0</td>
    <td>63.3</td>
    <td>44.4</td>
    <td>36.1</td>
    <td>47.2</td>
    <td>42.6</td>
    <td>54.4</td>
    <td>55.5</td>
    <td>54.9</td>
    <td>54.3</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT (7B)</td>
    <td>53.6</td>
    <td>68.7</td>
    <td>54.5</td>
    <td>70.9</td>
    <td>61.9</td>
    <td>42.0</td>
    <td>35.4</td>
    <td>36.1</td>
    <td>37.8</td>
    <td>48.7</td>
    <td>49.1</td>
    <td>48.9</td>
    <td>50.6</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision (7B)</td>
    <td>60.2</td>
    <td>76.1</td>
    <td>64.2</td>
    <td>84.0</td>
    <td>71.1</td>
    <td>53.3</td>
    <td>53.8</td>
    <td>52.9</td>
    <td>53.3</td>
    <td>53.0</td>
    <td>59.6</td>
    <td>56.3</td>
    <td>61.5</td>
  </tr>
  <tr>
    <td>Qwen2-VL (7B)</td>
    <td>63.2</td>
    <td>74.1</td>
    <td>53.0</td>
    <td>78.6</td>
    <td>67.2</td>
    <td>36.1</td>
    <td>33.5</td>
    <td>50.8</td>
    <td>40.1</td>
    <td>54.0</td>
    <td>55.2</td>
    <td>54.6</td>
    <td>55.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL (7B)</td>
    <td>59.3</td>
    <td>82.1</td>
    <td>58.9</td>
    <td>73.6</td>
    <td>68.5</td>
    <td>44.7</td>
    <td>31.6</td>
    <td>37.9</td>
    <td>38.1</td>
    <td>48.7</td>
    <td>54.1</td>
    <td>51.4</td>
    <td>54.1</td>
  </tr>
  <tr>
    <td>Janus-Pro (7B)</td>
    <td>57.1</td>
    <td>70.6</td>
    <td>59.3</td>
    <td>76.5</td>
    <td>65.9</td>
    <td>37.3</td>
    <td>33.5</td>
    <td>46.8</td>
    <td>39.2</td>
    <td>53.5</td>
    <td>54.3</td>
    <td>53.9</td>
    <td>54.0</td>
  </tr>
  <tr>
    <td>InstructBLIP (8B)</td>
    <td>45.6</td>
    <td>64.7</td>
    <td>50.4</td>
    <td>56.2</td>
    <td>54.2</td>
    <td>32.0</td>
    <td>28.5</td>
    <td>38.1</td>
    <td>32.8</td>
    <td>54.8</td>
    <td>52.7</td>
    <td>53.7</td>
    <td>47.0</td>
  </tr>
  <tr>
    <td>Idefics2 (8B)</td>
    <td>50.1</td>
    <td>45.3</td>
    <td>49.1</td>
    <td>58.5</td>
    <td>50.7</td>
    <td>22.5</td>
    <td>27.8</td>
    <td>40.1</td>
    <td>30.1</td>
    <td>51.0</td>
    <td>53.3</td>
    <td>52.1</td>
    <td>44.2</td>
  </tr>
  <tr>
    <td>InternVL2.5 (8B)</td>
    <td>62.2</td>
    <td>72.6</td>
    <td>64.6</td>
    <td>78.3</td>
    <td>69.4</td>
    <td>50.2</td>
    <td>43.8</td>
    <td>41.8</td>
    <td>45.3</td>
    <td>52.7</td>
    <td>53.6</td>
    <td>53.1</td>
    <td>57.3</td>
  </tr>
  <tr>
    <td>Qwen-VL (10B)</td>
    <td>55.9</td>
    <td>72.9</td>
    <td>59.2</td>
    <td>72.0</td>
    <td>65.0</td>
    <td>37.2</td>
    <td>29.9</td>
    <td>34.7</td>
    <td>33.9</td>
    <td>54.3</td>
    <td>54.3</td>
    <td>54.3</td>
    <td>52.0</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (11B)</td>
    <td>58.4</td>
    <td>53.5</td>
    <td>52.9</td>
    <td>67.2</td>
    <td>58.0</td>
    <td>29.8</td>
    <td>27.7</td>
    <td>40.8</td>
    <td>32.8</td>
    <td>60.7</td>
    <td>56.1</td>
    <td>58.4</td>
    <td>49.7</td>
  </tr>
  <tr>
    <td>Qwen2-VL (72B)</td>
    <td>62.8</td>
    <td>80.6</td>
    <td>60.8</td>
    <td>85.5</td>
    <td>72.4</td>
    <td>37.3</td>
    <td>38.0</td>
    <td><b>69.3</b></td>
    <td>48.2</td>
    <td>53.9</td>
    <td>55.0</td>
    <td>54.4</td>
    <td>59.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL (72B)</td>
    <td>63.7</td>
    <td>81.1</td>
    <td>69.9</td>
    <td>89.3</td>
    <td><b>76.0</b></td>
    <td>51.5</td>
    <td>55.7</td>
    <td>66.4</td>
    <td>57.9</td>
    <td>50.3</td>
    <td>56.4</td>
    <td>53.3</td>
    <td><b>64.3</b></td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (90B)</td>
    <td><b>66.0</b></td>
    <td>59.2</td>
    <td>65.2</td>
    <td>75.8</td>
    <td>66.5</td>
    <td>40.1</td>
    <td>31.8</td>
    <td>52.8</td>
    <td>41.6</td>
    <td>61.6</td>
    <td>55.3</td>
    <td><b>58.4</b></td>
    <td>56.2</td>
  </tr>
  <tr>
    <td>Gemini 2.0 Flash</td>
    <td>68.6</td>
    <td><b>82.1</b></td>
    <td>75.2</td>
    <td>86.9</td>
    <td><b>78.2</b></td>
    <td>50.3</td>
    <td>52.5</td>
    <td>49.7</td>
    <td>50.9</td>
    <td>48.0</td>
    <td>49.2</td>
    <td>48.6</td>
    <td><b>61.7</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>71.7</b></td>
    <td>76.6</td>
    <td>71.3</td>
    <td>89.4</td>
    <td><b>77.3</b></td>
    <td><b>54.4</b></td>
    <td><b>62.7</b></td>
    <td><b>58.8</b></td>
    <td><b>58.6</b></td>
    <td><b>66.0</b></td>
    <td><b>63.3</b></td>
    <td><b>64.7</b></td>
    <td><b>67.9</b></td>
  </tr>
  <tr>
    <th colspan="15">Spatial VLMs</th>
  </tr>
  <tr>
    <td>SpatialBot-RGB (3B)</td>
    <td>55.9</td>
    <td>67.7</td>
    <td>52.8</td>
    <td>73.0</td>
    <td>62.3</td>
    <td>38.5</td>
    <td>39.2</td>
    <td>38.7</td>
    <td>38.8</td>
    <td>57.6</td>
    <td><b>60.3</b></td>
    <td><b>59.0</b></td>
    <td>53.6</td>
  </tr>
  <tr>
    <td>SpatialBot (3B)</td>
    <td>55.5</td>
    <td>72.1</td>
    <td>51.9</td>
    <td>75.4</td>
    <td>63.7</td>
    <td>39.1</td>
    <td>36.1</td>
    <td>38.9</td>
    <td>38.0</td>
    <td>56.5</td>
    <td>54.6</td>
    <td>55.5</td>
    <td>53.1</td>
  </tr>
  <tr>
    <td>SpaceMantis (8B)</td>
    <td>52.7</td>
    <td>52.7</td>
    <td>57.0</td>
    <td>63.1</td>
    <td>56.4</td>
    <td>28.4</td>
    <td>43.7</td>
    <td>45.6</td>
    <td>39.2</td>
    <td>56.6</td>
    <td>54.2</td>
    <td>55.4</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td>SpatialRGPT-RGB (8B)</td>
    <td>59.3</td>
    <td>70.1</td>
    <td>59.2</td>
    <td>74.6</td>
    <td>65.8</td>
    <td>42.6</td>
    <td>46.2</td>
    <td>40.9</td>
    <td>43.2</td>
    <td>54.5</td>
    <td>53.2</td>
    <td>53.8</td>
    <td>55.3</td>
  </tr>
</table>

Table 3: Average accuracy (%) of VLMs on SPHERE tasks. Values highlighted in gold, silver, and bronze denote the first, second and third place for each task. The random baseline accuracy for multiple-choice questions is the probability of selecting the correct answer from given options purely by random guessing. * Average scores for random baseline excludes open-ended counting tasks.

For the single-skill tasks, Gemini 2.0 Flash has the highest average accuracy (78.2%). The best performances on Counting and Size are above 80%. The best performances on Position and Distance are between 70% and 80%, indicating that current VLMs still have difficulty in these two skills. In Section 4.3, we analyze how viewpoint affects model performance on understanding positions.

For the multi-skill tasks, GPT-4o has the highest average accuracy (58.6%). The multi-skill tasks are markedly more challenging than the single-skill tasks. In particular, we observe that half of the models perform worse than baseline random guessing on Distance + Size. Qwen2-VL (72B) obtains the highest 69.3% on Distance + Size, and 85.5% on Size. This gap reflects the reliance of the VLMs on the 2D representation of objects in an image, and measures object size as being directly proportional to the number of pixels occupied. The failure to incorporate the principle of size constancy may cause VLMs to struggle in 3D spaces.

For the reasoning tasks, GPT-4o has the highest average accuracy (64.7%). Most models perform close to random guessing, suggesting that their training datasets may lack sufficient examples emphasizing reasoning skills. Notably, Gemini 2.0 Flash has the worst average accuracy (48.6%), below the random guessing baseline.

In general, we note that the spatial VLMs evaluated performed comparably with or better than most similarly-sized general-purpose VLMs, except for LLaVA-OneVision. In particular, SpatialBot (3B) performed comparably with or better than larger models with up to 11B parameters. This shows that the techniques in spatial VLMs can benefit spatial capabilities. Possible reasons on why the spatial VLMs do not excel on the SPHERE tasks may be the lack of training on the specific skills emphasized by the SPHERE tasks, and the lack of model generalization to the SPHERE tasks.

We refer readers to Table 7 in the Appendix B for detailed validity scores of the models. The validity scores reflect the instruction-following capabilities of the VLMs. Idefics2 has above 95% validity, and
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Position</th>
    <th colspan="2">Position + Counting</th>
  </tr>
  <tr>
    <th>Allo.</th>
    <th>Ego.</th>
    <th>Allo.</th>
    <th>Ego.</th>
  </tr>
  <tr>
    <td colspan="5">General purpose VLMs</td>
  </tr>
  <tr>
    <td>Phi-3.5-Vision (4B)</td>
    <td>77.9</td>
    <td>44.5</td>
    <td>53.3</td>
    <td>29.7</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT (7B)</td>
    <td>69.3</td>
    <td>36.7</td>
    <td>47.6</td>
    <td>32.8</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision (7B)</td>
    <td>73.7</td>
    <td>45.7</td>
    <td>62.9</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>Qwen2-VL (7B)</td>
    <td>69.1</td>
    <td>56.0</td>
    <td>37.1</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL (7B)</td>
    <td>69.0</td>
    <td>48.8</td>
    <td>49.1</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>Janus-Pro (7B)</td>
    <td>65.0</td>
    <td>48.6</td>
    <td>37.1</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>InstructBLIP (8B)</td>
    <td>43.0</td>
    <td>48.4</td>
    <td>31.4</td>
    <td>32.8</td>
  </tr>
  <tr>
    <td>Idefics2 (8B)</td>
    <td>48.5</td>
    <td>51.9</td>
    <td>20.0</td>
    <td>26.6</td>
  </tr>
  <tr>
    <td>InternVL.2.5 (8B)</td>
    <td>75.7</td>
    <td>47.7</td>
    <td>59.8</td>
    <td>34.4</td>
  </tr>
  <tr>
    <td>Qwen-VL (10B)</td>
    <td>65.6</td>
    <td>45.5</td>
    <td>41.5</td>
    <td>30.0</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (11B)</td>
    <td>60.5</td>
    <td>56.0</td>
    <td>31.6</td>
    <td>26.9</td>
  </tr>
  <tr>
    <td>Qwen2-VL (72B)</td>
    <td>70.3</td>
    <td>55.6</td>
    <td>36.2</td>
    <td>35.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL (72B)</td>
    <td>78.7</td>
    <td>47.6</td>
    <td>58.1</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (90B)</td>
    <td>72.9</td>
    <td>58.6</td>
    <td>41.1</td>
    <td>38.4</td>
  </tr>
  <tr>
    <td>Gemini 2.0 Flash</td>
    <td>83.2</td>
    <td>52.9</td>
    <td>55.2</td>
    <td>42.2</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>79.5</td>
    <td>63.4</td>
    <td>62.9</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td colspan="5">Spatial VLMs</td>
  </tr>
  <tr>
    <td>SpatialBot-RGB (3B)</td>
    <td>68.6</td>
    <td>42.2</td>
    <td>47.6</td>
    <td>23.4</td>
  </tr>
  <tr>
    <td>SpatialBot (3B)</td>
    <td>69.7</td>
    <td>40.2</td>
    <td>46.7</td>
    <td>26.6</td>
  </tr>
  <tr>
    <td>SpaceMantis (8B)</td>
    <td>56.2</td>
    <td>48.8</td>
    <td>31.4</td>
    <td>23.4</td>
  </tr>
  <tr>
    <td>SpatialRGPT-RGB (8B)</td>
    <td>73.2</td>
    <td>44.4</td>
    <td>48.6</td>
    <td>32.8</td>
  </tr>
</table>

Table 4: Average accuracy (%) of VLMs on position-related questions asked from allocentric (allo.) and egocentric (ego.) viewpoints. Values highlighted in gold, silver and bronze denote the first, second and third place.

all other models have above 98% validity.

4.3 Further Analysis

Bias toward allocentric or egocentric viewpoint.
We further analyze model performance on position-related tasks by distinguishing between allocentric and egocentric viewpoints. This analysis provides insights into whether VLMs can comprehend object orientations and reason from both relative (object-to-object) and absolute (viewer-to-object) perspectives. Example questions are shown in Figure 4. From the results in Table 4, we observe that most VLMs demonstrate a significant bias toward the allocentric viewpoint, with performance gaps as large as 33.4% for Phi-3.5-Vision in the Position task, and 25.4% for LLaVA-OneVision in the Position + Counting task.

Effectiveness of additional information for reasoning. Since the reasoning tasks first require image understanding of the factual information presented in the images and then logical reasoning to conduct analysis and inference, we further disentangle the two types of abilities. We measure the strength of model reasoning capabilities alone by directly providing understanding-based information in the input prompts. That is, we prefix the final step question with the intermediate step question and ground-truth answer as:

“Given that for the question: <intermediate step question> The answer is: <intermediate step answer>. <final step question> Answer the question directly.”

For the example in Figure 3, the prompt will be “Given that for the question: Which object is thicker? Fire hydrant or tree trunk? The answer is: Tree trunk. Where can a child be hiding? Behind the fire hydrant or behind the tree? Answer the question directly.”

From Table 5, we observe that incorporating additional information improves final question accuracy across all models for the object occlusion task, but yields mixed results for the object manipulation task. Although the additional information can introduce relevant facts that the models might otherwise overlook, the models sometimes struggle to establish a clear connection between these facts and the specific reasoning they need to conduct.

Skill composition. The different performance of the evaluated models across tasks reflect that they have different compositions of skills, and that proficiency in one task does not necessarily translate to proficiency in related tasks. For instance, relatively weaker basic skills do not necessarily imply correspondingly weaker higher-order skills. In Table 3, SpaceMantis has 7.3% lower average accuracy than SpatialBot on the single-skill tasks, but has 1.2% higher average accuracy than SpatialBot on the multi-skill tasks. In Table 5, on the object manipulation task, Idefics2 and Llama-3.2-Vision (11B) have attained higher accuracy on the final reasoning questions (53.3% and 56.1%) than on the supposedly easier intermediate perception or understanding questions (51.1% and 53.7%).

5 Conclusion

This work introduces SPHERE, a comprehensive evaluation framework designed to address the critical gaps in spatial understanding and reasoning exhibited by current vision-language models. Through a hierarchy of tasks and a meticulously curated dataset, SPHERE highlights significant limitations in the ability of state-of-the-art models, especially in processing spatial cues such as distance, reasoning from both allocentric and egocentric viewpoints, and analyzing the depicted scene
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Object occlusion</th>
    <th colspan="3">Object manipulation</th>
  </tr>
  <tr>
    <th>Intermed.</th>
    <th>Final</th>
    <th>Final (w/ gt intermed.)</th>
    <th>Intermed.</th>
    <th>Final</th>
    <th>Final (w/ gt intermed.)</th>
  </tr>
  <tr>
    <th colspan="7">General purpose VLMs</th>
  </tr>
  <tr>
    <td>Phi-3.5-Vision (4B)</td>
    <td>60.6</td>
    <td>54.4</td>
    <td>67.0 (\( \uparrow 12.6 \))</td>
    <td>59.8</td>
    <td>55.5</td>
    <td>57.6 (\( \uparrow 2.1 \))</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT (7B)</td>
    <td>60.4</td>
    <td>48.7</td>
    <td>54.4 (\( \uparrow 5.7 \))</td>
    <td>58.5</td>
    <td>49.1</td>
    <td>48.8 (\( \downarrow 0.3 \))</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision (7B)</td>
    <td>63.9</td>
    <td>53.0</td>
    <td>59.8 (\( \uparrow 6.8 \))</td>
    <td>64.2</td>
    <td>59.6</td>
    <td>57.1 (\( \downarrow 2.5 \))</td>
  </tr>
  <tr>
    <td>Qwen2-VL (7B)</td>
    <td>63.4</td>
    <td>54.0</td>
    <td>60.8 (\( \uparrow 6.8 \))</td>
    <td>55.1</td>
    <td>55.2</td>
    <td>55.4 (\( \uparrow 0.2 \))</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL (7B)</td>
    <td>65.2</td>
    <td>48.7</td>
    <td>51.1 (\( \uparrow 2.4 \))</td>
    <td>61.1</td>
    <td>54.1</td>
    <td>55.4 (\( \uparrow 1.3 \))</td>
  </tr>
  <tr>
    <td>Janus-Pro (7B)</td>
    <td>58.4</td>
    <td>53.5</td>
    <td>55.0 (\( \uparrow 1.5 \))</td>
    <td>62.6</td>
    <td>54.3</td>
    <td>53.0 (\( \downarrow 1.3 \))</td>
  </tr>
  <tr>
    <td>InstructBLIP (8B)</td>
    <td>48.7</td>
    <td>54.8</td>
    <td>57.8 (\( \uparrow 3.0 \))</td>
    <td>52.9</td>
    <td>52.7</td>
    <td>45.7 (\( \downarrow 7.0 \))</td>
  </tr>
  <tr>
    <td>Idefics2 (8B)</td>
    <td>56.5</td>
    <td>51.0</td>
    <td>60.0 (\( \uparrow 9.0 \))</td>
    <td>51.1</td>
    <td>53.3</td>
    <td>60.7 (\( \uparrow 7.4 \))</td>
  </tr>
  <tr>
    <td>InternVL2.5 (8B)</td>
    <td>63.7</td>
    <td>52.7</td>
    <td>58.9 (\( \uparrow 6.2 \))</td>
    <td>62.9</td>
    <td>53.6</td>
    <td>57.2 (\( \uparrow 3.6 \))</td>
  </tr>
  <tr>
    <td>Qwen-VL (10B)</td>
    <td>57.8</td>
    <td>54.3</td>
    <td>57.3 (\( \uparrow 3.0 \))</td>
    <td>54.5</td>
    <td>54.3</td>
    <td>49.0 (\( \downarrow 5.3 \))</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (11B)</td>
    <td>52.3</td>
    <td>60.7</td>
    <td>62.5 (\( \uparrow 1.8 \))</td>
    <td>53.7</td>
    <td>56.1</td>
    <td>57.5 (\( \uparrow 1.4 \))</td>
  </tr>
  <tr>
    <td>Qwen2-VL (72B)</td>
    <td>65.6</td>
    <td>53.9</td>
    <td>62.4 (\( \uparrow 8.5 \))</td>
    <td>68.5</td>
    <td>55.0</td>
    <td>76.9 (\( \uparrow 21.9 \))</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL (72B)</td>
    <td>67.3</td>
    <td>50.3</td>
    <td>57.6 (\( \uparrow 7.3 \))</td>
    <td>64.2</td>
    <td>56.4</td>
    <td>66.9 (\( \uparrow 10.5 \))</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (90B)</td>
    <td>62.7</td>
    <td>61.6</td>
    <td>68.0 (\( \uparrow 6.4 \))</td>
    <td>59.8</td>
    <td>55.3</td>
    <td>74.0 (\( \uparrow 18.7 \))</td>
  </tr>
  <tr>
    <td>Gemini 2.0 Flash</td>
    <td>62.4</td>
    <td>48.0</td>
    <td>61.0 (\( \uparrow 13.0 \))</td>
    <td>65.5</td>
    <td>49.2</td>
    <td>58.3 (\( \uparrow 9.1 \))</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>67.3</td>
    <td>66.0</td>
    <td>72.0 (\( \uparrow 6.0 \))</td>
    <td>68.5</td>
    <td>63.3</td>
    <td>76.4 (\( \uparrow 13.1 \))</td>
  </tr>
  <tr>
    <th colspan="7">Spatial VLMs</th>
  </tr>
  <tr>
    <td>SpatialBot-RGB (3B)</td>
    <td>59.9</td>
    <td>57.6</td>
    <td>65.9 (\( \uparrow 8.3 \))</td>
    <td>60.8</td>
    <td>60.3</td>
    <td>54.0 (\( \downarrow 6.3 \))</td>
  </tr>
  <tr>
    <td>SpatialBot (3B)</td>
    <td>58.6</td>
    <td>56.5</td>
    <td>63.2 (\( \uparrow 6.7 \))</td>
    <td>59.2</td>
    <td>54.6</td>
    <td>56.1 (\( \uparrow 1.5 \))</td>
  </tr>
  <tr>
    <td>SpaceMantis (8B)</td>
    <td>63.1</td>
    <td>56.6</td>
    <td>62.4 (\( \uparrow 5.8 \))</td>
    <td>60.1</td>
    <td>54.2</td>
    <td>57.6 (\( \uparrow 3.4 \))</td>
  </tr>
  <tr>
    <td>SpatialRGPT-RGB (8B)</td>
    <td>61.1</td>
    <td>54.5</td>
    <td>63.2 (\( \uparrow 8.7 \))</td>
    <td>60.1</td>
    <td>53.2</td>
    <td>58.5 (\( \uparrow 5.3 \))</td>
  </tr>
</table>

Table 5: Average accuracy (%) of VLMs on the intermediate (intermed.) and final questions in the reasoning tasks. The approach ‘final (w/ gt intermed.)’ denotes that the intermediate questions and ground-truth answers are used as additional information to assist the answering of the final questions.

in a physical world context. Through this work, we seek to lay the groundwork for advancing vision-language models toward more human-like spatial perception and reasoning, and to set a benchmark for future research in this domain.

6 Limitations

As we seek to ensure the quality of the SPHERE dataset through manual annotations, the dataset may be limited in scope. The framework and the curated tasks may not encompass all possible spatial understanding and reasoning challenges encountered in practice. In addition, the dataset focuses on static images, which may not fully capture the challenges of dynamic spatial understanding and reasoning in real-world scenarios. As VLMs advance in spatial perception and reasoning, automatic annotation methods may become more reliable and warrant exploration in future work.

7 Ethics Statement

All real-world images used in our annotated dataset are sourced from the MS COCO-2017 dataset (Lin et al., 2014), a widely used and publicly available resource. Our annotations focus solely on the spatial properties of the entities in the images and do not involve any personally identifiable information. The authors have unanimously agreed that the annotated data will be used exclusively for academic research purposes, in compliance with the dataset’s terms of use (COCO Consortium, 2024).

Acknowledgements

We thank Chanwoo Hwang from Hanyang University for valuable discussions. This research is partially supported by the Nanyang Associate Professorship and the National Research Foundation Fellowship (NRF-NRFF13-2021-0006), Singapore.

References

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, and et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv.

Remyx AI. 2024a. SpaceLLaVA. https://huggingface.co/remyxai/SpaceLLaVA.

Remyx AI. 2024b. SpaceMantis. https://huggingface.co/remyxai/SpaceMantis.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL technical report. arXiv.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Diess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, and Brianna Zitkovich. 2023. RT-2: Vision-language-action models transfer web knowledge to robotic control. arXiv.

Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. 2024. SpatialBot: Precise spatial understanding with vision language models. arXiv.

Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. 2024a. SpatialVL-M: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-Pro: Unified multimodal understanding and generation with data and model scaling. arXiv.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024b. Intervl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. 2024. SpatialRGPT: Grounded spatial reasoning in vision-language models. arXiv.

COCO Consortium. 2024. COCO Terms of Use. https://cocodataset.org/#termsofuse.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Conference on Neural Information Processing Systems (NeurIPS).

Bhavika Suresh Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso, Yelena Menyaylenko, Barry-John Theobald, Jonathan Sheaffer, and Miguel Sarabia. 2024. Learning spatially-aware language and audio embeddings. In Conference on Neural Information Processing Systems (NeurIPS).

Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. 2024. EmbSpatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Annual Meeting of the Association for Computational Linguistics.

Google. 2024. Introducing Gemini 2.0: our new AI model for the agent era. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024.

HuggingFaceM4. 2024. idefics2-8B. https://huggingface.co/HuggingFaceM4/idefics2-8b.

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2016. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s “up” with vision-language models? Investigating their struggle with spatial reasoning. In EMNLP.

Isabel Leal, Krzysztof Choromanski, Deepali Jain, Kumar Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Ho Vuong, Tamás Sarlós, Kenneth Oslund, Karol Hausman, and Kanishka Rao. 2023. SARA-RT: Scaling up robotics transformers with self-adaptive robust attention. In International Conference on Robotics and Automation (ICRA).

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. LLaVA-OneVision: Easy visual task transfer. arXiv.

Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024b. Seedbench: Benchmarking multimodal large language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. 2024. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models. arXiv.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV).

Fangyu Liu, Guy Emerson, and Nigel Collier. 2023a. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In Conference on Neural Information Processing Systems (NeurIPS).

Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. Macaw-LLM: Multi-modal language modeling with image, audio, video, and text integration. arXiv.

Meta. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhi Gupta. 2022. R3M: A universal visual representation for robot manipulation. In Conference on Robot Learning.

OpenAI. 2024. Gpt-4o system card. arXiv.

Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, and Karol Hausman. 2023. Open-world object manipulation using pre-trained vision-language models. arXiv.

Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. 2024. CoDi-2: In-context, interleaved, and interactive any-to-any generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Anthony Meng Huat Tiong, Junqi Zhao, Boyang Albert Li, Junnan Li, Steven C.H. Hoi, and Caiming Xiong. 2024. What are we measuring when we evaluate large vision-language models? an analysis of latent factors and biases. In North American Chapter of the Association for Computational Linguistics.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv.

Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian R. Fisher, Abhilasha Ravichander, Khyathi Raghavi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. 2023. The generative ai paradox: "what it can create, it may not understand". arXiv.

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023. NExT-GPT: Any-to-any multi-modal LLM. arXiv.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. MM-Vet: Evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learning. PMLR.

Yunan Zeng, Yan Huang, Jinjin Zhang, Zequn Jie, Zhenhua Chai, and Liang Wang. 2024. Investigating compositional challenges in vision-language models for visual grounding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Yongqiang Zhao, Zhenyu Li, Zhi Jin, Feng Zhang, Haiyan Zhao, Chengfeng Dou, Zhengwei Tao, Xinhai Xu, and Donghong Liu. 2023. Enhancing the spatial awareness capability of multi-modal large language model. arXiv.

Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. 2024. BAT: Learning to reason about spatial sounds with large language models. arXiv.
A  Detailed Dataset Statistics

We provide additional details of our annotated SPHERE dataset.

Number of options. For non-counting tasks, questions are posed as multiple-choice questions (MCQs). Figure 7 shows the distribution of number of options across all tasks. All non-counting tasks except Distance + Size only have two-option MCQs, while around a third of MCQs in Distance + Size have three options. When comparing the size of two objects and asking the models to identify the smaller/bigger object, we provide options of the form ‘the closer object’, ‘the farther object’ and ‘the two objects are of similar size’. Such three-option MCQs are designed to evaluate if the model has an understanding of size constancy when similar objects may appear to be of different sizes in the image due to their positioning in the 3D scene.

Numerical answers. Counting tasks are phrased as open-ended numerical questions, designed to evaluate the models’ ability to recognize objects matching the required criterion while distinguishing such objects as distinct entities. Figure 8 shows the distribution of ground-truth answers across all counting-related tasks. We limit the number of objects to be counted to a maximum of 9 to avoid including overly small objects.

We check the count distributions by fitting a negative binomial model to each distribution using the goodfit function from R’s vcd package. The results of Pearson’s \( \chi^2 \) tests, shown in Table 6, indicate that all \( p \)-values exceeded 0.10. This suggests that the count distributions align well with the negative binomial model and are not excessively skewed toward specific values.

Formats. Figure 9a shows the distribution of the following option formats across all tasks:

• Boolean: True/false or other similar phrasings, e.g. “yes” vs. “no”.
• Name: Name of objects or their attributes, e.g. “empty space” vs. “window pane”; “black laptop” vs. “white laptop”, “curved” vs “straight”.
• Numeric: For counting tasks, a non-negative integer indicating the number of objects matching the required criterion. For non-counting tasks, MCQs may contain numbers in the provided options, e.g. “1/2” vs. “less than 1/4”, or require numerical estimation, e.g. “less than the length of the painting” vs. “more than the length of the painting”.
• Position: Position of objects, e.g. “left” vs. “right”, “on top” vs. “below”.

Viewpoints. Figure 9b shows the distribution of egocentric vs. allocentric viewpoints across all tasks. Position, Position + Counting, Distance + Counting and Object manipulation comprise both allocentric and egocentric questions, while all other tasks comprise only allocentric questions.

Object concepts. We assess the diversity of object concepts in the annotations by counting the occurrences of unique nouns. Nouns were extracted from each question using the en_core_web_sm model from spaCy, then manually filtered to exclude propositions and commonly referenced objects in question phrasings, such as ‘camera’, ‘viewer’, ‘photo’, ‘image’. The dataset contains a total of more than 600 object concepts. Figure 10 shows the distribution of object concepts across the dataset for the top 30 concepts.

B  Detailed Evaluation Results

Table 7 shows the validity scores of the models across all tasks. These scores reflect the instruction-following capabilities of the VLMs. Most models

![Bar chart showing distribution of number of options for MCQs across all non-counting tasks](page_1012_153_377_246.png)

Figure 7: Distribution of number of options for MCQs across all non-counting tasks.

<table>
  <tr>
    <th>Task</th>
    <th>\( \chi^2 \)</th>
    <th>df</th>
    <th>\( P(> \chi^2) \)</th>
  </tr>
  <tr>
    <td>Counting</td>
    <td>6.2420</td>
    <td>7</td>
    <td>0.51179</td>
  </tr>
  <tr>
    <td>Distance + Counting</td>
    <td>7.8268</td>
    <td>6</td>
    <td>0.25107</td>
  </tr>
  <tr>
    <td>Position + Counting</td>
    <td>7.8879</td>
    <td>5</td>
    <td>0.16252</td>
  </tr>
</table>

Table 6: Goodness-of-fit tests on ground-truth answers for counting tasks fitted to a negative-binomial model.
Figure 8: Distribution of ground-truth answers for all counting tasks.

Figure 9: Distribution of option formats and viewpoints across all tasks.

effectively follow instructions and provide valid responses, such as numerical answers for counting-related questions and selecting an option from the provided list for multiple-choice questions. Idefics2 has above 95% validity, and all other models have above 98% validity. Common types of invalid responses include providing an image caption instead of addressing the question, refusing to answer, or offering an answer that is not among the given options.

We provide example questions and answers for each task in Table 8, 9 and 10.
Figure 10: Distribution of object concepts across the dataset.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Single-skill</th>
    <th colspan="4">Multi-skill</th>
    <th colspan="3">Reasoning</th>
  </tr>
  <tr>
    <th>Pos.</th>
    <th>Count.</th>
    <th>Dist.</th>
    <th>Size</th>
    <th>Avg</th>
    <th>P + C</th>
    <th>D + C</th>
    <th>D + S</th>
    <th>Avg</th>
    <th>Occl.</th>
    <th>Manip.</th>
    <th>Avg</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td colspan="15">General purpose VLMs</td>
  </tr>
  <tr>
    <td>Phi-3.5-Vision (4B)</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.7</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.0</td>
    <td>99.7</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.9</td>
  </tr>
  <tr>
    <td>LLaVA-NeXT (7B)</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>98.8</td>
    <td>99.6</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.9</td>
  </tr>
  <tr>
    <td>LLaVA-OneVision (7B)</td>
    <td>99.7</td>
    <td>100.0</td>
    <td>99.8</td>
    <td>98.4</td>
    <td>99.5</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>97.3</td>
    <td>99.1</td>
    <td>99.7</td>
    <td>100.0</td>
    <td>99.8</td>
    <td>99.4</td>
  </tr>
  <tr>
    <td>Qwen-2-VL (7B)</td>
    <td>97.4</td>
    <td>100.0</td>
    <td>98.3</td>
    <td>99.1</td>
    <td>98.7</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>98.5</td>
    <td>99.5</td>
    <td>98.5</td>
    <td>99.5</td>
    <td>99.0</td>
    <td>99.0</td>
  </tr>
  <tr>
    <td>Qwen-2.5-VL (7B)</td>
    <td>94.0</td>
    <td>100.0</td>
    <td>98.8</td>
    <td>98.8</td>
    <td>97.9</td>
    <td>100.0</td>
    <td>99.4</td>
    <td>96.8</td>
    <td>98.7</td>
    <td>98.4</td>
    <td>97.3</td>
    <td>97.9</td>
    <td>98.2</td>
  </tr>
  <tr>
    <td>Janus-Pro (7B)</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.8</td>
    <td>99.9</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>InstructBLIP (8B)</td>
    <td>97.1</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.7</td>
    <td>99.2</td>
    <td>99.4</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>99.8</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>99.6</td>
  </tr>
  <tr>
    <td>Idefics2 (8B)</td>
    <td>92.9</td>
    <td>93.0</td>
    <td>96.5</td>
    <td>91.9</td>
    <td>93.6</td>
    <td>99.4</td>
    <td>96.8</td>
    <td>91.4</td>
    <td>95.9</td>
    <td>98.7</td>
    <td>96.0</td>
    <td>97.3</td>
    <td>95.3</td>
  </tr>
  <tr>
    <td>InternVL2.5 (8B)</td>
    <td>98.2</td>
    <td>100.0</td>
    <td>99.4</td>
    <td>98.9</td>
    <td>99.1</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.3</td>
    <td>99.8</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>99.5</td>
  </tr>
  <tr>
    <td>Qwen-VL (10B)</td>
    <td>99.0</td>
    <td>100.0</td>
    <td>99.5</td>
    <td>100.0</td>
    <td>99.6</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>98.4</td>
    <td>99.5</td>
    <td>96.8</td>
    <td>98.8</td>
    <td>97.8</td>
    <td>99.1</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (11B)</td>
    <td>99.0</td>
    <td>100.0</td>
    <td>99.0</td>
    <td>98.3</td>
    <td>99.1</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>98.6</td>
    <td>99.5</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>99.9</td>
    <td>99.4</td>
  </tr>
  <tr>
    <td>Qwen-2-VL (72B)</td>
    <td>97.6</td>
    <td>100.0</td>
    <td>99.5</td>
    <td>99.0</td>
    <td>99.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.3</td>
    <td>99.8</td>
    <td>99.6</td>
    <td>100.0</td>
    <td>99.8</td>
    <td>99.5</td>
  </tr>
  <tr>
    <td>Qwen-2.5-VL (72B)</td>
    <td>99.2</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>99.2</td>
    <td>99.2</td>
    <td>99.5</td>
    <td>99.6</td>
    <td>99.9</td>
    <td>99.8</td>
    <td>99.7</td>
  </tr>
  <tr>
    <td>Llama-3.2-Vision (90B)</td>
    <td>95.9</td>
    <td>100.0</td>
    <td>99.5</td>
    <td>98.3</td>
    <td>98.4</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>98.3</td>
    <td>99.4</td>
    <td>100.0</td>
    <td>98.8</td>
    <td>99.4</td>
    <td>99.0</td>
  </tr>
  <tr>
    <td>Gemini 2.0 Flash</td>
    <td>99.4</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.9</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>98.5</td>
    <td>99.5</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.8</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>99.2</td>
    <td>99.5</td>
    <td>98.5</td>
    <td>97.0</td>
    <td>98.5</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>94.0</td>
    <td>98.0</td>
    <td>98.5</td>
    <td>100.0</td>
    <td>99.2</td>
    <td>98.5</td>
  </tr>
  <tr>
    <td colspan="15">Spatial VLMs</td>
  </tr>
  <tr>
    <td>SpatialBot-RGB (3B)</td>
    <td>99.2</td>
    <td>99.0</td>
    <td>99.8</td>
    <td>98.4</td>
    <td>99.1</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.0</td>
    <td>99.7</td>
    <td>100.0</td>
    <td>99.5</td>
    <td>99.7</td>
    <td>99.4</td>
  </tr>
  <tr>
    <td>SpatialBot (3B)</td>
    <td>99.2</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>99.4</td>
    <td>99.3</td>
    <td>99.6</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.8</td>
  </tr>
  <tr>
    <td>SpaceMantis (8B)</td>
    <td>99.4</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.7</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>98.5</td>
    <td>99.5</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.7</td>
  </tr>
  <tr>
    <td>SpatialRGPT-RGB (8B)</td>
    <td>98.9</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.1</td>
    <td>99.5</td>
    <td>100.0</td>
    <td>100.0</td>
    <td>99.2</td>
    <td>97.9</td>
    <td>97.9</td>
    <td>95.9</td>
    <td>96.9</td>
    <td>98.9</td>
  </tr>
</table>

Table 7: Average validity (%) of VLM responses on SPHERE tasks.
<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3">
      <img src="page_186_246_172_120.png" alt="Image 1"></td>
    <td>If you are the man in the image, where is the girl relative to your position? Left or Right?</td>
    <td>Ground-truth: Left<br>GPT-4o: Right ✗<br>Gemini 2.0 Flash: Right ✗</td>
  </tr>
  <tr>
    <td>From the perspective of the viewer, where is the girl relative to the man? Left or Right?</td>
    <td>Ground-truth: Right<br>GPT-4o: Right ✓<br>Gemini 2.0 Flash: Right ✓</td>
  </tr>
  <tr>
    <td>If you are the man pouring wine, where is the man with glasses relative to your position? Left or Right?</td>
    <td>Ground-truth: Left<br>GPT-4o: Right ✗<br>Gemini 2.0 Flash: Right ✗</td>
  </tr>
</table>

(a) Task: Position

<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3">
      <img src="page_186_546_172_120.png" alt="Image 2"></td>
    <td>How many buses are visible?</td>
    <td>Ground-truth: 0<br>GPT-4o: 0 ✓<br>Gemini 2.0 Flash: 0 ✓</td>
  </tr>
  <tr>
    <td>How many trees are visible?</td>
    <td>Ground-truth: 4<br>GPT-4o: 2 ✗<br>Gemini 2.0 Flash: 3 ✗</td>
  </tr>
  <tr>
    <td>How many crows can be seen?</td>
    <td>Ground-truth: 3<br>GPT-4o: 3 ✓<br>Gemini 2.0 Flash: 3 ✓</td>
  </tr>
</table>

(b) Task: Counting
<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3">
      <img src="page_184_370_222_153.png" alt="Image 1"></td>
    <td>Which object is more distant from the plant? Chair or Table?</td>
    <td>Ground-truth: Chair<br>GPT-4o: Chair ✓<br>Gemini 2.0 Flash: Chair ✓</td>
  </tr>
  <tr>
    <td>Which group of people is farther away from the motorcycle? People sitting on the left or People standing on the right?</td>
    <td>Ground-truth: People sitting on the left<br>GPT-4o: People standing on the right ✗<br>Gemini 2.0 Flash: People sitting on the left ✓</td>
  </tr>
  <tr>
    <td>Which boat has a shorter distance to the bank? Boat with a cleaning sign or Boat with passengers?</td>
    <td>Ground-truth: Boat with passengers<br>GPT-4o: Boat with a cleaning sign ✗<br>Gemini 2.0 Flash: Boat with a cleaning sign ✗</td>
  </tr>
</table>

(c) Task: Distance

<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3">
      <img src="page_184_627_222_153.png" alt="Image 2"></td>
    <td>Which computer is the smallest? The leftmost one or The rightmost one?</td>
    <td>Ground-truth: The leftmost one<br>GPT-4o: The leftmost one ✓<br>Gemini 2.0 Flash: The leftmost one ✓</td>
  </tr>
  <tr>
    <td>Is the fish icon on the ground bigger than the star icon in the picture? Yes or No?</td>
    <td>Ground-truth: Yes<br>GPT-4o: Yes ✓<br>Gemini 2.0 Flash: Yes ✓</td>
  </tr>
  <tr>
    <td>Which cup is higher? The left one or The right one?</td>
    <td>Ground-truth: The right one<br>GPT-4o: The left one ✗<br>Gemini 2.0 Flash: The left one ✗</td>
  </tr>
</table>

(d) Task: Size

Table 8: Examples from single-skill tasks.
<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3"><img src="page_184_276_246_153.png" alt="A man sitting at a desk with two laptops in front of him"></td>
    <td>From the perspective of the viewer, how many laptops are on the left side of the man?</td>
    <td>Ground-truth: 2<br>GPT-4o: 2 ✓<br>Gemini 2.0 Flash: 1 ✗</td>
  </tr>
  <tr>
    <td>If you are the man in the image, how many laptops are on your left?</td>
    <td>Ground-truth: 0<br>GPT-4o: 2 ✗<br>Gemini 2.0 Flash: 2 ✗</td>
  </tr>
  <tr>
    <td>How many displays are to the right of the keyboard?</td>
    <td>Ground-truth: 1<br>GPT-4o: 2 ✗<br>Gemini 2.0 Flash: 3 ✗</td>
  </tr>
</table>

(a) Task: Position + Counting

<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3"><img src="page_184_579_246_153.png" alt="A person standing between a group of horses and a fence"></td>
    <td>From the perspective of the person, how many horses are farther from him than the fence?</td>
    <td>Ground-truth: 3<br>GPT-4o: 3 ✓<br>Gemini 2.0 Flash: 2 ✗</td>
  </tr>
  <tr>
    <td>From the perspective of the person, how many horses are closer to him than the fence?</td>
    <td>Ground-truth: 0<br>GPT-4o: 1 ✗<br>Gemini 2.0 Flash: 1 ✗</td>
  </tr>
  <tr>
    <td>How many crows are on the railing farther from the viewer?</td>
    <td>Ground-truth: 1<br>GPT-4o: 3 ✗<br>Gemini 2.0 Flash: 2 ✗</td>
  </tr>
</table>

(b) Task: Distance + Counting
<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3">
      <img src="page_184_629_172_120.png" alt="Image 1"></td>
    <td>Are the turquoise plates likely the same size? Yes or No?</td>
    <td>Ground-truth: Yes<br>GPT-4o: Yes ✓<br>Gemini 2.0 Flash: Yes ✓</td>
  </tr>
  <tr>
    <td>The chairs on the top half of the picture are differently sized? Yes or No?</td>
    <td>Ground-truth: No<br>GPT-4o: Yes ✗<br>Gemini 2.0 Flash: Yes ✗</td>
  </tr>
  <tr>
    <td>Which pillow is bigger? The left one or The right one or They are similar in size?</td>
    <td>Ground-truth: They are similar in size<br>GPT-4o: The left one ✗<br>Gemini 2.0 Flash: The right one ✗</td>
  </tr>
</table>

(c) Task: Distance + Size

Table 9: Examples from multi-skill tasks.
<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3">
      <img src="page_184_256_246_120.png" alt="A plate with food and a ketchup bottle on a checkered tablecloth"></td>
    <td>Where can the pepper shaker most likely be found? Behind the ketchup bottle or Behind the margarita glass?</td>
    <td>Ground-truth: Behind the ketchup bottle<br>GPT-4o: Behind the ketchup bottle ✓<br>Gemini 2.0 Flash: Behind the ketchup bottle ✓</td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_384_246_120.png" alt="A room with a suitcase on a white chair"></td>
    <td>Where can another teddy bear be found? Inside the suitcase on the white chair or Under the blue dress?</td>
    <td>Ground-truth: Inside the suitcase on the white chair<br>GPT-4o: Under the blue dress ✗<br>Gemini 2.0 Flash: Under the blue dress ✗</td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_512_246_120.png" alt="A group of people sitting at a table with pizza boxes"></td>
    <td>Is it likely that more glasses are hidden behind the pizza boxes? Yes or No?</td>
    <td>Ground-truth: No<br>GPT-4o: No ✓<br>Gemini 2.0 Flash: Yes ✗</td>
  </tr>
</table>

(a) Task: Object occlusion

<table>
  <tr>
    <th>Image</th>
    <th>Question</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td rowspan="3">
      <img src="page_184_768_246_120.png" alt="A woman in pink standing next to a man wearing glasses"></td>
    <td>From the perspective of the woman in pink, can the man wearing glasses stand on her left side? Yes or No?</td>
    <td>Ground-truth: Yes<br>GPT-4o: Yes ✓<br>Gemini 2.0 Flash: Yes ✓</td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_896_246_120.png" alt="Two cats lying on a bed"></td>
    <td>Can the dog lie between the two cats? Yes or No?</td>
    <td>Ground-truth: No<br>GPT-4o: Yes ✗<br>Gemini 2.0 Flash: No ✓</td>
  </tr>
  <tr>
    <td rowspan="2">
      <img src="page_184_1024_246_120.png" alt="Two wine glasses on a table"></td>
    <td>From the perspective of the viewer, can we pour all the wine from the nearer glass into the one behind? Yes or No?</td>
    <td>Ground-truth: Yes<br>GPT-4o: Yes ✓<br>Gemini 2.0 Flash: No ✗</td>
  </tr>
</table>

(b) Task: Object manipulation

Table 10: Examples from reasoning tasks.

Reviewer paper summaries: The paper introduces SPHERE, a new evaluation framework designed to probe spatial reasoning abilities in VLMs. SPHERE consists of a hierarchical set of tasks, progressing from single-skill spatial understanding (e.g., position, distance, and size) to multi-skill and complex reasoning tasks that require integrating multiple spatial and logical elements. The framework comes with a human-annotated dataset sourced from MS COCO-2017. The authors benchmark several state-of-the-art models, including GPT-4o, Gemini 2.0 Flash, and LLaVA variants, revealing weaknesses in spatial cognition.

The paper introduces a hierarchical benchmark, SPHERE, to assess spatial reasoning capabilities in Vision-Language Models (VLMs). This benchmark is originated from MS COCO and the hierarchies ranges from single-skill tasks to multi-skill tasks, i.e. the combinations of single-skill tasks, and two newly proposed reasoning tasks. 
The benchmark reveals that current models fall short to understand distance and proximity, and that performance on single basic skills does not necessarily corelated with higher-order skills.
The findings highlight the need for improved spatial perception and reasoning techniques in VLMs.
Summary of strengths: - The paper addresses a critical gap in VLM evaluation, proposing a structured and comprehensive benchmark for spatial reasoning. 
- The hierarchical framework effectively disentangles different levels of spatial understanding, making it possible to diagnose specific failure cases in models.
-  The evaluation methodology is rigorous as it uses repeated trials and randomized answer. Also results are well-presented, with insightful observations about the limitations of existing models (e.g., wrt handling distance and occlusion).

1. Novel aspect of evaluating spatial reasoning ability: This paper presents a hierarchical evaluation framework that disentangles spatial skills systematically, and introduces new reasoning tasks beyond basic spatial relationships.
2. Extensive evaluation of current models: This paper assesses both general-purpose and spatially-enhanced VLMs.
3. Questions are annotated from both allocentric and egocentric viewpoints, enabling a more comprehensive understanding of the spatial information than merely focusing on allocentric or egocentric views.
4. The hierarchies in the proposed benchmark could serve as detailed supervision for further models and applications, facilitating further advancements in spatial reasoning.
Summary of weaknesses: - While carefully annotated, the dataset is somewhat limited in scope. It focuses on static images and does not account for dynamic spatial reasoning (useful e.g. in robotics and real-world interactions in general). The lack of video-based evaluation restricts the applicability of SPHERE in assessing temporal-spatial reasoning.
- The paper would benefit from a clearer discussion of how different models process spatial relationships internally—whether through explicit geometric representations or learned statistical patterns.

1. Outdated models: Although this is a resubmission, newly released models ahead of the current submission circle should be considered to see if the benchmark is still challanging for recent models, such as IDEFICS 3 (released Aug 2024), and some newer versions (release a month ago) of the evaluated models.
2. Insufficient analysis: The authors address multi-skill abilities, but the correlation between skill combinations, as well as the relationship between basic skills and combined skills, should be discussed further. While the findings on skill composition are interesting, the authors only state that “relatively weaker basic skills do not necessarily imply correspondingly weaker higher-order skills” without offering an in-depth analysis. This conclusion is based solely on the overall accuracy of different tasks. Readers may expect more insight into which basic skills matter the most and how two basic skills can complement each other to improve performance when combined.
Comments, suggestions and typos: - The analysis of size constancy failures could be strengthened by including a breakdown of specific failure patterns, such as cases where models incorrectly judge relative sizes based on pixel proportions rather than depth cues.

- It would be useful to provide more details on how the models were prompted during evaluations. Specifically, did the prompts contain explicit spatial descriptors, or did models have to infer relationships purely from image-text pairs? A brief example of a full input-output pair could improve clarity.

Typos:
- Typo in the introduction: "capabilties" → "capabilities."
- In Section 3.2: "phyiscally" → "physically."

**Questions for rebuttal:**

See weaknesses: 
1. Please provide results on more recent version of employed models; 
2. Please provide detailed discussion or examples to further clarify the finding "relatively weaker basic skills do not necessarily imply correspondingly weaker higher-order skills", not just comparing the acc.

3. Additional question:
Spatial VLMs are specifically optimized for spatial reasoning. However, they do not significantly outperform general-purpose VLMs, even when excluding proprietary models like Gemini and GPT. Could the authors provide reasons or specific examples explaining the failures?

**Other comments:**

This is not a review comment, but I would like to say that I appreciate the discussion on the bias toward allocentric and egocentric viewpoint.
Overall assessment scores: 3.5

3.5