Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities

## Abstract

Abstract Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners.
While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored.
To address this issue, we present the CO nsistent M ultilingual F rame O f R eference T est ( COMFORT ), an evaluation protocol to systematically assess the spatial reasoning capabilities of VLMs.
We evaluate nine state-of-the-art VLMs using COMFORT .
Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages.
With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.

## 1 Introduction

The recent success of large language models has sparked breakthroughs in multi-modalities, leading to the development of many vision-language models (VLMs; Chen et al., ; OpenAI, ; Reid et al., , inter alia).
With some benchmarks developed to evaluate the downstream performance of these models (Liu et al., ; Yue et al., ), there has been growing excitement around evaluations and analyses inspired by human cognitive capabilities such as referential grounding (Ma et al., ), compositional reasoning (Ma et al., ), visual illusions (Zhang et al., ; Guan et al., ), and theory of mind (Jin et al., ).
One direction among them that captures significant attention is spatial language understanding and reasoning, leading to several benchmarks (Kamath et al., ; Liu et al., ) and enhanced models (Chen et al., ; Cheng et al., ).

Indeed, spatial cognition is a crucial part of human cognitive capability, developed since infancy and continuing through the elementary school years (Tommasi & Laeng, ; Vasilyeva & Lourenco, ).
Language is closely intertwined with spatial cognition, with each contributing to the acquisition of the other (Hayward & Tarr, ; Regier & Carlson, ; Pyers et al., ; Pruden et al., ; Gentner et al., ).
While spatial language and non-linguistic spatial representations in memory are closely correlated and share foundational properties, they are, to some extent, divergent—spatial conventions are not consistently preserved across different languages or tasks, and humans demonstrate flexibility in using multiple coordinate systems for both non-linguistic reasoning and linguistic expressions (Munnich et al., ; Shusterman & Li, ).
Thus, spatial language is inherently ambiguous, and as we quote:

In situated communication, even a simple spatial expression like “the basketball to the right of the car” may have multiple interpretations.
People may use different frames of reference (FoR; Levinson, ; Frank, , inter alia) to resolve ambiguity about the underlying coordinate system, as illustrated in Figure 1a.
The diversity of conventions across languages and cultures further complicates this ambiguity—different languages employ different conventions in choosing one FoR among multiple competing options.
As shown in Figure 1b, speakers may project themselves onto the ball or consider an imaginary listener facing them (Shusterman & Li, ).
These ambiguities are not easily resolvable based solely on linguistic expressions (Tenbrink, ; Liu et al., ).

Our main research question is not new: Do vision-language models represent space, and how?
Several benchmarks (Kamath et al., ; Liu et al., ) have been developed for this purpose, consisting of text-image pairs where objects may or may not follow certain spatial relations.
However, the aforementioned spatial ambiguities remain largely under-explored when studying VLM-based spatial language understanding and reasoning.
We emphasize that FoRs are crucial to studying spatial cognition across modalities, as they provide a foundational framework for understanding how spatial relationships are perceived, interpreted, and communicated (Levinson, ).

To fill this gap, we present COnsistent Multilingual Frame Of Reference Test (COMFORT), a framework that systematically evaluates the spatial reasoning capabilities of VLMs, emphasizing consistency in understanding ambiguous and disambiguated spatial expressions.
COMFORT introduces (1) a set of spatial reasoning tasks instantiated by synthetic 3D images and corresponding text describing spatial relations and (2) metrics to evaluate the robustness and consistency of the model responses.
We extend the setup to multilingual settings by evaluating models in 109 languages across 170 regions worldwide.
We find that VLMs show alignment with English conventions in spatial language understanding when resolving ambiguities.
However, they (1) are still far from achieving robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to linguistic and cultural conventions in cross-lingual tests, as English tends to dominate other languages.
With a growing effort to align vision-language models with human cognition, we highlight the ambiguous nature of spatial language and call for increased attention to cross-cultural diversity in spatial reasoning.

## 2 Background and Related Work

### 2.1 Spatial Language and Spatial Representation

Figure: Figure 1: In situated communication, spatial language understanding and reasoning are often ambiguous, leading to varying interpretations among people from different cultural backgrounds. Specifically: (a) different frames of reference can result in different interpretations of the same spatial expression; (b) speakers of different languages may use distinct coordinate frames for non-fronted reference objects; and (c) spatial relations extend beyond exact axes to include acceptable regions.
Refer to caption: x1.png

Some projective terms, such as the English words front, back, right, and left, convey meanings of spatial relations (Eschenbach, ).
These terms articulate the spatial relation between two entities within a designated frame of reference (FoR), often involving one entity as the reference object (relatum/ground) and another target object (referent/figure) that is positioned relative to the relatum along a specific axis/direction (Levinson, ; Frank, ).
In situated communication, speech act participants (e.g., an addressee) may also be considered (Danziger, ).
To determine acceptable uses of various spatial relations, existing theories suggest that people fit spatial templates, which are centered on the relatum and aligned with the FoR (Logan & Sadler, ), to parse out regions of acceptability of certain directions (Franklin et al., ; Carlson-Radvansky & Logan, ).

Ambiguities in frame of reference.
The choice of perspectives may lead to different FoRs, where Levinson () has identified three main types of FoR: absolute, intrinsic, and relative.
The absolute FoR uses cardinal directions, such as north and south, as fixed bearings.
The intrinsic FoR aligns the origin with the relatum, describing the referent’s position relative to the relatum’s inherent orientation.
The relative FoR positions a viewer (egocentric or addressee) as the origin, focusing on the observer’s intrinsic perspective.
Liu et al. () have highlighted the ambiguities in situated communication among three variations of intrinsic and relative FoRs (Figure 2): the egocentric relative, the addressee-centered relative, and the object-centered intrinsic FoRs.(^1^11We exclude the absolute FoR from our study as it introduces little ambiguity (Liu et al., ).)
When not specified, these FoRs are not easily distinguishable based solely on their linguistic expressions (Tenbrink, ).
To resolve the ambiguity, individuals from diverse linguistic and cultural backgrounds adopt different preferences and conventions in choosing FoRs (Majid et al., ; O’Meara & Báez, ; Bohnemeyer et al., ; Bender et al., ; Ogelo & Bylund, ).

Ambiguities in relative FoRs.
The variations of relative FoRs form another source of ambiguity.
After putting the origin of the coordination system on the viewer, multiple strategies specifying how to transform the axes can be considered (Figure 1b).
Different languages use different transformation conventions to resolve the ambiguity of the front-back and left-right of a relatum (Levinson, ; Shusterman & Li, ), including:
(1) translated projection (e.g., Hausa) where the coordinate frame of the speaker is directly applied, (2) rotated projection (e.g., Tamil), where the coordinate frame of the speaker is transformed with a 180-degree rotation, and (3) reflected projection (e.g., English), where only the front-back axis is reversed.

### 2.2 Spatial Understanding in Vision-Language Models

Large language models (LLMs) have exhibited strong adaptability that extends beyond text, encompassing 2D and 3D vision (Tsimpoukelli et al., ; Alayrac et al., ; Yang et al., ), their affordances in the physical embodiment (Driess et al., ; Qian et al., ; Yuan et al., ), and various other modalities (Yu et al., ).
Especially, a variety of vision-language models (VLM) have been developed by visual instruction tuning on paired text-image data (Dai et al., ; Liu et al., ; Dong et al., ).
With supervised fine-tuning using entity-phrase mappings in text-image pairs, grounded VLMs have been developed for fine-grained vision-language understanding at both the region (Chen et al., ; Bai et al., ; You et al., ; Peng et al., ) and pixel level (Lai et al., ; Xia et al., ; Rasheed et al., ; Zhang et al., ).

Figure: Figure 2: An illustrative example of how a frame of reference (FoR) specifies the reference system when describing the spatial relation between a target object (i.e., the ball) and a reference object (i.e., the car). When the FoR is not explicitly specified, English prefers an egocentric relative FoR, i.e., “the ball is behind the car.” We study FoRs that lead to ambiguity (Liu et al., ).
Refer to caption: x2.png

Spatial understanding is known to be challenging even for state-of-the-art VLMs and is receiving increasing attention (Achiam et al., ).
In addition to using explicit spatial language understanding modules (Rajabi & Kosecka, ), recent works such as SpatialVLM (Chen et al., ) and SpatialRGPT (Cheng et al., ) improve spatial reasoning in VLMs by leveraging 3D VQA or scene graph data for supervised fine-tuning.
Several benchmarks have also been developed to evaluate spatial reasoning in VLMs from various perspectives (Liu et al., ; Cheng et al., ; Kamath et al., ).
Still, these benchmarks overlook ambiguities related to the FoR, lack spatial continuity, and have not proposed metrics to evaluate the robustness and consistency of spatial reasoning.

## 3 Consistent Multilingual Frame of Reference Test ( COMFORT )

Figure: (a) Sample images from COMFORT-BALL. The 4 images on the left are selected every 90° interval along the rotational path out of 36 images. The 4 images on the right illustrate variations with a distractor, different object colors, sizes, or camera poses.
Refer to caption: x4.png

We introduce the COnsistent Multilingual Frame Of Reference Test (COMFORT), a new evaluation protocol with dataset, tasks, and comprehensive metrics, to study VLM behaviors in spatial language reasoning with FoR-related ambiguity.
This protocol accommodates spatial continuity and various ambiguities, drawing insights from several well-defined metrics to assess performance and prediction consistency.
Given our primary focus on the analytical inquiry of models’ linguistic competence (i.e., spatial knowledge encoded in the latent representations) rather than performance (i.e., behavioral evaluation) only (Warstadt & Bowman, ; Saxon et al., ),(^2^22Here, we use the terms competence and performance analogically to Chomsky ().) we additionally develop better evaluation and consistency metrics to deepen our understanding of model capabilities.

### 3.1 Task Formulation

Following the setups in object hallucination evaluation (Li et al., ; Chen et al., ), we formulate the task as a spatial relation inference problem.
In this task, a VLM $\mathcal{M}$ is presented with an RGB image $x_{\textrm{img}}$ and a textual question $x_{\textrm{query}}$.
The image shows the egocentric perception of a scene $s\in\mathcal{S}$, where $\mathcal{S}$ is the set of possible scenes in which the referent moves along a rotational trajectory with a constant radius from the relatum.
In contrast to fixing the referents on the standard canonical axes, this setup better mirrors the spatial continuity in common real-world scenarios.
A language prompt (Table 1) queries whether a spatial relation $r\in\mathcal{R}$ is satisfied by a referent-relatum pair in the image under FoR $f\in\mathcal{F}$ (Figure 2) in language $\ell\in\mathcal{L}$.
This work also examines models using queries with no FoR specified; therefore, a test case in COMFORT is defined as a 4-tuple in $\mathcal{S}\times\mathcal{R}\times\left(\mathcal{F}\cup\{\emptyset\}\right)
\times\mathcal{L}$.
While there are many spatial relations in daily languages, we primarily focus on four canonical directions; that is, the considered relation set $\mathcal{R}=\{\textit{to the left of},\textit{to the right of},\textit{in
front of},\textit{behind}\}$.
COMFORT covers $|\mathcal{L}|=109$ languages worldwide; however, we use English as an example to describe the data synthesis and evaluation processes for simplicity and clarity, and refer readers to Appendix A for more details.

### 3.2 Scene Setup

We render the scenes into images using Blender (Blender Online Community, ).
Each scene consists of a referent and a relatum.
The referent follows a rotational trajectory with a constant radius from the relatum to implement spatial continuity.
Starting from the canonical front direction, we move the referent with a uniform step of $10\degree$, totaling up to 36 images per scene.
In COMFORT, there are configurations determined by whether the relatum has an intrinsic semantic front:

- •
COMFORT-BALL: When the relatum is non-fronted (e.g., Figure 1b), we focus on the ambiguity of FoR conventions associated with different languages.
The split involves an observer’s egocentric perception of a referent (e.g., a red ball) and a non-fronted relatum (e.g., a blue ball).
We further randomize the dataset with object-level (colors, sizes, and shapes) and scene-level variations (camera positions and distractors) to consider more diverse yet reasonable settings (Figure 3(a)).
- •
COMFORT-CAR: When the relatum is fronted (e.g., Figure 1a), multiple FoRs can be explicitly adopted to interpret the scene.
A COMFORT-CAR image, therefore, involves the egocentric perception of a referent, a fronted relatum, and an additional human addressee.
One can interpret the spatial relations using either the Camera, Addressee, or Relatum (C/A/R) as the origin to resolve the reference frame ambiguity.
COMFORT-CAR has a set of 10 realistic objects in a typical household or outdoor scene, including horse, car, bench, laptop, rubber duck, chair, dog, sofa, bed, and bicycle, all of which have a clear semantic front.
We use a basketball as the referent and vary the relatum.
In addition to these objects, we include a human addressee in the scene.
To disentangle different FoRs as much as possible, we let the addressee face right, and let the relatum face either left or right in the rendered images from the rendering camera’s perspective (Figure 3(b)).

### 3.3 Language Query Setup

Given a pair of referent [A] and a relatum [B], together with a spatial relation of interest, the query is posed as “Is [A] [relation] [B]?”
Depending on whether or not and which FoR is specified, the query is appended after four different perspective prompts (Table 1): no perspective (nop), camera perspective (cam), addressee perspective (add), and relatum perspective (rel).
We only query from the camera egocentric perspective (cam) for COMFORT-BALL, focusing on the ambiguity introduced by variations of the relative FoR.
For COMFORT-CAR, we use all four possible language prompts to study how ambiguity in the reference system is resolved.
Overall, for English, the above data generation pipeline leads to 720 test cases in COMFORT-BALL, and 57.6k test cases in COMFORT-CAR.
The same method for dataset synthesis can be generalized to any other language; however, for computational efficiency, we only include the scenes corresponding to the four most prototypical directions (i.e., left, right, front, and back) in our multilingual analysis.

### 3.4 Metrics

Figure: Table 1: The origins of each coordinate system and the corresponding prompt templates for querying the FoR given a referent-relation-relatum triple.
Refer to caption: x6.png

Quantitatively assessing the spatial understanding and reasoning capabilities of models is challenging for two reasons.
First, the physical world is continuous, and spatial relations may extend beyond the precise canonical front-back and left-right axes.
As noted by Carlson-Radvansky & Logan (), there exists regions of acceptability where, for instance, an object slightly to the front-left might still be considered being “in front.”
Second, as Dentella et al. () pointed out, language models are biased towards affirmative responses.
However, the intermediate representations may be sensitive to variations in input and, to some extent, align with human perceptions of spatial cues.
Based on these concerns and findings, we introduce multiple metrics to evaluate the models’ competence to enable more nuanced analyses in addition to accuracy that measures performance.

Unless further clarified, we adopt a right-handed coordinate system with the thumb pointing upwards when describing angles.
We define the deviation angle $\theta\in(-180\degree,180\degree]$ as the angular displacement from the canonical direction $r$ to the vector connecting the relatum and target.
For example, in Figure 5, the deviation angle of canonical right from canonical front is $\theta=90\degree$.
Following Carlson-Radvansky & Logan (), we define the acceptable region for a spatial relation $r$ as the 180-degree hemisphere centered at the corresponding canonical direction.
For a VLM $\mathcal{M}$ and a test case indexed by $i$, we let $P_{i}(\textit{response};\mathcal{M})$ denote the probability of $\textit{response}\in\{\text{Yes},\text{No}\}$ assigned by $\mathcal{M}$, and abbreviate it as $P_{i}(\textit{response})$ if there is no confusion.

Accuracy. Given a spatial relation $r$ in the textual prompt, we assess whether the assigned response probabilities correspond to whether the referent lies within the acceptable region defined by the relatum and $r$.
Formally, we define the local probability of the model responding ‘Yes’ by $p_{i}=P_{i}(\text{Yes})/\left[P_{i}(\text{Yes})+P_{i}(\text{No})\right]$.
We consider the inference correct if (1) the scene falls into the acceptability region and $p_{i}>0.5$ or (2) the scene falls out of the acceptability region and $p_{i}\leq 0.5$.

Region Parsing Error.
To mitigate the known bias towards affirmative answers, where the expectation of $p_{i}$ is generally higher than $0.5$, we normalize it across all image-prompt pairs, resulting in the normalized probability

$$ $\displaystyle\widehat{p}_{i}:=\frac{p_{i}-\min_{j}p_{j}}{\max_{j}p_{j}-\min_{j }p_{j}}.$ $$

We adopt the root mean square error (RMSE) between the normalized acceptance probability $\widehat{p}$ and reference probability threshold $\lambda^{\textrm{ref}}$ that represents the actual regions of acceptability,

$$ $\displaystyle\varepsilon^{\textrm{ref}}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}{( \widehat{p}_{i}-\lambda^{\textrm{ref}})^{2}}},$ $$

where $\lambda^{\textrm{ref}}$ denotes the reference of the assigned probability, analogically to ground-truth labels in machine learning terms.

We introduce two analytically and geometrically motivated proposals defining $\lambda^{\textrm{ref}}$, $\lambda^{\textrm{hemi}}$ and $\lambda^{\textrm{cos}}$, based on hemispheres and cosine of angles, respectively.
First, the hemisphere-based reference $\lambda^{\textrm{hemi}}$ is defined as

$$ $\displaystyle\lambda^{\textrm{hemi}}(\theta):=\begin{cases}1&\text{if }\theta \in(-90\degree,90\degree)\\ 0&\text{if }\theta\in(-180\degree,-90\degree]\cup[90\degree,180\degree].\end{ cases}\vspace*{-10pt}$ $$

Here, $\theta=0\degree$ corresponds to the most prototypical spatial relation, and $\theta=180\degree$ corresponds to the opposite.
Intuitively, $\lambda^{\textrm{hemi}}=1$ denotes the test case falls into the acceptable region defined by the textual prompt, and otherwise not.

The second reference is derived from the cosine of the deviation angle.
Matching the range of the cosine function to that of probability, i.e., $[0,1]$, we define the cosine-based reference $\lambda^{\textrm{cos}}(\theta)$ by

$$ $\displaystyle\lambda^{\textrm{cos}}(\theta):=[\cos(\theta)+1]/2.$ $$

Figure 5 shows an example of the vanilla probability curve $p(\theta)$ from LLaVA-v1.5-7B (Liu et al., ), normalized probability curve $\widehat{p}(\theta)$, and two reference curves $\lambda^{\textrm{hemi}}(\theta)$ and $\lambda^{\textrm{cos}}(\theta)$.
We report both $\varepsilon^{\textrm{hemi}}(\theta)$ and $\varepsilon^{\textrm{cos}}(\theta)$ in experiments.
We also note that in human spatial cognition, the regions of acceptability are neither mutually exclusive 90° quadrants nor overlapping 180° hemispheres, as they vary across individuals and depend on the situational context (Franklin et al., ).
We discuss the limitations of this design in Section 5.4.

Figure: (a) An illustration of the spatial symmetry with respect to the (conventional) front. As the red ball rotates around the blue ball, spatial symmetry consistency ensures that each symmetric pair, with different deviation angles $\theta$ and $-\theta$, has the same probability of being identified as the front.
Refer to caption: x8.png

### 3.5 Robustness Metrics

Standard deviation.
In COMFORT, some images depict variations of the same scene, sharing identical spatial relations between the referent and the object but differing in terms of object colors, sizes, or distractors.
When the spatial relation and the query text remain unchanged, an ideal model should have consistent predictions for all variations.
To measure the robustness of the model prediction, we report the average standard deviation of the predicted probability $\widehat{p}_{i}$ across all deviation angles $\sigma:=\textrm{avg}_{\theta}{\sigma(\theta)}$.

#### Prediction noise.

Since our data is generated through interpolation, ideally, if a model well understands spatial relations, the probability curve with respect to the deviation angle should be low-frequency (i.e., smooth) rather than high-frequency (i.e., noisy).
Therefore, we measure the noise by the RMSE, denoted by $\eta$, between the predicted probability and a Butterworth Low Pass Filter (LPF; Butterworth et al., ):

$$ $\displaystyle\eta:=\sqrt{\frac{1}{n}\sum_{i=1}^{n}{[\widehat{p}_{i}-\textrm{ LPF}(\widehat{p}_{i})]^{2}}}.$ $$

A smaller value of $\eta$ indicates that the probability curve is smoother, which is more desirable.

### 3.6 Consistency Metrics

Spatial symmetric consistency.
A critical aspect of consistent spatial reasoning is geometric symmetry.
As our tested target object rotates around the relatum in a circular path that is spatially symmetric, we expect the probabilities of an ideal VLM to consistently reflect geometric symmetry (Figure 6(a)).
For a pair of test cases, indexed by $i$ and $j$, that have the same configurations but opposite deviation angles, i.e., $\theta_{i}+\theta_{j}=0\degree$, we define the symmetry consistency:

$$ $\displaystyle c^{\textrm{sym}}:=\sqrt{\frac{2}{n-2}\sum_{i,j}(\widehat{p}_{i}- \widehat{p}_{j})^{2}}.$ $$

Spatial opposite consistency.
Similarly, we expect the probabilities of an ideal VLM to consistently reflect geometric opposition (Figure 6(b)).
For example, the probability that a sample is accepted by the spatial relation “to the left” should be identical to the probability that it is rejected by “to the right.”
For a pair of opposite spatial relation $r,\textrm{opp}(r)\in\mathcal{R}$ with the same configurations including the deviation angles $\theta_{i}$, the opposition consistency is given as:

$$ $\displaystyle c^{\textrm{opp}}:=\sqrt{\frac{1}{n}\sum_{i=0}^{n}(\widehat{p}_{i }^{r}+\widehat{p}_{i}^{\textrm{opp}(r)}-1)^{2}}.$ $$

## 4 Empirical Experiments and Main Findings

The COMFORT framework enables us to investigate whether the internal representations of vision-language models encode spatial relations, and if they do, which underlying coordinate systems these representations capture.
This can further be broken down into two research questions:
(1) When presented with an ambiguous spatial expression, do VLMs follow conventions and exhibit specific preferred FoRs (and the coordinate transformation in relative FoRs) to resolve the ambiguity?
(2) How effectively can VLMs adopt different FoRs, when perspectives are explicitly specified to disambiguate spatial expressions paired with visual scenes?

In principle, COMFORT can be applied to all VLMs, whether multilingual or monolingual.
We note that most existing open-source VLMs are English-based language models; therefore, we begin our experiments on English conventions, where both relative and intrinsic FoRs are available, but there is a conventional preference for a relative FoR combined with a reflected coordinate transformation in the relative FoR (see Levinson, , Table 5.4).
We further extend our setup to multilingual settings by evaluating models in 109 languages across 170 regions worldwide.
To cover a variety of VLMs with different capabilities and training approaches, we evaluate the following models:

- •
VLMs build from supervised instruction fine-tuning: InstructBLIP-7B/13B (Dai et al., ), LLaVA-v1.5-7B/13B (Liu et al., ), InternLM-XComposer2-7B (Dong et al., );
- •
VLMs with both supervised fine-tuning and reinforcement learning alignment: MiniCPM-Llama3-V-v2.5-8B (Hu et al., ; Yu et al., );
- •
Mechanistically grounded VLMs: GLaMM-7B (Rasheed et al., );
- •
Multilingual VLMs: mBLIP-BLOOMZ-7B (Geigle et al., ) and GPT-4o (OpenAI, ).

Figure: (a) Behind in GPT-4o.
Refer to caption: x10.png

**Table 2: Preferred coordinate transformation mapping from the egocentric viewer (camera) to the relatum in the relative FoR. The cosine region parsing errors $\varepsilon^{\textrm{cos}}$ are computed against both the Same and Reversed directions relative to the egocentric viewer’s coordinate system. For example, native English speakers typically prefer a Reflected transformation, which maintains the lateral (left/right) axis but reverses the sagittal (front/back) axis relative to the viewer (Figure 1). We determine the preferred transformation based on the aggregated performance, with “-” for no significant preference.**
| Model | Back $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Front $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Left $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Right $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Aggregated | Preferred |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Same | Rev. | Same | Rev. | Same | Rev. | Same | Rev. | Tran. | Rot. | Ref. |  |  |
| InstructBLIP-7B | 45.6 | 39.0 | 31.6 | 52.0 | 37.2 | 48.0 | 47.5 | 37.8 | 40.5 | 44.2 | 43.9 | - |
| InstructBLIP-13B | 40.9 | 45.5 | 46.0 | 37.4 | 43.4 | 44.9 | 45.6 | 41.6 | 44.0 | 42.3 | 43.0 | - |
| mBLIP | 51.2 | 53.7 | 51.2 | 47.9 | 52.4 | 53.5 | 54.6 | 46.8 | 52.3 | 50.5 | 52.1 | - |
| GLaMM | 58.3 | 33.3 | 43.9 | 42.9 | 38.3 | 51.8 | 17.3 | 63.7 | 39.5 | 47.9 | 33.0 | Ref. |
| LLaVA-1.5-7B | 54.0 | 32.9 | 59.1 | 24.8 | 11.9 | 70.0 | 13.0 | 68.5 | 34.5 | 49.0 | 20.7 | Ref. |
| LLaVA-1.5-13B | 61.8 | 19.2 | 56.0 | 27.7 | 31.7 | 61.8 | 24.3 | 64.3 | 43.4 | 43.2 | 25.7 | Ref. |
| XComposer2 | 73.2 | 17.9 | 74.5 | 20.7 | 20.1 | 80.9 | 21.3 | 81.1 | 47.3 | 50.1 | 20.0 | Ref. |
| MiniCPM-V | 70.9 | 21.9 | 64.3 | 26.9 | 19.7 | 74.1 | 21.1 | 73.3 | 44.0 | 49.1 | 22.4 | Ref. |
| GPT-4o | 75.7 | 28.2 | 73.6 | 32.0 | 24.3 | 80.8 | 25.1 | 80.8 | 49.7 | 55.5 | 27.4 | Ref. |

**Table 3: Preferred frame of reference in VLMs. Models’ Cosine Region Parsing Errors $\varepsilon^{\textrm{cos}}$ are computed against the Intrinsic FoR (relatum origin), Egocentric relative FoR (camera origin), and Addressee-centric relative FoR (addressee origin). English typically prefers an egocentric relative FoR. We determine the preferred FoR based on the aggregated performance, with “-” indicating no significant preference.**
| Model | Back $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Front $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Left $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Right $\varepsilon^{\textrm{cos}}$ ($\downarrow$) | Aggregated | Prefer |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ego. | Int. | Add. | Ego. | Int. | Add. | Ego. | Int. | Add. | Ego. | Int. | Add. | Ego. | Int. | Add. |  |  |
| InstructBLIP-7B | 41.0 | 38.6 | 38.6 | 40.9 | 46.9 | 46.9 | 45.6 | 32.5 | 51.9 | 39.6 | 51.2 | 31.8 | 41.8 | 42.3 | 42.3 | - |
| InstructBLIP-13B | 32.9 | 34.4 | 34.4 | 52.5 | 48.5 | 48.5 | 47.8 | 56.2 | 27.8 | 40.6 | 27.6 | 56.6 | 43.5 | 41.7 | 41.8 | - |
| mBLIP-BLOOMZ | 52.2 | 53.2 | 53.2 | 45.3 | 44.6 | 44.6 | 47.8 | 47.6 | 48.1 | 45.4 | 48.4 | 42.4 | 47.7 | 48.4 | 47.1 | - |
| GLaMM | 28.0 | 49.1 | 49.1 | 30.0 | 40.2 | 40.2 | 14.0 | 56.8 | 41.5 | 13.7 | 53.0 | 46.6 | 21.4 | 49.8 | 44.4 | Ego. |
| LLaVA-1.5-7B | 20.9 | 43.0 | 43.0 | 34.5 | 32.6 | 32.6 | 13.4 | 53.5 | 47.4 | 14.3 | 53.6 | 49.3 | 20.8 | 45.7 | 43.1 | Ego. |
| LLaVA-1.5-13B | 31.9 | 38.8 | 38.8 | 24.8 | 57.1 | 57.1 | 11.7 | 51.1 | 51.1 | 27.5 | 57.4 | 48.7 | 24.0 | 51.1 | 48.9 | Ego. |
| XComposer2 | 12.7 | 49.3 | 49.3 | 15.2 | 48.3 | 48.3 | 18.8 | 61.2 | 53.7 | 16.5 | 58.4 | 54.5 | 15.8 | 54.3 | 51.4 | Ego. |
| MiniCPM-V | 34.2 | 40.7 | 40.7 | 35.5 | 53.4 | 53.4 | 18.0 | 53.9 | 58.4 | 19.0 | 58.1 | 52.7 | 26.7 | 51.5 | 51.3 | Ego. |
| GPT-4o | 38.3 | 36.7 | 36.7 | 43.1 | 50.2 | 50.2 | 34.7 | 59.3 | 56.5 | 24.3 | 57.3 | 61.7 | 35.1 | 50.9 | 51.3 | Ego. |

### 4.1 Most VLMs Prefer Reflected Coordinate Transformation Convention

In this section, we address the research question:
do VLMs have a preferred coordinate transformation convention, and if so, what is it?
Experiments are conducted on COMFORT-BALL using the camera perspective prompt (cam) that explicitly specifies an egocentric relative FoR (Table 2).
Table 7 in the appendix shows the complete evaluation including $\varepsilon^{\textrm{hemi}}$ and $\varepsilon^{\textrm{cos}}$.

We observe that almost all VLMs demonstrate a clear preference over the reflected transformation similar to English, except the BLIP series.
Still, some models are also affected by the ambiguity of multiple transformation conventions.
With the textual prompting specifying a relation, at $\theta=0$, GPT-4o and LLaVA-1.5-13B show a sharp drop of performance and a significant variance for behind and right, respectively (Figure 7), indicating that some models are sensitive to other transformations.

### 4.2 Most VLMs Prefer Egocentric Relative Frame of Reference

We now attempt to answer the research question:
do VLMs have a preferred frame of reference, and if so, what is it?
We conduct our study on COMFORT-CAR using the no perspective prompt (nop) that deliberately leaves the FoR ambiguous.
When calculating the performance with respect to relative FoRs (either egocentric or addressee-centered), we assume a reflected coordinate transformation convention.
Table 3
shows the results of preferred FoR in English measured by the region parsing error $\varepsilon^{\textrm{cos}}$, and Table 8 in the appendix shows the complete evaluation including both $\varepsilon^{\textrm{hemi}}$ and $\varepsilon^{\textrm{cos}}$.
Almost all VLMs demonstrate a significant preference for the egocentric relative FoR similar to English, again, except for the BLIP series.
Additionally, the models’ performances are inconsistent across different spatial relations—models generally perform better in the lateral directions (left and right) than the sagittal ones (front and behind), even in competitive industry models like GPT-4o.
For instance, GLaMM does not show a very strong preference when resolving ambiguities along the sagittal axes, but it demonstrates a significant preference when resolving lateral ambiguity.

**Table 4: The accuracy and cosine region parsing errors of VLMs when explicitly prompted to follow each frame of reference are provided (cam/rel/add). The values in parentheses indicate the performance change relative to the scenario with no perspective (nop) prompting.**
| Model | Egocentric | Intrinsic | Addressee | Aggregated |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc% $(\uparrow)$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}(\downarrow)$ | Acc% $(\uparrow)$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}(\downarrow)$ | Acc% $(\uparrow)$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}(\downarrow)$ | Acc% $(\uparrow)$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}(\downarrow)$ |  |
| InstructBLIP-7B | 47.2(+0.0) | 43.5(+1.7) | 47.2(+0.0) | 42.3(+0.0) | 47.2(+0.0) | 43.6(+1.3) | 47.2(+0.0) | 43.1(+1.0) |
| InstructBLIP-13B | 47.2(+0.0) | 43.8(+0.3) | 47.2(+0.0) | 43.2(+1.5) | 47.2(+0.0) | 42.9(+1.1) | 47.2(+0.0) | 43.3(+1.0) |
| mBLIP-BLOOMZ | 51.9(-0.9) | 55.4(+7.7) | 49.8(-3.0) | 54.2(+5.8) | 49.6(-3.2) | 55.8(+8.7) | 50.4(-2.4) | 55.1(+7.4) |
| GLaMM | 47.2(-10.6) | 23.3(-0.7) | 47.2(+0.8) | 44.2(-6.9) | 47.2(-2.8) | 42.8(-6.1) | 47.2(-4.2) | 36.8(-4.6) |
| LLaVA-1.5-7B | 55.2(-2.6) | 18.4(-3.0) | 48.3(+4.7) | 45.7(-4.1) | 48.2(-5.0) | 43.4(-1.0) | 50.6(-1.0) | 35.8(-2.7) |
| LLaVA-1.5-13B | 51.6(-15.0) | 23.9(+3.1) | 47.3(+0.8) | 45.0(-0.7) | 47.5(-3.8) | 38.9(-4.2) | 48.8(-6.0) | 35.9(-0.6) |
| XComposer2 | 85.6(-7.0) | 18.8(+3.0) | 51.0(+0.5) | 51.0(-3.3) | 53.2(-0.6) | 49.8(-1.6) | 63.3(-2.4) | 39.9(-0.6) |
| MiniCPM-V | 72.4(-4.8) | 24.6(-2.1) | 49.9(-2.6) | 47.8(-3.7) | 52.9(-0.5) | 45.1(-6.2) | 58.4(-2.6) | 39.2(-4.0) |
| GPT-4o | 78.3(+4.6) | 28.1(-7.0) | 53.4(-1.9) | 44.6(-6.3) | 49.1(-5.7) | 44.9(-6.4) | 60.3(-1.0) | 39.2(-6.6) |

**Table 5: A comprehensive evaluation of VLMs in egocentric relative FoR with reflected transformation, using an explicit camera perspective (cam) prompt, is conducted. The metrics considered include object hallucination (F1-score), accuracy (Acc), region parsing error ($\varepsilon$), prediction noise ($\eta$), standard deviation ($\sigma$), and consistency ($c$).**
| Model | Obj F1 ($\uparrow$) | Acc% ($\uparrow$) | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ ($\downarrow$) | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ ($\downarrow$) | $\sigma_{\times 10^{2}}$ ($\downarrow$) | $\eta_{\times 10^{2}}$ ($\downarrow$) | $c^{\textrm{sym}}_{\times 10^{2}}$ ($\downarrow$) | $c^{\textrm{opp}}_{\times 10^{2}}$ ($\downarrow$) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BALL | CAR | BALL | CAR | BALL | CAR | BALL | CAR | BALL | CAR | BALL | CAR | BALL | CAR | BALL | CAR |  |
| InstructBLIP-7B | 66.7 | 66.7 | 47.2 | 47.2 | 43.9 | 43.5 | 57.8 | 56.4 | 26.7 | 30.5 | 48.4 | 43.4 | 17.2 | 16.9 | 16.6 | 22.6 |
| InstructBLIP-13B | 67.3 | 50.3 | 47.2 | 47.2 | 43.0 | 43.8 | 55.5 | 55.9 | 27.1 | 36.8 | 48.2 | 46.4 | 17.3 | 17.0 | 21.0 | 21.9 |
| mBLIP-BLOOMZ | 99.1 | 33.3 | 47.5 | 51.9 | 52.1 | 55.4 | 62.1 | 65.6 | 43.7 | 48.6 | 54.1 | 60.7 | 29.1 | 30.1 | 33.8 | 42.0 |
| GLaMM | 100.0 | 99.8 | 47.2 | 47.2 | 33.0 | 23.3 | 45.2 | 37.6 | 29.9 | 23.4 | 45.0 | 28.4 | 10.1 | 9.4 | 13.7 | 14.6 |
| LLaVA-1.5-7B | 100.0 | 88.6 | 63.2 | 55.2 | 20.7 | 18.4 | 33.7 | 32.5 | 25.2 | 20.0 | 23.5 | 21.8 | 5.8 | 5.4 | 8.3 | 10.7 |
| LLaVA-1.5-13B | 100.0 | 98.6 | 55.3 | 51.6 | 25.7 | 23.8 | 37.6 | 37.1 | 19.3 | 20.8 | 24.9 | 29.9 | 7.0 | 5.8 | 9.3 | 10.8 |
| XComposer2 | 100.0 | 95.3 | 92.4 | 85.6 | 20.0 | 18.8 | 21.1 | 26.3 | 19.2 | 15.3 | 13.7 | 22.9 | 9.0 | 6.5 | 10.5 | 12.0 |
| MiniCPM-V | 66.8 | 81.5 | 81.0 | 72.4 | 22.4 | 24.6 | 32.8 | 35.8 | 19.2 | 19.2 | 29.8 | 22.7 | 10.1 | 9.2 | 12.4 | 14.9 |
| GPT-4o | 100.0 | 94.5 | 89.2 | 78.3 | 27.4 | 28.1 | 27.5 | 35.0 | 20.9 | 24.0 | 43.1 | 38.8 | 14.1 | 13.3 | 14.2 | 16.7 |
| Random (30 trials) | 50.0 | 50.9 | 46.3 | 58.7 | 28.3 | 26.6 | 42.5 | 44.2 |  |  |  |  |  |  |  |  |
| Always “Yes” | 50.0 | 47.2 | 61.2 | 68.7 | 0.0 | 0.0 | 0.0 | 100.0 |  |  |  |  |  |  |  |  |

### 4.3 VLMs Fail to Adopt Alternative Frames of Reference Flexibly

We now address the research question:
can VLMs adopt different FoRs when perspectives are explicitly specified to disambiguate spatial expressions?
We again use COMFORT-CAR; however, instead of using the no-perspective prompt (nop), we require VLMs to follow one FoR by explicitly specifying the perspective (cam/rel/add) in the textual prompts (Table 1).
Table 4 shows the results in accuracy and $\varepsilon^{\textrm{cos}}$ and the performance compared to when no perspective is specified, and Table 9 in the appendix gives the complete evaluation.
We find that all models, including the strong ones like GPT-4o and InternLM-XComposer2, show close-to-chance performance (50% accuracy) when being prompted to use the intrinsic or addressee-centered relative FoRs.
Compared to the same probing setup without a perspective specified (nop), we find generally marginal improvements in region parsing error ($\varepsilon$), whereas the accuracy decreases.
Overall, the results indicate that while VLMs can comprehend scenes using egocentric relative FoR, they struggle to adapt flexibly to alternative FoRs.

### 4.4 Spatial Representations in VLMs Are Not Robust and Consistent

In this section, we further ask:
are spatial representations in VLMs robust and consistent?
The considered metrics include accuracy (Acc), region parsing error ($\varepsilon$), prediction noise ($\eta$), standard deviation ($\sigma$), and consistency ($c$) as defined in Section 3.
One commonly considered possibility that VLMs underperform is that they suffer from object hallucination, where they misperceive objects in the scenes (Li et al., ; Chen et al., ).
Following the object probing setups, we prompt the VLMs to inquire about the presence of an existing object and a non-existing object in the scene, and compute the F1-score (Table 5).
We find that the BLIP models suffer from severe object hallucinations, which contribute to their underperformance in the previous evaluation.
Many VLMs, despite showing decent performance metrics in terms of spatial understanding and reasoning accuracy, demonstrate a lack of robustness and consistency.
For example, the spatial opposite consistency ($c^{\textrm{opp}}$) of GPT-4o is not significantly better than 30 random trials.
In contrast, VLMs that have undergone supervised fine-tuning on spatial relation tasks have a more robust and consistent spatial representation.
For instance, InternLM-XComposer2 and MiniCPM-V (on the COMFORT-BALL task, with no object hallucinations) show improved performance.
On the other hand, although GLaMM is mechanistically grounded to objects and exhibits minimal object hallucination, its spatial understanding capability is poor.
This suggests that improving visual entity grounding helps in recognizing individual objects but does not automatically translate to better spatial understanding between multiple objects.

### 4.5 A Cross-lingual and Cross-cultural Evaluation of Frame of Reference

Figure: Figure 8: A visualization of the world map that displays the preference of each region for using the intrinsic FoR over the relative FoR. The plot is based on the top three spoken languages in each region, as reported by The World Factbook (Central Intelligence Agency, ), and averages the cosine parsing error ($\varepsilon^{\textrm{cos}},\;\downarrow$), weighted by the speaking population. We present a quantitative comparison of English, Tamil, and Hausa, with the best-performing FoR marked in bold and the convention preferred by human speakers underlined.
Refer to caption: x12.png

All previous experiments are centered around English; however, individuals from diverse multilingual and cultural backgrounds may adopt different preferences and conventions to select their FoR in resolving ambiguities (Majid et al., ; O’Meara & Báez, ; Bohnemeyer et al., ; Ogelo & Bylund, ).
Our next research question naturally arises:
Does multilingual VLMs faithfully follow the preferences and conventions (associated with different languages) to select the FoR?
To extend the study of preferred FoR from English to a multilingual setting, we evaluate 109 languages worldwide to investigate whether each language shows a preferred FoR.
We translate the English prompts into the target languages using the Google Cloud Translate API.
Given that the open-source language models either lack strong multilingual capabilities or underperform in previous evaluations, we study this problem on the GPT-4o model (OpenAI, ).
We follow the setup similar to Section 4.2, but only evaluate the images corresponding to the four canonical directions using the nop prompt.
For each language, we compute $\varepsilon^{\textrm{cos}}$ for each FoR and coordinate transformation.
Figure 8 presents a visualization of the world map, displaying the preference of each region for using the (object-centered) intrinsic FoR over the relative FoR, where the latter corresponds to a low $\varepsilon^{\textrm{cos}}$ value.
Table 10 summarizes the results for all languages tested.

Nearly all tested languages demonstrate a preference towards the relative FoR, except several underrepresented languages, such as Konkani, Kurdish, and Amharic, which exhibit near-random performance without a significant preference.
In Figure 8, we present a classic comparison between English, Tamil, and Hausa similar to that of Levinson (), with the best-performing FoR marked in bold, and the preferred convention by humans underlined.
Although human speakers of these languages have different preferred coordinate transformation conventions, the English convention of reflected projection is observed for both Tamil and Hausa.
Although, for example, Hausa permits an English-like interpretation of front-back relations, this interpretation is generally less favored and may confuse Hausa speakers (Hill, ).
This raises concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs.

## 5 Discussions

### 5.1 Do vision-language models represent space and how?

It is insufficient to answer this question by simply querying the model with text-image pairs and comparing the output with a fixed ground truth.
We must, at least, query the models with awareness of the ambiguity in FoRs, which is essential in determining how the scenes in the physical world are mapped to spatial expressions (Levinson, ).
Our experiments confirm that many VLMs are equipped with reasonable spatial representations through vision-language training alone; in particular, most VLMs clearly prefer the egocentric relative FoR with reflected projection, aligning with English conventions.
However, our results also show these representations lack robustness and consistency in a continuous space.
Similar experimental setups can yield widely varying performance across different spatial relations—for example, GPT-4 shows minimal preferences for the egocentric relative FoR along the sagittal axis but a significant preference along the lateral one (Table 3).
As a result, VLMs demonstrate unsatisfactory consistency in their spatial performance (Table 5).
Future work is necessary to improve the consistency and robustness of spatial representations in these models.

Our research also aligns with the growing trend of grounding linguistic analysis in rich modalities representing the real world (Chai et al., ; Shi, ).
Language is not text in isolation; its meaning is significantly enriched when grounded in real-world contexts.
For example, the ambiguity of spatial terms, as discussed in this paper, becomes meaningful only when combined with FoRs, and these FoRs are much more intuitively illustrated when visual cues are available.
We advocate for future work that extends linguistic analysis to more grounded settings.

### 5.2 Perspective taking as a prerequisite of human-like spatial reasoning

Most languages support multiple FoRs.(^3^33Some languages, in very rare cases, have only one available spatial frame of reference. For example, Guugu Yimithirr exclusively uses the absolute FoR (Levinson, ).)
The ability to understand and reason about space from a non-egocentric perspective is an important foundation of the Theory of Mind, a basic building block of our situated communication skill that allows us to infer others’ mental states (Ma et al., ).
One of our key findings is that VLMs still struggle to adopt alternative FoRs flexibly, even when provided with explicit perspective-taking instructions (Section 4.3).
We hypothesize that this phenomenon may come from a reporting bias in the image-text datasets available on the internet—it is natural to take the reflected relative FoR to view images presented on a screen, but this does not always apply in real-world applications.
To address this issue, we suggest future work extend the current 2D VLMs to the 3D domain, by considering camera poses and multiview data (Yang et al., ) for training.

### 5.3 Cross-cultural conventions in cross-lingual spatial understanding

The conventions for resolving spatial ambiguities are not uniform, as individuals from diverse linguistic and cultural backgrounds select their FoR differently.
Cultural conventions can even be transmitted as individuals are exposed to other languages.
Interestingly, Bohnemeyer et al. () found that among native speakers of Indigenous languages (with various preferences in FoRs), those more proficient in Spanish tend to use the reflected relative FoR (Spanish convention) more in their native language.
This phenomenon has led to their Linguistic Transmission Hypothesis: “Using any language or linguistic variety – independently of its structures – may facilitate the acquisition of cultural practices of nonlinguistic cognition shared among the speakers of the language.”
Analogously, our experiment raises important concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs.
This is not surprising, as current training recipes for multilingual multimodal language models heavily rely on machine-translated captions (Chen et al., ; Geigle et al., ).
However, this practice can be problematic.
For instance, Hausa prefers an interpretation where the “front” aligns with the English concept of “back,” (Hill, ), where this approach may lead to English conventions overshadowing those of other languages.
At a high level, this issue is not limited to spatial reasoning—for example, Shi et al. () have demonstrated that English is always the best chain-of-thought language for math reasoning with multilingual LLMs, no matter what language is used for the problem description.
To enable similar linguistic transmission in AI models, exposure to naturally generated multilingual data is crucial (Romero et al., ).

### 5.4 Limitations and Future Work

The acceptance regions.
Using cosine and hemisphere as acceptance regions is analytical but might not capture some human cognitive biases.
In reality, regional angles might not be uniformly distributed per relation, nor are they exactly 90 degrees.
These angles vary across individuals and cultures (Franklin et al., ).

Spatial relations.
This work primarily focuses on the most basic types of spatial relations (front-back and left-right).
However, many other relations exist, such as away from and near (Logan & Sadler, ; Liu et al., ).
Additionally, not all languages possess terms for “left,” “right,” “front,” and “back.” Some languages, like Guugu Yimithirr, use only absolute frames of reference instead (Levinson, ).

Camera angle and occlusion.
Currently, there is minimal occlusion, and the camera angle is high.
Languages may differ in whether the speaker emphasizes these factors, such as the preference to use “behind” in cases of occlusion (Levinson, ).

Pragmatic aspect of spatial cognition.
Many conversational and pragmatic aspects of spatial cognition are simplified in this work, such as F-Formation (Kendon, ) and human-robot interaction (Liu et al., ).
For example, in human-robot interaction settings, users prefer an addressee-centered frame of reference to facilitate the robot’s comprehension of referents (Moratz & Tenbrink, ).

Multilingual prompts.
In this work, we used machine-translated text to construct the multilingual portion of the dataset.
Although we verified data quality through back translation, incorporating human annotations in the future would be a valuable future step.

## Acknowledgments

This work was supported in part by NSF IIS-1949634, NSERC RGPIN-2024-04395, and the Microsoft Accelerate Foundation Models Research (AFMR) grant program.
Ziqiao Ma is supported in part by the Weinberg Cognitive Science Fellowship.
The authors would like to thank Yinpei Dai, Run Peng, Jung-Chun Liu, and Xuejun Zhang for proofreading and feedback.

## Appendix A Dataset and Metric Details

### A.1 Dataset Configurations

The entire data generation pipeline produces 720 English test cases in COMFORT-BALL, and 57.6k English test cases in COMFORT-CAR.
For COMFORT-BALL: 1 object combination $\times$ 5 variants $\times$ 4 relations $\times$ 36 angles = 720 test cases.
For COMFORT-CAR: 20 object combinations $\times$ 5 variants $\times$ 4 relations $\times$ 36 angles $\times$ 4 prompts = 57,600 test cases.
The table below lists all possible variants and configurations for the dataset, and we describe our dataset configuration in detail as follows.

**Table 6: A test case in COMFORT is defined as a 4-tuple in $\mathcal{S}\times\mathcal{R}\times\left(\mathcal{F}\cup\{\emptyset\}\right) \times\mathcal{L}$. This table enumerates all possible variants and configurations of the dataset.**
| Test Case Setup | Possible Variants |
| --- | --- |
| Scene $\mathcal{S}$ | COMFORT-BALL: Relatum: red ball; Referent: blue ball; 36 samples uniformly collected along a rotational path. |
| COMFORT-CAR: Relatum: basketball; Referent: horse, car, bench, laptop, rubber duck, chair, dog, sofa, bed, bicycle; Addressee: woman; 36 samples uniformly collected along a rotational path. |  |
| Spatial Relation $\mathcal{R}$ | to the left of, to the right of, in front of, behind |
| Frame of Reference $\mathcal{F}$ | egocentric relative, addressee-centered relative, object-centered intrinsic |
| Language $\mathcal{L}$ | See Table 10. |

### A.2 List of Evaluated Languages

We started with 132 candidate languages supported by Google Translate API.(^4^44[https://cloud.google.com/translate](https://cloud.google.com/translate))
We removed 23 languages from our multilingual evaluation due to their failure to adhere to instructions for generating “yes” and “no” predictions, or because they did not pass the back-translation test for quality control: Aymara, Bambara, Croatian, Dhivehi, Dogri, Ewe, Guarani, Hmong, Kyrgyz, Luganda, Malayalam, Meiteilon (Manipuri), Mizo, Odia (Oriya), Punjabi, Quechua, Samoan, Tatar, Telugu, Tigrinya, Uyghur, Xhosa, Yoruba.

### A.3 Visualizations of Region Parsing Error

Figure: (a) $\varepsilon^{\textrm{hemi}}=10.7$
Refer to caption: x13.png

## Appendix B Addendum to Results

**Table 7: The full results for testing the preferred coordinate transformation mapping from the viewer to the relatum in the relative frame of reference.**
|  | Back | Front |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Same | Reversed | Same | Reversed |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |
| InstructBLIP-7B | 47.2 | 58.4 | 45.6 | 47.2 | 53.8 | 39.0 | 47.2 | 47.5 | 31.6 | 47.2 | 64.6 | 52.0 |
| InstructBLIP-13B | 47.2 | 55.9 | 40.9 | 47.2 | 56.6 | 45.5 | 47.2 | 60.0 | 46.0 | 47.2 | 53.0 | 37.4 |
| mBLIP | 56.1 | 60.2 | 51.2 | 47.2 | 64.8 | 53.7 | 51.1 | 61.4 | 51.2 | 47.8 | 58.0 | 47.9 |
| GLaMM | 47.2 | 71.1 | 58.3 | 47.2 | 46.3 | 33.3 | 47.2 | 55.4 | 43.9 | 47.2 | 55.9 | 42.9 |
| LLaVA-1.5-7B | 47.2 | 66.7 | 54.0 | 47.2 | 47.0 | 32.9 | 47.2 | 71.0 | 59.1 | 47.2 | 36.4 | 24.8 |
| LLaVA-1.5-13B | 47.2 | 73.8 | 61.8 | 47.2 | 36.3 | 19.2 | 42.8 | 67.3 | 56.0 | 51.7 | 39.1 | 27.7 |
| XComposer2 | 13.3 | 84.5 | 73.2 | 90.0 | 26.3 | 17.9 | 15.0 | 85.8 | 74.5 | 85.0 | 31.6 | 20.7 |
| MiniCPM-V | 13.9 | 84.1 | 70.9 | 80.6 | 35.6 | 21.9 | 26.1 | 77.0 | 64.3 | 75.0 | 35.3 | 26.9 |
| GPT-4o | 16.7 | 87.3 | 75.7 | 87.8 | 30.3 | 28.2 | 25.6 | 82.4 | 73.6 | 80.0 | 40.2 | 32.0 |
|  | Left | Right |  |  |  |  |  |  |  |  |  |  |
|  | Same | Reversed | Same | Reversed |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |
| InstructBLIP-7B | 47.2 | 51.5 | 37.2 | 47.2 | 61.6 | 48.0 | 47.2 | 61.4 | 47.5 | 47.2 | 52.0 | 37.8 |
| InstructBLIP-13B | 47.2 | 54.2 | 43.4 | 47.2 | 57.0 | 44.9 | 47.2 | 58.1 | 45.6 | 47.2 | 52.5 | 41.6 |
| mBLIP | 47.2 | 59.8 | 52.4 | 47.2 | 64.2 | 53.5 | 47.8 | 65.7 | 54.6 | 47.8 | 56.4 | 46.8 |
| GLaMM | 47.2 | 48.9 | 38.3 | 47.2 | 65.5 | 51.8 | 79.4 | 29.8 | 17.3 | 15.0 | 76.2 | 63.7 |
| LLaVA-1.5-7B | 47.2 | 25.3 | 11.9 | 47.2 | 83.4 | 70.0 | 47.2 | 26.0 | 13.0 | 47.2 | 80.9 | 68.5 |
| LLaVA-1.5-13B | 62.8 | 39.1 | 31.7 | 31.7 | 76.8 | 61.8 | 91.1 | 35.8 | 24.3 | 8.9 | 79.3 | 64.3 |
| XComposer2 | 97.8 | 11.3 | 20.1 | 3.3 | 95.6 | 80.9 | 96.7 | 15.2 | 21.3 | 3.3 | 95.8 | 81.1 |
| MiniCPM-V | 86.1 | 27.7 | 19.7 | 9.4 | 88.1 | 74.1 | 82.2 | 32.7 | 21.1 | 12.2 | 87.0 | 73.3 |
| GPT-4o | 94.4 | 20.4 | 24.3 | 11.1 | 92.6 | 80.8 | 94.4 | 19.0 | 25.1 | 11.1 | 92.8 | 80.8 |
|  | Aggregated |  |  |  |  |  |  |  |  |  |  |  |
|  | Translated | Rotated | Reflected | Preferred Transform |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |  |  |  |
| InstructBLIP-7B | 47.2 | 54.7 | 40.5 | 47.2 | 58.0 | 44.2 | 47.2 | 57.8 | 43.9 | Not Significant |  |  |
| InstructBLIP-13B | 47.2 | 57.1 | 44.0 | 47.2 | 54.8 | 42.3 | 47.2 | 55.5 | 43.0 | Not Significant |  |  |
| mBLIP | 50.6 | 61.8 | 52.3 | 47.5 | 60.9 | 50.5 | 47.5 | 62.1 | 52.1 | Not Significant |  |  |
| GLaMM | 55.3 | 51.3 | 39.5 | 39.2 | 61.0 | 47.9 | 55.3 | 45.2 | 33.0 | Reflected |  |  |
| LLaVA-1.5-7B | 47.2 | 47.3 | 34.5 | 47.2 | 61.9 | 49.0 | 47.2 | 33.7 | 20.7 | Reflected |  |  |
| LLaVA-1.5-13B | 61.0 | 54.0 | 43.4 | 34.9 | 57.9 | 43.2 | 63.2 | 37.6 | 25.7 | Reflected |  |  |
| XComposer2 | 55.7 | 49.2 | 47.3 | 45.4 | 62.3 | 50.1 | 92.4 | 21.1 | 20.0 | Reflected |  |  |
| MiniCPM-V | 52.1 | 55.4 | 44.0 | 44.3 | 61.5 | 49.1 | 81.0 | 32.8 | 22.4 | Reflected |  |  |
| GPT-4o | 57.8 | 52.3 | 49.7 | 47.5 | 64.0 | 55.5 | 89.2 | 27.5 | 27.4 | Reflected |  |  |

**Table 8: The full results for testing the preferred frame of reference in VLMs.**
|  | Back | Front |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Egocentric | Intrinsic | Addressee | Egocentric | Intrinsic | Addressee |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |
| InstructBLIP-7B | 47.2 | 51.4 | 41.0 | 47.2 | 53.0 | 38.6 | 47.2 | 53.0 | 38.6 | 47.2 | 54.2 | 40.9 | 47.2 | 60.7 | 46.9 | 47.2 | 60.7 | 46.9 |
| InstructBLIP-13B | 47.2 | 43.5 | 32.9 | 47.2 | 48.9 | 34.4 | 47.2 | 48.9 | 34.4 | 47.2 | 66.5 | 52.5 | 47.2 | 61.1 | 48.5 | 47.2 | 61.1 | 48.5 |
| mBLIP-BLOOMZ | 52.8 | 62.1 | 52.2 | 52.8 | 63.9 | 53.2 | 52.8 | 63.9 | 53.2 | 52.8 | 56.4 | 45.3 | 52.8 | 55.5 | 44.6 | 52.8 | 55.5 | 44.6 |
| GLaMM | 47.2 | 45.6 | 31.9 | 47.2 | 51.0 | 38.8 | 47.2 | 51.0 | 38.8 | 47.2 | 37.9 | 24.8 | 47.2 | 69.6 | 57.1 | 47.2 | 69.6 | 57.1 |
| LLaVA-1.5-7B | 49.2 | 41.6 | 28.0 | 47.5 | 60.3 | 49.1 | 47.5 | 60.3 | 49.1 | 48.6 | 43.2 | 30.0 | 48.6 | 52.9 | 40.2 | 48.6 | 52.9 | 40.2 |
| LLaVA-1.5-13B | 50.8 | 36.8 | 20.9 | 48.6 | 54.7 | 43.0 | 48.6 | 54.7 | 43.0 | 47.2 | 46.5 | 34.5 | 47.2 | 47.3 | 32.6 | 47.2 | 47.3 | 32.6 |
| XComposer2 | 91.4 | 25.0 | 12.7 | 53.6 | 59.9 | 49.3 | 53.6 | 59.9 | 49.3 | 87.8 | 26.6 | 15.2 | 55.0 | 59.3 | 48.3 | 55.0 | 59.3 | 48.3 |
| MiniCPM-V | 66.4 | 46.5 | 34.2 | 60.8 | 51.3 | 40.7 | 60.8 | 51.3 | 40.7 | 57.5 | 45.0 | 35.5 | 50.8 | 64.6 | 53.4 | 50.8 | 64.6 | 53.4 |
| GPT-4o | 64.2 | 49.1 | 38.3 | 66.4 | 45.4 | 36.7 | 66.4 | 45.4 | 36.7 | 58.1 | 54.8 | 43.1 | 53.6 | 61.0 | 50.2 | 53.6 | 61.0 | 50.2 |
|  | Left | Right |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Egocentric | Intrinsic | Addressee | Egocentric | Intrinsic | Addressee |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |
| InstructBLIP-7B | 47.2 | 59.0 | 45.6 | 47.2 | 45.3 | 32.5 | 47.2 | 62.0 | 51.9 | 47.2 | 53.1 | 39.6 | 47.2 | 61.7 | 51.2 | 47.2 | 45.3 | 31.8 |
| InstructBLIP-13B | 47.2 | 59.7 | 47.8 | 47.2 | 70.2 | 56.2 | 47.2 | 39.6 | 27.8 | 47.2 | 53.6 | 40.6 | 47.2 | 39.5 | 27.6 | 47.2 | 70.8 | 56.6 |
| mBLIP-BLOOMZ | 52.8 | 58.2 | 47.8 | 52.8 | 59.7 | 47.6 | 52.8 | 58.4 | 48.1 | 52.8 | 57.7 | 45.4 | 52.8 | 60.6 | 48.4 | 52.8 | 53.8 | 42.4 |
| GLaMM | 75.8 | 22.3 | 11.7 | 46.4 | 62.0 | 51.1 | 52.5 | 62.3 | 51.1 | 60.8 | 41.8 | 27.5 | 44.7 | 68.5 | 57.4 | 53.1 | 58.7 | 48.7 |
| LLaVA-1.5-7B | 76.7 | 25.6 | 14.0 | 33.9 | 68.2 | 56.8 | 64.4 | 52.7 | 41.5 | 56.4 | 28.5 | 13.7 | 44.2 | 64.6 | 53.0 | 52.5 | 57.3 | 46.6 |
| LLaVA-1.5-13B | 81.7 | 23.7 | 13.4 | 42.2 | 65.0 | 53.5 | 57.2 | 58.5 | 47.4 | 86.7 | 26.8 | 14.3 | 47.8 | 64.0 | 53.6 | 52.2 | 59.9 | 49.3 |
| XComposer2 | 95.0 | 18.8 | 18.8 | 45.6 | 70.5 | 61.2 | 54.4 | 64.0 | 53.7 | 96.1 | 17.1 | 16.5 | 47.8 | 68.1 | 58.4 | 52.2 | 64.6 | 54.5 |
| MiniCPM-V | 93.3 | 20.4 | 18.0 | 52.2 | 64.3 | 53.9 | 47.8 | 68.0 | 58.4 | 91.7 | 22.6 | 19.0 | 46.1 | 68.3 | 58.1 | 53.9 | 62.5 | 52.7 |
| GPT-4o | 78.6 | 42.1 | 34.7 | 48.1 | 69.4 | 59.3 | 51.9 | 65.8 | 56.5 | 93.9 | 21.8 | 24.3 | 52.8 | 67.0 | 57.3 | 47.2 | 71.0 | 61.7 |
|  | Aggregated |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Egocentric | Intrinsic | Addressee | Preferred FoR |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |  |  |  |  |  |  |  |  |  |
| InstructBLIP-7B | 47.2 | 54.4 | 41.8 | 47.2 | 55.2 | 42.3 | 47.2 | 55.2 | 42.3 | Not Significant |  |  |  |  |  |  |  |  |
| InstructBLIP-13B | 47.2 | 55.8 | 43.5 | 47.2 | 54.9 | 41.7 | 47.2 | 55.1 | 41.8 | Not Significant |  |  |  |  |  |  |  |  |
| mBLIP-BLOOMZ | 52.8 | 58.6 | 47.7 | 52.8 | 59.9 | 48.4 | 52.8 | 57.9 | 47.1 | Not Significant |  |  |  |  |  |  |  |  |
| GLaMM | 57.8 | 36.9 | 24.0 | 46.4 | 62.8 | 51.1 | 50.0 | 60.4 | 48.9 | Egocentric Relative |  |  |  |  |  |  |  |  |
| LLaVA-1.5-7B | 57.7 | 34.7 | 21.4 | 43.5 | 61.5 | 49.8 | 53.3 | 55.8 | 44.4 | Egocentric Relative |  |  |  |  |  |  |  |  |
| LLaVA-1.5-13B | 66.6 | 33.5 | 20.8 | 46.5 | 57.7 | 45.7 | 51.3 | 55.1 | 43.1 | Egocentric Relative |  |  |  |  |  |  |  |  |
| XComposer2 | 92.6 | 21.9 | 15.8 | 50.5 | 64.4 | 54.3 | 53.8 | 61.9 | 51.4 | Egocentric Relative |  |  |  |  |  |  |  |  |
| MiniCPM-V | 77.2 | 33.7 | 26.7 | 52.5 | 62.1 | 51.5 | 53.3 | 61.6 | 51.3 | Egocentric Relative |  |  |  |  |  |  |  |  |
| GPT-4o | 73.7 | 42.0 | 35.1 | 55.2 | 60.7 | 50.9 | 54.8 | 60.8 | 51.3 | Egocentric Relative |  |  |  |  |  |  |  |  |

**Table 9: The full results for benchmarking perspective-taking performance in VLMs.**
|  | Back | Front |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Egocentric | Intrinsic | Addressee | Egocentric | Intrinsic | Addressee |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |
| InstructBLIP-7B | 47.2 | 56.4 | 45.1 | 47.2 | 54.3 | 41.2 | 47.2 | 56.0 | 42.8 | 47.2 | 56.2 | 42.0 | 47.2 | 56.4 | 43.6 | 47.2 | 56.3 | 43.3 |
| InstructBLIP-13B | 47.2 | 49.2 | 38.1 | 47.2 | 54.0 | 40.4 | 47.2 | 53.8 | 40.9 | 47.2 | 63.0 | 49.8 | 47.2 | 58.6 | 46.2 | 47.2 | 59.7 | 47.6 |
| mBLIP-BLOOMZ | 52.9 | 65.4 | 55.3 | 52.4 | 64.7 | 54.8 | 50.8 | 66.3 | 57.1 | 52.6 | 66.2 | 56.3 | 52.1 | 63.8 | 52.8 | 53.1 | 67.1 | 58.9 |
| GLaMM | 47.2 | 46.2 | 32.7 | 47.2 | 54.8 | 42.4 | 47.2 | 62.6 | 49.9 | 47.2 | 40.4 | 25.3 | 47.2 | 55.1 | 41.6 | 47.2 | 51.0 | 38.3 |
| LLaVA-1.5-7B | 49.0 | 41.6 | 27.6 | 47.4 | 56.3 | 45.7 | 46.2 | 66.7 | 55.0 | 47.5 | 39.4 | 25.2 | 47.4 | 52.9 | 39.8 | 47.2 | 41.1 | 27.5 |
| LLaVA-1.5-13B | 47.2 | 38.3 | 22.4 | 47.2 | 53.2 | 41.1 | 47.2 | 52.1 | 39.9 | 47.2 | 48.8 | 36.8 | 47.2 | 54.7 | 41.2 | 47.2 | 41.9 | 26.8 |
| XComposer2 | 65.4 | 40.6 | 26.0 | 52.2 | 57.9 | 47.0 | 54.0 | 58.5 | 47.5 | 86.9 | 27.0 | 17.1 | 52.1 | 58.9 | 47.8 | 53.1 | 57.8 | 46.6 |
| MiniCPM-V | 55.7 | 48.6 | 36.0 | 46.9 | 57.7 | 45.9 | 53.8 | 47.4 | 36.3 | 54.7 | 45.9 | 35.0 | 52.2 | 58.4 | 45.9 | 52.2 | 58.5 | 46.9 |
| GPT-4o | 69.0 | 41.3 | 28.7 | 59.7 | 50.0 | 37.7 | 56.4 | 48.7 | 36.0 | 58.6 | 52.5 | 40.3 | 52.1 | 57.4 | 45.1 | 48.3 | 60.0 | 46.9 |
|  | Left | Right |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Egocentric | Intrinsic | Addressee | Egocentric | Intrinsic | Addressee |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ | Acc% | $\varepsilon^{\textrm{hemi}}_{\times 10^{2}}$ | $\varepsilon^{\textrm{cos}}_{\times 10^{2}}$ |
| InstructBLIP-7B | 47.2 | 56.3 | 43.3 | 47.2 | 56.0 | 43.0 | 47.2 | 57.9 | 47.1 | 47.2 | 56.8 | 43.5 | 47.2 | 52.9 | 41.5 | 47.2 | 54.4 | 41.0 |
| InstructBLIP-13B | 47.2 | 58.0 | 46.2 | 47.2 | 61.7 | 48.7 | 47.2 | 46.5 | 33.8 | 47.2 | 53.5 | 41.1 | 47.2 | 49.8 | 37.6 | 47.2 | 62.6 | 49.4 |
| mBLIP-BLOOMZ | 51.4 | 65.6 | 55.4 | 46.4 | 67.0 | 56.4 | 47.2 | 64.6 | 54.8 | 50.7 | 65.3 | 54.4 | 48.2 | 63.5 | 52.8 | 47.2 | 62.9 | 52.3 |
| GLaMM | 47.2 | 29.6 | 16.9 | 47.2 | 57.7 | 45.8 | 47.2 | 53.7 | 41.5 | 47.2 | 34.3 | 18.3 | 47.2 | 58.7 | 47.1 | 47.2 | 53.2 | 41.6 |
| LLaVA-1.5-7B | 64.9 | 23.7 | 12.1 | 50.4 | 60.2 | 48.9 | 49.9 | 56.3 | 45.3 | 59.3 | 25.2 | 8.7 | 47.9 | 59.8 | 48.5 | 49.6 | 56.7 | 45.7 |
| LLaVA-1.5-13B | 47.2 | 29.2 | 18.3 | 47.2 | 59.5 | 47.0 | 47.2 | 53.5 | 41.2 | 64.7 | 32.1 | 17.9 | 47.4 | 61.6 | 50.7 | 48.5 | 58.6 | 47.8 |
| XComposer2 | 95.6 | 18.4 | 16.4 | 49.7 | 64.8 | 54.5 | 54.0 | 62.4 | 51.3 | 94.4 | 19.2 | 15.7 | 49.9 | 64.9 | 54.8 | 51.8 | 64.3 | 53.8 |
| MiniCPM-V | 89.4 | 24.2 | 13.3 | 50.4 | 60.9 | 50.0 | 52.1 | 60.6 | 49.9 | 89.9 | 24.3 | 14.2 | 50.1 | 60.6 | 49.5 | 53.3 | 58.0 | 47.3 |
| GPT-4o | 91.7 | 24.0 | 22.8 | 52.5 | 60.1 | 48.6 | 46.7 | 59.9 | 47.6 | 93.9 | 22.1 | 20.5 | 49.2 | 59.4 | 47.0 | 45.1 | 61.0 | 49.1 |

**Table 10: The full results for the cross-lingual and cross-cultural evaluation of the preferred frame of reference in VLMs.**
| Code | Language | Intrinsic | Egocentric | Addressee | Code | Language | Intrinsic | Egocentric | Addressee |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Ref. | Rot. | Tran. | Ref. | Rot. | Tran. | Ref. | Rot. | Tran. | Ref. | Rot. | Tran. |  |  |  |  |  |  |
| af | Afrikaans | 50.9 | 33.7 | 57.8 | 49.2 | 56.6 | 55.5 | 57.5 | ku | Kurdish | 56.5 | 49.5 | 54.4 | 53.1 | 53.1 | 55.2 | 54.0 |
| ak | Akan | 51.8 | 39.6 | 52.2 | 48.8 | 50.4 | 50.6 | 53.8 | la | Latin | 52.2 | 43.9 | 49.8 | 55.5 | 55.1 | 50.8 | 56.6 |
| am | Amharic | 52.1 | 47.4 | 60.7 | 50.9 | 56.8 | 54.2 | 57.6 | lb | Luxembourgish | 54.7 | 35.6 | 57.6 | 50.3 | 58.6 | 53.0 | 59.9 |
| ar | Arabic | 55.7 | 35.8 | 59.0 | 51.0 | 56.6 | 55.8 | 59.8 | ln | Lingala | 52.6 | 45.7 | 50.3 | 59.4 | 54.6 | 51.3 | 58.0 |
| as | Assamese | 51.6 | 40.8 | 55.3 | 51.6 | 48.8 | 52.8 | 56.0 | lo | Lao | 55.8 | 40.1 | 55.0 | 53.7 | 54.7 | 55.7 | 55.5 |
| az | Azerbaijani | 49.8 | 41.9 | 56.2 | 51.6 | 50.7 | 52.4 | 55.5 | lt | Lithuanian | 54.4 | 35.5 | 58.3 | 51.6 | 57.4 | 56.5 | 59.1 |
| be | Belarusian | 54.4 | 39.7 | 61.7 | 46.5 | 51.1 | 51.9 | 58.9 | lv | Latvian | 55.8 | 35.5 | 57.7 | 53.8 | 57.9 | 58.7 | 58.8 |
| bg | Bulgarian | 56.9 | 32.8 | 56.0 | 51.4 | 55.7 | 55.6 | 58.9 | mai | Maithili | 52.7 | 39.8 | 55.1 | 49.9 | 51.2 | 50.6 | 56.5 |
| bho | Bhojpuri | 51.5 | 42.8 | 58.3 | 47.5 | 52.3 | 51.4 | 56.5 | mg | Malagasy | 47.1 | 37.1 | 52.2 | 48.1 | 53.1 | 50.9 | 53.5 |
| bn | Bengali | 55.8 | 34.5 | 57.1 | 50.2 | 53.8 | 56.1 | 57.4 | mi | Maori | 52.0 | 36.6 | 58.5 | 47.6 | 52.0 | 51.9 | 58.1 |
| bs | Bosnian | 55.1 | 35.2 | 58.5 | 49.6 | 54.3 | 53.1 | 59.4 | mk | Macedonian | 54.8 | 37.0 | 59.1 | 49.5 | 56.5 | 56.0 | 58.1 |
| ca | Catalan | 55.6 | 34.7 | 56.9 | 53.0 | 56.0 | 56.4 | 59.7 | mn | Mongolian | 54.1 | 36.7 | 56.8 | 47.4 | 54.7 | 53.6 | 54.7 |
| ceb | Cebuano | 52.9 | 40.0 | 52.7 | 54.9 | 55.3 | 49.5 | 60.3 | mr | Marathi | 52.9 | 34.5 | 55.0 | 48.3 | 51.7 | 52.4 | 55.4 |
| ckb | Sorani | 50.4 | 36.1 | 53.3 | 50.6 | 50.3 | 52.0 | 56.3 | ms | Malay | 54.2 | 33.1 | 55.8 | 50.5 | 55.9 | 55.4 | 57.9 |
| co | Corsican | 57.6 | 35.4 | 57.4 | 54.3 | 58.8 | 58.8 | 59.7 | mt | Maltese | 53.4 | 37.5 | 56.1 | 49.2 | 50.8 | 53.8 | 55.5 |
| cs | Czech | 56.4 | 35.7 | 58.2 | 52.4 | 57.0 | 58.5 | 58.0 | my | Myanmar | 54.9 | 39.3 | 58.7 | 51.8 | 54.2 | 56.1 | 58.1 |
| cy | Welsh | 55.1 | 36.7 | 59.5 | 48.7 | 54.9 | 54.8 | 58.5 | nb | Norwegian | 55.1 | 34.7 | 57.0 | 52.1 | 58.1 | 57.6 | 57.3 |
| da | Danish | 54.9 | 33.0 | 55.2 | 53.1 | 57.6 | 58.3 | 56.9 | ne | Nepali | 53.1 | 39.4 | 58.4 | 47.3 | 52.9 | 54.1 | 54.0 |
| de | German | 55.7 | 36.2 | 58.4 | 52.8 | 56.9 | 56.6 | 60.3 | nl | Dutch | 51.7 | 34.5 | 56.3 | 48.3 | 53.5 | 51.6 | 57.8 |
| el | Greek | 54.4 | 34.4 | 57.1 | 52.2 | 57.1 | 57.5 | 57.7 | nso | Sepedi | 53.8 | 42.6 | 51.0 | 57.1 | 46.3 | 54.2 | 53.7 |
| en | English | 50.9 | 35.8 | 57.3 | 53.7 | 58.8 | 51.3 | 58.8 | ny | Nyanja | 53.7 | 34.5 | 56.6 | 48.0 | 54.4 | 52.8 | 56.7 |
| eo | Esperanto | 58.0 | 34.3 | 56.4 | 54.6 | 58.2 | 58.2 | 60.2 | om | Oromo | 51.1 | 43.5 | 57.3 | 50.6 | 54.9 | 54.8 | 52.9 |
| es | Spanish | 56.9 | 36.2 | 58.1 | 53.3 | 57.0 | 58.5 | 59.0 | pl | Polish | 55.8 | 32.9 | 55.8 | 52.5 | 55.1 | 55.1 | 59.5 |
| et | Estonian | 53.7 | 35.1 | 56.0 | 51.7 | 55.0 | 54.8 | 58.5 | ps | Pashto | 53.0 | 34.6 | 57.4 | 48.9 | 53.7 | 54.6 | 57.4 |
| eu | Basque | 56.8 | 34.3 | 56.8 | 53.2 | 56.7 | 57.1 | 59.5 | pt | Portuguese | 56.3 | 35.9 | 58.2 | 51.9 | 59.1 | 59.3 | 57.6 |
| fa | Persian | 55.8 | 32.1 | 55.3 | 49.8 | 54.4 | 53.8 | 58.0 | ro | Romanian | 57.1 | 34.8 | 57.0 | 53.8 | 58.2 | 58.6 | 59.1 |
| fi | Finnish | 53.9 | 33.7 | 56.7 | 50.8 | 56.3 | 56.1 | 57.6 | ru | Russian | 56.2 | 36.9 | 58.8 | 53.0 | 56.8 | 56.3 | 60.8 |
| fil | Filipino | 50.9 | 31.1 | 54.1 | 49.2 | 54.3 | 54.0 | 55.7 | rw | Kinyarwanda | 53.2 | 35.2 | 56.7 | 48.9 | 54.4 | 54.1 | 57.2 |
| fr | French | 58.0 | 35.2 | 57.4 | 53.7 | 58.6 | 58.5 | 59.4 | sa | Sanskrit | 51.9 | 41.2 | 54.1 | 51.9 | 51.7 | 56.4 | 51.6 |
| fy | Frisian | 53.9 | 38.2 | 58.9 | 49.6 | 53.4 | 53.2 | 59.2 | sd | Sindhi | 51.3 | 40.3 | 56.5 | 49.1 | 54.8 | 49.8 | 57.4 |
| ga | Irish | 54.0 | 33.2 | 55.3 | 49.2 | 52.7 | 55.7 | 53.9 | si | Sinhala | 52.4 | 38.4 | 54.6 | 48.6 | 53.4 | 51.5 | 56.6 |
| gd | Scots Gaelic | 53.9 | 35.4 | 58.5 | 49.6 | 54.7 | 55.8 | 58.1 | sk | Slovak | 56.1 | 37.7 | 57.1 | 54.7 | 57.2 | 56.7 | 59.8 |
| gl | Galician | 56.6 | 37.1 | 59.0 | 53.4 | 57.9 | 57.9 | 60.0 | sl | Slovenian | 55.8 | 36.5 | 59.3 | 49.5 | 53.9 | 54.3 | 58.9 |
| gom | Konkani | 51.1 | 53.1 | 55.5 | 50.5 | 52.5 | 54.9 | 51.8 | sn | Shona | 56.0 | 34.7 | 56.0 | 52.2 | 54.8 | 55.6 | 58.5 |
| gu | Gujarati | 52.6 | 36.6 | 54.2 | 50.9 | 55.5 | 55.3 | 53.8 | so | Somali | 53.7 | 34.3 | 56.4 | 48.2 | 50.0 | 51.6 | 58.4 |
| ha | Hausa | 54.0 | 41.0 | 56.1 | 53.0 | 52.8 | 55.3 | 56.1 | sq | Albanian | 53.6 | 35.1 | 56.4 | 49.0 | 52.6 | 50.3 | 60.1 |
| haw | Hawaiian | 55.3 | 42.2 | 62.1 | 51.5 | 60.5 | 56.2 | 60.8 | sr | Serbian | 55.4 | 34.5 | 57.2 | 50.9 | 52.5 | 55.0 | 58.8 |
| he | Hebrew | 56.5 | 36.4 | 58.8 | 52.5 | 57.1 | 56.5 | 60.3 | st | Sesotho | 53.9 | 38.4 | 55.4 | 51.0 | 51.3 | 54.4 | 55.8 |
| hi | Hindi | 52.5 | 37.8 | 56.6 | 49.1 | 54.5 | 54.6 | 54.2 | su | Sundanese | 51.3 | 36.7 | 55.0 | 50.0 | 53.7 | 50.4 | 57.7 |
| ht | Haitian Creole | 56.1 | 36.0 | 58.3 | 53.6 | 58.4 | 58.2 | 59.6 | sv | Swedish | 54.0 | 33.5 | 56.7 | 51.7 | 55.8 | 56.3 | 58.2 |
| hu | Hungarian | 55.2 | 35.0 | 57.5 | 50.7 | 56.1 | 56.8 | 56.7 | sw | Swahili | 55.3 | 34.2 | 56.8 | 52.4 | 57.2 | 56.5 | 58.4 |
| hy | Armenian | 52.2 | 35.4 | 56.7 | 48.8 | 53.6 | 52.5 | 57.2 | ta | Tamil | 52.0 | 40.4 | 55.2 | 51.1 | 52.2 | 52.9 | 54.6 |
| id | Indonesian | 55.9 | 35.6 | 58.1 | 52.2 | 57.1 | 57.8 | 58.1 | tg | Tajik | 55.7 | 36.7 | 57.7 | 49.7 | 55.4 | 56.6 | 56.3 |
| ig | Igbo | 54.5 | 33.8 | 56.7 | 47.4 | 53.6 | 53.2 | 55.3 | th | Thai | 55.5 | 35.4 | 57.9 | 50.8 | 56.2 | 57.8 | 57.3 |
| ilo | Ilocano | 50.8 | 44.6 | 46.7 | 58.9 | 48.9 | 57.0 | 48.7 | tk | Turkmen | 52.3 | 45.3 | 59.0 | 51.5 | 52.6 | 51.2 | 59.1 |
| is | Icelandic | 55.9 | 34.2 | 57.0 | 52.2 | 56.5 | 58.0 | 57.3 | tr | Turkish | 55.3 | 33.6 | 56.3 | 50.8 | 56.2 | 57.0 | 56.3 |
| it | Italian | 56.8 | 35.6 | 57.6 | 53.6 | 57.9 | 58.2 | 59.6 | ts | Tsonga | 49.4 | 44.6 | 50.0 | 53.7 | 53.3 | 51.5 | 53.6 |
| ja | Japanese | 54.7 | 34.5 | 56.9 | 50.4 | 54.4 | 55.9 | 57.3 | uk | Ukrainian | 56.6 | 36.1 | 58.8 | 50.1 | 56.8 | 55.7 | 59.7 |
| jv | Javanese | 53.5 | 35.3 | 57.7 | 51.0 | 55.7 | 54.8 | 58.9 | ur | Urdu | 52.3 | 34.6 | 56.7 | 49.7 | 54.0 | 55.1 | 57.0 |
| ka | Georgian | 51.1 | 34.8 | 54.3 | 50.6 | 52.0 | 54.0 | 55.3 | uz | Uzbek | 52.6 | 34.5 | 56.4 | 48.1 | 51.7 | 53.2 | 56.6 |
| kk | Kazakh | 52.6 | 36.5 | 58.8 | 50.4 | 54.0 | 56.1 | 56.9 | vi | Vietnamese | 53.9 | 34.6 | 58.5 | 48.6 | 55.7 | 56.1 | 56.8 |
| km | Khmer | 55.6 | 37.6 | 60.2 | 50.5 | 56.9 | 56.6 | 59.3 | yi | Yiddish | 56.7 | 36.5 | 57.9 | 53.5 | 56.8 | 57.3 | 60.0 |
| kn | Kannada | 52.3 | 40.8 | 53.2 | 49.5 | 49.6 | 51.9 | 53.4 | zh | Yoruba | 54.6 | 35.5 | 58.3 | 51.4 | 56.9 | 57.2 | 57.8 |
| ko | Korean | 53.6 | 36.5 | 59.1 | 49.7 | 53.3 | 53.5 | 59.8 | zu | Chinese | 55.6 | 35.9 | 57.7 | 53.1 | 55.3 | 56.5 | 60.2 |
| kri | Krio | 58.3 | 36.2 | 57.1 | 51.2 | 56.1 | 53.2 | 60.2 |  |  |  |  |  |  |  |  |  |

Figure: Figure 10: All prediction plots for each model on COMFORT-BALL using the camera perspective prompt (cam). The raw probability $p(\theta)$ in gray, normalized probability $\widehat{p}(\theta)$ in black, and the reference probability $p_{\textrm{cos}}(\theta)$ of cam in red.
Refer to caption: x23.png

Figure: Figure 11: All prediction plots for each model on COMFORT-CAR using the camera perspective prompt (cam). The raw probability $p(\theta)$ in gray, normalized probability $\widehat{p}(\theta)$ in black, and the reference probabilities $p_{\textrm{cos}}(\theta)$ of cam in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel, we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMFORT-CAR with relatum facing right for front and behind relations.
Refer to caption: x59.png

Figure: Figure 12: All prediction plots for each model on COMFORT-CAR using the addressee perspective prompt (add). The raw probability $p(\theta)$ in gray, normalized probability $\widehat{p}(\theta)$ in black, and the reference probabilities $p_{\textrm{cos}}(\theta)$ of cam in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel, we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMFORT-CAR with relatum facing right for front and behind relations.
Refer to caption: x95.png

Figure: Figure 13: All prediction plots for each model on COMFORT-CAR using the relatum perspective prompt (rel). The raw probability $p(\theta)$ in gray, normalized probability $\widehat{p}(\theta)$ in black, and the reference probabilities $p_{\textrm{cos}}(\theta)$ of cam in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel, we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMFORT-CAR with relatum facing right for front and behind relations.
Refer to caption: x131.png

Figure: Figure 14: All prediction plots for each model on COMFORT-CAR without perspective prompt (nop). The raw probability $p(\theta)$ in gray, normalized probability $\widehat{p}(\theta)$ in black, and the reference probabilities $p_{\textrm{cos}}(\theta)$ of cam in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel, we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMFORT-CAR with relatum facing right for front and behind relations.
Refer to caption: x167.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Small grammar errors, eg 476: “Does multilingual VLMs..” -> “Do multilingual…” Maybe do a brief grammarly check?
soundness: 4
strengths: Deep utilization of synthetic pipeline to really probe spatial perception from several angles.

Demonstrations of anglewise judgements are compelling and surprising! (eg., figure 7)

Great to see the extra effort of testing 

Conclusions and discussion are thoughtful and well supported.
summary: The authors assess how VLMs represent space through the lens of “situated communication,” framing the meaning of spatial relations like “to the right of” under transformations, in different frames of reference, and others (situated communication).

They introduce the COMFORT dataset, tasks, and metrics to analyze spatial reasoning under these classes of transformations, and see which frame-of-reference preferences LMs have, and test whether these preferences are language-agnostic by extending their evaluation to multilingual settings. 

The images are rendered to enable simple dynamic generation of test samples. Their language queries that test the relations likewise have a simple structure and are programmatically produced, enabling translation into other languages. Testing the kinds of relations is simple using this methodology and allows them to test for continua such as how a model rates the “in front of” relation by angle between the two in the plane relative to the camera.

**Edit**: The authors did a great job adding some small case studies that **show** how their results generalize to other backgrounds and real images (although these are still pretty close to in-distribution to the test images). The use of a laptop rather than a ball in the real images in particular is a strong change. I have raised all the component scores to 4. **Although I would prefer an option to give a 9/10, since it isn't available I will raise from 8 to 10.**

These tests are then translated into metrics using the established concept of “acceptability regions”, which then allows them to score for correctness. Swapping object types, colors, etc in the same scene is easy using the synthetic pipeline and gives an evaluable sense of robustness.

They test a broad set of VLMs, including multilingual ones in 109 translated languages. Some models better represent egocentric vs object centric relations.

Overall, they find that the VLMs do have some acceptable egocentric understanding capabilities but they have severe robustness issues. They find that English notions of frame of reference are held across most test languages, not reflecting the diversity of notions.
weaknesses: ~Sole reliance on synthetic data.~ A brief ablation over natural images would be nice to see to give a sense of how well these findings generalize. They added the case study, the results generalize (though the case study is still close to ID the synth data)!

~Limited diversity of backgrounds. This is another form of variation that may have important implications for generalization.~ They added the case study, the findings generalize!

The multilingual experiments, while welcome, have limited time to breathe in the overall paper with just a few paragraphs and tables stuck in appendices. However, I think this is a great problem to have as the paper is densely packed with quality experimentation.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: **Suggestions**: 
* L212 "the query is appended after four different perspective prompts...": I'm assuming this meant to say "after *one of* four different..."? 

* I understand the authors may have been space limited (no pun intended), but I think adding a Conclusion section to reiterate contributions and takeaways (further than the Discussion section already does) could make the paper stronger. There's a large amount of content, so spoon-feeding the final takeaways to readers could aid clarity. 

* This is minor, but relatedly I think the paper could be a little clearer about what the intended usage of the framework is for readers. Are readers supposed to come away with an understanding that this will be a benchmark they can evaluate their own models on? I believe yes, but the wording of the paper at the beginning and end doesn't make this explicit. In the introduction, COMFORT is introduced as a "framework" and not a "benchmark", was this intentional? Secondly, the paper closes with only a discussion over the empirical findings in Section 5, without any explicit reiteration that it has introduced COMFORT. This omission at the end made it feel like the only point of the paper was to present a study of VLM spatial understanding -- this would be fine of course, but if I'm understanding correctly I think the paper would additionally want to explicitly market itself as offering COMFORT as a testbed for VLMs. It's subtle, but I think being explicit about this, especially at the end of the paper, could really help drive home the point the authors want to communicate.
soundness: 4
strengths: **Strong Backing in Theory**: The paper's framing and setup in sections 1 through 3 (which motivates the problem statement and framework design choices) is a joy to read. Motivating statements and scoping within existing work is very clear and compelling. 

**Conclusions Well Supported**: The experimental setup is quite thorough and shows clear trends which the authors use to justify their claims. A majority of models are clearly shown to have FoR preferences based on the metrics *cos* and *hem* error functions defined, with the BLIP family of VLMs being a notable exception. I particularly appreciated that the paper additionally presents a compelling explanation for the BLIP discrepancy by showing that these same models score poorly on an evaluation of object hallucinations. 

**Novelty**: In my understanding I believe that the presented framework is novel based on its continuous definition based on regions of acceptability. This seems to me to be an important distinction namely because these spatial relationships are known to lie within continuums and are not discrete. 

**Clarity**: The paper's writing is generally clear, and I found the organizational structure helpful for understanding the concepts introduced.
summary: **Main contributions**: 
The paper has two main contributions: 
1. It presents COMFORT, a framework for evaluating VLMs' understanding of spatial frame of reference expressions, which relate two objects to one another (e.g. "Is the basketball *to the right of* the car?"). Expressions are evaluated using a rendered image containing the two objects and which is used to query the VLM with (as a binary Yes/No question) for whether or not the relation holds for the image. 

2. It uses COMFORT to show that a set of contemporary VLMs have distinct preferences for frame of reference parameters. Specifically, the paper presents evidence for the preference of English language frame of reference parameters in VLMs. 

**Framework Structure**:  
Expressions in COMFORT vary across two main conditions:
1. Expressions which include two objects with a semantic "front" (e.g. such as the "front" of a person being the side with the person's face). In this scenario the spatial expression must be reasoned about in terms of an anchor frame of reference (i.e. either object A's, object B's, or the camera's frame). The selection of the frame of reference will determine which direction in the image corresponds to "left of", "back of" etc. 

2. Expressions which include objects without a semantic "front". In COMFORT, this is instantiated by scenarios with two balls. In such scenarios, a coordinate transformation convention must be assumed for determining what is "right", "left", "back", etc. Different languages have different standard conventions, and so this condition is set up to test this. 

Preferences are measured by evaluating the [Yes/No] probability for a given spatial expression in relation to a ground truth *region of acceptability*, a continuous region of rotation of the second object around the center object (see Figure 1c for reference) in which a spatial expression may hold true. This region of acceptability allows for not only for accuracy to be measured but also for the dynamics of the probability to be measured as functions of distance from the center of the region of acceptability -- intuitively, the score of "right of" should decrease smoothly as the object is rotated from 0 to 90 degrees from the center of the region. Specifically, the paper presents. 
These measures probe the robustness of model scores (e.g. evaluating std across object variations unrelated to space like color or shape) as well as the consistency (are scores symmetric at both equivalent angles from the acceptability region's center? Do they change smoothly as angle is varied?). 

**Results**
The paper uses the region of acceptability based error measures to show that VLMs generally skew towards preferring egocentric frames of references (condition 1) and the reflected coordinate transformation convention. Additionally, the paper also shows that these preferences cannot be reliably changed when the prompt explicitly defines the frame of reference to be used, or when a language (other than English) with different known preferences is used to prompt the model.

**Recommendation** 
This is a solid paper which I believe presents a unique contribution that stands to better capacitate the community in vetting the spatial understanding of VLMs. I have noted a few suggestions below under Weaknesses to improve the presentation of the paper. With that said, I think it is already at a good quality and I recommend acceptance.
weaknesses: **Cross-lingual Experiment Presentation**: I think some further refinement/elaboration of the cross lingual experiments (Section 4.5) could help strengthen the paper. Although the paper does state upfront that it spends most of the content focused around English experiments, the treatment of the section still felt a little too short for the emphasis that is placed in the introduction for this being a core piece of the framework (e.g. the M in COMFORT stands for Multilingual). Specifically I think a couple of details could be expanded upon. 

First, the results in Figure 8 are a little ambiguously presented. Is the world map mapping the preferences of GPT-4o, or is it mapping ground truth human preferences? I believe the former, but the wording in both the figure and the paper makes it sound like these are ground truth preferences. 
Second, for ground truth preferences, I'm wondering how these are obtained? -- I'm assuming this is pulled from the linguistics literature, but it is not explicitly stated and appears like a key detail to me. I would appreciate it if the authors could clarify both of these points. 

Lastly, although the results presented clearly show a preference for the same reference frames as English, I think they could be strengthened by also showing a notion of what the expected preferences would be if the VLM were to be adhering to the conventions of the language being used. For example, I think this could be done by (a) adding a second map showing a visualization of ground truth human preferences, or (b) adding the bold/underline convention from the table in Figure 8 to Table 10.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How do the findings impact the development of embodied AI systems and other applications like autonomous driving, robotics, or augmented reality?

2. How could improved spatial reasoning in vision-language models (VLMs) enhance performance in multilingual and multicultural contexts?

3. What are the key areas of future research to advance the spatial reasoning capabilities of VLMs, and what specific challenges or limitations need to be addressed?
soundness: 4
strengths: This paper introduces the innovative COnsistent Multilingual Frame Of Reference Test (COMFORT), which systematically evaluates vision-language models (VLMs) across multiple languages and cultural contexts. This novel approach highlights the importance of considering linguistic and cultural variations in model performance. The research is thorough and methodical, assessing nine state-of-the-art VLMs with detailed analyses of their robustness, consistency, and flexibility. The paper is well-organized and clearly written, making complex concepts accessible. Its findings underscore the need for more robust and flexible models, providing a valuable framework for future research in vision-language modeling and spatial reasoning.
summary: This paper explores the spatial reasoning capabilities of vision-language models (VLMs) and their handling of spatial ambiguities. It introduces the COnsistent Multilingual Frame Of Reference Test (COMFORT) to systematically evaluate these models. The study reveals that while VLMs show some alignment with English conventions in resolving spatial ambiguities, they exhibit significant shortcomings: poor robustness and consistency, lack of flexibility to accommodate multiple frames of reference (FoRs), and failure to adhere to language-specific or culture-specific conventions in cross-lingual tests. The paper emphasizes the need for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning in VLMs.
weaknesses: The scope of spatial relations is limited, focusing mainly on basic relations like front-back and left-right, while neglecting others such as near-far and above-below. The evaluation relies on synthetic 3D images, which may not fully capture real-world complexities, and lacks consideration for occlusion and varying camera angles. Additionally, the analysis could benefit from a more diverse linguistic and cultural context, as well as human annotations for the multilingual dataset. Addressing these weaknesses by expanding the scope of spatial relations, incorporating real-world scenarios, and improving the quality of the dataset with human annotations would provide a more comprehensive evaluation of VLMs' spatial reasoning capabilities.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the comments in the Weakness.
soundness: 4
strengths: 1. This paper delves into "Do VLM represent space and how". Specifically, it considers the concept of Frames of Reference (FoR) and constructs a framework for understanding how VLMs perceive spatial information, taking into account the ambiguities that FoR might bring. The COMFORT framework is systematic and comprehensive, and its task formulation is rigorous. The authors create challenging but realistic queries using synthetic 3D images and corresponding textual descriptions, and propose systematic metrics. From the perspective of benchmark construction, this is a systematic and rigorous work.

2. The authors tested different VLMs and analyzed how they internally understand spatial representations. The testing methods are reliable.

3. The authors extended the test cases to cover 109 languages across 170 regions, further testing how VLMs respond to ambiguities in spatial relationship descriptions under different languages.
summary: This paper discusses the performance of VLMs in handling ambiguity in spatial expressions. The authors constructed a new benchmark -COMFORT to systematically test the spatial reasoning capabilities of VLMs, and conducted analysis on COMFORT.
weaknesses: As the author claimed, "spatial expressions in situated communication can be ambiguous." Therefore, considering that humans also exhibit different understandings when dealing with similar issues of ambiguity, even if VLMs demonstrate different interpretations, it does not necessarily imply a clear superiority or inferiority. In other words, the evaluation metrics proposed in the paper are not absolute criteria, like egocentric is not naturally better than addressee-centered, but rather serve as perspectives for observing how VLMs understand spatial information. 

From this standpoint, I believe the author should include more analysis of how humans understand ambiguities in spatial relations (relevant information can be found in psychological and cognitive science research). This would help us better understand the differences between VLMs’ understanding of these ambiguities and the human perspective, and ultimately facilitate a better alignment of spatial reasoning in vision-language models with human cognition.

Hope to see further discussion on these points.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: NA
soundness: 3
strengths: - Unlike existing benchmarks, which typically focus on spatial language understanding within a single frame of reference or specific cultural context, this work broadens the scope by emphasizing ambiguity in spatial reasoning due to different Framework of References and cross-linguistic/cross-cultural diversity.

- The paper is well-structured, with clear explanations of key concepts such as frames of reference and spatial reasoning.

More importantly, 

- The paper’s contributions are highly significant in the context of both vision-language research and broader discussions on AI alignment with human cognition. By focusing on spatial reasoning, a core component of human cognition, this work addresses a critical gap in the evaluation of VLMs.

- The cross-lingual evaluation across diverse cultural backgrounds provides valuable insights into the limitations of current VLMs in handling non-English spatial conventions. This has implications for the development of more globally applicable models, as the dominance of English-centric FoRs could hinder the usability of VLMs in non-English contexts.
summary: - Investigates how vision-language models (VLMs) represent and reason about spatial relations, particularly when spatial language is ambiguous due to differing frames of reference (FoRs). It focuses on evaluating the robustness, consistency, and flexibility of VLMs in handling such ambiguities.

- The authors introduce a new evaluation protocol called the COnsistent Multilingual Frame Of Reference Test (COMFORT), which systematically assesses VLMs' spatial reasoning capabilities. The evaluation includes tasks with synthetic 3D images and text descriptions of spatial relations, with tests conducted in 109 languages across 170 regions worldwide.

- The study reveals that while VLMs show some alignment with English conventions for resolving spatial ambiguities, they struggle with: (A) Consistency and robustness in spatial reasoning, (B) Flexibility in adopting multiple FoRs, (C) Cross-linguistic and cross-cultural understanding of spatial relations, with English conventions often dominating over those of other languages.
weaknesses: - While the introduction of the COMFORT benchmark is a novel contribution, the idea of testing spatial reasoning in vision-language models is not entirely new. Several previous benchmarks, such as those mentioned in the paper, have evaluated spatial reasoning capabilities in VLMs. The novelty of this paper lies primarily in its focus on the frame of reference ambiguities, but the paper does not sufficiently clarify how this focus significantly advances beyond the existing benchmarks. But I agree that the frame of reference is a rather novel take on the same topic.

- Although the paper provides valuable insights into how English conventions dominate spatial reasoning in multilingual models, the discussion around cultural and linguistic biases remains relatively high-level.

- The paper identifies that most models tend to default to egocentric relative frames of reference, which aligns with English spatial conventions. However, it does not deeply investigate whether this bias toward egocentrism is inherently problematic across all contexts or if there are scenarios where this behavior is acceptable or even preferable. 

+ (not required) It would be beneficial to include a dedicated section discussing the practical relevance of the findings in LLM applications. Is there an area the the findings of this paper can seriously affect?

Meta-review: This paper proposes a novel evaluation protocol and benchmark, COnsistent Multilingual Frame Of Reference Test (COMFORT), to examine the spatial reasoning capabilities of vision and language models and evaluates 9 state-of-the-art models using this framework and demonstrate their  significant shortcomings. This is an interesting framework and accompanying analysis of models and reviewers suggest that this paper will be a good contribution to the conference.

Decision: Accept (Oral)