Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities
Authors: Zheyuan Zhang
Fengyuan Hu
Jayjun Lee
Freda Shi
Parisa Kordjamshidi
Joyce Chai
Ziqiao Ma
Venue: ICLR 2025 Oral (2025)
Keywords: vision-language models
spatial reasoning
multimodal reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We present an evaluation protocol to systematically assess the spatial reasoning capabilities of vision language models, and shed light on the ambiguity and cross-cultural diversity of frame of reference in spatial reasoning.

Abstract: Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners. While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored. To address this issue, we present the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to systematically assess the spatial reasoning capabilities of VLMs. We evaluate nine state-of-the-art VLMs using COMFORT. Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.

Published as a conference paper at ICLR 2025
Do VISION-LANGUAGE MODELS REPRESENT SPACE AND How? EVALUATING SPATIAL FRAME OF REFER- ENCE UNDER AMBIGUITIES
Zheyuan Zhang Ix
Fengyuan Hu
Jayjun Lee
Freda Shi
Parisa Kordjamshidi
Joyce Chail
Ziqiao Ma
'University of Michigan University of Waterloo
*Vector Institute "Michigan State University
https://ssattin comfort. thub.io/
ABSTRACT
Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners. While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored. To address this issue, we present the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol systematically assess the spatial reasoning capabilities of VLMs. We evaluate nine state-of-the-ard VLMs using COMFORT. Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMss: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language- specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. With growing effort to align vision-languaa models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.
INTRODUCTION
The recent success large language models has sparked breakthroughs in multi-modalities, leading to the development of many vision-language models (VLMs: Chen et al. 2023c OpenAl 2024] Reid
ct al. 2024| inter alia). With some benchmarks developed to evaluate the downstream performance
of these models (Liu et al.) [203cc] Yue et al., 2024), there has been growing excitement around evaluations and analyses inspired by human cognitive capabilities such as referential grounding (Mal
al.
2023a), compositional reasoning (Ma et al. 2023c), visual illusions (Zhang al. 2023 Guan
ct al.
2024). and theory of mind (Jin et al.) (024)). One direction among them that captures significant
attention spatial language understanding and reasoning, leading to several benchmarks (Mirzace et al.
2021 Mirzaee & Kordjamshidi 2022 Kamath et al. 2023 Liu et al. (023a) and enhanced
models (Chen al. [024a]] Cheng et al.) 2024 Premsri & Kordjamshidi
Indeed, spatial cogmition is crucial part of human cognitive capability developed in early ages (Tommasi & Laeng 2012 Vasilyeva & Lourenco) 2012). Language is closely intertwined with
spatial cognition, with each contributing to the acquisition of the other (Hayward & Tarr 1995] Regier& & Carlson, 2001 Pyers et al. Pruden et al., 201T Gentner et 2013) While spatial language and linglingcistic spatial representations in memory are closely correlated and share foundational properties, they are, to some extent, divergent- preserved across different languages or tasks, and humans demonstrate flexibility in using multiple coordinate systems for both non- non- -inguuiuiit reasoning and linguistic expressions (Munnich et al., 20011 Shusterman & Li 2016). Thus, spatial language is inherently ambiguous.
conventions are not consistently
In situated communication, even a simple spatial expression like "the basketball to the right of the car"' may have multiple interpretations. People may use different frames of reference (FoR; Levinson 1996; Frank 1998, inter alia) to resolve ambiguity about the underlying coordinate system, as illustrated in Figure Th. The diversity of conventions across languages and cultures further
"Authors contrihuted equally to this work.

---

Published as a conference paper at ICLR 2025
(a) Frame of Reference (FoR)
(b) Coordinate Transformation
(c) Spatial Continuity
the baskethall the right the car?
The ball the left/right front/back of the blue ball.
Is the red ball to the right of the blue ball?
Yes, from the camera's i viewpoint
Yes, from the woman's viewpoint
Translated: A/B/C/D
Yes, from the car's viewpoint
Rotated: B/A/D/C
Reflected:
A/B/D/C
Froez
Prest
Kal
Printt
Eig. Misser
Rg. Torns
E-g. Soyffih
Figure I: In situated communication, spatial language understanding and reasoning are often ambigu- ous, leading to varying interpretations among people from different cultural backgrounds. Specifically: (a) different frames of reference can result in different interpretations of the same spatial expression;
(b) speakers of different languages may use distinct coordinate frames for non-fronted reference objects; and (c) spatial relations extend beyond exact axes to include acceptable regions.
Origin
Frame of Reference
Example (English)
Addressee Referent/
Camera
Egocentric Relative FoR
(From the camera's viewpoint,) the hall is behind the car.
(Preferred)
Addressce- Centrerd Relative FoR
(From the woman's 's vicwpoint.) the ball is to the left of the car.
Addressee
(From the car's 's viewpoint,) the ball is to the right of the car.
Reference
Relatum (Cround
Intrinsic FoR
An illustrative example of how a frame of reference (FoR) specifies the reference system when describing the spatial relation between target object (i.e., the ball) and reference object (i.e., the car). When the FoR is not explicitly specified, English prefers egocentric relative FoR, i.e., "the ball is behind the car." We study FoRs that lead to ambiguity (Liu et al. 2010).
Figure 2:
complicates this ambiguity-different different languages employ different conventions in choosing one FoR among multiple competing options. As shown in Figure Tb, speakers may project themselves onto the ball or consider an imaginary listener facing them (Shusterman & T.i 2016). These ambiguities
are not easily resolvable based solely on linguistic expressions (Tenbrink] 2004 Liu et al. 2010).
Our main research question is not new: Do vision-language models represent space, and how? Several benchmarks (Kamath et al. 2023] Liu et al.) (2023a) have been developed for this purpose. consisting of text-image pairs where objects may or may not follow certain spatial relations. However, the aforementioned spatial ambiguities remain largely under-explored when studying YLM-based spatial language understanding and reasoning. We emphasize that FoRs crucial studying spatial cognition across modalities, as they provide a foundational framework for understanding how spatial relationships are perceived, interpreted, and communicated (Levinson) 2003).
To fill this gap, we present COnsistent Multilingual Frame Of Reference Test (COMFORT), a framework that systematically evaluates the spatial reasoning capabilities of VLMs, emphasizing consistency in understanding ambiguous and disambiguated spatial expressions. COMFORT introduces (1) set of spatial reasoning tasks instantiated by synthetic 3D images and corresponding text describing spatial relations and (2) metrics to evaluate the robustness and consistency of the model responses. We extend the setup multilingual settings by evaluating models in 109 languages across 170 regions worldwide. We find that VLMS show alignment with English conventions in spatial language understanding when resolving ambiguities. However, they (I) are still far from achieving robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to linguistic and cultural conventions in lingull tests, as English tends to dominate other languages. With a growing effort align vision- -anguuae models with human cognition, we highlight the ambiguous nature of spatial language and call for increased attention to cross-cus cultural diversity in spatial reasoning.
2
BACKGROUND AND RELATED WORK
2.1 SPATIAL LANGUAGE AND SPATIAL REPRESENTATION
Some projective terms, such as the English words front, back, right, and left, convey meanings of spatial relations (Eschenbach) 2004). These terms articulate the spatial relation between two entities within a designated frame of reference (FoR), often involving one entity as the reference object (relatumlground)) and another target object (referent/figure) that is positioned relative to the relatum
2

---

Published as a conference paper at ICLR 2025
along a specific axis/direction [Levinson 1996 Frank 1998). In situated communication, speech act participants (e.g., an addressee) may also be considered (Danziger 2010). To determine acceptable uses of various spatial relations, existing theories suggest that people htt spatial templates, which are centered on the relatum and aligned with the FoR (Lgan & Sadler 1996), to parse out regions of acceptability of certain directions (Franklin al. 1995 Carlson-Radvansly & Logan T997p.
Ambiguities in frame of reference. The choice of perspectives may lead to different FoRs, where Levinson (2003) has identified three main types of FoR: absolute, intrinsic, and relative. The absolute FoR uses cardinal directions, such as north and south, as fixed bearings. The intrinsic FoR aligns the origin with the relatum, describing the referent's position relative to the relatum's 's inherent orientation. The relative FoR positions a viewer (egocentric or addressee) as the origin, focusing on the observer's s intrinsic perspective. Liu et al. (010)) have highlighted the ambiguities in situated communication among three variations of intrinsic and relative FoRs (Figure :: the egocentric relative, the addressee- centered relative, and the object-centered intrinsic FoRs! When not specified, these FoRs are not easily distinguishable based solely on their linguistic expressions [Tenbrink] 2004). To resolve the ambiguity, individuals from diverse linguistic and cultural backgrounds adopt different preferences and conventions in choosing FoRs (Majid et al. 2004; O'Meara & Báez, 2011 Bohnemeyer et al.) 2014 Bender et al. 2020 Ogelo & Bylund| 2024).
Ambiguities in relative FoRs. The variations of relative FoRs form another source of ambiguity. After putting the origin of the coordination system on the viewer, multiple strategies specifying how to transform the axes can be considered (Figure [)). Different languages use different transformation
conventions to resolve the ambiguity of the froot-back and left-right of a relatum (Levinson) [0033 Shusterman & Li 2016), including: (1) translated projection (e.g., Hausa) where the coordinate frame of the speaker is directly applied, (2) rotated projection (e.g., Tamil), where the coordinate frame of the speaker is transformed with a 180-degree rotation, and (3) reflected projection (e.g..
English), where only the front-back axis is reversed.
2.2 SPATIAL UNDERSTANDING IN VISION-LANGUAGE MODELS
Large language models (LLMs) have exhibited strong adaptability that extends beyond text, encom- passing 2D and 3D vision Tsimpoukelli et al., 2021 Alayrac et al. 2022] Yang al., 2024), their affordances in the physical embodiment (Driess et al. [0033] Qian et al. 2024), and various other modalities (Yu et al. (024a). Especially, a variety of vision-language models (VLM) have been developed by visual instruction tuning on paired text-image data (Dai et al. 2023 Liu et al. 2023b] Dong et al., 2024). With supervised fine-tuning using entity-phrase mappings text-image pairs, grounded VLMs have been developed for fine-grained vision-language understanding at both the
region (Chen et al. 2023a Bai et al.) 2023
You et al. 2023 Peng el al. 2024) and pixel level (Lai
et al.
2024
Xia et al.] 2024 Rasheed et al. 2024 Zhang et al. 2024).
Spatial understanding is known to be challenging even for statc-of-the-ar art VLMs and is receiving
increasing attention (Achiam et al. 2023). In addition to using explicit spatial language understanding
modules (Rajabi & Kosecka| 2024). recent works such as Spatial (Chen et al., 2024a) and
SpatiaIRGPT (Cheng al.] (024) improve spatial reasoning in VLMs by leveraging 3D VQA or
scene graph data for supervised fine-tuning, Several benchmarks have also been developed to evaluate
spatial reasoning in VLMs from various perspectives (Liu et al., 2023a Cheng al. 2024 Kamath
et al. 2023). Still, these benchmarks overlook ambiguities related to the FoR, lack spatial continuity. and have not proposed metrics to evaluate the robustness and consistency of spatial reasoning.
3 CONSISTENT MULTILINGUAN FRAME OF REFERENCE TEST (COMFORT)
We introduce the COnsistent Multilingual Frame Of Reference Test (COMFORT), a new evaluation protocol with dataset, tasks, and comprehensive metrics, to study VLM behaviors in spatial language reasoning with FoR-related ambiguity. This protocol accommodates spatial continuity and various ambiguitics, drawing insights from several well-defined metrics to assess performance and prediction consistency. Given our primary focus on analytical and scientific inquiry rather than competitive testing only (Warstadt & Bowman 2022 Saxon et al.) 2024). in this work, we aim to develop better performance and consistency metries to deepen our understanding of model capabilities.
We exclude the absolute FoR from our study as it introduces little ambiguity (Lu al. 2010
3

---

Published as a conference paper at ICLR 2025
RALL.- of
Distractor
Object coloe
Bed
Bench
Bicydo
Lagtop
QX 9o"
901
pone
Dog
Sefa
180" 270
1807 270"
(a) Sample images from COMFORT-BALL -BALL. The images on the left are selected every 90" interval along the rotational path of 36 images. The images on the right illustrate variations with distractor, different object colors, sizes, or camera poses.
(b) Sample images from COMFORT-CAR The 4 images on the left are selected every 90% interval along the rotational path of 36 images. The images on the right sample images of each variation with different relatum objects.
Figure 3: Examples from the COMFORT-BALL and COMFORT-CAR datasets.
3.1 TASK FORMULATION
Following the setups object hallucination evaluation (I.i et al. 2023 Chen et al., 2024c), we formulate the task as spatial relation inference problem. In this task, VLM M presented with an RGB image Timg and a textual question Zqqeey. The image shows the egocentric perception of scene E S, where S is the set of possible scenes in which the referent moves along rotational trajectory with a constant radius from the relatum. In contrast to fixing the referents on the standard canonical axes, this setup better mirrors the spatial continuity in common real- world scenarios. A language prompt (Table queries whether a spatial relation E R. is satisfied by referent-relatum pair in the image under FoR work also examines models using queries with no FoR specified; therefore, test case in COMFORT L. While there are many spatial relations in
F (Figure 2 in language l E L. This
defined as 4-tuple in S x R (FU (0)) daily languages, we primarily focus on four canonical directions; that is, the considered relation set {to the left of, to the right of, in front of, behind). COMFORT covers |4] 109 languages worldwide; however, we use English an example to describe the data synthesis and evaluation processes for simplicity and clarity, and refer readers to Appendix for more details.
3.2 SCENE SETUP
We render the scenes into images using Blender (Blender Online Community. 2016). Each scene consists of referent and relatum. The referent follows rotational trajectory with a constant radius from the relatum to implement spatial continuity. Starting from the canonical front direction, we move the referent with a uniform step 10,, totaling up to images per scene. In COMFORT, there are configurations determined by whether the relatum has an intrinsic semantic front:
COMFORT-BALL When the relatum is non-fronted (e.g., Figure [p), we focus the ambiguity of FoR conventions associated with different languages. The split involves an observer's egocentric perception of a referent (e.g., red ball) and a non-fronted relatum (e.g., blue ball). We further randomize the dataset with object-level (colors, sizes, and shapes) and scene-level variations (camera positions and distractors) to consider more diverse yet reasonable settings (Figure 3a).
COMFORT-CAR: When the relatum fronted (e.g., Figure [Tb), multiple FoRs can explicitly adopted to interpret the scene. A COMFORT-CAR image, therefore, involves the egocentric perception referent, a fronted relatum, and an additional human addressee. One can interpret the spatial relations using either the Camera, Addressee, reference frame ambiguity. COMFORT-CAR has a set of 10 realistic objects in a typical household or outdoor scene, including horse, car, bench, laptop, rubber duck, chair, dog, sofa, bed, and bicycle, all of which have a clear semantic front. We use a basketball as the referent and vary the relatum. In addition these objects, we include human addressee in the scene. To disentangle different FoRs as much as possible, we let the addressee face right, and let the relatum face either left or right in the rendered images from the rendering camera's perspective (Figure
Relatum (C/A/R) as the origin resolve the
3.3 LANGUAGE QUERY SETUP
Given a pair of referent [A] and a relatum [B], together with spatial relation of interest, the query posed as "Is [A] [relation] [B]?" Depending whether not and which FoR is specified, the
4

---

Published as a conference paper at ICLR 2025
Origin
Prompt Template
1.00
nap
[A] [relation] [B]?
Conventional Right
0.75-
From the camera's viewpoint,
probabiitty
cam
is [A] [relation] [B]?
0.50
B
From the [addressee]'s viewpoint, [A] [relation] [B]?
add
0.25
Conventional Front
From the [relatum]'s viewpoint, is [A] [relation] [B]?
rel
0: deviation angle
-100
100
angle
Figure 4: A red ball with a deviation angle 90" relative to the conventional front (En- glish) of the blue ball.
Figure 5: The raw probability p(0) in gray, normalized proba- bility p(0) in black, and two ref- erence probability >hemi (0) and x0)( in red and purple.
Table 1: The origins of each coordinate system and the corresponding prompt templates for querying the FoR given eeférent-relationn-ellauu triple.
query is appended after four different perspective prompts (Table D no perspective (nop), camera perspective (cam), addressee perspective (add), and relatum perspective (rel). We only query from the camera egocentric perspective (cam) for COMFORT-BALL, focusing the ambiguity introduced by variations of the relative FoR. For COMFORT-CAR, we use all four possible language prompts to study how anbiguity in the reference system is resolved. Overall, for English, the above data generation pipeline leads to 720 test cases in COMFORT-BALL, and 57.6k test cases in COMFORT-CAR. The same method for dataset synthesis efficiency, we only include the scenes corresponding to the four most prototypical directions (i.c., left, right, front, and back) our multilingual analysis.
be generalized any other language; however, for computational
3.4 PERFORMANCE METRICS
Quantitatively assessing the spatial understanding and reasoning capabilities of models is challenging for two reasons. First, the physical world is continuous, and spatial relations may extend beyond the precise canonical front-back and left-right axes. As noted by Rallansky & Logan (1997). there exists regions of acceptability where, for instance, an object slightly to the front-left might still be considered being "in front." Second, as Dentella et al. (2023) pointed out, language models are biased towards affirmative responses. However, the intermediate representations may be sensitive to variations in input and, to some extent, align with human perceptions of spatial cues. Based on these concerns and findings, introduce multiple performance metrics, in addition to the vanilla accuracy, to enable more nuanced analyses.
Unless further clarified, we adopt right-handed coordinate system with the thumb pointing upwards when describing angles. We define the deviation angle 0 E (-180", 180") as the angular displacement from the canonical direction r the vector connecting the relatum and target. For example, in Figure[4] the deviation angle of canonical right from canonical front is = 90" Following Carlson- Radvansky Logan (1997), we define the acceptable region for a spatial relation hemisphere centered at the corresponding canonical direction. For VLM M and test case indexed by we let Presponsee; M) denote the probability of response E (Yes, No} assigned by M, and abbreviate it as P: (response) if there is no confusion.
the 180-degree
Accuracy. Given a spatial relation r in the textual prompt, we assess whether the assigned response probabilities correspond to whether the referent lies within the acceptable region defined by the relatum and r. Formally, we define the local probability of the model responding 'Yes' by 23 = P. P(Yess// [P. (Yes) + P. (No)]. We consider the inference correct if (1) the scene falls into the acceptability region and p: > 0.5 or (2) the scene falls out of the acceptability region and p; S 0.5.
Region parsing error. To mitigate the known bias towards affirmative answers, where Elp.] > 0.5. we normalize it across all image-prompt pairs, resulting in the normalized probability p. := (p: miny pj) (max, Py P--min -min, Pj). We adopt the root square error (RMSE) between the normalized acceptance probability p and reference probability threshold A that represents the actual regions of acceptability, eel v //EL (p: X /n. where A denotes the reference of the assigned probability, analogically to ground-truth labels in machine learning terms.
We introduce two analytically and geometrically motivated proposals defining Aee hemi and s based on hemispheres and cosine of angles, respectively. First, the bemisphere-baase reference heemi is defined X Abemi (0) 10 E (-90", (90)]. Here, 0t corresponds to the most prototypical
5

---

Published as a conference paper at ICLR 2025
behind
right
(a) An illustration of the spatial symmetry with respect
(b) Antonyms for spatial opposite consistency,
the (conventional) front. the red ball rotates around the
e.g., When evaluating if the red hall to left the blue ball, spatial opposite consistency ensures
blue ball, spatial symmetry consisteney ensures that cach symmetric pair, with different deviation angles 0 and -0, has the same probability of being identified the front.
the probability of accepting sample as left equals
the probability of identifying not right.
Figure Illustrations for the consistency metrics defined in COMFORT.
spatial relation, and 0 180 corresponds to the opposite. Intuitively, Auemi 1 denotes the test falls into the acceptable region defined by the textual prompt, and otherwise not. The second reference derived from the cosine of the deviation angle. Matching the range of the cosine function that of probability, i.e., [0, 1), we define the cosine -aased reference X=x( (8) by > (0) = (cos(0) + 11/2.
Figure shows an example of the vanilla probability curve p(0) from LLaVA-v1.5-77 (Liu al. 2023b), normalized probability curve p()), and two reference curves X hemmi (0) and X (f). We report both themi(() (0) and (0) in experiments. We also note that in human spatial cogmition, the regions of acceptability are neither mutually exclusive 90" quadrants nor overlapping 180 hemispheres, as they vary across individuals and depend on the situational context (Franklin al. 1995).
3.5 ROBUSTNESS METRICS
Standard deviation. COMFORT, some images depict variations of the same scene, sharing identical spatial relations between the referent and the object but differing in terms of object colors, sizes, or distractors. When the spatial relation and the query text remain unchanged, an ideal model should have consistent predictions for all variations. To measure model robustness, report the average standard deviation of the predicted probability p. across all deviation angles a == (O).
Prediction noise. Since our data is generated through interpolation, ideally, if a model well un- derstands spatial relations, the probability curve with respect to the deviation angle should be low-frequency (i.e., smooth) rather than high-frequency (i.e., noisy). Therefore, by the RMSE, denoted by between the predicted probability and Butterworth Low Pass Filter (LPF: Butterworth et al. 1930 == V p. - LPF( 12/n. A smaller value of 17 indicates that the probabilities are changing more smoothly, which more desirable.
measure the noise
3.6 CONSISTENCY METRICS
critical aspect of consistent spatial reasoning geometric symme- try. As our tested target object rotates around the relatum in circular path that is spatially symmetric, we expect the probabilities of an ideal VLM to consistently reflect geometric symmetry (Figure 6a). For pair of test cases, indexed by and j, that have the same configurations but opposite deviation
Spatial symmetric consistency.
angles, i.e., 0: + 00 = 0% we define the symmetry consistency: cxm :- E., f,)² 1).
Spatial opposite consistency. Similarly, we expect the probabilities of an ideal VLM to consistently reflect geometric opposition (Figure 6b). For example, the probability that sample is accepted by the spatial relation "to the left" should identical to the probability that it is rejected by "to the right." For pair of opposite spatial relation r, opp(r) R with the same configurations including the
deviation angles 0; the opposition consistency is given as: T = V o + P: capp(r) 1)2/n
4 EMPIRICAL EXPERIMENTS AND MAIN FINDINGS
The COMFORT framework enables to investigate whether the internal representations of vision- language models encode spatial relations, and they do, which underlying coordinate systems these representations capture. This can further be broken down into two research questions: (I) When presented with an ambiguous spatial expression, do VLMs follow conventions and exhibit specific preferred FoRs (and the coordinate transformation in relative FoRs) to resolve the ambiguity? (2) How effectively can VLMs adopt different FoRs, when perspectives are explicitly specified to disambiguate spatial expressions paired with visual scenes?

---

Published as a conference paper at ICLR 2025
Model
Back g00 (1)
Front gota
(1)
Left c
(L)
Right got
(1)
Aggregated
Preferred
Same
Rev.
Same
Rev.
Same
Rev.
Same
Rev.
Tran.
Rot.
Ref.
InstructBLIP..
LIP-BB
45.6
39.0
31,6
$2.0
37.2
48.0
47.5
37.8
40.5
44.2
43.9
DnstructBLIP-135
40.9
45.5
46.0
37.4
43,4
44.9
45.6
41.6
44,0
42.3
43.0
mBLIP-BLOOME
51.2
53.7
51.2
47.9
52.4
53.5
54.6
46.8
52.3
50.5
52.1
GLaMM
58.3
33.3
43.9
42.9
38.3
51.8
17.3
63.7
39.5
47.9
33.0
Ref.
LLaVA-1.5-7B
54.0
32.9
59.1
24.8
11.9
70.0
13.0
68.5
34.5
49.0
20.7
Ref.
LLaVA-1.5-13B
61.8
19.2
56.0
27.7
31.7
61.8
24.3
64.3
43.4
43.2
25.7
Ref.
XComposer2
73.2
17.9
74.5
20.7
20.1
80.9
21.3
81.1
47.3
50.1
20.0
Ref.
MiniCPM-V
70.9
21.9
64.3
26.9
19.7
74,1
21.1
73.3
44,0
49.1
22.4
Ref.
GPT-4o
75.7
28.2
73.6
32.0
24.3
80.8
25.1
80.8
49.7
55.5
27.4
Ref.
Table 2: Preferred coordinate transformation mapping from the egocentric viewer (camera) to the relatum in the relative FoR. The cosine region parsing errors goo are computed against both the Same and Reversed directions relative to the egocentric viewer's coordinate system. For example, native English speakers typically prefer Reflected transformation, which maintains the lateral (left/right) axis but reverses the sagittal (front/back) axis relative to the viewer (Figure D.. We determine the preferred transformation based on the aggregated performance, with "_" for no significant preference.
Back 0os (4)
Front go
Left g (L)
Right 200 (L)
Aggregated
Model
Prefer
Ego.
Int.
Add.
Ego.
Int.
Add.
Ego.
Int.
Add.
Ego.
Int.
Add.
Ego.
Int.
Add.
InstructBLIPP-
41.0
38,6
38.6
40.9
46.9
46.9
45.6
32.5
51.9
39.6
51.2
31.8
41.8
42.3
42.3
InstructBLIIP-138
32,9
34.4
34.4
52.5
48.5
48.5
47.8
56.2
27.8
40.6
27.6
56.6
43.5
41.7
41.8
mBL.I--B..OMMZ
52.2
53.2
53.2
45.3
44.6
44.6
47.8
47.6
48.1
45,4
48.4
42.4
47.7
48.4
47.1
GLaMM
28.0
49.1
49.1
30.0
40.2
40.2
14.0
56.8
41.5
13.7
53.0
46.6
21.4
49.8
44.4
Ego.
1.aVA-1.5-7B
20.9
43.0
43.0
34.5
32.6
32.6
13.4
53.5
47.4
14.3
53.6
49.3
20.8
45.7
43.1
Ego.
LLaVA-L.5-13B
31.9
38.8
38.8
24.8
57.1
57.1
11.7
51.1
51.1
27.5
57.4
48.7
24.0
51.1
48.9
Ego.
XComposer2
12.7
49.3
49.3
15.2
48.3
48.3
18.8
61.2
53.7
16.5
58.4
54.5
15.8
54.3
51.4
Ego.
MiniCPM-V
34.2
40.7
40.7
35.5
53.4
53.4
18.0
S3.9
58.4
19.0
58.1
52.7
26.7
51.5
51.3
Ego.
GPT-4o
38.3
36.7
36.7
43.1
50.2
50.2
34.7
59.3
56.5
24.3
57.3
61.7
35.1
50.9
51.3
Ego.
Table 3: Preferred frame of reference in VLMs. Models' Cosine Region Parsing Errors g are computed against the Intrinsic FoR (relatum origin), Egocentric relative FoR (camera origin), and Addressee-centric relative FoR (addressee origin). English typically prefers an egocentric relative FoR. We determine the preferred FoR based on the aggregated performance, with "_" indicating no significant preference.
In principle, COMFORT can be applied to all VLMs. whether multilingual or monolingual. We note that most existing open-source VLMs are English-based language models; therefore, we begin our experiments on English conventions, where both relative and intrinsic FoRs are available, but there is a conventional preference for a relative FoR combined with a reflected coordinate transformation in the relative FoR (see Levinson 2003 Table 5.4). We further extend our setup to multilingual settings by evaluating models in 109 languages across 170 regions worldwide. To cover a variety of VLMs with different capabilities and training approaches, evaluate the following models:
VLMs build from supervised instruction fine-tuning: LnstructBLIP-7B/1333
(Dai et al 2023).
LLa W--V-.----7/BB11.. 3B (Liu et al. 20236), InternLM-XCompoeerr (Dong et al. 2024);
VLMs with both supervised fine-tuning and reinforcement learning alignment: MiniCPM-Llama
V-v2.5-8B (Hu et al. 2024 Yu et al. 2024b);
Mechanistically grounded VLMs: GLaMM-7B (Rasheed et al. 2024
Multilingual VLM mBLIP-BLOOMZ-77 (Geigle et al. 2024) and GPT-4o OpenAl 2024
4.1 MOST VLMS PREFER REFLECTED COORDINATE TRANSFORMATION CONVENTION
In this section, we address the research question: do VLMS have preferred coordinate trans- formation convention, and if so, what is it? Experiments are conducted on COMFORT-BAR using the camera perspective prompt (cam) that explicitly specifies egocentric relative FoR (Table 2
Table 7 in the appendix shows the complete evaluation including = hhemi and g0
We observe that almost all VLMs demonstrate clear preference over the reflected transformation similar to English, except the BLIP series. Still, some models are also affected by the ambiguity of multiple transformation conventions. With the textual prompting specifying relation, 0 = 0. GPT-4o and LLaVA-1.5-13B show a sharp drop of performance and significant variance for behind and ri right, respectively (Figure 77. indicating that some models are sensitive to other transformations.
"The Pal.I series CCen al. 2023cb b 2024b) are closed soureed.
7

---

Published as a conference paper at ICLR 2025
Egocentric
Intrinsic
Addressee
Aggregated
Model
Acc (1)
(4)
Acc% (t)
exio=
Acc% (1) ex10= (4)
Acc% (1)
< 6x10= (+)
InstruciBLIP-77
47.2(+0.0)
(3.5
47.2(+0.0)
42.3(+0.0)
47.2(+0.0)
(3.6(+1.))
(7.2(+0.0)
43.1
InstructBLIP-
47.2(+0.0)
43.8(+0.3)
47.2(+0.0)
43.2(+1.5)
47.2(+0.0)
(2.9(+1.1)
47.2(+0.0)
43.3(+1.0)
mBLIP-BLOOMW
-..9
(5.4(47.7))
49.8;-3.0)
54.2(15.8)
49.61-3.2)
55.8(48.7))
(0.4(-2.4)
55.1(47.4)
GLaMM
-
2-.3)-
47.2(+0.8)
44.2,
47.2(-2.8)
42.8
47.2;-1.2)
36.8)-4.0)
LLa 1.577
25.2(-2.6)
18.4
48.3(44.7)
45.7(-4.1)
(8.2)-
-3.4(-1.0)
50.6;-100) 35.8
LLaVA-IL5-13BB
51.6.-15.0
(3.9
47.3
45.0
(7.5(-3.8)
(8.9(-4.)
35.9; (8.8(-6.0)
XComposer2
85.6
7.0)
(8.8
(1.0
51.0,
53.2
0.6]
(-.6) 63.3, 2.4) 39.9;
49.8
MiniCPM-V
72.4; 44.8)
24.6
47.8; (9.9.-2.))
(2.6)
52.9
0.5)
45.1
58.4,-26) 39.2,
58.4,
GPT-4o
(8.3(44.6)
28.1
53,4
-4.6
49.1 (-5.7)
44.9
60.3
69.2
Table 4: The accuracy and cosine region parsing errors of VLMs when explicitly prompted to follow each frame of reference are provided (cam/rel/add). The values in parentheses indicate the performance change relative to the scenario with perspective (nop) prompting.
Obj (f)
Acc% (1)
gane
(4)
hemmi (L)
x (L) 7xx10 (4)
PPP x 10
(4)
Model
"10
x10
x102
BALL
CAR
BALL
CAR
BALL
CAR
BALL
CAR
BALL
CAR
BALL
CAR
BALL
CAR
BALL
CAR
InstructBLIP-P1
66.7 66.7
47.2
47.2
43,9
43.5
57.8
56.4
26.7
30.5
48.4
43.4
17.2
16.9
16.6 22.6
InstructBLIP 13B
67.3 50.3
47.2
47.2
43.0
43.8
55.5
55.9
27.1
36.8
48.2
46.4
17.3
17.0
21.0
21.9
MBLIP-BILOOME
99.1
33.3
47.5
51.9
52.1
55.4
62.1
65.6
43.7
48.6
54.1
60.7
29.1
30.1
33.8
42.0
GLaMM
100.0
99.8
47.2
47.2
33.0
23.3
45.2
37.6
29.9
23.4
45.0
28.4
10.1
9.4
13.7
14.6
LLaVA-11.57B
100.0 88.6
63.2
55.2
20.7
18.4
33.7
32.5
25.2
20.0
23.5
21.8
588
5,4
8.3
10.7
LL5VA-L.5-13B
100.0 98.6
55.3
51.6
25.7
23.8
37.6
37.1
19.3
20.8
24.9
29.9
7.0
5.8
9.3
10.8
XComposer2
100.0 95.3
92.4
85.6
20.0
18.8
21.1
26.3
19.2
15.3
13.7
22.9
9.0
6.5
10.5
12.0
MiniCPM-V
66.8 81.5
81.0
72.4
22.4
24.6
32.8
35.8
19.2
19.2
29.8
22.7
10.1
9.2
12.4
14.9
GPT-4o
100.0 94.5
89.2
78.3
27.4
28.1
27.5
35.0
20.9
24.0
43.1
38.8
14.1
13.3
14.2
16.7
Random (30 trials)
50.0
50.9
46.3
58.7
28.3
26.6
42.5
44.2
Always "Yes"
50.0
47.2
61.2
68.7
0.0
0.0
0.0
100.0
Table 5: A comprehensive evaluation of VLMs in egocentric relative FoR with reflected transfor- mation, using an explicit camera perspective (cam) prompt, is conducted. The metrics considered include object hallucination (Fl-score), accuracy (Acc), region parsing error (#), prediction noise (1)) standard deviation (o), and consistency (c).
4.2 MOST VLMS PREFER EGOCENTRIC RELATIVE FRAME OF REFERENCE
We now attempt to answer the research ques- tion: do VLMs have a preferred frame of ref- erence, and if so, what is it? We conduct our study on COMFORT-CAR using the no perspective prompt (nop) that deliberately leaves the FoR ambiguous. When calculating the performance with respect relative FoRs (either egocentric or ceddresee--enterele
0.73-
grbbaillty
0.75
0,50-
0.50
0,00
-1.00
502
0:00
-200
0
120
we assume reflected coordinate transformation convention. Table shows the results of preferred FoR in English measured by the region parsing error gos and Table in the appendix shows the complete eval- uation including both heemi and gx Almost all VLMs demonstrate a significant preference for the egocentric relative FoR similar to English, again, except for the BLIP series. Additionally, the models' performances are inconsistent across different spatial relations moaiiosssmmolls models generally perform better in the lateral directions (left and right) than the sagittal ones (front and behind), even in competitive industry models like GPT-4o. For instance, GLaMM does not show a very strong preference when resolving ambiguities along the sagittal axes, but it demonstrates significant preference when resolving lateral ambiguity.
angle
argl
Behind in GPT-4o.
(b) Right in LLaVA-133
Figure 7: At 0 O, some models show sensitivity to multiple conventions.
4.3 VLMS FAIL TO ADOPT ALTERNATIVE FRAMES OF REFERENCE FLEXIBLY
We now address the research question: can VLMs adopt different FoRs when perspectives are explicitly specified to disambiguate spatial expressions? We again use COMFORT-CAR however, instead of using the no-perspective prompt (nop), require VLMs to follow one FoR by explicitly specifying the perspective (cam/rel/add) in the textual prompts (Table Table shows the results in accuracy and gos and the performance compared when no perspective is specified, and Table 9
8

---

Published as a conference paper at ICLR 2025
Language
English
Tamil
Hausa
Intrinsic
50.9
$2.0
54.0
Ref.
35.8
40.4
41.0
Ego-Rel
Rot.
57.3
55.2
56.
Tran.
53.7
51.1
53.0
(Radatiivv
Ref.
58.8
$2.2
52.8
Add-Rel
Rot.
51.3
52.9
55.3
Tran.
56.1
56.1
56.1
GPT-4o Prefer Human Prefer
Ego-Ref. Ego-Ref.
Ego-Ref. Ego-Rot.
Ego-Ref. Ego-Trans.
(tazinni))
Figure 8: GPT-4o's preferences for intrinsic FoR over the relative FoR across regions. The plot is based on the top three spoken languages in each region, as reported by The World Factbook (Central Intelligence Agency| 2009), and averages the cosine parsing error (etth J). weighted by the speaking population. We present a quantitative comparison of English, Tamil, and Hausa, with the best- performing FoR marked in bold and the convention preferred by human speakers underlined.
in the appendix gives the complete evaluation. We find that all models, including the strong ones like GPT-4 and InternL--XC MXCoomposerr, show close-to-chance performance (50% accuracy) when being prompted to the intrinsic addressee-centeeed relative FoRs. Compared the same probing setup without a perspective specified (nop), we find generally marginal improvements in region parsing error (E), whereas the accuracy decreases. Overall, the results indicate that while VLMs can comprehend scenes using egocentric relative FoR, they struggle to adapt flexibly to alternative FoRs.
4.4 SPATIAL REPRESENTATION, IN VLMS ARE NOT ROBUST AND CONSISTENT
In this section, we further ask: are spatial representations in VLMs robust and consistent? The considered metrics include accuracy (Acc), region parsing error (E), prediction noise (7), standard deviation (o). and consistency (c) as defined in Section 3 One commonly considered possibility that VLMS underperform is that they suffer from object hallucination, where they misperceive objects in the scenes (Li al.) 2023) Chen et al.) 2024c). Following the object probing setups, prompt the VLMs to inquire about the presence of an existing object and a non non-existing object in the scene, and compute the Fl-score (Table We find that the BLIP models suffer from severe object hallucinations, which contribute to their underperformance in the previous evaluation. Many VLMs, despite showing decent performance metrics in terms of spatial understanding and reasoning accuracy. demonstrate lack of robustness and consistency. For example, the spatial opposite consistency (C of GPT-4 is significantly better than 30 random trials. In contrast, VLMs that have undergone supervised fine- tuning on spatial relation tasks have more robust and consistent spatial representation. For instance, IntemLL-XXCom CCompose22 and MiniCPM-V (on the COMFORT-BA show improved performance. On the other hand, although GLaMM is mechanistically grounded to objects and exhibits minimal object hallucination, its spatial understanding capability is poor. This suggests that improving visual entity grounding helps in recognizing individual objects but does not automatically translate to better spatial understanding between multiple objects.
task, with no object hallucinations)
4.5 A CROSS-LINGUAL AND CROSS-CULTURAL EVALUATION OF FRAME OF REFERENCE
All previous experiments are centered around English; however, individuals from diverse multilingual and cultural backgrounds may adopt different preferences and conventions select their FoR in resolving ambiguities (Majid et al., 2004 O O'Meara & Báez 2011 Bohnemeyer et al.) 2014 Ogelo & Bylund 2024). Our next research question naturally arises: Do multilingual VLMs faithfully follow the preferences and conventions (associated with different languages) to select the FoR? To extend the study of preferred FoR from English a multilingual setting, we evaluate 109 languages worldwide to investigate whether each language shows preferred FoR. We translate the English prompts into the target languages using the Google Cloud Translate API. Given that the open-source language models either lack strong multilingual capabilities underperform in previous evaluations, we study this problem on the GPT-40 model (OpenAl 2024). We follow the setup similar to 4.2 but only evaluate the images corresponding to the four canonical directions using the nop prompt. For each language, we compute coo for each FoR and coordinate transformation. Figure g presents a visualization of the world map, displaying the preference of each region for using the (object-centered) intrinsic FoR over the relative FoR, where the latter corresponds to a low gees value. Table 10 in the Appendix summarizes the results across all tested languages.
Section

---

Published as a conference paper at ICLR 2025
Nearly all tested languages demonstrate preference towards the relative FoR, except several underrepresented languages, such as Konkami, Kurdish, and Amharic, which exhibit near-random performance without significant preference. In Figure [8] we present a classic comparison between English, Tamil, and Hausa similar to that of Levinsso (2003), with the best-performing FoR marked in bold, and the preferred convention by humans underlined. Although human speakers of these languages have different preferred coordinate transformation conventions, the English convention of reflected projection is observed for both Tamil and Hausa. Although, for example, Hausa permits an English-like interpretation of front-back relations, this interpretation is generally less favored and may confuse Hausa speakers (Hill) 1982). This raises concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs.
DISCUSSIONS
Do vision-language models represent space and how? It insufficient to answer this question by simply querying the model with text-image pairs and comparing the output with a fixed ground truth. We must, least, query the models with awareness of the ambiguity in FoRs, which is essential in determining how the scenes in the physical world are mapped to spatial expressions (Levinson 2003). Our experiments confirm that many VLMs are equipped with reasonable spatial representations through dision-langues training alone; in particular, most VLMs clearly prefer the egocentric relative FoR with reflected projection, aligning with English conventions. However, our results also show these representations lack robustness and consistency in a continuous space. Similar experimental setups can yield widely varying performance across different spatial relations- for example, GPT-4 shows minimal preferences for the egocentric relative FoR along the sagittal axis but a significant preference along the lateral one (Table 3 As a result, VLMs demonstrate unsatisfactory consistency in their spatial performance (Table 5) Future work is necessary to improve the consistency and robustness of spatial representations in these models.
Perspective taking as prerequisite of human-like spatial reasoning. Most languages support multiple FoRs F The ability to understand and reason about space from a non-egocentric perspective is an important foundation of the Theory of Mind, a basic building block of our situated communication skill that allows us to infer others' mental states (Ma et al.) (2023b). One of our key findings is that VLMs still struggle to adopt alternative FoRs flexibly, even when provided with explicit perspective- taking instructions (Section 4.3). We hypothesize that this phenomenon may come from a reporting bias in the image-text datasets available on the internet-ir it is natural to take the reflected relative FoR to view images presented on screen, but this does not always apply in real-world applications. To address this issue, we suggest future work extend the current 2D VLMs to the 3D domain, by considering camera poses and multiview data (Yang et al. 2024 Hao et al. 2024) for training.
Cross-cultural conventions in cross-lingual spatial understanding. The conventions for resolving spatial ambiguities are not uniform, individuals from diverse linguistic and cultural backgrounds select their FoR differently. Cultural conventions can even be transmitted as individuals are exposed to other languages. [Bohnemeyer et al. (2014) found that among indigenous language speakers with various preferences in FoRs, those more proficient in Spanish tend to use the reflected relative FoR (Spanish convention) more in their native language. This phenomenon has led to their Linguistic Transmission Hypothesis: "Using any language or linguistic variety independently of its structures may facilitate the acquisition of cultural practices of nonlinguistic cognition shared among the speakers of the language." Analogously, our experiment raises important concerns that English may dominate the FoR preference conventions of other languages in multilingual VLMs. This is not surprising, as current training recipes for multilingual multimodal language models heavily rely on captions (Chen et al., [2033 Geigle et al., 2024); however, this practice can be problematic: for instance, Hausa prefers an interpretation where the "front" aligns with the English concept of "back," " (Hill. 1982), where this approach may lead to English conventions overshadowing those of other languages. At high level, this issue is not limited to spatial reasoning: an example, Shi et al.] (2023) have demonstrated that English is always the best chain-of- of-thought language for math reasoning with multilingual L.Ms,, no matter what language is used for the problem description. To enable similar linguistic transmission in Al models, exposure to naturally generated multilingual image-text data is crucial (Romero et al. 2024).
SSoe languages, in very cases, have only one available spatial frame of reference. For example, Guugu Yimithirr exclusively uses the absolute FoR (Levinson 2003).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by NSF ISS-1999334, NSERC RGPIN-2024-043395, ONR N00014- 23-1-2417, and the Microsoft Accelerate Foundation Models Research (AFMR) grant program Ziqiao Ma is supported in part by the Weinberg Cognitive Science Fellowship. Any opinions, findings, conclusions, recommendations expressed in this material are those of the authors and do not necessarily reflect the views of these funding agencies. The authors would like to thank Yinpei Dai, Run Peng, Jung-Chur Liu, and Xuejun Zhang for proofreading and feedback.
REFERENCES
Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2333.088774 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, Advances in neural information processing systems, 35:1771-37357 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Andrea Bender, Sarah
Annelie Rothe-Wull Wulf, Miriam Seel, and Sieghard Beller. Being In Front is good- but where is In Front ? preferences for spatial referencing affect evaluation. Cognitive Science, (4(6):e12840) 2020.
Blender Online Community. Blender 3d modelling and rendering package. Blender Foundation,
Blender Institute, 2016.
Jürgen Bohnemeyer, Katharine Donelson, Randi Tucker, Elena Benedicto, Alejandra Capistrán Garza, Alyson Eggleston, Néstor Hernández Green, María de Jesús Selene Hernández Gómez, Samuel Herrera Castro, Carolyn O' Meara, al. The cultural transmission of spatial cognition: Evidence from a large-scale study. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 36, 2014.
Stephen Butterworth et al. On the theory of filter amplifiers. Wireless Engineer, 7(6):536-541) 541, 1930.
Laura A Carlson- Raddvnskk and Gordon D Logan. The influence of reference frame selection on spatial template construction. Journal of memory and language, 37(3):411-44) 437, 1997.
Central Intelligence Agency. The world factbook 2009. Central Intelligence Agency, 2009. https [Accessed: (Jun 1st 2024)].
7/www.cit cia. waa..gov//heee wwwp:/oddaaatooooieeee/////e///e
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing lasion-language models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. 15/95, 2023a.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint a7X1v22310.09199 2023b.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scale multilingual language-imag model. In The Eleventh International Conference on Learning Representations, 2023c.
11

---

Published as a conference paper at ICLR 2025
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. On scaling up a multilingual vision and language model. In Proceedings of the IEEE/CVP Conference on Computer Vision and Puttern Recognition, pp. 14432-14444, 2024b.
Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang. Sihan Xu, Shengyi Qian, Jianing Yang, David F Foubey, and Joyce Chai. Multi-object hallucination in vision-language models. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024c.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. arXiv preprint 2xXi:24006.01844, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In Advances in Neural Information Processing Systems, volume 36, 2023.
Eve Danziger. Deixis, gesture, and cognition in spatial frame of reference typology. Studies in Language. International Journal sponsored by the Foundation "Foundations of Language", 34(1): 167-185, 2010.
Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three language models reveals low language accuracy, absence of response stability, and yes-response bias. Proceedings of the National Academy of Sciences, 120651): 120, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao. Bin Wang. Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. diternlm-xcompose Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint 16X4::2401.16420, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et Palm-e: An embodied multimodal language model. In International Conference Machine Learning, pp. 1469-8488, PMLR, 2023.
Carola Eschenbach. Contextual, functional, and geometric components the semantics of projective terms. In Functional Features in Language and Space: Insights from Perception, Categorization, and Development. Oxford University Press, 12 2004.
Andrew U Frank. Formal models for cognition--axxonnyy of spatial location description and frames of reference. Spatial cognition: An interdisciplinary approach to representing and processing spatial knowledge. pp. 293-312, 1998.
Nancy Franklin, Linda A Henkel, and Thomas Zangas. Parsing surrounding space into regions. Memory & Cognition, 23:397 407, 1995.
Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavaš. mblip: Efficient bootstrapping of multilingual vision-llms. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR). pp. 7-25, 2024.
Dedre Gentner, Asli Özyürek, Özge Gürcanli, and Susan Goldin-Meadow. Spatial language facilitates spatial cognition: Evidence from children who lack language input. Cognition, 111-33338--30, 2013.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang. Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entan- gled language hallucination and visual illusion in large vision-language models. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375-14385, 2024.
Shengyu Hao, Wenhao Chai, Zhonghan Zhao, Meiqi Sun, Wendi Hu, Jieyang Zhou, Yixian Zhao, Qi Li, Yizhou Wang. Xi Li, et al. Ego3dt: Tracking every 3d object in ego-centric videos. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 2945-2955 2024.
12

---

Published as a conference paper at ICLR 2025
William G Hayward and Michael J Tarr. Spatial language and spatial representation. Cognition, 55 (1):39-84, 84, 1995.
Clifford Hill. Up/down, front/back, left/right. a contrastive study of hausa and english. Here and there: Cross-linguistic studies on deixis and demonstrution, 1342, 1982.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint mxXiv:2404.06392 2024.
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. MMTom- MA:: Multimodal theory of mind question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16077-16102, 2024.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's "up" with vision-language models? investigating their struggle with spatial reasoning. In EMNLP, 2023.
Adam Kendon. Spacing and orientation in co-present interaction. Development of Multimodal Interfaces: Active Listening and Synchrony: Second COST 2102 International Training School, Dublin, Ireland, March 23-27, 2009, Revised Selected Papers, pp. 1-15, 2010.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiay Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1579-95599 2024.
Stephen C Levinson. Frames of reference and molyneux's question: Crosslinguistic evidence. Language and Space, pp. 109-1700 170, 1996.
Stephen Levinson. Space language and cognition: Explorations cognitive diversity, volume 5. Cambridge University Press, 2003.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, 2023.
Changsong Liu, Jacob Walker, and Joyce Y Chai. Ambiguities in spatial language understanding in situated human robot dialogue. In 2010 AAAI Fall Symposium Series, 2010.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions the Association for Computational Linguistics, 11:635-651, 2023a.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, volume 36, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/ 2023c.
Gordon Logan and Daniel D Sadler. A computational analysis of the apprehension of spatial relations. Language and Space, pp. 493-530, 1996.
Ziqiao Ma, Jiayi Pan, and Joyce Chai. World-to torlddto-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 524-544, 2023a.
Ziqiao Ma, Jacob Sansom, Run Peng, and Joyce Chai. Towards a holistic landscape of situated theory of mind in large language models. Findings of Empirical Methody in Natural Language Processing. 2023b.
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision- languae foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 109 10-10221, 2023c.
13

---

Published as a conference paper at ICLR 2025
Asifa Majid, Melissa Bowerman, Sotaro Kita, Daniel BM Haun, and Stephen C Levinson. Can language restructure cognition? the case for space. Trends in cognitive sciences, 8(3):108-114, 2004.
Roshanak Mirzaee and Parisa Kordjamshidi. Transfer learning with synthetic corpora for spatial role labeling and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6148-6165, Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics. URL https:://clantthl.pgr org/ 2022.
413
Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. SPARTQA: A
textual question answering benchmark for spatial reasoning. In Proceedings the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4582-4598 Online, June 2021. Association for Computational Linguisties. doi: 10.18653/v1/202 maiaccmmaann.644.. htt.https:///at /aappttooo//ll org/ 222g/2221..2 naacl-main. 364
Reinhard Moratz and Thora Tenbrink. Spatial reference in linguistic human-robot interaction: Iterative, empirically supported development of a model of projective relations. Spatial cognition and computation, 6(1):63-107, 2006.
Edward Munnich, Barbara Landau, and Barbara Anne Dosher. Spatial language and spatial represen- tation: A cross-linguistic comparison. Cognition, 81(3): 2001.
Awino Ogelo and Emanuel Bylund. Spatial frames of reference in dholuo. Language Sciences, 104: 101614, 2024.
OpenAI. Hello gpt-4o, May 2024. URL httptss//ooeaa...co///////l/lllp
Carolyn O' Meara and Gabriela Pérez Báez. Spatial frames of reference in mesoamerican languages. Language Sciences, 33(6)8837-852, 2011.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shumiing Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations, 2024.
Tanawan Premsri and Parisa Kordjamshidi. Neuro-symbolic training for reasoning over spatial language. Proceedings of NAACI-2025. Association for Computational Linguistics, 2025.
Shannon M Pruden, Susan C Levine, and Janellen Huttenlocher. Children's ss spatial thinking: Does talk about the spatial world matter? Developmental science, 14(6):1417-1430 2011.
Jennie E Pyers, Anna Shusterman, Ann Senghas, Elizabeth S Spelke, and Karen Emmorey. Evidence from an emerging sign language reveals that language supports spatial cognition. Proceedings of the National Academy of Sciences, 107(27): 121111111112,2.,.111111122222111 2010.
Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Li Erran Li. Affordancellm: Grounding affordance from vision language models. In Proceedings of the Second Workshop on Open- Vocabulary 3D Scene Understanding, 2024.
Navid Rajabi and Jana Kosecka. Towards grounded visual spatial reasoning in multi-modal vision language models. In ICLR 2024 Workshop on Data-centric Machine Learning Research (DMLR): Harnessing Momentum for Science, 2024.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing. Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Terry Regier and Laura A Carlson. Grounding spatial language in perception: empirical and computational investigation. Journal of experimental psychology: General, 130(2):273, 2001.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
14

---

Published as a conference paper at ICLR 2025
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. Cvqa: Culverally-diverse multilingual visual question answering benchmark. arXiv preprint arXiv:2406.05967, 2024.
Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. Benchmarks as microscopes: A call for model metrology. In Proceedings of the First Conference on Language Modeling, 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are mul- tilingual chain-of ofttht reasoners. In The Eleventh International Conference on Learning Representations, 2023.
Anna Shusterman and Peggy Li. Frames of reference in spatial language acquisition. Cognitive 2016.
psychology,
Thora Tenbrink. Identifying objects on the basis of spatial contrast: An empirical study. In International Conference on Spatial Cognition, pp. 124-146. Springer, 2004.
Luca Tommasi and Bruno Laeng. Psychology of spatial cognition. Wiley Interdisciplinary Reviews: Cognitive Science, 3(6)::65-558)) 2012.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Processing Systems, volume 34, pp. 200-212, 2021.
Marina Vasilyeva and Stella F Lourenco. Development of spatial cognition. Wiley Interdisciplinary Reviews: Cognitive Science, 3(3):349-362, 2012.
Alex Warstadt and Samuel R Bowman. What artificial neural networks can tell us about human language acquisition. Algebraic structures in natural language, pp. 17-60. CRC Press, 2022.
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Jianing Yang, Xuweiyi Chen, Nikhil Madaan, Madhavan Iyengar, Shengyi Qian, David Fouhey, and Joyce Chai. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination. arXiv preprint arXiv:2406.05112,, 2024.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzh Du Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations, 2023.
Shoubin Yu, Jaehong Yoon, and Mohit Bansal. Crema: Multimodal compositional video reasoning via efficient modular adaptation and fusion. arXiv preprint arXiv:2402.05889, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He. Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.1 17220, 2024b.
Xiang Yue, Yuansheng Ni, Kai Zhang. Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipling multimodal under- standing and reasoning benehmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Yichi Zhang, Jiayi Pan, Yuchen Zhou, Rui Pan, and Joyce Chai. Grounding visual illusions in language: Do Dision-language models perceive illusions like humans? In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 5718-5728, 2023.
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
15

---

Published as a conference paper at ICLR 2025
A DATASET AND METRIC DETAILS
A.1 DATASET CONFIGURATIONS
The entire data generation pipeline produces 720 English test cases in COMFORT-BALL, and 57.6k English test cases COMFORT-CAR, For COMFORT-BALL: object combination variants relations 36 angles 720 test cases. For COMFORT-CAR: 20 object combinations x 5 variants relations x 36 angles x 4 prompts = 57,600 test cases. The table below lists all possible variants and configurations for the dataset, and we describe our dataset configuration in detail follows.
Test Case Setup
Possible Variants
COMFORT-BALL: Relatum: red ball; Referent: blue ball; 36 samples uniformly collected along a rotational path.
Scene S
COMFORT-CAR: Relatum: basketball; Referent: horse, car, bench, laptop, rubber duck,
chair, dog, sofa, bed, bicycle; Addressee: woman; 36 samples uniformly collected along a rotational path.
Spatial Relation R
to the left of. to the right of. in front of. behind
Frame of Reference F egocentric relative, addressee-centered relative, centered intrinsic
Language
See Table 10
Table 6: A test in COMFORT is defined a 4-tuple in S x R (FU(0)) x L. This table enumerates all possible variants and configurations of the dataset.
A.2 LIST OF EVALUATED LANGUAGES
We started with 132 candidate languages supported by Google Translate API We removed 23 languages from our multilingual evaluation due to their failure to adhere to instructions for generating "yes" and "no" predictions, because they did not the back-translation test for quality control: Aymara, Bambara, Croatian, Dhivehi, Dogri, Ewe, Guarani, Hmong, Kyrgyz, Luganda, Malayalam, Meiteilon (Manipuri). Mizo, Odia (Oriya), Punjabi, Quechua, Samoan, Tatar, Telugu, Tigrinya, Uyghur, Xhosa, Yoruba.
A.3 VISUALIZATIONS OF REGION PARSING ERROR
189
3.0
M
1.00
1.00
925
075
(8) hhemi
10.7
(b)
homi
24.7
(c) hemi
34.8
(d) 5 hemi
47.2
(c) c
hemi
55.3
000
130
0.0
20
000
(f) 004
10.4
(g)
-00
22.4
(h) 000
32.6
(i) o00
42.1
(i g00 50.3
Figure 9: visualization: (a-e) correspond hemi and (f-i) correspond goos
B LIMITATIONS AND INTENDED USE
B.1
LIMITATIONS
The acceptance regions. Using cosine and hemisphere as acceptance regions is analytical but might not capture some human cognitive biases. reality, regional angles might not be uniformly
https://cclus htt.://w....ooll....oot/tttsssss
16

---

Published as a conference paper at ICLR 2025
distributed per relation, nor they exactly 90 degrees. These angles vary across individuals and cultures (Franklin et al.) [995).
Spatial relations. This work primarily focuses on the most basic types of spatial relations (front- back and left-right). However, many other relations cxist, such as away from and near (Logan & Sadler
1996 Liu et al. 2023a). Additionally, not all languages possess terms for "left," "right," "front," and "back." Some languages, like Guugu Yimithirr, use only absolute frames of reference instead (Levinson] 2003).
Camera angle and occlusion. Currently, there is no occlusion, and the camera angle is high. Languages may differ in the importance placed these factors, such as the preference "behind" in cases of occlusion (Levinson 2003).
Pragmatic aspect of spatial cognition. Many conversational and pragmatic aspects of spatial cogni- tion are simplified in this work, such as F-Formation (Kendon (2010) and human-robot interaction (Liu al., 2010). For example, in human-robot interaction settings, users prefer an addressee-centerr frame of reference to facilitate the robot's ' comprehension of spatial referents (Moratz & Tenbbrik 2006).
Multilingual prompts. In this work, we used machine-translated text to construct the multilingual portion of the dataset. Although we verified data quality through back translation, incorporating human annotations in the future would be valuable future step.
B.2 INTENDED USE
We intentionally use the term "framework" to emphasize that this dataset and its associated evaluation metrics designed to assess cognitive similarity and alignment with human spatial cognition. While each studied language demonstrates preference, we do not position this as a lcaderboard-driver benchmark. However, the perspective-taking capability of VLMs as studied 4.3 can function as benchmark, as the input and ground truth are unambiguous. We encourage future work in VLM training address this specific challenge.
C CASE STUDIES
We added two case studies to augment the results in COMFORT:
C.1 ALTERNATIVE BACKGROUND
We first create a case study with brown background color brown from COMFORT-BALL. We keep the objects visible and generate images for each spatial relation. The images are shown below:
C.2
IMAGES
We added a pair of case studies to see results in synthetic images generalize to real images. We first create real-world version of the COMFORT-BALL dataset with no variations. We fixed the camera pose
17

---

Published as a conference paper at ICLR 2025
and manually rotated the target object around the reference object using equally sized red and blue balls, with all other setups identical to the Blender simulation. We took images in total along the rotational path for each spatial relation. We also create real-world version of the COMFORT-CAN dataset without the addressee. We used laptop and the red hall to collect a set of 36 images for each spatial relation. The images shown below:
C.3 RESULTS
For the case study, evaluated three VLMs: LLaVA-v1.5-7B/13B Liu al. (2023b) and GPT- 4o Ao 2024). Prediction plots under the camera perspective prompt (cam) are shown in Figure To Prediction plots under the camera perspective prompt (cam) shown in Figure [1] Prediction plots under the camera perspective prompt (cam) are shown in Figure [2] Our findings remain consistent even when applied to these case studies and the general trend holds. However, we did observe that models performed with more noise compared to the synthetic data. We hypothesize that this noise arises from small inconsistencies between images in the real dataset, such slight tilts in the positioning of the camera and the target object.
D ADDENDUM TO RESULTS
18

---

Published as a conference paper at ICLR 2025
Left
Right 142
Front
Back
112
40.5
LLAVVA .,578
Iebuakt
aigh-
4x30
xgg+
44.50
348
1.00
LL 1.5 1.3
8.75
025
8.50-
angh
xndx
aagl
23
218
52.1
1.00
25.7
GP750
8.50
a0gh
483b
Figure 10: All prediction plots for each model on COMFORT-BALL with brown background using the camera perspective prompt (cam). The probability p(0) in gray, normalized probability ((() in black, and the reference probability (o))(0) of cam in red.
Left
Right
Front
Back
125
1.00
37,
LLAVVA .,57
Imbaette
3.50
8.c0
angh
axy>
xgdx
aagl
41.
24.
1.00
1966
L.AVV 11A11113
3.55
0.5
Mw
kigh
s8.o
19K
1.00
1.00
1.00
Grr:BL1
GPT4o
3.50
a8go
Figure 11: All prediction plots for each model on real image version of COMFORT-BALL using the camera perspective prompt (cam). The probability p(0) in gray, normalized probability p(0) in black, and the reference probability (o)s()) of cam in red.
19

---

Published as a conference paper at ICLR 2025
Left
Right
Front
Back
433
1.00
90.4
LL 1,57
-
0.00
43.8
Syc 28.4
1.00
454
L.aVA, 51
1.
-00
0.01
ky 17.2
16.4
Gyt
Kur
GPT4o
0.0
Figure 12: All prediction plots for each model on the real image version of COMFORT using the
camera perspective prompt (cam). The raw probability p(0) in gray, normalized probability p(() in black, and the reference probability (0) of in red, rel in blue.
Back
Front
Same
Reversed
Same
Reversed
Acc%
Acc%
heni
Acc%
hunni
Acc%
elox
1x
x100
x100
xx
=x00
InstructBLIP-72
47.2
58.4
45.6
47.2
53.8
39.0
47.2
47.5
31.6
47.2
64.6
52.0
InstrucIBLIP--13B
47.2
55.9
40.9
47.2
56.6
45.5
47.2
60.0
46.0
47.2
53.0
37.4
mBLIP
56.1
60.2
51.2
47.2
64.8
53.7
51.1
61.4
$1.2
47.8
58.0
47.9
GLaMM
47.2
71.1
58.3
47.2
46.3
33.3
47.2
55.4
43.9
47.2
55.9
42.9
L.L.VA-L5--B
47.2
66.7
54.0
47.2
47.0
32.9
47.2
71.0
59.1
47.2
36.4
24.8
LLaVA-1.5-13B
47.2
73.8
61.8
47.2
36.3
19.2
42.8
67.3
56.0
51.7
39.1
27.7
XComposer2
13.3
84.5
73.2
90.0
26.3
17.9
85.8
74.5
85.0
31.6
20.7
MiniPPM-V
13.9
84.1
70.9
80.6
35.6
21.9
26.1
77.0
64.3
75.0
35.3
26.9
GPT-4o
16.7
87.3
75.7
87.8
30.3
28.2
25.6
82.4
73.6
80.0
40.2
32.0
Left
Right
Same
Reversed
Same
Reversed
Acc
humi x x04
glus 102
Acc
heni xx102
clas xx1
Acc
chumi xx12
elas x102
Acc
chmm x102
e 102
InstructBI
47.2
51.5
37.2
47.2
61.6
48.0
47.2
61.4
47.5
47.2
52.0
37.8
InstructBLIIP-130
47.2
54.2
43.4
47.2
57.0
44.9
47.2
58.1
45.6
47.2
52.5
41.6
mBLIP
47.2
59.8
52.4
47.2
64.2
53.5
65.7
$4.6
47.8
56.4
46.8
GLaMM
47.2
48.9
38.3
47.2
65.5
51.8
79.4
29.8
17.3
15.0
76.2
63.7
ALaVA-L.5-7B
47.2
25.3
11.9
47.2
83.4
70.0
47.2
26.0
13.0
47.2
80.9
68.5
LL5VA-LL.--3B
62.8
39.1
31.7
31.7
76.8
61.8
91.1
35.8
24.3
8.9
79.3
64.3
XComposer2
97.8
11.3
20.1
3.3
95.6
80.9
96.7
15.2
21.3
3.3
95.8
81.1
MiniCPM-V
86.1
27.7
19.7
9.4
88.
74.1
82.2
32.7
21.1
12.2
87.0
73.3
GPT-4o
94.4
20.4
24.3
11.1
92.6
80.8
94.4
19.0
25.1
11.1
92.8
80.8
Aggregated
Translated
Rotated
Reflected
Preferred Transform
Acc%
hemi xx14
gas
Acc%
hemi *x102
Acc
hemi xx10
x11
112
x102
InstructBLIPP771
47.2
54.7
47.2
58.0
44.2
$7.8
43,9
Significant
InstructBLIP-1333
47.2
57.1
44.0
47.2
54.8
42.3
47.2
55.5
43.0
Not Significant
mBLIP
50.6
61.8
52.3
47.5
60.9
50.5
47.5
62.1
52.1
Not Significant
GLaMM
55.3
39.2
61.0
47.9
55.3
45,2
33.0
Reflected
-l.VV--1.5-7B
47.2
47.3
34.5
47.2
61.9
49.0
47.2
33.7
20.7
Reflected
LLaVA-1.5-13B
61.0
54.0
43.4
34.9
57.9
43.2
63.2
37.6
25.7
Rellected
XComposer2
55.7
49.2
45.4
62.3
21.1
20.0
Reflected
MiniCPM-V
52.1
55.4
44.0
44.3
61.5
49.1
81.0
32.8
22.4
Reflected
GPT-4o
57.8
52.3
49.7
47.5
64.0
55.5
89.2
27.5
27.4
Reflected
Table 7: The full results for testing the preferred coordinate transformation mapping from the viewer to the relatum in the relative frame of reference.
20

---

Published as a conference paper at ICLR 2025
Back
Front Intrinsic
Egocentric
Intrinsic
Addressee
Egocentric
Addressee
Accy
s x102
Fxio
Accy
e x10e
Accy
Accy
Ex 10
x100
Accy,
x
*100
Accy,
x
xx1o
InstructB1.PP..
IPP-B
47.2
51.4
41.0
47.2
53.0
38.6
47.2
53.0
38.6
47.2
54.2
40.9
47.2
60.7
46.9
47.2
60.7
46.9
InstructBLIP-13B
47.2
43.5
32.9
47.2
48.9
34.4
47.2
48.9
34.4
47.2
66.5
52.5
47.2
61.1
48.5
47.2
61.1
48.5
BLOMMZ
52.8
62.1
52.2
52.8
63.9
53.2
52.8
63.9
53.2
52.8
56.4
45.3
52.8
55.5
44,6
52.8
55.5
44,6
GLaMM
47.2
45.6
31.9
47.2
51.0
38.8
47.2
51.0
38.8
47.2
37.9
24.8
47.2
69.6
$7.1
47.2
69.6
57.1
LLaVA-1.5-7B
49.2
41.6
28.0
47.5
60.3
49.1
47.5
60.3
49.1
48.6
43.2
30.0
48.6
52.9
40.2
48.6
52.9
40.2
LLaVA-1.5-13B
50.8
36.8
20.9
48.6
54.7
43.0
48.6
54.7
43.0
47.2
46.5
34.5
47.2
47.3
32.6
47.2
47.3
32.6
XComposer2
91.4
25.0
12.7
53.6
59.9
49.3
53.6
59.9
49.3
87.8
26.6
15.2
55.0
59.3
48.3
55.0
59.3
48.3
MiniCPM-V
66.4
46.5
34,2
60.8
51.3
40.7
60.8
51.3
40.7
57.5
45.0
35.5
50.8
64.6
53.4
50.8
64.6
53.4
GPT-4o
64.2
49.1
38.3
66.4
45.4
36.7
66.4
45.4
36.7
58.1
54.8
43.1
53.6
61.0
50.2
53.6
61.0
50.2
Left Intrinsic
Right Intrinsic
Egocentric
Addressee
Egocentric
Addressee
Accy,
x10e
Acey
x10e
Acpy
X10
Accg
x10
*x1ee
Accg
x1
1x0e
Accg
cot x
InstructBLIP-78
47.2
59.0
45.6
47.2
45.3
32.5
47.2
62.0
51.9
47.2
53.1
39.6
47.2
61.7
51.2
47.2
45.3
31.8
InstruciBLIP-
47.2
59.7
47.8
47.2
70.2
56.2
47.2
39.6
27.8
47.2
53.6
40.6
47.2
39.5
27.6
47.2
70.8
56.6
mB..P-BBL00OZ
52.8
58.2
47.8
52.8
59.7
47.6
52.8
58.4
48.1
52.8
57.7
45.4
52.8
60.6
48.4
52.8
53.8
42.4
GLaMM
75.8
22.3
11.7
46.4
62.0
51.1
52.5
62.3
51.1
60.8
41.8
27.5
44.7
68.5
57.4
53.1
58.7
48.7
LLaVA-1.5-7B
16.7
25.6
14.0
33,9
68.2
56.8
64.4
52.7
41.5
56.4
28.5
13.7
44.2
64.6
53.0
52.5
57.3
46.6
LLaVA-LL5-13B
81.7
23.7
13.4
42.2
65.0
53.5
57.2
58.5
47.4
86.7
26.8
14.3
47.8
64.0
53.6
52.2
59.9
49.3
XComposer2
95.0
18.8
18.8
45.6
70.5
61.2
54.4
64.0
53.7
96.1
17.1
16.5
47.8
68.1
58.4
52.2
64.6
54.5
MiniCPM-V
93.3
20.4
18.0
52.2
64.3
53.9
47.8
68
58.4
91.7
22.6
19.0
46.1
68.3
58
1
53.9
62.5
52.7
GPT-40
78.6
42.1
34.7
48.1
69.4
59.3
$1.9
65.8
$6.5
93.9
21.8
24.3
52.8
67.0
57.3
47.2
71.0
61.7
Aggregated
Egocentric
Intrinsic
Addressee
Preferred FoR
Acdy chami gane
Aceg
2014
Accy
X
InstruciBI.IP-11
47.2
54.4
41.8
47.2
55.2
42.3
47.2
55.2
42.3
Not Significant
InstructBLIP-
47.2
55.8
43.5
47.2
54.9
41.7
47.2
55.1
41.8
Not Significant
mBLIP-BLOOMZ
52.8
58.6
47.7
52.8
59.9
48.4
52.8
57.9
47.1
Not Significant
GLaMM
57.8
36.9
24.0
46.4
62.8
51.1
50.0
60.4
48.9
Egocentric Relative
1.577
57.7
34.7
21.4
43,5
61.5
49.8
53.3
55.8
44.4
Egocentric Relative
LLaVA-1.5-133B
66.6
33.5
20.8
46.5
57.7
45.7
5133
55.1
43.1
Egocentric Relative
XComposer2
92.6
21.9
15.8
50.5
64.4
54.3
53.8
61.9
$1.4
Egocentric Relative
MiniCPM-V
77.2
33.7
26.7
52.5
62.1
53.3
61.6
$1.3
Egocentric Relative
GPT-4o
73.7
42.0
35.1
55.2
60.7
50.9
54.8
60.8
51.3
Egocentric Relative
Table 8: The full results for testing the preferred frame of reference in VLMs.
Back Intrinsic
Front
Egacentric
Addressee
Egocentric
Intrinsic
Addressee
Accy,
chmmi x102
ctns
ige Accg e
hemi x102
10a
Accy E
hemi Xip
100
Accg
cow heme
x10a
Accg,
hemi cot
x10a
Accg e x
hrmi cons
xin
InstructBIIIP.I
47.2
56.4
45.1
47.2
54.3
41.2
47.2
56.0
42.8
47.2
56.2
42.0
47.2
56.4
43.6
47.2
$6.3
43.3
InstructBLIP-159
47.2
49.2
38,1
47.2
54.0
40.4
47.2
53.8
40.9
47.2
63.0
49.8
47.2
58.6
46.2
47.2
59.7
47.6
BBLIP-BLOOMZ
52.9
65.
55.3
52.4
64.7
54.8
50.8
66.3
57.1
52.6
66.2
56.3
52.1
63.8
52.8
53.1
67.1
58.9
GLaMM
47.2
46.2
32.7
47.2
54.8
42.4
47.2
62.6
49.9
47.2
40.4
25.3
47.2
55.1
41.6
47.2
51.0
38.3
LLaVA-1.5-7B
49.0
41.6
27.6
47.4
56.3
45.7
46.2
66.7
55.0
47.5
39.4
25.2
47.4
52.9
39.8
47.2
41.1
27.5
LLaVA-1.5-13B
47.2
38.3
22.4
47.2
53.2
41.1
47.2
52.1
39,9
47.2
48.8
36.8
47.2
54.7
41.2
47.2
41.9
26.8
XComposer2
65.4
40.6
26.0
52.2
57.9
47.0
54.0
58.5
47.5
86.9
27.0
17.1
52.1
58.9
47.8
53.1
$7.8
46.6
MiniCPM-V
55.7
48.6
36.0
46.9
57.7
45.9
53.8
47.
36.3
54.7
45.9
35.0
52.2
58.4
45.9
52.2
58.5
46.9
GPT-4o
69.0
41.3
28.7
59.7
50.0
37.7
56.4
48.7
36.0
58.6
52.5
40.3
52.1
57.4
45.1
48.3
60.0
46.9
Left Intrinsic
Right Intrinsic
Egocentric
Addressce
Egocentric
Addressee
Accy,
Jami 1x00
Acoy
Jhau K104
x00
Acpy
heni X10P
*1ne
Accy
Jhen X10*
xx0
Accg e
Jemi X18
110
Accg,
Jemi xII
Xx0
InstructBLIP-78
47.2
56.3
43.3
47.2
56.0
43.0
47.2
57.9
47.1
47.2
56,8
43.5
47.2
52.9
41.5
47.2
54.4
41.0
InstructBLIP-133
47.2
58.0
46.2
47.2
61.7
48.7
47.2
46.5
33,8
47.2
53.5
41.1
47.2
49.8
37.6
47
62.6
49.4
BBOOOZZ
51.4
65.6
55.4
46.4
67.0
56.4
47.2
64.6
54.8
50.7
65.3
54.4
48.2
63.5
52.8
47.2
62.9
52.3
GLaMM
47.2
29.6
16.9
47.2
57.7
45.8
47.2
53.7
41.5
47.2
34.3
18.3
47.2
58.7
47.1
47.2
53.2
41.6
LLaVA-1.5-7B
64.9
23.7
12.1
50.4
60.2
48.9
49.9
56.3
45.3
59.3
25.2
8.7
47.9
59,8
48.5
49.6
56.7
45.7
L.IVA--L55-33 13BB
47.2
29.2
18.3
47.2
59.5
47.0
47.2
53.5
41.2
64.7
32.1
17.9
47.4
61.6
50.7
48.5
58.6
47.8
XComposer2
95.6
18.4
16.4
49.7
64.8
54.5
54.0
62.4
51.3
94.4
19.2
15.7
49.9
64.9
54.8
51.8
64.3
53.8
MiniCPM-V
89.4
24.2
13.3
50.4
60.9
50.0
52
60.6
49.9
89.9
24.3
14.2
50.1
60.6
49.5
53.3
58.0
47.3
GPT-40
91.7
24.0
22.8
52.5
60.1
48.6
46.7
59.9
47.6
93.9
22.1
20.5
49.2
59.4
47.0
45.1
61.0
49.1
Table 9: The full results for benchmarking perspective-taking performance VLMs.
21

---

Published as a conference paper at ICLR 2025
Code Language
Intrinsic
Egocentric
Addressee
Code Language
Intrinsic
Egocentric
Addressee
Ref.
Rot.
Tran.
Ref.
Rot.
Tran.
Ref.
Rot. Tran.
Ref.
Rot.
Tran.
af Afrikaans
50.9
33.7 57.8
49.2
56.6
55.5
57.5
ku Kurdish
56.5
49.5 54.4
53.1
53.1
55.2
54,0
ak Akan
51.8
39.6
52.2
48.8
50.4
50.6
53.8
la Latin
$2.2
43.9 49.8
55.5
55.1
50.8
56.6
am Amharic
52.1
47.4
60.7
50.9
56.8
54.2
$7.6
Luxembourgish
54.7
35.6
58.6
53.0
59,9
Arabic
55.7
35.8 59.0
51.0
56.6
55.8
59.8
In Lingala
52.6
45.7
50.3 59.4
54.6
51.3
58.0
Assamese
51.6
40.8
55.3
51.6
48.8
52.8
56.0
Lao
55.8
40.1 55.0
53.7
54.7
55.7
55.5
Azerbuijani
49.8
41.9
56.2
51.6
50.7
52.4
55.5
It Lithuanian
54.4
35.5 58.3
51.6
57.4
56.5
59.1
Belarusian
54.4
39.7
61.7
46.5
51.1
51.9
58.9
Iv Latvian
55.8
35.5 57.7 53.8
57.9
58.7
58,8
bg Bulgarian
56.9
32.8 56.0
51.4
55.7
55.6
58.9
mai Maithili
52.7
39.8 55.1 49.9
51.2 50.6
56.5
bho Bhojpuri
51.5
42.8 58.3
47.5
52.3
51.4
56.5
mg Malagasy
47.1
37.1 52.2
48.1
53.1
50.9
53.5
bn Bengali
55.8
34.5
57.1 50.2
53.8
56.1
57.4
Maori
$2.0
36.6 58.5 47.6
52.0
51,9
58.1
bs Bosnian
55.1
35.2 58.5
49.6
54.3
53.1
59.4
mk Macedonian
54.8
37.0 59.1 49.5
56.5
56.0
58.1
Catalan
55.6
34.7
56.9
53.0
56.0
56.4
59.7
mn Mongolian
54.1
36.7 56.8
47.4
54.7
53,6
54.7
ceb Cebuano
52.9
40.0
52.7
54.9
55.3
49.5
60.3
mr Marathi
$2.9
34.5 55.0 48.3
51.7
$2.4
55.4
ckb Sorani
50.4
36.1 53.3
50.6
50.3
$2.0
56.3
ms Malay
54.2
33.1
50.5
55,4
57,9
Corsican
57.6
35.4 57.4
54.3
58.8
58.8
59.7
Maltese
53.4
37.5 56.1 49.2
50.8
53,8
55.5
Czech
56.4
35.7
58.2
52.4
57.0
58.5
58.0
my Myanmar
54.9
39.3 58.7
51.8
54.2
56.1
58,1
Welsh
55.1
36.7
59.5
48.7
54.9
54.8
58.5
ab Norwegian
55.1
34.7 57.0
52.1
58.1
57.6
57.3
da Danish
54.9
33.0 55.2
53.1
57.6
58.3
56.9
ne Nepali
53.1
39.4 58.4
47.3
52.9
54.1
54.0
de German
55.7
36.2 58.4
52.8
56.9
56.6
60.3
nl Dutch
51.7
34.5 56.3
48.3
53.5
51.6
57.8
el Greek
54.4
34.4 57.1
52.2
S7.1
57.5
57.7
nso Sepedi
53.8
42.6 51.0
57.1
46.3
54.2
53.7
English
50.9
35.8 57.3
53.7
58.8
51.3
58.8
Nyanja
53.7
34.5 56.6
48.0
54.4
52.8
56.7
co Esperanto
58.0
34.3 56.4
54.6
58.2
58.2
60.2
om Oromo
51.1
43.5 57.3 50.6
54.9
54.8
52.9
56.9
36.2 58.1
53.3
57.0
58.5
59.0
pl Polish
55.8
32.9 55.8
52.5
55.1
55.1
59.5
el Estunian
53.7
35.1 56.0
$1.7
55.0
54.8
58.5
Pashto
53.0
34.6 57.4
48.9
53.7
54.6
57.4
eu Basque
56.8
34.3 56.8
53.2
56.7
57.1
59.5
pt Portuguese
56.3
35.9 58.2
51.9
59.1
59,3
57,6
fa Persian
55.8
32.1 55.3
49,8
54.4
53.8
58.0
ro Romanian
57.1
34.8 57.0 53.8
58.2
58.6
59.1
Finnish
53.9
33.7 56.7
50.8
56.3
56.1
57.6
ru Russian
56.2
36.9 58.8
53.0
56.8
56.3
60.8
fil Filipino
50.9
31.1 54.1
49.2
$4.3
54.0
55.7
Kinyarwanda
53.2
35.2 56.7
48.9
54.4
54.1
57.2
fr French
58.0
35.2 57.4 53.7
58.6
58.5
59.4
sa Sanskrit
51.9
41.2 54.1 51.9
51.7
56.4
51.6
Frisian
53.9
38.2 58.9 49.6
53.4
53.2
59.2
sd Sindhi
$1.3
40.3 56.5
49.1
54.8
49.8
57,4
ga Irish
54.0
33.2 55.3 49.2
52.7
55.7
53.9
si Sinhala
52.4
38.4 54.6 48.6
$3.4
51.5
56.6
gd Scots Gaelic
53.9
35.4 58.5
49.6
54.7
55.8
58.1
Slovak
56.1
37.7 57.1 54.7
57.2
56.7
59.8
Galician
56.6
37.1 59.0
53.4
57.9
57.9
60.0
si Slovenian
55.8
36.5 59.3 49.5
53.9
54.3
58.9
gom Konkani
51.1
5 55.5
50.5
52.5
54.9
$1.8
Shona
56.0
34.7 56.0
55.6
58.5
Gujarati
52.6
36.6 54.2
50.9
55.5
55.3
53.8
Somali
53.7
34.3 56.4 48.2
50.0
51.6
58.4
ha Hausa
54.0
41.0 56.1
53.0
52.8
55.3
56.1
sq Albanian
53.6
35.1 56.4 49.0
52.6
50,3
60,1
haw Hawaiian
55.3
42.2 62.1
51.5
60.5
56.2
60.8
Sr Serbian
55.4
34.5 57.2 50.9
52.5
55.0
58.8
he Hehrew
56.5
36.4
58.8
52.5
57.1
56.5
60.3
st Sesotho
53.9
38.4 55.4 51.0
51.3
54.4
55.8
Hindi
52.5
37.8 56.6
49.1
$4.5
54.6
54.2
Sundanese
$1.3
36.7 55.0 50.0
53.7
50.4
57.7
ht Haitian Creole
56.1
36.0
58.3
53.6
58.4
58.2
59.6
SV Swedish
54.0
33.5 56.7 51.7
55.8
56.3
58.2
hu Hungarian
55.2
35.0
57.5
50.7
56.1
56.8
56.7
Swahili
55.3
34.2 56.8 52.4
57.2
56.5
58.4
hy Armenian
52.2
35.4 56.7
48.8
53.6
$2.5
57.2
ta Tamil
52.0
40.4 55.2 51.1
52.2
52.9
54.6
Indonesian
55.9
35.6
58.1
52.2
57.1
57.8
58.1
tg Tajik
55.7
36.7 57.7 49.7
55.4
56.6
56.3
ig Igho
54.5
33.8 56.7
47.4
53.6
53.2
55.3
th Thai
55.5
35.4 57.9 50.8
56.2
57.8
57.3
ilo Hlccano
50.8
46.7
58.9
48.9
$7.0
48.7
Turkmen tr Turkish
$2.3
45.3
$1.2
59.1
is Icelandic
55.9
34,2
57.0
52.2
56.5
58.0
57.3
55.3
33.6 56.3 50.8
56.2
57.0
56.3
it Italian
56.8
35.6
57.6
53.6
57.9
58.2
59.6
ts Tsonga
49.4
44.6 50.0 53.7
53.3
51.5
53.6
Japanese
54.7
34.5
56.9
50.4
54.4
55.9
57.3
uk Ukrainian
56.6
36.1 58.8
50.1
56.8
55.7
59.7
jv Javanese
53.5
35.3
57.7
$1.0
55.7
54.8
58.9
ur Urdu
52.3
34.6 56.7
49.7
54.0
55.1
57.0
ka Georgian
51.1
34,8
54.3
50.6
$2.0
54.0
55.3
Uzbek
$2.6
34.5 56.4
48.1
51.7
53.2
56.6
kk Kazakh
52.6
36.5
58.8
50.4
$4.0
56.1
56.9
vi Vietnamese
53.9
34.6 58.5
48.6
55.7
56.1
56.8
km Khmer
55.6
37.6
60.2
50.5
56.9
56.6
59.3
yi Yiddish
56.7
36.5 57.9
53.5
56.8
57.3
60.0
kn Kannada
52.3
40.8
53.2
49.5
49.6
51.9
53.4
7h Yoruba
$4.6
35.5
58.3
51.4
56.9
57.2
57.8
Korean
53.6
36.5
59.1
49.7
53.3
53.5
59.8
Chinese
55.6
35.9 57.7
53.1
55.3
56.5
60.2
kri Krio
58.3
36.2
57.1
51.2
56.1
53.2
60.2
Table 10: The full results for the cross-lingual and cross-cultural evaluation of the preferred frame of reference in VLMs.
22

---

Published as a conference paper at ICLR 2025
Left
Right
Front
Back
InstructB1 LP7B
maal
uxga
2.7
2.74
0.30
Instuct 13B
angho
Hight
zago
3.50
0.30
GLaMM MBL TPLIP LOOZZ77
mak
made
366
1.00
14.3
aigh
ando
xigl
ango
114
2.7
LLAVAL.57B
mak
made
2.00
1.30
22.
4.7m
2.3
LLAVAI5533
migh
xigl
anga
263
177
XComposer2
mah
1.0t
1.30
174
potdtss
8.75
4.20
0.36
MiniCPM
mak
male
aada
213
0.7-
GPT40
m0.
4.60
Figure 13: All prediction plots for each model on COMFORT-BAR using the camera perspective prompt (cam). The raw probability p(0) in gray, normalized probability (0) in black, and the reference probability Peos (0) of cam in red.
23

---

Published as a conference paper at ICLR 2025
Left
Right
Front
Back
Caf
Sy 314
Cor
InsrrucB LTP7B
6:6
-20
-206
-2.4
136
Syl 363
Cyl
Cg.
InstructBLIP LIP 13B
-0.
Ey.
Far
GLaMM MBLIPALOOMEA PPPLLOM27B
30
-20
136
Cyt 14.
Cyt
Syl
-054
0
-24
1.30
-256
0.00
-2.0
Fo. 5.9
Fg.
Fy.
LLAVA 1,578
200
-20
1.30
-200
300
-70
136
Eyc
Syc
Sy.
LLaVA 1.133
-254
9
30
-24
1.30
-25
0.00
-2.0
18
112
Aud 11A4
Ag:
XCompose22
-224
30
-20
1.30
-200
300
-2.0
15
Ky.
MiniCMM
-224
0
300
-00
0.00
-34
Jxc
Agc
Ag:
GPT40
1.10
300 -2.00 372
1.30
-250
300 0.00 -7.0
1x
Figure 14: All prediction plots for each model COMFORT-CAR using the camera perspective prompt (cam). The raw probability p(0) in gray, normalized probability (()) in black, and the reference probabilities Pess (0) of in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel, we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMF -CR with relatum facing right for front and behind relations.
24

---

Published as a conference paper at ICLR 2025
Left
Right
Front
Back
Caf d2
Ey M4
Gy au4
InsrrucBLI L1P7B
20
-20
-206
-24
136
215
Cel CAB
Cy:
Co.
InstructBLIP LIPI 13B
0.00
-16
12.
Cal 22
Fy.
Far
GLaMM TELIPALOOMZIR
30
-20
16
Cyc
Syl
-056
0
-24
1.30
-256
0.00
-14
65.
Zyl 0LA
Ag:
Eg: 56A
LLaVA 1,578
200
-20
1.30
-200
300
-70
136
Eye 47A
Ky:
Syl
LLaVA 1.133
-254
0
30
-20
1.30
-25
0.00
-2.0
16
Aer TTL8
Pyc
Ag:
XCompose22
0.50
-20
1.30
-300
300
-2.0
13
1.00
4gp 46.3
Kyc
Kg.
MiniCMM
-224
0
300
0.30
-00
0.00
-34
Pg: 43.5
Aue 473
Ay:
Ago
GPT40
300 -200
120 350 -250 12
Figure 15: All prediction plots for each model COMFORT-CAR using the addressee perspective prompt (add). The raw probability p(0) in gray, normalized probability p(0) in black, and the reference probabilities Pen. (0) of cam in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel. we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMF COMFORT-TO ORT-CAR with relatum facing right for front and behind relations.
25

---

Published as a conference paper at ICLR 2025
Left
Right
Front
Back
Cyl 423
Sy. 145
InsructBL L1P7B
20
-20
-256
-24
135
12
Cyl 222
Syl
Cg.
InstructBLn LIP 13B
1.1
-24
0.00
-14
Eal 26.1
Fy.
Far
GLaMM MBLIPALOOMEAI
-20
-206
300
15.
Cyl
Cyt CA..
Sg..
-056
0
30
-24
1.30
-256
0.00
-2.0
Ayl 147
Zyt. d9-7
Ag:
LLaVA 1,578
200
-20
1:39
-200
300
-200
136
13.4
Eyl 137
Syc
Syl
LLaVA 1.133
-254
9
-20
1.30
-25
0.00
-24
18
747
Zet TTLB
Ago
Ag:
XCompose22
-122
30
0.50
-20
1.39
-300
300
0.00
-200
1x
ry: 48.0
Syt 133
Tyc
Syc
MiniC
300
-00
-00
300
0.00
-24
Aug 40.3
Ay:
Ag:
GPT40
300 -200 322
1.00
300 0.00 -700
1x
Figure 16: All prediction plots for each model COMFORT-CAR using the relatum perspective prompt (rel). The raw probability p(0) in gray, normalized probability p(0) in black, and the reference probabilities Pass (0) of in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel, we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMF ORT-CAR with relatum facing right for front and behind relations.
26

---

Published as a conference paper at ICLR 2025
Left
Right
Front
Back
Gyr 433
Cal
S
InstrueB L1P7B
6:6
-054
20
-20
-25
-24
135
Cyl
Cyl
Syl
Co.
InstructBLn LIPI 13B
-24
145
Fy
Far
GLaMM MBLIPALOOMZII
30
-20
-20
136
2.
Cyt 273
Sy:
Sgl.
-056
0
0.30
-24
1.30
-25
0.00
-14
12.
Egt 17.4
Pac
Eg.
LLAVA 1,57B
300
-20
1:30
-200
300
-70
1x
Cyc 11A
Sy:
Sy.
LLaVA 1.33B
-254
0
30
0:50
-00
1.30
-25
300
0.00
-240
16
XCompose22
-222
0
300
-20
1.39
-300
300
0.00
-200
13
Sur 12.8
Sy.
MiniCMM
-204
9
300
-00
1.30
-24
300
0.00
-34
Agc
Ago
GPP40
1.10
300 -200 2.0
1.00
-200
300 0.00
-2.0
Figure 17: All prediction plots for each model COMFORT-CAR without perspective prompt (nop). The raw probability p00) in gray, normalized probability p()) in black, and the reference probabilities Peos(0) (0) of cam in red, add in orange, rel in blue. To avoid overlapping reference probabilities of add and rel, we use plots on COMFORT-CAR with relatum facing left for left and right relations and COMFORT-CAN with relatum facing right for front and behind relations.
27


Reviews:
[{"review_id": "7gBimuieyN", "reviewer": "Reviewer_QwFz", "summary": "The authors assess how VLMs represent space through the lens of “situated communication,” framing the meaning of spatial relations like “to the right of” under transformations, in different frames of reference, and others (situated communication).\n\nThey introduce the COMFORT dataset, tasks, and metrics to analyze spatial reasoning under these classes of transformations, and see which frame-of-reference preferences LMs have, and test whether these preferences are language-agnostic by extending their evaluation to multilingual settings. \n\nThe images are rendered to enable simple dynamic generation of test samples. Their language queries that test the relations likewise have a simple structure and are programmatically produced, enabling translation into other languages. Testing the kinds of relations is simple using this methodology and allows them to test for continua such as how a model rates the “in front of” relation by angle between the two in the plane relative to the camera.\n\n**Edit**: The authors did a great job adding some small case studies that **show** how their results generalize to other backgrounds and real images (although these are still pretty close to in-distribution to the test images). The use of a laptop rather than a ball in the real images in particular is a strong change. I have raised all the component scores to 4. **Although I would prefer an option to give a 9/10, since it isn't available I will raise from 8 to 10.**\n\nThese tests are then translated into metrics using the established concept of “acceptability regions”, which then allows them to score for correctness. Swapping object types, colors, etc in the same scene is easy using the synthetic pipeline and gives an evaluable sense of robustness.\n\nThey test a broad set of VLMs, including multilingual ones in 109 translated languages. Some models better represent egocentric vs object centric relations.\n\nOverall, they find that the VLMs do have some acceptable egocentric understanding capabilities but they have severe robustness issues. They find that English notions of frame of reference are held across most test languages, not reflecting the diversity of notions.", "questions": "Small grammar errors, eg 476: “Does multilingual VLMs..” -> “Do multilingual…” Maybe do a brief grammarly check?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Deep utilization of synthetic pipeline to really probe spatial perception from several angles.\n\nDemonstrations of anglewise judgements are compelling and surprising! (eg., figure 7)\n\nGreat to see the extra effort of testing \n\nConclusions and discussion are thoughtful and well supported.", "weaknesses": "~Sole reliance on synthetic data.~ A brief ablation over natural images would be nice to see to give a sense of how well these findings generalize. They added the case study, the results generalize (though the case study is still close to ID the synth data)!\n\n~Limited diversity of backgrounds. This is another form of variation that may have important implications for generalization.~ They added the case study, the findings generalize!\n\nThe multilingual experiments, while welcome, have limited time to breathe in the overall paper with just a few paragraphs and tables stuck in appendices. However, I think this is a great problem to have as the paper is densely packed with quality experimentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TCpnAWUyqs", "reviewer": "Reviewer_4ug1", "summary": "**Main contributions**: \nThe paper has two main contributions: \n1. It presents COMFORT, a framework for evaluating VLMs' understanding of spatial frame of reference expressions, which relate two objects to one another (e.g. \"Is the basketball *to the right of* the car?\"). Expressions are evaluated using a rendered image containing the two objects and which is used to query the VLM with (as a binary Yes/No question) for whether or not the relation holds for the image. \n\n2. It uses COMFORT to show that a set of contemporary VLMs have distinct preferences for frame of reference parameters. Specifically, the paper presents evidence for the preference of English language frame of reference parameters in VLMs. \n\n**Framework Structure**:  \nExpressions in COMFORT vary across two main conditions:\n1. Expressions which include two objects with a semantic \"front\" (e.g. such as the \"front\" of a person being the side with the person's face). In this scenario the spatial expression must be reasoned about in terms of an anchor frame of reference (i.e. either object A's, object B's, or the camera's frame). The selection of the frame of reference will determine which direction in the image corresponds to \"left of\", \"back of\" etc. \n\n2. Expressions which include objects without a semantic \"front\". In COMFORT, this is instantiated by scenarios with two balls. In such scenarios, a coordinate transformation convention must be assumed for determining what is \"right\", \"left\", \"back\", etc. Different languages have different standard conventions, and so this condition is set up to test this. \n\nPreferences are measured by evaluating the [Yes/No] probability for a given spatial expression in relation to a ground truth *region of acceptability*, a continuous region of rotation of the second object around the center object (see Figure 1c for reference) in which a spatial expression may hold true. This region of acceptability allows for not only for accuracy to be measured but also for the dynamics of the probability to be measured as functions of distance from the center of the region of acceptability -- intuitively, the score of \"right of\" should decrease smoothly as the object is rotated from 0 to 90 degrees from the center of the region. Specifically, the paper presents. \nThese measures probe the robustness of model scores (e.g. evaluating std across object variations unrelated to space like color or shape) as well as the consistency (are scores symmetric at both equivalent angles from the acceptability region's center? Do they change smoothly as angle is varied?). \n\n**Results**\nThe paper uses the region of acceptability based error measures to show that VLMs generally skew towards preferring egocentric frames of references (condition 1) and the reflected coordinate transformation convention. Additionally, the paper also shows that these preferences cannot be reliably changed when the prompt explicitly defines the frame of reference to be used, or when a language (other than English) with different known preferences is used to prompt the model.\n\n**Recommendation** \nThis is a solid paper which I believe presents a unique contribution that stands to better capacitate the community in vetting the spatial understanding of VLMs. I have noted a few suggestions below under Weaknesses to improve the presentation of the paper. With that said, I think it is already at a good quality and I recommend acceptance.", "questions": "**Suggestions**: \n* L212 \"the query is appended after four different perspective prompts...\": I'm assuming this meant to say \"after *one of* four different...\"? \n\n* I understand the authors may have been space limited (no pun intended), but I think adding a Conclusion section to reiterate contributions and takeaways (further than the Discussion section already does) could make the paper stronger. There's a large amount of content, so spoon-feeding the final takeaways to readers could aid clarity. \n\n* This is minor, but relatedly I think the paper could be a little clearer about what the intended usage of the framework is for readers. Are readers supposed to come away with an understanding that this will be a benchmark they can evaluate their own models on? I believe yes, but the wording of the paper at the beginning and end doesn't make this explicit. In the introduction, COMFORT is introduced as a \"framework\" and not a \"benchmark\", was this intentional? Secondly, the paper closes with only a discussion over the empirical findings in Section 5, without any explicit reiteration that it has introduced COMFORT. This omission at the end made it feel like the only point of the paper was to present a study of VLM spatial understanding -- this would be fine of course, but if I'm understanding correctly I think the paper would additionally want to explicitly market itself as offering COMFORT as a testbed for VLMs. It's subtle, but I think being explicit about this, especially at the end of the paper, could really help drive home the point the authors want to communicate.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Strong Backing in Theory**: The paper's framing and setup in sections 1 through 3 (which motivates the problem statement and framework design choices) is a joy to read. Motivating statements and scoping within existing work is very clear and compelling. \n\n**Conclusions Well Supported**: The experimental setup is quite thorough and shows clear trends which the authors use to justify their claims. A majority of models are clearly shown to have FoR preferences based on the metrics *cos* and *hem* error functions defined, with the BLIP family of VLMs being a notable exception. I particularly appreciated that the paper additionally presents a compelling explanation for the BLIP discrepancy by showing that these same models score poorly on an evaluation of object hallucinations. \n\n**Novelty**: In my understanding I believe that the presented framework is novel based on its continuous definition based on regions of acceptability. This seems to me to be an important distinction namely because these spatial relationships are known to lie within continuums and are not discrete. \n\n**Clarity**: The paper's writing is generally clear, and I found the organizational structure helpful for understanding the concepts introduced.", "weaknesses": "**Cross-lingual Experiment Presentation**: I think some further refinement/elaboration of the cross lingual experiments (Section 4.5) could help strengthen the paper. Although the paper does state upfront that it spends most of the content focused around English experiments, the treatment of the section still felt a little too short for the emphasis that is placed in the introduction for this being a core piece of the framework (e.g. the M in COMFORT stands for Multilingual). Specifically I think a couple of details could be expanded upon. \n\nFirst, the results in Figure 8 are a little ambiguously presented. Is the world map mapping the preferences of GPT-4o, or is it mapping ground truth human preferences? I believe the former, but the wording in both the figure and the paper makes it sound like these are ground truth preferences. \nSecond, for ground truth preferences, I'm wondering how these are obtained? -- I'm assuming this is pulled from the linguistics literature, but it is not explicitly stated and appears like a key detail to me. I would appreciate it if the authors could clarify both of these points. \n\nLastly, although the results presented clearly show a preference for the same reference frames as English, I think they could be strengthened by also showing a notion of what the expected preferences would be if the VLM were to be adhering to the conventions of the language being used. For example, I think this could be done by (a) adding a second map showing a visualization of ground truth human preferences, or (b) adding the bold/underline convention from the table in Figure 8 to Table 10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVCT78NLtI", "reviewer": "Reviewer_GnST", "summary": "This paper explores the spatial reasoning capabilities of vision-language models (VLMs) and their handling of spatial ambiguities. It introduces the COnsistent Multilingual Frame Of Reference Test (COMFORT) to systematically evaluate these models. The study reveals that while VLMs show some alignment with English conventions in resolving spatial ambiguities, they exhibit significant shortcomings: poor robustness and consistency, lack of flexibility to accommodate multiple frames of reference (FoRs), and failure to adhere to language-specific or culture-specific conventions in cross-lingual tests. The paper emphasizes the need for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning in VLMs.", "questions": "1. How do the findings impact the development of embodied AI systems and other applications like autonomous driving, robotics, or augmented reality?\n\n2. How could improved spatial reasoning in vision-language models (VLMs) enhance performance in multilingual and multicultural contexts?\n\n3. What are the key areas of future research to advance the spatial reasoning capabilities of VLMs, and what specific challenges or limitations need to be addressed?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper introduces the innovative COnsistent Multilingual Frame Of Reference Test (COMFORT), which systematically evaluates vision-language models (VLMs) across multiple languages and cultural contexts. This novel approach highlights the importance of considering linguistic and cultural variations in model performance. The research is thorough and methodical, assessing nine state-of-the-art VLMs with detailed analyses of their robustness, consistency, and flexibility. The paper is well-organized and clearly written, making complex concepts accessible. Its findings underscore the need for more robust and flexible models, providing a valuable framework for future research in vision-language modeling and spatial reasoning.", "weaknesses": "The scope of spatial relations is limited, focusing mainly on basic relations like front-back and left-right, while neglecting others such as near-far and above-below. The evaluation relies on synthetic 3D images, which may not fully capture real-world complexities, and lacks consideration for occlusion and varying camera angles. Additionally, the analysis could benefit from a more diverse linguistic and cultural context, as well as human annotations for the multilingual dataset. Addressing these weaknesses by expanding the scope of spatial relations, incorporating real-world scenarios, and improving the quality of the dataset with human annotations would provide a more comprehensive evaluation of VLMs' spatial reasoning capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ZO3bPfSC3", "reviewer": "Reviewer_Jhao", "summary": "This paper discusses the performance of VLMs in handling ambiguity in spatial expressions. The authors constructed a new benchmark -COMFORT to systematically test the spatial reasoning capabilities of VLMs, and conducted analysis on COMFORT.", "questions": "Please refer to the comments in the Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper delves into \"Do VLM represent space and how\". Specifically, it considers the concept of Frames of Reference (FoR) and constructs a framework for understanding how VLMs perceive spatial information, taking into account the ambiguities that FoR might bring. The COMFORT framework is systematic and comprehensive, and its task formulation is rigorous. The authors create challenging but realistic queries using synthetic 3D images and corresponding textual descriptions, and propose systematic metrics. From the perspective of benchmark construction, this is a systematic and rigorous work.\n\n2. The authors tested different VLMs and analyzed how they internally understand spatial representations. The testing methods are reliable.\n\n3. The authors extended the test cases to cover 109 languages across 170 regions, further testing how VLMs respond to ambiguities in spatial relationship descriptions under different languages.", "weaknesses": "As the author claimed, \"spatial expressions in situated communication can be ambiguous.\" Therefore, considering that humans also exhibit different understandings when dealing with similar issues of ambiguity, even if VLMs demonstrate different interpretations, it does not necessarily imply a clear superiority or inferiority. In other words, the evaluation metrics proposed in the paper are not absolute criteria, like egocentric is not naturally better than addressee-centered, but rather serve as perspectives for observing how VLMs understand spatial information. \n\nFrom this standpoint, I believe the author should include more analysis of how humans understand ambiguities in spatial relations (relevant information can be found in psychological and cognitive science research). This would help us better understand the differences between VLMs’ understanding of these ambiguities and the human perspective, and ultimately facilitate a better alignment of spatial reasoning in vision-language models with human cognition.\n\nHope to see further discussion on these points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L2jWm1JbhU", "reviewer": "Reviewer_EAzE", "summary": "- Investigates how vision-language models (VLMs) represent and reason about spatial relations, particularly when spatial language is ambiguous due to differing frames of reference (FoRs). It focuses on evaluating the robustness, consistency, and flexibility of VLMs in handling such ambiguities.\n\n- The authors introduce a new evaluation protocol called the COnsistent Multilingual Frame Of Reference Test (COMFORT), which systematically assesses VLMs' spatial reasoning capabilities. The evaluation includes tasks with synthetic 3D images and text descriptions of spatial relations, with tests conducted in 109 languages across 170 regions worldwide.\n\n- The study reveals that while VLMs show some alignment with English conventions for resolving spatial ambiguities, they struggle with: (A) Consistency and robustness in spatial reasoning, (B) Flexibility in adopting multiple FoRs, (C) Cross-linguistic and cross-cultural understanding of spatial relations, with English conventions often dominating over those of other languages.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike existing benchmarks, which typically focus on spatial language understanding within a single frame of reference or specific cultural context, this work broadens the scope by emphasizing ambiguity in spatial reasoning due to different Framework of References and cross-linguistic/cross-cultural diversity.\n\n- The paper is well-structured, with clear explanations of key concepts such as frames of reference and spatial reasoning.\n\nMore importantly, \n\n- The paper’s contributions are highly significant in the context of both vision-language research and broader discussions on AI alignment with human cognition. By focusing on spatial reasoning, a core component of human cognition, this work addresses a critical gap in the evaluation of VLMs.\n\n- The cross-lingual evaluation across diverse cultural backgrounds provides valuable insights into the limitations of current VLMs in handling non-English spatial conventions. This has implications for the development of more globally applicable models, as the dominance of English-centric FoRs could hinder the usability of VLMs in non-English contexts.", "weaknesses": "- While the introduction of the COMFORT benchmark is a novel contribution, the idea of testing spatial reasoning in vision-language models is not entirely new. Several previous benchmarks, such as those mentioned in the paper, have evaluated spatial reasoning capabilities in VLMs. The novelty of this paper lies primarily in its focus on the frame of reference ambiguities, but the paper does not sufficiently clarify how this focus significantly advances beyond the existing benchmarks. But I agree that the frame of reference is a rather novel take on the same topic.\n\n- Although the paper provides valuable insights into how English conventions dominate spatial reasoning in multilingual models, the discussion around cultural and linguistic biases remains relatively high-level.\n\n- The paper identifies that most models tend to default to egocentric relative frames of reference, which aligns with English spatial conventions. However, it does not deeply investigate whether this bias toward egocentrism is inherently problematic across all contexts or if there are scenarios where this behavior is acceptable or even preferable. \n\n+ (not required) It would be beneficial to include a dedicated section discussing the practical relevance of the findings in LLM applications. Is there an area the the findings of this paper can seriously affect?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)