Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes
Authors: Mohsen Gholami
Ahmad Rezaei
Zhou Weimin
Sitong Mao
Shunbo Zhou
Yong Zhang
Mohammad Akbari
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Model
Spatial Reasoning
Multiview Images
Primary area: datasets and benchmarks

Abstract: Understanding 3D spatial relationships remains a major limitation of current Vision-Language Models (VLMs). Prior work has addressed this issue by creating spatial question-answering (QA) datasets based on single images or indoor videos. However, real-world embodied AI agents—such as robots and self-driving cars—typically rely on ego-centric, multi-view observations. To this end, we introduce Ego3D-Bench, a new benchmark designed to evaluate the spatial reasoning abilities of VLMs using ego-centric, multi-view outdoor data. Ego3D-Bench comprises over 8,600 QA pairs, created with significant involvement from human annotators to ensure quality and diversity. We benchmark 16 SOTA VLMs, including GPT-4o, Gemini1.5-Pro, InternVL3, and Qwen2.5-VL. Our results reveal a notable performance gap between human level scores and VLM performance, highlighting that current VLMs still fall short of human level spatial understanding (SU). To bridge this gap, we propose Ego3D-VLM, a post-training framework that enhances 3D spatial reasoning of VLMs. Ego3D-VLM generates cognitive map based on estimated global 3D coordinates, resulting in 12% and 56% average improvements on multi-choice QA and absolute distance estimation, respectively. Ego3D-VLM can be integrated with any existing VLM. Together, Ego3D-Bench and Ego3D-VLM offer valuable tools for advancing toward human level SU in real-world, multi-view environments. Code is available in the supplementary materials.

Published as a conference paper at ICLR 2026
SPATIAL REASONING WITH VISION-LANGUAGE MOD- ELS IN EGO-CENTRIC MULTI-VIEW SCENES
Mohsen Gholami', Ahmad Rezaei' Zhou Weimin Sitong Mao? Shunbo Zhou²
Yong Zhang't Mohammad Akbari
Huawei Technologies Canada, 2 Huawei Cloud
Project Page
Code
Ego3D-Berd BeeD-BBnchh
Ego-Cantric
Object-Gentric
Egu30-Gent
Absolute Distance:
Human Level
objects
views?
Travel Time:
is the
Accuracy (86)
to reach specific location?
Relative Distance:
Which object is closer?
Motion Reasoning: What If we move north?
Localization: Where specific object?
Chance Level
10
20
30
40
50
60
70
50
Parometers
Ower ENV.
Fiamila
Orid
CFF-4o Va
Eyc30 M.Maarnn
Ege sar
Figure 1: RgocD-Rench is 3D spatial benchmark for VLMS using ego-centric multi-view im- ages. It spans ego- and object-centric perspectives across 5 categories. A significant gap exists between human and VLM performance; our method, VLM, consistently narrows this gap.
ABSTRACT
Understanding 3D spatial relationships remains a major limitation of current Vision-Language Models (VLMs). Prior work has addressed this issue by ere- ating spatial question-answering (QA) datasets based on single images or in- door videos. However, real-world embodied AI agents- srgents-such such robots and self-driving cars-typically rely on ego-centric, multi-view observations. To this end, introduce Rga3D-Rench,
benchmark designed to evaluate the spatial reasoning abilities of VLMs using ego-centric, multi-view outdoor data. Ege3D- BeDD-BBechhccccc comprises over 8,600 QA pairs, created with significant involve- ment from human annotators to ensure quality and diversity. We benchmark 16 SOTA VLMs, including GPT-4o, Gemini 1.5PPr, InterVLL,, and Qwen2.5-VL. Our results reveal notable performance gap between human level scores and VLM performance, highlighting that current VLMs still fall short of human level spatial understanding. To bridge this gap, we propose Ego3D- VLMM, training-free framework that enhances 3D spatial reasoning of VLMs. Ego3D- VLM generates cognitive map based estimated global 3D coordinates, resulting in 12% and 56% average improvements on multi-choice QA and absolute distance estimation, respectively. Ego3D-VLM can be integrated with any existing VLM. Together, Ego3D- -BDD-Benchhhccc and EgL3D-VLM offer valuable tools for advancing toward hu- man level spatial understanding in real-world, multi-view environments. Code is available in the supplementary materials.
"Coneeponding Author: mohsen.gholanii huawei.com CCo-Leads

---

Published as a conference paper at ICLR 2026
INTRODUCTION
3D spatial understanding is critical capability for embodied Al agents operating in the real world (Chaplot et al.) 202T Wang et al., 2024b). This includes perceiving the location of surrounding objects, estimating their distances, and reasoning about their motion. VL.Ms have recently emerged as powerful tools to integrate visual perception and language reasoning, making them promising components for building intelligent embodied AI systems (Ma et al. [024b;] Li et al. 2024b). Thus, enhancing and evaluating the spatial understanding of VLMs has become an incressngll important
research direction (Team 2025 Cheng et al.
2024| Chen et al. |024 /Zhoo et et [202 Zhang et al.,
2025b
Wang et al. 2024a Wu et al.
2024b
Song et al. 2025 Ray et al. (025).
Recent benchmarks on spatial understanding focus mainly spatial reasoning from single images (Cheng et al.] 2024] Chen et al. 2024] Liao et al.] (0244) or videos captured in static indoor envi- ronments Yang et al., [024] Daxberger al. (025). In such cases, spatial understanding is framed around passive observations, where a single camera moves in room to create video, and the model should infer spatial relationships in relatively static scene. This setup differs fundamentally from the perceptual experience of real-world embodied agents such as self- self-driving cars or mobile robots. These agents rely on ego-centric multi-view observations (Ma et al.) 2024b), provided by multiple cameras simultaneously capturing front, side, and rear views of their surroundings. These views are not interchangeable or purely visual; they carry explicit spatial semantics tied to the agent's frame of reference. E.g. "left" & "right" refer to fixed directions relative to the agent's 's body, and must be interpreted consistently over time as the agent moves through dynamic environments, This distinction is crucial: while prior video-based datasets may offer multiple viewpoints, they do not reflect the structured, directional, and temporally evolving nature of ego-centric multi-view inputs. Also, existing benchmarks do not evaluate VLMs' reasoning ability across these spatially grounded perspectives in dynamic, real-world scenes.
This gap motivates the need for new benchmarks that better align with the spatial reasoning demands of embodied agents. To this end, we introduce Fgo3D-Bench, a benchmark of 8.6K QA pairs carefully curated from the validation set of 3 public datasets: NuScenes (Caesar et al. 2020), Waymo Open Dataset (Sun et al.] 2020), and Argoverse (Chang et al. 2019). Human annotators played a central role in both the dataset construction and the rigorous quality review process to ensure the reliability of the benchmark. We focused specifically on ego-centric multi-view tasks, rather than building general- purpose benchmark, to complement existing monocular spatial benchmarks (e.g., VSI-Bench (Yang et al. 2024)). Thus, we excluded questions that can be answered based on each view independently across multiple images (e.g., counting objects) or general knowledge of LLMs (e.g.. size of well- objects). 2go3D-BBnnch extends existing multi-view benchmarks (Yang) ct al. 2024 Ych et al. 2025) to ego-centric scenario that is specialized for embodicd AL.
We evaluated 16 SOTA VLMs, including generalist and 3D spatial ones Ego Sggo3--Bench Bench, reveal- ing a significant gap between human performance and current VLMs. We hypothesize that a key limitation lies in the inability of VLMs to construct a coherent world model from multi-view images. In contrast, humans naturally integrate visual info from their left, right, and front views into a uni- fied spatial representation, enabling real-time reasoning and navigation. Prior work has attempted to bridge this gap by first generating 3D point-cloud [Team 2025 Huang et al. 2024 Hong et al.) 2023b Zhang et al. 2024a| Hong et al. 2023a) or rendering a bird-eye-view (BEV) image of the scene (Qi et al. 2024). Although point-clouds and BEV images offer rich spatial information, they are challenging to reconstruct dynamic environments, struggle with sparse multi-view inputs, and significantly increase inference time et al. 2024).
To this end, we propose, Ego3D- VLM, a training free method that improves 3D spatial understand- ing of VL.Ms. The main idea of Ego3D- -VMM is to create a textual cognitive map of the surrounding. The textual cognitive map defines a coordinate system center on the ego and locates important object (i.e.. those referred to in the input prompt) in 3D coordinate space. Unlike point-clouds and BEV image methods (Team 2025; Qi et al., 2024), our cognitive map only focuses on referred objects, making the number of input tokens significantly smaller and enabling efficient reasoning.
Given multi-vien images as input, we first use referring expression comprehension (REC) models to find the 2D location of referred expressions in pixel space. We also use metric depth estimator to estimate the depth values. We then convert the 2D points to 3D points in camera coordinate space and transform 3D points from all views the global coordinate space (i.e., front camera
2

---

Published as a conference paper at ICLR 2026
Dataset Collections 3D annotations
Filtering
Captioning
Taxil Free
objects
find
Arrisaatoos write castions forthe objects
del
Pai city tAS driiding the
MA
Source File
restew challenging categgaries
Question Generation Templates
north, does it closon to the pedestriaan with black bag
(oh51,, captton, 12 annotacion on,
Fruee
DAST
comero viei)
Answer Generation by 3D Annotations
[dbj2, caption, s0 annotacion,
2-]
and Rule-based Functions
pmototion, canera view),
Frame [(obs), cxption, 3>
DUl (8.10)
Figure 2: Overview of our Ego3D-Bench creation pipeline. Human annotators played a key role in the process. Right: the distribution of the samples. Ego.: Ego-centric, Obj.: Object-centric.
coordinate space). We define a cognitive map generator function that returns a textual cognitive map given 3D coordinates. The textual cognitive map defines coordinate space for the VLM and organizes detected objects (expressions) based on the view-point. The textual cognitive map consistently improves SOTA VLMs on spatial reasoning. The major contributions of our work are:
We introduce 3D-ehch,, an ego-centric multi-view benchmark to evaluate 3D spatial un- derstanding of VLMs.
We propose Ego3D- VLM, plug-and-play training-free framework to enhance 3D spatial under- standing of VLMs, especially in ego-centric multi-view scenarios.
Through extensive experiments, demonstrate that Ego3D-VVM significantly improves SOTA generalist as well as 3D spatial VLMs in 3D reasoning.
RELATED WORK
3D Spatial Benchmarks/Datasee for VLMs. We categorize prior datasets and benchmarks into
three groups based on the input data type.
(1) (Liu et al.
2023;
Kamath et al. 2023 Liao et al.)
2024 Song al.] 2025 Zhang et 20256 Cheng al.
2024;
Du et 2024) focus on single-
Li et al. 2025 Yang et al. 2025 Ray al., 2025) focus on videos captured from indoor statie scenes. As noted, this setup differs from the perceptual experience of real-world embodied agents, which involves ego-centric multi-view images. (3) All-Angle Ych et al. (2025) is the first multi-view benchmark for VLMs. However, in their setup, multiple cameras observe scene from different
view images. (2) Yang et al. 2024 Daxberger et al. 2025
to surveillance motion capture systems. In contrast, Ego3D-Bench is the first ego-centric multi-view benchmark for VLMs created from dynamic outdoor scenes.
3D Spatial VLMs. These models-alss referred to as 3D-LLMS or 3D-MLLMs-aing to perform tasks such 3D grounding, spatial reasoning, depth estimation, and distance measurement. We categorize existing approaches into two main groups: (1) models that take point-clouds as input or reconstruct them from multi-view images, and (2) models that operate directly on image data.
Hong et al. 2023b a Huang et al. 2024 Team 2025 Zhang et al. 2025a 2024a) fall in the first group. While point-cloud representations ofter rich spatial information, they are difficult to reconstruct in dynamic scenes, often struggle with sparse -ilt data, and significantly increase inference time by over 10x. The 2nd group includes VL-3A (Zhu et al. 2024). Video-3D LLM (Zeng et al. 2024), GPT4Scene al. 20244). MM-Spatial (Daxberger et al. (025). SpatialVI.M (Cen et al. 2024), SpatialRGPT (Cheng et al. 2024), and SpatialPIN (Ma al. 2024a). Our work
falls in this image-based category.
LL-VA and Video-3D LLM use depth maps and camera poses for 3D positional encoding. GPT4Scene leverages BEV to address 3D queries. However, these models are primarily trained on indoor, static scenes and limited in handling quantitative spatial relationships such as object dis- tances. SpatialVLM addresses this limitation by introducing synthetic data generation pipeline, enabling more robust spatial reasoning (remyxai 2024p. SpatialRGPT enhances input representa- tion using region proposals alongside the original image. MM-Spatial proposes VLM that supports
3

---

Published as a conference paper at ICLR 2026
8 witch cleser
object-Cmttic
Trauel
& usich ore is cinser gr cer, excovator?
Egp-Centric teletive Distance:
track In the front view, yellow excrvotor
the black endan
direetty towere
tlack Moyele lorted street sign In the
location of cltven oatan drteding next cenent stoee
the Front vinu with speed ave?
A. Sliver weden B. villow socevator
vellov secavator Black bicycle locked
street pole
Hore than secondls
aliis rrom peroeeciive aliver sedan
Localization:
S tgo
direection If the ego can roves 3 netere east, does st gtt
to
fromi shure yallow uecevalor loaaeet b. seart eace Left eiglt
steser to black endin car parked next
steycte
ip Object-Centric dbsolate Distance:
A. Yea No
steycle bast left vie from silver fuue tanter track in traffit in the from vinut 2t, extars
Notion
S The feart the morth diractionn If the silver sadien draving rect to comnt siser viee movess eeters north, beside bicycle the bock left vine?
Ego Cestric Absolute Distance:
trucs
@
Hos
bloycle parked next
K. Yes No
Epo-Centric
Oeject-Centric
8.3 metera 0:34 mebers C:22 meters 0.32 weters
14.33 exters
Figure 3: Ego- and abject-centir samples from each category in Ego Ago3D-Bench
Chain-of- Thought spatial reasoning involving 2D grounding and depth estimation, and can also leverage depth input via tool-use. Different from SpatialRGPT. SpatialVLM, and MM-Spatial, proposed Ego3D- VLV a training-free method, can be applied to any existing VLM, and enhances spatial understanding of VLMs and outperforms prior works ego-centric multi-view reasoning.
3 EGO3D-BENCH
Bgo3D--Bench is designed to quantitatively evaluate 3D spatial understanding of VLMs from multi-view outdoor images. A major distinction between Rgench-Bench and previous works is its focus ego-centric multi-view data-an approach particularly relevant for applications in self-driving and robotics. Ego3D-Bench contains over 8.6K QA pairs in 5 distinct categories. This benchmark is constructed from the validation sets of prominent outdoor multi-view datasets:
nuScenes (Caesar et al. 2020). Waymo Open Dataset (Sun et al. 2020), and Argoverse 1 (Chang al. 2019). featuring 6, and distinct camera views, respectively (Figure These datasets cover diverse outdoor environments, including urban areas, highways, and rural regions. Ego3D-Bes BBecch leverages the multi-view nature of these datasets formulate questions that require fine-grained visual comprehension across different viewpoints as well as reasoning over 3D spatial relationships. In this section, we describe the detailed process of constructing the questions of Ego 3D-Bench:
3.1 BENCHMARK CONSTRUCTION
Outdoor datasets, unlike their indoor counterparts, often feature multiple instances of the same object within each scene. While indoor environments typically contain unique items-such single oven or television per room-outdoo scenes commonly include numerous similar objects, like multiple cars or pedestrians. This makes it challenging to uniquely reference target object in the scene. Thus, annotators begin by carefully reviewing each scene to identify unique objects (called "Filtering" in Fi They then compose concise captions describing these objects (called "Captioning" in Figure 2. The descriptions are designed to be short, yet discrimina- tive such that each object can be uniquely identified. Furthermore, the ground-truth 3D annotations (bounding boxes) are collected from the source datasets. Object captions, 3D annotations, and the camera view from which the object is visible are used to construct source file for QA creation.
Creation of Source Files:
Each question category follows predefined template with placeholders for object names, such How far <objl>in <vrew/>>rrm <obj2>>n <view>>> ? (see Appendix for all question templates). Each question is constructed by placing the generated object annotation and camera views from the source file in the question template. To generate the answers, rule-based functions that leverage ground-trume 3D annotations. For challenging categories, such as motion reasoning, final human review is conducted ensure the accuracy of the QA pairs.
Creation of QAs:
4

---

Published as a conference paper at ICLR 2026
How far is the black sedan parked next blcycle the back left view silver fuel tanker truck in traffic view?
Pibhala
aßenāueT Lrrfe
Palsbellz...
Referring *
Cognitive Map (C)
Black audar is loca
Expression
coordinaces 1:28
In the
Comprehension
Cognitive Map
Frons VIeN, the Bllenen tunal
Project
Pana Pan
Function
Matric Depth Estimation
(Foo))
NO use Euc Edcad distarce Forula
Therefore, distence between
bicycle bast Left
Model
froe Blloor FuR tarter truckllo
trafficc Inoet xiee
apsroxlssttly
Vision
26.02 </arswer>
Figure 4: An overview of Ego3D-VLLM, our training-free 3D spatial understanding framework.
3.2 BENCHMARK DETAILS
Figure 3 Jillustrates different question categories in Rgo3D-Bench. We emphasize on questions that require understanding the 3D world by utilizing information from multiple views. Thus, we exclude questions that can be answered using a single-view or by analyzing each view independently (c.g.. counting the number of objects), as they do contribute multi-view spatial reasoning. We define question from the perspective of both ego and objects in the scene. To clearly indicate the perspective of each question, we categorize them into ego-centric or object-centric. the following, we describe the five tasks (each composed of ego-centric and object-centric types) in our benchmark.
(1) Absolute Distance Measurement. This category asks the VLM to estimate the absolute metric distance between the ego and another object in the scene or between two different objects from different views. This category is designed two forms of multi-choice QA and absolute meter.
(2) Relative Distance Measurement. In this task, the VLM is asked to determine which of two objects closer-either to the ego or to a specific object, designed the form of two-choice QA.
(3) Localization. This category is only object-centric and assesses the VLM's ability to localize objects within scene. Specifically, the model is asked to infer the location of object-1 from the perspective of object-2 in the form of multi-choice QA.
(4) Motion Reasoning. This category defines a coordinate system using cardinal directions and asks the VLM that if the ego object-1 moves in direction, whether not it gets closer to or farther away from object-2. Answering this yes/no question requires mapping the spatial relationship between the objects and how the distances would change if one object moves in a specific direction.
(5) Travel Time. Given a specific motion speed, this category asks (via multi-choice QA) to estimate the required time move from the location of either the ego object-1 to the location of object-2.
Evaluation Metrics. We design most of the questions as multi-choice QA and use accuracy as the evaluation metric. We also have two absolute distance estimation for which we use root mean squared error (RMSE) in meters as the evaluation metric.
4 TRAINING-FREE 3D SPATIAL UNDERSTANDING Ego3D-VLM
Figure shows an overview of our proposed framework, Ego3D-VLM VLM. Given a set of multi-view images and natural language prompt,
REC model detect all objects mentioned in the prompt. For each camera view E {1, 21.2...., V}. the REC model returns set of detected objects:
( = (100 .....)) N(
where ((²) 2 denotes the 2D bounding box coordinates for the i-th object in view v, and (()) is the corresponding referring expression match. We compute the 2D pixel-space center of each bounding box T E 2R2 to create list of 2D centers of the objects in the pixel space.
5

---

Published as a conference paper at ICLR 2026
Questiin:
The front view corresponds to the nurth the
Cognitive
Cognitive Map
direction. BM hatchoack harrler
go Car at 30 location [x, y,
while
E: ]-18, 8, B]. Positive x reans right
objects resain stationary, pedestrion red shirt crossing the street the front right vieu?
get
means left to the ago car. Positive neans in front of the
and negative
Barrier
Pediestrim
ego can and negative reans bohind
the
Answer:
Front Viexi detected the
Left delected
0.,
Front Right Rotected
pedestri an at 30 locet lon (31.) 3
(Right
Figure 5: Example question with associated textual and visual cognitive maps.
Camera Coordinate Transformation. To obtain 3D spatial info, we use metric depth estimator predict a dense depth map ( #R for each view v. For each detected object center u = (z,, y4) we extract the corresponding depth value dee) = D(*) (244, U:). We then project each center point into the 3D camera coordinate system using the camera intrinsics K(w) E R3x3,
2i ya
P cam,+ = -d"" (H((
-1
E R
Next, transform the 3D point from the local to the global coordinate system, i.e., defined the front camera view point coordinate system. This replicates the human perception mechanism given multi-view cameras by using the front view as the reference building the 3D world based on that. Using the rotation matrix ((() R333 and translation vector T(t) E R233 have:
R((e) 0
T(')
(v) P Peam
(e) P =
E R
(3)
1
This process effectively simulates human spatial perception by leveraging multi-view images to construct a unified 3D representation of the scene.
Relational Scaling. Humans estimate object sizes using known references e.g., knowing person (w)
1.7 m tall helps infer the size nearby objects. Inspired by this, we scale 3D points Pglobal, us- ing familiar object categories (c.g., sedans, humans, bikes) identified in a few representative frames across all camera views. We compute the average observed height hest and scale all 3D points by heas where hes is the canonical common sense height (e.g., 1.7 m for humans). This yields P s : producing physically plausible scales without ground-trute depth.
(e)
Creating a Cognitive Map. We define cognitive map generator function Fang, which takes as input the set 3D global coordinates and corresponding referring expressions for detected objects across all camera views. Specifically, for each object i detected from view v, denote its global 3D position as p P and its matched referring expression as c cognitive map C. defined as:
(v) The function outputs a textual
C= ((ppeole (0)
(4)
Feng constructs an ego-centric world model centered the agent. It integrates multi-view detections and links each referred object to its spatial position and originating viewpoint. The resulting cogni- tive map captures both linguistic references and spatial semantics form eaabeennlln grounded reasoning and situational awareness. Figure[Sillustrate iilluttrates a sample prompt, the corresponding multi-view images, and the generated cognitive maps.
compact, human- interprraree
Given VLM > that answers queries by visual-textual contexts, it takes as input the cognitive map C, a set of multi-view images I = {1(²) Jo, and natural language query to return an answer a:
ICC
(5)
The cognitive map C' provides structured spatial grounding, while I supplies rich visual cues such as appearance, color, and fine- -rainer context-noo captured C. Together, they guide the VLM in interpreting and answering the query.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison results of generalist VLMM vs. EgL Rgo3D-Berm
Accuracy (%)
RMSEL
Ego Dist.
Obj.
Ego Mot.
Obj.
Travel
Ego Obj.
Ego
Obj.
Model
Dist.
Loc.
Mot.
Time
Rel.
Rel.
Avg.t
Dist.
Dist.
Avg.l
Human Level
78.2
57.1
100
100
95.3
72.7
85.0
94.1
85.3
:
Chance Level
25.8
25.5
24.3
49.7
50.6
24.0
49.9
50.1
37.5
Closed-source Models
GPT-4o
51.4
38.4
47.7
86.9
71.1
35.8
65.9
84.2
56.7
8.5
29.5
19.2
Ego3D-VLL VLM GPT-40
76.3
58.9
57.3
87.3
89.7
59.9
70.6
85.3
73.2
6.3
8.4
7.4
Gemini-1.5
51.3
35.8
49.4
88.4
70.6
37.7
55.5
71.4
57.5
10.7
28.6
19.6
Ego3D- EgMM Gemini-1.5-Pro
65.0
62.0
67.2
93.9
91.4
58.4
66.2
80.6
73.1
6.6
7.8
7.2
Qwen2.5 Family
Owen2.5-3B
21.5
29.4
28.8
50.3
41.9
30.9
54.1
56.1
39.1
30.5
33.7
32.1
Ego3D-VL.. Quen2.5-38
35.0
30.0
29.8
52.4
56.4
29.6
60.1
62.0
44.4
12.7
12.5
12.6
Qwen2.5-7B
32.7
31.5
30.5
45.9
44.0
34.5
43.2
66.5
41.1
25.1
35.5
30.3
BVMM VLM Qwen2.5-7B
59.4
54.5
33.1
62.3
58.2
49.4
50.5
66.9
54.3
8.1
10.9
9.5
awvn2.553225
45.4
40.7
49.6
75.6
74.1
40.1
54.0
79.0
57.3
21.2
10.4
15.8
Ego3D-VLL. Qwen2.5-32B
63.7
62.6
54.5
87.7
86.2
40,8
62.6
72.0
65.5
11.6
15.9
13.7
Dwwn22.5-7BB
42.4
38.6
54.8
86.8
68.9
38.5
53.3
80.5
58.0
10.3
22.2
16.2
B-o3D- VLM Quan2.5-72R
62.1
61.4
58.2
94.0
84.5
56.0
63.1
76.6
69.5
6.8
8.3
7.5
InternVL3 Family
25.8
28.7
29.8
54.1
54.8
36.1
49.9
65.2
43.1
15.2
39.1
27,2
Ego3D- Ego3D-VLM VLM LnterVVL3-SB
65.4
56.1
37.0
73.0
71.4
49.0
63.5
66.0
60.1
6.8
9.0
8.0
VTter
46.0
35.6
35.9
63.2
65.9
41.6
55.5
70.1
51.7
10.6
24.5
17.6
B- VLM LmcmnVL3-148
70.3
60.9
50.5
79.8
83.1
50.2
63.0
70.7
66.1
6.6
8.8
7.7
InternVV..2 3-33-8BB
35.4
31.0
39.4
66.6
64.9
38.0
61.0
77.3
51.7
51.7
8.6
42.2
25.4
Ego3D- VLM InterVV1.-338B
55.1
64.5
53.4
87.2
88.9
51.9
66.6
76.5
68.0
8.2
8.7
8.5
InternVL3-78B
54.6
48.4
50.3
77.7
70.0
44.8
57.0
76.6
59.9
12.0
15.5
13,8
Ego3D-VLM IntenVVL3-78B
68.3
62.7
62.9
91.6
89.2
55.1
66.3
78.2
71.8
6.8
8.1
7.4
Ovis2 Family
Ovis2-4B
29.8
28.9
18.4
47.5
48.1
36.9
54.2
70.3
41.8
17.1
29.5
23.3
Ego3D-VLAN VLM Ovi2-1BB
62.1
46.9
20.1
49.1
51.9
33.3
62.0
60.4
48.2
6.5
10.4
8.5
Ovis2-8B
25.6
28.6
31.1
45.3
51.4
31.4
50.7
68.2
41.5
11.5
30.2
20.8
Ego3D-VILM Ovis2-8B
64.9
54.9
33.5
57.2
57.1
46.1
65.1
71.0
56.2
6.0
9.5
7.8
Ovis2-16B
41.7
36.5
41.1
50.5
52.5
27.9
50.9
74.9
47.0
10.8
16.6
13.7
Ego3D-VIA. VLM Ov12-1688
63.4
58.1
43.7
53.9
73.4
51.1
61.0
82.9
60.9
6.6
8.3
7.4
5 EVALUATION ON EGO3D-BENCH
In this section, we present a comprehensive evaluation of VLMs on Ego3D-Bench. We orga- nize our experiments into 4 parts by benchmarking (1) generalist VLMs (i.e., models trained for general vision-language tasks), (2) 3D-VLMs (i.e., models trained specifically for 3D SU), (3) VLM-Depu-REC (i.e., generalist VLMs augmented with depth and REC tools), and (4) ablation stud- ies. We use a fixed R R1-style prompt (Guo et al. 2025) the evaluations of all models (details in the Appendix [.9p. We use Grounding-Diinn Dino-Base Base (Research) 2023) as the REC model and Depth- Anything-V2-Merre Metriccc-Laargge (Contributors) 2024) as the metric depth estimator in all experiments.
Chance/Human Performance Levels. We provide frequency- random selection as the chance level baseline for the multi-choice questions. Furthermore, we conduct human evaluation on multi- choice questions, where 10% of the questions from each category are randomly sampled and eval- uated by each human annotator. Table [ presents the results of this analysis. While humans can accurately answer the questions that require reasoning about relative location of the objects in space, their performance degrades in questions that require estimation of the exact distance between ob- jects. This highlights the challenging nature of accurate distance estimation.
5.1 BENCHMARKING GENERALIST VLMS
We use GPT-4o OOpeAAl 2024 and Gemini-2-Flash (DeepMind) (023) closed-source VLMs, and 3 competitive families of open-source models: Intern VT.3 (et al. 2025). Qwen2.5-VI. (Qwen al.

---

Published as a conference paper at ICLR 2026
2025). and Ovis2 (Lu et al. 2024) open-source models. Inference time analysis. numerical results on more generalist VLMs, and qualitative examples are given in the Appendix A.6
A.5 and A.9
As given in [ the performance of the VLMs varies considerably across different model param- eter sizes. Smaller models (c.g.. 3B and 8B) operate near chance level, indicating limited capacity for multi-view 3D reasoning. In contrast, larger models demonstrate substantial improvements over the chance level, yet still exhibit a noticeable gap when compared to human level performance. Furthermore, our proposed Ego3D-VLM provides significant improvement across all model sizes and tasks (average 56% relative improvement in RMSE and 12% in Accuracy), underscoring the importance of providing structured spatial representations to the model in 3D understanding tasks.
Performance Analysis. Figure 6 shows the average per- formance of 4 leading models (GPT-40, Gemini-1.5-Pro with and without
Uecargen
Intern VL3-78B, and Qwen2.5-VL-7223 Ego3D- VLM integration across all multiple-choice QAs. Travel time, localization, and object-centric absolute dis- lance are the most challenging tasks for VLMs with 40%- 45% average accuracy. We argue that these categories require intricate spatial reasoning and the ability to con- struct accurate mental map by the relative positioning of objects. While humans can simply build such maps and achieve perfect accuracy, VLMs struggle to replicate this level of SU, indicating key area for further development. In the absolute distance tasks, the incorporation of 3D lo- cation data through the cognitive map substantially narrows the gap between model and human level accuracy. Notably, for object-centre absolute distance, VLMs augmented with Ego3D- VI.M even surpass human. This expected, as hu- man estimation of 3D distances in object-centric cases is prone to substantial eиror without explicit 3D spatial info. Conversely, in the localization task, VLMs continue to fall short of human proficiency, even with cognitive map support.
time
Pgg Centr
bys corpcc Distance
Notic Reasering
Ega Shatin
Human Level
Buseine Mean
Beseiines
Chence mvvel Fgo b Mever
Figure 6: Average performance of leading VLMs w/ and w/o Ego3D- VLM vs. chance human levels on each category of Ego3D-Bench.
Blind VLM Performance. This baseline evaluates how much spatial reasoning be achieved by VLMs using only textual input, without any visual information, relying solely on their world knowledge (Majumdar al., 2024). We report the average performance of GPT-4o and Gemini-1.5- Pro models. The blind VL.Ms perform 5% worse than vision-enabled VL.Ms (53.8% vs. 58.8%) and 16.4% better than chance level. These results consistent with findings from prior single-view and video-based spatial reasoning benchmarks (Cheng et al. 2024 Yang et al. 2024)..
Blind Ego3D- VLM Performance. In this section, run an experiment by feeding the cognitive map to text only LLM. Since LnternVL3-8B// 1.3-8B/14BB use Qwen2.5-7B/14P as their backbone LLM, we use the same models for fair comparison. Table shows that Ego3D- -LLM with VLM outperforms Ego3D- Dgo3D-VLM VLM with LLM since VLMs can ignore false positives in the cognitive map and remain robust when false negatives occur, whereas L.LMs suffer performance drops in those cases.
5.2 BENCHMARKING 3D-VLMS
3D-VLM models have been trained on datasets designed for 3D SU, such as ab- solute distance estimation and relative lo- cation inference. We benchmark Spatial- RGPT and two checkpoints trained with the Spatial-VLM framework (Chen et al. 2024]: SpaceThinker--Ween Qwen2.5-3B (remyxai 2024 and SpaceQwen2.5-33B The Spatial- RGPT (Cheng et al. 2024) model assumes that
Table 2: Blind Ego3D-VILM performance analysis.
Only LLM
Mult. Choice
Abs. Ans. (RMSE.)
Model
(Acc. 1)
Intern VL3-8B
43.1
27.2
Ego3D-Vi VLMuasviLsS
53.1
11.3
Ego3D- VL.MeevVi. 3-48
60.1
8.0
Intern VL VL3314BB
51.7
17.6
E VLMuusME33118
55.6
10.8
68.0
8.5
specific regions of the image annotated with bounding boxes and is trained to answer 3D ques- tions based on these regions. To evaluate SpatialRGPT on our Ego Ggo3D-Benne DD-Bench. we reformat the input: object names in the questions replaced with placeholder labels (e.g., region-i), and list
8

---

Published as a conference paper at ICLR 2026
of corresponding region captions is passed the REC model to generate bounding boxes. These estimated bounding boxes were then overlaid on the images before being fed to SpatialRGPT.
Table 3 presents the performance of the menveementioned 3D-VLMs
Bgo achieves the highest overall performance, surpassing both SpatialRGPT with 8B parameters and other generalist VLMs of similar scale. This outcome highlights the critical role of dedicated 3D spatial pretraining and end-to-end architecture in advancing VLM capabiliitess for spatial reasoning tasks. Moreover, augmenting SpaceThinker with Ego3D-VLNN leads an av- erage improvement of 3% on multiple-choice questions and a reduction of more than 4 meters in absolute distance RMSE, emphasizing the effectiveness of our proposed solution.
Table 3: Comparison results of 3D-VLMs and our method (Ego3D- VLLM) on Ego3D-Bench.
Accuracy (%)
RMSF
Ego
Obj-
Ego Mot.
Obj. Mot.
Travel Time
Ego
Obj. Rel.
Ego Dist.
Obj. Dist.
Model
Loc.
Rel.
Avg.
Avg.l
SpaceQwen2.5 VVL-3BB
18.1
21.4
16.1
35.6
35.2
30.2
31.2
32.0
27.5
10.6
15.7
13.2
SRGPI-VILA1.5-4
45.7
35.5
39.6
43.6
45.8
24.9
50.8
71.7
44.7
11.5
15.1
13.3
@Thnnker-Qwen2.5-33
38.9
39.0
21.9
57.5
53.4
27.1
52.8
71.6
45.2
15.2
16.9
16.0
Ego BB.33D-VLMMmnnee 5wen2.
50.6
44.2
26.4
53.4
57.0
42.6
54.7
59.8
48.6
11.1
12.2
11.6
5.3 BENCHMARKING VLM,DEPTE-REC"
This category enhances the base VLM with metric depth estimator and a REC model. We pass each query to the REC model (Research] 2023 extract BBox of the referred objects, and use a metric depth estimator (Contributors) 2024) to estimate depth values. We then construct list that pairs each matched referring expression with its corre- sponding BBox and estimated depth (i.e., distance from the ego). An example entry in this list is: [Front-View: Detected pedestrian
Table 4: BLL3DDVLM vs. generalist VLMs
with REC+depth tools on (3go3D-Bencct
Mult. Choice Abs, Ans.
Model
(Ace. f)
(RMSEL)
Intern VI.3-8R
43.1
27.2
Intern 3.3355 3 DDD
51.6
13.1
Ego3D-VLL VLM InemVL3 3B
60.1
8.0
Owen2.5-7B
41.1
30.3
Qwen2.5-1 2- D.e...
49,4
11.8
Ego3D-VLL VLM Qwa125577
54.3
9.5
with red hat at bbox [x1, yl, x2, y2], depth: z, Back-View:...
(examples in the Appendíx [A.)]. As shown in Table 4 equipping the VLM with REC and depth estimator models improves its baseline performance. highlighting the base model's limitations in accurately identifying objects and estimating their depths. However, even with these enhancements, it falls short compared to our proposed Ego3D- VLM framework. This underscores the importance of integrating depth information into a unified map representation, enabling the VLM reason more effectively about the 3D relationships between objects across different views.
5.4 EG03D-VLM ON FURTHER BENCHMARKS
Our EgL3D-VLM is designed for ego-centric multi-view cases, which are particularly relevant for Al agent applications. In this section, how- ever, evaluate the performance of Ego3D- VLM alternative multi-view settings, specifi- cally those used in All-Angle Bench (Yeh et al. 2025) and VSI-Bench (Yang et al., 2024). Table Eggo3D-VL. com-
Table 5: Comparison results on All-Angle
Bench and VSI-Bench (accuracy).
Model
All-Angle-Bench VSI-Bench
GPT-4o
47.8
34.0
Gemini-1.5.PP
47.4
45.4
Gemini-1.55 1,5-Flash
46.6
42.1
Intern VL3-8B
47.9
38.1
Ego3D- VLMMnN
49.5
presents the performance
VitemVLL3-141 FVo3D-VLMM B...D--VV.MMuevEE.... 14
50.3
38.2
pared to open- and closed-source baseline VLMs
52.1
on All-Angle Bench and VSI-Bench. Despite the differences in data settings compared to our primary benchmark, Eg03D-VLM still outperforms the respective baselines, demonstrating its adaptability across diverse multi-view scenarios. More de- tails given in the Appendix A.6
5.5 ABLATION STUDIES
Ego3D- VLM Components. ing from the baseline, Vo. incrementally add each component. In V1, use cognitive map
an ablation on Ego3D-VLM core components. Start-

---

Published as a conference paper at ICLR 2026
with estimated rotation (R), translation (T), and intrinsic parameters (K). Specifically, all cam- eras are positioned at the coordinate center with T = [0, 0, 0..). The front camera uses an identity rotation matrix, while all the other
frontt--ggt-thhhh right, back-right, back, etc. -are incre- mentally rotated 45 around the Y-axis. The focal length is estimated from the images' approximate the actual K and in V3 further use actual R, and T relative to the front camera, i.c., often available as a fixed parameter for an embodied Al agent. Va and Vi obtain comparable RMSE, while V3 is only 2.4% higher in multi-choice QAs. Thus, even with estimated camera parameters our cognitive map can significantly boost the baseline's performance. V4 adds relational scaling to cognitive map which decreases the RMSE by 2.5 meters. V4 is indeed Eg03D-VLM with all components. We provide two more ablations to evaluate the upper-bound of Ego3D-VLM. Vs assumes that the list of ob- jects (only their names) in the input query are provided which enhances the REC results. V, is the ground-truth cognitive map with ground-truth 3D locations. The difference between V6 and hu- man level is only 5% showing the upper bound of Rgo3D-VLM,
field of view. In V2,
Table 6:
Ablation on Ego3D-VLM main com- ponents. V4: all components.
Mult. Choice (Acc. f)
Abs, Ans, (RMSE.)
VLtemVL3--8
43.1
27.2
Vi
CogMap (est. R,T,K)
56.0
10.8
+ K
56.3
10.1
R.T
58.4
10.4
Scaling
60.1
8.0
Vs
List of objects
61.8
6.5
GT CogMap
79.4
1.3
Perception-Reasonniie Disentanglement. In or- der to disentangle perception from reasoning, we add the ground-truth 2D BBox for all objects the benchmark. This allows repeating our ex- periments with BBox overlaid, effectively isolat- ing the reasoning and perception. As expected, adding GT BBox as visual prompts to the images will help the VLM to handle perception casier, therefore improving the results. The results in Ta- ble 7 show that the major limitation of baselines not 2D perception, but 3D perception and reasoning. More reasoning analysis on Ego3D BegooDD-Bene Bench well Ego3D- VLM'' robustness in challenging conditions are given in Appendix A.4 and A.7
Table 7: Perception-reasoning disentanglement.
GT BBOX
Mult Choice (Acc. f)
Abs, Ans. (RMSE.)
Model
VI.3-81
Intern VL3-8B
50.2
Ego3D-VLMMunm
60.1
8.0
Ego 30DD-VLMunnVL3-1
62.2
IntemVV1.3144
$1.7
17.6
VL3-1VL3311B
51.8
16.2
Ego3D-VLLL VLM 148
66.5
7.7
Ego3D-ViL VLMaueVVL---1
66.8
7.5
Cognitive Map Format. We explore 3 different formats for our generated cognitive map: visual, JSON, and textual. Examples of the visual and textual maps are shown Figure 5 The aver- age results are presented in Table S Among the three, the textual and ISON formats achieve the best overall performance.
Table 8: Ablation on cognitive map format.
Mult. Choice Abs, Ans. (Acc. f)
(RMSEL)
Visual Cog-Map
50.9
14.4
ISON Cog-Map
60.0
8.4
Textual Cog-Map
60.1
CONCLUSION AND FUTURE WORK
In this work, we proposed Ego3D- benchmark for spatial understanding of VLMs ego- centric view images. Overall, the benehmark shows significant gap between human scores and VLMs. To address this limitation, we provided training-free solution named Ego3D-VLM to enhance the performance of VLMs. Future work should explore fine-tuning VLMs with ego-centric multi-view QAs and incorporating 3D projection modules proposed in EVL in the course of fine-tuning. The limitations of this work are provided in the Appendix (A.10).
REFERENCES
Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9392-9401, 2025.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1162-11663,, 2020.
10

---

Published as a conference paper at ICLR 2026
Kevin Cannons. Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaci, Mohsen Gho- lami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, and Mohammad Akbari. From seg- ments to scenes: Temporal understanding in autonomous driving via vision-language model, 2025.
http://///aaiioooo////////////// 05277
Ming-Fang Chang, John W Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang, Peter Cam, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d track- ing and forecasting with rich maps. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Russ Salakhutdinov. Seal: Self-supervised embodied active learning using exploration and 3d consistency. Proceed- ings of the Conference on Neural Information Processing Systems (NeurIPS), 34:1 13086-, 2021.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision- laniuue models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14455- 14465, June 2024.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2024.
Hugging door
Face
Contributors.
Depth anything
v2
metric out-
large.
https://huggingff co/ depth-anything/
Depth h- Anything V2 MetMeetricc-ee Seobrrio-otddooo--arrggeeee 2024. Accessed: 2025-05-
16.
Erik Daxberger, Nina Wenzel, David Griffiths, Haiming Gang, Justin Lazarow, Gefen Kohavi, Kai Kang, Marcin Eichner, Yinfei Yang. Afshin Dehghan, and Peter Grasch. Mm-spatial: Exploring 3d spatial understanding in multimodal llms, 2025.
arxiv. org/ ab03
1311
Google DeepMind. Gemini: A family of highly capable multimodal models. Google DeepMind Technical Report, 2023. URL
com/
deepminnd-mediageeet
/gmini v1 _i___v______________ report.
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial- bbnch:: Bench- marking spatial understanding for embodied tasks with large vision- models. In Lun Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Pupers). pp. 346-355, Bangkok, Thai- land, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2224.ad/ 33. URL httpt:::/ccaatttollllaa htt.::/w.......02222...e--......
Jinguo Zhu et al. Internv1:: Exploring advanced training and test-time recipes for open-source multimodal models, 2025.
2555//04.///52.....// 10479
Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL htttt:tt///aaaettt
org/ abs http://aas22444.1222
Mohsen Gholami, Mohammad Akbari, Tianxi Hu, Vaden Masrani, Wang, and Yong Zhang. Gold: Generalized knowledge distillation via nut-of-distribution-uuu language data generation. In Findings of the Association for Computational Linguistics: NAACI 2024, pp. 4380, 2024a.
Mohsen Gholami, Rabab Ward, and Z. Jane Wang. Posegen: Learning to generate 3d human pose dataset with nerf. Proceedings of the AAAI Conference on Artificial Intelligence, 38(3):1905- 1913, Mar. 2024b. doi: www.1660/aaii..vv333.. v3813.27960, php/ www.a//aaiiee/vee.......
URLhttta::// wtt.://oo.aaaa.o.o.......... wwg.iiii......e...rr index.
Mohsen Gholami, Mohammad Akbari, Kevin Cannons, and Yong Zhang. Casp: Compression of large multimodal models based on attention sparsity. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9372-9381, 2025.

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12944 2025.
Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images, 2023a. URL
abs/2303 11327
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2023b.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference on Machine Learning (ICML). 2024.
Amita Kamath, Jack Hessel, and Kai- Wei Chang. What's up with vision-language models? in- vestigating their struggle with spatial reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9161-9175 Singapore, December 2023. Association for Computational Linguis- URL /RLLL/tcss///allaatooooooooo///p
tics. doi:
emm ma in. 5687
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.085 033226, 2024a.
Dingming Li. Hongxing Li, Zixuan Wang. Yuchen Yan, Hang Zhang, Siqi Chen, Guiyang Hou, Shengpei Jiang, Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Viewspatial- bench: Evaluating perte-peesee spatial localization in vision-language models, 2025. URL
https:// wwtpxiv.org/b///555... 21500
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interllavee: Tackling multi-image, video, and 3d in large multimodal models, 2024b.
URL htttht::s///axiiii wwwp:xxwww.ooraaaoooooo abs /400/ 07895
Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. Reasoning paths with reference
objects elicit quantitative spatial reasoning in large vision-language models, 2024. URL https:
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the As- sociation for Computational Linguistics, 11:635 651, 2023. doi: 10.1166/ URL
https:
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. trXiv:2405.20797 2024.
Chenyang Ma, Kai Lu, Ta- Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2024a.
Yueen Ma. Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey vision- language-action models for embodied ai. arXiv preprint arXiv:2405. 14093, 2024b.
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Ra- jeswaran. Openeqa: Embodied question answering in the era of foundation models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16498, June 2024.
12

---

Published as a conference paper at ICLR 2026
OpenAl.
2024. Accessed: 2025-04-29.
Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Un- derstand 3d scenes from videos with vision-language models. arXiv preprint arX1v:2501.01428, 2024.
Qwen, and An Yang et al. Qwen2.5 technical report, 2025. WR. wwt.httsss//axxx..rr////// 2412. 15115
Arijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, and Kate Saenko.
Sat: Dynamic spatial aptitude training for multimodal language models, 2025. URL https:
/bb////b/// /atpp//2122227333
remyxai.
Vqasynth, 2024. URL http:s///gttaaaacccco///aa/////s
main
GitHub repository.
IDEA Research. Grounding dino base model on hugging face, 2023. URL https:// Salgginggface.coIDDeei htt.re/eoe..-eeeeeeeeeeeeeeeecee
dino---ino---- base Accessed: 2025-05-15.
Ahmad Rezaei, Mohsen Gholami, Saeed Ranjbar Alvar, Kevin Cannons, Mohammad Asiful Hos- sain, Zhou Weimin, Shunbo Zhou, Yong Zhang, and Mohammad Akbari. Cppo: Contrastive perception for vision language policy optimization, 2026. URL pesttss//arxivi /ttpp//////xxiorrrr/bb/////// 2601.00500
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024. [RL/pttts///xxxi wwwpaxwwwwwwpp./////0/a0000 abs42222 03300
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Ro- boSpatial: Teaching spatial understanding to 2D and 3D vision-language models for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. Oral Presentation.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 2446-2454, 2020.
ManyCore Research Team. Spatiallm: Large language model for spatial understanding. https:
//github. wtattuub..ommmmaycor..... rwwearch SwwprcchhSpaaaaaa... 2025.
Jayy Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 37:75392- 75421, 2024a.
Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19757-19767. 2024b.
Changli Wu, Qi Chen, Jiayi Ji, Haowei Wang, Yiwei Ma, You Huang, Gen Luo, Hao Fei, Xiaoshuai Sun, and Rongrong Ji. Rg-san: Rule-guided spatial awareness network for end-to-end 3d referring expression segmentation, 2024a.
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind's eye of Ilms: off elicits spatial reasoning in large language models. In Pro- ceedings of the Conference on Neural Information Processing Systems (NeurlPS), 2024b.
13

---

Published as a conference paper at ICLR 2026
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang. Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie. Yuxiang You, Kai Dong, Xingkai Yu, Haowei 7hang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12: Mixture-of- experts vision-language models for advanced multimodal understanding, 2024c. URL https:
10302
Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. arXiv preprint arXiv:2412. 14171, 2024.
Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li. Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, and Jiangmiao Pang. Mmsi-bench: A benchmark for multi-image spatial intelligence. arXiv preprint arXiv:2505.23764 2025.
Chun-Hsiao Yeh, Chenyu Wang, Shengbang Tong, Ta- Ying Cheng, Rouyu Wang, Tianzhe Chu, Yuexiang Zhai, Yubei Chen, Shenghua Gao, and Yi Ma. Seeing from another perspective: Eval- uating multi-view understanding milms. arXiv preprint arXiv:2504.15288 2025.
Jiawei Zhang, Chejian Xu, and Bo Li. Chatscene: Knowledge-enabled safety-critical scenario gener- ation for autonomous vehicles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15469, 2024a.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL
allaaaavlll.. vl. wtt.a//wwww..g..........//////// -01-://////2220--000--------- 10-vlava next sirt-viie video/
Yue Zhang. Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, and Lifu Huang. SPARTUN3d: Situated spatial understanding of 3d world in large language model. In The Thirteenth International Con- ference Learning Representations, 2025a. URL httt:::///penrevieew wwt//foo forum?
1d=FGMKSTANNR)
Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi, Joyce Chai, and Ziqiao Ma. Do languaee models represent space and how? evaluating spatial frame of reference under ambiguities. In The Thirteenth International Conference on Learning Representations, 2025b. URL https: http///ppereeieeeeeeeeeeeeeeeeee /t/hhttpsss//peeeeeiieeeee
n fow.foiddmn oew.eeeieeeeeeeeeoorrrrrddaadae
Baining Zhao, Jianjie Fang. Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jingiang Cui, Xinlei Chen, and Yong Li. Sobanvideo-bencle Benchmarking vision- language models on embodied intelligence with video data in urban spaces, 2025. URL https:
l/arxiv.orge abs///b//////// 06157
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning posttion-awaree video represen- tation for 3d scene understanding, 2024. ULL/:ttpe.//axxxxx.rrr///b/b///
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: simple yet effective pathway to empowering Imms with 3d-awareness. arXiv preprint arXiv:2409.1888 18125, 2024.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
In this appendix, we present the following additional discussions and experimental results corre- sponding to our proposed Ego3D-Berchh Bench and Ego3D-VLM:
Code and Ego3D-Bench
Bgo3D-Bench vs. Other Benchmarks
Further details on Rgo3D-Bench creation and question templates
Benchmarking the thinking of Rgo3DD-Bnnch as Open-QA
Results of the three source datasets: NuScenes, Waymo, and Argoverse
Benchmarking more generalist VLMMs
Inference time analysis of Sgo3DDVLLM vs. other baselines
Robustness of Ego3D-VLM D-LM in challenging conditions
Qualitative results and prompts structure
Limitations and future work
A.1 CODE
In order for results to be reproducible, share code supplementary materials, with detailed instructions included in the associated README.md file. We have also included the Ego3D-Bench in the supplementary material. However, due size limitation, images should be downloaded from the source datasets. Instructions to download images and models are provided in the README.md file.
A.2 1G03D-BENGK VS. OTHER BENCHMARKS
As described in the main body of this paper, our proposed Ego3D-VLM is designed for ego-centric multi-view scenarios, which are particularly relevant for AI agent applications. Ego3D-VLM is dif- ferent from alternative multi-view settings, specifically those used All-Angle Bench (Yeh et al, 2025) and VSI-Bench (Yang al. 2024). Figure 7 illlsttrates the configurations of these bench- marks in comparison to Ego3D- Bench. All-Angle Bench features stationary multi-view cameras observing the same scene from different angles-a setup commonly used in motion capture and surveillance systems. VSI-Bench, on the other hand, involves a single moving camera within a static indoor environment, typically used for static scene reconstruction and SU. In contrast, our Ego Ego3D-Bench assumes multiple cameras mounted on a moving ego agent, capturing the surround- ing environment a configuration suited for AI agent applications such as robotics and autonomous driving.
Allll Bench
VSI-Bench
Ego3D-Bench
Trajectory
Scene
Scene
Scene
Scene
EAG
Scene
Scene
Applications:
Applications:
Applications:
Motion Capture
Static Scene Understanding
A Agents: Robotics and autonomous driving
Surveillance Cameras
Figure Settings of multi-view/video spatial understanding benchmarks for VLMs.
15

---

Published as a conference paper at ICLR 2026
A.3 EGo3D-BENCH: FURTHER DETAILS
In this section, we provide the templates used to create questions of the benchmark. Figure shows templates used to create each category of Rgo3n- We replace <obj1>, <obj2>, and <obj3> in the templates with object descriptions. <viewl>, <view2>, and <view3> are placed with the camera views from which the object is visible (c.g.. Front-Right, Left, etc.). The placeholder <direction> in motion reasoning tasks is substituted with one of the four cardi- nal directions- "nnrrhh""s"""orth.,, "east", "west", or "south" and this process is repeated until all directions have been used. The placeholder <Y> is for motion reasoning tasks and is replaced with a number between 2 to 5 meters. <x1>, <x2>, <x3>, and <x4> serve as placeholders for multiple-choiee options in absolute distance estimation tasks. One of these options replaced with the ground-truth distance, while the remaining are filled with randomly generated values, ensuring that the distance between any two options is at least 8 meters.
A.4 BENCHIMRRKING THE THINKING OF EGO3D 1-GCC-BENCE
BENCE AS AN OPEN-QA
All our question- question-answer answer pairs include "thinking" step in open-ended format. In this section, we generate "GT thinking" for further open-ended type evaluation. To this end, we used the GT cog- nitive maps along with GPT-4o to generate the reasoning for two categories that require numerical answers: Ego-Centric/ Object- Centric Absolute Distance (in meters). Table 9 shows that Ego3D- VLM thinking is better than the baselines.
Table 9: Benchmarking the thinking of VLMs Ego3D-Bench
open-ended category. The GT thinking is generated using GT cognitive maps along with GPT-4o.
Thinking of Obj. Abs. Dist. Thinking of Ego Abs. Dist.
GPT-4o Score 10)1
GPT-4o Score (0-10)T
InternVL3-8B
1.9
2.0
Ego3D-VL. -VL InceraVL3--8
4.7
5.2
V Ego3D-VL.. ImcrnVL3--148
2.3
2.4
5.0
Intern VL,3-38
2.1
2.5
Ego3D- VLM ImcerNVL.-388R
7.1
3.7
2.1
1.7
Ego3D-VLM. In07rnV1.3-788
7.0
6.8
A.5 BENCHMARKING GENERALIST VLMs: FURTHER RESULTS
In the main paper, we reported results for Interv Qwen2.5-VL, and Ovis-2 as representa- tive open-source models, as they are, to the best of our knowledge, the current SOTA among open-source VLMs. Table To extends this comparison by including additional generalist VLMs on Bgo3D-Bench, such as Gemini- 1.5-Flash (DeepMind 2023), Gemini-2-Flash (DeepMind) ).23). Gemini-2.5-Flass (DeepMind 2023). Phi-3.5 al. 2024p. LLaVA-One- LiL---nn---V----- (Li al. 2024a), LLaVA-Next- LLaVA-Next-Video--7 (Zhang et al.] (024b), and DeepSeek- V1.2 (Wu et al.) 2024c). We observe that V.LVVA-One Vision, Phi-3.5, and DeepSeek- V1.2 underperform compared to In- tem and Qwen2.5-VL models of similar size. For instance, Intern Intem VL3-8B achieves an accuracy of 43%, while LLaVA-One VA-One- Visee--isio--- 7B achieves 38.7%. Likewise, Phi-3.5 (3.8B) attains an accuracy of 40.3%, compared 41% % from Ovis-2 (4B).
A.6 INFERENCE TIME ANALYSIS
Table II shows the inference time analysis of Ego3D-VL.N VLM compared to baseline models. We re- port End-to-End Latency (E2E Lat.) in seconds and Peak Memory in GB. For the E2E Lat., we measure the average end-to-end inference of models on 50 samples of Ego3D-Bench Bench and for the peak memory report the peak memory usage during inference on the same samples. Experi- ments are performed using flash-attention-2. The memory and latency overhead of Ego3D- VLM over Intern VL3-78B is 0.6% and 31% respectively. The main reason for the latency overhead is that the model reasons more when cognitive map is provided in the Ego3D- VLM. In order to make the inference more efficient and deal with the latency overhead of the VLMs, post-training tech-
16

---

Published as a conference paper at ICLR 2026
Travel Time:
How much time does it take to move from the location of <obj1> in <viewl> directly towards the location of <obj2> in the <view2> with the speed of (V} m/s? seconds More than 20 seconds
A.Less than seconds B.5-10 seconds
Localization:
Type1:
front-view corresponds to north direction. stand the location of <obj1> in the <view1> facing north, is the <obj2> to my left, right, front, or back?
A.Front B. Back
C. Right Left
Type2:
The front-view corresponds to the north direction. Assume <obj1> in the <view1> is facing north. From the perspective of <obj1> in the <viewl>, where is <obj2> in the <view2> located?
A. Front B. Back c. Right D. Left
O Ego Motion Reasoning:
The front view corresponds to the north direction. If the ego moves <Y> meters (direction) while all other objects remain stationary, does the ego car get closer <obj1> in the <viewl>? A. Yes B. No
Obbject Motion Reasoning:
The front view corresponds to the north direction. If the <obj1> in the <viewl> moves <Y> meters <direction> while all other objects remain stationary, does the cobj1> in the <view-1> get closer to the <obj2> in the <view2>?
Yes No
Q Objject-Cnntre Relative Distance:
which one is closer to <obj1> in the <viewl>, <obj2> in the <view2> <obj3> in the <view3>?
A. cobj2> B. <obj3>
Q Ego-Centric Relative Distance:
<obj1> in the <viewl> closer to the ego car than <obj2> in the <view2>?
A. Yes B. No
Ego-Centric Absolute Distance (Meter):
How far, in meters, is <obj1> in the <viewl> from the ego car?
Object-Centric Absolute Distance (Meter):
How far, in meters, is <obj1> in the <viewl> from <obj2> in the <view2>?
Ego-Centric Absolute Distance (Multi-choice):
How far, in meters, is <obj1> in the <viewl> from the ego car? A. <X1> meters B. <X2> meters C. <X3> meters D. <X4> meters
Obbject-Cenriic Absolute Distance (Multi-Choice):: How far, in meters, is <objl> in the <viewl> from cobj2> in the <view2>? A. <X1> meters B. <x2> meters C. <x3> meters D. <x4> meters
Figure 8: Templates used to create questions of Ego3D-Bench.
niques such as quantization GGhllami et al. 2025 token pruning Alvar el al. 2025 or knowledge distillation (Gholami al. 2024a) can used.
A.7 ROBUSTNESS OF Ego3D-VLM IN CHALLENGING CONDITIONS
To isolate tool performance, we included an ablation with ground-truth cognitive maps in the main body of the paper (T which improve the results from 60.1 1% to 79.4% accuracy. This confirms
17

---

Published as a conference paper at ICLR 2026
Table 10: Results of further generalist VLMMs Rgo3D-Bench.
Accuracy (%) f
RMSEL
Ego Dist.
Obj- Dist.
Ego
Qhj.
Travel
Ego
Ohi. Rel.
Ego Dist.
Obj- Dist.
Model
Loc.
Mor.
Mot.
Time
Rel.
Avg.1
Avg.
Gemini
Gemini-1 Flash
40.4
26.9
50.5
70.1
61.8
28.0
53.6
75.6
50.9
10.8
31.0
20.9
Gemini-2- Somini-2-Flashh
43.3
38.6
54.1
58.0
39.3
29.1
47.1
68.5
47.2
10.9
29.4
20.1
Gemini 2.5-Flash
41.3
25.6
65.1
93.4
84.8
20.1
63.5
68.5
57.8
11.4
19.6
15.5
Phi
Phi3.5
28.3
30.8
24.2
59.6
55.9
21.3
45.3
56.7
40.3
22.8
49.4
36.1
LL Family
LLaVA Next-Video-7B
30.2
26.6
42.2
60.8
58.0
48.3
55.1
73.5
49.3
19.5
22.8
21.2
LLaVA-OV-78
23.5
22.1
58.4
56.3
52.5
17.9
59.0
20.0
38.7
17.8
19.3
18.3
DeepSeek Family
DeepSeek- DeepSeek-VL2-tiny
34.7
29.0
19.9
60.0
59.3
46.6
57.8
65.0
46.5
14.0
18.7
16.3
DeepSeek VL2-Small
18.1
19.2
28.6
50.3
55.6
37.4
50.5
59.3
39.9
12.7
14.7
13.7
DeepSeek- VL2
22.3
25.3
32.4
60.0
59.1
21.4
56.7
62.0
42.4
20.3
17.4
18.8
Table 11: Inference time and memory usage of Ego3D-VLM compared to the baselines.
E2E Lat.
Memory (GB)
E2E Lat.
Memory (GB)
(sec)
(sec)
Intern VL33-B
5.2
18.1
Intern VL.3-388
16.9
80.0
Ego3D- Ego3D-VLM MLM.IeMM.......... V1..88
8.6
26.5
Ego3D- VLM V1..
19.1
84.6
Intern VLternVL3.. VL3-14B
15.5
33.1
InternVL3-778
35.0
161.7
Ego3D- ML.M IntemVL33.1p
16.4
40.2
Ego3D- IntemVL3.78B
46.9
162.4
that, there still a gap between the accuracy of such imperfect tools and ground-truth REC/depth. However, compared to the baseline, solution still achieves more robust results even in chal- lenging conditions such as low brightness, motion blur, and occlusion. To support this claim, we simulated these conditions and re-ran
specific experiments shown in the Table T2
Although G-DINO and Depth-Anything used Ego3D- VLM are among the best tools for REC and depth estimation, solution is orthogonal any such tools, and has no limitation in this regard.
Table 12: Robustness of external tools in scenarios involving occlusion, motion blur, and low light.
Mult. Choice
Abs. Ans. (RMSEL)
(Acc. ?)
Intern VL3-8B
43.1 60.1
27.2
Ego3D-VLL IntemVL3--B
8.0
Intemn [60% Darkness]
41.1
29.8
Ego3D-VLM, IntenVL3-sB [60% Darkness]
59.6
10.6
InternVL.3-8B [Motion Blur, 15x1 kernel]
42.5
28.5
Ego3D-VLM ImeenVL3-su [motion blur, 15x1 kerne]]
57.9
9.9
VL3-8B
[30% Occlusion]
42.0
28.9
Ego3D-VLM ImsnW11.-58 [30% Occlusion]
58.7
10.7
A.8 RESULTS OF DIFFERENT SOURCE DATASETS
Table 13 presents the results for different source dataset splits used create Ego3D-Bench. De- spite the varying number of camera viewpoints across the three datasets, the performance deviations minimal. This consistency underscores the reliability of Ego3D-Bench dicating that model performance is not heavily influenced by the specific choice of source data desirable property for robust and fair benchmark design.
benchmark, in-
A.9 QUALITATIVE RESULTS AND PROMPTS STRUCTURE
Figures 20 20 demonstrate example responses of Intern 78B and Ego3D-VLL. VLM VL--78 all categories of Ego 3D-BBnchh As seen, 3go3DBBench enhances the spatial reasoning ability of the baseline by providing the textual cognitive map.
18

---

Published as a conference paper at ICLR 2026
Table 13: Results samples of Rgo3D- generated from each of the source datasets.
NuScenes Acct RMSE]
Waymo Acct
Argaverse Acef
RMSEL
RMSEL
Closed Models
GPT-4o
60.9
21.2
60.0
19.0
59.4
17.1
Ego3D-VILS VLM GPT-4u
73.5
8.5
73.1
7.5
72.7
6.2
Gemini-1.5 1..-Prr
59.3
26.4
55.6
12.7
57.5
19.5
Ego3D- VLM Gemini-1.-PPo.
77.1
8.1
68.9
6.3
73.0
7.1
Qwen2.5 Family
Qwen2. 2-33B
39.7
33.4
38.6
31.5
38.9
31.2
Ego3DDVVis Qwen2.5-3B
44.7
13.0
40.3
12.1
48.1
12.6
Qwen2.
40.9
36.0
40.6
24.7
41.6
30.1
Bgo3DD-VI, VI.M Qwen2.5-7B
56.4
11.3
51.9
8.3
54.4
8.7
Owen2.5-321
56.6
17.5
56.0
11.0
59.2
18.7
Bgo3DD VT.M Qwan2,5-32R
64.3
17.1
64.8
10.6
64.0
Owen2.5.11 S572B
58.2
21.2
57.7
11.1
58.2
16.2
Ego3D-VLL Qwen2.5-728
74.4
8.5
64.4
6.4
69.5
7.6
Intern VL3 Family
InternVI. VL.3-8B
42.3
28.3
44.0
21.7
42.9
31.5
Ego3D- VLM IntemVL.3-8B
62.0
57.1
6.5
61.3
InternVL3.1ee Ego3D- IntemVL3-144
51.7
20.4
50.2
15.8
53.3
16.6
69.0
8.8
63.7
7.5
67.0
6.3
Intern VL3-38B
51.2
31.4
50.5
19.4
53.4
25.4
Ego3D-VIA, VL12mVLL3-36B
71.7
8.7
64.3
8.2
67.0
6.3
Intern |nternVL.3-7888 VL3-78B
60.0
16.7
59.7
10.7
60.1
13.7
3go3D-VLMM IntemVL3-788
76.5
8.3
67.0
6.5
71.0
7.5
Ovis2 Family
Ovis2-4B
41.2
29.2 2
43.0
18.1
40.9
22.5
Rgo3D-VI.M 0vi22-B
47.3
47.5
7.1
49.7
8.3
Ovis2-8B
41.5
21.4
41.9
20.7
41.0
20.2
Ego3D-VLMN VLM Ovis2-88 8B
55.6
9.4
55.1
6.8
57.9
7.0
Ovis2-16B
48.6
16.2
45.5
11.9
46.8
12.9
Ego3D- VLM Ovio2.18
62.6
8.5
57.9
7.15
62.2
6.5
A.10 LIMITATIONS AND FUTURE WORK
This work has several limitations that need further investigation. Ego3D- VIM does not improve the innate visual spatial reasoning of VI.Ms, but rather proposes framework that augments VLMs with structured 3D information. Future work, should consider end-to-end training of Ego3D-VLM using reinforcement learning Rezaei al. (2026); Shao et al. (2024) to enhance the spatial perception capability of VLMs. The proposed Ego3D-VLL relies on the reasoning capabilities the underly- ing vision- vision-language models (VLMs). Consequently, for models with limited reasoning ability (e.g., VILAI A.55-8)), we observe little to no improvement when combined with Ego3D-VLMM In this work, did not address spatial reasoning tasks in conjunction with temporal understanding. Future work should extend the proposed method to incorporate the temporal dimension Cannons et al. (2025).
Additionally, the REC models used in our pipeline provide 2D bounding box locations for all ex- pressions in the prompt. This results in redundant information within the cognitive maps, which may confuse or mislead the VLMs. Our ablation studies demonstrated that when the list of objects mentioned in the prompt is known in advance, the RMSE improves by approximately 1.5 meters.
Another limitation lies the accuracy of metric depth estimation in outdoor environments. To miti- gate this, we proposed relational scaling approach based on common-sens object sizes. However, this method is inherently approximate and not fully reliable. Future work should aim to address these issues by enhancing the depth understanding of VLMs or by improving metric depth estima- tion in outdoor settings. Better integration of spatial reasoning and scene geometry into VLMs could further improve performance in complex 3D environments, Another major limitation in spatial rea- soning datasets and benchmarks the availability ground -oound-truthha 3D. Previous work has shown the effectiveness of neural radiance fields in generating choto-realistic 3D data Gholami al. (2024b). Future work should investigate this direction for extending spatial reasoning dataset without having ground-truth 3D annotations.
19

---

Published as a conference paper at ICLR 2026
A.11 VSI-BENCH COGNITIVE MAP vs. EGO3D-VLM:
VSI-Bench uses the VLM itself to estimate cognitive-mnn map and then passes the generated cognitive- map to the VLM in a second call. However, we use external tools to create the map which leverages higher accuracy in depth estimation and 2D localization of objects. Moreover, creating the cogmap by external tools is faster than calling the VLM twice. In order to quantitatively evaluate, the method in VSI-Bench and Ego3D-VLM, we use VSI-Bench method create cognitive map and compare its results with Ego3D-VLM (Table 14p.
Table 14: Cognitive Map of VSI-Bench vs. Ego3D-VLM
InternL3--8B
Intern - Intern VL3-8B E3e
Acuracy
43.1
61.6
47.0
A.12 FURTHER DETAILS OF THE DATASET
Ego3D-Bench annotations were conducted internally by trained researchers within our organiza- tion who are experienced in 3D perception, autonomous driving, and dataset curation. No external crowd workers were used. In total, the annotation process required 112 numan-hours. All annota- tors were compensated according to their standard full-time employment contracts; no task-based or performance-ban payment structure was used. Since annotators were internal researchers rather than external contractors, no additional compensation scheme applied.
Figure shows the distribution of objects in our benchmark. The captioning process was inten- tionally designed not to rely rigid linguistic or structural templates. Instead, annotators were in- structed to produce concise, factual descriptions that focus on object identity, 3D spatial attributes, and scene-relevant properties (e.g.. "a white sedan parked the right lane" or "a pedestrian cross- ing in front of the ego vehicle"). This flexible guideline ensures that captions remain natural while still being compatible with downstream VLM training.
10a
10a
10
2212
de
dui
de
JO
snd Aauaq
uepies
Cleeee Suurrpo
and Jora
inpe
ajptoy
qu
Burres
Figure 9: Distribution of objects in Ego3D-Bench. To maintain objectivity, reliability, and neutrality, incorporated the following quality-control steps:
1. Clear annotation guidelines: Annotators were instructed to avoid subjective judgments (e.g., "dangerous driver," "reckless pedestrian"), emotional language, or speculative tent. Only directly observable attributes (category, appearance, relative position, size) were allowed.
20

---

Published as a conference paper at ICLR 2026
2. Structured attribute checklist (soft template): Although no fixed sentence template was enforced, annotators followed a checklist covering below items which acted as a soft tem- plate, improving consistency without forcing unnatural phrasing:
Object category
Color/shape if visible
3D relation to the ego (left/right/roon/bakk, distance tier)
Action or state (parked, moving, turning, standing still)
A.13 MULTI-VIEW IMAGE Vs. CONCATENATED IMAGES
For generality purposes we used the default setting of passing each view point separately in the main body. In this section we concatenate multi-views into a single image. Table shows concatenating multi-views enhances the results of both baseline and our method. We hypothesis that concatenating images reduces the number of vision tokens that enhances models performance.
shows that
Concatenate Images Intern VL3-8B Intern VL3-8B 8 InterVL3-8B Intern VL3-8BL3-BBBo B-gaDD-VLLL
Multiple Images
Acuracy
43.1
60.1
47.0
65.8
14 TEXTUAL VS. VISUAL COGNITIVE MAP
Ablation in the main body shows that textual cognitive map outperforms visual cognitive map. We hypothesis that the textual format outperforms the visual format because of the following two rea- sons. 1) Converting 3D information image domain and then decoding the information back to textual domain using visual encoders (c.g., siglip) poses errors. However, converting information directly to textual format lossless. 2) The information density in the textual format is higher than the visual format. In order to quantitatively evaluate this, we have pruned the cogmap tokens and evaluated the performance drop. Table below shows that with the same level of pruning, textual cogmap results in higher performance drop. This result, showcase that the information density in the textual cogmap is higher than visual cogmap. Here the relative accuracy drop is defined as (A ACCpruned) (aatuaa)(//ACoooooos (AcCego3d-Im ACCsasetine). It means that the relative accuracy drop in 0% pruning is zero for both methods and 100% pruning results in 100% relative accuracy drop for both methods.
Relative Accuracy Drop Textual-CogMap Visual-CogMap
CogMap Pruning
0%
0.0
0.0
10%
30.2
0.0
20%
31.0
5.3
30%
57.5
32.3
A.15 ANALYSIS ERROR PROPAGATION RELATIONAL SCALING
Estimating the relational scaling that proposed in the main body is prone to errors specifically in the case of partial visibility of objects (e.g. pedestrians), non-upright poses, motion blur, etc. We provide error propagation analysis showing how variations in scale estimation affect downstream results.
Let h denote the measured bounding- box height in pixels, the representative object depth, fy the vertical focal length, and H the assumed real-works object height. Using the pinhole model, the camera-space vertical extent of the object is
Theoretical A Analysis of Scale Sensitivity 2D Bounding- Box Errors:
h AyaD
fy
To enforce metric height of H, we estimate the global scale via
fg
8 Ayan = H
+
h z
21

---

Published as a conference paper at ICLR 2026
The sensitivity of to small perturbations in the pixel height h follows from
Os Hfy Oh 122
so a small error öh induces the first-order scale error
Hy 8h h22
ós
If we plug the typical values fy = 1000 px, H = 1.7m, bbox height h = 200 px, and the depth
2 10m:
8a 0.004 x dh
Therefore, the error in scale results the following error calculating the 3D positions:
E òs Pelobal (0.004 Palobll
Therefore, 10% errors in the bounding box (i.e., 8h x 20px), results in E' 0.08pglobal It means that for objects let's say 10 meters far from ego the distance error will be 0.08 x 10 0.8m and for objects 20 meters far from the ego the error will be 1.6m.
Empirical Analysis of Scale Sensitivity to 2D Bounding- Errors: We now perform an em- pirical analysis by adding errors in 2D bounding box detections and calculate the final accuracy of Inern BL3- on Ego3D-Benche At each scale the error was randomely applied to be +5h
+-5h (e.g., +1% 10%))
Scaling bbox error Accuracy
Intern VL3-8B
43.1
Intern VL3-8B Ego3D VLM
58.4
Intern FL3-&BEgodD-VLL
0%
60.1
10%
59.9
20%
57.6
+ 30%
55,4
A. STATEMENT ON LLMS ASSISTANCE
We declare that some portions of this document have been lightly refined using Large Language Models (e.g., ChatGPT) to enhance clarity and polish. All substantive content and ideas remain entirely our own.
22

---

Published as a conference paper at ICLR 2026
Ego-Centric Absolute Distance (Meter)
Prompt: These are five camera views nounted on an ego car.
Front <inage>
Front Left view: <image>
Left view: cimage>
Front Right view: <inage>
|Right vlew: <inage>
|How far, meters, pedestrian black jacket with white bag in left view from the ego car?
output the thinking process in <think> </think> and final answer (number only) in <answer> </answer> tags.
Answer: 10.29
Front View
Front View
Front-Right View
LofeN View
Left
Right
InternVL3-788
Eg03D-VLM mnemLt5-788
<thinks
<think>
First, I need to determine the distance jacket white bag fron the ego car in the left view. The image shows city street with multiple pedestrians pedestrian question walking the sidewalk, and the ego car is positioned at the intersection.
deternine how far the pedestrian in the black jacket with a white bag is from the ego car in the left view, we need consider the 30 location tensor provided that
pedestrian
for the left view is [-11, 2,
vehicles. The
The ego car is at the origin [0, 8, ej. The distance ego
calculated using the Euclidean distance formula in 30 space: W (Dtsst(D)stance) Vsqrt(x*2 y*2 2²2) W.
pedestrian
To estimate the distance, can use the perspective and relative size of the objects
Substituting the values from the tensor:
image. pedestrian appears to be few edge of the sidewalk, and the ego car is at
W (Dstace) \\sqrt((-11)2 + 2 (-3)
away from the
W]
W (text((Distance) VIsgrt(121
VM
intersection, which likely ieters ahead of the pedestrian. Considering typical city sidewalk widths and the apparent distance between the pedestrian and intersection, the estinated
W (DoxttDisstanc)) V1sqrt(134) Wi]
W (Ntext(Distance) Vappprox 11.58 Vitext( neters) W.
So, the pedestrian in the black jacket with white bag is approximately 11.58 meters away from the ego car in the view.</think>
distance is approximately meters. </tters.//ttkkk.
<answer>Sc/answen
X
Figure 10: Example responses of the baseline and Ego3D-VLM, VLM ego-centric absolute distance task.
23

---

Published as a conference paper at ICLR 2026
Object-Centric Absolute Distance (Meter)
Pronpt: These are five camera views mounted on
Front view: cinage>
<inage>
Left vlew: <image>
Front Right view: <image>
lRight view: cinage>
HHow far, in meters, is the postal delivery truck in the front left view from the silver sedan car Iturning right in the front right view?
joutput the thinking process in <think> </think> and final answer (number only) in <answer> </ </answer> taags.
Answer: 20.05
Left View
Right View
InternVL3-788
VL Miteaie
<Uhink>
<think
To deternine the distance between the postal delivery truck in the front left view and the silver
determine the distance between the postal delivery in front left and the silver sedan car turning right in the front right view, we need to analyze their respective
sedan car turning right the front right view,
need to consider the layout of the intersection and the positions of the vehicles.
"Identify Positions**:
1. *Poostll Delivery Truck in Front Left View:**
The postal delivery truck is on the left side of
the intersection, near the curb.
location: MM[-4, 2]11)/
**Silver Sedan Car in Front Right View:
intersection, turning right.
location: (V[14, 1, 8JVA)
2. **Estinate Intersection Dimensions**:
Typical city intersections can vary, meters.
Ne can calculate the Euclidean distance
width for city street about 10-12
between these two points using the formula: M (itex((Distance) Wsser(((X2 x_1) (y_2 y_1)²2 (z_2 z_1)"2} W.
Assuming the intersection is roughly square and about
wide, the diagonal distance across the intersection be estimated.
Plugging the coordinates: WI (Dtex((Distance) Vssrr(((14 (-4))²2 (1 0) + (8 2)* W.
**Calculate Distance**:
If the truck is near the left curb and the sedan
W ([text(Distance) \1sert((18)22 (1)
is turning right fron the opposite side, they
(6)^2)
are roughly across other.
intersection from each
W (itext(Distance) \1sqrt(324 36)
W
Using the Pythagorean theorem for rough estimate, each side the intersection is about meters, the diagonal distance would approximately WV Wigrrt(102 18^2)
W. (ttet(Distaacce) VIsart(361) W. W Vitet(Diistance) 19 (1]
Therefore, distance between postal delivery truck and the silver sedan is neters.</think>
\lsgrt(200) Wapprox 14.14 WV) meters.
Given these assumptions, the distance between the two vehicles is approximately 14 meters. meters.An</ttee \n</thinko
tanswer>19</answee
>14</answe>>
Figure 11: Example responses of the baseline and Ego3D-VLM on object-centric absolute distance task.
24

---

Published as a conference paper at ICLR 2026
Ego-Centric Absolute Distance (Multi-Choice)
PPompp: These five camera views
ego car
Front view: <image>
Front Left view: <image>
Left view: cimage>
Front Right view: <inage>
Right view: <image>
in motion the left view fron the ego car? IA.33 meters
[How far
[B.3 meters
JD.21 meters
(Output the thinking process in <think> </thinko and final answer (anly the letter of the choice) in
<enswer> </answer> tags.
Frant-Left View!
Front View
Front- Roont-Right VieW
Answer:
View
Right View!
Ego3D-VLM ntterit3-228
<think>
<think>
Okay, let's need to tigure out how far this car is from the ego car. The options are A. 33 neters, B. neters, c. 13 Ieterrs, and 21 meters.
left
motion.
To determine the distance of car the left view, we need to analyze the 30 location tensor provided: [[12.,, 2., -3.]'
motion from ego
First, I should consider the perspective of the left canera. Since it's mounted on the ego car, the distance to the car in the left view would be measured trom the ego car's position. The be the same road, moving in the same direction but in different lane. Looking
1. *"Understanding the Coordinates**
The x-coordinate (-12) indicates the horizontal position relative the ego
the left
appears
Negative means the is the left of the ego car.
very ego car. It's anead and the left. estimate based on typical lane widths and the apparent size of car, might be a neters away. However, the options don't have very close distance except for B. 3 neters, which seens too close. The car doesn't look like it's right next the ego car.
image,
The y-coordinate (2) indicates
vertical position, which cirectly relevant for the distance calculation this context.
The 2-coordinate (-3) indicates the position along the road. Negative means the car is behind the ego car. Distance**; The distance from the ego can be calculated using the Euclidean distance formula
2. **Cacuulttng
Option C. 13 meters seens plausible. It's moderate distance, not too far but not extremely close. Option possible,
it might too far considering the car is still visible in the left view. Option A. 33 neters seems too far because car clearly visible and not just small dot in the distance.
bit
VD \itext(Distance) Visgrt(x²2 y*2 z-2 WI
Plugging in the values: Wi'n Visqrt((-12)22 2n2 (-3)*2)
litext(Distance
Considering the typical speed and the visibility, meters might reasonable estimate. entirely sure, Maybe 21 neters is more accurate if the car further ahead. However, based on the image, the car appear be that better fit. \n</think>
\1sart(144 4 9) = (1sqrt(157)
\apppoox 1ltextf
meters) \1
3. **atching with Options**;
think
The closest option to 12,53 meters is meters. \n</think>>
IncanswersC</answer
<answer>cc/answer>
Figure 12: Example responses of the baseline and Ego3D-VLM on multi-choice ego-centric absolute distance task.
25

---

Published as a conference paper at ICLR 2026
Object-Centric Absolute Distance (Multi-Choice)
|Pronpt: These five
mounted ego
Front view: <image>
Front Left view: cinage>
Left cimage>
Front Right view: <inage>
RRght view: <image>
[Ho far car in motion in the front left view from car in motion the left view?
lA.19 meters
meters
IC.3 meters
[D.11 neters
joutput the thinking process in <think> </think> and final answer (only the letter of the choice)
(sanswer> </answer> tags.
Front-Left View!
Right
Answer:
Left
Right
IntemVL3-788
Ego3D-VLM ntterV13-720
<thānk>
"cthink>
To determine the distance between the front left view and the car in motion in the left view,
notion in the
To determine the distance between the cars,
calculate the Euclidean distance between their 30 coordinates.
need to consider the positions and relative distances these vehicles observed from the ego car's perspective.
"Identify the Positions**; The car in the front ieft left of the ego car, while the car in the left view is directly to the left of
1. **Front Left View Car Coordinates* W[[BB, 1, 12]W) *Leet Car Coordinates**; W[[-12, -3J())
ahead slightly
the ego car.
2. **Estimate Distances**:
The Euclidean distance formula 3D
The car in the front left view is likely further away the ego car compared to the car directly in the left view.
is:
W (DDxxt(Distance) (ssgt(((22 x_1)²2 (y_2 y_1)² (2_2 2_1)²2} WJ
The car in the left view is closer to the ego car, as it is directly adjacent.
**Compare
Options**;
Plugging in the coordinates:
Option A (19 meters) and Option B (29 meters) suggest significant distances, which night too far for typical closer.
WM (ttxt(Distaance Wssgr(((( 12)
(-8)) (2 1) ((-3)
urban highway scenarios where vehicles Option meters) is very close, which might be too near for confort in most driving situations.
12)^2) W/nM (\text(Distance)
\tsart((-4)22 (1) (-15)*2) Wi
W (text(Distancc) \lsgrt(16
Option (11 meters) seens a reasonable positions, especially
+ 2253 W. W (itext(Distance) = Vlsqrt(242)
vehicles in adjacent lanes they are noving in the same direction.
WJ
*CConclusion*: Given the typical spacing between vehicles traffic and the relative positions described, Option D (11 meters) appears to be the most plausible distance between the car in motion in the front left view the
ANI (itext(Distance) \lapprox 15.556
\itext( neters) W.
Since 15.556 neters 1s closest to 19 meters, the answer is A. Anc//think>
left view. \nc/think>
<answersD</answen
<answersAc/answer>
Figure 13: Example responses of the baseline and Fgo3D-VLM on multi-choice object-centric ab- solute distance task.
26

---

Published as a conference paper at ICLR 2026
Cbject-Centric Motion Reasoning
Prompt: are
ego
Front view: <inage>
Front Left view: cimage>
Left view: cimage>
Front Right view: cimage>
IRight view: <inage>
Ithe front view corresponds to the north direction. the minivan with skull sticker in the front Iview moves 2 neters east while all other objects remain stationary, does the minivan with skull Jsticker in the front view get closer to the two people walking in the front left view? Yes B. No
Output the thinking process in <think> </think> and final answer (only the letter of the choice )) canswer> c/answer> tags.
Annsurr: No
Front-Left View
Front Vicw
Tont- Bugnt View
Right View
Leit
InternVL3-788
Ego3D-VLM VLMaaarreL---
think>
<thinks
The front view shows
1. Understanding the Coordinates:
the minivan facing north.
The ego car is at the origin [e, 8, 0].
Positive x means right, negative means left.
Moving neters east
in front, negative 2. Initial Positions:
Positive
behind.
means the minivan moves the right from the front view perspective.
Minivan with skull sticker the front view: [1, 5].
Two people in the front left view: [-19, 8].
3. **Movemen of the Minivan:**
3. The front left view shows two people walking near the corner of the street.
The ninivan noves meters east, which means adding the x-
coordinate.
New position of the minivan: [1 2, 5] [3, 1, 5].
4. *Coompriing Positions:**
Moving from the front view would bring
Original x-coordinate
minivan: New x-coordinate of the minivan: 3.
the minivan closer the intersection and
x coordinate of the two people: -19.
**Distance Analysis:**
the area where the two
Initially, the minivan was
and the people were at =-19.
people are walking front left
After moving, the minivan is at x 3.
Ii
ninivan
people x-direction
5. Therefare, the minivan!
increases from 28 units (1 (-19)) 22 units (3 (-19)).
would get closer to people.
6. "CConclusion:**
away two people direction. \n</think>
ninivan
</think>
<answer>Yesc/answer>
cansuer>Moc/answer>>
Figure 14: Example responses of the bascline and Ego3D-VLM on object-centric motion reasoning task.
27

---

Published as a conference paper at ICLR 2026
Ego-Centric Motion Reasoning
Proupt: These are five cameea vlews mounted on an ego car.
Front view: cinage>
Front Left view: <image>
Left view: cinage>
Front Right view: <image>
IRight view: <inage>
The front view corresponds the north direction. If the ego car moves 5 meters north while all lother objects remain stationary, does the ego get closer the black in left lane tthe front left view?
(Output thinking process <thinks </thinks and final tags.
in </answer>
Front-Left
Answer: Yes
Laft View
Right View
InternVL3-788
Ego3D-VLM
<thānk>
<think>
To determine if the ego car gets closer the black car the left turn lane meters north, need to analyze the relative positions of the ego car and the black car the front left view.
deternine the ego
closer the black
the left turn lane when it moves meters north, need to
when it
analyze the relative positions the ego car and the black the front left view.
**Current Position Analysis**;
1. **Current Position Analysis**;
ego
origin [e, e, 8].
The ego car positioned the
The black car in the left turn lane is visible in the front left view.
front
the image.
The black car is in the left turn
lane, slightly to the left of the ego
*Movement of the Ego Car**:
Moving meters north means the ego car will move along positive
2. **Movement Analysis**:
(since positive in front of the ego car).
If the ego car moves 5 neters
north, it will shift its position further north along the road.
**Relative Position Change
The black car renains stationary in
moves north, it will be moving towards the front of the scene.
If the
left turn
3. "*Rllttive Position Change**;
turn lane is located the front left view, which means it is in the positive direction relative to the ego car's initial position.
black car in front of the ego
Moving north means the ega car will
nove away from the left side road where the black car located.
between ego car and the black will increase as the ego car noves north.
Therefore,
4. "CConclusion*:
Since the black car in front of ego car and moving north will bring the ego closer this front direction, the ego car will indeed get closer to the black car the left turn lane.
Based on this analysis, moving nerth will not bring the ego car closer to car; instead, it will move further away.
Therefore, the car will get closer to the black car in the left turn lane when it moves meters north.
</think>
canswersnor/answer>
canswer>yes</ansser
Figure 15: Example responses of the baseline and ELL3D-VLM on ego-centric motion reasoning task.
28

---

Published as a conference paper at ICLR 2026
Travel Time (Ego-Centrie)
|Pronpt: These are Five camera views mounted on en ego car.
[Frent <image>
Front Left view: cinage>
Left view: <image>
Frent Right
cinage>
Right view: <image>
nove the location ego car directly towards forange and delivery truck in the front view with the speed of m/s?
How
IA. Less than seconds
3-5 seconds
IC. 5-10 seconds
ID. More than 18 seconds
(Output thinking process <think </thinks and final (only the letter of choice)
<answer> </answer> tags.
Answer:
Frontseft Left View
Fraht VIEW
Front Bioh View
View
Right View
InternVL3-788
Ego3D-VLM P008D-VLMmeerrva11-220 13temv333375
<thinks
I1
To deternine the time it takes the ago car
II To determine the time it takes for the
to
the location of orange and
II move from its current location directly towards green delivery truck in the II view, need to calculate the distance between
delivery truck, we need to estinate the distance between the two vehicles.
orange
Il the ego car and the truck and then use the given Il speed to find time.
*"Estimate the distance**; In the front view, the ego car is directly behind the delivery truck.
il
appears roughly Il From the front view, the truck is directly in IIl front of the ego car, The location the tensor([A., 1., 10. Assuring the ego car's 30 location is [e, e, 0], the distance in the z-direction (which the forward direction)
length of lane, which typically around 6 neters.
is
time**; speed given n/s. Using the formula VV lltext(tine)
ego
units. AnknGiven the speed the ego car is n/s, can calculate the time usingl the formula:
(speed)) W))
WiD Vitext(time) Wfrrae( Witex(( neters)) \itext( m/s); 2.5 Witext( seconds)\n W1]
ll
Time Distance Speed
Time 10 units / m/s seconds Therefore,
This time falls within the range of 2-3 secands, which is closest to option (3-5 seconds).
seconds nove from thep location of the ego directly towards the location of the orange and green delivery truck.</think>
Therefore, the correct answer is B. B.c//thinno
Lanswer>Bc/answeer
canswerx<</answer>
Figure 16: Example responses the baseline and Ego3D-VLL VLM on travel time (ego-centric) task.
29

---

Published as a conference paper at ICLR 2026
Travel Time (Object-Centric)
iPronpt: These are five camera views nounted on an ego can.
Frant view: cinage>
Front Left view: <image>
Left vlew: <image>
Frontt Right view: <image>
Right view: cinage>
How nuch tire does take to nove from the location of the white delivery van in the front view directly towards the location the pedestrian standing brown jacket the right view Ithe speed of 2 n/s?
Less than seconds
K. 5-10 seconds
JE. More than 18 seconds
jOutput the thinking process in <think> </think> and final answer (only the letter of the choice) in
<answer> </answer> tags,
Answer:
Front-Left View
Left VIEW
Right
InternV13-788
Eg03D-VLM MLMDD--VMMMttm-521
<think>
<think>
To deternine the tine it
fron
To deternine the time it takes for the ego car to move delivery van front view to the location the pedestrian in the right view, we need to calculate the distance between these two points and then use the given speed of
the location of the white delivery van in
location
the front view the location of the pedestrian the right view, estinate the distance between these two points and then use the given speed of 2 n/s.
1. **Identify the locations:**
1. **Estinate the distance:** front shows van intersection.
white delivery
[15, 2, the front
Pedestrian in the right view: [8, 2, 3]
*Calculate the distance between the two points:**
The right view shows the pedestrian on
The x-ccordinate difference: [15
reters
the sidewalk near building.
The z-coordinate difference: [20 3) 17 meters
Visually, the distance
to roughly the length of city block, which typically around 189 neters.
The y-coardinate difference is 0, so it does not distance.
3. *use the Pythagorean theoren to find the distance:**
*"Calculate the time:**
Using the formula: time distance/ speee
Distance \u221a(7\u8eb2 17\u0eb2) \u221a(49
Distance 189 neters
289) \u221a338 (u2248 18.38 meters
Speed
4. *Calcullte the tine:**
Time 100 neters / n/s seconds
Distance Speed 18.38 meters m/s seconds
Giiven the options, 5e seconds falls into
seconds)" category.
Therefore, the correct answer 1s
Since 9.19 seconds falls within the range of 5-10 seconds, the
D. An</think>
A.An//thinko
canswer>D</answen>
<answer>C<</nnswer>
Figure 17: Example responses of the baseline and Ego3D-VVLM on travel time (object-centric) task.
30

---

Published as a conference paper at ICLR 2026
Prompt: These five Object-Centric Relative Distance
nounted ego
Front view: <inage>
Front Left view: <image>
Left view: cinage>
Front Right view: cimage>
leight view: cinage>
lkhich one closer to cyclist riding in the street in the front vlew?
JA. ego car
ja. three pedestrian
jotpput the thinking process <think </think> and final answer (only the letter of the choice) In canswer> </answer> tags.
Front-Le View
Front View
Frant Right View
Answer:
Left
Righl View
InternVL3-788
Ego3D-VLM 1122
<think
<think
cyclist the front view, need to compare the distances of the ego car and the three pedestrians in the left view fron the cyclist.
deternine which
To determine which is closer to the cyclist in the front view, we need compare the positions of the ego and three pedestrians in the left view relative to the cyclist.
1. **Ego Cart*: The cyclist is in front of the ego car, and they appeer to be the sane lane, indicating that the ego is directly behind the cyclist.
1. **Ego Position**; origin [0, 0, 0].
the 2. **Cyclist Position"*; The cyclist is in the front
ego
front view, which means they ego along the positive z-axis.
2. *TThree Pedestrians in the Left View**; The sidewalk
pedestrians are left view, which is different perspective from the front view. They are not in the
3. **Pedestrians Position**; The three pedestrians left view are located at [-17, 4, 4]. they
the left and slightly in front of the ego car.
same lane as the cyclist and appear further away terms of street distance.
since the cyclist directly in front of ego car and the pedestrians are to the left and not directly in the same path as the cyclist, the ego cyclist <answersAc/answer>
Given these observations, the ego car is closer cyclist than three pedestrians in the left view. \n</think>
car is closer are.</think>
pedestrians
Incanswer>A</ansseer
Figure 18: Example responses the baseline and Ego3D-VLM VLM on object-centric relative distance task.
31

---

Published as a conference paper at ICLR 2026
Ego-Centric Relative Distance
Prompt: These are five canera views nounted on an ego car.
Front view: <inage>
Front Left view: cimage>
Left view: <inage>
Front Right view: <image>
&Right vlew: cinage>
IIs the white car next to garbage hin in the front view claser to the ega car than the blue car Iparked the front right view?
Output the thinking process in <think> </thinks and final answer (yes or no) in canswer> </answer> tags.
Front-Left Left View
Front View!
Fent-fo View
Answer: No
View
Right View
InternVL3-788
Ego30-VLMatemtza
<think
<think>
deternine
garbage
To deternine if the white car the garbage
front view is closer to the than the blue parked
the front right view, we need to compare their positions relative
front view is closer to the ego car than the blue car parked in right view, need compare their 2-coordinates. The z-coordinate indicates the distance the ego car, higher z-value meaning further away.
ego car.
1. **Front View Analysis**:
The white car is parked on the right side of the street.
is positioned near blue garbage bin and black trash bin.
The garbage truck is on the left side of the street, indicating the ego car is facing forward.
**Front Right
Analysis**:
The blue car is parked in front of house with beige garage door.
The white in the front view has z-coordinate of 9. The blue car the front right coordinate of
The house numbers
positioning suggest further
down the street compared to the white
3. '"Relative Positioning**
In the front view, the white car is directly visible and appears closer to the ego
Since 3 (blue car) less than (white car), the blue is closer to the ego car than the white can, Ant/think)
In the front right view, the blue car is visible but appears
to be further down the street.
Based
the front closer to the car than the blue car in the front right view. \n</thinks
observations, the white
<answer>No /
<ansser>eec/ansseer
Figure 19: Example responses of the baseline and Eg03D-VLM ego-centric relative distance task.
32

---

Published as a conference paper at ICLR 2026
Localization Prompt: These are five camera views nounted car.
Front view: <image>
Front Left view:
Left view: cimage>
Front Right view: cinage>
Right view: cimage>
Ihee frant-view correspands to the north direction. If stand at the location of green parked car in facing north, is ego ny left, right, front, back?
Ithe
IA. Front
18. Back
IC. Right
10. Left
Outputt the thinking process in <think </thinko and final answer (only the letter of the choice) in
<answers </answer> tags.
Frant-Left View
FroneRight
Answer:
Left
Right View
InterVL3-778
Eg03D-VLM materi11-700
<think>
"cthink>
To determine the position of the ego car relative to the
The front-view is aligned with the
green parked car, let's analyze the camera views:
north direction.
shows green
1. **Front View**: Shows the parking lot with the building the right and the parking area on the left.
parked car and the ego car's position relative it.
**Left View*
the parking lot the green
proninently visible.
The ego is located thel ø], and the green parked car is at [-] 3, 1, -0] relative to the ego car. If facing north from green parked car, the car would be to the right because the x-coordinate of green
3. "*Front Left View**; Shows more of the parking lot and
green car from different angle.
**Front Right View**: Focuses the building entrance.
5. **Right View**; Shows the entrance with banner.
If you standing the location of the green parked facing north (towards the building), the ego can would be in the parking area to your left, as seen from the left and
negative, indicating car.
it is to the left of the 5. Therefore, the ego car is to the right the green parked car facing north. \ne/think>
front left
Therefore, the ego car is to left.</think
<answer>D</anssee DD/anseer>
canswersC</answer>
Figure 20: Example responses of the baseline and Ego3D-VLM on localization task.
33


Reviews:
[{"review_id": "l9edw9WP0j", "reviewer": "Reviewer_5Jz1", "summary": "In contrast to prior passive perception in static indoor scenes, this paper presents Ego3D-Bench, a benchmark for evaluating 3D spatial reasoning in ego-centric multi-view outdoor scenes, which comprises 8,600+ QA pairs from nuScenes, Waymo, and Argoverse. The authors also propose Ego3D-VLM, a framework that enhances VLMs via a textual cognitive map. Experiments on 16 SOTA VLMs (e.g., GPT-4o, InternVL3) show a large performance gap between humans and VLMs; and Ego3D-VLM narrows this gap, achieving 12% higher accuracy on multi-choice QA and 56% lower RMSE on absolute distance estimation, with adaptability to diverse multi-view scenarios. Moreover, the authors conduct detailed ablation studies regarding various model classes and model components.", "questions": "- What is the usage of scaled 3D points (Line 267)? Seems like it is used by default, rather than the global points, but the equation (4) is not.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike prior benchmarks (single-image/indoor video), this paper addresses a critical real-world gap by proposing Ego3D-Bench, which targets ego-centric multi-view outdoor scenarios, aligning with embodied AI needs such as robotics and autonomous driving.\n- Ego3D-VLM is efficient and flexible. The textual cognitive map can effectively represent spatial information to aid the 2D visual input. And the experimental results turn out to be very effective.\n- Extensive experiments and rigorous evaluations. This paper tests 16 kinds of VLMs (including generalist, 3D-specific, and depth+REC) and delivers ablation studies regarding various model components.", "weaknesses": "- The reliance on external perception modules (depth + REC) may complicate the inference pipeline. And such a design cannot train the inherent spatial perception capability of VLM in an end-to-end manner. It is more like providing extra input information by external modules though it does help.\n- Relatively weak transfer results to other scenes (All-angles and VSI-Bench). This raises my concerns on the generalizability of the proposed method. How could you address this issue, by polishing the methodology, or integrating more data?\n- The claim of \"post-training\" framework can be misleading. Post-training often relates to RL or RFT. However, I found no RL-related objective. This paper appears to only involve creating new data and SFT with that data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yLqFvF6R4v", "reviewer": "Reviewer_f2RE", "summary": "This paper introduces Ego3D-Bench, a new benchmark designed to evaluate the 3D spatial reasoning capabilities of VLMs using ego-centric, multi-view images from outdoor driving scenes. Comprising over 8,600 quality-controlled question-answer pairs, the benchmark reveals a significant performance gap between humans and state-of-the-art VLMs. To address this gap, the authors propose Ego3D-VLM, a training-free framework that enhances a given VLM's spatial understanding by generating a textual cognitive map. This map integrates 3D coordinate information of referred objects from multiple camera views and is provided as additional context to the VLM. Experimental results demonstrate that Ego3D-VLM consistently improves the performance of various VLMs on the proposed benchmark. The main contributions are the Ego3D-Bench benchmark and the plug-and-play Ego3D-VLM enhancement method.", "questions": "see weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The focus on ego-centric multi-view fusion is well-aligned with the needs of embodied AI and vehicular applications. The significant human-model performance gap observed on the benchmark provides a valuable measuring stick for future research.\n\n2. The method of constructing a textual/JSON cognitive map at inference time using tool-based models is straightforward and effective. The comparative analysis of different cognitive map formats also offers useful insights.\n\n3. The paper is well-structured, and the figures and examples are clear and illustrative.", "weaknesses": "1. Line 77: The claim that \"Ego3D-Bench is the first benchmark to evaluate spatial reasoning of VLMs given ego-centric multi-view inputs\" overstates its novelty. The related work section itself acknowledges existing multi-view benchmarks (e.g., All-Angle Bench, VSI-Bench), differing primarily in their setup. The contribution would be more accurately framed as a valuable extension within the specific ego-centric setting. The authors should more clearly articulate the unique challenges and benefits inherent to the ego-centric paradigm.\n\n2. Line 86: The claim that point-cloud/BEV methods \"significantly increase inference time—often by a factor of ten\" is not sufficiently supported. The cited work (Zhu et al., 2024) primarily deals with indoor scenes. In outdoor autonomous driving contexts, point clouds are sparser, and numerous real-time methods exist. The authors provide no direct, comparable efficiency comparison with such methods. Furthermore, Table 11 shows that Ego3D-VLM itself introduces non-negligible latency.\n\n3. Line 89: Describing Ego3D-VLM as a \"post-training method\" is a misnomer. The approach is fundamentally an inference-time augmentation or a form of prompt engineering.\n\n4. Line 266: The global scaling method based on common-sense object heights (e.g., person ≈ 1.7m) is highly susceptible to errors in dynamic traffic scenes due to pose, occlusion, and detection inaccuracies. The support for its reliability is limited (a single RMSE improvement in ablations), lacking analysis of error propagation or failure cases.\n\n5. Table 8: While the results indicate textual/JSON cognitive maps outperform visual ones, the authors provide no analysis for why this is the case. An investigation into the reasons behind this performance difference (e.g., alignment with pre-training, information density) is missing.\n\n6. The core concept of a \"cognitive map\" as a structured representation to aid spatial reasoning is not novel and bears similarity to ideas in prior works like VSI-Bench. The distinction and specific advancements of the proposed cognitive map should be more clearly delineated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdChFA83Pg", "reviewer": "Reviewer_ecVi", "summary": "The paper introduces Ego3D-Bench, a novel benchmark for evaluating the spatial reasoning capabilities of VLMs using ego-centric, multi-view outdoor data. It evaluates 16 SOTA VLMs on this new standard. Furthermore, the paper proposes Ego3D-VLM, a post-training framework that employs a cognitive map derived from 3D coordinates to enhance 3D spatial reasoning significantly. The experimental results show that this framework achieves clear and substantial improvements in performance on multi-choice QA and absolute distance estimation tasks.", "questions": "1. In the Ego3D-Bench construction pipeline, annotators write descriptive captions for unique objects. Do these captions follow specific structural or linguistic templates, and how is the objectivity, reliability, and unbiased nature of these captions ensured?\n2. Given the complex, multi-step process (multi-view images → 3D coordinates → 2D textual cognitive map), can the authors provide an intuitive explanation for why the Ego3D-VLM framework proves to be effective? What is the core cognitive insight this result provides regarding how VLMs process spatial information?\n3. The results in Table 1 show that VLM performance on Ego-Centric (Ego.) vs. Object-Centric (Obj.) Absolute Distance, Motion, and Relative Distance tasks is not consistently worse for Ego-Centric problems. Does this result contradict the expected challenge of Ego-centric problems, and if not, how should this be interpreted?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The Ego3D-Bench is highly valuable as its focus on outdoor, ego-centric data directly addresses and fills a critical gap in the current landscape of VLM spatial understanding benchmarks.\n2. The proposed post-training framework, Ego3D-VLM, is shown to be effective, adaptable across various VLM architectures, and demonstrably boosts the spatial reasoning abilities of several SOTA models.\n3. The paper provides a comprehensive and convincing experimental analysis of Ego3D-VLM, covering different VLM families, comparisons against other benchmarks, and inference time analysis in both the main body and the appendix, making the claims credible.", "weaknesses": "1. The paper lacks sufficient transparency regarding the human effort involved in the benchmark construction. Specifically, it should disclose the origin of the hired annotators, the total number of human hours dedicated to the Ego3D-Bench creation pipeline, and the compensation structure used for the annotators.\n2. The Ego3D-VLM explicitly reconstructs 3D points based on estimated depth, meaning the robustness of the entire pipeline is highly dependent on the accuracy of an external, potentially unreliable, outdoor metric depth estimator. The overall pipeline involves a seemingly over-complex process (multi-view images → 3D reconstruction → 2D cognitive map). The intuitive justification for mapping 3D information *back* to a 2D text representation seems questionable without further theoretical justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wge4hmcV8v", "reviewer": "Reviewer_QSvq", "summary": "the paper introduces Ego3D-Bench, a new benchmark with over 8,600 human-annotated QA pairs built from outdoor datasets like NuScenes, Waymo, and Argoverse 1. The benchmark tests five categories of spatial tasks from both ego-centric and object-centric perspectives, including absolute/relative distance, localization, motion reasoning, and travel time. An evaluation of 16 SOTA VLMs reveals a significant performance gap compared to human-level scores. To bridge this gap, the paper proposes Ego3D-VLM, a plug-and-play post-training framework. This method uses external REC and metric depth estimation models to identify objects and their 3D coordinates , transforms them into a global reference frame , and generates a \"textual cognitive map\" that is fed to the VLM as additional context along with the images. The authors report that this framework improves average multi-choice QA accuracy by 12% and absolute distance estimation by 56%.", "questions": "Please refer to the weakness.\n\nWill this dataset be released?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposes Ego3D-Bench, which is well-constructed with over 8,600 QA pairs. It's built on real-world datasets. Critically, it also involves human annotators for filtering, captioning, and final review, ensuring high-quality, non-trivial questions.\n\n2. The evaluation is thorough, testing 16 SOTA VLMs with important ablations.\n\n3. The paper proposes Ego3D-VLM, a plug-in framework. This approach generates a textual cognitive map, is presented as an efficient alternative to computationally expensive methods like full point-cloud reconstruction. Even if it did not improve the spatially reasoning ability of the base VLM.\n\n4. The paper is well-written and easy to follow.", "weaknesses": "1. A minor problem is that the proposed Ego3D-VLM is repeatedly called a \"post-training framework\", which is misleading. Post-training usually implies fine-tuning or adapting model weights. The Ego3D-VLM framework is a purely inference-time pipeline that prepends the cognitive map to guide a frozen, underlying VLM.\n\n2. The Ego3D-VLM is essentially transforming the 3D vision problem into a textual reasoning problem. As the qualitative examples show in Fig. 9, 10, 11, the baseline VLM is visually analyzing distances while the Ego3D-VLM version is simply given the 3D coordinates in text and performs Euclidean distance calculations. So the improvement is not on the VLM's visual spatial reasoning ability, but more like a work-around, that reads more accurate answers from a pre-processed text map. \n\n3. The paper can be improved by providing more detailed information about the dataset. Like the distribution of types of objects involved (people, cars, or buildings). And some details on how many works human annotators need to be involved (what percentage of data needs to be filtered out?)\n\n4. For egocentric VQA questions, what if instead of giving each image separately as input, we concat them next to each other to form one larger picture as input? what will happen in this case? Will it be better than inputing several images at the same time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)