Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View
  Scenes
Authors: name: Mohsen Gholami, name: Ahmad Rezaei, name: Zhou Weimin, name: Yong Zhang, name: Mohammad Akbari
Organization: Huawei's Vancouver AI Lab

Understanding 3D spatial relationships remains a major limitation of current
Vision-Language Models (VLMs). Prior work has addressed this issue by creating
spatial question-answering (QA) datasets based on single images or indoor
videos. However, real-world embodied AI agents such as robots and self-driving
cars typically rely on ego-centric, multi-view observations. To this end, we
introduce Ego3D-Bench, a new benchmark designed to evaluate the spatial
reasoning abilities of VLMs using ego-centric, multi-view outdoor data.
Ego3D-Bench comprises over 8,600 QA pairs, created with significant involvement
from human annotators to ensure quality and diversity. We benchmark 16 SOTA
VLMs, including GPT-4o, Gemini1.5-Pro, InternVL3, and Qwen2.5-VL. Our results
reveal a notable performance gap between human level scores and VLM
performance, highlighting that current VLMs still fall short of human level
spatial understanding. To bridge this gap, we propose Ego3D-VLM, a
post-training framework that enhances 3D spatial reasoning of VLMs. Ego3D-VLM
generates cognitive map based on estimated global 3D coordinates, resulting in
12% average improvement on multi-choice QA and 56% average improvement on
absolute distance estimation. Ego3D-VLM is modular and can be integrated with
any existing VLM. Together, Ego3D-Bench and Ego3D-VLM offer valuable tools for
advancing toward human level spatial understanding in real-world, multi-view
environments.