Towards Embodied Cognition in Robots via Spatially Grounded Synthetic
  Worlds
Authors: name: Joel Currie; name: Gioele Migno; name: Enrico Piacenti; name: Maria Elena Giannaccini; name: Patric Bach; name: Davide De Tommaso; name: Agnieszka Wykowska
Published: 2025-05-20 13:49:09

Abstract: We present a conceptual framework for training Vision-Language Models (VLMs)
to perform Visual Perspective Taking (VPT), a core capability for embodied
cognition essential for Human-Robot Interaction (HRI). As a first step toward
this goal, we introduce a synthetic dataset, generated in NVIDIA Omniverse,
that enables supervised learning for spatial reasoning tasks. Each instance
includes an RGB image, a natural language description, and a ground-truth 4X4
transformation matrix representing object pose. We focus on inferring Z-axis
distance as a foundational skill, with future extensions targeting full 6
Degrees Of Freedom (DOFs) reasoning. The dataset is publicly available to
support further research. This work serves as a foundational step toward
embodied AI systems capable of spatial understanding in interactive human-robot
scenarios.