MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
Authors: name: Zhenyu Pan; name: Han Liu
Published: 2025-03-24 09:18:01

Abstract: We present MetaSpatial, the first reinforcement learning (RL)-based framework
designed to enhance 3D spatial reasoning in vision-language models (VLMs),
enabling real-time 3D scene generation without the need for hard-coded
optimizations. MetaSpatial addresses two core challenges: (i) the lack of
internalized 3D spatial reasoning in VLMs, which limits their ability to
generate realistic layouts, and (ii) the inefficiency of traditional supervised
fine-tuning (SFT) for layout generation tasks, as perfect ground truth
annotations are unavailable. Our key innovation is a multi-turn RL-based
optimization mechanism that integrates physics-aware constraints and rendered
image evaluations, ensuring generated 3D layouts are coherent, physically
plausible, and aesthetically consistent. Methodologically, MetaSpatial
introduces an adaptive, iterative reasoning process, where the VLM refines
spatial arrangements over multiple turns by analyzing rendered outputs,
improving scene coherence progressively. Empirical evaluations demonstrate that
MetaSpatial significantly enhances the spatial consistency and formatting
stability of various scale models. Post-training, object placements are more
realistic, aligned, and functionally coherent, validating the effectiveness of
RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game
development applications. Our code, data, and training pipeline are publicly
available at https://github.com/PzySeere/MetaSpatial.