MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Authors: name: Erik Daxberger, name: Nina Wenzel, name: David Griffiths, name: Haiming Gang, name: Justin Lazarow, name: Gefen Kohavi, name: Kai Kang, name: Marcin Eichner, name: Yinfei Yang, name: Afshin Dehghan, name: Peter Grasch

Multimodal large language models (MLLMs) excel at 2D visual understanding but
remain limited in their ability to reason about 3D space. In this work, we
leverage large-scale high-quality 3D scene data with open-set annotations to
introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation
benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data
covers diverse spatial tasks including spatial relationship prediction, metric
size and distance estimation, and 3D grounding. We show that CA-VQA enables us
to train MM-Spatial, a strong generalist MLLM that also achieves
state-of-the-art performance on 3D spatial understanding benchmarks, including
our own. We show how incorporating metric depth and multi-view inputs (provided
in CA-VQA) can further improve 3D understanding, and demonstrate that data
alone allows our model to achieve depth perception capabilities comparable to
dedicated monocular depth estimation models. We will publish our SFT dataset
and benchmark.