Thinking in Space: How Multimodal Large Language Models See, Remember,
  and Recall Spaces
Authors: name: Jihan Yang; name: Shusheng Yang; name: Anjali W. Gupta; name: Rilyn Han; name: Li Fei-Fei; name: Saining Xie
Published: 2024-12-18 18:59:54

Abstract: Humans possess the visual-spatial intelligence to remember spaces from
sequential visual observations. However, can Multimodal Large Language Models
(MLLMs) trained on million-scale video datasets also ``think in space'' from
videos? We present a novel video-based visual-spatial intelligence benchmark
(VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit
competitive - though subhuman - visual-spatial intelligence. We probe models to
express how they think in space both linguistically and visually and find that
while spatial reasoning capabilities remain the primary bottleneck for MLLMs to
reach higher benchmark performance, local world models and spatial awareness do
emerge within these models. Notably, prevailing linguistic reasoning techniques
(e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve
performance, whereas explicitly generating cognitive maps during
question-answering enhances MLLMs' spatial distance ability.