Title: Visual Spatial Tuning
Primary area: foundation or frontier models, including LLMs
Keywords: vision-language model; spatial understanding

Abstract: Capturing spatial relationships from visual inputs is a cornerstone of human-like general intelligence. Several previous studies have tried to enhance the spatial awareness of Vision-Language Models (VLMs) by adding extra expert encoders, which brings extra overhead and usually harms general capabilities. To enhance the spatial ability in general architectures, we introduce Visual Spatial Tuning (VST), a comprehensive framework to cultivate VLMs with human-like visuospatial abilities, from spatial perception to reasoning.
We first try to enhance spatial perception in VLMs by constructing a large-scale dataset termed VST-P, which includes 4.1M samples spanning 19 skills across single view, multiple images, and videos. 
Then, we present VST-R, a curated dataset with 135K samples that instruct models to reason in space. In particular, we adopt a progressive training pipeline: supervised fine-tuning to build foundational spatial knowledge, followed by reinforcement learning to improve spatial reasoning abilities further. 
Without the side-effect to general capabilities, the proposed VST consistently achieves state-of-the-art results on several spatial benchmarks, including $34.8\%$ on MMSI-Bench and $61.2\%$ on VSI-Bench. 
It turns out that the Vision-Language-Action models can be significantly enhanced with the proposed spatial tuning paradigm, paving the way for more physically grounded AI.

TL;DR: We present a comprehensive framework that builds fundational visuospatial abilities in vision-language models.