VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
Authors: name: Shijie Zhou, name: Alexander Vilesov, name: Xuehai He, name: Ziyu Wan, name: Shuwang Zhang, name: Aditya Nagachandra, name: Di Chang, name: Dongdong Chen, name: Xin Eric Wang, name: Achuta Kadambi

Vision language models (VLMs) have shown remarkable capabilities in
integrating linguistic and visual reasoning but remain fundamentally limited in
understanding dynamic spatiotemporal interactions. Humans effortlessly track
and reason about object movements, rotations, and perspective shifts-abilities
essential for robust dynamic real-world understanding yet notably lacking in
current VLMs. In this paper, we introduce VLM4D, the first benchmark
specifically designed to evaluate the spatiotemporal reasoning capabilities of
VLMs. Our benchmark comprises diverse real-world and synthetic videos
accompanied by carefully curated question-answer pairs emphasizing
translational and rotational motions, perspective awareness, and motion
continuity. Through comprehensive evaluations of state-of-the-art open and
closed-source VLMs, we identify significant performance gaps compared to human
baselines, highlighting fundamental deficiencies in existing models. Extensive
analysis reveals that VLMs struggle particularly with integrating multiple
visual cues and maintaining temporal coherence. We further explore promising
directions, such as leveraging 4D feature field reconstruction and targeted
spatiotemporal supervised fine-tuning, demonstrating their effectiveness in
enhancing spatiotemporal comprehension. Our work aims to encourage deeper
exploration into improving VLMs' spatial and temporal grounding, paving the way
towards more capable and reliable visual intelligence for dynamic environments.