VideoMolmo: Spatio-Temporal Grounding Meets Pointing
Authors: name: Ghazi Shazan Ahmad, name: Ahmed Heakl, name: Hanan Gani, name: Abdelrahman Shaker, name: Zhiqiang Shen, name: Ranjay Krishna, name: Fahad Shahbaz Khan, name: Salman Khan
Organization: Mohamed Bin Zayed University of Artificial Intelligence

Spatio-temporal localization is vital for precise interactions across diverse
domains, from biological research to autonomous navigation and interactive
interfaces. Current video-based approaches, while proficient in tracking, lack
the sophisticated reasoning capabilities of large language models, limiting
their contextual understanding and generalization. We introduce VideoMolmo, a
large multimodal model tailored for fine-grained spatio-temporal pointing
conditioned on textual descriptions. Building upon the Molmo architecture,
VideoMolmo incorporates a temporal module utilizing an attention mechanism to
condition each frame on preceding frames, ensuring temporal consistency.
Additionally, our novel temporal mask fusion pipeline employs SAM2 for
bidirectional point propagation, significantly enhancing coherence across video
sequences. This two-step decomposition, i.e., first using the LLM to generate
precise pointing coordinates, then relying on a sequential mask-fusion module
to produce coherent segmentation, not only simplifies the task for the language
model but also enhances interpretability. Due to the lack of suitable datasets,
we curate a comprehensive dataset comprising 72k video-caption pairs annotated
with 100k object points. To evaluate the generalization of VideoMolmo, we
introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five
real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving,
Video-GUI Interaction, and Robotics. We also evaluate our model on Referring
Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to
existing models, VideoMolmo substantially improves spatio-temporal pointing
accuracy and reasoning capability. Our code and models are publicly available
at https://github.com/mbzuai-oryx/VideoMolmo.