Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of
  Images and Videos
Authors: name: Haobo Yuan; name: Xiangtai Li; name: Tao Zhang; name: Zilong Huang; name: Shilin Xu; name: Shunping Ji; name: Yunhai Tong; name: Lu Qi; name: Jiashi Feng; name: Ming-Hsuan Yang
Published: 2025-01-07 18:58:54

Abstract: This work presents Sa2VA, the first unified model for dense grounded
understanding of both images and videos. Unlike existing multi-modal large
language models, which are often limited to specific modalities and tasks,
Sa2VA supports a wide range of image and video tasks, including referring
segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA
combines SAM-2, a foundation video segmentation model, with LLaVA, an advanced
vision-language model, and unifies text, image, and video into a shared LLM
token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2
in producing precise masks, enabling a grounded, multi-modal understanding of
both static and dynamic visual content. Additionally, we introduce Ref-SAV, an
auto-labeled dataset containing over 72k object expressions in complex video
scenes, designed to boost model performance. We also manually validate 2k video
objects in the Ref-SAV datasets to benchmark referring video object
segmentation in complex environments. Experiments show that Sa2VA achieves
state-of-the-art across multiple tasks, particularly in referring video object
segmentation, highlighting its potential for complex real-world applications.