SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a
  Training-Free Memory Tree
Authors: name: Shuangrui Ding; name: Rui Qian; name: Xiaoyi Dong; name: Pan Zhang; name: Yuhang Zang; name: Yuhang Cao; name: Yuwei Guo; name: Dahua Lin; name: Jiaqi Wang
Published: 2024-10-21 17:59:19

Abstract: The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation
model for object segmentation in both images and videos, paving the way for
various downstream video applications. The crucial design of SAM 2 for video
segmentation is its memory module, which prompts object-aware memories from
previous frames for current frame prediction. However, its greedy-selection
memory design suffers from the "error accumulation" problem, where an errored
or missed mask will cascade and influence the segmentation of the subsequent
frames, which limits the performance of SAM 2 toward complex long-term videos.
To this end, we introduce SAM2Long, an improved training-free video object
segmentation strategy, which considers the segmentation uncertainty within each
frame and chooses the video-level optimal results from multiple segmentation
pathways in a constrained tree search manner. In practice, we maintain a fixed
number of segmentation pathways throughout the video. For each frame, multiple
masks are proposed based on the existing pathways, creating various candidate
branches. We then select the same fixed number of branches with higher
cumulative scores as the new pathways for the next frame. After processing the
final frame, the pathway with the highest cumulative score is chosen as the
final segmentation result. Benefiting from its heuristic search design,
SAM2Long is robust toward occlusions and object reappearances, and can
effectively segment and track objects for complex long-term videos. Notably,
SAM2Long achieves an average improvement of 3.0 points across all 24
head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term
video object segmentation benchmarks such as SA-V and LVOS. The code is
released at https://github.com/Mark12Ding/SAM2Long.