Surgical SAM 2: Real-time Segment Anything in Surgical Video by
  Efficient Frame Pruning
Authors: name: Haofeng Liu; name: Erli Zhang; name: Junde Wu; name: Mingxuan Hong; name: Yueming Jin
Published: 2024-08-15 04:59:12

Abstract: Surgical video segmentation is a critical task in computer-assisted surgery
and is vital for enhancing surgical quality and patient outcomes. Recently, the
Segment Anything Model 2 (SAM2) framework has shown superior advancements in
image and video segmentation. However, SAM2 struggles with efficiency due to
the high computational demands of processing high-resolution images and complex
and long-range temporal dynamics in surgical videos. To address these
challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to
utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate
real-time surgical video segmentation. The EFP mechanism dynamically manages
the memory bank by selectively retaining only the most informative frames,
reducing memory usage and computational cost while maintaining high
segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2
significantly improves both efficiency and segmentation accuracy compared to
the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3times FPS compared with
SAM2, while also delivering state-of-the-art performance after fine-tuning with
lower-resolution data. These advancements establish SurgSAM-2 as a leading
model for surgical video analysis, making real-time surgical video segmentation
in resource-constrained environments a feasible reality.