Efficient Track Anything
Authors: name: Yunyang Xiong; name: Chong Zhou; name: Xiaoyu Xiang; name: Lemeng Wu; name: Chenchen Zhu; name: Zechun Liu; name: Saksham Suri; name: Balakrishnan Varadarajan; name: Ramya Akula; name: Forrest Iandola; name: Raghuraman Krishnamoorthi; name: Bilge Soran; name: Vikas Chandra
Published: 2024-11-28 05:52:10

Abstract: Segment Anything Model 2 (SAM 2) has emerged as a powerful tool for video
object segmentation and tracking anything. Key components of SAM 2 that drive
the impressive video object segmentation performance include a large multistage
image encoder for frame feature extraction and a memory mechanism that stores
memory contexts from past frames to help current frame segmentation. The high
computation complexity of multistage image encoder and memory module has
limited its applications in real-world tasks, e.g., video object segmentation
on mobile devices. To address this limitation, we propose EfficientTAMs,
lightweight track anything models that produce high-quality results with low
latency and model size. Our idea is based on revisiting the plain,
nonhierarchical Vision Transformer (ViT) as an image encoder for video object
segmentation, and introducing an efficient memory module, which reduces the
complexity for both frame feature extraction and memory computation for current
frame segmentation. We take vanilla lightweight ViTs and efficient memory
module to build EfficientTAMs, and train the models on SA-1B and SA-V datasets
for video object segmentation and track anything tasks. We evaluate on multiple
video segmentation benchmarks including semi-supervised VOS and promptable
video segmentation, and find that our proposed EfficientTAM with vanilla ViT
perform comparably to SAM 2 model (HieraB+SAM 2) with ~2x speedup on A100 and
~2.4x parameter reduction. On segment anything image tasks, our EfficientTAMs
also perform favorably over original SAM with ~20x speedup on A100 and ~20x
parameter reduction. On mobile devices such as iPhone 15 Pro Max, our
EfficientTAMs can run at ~10 FPS for performing video object segmentation with
reasonable quality, highlighting the capability of small models for on-device
video object segmentation applications.