Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
Authors: name: Chaoning Zhang; name: Dongshen Han; name: Yu Qiao; name: Jung Uk Kim; name: Sung-Ho Bae; name: Seungkyu Lee; name: Choong Seon Hong
Published: 2023-06-25 16:37:25

Abstract: Segment anything model (SAM) is a prompt-guided vision foundation model for
cutting out the object of interest from its background. Since Meta research
team released the SA project, SAM has attracted significant attention due to
its impressive zero-shot transfer performance and high versatility of being
compatible with other models for advanced vision applications like image
editing with fine-grained control. Many of such use cases need to be run on
resource-constraint edge devices, like mobile Apps. In this work, we aim to
make SAM mobile-friendly by replacing the heavyweight image encoder with a
lightweight one. A naive way to train such a new SAM as in the original SAM
paper leads to unsatisfactory performance, especially when limited training
sources are available. We find that this is mainly caused by the coupled
optimization of the image encoder and mask decoder, motivated by which we
propose decoupled distillation. Concretely, we distill the knowledge from the
image encoder ViT-H in the original SAM to a lightweight image encoder, which
can be automatically compatible with the mask decoder in the original SAM. The
training can be completed on a single GPU within less than one day, and the
resulting lightweight SAM is termed MobileSAM which is more than 60 times
smaller yet performs on par with the original SAM. For inference speed,
MobileSAM runs around 10ms per image: 8ms on the image encoder and 2ms on the
mask decoder. With superior performance and a higher versatility, our MobileSAM
is 7 times smaller and 4 times faster than the concurrent FastSAM, making it
more suitable for mobile applications. The code for MobileSAM project is
provided at https://github.com/ChaoningZhang/MobileSAM