TinySAM: Pushing the Envelope for Efficient Segment Anything Model
Authors: name: Han Shu; name: Wenshuo Li; name: Yehui Tang; name: Yiman Zhang; name: Yihao Chen; name: Houqiang Li; name: Yunhe Wang; name: Xinghao Chen
Published: 2023-12-21 12:26:11

Abstract: Recently segment anything model (SAM) has shown powerful segmentation
capability and has drawn great attention in computer vision fields. Massive
following works have developed various applications based on the pretrained SAM
and achieved impressive performance on downstream vision tasks. However, SAM
consists of heavy architectures and requires massive computational capacity,
which hinders the further application of SAM on computation constrained edge
devices. To this end, in this paper we propose a framework to obtain a tiny
segment anything model (TinySAM) while maintaining the strong zero-shot
performance. We first propose a full-stage knowledge distillation method with
online hard prompt sampling strategy to distill a lightweight student model. We
also adapt the post-training quantization to the promptable segmentation task
and further reduce the computational cost. Moreover, a hierarchical segmenting
everything strategy is proposed to accelerate the everything inference by
2times with almost no performance degradation. With all these proposed
methods, our TinySAM leads to orders of magnitude computational reduction and
pushes the envelope for efficient segment anything task. Extensive experiments
on various zero-shot transfer tasks demonstrate the significantly advantageous
performance of our TinySAM against counterpart methods. Pre-trained models and
codes will be available at https://github.com/xinghaochen/TinySAM and
https://gitee.com/mindspore/models/tree/master/research/cv/TinySAM.