SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial
  Understanding
Authors: name: Haoxiang Wang; name: Pavan Kumar Anasosalu Vasu; name: Fartash Faghri; name: Raviteja Vemulapalli; name: Mehrdad Farajtabar; name: Sachin Mehta; name: Mohammad Rastegari; name: Oncel Tuzel; name: Hadi Pouransari
Published: 2023-10-23 19:21:57

Abstract: The landscape of publicly available vision foundation models (VFMs), such as
CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed
with distinct capabilities stemming from their pre-training objectives. For
instance, CLIP excels in semantic understanding, while SAM specializes in
spatial understanding for segmentation. In this work, we introduce a simple
recipe to efficiently merge VFMs into a unified model that assimilates their
expertise. Our proposed method integrates multi-task learning, continual
learning techniques, and teacher-student distillation. This strategy entails
significantly less computational cost compared to traditional multi-task
training from scratch. Additionally, it only demands a small fraction of the
pre-training datasets that were initially used to train individual models. By
applying our method to SAM and CLIP, we derive SAM-CLIP: a unified model that
amalgamates the strengths of SAM and CLIP into a single backbone, making it apt
for edge device applications. We show that SAM-CLIP learns richer visual
representations, equipped with both localization and semantic features,
suitable for a broad range of vision tasks. SAM-CLIP obtains improved
performance on several head probing tasks when compared with SAM and CLIP. We
further show that SAM-CLIP not only retains the foundational strengths of its
precursor models but also introduces synergistic functionalities, most notably
in zero-shot semantic segmentation, where SAM-CLIP establishes new
state-of-the-art results on 5 benchmarks. It outperforms previous models that
are specifically designed for this task by a large margin, including +6.8% and
+5.9% mean IoU improvement on Pascal-VOC and COCO-Stuff datasets, respectively.