Segment Anything with Multiple Modalities
Authors: name: Aoran Xiao; name: Weihao Xuan; name: Heli Qi; name: Yun Xing; name: Naoto Yokoya; name: Shijian Lu
Published: 2024-08-17 03:45:40

Abstract: Robust and accurate segmentation of scenes has become one core functionality
in various visual recognition and navigation tasks. This has inspired the
recent development of Segment Anything Model (SAM), a foundation model for
general mask segmentation. However, SAM is largely tailored for single-modal
RGB images, limiting its applicability to multi-modal data captured with
widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal
plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that
supports cross-modal and multi-modal processing for robust and enhanced
segmentation with different sensor suites. MM-SAM features two key designs,
namely, unsupervised cross-modal transfer and weakly-supervised multi-modal
fusion, enabling label-efficient and parameter-efficient adaptation toward
various sensor modalities. It addresses three main challenges: 1) adaptation
toward diverse non-RGB sensors for single-modal processing, 2) synergistic
processing of multi-modal data via sensor fusion, and 3) mask-free training for
different downstream tasks. Extensive experiments show that MM-SAM consistently
outperforms SAM by large margins, demonstrating its effectiveness and
robustness across various sensors and data modalities.