Segment Anything with Multiple Modalities
Authors: name: Aoran Xiao, name: Weihao Xuan, name: Heli Qi, name: Yun Xing, name: Naoto Yokoya, name: Shijian Lu

Robust and accurate segmentation of scenes has become one core functionality
in various visual recognition and navigation tasks. This has inspired the
recent development of Segment Anything Model (SAM), a foundation model for
general mask segmentation. However, SAM is largely tailored for single-modal
RGB images, limiting its applicability to multi-modal data captured with
widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal
plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that
supports cross-modal and multi-modal processing for robust and enhanced
segmentation with different sensor suites. MM-SAM features two key designs,
namely, unsupervised cross-modal transfer and weakly-supervised multi-modal
fusion, enabling label-efficient and parameter-efficient adaptation toward
various sensor modalities. It addresses three main challenges: 1) adaptation
toward diverse non-RGB sensors for single-modal processing, 2) synergistic
processing of multi-modal data via sensor fusion, and 3) mask-free training for
different downstream tasks. Extensive experiments show that MM-SAM consistently
outperforms SAM by large margins, demonstrating its effectiveness and
robustness across various sensors and data modalities.