Unifying Segment Anything in Microscopy with Multimodal Large Language
  Model
Authors: name: Manyu Li, name: Ruian He, name: Zixian Zhang, name: Weimin Tan, name: Bo Yan

Accurate segmentation of regions of interest in biomedical images holds
substantial value in image analysis. Although several foundation models for
biomedical segmentation have currently achieved excellent performance on
certain datasets, they typically demonstrate sub-optimal performance on unseen
domain data. We owe the deficiency to lack of vision-language knowledge before
segmentation. Multimodal Large Language Models (MLLMs) bring outstanding
understanding and reasoning capabilities to multimodal tasks, which inspires us
to leverage MLLMs to inject Vision-Language Knowledge (VLK), thereby enabling
vision models to demonstrate superior generalization capabilities on
cross-domain datasets. In this paper, we propose using MLLMs to guide SAM in
learning microscopy crose-domain data, unifying Segment Anything in Microscopy,
named uLLSAM. Specifically, we propose the Vision-Language Semantic Alignment
(VLSA) module, which injects VLK into Segment Anything Model (SAM). We find
that after SAM receives global VLK prompts, its performance improves
significantly, but there are deficiencies in boundary contour perception.
Therefore, we further propose Semantic Boundary Regularization (SBR) to prompt
SAM. Our method achieves performance improvements of 7.71% in Dice and 12.10%
in SA across 9 in-domain microscopy datasets, achieving state-of-the-art
performance. Our method also demonstrates improvements of 6.79% in Dice and
10.08% in SA across 10 out-ofdomain datasets, exhibiting strong generalization
capabilities. Code is available at https://github.com/ieellee/uLLSAM.