SAMed-2: Selective Memory Enhanced Medical Segment Anything Model
Authors: name: Zhiling Yan; name: Sifan Song; name: Dingjie Song; name: Yiwei Li; name: Rong Zhou; name: Weixiang Sun; name: Zhennong Chen; name: Sekeun Kim; name: Hui Ren; name: Tianming Liu; name: Quanzheng Li; name: Xiang Li; name: Lifang He; name: Lichao Sun
Published: 2025-07-04 16:30:38

Abstract: Recent "segment anything" efforts show promise by learning from large-scale
data, but adapting such models directly to medical images remains challenging
due to the complexity of medical data, noisy annotations, and continual
learning requirements across diverse modalities and anatomical structures. In
this work, we propose SAMed-2, a new foundation model for medical image
segmentation built upon the SAM-2 architecture. Specifically, we introduce a
temporal adapter into the image encoder to capture image correlations and a
confidence-driven memory mechanism to store high-certainty features for later
retrieval. This memory-based strategy counters the pervasive noise in
large-scale medical datasets and mitigates catastrophic forgetting when
encountering new tasks or modalities. To train and evaluate SAMed-2, we curate
MedBank-100k, a comprehensive dataset spanning seven imaging modalities and 21
medical segmentation tasks. Our experiments on both internal benchmarks and 10
external datasets demonstrate superior performance over state-of-the-art
baselines in multi-task scenarios. The code is available at:
https://github.com/ZhilingYan/Medical-SAM-Bench.