MedSAMix: A Training-Free Model Merging Approach for Medical Image
  Segmentation
Authors: name: Yanwu Yang; name: Guinan Su; name: Jiesi Hu; name: Francesco Sammarco; name: Jonas Geiping; name: Thomas Wolfers
Published: 2025-08-14 19:35:57

Abstract: Universal medical image segmentation models have emerged as a promising
paradigm due to their strong generalizability across diverse tasks, showing
great potential for a wide range of clinical applications. This potential has
been partly driven by the success of general-purpose vision models such as the
Segment Anything Model (SAM), which has inspired the development of various
fine-tuned variants for medical segmentation tasks. However, fine-tuned
variants like MedSAM are trained on comparatively limited medical imaging data
that often suffers from heterogeneity, scarce annotations, and distributional
shifts. These challenges limit their ability to generalize across a wide range
of medical segmentation tasks. In this regard, we propose MedSAMix, a
training-free model merging method that integrates the strengths of both
generalist models (e.g., SAM) and specialist models (e.g., MedSAM) for medical
image segmentation. In contrast to traditional model merging approaches that
rely on manual configuration and often result in suboptimal outcomes, we
propose a zero-order optimization method to automatically discover optimal
layer-wise merging solutions. Furthermore, for clinical applications, we
develop two regimes to meet the demand of domain-specificity and
generalizability in different scenarios by single-task optimization and
multi-objective optimization respectively. Extensive evaluations on 25 medical
segmentation tasks demonstrate that MedSAMix effectively mitigates model bias
and consistently improves performance in both domain-specific accuracy and
generalization, achieving improvements of 6.67% on specialized tasks and 4.37%
on multi-task evaluations.