Title: Segment Anything Model Meets Semi-supervised Medical Image Segmentation: A Novel Perspective
Venue: NeurIPS 2025
Accepted: yes
Keywords: Semi-supervised Segmentation

Segment Anything Model

Knowledge Distillation
Abstract: The scarcity of annotated medical imaging data has driven significant progress in semi-supervised learning to alleviate reliance on expensive expert labeling. While foundational vision models such as the Segment Anything Model (SAM) exhibit robust generalization in generic segmentation tasks, their direct application to medical images often results in suboptimal performance. To address this challenge, in this work, we propose a novel fully SAM-based semi-supervised medical image segmentation framework and develop the corresponding knowledge distillation-based learning strategy. Specifically, we first employ an efficient SAM variant as the backbone network of the semi‑supervised framework and update the default prompt embedding of SAM to unleash its full potential. Then, we utilize an original SAM, which is rich in prior knowledge, as the teacher to optimize our efficient student SAM backbone through hierarchical knowledge distillation and a dynamic loss weighting strategy. Extensive experiments on various medical datasets demonstrate that our method outperforms state-of-the-art semi-supervised segmentation approaches. Especially, our model requires less than 10% of the parameter size of the original SAM, enabling substantially lower deployment and storage overhead in real-world clinical settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel fully SAM-based semi-supervised medical image segmentation (SSMIS) framework designed to address the scarcity of annotated medical imaging data. It tackles the suboptimal performance of generalist SAM models on medical images by integrating an efficient SAM variant as the backbone and updating its default prompt embedding.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper proposes a novel framework that fully leverages an efficient SAM variant as the backbone for semi-supervised medical image segmentation.\n2. The sophisticated knowledge distillation strategy from an original SAM teacher to optimize the efficient student SAM backbone is interesting.\n3. It updates SAM's default prompt embedding, and mitigate the negative impact of inaccurate prompts.\n\n**Weakness:**\n1. The student model's performance is inherently tied to the capabilities of the original SAM teacher. If the teacher, even after fine-tuning on medical data, struggles with certain rare pathologies or complex anatomical variations, these limitations will be propagated, potentially capping the student's accuracy in critical cases.\n2. While \"various medical datasets\" are mentioned, the specific range of medical imaging modalities (e.g., MRI, CT, X-ray, ultrasound, different anatomical regions, varying image qualities) and their pathological diversity are not explicitly detailed in the abstract. SAM's generalization can still be challenging across wildly different medical image characteristics.\n\nQUESTIONS:\n1. Could the authors provide a more granular breakdown of the \"various medical datasets\" used, specifying modalities, anatomical regions, and the range of pathological conditions to better assess the true generalizability?\n2. How sensitive is the model's performance to the choice of distillation strategy parameters and the dynamic loss weighting schedule? Is there an automated way to tune these, or does it require significant manual effort per dataset?\n\nLIMITATIONS:\n1. **teacher-student model dependency:** The fundamental effectiveness of the proposed semi-supervised framework relies on the \"teacher\" (original SAM) providing accurate and rich knowledge. If the teacher struggles with the inherent complexities, ambiguities, or out-of-distribution characteristics of certain medical images, the student's performance will be capped, potentially leading to critical errors in clinical diagnoses or interventions.\n2. **generalizability to highly fiverse medical modalities/anatomies:** Medical imaging is incredibly diverse. A model performing well on one type of medical data (e.g., cardiac MRI) may not directly generalize to another (e.g., lung CT scans with diffuse pathologies) due to different image characteristics and anatomical contexts. This limits the \"plug-and-play\" nature of the solution for broad clinical applicability and necessitates careful validation for each new application.\n3. **complexity of hierarchical knowledge transfer and dynamic weighting:** The intricate distillation and weighting mechanisms, while contributing to performance, increase the complexity of model development and deployment. Achieving optimal results likely requires expert-level hyperparameter tuning, which could be a barrier for wider adoption in clinical research or practice where ease of use is paramount.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting Challenge 1: Annotated medical imaging data is scarce and expensive to obtain.  Approach: Leverage unlabeled data for semi-supervised learning to reduce reliance on annotated data.\n\nExisting Challenge 2: Although foundational vision models like SAM demonstrate strong generalization in general segmentation tasks, their direct application to medical images often results in suboptimal performance.  Approach: Fine-tune the original SAM using a small amount of labeled data to serve as a teacher network, and then use knowledge distillation to train a more lightweight SAM variant (student network) with the help of unlabeled data.\n\nSTRENGTHS AND WEAKNESSES:\nThe Related Work and Methodology are clearly written and easy to follow. However, the claimed innovation is questionable. The authors state that the core contribution lies in leveraging unlabeled data for semi-supervised learning. Actually, unlabeled data is only used for knowledge distillation and not for effectively improving SAM's adaptation to the target domain.\n\nOverall, the work does not constitute a true semi-supervised learning framework. The teacher network is fine-tuned using labeled data, while the student network is trained with hard labels and distills knowledge by aligning soft predictions on unlabeled data with those from the teacher. Unlabeled data serves merely as a medium for distillation, rather than directly enhancing the model’s generalization ability.\n\nMoreover, the paper lacks critical experiments, such as evaluating the performance of the teacher network itself and quantifying the performance gap between the teacher and student networks.\n\nQUESTIONS:\n1.\tEffectiveness of the teacher network:\nKnowledge distillation is essentially a model compression technique where the performance of the student network is inherently dependent on the quality of the teacher. If the teacher network performs poorly, the student network is unlikely to perform well either. Therefore, the paper should provide evidence of the teacher network’s effectiveness. \n\n2.\tImpact of fine-tuning the teacher network:\nThe paper adopts a static teacher network, which is obtained by fine-tuning SAM on labeled data and then freezing it. To validate the effect of fine-tuning, the paper should compare the performance of the original SAM and the fine-tuned SAM. \n\n3.\tClarification on the role of unlabeled data:\nThe role of unlabeled data in the framework needs further clarification. In Section 3.3, the paper states: “After fine-tuning, we further optimize the model through semi-supervised learning.” What does this “optimization” refer to exactly? Is the unlabeled data only used to help align the student’s predictions with those of the teacher?\n\n4.\tEvaluation of model deployment and storage optimization:\nAlthough the paper claims that the proposed model has less than 10% of the original SAM parameters, it does not provide a clear comparison of actual storage sizes (e.g., in MB or GB), nor does it report key deployment metrics such as inference speed or latency in clinical scenarios.\n\n5.\tDoes offline distillation limit the model's adaptability?\nThe teacher model is frozen after fine-tuning, while the student model continues to be updated during training. This offline distillation approach may prevent the teacher from adapting to the student’s evolving learning status or distribution shift. Has the author considered using a soft update strategy (e.g., EMA) to enhance the adaptability of the model?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a SAM-based Semi-supervised learning method to address that the direct application of SAM-based methods to medical images often results in suboptimal performance. The authors propose a teacher-student framework that utilises an original SAM, which is rich in prior knowledge, as the teacher to optimise the student SAM backbone through hierarchical knowledge distillation and a dynamic loss weighting strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper is easy to read, and the motivation is clear.\n\n2.\tUtilising efficient SAM is an interesting application for semi-supervised medical image segmentation.\n\n3.\tThe authors have conducted extensive experiments, along with ablation studies.\n\n4.\tResults show the method achieves SOTA performance.\n\nWeaknesses:\n\n1.\tIn the experimental evaluation section, the most recent SOTA in SSMIS is missing, for example, MC-Net+ [1], Co-BioNet [2], Correlation-based switching mean teacher for semi-supervised medical image segmentation [3].\n\n2.\tAdditionally, comparisons with SAM-variants for medical image segmentation are lacking, for example, MedSAM and SAM-Med2D.\n\n3.\tAuthors mentioned, “The success of efficient SAM variants inspires us to consider whether they can serve as backbone networks in semi-supervised learning frameworks, replacing traditional CNN-based backbones to fully leverage their capabilities”, yet the complexity analysis between SSMIS and the proposed method is missing.\n\n4.\tDetails on experimental result reproducibility and code repository are missing, despite the criteria for this being marked as ‘Yes’ in the checklist.\n\n[1] Y. Wu, Z. Ge, D. Zhang, M. Xu, L. Zhang, Y. Xia, J. Cai, Mutual consistency learning for semi-supervised medical image segmentation, Med. Image Anal. 81 (2022) 102530. \n\n[2] H. Peiris, M. Hayat, Z. Chen, G. Egan, M. Harandi, Uncertainty-guided dual-views for semi-supervised volumetric medical image segmentation, Nat. Mach. Intell. (2023) 1–15.\n\n[3] G. Deng, H. Sun, W. Xie, Correlation-based switching mean teacher for semi-supervised medical image segmentation, Neurocomputing (2025) 129818.\n\nQUESTIONS:\n1.\tThe Brats dataset is multiclass (edema, NET, ET), so what are the tumour regions focused on in the experiment? \n\n2.\tAlso, does the mask decoder generate multiple masks? Or, as in the VFM setting, is it trained with an individual class at a time with a binary mask?\n\n3.\tIsn’t mitigating the prompt encoder during the mask generation like removing a fundamental block from SAM? What will happen if a prompt encoder is used? Is there a significant performance gain to be had by not using a prompt encoder?\n\nLIMITATIONS:\nYes, mentioned in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a semi-supervised segmentation method for the Segment Anything Model (SAM) which distills a standard SAM segmentation backbone to a more efficient one, using both labeled and unlabeled images. As in standard Teacher-to-Student distillation approaches for semi-supervised learning (SSL), the predictions of the Teacher for unlabeled images are used as pseudo-supervision to train the student, in addition to a standard supervised segmentation loss on labeled images. Two main modifications are made to this approach. First, a distillation loss is applied to both the output of the image encoder and the segmentation decoder. Second, the distillation transfers knowledge from the Teacher to a more efficient Student backbone. The proposed method also introduces a fine-tuning step to learn task-specific features and a dynamic adjustment of the distillation loss during SSL training. The method is evaluated on Left Atrium (LA), Brats-2019, PROMISE12 and Synapse datasets, achieving a better performance than recent SSL approaches for medical image segmentation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Existing works typically use Teacher-to-Student knowledge distillation for learning a more efficient model or to obtain pseudo-labels for unlabeled examples. As mentioned in the paper, a contribution of the proposed method is to do both these steps in a single framework.\n\n* The proposed method achieved outstanding performance on a varied set of segmentation datasets.\n\n* The paper is well-written and easy to follow.\n\nWeaknesses:\n\n* Despite the contributions mentioned above, the methodological novelty of the proposed approach remains somewhat limited, as it combines two well-known distillation strategies in a single framework. \n\n* Although the method outperforms strong SSL baselines, the comparison is a bit tricky since most (if not all) of these baselines are based on a CNN backbone, while the method employs a transformer (ViT-B) backbone. \n\n* Some recently published papers report a better performance than those in the paper, for example: \"DyCON: Dynamic Uncertainty-aware Consistency and Contrastive Learning for Semi-supervised Medical Image Segmentation\". I understand that the paper was put on Arxiv one month before the NeurIPS deadline, so I do not blame authors for this omission.\n\n* The ablation study shows that the method's performance is affected by the choice of hyperparameters, such as lamba_out and lambda_emb. However, unless I missed that part, it is unclear how these hyperparameters were selected (if a validation set was employed and if they were tuned specifically for each dataset).\n\nQUESTIONS:\n* The proposed method starts with a pre-trained SAM model which is fine-tuned to specific medical image segmentation tasks. Why not start with a model pre-trained on medical images, such as MedSAM ?\n\n* Can authors provide more information on the default embeddings for their mask decoder (size, which decoder inputs, etc.) ?  \n\n* In eq (3), how do you compute the KL divergence between feature maps? Do you add a learnable projector + softmax to map features into probability distributions?\n\n* For dynamic loss weighting, two factors can impact the performance difference between the Teacher and Student: 1) the model capacity (the Teacher has a greater capacity), and 2) access to training gradient from unlabeled images (only the Student). How can you separate these two factors when choosing lambda_kd?\n\n* In Section 4.1: \" As shown in Table 1, our approach outperforms these competitors\". Technically, BCP is better in terms of surface-to-surface error (HD and ASD).\n\n* Instead of the proposed method, it would also be possible to fine-tune the SAM-based Teacher with both labeled and unlabeled data, and only after perform distillation. Has this approach been tested? If so, how does it compare?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}