UnCLe SAM: Unleashing SAM’s Potential for Continual Prostate MRI Segmentation

Authors: amin  ranem
moritz  fuchs
anirban  mukhopadhyay
Venue: MIDL_io_2024_Conference
Published: 2024-01-19 14:49:45

Abstract: Continual medical image segmentation primarily explores the utilization of U-Net and its derivatives within the realm of medical imaging, posing significant challenges in meeting the demands of shifting domains over time. Foundation models serve as robust knowledge repositories, offering unique advantages such as general applicability, knowledge transferability, and continuous improvements. By leveraging pre-existing domain insights, adaptability, generalization, and performance across diverse tasks can be enhanced.
In this work, we show how to deploy Segment Anything Model's (SAM) natural image pretraining for the continual medical image segmentation, where data is sparse.
We introduce UnCLe SAM, a novel approach that uses the knowledge of the pre-trained SAM foundation model to make it suitable for continual segmentation in dynamic environments.
We demonstrate that UnCLe SAM is a robust alternative to U-Net-based approaches and showcase its state-of-the-art (SOTA) continual medical segmentation capabilities.
The primary objective of UnCLe SAM is to strike a delicate balance between model rigidity and plasticity, effectively addressing prevalent pitfalls within CL methodologies.
We assess UnCLe SAM through a series of prostate segmentation tasks, applying a set of different CL methods. Comparative evaluations against the SOTA Lifelong nnU-Net framework reveal the potential application of UnCLe SAM in dynamically changing environments like healthcare.
Our code base will be made public upon acceptance.

Main paper: 
 introduction: 
 Continual learning (CL) holds immense significance in safety-critical applications of Deep Learning. This is evident in healthcare, where models must adapt to data changes over time while maintaining high performance on older data (Gonzalez et al., 2020) . Traditional U-Net architectures encounter difficulties in seamlessly adapting to domain changes in data distribution, particularly when faced with new imaging protocols or variations in patient populations or diseases (Sanner et al., 2021; Derakhshani et al., 2022; González et al., 2022; Fuchs et al., 2022) . The challenge lies in training models that exhibit superior performance when using datasets with limited temporal availability. Finding a good trade-off between rigidity which hinders learning new tasks and plasticity causing catastrophic forgetting on previous tasks is therefore important (Kirkpatrick et al., 2017; Hadsell et al., 2020 ; De Lange et al., 2021) . Existing CL methods, when applied to medical data, often result in segmentations that fall short of basic semantic standards like semantic coherence over time (Ranem et al., 2022; González et al., 2023) . We introduce UnCLe SAM (Unleashing Continual Learning for SAM), a novel approach that leverages the Segment Anything Model (SAM) (Kirillov et al., 2023) for enhanced domain adaptation in continuous medical setups. Our method involves continually adapting the prompt for SAM while leveraging the knowledge base of SAM without a full re-training as in MedSAM (Ma and Wang, 2023) .The Lifelong nnU-Net framework and other CL methods, such as Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) , Riemannian Walk (RWalk) (Chaudhry et al., 2018) , or basic replay methods like iCARL (Rebuffi et al., 2017) , struggle to adequately adapt to changing domains (González et al., 2023) . Using replay, regularization, or knowledge distillation has its advantages and disadvantages when it comes to domain shifts in continuous setups. For instance, there is a rigidity/plasticity trade-off or computational burden on performance. Exploring CL by using foundation models, until now, remained an unexplored approach, Figure 1 .UnCLe SAM strategically addresses challenges for domain adaptation known when applying U-Net-based architectures. U-Nets encounter challenges in maintaining segmentation accuracy amidst variations in imaging protocols or discrepancies in patient populations, resulting in compromised performance and reduced reliability of the model in dynamic clinical scenarios (Gonzalez et al., 2020; Ranem et al., 2022; Sanner et al., 2021) . Leveraging the robust knowledge from SAM while continually adapting the prompting adapter reduces such challenges.In this work, we use the pre-trained SAM architecture for continual prostate MRI segmentation that leverage the foundation model's knowledge to properly adapt to shifting domains. Rather than attempting to apply regularization to the network, we opt to freeze certain architectural components such as the Vision Transformer, i.e., SAM's Encoder while continually adapting the prompt for SAM. With this approach, the learned visual representation can be transfered to different domains (Ma and Wang, 2023) .SAM's ability to segment anything in diverse contexts becomes a valuable asset for continual adaptation, ensuring that the model maintains high segmentation performance across evolving datasets with simple fine-tuning techniques. By using a pre-trained ResNet-50 network, (He et al., 2016) as an Adapter to continually updating the SAM prompt, UnCLe SAM effectively handles challenges that come with domain adaptations commonly faced by U-Net-based architectures. UnCLe SAM does not require a long time to train on a new domain, which makes it superior in terms of applicability while achieving SOTA performance. To validate our approach, we focus on the critical task of prostate segmentation for T2weighted MRIs, which plays an important role in prostate cancer diagnosis and treatment planning. Our contributions are three-fold: We (1) introduce a Continual prompting of foundation model for medical image segmentation, that can (2) successfully respond to domain adaptation by achieving (3) superior performance than Lifelong nnU-Net Framework. 
methodology: 
 Fundamentals We start by introducing some key terminology: Ω ⊂ R 3 defines a 3D spatial domain as we work with three-dimensional Magnetic Resonance (MR) scans. T i ⊂ Ω T is referred to a single task i, whereas Ω T represents a set of tasks. A stage j in a continual setup defines the process of training the model on task T j after it has been trained on all previous {T 1 , . . . , T j-1 } tasks using some CL method.Basic components of SAM SAM (Kirillov et al., 2023) , consists of a Vision Transformer (ViT) (Dosovitskiy et al., 2020) as its core feature extractor and a segmentation head in form of a mask decoder, responsible for generating precise segmentation masks. Trained on a large database of general images, SAM has garnered a robust knowledge base that facilitates its adaptability across various domains opening doors for continual setups.The ViT feature extractor within SAM effectively captures visual information from input images, creating detailed embeddings. SAM's segmentation head complements the ViT feature extractor by processing the embeddings to produce detailed segmentation masks. This mask decoder is trained to accurately delineate regions of interest within the input images. Moreover, the segmentation head can makes use of different prompts such as 2D points and bounding boxes to further guide the segmentation process. SAM was trained on a large database of general images to establish a strong foundation for adaptable segmentation tasks. This foundation enables SAM to excel in various domains, making it particularly well-suited for continual setups where adaptability is paramount.UnCLe SAM: Continual Prompting for Enhanced Adaptability UnCLe SAM builds upon the foundation of SAM by introducing a novel approach to enhance adaptability over time. Since SAM generalizes to different domains, UnCLe SAM enhances this adaptability by introducing continual prompting using a ResNet-50 adapter. The architecture of UnCLe SAM is carefully crafted to leverage the global knowledge stored within the pre-trained SAM, offering adaptability in dynamic environments like healthcare. A key aspect of our methodology is the decision to keep SAM's ViT backbone frozen, ensuring consistent feature extraction across different datasets and imaging modalities. This approach not only enhances feature extraction reliability but also lays the groundwork for seamless adaptation to changing domain characteristics, see Figure 2 . By leveraging the frozen ViT backbone, we implement a pre-processing step to extract embeddings from both training and testing sets, aligning with the approach proposed by MedSAM (Ma and Wang, 2023) .ResNet 50 A key component of UnCLe SAM lies in the continuous adaptation of the ResNet-50 adapter, which plays a crucial role in guiding the segmentation process by generating adaptive prompts. These adaptive prompts are based on the embeddings extracted by the ViT feature extractor. By incorporating a transposed convolutional layer, the ResNet-50 adapter effectively translates the embeddings into actionable prompts, improving segmentation accuracy. Based on the extensive ablations of SAM in radiology (Ranem et al., 2023) , the proposed adapter is designed to predict 100 2D points and four coordinates representing a bounding box.Bounding Box ℒ !"# ℒ !"# ℒ !"# ℒ !"# (𝑠𝑒𝑔, 𝑠𝑒𝑔 $% ) ℒ &'# (()* 𝑏𝑏𝑜𝑥, 𝑏𝑏𝑜𝑥 $% ℒ &'# +,-./0+ 𝑠𝑎𝑚𝑝𝑙𝑒𝑠, 𝑠𝑎𝑚𝑝𝑙𝑒𝑠 $% Losses ℒ %)%,/ = ℒ &'# +,-./0+ + ℒ &'# (()* + ℒ !"#Moreover, UnCLe SAM is designed to adapt to changing domain characteristics over time. While SAM's basic components provide a strong foundation for segmentation tasks, UnCLe SAM's continual prompting mechanism ensures that the model can dynamically adjust to evolving datasets and domain shifts. This adaptability is essential for maintaining high performance across diverse medical imaging environments.In summary, UnCLe SAM's architectural composition integrates the robustness of pretrained SAM and the feature extraction capabilities of ResNet-50, forming a comprehensive base model for domain adaptation in medical image segmentation over time. By strategically freezing SAM's ViT backbone, coupled with the ResNet-50 Adapter, UnCLe SAM is the first method to effectively combine the strengths of both a foundation model and CL to achieve accurate and adaptable segmentation results for medical segmentation tasks. 
experimental setup: 
 Datasets We explore the problem of continual image segmentation for prostate MRIs.To ensure reproducibility, we use only openly available datasets, whereas every data source acts as one task {T 1 , . . . , T n }. The prostate data corpus consists of four publicly available T2-weighted MRI datasets as provided in the Multi-site Dataset for Prostate MRI Segmentation Challenge for sites A (ISBI), C (I2CVB) and D (UCL) and DecathProst from the Medical Segmentation Decathlon (Litjens et al., 2014; Bloch et al., 2015; Lemaître et al., 2015; Liu et al., 2020a,b; Antonelli et al., 2021) . For all datasets, we randomly divide 20% of the data for test purposes and maintain this split across all experiments.Training setup All nnU-Net (Isensee et al., 2021) experiments train for 250 epochs with 250 steps each using the Lifelong nnU-Net framework (González et al., 2023) with default optimizer and scheduler. SAM experiments also run for 250 epochs using Adam optimizer with weight decay of 1e -4 , learning rates of 1e -4 and 1e -3 for the SAM segmentation head and the Adapter respectively. Our loss function combines Dice-Cross-Entropy (DCE, L DCE ) from (Consortium, 2020) and Mean-Squared-Error (MSE, L M SE ) with early stopping (patience of 15). The MSE loss-term is used for predicted samples (L samples M SE ) and bounding box coordinates (L BBox M SE ):L SAM = L DCE + L samples M SE + L BBox M SE .All models are trained on a single NVIDIA A40 GPU (48 GB).Metrics For every CL setup, we report the mean Dice and standard deviation across the test images from all tasks {T i } i≤|Ω T | as well as average forwards (FWT) and backwards (BWT) transferability (Díaz-Rodríguez et al., 2018) . FWT measures the impact of the current training stage {T i } i≤|Ω T | on test data from an untrained stage T j ; j > i. BWT, on the other hand, indicates the amount of maintained knowledge on test samples from T j during training on different stages {T i } i≤|Ω T | ; j < i over time. Models that achieve a higher FWT have high plasticity and are able to learn new knowledge, while models with a higher BWT maintain most knowledge from previous tasks, i.e. prevent catastrophic forgetting. More information on the CL metrics can be found in the Appendix B.Baselines To get a proper evaluation of our approach, we compare against conventional sequential training, rehearsal training, and two well-known CL methods: EWC (Kirkpatrick et al., 2017) and RWalk (Chaudhry et al., 2018) . For both CL methods we are inspired by the Lifelong nnU-Net (González et al., 2023) hyperparameter setup for all our experiments (EWC: λ = 0.4, RWalk: α = 0.9, λ = 0.4). 
continual learning performance: 
 In this section, we compare UnCLe SAM with sequential Lifelong nnU-Net and two established CL methods -EWC and RWalk. Additionally, we evaluate against the upper bound of rehearsal training, which involves storing randomly 20% from each task. Rehearsal serves as an upper bound for Lifelong nnU-Net but is impractical due to privacy policy constraints on storing patient images. Table 2 and Figure 3 show that Lifelong nnU-Net achieves certain benefits depending on which CL method is used, however gets significantly outperformed by UnCLe SAM. UnCLe SAM demonstrates superiority over Lifelong nnU-Net as it successfully leverages the rich knowledge base embedded in the foundation model, enabling robust adaptation to domain shifts within the data. The 23% and 18% performance increase for BWT and FWT, UnCLe SAM compared to the rehearsal upper bound, contrasts with traditional methods, which struggle to handle domain variations effectively over time.To analyze the robustness of our proposed method, we illustrate segmentation masks in Figure 4 for UnCLe SAM and Lifelong nnU-Net using EWC, RWalk and rehearsal.UnCLe In a static training condition, a model is trained on one single task and validated across all existing tasks, as shown in Table 3 demonstrates the superior performance and greater generalizability of UnCLe SAM compared to nnU-Net for prostate segmentation. The method consistently achieves higher Dice scores across diverse datasets, achieving proper generalizability, highlighting UnCLe SAM's robustness and adaptability. It showcases promising potential for domain adaptation and continual learning in medical image segmentation, providing a stable and adaptable solution across diverse datasets. For additional results, we refer the reader to Appendix A. 
conclusion: 
 We propose UnCLe SAM, a novel approach that leverages the knowledge base of the pretrained SAM foundation model to address domain adaptation challenges in continual medical image segmentation. By leveraging SAM's robust capabilities, our method achieves superior adaptability and performance compared to traditional U-Net architectures like the Lifelong nnU-Net framework. Through extensive evaluation of a set of four different prostate datasets, UnCLe SAM demonstrates its effectiveness in maintaining knowledge from early stages while adapting to evolving datasets over time. Our approach not only outperforms existing methods in terms of segmentation accuracy in a continuous setup but also offers a more generalizable solution, showcasing a significant performance improvement even when trained statically with data from a single site. UnCLe SAM paves the way for a balanced approach between rigidity and plasticity in continual learning setups without using actual CL methods like EWC while achieving better results than rehearsal. By releasing our code base, we hope to inspire research in CL that goes beyond traditional U-Net-based segmentation for medical settings by leveraging the knowledge base of foundation models like SAM. Figure 6 illustrates the workflow process for SAM, MedSAM and UnCLe SAM, highlighting their distinct training methodologies. SAM and MedSAM are trained centralized from scratch, while having an increased CO2 emission. Additionally, having centralized training of MedSAM may inadvertently contain samples from publicly used datasets, resulting in data leakage and privacy violations in CL comparisons. In contrast, UnCLe SAM utilizes a continuous training paradigm, minimizing CO2 emissions by leveraging the pre-trained SAM model and enabling adaptation to evolving data leveraging the pre-trained SAM. 
 
 Appendix:  


Decision: All reviewers found the proposed method to be novel and the results promising. 2 weak reject, 1 weak accept, and 2 strong accept recommendations from reviewers.

Reviews: review: paper_summary: paper_summary: This paper proposed UnCLeSAM, an extension of the Segment Anything Model for continual learning. UnCLeSAM added to the knowledge of the pre-trained SAM continual prompting using ResNet-50 adapter. Using the image embedding output by the ViT,  the adapter generates new prompts in the form of Samples and Bounding Box to feed into the segmentation head. The approach is tested on prostate segmentation, where each task is simulated by a different dataset and compared to Lifelong nn-Unet and two CL approaches (EWC and RWalk). Results showed that UnCLeSAM outperformed comparative approaches.
main_review: main_review: The experimental part is not clear and robust enough to justify the statement of superior performance of the method. The approach should be tested on more than one task and numerical results on each datasets should be provided.
strength_weakness: strength_weakness: - Experimental part is too small to support the claim, authors used only one (easy) segmentation task (segmentation of the whole prostate gland) and simulated the different tasks only by using datasets with different resolutions. Datasets are very small. Comparative methods are quite old (2017-2018)   
-Experimental part is unclear
questions: questions: Authors should 
- explain how did they choose the datasets.
- provide numerical results of the CL on the single results. The results presented on table 2 for Lifelong nnU-Net are quite different from the one presented on the original paper (Lifelong nnU-Net: a framework for standardised medical continual learning) 
- give more details on the experimental parts (see detailed comments)
paper_summary: paper_summary: The paper introduced an approach uses the pre-trained SAM model for segmentation in different settings of prostate MRI dataset, in the context of continual learning. The proposed UnCLe SAM is composed by a SAM model and a ResNet-50 based segmentation header. They try to demonstrate the proposed UnCLe SAM is a robust alternative to U-Net-based approaches using multiple prostate MRI dataset. However, there are some issues need to be addressed.
main_review: main_review: The authors partially solved my concerns, and I still think the authors' contribution was relatively limited : not be able to pre-train the baseline UNet-like models in same amount of data compared with the SAM, and thus lack of fairly comparisons. Therefore, I will keep my rating here.
strength_weakness: strength_weakness: 1) It is not clear if this better performance was achieved by the more trainable parameters SAM model has. Similar numbers of trainable parameter settings should be given if the authors wanted to conclude the SAM-like models are better candidate of Unet-like models. Unet-like models can also have large number of trainable parameters.
2) It is not clear if the better performance was achieved by the impact of a good pretraining with enough amount of data observation. For example, if a Unet-like model with comparable capacity pretrained with similar amount of data seen by the pretrained SAM, will the discrepancies still exist on the prostate segmentation tasks?
3) If training from scratch in the continual learning setting, I highly doubt it is enough when there are only ~70 cases for Unet-like models to train on. In addition, when only using single dataset for training, the nnUNet will only take <20 cases for training, and it is hard to believe the nnUNet is trained with enough data observations in this case.  Fair comparisons between model performances should happen at least when Unet-like models are approximately trained enough.
4) Experiment setting sometimes not clear. Author should briefly state what is the ‘default setting’ of the Lifelong nnUNet framework. There are multiple settings in the paper mentioned in (Gonz´alez et al., 2023).
5) I saw the performance is the worst in DecathProst. Is this because of the resolution differences? Sinc e DecathProst has different resolutions compared with other 3 datasets, how did you do the pre-processing?
questions: questions: Please refer to the point listed in Weaknesses section.
paper_summary: paper_summary: The paper introduces an adapted version of the SAM (Self-Attention Model) by modifying it to retain the feature representations of SAM's ViT (Vision Transformer) image encoder while introducing adapters based on ResNet and multi-task learning. This adaptation enhances SAM's task generalization performance in medical image detection tasks. The method shares similarities with the latest PEFT (Pseudo-Ensemble Fine-Tuning) method and additionally introduces two loss functions: BBOX regression loss and MSE category prediction loss, which can be seen as multi-task learning. These two additional task loss functions assist and optimize the true image segmentation task.
main_review: main_review: This paper effectively leverages the latest SAM model in Continual Learning (CL), introducing a unique optimization approach distinct from MedSAM and validated through experiments, showcasing its relevance. It utilizes prior knowledge from SAM in ViT feature map encoding for image BBOX detection. However, it lacks clarity in the need for ResNet feature extraction after ViT encoding and ambiguity in depicting SAM's components in Figure 2, raising uncertainties about the role of the Prompt encoder and Segmentation Head.
strength_weakness: strength_weakness: The drawbacks of this paper include: 
1. In Figure 2, since the embeddings are derived from the feature maps encoded by ViT, I believe that these embeddings already contain rich features. Therefore, I wonder if the subsequent step of extracting features with ResNet might be redundant.
I suggest that you could use a simple attention layer (or even directly use a few fully connected layers) for fine-tuning. I truly believe that the embeddings output by ViT do not need to undergo ResNet encoding. You could conduct some ablation experiments to test whether ResNet is redundant.

2. The most crucial point is that we know SAM comprises the ViT image encoder, the Prompt encoder, and the Mask decoder for the final region output. However, this is not clearly depicted in Figure 2, which raises uncertainties. My question is whether the Segmentation Head is a standalone Mask decoder or a combination of the Mask decoder and the Prompt encoder. Additionally, what is the Prompt input for this model? If Prompt input is not necessary, SAM can omit it. I suggest that in Figure 2, you represent your improvements more clearly by basing it on the complete SAM architecture and then adding your Adapter. This way, it will be clearer to indicate your modifications rather than being confusing, leaving readers uncertain about your configuration of SAM's Prompt Encoder.
questions: questions: I do not care rebuttal. I suggest acceptance. Author should remember release their code.
paper_summary: paper_summary: This paper introduces UnCLe SAM, a novel approach that leverages the Segment Anything Model (SAM) for continual prostate MRI segmentation. UnCLe SAM strategically addresses challenges faced by traditional U-Net-based architectures in maintaining segmentation accuracy amidst variations in imaging protocols and patient populations, presenting a promising approach for continual medical image segmentation in dynamic clinical scenarios.
main_review: main_review: The idea to leverage SAM for continual segmentation is novel. The paper is well-written and conducted comprehensive analysis and comparison to demonstrate the effectiveness. The demonstrated superior performance compared to the state-of-the-art methods adds significant value to the proposed approach.
strength_weakness: strength_weakness: Overall the method and experiments are sufficient. I have several minor suggestions. 
- It would be better to compare with original SAM and other medical variants like MedSAM. 
- What does 'UnCLe' means in the name? U-Net-based Continual Learning? It would be better to specify this in the revised version.
questions: questions: See above.
paper_summary: paper_summary: The authors present a CL method for MRI segmentation based on the pre-trained Segment Anything Model with an appended ResNet50 adapter that improves domain adaptation and achieves superior performance than the "state-of-the-art" Lifelong nnUNet.
main_review: main_review: Thank you for addressing my concerns, and taking an active part in the rebuttal. Although I agree with some of the other reviewers about the added novelty of the proposed method, I believe that showing the clear advantages of using pre-trained models is an important one, especially in scenarios where the model was pre-trained on images from another domain. Highlighting these advantages can help researchers develop models in areas where data is scarce, and the use of a publicly available model helps with the reproducibility as well.
strength_weakness: strength_weakness: The authors point out, that their model outperforms the 'SOTA' Lifelong nnUNet, however their referenced publication (González et. al.) does not claim state-of-the-art results. The authors should reference what they base their assumption on that the Lifelong nnUNet is state-of-the-art, or preferrably, rephrase and simply claim that they largely outperform the nnUNet model, which in itself is a great achievement.
Most of the paper is well-written and very detailed, but there are parts where the text is too high level, and the motivation behind some paragraphs is unclear. I will give more details later.
questions: questions: - Do the authors have an assumption on how the pre-trained SAM model in itself (without the adapter) would perform for MRI segmentation? If so, it would be interesting to include the results in the static evaluations. A strong motivation between the much better performance of UnCLe SAM over the static nnUNet could be simply that SAM has been extensively pre-trained before training here.
- The need for domain adaptation is clear, and it's a strong motivation of the paper, however the lack of access to previous training data is less obvious. Could the authors explain a scenario where access to previous training data is lost?
- The paragraph "Basic components of SAM" is very high-level and the length of the paragraph isn't really motivated. I believe shortening it would improve the focus of the manuscript.

Comments: title: Not convinced
comment: Thank you for your thoughtful comment regarding the potential impact of better pre-training with larger datasets on the performance of U-Net-like models. We understand your concern and appreciate your insights. Unfortunately, reliable U-Nets trained on massive datasets are not readily available in the public domain, unlike SAMs. Given the limitations of university-level compute infrastructures and the timeframe for empirical validation, training a U-Net on such massive datasets from scratch is not feasible. We acknowledge the importance of your feedback for future improvements and appreciate your understanding of the constraints we face in addressing this issue within the rebuttal period.
I still hold my doubt on the 2nd question I asked, and believe if having better pre-training with a large cohort of dataset, the UNet-like models will perform much better. The size of the datasets you mentioned used in Table 2 and Figure 3 is way less than how SAM model was trained. 

Also, for the 3rd question, I still believe in order to have a fair comparison, all baseline models, like the UNet-like models you mentioned, should have pretrained, or seen, enough dataset for pretraining, and then we can safely say about how we can compare using the following CL framework.

I know this is hard to solve within the rebuttal time, but I will leave my comment here for your future improvement. I will keep my rating.
We appreciate your detailed observation and answer your comments in a point-by-point fashion:
- In response to your feedback, we have addressed this concern by adding a figure in the Appendix (Figure 6, Appendix D), illustrating the overall problem. This figure highlights the differences between SAM, MedSAM, and our UnCLe SAM approach. SAM and MedSAM are foundation models trained in a centralized way, while UnCLe SAM is trained continuously. Meaning, that having a centralized training of MedSAM might already contain samples from our publicly used datasets, which would lead to results that are not comparable in a CL comparison, as this would be data-leakage/privacy violation in CL. To have a comparison for continuous setups, the MedSAM and UnCLe SAM should be run on a set of private datasets to ensure neither model was potentially trained on such a dataset. Unfortunately, due to limitations in accessing private datasets, we were unable to conduct comparative experiments on such datasets.
- We have updated the revised manuscript and introduced the UnCLe SAM abbreviation in the text to avoid any ambiguity.
1. "the computational time required to train (Med)SAM from scratch", since SAM/MedSAM are zero-shot segmentation foundation models, which do not require additional training. Despite its limited performance on medical image segmentation tasks, it should also be added for comparison.
2. I would suggest to add the "Unleashing Continual Learning for SAM" in the revised manuscript to avoid ambiguity.
Thank you for your insightful comments, which have helped us improve the clarity and focus of our manuscript. If you have any further suggestions or questions, please let us know. We addressed each of your points:

1. **Performance of Pre-trained SAM Model**: In our recent work on "Exploring SAM Ablations for Enhancing Medical Segmentation in Radiology and Pathology," we observed that even when SAM is pre-trained and used with ground truth labels, it fails to achieve very high performances for radiology cases. For example, the highest performance achieved in the BraTS dataset was 84%. This insight suggests that the strong performance of UnCLe SAM over the static nnU-Net cannot solely be attributed to extensive pre-training of SAM.

2. **Explanation of the Lack of Access to Previous Training Data**: Consider the limited availability of patient data due to factors such as time constraints or changes in population or acquisition methods. In the case of prostate segmentation, changes in acquisition methods, such as the use of prostate coils that were previously used but are now outdated, can contribute to severe domain shifts. Additionally, privacy constraints over time may prohibit the long-term storage of patient scans, further limiting access to previous training data.

3. **Simplification of the "Basic Components of SAM" Paragraph**: We have reformulated and shortened the paragraph to make it easier to understand and to improve the focus of the manuscript.
Thank you for your insightful comments. We have considered your suggestions and addressed them accordingly in the revised manuscript. Regarding our used term “Segmentation Head”, this only includes the standalone mask decoder, not the prompt encoder.

1. **Comparison with Original SAM and Other Medical Variants**: We acknowledge the importance of comparing our approach with original SAM and other medical variants like MedSAM. However, due to the submission deadline and the computational time required to train (Med)SAM from scratch, we were unable to perform these experiments. It's worth noting that our proposed training strategy is environmentally friendly, as three rounds of continual training of SAM from scratch would require a substantial amount of training time. Compared to the approximately 3 hours required per stage for a UnCLe SAM, this duration does not include the additional time needed for re-training the ViT in traditional SAM or MedSAM models.

2. **Clarification of 'UnCLe'**: We want to clarify that 'UnCLe SAM' stands for "Unleashing Continual Learning for SAM." This designation denotes our approach of leveraging continual learning techniques to unleash the performance of SAM.
Thank you very much for your valuable suggestions. We greatly appreciate your input, and we want to inform you that we have incorporated your suggestions into the revised manuscript. Your feedback has been instrumental in improving the quality and clarity of our work. If you have any further comments or concerns, please do not hesitate to let us know.
3. **Training Adequacy in Continual Learning Setting**:  In addition to the segmentation performance of individual tasks, it is crucial to evaluate the sufficiency of training in the Continual Learning (CL) setting using comprehensive metrics such as FWT (Forward Transfer) and BWT (Backward Transfer). While our trained nnU-Nets achieve a Dice score of 85% on average (with 93% for ISBI, for example), which is considered decent segmentation performance for prostate tasks, it is important to note that this paper focuses primarily on evaluating performance in the CL setting.
Forward Transfer (FWT) measures the ability of a model to adapt to new tasks or datasets while retaining the knowledge learned from previous tasks. It evaluates how well the model generalizes to unseen data distributions encountered during training, making it a critical metric for assessing the effectiveness of CL approaches. In contrast, Backward Transfer (BWT) assesses the impact of new learning on previously learned tasks. It measures the extent to which training on new tasks affects the performance of previously learned tasks, providing insights into the stability and robustness of CL models over time while considering catastrophic forgetting..
In the context of medical imaging, where data availability is limited and new datasets may exhibit varying characteristics, FWT and BWT assume greater importance than traditional performance metrics. The ability of a model to continually adapt to new data distributions and retain previously learned knowledge is paramount for ensuring consistent and reliable performance in real-world applications.
While individual task performance provides valuable insights into the capabilities of a model, it is the ability to adapt and learn from new data distributions over time that distinguishes CL approaches. By prioritizing FWT and BWT in our evaluation, we aim to provide a more comprehensive assessment of the effectiveness of our proposed CL framework in the context of medical image segmentation.
In summary, while achieving high segmentation performance on individual tasks is important, evaluating performance in the CL setting using metrics such as FWT and BWT is crucial for assessing the robustness and adaptability of the models.


4. **Clarification of Experiment Settings**: The Lifelong nnU-Net framework's default settings involve 250 epochs with 250 steps per epoch, utilizing the optimizer and scheduler as specified in (González et al., 2023). Adhering to these standard settings ensures consistency and facilitates comparisons across experiments. We added this information in the revised manuscript. Our use of these defaults enables transparent and reproducible evaluation of our continual learning framework's performance against existing methodologies.

5. **Performance Discrepancy in DecathProst Dataset**: The lower performance observed in DecathProst can indeed be attributed to resolution differences. DecathProst has different resolutions compared to the other three datasets. Our UnCLe SAM preprocessing involves resampling to the ViT Encoder’s input size, as done in the MedSAM tutorial. Datasets are downloaded and directly used without additional preprocessing.

Regarding the usage of the Lifelong nnU-Net framework, we want to assure you that the framework has been utilized correctly, as we are the developers of the Lifelong nnU-Net framework and have ensured its correct implementation. We appreciate your feedback and have taken into account the points raised to enhance the clarity and quality of our work.
Thank you for the valuable feedback. We appreciate the insights provided and have addressed each point below:

1. **Comparison of SAM-like Models with U-Net-like Models**: We acknowledge the need for clarity regarding the comparison between SAM-like models and U-Net-like models. Our intention in this study was not to assert the general superiority of SAM-like models over U-Net-like models. Instead, we aimed to introduce and evaluate a novel training strategy that capitalizes on the strengths of foundation models within a Continual Learning (CL) framework. By leveraging the knowledge encoded in the foundation models, we sought to address the challenges associated with adapting to new data distributions over time, particularly in medical imaging tasks such as prostate segmentation from MRI scans, as the title of our paper states: “UnCLe SAM: Unleashing SAM’s Potential for Continual Prostate MRI Segmentation”.
It is important to note that our conclusion regarding the performance of SAM-like models was specific to the context of this training strategy and the metrics used for evaluation. Through comprehensive CL metrics, we demonstrated that our approach yields superior results compared to traditional nnU-Nets, particularly in the context of prostate segmentation.
In summary, while our study showcases promising results for SAM-like models in the context of prostate segmentation from MRI scans, we emphasize the significance of our proposed training strategy within a CL framework and its potential to enhance performance in specific applications. We hope this clarification provides a comprehensive understanding of our approach and its implications within the broader context of medical image segmentation research.


2. **Impact of Pre-training on Better Performance**: We do not agree that pre-training with sufficient data observation may impact performance. Even with pre-training on the first prostate dataset and subsequent fine-tuning on additional datasets, as demonstrated in Table 2 and Figure 3, U-Net-based architectures, such as nnU-Nets, still struggle to achieve optimal segmentation performance. This underscores the complexity of medical image segmentation tasks and the necessity for more sophisticated approaches. We contend that the impact of pre-training with sufficient data observation may be limited in the context of medical imaging tasks. The inherently limited availability of annotated medical imaging data poses a significant challenge for pre-training based approaches.
In contrast, SAM  benefits from pre-training on a much larger variety of data, leveraging the diverse range of medical imaging datasets to enhance its performance. Transformer architectures, such as SAM, have demonstrated favorable scaling laws when trained on large and diverse datasets. While U-Net-based architectures also exhibit scalability, we opted to leverage the already expended resources in pre-training SAM rather than introducing entirely new foundation models. This decision was driven by practical considerations, including space and time constraints in exploring the continual learning paradigm comprehensively.
By capitalizing on SAM's pre-trained representation and integrating it within our continual learning framework, we aimed to maximize the utilization of available resources and ensure efficient adaptation to new data distributions over time. This approach not only streamlines the training process but also offers a promising avenue for addressing the challenges posed by limited annotated medical imaging data.
In summary, while pre-training remains an essential component of many machine learning approaches, its impact on performance may be constrained by the availability of annotated data, particularly in the domain of medical imaging. Leveraging pre-trained models such as SAM presents a viable solution for maximizing the utility of existing resources and advancing the state-of-the-art in medical image segmentation.
Thank you for the detailed feedback. We appreciate the constructive comments and have addressed each point below:

1. **Dataset Selection & Usage of All Datasets**: For dataset selection and the usage of all datasets, we opted for the Prostate datasets, aligning with the experimental methodology we introduced for standardized comparison of medical continual learning research in our Lifelong nnU-Net paper. However, we made the deliberate decision not to utilize all datasets employed by Lifelong nnU-Net due to several factors. One primary consideration was the constraint posed by the limited space available in the MIDL paper (8 pages of content) restricting our capacity to include all experiments. Additionally, the significant time investment required for training all nnU-Nets, as elucidated in the Lifelong nnU-Net paper, where each continual learning (CL) method demands substantial computational resources. 
In the medical domain, evolving technologies and practices often necessitate adjustments in data acquisition methodologies. For instance, in the case of prostate segmentation, historical practices involved the use of prostate coils, which have since become obsolete. Consequently, the availability of data from these outdated methods may be limited or non-existent in contemporary datasets.
Despite these limitations, we assure the reviewer that all experiments will be comprehensively included in the journal version of our work. This decision is motivated by our commitment to providing a thorough and exhaustive analysis of our methodology and results.

2. **Numerical Results of Continual Learning (CL)**: We have included numerical results of the CL experiments on the single results in the Appendix (see revised manuscript). Regarding the differences in results between Table 2 and the original paper on Lifelong nnU-Net, we acknowledge this variation. The discrepancy arises because we randomly split the dataset, ensuring equal distributions of train and test sets for all experiments. This differs from the random splits used in Lifelong nnU-Net, as the random splitting was performed internally with nnU-Net random splitting method. There is a significant difference in the performance between Lifelong nnU-Net and UnCLe SAM, however, the performance of Lifelong nnU-Net across different splits is similar.

3. **Reference for Sentence about U-Nets**: We have added a reference to support the statement regarding challenges encountered by U-Nets in maintaining segmentation accuracy in clinical scenarios.

4. **Training of the Adapter**: The adapter is trained using the same data as SAM. Specifically, SAM is trained solely on dataset 79, while the adapter is trained on datasets UCL, I2CVB, ISBI and DecathProst.

5. **Clarification on Continuous Learning in Experimental Part**: Our experimental setup involves training SAM and the adapter together on a dataset UCL initially. Subsequently, we freeze SAM and exclusively continue training the adapter on datasets I2CVB, ISBI, and DecathProst.
We thank all the reviewers for their insightful comments and suggestions. Their feedback has been invaluable in improving the quality of our manuscript. We greatly appreciate their time and effort in providing constructive feedback, which has helped us to refine our work for the better, and we hope our provided rebuttal solves their mentioned weaknesses. All changes have been done in a bluish color to be recognizable from the original/unmodified text.