Training-free Prompt Placement by Propagation for SAM Predictions in Bone CT Scans

Authors: caroline  magg
lukas p e verweij
maaike a ter wee
george s buijs
johannes g g dobbe
geert j streekstra
leendert  blankevoort
clara i sánchez
Venue: MIDL_io_2024_Conference
Published: 2024-01-25 14:05:16

Abstract: The Segment Anything Model (SAM) is an interactive foundation segmentation model, showing impressive results for 2D natural images using prompts such as points and boxes. Transferring these results to medical image segmentation is challenging due to the 3D nature of medical images and the high demand of manual interaction. As a 2D architecture, SAM is applied slice-per-slice to a 3D medical scan. This hinders the application of SAM for volumetric medical scans since at least one prompt per class for each single slice is needed. In our work, the applicability is improve by reducing the number of necessary user-generated prompts. We introduce and evaluate multiple training-free strategies to automatically place box prompts in bone CT volumes, given only one initial box prompt per class. 
The average performance of our methods ranges from 54.22% Dice to 88.26% Dice. At the same time, the number of annotated pixels is reduced significantly from a few millions to two pixels per class. These promising results underline the potential of foundation models in medical image segmentation, paving the way for annotation-efficient, general approaches.

Main paper: 
 introduction: 
 Medical image segmentation (MIS) plays a crucial role in various clinical applications, among others the segmentation of bones as part of the musculoskeletal system for medical interventions and for examining changes in anatomical structures (Bonaldi et al., 2023) . In order to reduce user interaction and time-consuming manual delineations, deep learning (DL) methods have been developed to automatically generate a segmentation mask of diverse bone structures in CT imaging (Lindgren Belal et al., 2019; Deng et al., 2022; Wasserthal et al., 2023; Magg et al., 2024) . However, the best performing methods so far developed are trained in a fully supervised manner requiring pixel-based annotation. These models cannot be transferred to other unseen structures without new annotations and training. Foundation Models (FMs) offer better generalisability as they are generalpurpose models that are trained on large data sets and can be used for a wide range of downstream tasks. New advances for segmentation FMs have been made with the promptbased Segment Anything Model (SAM) by Kirillov et al. (2023) . This model relies on the user to mark a point in the object or bound the region of interest, which is referred to as placing a prompt. However, current segmentation FMs are not suitable for MIS because of two main challenges. First, SAM has been trained on natural images. Several studies (He et al., 2023; Mazurowski et al., 2023; Ma et al., 2023; Cheng et al., 2023a) evaluated SAM for MIS and reported unstable performance across diverse datasets which can be explained by the difference between natural and medical images. To overcome this challenge, adaptation techniques (Wu et al., 2023; Li et al., 2023) and fine-tuning on medical data (Ma et al., 2023; Cheng et al., 2023b; Xie et al., 2024) are introduced. Second, the 2D SAM architecture does not account for the 3D nature of volumetric medical scans. As at least one prompt per slice is required, the interactive approach depends on a high manual interaction if applied to volumetric MIS. This reduces the applicability in high-throughput or less supervised medical tasks. To alleviate the second challenge, we propose multiple training-free strategies that automatically place box prompts for bone CT scans. In contrast to our training-free method, Lei et al. (2023) and Wang et al. (2023) addressed this challenge using approaches that required an extra training phase, either to predict bounding boxes locations or to adapt SAM architecture to 3D, respectively. Most similar to our work is EviPrompt by Xu et al. (2023) which is an evidence-based method for generating point prompts for SAM based on uncertainty estimates. A medical image-annotation pair is used as reference to create a shared embedding in the feature space and extract pixel similarity compared to the sample of interest. In this work, we propose four different training-free strategies of prompt placement by propagation to adapt SAM for bone segmentation in 3D CT scans. In contrast to EviPrompt, only one initial box prompt is required and the remaining box prompts are inferred from predictions in adjacent slices. We evaluated our methods on three datasets containing different bone structures, and showed the annotation-efficiency of our methods by analyzing performance with respect to the required numbers of annotated pixels. 
data: 
 Three different datasets are used for our experiments, i.e., two internal datasets, namely Ds and Dk, and one publicly available dataset, namely Dt (see Appendix A, Figure 4 ). The first internal dataset (Ds) contains 15 bilateral CT scans of the shoulder joint with annotations of the left and right humerus and scapula. The shoulder CT scans were acquired with a Brilliance 64-channel CT Scanner (Philips Healthcare, Best, The Netherlands) with 250 mAs, 120 kV, pixel resolution of 0.83-0.98mm and slice thickness 0.3-1.0 mm. The second internal dataset (Dk) consists of 25 unilateral knee CT scans with knee implants, including 20 cadaveric and 5 patient scans. The tibial cortex bone and the tibial component of a knee prosthesis are labeled in the knee CT scans which were acquired with a Brilliance 64channel CT Scanner (Philips Healthcare, Best, The Netherlands) or a Siemens SOMATOM Force with 160 mAs, 120 kV, isotropic voxel spacing of 0.45mm. The internal datasets were annotated using an in-house annotation software (Dobbe et al., 2014) and ITK-Snap (Yushkevich et al., 2006) . The external dataset, Dt, is extracted from a publicly available dataset (https://doi.org/10.5281/zenodo.6802613, Wasserthal et al. (2023) ). Two different subsets, Dt1 and Dt2, are extracted from the provided test set: Dt1 consists of 33 CT scans containing the left and right humerus and scapula, in analogue to Ds; and Dt2 consists of 53 CT scans containing the left and right hip and femur. As the smallest dataset, Ds was used for the method development and preliminary hyperparameter testing. Dk and Dt were only used for additional evaluation purposes. 
methods: 
 Based on the prompt-based mode of SAM, our method is an automated prompt placement approach which is initialized by defining a box prompt. Assuming that the boundary of the object of interest in a 3D CT scan, in our case bone structures, can be modeled by a smooth surface over slices, local changes of prompts between neighboring slices could be considered minimally. Based on this, we present four different strategies that iterative propagate predictions from adjacent slices to infer a new prompt.SAM by Kirillov et al. ( 2023) is a prompt-based FM developed for interactive 2D segmentation. The architecture consists of three components: an image encoder, a prompt encoder and a mask decoder. First, the input image I ∈ R H×W is embedded by a Vision Transformer (ViT) pre-trained as Masked Auto-encoder (MAE). Then, the prompt encoder creates a prompt encoding from geometric prompts (namely bounding box or point). Multiple studies illustrate that box prompts achieve better results compared with point prompts (Ma et al., 2023; Mazurowski et al., 2023; He et al., 2023) . Thus, we focus on box prompts P ∈ N 2×2 represented by an embedding pair, i.e., the positional encoding of its top-left and bottom-right corner. Finally, a lightweight mask decoder maps the image and prompt embedding to an output mask M = f SAM (I, P ) with M ∈ R H×W .For volumetric data, SAM is currently applied in a slice-by-slice manner per class (Mazurowski et al., 2023) . Therefore, given a 3D CT scan I ∈ R H×W ×D with the i-th slice I i ∈ R H×W and the corresponding mask prediction M ∈ R H×W ×D with M i ∈ R H×W , one prompt P i for each slice i is required, so that M i = f SAM (I i , P i ). In our approaches, P i is defined as a bounding box enclosing the object of interest and described byP i = f BOX (M i ) = {(x t i , y t i ), (x b i , y b i )}, with the coordinates of the top-left (x t i , y t i ) and bottom-right (x b i , y b i ) pixel.Given an initial slice S j ∈ I and a manually provided prompt P j , we propose four approaches to estimate P i , and thus obtain M i . Figure 1 visualizes the idea behind three of the strategies. Without any loss of generality, j < i applies in the following.Baseline (f b ): In this approach, given the initial slice, the box prompt is propagated uni-directional through the volume, inferred from the previous prediction as follows:M i = f b (I i , M i-1 ) = f SAM (I i , P i ), with P i = f BOX (M i-1 ).(1)Stochastic Approach (f s ): For this approach, the box prompt is also propagated in one direction, i.e., uni-directional, but relies on multiple modifications of the previous prompt prediction. Specifically, K prompts are generated by randomly shifting the coordinates of the box enclosing the previous prediction as follows:P (k) i = {(x t i , y t i ) + (δ (k) x , δ (k) y ), (x b i , y b i ) + (δ (k) x , δ (k) y )}, ∀k = {1, . . . , K}, with δ (k) x , δ (k) y ∈ N.Based on this, M i is estimated as follows:M i = f s (I i , M i-1 , K) = K k=1 f SAM (I i , P (k) i ) (2) Nested Approach (f n ):The nested approach performs bi-directional box prompt propagation. The information of the previous prediction M i-1 and the following slice I i+1 are used to perform propagation steps forwards and backwards as follows:M i = f n (I i , I i+1 , M i-1 ) = f SAM (I i , P i ), with P i = f BOX (f SAM (I i+1 , f BOX (f SAM (I i , f BOX (M i-1 ))))).(3)Combined approach (f c ): The nested and stochastic approaches can be combined asEQUATIONby using f s from Equation (2).The initial slice S j is chosen randomly around the center of the object, because there will be a higher certainty of the correctness of the manually placed prompt P j due to the larger object area contained in this region. The obtained masks are postprocessed in two steps. First, the largest component associated with the prediction generated by the initial prompt is retained and all other disconnected components are removed (pp1). Second, predictions are removed if the corresponding box is significantly smaller than the initial box (pp2). 
experimental setup: 
 For our experiments, we used the pre-trained SAM ViT-B, as described in (Kirillov et al., 2023) . The 2D architecture processes the 3D CT scans in axial slices, since the axial plane has usually a higher resolution in CT scans. For each slice, a binary mask per class is generated. In order to combine all binary masks to one multi-class mask, a class processing sequence was defined: For Dk and Dt1, humerus before scapula; for Dk, tibia bone before tibial implant; for Dt2, hip before femur. Following Mazurowski et al. ( 2023), a maximum of three bounding boxes, each larger than 5% of the 2D mask area, was extracted from a prediction. The values δ (k)x and δ (k) y are sampled from a uniform distribution between 0 and 5. The size of threshold in pp2 is empirically set to 10% of the size of P j . Other parameters, such as the initial slice position and K, were studied in hyperparameter experiments on Ds without postprocessing and after pp1 to determine optimal values. As comparison to FMs, two 3D full resolution nnUNets (Isensee et al., 2021) for Ds and Dk were trained. The training details are reported in Appendix B. For Dt, the trained nnUNet from Wasserthal et al. (2023) was deployed. Another 3D full resolution nnUNet was trained In addition, we analysed the number of annotated pixels in relation to the Dice to investigate the annotation-efficiency. 
results: 
 The evaluation results of our prompt propagation approaches without postprocessing (no pp) and after applying both postprocessing steps (pp), the traditional SAM, and the nnUNet predictions are shown in 2 .Figure 3 shows the annotation-efficiency of our approaches compared to fully supervised methods. An ablation study about the number of prompts used for initialization considering performance and annotation time is given in Appendix C.3. In Appendix C.4, performance across different slice ranges surrounding the initial slice is evaluated to understand performance evolution across volumes. The results after training a nnUNet with the masks obtained by combination pp1 are 90.19% Dice and 2.77mm HD95, while for training a nnUNet with the ground truth labels the results are 98.35% Dice and 0.92mm HD95.According to Figure 8 in Appendix C.5, the best results in the hyperparameter experiments are achieved with the following settings: The initial slice is sampled from -25% to +25% around the object's center. For the stochastic-based approach, K is set to 10 and if at least 5 predictions include a particular pixel, it is included in the final mask.  
discussion: 
 In this work, we have presented multiple training-free strategies to apply SAM with box prompts to 3D bone CT scans starting with only two pixels annotated. We have introduced four methods, following different strategies of using the adjacent information. Our experiments do not reveal a definitive trend favoring one of our methods. However, our methods significantly reduce the number of annotated pixels and compared to using SAM in the traditional slice-by-slice manner or a fully supervised segmentation method, while maintaining a certain level of segmentation performance and showing promising results. Metal artefacts, similar structures in close proximity to each other or interlocking structures, as well as objects that split into multiple components have a negative effect on our training-free strategies. In some cases, they can lead to segmentation overflow to surrounding structures or a lack of components tracking during a split (see Figure 12 ). Aside from that, a lower contrast between bone and background decreases the performance of our methods, as seen in Dt1, aligning with the limitations of SAM (Mazurowski et al., 2023) . However, our proposed strategies show very good results when the segmentation task is easier and there is a good bone-background contrast. For example, for Ds, the Dice Coefficients are only 2 -3% off the traditional SAM (see Table 1 ). As a very thin structure, the scapula is difficult to manually annotate in a smooth 3D manner, resulting in some holes in the ground truth due to annotation errors (see Figure 4 Ds). Even when the reference standard is not completely correct, our methods produce smooth surfaces (see Figure 2(a) ).Analysing the individual methods reveals that baseline and nested approach can show severe over-propagation if an error is propagated (see Figure 12 ). In contrast, stochasticbased methods are more conservative, requiring agreement between multiple predictions. However, they suffer from vanishing boxes and thus, missing segmentation, when agreement is lacking (see Figure 2 (b), Figure 12(b) ). Combining stochastic and nested strategy does not balance weaknesses and may even amplify them (see Figure 12 ). Our current postprocessing steps target two reoccurring segmentation mistakes (see Figure 11 ). SAM can predict multiple disconnected components for one box, detecting structures similar to the object of interest. Removing all components, that are not connected to the largest component predicted from the initial box (pp1), improves performance. Moreover, SAM lacks spatial context, leading to the box not always vanishing at the top and bottom of the object. Restricting the box size (pp2), corrects for small additional structures. Using only pp1 yields greater performance gains, as the corrections are generally larger than pp2 corrections made at the object's boundary (see Table 3 , Figure 11 ). However, since each postprocessing step addresses another error pattern, the best results are achieved applying both (see Table 1 ).As shown in Figure 3 , there is a trade-off between performance and annotated pixels. While nnUNet consistently outperforms other solutions, it requires extensive ground truth labels for each new task and shift in data characteristics. In contrast, traditional SAM reduces annotated pixels by a magnitude of 2, and it seems robust to data and task changes. However, SAM lacks scalability for high-throughput 3D segmentation tasks due to the demand of manual interactions per slice. Our approaches are independent of the number of slices. Figure 5 shows that more user-provided prompts enhance performance in our approaches until reaching the fully prompted SAM level, which comes with the cost of additional annotation time. The annotation-efficient predictions can be used for pseudolabel training or as a starting point for model-assisted labeling. Training nnUNet with SAM predictions (see Figure 13 ) achieves 90% Dice, demonstrating the feasibility of creating a training dataset for supervised learning. Aside annotated datasets, our study provides insights for selecting training samples for efficient fine-tuning. For example, analyzing performance evolution throughout the volume reveals that for some slices the performance is closer to nnUNet and better than fully-prompted SAM (see C.4). Thus, our methods have value for some slices, but more challenging ones may require supervised approaches.A limitation of our method is relying solely on prediction information from adjacent slices, excluding image information. To enhance segmentation performance without increasing annotation effort, incorporating a priori information like shape could be beneficial. Further improvements in prediction quality could be achieved by exploiting the full range of geometric prompts, combining box with positive and negative point prompts to reduce ambiguity. The performance of our methods depends on the underlying SAM version, which should be investigated further in combination with models fine-tuned on medical data, like Med-SAM (Wu et al., 2023) or SAM-Med2D (Cheng et al., 2023b) . It is important to note that our goal is not to achieve state-of-the-art results but to offer a training-free strategy for using prompt-based 2D segmentation FMs, like SAM, in 3D medical data with one prompt per class per volume rather than one prompt per slice. 
conclusion: 
 We have presented multiple training-free strategies to apply the box-prompt mode of SAM to 3D bone CT scans with only one initial box prompt. Our method significantly decreases the number of annotated pixels while maintaining a certain level of segmentation performance. This work is another step towards the applicability of FMs for 3D bone segmentation. 
 
 Appendix: In order to evaluate how the performance changes if multiple prompts are used for initialization, the number of initialized slices is increased to 3, 5, 7, 10, 20, 30, 40, 50 for Ds. The slice chosen randomly around the center of the object is kept the same, the additional slices are chosen randomly in the volume of the object. The propagation is started from the center slice. When a slice with given prompt is reached, this initialized prompt is used instead of the prompt generated from prediction for the propagation anew. The Dice coefficient for the different numbers of prompts is shown in Figure 5 (blue dots).An exact time analysis of the prompt annotation process was not conducted, since the prompts have been generated automatically from the existing ground truth labels. In a real-world scenario without ground truth labels, the prompts need to be drawn manually. In order to assess the annotation time depending on the prompts used for initialization of our approaches, an annotation time estimation is calculated as:EQUATIONwith the annotation time for one prompt per class t a , the number of classes c, the time to scroll and navigate through the 3D volume per slice t s and the number of slices N . For Ds, the number of classes is 4. Under the assumption that drawing one prompt per slice takes 5 seconds per class and scrolling through the volume takes 2 seconds per slice, Equation ( 5) is only dependent on N which we set equal to the number of initialized prompts. For Ds, the average number of slices containing all structures is maximal 269, which we use as N for fully-prompted SAM. Figure 5 (red line and markers) shows the estimated annotation time for the same set of prompt numbers as used for the performance gain analysis. To show the performance evolution across volumes, the performance is evaluated for different subsets of slices surrounding the slice with initial prompt. Figure 6 and Figure 7 shows the results for all four datasets and different ranges of slices. The trend is a decrease of performance since errors accumulate and are propagated towards the boundary of the object. Dataset Ds is an exception of this trend due to the discrepancy between full-bone and cortical-bone segmentation. While ground truth labels represent full-bone segmentation, predictions primarily capture cortical bone, particularly in the center of the object (see Figure 9 ). Increasing the number of slices towards the boundary, where predictions are filled, improves Dice scores.  


Decision: The proposed work presented several prompt strategies to leverage the 2D SAM model for 3D bone segmentation. In the experiment, the authors evaluated these strategies and provided comparisons with other methods.

Reviewers HjpC and BeaE agreed that the proposed work facilitates the generation of pseudo labels, thereby reducing the annotation time. However, two reviewers (GcmQ and BeaE) also questioned the idea of using SAM for CT segmentation, noting that SAM was not originally designed for medical imaging. I also agree with the reviewer BeaE that an existing pretrained out-of-box tool,  such as TotalSegmentator (TS), can segment bone structures as discussed in this paper. I am particularly curious why the TS only achieved the DSC of 0.5 on the internal Ds dataset. It might be due to the domain shift issues, as explained by the authors.

Nevertheless, as mentioned in the rebuttal, this paper aims to demonstrate how SAM can be used for medical image segmentation through different prompts strategies. The proposed paper empirically and effectively showcases this concept. As pointed out by one reviewer, applying vision foundation models in medical imaging is still new. This manuscript provides valuable insights for other researchers to explore this emerging field.

Reviews: review: paper_summary: paper_summary: This paper introduces a training-free approach for extending 2D SAM to 3D segmentation using prompt propagation, reducing labeling efforts from pixel-level annotations to a single initial box prompt. The exploration of four box prompt placement strategies aims to optimize segmentation performance. The proposed method demonstrates competitive results against the nnUNet benchmark and fully-prompted SAM, underscoring the potential of prompt-based methods in 3D medical segmentation.
main_review: main_review: The author addressed my concerns with detailed illustrations. The paper provides experimental results for future work on reducing the annotation workload with SAM-based methods, leading me to increase the score in the final judgment.
strength_weakness: strength_weakness: 1. The distinctions among the three prompt propagation strategies appear minimal, lacking a definitive conclusion on their comparative effectiveness.
2. A noticeable performance gap persists when compared to nnUNet and fully-prompted SAM, indicating room for improvement.
questions: questions: While there's a performance disparity with nnUNet and fully-prompted SAM, the concept of training-free 3D segmentation using a single box prompt remains compelling. Addressing the questions raised about the weaknesses and detailed comments, along with a more thorough discussion and theoretical illustration for the box placement strategies, could substantially improve the manuscript's contribution as a methodological and validation study.
paper_summary: paper_summary: -The authors introduce several automatic box-prompt generation methods for the Segment-Anything Model (SAM), given only one initial box prompt on a certain slice of a 3D volume. This would then propagate to the rest of the slices.

-The experiments were conducted on three different datasets: two in-house and one public, for bone segmentation
main_review: main_review: I was wondering if other pre-trained models for medical imaging could deliver high-quality segmentations, i.e., above 90% in terms of the Dice coefficient, or perform better than SAM. In such cases, the proposed algorithm could be developed based on these models, or we could directly use these models to obtain segmentations since generating prompts is not considered time-consuming and does not require dealing with large datasets. However, the authors have addressed my concern.

In addition, the authors provide more details on SAM Med 2D. However, the training-free strategy does not work well, and I believe it offers a valuable experimental suggestion for future studies in the field of reduced prompting. This field has not been thoroughly explored, and it is important for practical applications. This also highlights their limitations with different models.
strength_weakness: strength_weakness: Overall motivation (not about the performance):

-I understand the importance of training-free strategies for the field. However, I am curious about the time it would take to input additional box prompts for more slices. If it does not significantly increase the time, why not use more box prompts to enhance overall performance

-I am unclear about the next steps after segmentation is produced. I speculate that it could be utilized for pseudo-label learning. However, I question its feasibility given the unimpressive Dice scores. Additionally, considering the inherent Hounsfield Unit (HU) values make bone segmentation from CT images relatively straightforward, why not employ a pretrained model for direct segmentation.
questions: questions: -please see the detailed comments and weaknesses. I need the authors to address the trade-off between performance and the number of prompts, including the time required to provide them. Such an ablation study could demonstrate the contribution of the proposed work.
paper_summary: paper_summary: In this paper, 3 strategies for sequential box generation as input to the Segment Anything Model (SAM) are proposed for 3d CT data. 2 private and 1 public data set are analyzed. The results are compared with 3D nnUNet, and while the strategies improve blind usage of SAM, the results are far from state-of-the-art.
main_review: main_review: How to reduce the amount of human labor when training networks is an important question. From the discussions, it seems that the authors also wish to discuss the concept of Foundational Models and that this is the reason why SAM was used. However, since the paper only discusses SAM, it is not a discussion on the definition of what a Foundational Model is. And while the concept of clever usage of prompt boxing is interesting, SAM is still the wrong model for the proposed problem, analogously trying to make a car fly by showing it where to drive: Even if you drive off a cliff and fly momentarily, it still is a weird strategy for designing machines that can fly.
strength_weakness: strength_weakness: In general, I find the use of SAM for the problem of CT segmentation a weird choice since it is not trained for this purpose and it is known to work less than optimal as the authors also claim. It is not well argued that the choice of prompt boxes should solve this problem.

The paper contains many errors which make it difficult to read: There is a spelling error in the title "Propgation" I presume. The logical connection between the abstract's two parts before and after "In our work" is not easy to understand. The notion of "prompt placement" is not explained until late in the article, making it unclear. Reporting results on private data makes the results nonrepeatable by the reader. The term "unidirectional" above (1) is confusing since the Stochastic approach explained below is also unidirectional. The parameters for the distribution above (2) are not defined. Also in (2), I'm guessing that "with P_i ..." is to be deleted.
questions: questions: I don't find this work ready for publication, and specifically, the authors should somehow demonstrate that the choice of prompt is more important than retraining the model for the specific problem.

Comments: title: The discussion period begins
Rebuttal for Reviewer GcmQ
Rebuttal For Reviewer BeaE
Rebuttal for Reviewer HjpC
General Comment
comment: Our latest analysis shows that the performance of our strategies indeed depends on the underlying model, i.e., SAM and SAM Med 2D, which should be further investigated. Considering the observed drop in performance with pretrained models, such as SAM Med 2D, the performance of SAM Med2D with our strategies is consequently affected. We believe that this observation is indeed worth mentioning. If we are still allowed to make a change to our manuscript, we would mention this observation in our discussion. 

We thank the Reviewer for the discussion which helped us making this observation.
Thank you for addressing my concerns. It appears that your algorithm does not work with different models (the inconsistent performance), which might be a limitation worth mentioning in the paper, even if only in a sentence. The area of training-free or reduced prompting has not been well explored, and I believe it presents a valuable experimental suggestion for future research. Good luck!
Based on the Reviewer’s question, we have evaluated our strategies based on the pretrained SAM Med 2D on Ds. The results after postprocessing can be found in the table below with the comparison to SAM performances for more convenience.

Examples for all four strategies can be found [here](https://www.dropbox.com/scl/fi/6dhw3oseewgm5voavuce5/MIDL_2024_examples02.pdf?rlkey=dfu49g8moir8pfsrzclh3aq8j&dl=0) (this link will be valid until the decision deadline).

| Method                    | Dice (%) | HD95 (mm) |
|---------------------------|----------|-----------|
| SAM fully-prompted        | 90.0     | 2.5       |
| SAM baseline              | 87.6     | 4.2       |
| SAM stochastic            | 87.6     | 4.0       |
| SAM nested                | 88.1     | 4.1       |
| SAM combined              | 87.9     | 4.5       |
|                           |          |           |
| SAM Med 2D fully-prompted | 75.3     | 9.4       |
| SAM Med 2D baseline       | 43.2     | 63.9      |
| SAM Med 2D stochastic     | 46.0     | 60.6      |
| SAM Med 2D nested         | 46.3     | 56.1      |
| SAM Med 2D combined       | 44.5     | 61.2      |
Thanks for these. A quick question: what's the performance of your algorithm based on pretrained sam med 2D.
The Reviewer is correct that the findings in previous literature studies show variations in performance across different datasets and tasks. However, as already mentioned for another Reviewer, several studies have identified specific characteristics where SAM performs better, such as foreground-background contrast and well-defined object borders [1,2]. These criteria are met in bone segmentation from CT scans. Thus, we hypothesized that SAM will show promising results on our datasets.

The improvement of our method lies in the application of SAM, or other related fine-tuned versions. As stated in our manuscript, our goal is not to improve the performance of SAM, but its applicability for 3D data. By design, our strategies are limited to the underlying SAM-version. However, we believe that our training-free strategies will help improve the performance in the future, due to the potential next steps as mentioned for Q2.

Nevertheless, in order to address the Reviewer’s question about a performance, we performed a preliminary comparison. The Reviewer is correct that there are pretrained models publicly available for medical imaging. On the one hand, there are pretrained models that have been trained in a fully-supervised manner on a large CT dataset using pixel-level annotations. On the other hand, Foundation Models (FM) offer generalisability and by definition, they can be applied to unseen structures without requiring additional training or annotations. Since we were not sure which of the two categories the Reviewer was referring to, we would like to discuss concrete examples from both categories. We performed preliminary evaluations for the mentioned models on our dataset Ds and we provide examples of the predictions [here]( https://www.dropbox.com/scl/fi/7xqaatmpxv4h0a6g5a0r7/MIDL_2024_examples.pdf?rlkey=g6fgx990pcxlfzawqbu6ned66&dl=0) (this link is valid until the decision deadline).

**TotalSegmentator [4]**

This is an example for a fully supervised segmentation network, pretrained on CT scans. Among others, it was trained on humerus and scapula segmentations. Thus, we have made preliminary experiments on our internal dataset Ds. The averaged Dice is below 50%. We believe that a difference in acquisition protocol and scanned anatomical views might be the reason for the low performance. However, this shows, that fully supervised models are specialized and cannot be transferred easily to other structures or data samples.

**Med-SAM [1] and SAM Med 2D [2]**

Both models were mentioned by the Reviewer as SAM version fine-tuned for medical data. We have evaluated both models in a fully-prompted manner on our dataset Ds. The average Dice is around 50% and around 75% for Med-SAM and SAM Med 2D, respectively. Both performances are far below the performance of SAM with 90%. Visual inspection of the predictions shows a non-smooth prediction with block-like structures.
Although these are preliminary results of Med-SAM and SAM Med 2D on Ds, it shows that out-of-the-box application of fine-tuned SAM versions do not automatically perform better than SAM. As stated in our discussion, we believe that these need to be investigated further.

**SegmentAnyBone [3]**

The authors of [3] also identified the prompt-based application of SAM as a challenge and thus, introduced a fully automated mode in SegmentAnyBone. The fully automated mode is realized by hybrid prompting engineering, i.e., the prompt input during 70% of the training iterations is omitted. As already mentioned in our previous response, SegmentAnyBone was trained on MRI data and thus, we hypothesized that it does not perform well on CT data. In order to show that, we generated some predictions with the fully automated mode in a 3D and slice-by-slice manner (with and without attention mechanism), which shows very unsatisfying results (see examples). Unfortunately, the prompt-based is not further explained and we couldn't find direct support in the repository, so it was not possible at this moment to test it with our data.

**Comparison on Ds**
| Model            | Dice (%) | HD95 (mm) |
|------------------|----------|-----------|
| SAM*              | 90.0     | 2.5       |
| SAM Med 2D*       | 75.3     | 9.4       |
| Med-SAM*          | 51.8     | 12.1      |
| TotalSegmentator | 48.8     | 186.5     |
'* fully-prompted

[1] Ma, Jun, et al. "Segment anything in medical images."

[2] Cheng, Junlong, et al. "Sam-med2d."

[3] Gu, Hanxue, et al. "SegmentAnyBone: A Universal Model that Segments Any Bone at Any Location on MRI."

[4] Wassertheurer, Jakob, et al. “TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images“
Thanks for the feedback.

Let's directly jump into the Q4 and 5. 

As many studies have shown that SAM doesn't produce good results on medical images. There are some pretrained models, with CT or without CT, publicly available for medical imaging. What if their performances are better than yours or what are the improvements between their pretrained models and your algorithm. This point makes a clear contribution of submission. I don't think the improvements based on SAM is reasonable at the current timepoint.
Dear reviewers and authors,

Thank you for your contribution to MIDL24. The discussion period begins! I encourage all reviewers and authors to participate in the discussion to address questions and clarify uncertainties. 

Thank you!
We have addressed the Reviewer’s concerns by further clarifying our motivation and contribution. In addition, following the Reviewer’s suggestion, some changes to the manuscript have been made to improving clarity and readability. However, we would have appreciated a more elaborated and justified assessment in order to provide a more succinct answer to the comments.

**Q1) In general, I find the use of SAM for the problem of CT segmentation a weird choice since it is not trained for this purpose and it is known to work less than optimal as the authors also claim. It is not well argued that the choice of prompt boxes should solve this problem.**

A1) As the Reviewer correctly observed, SAM has been trained on natural images, not CT scans. However, SAM is introduced as segmentation FM and, by definition, FMs are capable of performing a wide variety of tasks, even those they were not explicitly trained for. The capacity for generalizability implies that it can be applied to unseen structures without requiring additional training, making it particularly valuable in domains challenged by data and annotation scarcity. Due to these attributes, the medical image segmentation community has already adopted SAM and evaluated its performance  [1,2,3,4]. The findings from previous literature studies indicate valuable contributions but significant variations in performance across different datasets and tasks. Several studies have identified specific characteristics where SAM performs better, such as clear foreground-background contrast and well-defined object borders [3,4]. These criteria are met in bone segmentation from CT scans. Considering that manual labeling of 3D CT scans is highly time consuming, specially in large structures such as bones, we believe that SAM can be effectively utilized for this task. However, we encounter two separate challenges: 1) the performance gap compared to fully supervised methods like nnUNet, and 2) the manual intervention necessary for providing prompts in the 2D architecture. The first challenge is addressed through adaptation techniques and fine-tuning, both requiring annotated datasets. 

Our strategies are focused on addressing the second problem, not the first. As outlined in our manuscript, our aim is not to produce new state-of-the-art results, but rather to offer an approach for utilizing prompt-based 2D FM, such as SAM, for 3D scans in an annotation-efficient way. The advantages of our strategies include being training-free and requiring only one bounding box per class per volume, which significantly reduces annotation time. However, we acknowledge that the limitations in SAM's performance cannot be overcome, given that the design of our work is centered around its usage. As the Reviewer states, our strategies improve blind usage of SAM. In our opinion, it is equally important not to blindly fine-tune SAM, but rather to prioritize informed decision-making in the fine-tuning process to optimize label and computational efficiency. Thus, we consider our results as promising because the predictions generated can be utilized in model-assisted labeling to obtain annotated datasets for fine-tuning SAM or for other fully-supervised networks. We hope that our study provides insights that can be leveraged alongside these datasets for efficient training strategies, ultimately enhancing performance.

**Q2) The logical connection between the abstract's two parts before and after "In our work" is not easy to understand. The notion of "prompt placement" is not explained until late in the article, making it unclear. Reporting results on private data makes the results nonrepeatable by the reader. The term "unidirectional" above (1) is confusing since the Stochastic approach explained below is also unidirectional. The parameters for the distribution above (2) are not defined. Also in (2), I'm guessing that "with P_i ..." is to be deleted.** 

A2) We recognize that reporting on private data makes the results non-reproducible by the reader. In order to overcome this limitation, we included a publicly available dataset in our analysis to show results that could be reproduced by the reader. After acceptance, we also plan on making our code publicly available.
In (2), we wanted to clarify that the prompt P_i is based on the previous segmentation mask M_i-1. We made additions to our manuscript regarding the logical connection in the abstract, the explanation of “placing a prompt”, the term “unidirectional”, and the distribution for delta (assuming that this was unclear).

[1] Maciej A. Mazurowski, et al. Segment anything model for medical image analysis: An experimental study. Medical Image Analysis, 89:102918, 2023.
[2] Dongjie Cheng, et al. SAM on medical images: A comprehensive study on three prompt modes, arXiv, 2023
[3] Jun Ma, et al. Segment anything in medical images, arXiv, 2023
[4] Sheng He, et al. Accuracy of segment-anything model (SAM) in medical image segmentation tasks. arXiv 2023
We thank the reviewer for the comprehensive review and we address the questions and concerns below:

**Q1) Annotation time of additional box prompts**

A1) We agree with the Reviewer that more prompts are beneficial for the prediction performance, as can already been seen by the comparison between our methods and fully-prompted SAM. Based on the Reviewer’s comment, we added an analysis testing different numbers of initialized prompts for Ds (Appendix C.3.). As expected, the results show an increase in performance with increasing number of prompts.

We did not perform an exact time analysis of the prompt annotation process since the prompts are generated automatically from the existing ground truth labels. However, in order to assess the annotation time depending on the prompts used for initialization of our approaches, an annotation time estimation is added in Appendix C.3. In the results, we can for example see that a gain in performance of 1.2% Dice comes with an annotation time of over 3 additional minutes based on our assumptions. It should be noted, that for other datasets, the performance gain might be higher compared to the additional annotation cost. However, this trade-off needs to be considered for downstream tasks. For example, in time constrained tasks, this time increase might not be acceptable.  The analysis as well as the insights gained from it, have been included in the manuscript.

**Q2) Next steps after segmentation and feasibility.**

A2) As the reviewer correctly suggested, the next step after segmentation could be pseudo-label training. The generated predictions can also be used as a starting point for model-assisted labeling process, combined with active learning strategies. In our experiment, where we trained a nnUNet on the predictions generated by our approaches, our aim was to demonstrate feasibility of one scenario, i.e. these predictions provide an approximation of the ground truth labels, enabling supervised training with significantly reduced annotation time. 

Aside from using the segmentations for a next step, our approaches also provide insight into the usage of SAM which can be leveraged for the selection of training samples for efficient fine-tuning. We conducted an additional evaluation of performance for various slice ranges surrounding the initial slice (Appendix C.4.). Results show that slices close to the initial prompt have more stable performance compared to boundary slices. This is due to error accumulation and propagation. An exception to that is Ds, which is discussed in C.4.. Understanding trends about performance distributions in volumes can inform slice-based correction strategies. Additionally, implementing a failure detection method atop predictions can help in identifying volumes or slices requiring human intervention (i.e., pre-select data for manual corrections). Our results indicate that some datasets or object regions have a higher need for attention to achieve satisfactory outcomes. While fine-tuning enhances performance, it demands significant data and computational resources. Thus, we believe that informed decision-making in fine-tuning is crucial for optimizing label and computational efficiency, especially in constrained domains like medical settings. 

The mentioning of next steps, and an additional evaluation of performance as well as the gained insights have been added to the manuscript.

**Q3) Considering the inherent Hounsfield Unit (HU) values make bone segmentation from CT images relatively straightforward, why not employ a pretrained model for direct segmentation.** 

A3) We believe that employing a pretrained model for direct segmentation is not straightforward because of 1) differences in bone HU coming from different scanners and CT scanning protocols and 2) differing artifacts across tasks (e.g., metal artifacts in CT, as seen in Dk). Nevertheless, bone segmentation in CT scans shows some characteristics, such as a good foreground-background contrast and a distinctive object border, that makes this task relatively easy compared to other medical image segmentation tasks, e.g., lesion segmentation. Evaluation studies of SAM [1] support the notion that these data characteristics are advantageous for SAM.

**Q4) Comparison to 'Segment Anything in Medical Images' [1] and 'SAM MED 2D' [2]?.** 

A4) We would like to clarify, that our method is not depending on the underlying 2D version of SAM and could be applied to Med-SAM [1] and SAM MED 2D [2] as well. We agree with the reviewer, an application to other FM models specialized on medical data is possible and as stated in our discussion, we would like to perform such a comparison in the future.

**Q5) SegmentAnyBone** 

A5) Since our datasets consists of CT scans and SegmentAnyBone was trained on MRI data, we believe that the performance of the pretrained model suffers from the modality shift. To the best of our knowledge there is no equivalent of SegmentAnyBone trained on (bone) CT data.
We thank the Reviewer for the helpful comments and comprehensive review.

**Q1) The distinctions among the three prompt propagation strategies appear minimal, lacking a definitive conclusion on their comparative effectiveness. Clarification on the statistical significance (p-values) between different methods in Table 1 would enhance the credibility of the results.** 

A1) We agree that the differences between the prompt propagation strategies could be highlighted. For that, we have included Figure 1 in the manuscript illustrating each strategy. Additionally, we have performed an analysis on statistical differences between the proposed strategies using bootstrapping. After Bonferroni correction, it is shown that no statistical differences could be demonstrated. It is important to note that the number of samples is very small in each of the datasets. 
However, qualitative analysis of the results showed noticeable differences of their effectiveness. Specifically, the baseline and nested approach suffer in some cases from over-segmentation. The reason for that is the error propagation, especially at the top or bottom of an object. The bounding box is not disappearing and includes mainly background (see Figure 10). In contrast, the stochastic approach is more conservative, suffering from under-segmentation in some cases (see Figure 10). The reason is that the required agreement between multiple predictions can lead to vanishing boxes if this agreement is missing.

The statistical analysis can be found in Appendix C.1. The general analysis of the approaches and discussion of visual inspection is discussed in Section 6 (3rd paragraph).

**Q2) A noticeable performance gap persists when compared to nnUNet and fully-prompted SAM, indicating room for improvement.**  

A2) The Reviewer is correct that a performance gap still remains between nnUNet and FM as previously reported in literature [1]. The goal of our work was 1) to shed light on the applicability of SAM in medical imaging, i.e., specifically the need of a 3D approach, and 2) to provide efficient, training free strategies to facilitate this application. Although we managed to show that it is possible to have an approach to segment 3D scans with almost negligibly annotation time, in new experiments (see Appendix C.3.), we saw that the performance gap to fully-prompted SAM could be reduced by increasing the number of needed prompts. 

Please note that our study did not initially aim at closing the performance gap between SAM and nnUNet, but we believe it opens the way for more efficient strategies for training segmentation approaches. As also commented by another Reviewer, our predictions can be used for various follow-up steps, among others pseudo-label learning or model-assisted labeling. As shown in an experiment in the discussion section, the obtained training-free segmentations can be used as training samples for fully-supervised approaches and the a performance of 90% Dice has been achieved. On the one hand, this helps identifying labels for fine-tuning FM in a more efficient way than just random selection as currently done. For example, in our dataset Dk we would choose to annotate the tibia bone, as this is a label that SAM doesn’t perform well on. Additionally, it gives insight which datasets or tasks even benefit from fine-tuning and which already show sufficient or stable performance. We added a discussion of next steps in our manuscript.

[1] Cheng, Dongjie et al., SAM on medical images: A comprehensive study on three prompt modes, 2023

**Q3) The paper should provide insights or discussions on instances where pp2 underperforms pp1.**  

A3) We agree with the Reviewer that our representation of the results may have caused some misunderstanding regarding the application of the postprocessing steps. To clarify the result representation, Table 1 has been adapted to only show results without postprocessing and with both postprocessing steps (pp1+pp2). Results from only applying pp1 or pp2 are included in the Appendix C.2. as ablation study. In addition, an example comparing the predictions without pp, with pp2, with pp1+pp2 (Figure 8 ) and an example without pp, after pp1, after pp2, and after pp (pp1+pp2) (Figure 9) are added to the appendix. From the results and the example, it can be seen that only applying pp2 underperforms pp1. The reason is that corrections made in pp1 are larger than for pp2. In pp1, disconnected structures, e.g., the rib in scapula predictions in Ds, are removed. These are commonly larger structures. In contrast, the corrections in pp2 are mainly made in a few slices at the top and bottom of the object, with fewer pixels affected. Following the Reviewer’s suggestions, these insights are added to the discussion.
We thank the Reviewers for their valuable feedback, which helped us improve the paper. We have carefully addressed each of their insights and suggestions, enhancing the clarity and exposition of the contribution of our work. We would have appreciated, though, a more elaborated and justified feedback by Reviewer GcmQ in order to provide this Reviewer a more succinct answer. The updated manuscript includes now insights about the differences between the prompt propagation strategies, as well as between postpocressing steps, with new experiments and qualitative analysis. Additionally, we have extended the quantitative analysis of the trade-off between annotation time and performance gains with additional prompts. Finally, we have expanded upon the significance and applicability of the results to enhance labeling efficiency, providing quantitative insights of its potential. We believe that these improvements have considerably enhanced the paper, adequately addressing all concerns brought up by the reviewers.