# Melanoma Segmentation with SAM-Like Models: Assessing the Influence and Limits of Bounding Box Input

Nicolas Martin

PEEKTORIA &

Univ. Grenoble Alpes,
CNRS, Grenoble INP§, LIG

F-38000 Grenoble France

nicolas.martin@

peektoria.com

0000-0002-2788-1042

Philippe Mulhem

Univ. Grenoble Alpes,
CNRS, Grenoble INP§, LIG
F-38000 Grenoble France

philippe.mulhem@

univ-grenoble-alpes.fr

0000-0002-3245-6462

Jean-Pierre Chevallet

Univ. Grenoble Alpes,
CNRS, Grenoble INP§, LIG
F-38000 Grenoble France

jean-pierre.chevallet@

univ-grenoble-alpes.fr

0000-0002-5945-9444

**Abstract**—While the Segment Anything Model (SAM) has demonstrated remarkable capabilities in general image segmentation, its direct application to medical images faces significant challenges, leading to domain-specific adaptations like MedSAM. In current medical literature, SAM-like models are often evaluated using “simulated” bounding box prompts derived from ground truth annotations, potentially overestimating their real-world performance. This paper addresses this underexplored problem by systematically evaluating the influence of bounding box quality on melanoma segmentation performance using the ISIC 2018 dataset (RGB skin image corpus).

Experiments were conducted on both the foundational SAM and MedSAM, utilizing a structured protocol that includes ideal bounding boxes, baseline heuristic prompts (full image, most probable location, centered point for SAM), and controlled perturbations. These perturbations comprise five levels of expanding, shrinking, and shifting the ideal bounding box (random disturbance between 5% and 25%).

Our results demonstrate a significant dependence of segmentation performance on bounding box quality for both models. Depending on the type of bounding box perturbation, the Dice score decreases from 84% under ideal conditions to 37% with inaccurate bounding boxes, corresponding to a reduction of more than 50%. Notably, shrinking the bounding box has a more detrimental effect than expanding it, with accuracy dropping substantially (e.g., MedSAM's Dice score decreases from 91.5 to 60.9 under a 25% shrinkage). Moreover, medical pre-training significantly improves robustness, as MedSAM outperforms SAM by approximately 7 Dice points under ideal conditions.

These findings highlight that although SAM-like models offer promising capabilities, their reliance on high-quality bounding box prompts currently limits their applicability for fully unsupervised segmentation in medical image processing pipelines. This underscores the need for either human intervention or more advanced automated prompting strategies in clinical applications.

**Index Terms**—Prompt-based segmentation model, Segment Anything, Melanoma

# I. INTRODUCTION

This work has been supported by MIAI@Grenoble Alpes (ANR-19-P3IA-0003). This work benefited from state aid managed by the National Research Agency under France 2030 bearing the reference ANR-23-IACL-0006.

The segmentation of medical images is a crucial step in many medical image processing pipelines [1]. Building such systems requires pixel-level annotations, which are particularly difficult and costly to obtain [2]. Recently, the Segment Anything Model (SAM) [3] was introduced as an approach for solving (almost) any type of segmentation task. This prompt-based model, which relies on a transformer network, was trained on more than one billion natural images to segment objects from prompts. Despite the initial claim of solving segmentation problem, these models are unable to properly cope with specific image domains [4], such as medical ones, and therefore require adaptation (e.g., MedSAM [5] for medical images). Apart from the need for specialized adaptation to the fields of application, such domain-specific segmentation models rely heavily on “prompts” (e.g., bounding boxes, points) that serve as strong indicators of the image region to consider for segmentation. In this paper, we explored the influence of prompt quality on segmentation performance for melanoma RGB images (ISIC 2018 dataset [6]). An overview of SAM and the adaptation to the medical domain through MedSAM, along with related work on the influence of input prompts, is presented in Section II. The experiments conducted are described in Section III, and results are presented in Section 4. The discussion and conclusions are presented in Section 5.

# II. PROMPT-BASED SEGMENTATION: SAM, MEDSAM, AND RELATED WORK

SAM-like models [3] have been reported in the literature as highly effective for segmentation tasks on everyday photographs. These models are composed of an image encoder (a Vision Transformer with attention mechanisms) and a mask decoder, enabling segmentation from various prompts such as points, bounding box or mask [3]. Outside the medical domain, prior research has shown that bounding box prompts often outperform point-based prompts [7].
From a methodological perspective, and as illustrated in Fig. 1, SAM encodes the image using a Vision Transformer [8]. Simultaneously, the input prompt (e.g., a point or a bounding box) is encoded into a 256-dimensional embedding vector. For bounding boxes, this involves summing positional encodings with learned embeddings to form a pair of embeddings: one combining the positional encoding of the top-left corner with a learned “top-left” embedding, and another for the bottom-right corner. The mask decoder then refines the image and prompt embeddings through a modified Transformer block that incorporates both self-attention and cross-attention. The refined embedding is subsequently upsampled, and a Multilayer Perceptron (MLP) maps the output token to per-pixel mask probabilities. Finally, these models can also operate without an explicit prompt: in this case, the prompt is simulated by generating a regular 32x32 pixel grid as input to the prompt encoder.

<figure><img src="image_1.png" /><figcaption>Fig. 1. Overview of segmenting from bounding box with Segment Anything Model (SAM) [3]. The red bounding box corresponds to the ideal bounding box prompt.</figcaption></figure>

Previous studies [9], [10] have reported that the original SAM exhibits limitations when applied to medical images, primarily because general-purpose models struggle to generalize effectively to specialized domains. Consequently, adaptations for medical imaging (e.g., MedSAM [5], [11], SAM-Med3D [12]) have been proposed. These models are trained on pairs of medical images and their corresponding segmentation masks, and they typically rely on a bounding box prompt during inference. Accordingly, experiments reported in [5] demonstrate that MedSAM outperforms the generic SAM model in the medical domain.

In the medical domain, when evaluating SAM models on test sets, existing literature (e.g., [3], [5]) typically generates “simulated” bounding boxes from ground-truth segmentation annotations, which are then used as prompts to the SAM-like system. For example, in the reported experiments of [5], ideal bounding boxes, i.e., axis-aligned minimum bounding boxes around the ground-truth segmentation mask (see Fig. 1), are used as prompts on the test data<fnref n="1" />. Such prompts may bias eval-

uation results by placing the system under artificially favorable conditions, a phenomenon already reported in the natural image domain [13]. Indeed, in real-world applications, where physicians face heavy workloads, bounding boxes cannot be expected to be ideal. It is therefore essential to systematically assess the extent to which a SAM-like segmentation process can cope with inaccurate bounding boxes. To our knowledge, this effect has not yet been systematically studied or reported in medical domain.

### III. BOUNDING BOXES STUDY PROTOCOL

In this paper, we systematically investigate the impact of perturbed bounding box prompt on the segmentation performance of SAM and its medically specialized variant, Med-SAM.

The experimental protocol is defined as follows. Consider a Segment Anything Model (SAM) that requires a bounding box prompt to segment a medical image from a test corpus $C = c_i$, where each $c_i$ has an associated ground-truth segmentation mask $s_i$:

1. Evaluate SAM using ideal bounding boxes constructed directly from the ground-truth segmentation.

2. Evaluate SAM using heuristic prompts derived from the ground-truth segmentation.

3. Evaluate SAM using controlled perturbed bounding boxes, generated by perturbing the ideal bounding boxes according to predefined functions.

To establish a reference, we define the ideal case as the setting in which bounding boxes are constructed as axis-aligned minimum rectangles enclosing the ground-truth segmentation mask (see Fig. 1). This evaluation will be referred to as *Ideal* throughout the paper.

#### A. Heuristic (Non-Perturbed) Prompts

In addition to the ideal case, other reference baselines were considered where bounding boxes are automatically generated through simple heuristics:

* **Agnostic prompt**: centered bounding box covering a fixed proportion of the image;

* **Full image**: bounding box covering the entire image;

* **Most probable location**: A data-driven prompt, defined from the most probable location and size estimated from the training set. This non-agnostic prompt is expected to facilitate segmentation more effectively;

* **Centered point-based**: a single point placed at the image center.

#### B. Perturbed Prompts

The core of our analysis focuses on perturbed bounding boxes, i.e., modified versions of the ideal boxes obtained through controlled and reproducible transformations. Let a bounding box be represented by two corner coordinates, $top\_left = (x_{tl}, y_{tl})$ and $bottom\_right = (x_{br}, y_{br})$. A perturbation $p$ transforms the ideal bounding box $<top\_left_{ideal}, bottom\_right_{ideal}>$ into a perturbed box $<$

<footnote n="1">¹On the training data, random perturbations ranging from 0 to 20 pixels were applied to the ideal bounding boxes, corresponding to a maximum of 1.9% of the image size [5].</footnote>
top_left$_{perturb}$, bottom_right$_{perturb}$ >. The following perturbation types have been investigated:

* **Shift**: the ideal bounding box is translated along the x and/or y axis, without altering its dimensions.

* **Shrink**: the bounding box is contracted, thereby excluding parts of the ground truth (i.e., $x_{tl}$ and $y_{tl}$ increase, and/or $x_{br}$ and $y_{br}$ decrease).

* **Expand**: the bounding box is enlarged, extending beyond the ideal region (i.e., decreasing $x_{tl}$, $y_{tl}$ and/or increasing $x_{br}$, $y_{br}$).

The experiments are designed to determine whether expansions degrade segmentation using SAM models differently from shrinkages, and to assess the effect of shifts.

Finally, inspired by classical random image transformations in PyTorch<fnref n="2" />, we also simulate realistic scenarios of not ideal prompt by generating random expansions and shrinkages, and analyze the stability of segmentation performance under such stochastic perturbations.

# IV. EXPERIMENTS

## A. Dataset

The dataset provided in the ISIC 2018 Challenge [6], [14] was utilized for our experiments. This dataset consists of RGB dermoscopic images of melanoma, accompanied by corresponding binary segmentation masks that distinguish foreground lesions from the background. An illustrative example of an image and its annotation is shown in Fig. 2. The dataset is well-suited to this study, offering heterogeneous and clinically validated dermoscopic images collected from multiple international centers, thereby capturing the variability encountered in real-world diagnostic practice. In this study, the images and annotations from Task 1 of the challenge was specifically used. This subset contains 2,594 training images, 100 validation images, and 1,000 test images. Each image is annotated for binary segmentation, enabling the precise identification of melanoma regions within the skin images.

## B. Evaluation metric

In this study, we perform binary segmentation. For each image $i$, the ground truth segmentation $Sg_i$ assigns pixels to either melanoma ($Sg_i^m$) or background ($Sg_i^b$). Similarly, the model's automatic segmentation $Sa_i$ produces labeling into sets $Sa_i^m$ and $Sa_i^b$. To evaluate the accuracy of this segmentation, we employ the Dice similarity coefficient, computed for each image, defined by equation (1):

$$ \text{Dice}(Sg_i, Sa_i) = \frac{2|Sg_i^m \cap Sa_i^m|}{|Sg_i^m| + |Sa_i^m|} \quad (1) $$

The reported performance corresponds to the mean of these per-image Dice scores across the test dataset. The coefficient ranges from 0 to 1, with 1 indicating perfect agreement between the predicted and ground-truth masks. For each experiment, we report the mean Dice score together with the

standard deviation and the 95% confidence interval, computed across all test samples using the Student's t-distribution.

<figure><img src="image_2.png" /><figcaption>Fig. 2. One image and its annotation, from ISIC 2018 [6] - Task 1 (ISIC_0000009)</figcaption></figure>

For reference, [15] reports an average Dice value of 86.84% on the ISIC 2018 dataset with a specialized U-Net. U-Net [16] is a widely recognized and strong baseline in medical image segmentation, and remains one of the most commonly used architectures for benchmarking segmentation models, particularly in melanoma segmentation [17].

## C. Reference runs and perturbations generation

This section outlines the various reference runs established for our experiments. For the evaluated corpus, the run characteristics are detailed below:

* **Ideal**: The bounding boxes are defined to be axis-aligned minimum around the actual region to be segmented from the ground truth.

* **Agnostic bounding boxes**: For an image $i$, these are defined by $top\_left = ([height_i*0.25], [width_i*0.25])$ and $bottom\_right = ([height_i*0.75], [width_i*0.75])$. This bounding box corresponds to a centered rectangle whose height and width are half of the initial image's respective dimensions.

* **Most probable location**: Based on the training dataset, the most probable bounding box region is defined by the average top-left coordinates and the average bottom-right coordinates calculated over the entire training set. In our case, these points are defined as $top\_left = (168, 24)$ and $bottom\_right = (487, 333)$.

* **Full image**: A bounding box covering the entire input image is provided.

* **Centered Point-based**: For the original SAM, which is designed to handle various prompt types, we also evaluated performance when providing a single point prompt centered on the ground truth object. Note that centered point prompts were only applied to SAM, as MedSAM's training focused primarily on bounding box inputs.

* **Naive model**: One naive model that predicts the foreground (1) for all pixels<fnref n="3" />.

<footnote n="3">³Given that every image has at least one foreground pixel, the naive all-zero prediction model invariably results in a Dice score of 0, and will not be reported.</footnote>

<footnote n="2">²https://docs.pytorch.org/vision/0.9/transform.html</footnote>
We now detail the specific types of bounding box perturbations investigated, noting that one perturbation is applied per image in the test set during evaluation. These perturbations are:

* **Shifting perturbations:** For these, the ideal bounding box was translated along the x and/or y axis. We evaluated five distinct levels of shift. For each level, the displacement in both x and y directions was a random percentage of the ideal bounding box's dimensions, with the maximum shift within the following ranges: [0%, 5%], [0%, 10%], [0%, 15%], [0%, 20%], and [0%, 25%]. This means both the top-left and bottom-right corners were displaced by these random percentage-based amounts while maintaining the original box dimensions.

* **Shrinking perturbations:** For these, the ideal bounding box was reduced in size. We evaluated five distinct levels of shrinkage. For each level, the reduction in both x and y directions was a random percentage within the following ranges: [0%, 5%], [0%, 10%], [0%, 15%], [0%, 20%], and [0%, 25%]. This reduction was applied symmetrically around the center of the ideal bounding box, increasing the top-left corner coordinates and decreasing the bottom-right corner coordinates proportionally.

* **Expanding perturbations:** For these, the ideal bounding box was expanded. We evaluated five distinct levels of extension. For each level, the expansion in both x and y directions was a random percentage within the following ranges: [0%, 5%], [0%, 10%], [0%, 15%], [0%, 20%], and [0%, 25%]. This expansion was applied symmetrically around the center of the ideal bounding box, decreasing the top-left corner coordinates and increasing the bottom-right corner coordinates proportionally.

<figure><img src="image_3.png" /><figcaption>Fig. 3. Examples of ideal bounding boxes and each type of perturbed prompt</figcaption></figure>

For each perturbation type (i.e., Shift, Shrink, Expand), five distinct perturbation levels were evaluated. At each level, every test image was perturbed independently by a random transformation within the specified range (e.g., [0%, 5%]): examples for each perturbations are presented in Fig. 3. The model was evaluated separately for each level, and the reported performance corresponds to the mean Dice score across all images at that level. No cross-validation was used; unless otherwise noted, results are from a single fixed test set with a fixed random seed. A summary of the evaluated prompt parameters is provided in Table I.

### D. Evaluated models

To evaluate the impact of bounding box perturbations, experiments were conducted using two models: the foundation

<table><caption>TABLE I
SUMMARY OF EVALUATED PROMPTS AND PARAMETERS</caption><thead><tr><td>Parameter</td><td>Values / Levels</td></tr></thead><tbody><tr><td>Prompt type</td><td>Ideal, Full image, Centered point, Agnostic, Most probable location</td></tr><tr><td>Perturbation type</td><td>Shift, Shrink, Expand</td></tr><tr><td>Perturbation levels</td><td>[0-5]%, [0-10]%, [0-15]%, [0-20]%, [0-25]%</td></tr></tbody></table>

Segment Anything Model (SAM) [3] and its medical adaptation, MedSAM [5]. The comparison across varying perturbation levels was designed to assess whether MedSAM's domain-specific training improves robustness to imprecise prompts relative to the general-domain SAM trained on natural images. For consistency and fair comparison, the ViT-B variant of SAM was used, as this architecture was employed exclusively in the training of MedSAM.

Both SAM and MedSAM were used with their publicly available pretrained weights, without any additional training or fine-tuning on the ISIC 2018 dataset. The models were directly evaluated on the test set using the provided prompts. No post-processing operations were applied to the predicted segmentation masks.

The code for these experiments is publicly available at https://github.com/martinicmrim/sam_bbox_perturbation.

## V. RESULTS

Tables III and IV report the segmentation performance of SAM and MedSAM under bounding box perturbations, while Table II summarizes their performance with heuristic (non-perturbed) prompts. A naïve baseline that predicts a full foreground mask for every pixel achieved a Dice score of 39.7 (SD=24.0, 95% CI=[38.2, 41.2]), establishing a lower bound for comparison.

Under ideal bounding boxes, SAM reached 84.7 and MedSAM 83.5 mean Dice. When heuristic prompts were used, MedSAM consistently surpassed SAM. For instance, with *agnostic* boxes, MedSAM achieved 81.6 versus 61.8 for SAM; with *full-image* prompts, 79.3 versus 37.2. Even with simpler heuristics (*most-probable* or *centered-point*), MedSAM maintained Dice scores above 73%, whereas SAM remained near 43%.

Across perturbation experiments, both models exhibited a progressive decline in Dice score as perturbation magnitude increased, confirming their sensitivity to prompt accuracy. Among the three perturbation types, *Shrink* perturbations had the most detrimental effect, followed by *Shift*, whereas *Expand* perturbations were less harmful. MedSAM consistently outperformed SAM across all perturbation levels (e.g., 74.7% vs. 72.4% Dice at 25% shift) and retained higher accuracy even under severe perturbations.

MedSAM consistently outperformed SAM across all conditions, demonstrating greater robustness to bounding box inaccuracies. For instance, under *Shift* perturbations, MedSAM achieved a Dice of 89.6% at 5% and 74.7% at 25%, compared with 84.6% and 72.4% for SAM. Similar trends were observed
<table><caption>TABLE II
SEGMENTATION PERFORMANCE (DICE SCORE: MEAN / (SD) / [95% CONFIDENCE INTERVAL]) OF THE SAM MODEL [3] AND MEDSAM [5] USING HEURISTIC (NON-PERTURBATIVE) PROMPTS.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Heuristic (Non-Perturbative) Prompts</th></tr><tr><th><em>Ideal</em></th><th>Agnostic</th><th>Full image</th><th>Most probable location</th><th>Centered Point-based</th></tr></thead><tbody><tr><td rowspan="3">SAM</td><td>84.7<br>(1.6)</td><td>61.8<br>(2.9)</td><td>37.2<br>(2.6)</td><td>42.9<br>(3.1)</td><td>42.7<br>(3.6)</td></tr><tr><td>[83.7, 85.7]</td><td>[60.0, 63.5]</td><td>[35.6, 38.9]</td><td>[41.0, 44.9]</td><td>[40.4, 44.9]</td></tr><tr><td colspan="5"></td></tr><tr><td rowspan="3">MedSAM</td><td>91.5<br>(0.6)</td><td>60.2<br>(2.1)</td><td>41.5<br>(2.1)</td><td>51.3<br>(2.0)</td><td>NA</td></tr><tr><td>[91.1, 91.6]</td><td>[59.0, 61.5]</td><td>[40.2, 42.9]</td><td>[50.1, 52.6]</td><td></td></tr><tr><td colspan="5"></td></tr></tbody></table>

<table><caption>TABLE III
SEGMENTATION PERFORMANCE (DICE SCORE (%): MEAN / (SD) / [95% CONFIDENCE INTERVAL]) OF SAM [3] UNDER VARIOUS BOUNDING BOX PERTURBATIONS INPUT CONDITIONS</caption><thead><tr><th colspan="5">Perturbed Prompts</th></tr><tr><th colspan="5">Shift</th></tr><tr><th>5%</th><th>10%</th><th>15%</th><th>20%</th><th>25%</th></tr></thead><tbody><tr><td>84.6<br>(1.6)</td><td>83.0<br>(1.7)</td><td>79.5<br>(2.0)</td><td>76.6<br>(2.2)</td><td>72.4<br>(2.3)</td></tr><tr><td>[83.7, 85.6]</td><td>[81.9, 84.1]</td><td>[78.2, 80.7]</td><td>[75.3, 77.9]</td><td>[70.9, 73.8]</td></tr><tr><th colspan="5">Shrink</th></tr><tr><th>-5%</th><th>-10%</th><th>-15%</th><th>-20%</th><th>-25%</th></tr><tr><td>84.4<br>(1.6)</td><td>81.4<br>(1.7)</td><td>76.9<br>(1.9)</td><td>71.8<br>(2.0)</td><td>65.0<br>(2.2)</td></tr><tr><td>[83.4, 85.4]</td><td>[80.3, 82.4]</td><td>[75.7, 78.1]</td><td>[70.5, 73.0]</td><td>[63.7, 66.4]</td></tr><tr><th colspan="5">Expand</th></tr><tr><th>+5%</th><th>+10%</th><th>+15%</th><th>+20%</th><th>+25%</th></tr><tr><td>83.5<br>(1.8)</td><td>81.6<br>(2.0)</td><td>79.3<br>(2.2)</td><td>76.0<br>(2.4)</td><td>73.2<br>(2.6)</td></tr><tr><td>[82.4, 84.6]</td><td>[80.4, 82.8]</td><td>[77.9, 80.6]</td><td>[74.5, 77.5]</td><td>[71.5, 74.8]</td></tr></tbody></table>

<table><caption>TABLE IV
SEGMENTATION PERFORMANCE (DICE SCORE(%): MEAN / SD / [95% CONFIDENCE INTERVAL]) OF MEDSAM [5] UNDER VARIOUS BOUNDING BOX PERTURBATIONS INPUT CONDITIONS</caption><thead><tr><th colspan="5">Perturbed Prompts</th></tr><tr><th colspan="5">Shift</th></tr><tr><th>5%</th><th>10%</th><th>15%</th><th>20%</th><th>25%</th></tr></thead><tbody><tr><td>89.6<br>(0.7)</td><td>86.4<br>(0.8)</td><td>82.6<br>(0.9)</td><td>79.1<br>(1.1)</td><td>74.7<br>(1.2)</td></tr><tr><td>[89.2, 90.0]</td><td>[85.9, 86.9]</td><td>[82.0, 83.1]</td><td>[78.4, 79.7]</td><td>[74.0, 75.5]</td></tr><tr><th colspan="5">Shrink</th></tr><tr><th>-5%</th><th>-10%</th><th>-15%</th><th>-20%</th><th>-25%</th></tr><tr><td>87.0<br>(0.8)</td><td>80.8<br>(1.0)</td><td>74.1<br>(1.2)</td><td>68.0<br>(1.4)</td><td>60.9<br>(1.6)</td></tr><tr><td>[86.5, 87.5]</td><td>[80.1, 81.4]</td><td>[73.4, 74.8]</td><td>[67.1, 68.8]</td><td>[59.9, 61.9]</td></tr><tr><th colspan="5">Expand</th></tr><tr><th>+5%</th><th>+10%</th><th>+15%</th><th>+20%</th><th>+25%</th></tr><tr><td>92.0<br>(0.6)</td><td>91.2<br>(0.6)</td><td>89.2<br>(0.7)</td><td>86.8<br>(0.9)</td><td>84.2<br>(1.0)</td></tr><tr><td>[91.7, 92.4]</td><td>[90.8, 91.6]</td><td>[88.7, 89.6]</td><td>[86.3, 87.4]</td><td>[83.6, 84.8]</td></tr></tbody></table>

for *Shrink* (87.0% to 60.9%) and *Expand* (92.0% to 84.2%) perturbations.

Overall, MedSAM maintained higher segmentation accuracy across both heuristic and perturbed prompting scenarios, although both models exhibited substantial performance degradation when the bounding box deviation exceeded 15–20% from the ideal case.

## VI. DISCUSSION

Firstly, the findings underscore the critical role of pre-training on medical data. The observed difference of approximately seven Dice points between SAM and MedSAM under ideal conditions highlights the substantial benefit of domain-specific adaptation. This supports the view that, although foundational models like SAM exhibit strong zero-shot capabilities, fine-tuning or training on relevant medical datasets, as performed in MedSAM, yields significant performance gains for specialized tasks such as skin lesion segmentation [5]. These results emphasize the domain gap between general-purpose image understanding and the specific characteristics of medical image analysis.

Secondly, the analysis of bounding box perturbations reveals that both reductions and enlargements of the input bounding

box adversely affect segmentation accuracy. Shrinking the bounding box induces a more severe degradation than enlargement. For example, as shown in Table IV, MedSAM’s Dice score decreases from 91.5 to 84.5 with a +25% size increase, and to 60.9 with a -25% decrease. This indicates that, while SAM-like models leverage bounding boxes as strong spatial priors, they are more sensitive to the exclusion of relevant object regions than to the inclusion of additional background. Consequently, the bounding box acts as a crucial spatial guide, even without hard masking outside its boundaries. In line with [18], employing SAM-like models with slightly larger bounding boxes may be advantageous for generating initial segmentations, which can then be refined to train specialized, lightweight architectures (e.g., U-Net [16]) tailored to the target task. Such a strategy could be particularly useful within active learning frameworks to reduce annotation efforts [19]. Some recent studies (e.g., [13], [20]) have also explored fine-tuning SAM-like models to reduce their sensitivity to such perturbations.

Furthermore, the evaluation of heuristic prompting strategies—such as using the entire image, the most probable location, or a centered bounding box—consistently yielded degraded performance, despite the general centering of
melanomas within the dataset. This outcome highlights the models' reliance on an accurate spatial prior and suggests that simple heuristics for automated bounding box generation are insufficient to achieve high segmentation quality with current SAM-like approaches.

Although the documentation provided by the authors of MedSAM [5] indicates the exclusion of the ISIC 2018 dataset from its training corpus, the inclusion of closely related datasets (e.g., other ISIC challenge iterations) cannot be fully excluded. In contrast, SAM was trained exclusively on a large proprietary dataset of natural images. Therefore, the evaluation of MedSAM on ISIC 2018 may carry potential biases arising from incomplete transparency regarding its full training data composition. Future research should further investigate the impact of bounding box perturbations on more complex medical datasets, particularly those involving multiple classes or alternative imaging modalities, to assess the generalizability of these observations.

## VII. CONCLUSION

This study systematically investigated the influence of bounding box quality on the segmentation performance of SAM and its medical adaptation, MedSAM, using melanoma images from the ISIC 2018 dataset. The results clearly demonstrate that segmentation accuracy is highly sensitive to the quality of the bounding box prompt. Even moderate deviations from the ideal bounding box (i.e., shift, shrinking, or expansion) led to substantial performance degradation for both models.

A consistent and significant performance gap was observed between the two models: MedSAM outperformed SAM across all conditions, achieving approximately 7 Dice points higher accuracy under ideal prompts. This confirms the importance of domain-specific pretraining for medical image segmentation tasks and highlights the limitations of general-purpose models such as SAM when applied directly to specialized domains without adaptation.

These findings indicate that, while SAM-like models show remarkable zero-shot segmentation capabilities, their reliance on accurate bounding box prompts currently limits their direct integration into fully automated medical image analysis pipelines. Human intervention or robust automatic prompt generation mechanisms remain necessary to ensure reliable performance in clinical settings.

Building on these insights, several directions for future work are envisioned. First, the evaluation will be extended to additional medical datasets spanning diverse imaging modalities (e.g., radiography, MRI, CT) and multi-class segmentation tasks, to assess the generalizability of the observed trends. Second, multiple experimental runs with varying random seeds will be conducted to evaluate the stability and statistical significance of the results. Third, more realistic perturbation scenarios (e.g., combined translations and scalings) will be explored, alongside strategies for automatic bounding box refinement to enhance robustness. Finally, integrating SAM-like models into active learning frameworks or hybrid pipelines

(e.g., combination of SAM-based segmentations with specialized lightweight architectures such as U-Net) represents a promising direction to reduce annotation costs while maintaining high segmentation accuracy.

Overall, this work emphasizes both the potential and the current limitations of SAM-like models in medical image segmentation. By addressing prompt sensitivity and leveraging domain-specific adaptation, future research can further bridge the gap between foundation models and clinical-grade segmentation systems.

## REFERENCES

[1] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der Laak, B. van Ginneken, and C. I. Sánchez, "A Survey on Deep Learning in Medical Image Analysis," *Medical Image Analysis*, vol. 42, pp. 60–88, Dec. 2017, arXiv: 1702.05747. [Online]. Available: http://arxiv.org/abs/1702.05747

[2] S. Wang, C. Li, R. Wang, Z. Liu, M. Wang, H. Tan, Y. Wu, X. Liu, H. Sun, R. Yang, X. Liu, J. Chen, H. Zhou, I. Ben Ayed, and H. Zheng, "Annotation-efficient deep learning for automatic medical image segmentation," *Nature Communications*, vol. 12, no. 1, p. 5915, Oct. 2021. [Online]. Available: https://www.nature.com/articles/s41467-021-26216-9

[3] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, "Segment anything," in *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, 2023, pp. 3992–4003.

[4] Y. Huang, X. Yang, L. Liu, H. Zhou, A. Chang, X. Zhou, R. Chen, J. Yu, J. Chen, C. Chen, S. Liu, H. Chi, X. Hu, K. Yue, L. Li, V. Grau, D.-P. Fan, F. Dong, and D. Ni, "Segment anything model for medical images?" *Medical Image Analysis*, vol. 92, p. 103061, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841523003213

[5] J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, "Segment anything in medical images," *Nature Communications*, vol. 15, no. 1, pp. 1–9, December 2024. [Online]. Available: https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-44824-z.html

[6] N. C. F. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, and A. Halpern, "Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI)," hosted by the international skin imaging collaboration (ISIC)," in *2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)*, Apr. 2018, pp. 168–172, iSSN: 1945-8452. [Online]. Available: https://ieeexplore.ieee.org/document/8363547

[7] X. Zhong, C. Hu, J. Xie, J. Yi, Y. Zhou, D. Zeng, L. Zhou, K. Chen, W. Liu, and Z. Wu, "Segment Anything Model for Industrial Vision: A Comprehensive Evaluation with a New Metric, a New Dataset, and a Toolbox," Apr. 2024. [Online]. Available: https://www.researchsquare.com/article/rs-4257722/v1

[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," *arXiv:2010.11929 [cs]*, Oct. 2020, arXiv: 2010.11929. [Online]. Available: http://arxiv.org/abs/2010.11929

[9] W. Ji, J. Li, Q. Bi, T. Liu, W. Li, and L. Cheng, "Segment anything is not always perfect: An investigation of sam on different real-world applications," *Machine Intelligence Research*, vol. 21, no. 4, p. 617–630, Apr. 2024. [Online]. Available: http://dx.doi.org/10.1007/s11633-023-1385-0

[10] T. Zhou, Y. Zhang, Y. Zhou, Y. Wu, and C. Gong, "Can sam segment polyps?" 2023. [Online]. Available: https://arxiv.org/abs/2304.07583

[11] J. Ma, Z. Yang, S. Kim, B. Chen, M. Baharoon, A. Fallahpour, R. Asakereh, H. Lyu, and B. Wang, "MedSAM2: Segment Anything in 3D Medical Images and Videos," Apr. 2025, arXiv:2504.03600 [eess]. [Online]. Available: http://arxiv.org/abs/2504.03600

[12] H. Wang, S. Guo, J. Ye, Z. Deng, J. Cheng, T. Li, J. Chen, Y. Su, Z. Huang, Y. Shen, B. Fu, S. Zhang, J. He, and Y. Qiao, "SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images," Sep. 2024, arXiv:2310.15161 [cs].
[13] X. Zhao, Y. Pang, S. Chang, Y. Zhao, L. Zhang, C. Yu, H. Liu, J. Zuo, J. Ouyang, W. Lin, G. E. Fakhri, H. Lu, and X. Liu, "Inspiring the Next Generation of Segment Anything Models: Comprehensively Evaluate SAM and SAM 2 with Diverse Prompts Towards Context-Dependent Concepts under Different Scenes," Aug. 2025, arXiv:2412.01240 [cs]. [Online]. Available: http://arxiv.org/abs/2412.01240

[14] P. Tschandl, C. Rosendahl, and H. Kittler, "The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions," *Scientific Data*, vol. 5, no. 1, p. 180161, Aug. 2018, publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/sdata2018161

[15] S. Perera, Y. Erzurumlu, D. Gulati, and A. Yilmaz, "MobileUNETR: A Lightweight End-To-End Hybrid Vision Transformer For Efficient Medical Image Segmentation," Sep. 2024, arXiv:2409.03062 [cs]. [Online]. Available: http://arxiv.org/abs/2409.03062

[16] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," arXiv:1505.04597 [cs], May 2015, arXiv: 1505.04597. [Online]. Available: http://arxiv.org/abs/1505.04597

[17] Z. Mirikharaji, K. Abhishek, A. Bissoto, C. Barata, S. Avila, E. Valle, M. E. Celebi, and G. Hamarneh, "A survey on deep learning for skin lesion segmentation," *Medical Image Analysis*, vol. 88, p. 102863, Aug. 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841523001238

[18] C. Mattjie, L. V. d. Moura, R. C. Ravazio, L. S. Kupssinskü, O. Parraga, M. M. Delucis, and R. C. Barros, "Zero-shot performance of the Segment Anything Model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guidelines," May 2023, arXiv:2305.00109 [cs]. [Online]. Available: http://arxiv.org/abs/2305.00109

[19] N. Martin, J.-P. Chevallet, P. Mulhem, and G. Quénot, "Combining Image and Region Uncertainty-Based Active Learning for Melanoma Segmentation," in *2024 International Conference on Content-Based Multimedia Indexing (CBMI)*. Reykjavik, Iceland: IEEE, Sep. 2024, pp. 1-7. [Online]. Available: https://ieeexplore.ieee.org/document/10859208/

[20] B. Li, Y. Yuan, and W. Tan, "Optimization of MedSAM Model Based on Bounding Box Adaptive Perturbation Algorithm," in *Advanced Intelligent Computing Technology and Applications*, D.-S. Huang, H. Chen, B. Li, and Q. Zhang, Eds. Singapore: Springer Nature, 2025, pp. 13-24.