Stable Segment Anything Model

## Abstract

Abstract The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts which,
however, often require good skills to specify. To make SAM robust to casual prompts, this paper presents
the first comprehensive analysis on SAM’s segmentation stability across a diverse spectrum of prompt qualities,
notably imprecise bounding boxes and insufficient points. Our key finding reveals that given such low-quality
prompts, SAM’s mask decoder tends to activate image features that are biased towards the background or confined to specific object parts.
To mitigate this issue, our key idea consists
of calibrating solely SAM’s mask attention by adjusting the sampling locations and amplitudes of image features, while the original SAM model architecture and weights remain unchanged. Consequently, our deformable sampling plugin (DSP)
enables SAM to adaptively shift attention to the prompted target regions in a data-driven manner. During inference,
dynamic routing plugin (DRP) is proposed that toggles SAM between the deformable and regular grid
sampling modes, conditioned on the input prompt quality. Thus, our solution, termed Stable-SAM, offers several advantages:
1) improved SAM’s segmentation stability across a wide range of prompt qualities, while 2) retaining
SAM’s powerful promptable segmentation efficiency and generality, with 3) minimal learnable parameters (0.08 M) and fast adaptation. Extensive experiments
validate the effectiveness and
advantages of our approach, underscoring Stable-SAM as a more robust solution for
segmenting anything. † † Qi Fan (fanqics@gmail.com) done this work at Kuaishou Technology. † † Xin Tao (jiangsutx@gmail.com) is the corresponding author.

## 1 Introduction

Figure: Figure 1: The top row shows a performance comparison among SAM, HQ-SAM and our Stable-SAM, when provided with suboptimal prompts. Our Stable-SAM consistently surpasses other methods across prompts of different quality, demonstrating better or comparable performance to the SAM prompted by ground truth box. The bottom row displays the predicted masks and sampled important image features of SAM and Stable-SAM, with orange circles representing the attention weights, where a larger radius indicates a higher score. (a) SAM yields satisfactory segmentation results when provided with a high-quality box prompt. (b) Even a minor prompt modification leads to unstable segmentation output. SAM incorrectly segments the background, where the inaccurate box prompt misleads SAM to spend more attention to the background. (c) Our Stable-SAM accurately segments the target object by shifting more feature sampling attention to it.
Refer to caption: x1.png

The recent Segment Anything Model (SAM []) stands a significant milestone in image segmentation, attributed to its superior zero-shot generalization ability on new tasks and data distributions.
Empowered by the billion-scale training masks and the promptable model design,
SAM generalizes to various visual structures in diverse scenarios through flexible prompts, such as box, point, mask or text prompts.
Facilitated by high-quality prompts, SAM has produced significant performance benefit for various important applications, such as healthcare [, ], remote sensing [, ], self-driving [, ], agriculture [, ], *etc*.

Previous works mainly focus on improving SAM’s segmentation performance
assuming high-quality prompts are available, such as a tight bounding box (*e.g*., produced by SOTA detectors [, , ]) or sufficient points (*e.g*., 10 points) for the target object.
However, in practice SAM or in fact interactive segmentation often encounters inaccurate or insufficient prompts, casually marked up by users as inaccurate box or very sparse points, especially in the crowd-sourcing annotation platform.
Such inaccurate prompts often mislead SAM to produce unstable segmentation results as shown in Figure [1](#S1.F1).
Unfortunately, however, this critical issue has been largely overlooked, even though the suboptimal prompts and the resulting segmentation stability problem are quite prevalent in practice .

Note that there is no proper off-the-shelf solution for solving SAM’s segmentation stability problem with inaccurate prompts.
Simply finetuning SAM’s mask decoder with imprecise prompts may easily lead to catastrophic forgetting, undermining the integrity of the highly-optimized SAM model and thus sacrificing the zero-shot segmentation generality.
Although in the image domain deformable attention [, ] has shown impressive efficacy on adaptively shifting the model attention to informative regions, which may naturally address the attention drift issue caused by the misleading prompts,
a straightforward implementation of this idea can again compromise SAM’s integrity.

In this paper we present the first comprehensive analysis on SAM’s segmentation stability across a wide range of prompt qualities, with a particular focus on low-quality prompts such as imprecise bounding boxes or points.
Our findings demonstrate that, when fed with imprecise prompts, the SAM’s mask decoder is likely to be misguided to focus on the background or specific object parts, where
the cross-attention module is inclined to aggregate and activate image features of these regions when mutually updating the prompt and image tokens.
Such collaborative token updating mechanism usually suffers from attention drift, which is accumulated and propagated from the suboptimal prompt to the unsatisfactory segmentation results.

To address this issue, we present a novel deformable sampling plugin (DSP) with two key designs to improve SAM’s stability while maintaining its zero-shot generality.
Our key idea is to adaptively calibrate SAM’s mask attention by adjusting the attention sampling positions and amplitudes, while keeping the original SAM model unchanged:
1) we employ a small offset network to predict the corresponding offsets and feature amplitudes for each image feature sampling locations, which are learned from the input image feature map;
2) then, we adjust the feature attention by resampling the deformable image features at the updated sampling locations for keys and values of the cross-attention module in SAM’s mask decoder, keeping the original SAM model unchanged.
In doing so, we can shift the feature sampling attention toward informative regions which is more likely to contain target objects, and meanwhile avoiding the potential model disruption of the original highly-optimized SAM. Finally,
to effectively handle both the high- and low-quality prompts, we propose a dynamic routing module to toggle SAM between deformable and regular grid sampling modes.
A simple and effective robust training strategy is proposed to facilitate our Stable-SAM to adapt to prompts of diverse qualities.

Thus, our method is unique in its idea and design on solely adjust the feature attention without involving the original model parameters.
In contrast, the conventional deformable attention methods [, ] updates the original network parameters, which is undesirable when adapting powerful foundation models, especially when finetuning large foundation models. Our method thus improves SAM’s segmentation stability across a wide range of prompt qualities with minimal learnable paramters and fast adaptation, and meanwhile retains SAM’s powerful promptable segmentation efficiency and generality.

## 2 Related Works

Improving Segmentation Quality. Researchers have proposed various methods to enhance the quality and accuracy of semantic segmentation methods. Early methods incorporate graphical models such as CRF [] or region growing [] as an additional post-processing stage, which are usually training-free. Many learning-based methods design new operators [, , ] or utilize additional refinement stage [, ].
Recently, methods such as Mask2Former [] and SAM [] have been introduced, which address open-world segmentation by introducing prompt-based approaches. Along this line, a series of improvements [, ] have been proposed, focusing on prompt-tuning and improving the accuracy of segmentation decoders. However, these methods overlook a crucial aspect, which is how to generate high-quality segmentation results in cases where the prompt is inaccurate. This is precisely the problem that our method aims to address.

Tuning Foundation Models. Pretrained models have played an important role since the very beginning of deep learning [, , ]. Despite zero-shot generalization grows popular in foundation models of computer vision and natural language processing [, ], tuning methods such as adapter [] and prompt-based learning [, ] have been proposed to generalize these models to downstream tasks [, ]. These methods typically involves additional training parameters and time. We propose a new method that makes better use of existing features with minimal additional methods and can also produce competitive results.

Deformable Attention. Deformable convolution [, ] has been proved effective to help neural features attend to important spatial locations. Recently, it has also been extended to transformer-based networks [, , , ]. Such deformed spatial tokens are especially suitable for our task, which requires dynamically attending to correct regions given inaccurate prompts. However, previous deformable layers involve both offset learning and feature learning after deformation. In this paper, we propose a new approach to adjust the feature attention by simply sampling and modulating the features using deformable operations, without the need to train subsequent layers.

## 3 SAM Stability Anlaysis

In this section, we present a comprehensive investigation into the stability of SAM under prompts of varying quality.

### 3.1 Segmentation Stability Metric

Prior segmentation studies have focused on achieving high prediction accuracy, gauged by the Intersection-over-Union (IoU) between the predicted and ground truth masks.
This focus on high performance is justified as segmentation models typically produce deterministic masks for given input images, without requiring additional inputs.

However, SAM’s segmentation output depends on both the image and the prompts, with the latter often varying in quality due to different manual or automatic prompt generators.
In practical applications of SAM, segmentation targets are typically clear and unambiguous, independent of prompt quality.
For instance, in autonomous driving applications, the goal is to segment the entire car stably and consistently, regardless of whether the prompt—be it a point or a bounding box—initially focuses on a specific part such as the wheel or the car body.

Motivated by this application requirement, we introduce the segmentation stability metric. Specifically, SAM is capable of producing a set of binary segmentation maps $M\in\mathcal{R}^{B\times H\times W}$ for a single target object using $B$ prompts of differing qualities.
We define the segmentation stability (mSF) within the set as:

$$ $S=\frac{1}{B}\sum_{i=1}^{B}\mathrm{IoU}(M_{i},M_{\mathrm{union}}),$ (1) $$

where $\mathrm{IoU}(M_{i},M_{\mathrm{union}})$ represents the Intersection-over-Union between the $i$-th segmentation map $M_{i}$ and the collective foreground region $\bigcup_{i}^{B}M_{i}$ of all maps.
This new metric assesses the consistency across segmentations in each prediction, serving as a reliable indicator of stability, even without access to the ground truth masks.

### 3.2 SAM Segmentation Instability

We perform empirical studies to illustrate the segmentation instability of the current SAM with prompts of differing quality, thereby justifying our Stable SAM approach.

Model and Evaluation Details. The released SAM is trained with crafted prompts on large-scale SA-1B dataset.
We evaluate the segmentation accuracy and stability of the ViT-Large based SAM with different prompt types and qualities, including box prompts with added noise (noise scale 0.4) and point prompts with varying numbers of points (1, 3, 5, 10 positive points randomly selected from the ground truth mask).
For every input image and prompt type, we randomly select 20 prompts to compute their segmentation stability, average mask mIoU, and boundary mBIoU scores.
The evaluation utilizes four segmentation datasets as in HQ-SAM []: DIS [] (validation set), ThinObject-5K [] (test set), COIFT [], and HR-SOD [].

Table [1](#S3.T1) tabulates that SAM’s segmentation accuracy and stability significantly decrease with low-quality prompts, such as imprecise box prompts or point prompts with minimal points.
These analysis are performed on the four aforementioned segmentation datasets.
The varying segmentation accuracy and stability indicates that SAM’s mask decoder performs distinctly when dealing with prompts of varying qualities.

Figure: (a)
Refer to caption: extracted/5273614/figures/fig2/box_0_0_0_0.7366_0.5742.png

**Table 1: SAM’s segmentation accuracy and stability under prompts of varying quality.**
| Metric | GT Box | Noisy Box | 1 Point | 3 Points | 5 Points | 10 Points |
| --- | --- | --- | --- | --- | --- | --- |
| mIoU | 79.5 | 48.8 | 43.3 | 78.7 | 83.3 | 84.8 |
| mBIoU | 71.1 | 42.1 | 37.4 | 69.5 | 74.2 | 76.0 |
| mSF | - | 39.5 | 45.1 | 79.3 | 84.7 | 87.5 |

We visualize the image activation map for the token-to-image cross-attention in SAM’s second mask decoder layer to better understand its response to low-quality prompts.
We focus on the second mask decoder layer for visualization because its cross-attention is more representative, benefiting from the input tokens and image embedding collaboratively updated by the first mask decoder layer.
Figure [2](#S3.F2) demonstrates that an inaccurate box prompt causes SAM’s mask decoder to miss regions of the target object while incorrectly incorporating features from the background, or focusing on specific object parts.
It consequently leads to degraded segmentation accuracy and stability.

Overall, the above empirical evidence suggests that SAM potentially suffers from the attention drift issue, where suboptimal prompts misleadingly shift attention from the target object to background areas or specific object parts, thereby compromising the accuracy and stability of the segmentation results.
This motivates us to calibrate SAM’s mask attention by leveraging learnable offsets to adjust the attention sampling position towards the target object regions, thus boosting segmentation accuracy and stability.

## 4 Stable Segment Anything Model

### 4.1 Preliminaries

We first revisit the recent Segment Anything Model (SAM) and deformable attention mechanism.

Segment Anything Model.
SAM [] is a powerful promptable segmentation model.
It comprises an image encoder for computing image embeddings, a prompt encoder for embedding prompts, and a lightweight mask decoder for predicting segmentation masks by combining the two information sources.
The fast mask mask decoder is a two-layer transformer-based decoder to collaboratively update both the image embedding and prompt tokens via cross-attention.
SAM is trained on the large-scale SA-1B dataset.

Figure: Figure 3: (a) An illustration of our deformable sampling plugin (DSP) and deformable routing plugin (DRP) in SAM’s mask decoder transformer. DSP employs a small (b) offset network to predict the feature sampling offsets and amplitudes. Subsequently, DSP calibrates the feature attention by resampling deformable image features at the updated sampling locations, and feeds them into SAM’s token-to-image attention. DRP employs a small (c) MLP network to regulate the degree of DSP activation based on the input prompt quality. Note that our DSP adaptively calibrates solely SAM’s mask attention without altering the original SAM model.
Refer to caption: x3.png

Deformable Attention. Deformable attention [] is a mechanism that enables the model to focus on a subset of key sampling points instead of the entire feature space. This mechanism naturally addresses the attention shift problem in SAM caused by low-quality prompts.

In the standard self-attention, given a feature map $x\in\mathcal{R}^{H\times W\times C}$, the attention weights are computed across all spatial locations within the feature map.

In the deformable attention [], a uniform grid of points $r\in\mathcal{R}^{H_{G}\times W_{G}\times 2}$ are first generated as the references(^1^11With the grid size downsampled from the input feature map spatial size (H, W) by a factor of $s$, thus $H_{G}=H/s$ and $W_{G}=W/s$.) with the sampled image feature $x_{r}\in\mathcal{R}^{H_{G}\times W_{G}\times C}$.
Subsequently, a convolutional offset network $\theta_{\mathrm{offset}}$ predicts the offset $\Delta r=\theta_{\mathrm{offset}}(x_{r})$ for each reference point.
The new feature sampling locations are given by $r+\Delta r\in\mathcal{R}^{H_{G}\times W_{G}\times 2}$.
The resampled deformable image features $x_{r+\Delta r}\in\mathcal{R}^{H_{G}\times W_{G}\times C}$ are then utilized as the key and value features in the attention module.

Note that conventional deformable attention optimizes both the offset network and attention module.
Thus directly applying deformable attention to SAM is usually suboptimal,
because altering SAM’s original network or weights, *e.g*., substituting SAM’s standard attention with deformable attention and retraining, may compromise its integrity.

### 4.2 Deformable Sampling Plugin

To address the attention drift issue while preserving the SAM’s integrity, we propose a novel deformable sampling plugin (DSP) module on top of SAM’s original token-to-image cross-attention module, as shown in Figure [3](#S4.F3).

Specifically, given the prompt token feature $t\in\mathcal{R}^{\mathrm{T}\times C}$ and image feature $x_{p}\in\mathcal{R}^{H\times W\times C}$, the token-to-image cross-attention is:

$$ $\text{CAttn}(t,x)=\sigma(Q(t)\cdot K(x_{p})^{T})\cdot V(x_{p}),$ (2) $$

where $p\in\mathcal{R}^{H\times W\times 2}$ represents the image feature spatial sampling locations, $\sigma$ denotes the softmax function, and $Q,K,V$ are the query, key, and value embedding projection functions, respectively

Our DSP adaptively calibrate the feature attention by adjusting solely image feature sampling locations and amplitudes without altering the original SAM model.
Specifically, we utilize an offset network $\theta_{\mathrm{offset}}$ to predict the feature sampling offset $\Delta p\in\mathcal{R}^{H\times W\times 2}$, akin to that in deformable attention:

$$ $\Delta p=\theta_{s}(\theta_{\mathrm{offset}}(x_{p})),$ (3) $$

where $\theta_{s}$ is a scale function $s_{p}\cdot\mathrm{tanh}(*)$ to prevent too large offset, and $s_{p}$ is a pre-defined scale factor.
The offset network $\theta_{\mathrm{offset}}$ consists of a $1\times 1$ convolution, a $5\times 5$ depthwise convolution with the layer normalization and GELU activation, and a $1\times 1$ convolution. The updated feature sampling locations are $p+\Delta p$.
The numerical range of both $p$ and $p+\Delta p$ lies in $\{(0,0),...,(H-1,W-1)\}$, which is then normalized to the range $[-1,1]$ for feature sampling.
The feature amplitudes are predicted by the first convolutional layer and the image features $x_{p}$ are thus updated as $x_{p}^{\star}$, which are used solely for computing the feature attention.

Subsequently, we resample and modulate deformable image features $x_{p+\Delta p}^{\star}\in\mathcal{R}^{H\times W\times C}$ at the updated sampling locations $p+\Delta p$ with the learned feature amplitudes for keys and values.
Thus, our DSP calibrates the token-to-image cross-attention of SAM’s mask decoder as:

$$ $\text{DCAttn}(t,x)=\sigma(Q(t)\cdot K(x_{p+\Delta p}^{\star})^{T})\cdot V(x_{p +\Delta p}^{\star}).$ (4) $$

As $p+\Delta p$ is fractional, we apply a bilinear interpolation to compute $x_{p+\Delta p}^{\star}$ as in Deformable DETR [].

Note that our DSP only trains the deformable offset network to predict new feature sampling locations $p+\Delta p$ and feature amplitudes, and feeds the resampled and modulated deformable features $x_{p+\Delta p}^{\star}$ to SAM’s cross-attention module. Thus, the original SAM model remains unchanged.

### 4.3 Dynamic Routing Plugin

While our DSP can effectively handle suboptimal and even erroneous prompts, by redirecting SAM’s attention to informative regions which are more likely to contain the target objects, high-quality prompts can typically direct the model’s attention correctly to target regions. Thus, it is essential to properly control the DSP’s activation to prevent unwanted attention shifts.

To address this issue, we propose a novel dynamic routing plugin (DRP) that regulates the degree of DSP activation based on the input prompt quality.
The DRP can be formulated as follows:

$$ $\alpha=\sigma(\mathrm{MLP}(t_{o}))\cdot s,$ (5) $$

where $t_{o}\in\mathbb{R}^{\mathrm{1}\times C}$ is the prompt token feature corresponding to the output mask, $\mathrm{MLP}$ refers to a small MLP network that includes an MLP layer with LayerNorm and GELU activation, as well as an output MLP layer;
$s$ denotes a learnable scale and $\sigma$ denotes the softmax function.

We utilize the predicted values of $\alpha=[\alpha_{1},\alpha_{2}]\in\mathbb{R}^{1\times 2}$ to adaptively route SAM between DSP and original SAM’s attention mechanism.
Consequently, the token-to-image cross-attention output $\mathrm{O}(t,x)$ can be formulated as:

$$ $\mathrm{O}(t,x)=\mathrm{CAttn}(t,\alpha_{1}\cdot x_{p+\Delta p}^{\star}+\alpha _{2}\cdot x_{p})$ (6) $$

This soft dynamic routing strategy allows SAM to benefit from both DSP and its original zero-shot generality, contingent upon the quality of the prompt.

### 4.4 Robust Training Strategy

We propose a simple and effective robust training strategy (RTS) to assist our model to learn how to correct SAM’s attention when adversely affected by bad prompts.

Robust Training Against Inaccurate Prompts. SAM’s training, including HQ-SAM [], typically utilizes high-quality prompts given by precise bounding boxes or multiple points to accurately identify the target object. To address inaccurate prompts, our RTS incorporates prompts of varying qualities during training.
These prompts include groundtruth boxes, box prompts with added noise (noise scale 0.4), and point prompts with varying numbers of points (1, 3, 10 positive points randomly chosen from the ground truth mask).

Robust Training Against Ambiguous Prompts.
In real segmentation scenarios, target objects often occur
in cluttered environment, either occluding others or being occluded.
Even given an accurate, tight bounding box,
objects other than the target object will be enclosed. On the other hand, target objects are typically unambiguous even other objects are enclosed.
For instance, in MS COCO,
beds (occluded by quilt) are consistently regarded as target objects; the model must accurately segment the entire bed including accessories such as pillows and bedding.
Thus, SAM’s original ambiguity-aware solution,
which predicts multiple masks for a single prompt,
is generally suboptimal in well-defined realistic applications.
To address such “ambiguous” prompts,
our RTS incorporates synthetic occlusion images to make SAM conducive to accurately segment target objects. The occlusion images are synthesized by randomly introducing other objects to simulate “occluder” and “occludee” relationship.

Our RTS is general and applicable to various SAM variants to improve their segmentation stability.
Notably, our Stable-SAM with DSP and DRP experience the most substantial improvements from the application of RTS.

## 5 Experiments

**Table 2: Comparison on four HQ datasets among SAM, DT-SAM (finetuning SAM’s mask decoder), PT-SAM (finetuning SAM’s prompt token and its corresponding output MLP layer), HQ-SAM and our Stable-SAM, under prompts of varying quality.**
|  |  | Noisy Box | 1 Point | 3 Points |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Epoch | mIoU | mBIoU | mSF | mIoU | mBIoU | mSF | mIoU | mBIoU | mSF |
| SAM (baseline) | - | 48.8 | 42.1 | 39.5 | 43.3 | 37.4 | 45.1 | 78.7 | 69.5 | 79.3 |
| DT-SAM | 12 | 70.6 | 60.4 | 64.0 | 43.1 | 43.2 | 37.9 | 80.3 | 71.6 | 80.5 |
| PT-SAM | 12 | 70.8 | 60.2 | 64.1 | 43.0 | 42.9 | 38.3 | 80.1 | 71.8 | 80.4 |
| HQ-SAM [] | 12 | 72.4 | 62.8 | 65.5 | 43.2 | 44.6 | 37.4 | 81.8 | 73.7 | 81.4 |
| Stable-SAM | 1 | 82.3 | 74.1 | 82.3 | 76.9 | 68.4 | 71.1 | 84.0 | 75.8 | 84.9 |

**Table 3: Comparison on MS COCO and SGinW datasets. All models are prompted by noisy boxes (N-Box) that overlap with the ground truth boxes, with IoU ranges of 0.5-0.6 and 0.6-0.7.**
|  |  | MS COCO | SGinW |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | N-Box (0.5-0.6) | N-Box (0.6-0.7) | N-Box (0.5-0.6) | N-Box (0.6-0.7) | Learnable |  |  |  |  |  |
| Model | Epoch | mAP | mAP${}_{\mathrm{50}}$ | mAP | mAP${}_{\mathrm{50}}$ | mAP | mAP${}_{\mathrm{50}}$ | mAP | mAP${}_{\mathrm{50}}$ | Params | FPS |
| SAM (baseline) | - | 27.3 | 60.2 | 40.9 | 75.0 | 26.0 | 60.8 | 39.5 | 73.2 | (1191 M) | 5.0 |
| DT-SAM | 12 | 12.2 | 22.7 | 15.8 | 28.7 | 10.4 | 21.5 | 13.6 | 27.1 | 3.9 M | 5.0 |
| PT-SAM | 12 | 30.2 | 63.4 | 41.3 | 76.5 | 32.1 | 66.4 | 41.1 | 74.3 | 0.13 M | 5.0 |
| HQ-SAM [] | 12 | 31.9 | 65.5 | 42.9 | 77.1 | 33.6 | 68.4 | 42.2 | 75.9 | 5.1 M | 4.8 |
| Stable-SAM | 1 | 44.8 | 76.4 | 50.5 | 81.1 | 43.3 | 75.6 | 48.6 | 79.4 | 0.08 M | 5.0 |

Datasets.
For fair comparison we keep our training and testing datasets same as HQ-SAM [].
Specifically, we train all models on HQSeg-44K dataset, and evaluate their performance on four fine-grained segmentation datasets, including DIS [] (validation set), ThinObject-5K [] (test set), COIFT [] and HR-SOD [].
Furthermore, we validate the model’s zero-shot generalization ability on two challenging segmentation benchmarks, including COCO [], and SGinW [], where SGinW contains 25 zero-shot in-the-wild segmentation datasets.
More experimental results are included in the supplementary material.

Input Prompts.
We evaluate model’s accuracy and stability with prompts of differing type and quality, as described in Sec. [3.2](#S3.SS2).
For MS COCO and SGinW,
we do not use the boxes generated by SOTA detectors [, ] as the box prompt.
This is because their predicted boxes are typically of high quality and cannot effectively evaluate the model’s segmentation stability in the presence of inaccurate boxes.
Instead, we introduce random scale noises into the ground truth boxes to generate noisy boxes as the prompts.
Specifically, to simulate inaccurate boxes while still having some overlap with the target object, we select noisy boxes that partially overlap with the ground truth boxes with IoU ranges of 0.5–0.6 and 0.6–0.7.
We also evaluate our method using the box prompts generated by SOTA detectors.

Evaluation Metrics.
We select suitable evaluation metrics depending on testing datasets, *i.e*., 1) mask mIoU, boundary mBIoU and mSF for DIS, ThinObject-5K, COIFT, and HR-SOD;
2) mask mAP and mAP${}_{\mathrm{50}}$ for COCO and SGinW.

Figure: (a)
Refer to caption: extracted/5273614/figures/fig4/box/0_118_4_0.8621_0.7441.png

### 5.1 Comparison with SAM Variants

We compare our method with SAM and three powerful SAM variants.
HQ-SAM is a recent powerful SAM variant for producing high-quality masks. We also try two simple SAM variants by finetuning its mask decoder and the prompt token, *i.e*., DT-SAM and PT-SAM, respectively.
All our Stable-SAM models are trained by just one epoch for fast adaptation unless otherwise stated.

Stability Comparison on Four HQ Datasets.
Table [2](#S5.T2) shows the segmentation accuracy and stability on four HQ datasets, when models are fed with suboptimal prompts.
Notably, the use of noisy box prompts significantly reduces SAM’s performance, as evidenced by the drop from 79.5/71.1 (as shown in Table [1](#S3.T1)) to 48.8/42.1 mIoU/mBIoU, accompanied by a low stability score of 39.5 mSF.
This is probably because SAM was trained with solely high-quality prompts, thus seriously suffers from the low-quality prompts during inference.
The other three SAM variants, namely HQ-SAM, DT-SAM, and PT-SAM, demonstrate relatively better stability in dealing with noisy boxes, which can be attributed to their long-term training on the HQSeg-44K dataset.
Note our Stable-SAM can effectively address inaccurate box prompts, by enabling models to shifts attention to target objects.
Given a single-point prompt, both SAM and its variants exhibit the lowest accuracy and stability.
This indicates they are adversely affected by the ambiguity problem arising from the use of a single-point prompt.
Although, in most practical applications, users prefer minimal interaction with clear and consistent segmentation targets.
Our method maintains much better performance and stability when handling ambiguous one-point prompt, owing to our deformable feature sampling and robust training strategy against ambiguity.
When point prompts increase to 3, all methods performs much better, while other methods still under-perform compared with ours.

Generalization Comparison on MS COCO and SGinW.
Table [3](#S5.T3) presents the segmentation accuracy and stability when the models are generalized to MS COCO and SGinW with noisy box prompts.
Note that the DT-SAM performs the worst, probably due to overfitting on the training set, which compromises its ability to generalize to new datasets.
Our method consistently surpasses all competitors, particularly in handling inaccurate boxes (N-Box 0.5–0.6), where all noisy boxes have an IoU range of 0.5–0.6 with the ground truth boxes.
Note that our method has a minimal number of extra learnable parameters (0.08M) and can be quickly adapted to new datasets by just one training epoch.

Comparison Based on Detector Predicted Box Prompts.
Existing zero-shot segmentation methods typically choose powerful object detection model to generate high-quality boxes as the input prompts, such as FocalNet-L-DINO [, ].
We also evaluate our method in such setting.
Table [4](#S5.T4) presents that our model achieves comparable performance as SAM and PT-SAM when using the FocalNet-L-DINO generated high-quality boxes as prompts.
When using the R50-H-Deformable-DETR [] as the box prompt generator, our method achieves comparable performance as HQ-SAM.
Note that training and implementing SOTA detectors typically require large computational resources and the cross-domain generalization is still very challenging.
In practice, users tend to leverage interactive tools to annotate objects for their personalized datasets.
Our method substantially surpasses other competitors in such scenario, when the box can roughly indicate the target object.

**Table 4: Comparison on MS COCO with the box prompts generated by SOTA detectors or noisy box prompts.**
|  | FocalNet-L-DINO | R50-H-D-DETR | Noisy Box |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | mAP | mAP${}_{\mathrm{50}}$ | mAP | mAP${}_{\mathrm{50}}$ | mAP | mAP${}_{\mathrm{50}}$ |
| SAM | 48.5 | 75.3 | 41.5 | 63.7 | 27.3 | 60.2 |
| PT-SAM | 48.6 | 75.5 | 41.7 | 64.2 | 30.2 | 63.4 |
| HQ-SAM | 49.5 | 75.7 | 42.4 | 64.5 | 31.9 | 65.5 |
| Ours | 48.3 | 74.8 | 42.2 | 64.0 | 44.8 | 76.4 |

### 5.2 Analysis on Stable-SAM

Deformable Sampling Plugin.
Table [5](#S5.T5) shows DSP can be trained with high-quality prompts (without RTS) to improve the performance and stability on low-quality prompts, although the model still exhibits some instability.
When equipped with RTS, DSP can effectively learn to shift SAM’s attention to target objects when subjecting to inaccurate prompts.
To delve deeper into the deformable sampling mechanism, we visualize the sampled feature points and their corresponding attention weights.
Figure [4](#S5.F4) illustrates how our DSP effectively shifts model’s attention to the target object, resulting in increased attention weights.
Consequently, the cross-attention module aggregates more target object features into the prompt tokens, thereby improving the segmentation quality of the target objects.

Dynamic Routing Plugin.
We leverage DSP to dynamically route the model between the regular and deformable feature sampling modes, conditioned on the input prompt quality.
We find that DRP tends to route more DSP features when dealing with worse prompts.
The DSP routing weight $\alpha_{1}$ is increased from 0.469 to 0.614 when we change the point prompt from three points to one point.
It indicates that lower-quality prompts rely more on DSP features to shift attention to the desirable regions.
Table [5](#S5.T5) shows that DRP can further improve model’s performance, especially when handling the challenging one-point prompt scenario.

Robust Training Strategy.
Robust training is critical for improving model’s segmentation stability, but is usually overlooked in previous works.
RTS can guide the model, including our DSP, to accurately segment target objects even when provided with misleading low-quality prompts.
Table [6](#S5.T6) shows that RTS substantially improves the segmentation stability of all the methods,
albeit with a slight compromise in performance when dealing with high-quality prompts.
Note that our Stable-SAM benefits the most
from the application of RTS, which can be attributed to our carefully designed deformable sampling plugin design.

**Table 5: Ablation study on deformable sampling plugin (DSP), dynamic routing plugin (DRP) and robust training strategy (RTS).**
|  | Noisy Box | 1 Point |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | mIoU | mBIoU | mSF | mIoU | mBIoU | mSF |
| SAM (baseine) | 48.8 | 42.1 | 39.5 | 43.3 | 37.4 | 45.1 |
| + DSP | 69.9 | 60.2 | 67.2 | 46.8 | 40.8 | 48.0 |
| + DSP + RTS | 81.7 | 73.5 | 81.6 | 75.9 | 67.5 | 70.6 |
| + DSP + DRP + RTS | 82.3 | 74.1 | 82.3 | 76.9 | 68.4 | 71.1 |

Model Scalability.
Our method solely calibrates SAM’s mask attention by adjusting model’s feature sampling locations and amplitudes using a minimal number of learnable parameters (0.08 M), while keeping the model architecture and parameters intact.
This plugin design grants our method with excellent model scalability.
Table [6](#S5.T6) shows that our model can be rapidly optimized by just one training epoch, achieving comparable performance and stability.
By scaling the training procedure to 12 epochs, our method achieves the best performance across all prompting settings.
Additionally, our method can cooperate with other SAM variants.
For instance, when combined with HQ-SAM, the performance and stability are further improved.

Low-Shot Generalization.
Customized datasets with mask annotation are often limited, typically consisting of only hundreds of images.
For a fair comparison, all methods in Table [7](#S5.T7) are trained with RTS by 12 training epochs.
Table [7](#S5.T7) shows that HQ-SAM performs worst when trained with a limited number of images (220 or 440 images), which can be attributed to its potential overfitting problem caused by the relatively large learnable model parameters (5.1 M).
In contrast, PT-SAM’s better performance with minimal learnable parameters (0.13 M) further validates this hypothesis.
Our plugin design, coupled with minimal learnable parameters, enables effective low-shot generalization, and thus achieves the best performance in such scenario.

**Table 6: Model scalability study.**
|  | Groundtruth Box | Noisy Box |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | mIoU | mBIoU | mIoU | mBIoU | mSF |
| SAM (baseine) | 79.5 | 71.1 | 48.8 | 42.1 | 39.5 |
| Without RTS: |  |  |  |  |  |
| PT-SAM | 87.6 | 79.7 | 70.6 | 60.4 | 64.0 |
| HQ-SAM | 89.1 | 81.8 | 72.4 | 62.8 | 65.5 |
| Ours (1 epoch) | 87.4 | 80.0 | 69.6 | 60.0 | 66.5 |
| Ours (12 epochs) | 89.1 | 82.1 | 72.7 | 63.2 | 67.4 |
| With RTS: |  |  |  |  |  |
| PT-SAM | 86.8 | 78.4 | 82.1 | 73.1 | 78.7 |
| HQ-SAM | 87.4 | 79.8 | 82.9 | 74.5 | 80.4 |
| Ours (1 epoch) | 86.0 | 78.4 | 82.3 | 74.1 | 82.3 |
| Ours (12 epochs) | 87.4 | 80.1 | 84.4 | 76.7 | 85.2 |
| HQ-SAM + Ours | 88.7 | 81.5 | 86.1 | 78.7 | 86.3 |

**Table 7: Low-shot generalization comparison. All models are trained with RTS by 12 training epochs.**
|  | Noisy Box | 1 Point |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | mIoU | mBIoU | mSF | mIoU | mBIoU | mSF |
| SAM (baseine) | 48.8 | 42.1 | 39.5 | 43.3 | 37.4 | 45.1 |
| 220 train images: |  |  |  |  |  |  |
| PT-SAM | 77.6 | 67.7 | 72.6 | 71.8 | 63.2 | 73.0 |
| HQ-SAM | 73.5 | 62.3 | 67.7 | 71.3 | 62.6 | 72.4 |
| Ours | 78.8 | 70.0 | 78.9 | 73.0 | 64.7 | 74.5 |
| 440 train images: |  |  |  |  |  |  |
| PT-SAM | 78.6 | 69.0 | 74.4 | 76.2 | 67.4 | 75.0 |
| HQ-SAM | 77.4 | 67.1 | 75.6 | 74.6 | 64.6 | 71.9 |
| Ours | 81.6 | 73.5 | 82.6 | 79.8 | 71.5 | 82.5 |

## 6 Conclusion

In this paper, we present the first comprehensive analysis on SAM’s segmentation stability across a wide range of prompt qualities.
Our findings reveal that SAM’s mask decoder tends to activate image features that are biased to the background or specific object parts.
We propose the novel Stable-SAM to address this issue by calibrating solely SAM’s mask attention, *i.e*., adjusting the sampling locations and amplitudes of image feature using learnable deformable offsets, while keeping the original SAM model unchanged.
The deformable sampling plugin (DSP) allows SAM to adaptively shift attention to the prompted target regions in a data-driven manner.
The dynamic routing plugin (DRP) toggles SAM between deformable and regular grid sampling modes depending on the quality of the input prompts.
Our robust training strategy (RTS) facilitates Stable-SAM to effectively adapt to prompts of varying qualities.
Extensive experiments on multiple datasets validate the effectiveness and advantages of our Stable-SAM.

## Appendix

**Table 8: Comparison on Multi-domain Evaluation of Semantic Segmentation (MESS) benchmark, consisting of 22 downstream datasets, 448 classes, and 25,079 images.**
| Model | Prompt | General | Earth<br>Monitoring | Medical<br>Sciences | Engineering | Agriculture<br>& Biology | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SAM | Oracle<br>Point | 46.51 | 42.31 | 55.92 | 51.57 | 57.43 | 49.67 |
| HQ-SAM | 44.05 | 40.82 | 61.0 | 54.18 | 53.92 | 49.58 |  |
| Stable-SAM (Ours) | 49.10 | 43.23 | 54.45 | 53.52 | 60.92 | 51.15 |  |
| SAM | Oracle<br>Box | 77.85 | 73.03 | 64.89 | 73.03 | 86.75 | 74.74 |
| HQ-SAM | 76.63 | 68.03 | 65.1 | 74.78 | 85.34 | 73.43 |  |
| Stable-SAM (Ours) | 77.91 | 72.84 | 62.93 | 73.12 | 85.63 | 74.21 |  |
| SAM | Random<br>Point | 40.21 | 36.74 | 53.20 | 47.15 | 49.75 | 44.34 |
| HQ-SAM | 38.81 | 36.62 | 58.27 | 51.43 | 44.51 | 44.92 |  |
| Stable-SAM (Ours) | 42.43 | 37.30 | 50.89 | 48.96 | 53.42 | 45.49 |  |
| SAM | Noisy<br>Box | 55.97 | 55.12 | 62.45 | 62.34 | 64.26 | 59.24 |
| HQ-SAM | 52.02 | 47.15 | 62.57 | 62.02 | 60.56 | 55.81 |  |
| Stable-SAM (Ours) | 59.82 | 58.86 | 62.19 | 64.56 | 68.84 | 62.13 |  |

**Table 9: Dataset and comparison details on MESS benchmark. Models are prompted with noisy boxes. “HQ” denotes HQ-SAM.**
| Dataset | Domain | Sensor type | Mask size | # Classes | # Images | Task | SAM | HQ | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BDD100K [] | General | Visible spectrum | Medium | 19 (Medium) | 1,000 | Driving | 48.9 | 43.66 | 51.84 |
| Dark Zurich [] | Visible spectrum | Medium | 20 (Medium) | 50 | Driving | 54.34 | 50.85 | 56.81 |  |
| MHP v1 [] | Visible spectrum | Small | 19 (Medium) | 980 | Body parts | 66.64 | 62.63 | 69.11 |  |
| FoodSeg103 [] | Visible spectrum | Medium | 104 (Many) | 2,135 | Ingredients | 57.98 | 56.22 | 64.12 |  |
| ATLANTIS [] | Visible spectrum | Small | 56 (Many) | 1,295 | Maritime | 56.23 | 48.7 | 60.04 |  |
| DRAM [] | Visible spectrum | Medium | 12 (Medium) | 718 | Paintings | 51.73 | 50.06 | 57.03 |  |
| iSAID [] | Earth<br>Monitoring | Visible spectrum | Small | 16 (Medium) | 4,055 | Objects | 65.2 | 62.6 | 67.46 |
| ISPRS Potsdam [] | Multispectral | Small | 6 (Few) | 504 | Land use | 47.32 | 38.43 | 50.92 |  |
| WorldFloods [] | Multispectral | Medium | 3 (Binary) | 160 | Floods | 57.61 | 49.46 | 60.81 |  |
| FloodNet [] | Visible spectrum | Medium | 10 (Few) | 5,571 | Floods | 51.85 | 39.82 | 58.31 |  |
| UAVid [] | Visible spectrum | Small | 8 (Few) | 840 | Objects | 53.62 | 45.43 | 56.82 |  |
| Kvasir-Inst. [] | Medical<br>Sciences | Visible spectrum | Medium | 2 (Binary) | 118 | Endoscopy | 83.62 | 81.32 | 85.87 |
| CHASE DB1 [] | Microscopic | Small | 2 (Binary) | 20 | Retina scan | 34.59 | 37.5 | 33.88 |  |
| CryoNuSeg [] | Microscopic | Small | 2 (Binary) | 30 | WSI | 72.87 | 72.78 | 73.09 |  |
| PAXRay-4 [] | Electromagnetic | Large | 4x2 (Binary) | 180 | X-Ray | 58.72 | 58.66 | 55.94 |  |
| Corrosion CS [] | Engineering | Visible spectrum | Medium | 4 (Few) | 44 | Corrosion | 53.05 | 51.86 | 55.47 |
| DeepCrack [] | Visible spectrum | Small | 2 (Binary) | 237 | Cracks | 56.36 | 60.84 | 58.2 |  |
| ZeroWaste-f [] | Visible spectrum | Medium | 5 (Few) | 929 | Conveyor | 70.23 | 69.52 | 74.01 |  |
| PST900 [] | Electromagnetic | Small | 5 (Few) | 288 | Thermal | 69.71 | 65.88 | 70.55 |  |
| SUIM [] | Agriculture<br>& Biology | Visible spectrum | Medium | 8 (Few) | 110 | Underwater | 56.98 | 47.51 | 62.35 |
| CUB-200 [] | Visible spectrum | Medium | 201 (Many) | 5,794 | Bird species | 56.3 | 54.54 | 64.57 |  |
| CWFID [] | Visible spectrum | Small | 3 (Few) | 21 | Crops | 79.49 | 79.63 | 79.61 |  |
| Mean IoU | – | – | – | – | – | – | 59.24 | 55.81 | 62.13 |

## 7 More Experimental Results

### 7.1 MESS

The recently released Multi-domain Evaluation of Semantic Segmentation (MESS) [] is a large-scale benchmark for holistic analysis of zero-shot segmentation performance.
MESS consists of 22 downstream tasks, a total of 448 classes, and 25079 images, covering a wide range of domain-specific datasets in the fields of earth monitoring, medical sciences, engineering, agriculture and biology and other general domains.
We evaluate SAM [], HQ-SAM [] and our Stable-SAM on MESS benchmark using the official MESS evaluation code, and report the mean of class-wise intersection over union (mIoU).

Following MESS’s model settings, our Stable-SAM selects the first mask of the predicted multiple masks as the output.
For a fair comparison, our Stable-SAM follows HQ-SAM to fuse the SAM’s original prediction map into our predicted segmentation map.
We provide four prompt types for evaluation.
The oracle point refers to a single point sampled from the ground-truth mask using the point sampling approach RITM [].
The random point refers to a single point randomly sampled from the ground-truth mask of the target object.
The oracle box refers to a single box tightly enclosing the ground-truth mask of the target object.
The noisy box refers to a single box generated by adding noise (noise scale 0.4) to the oracle box.

Table [8](#Sx1.T8) tabulates the zero-shot semantic segmentation performance comparison on MESS.
Our Stable-SAM performs best when prompted with oracle point, random point and noisy box, and achieves comparable performance when provided with oracle box.
Our competitive performance on the large-scale MESS benchmark further consolidates the powerful zero-shot generalization ability inherent in our Stable-SAM.
Table [9](#Sx1.T9) shows the dataset and comparison details on 22 tasks of MESS benchmark. Our Stable-SAM performs best on 19 out of 22 datasets.

**Table 10: Comparison on MS COCO and four HQ datasets for different backbone variants.**
|  |  | MS COCO | Four HQ Datasets |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | N-Box (0.5-0.6) | Noisy Box | 1 Point | Params (M) | FPS | Mem. |  |  |  |  |  |  |
| Model | Epoch | mAP | mAP${}_{\mathrm{50}}$ | mIoU | mBIoU | mSF | mIoU | mBIoU | mSF | Total | Trainable |  |  |
| SAM-Huge | - | 25.6 | 56.8 | 50.1 | 43.2 | 40.4 | 44.5 | 38.3 | 46.5 | 2446 | 2446 | 3.5 | 10.3G |
| HQ-SAM-Huge | 12 | 30.0 | 62.6 | 75.2 | 65.5 | 69.3 | 48.0 | 41.1 | 49.5 | 2452.1 | 6.1 | 3.4 | 10.3G |
| Stable-SAM-Huge | 1 | 43.9 | 75.7 | 81.8 | 73.5 | 82.3 | 77.2 | 68.9 | 74.6 | 2446.08 | 0.08 | 3.5 | 10.3G |
| SAM-Large | - | 27.3 | 60.2 | 48.8 | 42.1 | 39.5 | 43.3 | 37.4 | 45.1 | 1191 | 1191 | 5.0 | 7.6G |
| HQ-SAM-Large | 12 | 31.9 | 65.5 | 72.4 | 62.8 | 65.5 | 43.2 | 44.6 | 37.4 | 1196.1 | 5.1 | 4.8 | 7.6G |
| Stable-SAM-Large | 1 | 44.8 | 76.4 | 82.3 | 74.1 | 82.3 | 76.9 | 68.4 | 71.1 | 1191.08 | 0.08 | 5.0 | 7.6G |
| SAM-Base | - | 19.7 | 49.2 | 41.6 | 35.8 | 33.4 | 35.1 | 29.2 | 36.7 | 358 | 358 | 10.1 | 5.1G |
| HQ-SAM-Base | 12 | 24.7 | 56.1 | 68.7 | 59.1 | 63.2 | 40.6 | 35.7 | 42.7 | 362.1 | 4.1 | 9.8 | 5.1G |
| Stable-SAM-Base | 1 | 31.2 | 63.3 | 74.7 | 64.8 | 75.9 | 68.9 | 59.5 | 67.1 | 358.08 | 0.08 | 10.1 | 5.1G |

**Table 11: Comparison on MS COCO and four HQ datasets for different Stable-SAM variants. The “finetuning decoder” denotes finetuning the mask decoder when training Stable-SAM.**
|  | MS COCO | Four HQ Datasets |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | N-Box (0.5-0.6) | Noisy Box | 1 Point |  |  |  |  |  |
| Model | mAP | mAP${}_{\mathrm{50}}$ | mIoU | mBIoU | mSF | mIoU | mBIoU | mSF |
| SAM | 27.3 | 60.2 | 48.8 | 42.1 | 39.5 | 43.3 | 37.4 | 45.1 |
| Stable-SAM (finetuning decoder) | 25.7 | 56.5 | 78.5 | 69.2 | 79.9 | 76.0 | 67.1 | 78.2 |
| Stable-SAM (spatial attention) | 29.8 | 64.9 | 69.3 | 59.8 | 57.8 | 51.6 | 44.5 | 49.1 |
| Stable-SAM | 44.8 | 76.4 | 82.3 | 74.1 | 82.3 | 76.9 | 68.4 | 71.1 |

Figure: Figure 5: Method difference between our deformable sampling plugin and conventional deformable attention.
Refer to caption: x4.png

### 7.2 Backbone Variants

Table [10](#S7.T10) tabulates the performance comparison on different backbone variants.
Our Stable-SAM consistently performs better than other methods on all backbone variants.

## 8 Relation to Other Methods

Deformable Attention.
Our method is unique in its idea and design on solely adjusting the feature sampling locations and amplitudes by training the offset network, without involving the original model parameters. In contrast, conventional deformable attention methods [, ] train both the offset network and original network parameters, which is undesirable when adapting powerful foundation models in deployment, especially in finetuning large foundation models.
Figure [5](#S7.F5) shows the difference between our deformable sampling plugin and conventional deformable attention.

We apply the conventional deformable attention in our Stable-SAM by finetuning the mask decoder during training.
Table [11](#S7.T11) shows that the conventional deformable attention (Stable-SAM (finetuning decoder)) exhibits the worst generalization ability on MS COCO, even worse than the original SAM model.
This further validates the necessity and better performance of our deformable sampling plugin paradigm, *i.e*., adapting the foundation model by only adjusting the feature sampling locations and amplitudes, while fixing the original model features and parameters.

Spatial Attention.
The spatial attention [, ] can adjust the image spatial feature weights, and thus can be regarded as a soft feature sampling method.
We directly replace DSP with spatial attention in our Stable-SAM to investigate if spatial attention offers comparable effectiveness.
Table [11](#S7.T11) shows that spatial attention performs much worse than our DSP, although it consistently improves the segmentation performance and stability on all datasets.
This indicates that simply adjusting the feature weights is insufficient to adapt SAM for handling suboptimal prompts.

## 9 Implementation Details

During training, we only train DSP and DRP on HQSeg-44K dataset while fixing the model parameters of the pre-trained SAM model.
We train Stable-SAM on 8 NVIDIA Tesla V100 GPUs with a total batch size of 32, using Adam optimizer with zero weight decay and 0.001 learning rate.
The training images are augmented using large-scale jittering [].
The input prompts are randomly sampled from mixed prompt types, including ground truth bounding boxes, randomly sampled points (1, 3, 5, 10 positive points randomly selected from the ground truth mask), noisy boxes (generated by adding noise (noise scale 0.4) to the ground truth bounding boxes, where we ensure the generated noisy boxes have at least 0.5 overlap IoU with the ground truth boxes), and coarse masks (generated by adding Gaussian noise in the boundary regions of the ground truth masks).
The model is optimized using cross entropy loss and dice loss [].

We follow the same inference pipeline of the original SAM.
The mask decoder first predicts a small mask in $256\times 256$ spatial resolution for each prompt, which is then up-sampled to the original resolution $1024\times 1024$ as the output mask.

## 10 Stability Visualization

Figure 6-16 show extensive visualization comparisons between SAM and Stable-SAM, under box, 3-points and 1-point prompts of diverse qualities.
We also visualize the image activation map for the token-to-image cross-attention in SAM’s second mask decoder layer to better understand its response to low-quality prompts.
The important features are highlighted by the orange circles, with larger radius indicating higher attention score.
SAM yields unsatisfactory segmentation results when provided with low-quality prompts, and even a minor prompt modification leads to unstable segmentation output.
In contrast, our Stable-SAM produces consistent and accurate mask predictions even under prompts of diverse qualities, by shifting more feature sampling attention to the target object.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: One open question: the module is efficient in improving the stability, considering both the training and inference budget. However, will scaling up the training data with more noisy data also improve the segmentation stability? How to treat the two factors?
soundness: 3
strengths: 1. The topic this paper focuses on is interesting and important, that may be ignored in previous research. Though SAMs produce surprising segmentation results, they may fail when the input prompts are not that accurate, especially in real-life applications. 
2. The proposed method is simple but effective. The promotion over previous SAM and high-quality SAM methods is notable on several benchmarks and tasks.
3. The writing is easy to follow.
summary: This paper analyzes the stability issue of segment anything models (SAMs). It is observed that SAMs tend to focus on background or some undesired parts when given low-quality prompts. A metric of segmentation stability is introduced to better evaluate this capability. Two main modules are proposed to improve the stability of SAMs, i.e. deformable sampling to organize the image feature and dynamic routing to allocate the magnitude of deformed activation.
weaknesses: 1. One potential negative effect to discuss. Though this paper targets strengthening the ability of SAM to focus on the foreground object, will it tend to produce wrong segmentation results when the desired segmenting part is the background? It will be interesting to discuss the bias this method may introduce.
2. It is easy to demonstrate the improvement when the number of prompts reduces to 1 or 3 points. However, noisy boxes are usually determined by the user, which are more subjective. Given a user has priors to roughly locate the desired object, the noise may not be that big. Therefore, a user study is recommended to better demonstrate how much benefit the method can actually bring when applied to real scenarios. The users can be required to give prompts to the same set of testing data, which can be more realistic than generated noisy prompts. 
3. Though the proposed modules are light and parameters-friendly, it is also necessary to evaluate their real latency and compare them with other methods.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. While the authors have specified the number of trainable parameters, could they also provide the inference time of Stable-SAM compared to other baselines? I believe this is another important dimension to examine the efficiency.

2. (Minor) There is a typo: a period is missing at the end of Line 259.
soundness: 3
strengths: 1. The writing is clear, making the paper easy to follow.
2. The motivation to develop stable prompting for SAM is well-articulated and addresses a less-rexplored but valuable area. The empirical studies in Section 3 are solid and clearly presented.
3. The proposed DSP and DRP modules are novel, effectively enhancing SAM's stability without compromising its powerful pre-trained representations.The inclusion of a dynamic routing mechanism is practical and well-justified.
4. Stable-SAM demonstrates clear performance improvements over SAM and its listed variants.
summary: This paper introduces Stable-SAM, a variant of the Segment Anything Model (SAM), aimed at addressing SAM's instability when given low-quality prompts—either biased towards the background or confined to specific object parts. The authors propose two key modules integrated into SAM's mask decoder transformer: a Deformable Sampling Plugin (DSP) and a Deformable Routing Plugin (DRP). DSP adjusts the feature sampling offsets and amplitude, while DRP resamples deformable image features at updated sampling locations, refining SAM's token-to-image attention. Additionally, a dynamic routing mechanism is introduced to selectively activate DSP, preventing unwanted attention shifts. A robust training strategy is also proposed to help the model correct SAM's attention when negatively impacted by poor-quality prompts. Low-quality prompts are generated on fine-grained segmentation datasets, including DIS, ThinObject-5K, COIFT and HR-SOD, and the approach demonstrates clear performance improvements, outperforming SAM, SAM fine-tuned with LoRA, SAM fine-tuned with an adapter, and HQ-SAM.
weaknesses: 1. The range of baseline models is limited. While many SAM-based models have recently been proposed to enhance its performance across various domains, this paper does not include comparisons with these relevant models.
2. The implementation of LoRA and adapters in SAM is not detailed. Since implementation choices can significantly affect performance, providing this information is crucial for evaluating the performance comparison as well as transparency and reproducibility.
3. The paper does not review SAM 2, which has been released and open-sourced. It would be nice to assess whether Stable-SAM is compatible with SAM 2 and if similar performance improvements can be achieved.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Please clarify the results for SAM in the SBD tests presented in Table 6.
2. How does Stable-SAM perform compared to other models with sufficient points?
3. Could you provide an analysis of the computational efficiency of Stable-SAM?
soundness: 3
strengths: 1. This work provides new insights of SAM’s stability in segmentation and introduces a segmentation stability metric that effectively quantifies the stability.
2. The proposed DSP and DRP modules provide novel insights into the improvement of SAM.
3. Stable-SAM shows marked improvement with insufficient prompts.
4. Generally, the paper is easy to follow.
summary: The paper presents Stable-SAM for enhancing SAM's segmentation stability when handling low-quality prompts. Stable-SAM employs DSP to adjust the sampling positions and amplitudes of image features without altering SAM's original architecture, which refines the mask attention. Additionally, the proposed DRP allows for switching between conventional and deformable sampling modes based on prompt quality, further improving the model's performance across varying prompt qualities. Experiments validate the method's effectiveness on multiple datasets.
weaknesses: 1. In Table 6, the interactive segmentation results for SAM on the SBD dataset appear inconsistent with those reported in other studies, showing higher performance. For example, in the works [1-4], the NoC90 metric for SAM-B/L/H exceeds 7.6, while Table 6 reports an SBD NoC90 of only 5.76. This discrepancy raises concerns given that this work builds upon SAM.

2. The core focus of this work is to address the issues arising from ambiguous prompts, yet SAM's practical application relies more on sufficiently detailed prompts, where SAM's inherent robustness may suffice. Moreover, Table 2 shows that increasing the number of points significantly reduces the advantage of Stable-SAM. Therefore, further experiments with more points are necessary to validate the effectiveness of Stable-SAM.

3. From another perspective, under precise prompts, the DSP may introduce shifts, as there remains a difference between DSP and human intent. If the intent is to segment small parts of the object, DSP may still guide the model to segment the main parts.

4. The paper lacks an analysis of computational efficiency. The DRP seems to run the SAM decoder twice (once with the new module proposed in the paper), leading to double the computational load of the SAM decoder. Thus, the efficiency analysis is necessary to confirm that the core performance benefits come from the new modules and not just increased computational demands. 

Reference
1. ClickAttention: Click Region Similarity Guided Interactive Segmentation
2. FocSAM: Delving Deeply into Focused Objects in Segmenting Anything
3. MST: Adaptive Multi-Scale Tokens Guided Interactive Segmentation
4. PiClick: Picking the desired mask from multiple candidates in click-based interactive segmentation

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses.
soundness: 3
strengths: 1. The paper is well-written and well-presented.
2. The problem addressed in this paper is novel and focuses on the segmentation capabilities of SAM under low-quality prompts.
3. The results of this paper are good and the experiments are sufficient.
summary: This paper proposes Stable SAM, which aims to improve the segmentation ability of SAM under low-quality prompts. It focuses on the segmentation stability of SAM under different quality prompts and proposes a stability metric called ST. To improve the stability, Stable SAM proposes deformable sampling plugin (DSP) and dynamic routing plugin (DRP), which enable SAM to adaptively shift attention to the prompted target regions and switch SAM between the deformable and regular grid sampling modes.
weaknesses: 1. Why the adjustment of the attention sampling position can achieve calibration of noisy prompts through learnable offsets? This lacks intuitive motivation and theoretical justification. The visualization comparison in the paper does proves that this method can produce good generalization effects, but the transition here is a bit abrupt. I hope the author can explain the motivation behind this idea in more detail.

2. Line 245 of the paper mentions "Note that conventional deformable attention optimizes both the offset network and attention module. Thus directly applying deformable attention to SAM is usually suboptimal, because altering SAM’s original network or weights", but intuitively, retraining all components may be the optimal result. Freezing the parameters trained in a predetermined way and re-adding some parameters for adaptation seems to be a compromise solution and result. However, this is still a conjecture and there is no necessary evidence to prove this statement. Perhaps SAM needs to be pre-trained again.

3. The purpose of DRP is to control the DSP’s activation to prevent unwanted attention shifts. Are there any experiments that prove that the output of DRP is consistent with the expected intuition under different quality prompt inputs? In addition, in the case of precise prompt input, whether increasing the output strength of DRP leads to worse results, and vice versa. I think sufficient experiments are needed here to verify the intuitive conjecture of DRP.

4. RTS makes the model produce better performance, I think it is necessary to compare RTS+SAM (w/ and w/o deformable attention).

5. According to Table 7, I cannot understand why SAM+DSP can achieve a significant performance improvement (even in the case of a single click) with only a small amount of fine-tuning using high-quality prompts. I think this phenomenon is not in line with common sense and requires sufficient evidence for demonstration and analysis.

6. The strategy proposed in the "Robust Training Against Ambiguous Prompts" section will cause the model to tend to segment larger objects when faced with ambiguity, and cannot handle the possible multiple segmentation results expected by the user, which is not conducive to solving the ambiguity problem of interactive segmentation.

Meta-review: The paper proposes a method to enhance the performance of SAM when provided with low-quality prompts through two novel modules: the Deformable Sampling Plugin (DSP) and the Dynamic Routing Plugin (DRP). Overall, the idea is novel and the results are solid. While the original submission lacked some details and experimental results, these shortcomings have been effectively addressed in the authors' feedback. Finally, all four reviewers provided positive scores. Based on the novelty of the approach, the solid experimental results, and the thoroughness of the authors' responses, the decision is to recommend this paper for acceptance.

Decision: Accept (Poster)