Stable Segment Anything Model
Authors: Qi Fan
Xin Tao
Lei Ke
Mingqiao Ye
Di ZHANG
Pengfei Wan
Yu-Wing Tai
Chi-Keung Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Segment Anything Model
Interactive Segmentation
Segmentation Stability
Deformable Feature Sampling
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Our Stable-SAM analyzes and improves SAM’s segmentation stability across a wide range of prompt qualities.

Abstract: The Segment Anything Model (SAM) achieves remarkable promptable segmentation given high-quality prompts which, however, often require good skills to specify. To make SAM robust to casual prompts, this paper presents the first comprehensive analysis on SAM’s segmentation stability across a diverse spectrum of prompt qualities, notably imprecise bounding boxes and insufficient points. Our key finding reveals that given such low-quality prompts, SAM’s mask decoder tends to activate image features that are biased towards the background or confined to specific object parts. To mitigate this issue, our key idea consists of calibrating solely SAM’s mask attention by adjusting the sampling locations and amplitudes of image features, while the original SAM model architecture and weights remain unchanged. Consequently, our deformable sampling plugin (DSP) enables SAM to adaptively shift attention to the prompted target regions in a data-driven manner. During inference, dynamic routing plugin (DRP) is proposed that toggles SAM between the deformable and regular grid sampling modes, conditioned on the input prompt quality. Thus, our solution, termed Stable-SAM, offers several advantages: 1) improved SAM’s segmentation stability across a wide range of prompt qualities, while 2) retaining SAM’s powerful promptable segmentation efficiency and generality, with 3) minimal learnable parameters (0.08 M) and fast adaptation. Extensive experiments validate the effectiveness and advantages of our approach, underscoring Stable-SAM as a more robust solution for segmenting anything. Codes are at https://github.com/fanq15/Stable-SAM.

Published as a conference paper at ICLR 2025
STABLE SEGMENT ANYTHING MODEL
Fan', Xin T Lei Ke", Mingqiao Ye", Pengfei Wan", Di Zhang",
Yu- Ku-Wing Tai Chi-Keung Tang
1 Nanjing University. Kuaishou Technology, 3 CCamegie Mellon University, EPFL. Dartmouth College, 6 The Hong Kong University of Science and Technology
ABSTRACT
The Segment Anything Model (SAM) achieves remarkable promptable segmenta- tion given high-quality prompts which, however, often require good skills to specify. To make SAM robust to casual prompts, this paper presents the first comprehensive analysis on SAM's segmentation stability across a diverse spectrum of prompt qualities, specifically imprecise bounding boxes and insufficient points. Our key finding reveals that given such low-quality prompts, SAM's mask decoder tends to activate image features that are biased towards the background, or confined to specific object parts. To mitigate these issues, our solution consists of calibrating solety SAM's mask attention by adjusting the sampling locations and amplitudes of image features, while the original SAM model architecture and weights remain unchanged. Consequently, our deformable sampling plugin (DSP) enables SAM to adaptively shift attention to the prompted target regions in data-driven man- ner. During inference, dynamic routing plugin (DRP) is proposed that toggles SAM between the deformable and regular grid sampling modes, conditioned on the input prompt quality. Thus, our solution, termed Stable-SAM, offers several advantages: 1) improved SAM's segmentation stability across wide range of prompt qualities, while 2) retaining SAM's s powerful promptable segmentation efficiency and generality, with 3) minimal learnable parameters (0.08 M) and fast adaptation. Extensive experiments validate the effectiveness and advantages of our approach, underscoring Stable-SAM as more robust solution for segmenting anything. Codes are at https: /ttpps://////ee//eeeeeeeeeeeeest thub. ccm fanql/ 5 Stable e SAMM.
INTRODUCTION
The recent Segment Anything Model (SAM (Kirillov et al., 2023)) stands a significant milestone in image segmentation, attributed its superior zero-shot generalization ability new tasks and data distributions. Empowered by the billion-scale training masks and the promptable model design, SAM generalizes to various visual structures in diverse scenarios through flexible prompts, such as box, point, mask or text prompts. Facilitated by high-quality prompts, SAM has produced significant performance benefit for various important applications. such as healtheare (Huang et al., 2023b; Mazurowski et al., 2023), remote sensing (Wen et al., 2023; Ding et al., 2023), self-driving (Dikshit al., 2023; Fan et al., 2022b), agriculture (Nguyen al., 2023; Liu, 2023), etc.
Previous works mainly focus on improving SAM's segmentation performance assuming high-quality prompts are available, such as a tight bounding box (e.g., produced by SOTA detectors (Jia et al., 2023; Zhang et al., 2023a; Yang et al., 2022)) or sufficient points (Ke et al., 2023) (e.g.. 10 points) for the target object. However, in practice SAM and interactive segmentation are often given inaccurate or insufficient prompts casually marked up by users as inaccurate box, or very sparse points are given, especially in the crowd-sourcing annotation platform. Such inaccurate prompts often mislead SAM to produce unstable segmentation results shown in Figure 1. Unfortunately, however, this critical issue has been largely overlooked, even though the suboptimal prompts and the resulting segmentation stability problem are quite prevalent in practice.
Note that there is no proper off-the-shelf solution for solving SAM's segmentation stability problem with inaccurate prompts. Simply finetuning SAM's mask decoder with imprecise prompts may easily
Corresponding Author: Xin Tao

---

Published as a conference paper at ICLR 2025
miou of fferent Methods
H0-S0M
Our
21.5
$4.
78.7
72.4
46,0
<1.3 43.2
Noley Box
Point Prompt
Prompt
(8)
(b)
(c)
Figure 1: To illustrate SAM's instability, the left figure compares the performance among SAM, HQ- SAM and our Stable-SAM, when provided with suboptimal prompts. Our Stable-SAM consistently surpasses other methods across prompts of different quality, demonstrating better or comparable performance to the SAM prompted by ground truth box. The right figure displays the predicted masks and sampled important image features of SAM and Stable-SAM prompted by the bounding box (in green color), with larger orange circles indicating higher attention weights. (a) SAM yields satisfactory segmentation results when provided with a high-quality box prompt. (b) SAM can be very unstable, as shown here even minor prompt modification makes SAM segment the background instead. SAM incorrectly segments the background, where the inaccurate box prompt misleads SAM spend more attention to the background. (c) Our Stable-SAM accurately segments the target object by shifting more feature sampling attention to it.
lead to catastrophic forgetting, undermining the integrity of the highly-optimized SAM model and thus sacrificing the zero-shot segmentation generality. Although in the image domain deformable attention (Dai et al., 2017) has shown impressive efficacy on adaptively shifting the model attention to informative regions, which may naturally address the attention drift issue caused by the misleading prompts, a straightforward implementation of this idea can again compromise SAM's 's integrity.
In this paper present the first comprehensive analysis on SAM's segmentation stability across wide range of prompt qualities, with a particular focus on low-quality prompts such as imprecise bounding boxes or points. Our findings demonstrate that, when fed with imprecise prompts, the SAM's mask decoder is likely to be misguided to focus on the background or specific object parts, where the cross-attention module is inclined to aggregate and activate image features of these regions when mutually updating the prompt and image tokens. Such collaborative token updating mechanism usually suffers from attention drift, where the suboptimal prompts misleadingly shift attention from the target object to the background areas or specific object parts. The attention drift is accumulated and propagated from the suboptimal prompt to the unsatisfactory segmentation results.
To address this issue, we present novel deformable sampling plugin (DSP) with two key designs to improve SAM's stability while maintaining its zero-shot generality. Our key idea to adaptively calibrate SAM's mask attention by adjusting the attention sampling positions and amplitudes, while keeping the original SAM model unchanged: 1) we employ a small offset network to predict the corresponding offsets and feature amplitudes for each image feature sampling locations, which are learned from the input image feature map: 2) then, we adjust the feature attention by resampling the deformable image features at the updated sampling locations of the cross- -tts-nattnntion module in SAM's mask decoder, keeping the original SAM model unchanged. In doing so, we can shift the feature sampling attention toward informative regions which is more likely contain target objects, and meanwhile avoiding the potential model disruption of the original highly-optimized SAM. Finally, to effectively handle both the high- and low-quality prompts, we propose a dynamic routing module to toggle SAM between deformable and regular grid sampling modes. A simple and effective robust training strategy is proposed to facilitate our Stable-SAM to adapt to prompts of diverse qualities.
Thus, our method is unique in its idea and design on solely adjusting the feature attention without involving the original model parameters. In contrast, conventional deformable attention methods (Dai et al., 2017; Xia et al., 2022) update the original network parameters, which is undesirable when adapting powerful foundation models involving finetuning such large models data-scarce scenarios.
Our model, Stable-SAM., benefits both the selective deformable attention and the powerful original SAM model, with minimal addition of computational overhead and parameters. First, the SAM's segmentation stability is substantially improved across wide range of prompt qualities, especially with low-quality prompts. Besides, the original SAM's powerful promptable segmentation efficiency and generality are preserved well even in the data-scarce scenarios. Extensive experiments across multiple datasets validate the effectiveness and advantages of our approach, underscoring its potential as robust solution for segmentation tasks.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
Segment Anything Model. The recent Segment Anything Model (Kirillov et al., 2023; Ravi et al., 2024) has gained widespread recognition, attributed to its remarkable performance and generalization in image segmentation. SAM has been applied in wide range of downstream tasks and applications, including medical images (Ma et al., 2024; Zhang & Liu, 2023; Liu et al., 2024; Leng et al., 2024), object tracking (Zou et al., 2024), data annotation (He et al., 2023; Wang et al., 2024), 3D reconstruction (Cen et al., 2024; Yin et al., 2023), robotics (Huang et al., 2023a), and multimodal tasks (Mo & Tian, 2023; Zhang et al., 2023b; Wang ct al., 2023). Some researchers attempt to address SAM's computational limitations and improve its efficiency. Some works (Zhang et al., 2023e; Liu et al., 2023) focus improving SAM's segmentation quality and generalization to downstream applications. Thus the foundation model finetuning methods (Chen et al., 2023; Wu et al., 2023) are widely adopted for fast and effective SAM adaptation in specific segmentation scenarios.
Improving Segmentation Quality. Researchers have proposed various methods to enhance the quality and accuracy of semantic segmentation methods. Early methods incorporate graphical models such as CRF (Krähenbühl & Koltun, 2011) or region growing (Dias & Medeiros, 2019) as an additional post-processing stage, which are usually training-free design new operators (Ke et al., 2022a;b; Kirillov et al., 2020) or utilize additional refinement stage (Cheng et al., 2020; Shen al., 2022). Recently, methods such as Mask2Former (Cheng et al., 2022) and SAM (Kirillov et al., 2023) have been introduced, which address open-world segmentation by introducing prompt-based approaches. Along this line, a series of improvements (Ke et al., 2023; Li et al., 2023) have been proposed, focusing prompt-tuning and improving the accuracy of segmentation decoders. However, these methods overlook a crucial aspect, which is how to generate high-quality segmentation results in cases where the prompt is inaccurate. This is precisely the problem that our method aims to address.
Many learning-based methods
Pretrained models have played important role since the very beginning of deep learning (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016). Despite zero-shot generalization grows popular in foundation models of computer vision and natural language processing (Bommasani et al., 2021; Brown et al., 2020), tuning methods such as adapter (Hu al., 2022) and prompt-based learning (Houlsby al., 2019; Hu al., 2022) have been proposed to generalize these models to downstream tasks (Fan et al., 2020; 2022a). These methods typically involves additional training parameters and time. We propose a new method that makes better use of existing features with minimal additional methods and can also produce competitive results.
Tuning Foundation Models.
Deformable Attention. Deformable convolution (Dai et al., 2017; Zhu et al., 2019) has been proved effective to help neural features attend to important spatial locations. Recently, it has also been extended to transformer-based networks (Chen et al., 2021; Yue et al., 2021; Zhu et al., 2020; Xia et al,, 2022). Such deformed spatial tokens are especially suitable for our task, which requires dynamically attending to correct regions given inaccurate prompts. However, previous deformable layers involve both offset learning and feature learning after deformation. In this paper, propose new approach to adjust the feature attention by simply sampling and modulating the features using deformable operations, without the need to train subsequent layers.
3 SAM STABILITY ANLAYSIS
We perform empirical studies to illustrate the segmentation instability of the current SAM with prompts of differing quality, thereby justifying our Stable SAM approach.
Prior segmentation studies have focused on achieving high prediction accuracy, gauged by the Detrreection-vvr-Uniiiii (IoU) between the predicted and ground truth masks. This focus on high performance is justified as segmentation models typically produce deterministic masks for given input images, without requiring additional inputs. However, SAM's 's segmentation output depends on both the image and the prompts, with the latter often varying in quality due to different manual or automatic prompt generators. In practical applications of SAM, segmentation targets are typically clear and unambiguous, independent of prompt quality.
Segmentation Stability Metric. Motivated by this application requirement, introduce the segmentation stability metric. Specifically, SAM is capable of producing a set of binary segmentation maps M Bx for single target object using B prompts of differing qualities. We define the
3

---

Published as a conference paper at ICLR 2025
Figure 2: SAM performs badly when dealing with suboptimal prompts. This is mainly caused by the undesirable feature attention, focusing on the background or specific object parts. The important features are highlighted by the orange circles, with larger radius indicating higher attention score. The green boxes denote the box prompts with added noise to the groundtruth boxes. The stars denote the point prompts which are randomly sampled from the groundtruth masks.
segmentation stability (ST) within the set as:
1 B
ST = Σ
MM, M Munion);
B (=1
where (M, Munion) represents the Intersection-over-UUU
between the i-th segmentation map M; and the collective foreground region -, M, of all maps. This new metric assesses the consistency across segmentations in each prediction, serving as a reliable indicator of stability, even without access to the ground truth masks.
Model and Evaluation Details. The released SAM is trained with crafted prompts on large-scale SA-1B dataset. We evaluate the segmentation accuracy and stability of the ViT-Large based SAM with different prompt types and qualities, including box prompts with added noise (we insert the uniform noise with the noise scale 0.4 into the box height, width and center position) and point prompts with varying numbers of points (1, 3, 5, 10 positive points randomly selected from the ground truth mask). For every input image and prompt type, we randomly select 20 prompts to compute their segmentation stability, average mask mloU, and boundary mBIoU scores. The evaluation utilizes four segmentation datasets in HQ HQ-SAM: DIS (Qin et al., 2022) (validation set), ThinObject-5k (Liew et al., 2021) (test set), COIFT (Mansilla & Miranda, 2019), and HR-SOD (Zeng et al., 2019).
Table tabulates that SAM's segmentation accuracy and stability significantly decrease with low-quality prompts, such as imprecise box prompts or point prompts with minimal points. The varying segmen- tation accuracy and stability indicates that SAM's mask decoder performs distinctly when dealing with prompts of varying qualities.
Table 1: SAM's segmentation accuracy and stability under prompts of varying quality. All evaluation metrics are averaged on four HQ datasets.
Metric
Naisy Point
mloU
79.5
48.8
43.3
78.7
83.3
84.8
mBtol
76.0
ST
39.5
45.1
79.3
84.7
87.5
We visualize the image activation map for the token- to-image cross-attention in SAM's second mask decoder layer to better understand its response to low-quality prompts. We focus the second mask decoder layer for visualization because its cross-attention is more representative, benefiting from the input tokens and image embedding collaboratively updated by the first mask decoder layer. Figure 2 demonstrates that inaccurate box prompt causes SAM's mask decoder to miss regions of the target object while incorrectly incorporating features from the background, or focusing specific object parts. It consequently leads to degraded segmentation accuracy and stability.
Overall, the above empirical evidence suggests that SAM potentially suffers from the attention drift issue, where suboptimal prompts misleadingly shift attention from the target object to background areas or specific object parts, thereby compromising the segmentation accuracy and stability. This motivates us to calibrate SAM's mask attention by leveraging learnable offsets to adjust the attention sampling position towards the target object regions, thus boosting segmentation accuracy and stability.
STABLE SEGMENT ANYTHING MODEL
We first revisit the recent Segment Anything Model (SAM) and deformable attention mechanism. Segment Anything Model. SAM (Kirillov et al., 2023) is powerful promptable segmentation model. It comprises image encoder for computing image embeddings, prompt encoder for embedding
4

---

Published as a conference paper at ICLR 2025
E 32l1x022
Imagio Foat
image token artn.
DWCConv EXG GELU
offset net
Decoder Layer
featurr
dp
Sia sammpling
Cony 1xt
E RAWMW
image
Prompp Faat
DRP
(c)
xp
Deformed Feat
Output
selt
Decoder
Trainable Module
(a)
Frazen Module
f, e eeti
Figure 3: (a) An illustration of our deformable sampling plugin (DSP) and deformable routing plugin (DRP) in SAM's mask decoder transformer. DSP employs a small (b) offset network to predict the feature sampling offsets and amplitudes. Subsequently, DSP calibrates the feature attention by resampling deformable image features at the updated sampling locations, and feeds them into SAM's token-to-image attention. DRP employs small (c) MIP network regulate the degree of DSP activation based on the input prompt quality. Note that our DSP adaptively calibrates solely SAM's mask attention without altering the original SAM model.
prompts, and a lightweight mask decoder for predicting segmentation masks by combining the two information sources. The fast mask mask decoder is two-layer transformer-based decoder to collaboratively update both the image embedding and prompt tokens via cross-attention. SAM is trained on the large-scale SA-IB dataset.
Deformable Attention. Deformable attention (Xia et al., 2022) is a mechanism that enables the model focus subset of key sampling points instead of the entire feature space. This mechanism naturally addresses the attention shift problem in SAM caused by low-quality prompts.
In the standard self-attention, given feature map x E RxCWxCC, the attention weights are computed across all spatial locations within the feature map.
In the deformable attention (Xia al., 2022), uniform grid of points RHax are first generated as the references' with the sampled image feature E R RRox XXxC Subsequently, convolutional offset network foffset predicts the offset Ar (x for each reference point. The new feature sampling locations are given by Ar E RG Wa The resampled deformable image features Ar+ E Ha HaaxWaxx are then utilized the features in the attention module.
Note that conventional deformable attention optimizes both the offset network and attention module. Thus directly applying deformable attention to SAM is usually suboptimal, because altering SAM's original network or weights, e,g., substituting SAM's standard attention with deformable attention and retraining, may compromise its integrity.
4.1 DEFORMABLE SAMPLING PLUGIN
To address the attention drift issue while preserving SAM's integrity, propose novel deformable sampling plugin (DSP) module on top of SAM's original token-to-image cross-attention module, as shown in Figure 3.
Specifically, given the prompt token feature E TxC and image feature Ip E RXXXC the token-to-image cross-attention is:
(Atn(t=))=oQQU) (Q(1)) K(x,)²) V(ap),
where p R Hxxx x2 represents the image feature spatial sampling locations, o denotes the softmax function, and Q, K, V are the query, key, and value embedding projection functions, respectively.
Our DSP adaptively calibrate the feature attention by adjusting solely feature sampling locations and amplitudes without altering the original SAM model. Specifically, we utilize an offset network foflset predict the feature sampling offset Ap E Rx akin that in deformable attention:
Ap (0 (xx)),
With the grid size downsampled from the input feature map spatial size (H, W) by factor of thus He H/s and Wa W/s.
5

---

Published as a conference paper at ICLR 2025
where 9, is scale function 8p -tan(*) to prevent too large offset, and Sp is pre-defined scale factor. The offset network foffstt consists of a 1 x 1 convolution, a 5 5 depthwise convolution with the layer normalization and GELU activation, and a 1 X convolution. The updated feature sampling locations are p + Ap. The numerical range of both p and p + Ap is clamped {(0, ().,))... (H 1, W 1)}, which is then normalized predicted by the first convolutional layer and the image features Tp are thus updated as zp. which are used solely for computing the feature attention.
the range [-1,1] 1] for feature sampling. 'hee feature amplitudes
Subsequently, we resample and modulate deformable image features Tz+Ap E R#xxxC at the updated sampling locations p + Ap with the learned feature amplitudes for keys and values. Thus, our DSP calibrates the token-to-inen
cross-attention of SAM's mask decoder as:
DCAun(t, x² = (Q(t)) K ((T++p))) (p))
As p + Ap is fractional, we apply bilinear interpolation to compute pp-Ap as in Deformable DETR (Zhu et al., 2020).
Note that our DSP only trains the deformable offset network to predict new feature sampling locations p + Ap and feature amplitudes, and feeds the resampled and modulated deformable features pp+Ap
to SAM's cross-attention module. Thus, the original SAM model remains unchanged.
4.2 DYNAMIC ROUTING PLUGIN
While our DSP can effectively handle suboptimal and even erroneous prompts, by redirecting SAM's attention to informative regions which are more likely to contain the target objects, high-quality prompts can typically direct the model's attention correctly to target regions. Thus, it is essential to properly control the DSP's activation to prevent unwanted attention shifts.
To address this issue, we propose novel dynamic routing plugin (DRP) that regulates the degree of DSP activation based on the input prompt quality. The DRP can be formulated as follows:
a = o (MLP(tt)) ss,
(5)
where to E R is the prompt token feature corresponding to the output mask, MLP refers to small MLP network that includes an MLP layer with LayerNorm and GELU activation, as well as an output MLP layer; denotes learnable scale and denotes the softmax function.
We utilize the predicted values of a = [ai, E R to adaptively route SAM between DSP and original SAM's attention mechanism. Consequently, the token-to-image cross-attention output Of,, x) can be formulated as:
Ott, x) CAttn aj ETTAP + 02 xp))
(6)
This soft dynamic routing strategy allows SAM to benefit from both DSP and its original zero-shot generality, contingent upon the quality of the prompt.
4.3 ROBUST TRAINING STRATEGY
We propose a simple and effective robust training strategy (RTS) assist our model learn how to correct SAM's attention when adversely affected by bad prompts.
Robust Training Against Inaccurate Prompts. SAM's training, including HQ-SAM (Ke et al., 2023), typically utilizes high-quality prompts given by precise bounding boxes or multiple points to accurately identify the target object. To address inaccurate prompts, our RTS incorporates prompts of varying qualities during training. These prompts include groundtruth boxes, box prompts with added noise (noise scale 0.4), and point prompts with varying numbers of points (1, 3, 10 positive points randomly chosen from the ground truth mask).
Robust Training Against Ambiguous Prompts. real segmentation scenarios, target objects often occur in cluttered environment, either occluding others or being occluded. Even given an accurate, tight bounding box, objects other than the target object will be enclosed. On the other hand, target objects typically unambiguous even other objects COCO, beds (oceluded by quilt) are consistently regarded as target objects; the model must accurately segment the entire bed including accessories such pillows and bedding. Thus, SAM's original
enclosed. For instance, in MS

---

Published as a conference paper at ICLR 2025
Table 2: Method comparison on four HQ datasets under prompts of varying quality. All models
(except for methods in the first group) are trained on HQSeg-444 44K dataset.
Noisy Box
Point mBloU
3 Points mBIoU
Model
Epoch mloU
mBloU
ST
mloU
ST
mloU
ST
SAM (baseline)
48.8
42.1
39.5
43,3
37.4
45.1
78.7
69.5
79.3
PA-SAM (Xie et al., 2024)
51.2
44.4
41.8
45.3
39.5
47.2
79.6
70.1
80,0
CAT-SAM (Xiao
2024)
51.5
44.8
42.1
45.7
39.9
47.6
80.0
70.6
80.5
RobustSAM (Chen et al., 2024)
51.7
44.9
42.3
45.9
40.2
47.7
80.4
71.1
81.0
SAM 2 (Ravi ct 2024)
52.4
45.3
43.1
46.7
41.1
48.5
81.1
71.8
81.7
FT-SAM (finctuning SAM's whole model)
12
32.5
27.7
24.1
28.6
22.8
30,3
46.2
35.4
43.1
DT-SAM (finetuning SAM's mask decoder)
12
70.6
60.4
64.0
43.1
43.2
37.9
80.3
71.6
80.5
PT-SAM (linetuning SAM's prompi loken)
70.8
64.1
43.0
42.9
38.3
80.1
71.8
SAM with LoRA (Hu et al., 2022)
12
70.3
60.6
63.7
42.3
43.2
37.2
79.5
71.2
79.6
SAM with Adapter (Chen al., 2022)
12
70.5
63.2
42.7
43.3
37.5
79.8
71.4
80.0
HQ-SAM (Kc al., 2023)
12
72.4
62.8
65.5
43.2
44.6
37.4
81.8
73.7
81.4
Stable-SAM Stable-SAM
82.3
74.1
82.3
76.9
68 4
71.1
84.0
75.8
84.9
83.5
75.3
83.4
78.0
69.6
72.2
85.1
76.9
86.0
ambiguity aware solution, which predicts multiple masks for single prompt, is generally suboptimal in well-defined realistic applications. To address such "ambiguous" prompts, our RTS incorporates synthetic occlusion images make SAM conducive accurately segment target objects. We include the implementation details of the occlusion image synthesis in the supplementary materials.
Our RTS is general and applicable to various SAM variants improve their segmentation stability. Notably, our Stable-SAM with DSP and DRP experience the most substantial improvements from the application of RTS.
EXPERIMENTS
Datasets. For fair comparison keep our training and testing datasets as HQ-SAM (Ke et al., 2023). Specifically, we train all models on HQSeg-44K dataset, and evaluate their performance on four fine-grained segmentation datasets, including DIS (Qin et al., 2022) (validation set), ThinObject- 5K (Liew et al., 2021) (test set), COIFT (Mansilla & Miranda, 2019) and HR-SOD (Zeng et al., 2019). Furthermore, we validate the model's zero-shot generalization ability on three challenging segmentation benchmarks, including COCO (Lin et al., 2014), SGinW (Zou et al., 2023) and MESS (Blumenstiel et al., 2023). SGinW contains 25 zero-shot in-the- wild segmentation datasets. MESS is a large-scale benchmark for holistically evaluating the zero-shot segmentation performance.
We evaluate model's accuracy and stability with prompts of differing type and quality, as described in Sec. 3. For MS COCÓ and SGinW, we do not use the boxes generated by SOTA detectors (Zhang et al., 2023a; Jia et al., 2023) as the box prompt. This is because their predicted boxes are typically of high quality and cannot effectively evaluate the model's segmentation stability in the presence of inaccurate boxes. Instead, we introduce random scale noises into the ground truth boxes to generate noisy boxes as the prompts. Specifically, to simulate inaccurate boxes while still having some overlap with the target object, we select noisy boxes that partially overlap with the ground truth boxes with IoU ranges of 0.5-0.6 and 0.6-0.7. We also evaluate method using the box prompts generated by SOTA detectors.
Input Prompts.
5.1 COMPARISON WITH SAM VARIANTS
We compare our method with SAM and three powerful SAM variants. HQ-SAM a recent powerful SAM variant for producing high-quality masks. We also try two popular model finetuning methods, LoRA (Hu et al., 2022) and Adapter (Chen al., 2022), and three simple SAM variants by finetuning the SAM's whole model. its mask decoder and the prompt token, i.e.. FT-SAM, DT-SAM and PT-SAM, respectively. All our Stable-SAM models are trained by just one epoch for fast adaptation unless otherwise stated. All other models are trained 12 epochs. More experimental results and implementation details are included in the supplementary material.
Stability Comparison Four HQ Datasets. Table 2 shows the segmentation accuracy and stability on four HQ datasets, when models are fed with suboptimal prompts. Notably, the use of noisy box prompts significantly reduces SAM's performance, evidenced by the drop from 79.5/71.1 (as
7

---

Published as a conference paper at ICLR 2025
Table Comparison on MS COCO and SGinW datasets. All models (except for the SAM baseline) are trained on HQSeg-44K dataset. All models are prompted by noisy boxes (N-Box) that overlap with the ground truth boxes, with IoU ranges of 0.5-0.6 and 0.6-0.7.
MS COCO
SGinW
N-Box
(0.5-0.6)
N-Box (0.6-0.7)
N-Box (0.5-0.6)
N-Box (0.6-0.7)
Learnable Params
Model
Epoch
mAP
mAPs0
mAP
mAP
mAP
mAP50
mAP
mAP50
Mem.
FPS
SAM (baseline)
27.3
60.2
40.9
75.0
26.0
60.8
39.5
73.2
[(119] M)
7.6G
DT-SAM
12.2
22.7
15.8
28.7
10.4
21.5
13.6
27.1
3.9
76G
PT-SAM
12
30.2
63.4
41.3
76.5
32.1
66.4
41.1
74.3
0.13
7.6
5.0
HQ-SAM
12
31.9
65.5
42.9
77.1
33.6
68.4
42.2
75.9
5,1 M
7.6
4.8
Stable-SMM
1
44.8
76.4
50.5
81.1
43.3
75.6
48.6
79.4
0.08 M
7.6
5.0
shown in Table 1) to 48.8/42 18.8/42.1 mloU/mBIoU,, accompanied by low stability score of 39.5 ST. This is probably because SAM was trained with solely high-quality prompts, thus seriously suffers from the low-quality prompts during inference. Finetuning SAM's 's whole model greatly impairs performance, because it destroys the integrity of the highly-optimized SAM model and thus sacrificing the zero-shot segmentation generality. The other five SAM variants, namely HQ-SAM, DT-SAM, and PT-SAM, SAM with LoRA and SAM with Adapter, demonstrate relatively better stability dealing with noisy boxes. which can be attributed to their long-term training on the HQSeg-44K dataset. Note our Stable-SAM can effectively address inaccurate box prompts, by enabling models to shift attention to target objects. Given a single-point prompt, both SAM and its variants exhibit the lowest accuracy and stability. This indicates they are adversely affected by the ambiguity problem arising from the use of single-point prompt. Although, in most practical applications, users prefer minimal interaction with clear and consistent segmentation targets. Our method maintains much better performance and stability when handling ambiguous one-point prompt, owing robust training strategy against ambiguity. When point prompts increase to 3, all methods perform
deformable feature sampling and much better, while other methods still under-perform compared with ours.
Segment Anything Model (SAM 2) (Ravi et al., 2024) is a unified model for video and image- based promptable segmentation. SAM 2 outperforms SAM, due to its stronger backbone and larger pretraining dataset. However, SAM 2 still suffers significantly from low-quality prompts, owing to the overlooked segmentation stability problem. Our method can be seamlessly integrated into SAM to enhance segmentation stability and performance under prompts of varying quality. Stable- 2 exhibits substantial improvements in segmentation quality and stability, outperforming the original Stable-SAM model.
Generalization Comparison on MS COCO and SGinW. Table 3 presents the segmentation accuracy and stability when the models are generalized to MS COCO and SGinW with noisy boxes. Note that the DT-SAM performs the worst, probably due to overfitting on the training set, which compromises its ability to generalize to new datasets. Our method consistently surpasses all competitors, particularly handling inaccurate boxes (N-Box 0.5-0.6), where all noisy boxes have an IoU range of 0.5-0.6 with the ground truth boxes. Note that our method has minimal number of extra learnable parameters (0.08M) and can be quickly adapted to new datasets by just one training epoch. We also evaluate the learnable parameters, training memory and inference speed of our method. The results demonstrate that our approach is lightweight and efficient, with the negligible addition of 0.08 M parameters having no impact on the efficiency of the original SAM model.
Comparison Based on Detector Predicted Box Prompts. Existing zero-shot segmentation methods typically choose powerful object detection models to generate high-quality boxes as the input prompts, such as FocalNet-L-DINN
Table 4: Comparison on MS COCO with the box prompts generated by SOTA detectors (FocalNet-L-DINO and R50-H-D-DETR, or noisy box prompts that overlap with the ground truth boxes, with IoU ranges of 0.5- 0.6. All models (except for SAM) are trained on HQSeg-44K dataset.
(Zhang et al., 2023a; Yang al, 2022). We also evaluate our method in such set- ting. Table 4 presents that our model achieves compa- rable performance as SAM and PT-SAM when using the FocalNet-L-DINO generated high-quality boxes as prompts. When using the 550-H-Deffommble DETR (Zhu al., 2020) the box prompt generator, our method achieves comparable performance as HQ- SAM. Note that training and implementing SOTA
RocalNer-LL-DiN
RSD-IL--DDI
Noisy Box mAP
Model
mAP
mAP
mAP
mAP
mAP
SAM
41.5
63.7
27.3
60.2
PT-SAM
48.6
75.5
41.7
64.2
30.2
63.4
HQ-SAM
49.5
75.7
42.4
64.5
31.9
65.5
Ours
48.3
74.8
42.2
64.0
448
76.4
8

---

Published as a conference paper at ICLR 2025
Figure 4: Visual results for box prompts (Ist row), for point prompts (2nd row). Within each image group in the first three rows, the three figures represent the results of SAM with GT box prompt, SAM with noisy box prompt, and Stable-SAM with noisy box prompt, respectively. The last two rows display the results of SAM and Stable-SAM with point prompts.
detectors typically require large computational resources and the cross-domain generalization is still very challenging. In practice, users tend to leverage interactive tools to annotate objects for their personalized datasets. Our method substantially surpasses other competitors in such scenario, when the box can roughly indicate the target object.
5.2 ANALYSIS ON STABLE-SAM
We perform detailed analysis on Stable-SAM on its network modules, model scalability, low-shot generalization, point prompt quality, backbone variants, relation to other methods, and stability visualization. More experimental analysis are included in the supplementary material.
Deformable Sampling Plugin. Table 5 shows DSP can be trained with high-quality prompts (without RTS) to improve the performance and stability on low-quality prompts, although the model still exhibits some instability. When equipped with RTS, DSP can effectively learn to shift SAM's attention to target ob- jects when subjecting inaccurate prompts. To delve deeper into the deformable sampling mechanism, we visualize the sampled feature points and their corre- sponding attention weights. Figure 4 illustrates how our DSP effectively shifts model's attention to the target object, resulting in increased attention weights. Consequently, the cross-altention module aggregates more target object features into the prompt tokens, thereby improving the segmentation quality of the target objects.
Table 5: Ablation study on deformable sam- pling plugin (DSP), dynamic routing plugin (DRP) and robust training strategy (RTS). All models (except SAM) are trained on HQSeg-44h 44K dataset.
Noisy Box mloli mBlol:
I Point mloU mBloUl ST
Model
SAM (baseline)
48.8
42.1
39.5
43,3
37.4 45.1
48.0
DSP + RIS
81.7
73.5
81.6
75.9
67.5
70.6
DSP DRP RTS
82.3
82.3
76.9
68.4
71.1
Dynamic Routing Plugin. We leverage DSP to dynamically route the model between the regular and deformable feature sampling modes, conditioned on the input prompt quality. We find that DRP tends to route more DSP features when dealing with worse prompts. The DSP routing weight aj is increased from 0.469 0.614 when change the point prompt from three points to point. indicates that lower-quality prompts rely more on DSP features to shift attention to the desirable regions. Table 5 shows that DRP can further improve model's performance, especially when handling the challenging one-point prompt scenario.
Robust Training Strategy. Robust training is critical for improving model's segmentation stability, but is usually overlooked in previous works. RTS can guide the model, including our DSP, to accurately segment target objects even when provided with misleading low-quality prompts. Table shows that RT'S substantially improves the segmentation stability of all the methods, albeit with a slight compromise performance when dealing with high-quality prompts. Note that from the application of RTS, which can be attributed to our carefully designed deformable sampling plugin design.

---

Published as a conference paper at ICLR 2025
Model Scalability. Our method solely calibrates SAM's mask attention by adjusting model's feature sampling locations and amplitudes using minimal number of learnable parameters (0.08 M), while keep- ing the model architecture and parameters intact. This plugin design grants our method with excellent model scalability, Table 6 shows that our model can be rapidly optimized by just one training epoch, achiev- ing comparable performance and stability. By scal- ing the training procedure to 12 epochs, our method achieves the best performance across all prompting settings. Additionally, our method can cooperate with other SAM variants. For instance, when combined with HQ-SAM, the performance and stability are fur- ther improved.
Table 6: Ablation study on Robust Training Strategy (RTS) and model scalability. All models in this table (except for SAM) are trained on HQSeg-44K dataset by 12 training epochs, unless stated otherwise, with or with- out Robust Training Strategy (RTS). All mod- els are evaluated on four HQ datasets with GT box prompt and noisy box prompt.
Groundmath Box
Noisy Box mBloU
Model
mloU
mBlol
mloU
ST
SAM (baselinc)
79.5
71.1
48.8
42.1
39.5
Without RTS:
PT.SAM
87.6
79.7
70.6
60.4
64.0
HQ-SAM
89.1
81.8
72.4
62.8
65.5
Ours (1 epoch)
87.4
80.0
69.6
60.0
66.5
Ours (12 opochs)
89.1
82.1
72.7
63,2
67.4
Low-Shot Generalization. Customized datasets with mask annotation are often limited, typically con- sisting of only hundreds of images. For fair com- parison, all methods in Table 7 are trained with RTS by training epoch. Table 7 shows that HQ-SAM performs worst when trained with limited number of images (220 or 440 images), which can be attributed its potential overfitting problem caused by the relatively large learnable model parameters (5.1 M). In contrast, PT-SAM's better performance with minimal learnable parameters (0.13 13 M) further validates this hypothesis. Our plugin design, coupled with minimal learnable parameters, enables effective low-shot generalization, and thus achieves the best performance in such scenario.
With RTS:
PT-SAM
86.8
78.4
82_1
73.1
78.7
IQQ.AM
87.4
79.8
82.9
74.5
80.4
Ours (1 (pech)
86.0
78.4
82.3
74.1
82.3
Ours (12 epochs)
87.4
80.1
84.4
76.7
85.2
HQ:SAM Ours
88.7
81.5
86.1
78.7
86.3
Discussion on Potential Segmentation Bias. We emphasize that our method is designed without inher- ent bias towards large or small objects. If the prompt explicitly specifies the background, our method adapts accordingly, without constraining the model to prioritize the foreground. Although our method avoids introducing bias, we acknowledge that the model may still be influenced by dataset bias. For in- stance, if the training dataset predominantly contains foreground objects, the model may skew predictions towards the foreground, potentially neglecting back- ground regions. Thus, we also highlight the flexi- bility of our method. If users wish to personalize segmentation targets, such as focusing on specific background regions, Stable-SAM can be easily fine- tuned to meet this requirement. This adaptability is key strength, enabling our approach to effectively mitigate dataset bias and address wide range of user needs and scenarios beyond typical foreground segmentation.
Table 7: Low-shot generalization comparison. All models are trained with RTS by I training epoch, with 220/440 train images. All models are evaluated on four HQ datasets with noisy box prompt and 1 point prompt.
Noisy Box
I Point
Model
mloU mBloll
ST mloU mBloll
ST
SAM (haseline)
48.8
42.1
39.5|
43.3
37.4
45.1
220 main images:
PT.SAM
77.6
67.7
72.6
71.8
63.2
73.0
HQ-SAM
73.5
62.3
67.7
71.3
62.6
72.4
Ours
78.1
68.8
78.6
73.0
64.7
74.5
train images:
PT-SAM
78.6
69.0
74.4
73.1
65.5
71.1
HQ-SAM
77.4
67.1
75.6
71.7
61.8
68.8
Ours
79.5
70.3
81.7
75.6
67.5
71.5
CONCLUSION
In this paper, we present the first comprehensive analysis on SAM's segmentation stability across a wide range prompt qualities. Our findings reveal that SAM's mask decoder tends to activate image features that are biased to the background or specific object parts. We propose the novel Stable-SAM to address this issue by calibrating solely SAM's 's mask attention, iee., adjusting the sampling locations and amplitudes of image feature using learnable deformable offsets, while keeping the original SAM model unchanged. The deformable sampling plugin (DSP) allows SAM to adaptively shift attention to the prompted target regions in data-driven manner. The dynamic routing plugin (DRP) toggles SAM between deformable and regular grid sampling modes depending on the quality of the imput prompts. Our robust training strategy (RTS) facilitates Stable-SAM effectively adapt to prompts of varying qualities. Extensive experiments on multiple datasets validate the effectiveness and advantages of our Stable-SAM.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research was supported in part by the Natural Science Foundation of Jiangsu Province (BK202412000 National Natural Science Foundation of China (62406140), Research Grant Council of the HKSAR (16201420), and Kuaishou Technology.
REFERENCES
Benedikt Blumenstiel, Johannes Jakubik, Hilde Kühne, and Michael Vössing. What MESS:
Multi-Domain Evaluation of Zero-shot Semantic Segmentation. In NeurIPS Workshop, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurIPS, 2020.
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. Segment anything in with nerfs. NeurIPS, 2024.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurlPS, 2022.
Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam fails to segment unything--sam-adape Adapting sam in underperformed scenes: Camouflage, shadow, and more. arXiv preprint urXiv:2304.09 04144, 2023.
Wei-Ting Chen, Yu-Jiet Vong, Sy-Yen Kuo, Sizhou Ma, and Jian Wang. Robustsam: anything robustly on degraded images. In CVPR, 2024.
Segment
Zhiyang Chen, Yousong Zhu, Chaoyang Zhao, Guosheng Hu, Wei Zeng, Jinqiao Wang, and Ming Tang. Dpt: Deformable patch- transformer for visual recognition. In ACM MM, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
Ho Kei Cheng. Jihoon Chung. Yu-Wing Tai, and Chi-Keung Tang. Cascadepsp: Toward alss-agnnostic and very high-resolution segmentation via global and local refinement. In CVPR, 2020.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Philipe Ambrozio Dias and Henry Medeiros. Semantic segmentation refinement by monte carlo region growing of high confidence detections. In ACCV, 2019.
Atharva Dikshit, Alison Bartsch, Abraham George, and Amir Barati Farimani. Robochop: Au- tonomous framework for fruit and vegetable chopping leveraging foundational models. arXiv preprint arXiv:2307. 1315,, 2023.
Lei Ding, Kun Zhu, Daifeng Peng, Hao Tang, and Haitao Guo. Adapting segment anything model for change detection in hr remote sensing images. arXiv preprint 2:X9022309.01449, 2023.
Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. In CVPR, 2020.
Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang. Self-support few-shot semantic segmentation. In ECCV, 2022a.
Qi Fan, Mattia Segu, Yu- Wing Tai, Fisher Yu, Chi-Keung Tang. Bernt Schiele, and Dengxin Dai. Towards robust object detection invariant real-world domain shifts. In ICLR, 2022b.

---

Published as a conference paper at ICLR 2025
Haibin He, Jing Zhang, Mengyang Xu, Juhua Liu, Bo Du, and Dacheng Tao. Scalable mask annotation for video text spotting. arXiv preprint arXiv:2305.01443 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305. 176, 2023a.
Yuhao Huang, Xin Yang. Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen. Chaoyu Chen, al. Segment anything model for medical images? arXiv preprint arXiv:2304. /4660, 2023b.
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu, Detrs with hybrid matching. In CVPR, 2023.
Lei Ke, Martin Daneljjan, Xia Li, Yu-Wing Tai, Chi- Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In CVPR, 2022a.
Lei Ke, Henghui Ding, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Video mask transfiner for high-quality video instance segmentation. In ECCV, 2022b.
Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality. In NewrIPS, 2023.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR, 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In ICCV, 2023.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. NeurIPS, 2011.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. NewrlPS, 2012.
Tianang Leng, Yiming Zhang, Kun Han, and Xiaohui Xie. Self-sampling meta sam: Enhancing few-shot medical image segmentation with meta-learning In WACV, 2024.
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint 230.007707. 2023.
Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection. In WACV, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Xuanyu Liu. A sam-based method for large-scale crop field boundary delineation. In SECON, 2023.
Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all all-pur feature matching, arXiv preprint 23X. 2023.
12

---

Published as a conference paper at ICLR 2025
Zhizhe Liu, Shuai Zheng, Xiaoyi Sun, Zhenfeng Zhu, Yawei Zhao, Xuebing Yang, and Yao Zhao. The devil is in the boundary: Boundary-enhanced polyp segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communicutions, 2024.
Lucy Alsina Choque Mansilla and Paulo André Vechiatto de Miranda. Object segmentation by oriented image foresting transform with connectivity constraints. In IC/P,,019)
Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis, 2023.
Shentong Mo and Yapeng Tian. Av-sam: Segment anything model meets audio-visua localization and segmentation. arXiv preprint 2/X1v 2023.
Khoa Dang Nguyen, Thanh-Hai Phung, and Hoang-Giang Cao. A sam-based solution for hierarchical panoptic segmentation of crops and weeds competition. In ICCVW, 2023.
Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Lue Van Gool. Highly accurate dichotomous image segmentation. In ECCV, 2022.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 007 2024.
Tiancheng Shen, Yuechen Zhang, Lu Qi, Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Di Wang, Jing Zhang, Bo Du, Minqiang Xu, Lin Liu, Dacheng Tao, and Liangpei Zhang. Samrs: Scaling-up remote sensing segmentation dataset with segment anything model. NeurIPS, 2024.
Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint arXiv:2305.02677 2023.
Congcong Wen, Yuan Hu, Xiang Li. Zhenghang Yuan, and Xiao Xiang Zhu. Vision-language models in remote sensing: Current progress and future trends. arXiv preprint arXiv:2305.05726, 2023.
Junde Wu, Rao Fu, Huihui Fang. Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint 2304 /2620, 2023.
Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In CVPR, 2022.
Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Ruijie Ren, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Cat-sam: Conditional tuning for few-shot adaptation of segment anything model. ECCV, 2024.
Zhaozhi Xie, Bochen Guan, Weihao Jiang. Muyang Yi. Yue Ding, Hongtao Lu, and Lei Zhang. Pa-sam: Prompt adapter sam for high-quality image segmentation. In ICME, 2024.
Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. In NeurIPS, 2022.
Youtan Yin, Zhoujie Fu, Fan Yang, and Guosheng Lin. Or-nerf:
Object removing from 3d scenes guided by multiview segmentation with neural radiance fields. arXiv preprint (tt.:2305..0003 2023.
13

---

Published as a conference paper at ICLR 2025
Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip HS Torr, Wayne Zhang, and Dahua Lin. Vision transformer with progressive sampling. In ICCV, 2021.
Yi Zeng, Pingping Zhang, Jianming Zhang, Zhe Lin, and Huchuan Lu. Towards high-resolution salient object detection. ICCV, 2019.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yenr Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2023a.
Jielu Zhang, Zhongliang Zhou, Gengchen Mai, Lan Mu, Mengxuan Hu, and Sheng Li. Text2seg: Remote sensing image semantic segmentation via text-guided visual foundation models. arXiv preprint arXiv:2304. 10597, 2023b.
Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13755 2023.
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hong- sheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.0304 03048, 2023c.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In CVPR, 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. CVPR, 2023.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang. Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. NewrIPS, 2024.
14


Reviews:
[{"review_id": "T7NKR2dGOt", "reviewer": "Reviewer_SsTF", "summary": "This paper analyzes the stability issue of segment anything models (SAMs). It is observed that SAMs tend to focus on background or some undesired parts when given low-quality prompts. A metric of segmentation stability is introduced to better evaluate this capability. Two main modules are proposed to improve the stability of SAMs, i.e. deformable sampling to organize the image feature and dynamic routing to allocate the magnitude of deformed activation.", "questions": "One open question: the module is efficient in improving the stability, considering both the training and inference budget. However, will scaling up the training data with more noisy data also improve the segmentation stability? How to treat the two factors?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The topic this paper focuses on is interesting and important, that may be ignored in previous research. Though SAMs produce surprising segmentation results, they may fail when the input prompts are not that accurate, especially in real-life applications. \n2. The proposed method is simple but effective. The promotion over previous SAM and high-quality SAM methods is notable on several benchmarks and tasks.\n3. The writing is easy to follow.", "weaknesses": "1. One potential negative effect to discuss. Though this paper targets strengthening the ability of SAM to focus on the foreground object, will it tend to produce wrong segmentation results when the desired segmenting part is the background? It will be interesting to discuss the bias this method may introduce.\n2. It is easy to demonstrate the improvement when the number of prompts reduces to 1 or 3 points. However, noisy boxes are usually determined by the user, which are more subjective. Given a user has priors to roughly locate the desired object, the noise may not be that big. Therefore, a user study is recommended to better demonstrate how much benefit the method can actually bring when applied to real scenarios. The users can be required to give prompts to the same set of testing data, which can be more realistic than generated noisy prompts. \n3. Though the proposed modules are light and parameters-friendly, it is also necessary to evaluate their real latency and compare them with other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BKvZOsV0If", "reviewer": "Reviewer_UZpL", "summary": "This paper introduces Stable-SAM, a variant of the Segment Anything Model (SAM), aimed at addressing SAM's instability when given low-quality prompts—either biased towards the background or confined to specific object parts. The authors propose two key modules integrated into SAM's mask decoder transformer: a Deformable Sampling Plugin (DSP) and a Deformable Routing Plugin (DRP). DSP adjusts the feature sampling offsets and amplitude, while DRP resamples deformable image features at updated sampling locations, refining SAM's token-to-image attention. Additionally, a dynamic routing mechanism is introduced to selectively activate DSP, preventing unwanted attention shifts. A robust training strategy is also proposed to help the model correct SAM's attention when negatively impacted by poor-quality prompts. Low-quality prompts are generated on fine-grained segmentation datasets, including DIS, ThinObject-5K, COIFT and HR-SOD, and the approach demonstrates clear performance improvements, outperforming SAM, SAM fine-tuned with LoRA, SAM fine-tuned with an adapter, and HQ-SAM.", "questions": "1. While the authors have specified the number of trainable parameters, could they also provide the inference time of Stable-SAM compared to other baselines? I believe this is another important dimension to examine the efficiency.\n\n2. (Minor) There is a typo: a period is missing at the end of Line 259.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear, making the paper easy to follow.\n2. The motivation to develop stable prompting for SAM is well-articulated and addresses a less-rexplored but valuable area. The empirical studies in Section 3 are solid and clearly presented.\n3. The proposed DSP and DRP modules are novel, effectively enhancing SAM's stability without compromising its powerful pre-trained representations.The inclusion of a dynamic routing mechanism is practical and well-justified.\n4. Stable-SAM demonstrates clear performance improvements over SAM and its listed variants.", "weaknesses": "1. The range of baseline models is limited. While many SAM-based models have recently been proposed to enhance its performance across various domains, this paper does not include comparisons with these relevant models.\n2. The implementation of LoRA and adapters in SAM is not detailed. Since implementation choices can significantly affect performance, providing this information is crucial for evaluating the performance comparison as well as transparency and reproducibility.\n3. The paper does not review SAM 2, which has been released and open-sourced. It would be nice to assess whether Stable-SAM is compatible with SAM 2 and if similar performance improvements can be achieved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kC4mNtaaOS", "reviewer": "Reviewer_Fktg", "summary": "The paper presents Stable-SAM for enhancing SAM's segmentation stability when handling low-quality prompts. Stable-SAM employs DSP to adjust the sampling positions and amplitudes of image features without altering SAM's original architecture, which refines the mask attention. Additionally, the proposed DRP allows for switching between conventional and deformable sampling modes based on prompt quality, further improving the model's performance across varying prompt qualities. Experiments validate the method's effectiveness on multiple datasets.", "questions": "1. Please clarify the results for SAM in the SBD tests presented in Table 6.\n2. How does Stable-SAM perform compared to other models with sufficient points?\n3. Could you provide an analysis of the computational efficiency of Stable-SAM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work provides new insights of SAM’s stability in segmentation and introduces a segmentation stability metric that effectively quantifies the stability.\n2. The proposed DSP and DRP modules provide novel insights into the improvement of SAM.\n3. Stable-SAM shows marked improvement with insufficient prompts.\n4. Generally, the paper is easy to follow.", "weaknesses": "1. In Table 6, the interactive segmentation results for SAM on the SBD dataset appear inconsistent with those reported in other studies, showing higher performance. For example, in the works [1-4], the NoC90 metric for SAM-B/L/H exceeds 7.6, while Table 6 reports an SBD NoC90 of only 5.76. This discrepancy raises concerns given that this work builds upon SAM.\n\n2. The core focus of this work is to address the issues arising from ambiguous prompts, yet SAM's practical application relies more on sufficiently detailed prompts, where SAM's inherent robustness may suffice. Moreover, Table 2 shows that increasing the number of points significantly reduces the advantage of Stable-SAM. Therefore, further experiments with more points are necessary to validate the effectiveness of Stable-SAM.\n\n3. From another perspective, under precise prompts, the DSP may introduce shifts, as there remains a difference between DSP and human intent. If the intent is to segment small parts of the object, DSP may still guide the model to segment the main parts.\n\n4. The paper lacks an analysis of computational efficiency. The DRP seems to run the SAM decoder twice (once with the new module proposed in the paper), leading to double the computational load of the SAM decoder. Thus, the efficiency analysis is necessary to confirm that the core performance benefits come from the new modules and not just increased computational demands. \n\nReference\n1. ClickAttention: Click Region Similarity Guided Interactive Segmentation\n2. FocSAM: Delving Deeply into Focused Objects in Segmenting Anything\n3. MST: Adaptive Multi-Scale Tokens Guided Interactive Segmentation\n4. PiClick: Picking the desired mask from multiple candidates in click-based interactive segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RzGtgVRGoW", "reviewer": "Reviewer_wxwP", "summary": "This paper proposes Stable SAM, which aims to improve the segmentation ability of SAM under low-quality prompts. It focuses on the segmentation stability of SAM under different quality prompts and proposes a stability metric called ST. To improve the stability, Stable SAM proposes deformable sampling plugin (DSP) and dynamic routing plugin (DRP), which enable SAM to adaptively shift attention to the prompted target regions and switch SAM between the deformable and regular grid sampling modes.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-presented.\n2. The problem addressed in this paper is novel and focuses on the segmentation capabilities of SAM under low-quality prompts.\n3. The results of this paper are good and the experiments are sufficient.", "weaknesses": "1. Why the adjustment of the attention sampling position can achieve calibration of noisy prompts through learnable offsets? This lacks intuitive motivation and theoretical justification. The visualization comparison in the paper does proves that this method can produce good generalization effects, but the transition here is a bit abrupt. I hope the author can explain the motivation behind this idea in more detail.\n\n2. Line 245 of the paper mentions \"Note that conventional deformable attention optimizes both the offset network and attention module. Thus directly applying deformable attention to SAM is usually suboptimal, because altering SAM’s original network or weights\", but intuitively, retraining all components may be the optimal result. Freezing the parameters trained in a predetermined way and re-adding some parameters for adaptation seems to be a compromise solution and result. However, this is still a conjecture and there is no necessary evidence to prove this statement. Perhaps SAM needs to be pre-trained again.\n\n3. The purpose of DRP is to control the DSP’s activation to prevent unwanted attention shifts. Are there any experiments that prove that the output of DRP is consistent with the expected intuition under different quality prompt inputs? In addition, in the case of precise prompt input, whether increasing the output strength of DRP leads to worse results, and vice versa. I think sufficient experiments are needed here to verify the intuitive conjecture of DRP.\n\n4. RTS makes the model produce better performance, I think it is necessary to compare RTS+SAM (w/ and w/o deformable attention).\n\n5. According to Table 7, I cannot understand why SAM+DSP can achieve a significant performance improvement (even in the case of a single click) with only a small amount of fine-tuning using high-quality prompts. I think this phenomenon is not in line with common sense and requires sufficient evidence for demonstration and analysis.\n\n6. The strategy proposed in the \"Robust Training Against Ambiguous Prompts\" section will cause the model to tend to segment larger objects when faced with ambiguity, and cannot handle the possible multiple segmentation results expected by the user, which is not conducive to solving the ambiguity problem of interactive segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)