SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement
Authors: Yuqi Lin
Hengjia Li
Wenqi Shao
Zheng Yang
Jun Zhao
Xiaofei He
Ping Luo
Kaipeng Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Mask Refinement
Segment Anything Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In this paper, we explore a principal way to enhance the quality of widely pre-existing coarse masks, enabling them to serve as reliable training data for segmentation models to reduce the annotation cost. In contrast to prior refinement techniques that are tailored to specific models or tasks in a close-world manner, we propose SAMRefiner, a universal and efficient approach by adapting SAM to the mask refinement task. The core technique of our model is the noise-tolerant prompting scheme. Specifically, we introduce a multi-prompt excavation strategy to mine diverse input prompts for SAM (\ie, distance-guided points, context-aware elastic bounding boxes, and Gaussian-style masks) from initial coarse masks. These prompts can collaborate with each other to mitigate the effect of defects in coarse masks. In particular, considering the difficulty of SAM to handle the multi-object case in semantic segmentation, we introduce a split-then-merge (STM) pipeline. Additionally, we extend our method to SAMRefiner++ by introducing an additional IoU adaption step to further boost the performance of the generic SAMRefiner on the target dataset. This step is self-boosted and requires no additional annotation. The proposed framework is versatile and can flexibly cooperate with existing segmentation methods. We evaluate our mask framework on a wide range of benchmarks under different settings, demonstrating better accuracy and efficiency. SAMRefiner holds significant potential to expedite the evolution of refinement tools. Our code is available at https://github.com/linyq2117/SAMRefiner.

Published as a conference paper at ICLR 2025
SAMREFINER: TAMING SEGMENT ANYTHING MODEL FOR UNIVERSAL MASK REFINEMENT
Yuqi Lin 1.2 Hengjia Li', Wenqi Shao?, Zheng Yang**, Jun Zhao
Xiaofei Hel. Ping Luo2.1 Kaipeng Zhang
1 Sttate Key Lab of CAD&CG, College of Computer Science, Zhejiang University Shanghai AI Laboratory
FABU Inc. The University of Hong Kong
1inyq55 6 60 gmail. com
kp pp_zhange foxmail com com
ABSTRACT
In this paper, we explore a principal way to enhance the quality of widely pre- existing coarse masks, enabling them to serve reliable training data for segmenta- tion models to reduce the annotation cost. contrast to prior refinement techniques that are tailored to specific models or tasks in close-world manner, we propose SAMRefiner, a universal and efficient approach by adapting SAM to the mask refinement task. The core technique of our model is the noise-tolerant prompting scheme. Specifically, we introduce diverse input prompts for SAM (i.e, distance-guided points, context-aware elastic bounding boxes, and Gaussian-stylee masks) from initial coarse masks. These prompts can collaborate with each other to mitigate the effect of defects in coarse masks. In particular, considering the difficulty of SAM to handle the multi-object case in semantic segmentation, we introduce a split-then-mengg Additionally, we extend our method to SAMRefiner++ by introducing an additional IoU adaption step to further boost the performance of the generic SAMRefinnr on the target dataset. This step is self-booster and requires no additional annota- tion. The proposed framework is versatile and can flexibly cooperate with existing segmentation methods. We evaluate our mask framework benchmarks under different settings, demonstrating better accuracy and efficiency. SAMRefiner holds significant potential expedite the evolution of refinement tools. Our code is available at SAMRefiner.
excavation strategy to mine
(STM) pipeline.
wide range of
INTRODUCTION
Image segmentation aims to assign a label to each pixel in an image such that pixels with the same label share certain characteristics. There are different notations about the group labels, such as semantic categories instances. In the past few years, although significant progress has been made in image segmentation, the prevailing approaches rely on fully annotated training images, which are tedious to obtain. To reduce human labor, a labor-efficient alternative is generating segmentation masks by preceding models, especially those designed under incomplete supervisions (e.g. unsupervised, weakly supervised or supersupe annotations Wang et al. (2023b; 2022); Lin et al. (2023)). These generated segmentation masks can serve as pseudo labels to train advanced segmentation models or iteratively upgrade existing models Zhu al. (2021); Yang et al. (2022). With the ever-increasing data amount, this pseudo-labeling paradigm showcases great practicality and potential to expand dataset volume for large-scale learning. However, the initial pseudo masks are usually noisy and lack fine details, particularly in object boundaries or in high- frguufrey regions (seeing Fig. la), hindering them from providing reliable supervision for model training.
Several mask refinement techniques have been proposed to improve the mask quality. but they suffer from major drawbacks: 1) model-dependent: Some methods develop custom refinement modules tailored specific networks and train them in end-to-end fashion Zhang et al. (2021); Ke et al. (2022a), making them fail to work on different models. 2) task-specific: Another group of techniques Chen et al. (2022); Cheng et al. (2020); Shen et al. (2022) resort to model-agnostic
Corresponding authors.

---

Published as a conference paper at ICLR 2025
PASCAL VOU
(W/955)
PASCAL (DespLab)22)
(USS)
COCO
(UBS)
DAVIS-185
Bodin:
Ct
Initial Mask
CRF
Ours
CM
Secfatrer
COCO (WSIS)
(Miak RCNN)
COCO
COCO (WSSIS)
(a) Visualizations of segmentation masks. Left: The
initial masks generated by Lin al. (2023). Mid: Masks refined by dense CRF Krähenbühl & Koltun (2011). Right: Masks refined by our framework.
(b) The performance of our proposed mask refinement framework SAMRefiner on different benchmarks and comparisons with related works.
Figure 1: Visualization of segmentation masks and performance.
refinement mechanisms but they usually focus on specific task (e.g, semantic segmentation or instance segmentation). 3) category-limited: Most previous works require training on target datasets with annotated data, limiting them to generalize to unseen categories and granularity. 4) time-inefficiien: Recent works Shen et al. (2022); Wang et al. (2023a) demonstrate better performance but refine one instance at a time, which is inefficient in complex instance segmentation tasks.
Recently, Segment Anything Model (SAM) Kirillov et al. (2023), interactive image segmentation model that segments intended objects by user-provided prompts (e.g, point, box), has been proposed and achieved significant success in many image segmentation tasks. Some researchers have en- deavored to adapt to various tasks in order to take advantage of SAM's powerful representation capability to alleviate inadequate training samples. However, most of these studies focus predicting masks from scratch and how to adapt SAM for the mask refinement task with pre-existing coarse masks remains an unexplored and challenging problem. We argue that this task is of great value in practical applications due to the widespread pre-existing masks (e.g, masks provided by offline models, inaccurate human annotations, or other forms of pre-processing Making modifications on them could facilitate the annotation and benefit various downstream tasks.
However, since SAM prompt-driven, adapting SAM to the refinement task not trivial because is difficult to obtain accurate prompts for SAM merely from coarse masks. Applying SAM directly mask refinement using naive strategies would suffer from distorted prompts caused by noise and result in inferior performance. For example, in Fig. 2, we adopt the commonly used box prompt (tight box of coarse mask) and observe that this naive approach fails to obtain satisfactory performance because diverse types of errors (e.g. false- negative, false- possttvve contained in the coarse mask would mislead the prompt extraction. Besides, results of directly taking the coarse mask as prompt are also terrible for SAM (the 4th column in Fig. 2) due its inherent nature in pre-training. (More prompt analyses are provided in the Method Section and Appendix.) Therefore, how to mine noise-tolerant prompts from the coarse mask poses a great challenge.
In this paper, we tame SAM for the mask refinement tasks, which have unique characteristics compared to other segmentation tasks for the existence of coarse masks. We propose a universal and efficient framework called SAMRefiner, the core technique of which is the noise-tolerant prompting scheme. Specifically. to mitigate the effect of defects in coarse masks to prompt generation, we propose a guided points, context-aware elastic bounding boxes (CEBox), and Gaussian-style masks. These multi-prompts can collaborate with each other to generate high-quality masks and are more robust to noise than the single prompt. To overcome the confusion caused by multi-object cases, we introduce a split-then-men -erge (STM) pipeline to make it better suited for semantic segmentation. Meanwhile, given that the original SAM lacks dataset-specific priors, resulting in inaccurate IoU branch predictions, we propose SAMRefiner++. This approach incorporates additional IoU adaptation step to enhance SAM's prediction accuracy on specific datasets by leveraging coarse mask priors. This minimal adaption startegy operates in self-boosted manner and requires extra annotations.
excavation strategy to mine diverse and seemly prompts, including distance-
2

---

Published as a conference paper at ICLR 2025
GT
Coarse Mask
Tight Box Prompt
Mask Prompt
Ours
Figure 2: Failure cases of SAM using the tight box of the coarse mask (red box) and directly using the coarse mask as the prompt. The tight box sensitive to the false negative (first row) and false positive (last row) errors in the coarse mask, which would mislead SAM's predictions. And the separate mask prompt fails to work for SAM. Our proposed multi-prompt excavation strategy is robust the noise.
We conduct experiments wide range of semantic and instance segmentation settings, with pseudo masks generated from incomplete supervisions, existing models, and synthetic data. Experimental results demonstrate the outstanding mask refinement capability of SAMRefiner (seeing Fig. lb). Our approach generic post-processing tool and can be incorporated into any image segmentation approach in self-training fashion with constant performance improvement.
Our contributions are summarized as follows:
New Roadmap:
SAMRefiner offers the first solution address the mask refinement task based on SAM, which of great value in practical applications.
New Method: We uncover the deficiency of SAM in the mask refinement task and propose effective and efficient framework to mine noise-tolerant prompts, successfully addressing the challenging universal mask refinement task.
Novel Insights: While work is based SAM, it offers several novel insights and observations like the impact of mask prompt and the loU adaption strategy.
Stronger practicality and performance: This framework is versatile and can flexibly co- operate with existing segmentation methods under various setting. It significantly enhances the pseudo mask quality (e.g, over 10% for WSSIS) while taking less time (e.g, 5 faster than CascadePSP).
2 RELATED WORKS
2.1 COARSE MASKS IN IMAGE SEGMENTATION
Coarse masks are common and ubiquitous in the image segmentation task due to their strict standard of pixel-accurate annotations. To relieve human burden, some works adopt the pseudo-labeling paradigm to obtain segmentation masks. These approaches usually leverage incomplete annotations (e.g. none, point, box, image-level labels partially fully-labeled data) obtain segmentation masks, which can be roughly categorized into unsupervised Cho et al. (2021): )); Ziegler & Asano (2022); Ke et al. (2022b); Hwang et al. (2019): Van Gansbeke al. (2021); Zhou al. (2022); Shin et al. (2022; 2023), weekky-superrviee Lin et al. 2) Dai et al. (2015); Papandreou et al. (2015); Ahn & Kwak (2018); Xie et al. (2022); Wang al. (2020b); Xu al. (2022b), and semi-supervised Wang et (2022); Filipiak et al. (2022): Yang et al. (2023b); Xu et al. (2022a). Although labor- effocieefi,intt, the quality of pseudo mask unsatisfactory, which can heavily impair the performance of subsequent segmentation model training. The noisy labels even exist in human annotation (e.g, MS COCO Lin et al. (2014)),
which is inevitable for achieving aixel-aceurate annotations at scale. This paper focuses on enhancing the quality of the coarse mask and consequently contributes to subsequent model training.
2.2 MASK REFINEMENT TECHNIQUE
To overcome the inaccuracy of coarse masks, several mask refinement methods have been ex- plored Zhang et al. (2021); Kirillov et al. (2020); Xu al. (2017); Zhang al. (2019); Yuan et al.

---

Published as a conference paper at ICLR 2025
SAMRethnr
Top-1
Predkteed
Prompt Excavation
Negatve
X
ML Pws
LoKA Actattor
Barilary Mio
(a) SAMRefiner
IoU Adaption
Figure 3: (a) An overview of our proposed framework. SAMRefiner leverages SAM to refine coarse masks by automatically generating prompts from coarse masks, including distance-guided points, context-aware elastic boxes and Gaussian-style masks. We select the best mask from multiple generated masks based on SAM's Is IoU predictions. (b) An overview of the introduced IoU adaption step, which aims to enhance the IoU prediction ability of SAM on specific datasets. We adopt LoRA-style adaptor at the last layer of IoU MLP and a ranking loss used to improve the top- accuracy of IoU predictions. This step is self- hiffbbooted and requires additional annotation.
(2020). Most existing works designed for specific networks or tasks and thus lack generality and flexibility. For example, PointRend Kirillov et al. (2020) and RefineMask Zhang et al. (2021) built upon Mask RCNN He et al. (2017) for instance segmentation, BPR Tang al. (2021) propose a model -agnos-agnottic post-processing mechanism but mainly focuses on instance segmentation. The dataset-dependan training in a close-world paradigm makes them overfit to specific datasets. CascadePSP Cheng et al. (2020) and CRM Shen et al. (2022) train on a large merged dataset and perform well across different semantic segmentation datasets, but the performance is poor on the complex instance segmentation setting. SegRefiner interprets segmentation refinement as data generation process but the diffusion step is inefficient for practical use. Dense CRF Krähenbühl Koltun (2011) is training-free free post-process approach but it lacks high-level semantic context and usually struggles to work in complex scenarios. Differently, we aim to design a versatile, generic and efficient post-processing tool across diverse segmentation models, tasks and datasets, which makes it a highly meaningful and valuable tool with broad applications.
2.3
SEGMENT ANYTHING MODEL
Segment Anything Model (SAM) has been considered as a milestone vision foundation model for promptable image segmentation. Several works have used this powerful foundation model to benefit downstream vision tasks, including object tracking Cheng et al. (2023); Yang et al. (2023a), image editing Gao et al. (2023), 3D object reconstruction Shen et al. (2023) and many real-world scenarios Ma et al. (2024); Han et al. (2023); Tang et al. (2023), while the potential of SAM in segmentation refinement task and the effect of different prompt types has been barely explored.
METHOD
In this section, introduce our proposed mask refinement framework, is shown in Fig. 3. We first review the architecture of SAM and its usage. Then, we introduce multi prompt excavation strategies exploit SAM. We further present an efficient adaption variant to enhance the accuracy of IoU predictions in belf-boosted manner.
3.1 REVIEW OF SAM
We start by introducing the components of SAM, which consists of an image encoder, a prompt encoder, and mask decoder. 1) The image encoder is based standard Vision Transformer
4

---

Published as a conference paper at ICLR 2025
Dog
Carra Mank
Tight Bas Prasept
Flaute Bax Prenept
Psint-Bes Pronpt
Prompt
Sppti-T Margo
Caara Mank
wia Mask Pronpt
Wi Mask Prompt
GT
(a) Effects of the context-aware elastic box (top two rows) and mask prompt (last two rows).
(b) Effects of the proposed split-then-merge (STM) strategy.
Figure 4:
Visualizations our proposed techniques effects. All of them play a crucial role in mitigating the impact of defects in coarse masks.
(ViT) pre-trained by MAE He et al. (2022). It generates a 16x downsampled embedding of the input image. 2) The prompt encoder can be either sparse (points, boxes, text) dense (masks). For sparse prompts, points and boxes are represented as positional encodings summed with learned embeddings. Text prompts are processed by the text encoder of CLIP Radford al. (2021). Dense prompts are directly convolved with the image embeddings and summed element-wise. 3) The mask decoder employs prompt-based self-attention and two-way cross-ane auss-aattntioo. This allows interaction between prompt-to-image and image-to-prone to-ommpptppp embeddings, enabling simultaneous updates to the encoded image and prompt features. After two decoder layers, the output mask tokens are processed by 3-layer MLP and then perform spatially point-wise product with the upsampled image embedding to get target masks.
SAM is able to produce both single mask multiple masks (i.e, three masks) for each input prompt. The multi-mask mode is designed to address the ambiguity problem and an additional IoU token is adopted to learn the confidence of each mask, which reflects the IoU between each predicted mask and the target object. In Fig. 5a, we empirically find that the multi-mask mode is generally superior to the single-mask mode by simply selecting the mask with the best IoU predictions adopt the multi-mask mode in our experiments.
3.2 PROMPT EXCAVATION
As promptable segmentation model, the input prompts play a crucial role in SAM because these prompts provide localization guidance of intended objects. To employ SAM for mask refinement, we need mine prompts merely based the initial coarse masks, which is challenging for the existence of noise and defects. Unlike previous works that mostly use one type of prompt Dai et al. (2023), our prompt excavation strategies aim mine diverse and scemly prompts (including points, boxes and masks), making them collaborate with each other to mitigate the effect of defects in coarse masks. Note that SAM fails work by merely using the mask as an input prompt, and we provide analysis in subsequent parts.
The point prompt can provide position information for either foreground background objects. However, it is difficult to determine the most salient point when using binary coarse masks. To solve this challenge, we leverage a simple but empirically effective object-centric prior: The center of object tends to be positive and boundaries. Based on this criteria, we select the foreground point that has a maximum distance to the nearest background position the positive prompt. Similarly, the negative prompt should satisfy the following principle: 1) the point is farthest away from the foreground region; 2) the point is within the bounding box of the foreground region.
Points.
while uncertainty is mostly located along
Boxes. The box prompt shows a more powerful localization ability for the abundant cues it contains. Given binary mask, is simple find the maximum bounding rectangle (tight box) of foreground
5

---

Published as a conference paper at ICLR 2025
regions the box prompt. However, the false-negative pixels in the coarse mask may hinder the quality of the bounding box, resulting in incomplete coverage of the potential object (Fig. 4a).
To address this, propose context-aware elastic box (CEBox) to adjust the tight box conditionally. The bounding box can be expanded in four directions according to the surrounding context. Specif- ically, the input image I E RH xfxx x is encoded feature embedding Fim. RAxwxe in SAM latent space by the image encoder, where (H, W), (h, w) denote original image size and embedding size. The coarse mask Mcoaree E "R' is resized to M E Rhxx to keep aligned with F.m. We calculate the mean feature embedding of the coarse mask (denoted as query embedding) as follows:
F query
1
Σ [Iiss(Fiml
(1)
[1mool
where 1 م4so {0, 1) is the indicator function to determine foreground regions, | represents the number of elements. We calculate the affinity between Fquery and each spatial location in resized image embedding Fimm E RfXxxxe to obtain similarity map Sim E R#f and binary it by 0.5:
Sim [Fquery Fiml>=0.5
(2)
For each direction in [left, right, up, downJ, we enlarge the tight box B by 10% of the corresponding side length and approximate the positive ratio in the enlarged region Simcontezt- A threshold is used to determine the necessity to expand the current box in this direction. To avoid over-enlarge, limit the maximum expanding pixels each time and run multiple iterations for progressive expansion.
Masks. Most existing works employ point or box as the initial prompt while mask prompt is usually discarded Dai et al. (2023); Chen et al. (2023); Zhang et al. (2023). The mask fails to serve as the initial input prompt for SAM separately (seeing qualitative results in Fig. 2 and quantitative results in Tab. 1). This is because the mask prompt merely acts as an auxiliary for point and box in the cascade refinement during SAM pre-training, with the predicted logits of the previous iteration as input to guide the next one. However, we argue that the mask prompt is vital to distinguish foreground and background in the mask refinement task, especially the case that the box prompt fails to work (e.g. the oversized box results in falsely detected objects or background in Fig. 4a). Considering the inaccuracy of coarse mask, we leverage a Gaussian-style mask GM based on distance transform used in point prompt:
GM = w exppp (x v0)) + (y yo)?
(3)
M .7
where GM(z, y) represents the mask prompt at location (I, v), (Fo, yo) is the mask center point that farthest the background regions, w, y are the factors control the amplitude and span of the distribution. We provide detailed analysis of the Gaussian Mask in the Appendix Appendix C.2.
Application on Semantic Segmentation. The semantic segmentation mask category-wiss and there may exist many objects in semantic mask. In Fig. 4b, we find that SAM struggles to segment multiple objects with large span (either miss-detect or falsely detect) using common prompts. Although the mask prompt can mitigate this problem, it fails when objects of different categories are mingled. We further propose a split-then-merge (STM) pipeline to solve it. 1) Split: we split the mask by finding all connected regions. Note that some regions noisy and trivial due to the inaccuracy of coarse masks. 2) Merge: To form semantically meaningful regions, we iteratively merge the close regions based on the box area variation and mask area occupancy. Two regions will be merged only if the change of box area of (before and after region merging) is small and the mask area occupancy of the merged box is enough. An elaboration of this strategy is provided in Algorithm 1.
3.3 IOU ADAPTION
For SAM, the quality of the generated masks determined by the input prompts, while the selection of the best mask is based on IoU predictions (denoted as loU Traditional SAM leverages an individual token to produce the mask when multiple prompts are given ssinglemaskk mode). However, in Fig. 5a, empirically observe that selecting the best mask from multiple predictions of SAM based on IoU pred is generally superior to the single-man mode under all prompt combinations. We give detailed analysis in the Appendix Appendix C.4. However, in Fig. 5b, we find that

---

Published as a conference paper at ICLR 2025
ed in Inkeeiiiissln
Topa Aee
muskiii (611
Sifal fox
Pronge Type
Prosge Type
Pronupt Type
(a) Mask quality using different prompt types.
(b) The effect of different IoU lection criteria.
(c) Top-1 accuracy using different
IoU selection criteria.
Figure 5: The effect of different prompt types, mask modes and IoU selection criteria on DAVIS-
mask selection based on pred still falls short of the upper limit (select mask by ground-trute
loUs loUGr), which indicates the inaccuracy of SAM's top-1 loU prediction. This is because the abovementioned SAMRefiner, which is training-free free and generalized to most cases, is agnostic to downstream categories. SAM's IoU head is not specifically trained for intended objects, leading to suboptimal IoU predictions.
For the mask refinement task, where the GT' masks are unavailable, we propose that coarse masks can act as effective priors guide IoU predictions for domain-specific categories. To verify it, we denote the IoU between SAM's output mask and coarse mask as LoUcooree and compare the top-1 IoU accuracy of IoU'oarse against poUpred Results in Fig. 5c indicate that the top-1 performance based on IoUcoarse outperforms ToUpred for simple point and box prompts, which is close to that based on ground truth. contrast, the coarse IoU performs poorly in multi-prompt cases. It is likely that the less prompt provides ambiguous guidance for SAM and results in variant masks, enabling the coarse masks to provide effective guidance selecting the intended one. However, multi-p pooltitpropps have better quality than the coarse mask and thus may mislead the selection. To pursue better performance, we enhance SAM's IoU ranking ability by training under the single prompt case supervised by ToUcoarse and expect it to benefit multi-pron prompt cases. This process is conducted in self-hoosted manner and requires no extra annotations.
masks generated by
Specifically, focus minimal adaptation of SAM toward better IoU predictions. To preserve the zero-shot transfer capability of SAM, we fix the model parameters of the pre-trained SAM and only add LoRA-style adaptor Hu et al. (2021) in the IoU head, as is shown in Fig. 3b. Considering the inaccuracy of ToU Coureeee. we adopt a ranking-based loss instead of a regression loss. In particular, for each SAM's predicted mask M, and its predicted IoU x;, we calculate their coarse IoU and denote the index of the mask with the best coarse IoU as j. The pairwise ranking loss is computed as:
loss =
Σ max(0, x, 8j + m) $=1,8f
where is the number of total masks (3 for SAM), m is the margin to control the minimal difference. This loss encourages the best IoU score xy to be higher than the remaining ones, thus promoting the accuracy of top-1 prediction. We train the adaptor based on the single prompt and use multi- prompt during inference. Note that despite LoRA's popularity. its optimal placement remains unclear. Previous works empirically place LoRA layers in some specific layers (e.g, backbone), altering existing knowledge to adapt to new domains, which modify the learned knowledge and affect the mask generation. In contrast, our approach inserts the LoRA layer in the IoU head, preserving SAM's full capability to generate high-quality masks while improving mask selection. To our knowledge, this minimal adaptation is underexplored and may provide new insights for the field. We denote SAMRefiner with this adaption step as SAMRefiner++, which only focuses on selecting better masks on the target dataset and has no effect on mask generation.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets and Implementation Details. For a comprehensive evaluation of the mask refinement performance of SAMRefiner, conduct experiments wide range of benchmarks, including those
7

---

Published as a conference paper at ICLR 2025
Table 1: The quality of refined masks using different prompts and the effect of IoU adaption on DAVIS- 185.5555. Results are presented as SAMRefiner/AA SAMRefiner++
Prompt Type
IoU
boundary IoU
Top-1 Acc
Coarse Mask
81.4
71.4
Point
53.7 / 56.4
49.9/533
51.5 / 62.1
Box
68.8 70.8
63.9/63.5
36.4 56.8
Mask
37.3 / 40.4
32.6/33.5
28.4/30.8
Point + Box
60.0/
37.1/
Point Mask
77.5 80.6
/7.7
43.2
Box + Mask
84.6 85.1
7
36.2 60.7
ALL
86.9 /7.1
75.1/75.4
designed for mask refinement (DAVIS-585Chen) et (2022)), instance segmentation (COCOLin et al. (2014)), semantic segmentation (VOCEveringham et al. (2010)) under different settings. As mask refiner, our method keeps the same setting as each baseline for pair comparison. The metrics we used include (boundary) loU Cheng et al. (2021), (boundary) mask AP and mloU. The threshold and μ used in the box and mask prompt are set to 0.1 and 0.5 respectively. The factors y for Gaussian distribution are set to 15 and 4 by default. We adopt pre-trained SAM with the VIT-H image encoder as the segmentation foundation model and more details are provided the Appendix.
4.2 ABLATION EXPERIMENTS
In this section, we conduct detailed ablation studies to analyze the effect of each component in our framework. We mainly experiment DAVIS-585, as it is specifically designed for mask correction and contains various defects in the mask. We also leverage popular COCO and VOC to evaluate our method for specific scenarios.
Effect of different prompts and IoU adaption. Tab. shows the performance of using different prompts for SAMRefiner. The results indicate that our proposed pulti-ppomptt excavation strategy performs better than the single prompt. The mask prompt, which is barely considered in previous works, shows poor performance on its own but can bring an obvious advantage of nearly 20% IoU for point and box. Besides, we compare the mask selection by SAM's original IoU predictions (number before /) and our adapted IoU head (number after /). The loU adaption step can significantly boost the top-1 accuracy of the best mask selection and further benefit the final IoU performance. We provide more ablation studies in Appendix for the page limit.
Effect of different design choices in SAMRefiner. We analyze the impact of different design choices for each prompt and report their relative contribution in Tab. (1) We compare different strategies to sample the positive point prompt in Tab. 2a, including randomly choosing from the coarse mask, selecting the center of the bounding box, and selecting the point having a maximum distance to the background. Compared random choice, using the box center shows even worse performance. This is because the bounding box is sensitive to the noise in the mask (e.g, the distant false positives), resulting in the inaccuracy of the box center. Our distance-guided and can obtain 52.5% IoU with only positive point. The performance can be further improved to 53.7% by adding the extra negative point mentioned in Sec. 3.2. (2) Tab. 2b shows the impact of using the tight box and context-aware elastic box (CEBox). We adopt the coarse masks generated from PointWSSIS Kim al. (2023) on COCO, which usually suffers from incomplete masks. The proposed CEBox can produce better boxes with higher A and benefit mask generation. (3) In Tab. 2c, we compare the commonly used mloU with/without STM. Results show that STM can bring remarkable improvement for extremely coarse masks (i.e, 6.2% for MaskCLIP) and can constantly promote performance on better initial masks.
more robust to the noise
4.3 APPLICATION ON INCOMPLETE SUPERVISION
Instance Segmentation. To verify the effectiveness of our framework, we apply it to various typical methods, including unsupervised (CutL.RR Wang et al. (2023b)), semi-supervised (NoisyBound-
8

---

Published as a conference paper at ICLR 2025
Table 2: Ablation study of our proposed strategies on different cases.
(a) Point sampling strategy.
(b) Context-aware elastic box.
(c) STM strategy.
Point
IoU
bloU
CEBox
Apha
Apmass
Aphoundury
STM
MaskCLIP
CLIP-ES
Random
Box Center
22.2
37.8
26.3
38.5
x
36.7
37.5
25.6
X
51.1
79.1
Distance-
53.7
56.4
38.2
37.8
25.9
57.3
79.3
Table 3: Results of instance segmentation under different supervisions on COCO 2017. The Anno- tations denote the supervision type, including 24 ((unlabeled), P(point-level label), F(full labeled). Networks represent the final segmentation model trained based on the pseudo masks. We follow the default setting of each baseline method.
COCO trainšK
COCO val2017
Methods
Annotations
Networks
Apmi
Aprastry
Aprra
Aphumbry
Unsupervised
CuILER
None None
Cascade R CNN
8.8
2.8
SAMRRfiner
Cascade R-CNN
12.1 12.1( (+3.3)
(.0(+).2)
Semi-supervised
NB
J 1% 14 99%
Mask R CNN
4.4
1.6
6.7
2.3
SAMRefiner
F 1% + 99%
Mask R-CNN
(.9(2
4.4[ +2.8)
11.8( (5.1)
6.5(+44.2)
NB
F 5% 4 95%
Mask R-CNN
18.3
8.8
24.0
12.4
+SMMReffnr
F 5% L4 95%
Mask R-CNN
(2.0(+4.0)
14.4 (5.6)
27.4(+33.0
16.5 (411)
NB
110% 90%
Mask CNN
11.8
28.9
16.3
SAMRefiner
F' 10% 7/ 90%
Mask R-CNN
26.
17.0 +5.2)
30.5 (1.6)
18.6 (2233)
Weakly Semi-supervised
PSSnIWSSIS
Jr 11%
SOLOv2
11.5
SAMReffner
J' 1%% 7 99%
SOL.v2
25.4(+10.3)
16.3 (.06)
30.2( +6.3)
18.2 (+6.7)
PointWSSIS
F.5%%
95%
SOLOv2
32.3
19.7
35.4
19.6
+SAMRefner
5.5% + 95%
SOLOv2
37.7(+3.4)
25.9 (62)
34.6 (122)
21.6(42.0)
PointWSSIS
710%
909
SOLOv2
39.9
26.4
35.5
21.9
+SAMRofner
-100% 90%
501.02
42.8 +2.9)
30.2 3.8)
(60.6)))))
22.9(+
ary Wang et al. (2022)) and weakly supervipe (PointWSSIS Kim al. (2023)). Experiments are conducted on COCO following these methods. We evaluate the mask quality in terms of two aspects: 1) the performance of pseudo masks on the train set and 2) the performance of the final segmentation model trained based on these pscudo masks. The pscudo masks are evaluated subset of COCO train set (train 5K) and the final segmentation model Cai & Vasconcelos (2018); He al. (2017); Wang et al. (2020a) is evaluated on the validation set. We compare both the commonly used mask AP and boundary AP. Results in Tab. 3 demonstrate the superiority of our framework. can constantly boost the quality of pseudo masks in all settings, especially for those label-limited scenarios (e.g, the improvement for PointWSSIS with I 1% annotations can reach 10.3%). Besides, the segmentation model can also benefit from refined masks, with significant improvement under different settings, demonstrating that SAM can offer valuable knowledge and cues to improve these scenarios.
Semantic Segmentation. Tab, 4 shows the improvement of pseudo masks generated by unsupervised semantic segmentation (MaskCLIP Zhou et al. (2022)) and weakly supervised semantic segmenta- tion (BECO Rong et al. (2023) and CLIP-ES Lin et al. (2023)). We refine the pseudo masks on the train set and use them to train a DeepLab V2 Chen al. (2017) model following Lin et al. (2023). The results show that our method brings obvious performance gains for both the pseudo masks and segmentation models. The average improvement of pseudo masks is more than 5% and even approaches 10% for MaskCLIP and CLIP-ES. The superior performance across various datasets and settings demonstrates the generalization and flexibility of our framework.
4.4 COMPARISON WITH -TATE-OF-THE-ATT
In Tab. 5, we compare our SAMRefiner with state-of-the-art model-agnostic refinement methods, including dense CRFKrähenbühl & Koltun (2011), CascadePSPCheng et al. (2020), CRMShen et al. (2022) and SegRefinerWang et al. (2023a). We first conduct experiments previously used DAVIS- 585, COCO and VOC. The results prove that 1) CRF shows inferior performance due to the lack of level semantic context and unfitness for the binary mask. 2) CascadePSP and CRM
9

---

Published as a conference paper at ICLR 2025
Table 6: Performance of refined masks on COCO val set using LVIS annotations.
(a) Results MaskRCNN.
(b) Results on more segmentation models.
Method
Apraat
Apheesdayy
Method
Aprza
Aphassay
Method
Apeza
Aphosstay
MRCNN(RN50
39.8
27.3
PaintRend
30.6
SOLO
+SagFix
+NegReliner
42.8
53.7
+SegRetiner
40.3
31.3
+BRP
30.4
+SAMReliner
45.50=10)
36.0
+SAMReliner
44.1
34.2(-9.5)
+SegRefiicc
11.9
32.6
ASAMReliner
45 15.3 5)
35,9++8.6)
RefineMask
41,2
30.5
Condlnst
+SegRelincr
41,9
$3.0
+SegReliner
41.1
32.2
MRCNNIRNIOI)
SAMReeliner
4.57
34.3(448))
4SAMReliner
45, 45-24))).))
35.81
0.6)
+SegFix
42.2
30.6
+BRP
42.8
32.0
MaskTrunsifiee
42.2
31.6
MaskeFormer
SSggRefiner
43.6
34,1
SSgRRefiner
43.3
34.4
SSeRRefiner
47.4
38.8
+SAMRefiner
(6.6(+5.0)
(6.9(+7799)
+SAMRefiner
46.30
36, 3643(44.7)
4SAMReliner
O5 +22.)
39.01
2200)
show competitive performance semantic segmentation (VOC), but the improvement is limited or even worse than coarse masks on instance segmentation (DAVIS-585 and COCO). It is likely that these methods are trained on a merged dataset consisting of extremely accurate mask annotations, which has strong relation to VOC and makes them fail to generalize to complex scenarios like COCO. We also explore the use of high-quality datasets on SAM (i.e, HQ-SAM Ke al. (2023)) in the Appendix. 3) SegRefiner's performance is not stable across different settings because it lacks the ability to process diverse defects in the coarse masks. 4) SAMRefiner is more generic and can improve performance remarkably on various datasets due to its better robustness to the mask noise.
Table 4: Results of semantic segmentation un- der different supervisions on PASCAL VOC 2012. The Annotations denote the supervi- sion type, including (uunabbeled), (imagee- level label). Results val set based training a Deepl L2 model.
Table 5: Comparisons with SOTA methods. CM represents Coarse Mask.
Source
CM
CRF
PSP
CRM
SR
Ours
DAVIS-585
DAVIS-585
81.4
81.0
81.9
82,9
80.3
87.1
COCO
NB
15.2
13.9
15.9
15.1
15.8
18.4
PointWSSIS
29.1
24.4
28.9
25.6
29.7
35.3
Methods
Amactations
mioU(train)
mioU(val)
MaskRCNN
35.2
31,5
34.6
31.7
35.4
36.5
MasKCLIP
24
47.8
47.3
PASCAL VOC
+SAMReliner
24
(49..)
53.5 63.5
MaskCLIP
47.8
48.2
55.3
56.8
58.5
57.3
BECO
66.3
69.5
BECO
66.5
+SAMRefiner
I
71.8 (5.5)
70.9 (+1.4)
CLIP-ES
70.8
72.6
76.9
78.7
74.7
79.3
CLIP-ES
70.8
70.3
DeeplaabV2
76.5
77.8
81.2
81.6
83.1
78.8
+SAMRefiner
I
(
+3.6)
Time (h)
1.0
3.4
1.5
1.4
0.6
Besides, we compare the total time refine masks for COCO train5K (with about 5K images and 37K masks). CRF is tested with 16 workers, and others are based on one 3090 GPU. SAMRefiner takes less than half the inference time compared to previous methods because SAM can batch process multiple masks in an image simultaneously, while other methods can only refine one mask each time. The batch processing capability makes SAMRefiner more efficient and competitive practical use.
In addition, considering the original ground- ground-truth truth annotations used in the COCO dataset are not accurate, we follow SegRefiner Wang et al. (2023a) to evaluate the predictions of different fully supervised segmentation models on COCO val set using LVISGupta et al. (2019) annotations. Results in Tab. 6 indicate that our method outperforms other works by a large margin and can consistently enhance the mask quality generated by various networks (e.g, both CNN and Transformer), validating its generality for broad applications.
CONCLUSION
This paper uncovers the deficiency of SAM in the mask refinement task and proposes a universal and efficient framework called SAMRefiner adapt SAM for mask refinement. We propose excavation to generate diverse prompts that are robust to the defects in coarse masks. the target dataset
An optional loU adaption step is introduced to further boost the performance without additional annotated data. We evaluate SAMRefiner on a wide range of image segmentation benchmarks under different settings, demonstrating its consistent accuracy and efficiency.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENT
This work was supported in part by The National Key R&D Program of China (NO.2022240010110))
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR, 2018. 3
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, pp. 6164-6162, 2018. 9
Blang-Chied Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI, 40(4):834-848, 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, pp. 801-818, 2018. 18
Tianle Chen, Zheda Mai, Ruiwen Li, and Wei-lun Chao. Segment anything model (sam) enhanced pseudo labels for weakly supervised semantic segmentation. arXiv preprint arXiv:2305.05M0 2023. 6, 16
Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Focalclick: Towards practical interactive image segmentation. In CVPR, pp. 1300-1309, 2022. 1, 8, 15
Bowen Cheng. Ross Girshick. Piotr Dollar, Alexander C. Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In CVPR, pp. 15334-15342, June 2021. 8
Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi- Tang. Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In CVPR, pp. 8890-8899, 2020. 1, 4, 9, 18
Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang. Wenguan Wang, and Yi Yang. Segment and track anything. arXiv preprint arXiv:2305.06538 2023. 4
Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In CVPR, pp. 16794 16804, 2021.
Haixing Dai, Chong Ma, Zhengliang Liu, Yiwei Li, Peng Shu, Xiaozheng Wei, Lin Zhao, Zihao Wu, Dajiang Zhu, Wei Liu, al. Samaug: Point prompt augmentation for segment anything model. arXiv preprint arXiv:2307.01 187, 2023. 5, 6
Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes supervise convolutional networks for semantic segmentation. In ICCV, 2015. 3
Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2):303-33 338-338, 2010. 8
Dominik Filipiak, Andrzej Zapala, Piotr Tempezyk, Anna Fensel, and Marek Cygan. Polite teacher: Semi-supervised instance segmentation with mutual learning and pseudo-label thresholding. arXiv preprint arXiv:221 1.008500 2022. 3
Shanghua Gao, Zhijie Lin, Xingyu Xie, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Editany- thing: Empowering unparalleled flexibility in image editing and generation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 9414 9416, 2023. 4
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In CVPR, pp. 5356-5364, 2019. 10, 18

---

Published as a conference paper at ICLR 2025
Dongsheng Han, Chaoning Zhang. Yu Qiao, Maryam Qamar, Yuma Jung, SeungKyu Lee, Sung- Ho Bae, and Choong Seon Hong. Segment anything model (sam) meets glass: Mirror and transparent objects cannot be easily detected. arXiv preprint arXiv:2305.0000
2023. 4
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, pp. 991-998, 2011. 15
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961-2969, 2017. 4,
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp- 16000-16009, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXiv:2/06009668 2021. 7
Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chiel Chen. Segsort: Segmentation by discriminative sorting of segments. In ICCV, pp. 7334-7344, 2019..
Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask transfiner for high-quality instance segmentation. In CVPR, pp. 4412-4421, 2022a.
Lei Ke, Mingqiao Ye, Martin Danelljan, Yifan Liu, Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu.
Segment anything in high quality. arXiv preprint arXiv:2306.01567. 2023. 10, 17
Tsung-We Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X Yu. Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers. In CVPR, pp. 2571-2581, 20226. 3
Beomyoung Kim, Joonhyun Jeong, Dongyoon Han, and Sung Hwang. The devil is in the points: Weakly semi-supervised instance segmentation via point-guided mask representation. In CVPR, pp. 11360-11370, 2023. 8, 9, 15, 17
Alexander Kirillov, Yuxin Wu, Kaiming Hc, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR, pp. 9799-9999999 7808, 2020. 3,4
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint vrXiv:2304.02643 2023. 2
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NeurIPS, 2011. 2, 4,
Jungbeom Lee, Eunji Kim, and Sungroh Yoon. Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation. In CVPR, pp. 4071-4080, 2021. 15
Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervieer convolu- tional networks for semantic segmentation. CVPR, 2016.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 3, 8, 18
Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofei He. Clip is also efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In CVPR, pp. 15305-15314, 2023. 1, 2, 9, 15, 17
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
Alexander Neubeck and Luc Van Gool. Efficient non- maximum suppression. In ICPR, volume 3, pp. 850-855. IEEF, 2006. 16
12

---

Published as a conference paper at ICLR 2025
George Papandreou, Liang-Chine Chich Chen, Kevin P. Murphy, and Alan Loddon Yuille. Weakly-and semi-suuerrised learning of a deep convolutional network for semantic image segmentation. In ICCV, 2015. 3
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32, 2019. 15
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, PMLR, 2021. 5
Shenghai Rong, Bohai Tu, Zilei Wang, and Junjie Li. Boundary-enhanced co-training for weakly supervised semantic segmentation. In CVPR, pp. 19574-19584, 2023. 9, 17
Qiuhong Shen, Xingyi Yang, and Xinchao Wang. Anything-3d: Towards single-view anything reconstruction in the wild. arXiv preprint arXiv:2304. /0261, 2023. 4
Tiancheng Shen, Yuechen Zhang, Lu Qi. Jason Kuen, Xingyu Xie, Jianlong Wu, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, pp. 1310-1319, 2022. 2, 18
Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. In NeurlPS, 2022. 3
Gyungin Shin, Weidi Xie, and Samuel Albanic. Namedmask: Distilling segmenters from comple- mentary foundation models. In CVPR, pp. 4960-4969, 2023. 3
Chufeng Tang, Hang Chen, Xiao Li, Jianmin Li, Zhaoxiang Zhang, and Xiaolin Hu. Look closer to segment better: Boundary patch refinement for instance segmentation. In CVPR, pp. 13926-13935. 2021. 4
Lv Tang, Haoke Xiao, and Bo Li. Can sam segment anything? when sam meets camouflaged object detection. arXiv preprini 0447: 2023. 4
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In ICCV, pp. 10052-10062, 2021.
Mengyu Wang, Henghui Ding, Jun Hao Liew, Jiajun Liu, Yao Zhao, and Yunchao Wei. SegRefiner:
Towards model-agnostic segmentation refinement with discrete diffusion process. In NeurIPS, 2023a. 2, 9, 10
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. NeurIPS, 2020a.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In CVPR, pp. 3124-3134, 2023b. 1, 8
Yude Wang, Jie Zhang, Meina Kan, S. Shan, and Xilin Chen. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In CVPR, 2020b.
Zhenyu Wang, Yali Li, and Shengjin Wang. Noisy boundaries: Lemon or lemonade for semi-
supervised instance segmentation? In CVPR, pp. 16826-16835, 2022. 1, 3. 9, 15, 17
Jinheng Xie, Xianxu Hou, Kai Ye, and Linlin Shen. CLIMS: Cross language image matching for
weakly supervised semantic segmentation. In CVPR, June 2022.
Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmentation
with prototype- proset based consistency regularization. NewrlPS, 35:26007-26020, 2022a. 3
Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multi-class token transformer for weakly supervised semantic segmentation. In CVPR, 2022b. 3
Ning Xu. Brian Price, Scott Cohen, and Thomas Huang. Deep image matting, In CVPR, pp. 2970-2979, 2010-
13

---

Published as a conference paper at ICLR 2025
Jinyu Yang. Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang. and Feng Zheng. Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.1166. 2023a.
Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi-supervised semantic segmentation. In CVPR, pp. 4268-4277, 2022. 1
Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting yeak-to-strong
consistency in semi sum-rvieee semantic segmentation. In CVPR, pp. 7236-7246, 2023b. 3
Yuhui Yuan, Jingyi Xie, Xilin Chen, and Jingdong Wang. Segfix: Model-agnostic boundary refine- ment for segmentation. In ECCV, pp. 489-506. Springer, 2020. 3
Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmenta- tion networks with iterative refinement and attentive few-shot learning. In CVPR, pp. 5217-5226, 2019. 3
Gang Zhang. Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Li, and Xiaolin Hu. Refinemask: Towards high-quality instance segmentation with fine-grained features. CVPR, pp. 6861-6869, 2021. 1, 3,4
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hong- sheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048; 2023. 6
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022.
Yi Zhu, Zhongyue Zhang, Chongruo Wu, Zhi Zhang, Tong He, Hang Zhang, R Manmatha, Mu Li, and Alexander Smola. Improving semantic segmentation via efficient self-training. IEEE TPAMI, 2021. 1
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In CVPR, pp. 14502-14511, 2022. 3
14

---

Published as a conference paper at ICLR 2025
Algorithm The Region Merging Strategy
Input:
Region number R; region label for each mask pixel Miaset; a hyper-parameter Jr.
Output:
Merged regions masks M
1: Initialize M Merge Miobels M Ø.
2: for 1 to R do
3: B; Extract minimum bounding box in M iabel-
4: af a: aquask <- Compute areas for B, M Mabeet
5:
for = i to R do
6:
Bj + Extract minimum bounding box in M.
7:
agor, . .apask + Compute areas for Bj. MM
8:
(B, @bor)= + Find merged boxes for (B, and compute its area.
9:
if (afor + a > f ahhor and (ammask + a maskk) p bor then Merge region and region j. Update Mmerge.
10:
12: end if
end for
14: end for
15: G f Extract merged region labels from Mmerge
16: for k 9 do
17: M hwwpandpppddde.cc M Meegger.
18: end for
19: return M
A ADDITIONAL DETAILS
A.1 DATASETS DETAILS
DAVIS-585. DAVIS-585 is proposed in FocalClick Chen (2022) evaluate the interactive mask correction task. It consists of 585 samples and generates the flawed initial masks by simulating the defects on ground-truth masks using super-pixels. There are different types of defects, egg, boundary error, external false positive, and internal true negative, making it comprehensive benchmark for the mask correction task.
MS COCO 2017. COCO comprises 80 object classes and one background class, with 118,287 training samples and 5,000 validation samples. We perform instance segmentation experiments on COCO following previous works Wang et al. (2022); Kim al. (2023). To ensure fair comparison, we maintain the same split of data subsets (e.g. 1 1%, 5%, 10%) as each baseline method. We assess pseudo labels quality by randomly sampling 5,000 images in the train (denoted train5K) that have no intersection with annotated data subsets.
PASCAL VOC 2012. We conduct semantic segmentation experiments on PASCAL VOC 2012 following Lin et al. (2023); Lee et al. (2021). It contains 20 categories and one background category. We evaluate the pseudo mask quality on the train set with 1464 images. An augmented set with 10,582 images Hariharan et al. (201 1) is usually used for training in the WSSS task.
A.2 IMPLEMENTATION DETAILS
We implement our method with PyTorch Paszke et al. (2019). For SAMRefiner, we didn't use multi-scale strategy and images are kept at their original sizes before being processed by SAM. For IoU adaption step, we use SGD optimizer with 0.01 learning rate. The batch size is set to and we only train for I epoch. The learning rate is reduced to one-tenth at steps 60 and 100. We use margin ranking loss with the margin as 0.02 and the LoRA rank is set to 4. Note that the IoU adaption step is optional and only adopt
DAVIS-585. The time cost reported in the paper is tested 3090 GPU. For instance segmentation, the threshold À is set to 0. for the box prompt. For semantic segmentation, y used in the STM is set 0.5. The factors w, Y for Gaussian distribution
single
to 15
15

---

Published as a conference paper at ICLR 2025
Table 7: Quantitive comparison between automatic grid point prompt and our prompt strategy on DAVIS-585.
Prompt Type
IoU
boundary IoU
Time (minute)
Coarse Mask
81.4
71.4
Max IoU
70.6
65.5
8.0
Merge
81.9
73.1
8.0
Ours
86.9
75.1
1.6
GT
Coarse
Everything
IoU
Merging
Ours
Figure 6: Qualitive comparison with grid point prompts.
and by default. We present the pseudo-code for the region merging strategy in Algorithm which is an important component of our split-then-merge (STM) pipeline for semantic segmentation.
B ADDITIONAL EXPERIMENTS
COMPARISON WITH AUTOMATIC MASK GENERATOR
SAM can produce masks for an entire image by sampling grid of points over the image prompts. This automatic manner can be used for mask refinement by matching the potential masks to the coarse mask. We validate its performance leveraging two matching criteria: 1) Max IoU: For each coarse mask, we select the SAM-generated segments with the highest IoU as the refined mask. 2) Merging: For each SAM-generater segment, is viewed part of the final refined mask if the overlap area between this segment and coarse mask exceeds a certain percentage (e.g, 0.5) this segment Chen et al. (2023).
We compare our prompt excavation strategy with these two automatic grid-style point prompts in Tab. 7. We note that the performance drops severely for the Max loU approach and barely improves for the Merging approach on DAVIS-585. It stems from the inherent drawbacks of this prompt generation manner, which is shown in Fig. 6. First, the grid point prompts split an image into several fine-grained masks and it is difficult to control the granularity. The best-matched mask selected by Max IoU usually fails cover the whole object. Second, although the Merging strategy can obtain relatively complete objects, it is susceptible to defects in the coarse mask (e.g. false positives) and tends result in over-detected. Thirdly, the SAM-generated segments are not exclusive and sometimes an object may be included in multiple masks with different granularity. It remains challenging to filter them out by the strategies above. contrast to this bottom-up paradigm, prompt excavation strategy directly produces diverse prompts for the target object (lop-down paradigm), which is more purposive, accurate and robust to the noise coarse masks. In addition, these grid-based prompts inefficient (i.e, taking 5x more time than ours) because of the massive prompts and time-conse consumnng post-pros pest-srocssiig (e.g. NMS Neubeck Van Gool (2006)) to filter low-quality and duplicate masks.
16

---

Published as a conference paper at ICLR 2025
10.0
57.
14
Inll
VTB
25.0
VTL
VTFI1
725
71
700
Iteration
Iteration
(a) Effects of different backbones and cascaded post-refinement
(b) Comparison with SAM predicted mask logits.
Figure 7: Ablation study of different backbones and cascaded post-refinement strategies as well as mask logits.
B.2 EFFECTS OF DIFFERENT BACKBONES AND CASCADED POST-REFINEMENT
The pre-trained SAM models are available with three backbone sizes and the mask can be iteratively processed by cascaded refinement. We compare the impact of using different backbones and iterations Fig. 7a. The results show that the largest ViT-H outperforms other backbones at the first iteration, and multiple iterations can further improve the mask quality, especially for the ViT-L backbone.
Note that the mask prompt of each iteration in the cascaded refinement from our prompt exvacation strategy (i.e, Gaussian-style mask). We also compare with some typical practices used in the original SAM, including (a) only using point and box prompt at the first iteration and adding SAM's predicted mask logits (produced by the previous iteration) subsequent iterations; (b) using all prompts generated by our method at the first iteration but replacing the Gaussian-style mask with SAM predicted mask logits for subsequent iterations. Results in Fig. 7b demonstrate that SAM's mask logits can contribute to point and box (condition a) in the cascaded refinement but fail to work when our mask prompt adopted in the initial step (condition b). This indicates that our Gaussian-style mask can provide more powerful guidance than the mask logits, which not only produce high-quality masks the initial step but also more advantageous for cascaded refinement.
B.3 UPGRADED RESULTS BASED ON HQ-SAM
HQ-SAM Ke et al. (2023) is an advanced version of SAM that enable more accurate segmentation. Our framework can also be applied to this powerful variant, and we conduct experiments on various benchmarks based on it to pursue better performance. In Tab. 8, compare the performance of our framework using SAM and HQ-SAM on DAVIS-585, VOC (BECO Rong et al. (2023) and CLIP-ES Lin et al. (2023)) and COCO (NB Wang et al. (2022) and WSSIS Kim (2023)). Results show that there is significant improvement on SAAIS-558 and VOC while the performance is fair to SAM on COCO. This is because HQ-SAM enhances original SAM by specifically training on high-quality dataset with large and salient objects, which aligns well with the characteristics of datasets like DAVIS-585 and VOC. In contrast, COCO has plenty of small objects and may not benefit much from HQ-SAM.
Table Comparison between SAM and HQ-SAM. We report IoU boundary IoU on DAVIS-585. AP/ boundary AP on COCO and mloU on VOC.
Model
DAVIS-582
NB
PointWSSIS
BECO
CLIP-ES
SAM
87.7 78.9
18.4/ 11.8
35.3/24.1
71.8
79.3
HQ-SAM
90.6/81.7
18.4/ 12.2
35.0/24.3
73.6
81.0
17

---

Published as a conference paper at ICLR 2025
Table 9: Additional experiment results BIG and relabeled PASCAL VOC datasets. The coarse
masks are produced from DeepLabV3+ Chen et al. (2018).
(a) Results on BIG dataset.
Method
Coarse Mask
SegFix
PSP
CRM
Ours
IoU
89.4
90.0
92.2
91.8
93.9
mBA
60.2
69.3
74,6
75.0
74.8
(b) Results relabeled VOC.
Method
Coarse Mask
SegFix
PSP
CRM
Ours
IoU
87.1
88.0
89.0
88.3
89.6
mBA
61.7
66.4
72.1
72.3
71.9
COCO
SAMRefiner
Figure Visualizations of COCO annotations and our refined annotations.
B.4 APPLICATIONS ON DIFFERENT TASKS
Application on high-resolution images. We evaluate our SAMRefiner on the BIG dataset, which includes ultra-high resolution images ranging from 2K to 6K. We directly refine the coarse masks generated by DeepLabV3+ Chen et al. (2018) based on SAM without dataset-specific finetuning, using IoU and mean Boundary Accuracy (mBA) as metries following Cheng et al. (2020); Shen et al. (2022). Results in Tab. 9a show that our framework can effectively improve the quality of coarse masks and is superior comparable the previous methods by using the powerful HQ-SAM.
Application on relabeled VOC. CascadePSP Cheng et al. (2020) introduces relabeled VOC dataset with accurate boundary annotations for better evaluation. We follow this setting validate framework on this benchmark. Results in Tab. 9b demonstrate the effectiveness of our method, with better IoU than existing methods. Note that the semantic obscurity may result in the inconsistency between human subjective annotations and SAM predictions, hindering SAM from obtaining better performance.
Application on human annotations correction. The human- -nnootated masks can also be coarse due the strict standard of pixel-accurate annotations. For example, COCO Lin al. (2014) masks are annotated in the polygon format, which is inaccurate in the boundary area (seeing Fig. 8). LVIS Gupta al. (2019) constructs more precise annotations for COCO images. We refine the mask in the COCO val set using our SAMRefiner and evaluate them based on LVIS annotations. Results in Tab. 10 show that our methods can also work for inaccurate human anmotations. There is a remarkable increase (i.e.
Table 10: Performance of refined masks COCO2017 val.
Data
Apmask
A pboundary
COCO
38.3
27.3
+Ours 43.5(+3.2)
33.0 +5.7)
18

---

Published as a conference paper at ICLR 2025
0
EMML
0.007
1999
0203
Ar
rauk AP
0.80
CO0L
8.002
35
0.5
(a) Analysis of w.Y. T.
(b) Analysis of X.
Figure Ablation study of (a): w, y and (b): À
3.2% mask AP and 5.7% boundary AP) for the mask quality. We provide qualitative comparisons in Fig. 8.
B.5 ADDITIONAL ABLATION STUDIES
Analysis of 7 in the mask prompt. We leverage a Gaussian-style mask in our prompt excavation strategy, with two factors w, controlling the amplitude and span of the distribution. We perform sensitive analysis of these two parameters in Fig. 9a. When is too small (i.e, w 1), the effect of mask prompts is negligible since the mask inputs the original SAM are the predicted logits, which are not scaled to 0-1. We note that relatively higher value for to benefit mask refinement, and the performance not sensitive to these higher w as well as 7
promote mask prompts
Analysis of À in the context-aware elastic box. We introduce threshold À in CEBox determine whether to expand current boxes based on context features, which controls the trade-off of box sizes. We give analysis of À in Fig. 9b. The proposed CEBox has consistent bonuses compared to the baseline (A = 0) under different thresholds. We set = 0.1 in our experiments to avoid over-enlargement.
Split
Merge
SAMRefiner
Region Region Region 3
Merged 1 Merged 2
Figure 10: Illustrations of Split-Then-Merge (STM) pipeline. The Region 3 in red color is small. Please zoom in for better visibility.
C TECHNICAL DETAILS
C.1 FURTHER ELABORATIONS STM
The Speit- (STM) pipeline is proposed to solve the multi-object case in the semantic segmentation. In this case, SAM struggles to segment multiple objects with a large distance using common prompts, resulting in either missed detection false detection (Fig. 4b). We propose STM to convert semantic masks with multiple objects into instance masks to ensure better compatibility with SAM. As shown in Fig. 10, STM includes two stages: 1) Split: split the mask by finding all
19

---

Published as a conference paper at ICLR 2025
Prompt Extraction
1
Without STM
*
Prompt Extraction
With STM
Figure 11: Visual comparisons between STM and baseline (without STM).
Original
w=1, y=0.1
w=1, y=0.2
w=1, y=1
Mask
Figure 12: Visualization of Gaussian-style Masks under different 7.
connected regions, which tends to be messy and noisy: 2) Merge: iteratively merge the adjacent regions form semantically meaningful regions based the box variation and mask occupancy (Algorithm 1). The STM is performed before prompt extraction. Once finished, we can produce prompts based the merged mask and leverage SAMRefiner for refinement. As shown in Fig. 11, STM can effectively mitigate the impact of multiple objects in semantic segmentation, yielding better results than the baseline.
C.2 FURTHER ELABORATIONS OF GAUSSIAN-STYLE MASK
Note that the central point is not the geometry central point of the mask, but the farthest positive point selected by the previous point prompt step. We only apply the Gaussian operation the foreground region of the mask, and the Gaussian-style mask is a generalized form of the coarse mask. For instance, when the amplitude w is set to and the span y is sufficiently large, the Gaussian-style mask is equivalent to the original coarse mask. Visualizations of the Gaussian Mask are presented in Fig. 12.
There are two main reasons for using the Gaussian-style mask: 1) Compatibility with SAM: The original SAM doesn't 't support the binary masks as prompts. This is because the mask prompt merely acts as an auxiliary for point and box in the cascade refinement during SAM pre-training, with the predicted logits of the previous iteration as input to guide next one. Therefore, the mask input for SAM requires logits with continuous values, while the original coarse mask is discrete-valued (0 and 1). The Gaussian operation can convert the binary mask to continuous, making it compatible with SAM. 2) The object-centrie prior: The center of object tends to be positive and feature- diseriminative, while uncertainty mostly located along boundaries. The Gaussian-style mask effectively reduces the weights
boundaries. As shown in Fig. 9a, when drops significantly due to the incompatible value space, while the Gaussian transformed mask can consistently outperform the original coarse mask under different w and 7.
the performance
C.3 FURTHER ELABORATIONS OF CEBOX
For SAM, the image features of different instances (even within the same category) exhibit distinct characteristics. This enables SAM to produce fine- grained component-lever segments, making support variety downstream applications. To illustrate this, we analyze feature similarity between different masks in Fig. 13a. As shown, the features of different instances, even within the same
20

---

Published as a conference paper at ICLR 2025
Simc 0.702
Segmented Image
Segmented Image
Simc 0.833
0.248
Sim: 8.871
Sim: 0.624
(a) Instance-level feature similarity.
(b) Component- -leeel feature similarity.
Figure 13: Visualizations feature similarity between base and other masks.
class, display certain differences. This characteristic allows SAM to distinguish between instances effectively (e.g, adjacent books). Similar conclusions can also be drawn for the part segmentation, as shown in Fig. 13b. On this basis, we can flexibly adjusting A, a threshold to determine the necessity expand the current box in each direction based on image feature similarity, according to different settings. For instance, relaxed threshold could be applied for general segmentations, while stricter threshold may be more suitable for fine-grained segmentations, such as distinguishing different instances or components.
C.4 FURTHER BLABORATIONS IOU ADAPTION
Although the original SAM uses an individual token when multiple prompts are provided, we empirically observe that selecting the best mask from the remaining three masks based on the loU prediction yields better performance than the fourth mask, as shown in Fig. 5a. This is because although the three predictions converge, some details remain different and usually better than the fourth token. We provide visualizations in Fig. 14 to compare the masks generated by different tokens. Though the improvement may not be remarkable, the advantage IoU adaptation is that it doesn't require any additional annotated data and only takes advantage of the priors contained in the target dataset. SAMRefiner++ serves as a complementary enhancement to SAMRefiner when coarse masks on target datasets can provide high-quality guidance and this step is mandatory.
GT
Coarse Mask
token GT loU: 0.886
GT
Coarse Mask
4th token
GT loU: 0.927
1st token GT loU: 0,904 Pred loU: 0.979
2nd token GT loU: 0.929
token GT loU: 0.886
1st token loU: 0.888
2nd token GT loU: 0.927 Predi IoU: 1.0
3rd token GT loU: 0.969
Pred loU: 0.996
Pred IoU: 0.964
Pred IoU: 1.0
Figure 14: Visualization of masks generated by different tokens in SAM decoder.
C.5 IMPACT OF THE DISTANCE-GUIDED POINT SAMPLING STRATEGY
The distance-guided point sampling strategy outperforms the box-center method it effectively mitigates the impact of false false- positive noise, which often distorts the bounding box and causes the box center deviate from the actual object, shown in Fig. 15.
21

---

Published as a conference paper at ICLR 2025
Box Center Point
Distance- iustance-guided Point
Image
Coarse Mask
Figure 15: Comparison between box center point and distance-guided point.
C.6 ANALYSIS OF THE QUALITY OF COARSE MASKS
In Fig. 16, we provide visualizations of the refined masks based on coarse masks with varying levels of quality. The results show that SAMRefnner works effectively when the coarse masks meet certain quality standard but may fail when the coarse masks are extremely inaccurate. This because the mask refinement task becomes an ill-posed problem if the initial mask is too coarse. For example, if the coarse mask only covers person's 's head, reconstructing the entire person would be impossible without additional information due to the inherent ambiguity. Fortunately, most real-world coarse masks, such as those generated by model predictions, usually meet certain quality standard and can be effectively handled by our proposed approach.
Image
Coarse Mask
Refined Mask
Coarser
GT Mask
Figure 16: Visualizations of refined masks based coarse masks with varying levels of quality.
D DISCUSSIONS AND LIMITATIONS
D.1 COMPARISON OF DIFFERENT MASK REFINEMENT METHODS.
We provide detailed discussion of the differences between SAMRefiner and related methods (dense CRF, Cas CRM) in terms of the design principle, architecture, training data, advantages, and drawbacks in Tab. II. Among these methods, dense CRF is training-free post-process approach based on low-level color characteristics, making it efficient and easy to use. However, it struggles in complex scenarios due to its lack of high-level semantic context. CascadePSP and CRM, on the other hand, focus on aligning the feature map with the refinement target using CNN-based architectures. They trained combined dataset with extremely accurate mask annotations and demonstrate
22

---

Published as a conference paper at ICLR 2025
Table 1I: Comparison of different mask refinement methods.
Design Principle
Archilecture
Traiiin Data
Advantages Training-fret, Easy to use
Drawbacks insocumtt
dense
Matimiae lahel agreement
None
between pizels with
sinelar low-level coler
CSscddePSP
Align the foorare map
CNN
MSRAA TOK. DUT-OMRON,
Clas- Agnes-Agoshicc Accurste on semantic segmentazion
Innkedependem Irofficient
with the refinement tirget in cascode fashion
ECSSD.
FSS-1000
CRM
Align the feorare with the
CNN
MSSKA- TOK,
Cl Agnostic. segmentation
Task-dependent Ineflicient
turget
DURON
contauoasly
ECSSD:
Design noise-folerane pcomipts couble SAM
Transfomerr
lass-agnoshi
Task-agnonti, Accurate Bificient
intricate architecture
refinement
strong performance on semantic segmentation tasks. Nevertheless, their performance on instance segmentation is less competitive, primarily due the absence of complex cases in their training data and the inherent limitations of CNNs. Additionally, the cascade structure of CascadePSP and the multi-resolution inference required by CRM make them inefficient when handling masks with large number of objects.
In contrast, SAMRefiner leverages the strengths of SAM by designing noise-tolerant prompts specifi- cally for mask refinement tasks. This approach achieves better accuracy and efficiency compared to existing methods. Nevertheless, may underperform for objects with intricate structures, limitation inherited from SAM itself. This issue can be addressed using enhanced variants, such as HQ-SAM, as the experiments conducted Appendix B.3.
D.2 THE TARGET AND RELEVANCE OF SAMREFINER.
SAMRefiner is designed to be a universal framework for correcting coarse pseudo masks generated by various sources. This task is significant due to the wide source of coarse masks in practical scenarios, such as model predictions and even inaccurate human annotations. Our SAMRefiner can be treated as an effective proseproesssing method to improve data quality and the refined masks can then be used to train advanced models for specific tasks, which we denote as the pseudo-labeling paradigm. Although some recent works attempt construct large foundation models to achieve open-vocabulary capability and show impressive performance, we argue that the pseudo-labeling paradigm still remains meaningful for certain application scenarios. First, people usually focus on limited semantic categories in specific practical use (e.g, automatic driving). The opea-vocabyyary setup is unnecessary and sometimes even detrimental to the performance of focused objects. Second, the foundation models tend to be large and onference-inefficienn, which is not suitable for resource- limited and time-sensitive settings. Therefore, is more efficient to customize specific models for different application scenarios instead of directly using large foundation models. Our framework treats the foundation model as a separate post-processor to improve the data quality of customized models, which is more generic and flexible. It can complement various segmentation methods and has the potential to be complementary to other refinement techniques and foundation models.
D.3 APPLICATION SCENARIOS AND LIMITATIONS OF SAMREFINER(++).
Application scenarios. In this paper, we propose an effective mask refinement method SAMRefiner and its variant SAMRefiner++. SAMRefiner is a training-free method that refines masks using noise-tolerant prompts. It retains most of the characteristics of the original SAM and inherits its "universal capability." " In contrast, SAMRefinne+++ refer to the combination of SAMRefiner and IoU Adaption, which require additional training on target datasets. This method is specifically tailored for certain conditions and has strict prerequisites, such as the quality of coarse masks, which dataset-dependent and may not achieve remarkable results on all datasets. As a result, SAMRefiner++ is not intended to be a universal method. Instead, it offers a potential approach to achieve further improvements without requiring additional annotations.
Limitations, For the mask refinement task, the final performance is highly affected by the quality of the initial coarse masks. The defects in masks diverse, making it challenging to design single effective method that applies to all scenarios. Our prompt excavation strategy proposes diverse prompt types mitigate the effect of defects in the coarse masks. Although more noise-robust than
23

---

Published as a conference paper at ICLR 2025
Coarse Mask
SAMRefiner
GT
Figure 17: Failure cases semantic segmentation.
previous works, it still fails to work when the initial masks are extremely noisy, as is shown in the last few rows in Fig. 18. Besides, there may exist semantic obscurity between SAM predictions and our subjectivity (e.g, whether the category table should contain the items on the table in Fig. 17), which inevitable due the lack of downstream data and a potential solution dataset-specific adaption. Finally, SAM struggles to process multiple objects in semantic segmentation due to the absence of this condition during pre- pre-training. Although our proposed STM can partly mitigate this, sometimes fails to work (second row in Fig. 17). An option consider is to finetune SAM to make it adapted to this setting.
Coarse Mask
CascadePSP
CRM
SAMRefiner
GT
Figure 18: More visualizations on VOC. The last three rows show some failure cases.
24

---

Published as a conference paper at ICLR 2025
E MORE QUALITATIVE RESULTS
In Fig. 18 and Fig. 19, we provide more qualitative results our refined masks and previous works on PASCAL VOC 2012 and COCO 2017. We can observe that our SAMRefiner produces satisfactory segmentation results on boundaries and detailed structures, It is effective in both simple and complex scenes. The failure cases mainly stem from the extreme inaccuracy of coarse masks, resulting false activation or missed detection.
Coarse Mask
CascadePSP
CRM
SAMRefiner
GT
Figure 19: More visualizations COCO. The last rows show some failure cases.
25


Reviews:
[{"review_id": "kWCbPsAIj5", "reviewer": "Reviewer_NC8S", "summary": "The paper proposes SAMRefiner, a novel framework for mask refinement using the Segment Anything Model (SAM). It contains several techniques: noise-tolerant prompting scheme, split-then-merge strategy, and IoU adaption. By combining these techniques, the proposed SAMRefiner improves the quality of the SAM-generated masks.", "questions": "See \"Weakness\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed SAMRefiner offers a unique solution to the mask refinement problem, which is an important yet under-explored area. By adapting SAM for this task, the authors address a practical issue of improving the quality of pre-existing coarse masks, which can have significant implications for reducing annotation costs and enhancing the performance of downstream segmentation models.\n\n2. The introduction of the noise-tolerant prompting scheme, including the multi-prompt excavation strategy, is interesting. \n\n3. The split-then-merge (STM) is a practical solution to handle the challenges of multiple objects in the mask. \n\n4. The IoU adaption step is another interesting technique. By introducing a self-boosted method to enhance SAM's IoU prediction ability on specific datasets using coarse mask priors, the authors demonstrate a way to further improve the performance of the framework without additional annotation.\n\n5. The authors conduct a comprehensive set of experiments on various benchmarks, including DAVIS-585, COCO, and PASCAL VOC, under different settings (such as instance and semantic segmentation with incomplete supervision). This wide range of evaluations helps to demonstrate the versatility and effectiveness of SAMRefiner in different scenarios.\n\n6. The comparison with state-of-the-art model-agnostic refinement methods is thorough. The detailed performance metrics presented (e.g., IoU, mask AP, mIoU) and the analysis of the results provide strong evidence of the superiority of SAMRefiner in many cases, especially in terms of its robustness to mask noise and its ability to improve performance across diverse datasets.", "weaknesses": "1. The explanation of how the proposed method overcomes the limitations of SAM in handling multi-object cases, especially in the context of semantic segmentation, could be more detailed. While the STM pipeline is introduced, its interaction with the overall framework and the specific advantages it brings compared to other possible approaches are not entirely clear. In addition, I suggest the authors to use a flowchart and visualize some examples after each stage in STM.\n\n2. Although the paper compares with several state-of-the-art refinement methods, it could provide a more in-depth analysis of why some methods perform better or worse in different scenarios. For example, a more detailed discussion of the differences between SAMRefiner and methods like CascadePSP and CRM in terms of their design principles and how these differences lead to performance variations would be beneficial. It is better to compare the key design principles of SAMRefiner with others in a table or a figure.\n\n3. Some implementation details are missing or could be further elaborated. For instance, the specific hyperparameters used in different experiments (beyond the ones mentioned for the IoU adaption step) are not always clearly stated. This makes it difficult for other researchers to fully reproduce the experiments and evaluate the method under different conditions. The authors are suggested to include these implementations and put them in the appendix.\n\n4. The paper would benefit from more visualizations to support the understanding of the proposed method and the experimental results. For example, visual comparisons of the mask refinement process using different prompts and techniques could provide a better understanding of how the method works and why it is effective. Additionally, visualizations of the failure cases and their analysis could help to identify the limitations of the method more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LC3W4aZHn", "reviewer": "Reviewer_XLPd", "summary": "This paper aims to adapt the segment anything model for the mask refinement task. It aims to go beyond the previous mask refinement networks that are 1) model-dependent, 2) task-specific, and 3) category-limited. It also aims to improve the time efficiency of the existing method. The key of the proposed method is prompting SAM through prompt excavation design and IoU adaptation.  To this end, the proposed method has been validated over DAVIS-585 (mask refinement), COCO (instance segmentation), and VOC (semantic segmentation).", "questions": "See the weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Given that inaccurate pseudo-labels and coarse human annotations are widely used to train the deep segmentation model, mask refinement is a meaningful task to improve the training data and inference performance. Specialized deep learning mask refinement methods and non-learning-based methods have been proposed previously. However, they either fail to generalize to more diverse tasks or are limited in being adaptive to different samples. The proposed method, SAMRefiner,  aims to adapt SAM, a segmentation foundation model for different masks, which is of practical meaning. \n\nThe key contribution of the proposed method is to design noise-tolerant prompts based on the coarse mask. Since SAM is trained in a specific way of prompt combination. The author designed several strategies to adapt the SAM for the mask refinement task by combining point, box, and mask prompt. \n\nBesides, IoU adaptation is used to adapt SAMRefiner for tasks and dataset-specific applications.", "weaknesses": "I have concerns about the novelty and the experiment:\n\n1. Point Prompts Sampling Method: The distance-based sampling method used is not original, as it is a widely adopted approach for evaluating interactive segmentation models. The author suggests sampling one positive and one negative point, but there is no ablation study to support this design choice (e.g. why not other numbers). Different samples or tasks may have varying characteristics, so the author should provide analysis or empirical ablation to justify this approach.\n\n2. Box Prompt Generation Method: Similar concerns apply to the box prompt generation method. A simple baseline would be to take the bounding box over the coarse masks. The appropriateness of the proposed method heavily depends on the quality and characteristics of the coarse mask. While the author provides an ablation study using coarse masks generated by PointWSSIS, this is specific to the dataset and method. The author claims the proposed method is general, but additional analysis or empirical observations are needed to support this claim. The authors are suggested to evaluate the box prompt generation method on a wider range of datasets and coarse mask types to demonstrate its generality. Additionally, proposing a comparison against simpler baselines like the tight bounding box across various datasets would more clearly show the advantages of the proposed approach.\n\n\n3. Effectiveness of the Gaussian-Style Mask: I question the effectiveness of the proposed Gaussian-style mask compared to directly using the coarse mask. Although there is an ablation study in Figure 7 of the Appendix, no experiments address the scenario where the coarse mask is used in the first iteration. Despite potential noise, the coarse mask may still be more informative than a Gaussian-style mask based solely on the distance to a central point, especially for objects with complex structures. The description of “Considering the inaccuracy of coarse mask” is vague, and the author should provide more context on the advantages of the proposed method.  The author is advised to provide more detailed analysis or examples of when and why the Gaussian-style mask is advantageous over the coarse mask.\n\n4. IoU Adaptation: The IoU adaptation strategy seems questionable. When multiple prompts are provided, SAM is trained to predict only one mask via the fourth output token, and the three masks corresponding to the first three output tokens are not utilized. The author proposes a multi-prompt strategy for SAM, but in this context, using the three masks from the single prompt-only scenario seems unnecessary. Additionally, the proposed IoU selection strategy does not show significant improvement (87.1 vs 86.9), as the three predictions converge. I would like the author to clarify why they use the three masks from the single-prompt scenario in their multi-prompt strategy. Additionally,  the author should also provide a more in-depth analysis of the benefits of their IoU adaptation approach given the small improvement observed. For instance, it would be beneficial to examine specific cases where it provides more significant gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DUEHoPJC90", "reviewer": "Reviewer_a711", "summary": "The paper introduces SAMRefiner for refining coarse masks using the SAM. SAMRefiner utilizes a prompting scheme that generates varied prompts to address defects in initial masks, making it robust across different segmentation tasks. The paper also presents an extended version to add an IoU adaptation step to improve performance without requiring extra annotations. Experiments show that the proposed method enhances mask accuracy and efficiency, offering potential for reducing annotation efforts in training segmentation models.", "questions": "Based on the concerns raised in the weaknesses section, please explain in detail the direct advantages of using SAMRefiner to enhance downstream segmentation model performance compared to the simple use of SAM, such as through distillation or pre-trained weights.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work introduces a novel method to enhance downstream segmentation models using SAM.\n2. The exploration of prompt design in SAM is thorough, with comprehensive ablation studies provided to validate the design choices.\n3. The IoU adaptation is interesting and provides new insights into SAM’s IoU prediction.\n4. Overall, the paper is well-written, with elaboration on the prompt design.", "weaknesses": "1. Although this paper presents technical novelty, it needs to address a fundamental issue. For instance, the comparisons in Tables 3 and 6 involve SAMRefiner using SAM trained on a larger dataset, which introduces fairness concerns. To address this, the core of the paper should focus on how to better utilize SAM to enhance segmentation models. Indeed, the use of SAM is not limited to the refinement of Coarse Masks to indirectly improve downstream segmentation models. Alternative methods, such as straightforward distillation or the use of pre-trained weights, could also be explored. The paper should demonstrate the advantages of the proposed method in utilizing SAM from perspectives of training costs and downstream model performance, to justify its practicality and address fairness concerns.\n\n2. The description of the IoU Adaptation section is somewhat confusing. It would benefit from using more precise symbols to differentiate between different types of IoU and to clearly describe the corresponding training strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hzUOSM3f5F", "reviewer": "Reviewer_ukvm", "summary": "This paper proposes a new method to refine coarse segmentation masks by utilizing the Segment Anything Model (SAM). The method simultaneously leverages point-, box-, and mask-based prompts to produce high-quality refined masks. Additionally, an IoU-based adaptation method is proposed to improve the performance of the IoU head. Experiments on multiple tasks and datasets demonstrate the effectiveness of the proposed method.", "questions": "Please see the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written and easy to follow. \n2. The proposed method seems to be reasonable. The new designs of the point-, box-, and mask-based prompts used in this paper are novel and interesting.\n3. The experiments are relatively comprehensive, demonstrating the effectiveness of the method across multiple tasks.", "weaknesses": "1. The performance of SAMRefiner++ is not reported in the experimental section, thus the effectiveness of the proposed IoU adaptation method cannot be evaluated.\n2. It would be interesting if the authors could analyze the robustness of the method. For example, can the proposed method still maintain good performance when the initial mask becomes coarser?\n3. As shown in Table 2(a), the proposed distance-guided point sampling method performs significantly better than the box center method. Therefore, it is suggested to provide a more detailed explanation of the advantages of the distance-guided method over the box center approach.\n4. This method involves a lot of hyperparameters, such as λ, μ,  ω, and γ. It is suggested to include an ablation study on these hyperparameters for a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)