Segment Anything without Supervision
Authors: Xudong Wang
Jingfeng Yang
Trevor Darrell
Venue: NeurIPS 2024 poster (2024)
Keywords: Unsupervised Segmentation
Segment Anything
Primary area: machine_vision
TL;DR: We present Unsupervised SAM (UnSAM), a segment anything model for interactive and automatic whole-image segmentation which does not require human annotations.

Abstract: The Segmentation Anything Model (SAM) requires labor-intensive data labeling. We present Unsupervised SAM (UnSAM) for promptable and automatic whole-image segmentation that does not require human annotations. UnSAM utilizes a divide-and-conquer strategy to “discover” the hierarchical structure of visual scenes. We first leverage top-down clustering methods to partition an unlabeled image into instance/semantic level segments. For all pixels within a segment, a bottom-up clustering method is employed to iteratively merge them into larger groups, thereby forming a hierarchical structure. These unsupervised multi-granular masks are then utilized to supervise model training. Evaluated across seven popular datasets, UnSAM achieves competitive results with the supervised counterpart SAM, and surpasses the previous state-of-the-art in unsupervised segmentation by 11% in terms of AR. Moreover, we show that supervised SAM can also benefit from our self-supervised labels. By integrating our unsupervised pseudo masks into SA-1B’s ground-truth masks and training UnSAM with only 1% of SA-1B, a lightly semi-supervised UnSAM can often segment entities overlooked by supervised SAM, exceeding SAM’s AR by over 6.7% and AP by 3.9% on SA-1B.

Segment Anything without Supervision
XuDong Wang
Jingfeng Yang
Trevor Darrell
UC Berkeley
code: http://gthhb..om//ffnttwwwaaaag/
Abstract
The Segmentation Anything Model (SAM) requires labor-intensive data labeling- We present Unsupervised SAM (UnSAM) for promptable and automatic whole- image segmentation that does not require human annotations. UnSAM utilizes a hivide-and-conquer strategy to "discover" the hierarchical structure of visual scenes. We first leverage top-down clustering methods to partition an unlabeled image into instance/semantic level segments. For all pixels within segment, bottom-up clustering method employed to iteratively merge them into larger groups, thereby forming hierarchical structure. These unsupervised multi-ge granular masks then utilized to supervise model training. Evaluated across seven popular datasets, UnSAM achieves competitive results with the supervised counterpart SAM, and surpasses the previous state-of-the-ar in unsupervised segmentation by 11% terms of AR. Moreover, self-supervised labels. By integrating our unsupervised pseudo masks into SA-1B ground-truth masks and training UnSAM with only 1% of SA-1B, lightly semi- supervised UnSAM can often segment entities overlooked by supervised SAM, exceeding SAM's AR by over 6.7% and AP by 3.9% SA-1B.
show that supervised SAM can also benefit from
Introduction
Trained massive unlabeled data using self superviid learning methods, Large Language Models (LLMs) [5, 34, 33, 46, 2, 19] in natural language processing have revolutionized our world and redefined human-computer interactions. In the domain of computer vision, the recent introduction of the Segment Anything Model (SAM) [21] has dramatically transformed the field with its exceptional ability to handle diverse image segmentation tasks. However, the need for comprehensive manual labeling of training data-over 20 minutes per image [2i]-limits SAM from following the scaling laws that benefit L.MSs [20]. As result, despite SA-IB [21] being the most extensive segmentation dataset available, contains only about 11 million images. Moreover, numan-annotate data often introduces significant biases based the annotators' perceptions of "what constitutes instance". which frequently leads to the oversight of small entities within the images.
This challenge raises crucial question addressed in this paper: Can "segment anything" without supervision? In response, we present UnSAM, an innovative unsupervised learning method capable of performing both interactive and whole-image segmentation without the need for supervision.
How can we achieve fine -maiind and multi granular segmentation masks comparable to those in SA-1B [21] without supervision? Insights from neuroscience suggest that the human visual system exploits the structure visual scenes by decomposing dynamic scenes into simpler parts or motions. This perception of hierarchically organized structures implies powerful "divide-and- conquer" strategy for parsing complex scenes [4, 27]. Drawing inspiration from this, we introduce a divide-and-con
approach designed to generate hierarchical image segmentation results directly from raw, unlabeled images. The divide-and-conquee approach is a crucial element UnSAM, enabling it to effectively parse and segment images at multiple levels of granularity.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prev. Unsup, SOTA UnSAM
Raw Image
SAM
Mosk
SAM
UnSAM-
0000
UnSAM
195
50
Erity
point prompt
PADO
20:
Misk All (h)
SAM
UnSAM
UnSAM+
UnSAM
COCO
50
Raw Image
Point s-bod (Chacie)
Figure 1: UaSAM significantly surpasses the performance of the previous SOTA methods in unsupervised seg- mentation, and delivers impressive whole image and promptable segmentation results, rivaling the performance the supervised SAM [21]. This comparative analysis features our unsupervised UnSAM, the supervised SAM, and an enhanced version, UnSAM+, across a variety of datasets. The top section displays raw images (row 1) alongside whole image segmentation outputs from UnSAM (row 3), and SAM (row 2). The bottom section highlights our promptable segmentation results using a point prompt (i.e. the star mark). The right panel quantitatively compares the performance across models, including metrics like Mask AR (%) and Point IoU.
Our ascudo-mask generation pipeline initiates with a top-down clustering approach (i.e the divide stage), to extract initial semantic and instance-level masks using Normalized Cuts-based method CutLER [39, 31]. Subsequently, UnSAM refines these masks using a bottom-up clustering method (i.e., the conquer stage): within each mask, iteratively merge semantically similar pixels into larger segments based on various similarity thresholds. The resulting masks at different thresholds in the conquer stage, along with the masks produced in the divide stage, create hierarchical structure. Technically, we can generate a vast range of granularities with minimal extra cost! Furthermore, UnSAM captures more subtle details that pose challenges for human annotators, significantly enriching the granularity and utility of unsupervised segmentation models.
Equipped with these sophisticated multi-granule granular pseudo masks "ground- truth"""t""" labels, UnSAM is adeptly trained to perform both interactive and automatic whole-image segmentation, demonstrating remarkable versatility across various segmentation scenarios. We have observed that our UnSAM model frequently identifies objects that SAM [21] overlooks, particularly types of objects or parts typically missed by ground-truth -rruth annotations of SA-1B [21], such human ears, animal tails, etc.
The capabilities of UnSAM are rigorously tested across seven major whole-entity and part segmenta- tion datasets, e.g., MSCOCO [24], LVIS [15], SA-1B B [21]. ADE [48], Entity [29], PartImageNet [16] and PACO [30]. As illustrated Fig. we demonstrate some noteworthy behaviors:
The performance gap between unsupervised segmentation models and SAM can be significantly reduced: By training on just 1 1% of SA-1B's unlabeled images with ResNet50 backbone, UnSAM not only advances the state-of-the-an
unsupervised segmentation by 10% but also achieves comparable performance with the labor-intensive, fully-supervised SAM.
The supervised SAM can also benefit from our self-supervised labels: integrating our unsuper- vised pseudo masks with SA-1B's ground-truth data and retraining UnSAM on this combined data enables UnSAME to outperform SAM's AR by over 6.7% and AP by 3.9%. We observed that UnSAM and UnSAM+ can often discover entities missed by SAM.

---

2 Related Works
2.1 Self-supervised Image Segmentation
Recent advances in unsupervised image segmentation [39. 28, 44, 6, 8, 41, 38, 42, 35, 12, 7, 37] have leveraged the emergent segmentation capabilities of self-supervised Vision Transformers (ViT) 18, 14, 17] to "discover" objects within images. Initial efforts, such as TokenCut [44) and LOST [32], have produced semantically meaningful pixel groupings for salient objects by utilizing the class-attention mechanism of self-supervised ViTs. As a representative work in the unsupervised segmentation domain, CutLER [39] introduced a cut-and-learn pipeline for unsupervised object de- tection and image segmentation. CutLER initially generates high-quality pseudo masks for multiple objects using MaskCut [39], followed by learning a detector these masks using a loss dropping strategy. Extending this approach, VideoCutLER [40] employs a cut-syytheeis--nde and-lerr strategy for segmenting and tracking multiple instances across video frames without supervision. Additionally, SOHES [6] introduced the global-local selff-xploration method to cluster image features from high to low cosine similarity, obtaining pseudo masks that cover multiple hierarchical levels.
In contrast, UnSAM introduces a divide-and-conquer pipeline that generates more pseudo masks per image at the same processing speed, but with enhanced quality and broader coverage across hierarchical levels. Furthermore, UnSAM captures more subtle details that pose challenges for human annotators, significantly enriching the granularity and utility of unsupervised segmentation models.
2.2 Promptable Image Segmentation
Tradition segmentation models have focused on predicting masks for all instances or semantic parts within single image simultaneously. Recently, however, models have begun to interact with users, generating segmentation masks based on user inputs such as points [21, 23, 47, 45, 11], text descriptions [26], or bounding boxes [21]. Moreover, some approaches now frame segmentation tasks within an in-context learning framework [43, 3], utilizing in-context examples to define distinct segmentation tasks. For example, the Segment Anything model [21] can produce masks in zero- shot manner based on different types of prompts. One limitation of SAM is that it only produces three class-agnostic masks. An extension, Semantic-SAM, [23], aims segment and recognize objects at multiple granularities through a multi-choice learning scheme, allowing each click point to produce masks at multiple levels along with their semantic labels. Nevertheless, both models supervised and rely on large-scale, auman-annotated data, which introduces issues of annotator bias and scalability limitations.
In contrast, our unsupervised UnSAM and lightly semi-supervised UnSAM+ model demonstrate superior performance in the promptable segmentation task, offering robust alternative to these fully-supervised approaches.
Preliminaries
3.1 Cut and Learn (CutLER) and MaskCut
CutLER [39] introduces a cut-and-learn pipeline precisely segment instances without supervision. The initial phase, known as the cut stage, uses a normalized cut-based method, MaskCut [39], to gener- ate high-quality instance masks given the patch-wise cosine similarity matrix W T where K₁ is "key" features of patch i in the last attention layer of unsupervised ViT. To extract multiple in- stance masks from a single image, MaskCut repeats this operation but adjusts by masking out patches
K.K,
(K. Ei MG,) (K, Ei MG)
from previously segmented instances in the affinity matrix: i
|
Subsequently, CutLER's learning stage trains a segmentation/ddteetion model on these pseudo-masks with drop-loss. Please check Appendix A.2 for more details on CutLER.
3.2 Segment Anything Model (SAM) and SA-1B
Segment Anything [21] tackles the promptable segmentation task. At its core lies the Segment Anything Model (SAM), which is capable of producing segmentation masks given user-provided points, boxes, and masks in zero-shot manner. One significant contribution of SAM is the release of
3

---

Iterative Merging
Iterative Merging
SSL
Model
corse grained
fine grained
Top-down Clustering (Divide Stage)
Bottom-up Clustering
(Conquer Stage)
Figure Our divide-and- conquer pipeline for generating the "ground-truue truth" pscudo masks used for training UnSAM without human supervision begins with top-down clustering approach (i.e., the divide stage), to extract initial semantic/imstancelleww masks using Normalized Cuts [31 3hased CutLER [39]. Subsequently, we refine these masks using bottom-up clustering method (i.e., the conquer stage): within each mask, we iteratively merge semantically similar pixels into larger segments using various similarity thresholds. The resulting masks different thresholds create a hierarchy. zoom-in selected regions to visualize details.
the SA-1B dataset [21], which comprises 11M high-resolution images and 1.1 billion segmentation masks, providing a substantial resource for training and evaluating segmentation models. While SAM significantly accelerates the labeling of segmentation masks, annotating an image still requires approximately 14 seconds per mask. Given that each image contains over 100 masks, this equates to more than 30 minutes per image, posing substantial cost and making it challenging to scale up the training data effectively. For more details on SAM and SA-IB, please check Appendix A.3.
4 UnSAM: Segment Anything without Supervision
4.1 Divide-and-Conquee for Hierarchical Image Segmentation
Our segment anything without supervision model starts by generating pseudo masks that respect the hierarchical structure visual scenes without supervision. This approach is motivated by the observation that the "divide and conquer" strategy is fundamental organizational principle employed by the human visual system to efficiently process and analyze the vast complexity of visual information present in natural scenes [4, 27]. Our pseudo-mask generation pipeline divide-and- conquer, which is summarized in Alg. 1 and illustrated in Fig. 2, consists of two stages:
Divide stage: leverage Normalized Cuts -NCuts)-based method, CutLER [39, 31], to obtain semantic and instance-level masks from unlabeled raw images. CutLER's cut-and-learn pipeline and its MaskCut method are discussed in Sec. 3.1. However, the Searser-granulaiig masks predicted by CutLER can be noisy. To mitigate this, we filter out masks with a confidence score below a threshold т. Empirically, salient semantic and instance-level entities typically encompass richer part-level entities (for example, a person has identifiable parts such as legs, arms, and head, whereas background sky contains few hierarchical structure, we employ a conquer phase.
sub-level entities). To extract these part-level entities with
Conquer stage: for each instancee/semantic-lvee mask discovered in the previous stage, we em- ploy iterative merging [l, 6] to decompose the coarse- maasee-rrined mask into simpler parts, forming a hierarchical structure.
More specifically, we first crop local patches using the masks we obtained in the divide phase, and bi-linearly interpolate local patches to the resolution of 256 x 256. We then feed them into DINO pre-trained VIT-B/8 [8] encoder f()), and extract "key" features k; = f(x)) from the last attention layer as patch-wise features for local patches ps. Subsequently, the conquer phase employs iterative merging [1, 6] to group patches into larger clusters, with pre-defined cosine similarity thresholds at 0 E {01..., 0r), where I is the predefined granularity levels.

---

Algorithm Divide and Conquer
Tresived < input image resized to 1024 x 1024
M
{m : E Curs.ER(Treizee)) A Miccrre r}
for m E M do
Add m into So
bbox - bounding box [T1, V1, Z2. y2l of m Loval 6 Lesized cropped by bboz, resized 256 256
K < (INOO(
for 02 E El......6
01 do
if = then
Initialize ki + K. P: Vi, (, f- -1
where pi is corresponding patch of K;, add pi into S Vi
clse
Initialize S. + Si+1, Kf + k C O+ vi
end if
while 0, do
Identify adjacent Pa, Pj with i.j + argmax
[ a all a < max
Identify cluster C Ch where Pr C Pj Ch
Remove C and C from St
C C U.U. C add O into St
am ki tanks where am is the size of cluster C₁ and pt E C
Vpz E C", kgg
den +an
end while
end for end for
In iteration f, our method finds two adjacent patches (pt,P)) from two separate clusters (Ct, Ch)) A5M;
with the highest cosine similarity ||*]iaa||AÍll merges them into one cluster, and updates kς and kjj to em emk(+enks tm. where am is the number of patches in cluster C (p; E C.t.). The conquer stage repeats this step until the maximum cosine similarity is less than 0,, collects all merged clusters as new part-level pseudo masks, and uses smaller threshold 0.41 iterate again. Each coarse grained mask discovered in the divide stage can form hierarchical structure H after the conquer stage:
H (So, .i..... S., ...... Si), where S, (Ct,..., Ctt, , n S my if <
(1)
n4 is the number of clusters/masks belonging to granularity level I and
Mask merging: The new part-level pseudo masks discovered in the conquer stage are added back the semantic and instance-level masks identified in the divide stage. We then use Non-Maximum Suppression (NMS) eliminate duplicates. Following previous works in unsupervised image segmentation [39. 28, 6], we also employ off-the- the-shelf mask refinement methods, such as Conditional Random Fields (CRF) [22] and CascadePSP [10], to further refine the edges of the pseudo masks. Finally, we filter out the post-processed masks that exhibit significant differences in Intersection-over- Union (IoU) before and after refinement.
Preliminary results: The divide-and-conquer pipeline achieves a pseudo mask pool with more entities, a broader range of granularity levels, and superior quality compared to previous work, e.g., CutLER [39], U2Seg [28] and SOHES [6]. As shown in Table 3, its pseudo masks reach 23.9% AR on 1000 randomly selected validation images from the SA-1B dataset [21], representing 45.7% improvement over the state-of-thh-an
Key distinctions employed by UnSAM sets it apart from previous works:
prior works pseudo-mask generation: The divide-and-conguee strategy
[39, 28] rely solely on top-down clustering methods, providing only instance and semantic-level masks, and thereby missing the hierarchical structure present in complex images. In contrast, our pipeline captures this hierarchical structure by identifying more fine-grained pixel clusters.
5

---

While [6] does incorporate some hierarchical structure through bottom- up clustering with iterative merging, it still misses many fine-grained instances and some large-scale instance masks. Additionally, the iterative merging in [6] focuses on small regions below a certain mask size threshold, primarily to refine noisy small masks, limiting its ability detect a full range of entity sizes. Our experimental results demonstrate qualitatively and quantitatively superior performance compared to prior works, particularly in producing high-quality, detailed pseudo-masks that better capture the hicrarchical complexity of visual scenes.
4.2 Model Learning and Teff-Training
Although the pseudo masks generated by our pipeline are qualitatively and quantitatively superior to those from prior works, they can still be somewhat noisy. Our self-supervised pipeline has limitations in identifying certain types of instances. For example, iterative merging sometimes fails to correctly associate disconnected parts of the same entity. To address this, we utilize a self-training strategy to further enhance UnSAM's model performance. UnSAM learns image segmentation model strategy. It has been observed that self-training
using the masks discovered by the divide-and-congue enables the model to "clean" the pseudo masks and predict masks of higher quality [39]. Once we have prepared the pseudo-masks, UnSAM can be integrated with any arbitrary whole-image or promptable image segmentation models during the model learning or self-training stage.
Whole-image segmentation. We choose the vanilla Masked Attention Mask Transformer (Mask2Former) [9] for simplicity. The key innovation of Mask2Former is the introduction of masked attention mechanism in the transformer's 's cross-attention block, defined as softmax (M + QKTVV, where the attention mask M at feature location (x. is given by: Mzr
(0 if (zx This mechanism constrains attention within the region of the predicted -00 otherwise
mask. UnSAM is then trained using the following mask prediction loss:
L Ace + Adicehuice
where Lee and Ldice is the cross-entropy and Dice loss, with Ace and Adice as their respective weights.
After one round of self-training UnSAM on the pseudo-masks, we perform second round of self- training by merging high-confidence mask predictions (with a confidence score greater than Tsetf-rriin) as the new 'ground-tuuth' annotations. To avoid duplication, filter out ground truth masks that have an IoU greater than 0.5 with the predicted masks.
Promptable Image Segmentation. Similar to SAM [21], our unsupervised SAM can also produce high-quality object masks from input prompts such as points. We utilize Semantic-SAM [23] as the base model for predicting multiple granularity levels of masks from single click. During the learning process, we randomly sample points within an inner circle (radius S 6.1.min(M mainkaskwiahh Maskneight)) of the mask to simulate user clicks.
4.3 UnSAM+: Improving Supervised SAM with Unsupervised Segmentation
The supervised SAM model's [21] reliance Human-annotalee data introduces significant bias based on the annotator's perception of 'what constitutes an instance', frequently missing some entities within the image. In contrast, since mask generation pipeline does not rely on human supervision, it can often identify valid objects or parts that are overlooked by SA-1B's (21) ground- truth annotations.
Motivated by this observation, we leverage UnSAM to improve the performance of the supervised SAM [21] by implementing a straightforward yet effective strategy: merging SA-IB's ground-truth masks DSA-I with our unsupervised segmentation masks DUSSAM based on the IoU, formulated as:
DiNSAM+ = Di U (VCm E DÚNSAM if IoU muax (Cm, YC,, E D'A--B) S T
(3)
TUSSAM+ is the IoU threshold, ma is the maximum IoU between Cm and any mask C₂ in DiA-IB- and D' DIA-IB and Dj DU is the set of SA-1B and unsupervised masks within image i, respectively. We then train UnSAM+ on DUSSAM+ for promptable image segmentation and whole-image segmenta- tion. The fusion approach leverages the strengths of both supervised and unsupervised annotations, addressing the limitations inherent in human-annotate diversity and comprehensiveness of the training data. This results in more robust and generalizable segmentation model UnSAM+, surpassing the performance of SAM.
datasets while significantly enriching the

---

Rew images
SA-1B's Ground-truth
UnSAM's Unsupervised Labels
Figure 3: Unsupervised pseudo-masks generated by our divide-and-conquet pipeline only contain precise
masks for coarse- coarse-grained instances (column 5), e.g., cameras and persons, but also capture fine-grained parts (column 3), e.g., digits and icons on a tiny camera monitor that are missed by SA-IB's [21] ground truth labels.
Backbone
Datasets with Whole Entities
Datasets w/ Parts
Methods
Avg.
(# params)
images
COCO
LVIS
ADE Entity SA-1B
Ptln
PACO
SAM (supervised)
VIT-B/8 (85M)
11M
42.1
49.6
46.1
45.8
45.9
60.8
28.3
18.1
FreeSOLO [41]
TOT (45M)
1.3M
7.3
11.6
5.9
7.3
8.0
2.2
13.8
2.4
CutLER [39]
RN-50 (23M)
1.3M
21.8
28.1
20.2
26.3
23.1
17.0
28.7
8.9
SOHES [6]
VTT-B/8 (85M)
0.2M
30.1
30.5
29.1
31.1
33.5
33.3
36.0
17.1
UnSAM
RN-50 (23M)
0.1M
39.2
40.5
37.7
35.7
39.6
41,9
51.6
27.5
UnSAM
RN-50 (23M)
0.2M
40.4
41.2
39.7
36.8
40.3
43.6
52.1
29.1
UnSAM
RN-50 (23M)
0.4M
41.1
42.0
40.5
37.5
41.0
44.5
52.7
29.7
vs. prev. SOTA
+11.0
+11.5
+11.4
+6.4
+7.5
+11.2
+16.7
+12.6
Table I: UnSAM achieves the state-of-the-art results on unsupervised image segmentation, using a backbone of ResNet50 and training with only 1% of SA-IB [21] data. We perform a zero-shot evaluation on various image segmentation benchmarks, including whole entity datascts, egg., COCO and ADE, and part segmentation datascts, egg., PACO and PartImageNet. The evaluation metric is average recall (AR).
Experiments
5.1 Model Training Settings
We provide brief overview of the model training settings and include more details in Appendix A.
Pseudo mask generation. In the divide stage, we set the confidence threshold 7=0.3; in the conquer stage, we choose threshold Omerge [0.6, 0.5, 0.4, 0.3, 0.2, 0.1). When merging the pseudo masks with the ground truths for training UnSAM+ we select TUSSAM+ 0.02. Whole-image segmentation. UnSAM picks DINO [8] trained Re [18] as the backbone and Mask2former [9] as the mask decoder. The default learning rate is 5 x 10 with a batch size of 16 and a weight decay of 5 x 10-2 We train the model for epochs. Promptable segmentation. UnSAM uses the self-supervised pre-trained Swin-Trrasforme
[25] Tiny model as the backbone, and leverages Semantic-SAM [23] as the base model. We set the number of hierarchy levels to 6, which is also the number of predicted masks UnSAM generates per prompt during inference. One can easily train with a different number of granularity levels as needed. For all experiments, we train UnS AM with 1~4% unlabeled images from SA- IB dataset [21].
5.2 Evaluation Datasets and Metrics
Whole-image segmentation. To evaluate our model's performance, we test our models on various datasets in a zero-shot manner to evaluate the performance of segmenting entities from all granularity levels. We choose COCO [24], LVIS [15], ADE20K [48], EntitySeg [29]. and SA-1B [21] that mainly encompass rmmantic-iinstacce entities. The SA-IB test set consists of randomly selected 1000 images not included in our training set. Notably, each dataset only covers entities from certain hierarchical levels and certain pre-defined classes, while our model generates masks from all levels and classes. Hence, the COCO Average Precision (AP) metric could not reflect our model's 's authentic performance in segmenting all entities in the open-world. Following prior work [39, 6], we mainly consider Average Recall (AR) to compare with different models.
entities; PartImageNet [16] and PACO [30] that cover leve-
7

---

Raw Image
SAM
UnSAM
UnSAM+
Figure UnSAM has competitive dense object segmentation results compared to the supervised SAM [21].
Backbone
Sup. Unsup. #
Datasets with Whole Entities
Datasets w/ Parts
Methods
Avg.
(# params)
Labels Labels images
COCO
LVIS ADE Entity SA-1B
Pila
PACO
SAM
VIT-8/8 (85M)
TTM
42.1
49.6
46.1
45.8
45.9
60.8
28.3
UnSAM RN-50 (23M)
X
/
0.1M
39.2
40.5
37.7
35.7
39.6
41.9
51.6
27.5
UnSAM+ RN-50 (23M)
7
/
0.1M
48.8
52.2
50.8
45.3
49,8
64.8
46.0
32.3
SAM
+6.7
+2.6
+4.7
-0.5
+3,9
+4.0
+17.7
+14.2
Table 2: UnSAM+ can outperform SAM [21] on most experimented benchmarks (including SA-1B [21), when training UnSAM on 1% of SA-1B with both ground truth masks and our unsupervised labels. This demonstrates that our unsupervised pseudo masks can serve as a powerful add-on to the densely annotated SA-1B masks!
Methods
AR00
ARs
ARM
ARE.
Methods
AP
ARg
ARM
ARt
AR
SOHES (CRF [22])
12.0
9.5
20.7
SAM
38.9
20.0
59.9
82.8
60.8
SOHES (CascadePSP [10])
16.4
6.0
15.8
22.6
UaSAM+
42.8
36.2
65.9
76.5
64.8
UnSAM (CRF [22])
15.3
2.3
11.9
27.7
vs. sup. SAM
+3,9 16.2 +6.0 -6.3
+4.0
UnSAM (CascadePSP [10))
23.9
7.9
22.4
34.0
vs. prev. SOTA
+7.5
+1.9
+6.6
+11.4
Table 4: Quantitative comparisons between our
Table 3: Evaluation on unsupervised pscudo masks using
lightly semi-supervised SAM, UnSAM+, and
SA- AB's [21] ground-trute truth annotations.
the fully-supervised SAM [21] on SA-1B [21].
Point-based promptable segmentation. We evaluate our point-based interactive segmentation model on MSCOCO 2017 [24]. Following the previous work on promptable image segmentation [21, 23], we pick two metries for model evaluation MaxIoU and OracleloU. For each point prompt, UnSAM predicts 6 masks representing different granularity levels. MaxIoU calculates the IoU between the mask with the highest confidence score among masks, whereas OracleloU picks the highest IoU between 6 predicted masks and the ground truth. For each mask in test image, we select its center as the point prompt.
5.3 Evaluation Results
Unsupervised pseudo-masks. Unsupervised pseudo-masks generated by our divide-and-conquer pipeline not only contain precise masks for grasiee instances, but also capture fine-grained parts that are often missed by SA-1B's s [21] ground- labels, as shown in Fig. 3.
Whole-image segmentation. Remarkably, UnSAM outperforms the previous state-of-the-ee methods across all evaluation datasets as summarized in Table 1. UnSAM demonstrates superior performance compared to the SOTA method even when trained with only 1 1% SA- 1B training data and a backbone of 500 with only 23M parameters, while the SOTA utilizes twice training data and backbone with nearly four times the parameters. This implies that UnSAM is a lightweight, easier to train, and less data-hungry model with better zero-shot performance segmenting entities in the open-world as shown in Figs. 4 and 5. On average, UnSAM surpasses the previous SOTA by 11.0% in AR. When evaluated on PartImageNet [16] and PACO [30] benchmarks, UnSAM exceeds the SOTA by 16.6% and 12.6 %, respectively.
8

---

Raw Image
Prev. Unsup SOTA
UnSAM
Figure 5: UnSAM not only discovers more fine-grained masks than the previous state-of-the-art unsupervised segmentation method [6], but also provides segmentation masks with wide range of granularity. We show qualitative comparisons between UnSAM (with 3 levels of granularity) and baseline models on SA-1B [21].
SAM
UnSAM
UnSAM+
Figure 6: Qualitative comparisons of promptable image segmentation between the fully-supervised SAM [21]. unsupervised UnSAM, and the lightly semi-supervised UnSAM+. Both UnSAM and UnSAM+ consistently deliver high-quality, aulti-granular segmentation masks in response to the point prompts (i.e., the star mark).
Backbone
Sup. Labels
Unsup. Labels
Point (Max)
Point (Oracle)
Methods
% of SA-1B
(# params)
I-loU
1-loU
SAM (B)
ViT-B/8 (85M)
V
X
100%
52.1
68.2
UnSAM
Swin-Tiny (25M)
X
40.3
59.5
UnSAM
Swin-Tiny (25M)
1%
52.4
69.5
Table 5: Despite using backbone that is 3x smaller and being trained on only 1% of SA- IB, our lightly
superviid UnSAM+ surpasses the sully-sue SAM in promptable segmentation task COCO.
When compared to the supervised SAM [21], UnSAM's AR across all datasets is already very close, with only 1 %% difference. On PartImage [16] and PACO [30], UnSAM surpasses SAM by 24.4% and 11.6%. This further demonstrates the excellent capability of our divide-and -nde--u pipeline in discovering details that human annotators tend to miss.
Furthermore, our UnSAM+, trained with integrated unsupervised pseudo masks and SA-1B [21] ground truth, outperforms SAM's [21] AR by over 6.7% and AP by 3.9% as shown by Table 2 and 4. UnSAM+ demonstrates superior average recall compared to SAM across all evaluation datasets except for ADE20K [48], which is dominated by semantic-level annotations. UnSAM+'s significantly 16.2 % higher AR on small entities further confirms that our pscudo masks can effectively complement the SA-IB datasets with more details it ignores and the UnSAM+ often discover entities missed by SAM as demonstrated in Fig. 4 and Fig. 7.
bhnnt promptable segmentation. As shown in Table 5, UnSAM trained with our pseudo masks achieve 40.3% MaxIoU and 59.5% OracleloU on COCO. Notably, we train the model with only 1 % of the data that SAM [21] uses and backbone with 4x fewer parameters. Moreover, the UnSAM+ trained with integrated pseudo masks and SA-1B ground truths outperforms SAM on both MaxIoU and OracleloU with 0.3% and 1.3% respectively. Qualitative results are shown in Fig. 6.

---

Figure More visualizations on SA-IB [21]. From top bottom are raw images, segmentation by SAM, segmentation by UnSAM, and segmentation by UnSAM+
6
Summary
Image segmentation a fundamental task in computer vision, traditionally relying on intensive human annotations achieve detailed understanding of visual scenes. We propose UnSAM, an unsupervised segmentation model that significantly surpasses the performance of previous state- of-the-art methods in unsupervised image segmentation. Additionally, our unsupervised UnSAM model delivers impressive results, rivaling the performance the cutting-edge supervised SAM, and exceeding it in certain semi-supervised settings.
10

---

Acknowledgement. We thank helpful discussions with Jitendra Malik, Cordelia Schmid, Ishan Misra, Xinlei Chen, Xingyi Zhou, Alireza Fathi, Renhao Wang. Stephanie Fu, Qianqian Wang, Baifeng Shi, Max Letian Fu, Tony Long Lian, Songwei Ge. Bowen Cheng and Rohit Girdhar. We thank Shengcao Cao and Hao Zhang for their help reproducing baseline results. XuDong Wang and Trevor Darrell were funded by Dol including DARPA LwLL and the Berkeley Al Research (BAIR) Commons.
References
[1] P. Arbelacz, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5)8898-116, 2010.
[2] J. Bai, S. Bai. Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[3] Y. Bai, X. Geng, K. Mangalam, A. Bar, A. Yuille, T. Darrell, J. Malik, and A. A. Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.0078: 2023.
[4] J. Bill, H. Pailian, S. J. Gershman, and J. Drugowitsch. Hicrarchical structure is employed by humans during visual motion perception. Proceedings of the National Academy of Sciences, 117(39)24581-24589) 2020.
[5] T. Brown, B. Mann. N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry. A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] S. Cao, J. Gu, J. Kuen, H. Tan, R. Zhang. H. Zhao, A. Nenkowa, L. Gui, T. Sun, and Y.-X. Wang. SOHES: Self-supervised open-world hierarchical entity segmentation. In The Twelfth International Conference on Learning Representations, 2024.
[7] S. Cao, D. Joshi, L. Gui, and Y.-X. Wang. HASSOD: Hierarchical adaptive self-supervised object detection. In NeurlPS, 2023.
[8] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers, In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[9] B. Cheng, I. Misra, G. Schwing, Kirillov, and Girdhar. Masked-attention mask transformer for universal image segmentation. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299, 2022.
[10] H. K. Cheng, Chung, Y.-W. Tai, C.-.. Tang. Cascadepsp: Toward class annsstagnstic and very high- resolution segmentation via global and local refinement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8890-8899, 2020.
[11] Cheng, J. Ye, Z. Deng, J. Chen, Li, H. Wang. Y. Su, Z. Huang. Chen, L. J. H. Sun, J. He, S. Zhang, M. Zhu, and Y. Qiao. Sam-med2d, 2023.
[12] Cho, Kwak, C. Schmid, and Ponce. Unsupervised object discovery and localization the wild: Part- Part-based matching with bottom-up region proposals, 2015.
[13] Deng, W. Dong. Socher, L.-J. Li, Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEF Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, Heigold, Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 20t0:22000.19299 2020.
[15] A. Gupta, Dollar, and R. Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3366-5364, 2019.
[16] He, S. Yang, S. Yang, A. Kortylewski, X. Yuan, J.NN. Chen, S. Liu, C. Yang, Q. Yu, and A. Yuille. Partimagenet: A large, high-quality dataset of parts. In European Conference on Computer Vision, pages 128-145. Springer, 2022.
[17] K. He, X. Chen, S. Xie, Y. Li. P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000- 16009, 2022.

---

[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[19] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D S. Chaplot, D. d. Casas, F. Bressand, G. Lengyel,
G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06925, 2023.
[20] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and
D. Amodci. Scaling laws for neural language models, arXiv preprint arXiv:2001.0880 2020.
[21] A. Kirillov, Mintun, N. Ravi, H. Mao, C. Rolland, L, Gustafson, Xiao, S. Whitchead, A, C. Berg. W.-Y P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02644 2023.
[22] J. Lafferty, A. McCallum, F. Pereira, et al. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Ieml, volume I, page 3. Williamstown, MA, 2001.
[23] F. Li, H. Zhang, P. Sun. X. Zou, S. Liu, J. Yang, C. Li, L. Zhang, and J. Gao. Semantic-sam: Segment and recognize anything any granularity. arXiv preprint (tt:::307.00767., 2023.
[24] T.-.. Lin, M. Maire, S. Belongic, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[25] Z. Liu, Y. Lin, Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021.
[26] T. Lüddecke and A. Eeker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086-7096, 2022.
[27] S. R. Mitroff, B. J. Scholl, and K. Wynn. Divide and conquer: How object files adapt when persisting object splits into two. Psychological Science, 15(6))420-425, 2004.
[28] D. Niu, X. Wang, X. Han, L. Lian, R. Herzig, and T. Darrell. Unsupervised universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[29] L. Qi. J. Kuen, Y. Wang. Gu, Zhao, P. Torr, Z. Lin, and J. Jia. Open world entity segmentation. JEEE Transactions on Pottern Analysis and Machine Intelligence, 2022.
[30] V. Ramanathan, A. Kalia, V. Petrovic, Y. Wen, B. Zheng, B. Guo, R. Wang. A. Marquez, R. Kovvuri, A. Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7141-7151, 2023.
[31] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-990 905, 2000.
[32] O. Siméoni, G. Puy, H. V. Vo, S. Roburin, S. Gidaris, A. Bursue, P. Pérez, R. Marlet, and J. Ponce. Localizing objects with self- supervied transformers and no labels. arXiv preprint NXX1:22099.12799 2021.
[33] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.BB Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
[34] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 133711 2023.
[35] W. Van Gansbeke, S. Vandenhende, and L. Van Gool. Discovering object masks with transformers for unsupervised semantic segmentation. arxiv preprint arxiv:2206.065 06366, 2022.
[36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, L.. Kaiser, and I. Polosukhin. Attention all you need, 2023.
[37] H. V. Vo, F. Bach. M. Cho, K. Han, Y. LeCun, P. Perez, and Ponce. Unsupervised image matching and object discovery as optimization, 2019.
[38] H. V. Vo, P. Pérez, and J. Ponce. Toward unsupervised, multi-object discovery in large-scale image collections. In Computer Msion-EECV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXIII 16, pages 779-795. Springer, 2020.
12

---

[39] X. Wang, R. Girdhar, S. X. Yu, and I. Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proceedingy of the IEEE/CVF conference computer vision and pattern recognition. pages 3124-3134, 2023.
[40] X. Wang. Misra, Z. Zeng, R. Girdhar, and T. Darrell. Videocutler: Surprisingly simple unsupervised video instance segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22755-22664 2024.
[41] X. Wang. S. Mello, J. Kautz, Anandkumar, C. Shen, and M. Alvarez. Freesolo: Learning segment objects without annotations. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages
2022.
[42] X. Wang. R. Zhang. T. Kong. L. Li. and C. Shen, Solov2: Dynamic and fast instance segmentation. Proc. Advances in Neural Information Processing Systems (NeurlPS), 2020.
[43] X. Wang. X. Zhang. Y. Cao, W. Wang. C. Shen, and Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304 032844444 2023.
[44] Y. Wang, X. Shen, Y. Yuan, Y. Du, M. Li, S. X. Hu, J. L. Crowley, and D. Vaufreydaz. Tokencut: Segmenting objects in images and videos with self supervised transformer and normalized cut. IEEE transactions pattern analysis and machine intelligence, 2023.
[45] Y. Xiong, B. Varadarajan, L. Wu, X. Xiang. F. Xiao, C. Zhu, X. Dai, D. Wang, F. Sun. F. landola. R. Krishnamoorthi, and V. Chandra. Efficientsam: Leveraged masked image pretraining for efficient segment anything, 2023.
[46] A. Young. Chen, C. Li, C. Huang, G. Zhang. G. Zhang. H. Li. J. Zhu, J. Chen, J. Chang, al. Yi: Open foundation models by 01. arXiv preprint arXiv:2403.044 040.2442.,. 2024.
[47] X. Zhao, W. Ding, Y. AD, Y. Du, T. Yu, M. Li, M. Tang, and J. Wang. Fast segment anything, 2023.
[48] B. Zhou. H. Zhao, X. Puig. T. Xiao, S. Fidler. A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321. 2019.
13

---

A Appendix
A.1 Training Details
Pseudo mask preparation details. Empirically, the divide stage, we set the confidence threshold = 0.3; in the conquer stage, we choose threshold merge = [0.6, 0.5, 0.4, 0.3, 0.2, 0.1). For each image, the divide-and-conquee pipeline generates on average 334 pseudo masks. In the self-training phase, the Tself-train 0.7, and each image has 448 pseudo masks per image after merging high- confidence mask predictions generated by UnSAM. When merging the pseudo masks with the ground truths for training UnSAM+, select TUSSAM+ 0.02.
Whole-image segmentation. UnSAM picks DINO [8] pre-trained ResNet-50 [18] as the backbone and Mask2former [9] as the mask decoder. Given the abundant number of pseudo masks generated, UnSAM augments data only by cropping a 1024 x 1024 region from the original image. To cope with large amount of 'ground-truth' masks per image, we find that having 2000 learnable queries produces the best result. We randomly select at most 200 'ground-truth' masks per image to speed up the training
process. The default learning rate is 5 x 10 with batch size equals 16 and weight decay 5 x 10 p We train the model for 8 epochs. All model training in this paper was conducted using either 4 A100 GPUs RTX 3090 GPUs.
Promptable segmentation. UnSAM uses the self-supervised pre-trained Swin- Wvin-Transformer [25], specifically the Swin-Ting model, as the backbone and leverages Semantic-SAN -SA [23] as the base model. Given at most 6 levels of masks corresponding to one input point SA-IB [21], we set the number of hierarchy levels to 6, which is also the number of predicted masks UnSAM generates per prompt during inference. However, one can easily train with a different number of granularity levels as needed. The default learning rate is x 10 with a batch size of 8. The learning rate decreases by a factor of 10 at 90% and 95% of the training iterations. We train the model for 4 epochs.
A.2 Preliminary: Cut and Learn (CutLER) and MaskCut
CuiLER [39] introduces a cut-and-learn pipeline to precisely segment instances without supervision. The initial phase, known as the cut stage, uses normalized cut-based method, MaskCut [39], to generate high-quality instance masks that serve as pscudo-labels for subsequent learning phases, MaskCut begins by harnessing semantic information extracted from "key" features K; of patch i in the last attention layer of unsupervised vision transformers. It then calculates patch-wise cosine similarity matrix Waj ,
K.K, To extract multiple instance masks from single image,
MaskCut initially applies Normalized Cuts [31], which identify the eigenvector x corresponding to the second smallest eigenvalue. The vector is then bi-partitioned to extract the foreground instance mask M Subsequent iterations repeat this operation but adjust by masking out patches K, Ei M;, (K, Eis Mú,)
from previously segmented instances in the affinity matrix: W
|| Subsequently, CutLER's learning stage trains a segmentation//eteetiin model with drop-loss, which encourages the model to explore areas not previously identified by MaskCut, An iterative self-training phase is employed for continuously refining the model's performance.
A.3 Preliminary: Segment Anything Model (SAM) and SA-1B
Inspired by achievement the NLP field, the Segment Anything project [21] introduces the novel promptable segmentation task. At its core lies the Segment Anything Model (SAM) [21], which is capable of producing segmentation masks given user- provided text, points, boxcs, and masks in zero-shot manner. SAM comprises three key components: an MAE [17] pre-trained Vision Transformer [14] that extracts image embeddings, the prompt encoders that embed various types of prompts, and lightweight Transformer [36] decoder that predicts segmentation masks by integrating image and prompt embeddings.
One significant contribution of SAM [21] is the release of the SA- 1B dataset, which comprises 11 million high- high-resolution images and 1. billion segmentation masks, providing substantial resource for training and evaluating segmentation models. In particular, annotators interactively used SAM to annotate images, and this newly annotated data was then utilized to iteratively update SAM. This cycle repeated multiple times to progressively enhance both the model and the dataset.
14

---

While SAM [21] significantly accelerates the labeling of segmentation masks, annotating an image still requires approximately 14 seconds per mask. Given that each image contains over 100 masks, this equates to more than 30 minutes per image, posing a substantial cost and making it challenging to scale up the training data effectively.
A.4 Evaluation Datasets
COCO (Common Objects in Context) [24] is widely utilized object detection and segmentation dataset. consists of 115,000 labeled training images, 5,000 labeled validation images, and more than 200,000 unlabeled images. Its object segmentation covers 80 categories and is mainly on the instance-level. We evaluate our model on COCO Va12017 with 5000 validation images without training or fine-tuning on any images from the COCO training set. The metrics we choose are class-agnostic COCO style averaged precision and averaged recall for the whole-image inference task, and MaxIoU and OracleloU for the promptable segmentation task.
SA-1B [21] consists of 11 million high-resolution (1500 on average) images and 1.1 billion segmen- tation masks, approximately 100 masks per image. All masks are collected in a class-agnostic manner with various subject themes including locations, objects, and scenes. Masks cover a wide range of granularity levels, from large-scale objects to fine aie-ggained details. In the whole-image inference task, we randomly selected 1000 SA-IB images that are not used to generate pseudo labels as the validation set.
LVIS (Large Vocabulary Instance Segmentation) [15] has 164,000 images with more than 1,200 categories and more than 2 million high-quality instance-level segmentation masks. It has a long tail distribution that naturally reveals large number of rare categories. In the whole-image inference task, we evaluate our model using its 5000 validation images in zero-shot manner.
EntitySeg [29] is an open-world, class-agnostic dataset that consists of 33277 images in total. There are on average 18.1 entities per image. More than 80% of its images are of high resolution with at least 1000 pixels for the width. EntitySeg also has more accurate boundary annotations. In the whole-image inference task, we evaluate our model with 1314 low-rcsolution version images (800 x 1300 on average) in zero-shot manner.
PACO (Parts and Attributes of Common Objects) [30] a detection dataset that provides 641,000 masks for part-level entities not included in traditional datasets. It covers 75 object categories and 456 object-part categories. In the whole-image inference task, we evaluate our model with 2410 validation images in a zero-shot manner.
PartImageNet [16] a large-scale, high-quality dataset with rich part segmentation annotations
general set of classes with non-rigid, articulated objects. It includes 158 classes and 24,000 images from ImageNet [13]. In the whole-image inference task, we evaluate our model with 2956 validation images in zero-shot manner.
ADE20K [48] is composed of 25,574 training and 2,000 testing images spanning 365 different scenes. mainly covers semantic-leww segmentation with 150 semantic categories and 707,868 objects from 3,688 categories. In the whole-image inference task, we evaluate our model with 2000 testing images in zero-shot manner.
A.5 More Visualizations
We provide more qualitative results of UnSAM and UnSAM+ in zero-shot manner in Figure Al, and Figure A2.
15

---

Figure A1: More visualizations COCO [24]. From top to bottom are raw images, segmentation by SAM, segmentation by UnSAM, and segmentation by UnSAM+
16

---

Figure A2: More visualizations on PACO [30]. From top to bottom raw images, segmentation by SAM. segmentation by UnSAM, and segmentation by UnSAM+
17

---

Figure A3: Failure cases of UnSAM. From left to right are raw images, segmentation by SAM, and segmentation by UnSAM.
A.6 Limitations
In images with very dense fine-grained details, UnSAM tends to miss repetitive instances with similar texture. As shown in Figure A3, in the first row, although UnSAM accurately segments the leaves in the center of the picture, it misses some leaves located at the top of the image. Additionally, UnSAM occasionally over-segment images. In the second row, the right sleeve cuff of the dancer has meaningless segmentation masks. This issue mainly arises because the unsupervised clustering method mistakenly considers some information, such as folds and shadows on clothing, as criteria for distinguishing different entities. In contrast, human annotators can use prior knowledge to inform the model that such information should not be valid criteria. In this regard, unsupervised methods still need to close the gap with supervised methods.
A.7 Ethical Considerations
We train UnSAM and UnSAM+ on ground truths of and pseudo masks generated on SA- IB [21]. SA-IB contains licensed images that are filtered for objectionable content. It is geographically diverse, but some regions and economic groups are underrepresennted. Downstream use of UnSAM and UnSAM+ may create their own potential biases.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The main claims made in our abstract and introduction precisely summarize this paper's contributions, assumptions, and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
We discuss limitations of our method in appendix A.6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All information needed to reproduce the main experimental results included in Appendix A.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will open-source the codes and models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ //nsss////ss.///////e cc/
public guides CooeeSuomiiiic wsionnolll. comooicc) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (ht (https: / /nips. cc/ cttppc/ppccccccccccccc.cccc....l i c// i des Code5 Suebubbn ss i onnol iiPolicc icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All basic settings of the pseudo mask preparation process and model training are included in Appendix A.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following established protocols from prior studies, report experimental results. We observed robustness in our results against the variability of random seeds used for initializing model weights. Additionally, our pseudo-mask generation process does not require retraining a parameterized model, thus ensuring deterministic results.
Guidelines:
The answer NA
that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide full information on the compute resources we use in Appendix A.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /tt////euue /neuripss
Answer: [Yes]
Justification:
The research conducted in this paper fully conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Broader impacts of our research are discussed in Appendix A.7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
23

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersw wiwttt sttccooeee com/ da tattssess tes has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper doesn't include crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper doesn't include crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "brx8hXlOVj", "reviewer": "Reviewer_S6W7", "summary": "This paper proposes an approach to generate masks from images in an unsupervised manner, which are then used to train segmentation models. The major distinction from previous works is the proposed divide-and-conquer strategy, which first adopts the CutLER to obtain coarse instance/segmentation masks and then uses iterative merging like the SOHES method to generate fine-grained masks. Experiments are conducted on different datasets with detailed analysis and advanced performance.", "questions": "Please refer to the weaknesses for the concerns. The reviewer would appreciate it if these concerns could be addressed after the rebuttal.", "limitations": "The authors discussed the limitations in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. Many technical details are provided. The reviewer believes the results should be easy to reproduce.\n- The two-stage approach that adopts both CutLER and SOHES to generate pseudo masks is reasonable.\n- The overall performance surpasses previous SOTA methods.", "weaknesses": "- The major concern is about technical novelty. Though effective, the proposed method technically adds a candidate extraction method before the previous method's iterative refinement stage. All these methods already exist in the unsupervised segmentation field. Therefore, although the performance improvement is reasonable, the reviewer thinks its technical novelty is limited as a NeurIPS research paper.\n- The comparison with SAM is based on different backbones. Meanwhile, some details, such as the number of masks (UnSAM's 6 vs. SAM's 3), are different. It may have a subtle influence on the performance comparison.\n- Some typos exist. For example, the $\\theta_{t+1}$ in line-147 should be $\\theta_{t-1}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer S6W7, we appreciate your invaluable insights and thoughtful comments! In the following sections, we address the questions you have raised: \n\n**[W1] Contributions and Insights** \n1. *Novel and Simple Pipeline:* We introduce a simple yet effective divide-and-conquer pipeline for producing high-quality pseudo-masks for unsupervised image segmentation. As no previous work has used this pipeline for unsupervised image segmentation, we believe that our overall pipeline is novel and adds new knowledge to the field. *We believe in simple science: we think a simple change leading to substantial performance gains is far more valuable and intriguing than a complex method yielding marginal gains.* \n2. UnSAM, for the first time, demonstrates that unsupervised image segmentation method can achieve competitive results with the supervised counterpart SAM. In addition, UnSAM achieves over 12% better AR than the previous unsupervised segmentation methods.\n3. UnSAM is also the first to show that the state-of-the-art supervised segmentation method, SAM, can benefit from our self-supervised labels—a discovery that has not been previously reported. UnSAM+ exceeds SAM’s AP by 3.9% and AR by over 6.7% on SA-1B.\n\n**[W2.1] Model Backbones**\n\nThe results for UnSAM reported in the paper were obtained using a significantly smaller backbone and fewer training samples than those used for SAM, placing our method at a disadvantage. To fully address your question, we conducted additional experiments using a larger backbone, ViT-Base, and have presented the comparative results on SA-1B below:\n\n| Method   | Backbone    | Training Data    | AP       | AR      |\n|:-------- |:--------:   | --------:        | --------:|--------:|\n| SAM      |  ViT-Base   |     11M SA-1B    | 38.9     | 60.8\n| UnSAM+   |  RN50       |     0.1M SA-1B   | **42.8**     | **64.8**\n| UnSAM+   |  ViT-Base   |     0.1M SA-1B   | **44.6**     | **67.2**\n\nUnSAM demonstrates superior performance with a larger backbone, outperforming SAM in terms of both Average Precision (AP) and Average Recall (AR), despite being trained on significantly fewer samples.\n\n**[W2.2] Number of Masks Per Point**\n\nGreat question! We employed additional granularity levels (i.e., more masks per click) because our unsupervised pseudo-labeling method can create a hierarchical structure with more granularity levels than the ground-truth masks from SA-1B. We increased the number of masks per click to fully utilize the advantages of our hierarchically structured data. Despite increasing the number of masks for SAM (two clicks producing six output masks), the performance improvement on MSCOCO was relatively marginal.\n\n| Method   | Backbone (# params)    | Training Data    | # Masks  | 1-IoU    |\n|:-------- |:--------:   | --------:        | --------:| --------:|\n| SAM      |  ViT-Base (85M)  |     11M SA-1B    | 3     | 68.2     | \n| SAM      |  ViT-Base (85M)  |     11M SA-1B    | 6     | 69.0     | \n| UnSAM+   |  Swin-Tiny (25M) |     0.1M SA-1B   | 6     | **69.5**     | \n| UnSAM+   |  Swin-Tiny (25M) |     0.4M SA-1B   | 6     | **70.4**     | \n\nAs indicated in the table, UnSAM+ surpasses SAM by over 1.4% despite being trained with 100 times fewer samples.\n\n**[W3] Typos**\n\nThank you for pointing out these typos! We will correct them and thoroughly review the manuscript before finalizing the paper.\n\n*Hope our explanation and experiments can address your inquiries. We will integrate all your valuable comments into our revision!*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vVE0qhM46W", "reviewer": "Reviewer_p86w", "summary": "The paper explores a new way to generate hierarchical pseudo-masks to train downstream segmentation models without human annotations. First, the image is segmented using CutLER. Then, within CutLER proposed masks (with cropping and resizing), DINO features are extracted, and patches are iteratively merged based on proximity in cosine distance. The masks are then refined and filtered using CRF/CascadePSP. The pool of pseudo-labels is used to train Mask2Former architecture or Semantic-SAM architecture, for whole-image or prompt-able segmentation tasks, respectively. Additional self-training is performed to improve results. The evaluation is carried out on several datasets using the average recall (AR) metric. UnSAM shows significant improvement over other models in terms of recall of generated masks and final outputs.", "questions": "The central questions to address in the rebuttal would be around the evaluation protocol. Some more explanation about the contribution of the paper would also be helpful.\n\n\nWhile the paper presents strong results, the appropriateness of the evaluation protocol is somewhat questionable. Furthermore, there are some questions around the difference of the proposed scheme to prior work, and lack of experiments and analysis to explain it. While a rebuttal can address these issues, I currently rate the paper as a Borderline Reject.", "limitations": "Limitation are appropriately addressed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1) The method presented in the paper shows improved results in unsupervised segmentation. Reducing the recall gap between supervised SAM and unsupervised methods.\n2) The additional masks generated using the unsupervised approach in combination with manual annotatons to enhance the performance further, surpassing that of SAM.", "weaknesses": "#### What is \"UnSAM\"? \nThe naming scheme adopted in the paper is slightly confusing. UnSAM refers to all:\n - the hierarchical pseudo-labelling scheme building on top of CutLER, \n - the distillation of such scheme to Mask2Former architecture,\n - distillation of such a scheme into semantic-SAM architecture. \n\nWhile it still is possible to parse the results in the tables based on the setting, the ability to understand and scrutinise the text is severely impacted. Note the distinction is important as the output of three of these appears to be different.\n\n#### Is AR a sensible metric?\nThe proposed model seems to generate an extremely large amount of masks (up to 2000, L407). While it seems that some useful masks are in this set, the cardinality of the output makes one wonder if it is useful. What is the number of masks used for assessing recall? Is it 1000? Cutler has a setting of 100-200 from published configs. Clearly, increasing the number of masks boosts recall. It is important that the evaluation maintains the same amount of masks for all methods. \n\nIt also raises a question whether some classical hierarchical segmentation methods such as MCG [A] or Arbelaez et al. [B] would get similar or close performance when such large pool of candidates is allowed.\n\n#### What is the effective difference between a proposal and SOHES?\nThe hierarchical merging scheme follows SOHES formulation, with the difference being the CutLER region \"proposal\" as a constraint. Is that all? It is important to highlight the differences here to both figure out the novelty and correctly attribute improvements in performance to different components/steps of the proposal. Additionally, it appears that distillation to Mask2Former abandons the hierarchical association. Does this mean that the output is no longer organised in a hierarchical manner?\n\n#### What makes the method work better than prior works?\nThere is some lack of ablations that explore the influence and sensitivity of various components in the pipeline. The paper only reports on the construction and technical details of the UnSAM methods and the evaluation of this approach. It lacks analysis and insights into what the makes the construction effective. The advancement of knowledge offered by the paper is somewhat limited. \n\n#### Are different two different models required?\nIt is not entirely clear why the two modes of operation, \"whole image\" and \"prompt-able\" require different models/architectures. Could the Semantic-SAM-based model be prompted in a similar fashion to SAM (i.e. a grid of points) to perform the whole image segmentation? Is this disadvantageos for the whole-image segmentation task?\n\n#### Inaccuracies \nFinally, it is important to note that CascadePSP relies on manual annotations. Thus, unSAM + CascadePSP is _slightly_ supervised. \n\nSince UnSAM partly distils CutLER outputs, would it not be more accurate to incorporate the #images used in CutLER to those in UnSAM, writing e.g. 1.4M in Table 1 instead of 0.1M?\n\n[Nit] L135: \"<...> we employ iterative _merging_ to _decompose_\" might require rephrasing as it is currently a slight oxymoron. \n\n---\n\n[A] Arbelaez et al. \"Multiscale combinatorial grouping\"\n\n[B] Arbelaez et al. \"Contour detection and hierarchical image segmentation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer p86w, thank you for your thoughtful comments. We will provide detailed responses to your questions below:\n\n**[W1] What is \"UnSAM\"?**\n\nNice suggestion. In the paper, UnSAM stands for **Un**supervised **S**egment **A**nything **M**odel. The pseudo-labeling strategy discussed in our paper is named as divide-and-conquer. I agree that we can make this clearer, and we plan to designate the terms UnSAM (pseudo), UnSAM (whole-image), and UnSAM (promptable) to distinctly refer to the pseudo-labeling method, whole-image segmentation model, and promptable image segmentation models, respectively. We would appreciate any suggestions you might have!\n\n**[W2] Why Using AR as One of The Main Evaluation Metrics?**\n\nExcellent question. We explained the rationale for using AR as a primary evaluation metric in lines 234-238, following previous works like CutLER (CVPR2023) and SOHES (ICLR2024). But **why is AR more appropriate than AP for unsupervised segmentation?** The main reason is that all human-labeled datasets, including SA-1B, only label a subset of objects, and AR doesn't penalize models for detecting objects not labeled in these datasets.\n\nBelow is a comparison of results on the MSCOCO dataset that further illustrates why AR is preferred:\n\n| Method   | Venue  |  Setting |Backbone | Training Data    | AP       | AR       |\n|:-------- |:--------   |:-------- | --------:        | --------:| --------:| --------:|\n| SAM      |  ICCV 2023   |  supervised     | ViT-Base   |     11M SA-1B    | 5.9      | 49.4     | \n| CutLER  |  CVPR 2023 |  unsupervised | RN-50       |     1.2M ImageNet    | 6.5      | 31.4     | \n| SOHES  |  ICLR 2024   |  unsupervised | ViT-Base   |     0.2M SA-1B   | 2.0      | 30.5     | \n| **UnSAM**   |  Ours      |  unsupervised | RN-50      |     0.4M SA-1B   | 3.6      | 42.0     | \n| **UnSAM+**   |  Ours    |  lightly supervised | RN-50      |     0.1M SA-1B   | **7.1**      | **52.3**     | \n\nDespite SAM's overall better performance, its lower AP compared to CutLER highlights that AP can unfairly penalize models for detecting unannotated objects in evaluation datasets, failing to truly reflect a model’s capabilities. I agree that AR is not without its imperfections, but it is a more suitable metric than AP for unsupervised segmentation. Future research on developing new metrics is needed.\n\n**[W3.1] The Proposed Model Seems to Generate an Extremely Large Amount of Masks.**\n\nSorry for the confusion—the 2000 queries mentioned in L407 refer to the number of learnable queries in Mask2Former, not the number of masks predicted. Before outputting the final model predictions, we apply non-maximum suppression, confidence score thresholding, etc. Consequently, the average number of masks our model predicts is around 500.\n\n**[W3.2] What is The Number of Masks Used for Assessing Recall? Why Using AR$_{1000}$?**\n\nWe assessed recall using AR$_{1000}$, following the approach used by SOHES. The maximum number of masks per image is set at 1000 for UnSAM and CutLER, and 32*32 (=1024) for SAM. We set the maximum number of masks as 1000 in CutLER's config. \n\n**[W4 W5] What's the Difference Between UnSAM and SOHES? Why UnSAM Performs Better Than Prior Works?**\n\nGood question! We have outlined the key differences between UnSAM and prior works (e.g. CutLER and SOHES) in lines 162-173. These distinctions are further detailed in the global rebuttal. Thank you for checking!\n\n**[W6] Are Different Two Different Models Required?**\n\nIt is indeed feasible to use the same model for both tasks. However, we choose to employ two distinct models primarily due to differences in inference time. While the Semantic-SAM-based model can be prompted similarly to SAM (using a grid of points) with only a minor performance disparity (< 2~3%), its processing speed is significantly slower—at least 5 times slower than a specialized whole-image segmentation framework like Mask2Former. In future research, we plan to utilize models such as FastSAM or SAM-2 to enhance the speed of interactive segmentation models.\n\nAdditionally, we can still effectively construct the hierarchical structure using the masks from Mask2Former by post-processing the results based on mask overlaps. \n\n**[W7] CascadePSP**\n\nGreat question! We follow previous works CutLER (CVPR 2023) and SOHES (ICLR 2024), and utilize CascadePSP (from SOHES) or CRF (from CutLER) for mask refinement (Table 3). We opted for CascadePSP as our default method primarily for two reasons:\n\n1) **Efficiency**: The primary advantage of CascadePSP is not actually in terms of model performance, but rather in terms of inference speed. In our local small scale experiments, we discovered that the performance difference (in terms of AR) between training UnSAM with CascadePSP-refined pseudo-masks and training UnSAM with CRF-refined masks is only about 2-3%, but the speed difference in refining pseudo-masks is about 5-10 times. Due to our limited computing resources, refining pseudo-masks using CRF would take approximately 3-4 months, making it unfeasible.\n\n2) **Consistency with Previous Works**: To maintain consistency with SOHES, we used the same refinement method.\n\nTo fully answer your question, how can we speed up the overall process for CRF-based mask refinement? One strategy could be to train a CascadePSP model using the \"ground-truth\" generated by CRF to achieve \"fully-unsupervised\" mask refinement quickly. \n\n**[W8] #images Should Include ImageNet**\n\nSince ImageNet was actually used by all methods for pre-training the backbone, including SAM, SOHES and CutLER, and UnSAM was never trained on pseudo-labels from ImageNet, we didn't include it as training data. We'll explain it in our paper. \n\n**[W9] Typos and Minors Issues** will be addressed in the revision! Thank you for pointing them out! \n\n*Hope our explanation and experiments can address your questions. We will integrate your valuable comments into our revision!*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ez7sUXvl9Y", "reviewer": "Reviewer_wivK", "summary": "The paper presents Unsupervised Segment Anything Model (UnSAM) for image segmentation whose training does not have access to human annotations. UnSAM employs a divide-and-conquer approach to hierarchically segment the image. In the divide stage, it uses CutLER [39] to obtain masks, and in the conquer stage, it applies an iterative merging method from SHOES [6]. The proposed method achieves SOTA results.", "questions": "1. The original CutLER uses ImageNet for unlabeled training, while UnSAM uses the SA-1B dataset. Since the proposed method and baseline use different training data, it is unclear if the proposed method is truly better. Similarly, for the comparison with SOHES on the 0.2M data, did both methods use the same set of 0.2M images?\n\n2. If different sets of 2% unlabeled training data from SA-1B are used, are the results stable? Are the threshold hyperparameters in UnSAM robust to variations of the selected training data? \n\n3. What does “the two extreme cases” in Line 169 refer to?", "limitations": "The paper lacks a limitation section. Including a discussion on failure cases would improve the paper.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This proposed approach achieves good image segmentation results.", "weaknesses": "1. Unclear technical contributions -- limited novelty: The paper uses existing work, including CutLER, SHOES, DINO and Mask2former. The key differences are not explained (well), beyond that these existing methods are put together for unsupervised image segmentation.\n\n2. The performance gain seems to come from additional thresholds used (Lines 130, 142) and potentially unfair comparisons (please see the point 3 and the Questions section below). The thresholds are set in an ad hoc manner.\n\n3. Backbone comparison: In Section 4 (UnSAM), Line 139, the paper mentions using the DINO pre-trained ViT-B/8 encoder, yet none of the tables show results with the ViT-B backbone. Recent methods (e.g., SOHES and SAM) use the ViT-B/8 backbone. A fair comparison with these methods should use the same backbone. Although one might argue that RN-50 and Swin-Tiny (used in this paper) are lighter backbones, using ViT-B/8 would provide a fair comparison.\n\n4. Clarity can be improved:\n   - 4.1. Inconsistent Figure Caption: In Figure 1, the caption and figure are not consistent. The caption states “UnSAM (row 2) and SAM (row 3)”, but the figure shows SAM in row 2 and UnSAM in row 3. It is unclear which result corresponds to which method.\n   - 4.2. Key Distinctions: The second part of “Key distinctions over prior works on pseudo-mask generation,” Line 167, is unclear. The paper summarizes SOHES [6] but does not explain novelty of the proposed work.\n\n5. Additional comparisons (optional): The paper could be improved by including another comparison with \"Unsupervised Universal Image Segmentation,\" CVPR 24 (published after the NeurIPS deadline, though).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer wivK, we appreciate your invaluable insights and thoughtful comments. In the following sections, we address the questions you have raised:\n\n**[W1.1] Technical Contributions**\n\nPlease check our answers in the global rebuttal. Thank you!\n\n**[W1.2] Differences with Prior Works**\n\nWe explained the key differences between UnSAM and prior works (e.g. CutLER and SOHES) in lines 162-173, and described the distinctions in the global rebuttal. Thank you! \n\n**[W2] Performance Gain Comes From Additional Thresholds Used?**\n\nSince SOHES didn't release its codes, models and data, we reproduced SOHES's results and reported the results of SOHES using exactly the same threshold settings as below:\n\n| Pseudo-Labeling   | Threshold Setting  |  AR       | AR$_S$       |  AR$_M$       |  AR$_L$       | \n|:-------- |:--------          | --------:  | --------: | --------: | --------: |\n| SOHES (reported)     |  Official   | 16.4  | 6.0 | 15.8 | 22.6 |\n| SOHES (reproduced)   |  Official   | 16.5  | 5.7 | 16.1 | 23.0 |\n| SOHES (reproduced)   |  Ours       | 16.8  | 6.1 | 16.2 | 22.9 |\n| UnSAM                        |  Ours       | **23.9** | **7.9** | **22.4** | **34.0** | \n   \nThe performance gain from using additional thresholds is marginal, at only about 0.3%. These results indicate that it is our divide-and-conquer strategy, not the threshold settings, that contributes to the significant performance improvements over SOHES.\n\n**[W3.1] UnSAM Uses a Smaller Backbone (RN-50 / Swin-Tiny) and May be Unfair**\n\nThe results for UnSAM reported in the paper were obtained using a significantly smaller backbone and fewer training samples than those used for SAM, placing our method at a disadvantage. To fully address your question, we conducted additional experiments using a larger backbone, ViT-Base, and have presented the comparative results on SA-1B below:\n\n| Method   | Backbone    | Training Data    | AP       | AR      |\n|:-------- |:--------:   | --------:        | --------:|--------:|\n| SAM      |  ViT-Base   |     11M SA-1B    | 38.9     | 60.8\n| UnSAM+   |  RN50       |     0.1M SA-1B   | 42.8     | 64.8\n| UnSAM+   |  ViT-Base   |     0.1M SA-1B   | **44.6**     | **67.2**\n\nUnSAM demonstrates superior performance with a larger backbone, outperforming SAM in terms of both Average Precision (AP) and Average Recall (AR), despite being trained on significantly fewer samples.\n\n**[W3.2] ViT-Base in Line 139** is a typo. It will be addressed in the revision, thank you for pointing it out!\n\n**[W4.1] Inconsistent Figure Caption**: Thank you for pointing this out! We will correct the figure caption accordingly. UnSAM's results are displayed in row 3, while SAM's results are in row 2.\n\n**[W4.1] Clarification on Key Distinctions with SOHES**\n\nCompared to SOHES, UnSAM has 1) better segmentation quality for coarse-grained instance/semantic-level masks. 2) better segmentation quality for fine-grained sub-part masks. For detailed explanations of these improvements, please refer to our responses in the global rebuttal. \n\n**[W5] Additional Comparisons with U2Seg**\n\nNice question! We have included the comparisons with U2Seg as below:\n\n| Methods   | Venue      |  Backbone |  MSCOCO     |   SA-1B  |    Part-ImageNet  | \n|:--------              |:--------:  | --------:   | --------:| --------:|  --------:| \n| CutLER                |  CVPR 2023 |  RN-50      |    28.1  | 17.0 | 28.7\n| U2Seg                 |  CVPR 2024 |  RN-50      |    27.5  | 19.3 | 29.1 \n| SOHES                 |  ICLR 2024 |  ViT-Base   |    30.5  | 33.3 | 36.0 \n| UnSAM             |  Ours      |  RN-50      |    **42.0**  | **44.5** | **52.7**\n\nUnSAM outperforms CutLER, U2Seg and SOHES by a large margin on all experimented benchmarks. \n\n**[Q1 Q2] Did UnSAM and SOHES Use the Same Set of 0.2M Images? If Different Sets of 2% Unlabeled Training Data From SA-1B Are Used, Are The Results Stable?***\n\nUnfortunately, SOHES has not released their codes, models, and training data, preventing us from using the identical set of 0.2M images. However, our results remain robust when training the model with different subsets of 0.2M images.\n\n| Methods        |  Seed |  MSCOCO     |   SA-1B  |    Part-ImageNet  | \n|:--------       | --------:   | --------:| --------:|  --------:| \n| Prev. SOTA     |  -    |    30.5  | 33.3 | 36.0 \n| UnSAM          |  1    |    41.2  | 43.6 | 52.1\n| UnSAM          |  2    |    42.4  | 44.5 | 52.7\n| UnSAM          |  3    |    41.7  | 44.0 | 51.8\n\nAs shown in the table, we observed that the model's performance remains stable across different sets of 2% unlabeled training data (sampled with 3 different seeds). All three models, each trained with a distinct set, consistently outperformed previous state-of-the-art methods by a large margin.\n\n**[Q3] What Does “the two extreme cases” in Line 169 Refer To?**\n\nThe term \"two extreme cases\" refers to the semantic/instance-level masks (the coarsest granularity), and the subpart-level masks (the finest granularity). We will clarify it in the revision. \n\n**[L1] Limitation Section**\n\nWe discussed UnSAM's limitations in Section A6 of our submission and will highlight this more prominently in the main paper. Specifically, UnSAM struggles with images containing dense, fine-grained details, often missing repetitive instances with similar textures. Additionally, it tends to over-segment images due to the unsupervised clustering method mistaking details like folds and shadows on clothing as distinct entities—a contrast to human annotators who use prior knowledge to disregard such information. This underscores the ongoing challenge for unsupervised methods to match the performance of supervised approaches.\n\n*Hope our explanation and experiments can address your inquiries. We will integrate all your valuable comments into our revision!*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Quueqkc70a", "reviewer": "Reviewer_ecwU", "summary": "This paper presents Unsupervised SAM (UnSAM) for interactive and automatic whole-image segmentation which does not require human annotations.\nThis method uses top-down clustering and bottom-up merging to obtain multi-granularity pseudo labels for supervised SAM training. This unsupervised training of SAM achieved good performance on specific datasets.\nIn addition, this paper finds that SAM can achieve better performance by combining pseudo labels and a small amount of GT from SA-1B for model training.", "questions": "See weaknesses for details. If the author can answer the above questions positively, especially question 2, I will consider raising the score.", "limitations": "The problems mentioned in the paper do exist and are difficult to solve, and unsupervised pseudo noise is difficult to avoid.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The results of this paper are solid.\n2. The improvement of the paper on SAM is significant, both in terms of quantitative and partial qualitative results provided.", "weaknesses": "1. The motivation of this work is to extend SAM, but it has not been experimentally proven that unsupervised training of UnSAM outperforms fully supervised SAM by continuously increasing the size of the dataset. Instead, it only provides better semi-supervised results.\n2. The method used in this paper is very similar to the one published a year ago in [1], which first proposed unsupervised interactive segmentation using top-down clustering and bottom-up merging to obtain hierarchical masks for training interactive segmentation models. I believe that the author needs to provide a clear difference in design compared to [1], rather than just details or differences in model structure and source data.\n3. The paper lacks tests of interactive segmentation performance, such as evaluation of NoC (number of clicks) metrics, and should provide a comparison with previous interactive segmentation methods, such as SimpleClick [2] and its subsequent improvement work, etc.\n\n[1] Li, Kehan, et al. \"Multi-granularity interaction simulation for unsupervised interactive segmentation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Liu, Qin, et al. \"Simpleclick: Interactive image segmentation with simple vision transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ecwU, thank you for your insightful comments, and we really appreciate that you are willing to increase our score if we can answer the questions positively.\nWe will provide detailed responses to each of them below. \n\n**[W1] It Has Not Been Experimentally Proven that Unsupervised Training of UnSAM Outperforms Fully Supervised SAM by Continuously Increasing the Size of the Dataset.**\n\nThanks for your question. **We respectfully argues that surpassing the heavily supervised segmentation model SAM with an unsupervised model like UnSAM is not trivial.** We are pleased to report that UnSAM's performance improves with an increase in training samples—from 0.1% of SA-1B to 0.4% of SA-1B. We found that by increasing the training samples to 0.4% and using a larger backbone, UnSAM's performance already surpasses that of SAM.\n\n| Methods        | Setting      |    # imgs |  Avg. | COCO | LVIS | ADE | Entity | SA-1B | Part-IN | PACO  | \n|:--------       | :--------   | :--------  | --------:| --------:|  --------:| --------:| --------:|  --------:| --------:| --------:|\n| *SAM*     |  *Supervised*    | *11M*  | *42.1* | *49.6* | *46.1* | *45.8* | *45.9* | *60.8* | *28.3* | *18.1*\n| Prev. UnSup. SOTA     |  Unsupervised | 0.2M | 30.1 | 30.5 | 29.1 | 31.1 | 33.5 | 33.3 | 36.0 | 17.1\n| **UnSAM (RN-50)**          |  Unsupervised | 0.1M | 39.2 | 40.5 | 37.7 | 35.7 | 39.6 | 41.9 | 51.6 | 27.5\n| **UnSAM (RN-50)**          |  Unsupervised | 0.4M | 41.1 | 42.0 | 40.5 | **37.5** | 41.0 | 44.5 | 52.7 | 29.7\n| **UnSAM (ViT-B)**          |  Unsupervised | 0.4M | **43.0** | **44.0** | **42.7** | 37.2 | **44.4** | **47.2** | **55.1** | **31.1**\n| *vs. Prev. SOTA*          |  - | - | *+12.9* | *+13.5* | *+13.6* | *+6.1* | *+10.9* | *+13.9* | *+19.1* | *+14.0*\n\nDue to limitations in computing resources, completing the model training on the full 100% SA-1B dataset could take over two months, which may not be feasible within a short rebuttal period. However, the current results already show that UnSAM performs better than SAM on average across seven datasets. We are confident that these performance gains over SAM will further increase as we continue training with more samples.\n\n\n**[W2] Difference in Design Compared to MIS**\n\n- High-level Main Differences in Methods: Unlike MIS, which employs top-down clustering primarily for selecting proposals *WITHOUT* the capability to discover new objects missed by bottom-up clustering methods. In contrast, both bottom-up and top-down clustering in our divide-and-conquer strategy contributes to discovering new objects and parts within an image. This leads to two significant distinctions: 1) Bottom-up clustering relies heavily on low-level features like color and texture, resulting in lower-quality pseudo-masks for instance and semantic segmentation, particularly when dealing with disconnected or overlapping instances. 2) The bottom-up clustering in MIS merges only adjacent groups, often resulting in incomplete segmentation of instances with disconnected parts. \n- Minor Differences in Bottom-up Clustering: While MIS utilizes a cut-based method for bi-partitioning the affinity matrix, our approach employs mean-shift clustering, which leverages cosine similarity for merging groups into larger entities. This makes our method more resilient to noise and outliers in the training data.\n- Model Performance: To ensure a fair comparison, we trained UnSAM using the same model as MIS, ViT-Base, and present the NoC@85 results below:\n\n| Methods        | GrabCut (NoC@85 &#8595;) | Berkeley (NoC@85 &#8595;) | SBD (NoC@85 &#8595;) | DAVIS (NoC@85 &#8595;) |\n|:-------- | --------:| --------:| --------:| --------:|\n| MIS (unsupervised)           | 1.94 | 3.09 | 6.91 | 6.33 |\n| **UnSAM** (unsupervised)     | **1.39** | **1.42** | **3.04** | **4.02** |\n| SimpleClick (supervised)     | 1.40 | 1.44 | 3.28 | 4.10 |\n| **UnSAM+** (semi-supervised) | **1.30** | **1.37** | **2.74** | **3.11** |\n\nThe results for MIS and SimpleClick are copied directly from MIS. Our unsupervised model UnSAM significantly surpasses the previous state-of-the-art in unsupervised interactive segmentation and delivers performance comparable to the supervised SimpleClick. And the semi-supervised model UnSAM+ greatly surpasses the SimpleClick model.\n\nIt's important to note that datasets such as GrabCut, Berkeley, SBD, and DAVIS are easier and primarily focus on instance and semantic-level masks, typically labeling only a few dominant instances in each image. Consequently, we observe larger performance improvements on the MSCOCO dataset, as detailed in the following table.\n\n**[W3] Comparisons with SimpleClick**\n\nThank you for introducing us to SimpleClick! Beyond the results shared earlier, we have also measured the 1-IoU for SimpleClick on the MSCOCO dataset and present the results below:\n\n| Methods    | Setting           | 1-IoU  |\n|------------|-------------------|--------|\n| SimpleClick| Supervised        | 52.3   |\n| SAM        | Supervised        | 68.2   |\n| **UnSAM**  | **Unsupervised**  | **59.5** |\n| **UnSAM+** | **Semi-supervised** | **69.5** |\n\nOur unsupervised model, UnSAM, outperforms the supervised SimpleClick by over 7%, and these gains increase to 17.2% under a semi-supervised setting. The more significant performance improvement on MSCOCO, compared to datasets like GrabCut, Berkeley, SBD, and DAVIS, can be attributed to MSCOCO's complexity with numerous small and heavily overlapped instances, which presents a greater challenge than the datasets frequently used for interactive segmentation studies.\n\n*Hope our explanation and experiments can address your inquiries. We will integrate all your valuable comments into our revision!*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their valuable feedback. In this paper, we present Unsupervised SAM (UnSAM) for promptable and automatic whole-image segmentation that does not require human annotations. We are encouraged by the acknowledgements on: 

- **Comprehensive Experiments and SOTA Results**: We are especially glad that the reviewers believe that "the results of this paper are solid"(ecwU). Reviewers also noted that UnSAM "achieves improved results over state-of-the-art (SOTA) methods" (S6W7, wivK, p86w), and that "the improvement of the paper on SAM is significant, both in terms of quantitative and partial qualitative results provided" (ecwU).
- **Novel Setup**: UnSAM not only demonstrates SOTA performance in unsupervised segmentation but also shows that "the additional masks generated using the unsupervised approach, in combination with manual annotations, enhance performance further, surpassing that of SAM" (p86w). And it was agreed that "the two-stage approach to generate pseudo masks is reasonable." (S6W7)

We really appreciate that reviewers ecwU and p86w have indicated they might increase our score based on the content of our rebuttal. We are more than happy to discuss with all reviewers to address any additional questions during the discussion period. 

In this section, we commence by tackling the concerns that have been collectively raised. 

**Technical Contributions of UnSAM**
- *Novel and Simple Pipeline:* We introduce a simple yet effective divide-and-conquer pipeline for producing high-quality pseudo-masks for unsupervised image segmentation. As no previous work has used this pipeline for unsupervised image segmentation, we believe that our overall pipeline is novel and adds new knowledge to the field.
- UnSAM, for the first time, demonstrates that *unsupervised image segmentation method can achieve competitive results with the supervised counterpart SAM*. As shown in the table below, UnSAM achieves over 12% better AR than the previous unsupervised segmentation methods.
- UnSAM is also *the first to show that the state-of-the-art supervised segmentation method, SAM, can benefit from our self-supervised labels*—a discovery that has not been previously reported. As illustrated in the table below, UnSAM+ surpasses SAM's Average Precision (AP) by 3.9% and Average Recall (AR) by over 6.7% on SA-1B.
- *We believe in simple science*: We think a simple change leading to substantial performance gains is far more valuable and intriguing than a complex method yielding marginal gains. 

| Methods        | Setting      |    # imgs |  Avg. | COCO | LVIS | ADE | Entity | SA-1B | Part-ImageNet | PACO  | 
|:--------       | :--------   | :--------  | --------:| --------:|  --------:| --------:| --------:|  --------:| --------:| --------:|
| Prev. UnSup. SOTA (ViT-B)     |  Unsupervised | 0.2M | 30.1 | 30.5 | 29.1 | 31.1 | 33.5 | 33.3 | 36.0 | 17.1
| UnSAM (RN-50)          |  Unsupervised | 0.1M | 39.2 | 40.5 | 37.7 | 35.7 | 39.6 | 41.9 | 51.6 | 27.5
| UnSAM (RN-50)          |  Unsupervised | 0.4M | 41.1 | 42.0 | 40.5 | **37.5** | 41.0 | 44.5 | 52.7 | 29.7
| UnSAM (ViT-B)          |  Unsupervised | 0.4M | **43.0** | **44.0** | **42.7** | 37.2 | **44.4** | **47.2** | **55.1** | **31.1**
| *vs. Prev. SOTA*          |  - | - | *+12.9* | *+13.5* | *+13.6* | *+6.1* | *+10.9* | *+13.9* | *+19.1* | *+14.0*

| Methods        | Setting      |    # imgs |  Avg. | COCO | LVIS | ADE | Entity | SA-1B | Part-ImageNet | PACO  | 
|:--------       | :--------   | :--------  | --------:| --------:|  --------:| --------:| --------:|  --------:| --------:| --------:|
| SAM     |  Fully-Supervised                                           | 11M  | 42.1      | 49.6      | 46.1      | **45.8** | 45.9       | 60.8      | 28.3 | 18.1
| UnSAM+ (RN-50)          |  Lightly-Supervised | **0.1M** | **48.8** | **52.2** | **50.8** |   45.3    | **49.8**  | **64.8** | **46.0** | **32.3**
| *vs. SAM*          |  - | - | *+6.7* | *+2.6* | *+4.7* | -0.5 | *+3.9* | *+4.0* | *+17.7* | *+14.2*

**What's the Difference Between UnSAM and Prior Works? Why UnSAM Performs Better Than Prior Works?**

- **Comparison with CutLER and U2Seg**: These models are limited to providing only instance/semantic-level masks, missing the hierarchical structure that is often present in complex visual scenes. In contrast, our pipeline captures this hierarchical structure by identifying more fine-grained pixel clusters.

- **Comparison with SOHES**: 
  1. Segmentation Quality at Instance/Semantic Level: SOHES heavily utilizes low-level feature similarities between patches for cluster merging, which often leads to missing many instance masks that have disconnected or occluded components. SOHES struggles to recognize that visually distinct patches (e.g., red-colored t-shirts and green-colored pants) may belong to the same instance (e.g., a person). In contrast, our divide-and-conquer strategy employs a cut-based method that evaluates both the total dissimilarity between different pixel groups and the total similarity within these groups during image partitioning. UnSAM often identifies instance/semantic-level masks that SOHES overlooks. Consequently, UnSAM achieves a recall that is **1.5 times higher than SOHES for large objects** on SA-1B.

   2. Segmentation Quality for Part/Sub-part Masks: Our use of top-down clustering also allows the model to zoom in on selected regions provided by our cut-based clustering method, resulting in more detailed and fine-grained masks for small objects. Consequently, UnSAM exhibits a recall rate that is **1.3 times higher than SOHES for small objects** on SA-1B. 

Because UnSAM can produce higher quality pseudo-labels than both CutLER and SOHES, the resulting segmentation model trained on these masks can get better model performance. 

**We will integrate all the valuable suggestions into our final version and open-source the code.** Next, we address all concerns raised by the reviewers below.

Meta-review: After rebuttal and discussion, all reviewers assess the manuscript with a weak accept (note that in one case this was only written in a comment and the numerical score has not been updated). The main strengths are the good presentation of the proposed method and the high performance. The main weakness is the limited novelty.
Decision: Accept (poster)