Segment Anything without Supervision

Authors: xudong  wang
jingfeng  yang
trevor  darrell
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 06:25:53

Abstract: The Segmentation Anything Model (SAM) requires labor-intensive data labeling. We present Unsupervised SAM (UnSAM) for promptable and automatic whole-image segmentation that does not require human annotations. UnSAM utilizes a divide-and-conquer strategy to “discover” the hierarchical structure of visual scenes. We first leverage top-down clustering methods to partition an unlabeled image into instance/semantic level segments. For all pixels within a segment, a bottom-up clustering method is employed to iteratively merge them into larger groups, thereby forming a hierarchical structure. These unsupervised multi-granular masks are then utilized to supervise model training. Evaluated across seven popular datasets, UnSAM achieves competitive results with the supervised counterpart SAM, and surpasses the previous state-of-the-art in unsupervised segmentation by 11% in terms of AR. Moreover, we show that supervised SAM can also benefit from our self-supervised labels. By integrating our unsupervised pseudo masks into SA-1B’s ground-truth masks and training UnSAM with only 1% of SA-1B, a lightly semi-supervised UnSAM can often segment entities overlooked by supervised SAM, exceeding SAM’s AR by over 6.7% and AP by 3.9% on SA-1B.

Main paper: 
 introduction: 
 Trained on massive unlabeled data using self-supervised learning methods, Large Language Models (LLMs) [5, 34, 33, 46, 2, 19] in natural language processing have revolutionized our world and redefined human-computer interactions. In the domain of computer vision, the recent introduction of the Segment Anything Model (SAM) [21] has dramatically transformed the field with its exceptional ability to handle diverse image segmentation tasks. However, the need for comprehensive manual labeling of training data-over 20 minutes per image [21] -limits SAM from following the scaling laws that benefit LLMs [20] . As a result, despite SA-1B [21] being the most extensive segmentation dataset available, it contains only about 11 million images. Moreover, human-annotated data often introduces significant biases based on the annotators' perceptions of "what constitutes an instance", which frequently leads to the oversight of small entities within the images. This challenge raises a crucial question addressed in this paper: Can we "segment anything" without supervision? In response, we present UnSAM, an innovative unsupervised learning method capable of performing both interactive and whole-image segmentation without the need for supervision.How can we achieve fine-grained and multi-granular segmentation masks comparable to those in SA-1B [21] without supervision? Insights from neuroscience suggest that the human visual system exploits the structure of visual scenes by decomposing dynamic scenes into simpler parts or motions. This perception of hierarchically organized structures implies a powerful "divide-andconquer" strategy for parsing complex scenes [4, 27] . Drawing inspiration from this, we introduce a divide-and-conquer approach designed to generate hierarchical image segmentation results directly from raw, unlabeled images. The divide-and-conquer approach is a crucial element of UnSAM, enabling it to effectively parse and segment images at multiple levels of granularity. UnSAM significantly surpasses the performance of the previous SOTA methods in unsupervised segmentation, and delivers impressive whole image and promptable segmentation results, rivaling the performance of the supervised SAM [21] . This comparative analysis features our unsupervised UnSAM, the supervised SAM, and an enhanced version, UnSAM+, across a variety of datasets. The top section displays raw images (row 1) alongside whole image segmentation outputs from UnSAM (row 3), and SAM (row 2). The bottom section highlights our promptable segmentation results using a point prompt (i.e., the star mark). The right panel quantitatively compares the performance across models, including metrics like Mask AR (%) and Point IoU.Our pseudo-mask generation pipeline initiates with a top-down clustering approach (i.e., the divide stage), to extract initial semantic and instance-level masks using a Normalized Cuts-based method CutLER [39, 31] . Subsequently, UnSAM refines these masks using a bottom-up clustering method (i.e., the conquer stage): within each mask, we iteratively merge semantically similar pixels into larger segments based on various similarity thresholds. The resulting masks at different thresholds in the conquer stage, along with the masks produced in the divide stage, create a hierarchical structure. Technically, we can generate a vast range of granularities with minimal extra cost! Furthermore, UnSAM captures more subtle details that pose challenges for human annotators, significantly enriching the granularity and utility of unsupervised segmentation models.Equipped with these sophisticated multi-granular pseudo masks as "ground-truth" labels, UnSAM is adeptly trained to perform both interactive and automatic whole-image segmentation, demonstrating remarkable versatility across various segmentation scenarios. We have observed that our UnSAM model frequently identifies objects that SAM [21] overlooks, particularly types of objects or parts typically missed by ground-truth annotations of SA-1B [21] , such as human ears, animal tails, etc.The capabilities of UnSAM are rigorously tested across seven major whole-entity and part segmentation datasets, e.g., MSCOCO [24] , LVIS [15] , SA-1B [21] , ADE [48] , Entity [29] , PartImageNet [16] and PACO [30] . As illustrated in Fig. 1 , we demonstrate some noteworthy behaviors:• The performance gap between unsupervised segmentation models and SAM can be significantly reduced: By training on just 1% of SA-1B's unlabeled images with a ResNet50 backbone, UnSAM not only advances the state-of-the-art in unsupervised segmentation by 10% but also achieves comparable performance with the labor-intensive, fully-supervised SAM. • The supervised SAM can also benefit from our self-supervised labels: integrating our unsupervised pseudo masks with SA-1B's ground-truth data and retraining UnSAM on this combined data enables UnSAM+ to outperform SAM's AR by over 6.7% and AP by 3.9%. We observed that UnSAM and UnSAM+ can often discover entities missed by SAM.2 Related Works 
self-supervised image segmentation: 
 Recent advances in unsupervised image segmentation [39, 28, 44, 6, 8, 41, 38, 42, 35, 12, 7, 37] have leveraged the emergent segmentation capabilities of self-supervised Vision Transformers (ViT) [8, 14, 17] to "discover" objects within images. Initial efforts, such as TokenCut [44] and LOST [32] , have produced semantically meaningful pixel groupings for salient objects by utilizing the class-attention mechanism of self-supervised ViTs. As a representative work in the unsupervised segmentation domain, CutLER [39] introduced a cut-and-learn pipeline for unsupervised object detection and image segmentation. CutLER initially generates high-quality pseudo masks for multiple objects using MaskCut [39] , followed by learning a detector on these masks using a loss dropping strategy. Extending this approach, VideoCutLER [40] employs a cut-synthesis-and-learn strategy for segmenting and tracking multiple instances across video frames without supervision. Additionally, SOHES [6] introduced the global-local self-exploration method to cluster image features from high to low cosine similarity, obtaining pseudo masks that cover multiple hierarchical levels.In contrast, UnSAM introduces a divide-and-conquer pipeline that generates more pseudo masks per image at the same processing speed, but with enhanced quality and broader coverage across hierarchical levels. Furthermore, UnSAM captures more subtle details that pose challenges for human annotators, significantly enriching the granularity and utility of unsupervised segmentation models.Tradition segmentation models have focused on predicting masks for all instances or semantic parts within a single image simultaneously. Recently, however, models have begun to interact with users, generating segmentation masks based on user inputs such as points [21, 23, 47, 45, 11 ], text descriptions [26] , or bounding boxes [21] . Moreover, some approaches now frame segmentation tasks within an in-context learning framework [43, 3] , utilizing in-context examples to define distinct segmentation tasks. For example, the Segment Anything model [21] can produce masks in a zeroshot manner based on different types of prompts. One limitation of SAM is that it only produces three class-agnostic masks. An extension, Semantic-SAM [23] , aims to segment and recognize objects at multiple granularities through a multi-choice learning scheme, allowing each click point to produce masks at multiple levels along with their semantic labels. Nevertheless, both models are supervised and rely on large-scale, human-annotated data, which introduces issues of annotator bias and scalability limitations.In contrast, our unsupervised UnSAM and lightly semi-supervised UnSAM+ model demonstrate superior performance in the promptable segmentation task, offering a robust alternative to these fully-supervised approaches. 
cut and learn (cutler) and maskcut: 
 CutLER [39] introduces a cut-and-learn pipeline to precisely segment instances without supervision. The initial phase, known as the cut stage, uses a normalized cut-based method, MaskCut [39] , to generate high-quality instance masks given the patch-wise cosine similarity matrixW ij = KiKj |Ki|2|Kj |2, where K i is "key" features of patch i in the last attention layer of unsupervised ViT. To extract multiple instance masks from a single image, MaskCut repeats this operation but adjusts by masking out patches from previously segmented instances in the affinity matrix:W t ij = (Ki t s=1 M s ij )(Kj t s=1 M s ij ) ∥Ki∥2∥Kj ∥2Subsequently, CutLER's learning stage trains a segmentation/detection model on these pseudo-masks with drop-loss. Please check Appendix A.2 for more details on CutLER.Segment Anything [21] tackles the promptable segmentation task. At its core lies the Segment Anything Model (SAM), which is capable of producing segmentation masks given user-provided points, boxes, and masks in a zero-shot manner. One significant contribution of SAM is the release of Our divide-and-conquer pipeline for generating the "ground-truth" pseudo masks used for training UnSAM without human supervision begins with a top-down clustering approach (i.e., the divide stage), to extract initial semantic/instance-level masks using a Normalized Cuts [31] -based CutLER [39] . Subsequently, we refine these masks using a bottom-up clustering method (i.e., the conquer stage): within each mask, we iteratively merge semantically similar pixels into larger segments using various similarity thresholds. The resulting masks at different thresholds create a hierarchy. We zoom-in selected regions to visualize details.the SA-1B dataset [21] , which comprises 11M high-resolution images and 1.1 billion segmentation masks, providing a substantial resource for training and evaluating segmentation models. While SAM significantly accelerates the labeling of segmentation masks, annotating an image still requires approximately 14 seconds per mask. Given that each image contains over 100 masks, this equates to more than 30 minutes per image, posing a substantial cost and making it challenging to scale up the training data effectively. For more details on SAM and SA-1B, please check Appendix A.3. Our segment anything without supervision model starts by generating pseudo masks that respect the hierarchical structure of visual scenes without supervision. This approach is motivated by the observation that the "divide and conquer" strategy is a fundamental organizational principle employed by the human visual system to efficiently process and analyze the vast complexity of visual information present in natural scenes [4, 27] . Our pseudo-mask generation pipeline divide-andconquer, which is summarized in Alg. 1 and illustrated in Fig. 2 , consists of two stages:Divide stage: we leverage a Normalized Cuts (NCuts)-based method, CutLER [39, 31] , to obtain semantic and instance-level masks from unlabeled raw images. CutLER's cut-and-learn pipeline and its MaskCut method are discussed in Sec. 3.1. However, the coarser-granularity masks predicted by CutLER can be noisy. To mitigate this, we filter out masks with a confidence score below a threshold τ . Empirically, salient semantic and instance-level entities typically encompass richer part-level entities (for example, a person has identifiable parts such as legs, arms, and head, whereas a background sky contains few or no sub-level entities). To extract these part-level entities with a hierarchical structure, we employ a conquer phase.Conquer stage: for each instance-/semantic-level mask discovered in the previous stage, we employ iterative merging [1, 6] to decompose the coarse-grained mask into simpler parts, forming a hierarchical structure.More specifically, we first crop local patches using the masks we obtained in the divide phase, and bi-linearly interpolate local patches to the resolution of 256 × 256. We then feed them into DINO pre-trained ViT-B/8 [8] encoder f (•), and extract 'key' features k i = f (p i ) from the last attention layer as patch-wise features for local patches p i . Subsequently, the conquer phase employs iterative merging [1, 6] to group patches into larger clusters, with pre-defined cosine similarity thresholds at θ ∈ {θ 1 , ..., θ l }, where l is the predefined granularity levels.I resized ← input image I resized to 1024 × 1024 M ← {m : m ∈ CutLER(I resized ) ∧ m score > τ } for m ∈ M do Add m into S 0 bbox ← bounding box [x 1 , y 1 , x 2 , y 2 ] of m I local ← I resized cropped by bbox, resized to 256 × 256 K ← DINO(I local ) for θ t ∈ θ l , . . . , θ 1 do if t = l then Initialize k t i ← K i , C t i ← p i ∀i, a ← 1 where p i is corresponding patch of K i , add p i into S l ∀i else Initialize S t ← S t+1 , k t i ← k t+1 i , C t i ← C t+1 i ∀i end if while a ≥ θ t do Identify adjacent p i , p j with i, j ← argmax i,j k t i k t j ||k t i ||2||k t j ||2 , a ← max i,j k t i k t j ||k t i ||2||k t j ||2 Identify cluster C t m , C t n , where p i ∈ C t m , p j ∈ C t n Remove C t m and C t n from S t C t ← C t m ∪ C t n , add C t into S t ∀p z ∈ C t , k t z ← amk t i +ank t j am+an, where a m is the size of cluster C t m and p i ∈ C t m end while end for end for In iteration t, our method finds two adjacent patches (p i , p j ) from two separate clusters (C t m , C t n ) with the highest cosine similarityk t i k t j ||k t i ||2||k t j ||2, merges them into one cluster, and updates k t i and k t j toamk t i +ank t j am+an, where a m is the number of patches in cluster C t m (p i ∈ C t m ). The conquer stage repeats this step until the maximum cosine similarity is less than θ t , collects all merged clusters as new part-level pseudo masks, and uses smaller threshold θ t+1 to iterate again. Each coarse-grained mask discovered in the divide stage can form a hierarchical structure H after the conquer stage:EQUATIONn t is the number of clusters/masks belonging to granularity level t and n 0 = 1.The new part-level pseudo masks discovered in the conquer stage are added back to the semantic and instance-level masks identified in the divide stage. We then use Non-Maximum Suppression (NMS) to eliminate duplicates. Following previous works in unsupervised image segmentation [39, 28, 6] , we also employ off-the-shelf mask refinement methods, such as Conditional Random Fields (CRF) [22] and CascadePSP [10] , to further refine the edges of the pseudo masks. Finally, we filter out the post-processed masks that exhibit significant differences in Intersection-over-Union (IoU) before and after refinement.Preliminary results: The divide-and-conquer pipeline achieves a pseudo mask pool with more entities, a broader range of granularity levels, and superior quality compared to previous work, e.g., CutLER [39] , U2Seg [28] and SOHES [6] . As shown in Table 3 , its pseudo masks reach 23.9% AR on 1000 randomly selected validation images from the SA-1B dataset [21] , representing a 45.7% improvement over the state-of-the-art.Key distinctions over prior works on pseudo-mask generation: The divide-and-conquer strategy employed by UnSAM sets it apart from previous works:[ 39, 28] rely solely on top-down clustering methods, providing only instance and semantic-level masks, and thereby missing the hierarchical structure present in complex images. In contrast, our pipeline captures this hierarchical structure by identifying more fine-grained pixel clusters.While [6] does incorporate some hierarchical structure through bottom-up clustering with iterative merging, it still misses many fine-grained instances and some large-scale instance masks. Additionally, the iterative merging in [6] focuses on small regions below a certain mask size threshold, primarily to refine noisy small masks, limiting its ability to detect a full range of entity sizes. Our experimental results demonstrate qualitatively and quantitatively superior performance compared to prior works, particularly in producing high-quality, detailed pseudo-masks that better capture the hierarchical complexity of visual scenes. 
model learning and self-training: 
 Although the pseudo masks generated by our pipeline are qualitatively and quantitatively superior to those from prior works, they can still be somewhat noisy. Our self-supervised pipeline has limitations in identifying certain types of instances. For example, iterative merging sometimes fails to correctly associate disconnected parts of the same entity. To address this, we utilize a self-training strategy to further enhance UnSAM's model performance. UnSAM learns an image segmentation model using the masks discovered by the divide-and-conquer strategy. It has been observed that self-training enables the model to "clean" the pseudo masks and predict masks of higher quality [39] . Once we have prepared the pseudo-masks, UnSAM can be integrated with any arbitrary whole-image or promptable image segmentation models during the model learning or self-training stage.Whole-image segmentation. We choose the vanilla Masked Attention Mask Transformer (Mask2Former) [9] for simplicity. The key innovation of Mask2Former is the introduction of a masked attention mechanism in the transformer's cross-attention block, defined as softmax(M + QK T )V , where the attention mask M at feature location (x, y) is given by:M (x, y) = 0 if M (x, y) = 1 -∞ otherwise. This mechanism constrains attention within the region of the predicted mask. UnSAM is then trained using the following mask prediction loss:L = λ ce L ce + λ dice L dice (2)where L ce and L dice is the cross-entropy and Dice loss, with λ ce and λ dice as their respective weights.After one round of self-training UnSAM on the pseudo-masks, we perform a second round of selftraining by merging high-confidence mask predictions (with a confidence score greater than τ self-train ) as the new 'ground-truth' annotations. To avoid duplication, we filter out ground truth masks that have an IoU greater than 0.5 with the predicted masks.Promptable Image Segmentation. Similar to SAM [21] , our unsupervised SAM can also produce high-quality object masks from input prompts such as points. We utilize Semantic-SAM [23] as the base model for predicting multiple granularity levels of masks from a single click. During the learning process, we randomly sample points within an inner circle (radius ≤ 0.1 • min(Mask width , Mask height )) of the mask to simulate user clicks.The supervised SAM model's [21] reliance on human-annotated data introduces a significant bias based on the annotator's perception of 'what constitutes an instance', frequently missing some entities within the image. In contrast, since our mask generation pipeline does not rely on human supervision, it can often identify valid objects or parts that are overlooked by SA-1B's [21] groundtruth annotations.Motivated by this observation, we leverage UnSAM to improve the performance of the supervised SAM [21] by implementing a straightforward yet effective strategy: merging SA-1B's ground-truth masks D SA-1B with our unsupervised segmentation masks D UnSAM based on the IoU, formulated as:EQUATION) τ UnSAM+ is the IoU threshold, IoU max is the maximum IoU between C m and any mask C n in D i SA-1B , and D i SA-1B and D i UnSAM+ is the set of SA-1B and unsupervised masks within image i, respectively. We then train UnSAM+ on D UnSAM+ for promptable image segmentation and whole-image segmentation. The fusion approach leverages the strengths of both supervised and unsupervised annotations, addressing the limitations inherent in human-annotated datasets while significantly enriching the diversity and comprehensiveness of the training data. This results in a more robust and generalizable segmentation model UnSAM+, surpassing the performance of SAM.UnSAM's Unsupervised LabelsFigure 3 : Unsupervised pseudo-masks generated by our divide-and-conquer pipeline not only contain precise masks for coarse-grained instances (column 5), e.g., cameras and persons, but also capture fine-grained parts (column 3), e.g., digits and icons on a tiny camera monitor that are missed by SA-1B's [21] ground-truth labels.  
model training settings: 
 We provide a brief overview of the model training settings and include more details in Appendix A.1.Pseudo mask generation. In the divide stage, we set the confidence threshold τ =0.3; in the conquer stage, we choose threshold θ merge = [0.6, 0.5, 0.4, 0.3, 0.2, 0.1]. When merging the pseudo masks with the ground truths for training UnSAM+, we select τ UnSAM+ = 0.02. Whole-image segmentation. UnSAM picks DINO [8] pre-trained ResNet-50 [18] as the backbone and Mask2former [9] as the mask decoder. The default learning rate is 5 × 10 -5 with a batch size of 16 and a weight decay of 5 × 10 -2 . We train the model for 8 epochs. Promptable segmentation. UnSAM uses the self-supervised pre-trained Swin-Transformer [25] Tiny model as the backbone, and leverages Semantic-SAM [23] as the base model. We set the number of hierarchy levels to 6, which is also the number of predicted masks UnSAM generates per prompt during inference. One can easily train with a different number of granularity levels as needed. For all experiments, we train UnSAM with 1∼4% unlabeled images from SA-1B dataset [21] .Whole-image segmentation. To evaluate our model's performance, we test our models on various datasets in a zero-shot manner to evaluate the performance of segmenting entities from all granularity levels. We choose COCO [24] , LVIS [15] , ADE20K [48] , EntitySeg [29] , and SA-1B [21] that mainly encompass semantic-/instance-level entities; PartImageNet [16] and PACO [30] that cover part-level entities. The SA-1B test set consists of randomly selected 1000 images not included in our training set. Notably, each dataset only covers entities from certain hierarchical levels and certain pre-defined classes, while our model generates masks from all levels and all classes. Hence, the COCO Average Precision (AP) metric could not reflect our model's authentic performance in segmenting all entities in the open-world. Following prior work [39, 6] , we mainly consider Average Recall (AR) to compare with different models. 4 : Quantitative comparisons between our lightly semi-supervised SAM, UnSAM+, and the fully-supervised SAM [21] on SA-1B [21] .Point-based promptable segmentation. We evaluate our point-based interactive segmentation model on MSCOCO Val2017 [24] . Following the previous work on promptable image segmentation [21, 23] , we pick two metrics for model evaluation MaxIoU and OracleIoU. For each point prompt, UnSAM predicts 6 masks representing different granularity levels. MaxIoU calculates the IoU between the mask with the highest confidence score among 6 masks, whereas OracleIoU picks the highest IoU between 6 predicted masks and the ground truth. For each mask in a test image, we select its center as the point prompt.Unsupervised pseudo-masks. Unsupervised pseudo-masks generated by our divide-and-conquer pipeline not only contain precise masks for coarse-grained instances, but also capture fine-grained parts that are often missed by SA-1B's [21] ground-truth labels, as shown in Fig. 3 .Whole-image segmentation. Remarkably, UnSAM outperforms the previous state-of-the-art methods across all evaluation datasets as summarized in Table 1 . UnSAM demonstrates superior performance compared to the SOTA method even when trained with only 1% SA-1B training data and a backbone of ResNet-50 with only 23M parameters, while the SOTA utilizes twice training data and a backbone with nearly four times the parameters. This implies that UnSAM is a lightweight, easier to train, and less data-hungry model with better zero-shot performance in segmenting entities in the open-world as shown in Figs. 4 and 5 . On average, UnSAM surpasses the previous SOTA by 11.0% in AR. When evaluated on PartImageNet [16] and PACO [30] benchmarks, UnSAM exceeds the SOTA by 16.6% and 12.6 %, respectively. UnSAM not only discovers more fine-grained masks than the previous state-of-the-art unsupervised segmentation method [6] , but also provides segmentation masks with a wide range of granularity. We show qualitative comparisons between UnSAM (with 3 levels of granularity) and baseline models on SA-1B [21] . 5 : Despite using a backbone that is 3× smaller and being trained on only 1% of SA-1B, our lightly semi-supervised UnSAM+ surpasses the fully-supervised SAM in promptable segmentation task on COCO.When compared to the supervised SAM [21] , UnSAM's AR across all datasets is already very close, with only a 1% difference. On PartImageNet [16] and PACO [30] , UnSAM surpasses SAM by 24.4% and 11.6%. This further demonstrates the excellent capability of our divide-and-conquer pipeline in discovering details that human annotators tend to miss. Furthermore, our UnSAM+, trained with integrated unsupervised pseudo masks and SA-1B [21] ground truth, outperforms SAM's [21] AR by over 6.7% and AP by 3.9% as shown by Table 2 and 4 . UnSAM+ demonstrates superior average recall compared to SAM across all evaluation datasets except for ADE20K [48] , which is dominated by semantic-level annotations. UnSAM+'s significantly 16.2 % higher AR on small entities further confirms that our pseudo masks can effectively complement the SA-1B datasets with more details it ignores and the UnSAM+ can often discover entities missed by SAM as demonstrated in Fig. 4 and Fig. 7 .Point-based promptable segmentation. As shown in Table 5 , UnSAM trained with our pseudo masks achieve 40.3% MaxIoU and 59.5% OracleIoU on COCO. Notably, we train the model with only 1% of the data that SAM [21] uses and a backbone with 4× fewer parameters. Moreover, the UnSAM+ trained with integrated pseudo masks and SA-1B ground truths outperforms SAM on both MaxIoU and OracleIoU with 0.3% and 1.3% respectively. Qualitative results are shown in Fig. 6 .  
summary: 
 Image segmentation is a fundamental task in computer vision, traditionally relying on intensive human annotations to achieve a detailed understanding of visual scenes. We propose UnSAM, an unsupervised segmentation model that significantly surpasses the performance of previous stateof-the-art methods in unsupervised image segmentation. Additionally, our unsupervised UnSAM model delivers impressive results, rivaling the performance of the cutting-edge supervised SAM, and exceeding it in certain semi-supervised settings.While SAM [21] significantly accelerates the labeling of segmentation masks, annotating an image still requires approximately 14 seconds per mask. Given that each image contains over 100 masks, this equates to more than 30 minutes per image, posing a substantial cost and making it challenging to scale up the training data effectively.COCO (Common Objects in Context) [24] is a widely utilized object detection and segmentation dataset. It consists of 115,000 labeled training images, 5,000 labeled validation images, and more than 200,000 unlabeled images. Its object segmentation covers 80 categories and is mainly on the instance-level. We evaluate our model on COCO Val2017 with 5000 validation images without training or fine-tuning on any images from the COCO training set. The metrics we choose are class-agnostic COCO style averaged precision and averaged recall for the whole-image inference task, and MaxIoU and OracleIoU for the promptable segmentation task. [21] consists of 11 million high-resolution (1500 on average) images and 1.1 billion segmentation masks, approximately 100 masks per image. All masks are collected in a class-agnostic manner with various subject themes including locations, objects, and scenes. Masks cover a wide range of granularity levels, from large-scale objects to fine-grained details. In the whole-image inference task, we randomly selected 1000 SA-1B images that are not used to generate pseudo labels as the validation set.LVIS (Large Vocabulary Instance Segmentation) [15] has 164,000 images with more than 1,200 categories and more than 2 million high-quality instance-level segmentation masks. It has a long tail distribution that naturally reveals a large number of rare categories. In the whole-image inference task, we evaluate our model using its 5000 validation images in a zero-shot manner.EntitySeg [29] is an open-world, class-agnostic dataset that consists of 33277 images in total. There are on average 18.1 entities per image. More than 80% of its images are of high resolution with at least 1000 pixels for the width. EntitySeg also has more accurate boundary annotations. In the whole-image inference task, we evaluate our model with 1314 low-resolution version images (800 × 1300 on average) in a zero-shot manner.PACO (Parts and Attributes of Common Objects) [30] is a detection dataset that provides 641,000 masks for part-level entities not included in traditional datasets. It covers 75 object categories and 456 object-part categories. In the whole-image inference task, we evaluate our model with 2410 validation images in a zero-shot manner.PartImageNet [16] is a large-scale, high-quality dataset with rich part segmentation annotations on a general set of classes with non-rigid, articulated objects. It includes 158 classes and 24,000 images from ImageNet [13] . In the whole-image inference task, we evaluate our model with 2956 validation images in a zero-shot manner.ADE20K [48] is composed of 25,574 training and 2,000 testing images spanning 365 different scenes. It mainly covers semantic-level segmentation with 150 semantic categories and 707,868 objects from 3,688 categories. In the whole-image inference task, we evaluate our model with 2000 testing images in a zero-shot manner.We provide more qualitative results of UnSAM and UnSAM+ in a zero-shot manner in Figure A1 , and Figure A2 . In images with very dense fine-grained details, UnSAM tends to miss repetitive instances with similar texture. As shown in Figure A3 , in the first row, although UnSAM accurately segments the leaves in the center of the picture, it misses some leaves located at the top of the image. Additionally, UnSAM occasionally over-segment images. In the second row, the right sleeve cuff of the dancer has meaningless segmentation masks. This issue mainly arises because the unsupervised clustering method mistakenly considers some information, such as folds and shadows on clothing, as criteria for distinguishing different entities. In contrast, human annotators can use prior knowledge to inform the model that such information should not be valid criteria. In this regard, unsupervised methods still need to close the gap with supervised methods.We train UnSAM and UnSAM+ on ground truths of and pseudo masks generated on SA-1B [21] . SA-1B contains licensed images that are filtered for objectionable content. It is geographically diverse, but some regions and economic groups are underrepresented. Downstream use of UnSAM and UnSAM+ may create their own potential biases.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Broader impacts of our research are discussed in Appendix A.7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: This paper doesn't include crowdsourcing or research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: After rebuttal and discussion, all reviewers assess the manuscript with a weak accept (note that in one case this was only written in a comment and the numerical score has not been updated). The main strengths are the good presentation of the proposed method and the high performance. The main weakness is the limited novelty.

Reviews: review: paper_summary: paper_summary: This paper proposes an approach to generate masks from images in an unsupervised manner, which are then used to train segmentation models. The major distinction from previous works is the proposed divide-and-conquer strategy, which first adopts the CutLER to obtain coarse instance/segmentation masks and then uses iterative merging like the SOHES method to generate fine-grained masks. Experiments are conducted on different datasets with detailed analysis and advanced performance.
strength_weakness: strength_weakness: - The major concern is about technical novelty. Though effective, the proposed method technically adds a candidate extraction method before the previous method's iterative refinement stage. All these methods already exist in the unsupervised segmentation field. Therefore, although the performance improvement is reasonable, the reviewer thinks its technical novelty is limited as a NeurIPS research paper.
- The comparison with SAM is based on different backbones. Meanwhile, some details, such as the number of masks (UnSAM's 6 vs. SAM's 3), are different. It may have a subtle influence on the performance comparison.
- Some typos exist. For example, the $\theta_{t+1}$ in line-147 should be $\theta_{t-1}$.
questions: questions: Please refer to the weaknesses for the concerns. The reviewer would appreciate it if these concerns could be addressed after the rebuttal.
limitations: limitations: The authors discussed the limitations in the appendix.
paper_summary: paper_summary: The paper explores a new way to generate hierarchical pseudo-masks to train downstream segmentation models without human annotations. First, the image is segmented using CutLER. Then, within CutLER proposed masks (with cropping and resizing), DINO features are extracted, and patches are iteratively merged based on proximity in cosine distance. The masks are then refined and filtered using CRF/CascadePSP. The pool of pseudo-labels is used to train Mask2Former architecture or Semantic-SAM architecture, for whole-image or prompt-able segmentation tasks, respectively. Additional self-training is performed to improve results. The evaluation is carried out on several datasets using the average recall (AR) metric. UnSAM shows significant improvement over other models in terms of recall of generated masks and final outputs.
strength_weakness: strength_weakness: #### What is "UnSAM"? 
The naming scheme adopted in the paper is slightly confusing. UnSAM refers to all:
 - the hierarchical pseudo-labelling scheme building on top of CutLER, 
 - the distillation of such scheme to Mask2Former architecture,
 - distillation of such a scheme into semantic-SAM architecture. 

While it still is possible to parse the results in the tables based on the setting, the ability to understand and scrutinise the text is severely impacted. Note the distinction is important as the output of three of these appears to be different.

#### Is AR a sensible metric?
The proposed model seems to generate an extremely large amount of masks (up to 2000, L407). While it seems that some useful masks are in this set, the cardinality of the output makes one wonder if it is useful. What is the number of masks used for assessing recall? Is it 1000? Cutler has a setting of 100-200 from published configs. Clearly, increasing the number of masks boosts recall. It is important that the evaluation maintains the same amount of masks for all methods. 

It also raises a question whether some classical hierarchical segmentation methods such as MCG [A] or Arbelaez et al. [B] would get similar or close performance when such large pool of candidates is allowed.

#### What is the effective difference between a proposal and SOHES?
The hierarchical merging scheme follows SOHES formulation, with the difference being the CutLER region "proposal" as a constraint. Is that all? It is important to highlight the differences here to both figure out the novelty and correctly attribute improvements in performance to different components/steps of the proposal. Additionally, it appears that distillation to Mask2Former abandons the hierarchical association. Does this mean that the output is no longer organised in a hierarchical manner?

#### What makes the method work better than prior works?
There is some lack of ablations that explore the influence and sensitivity of various components in the pipeline. The paper only reports on the construction and technical details of the UnSAM methods and the evaluation of this approach. It lacks analysis and insights into what the makes the construction effective. The advancement of knowledge offered by the paper is somewhat limited. 

#### Are different two different models required?
It is not entirely clear why the two modes of operation, "whole image" and "prompt-able" require different models/architectures. Could the Semantic-SAM-based model be prompted in a similar fashion to SAM (i.e. a grid of points) to perform the whole image segmentation? Is this disadvantageos for the whole-image segmentation task?

#### Inaccuracies 
Finally, it is important to note that CascadePSP relies on manual annotations. Thus, unSAM + CascadePSP is _slightly_ supervised. 

Since UnSAM partly distils CutLER outputs, would it not be more accurate to incorporate the #images used in CutLER to those in UnSAM, writing e.g. 1.4M in Table 1 instead of 0.1M?

[Nit] L135: "<...> we employ iterative _merging_ to _decompose_" might require rephrasing as it is currently a slight oxymoron. 

---

[A] Arbelaez et al. "Multiscale combinatorial grouping"

[B] Arbelaez et al. "Contour detection and hierarchical image segmentation"
questions: questions: The central questions to address in the rebuttal would be around the evaluation protocol. Some more explanation about the contribution of the paper would also be helpful.


While the paper presents strong results, the appropriateness of the evaluation protocol is somewhat questionable. Furthermore, there are some questions around the difference of the proposed scheme to prior work, and lack of experiments and analysis to explain it. While a rebuttal can address these issues, I currently rate the paper as a Borderline Reject.
limitations: limitations: Limitation are appropriately addressed.
paper_summary: paper_summary: The paper presents Unsupervised Segment Anything Model (UnSAM) for image segmentation whose training does not have access to human annotations. UnSAM employs a divide-and-conquer approach to hierarchically segment the image. In the divide stage, it uses CutLER [39] to obtain masks, and in the conquer stage, it applies an iterative merging method from SHOES [6]. The proposed method achieves SOTA results.
strength_weakness: strength_weakness: 1. Unclear technical contributions -- limited novelty: The paper uses existing work, including CutLER, SHOES, DINO and Mask2former. The key differences are not explained (well), beyond that these existing methods are put together for unsupervised image segmentation.

2. The performance gain seems to come from additional thresholds used (Lines 130, 142) and potentially unfair comparisons (please see the point 3 and the Questions section below). The thresholds are set in an ad hoc manner.

3. Backbone comparison: In Section 4 (UnSAM), Line 139, the paper mentions using the DINO pre-trained ViT-B/8 encoder, yet none of the tables show results with the ViT-B backbone. Recent methods (e.g., SOHES and SAM) use the ViT-B/8 backbone. A fair comparison with these methods should use the same backbone. Although one might argue that RN-50 and Swin-Tiny (used in this paper) are lighter backbones, using ViT-B/8 would provide a fair comparison.

4. Clarity can be improved:
   - 4.1. Inconsistent Figure Caption: In Figure 1, the caption and figure are not consistent. The caption states “UnSAM (row 2) and SAM (row 3)”, but the figure shows SAM in row 2 and UnSAM in row 3. It is unclear which result corresponds to which method.
   - 4.2. Key Distinctions: The second part of “Key distinctions over prior works on pseudo-mask generation,” Line 167, is unclear. The paper summarizes SOHES [6] but does not explain novelty of the proposed work.

5. Additional comparisons (optional): The paper could be improved by including another comparison with "Unsupervised Universal Image Segmentation," CVPR 24 (published after the NeurIPS deadline, though).
questions: questions: 1. The original CutLER uses ImageNet for unlabeled training, while UnSAM uses the SA-1B dataset. Since the proposed method and baseline use different training data, it is unclear if the proposed method is truly better. Similarly, for the comparison with SOHES on the 0.2M data, did both methods use the same set of 0.2M images?

2. If different sets of 2% unlabeled training data from SA-1B are used, are the results stable? Are the threshold hyperparameters in UnSAM robust to variations of the selected training data? 

3. What does “the two extreme cases” in Line 169 refer to?
limitations: limitations: The paper lacks a limitation section. Including a discussion on failure cases would improve the paper.
paper_summary: paper_summary: This paper presents Unsupervised SAM (UnSAM) for interactive and automatic whole-image segmentation which does not require human annotations.
This method uses top-down clustering and bottom-up merging to obtain multi-granularity pseudo labels for supervised SAM training. This unsupervised training of SAM achieved good performance on specific datasets.
In addition, this paper finds that SAM can achieve better performance by combining pseudo labels and a small amount of GT from SA-1B for model training.
strength_weakness: strength_weakness: 1. The motivation of this work is to extend SAM, but it has not been experimentally proven that unsupervised training of UnSAM outperforms fully supervised SAM by continuously increasing the size of the dataset. Instead, it only provides better semi-supervised results.
2. The method used in this paper is very similar to the one published a year ago in [1], which first proposed unsupervised interactive segmentation using top-down clustering and bottom-up merging to obtain hierarchical masks for training interactive segmentation models. I believe that the author needs to provide a clear difference in design compared to [1], rather than just details or differences in model structure and source data.
3. The paper lacks tests of interactive segmentation performance, such as evaluation of NoC (number of clicks) metrics, and should provide a comparison with previous interactive segmentation methods, such as SimpleClick [2] and its subsequent improvement work, etc.

[1] Li, Kehan, et al. "Multi-granularity interaction simulation for unsupervised interactive segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Liu, Qin, et al. "Simpleclick: Interactive image segmentation with simple vision transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
questions: questions: See weaknesses for details. If the author can answer the above questions positively, especially question 2, I will consider raising the score.
limitations: limitations: The problems mentioned in the paper do exist and are difficult to solve, and unsupervised pseudo noise is difficult to avoid.

Comments: title: The rebuttal successfully addressed my comments
Thank you for considering to raise the score!
comment: Dear Reviewer S6W7,

Thank you very much for raising our score! We are pleased that our rebuttal addressed your concerns. Hope you have a nice day!

Best regards,

UnSAM Authors
Dear Reviewer p86w, 

Thank you very much for raising our score! We are pleased that our rebuttal addressed your concerns. We truly appreciate the constructive discussions with you, which can definitely help us refine our paper. 

Best regards, 

UnSAM Authors
Thanks for the response and additional information. It addressed my major concerns. Therefore, I think the recommendation score can be improved to Weak Accept.
I thank the authors for their thorough reply. Simple improvements that lead to significant gains are noteworthy and important. I was merely looking to confirm I understood the formulation that led to such improvements correctly. Lines 167-173 give the impression that bottom-up clustering in [6] is formulated differently.

I have updated my recommendation.
Dear Reviewer S6W7,

Thank you for your thorough review and insightful feedback on our paper. Your input has been instrumental in refining the quality of our work.

We are pleased to report that we have carefully addressed each of the concerns you raised. Here's a brief summary of our actions:

- **Technical Contributions**: We have detailed our key technical contributions in the global rebuttal and provided quantitative results that underscore the significant performance improvements these contributions bring.

- **Ablation Study on Backbones**: In response to your comments, we conducted further experiments with various backbones. The results, presented in the tables at W2.1, indicate that a larger backbone significantly enhances our outcomes.

- **Ablation Study on Number of Masks**: In response to your comments, we conducted further experiments with various number of masks. The results, presented in the tables at W2.2, indicate that UnSAM can still achieve performance gains over SAM as we increase the number of masks of SAM.

We remain open and committed to addressing any additional questions or concerns. Your feedback continues to shape our research, and we appreciate your contribution to the improvement of our paper.

Please feel free to reach out with further questions or suggestions. We look forward to potentially discussing these matters in greater depth and continuing to refine our work with your valued expertise.

Warm regards,

UnSAM Authors
Dear Reviewer wivk,

Thank you so much for raising our score from 4 to 6! Hope you have a wonderful day! 

Best,

UnSAM Authors
Dear Reviewer wivK,

Thank you for your thorough review and insightful feedback on our paper. Your input has been instrumental in refining the quality of our work.

We are pleased to report that we have carefully addressed each of the concerns you raised. Here's a brief summary of our actions:

- **Technical Contributions**: We have detailed our key technical contributions in the global rebuttal and provided quantitative results that underscore the significant performance improvements these contributions bring.

- **Ablation Study on Thresholds and Backbones**: In response to your comments, we conducted further experiments with various thresholds and backbones. The results, presented in the tables at W2 and W3.1, indicate that a larger backbone significantly enhances our outcomes.

- **Comparison with U2Seg**: We have included a comparison with U2Seg in section W5. The results demonstrate that UnSAM significantly outperforms previous methods in unsupervised image segmentation.

- **Robustness Across SA-1B Subsets**: We have tested our method across various subsets of SA-1B, consistently achieving over 10% performance gains, which validates the robustness of our approach.

We remain open and committed to addressing any additional questions or concerns. Your feedback continues to shape our research, and we appreciate your contribution to the improvement of our paper.

Please feel free to reach out with further questions or suggestions. We look forward to potentially discussing these matters in greater depth and continuing to refine our work with your valued expertise.

Warm regards,

UnSAM Authors
The authors' response successfully addressed my concerns, and I would like to increase the paper's rating to "weak accept".
Dear Reviewer p86w,

Thank you for your response. We are pleased that our rebuttal has addressed some of your concerns! Here are our detailed answers to your additional queries:

**1) SOHES heavily utilizes low-level feature similarities between patches for cluster merging. It seems that this does not create a problem for UnSAM due to the use of CutLER as the overall limit of the highest level in the hierarchy, correct?**

Yes, the issue of over-relying on low-level similarities is mitigated by our overall divide-and-conquer pipeline. Divide-and-conquer enables our bottom-up clustering to focus on selected regions, performing cluster merging effectively without being influenced by outliers outside these regions that are identified during the divide stage.

**2) Distinctiveness of Divide-and-Conquer:**

Beyond the differences listed in our global rebuttal, here we outline additional distinctions:
- **Major Differences in the Overall Pipeline**: *We wish to emphasize that the core contribution of this work lies in the integration of both bottom-up and top-down clustering—termed the Divide-and-Conquer strategy, a simple yet effective change with significant performance gains.* As no previous work has used this pipeline for unsupervised image segmentation, we believe that our overall pipeline is novel and adds new knowledge to the field. With advancements in either top-down or bottom-up clustering methods, we believe the performance of our Divide-and-Conquer approach can be further enhanced. Thus, we assert that the significant and innovative contribution of our work is the overall pipeline itself, rather than the individual components of each stage. 

- **Major Difference Between Our Bottom-up Clustering and SOHES**: The use of the divide phase significantly impacts our bottom-up clustering in two key ways:
   1) **Outlier Removal**: Our bottom-up clustering focuses exclusively on regions identified during the top-down clustering phase, which sharpens the model's focus and effectively filters out many noisy outliers outside these regions that are identified during the divide stage. 
   2) **Two Stage Pseudo-labeling**: The instance/semantic-level masks generated by CutLER specify candidate regions within the image. Similar to the use of selective search in R-CNN or the Region Proposal Network (RPN) in Faster/Mask R-CNN, CutLER primarily functions to provide initial candidate regions for subsequent detection or segmentation stages. This two-stage approach allows our bottom-up clustering to zoom in on these defined regions, facilitating the detection of smaller objects that SOHES often misses.

- **Performance Comparison**: We have reported substantial performance gains over SOHES, particularly in AR$_L$, supporting our argument that top-down clustering is more effective at identifying instance/semantic-level masks compared to solely bottom-up approaches. Due to the lack of publicly available codes and models from SOHES, we compare against our re-implemention of SOHES for fair comparisons. 

    | Method                        | AR    | AR$_S$ | AR$_M$ | AR$_L$ |
    |-------------------------------|-------|--------|--------|--------|
    | SOHES                         | 16.5  | 5.7    | 16.1   | 23.0   |
    | UnSAM                         | **23.9** | **7.9**  | **22.4** | **34.0** |
    | *Improvement over SOHES*      | +7.4  | +2.2   | +6.3   | +11.0  |

The below table showcases how training segmentation models on high-quality pseudo-masks from our strategy significantly advances the state-of-the-art in unsupervised segmentation across multiple datasets by an average of 12.9%.

| Methods        |  # imgs |  Avg. | COCO | LVIS | ADE | Entity | SA-1B | Part-IN | PACO | 
|:--------       | --------:   | --------:| --------:|  --------:| --------:| --------:|  --------:| --------:| --------:|
| SOHES (ViT-Base)       | 0.2M | 30.1 | 30.5 | 29.1 | 31.1 | 33.5 | 33.3 | 36.0 | 17.1 |
| UnSAM (RN-50)          | 0.1M | 39.2 | 40.5 | 37.7 | 35.7 | 39.6 | 41.9 | 51.6 | 27.5 |
| UnSAM (RN-50)          | 0.2M | 40.4 | 41.2 | 39.7 | 36.8 | 40.3 | 43.6 | 52.1 | 29.1 |
| UnSAM (RN-50)          | 0.4M | 41.1 | 42.0 | 40.5 | 37.5 | 41.0 | 44.5 | 52.7 | 29.7 |
| UnSAM (ViT-B)            | 0.4M | **43.0** | **44.0** | **42.7** | **37.2** | **44.4** | **47.2** | **55.1** | **31.1** |
| *vs. SOHES*               |          | *+12.9* | *+13.5* | *+13.6* | *+6.1* | *+10.9* | *+13.9* | *+19.1* | *+14.0* |
    

**We would like to emphasize our commitment to simple science: we hold the view that straightforward changes leading to significant performance improvements are much more valuable than complex modifications that yield only minimal benefits.**

Thank you once again for your insights! We welcome any further questions. Have a fantastic day!

Best regards,

UnSAM Authors
I thank the authors for their response. I agree with the proposed changes. 

#### Difference to SOHES

I thank the authors for the added explanation highlighting the output difference between SOHES and UnSAM. The question was about the formulation. 

> SOHES heavily utilizes low-level feature similarities between patches for cluster merging.

My current understanding is that UnSAM does as well (L140-142). It seems that this does not create a problem in this case due to the use of CutLER as the overall limit of the highest level in the hierarchy, correct?

Other than the use of CutLER to limit these masks, are there any other noteworthy differences in the proposed Divide-and-Conquer?
Dear Reviewer ecwU,

Thank you so much for raising our score! Hope you have a great weekend! 

Best,

UnSAM Authors
My main concerns have been addressed and I will raise my rating from 4 to 6. I am very grateful to the authors for their serious approach to the paper and the rebuttal.
Dear Reviewer ecwU,

We are thrilled to hear that you are considering raising the score!

Regarding your additional questions about the potential degradation in UnSAM's performance when using only a bottom-up merging approach to generate masks, we have conducted experimental evaluations. We assessed the quality of pseudo-masks generated solely through a bottom-up clustering method on 1,000 images from SA-1B. The results are presented in the table below:

| Method                        | AR   | AR$_S$ | AR$_M$ | AR$_L$ |
|-------------------------------|------|--------|--------|--------|
| Bottom-up Clustering          | 16.5 | 5.7    | 16.1   | 23.0   |
| Bottom-up + Top-down Clustering | 23.9 | 7.9    | 22.4   | 34.0   |

The result clearly illustrates that incorporating both bottom-up and top-down clustering methods in our divide-and-conquer strategy significantly enhances performance compared to using bottom-up clustering alone. Notably, the most substantial gains are observed in AR$_L$, underscoring our discussion in the rebuttal that top-down clustering often identifies more instance/semantic-level masks compared to bottom-up methods alone. Unfortunately, due to the limited time for the discussion period, we were unable to complete pseudo-mask generation and segmentation model training for all training samples. However, we anticipate that the performance gains post-model training will align closely with the quality of the pseudo-masks.

Thank you again for your feedback! Please let us know if there are any more questions. We hope you have a wonderful day!

Best regards,

UnSAM Authors
Thank you for the response from the authors. I agree with the performance of unSAM. However, I still have some doubts about the novelty of the method, i.e. **W2**. MIS uses the bottom-up merging strategy to generate pseudo-labels. I think that it is able to merge features of non-adjacent groups according to SSE Cost in MIS. The top-down sampling in MIS is to balance the sampling probability of multi-granularity masks during training, which is indeed different from unSAM.

So I'll clarify my question again. I think unSAM's bottom-up clustering is very similar to MIS's bottom-up merging, except that unSAM further subdivides the instance masks produced by top-down clustering. In fact, the result of bottom-up merging contains masks of various granularities (including instance-level masks), so it seems that top-down clustering is not necessary.

But I would still consider raising my score, and if the author has time, could you provide some information on how much unSAM performance would be degraded by using only MIS bottom-up merging to generate the mask?