The Devil is in the Object Boundary: Towards Annotation-free Instance Segmentation using Foundation Models
Authors: Cheng Shi
Sibei Yang
Venue: ICLR 2024 poster (2024)
Keywords: object detection
annotation-free
instance segmentation
open-vocabulary
SAM
CLIP
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: One Annotation-Free Open Vocabulary Object Detector and Segmenter using Foundation models.

Abstract: Foundation models, pre-trained on a large amount of data have demonstrated impressive zero-shot capabilities in various downstream tasks. However, in object detection and instance segmentation, two fundamental computer vision tasks heavily reliant on extensive human annotations, foundation models such as SAM and DINO struggle to achieve satisfactory performance. 
In this study, we reveal that the devil is in the object boundary, $\textit{i.e.}$, these foundation models fail to discern boundaries between individual objects. 
For the first time, we probe that CLIP, which has never accessed any instance-level annotations, can provide a highly beneficial and strong instance-level boundary prior in the clustering results of its particular intermediate layer. Following this surprising observation, we propose $\textbf{\textit{Zip}}$ which $\textbf{Z}$ips up CL$\textbf{ip}$ and SAM in a novel classification-first-then-discovery pipeline, enabling annotation-free, complex-scene-capable, open-vocabulary object detection and instance segmentation. 
Our Zip significantly boosts SAM's mask AP on COCO dataset by 12.5\% and establishes state-of-the-art performance in various settings, including training-free, self-training, and label-efficient finetuning. Furthermore, annotation-free Zip even achieves comparable performance to the best-performing open-vocabulary object detecters using base annotations. Code is released at https://github.com/ChengShiest/Zip-Your-CLIP

Published as a conference paper at ICLR 2024
THE DEVIL IS IN THE OBJECT BOUNDARY: TOWARDS ANNOTATION-FREE INSTANCE SEGMENTATION USING FOUNDATION MODELS
Cheng Shi & Sibei Yang
School of Information Science and Technology
ShanghaiTechh University
[shicheng2022, yangsb) @esanngaitech.edeee
ABSTRACT
Foundation models, pre-trained on large amount of data have demonstrated im- pressive zero-shot capabilities in various downstream tasks. However, in object detection and instance segmentation, two fundamental computer vision tasks heav- ily reliant on extensive human annotations, foundation models such as SAM and DINO struggle to achieve satisfactory performance. In this study, we reveal that the devil is in the object boundary, i.e., these foundation models fail to discern boundaries between individual objects. For the first time, we probe that CLIP, which has never accessed any instance-level annotations, beneficial and strong instance-level boundary prior in the clustering results its particular intermediate layer. Following this surprising observation, propose Zip which Zips up CLip and SAM in a novel classification sirtrioo--frrtttee-iissoooooo pipeline, enabling annotation-free, detection and instance segmentation. Our Zip significantly boosts SAM's mask AP on COCO dataset by 12.5% and establishes srute-of-the-at performance in various settings, including vraining-free self eeff-trainnng and label-efficient finetuning. Furthermore, annotation-free Zip even achieves comparable performance the best-performing open- vocabulary object detecters using base annotations. Code is released /ttpss/ggitubb.ooCCCennniSiiieee
provide highly
open-vocabulary object
Your-CLIP
INTRODUCTION
The primary objective of both object detection and instance segmentation is to localize and classify objects within image. Although significant progress has been made in them, the costly, challenging. and sometimes even infeasible annotation collection limits fully-supervised approaches (Carion et al.
2020: He et al.] 2017 Ren al., 2015)] Girshick 2015). Recently, wave of vision foundation models, such as DINO (Caron et al. [021] || Oquab et al. 2023). CILP (Radford et al. 2021), and SAM (Kirillov al. 2023). has emerged and demonstrated impressive zero-shot generalization capabilities at various downstream tasks (Wang et al. 2023 Zhou et al., 2022c Kirillov et al. 2023). straightforward idea is directly transfer and synergistically utilize foundation models for object detection and instance segmentation, thereby eliminating the need for additional human annotations.
In this work, we study annotation-free object detection and instance segmentation, where the item "annotation-free" represents without needing any instance annotations and in in-domain human labels. We aim to directly generalize foundational models in a zero-shot manner to achieve object detection and instance segmentation. To attain this objective, the discovery and precise localization of object proposals are indispensable. Given that DINO has been employed for unsupervised object discovery in object-centric images from ImageNet (Deng al. (009), and SAM is capable of generating valid segmentation masks for any things and stuff, we intiilly endeavor to harness them to localize multiple objects more complex scenes from COCO (Lin al. 2014):
1 Corresponniig author

---

Published as a conference paper at ICLR 2024
DINO
CuiLERR
SAM-H
Clustering
Ours
Comparison of different annotation-free object discovery methods and zero-shot SAM-H. Previous state-of-the-ar methods rely on DINO, struggle to discern between instance-level objects, and often miss potential objects within the background. On the other hand, SAM generates valid segmentation masks but struggles to determine the confidence of a particular mask representing an object. In contrast, our approach, employing clustering outcomes of CLIP's intermediate features, can effectively outline boundaries between objects to differentiate them.
Figure 1:
Object discovery relying upon the foreground object mask from DINO struggles to discern between different object instances, such different oranges in Figure T resulting in only being able to localize salient regions in simple scenes instead of instance-level objects.
SAM produces numerous valid segmentation masks, including some object instances, achieving superior recall rates compared to DINO in object localization. However, SAM still yields relatively low precision due to its semantic-unaware and mostly edge-oriented approach, making it hard to discern the confidence of particular mask representing an object. We observe that SAM tends to assign higher confidence scores to smoother "planes", as shown in Appendix B Furthermore, considering CLIP enables the recognition of image regions, we combine the CLIP classification score with the SAM localization score using geometric mean. This enhancement elevates the confidence scores of masks, resulting in a marginal increase precision, though it falls short of complete satisfaction.
At its essence, both DINO and SAM are limited to distinguish objects, that is, discern the boundaries between objects. DINO cannot detect the edges between similar and closely positioned objects, whereas SAM can clearly delineate edges but cannot distinguish which ones are object boundaries of interest and which are part of the objects' interiors. For instance, in the case of an elephant's s mask, there may be distinct edges between head and ear, but these edges do not serve as boundaries distinguishing the elephant from other objects. The devil is in the object boundaries!
To overcome the challenge, we probe the CLIP's ability to discern the boundaries between objects instead of utilizing DINO or SAM to discover object boundaries. Our key insights are: 1) CLIP's features encompass the rich semantics learned from a vast corpus of image-text pairs, and semantics necessary condition for defining what constitutes an object boundary. For instance, in the case of "elephant", edges between "nose" and "head" are not considered object boundaries, but they so for "nose" itself. 2) Although CLIP is trained image-level alignment with text, it may have encountered fine-grained image-text pairs describing multiple local objects and their relations during training, thereby enabling identify and distinguish objects. After encountering several unsuccessful attempts, we observe that clustering the features of CLIP's specific middle layer (as shown in Figure Z7 can delineate the boundaries between individual objects! The visualization results clearly outline the objects in various complex scenes, such as three elephants and clusters of oranges. These widely observed clustering results encourage our study. To the best of our knowledge, we are the first to discover CLIP'ss ability to clearly outline object boundaries. In contrast, some semantic segmentation methods (Zhou et al. 2022a Li et al. 2023) rely on clustering CLIP's features to extract regions with the same semantics but cannot distinguish objects. Then, propose specific boundary score metric to extract object boundaries and combine with semantics to detect multiple objects in Figure 3] as detailed in Section 3.4 Notably, generate instance-level segmentation masks! Indeed, the issue SAM's inability to distinguish whether edges belong object boundaries or the inner edges of objects has been overcome by CLIP's
SAM can be effectively utilized to
2

---

Published as a conference paper at ICLR 2024
ability to outline object boundaries. For the first time, achieved the zero-shot generalization of CLIP and SAM foundation models to instance- level object localization and segmentation.
Furthermore, one incidental benefit of using CLIP to discover objects is to subsequently employ it for classifying the discovered object proposals, achieving open-vocabular instance segmentation. Unfortunately, the image-level alignment of CLIP makes it susceptible to contexts when classifying proposals, resulting in objects being classified with low confidence and even misclassified. In F of Appendix FP CLIP even misclassifies the salient object "car" as a "cat" in such simple scene
Therefore, we propose a classification--irs first--een---iisooveee pipeline rather than the conventionally used -owleooeery-flllooed----------e bycclassification one. As shown in Figure 3 for given -occbaabbyl category, we first extract the class-specific semantic clues in the image by computing the similarity between the category and the features of CLIP's s final layer. Notably, the semantic clues capture per-pixel alignment instead of per-image, which can avoid the previously mentioned mismatch issue between the CLIP's scene-centric training and our object-centric requirement, despite only capturing relatively rough regions. Then, we detect objects with precise boundaries by first clustering on features of CLIP's middle layer to extract fragments and then performing fragment selection to form objects according to the boundary score metric and the semantic clues. Finally, SAM can be prompted by our individual instance-level object mask, leading to a more accurate mask.
Our contributions are multi-fold: 1) We discover that the devil is in the object boundaries when employing foundation models such as DINO and SAM for annotation- -ree object detection and instance segmentation. Our study is the first to discover the potential of CLIP's particular middle layer in outlining object boundaries to overcome the challenge. 2) We propose a novel classification- fist-theendiscoverry framework, namely Zip, enabling annotation-free, complex-scene-capale vocabulary object detection and instance segmentation. Our Zip pipeline is the first to effectively zip up CLIP and SAM foundation models segment objects an innotation-free manner, according to object boundaries and semantic clues. 3) Zip in a training- free manner, i.e., without the need for any additional training and by directly leveraging off-the-shee foundation models, exhibits strong zero-shot performance across various evaluation settings, significantly improving class-agnostic (+8.7% AP), class-aware (+8.8% AP), and comparable performance on open-vocabulary object detection without any base annotation. 4) Zip, by combining training strategies such as self-training and data-efficient fine-tuning (with 5% data), can further improve AP to 18.2% and 28. 18.1%. Zip with seff-training significantly outperforms state-of-the-ee tht annotation free instance segmenter by 14.4% AP50 under the same settings.
open-
RELATED WORK
Visual Foundation Models. Without the need for finetuning, the foundation models (Brown et al.
2020
Ouyang et al.) 2020 Kirillov et al. 2023 Radford et al.) 2021; Li et al.) 2021 Caron et al.
2021 Oquab et al. 2023; Chowdhery el al.] 2022). pre-trained on carefully designed tasks have
demonstrated impressivv impressive sero-shot capabilities for various downstream tasks. The recently introduced DINO-v2 (qquab et al. 2023), pre-trained in sulf-super manner on a diverse range of curated data sources, shows astounding performance across various downstream tasks. The contrastive vision- language models (VLMs) such as CLIP (Radford et al. 2021) and ALIGN (Li et al., 2021), train a dual-modality encoder on large-scale image-text pairs to learn transferable visual representations under text supervision using contrastive loss. The transferable visual representations bolster a myriad of downstream tasks, including image classification (Zhou et al. 2022c by Shi & Yang 2023b). image
synthesis (Rombach et al. [021; Huang et al.] 2024), video comprehension (Lin et al. 2022 Tang
al., (203b)), semantic segmentation (Zhou et 2022a, Li et al., 2022), referring segmentation Dai & Yangl (2024); Tang et al. (2023a), and object detection (Zhou et al.) 2022d] Feng et al. 2022 Gu
let al.22222)) Shi & Yang 2023a). SAM (Kirillov et al. (023) introduces the pioneering foundation
model for image segmentation, pre- pre-trained using one billion masks and designed to respond to diverse input prompts such as points, bounding boxes, masks, and text. In standard object detection, the
potential of the foundation model has yet be validated. Our goal is to leverage the foundation model to explore its performance in object detection and instance segmentation tasks.
Unsupervised Object Detection and Instance Segmentation. As fundamental tasks in computer vision, object detection and instance segmentation require extensive human annotation to yield good performance (He et al. 2017 Carion et al. 2020). Many previous works (Wang et al. 2022b 2023
3

---

Published as a conference paper at ICLR 2024
Siméoni al. 2021 Wang al. 2022a Van Gansbeke et al. 2022) have explored unsupervised
approaches to circumvent this bottleneck. DINO (Caron et al. [021] 1) first unveiled that the self-
attention modules of the terminal block in self-supervised vision (Dosovitskiy et al. 2020) will automatically learn to segment the foreground objects. Following works (Wang et al. 2022b 2023]
Siméoni et al.] (02T) build upon DINO's observation and utilize DINO's patch features construct various similarity matrices for separating single foreground object in each image. To detect multiple objects, CutLER (Wang al. 2023 proposes MaskCut, which masks out the similarity matrix using the foreground mask, and then iteratively separates the objects. However, as shown in Figure [ foreground objects do not inherently contain instance information. Therefore, these previous methods fail to detect individual instances when multiple objects are densely clustered together, and they also miss inconspicuous objects in the background. Simultaneously, due to the lack of semantic understanding in pre-trained networks, they can only localize objects without being able to classify them. In contrast, our work not only leverages CLIP (Radford et al. 202I) to discover objects based on our observations that the intermediate output of CLIP offers strong prior for object discovery but also to classify objects based on the image-text alignment from the CLIP pre-training.
3 METHOD
In this section, we first revisit the CLIP model and introduce how to extract dense semantic clues from in Section 3.1] Then, we introduce our new discovery that clustering the features of CLIP's particular middle layer can delineate the boundaries between objects (Section 3.2). To synergistically leverage CL.PP's property for object boundary discovery and SAM's 's ability to generate segmentation masks, we propose novel object detection and instance segmentation (Section 3.3). In Section we provide details on how Zip derives the localization of individual objects from CLIP's clustering results.
Zip framework, enabling annotation-
3.1 PRELIMINARY
A Revisit of Contrastive Language-Image Fre-Training. CLIP (Radford et al. 2021) is trained on a vast corpus of image-text pairs using the contrastive learning approach. encodes the input image I and input text T by an image encoder Ener() :) and a text encoder Encr(-), respectively. To compute the similarity between the image and text, CLIP employs the following formula:
sime (I, = cos cosc(nn, Ency
(I)
where cos(:)) denotes the cosine similarity.
Extract Dense Semantic Clues from CLIP. As CLIP is primarily intended for image-level pre- dictions, it is not readily feasible to extract local patch predictions (i.e., similarities between image patches Ipatch and text 7) from CLIP. Previous works (Zhou et al. 2022a Li et al.. 2023) propose a specific modification in the last attention pooling layer of Eney( (-) to achieve the local predictions.
Given an image I E h with the size of h x w and its patch features obtained by passing the image through the Enc,() (-) except for the last layer, the similarities between the patches Ipatch and the text T are computed as follows,
SCLIP (Tpnch, 7) cos (Proj(Encc (Ipunc))
(Tmucc) )), (T)),
where Proj(-) is the value- embedddnn projection of the last attention attention-pooling -pooling layer and Ency (-) is the encoder Ency() (-) except for this pooling layer. When a class name like "elephant" is given as the
input text, the S (:) function allows us to calculate similarities between image patches and the class, and thus obtain to activation map indicating rough regions for that class in the image. One resulting region is visualized in "Semantic Clues" of Figure 3
3.2 PROBING CLIP FOR DISCERNING OBJECT BOUNDARY
In this section, we probe a new property of CLIP, which is the discovery of boundaries between objects. Figure presennss the results of clustering using features from the particular middle layers of CLIP (conv4_ x layer). As depicted, under various challenging scenarios, CLIP can distinctly outline the edges of objects of interest, represented by the gray cluster in Figure 2 and the orange cluster
4

---

Published as a conference paper at ICLR 2024
DINO
A
Ours
SAM
Figure 2: Clustering result and boundary comparison between diffeernt methods. In our clustering results from to D. the boundaries between objects are successfully clustered into particular clusters marked in orange gray, serving as strong priors for segmenting objects of the same category. the middle, we showcase the edges extracted by various methods. DINO provides only prominent outer edges, while SAM segments all edges, but such edges cannot distinguish which ones are object boundaries of interest. The devil is in the boundary.
in other cases. These clusters of edges define the boundaries between objects ("Ours" in Figure 2 enabling the localization of each individual object. contrast, DINO can only obtain the outer edges of the salient regions, while SAM captures all prominent edges. However, unlike ours, the edges extracted by both SAM and DINO do not directly correspond to boundaries between instance-level objects. Notably, this observation is universal, and these four cases from A to D are randomly selected for visualization. For a comprehensive analysis of this general observation, please see Appendix C Specifically, we employ standard K-Means clustering to obtain such impressive results. Please refer to Appendix C for details.
3.3 PROBLEM STATEMENT AND ZIP FRAMEWORK
In this section, we first formulate the problem statement. Then, we introduce our classification first- then-discovery pipeline, which is the first to effectively and synergistically utilize CLIP and SAM foundation models to detect and segment objects in an annotation manner, according to semantic clues (Equ. 2 and object boundaries (Sec. 3.2)
Annotation- Free Object Detection and Instance Segmentation seeks to detect and segment objects in an image belonging to set of object categories Telass without instance annotations and in-domain human labels. More specifically, given an image I E Rhxwwx for each category with a text name Te E Telass. the segmenter is required predict all the instances belongs to the category T and their segmentation masks. Considering the flexibility and adaptability of the ooea-vocabulary textual category, this task can facilely extend to various object detection and instance segmentation tasks. In this section, we introduce our instance segmentation method to simplify the demonstration because instance masks can be converted into bounding boxes of objects.
Our Zip pipeline, which follows classification-fist- -he----ee approach, consists of four steps, as illustrated in Figure 3
Classification first to obtain semantic clue prior. For each textual category T. we first obtaín an image-level similarity s and patch-level activation map Sr via Equ. and Equ. respectively. Considering that the semantic activation map Sy ("Semantic Clues" in Figure provides only rough object regions and lacks the ability to distinguish clustered objects of the same class, we depart from the CLIP+SAM (Li et al. 2023) of directly selecting points from the activation map to prompt SAM for instance segmentation. Instead, we leverage the semantic activation map as a strong prior in two distinct ways: 1) To assist CLIP in obtaining stable clustering results for outlining object boundaries, as discussed in Section 3.2 2) To identify the group of clustered fragments belonging to the same object, thereby enabling the localization of bounding boxes for each individual object, will be detailed in Section 3.4
Clustering to discover object boundary. We run the standard K-Means clustering algorithm with our semantic-aware initialization on the feature map from the second-to-last layer the Ency(-) to obtain K clustering results in CI.K. Each cluster C₁ E Cl.K is represented as {0, 1 hkw where C 1 means that the point (i, belongs to the k-th cluster.
5

---

Published as a conference paper at ICLR 2024
Middle-level Clustering
Fragment Selection
Prompting SAM
Image
Semantic Clues
Clustering
Pragments
Precise Masks
Figure 3: Zip: Multi-object Discovery with No Supervision. Zip follows classification-fisst first-then---- discovery approach, consisting of four steps: 1) Classification first to obtain semantic clues provided by CLIP, where the semantic clues indicate the approximate activation regions of potential objects. 2) Clustering on CLIP's features at a specific intermediate layer to discover object boundaries with the aid of our semantic- aware initialization. The semantic-awee aware initialization leverages semantic activation to automatically initialize clustering centers and determine the number of clusters. 3) Localization of individual objects by regrouping dispersed clustered fragments that have the same semantics, all while adhering to the detected boundaries. 4) Prompting SAM for precise masks for each individual object.
Localization of individual objects via fragment selection. To find each individual object, we further identify the clusters representing boundaries within the clustering results C1:K and regroup the scattered clustering fragments for each object. Specifically, we define the boundary score metric and utilize the objective function Equ. to detect boundaries and facilitate the grouping, which reformulates the instance segmentation task into fragment selection problem and employ the activation map ST help approximate the selection in annotation-free manner (Sec. (34).
Prompting SAM for precise masks. Although preliminary localization for individual objects is extracted, the segmentation masks generated by clustering are very coarse. Therefore, prompt SAM to attain finer segmentation results by initially prompting it with the preliminary bounding boxes and subsequently with the center points of the refined boxes. See Appendix for the analysis the prompting method.
3.4 LOCALIZATION OF INDIVIDUAL OBJECTS VIA FRAGMENT SELECTION
In this section, we localize objects from the clustering results by framing object detection and instance segmentation as a fragment selection problem.
Motivation and Observation: 1) Clustering results CL:K be transformed into superpixel- like fragments by considering the connectivity to eight neighbors in the image points in each cluster, as shown in "Fragments" in Figure 3 2) A specific of fragments
specific object. In Figure 3 the two fragments in the bounding box coalesce to form the instance "orange". Thus, we transform the instance segmentation task into fragment selection task. For symbolic consistency, without loss of generality, we assume that each cluster C2 is connected fragment. If this condition met, treat each connected fragment within the cluster new cluster.
correspond
Fragment Selection. Specifically, we select the fragment according to the activation map Sç IR. " and the elustering results C1.k, where the former roughly activates scattered fragments for objects of interest, while the latter is used to group fragments into each individual instance. First, we identify fragments that belong to the category T but are not on the boundary regions. The objective function is:
cluster-region activation score
(xT(XT)= = Σ CaSS, ((x)) - 1(X))
((CW)/(C 1(XT))
boundary score metric
where the X E xW is the indicator variable where X - signifies that the k-th cluster is identified to belong to category T and is not on the object boundary. (...) represents the inner product of matrices, and the f( ((Cx) returns the size of C 's smallest enclosing bounding box. The term cluster-region activation score encourages clusters belonging to the category T to be selected. The term boundary score metric penalizes clusters the object boundaries. It is computed as the

---

Published as a conference paper at ICLR 2024
Annotation Free
AR
AR 100
AP
AP
Method
AP,
APm
AP,
AP
10
75
50
COCO class-agnostic mask
1 SAM-B Kirilllvv ct al.
2023
10.1
36.9
1.4
3.3
5.9
2.7
4.6
2.6
2 SAM-H Kinllov ct al.
2023
17.4
44.5
0.9
6.1
15.0
6.9
10.4
6.6
3 DINOv2 (Qquab ct al.
2023
2.0
0.8
2.2
1.9
0.6
1.7
0.8
4 CutLER Wang al.
2023
v
16.5
2.4
8.8
24.3
9.2
18.9
9.7
5 Ours (SAM-B)
13.8
29.8
2.8
16.3
12.0
19.0
11.3
6 Ourst (SAM-B)
23.1
35.1
5.9
19.0
31.3
15.3
24.3
15.1 +12.5
COCO class-aware mask
1 CLIP+SAM-B Li al.
2023
17.9
23.8
0.9
5.4
10.0
2.8
5.7
3.0
2 CLIP+SAM-H L1
2023
21.9 27.6
1.2
7.2
13.2
4.1
7.5
4.2
3 DINOv2 Oquab et al.
2023
2.8
0.2
2.7
2.3
1.0
2.3
1.2
4 CutLER+ Wang 2023
2.7
7.4
8.2
15.7
8.5
5 Ours (BAM-B)
21.9
31.2
4.8
13.1
20.1
12.0
20.0+14..3
11.8
+8.8
6 Ours (SAM-B)
30.3
39.2
5.0
19.6
31.7
19.1
30.1
+24.4
18.2 +15.2
Table 1: Annotation-free instance segmentation on the COCO val2017. For fair comparison, keep the same settings the previously Dest-performing CutLER in the self-training setting, and Zip outperforms all previous methods on all evaluation metrics. with self-training, 1 our re-implemented based Neut (Shi & Malik 2000).
ratio between the area of the bounding rectangle and the area of the object itself because indicates the potential of becoming boundary. In our implementation, we found that the heuristic solution for Equ. works well as follows:
1, if (C.S X
(C..1) 2 01 and
(Cx,3) S 0g
f(C+)
0, otherwise
where we decouple the two terms in Equ. and 01 and 0g are arper-parameter set to 0.3 and 3 for all the experiments. Second, we execute standard connected graph algorithm (Caron al. [02T] on the identified clusters Σ (x1( to obtain the instance segmentation results. We naturally classify these instances to category T with the confidence score being the average activation score within the regions of instances.
4 EXPERIMENTS
We evaluate our approach on several object detection and instance segmentation settings, including annotation-free object localization, detection, and segmentation in Section and open-vocabulary object detection in Section 4.2 In Section 4.3 we show that our Zip can be an effective annotation- free initialization for supervised object detection and segmentation tasks, Finally, conduct the analysis and ablation experiments in Section 4.4
Implementation Details. Zip does utilize any instance annotations and in-domain image annotations aside from fine-tuning downstream tasks in Section 4.3 In order to match the model size utilized by previous methods (Siméoni et al. 2021 Wang et al. [202 Van Gansbeke al. 2022). we choose the RN50x64 CL.P model (Radford et al. 2021), adjusting the image size to 2048 x 2048 to achieve superior clustering results. For fair comparison, we use the same prompting method to employ SAM-B to refine original results of all compared methods.
Self-training Details. Our method is compatible with any architecture of object detectors, such as DETR (Carion al., 2020) Mask R-CNN (He et al. 2017). For fair comparison with previous methods (Wang et al. 2023). we adopt the same detection architecture of CutLER (Wang et al., 2023). Unless specifically noted, employ the Cascade Mask R R-CNN by default in all experiments. We maintain the identical training strategy as CutLER. Please refer to Appenidx for more implementation and self-training details.

---

Published as a conference paper at ICLR 2024
4.1 ANNOTATION FREE OBJECT LOCALIZATION, DETECTION, AND SEGMENTATION
We conduct the annotation-free experiment on COCO in et al. 2014 shown in Table [ and an additional experiment on Pascal VOC (Everingham et al. 2010 in Appendix D We employ the same dataset partitioning on COCO dataset as previous methods and evaluate the standard COCO metrics. The primary methods we compare include: 1) the various variants of SAM and the naive combination of SAM and CLIP: 2) The previous state-of-the-er the DINO, i.e., CutLER. Additionally, we list fully-super baselines for reference. Notably, for unsupervised methods originally not requiring SAM, to ensure fair comparison, we also refine their outputs by SAM. Regarding the methodologies related to SAM, we introduce how we tailor SAM for parallel computation, supplemented with pseudocode in Appendix A.I
unsupervised object detection technique utilizing
Comparison on class-agnostic setting: 1) SAM-B demonstrates superior object localization capa- bilities compared to DINO (+36.9%) on AR100 by pretraining on a large amount of segmentation masks. Furthermore, SAM-H with a larger model size further improves the AR100 (+7.6%). 2) However, both SAM-B and SAM-H perform subpar in terms of the Average Precision (AP), showing a significant gap between the edge-oriented segmentation and the downstream instance segmentation. We conduct additional analysis and illustrate why SAM fails to yield satisfactory results the COCO dataset in Appendix B 3) Compared with SAM, our method without self-training surpasses SAM by +8.7% on AP, and after self-training. the lead extends to + +12.5%. This improvement demonstrates that our Zip effectively applies the foundation models to the downstream instance segmentation task in zero-shot manner. 4) Compared with previously best-performing CutLER (Wang al. (023). which leverage DINO's ability to discover object. Our Zip, even devoid of self-training, achieves higher AP (+1.6%) than CutLER subjected to self-training. Employing the same self-training, we surpass the state-of-the-an unsupervised method by margin of +5.4% AP.
Comparison on class-aware setting. To ensure fair comparison, we employ the identical CLIP model to classify the segmentation masks extracted by all compared methods. The even more significant improvement compared to the class-agnostic comparison shows the effectiveness of our pipeline. Specifically, surpass CutLER by margin of +14.4%
classification firtstttiinn-ittt-eeeteeeee fhrss-tten-discoveeee on AP50 and +9.7% on AP. In comparison, the previous CLIP+SAM (Li et al. 2023) results in less-than-ideal performance, highlighting the non-trivial nature of utilizing CLIP and SAM for annotation instance segmentation.
Furthermore, our Zip model consistently and significantly improves annotation-free object detection on the Pascal VOC dataset (Table 8 in Appendix It enhances the state-of-the-art AP50 by +10.2% in the class-agnostic setting and +5.6% the class-aware setting. In addition, we compare Zip with semantic segmentation models (with text-image pairs for pretraining) combined with SAM, achieving significant performance improvement on annotation object detection, as detailed in Appendix
4.2 OOEA-VOCA OBJECT DETECTION
open-vocabulary object detection (OVD) methods are trained using annotations of objects from base categories but are re- quired to detect and recognize objects belonging to both base and novel categorics. Without resorting to a single COCO im- age in either base or novel categories, annotation-free and training-fres Zip achieves comparable performance to OVD methods trained
Method
AP50
with annotation
ViLD (RN50)
27.6
RegionCLIP (RN50)
26.8
annotations of base categories. For in- stance, our Zip achieves comparable performance to ViLD GGu et al. 2021) in a training-free and zero-shot manner. Follow- ing stainnga it not only significantly outstrips the base- lines (Zhong et al., 2021; Zhou et al.] 2022d] (hi & Yang 2023a (+12.5%) but also approaches performance close to the current state-of-the-eeeee---
Detic (RN50)
27.8
EdaDet (RN50)
37.8
CORA+ (RN50X4)
43.1
w/o any annotation
Ours (RN50)
23.5
Ours (RN50x64)
27.1
models CORA+ (Wu et al. 2023).
Ours! (RN50)
40.3
In addition, we also observe a performance drop from the novel category (27.1% to all COCO 80 categories (20.0%), which is analyzed in Appendix G
Table 2: OVD results on COCO. We experiment with transductive set- tings where all category names are known.
4.3 FEW-SHOT AND LABEL- TUNING
8

---

Published as a conference paper at ICLR 2024
We now evaluate Zip as pretraining model for training object detection and instance seg- mentation models following the CutLER set- ting. We use the self-supervised Zip to ini-
Segnentationn AP (96
Detection: AP (%)
12
4.44%
4:3%
tialize the detector and fine-tune it varying amounts of labeled data on the COCO dataset. The comparisons between Zip with the unsuper- vised object detection method CutLER and the
CurlEl 0223
18
babeled Data
1. of labeled Data
self-supervised MoCo-v2 (Chen et al.] 2020) are shown in Figure 4 We report both box AP and mask AP of models using from 1% to 30% labeled data. Under 1% of labeled data, our method's performance stands at 21.1% box AP, while MoCo-v2 and CutLER exhibit perfor- mances of 16.8% and 11.8%, demonstrating a +4.3% improvement of our method. With the increment in labeled data, method continues improve, reaching a performance of 39.2% with 30% labeled data. These findings denote that our method consistently outperforms in different volumes of labeled data. Relative MoCo-v2 and CutLER, our method demonstrates a more significant improvement with less labeled data.
Figure 4: Label-efficient Tuning COCO. We initialize the weights derived from various self- supervised methods and then fine-tune them varying proportions of labeled data.
4.4 ANALYSIS OF OUR PIPELINE
We analyze the impact of each component on Zip and the robustness of the detector selection on the Pascal VOC val set. (1) Our baseline employs the "Segment Anything" mode from SAM (Kirillov al. 2023) to extract proposals from the entire image, which CLIP then classifies. Although SAM pre-trained on one billion mask data, its high AR yet low AP underscores its limitations. (2) We utilize the method from DINO (Caron et al. 202T) on CLIP semantics, yielding modest results due to the absence of instance-level information. (3) However, with the further utilization of clustering information and boundary removal to acquire instance information, the performance escalated by +20.4%. Note that we only discovery boundaries in specific layer. (4) Refining the results on this basis using SAM improved the result by +4.9% substantiating the essentiality of each module in our methodology. (5) For any given detector architecture, our generated annotations consistently yield stable performance, with the optimal combination achieving the 52.1% AP50.
Analysis on Our Components
AP50
AR100
Analysis on the Architecture
AP50
AR100
"SAM CLIP"
16.6
$1.2
"DETR" "Mask R- R-CNN"
44.9
38.9
CLIPP semantic only"
2.2
3.6
46.1
40.3
"+ boundary detection"
22.4
28.4
"Cascade Mask R-CNN"
50.3
46.
"+ two-step SAM"
27.3
32.8
"NNIOI"
52.1
47.6
Table 3: Analysis of component our pipeline and its robustness across different detectors.
Further analysis regarding the results for boundary discovery, and algorithm complexity are given in the Appendix F
pipeline, general and stable clustering
A and
CONCLUSION
This paper introduces Zip, an annotation-free, segmenter. Zip combines two different foundation models CLIP and SAM to achieve robust instance segmentation performance on downstream datasets. First, we highlight our observation that the clustering outcome from the intermediate layer output in CLIP is keenly attuned to the boundaries between objects. Building on this, we introduce our convert the instance segmentation task into a fragment selection task. Our approach significantly surpasses previous methods across a wide range of settings. Ethics Statement: It is possible that any existing biases, stereotypes, and controversies inherent within the image-text pairs utilized for training the CLIP could inadvertently permeate our models.
object detector and instance
pipeline and
Acknowledgment: This work was supported by the National Natural Science Foundation of China (No.62206) 174) and MoE Key Laboratory of Intelligent Perception and Human-Maun Machine Collaboration (ShanghaiTTech University).

---

Published as a conference paper at ICLR 2024
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCC Ecion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Qiyuan Dai and Sibei Yang. Curriculum point prompting for weakly-supervised referring segmenta- tion. 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010. 19929, 2020.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Expand your detector vocabulary with uncurated images, arXiv preprint arXiv:2203.16513. 2022.
Ross Girshick, Fast r-cnn. In Proceedings of the IEEE international confereeeecccnccceeeeeccc on computer vision, pp. 1440-1448, 2015.
Xiuye Gu, Tsung- Yi Lin. Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 2021.
Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. International Conference on Learning Representations, 2022.
John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series e (applied statistics), 28(1):100-10) 1979.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with Ilm director and Idm animator. Advances in Neural Information Processing Systems, 36, 2024.
10

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint 30XY::2304.026443, 2023.
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209. 15639, 2022.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In The European Conference Computer Vision, 2014.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pp. 388 404. Springer, 2022.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 9748-8763. PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems, 28, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Cheng Shi and Sibei Yang. Edadet: Open-vocabulary object detection using early dense alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15724-15734, 2023a.
Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can be good visual prompts for vision-language models. arXiv preprint arXiv:2309.01 155, 2023b.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions pattern analysis and machine intelligence, 22(8):888 905, 2000.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-suue transformers and no labels. arXiv preprint arXiv: 2 109. 14279, 2021.
Jiajin Tang, Ge Zheng, Cheng Shi, and Sibei Yang. Contrastive grouping with transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 33570-23580 2023a.

---

Published as a conference paper at ICLR 2024
Jiajin Tang, Ge Zheng, and Sibei Yang. Temporal collection and distribution for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15466-15476, 2023b.
Wouter Van Gansbeke, Simon Vandenhende, and Luc Van Gool. Discovering object masks with transformers for unsupervised semantic segmentation. arXiv preprini arXiv:2206.06365 2022.
Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. FreeSOLO: Learning to segment objects without annotations. arXiv preprint arXiv:2202. 12181, 2022a.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and lear for unsupervised object detection and instance segmentation. arXiv preprint arXiv:2301. /320,, 2023.
Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufrey- daz. Self-supervised transformers for unsupervised object discovery using normalized cut. In Conference on Computer Vision and Pattern Recognition, 2022b.
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. ArXiv, abs/2303.13074 2023.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiao- long Wang. Groupvit: Semantic segmentation emerges from text supervision. arXiv preprint arXiv:2202. 10094, 2022.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-bases language-imag 2021.
pretraining. arXiv preprint 1rXiv:2112.09101
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision (ICCV), 2022c.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision, arXiv preprint urXiv:2201.0221 .22605, 2022d.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
In Appendix, we provide additional information regarding,
Implementation Details (Appendix A
Why does SAM Fail AP? (Appendix
Clustering with Semantic-aware initialization (Appendix
Additional Results on Pascal VOC (Appendix D
Is SAM Free Lunch? (Appendix) F
Why Classification then Discovery (Appendix
F
A Closer Look at COCO 80 Categories (Appendix G
Qualitative Results (Appendix
Compared with Weakly supervised Semantic Segmentation methods (Appendix)
Limitations (Appendix J
Datasets License Details (Appendix
Understanding the Process Clustering and Fragment Selection (Appendix L
Comparison between DINO and CLIP feature (Appendix N
A IMPLEMENTATION DETAILS
A.1 SAM AUTOMATICALLY MASK GENERATE IN PARALLEL
The official implementation of SAM's GaanAttooaaiceMassGenneaa does not support the parallel processing of multiple images. Utilizing MamnAutoaaiieMaslGGennenn to extract annotations from large-scale dataset proves be time-consuming. For instance, extracting proposals from the entire COCO dataset requires approximately 48 hours. We dissected the various modules within Lam.AutomaaccMMakeGeerrtoo modified portion of the code, and by uniformly scattering points within the images, enabled the model to accept parallel inputs, markedly accelerating the process. In comparison the prior implementation, our version merely requires 15 minutes on 8 A100 GPUs to complete the task.
SAM is comprised of three components: an image encoder, prompt encoder, and mask decoder
+ We take batch of images as inputs [...]
3 Batch_inpure
$ Encode the image:
6 icage_fcatures SAM. image_encoder (Batch_inputs)
8 $ Generate grid point prompt:
num_intervals 32
11 Create grid
12 x torchaiinspace (0, i, num intervals)
torch.linspace (0, 1, num_intervals)
xgrrid, y-grid torch rorch..eshgriddx y)
points torch. cat (oceattt(ttitttttt resggrid (-1, 1), reshad 1)), dim
=1) unsqueeeee (1)
labels_torch torch. like (pointss
01)
prompt_input (points, labels_torch Sparse_cnbeddingg cense_embeddings p:M.prompt_encoder
20
points=prompt
21
boxes=None, masks-None,
13

---

Published as a conference paper at ICLR 2024
Generate Mask in parallel
masks, scores, logits SAM.mask_decoder
26
27
()
31
Modifications on the mask decoder.
4 Borrew from SAM GitHub issue and MedSAM
2 4 Line 126 Refore
sre = corch.repeat
wtteerreeeet_tteeearaeeeee interLeave emmage_embeddingg, shape (0), dim=0)
Line 126 After
0] tokens. shape 0]:
6
src torch. renchrrepeattiteee eave mmage_embeddiinge cokens.shape shape [01, dim
-0)
else:
image embeddings
We also compared the ARIO and ARI00 among different methods in Table We discovered that the complex yet non- non-parallelizable post-prose processinn indeed augmented the Average Recall (AR), however, own model also surpassed the original SAM in terms of AR10.
Method
1R
AR 100
SAM-B E32
11.4
39.6
SAM-H E32
18.5
49.5
SAM-B-Parallel E32
10.1
36.9
SAM----arallel E32
17.4
44.5
Ours
15,8
29.8
Ours
23.1
35.1
Table 4: Comparison of different SAM models at varying sampling points, contrasting the original and parallelized versions terms of AR (Average Recall).
A.2 SAM+CLIP IN PARALLEL
Inspired by F-VLM al. 2022). in order avoid redundant cropping of images and input the entire proposal bounding box region into CLIP, we use RolAlign to feed the corresponding visual features into the pooling layer of CLIP. We also compared the SAM+CLIP in parallel with the naive method which crops the proposal from the image and resizes it to 224 then feeds it into CLIP SAM+CLIP in Parallel boosted efficiency by factor of seven without compromising performance.
Load CLIP
2 clip model, preprocess = Load ("RN50")
3 clip_model_ list
backbone nn. Sequential (* (clip__odel_[:-1]] )
pooling nn. Sequentiaal (clip_model_[-1)
6
7 $ After obtaining several proposals
proposal_bbex = [...] $ xyxy
After obtaining clip backbone feature
11 clipout = backbone (preprocess (image))
ROT bbox feature feeds into pooling from sorchvision.or inport roi_align
14

---

Published as a conference paper at ICLR 2024
outputsize [7
index torch. zeros (proposal_bbox wtwpsaallloxo...pllllpee [C]) uussuueze (-1)
17 interest_bbox = torch. cat (indexx proposal_bbox) dim -1)
Interost_roi_fottur
zoi_align (elpput, Intercst_bbox, outputsizc,
spatial_scale-1e aligned True)
clip_featurs pooling (interest_roo ttaterresttroofeaturre fea_fee)
21 clip_feature clip_feature clip_feature norm (dim=-1, keepdim=True)
Method
AP
Efficiency
SAM+CLIP Crop
3.2
1X
SAM+CLIP Parallel
3.0
7.5X
Table 5: Comparison of AP and Efficiency of Parallel implementation in COCO.
A.3 SELF-TRAINING DETAILS
We provide a detailed implementation for the self-training phase. Note that we adhere to the strategies outlined in CutLER (Wang al. 2023), with the singular exception that employ single round of self-training, in contrast to the three rounds utilized CutLER. By default, we employ the Cascade Mask R-CNN (He al. 2017) with ResNet-50 (He et al. using the self-supervised pre-trained DINO (Caron et al. 160K iterations with a batch size of 16. For data augmentation, we adopt variant of the copy-paste augmentation identical to CutLER, which entails randomly downsampling the mask with a scalar uniformly sampled between 0.3 and 1.0. For the optimizer, leverage SGD with initial learning rate of 0.005 which is diminished by factor of 5 after 80K iterations, weight decay of 5x10 and momentum of 0.9.
2016) backbone. We initialize the weights
2021) model and carry out the process for
A.4 LABEL-EIFICIENT TUNING DETAILS
We start from weight initialization obtained through self-supervised training. To ensure a fair comparison with CutLER, we adopt the same training strategies of CutLER, which adjusts the learning rates according varying quantities of tuning data, as illustrated in Table 6
%
Max Iters
Diminishing Steps
WarmUp Iters
1
3,600
2,400 / 3,200
1,000
2
3,600
2,400 / 3,200
1,000
5
4,500
3,000 4,000
1,000
10
9,000
6,000 8,000
0
20
18,000
12,000 / 16,000
0
30
27,000
18,000 / 24,000
0
Table 6: Summary of training strategies of CutLER and ours.
A.5 DATASETS DETAILS
We have employed two commonly used datasets in object detection and instance segmentation, COCO (Lin et al. 2014) and Pascal VOC (Everingham et al., 2010), as benchmarks. To compare with previous methods, we use the same test sets as previous methods (Wang et al. 2022b 2023 Siméoni et al. 2021 Van Gansbeke et al. 2022). In Table we enumerate the specific conditions of the training and test sets used under different settings. Configuration A and are applied Table 1, while Configuration C utilized in Section 4.4.
15

---

Published as a conference paper at ICLR 2024
settings
training data
testing data
#images
segmentation
COCO tmaal n2017 split
va12017 split
5,000
V
COCO tra n2017 split
trainval0 split
9,963
X
VOC train07 split
va107 split
2,510
X
Table 7: Datasets details under different settings.
B WHY DOES SAM FAIL IN AP
We visualized the ground truth alongside the top 10/30 predictions based on SAM's confidence scores. We discerned that SAM prefers to predict smooth, regular pixel blocks, smaller objects since SAM's edge-oriented property. These objects, in the prevailing dataset annotations, are highly likely not to be the targeted objects for detection, hence SAM yielded comparatively low Average Precision (AP).
A
Ground Truth
Top-10 Prediciton
Top-30 Prediciton
C CLUSTERING WITH SEMANTIC-AWARD INITIALIZATION
Motivation and Observation. Although the activation map ST offers semantic information at the dense level, two serious drawbacks hinder from performing accurate instance segmentation: (1) Rough semantic segmentation results. Since mask-level annotations training, the activation map of CLIP fails to accurately activate the local regions especially object boundaries, resulting in coarse localization. (2) Lack of instance information. As the oranges
never exposed to CLIP for
16

---

Published as a conference paper at ICLR 2024
shown in Figure B] due their close proximity, all instances in single connected semantic segment and cannot be distinguished. To overcome these shortcomings, our key observation is that after clustering the features of a specific middle layer in CLIP, not only can pixels be grouped into ceapper-piels/fragmenne
but what is even more astonishing is that is highly sensitive to object edges, resulting in the formation of object boundaries shown in the Figure Clustering).
Semantic-Aware Initialization. To obtain impressive clustering results via K-Means (Hartigan & Wong 1979), we utilize the activation map initialize the clustering centers and determine the number of clusters as the activation map serves a strong prior where the objects localize. Our principle that active regions are prioritized over those that are far away from them, meanwhile, the cluster centers should be more located near the edges between the active regions and the inactive ones to identify the boundary. To implement the principle, we first convert the activation map into a binary map M by applying a threshold T:
f1, 0.
if S Z mean (S+) otherwise
M
(5)
where mean(-) represents the computation of the average activation score for the activation map, is set as 0.7 for all datasets. Then, we obtain the smallest enclosing bounding box of the region where M and specified to its top left and bottom right coordinates (Xminn Ymin) and (Nmax Ymax). respectively. The bounding box divides the active regions and the inactive ones.
To align with our principle that the cluster centers are expected to be near the edges between the active regions and the inactive ones, we compute the distance dd from each point (i,j) in the map to the bounding box and apply a Gaussian kernel on the distance d3 to obtain the heatmap Y:
d' min ((i Xmin )2, (i Xmax) )2) + min ((j Ymin) )2, (j >mma)))
d
(6)
yiu = cxp
2o
where a is adaptive based on the size of the feature map. The score Y of the heatmap roughly reveals how close a point (i. is to the contours of active and inactive regions. Therefore, we sample the cluster centers based on the normalized heatmap.
Regarding the number of clustering centers K for one image I, expect it be positive correlation to the number of categories of interest appearing in the image because one cluster is expected to be corresponding to some regions of one category. Therefore, we empirically define it as K = max (20, (Q(Z) 1))), where QZZ) is:
Q(I)
Σ TETaB
(S (I, (I,T)>
where Telass is the set of all interest categories.
Clustering. Notice that we first obtain the heatmap individually for each category T and then cluster once considering all categories Tolass The clustering centers are sampled based the probability normalized from the weighted sum on heatmaps for all categories. With our semantic- aware initialization, second-to- last layer of the Ency(-) to obtain K clustering results in C1.K- Each cluster C₁ E C is represented as {0, 1 h,e where Cy means that the point (i, j) belongs to the k-th cluster.
the standard K-Means clustering algorithm on the feature map from the
Analysis of Computational Complexity. Our method more computationally efficient than previous algorithms that require computing the eigenvectors of the Laplace matrix. Our main computation is the K K-means clustering, whose time complexity is times the square of the number of patches. In practical implementation, compared to the previous best persfrrerfrming CutLER under default settings, our algorithm generally proffers 10x increase in processing speed.
C.I CLUSTERING RESULTS ANALYSIS.
Clustering Results Analysis. Figure 5 shows the impact of using CLIP's different layers and K-Means's 's initialization on clustering results. From left to right, the clustering features are derived from shallower deeper layers, with the middle three stemming from the intermediate layer but with
17

---

Published as a conference paper at ICLR 2024
Skadom
Figure 5: The influence of different layers of CLIP and initialization methods for clustering results.
different initialization methods. Layers either too shallow or too deep do not attend to information well-suited for instance segmentation, and the intermediate layer's results are quite stable in the absence of proper initialization. Our method delivers stable clustering and boundary discovery. Since we only find CLIP's ResNet variants can clustering boundaries. further discuss why CLIP's intermediate features have the potential for instance-level localization below.
Hierarchical representation structure of CNN Architecture: The CNN representation structure with progressively increasing receptive fields exhibits pattern where lower layers focus on local and contour details, while higher layers attend to global and semantic information. discussed in [1], ViTs have more highly similar representations across all layers and incorporate more global information at lower layers due to self-attention. As result, ViT's intermediate layers lose instance-level localization information.
Distinctive pretraining approach and training data: The ImageNet supervised pretrained model (ResNet) trained object-center images for single-label classification without the requirement for learning instance-level information on complex scenes. In contrast, CLIP's s pretraining approach utilizes image-text pairs, where the could describe multi-instances present in the image and their relations. Image features must capture instance-level information in complex scenes to match highly with the text.
To prove the boundary clustering results are general in the entire COCO dataset, we further provide qualitative results across the entire dataset. We count the boundary scores of fragments that are and not the object edges on the COCO dataset (see Figure 13 indicating that fragments on object edges indeed have higher boundary scores in general.
300000
Fragments not on Object Edges
14000
Number of Fragments
Mean
250000
12000
Number of Fragments
Fragments on Object Edges
10000
200000
8000
150000
6000
100000
4000
50000
2000
2
4
5
8
Figure 6: Boundary scores of fragments on and off object edges sampled from the COCO dataset. The x-axis corresponds the boundary score.
D ADDITIONAL RESULTS ON PASCAL VOC
In Table g give the annotation-fres object detection results on the VOC dataset. For VOC, given that VOC's training set is also employed for testing, we directly utilize the overlapping 18 categories from COCO for training while maintaining tests 20 classes. Consequently, this is not an "apple-to-ae to-apple" comparison. But considering the non-trivial improvement of 19.7% in AP, it's evident that approach boasts robust dataset transferability capabilities.
18

---

Published as a conference paper at ICLR 2024
Class-aware
VOC Box
Model
Self-Train
AP
AP50
AP75
DINOv2
(Qquab et al.
2023
0.9
2.8
0.4
TokenCut
Wang ct a..
20226
4.7
8.8
4.3
CutLER
Wang ct al.
2023
5.4
9.9
5.8
Zip (Ours)
16.5
25.5
17.3
vs. prev. SOTA
+11.1
+15.6
+11.5
CutLER (Wang
2023
7.6
13.3
7.4
Zip (Ours)
27.3
49.4
26.9
Class-agnostic Model
VOC Box
Self Belf-Train
AP
AP50
AP75
DINOv2 Oquab 2023
0.7
2.5
0.3
TokenCut Wang
3.6
6.7
3.2
CutLER Wang 2025
8.5
10.2
6.6
Zip (Ours)
14.1
20.4
14.5
Vs. prev. SOTA
+5.6
+10.2
+7.9
LOST (Siméoni et al.
2021
6.7
19.8
FreeSOLO (aang (Wans ct al. 2022a
5.9
15.9
3.6
CutLER (Wang et al. Zip (Ours)
2023
20.2
36.9
19.2
20.7
37.7
20.3
Table 8: Annotation-free object detection on VOC. For fair comparison, we keep the same settings as the previously best-performing CutLER in the self-training setting, and Zip outperforms all previous methods on all evaluation metrics.
E Is SAM A FREE LUNCH?
SAM (Kirillov et al., 2023). as pre-trained foundational model for segmentation, capable of accepting a variety of prompt inputs. One straightforward application of SAM involves its deployment refine relatively coarse masks. We prompt SAM in steps, in the first step, we prompt the SAM with the original predicted bounding boxes. For the second step, we prompt the SAM by the center points of boxes that output from step one. However, in our attempts, we discovered that employing SAM for refinement does not invariably yield favorable results, shown in Figure 7 In the case illustrated in Figure a SAM aids in refining the object boundaries and replenishing any deficits in the object. However, in Figure 7b)), despite Zip providing generally accurate results, the outcomes, post SAM refinement, either overflow or become a part of the object. We hypothesize the plausible cause to be that SAM lacks the concept of object level, which could potentially lead to the phenomenon the primary subject being lost. Despite our empirical evidence demonstrating that SAM can generally enhance segmentation results, model's 's superior performance does not mainly come from SAM's segmentation capabilities. However, the question of how to utilize SAM more effectively remains open-ended.
SAM
(b)
Figure 7: The impact of SAM refinement. (a) SAM refinement has elevated the mask AP. (b) SAM refinement has diminished the mask AP.
19

---

Published as a conference paper at ICLR 2024
F WHY CLASSIFICATION THEN DISCOVERY
In our experiments, we have found that when using CLIP for zero-shot object classification, it is highly susceptible interference from in-context information the images. In Figure 8 we have illustrated an example called the "cat-car dilemma" phenomenon. When cat is sitting on car, the proposal for the car inherits the in-context information of the cat. In supervised setting, the model might learn that because the bounding box is tighter around the car, it should be classified as car. However, in the zero-shot scenario, we observe that the score for the "cat" category is higher. This phenomenon may be related to biases present in CLIP. How we address this phenomenon is first to perform per-pixel classification. This approach separates the semantics of the cat and the car. Subsequently, generate masks based on semantic priors, effectively avoiding the cat-car dilemma.
Cat? or Car?
Proposal Extract
Crop and Feed to CLIP
Figure 8: The cat-car dilemma.
G A CLOSER LOOK AT COCO 80 CATEGORIES
We observe performance drop from the novel category (27.1%) to all COCO 80 categories (20.0%). Due to the model's ability to recognize objects stemming from CLIP, in this section, will randomly select 100 objects from each category on COCO validation set. These objects will be cropped and fed into CLIP. Similarity scores with text embeddings from each of the 80 COCO categories will be computed, resulting in the following Figure The blue line represents novel objects. We observe that CLIP's recognition of categories is uneven. For certain novel categories such "cat" and "dog." CLIP exhibits a recognition probability exceeding 90%. Surprisingly, for the common category "person", CLIP demonstrates minimal familiarity, with an accuracy rate falling below 20%. The average accuracy in the 18 novel categories is 49.8% and 43.2% for all categories, which aligns with the performance trend of our model after self-training.
A Closer Look at COCO 80 Categories
m50
A20
Figure 9: CLIP accuracy on 80 categories.
20

---

Published as a conference paper at ICLR 2024
H QUALITATIVE RESULTS
provide qualitative results of unsupervised Zip in Figure 10 ambiguity of the scene, Zip consistently performs commendably.
Regardless of the complexity or
Figure 10: Visualizations of Zip's predictions under a diverse array scenarios.
21

---

Published as a conference paper at ICLR 2024
1 COMPARED WITH WEAKLY SUPERVISED SEMANTIC SEGMENTATION METHODS
Since CLIP was not designed for per-pixel classification, we have explored alternative weakly supervised semantic segmentation method (Xu et al. 2022 [L et al. (023) to serve as semantic priors. However, since the clustering results from other networks do exhibit the boundary phenomenon
we have identified, the final outcome closely aligns with using CLIP alone for semantic prior. The relatively lower performance also indicates the challenge of achieving instance segmentation solely based on semantics. We also supply the outcomes achieved by substituting DINO features in place of CLIP's. The results demonstrates that the CLIP's boundary phenomenon important for object detection.
Semantic Method
AP50
100
Group ViT (Xu et al.
2022
SAM-B
5.5
7.2
CLIP Radford al. 2021)
+ SAM-B
2.2
3.6
CLIPSurgery (Li et al.
2023)
+ SAM-B
4.7
6.9
CLIP + DINO SAM-B
6.1
8.8
Ours
27.3
32.8
Table 9: Comparison of different Semantic Segmentation Method on Pascal VOC.
J LIMITATIONS
Zip achieves twice the performance of previous methods under identical settings without employing any annotations. However, there remains certain gap when compared with fully supervised methods. We have attempted to bridge this gap by utilizing more training data and conducting multiple rounds of self-training. However, experiments have revealed that training the model with initial objects obtained from unsupervised methods tends to trap the model as sub-optimal. The misalignment between training objectives and ground-truth annotations hinders the further improvement of self- training. Addressing this issue is potential future work that may require the inclusion of ground-truth examples.
J.1 ETHICAL CONSIDERATIONS AND POTENTIAL NEGATIVE SOCIAL IMPACTS
Our open-vocabulary prowess is entirely predicated upon the pre-trained foundational model, CLIP. However, given that CLIP was trained on a dataset assembled from the internet, any undesirable biases may still potentially permeate our models despite data cleansing. Any biases within the pre- trained model should be mitigated before employing our method for downstream tasks to circumvent deleterious influences. Furthermore, the detector vocabulary flexible and can manipulated to display racial bias during human detection. For instance, the text prompt can be set as specialized biased descriptions, which is a critical factor to be taken into account.
K LICENSE DETAILS
All utilized codes and licenses are cataloged Table 10 ShareAlike, CCA: Creative Commons Attribution.
ANCSA: AltributionnNoonCommeettt
Table 10: The used Datasets/Model Weights/Codes and License.
Datasets/Weighs//Coodee
citations
License
COCO
Lin et al. 2014
CCA 4.0 License
PASCAL VOC
Everinghmm et al. 2010
NO License
Radford 2021
MIT License
Caron al. Wang et al. 2023
Apache License
ANCSA4.0
22

---

Published as a conference paper at ICLR 2024
L UNDERSTANDING THE PROCESS OF CLUSTERING AND FRAGMENT
SELECTION
Middle-level
(a) Image
(b) Semantic Activation
Binary Map M in Equ.5
(d) Contour of the oranges
Clustering with Semantic-
Map
aware Initialization
Outer
edge
The cluster centers should be located near the edge of the map M identify the boundary feature, Equ.6.
"Boundary" Clusters
boundary
Objeck
fidentified via boundary
score metric in Equ.3)
()) Refine each individual object SAM
(i) Run connected graph algorithm on the segments based on their connections.
(h) Adjacent Segments are connected with each other if both of their Xg So, two segments are connected but NOT tto boundary's X boundary.
(g) Two Segments
(f) Boundary WITHIN the Semantic Activation
belong to the same
class and
the
Region
boundary, i.e, their Xx = 1 in Bqu.4. The
Figure 11: The visualization for an explanation of the clustering motivation and process.
The Importance of Sewantic-Aware Initialization and Its Motivation: Semantic-aware initial- ization leverages the semantic activation map automatically initialize clustering centers and determine the number of clusters.
Importance: Without our semantic-avee aware initialization, K-Means fails to produce general and reliable clustering results for delineating object boundaries. As shown in Figure despite clustering the same intermediate features in CLIP, it is only when utilizing the proposed initialization technique that stable clustering and boundary discovery are achieved.
Motivation: By initializing certain clustering centers near the outer edges of the semantic activation map, boundaries between objects within the activation map can be clustered and outlined. This is because whether they are outer edges or inner object boundaries, they are all edges with high feature similarity.
The explanation of clustering process:
We convert the activation map into a binary map M using Equation as shown in Figure |I| c). This conversion enables us make preliminary estimation of the outer edges of the activation map, as demonstrated in Figure (T..). While the outer edges may not distinguish individual instances of oranges, they consistently delineate the boundaries between the cluster of oranges and the background. Our intention is to leverage the edge features these outer edges to aid in identifying additional edges, such those between oranges, similarity between the outer edges and the oranges' boundaries.
hypothesize significant
Therefore, we compute the heatmap Y in Equ. 6 roughly reveals how close a pixel (i.j) is to the outer edges. Then, sample the cluster centers based on Y, so that the cluster centers can be initialized by outer edges' features, as shown in Figure dd). For the number of clusters, we roughly estimate it based on estimating the number of object categories in single images through Equ. 7 reality, our method only needs to set the number of clusters within a certain range (15 to 30) work effectively, thanks to our reformulation instance segmentation by fragment selection.
23

---

Published as a conference paper at ICLR 2024
Our method is even robust to ver-seegentttion caused by slightly larger number of clusters (as long as it is not too small) because our subsequent fragment selection can regroup over-segmented fragments into instance.
By performing the K-Means clustering algorithm with the above initialization, the clustering results shown in Figure (T)e e) are obtained, with each cluster represented by a distinct color. The "boundary" clusters also be identified via the boundary score metric in Equ. 3
In summary, we accomplish the transformation from the outer edges of the activation map to the boundaries of individual objects within the activation map!
The process of fragment selection is visualized in Figure 11 (ij for more intuitive understanding of the Section 3.4
M CLUSTERING CLIP FEATURES ON THE HUMAN PART SEGMENTATION
BRERALING
BOUEK
Figure 12: Clustering CLIP features on the Human Part Segmentation.
We test the clustering results by CLIP (Rafford et al., 2021 on images from the Pascal VOC Part (Everingham et al., 2010) in Figure 12 The experimental results demonstrate that using clustering method on CLIP middle-layer features yields robust clustering effects across different domains.
The boundaries of individuals remain clear. Despite the variation in image domains. transitioning from COCO common objects human-centric images, the boundary information of objects or individuals remains well-preserved. This highlights the robustness of CLIP features and our Zip.
More surprisingly, the clustering results align well with human part semantics, including features like the nose, mouth, eyes, arms, and hair. This may also unveil the potential for using CLIP features to aid part segmentation in future research.
24

---

Published as a conference paper at ICLR 2024
N
COMPARISON BETWEEN DINOV2 AND CLIP FEATURE
Image
CLIP
DINO
(a)
(b)
(c)
(d)
(e)
Figure 13: In the comparison of clustering the boundaries of "people" between the CLIP middle-layer feature and the DINOv2 feature, all clustering settings are kept the same.
We compare the results of the CLIP-middle layer and DINOv2 middle layer under the same clustering method. In nutshell, DINOv2's clustering results appear smoother, with larger contiguous blocks sharing similar semantics, but it lacks instance-specte results are comparatively "noisier" yet provide a clear delineation of object boundaries. As shown in (a)-(c,, the boundaries of all individuals ("person") are clustered, with pink in (a), red in (b), and green in (c). As shown in (d)-(e), CLIP's features focus on more inconspicuous objects, such as the person in the television (d) and the three individuals in (c).
information. On the other hand, CLIP's
O COMPARISON BETWEEN OPEN CLIP AND OFFICIAL CLIP
(a) (altention pooling layor)
(b) Convnext (global average pool)
(c) Convnext Large (global average pool)
Open CLIP
(d) Modified RN101 (attention pooling layer)
(0) Modifed RN50 (attention pooling layer)
(1) Modified RN50x64 (attention pooling layer)
Official CLIP
Figure 14: In the comparison of clustering the boundaries of "people" between the official CLIP middle-layer feature and the Open CLIP feature, all clustering settings are kept the same. Boundary color also show in
We compare the results of the official CLIP-middle layer and Open CLIP middle layer under the same clustering method in Figure 14 Only model with modified attention pooling layer outline boundary. In official CLIP, attention pooling with the [CLS] token is used to obtain global image features, offering attention maps for free and allowing features with low attention scores to focus on aspects beyond "category" and semanties, such as boundaries. However, Open CLIP uses global
25

---

Published as a conference paper at ICLR 2024
average pooling acquire global image features, emphasizing that features from different spatial positions pay more attention to aspects related to "category" and semantics.
26


Reviews:
[{"review_id": "5zgN5bvK2B", "reviewer": "Reviewer_11pM", "summary": "Authors propose an annotation free instance segmentation through combining CLIP and SAM, where the authors claim that CLIP has a better capability of obtaining the boundaries. They evaluate their method on COCO and PASCAL VOC datasets. Their framework named Zip outperforms some of the unsupervised annotation free methods from SOA.", "questions": "- Table 3 analysis on the architecture its not clear how the AP50 climbed to 44.9%, I am not sure how did this happen and is this still annotation free? Why it is not the final results compared in Table 1.\n\n- F.g 6 C.1 its not clear what’s the x-axis?\n\n- How did you retrieve CutLER results in Table 1? Why is there one class aware and class agnostic?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- interesting results in Table 1 and Table 2 improving over the compared methods\n- Interesting direction to work on establishing methods for annotation free instance segmentation", "weaknesses": "- Weak novelty as the method is simply utilizing other foundation models without proposing anything specific to the instance segmentation task. The clustering techniques are not really showing anything novel in its formulation. Novelty is more in the pipeline.\n\n- There are no quantitative results to support their claim that DINO is worse than CLIP in identifying the boundaries as far as I have seen but if there are please clarify in the response. Also is CLIP better than Up-DetR or better than DINO, DINOv2 in identifying boundaries. Meaning if they apply same clustering and everything on these models' features how would it fair against CLIP.\n\n- Up-DetR is not clearly stated in the related work and I am wondering if the authors have inspected its use\nDai, Zhigang, et al. \"Up-detr: Unsupervised pre-training for object detection with transformers.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n- Equation 3 and the method is not quite clear for example when referring to performing inner product of matrices. But there is no mention of the matrices shapes which can help clarify ambiguities in the mathematical formulation.\n\n- The method is quite heuristic depending on hyper parameters used in Eq. 4 theta_1, theta_2.\n\n- Their results are quite low in Table 1 compared to for example SAM when coupled with ViTDet as reported in original SAM paper. It has to be highlighted on why the previous results weren’t compared to. For example if it was not fully annotation free in SAM paper, please detail that in the paper then.\n\n- Typos needs to be fixed e.g. “classification-frist-then-discovery “", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cxAeQOrRHl", "reviewer": "Reviewer_g6s6", "summary": "This paper introduces a novel approach to annotation-free instance segmentation. The authors identify that a specific middle-level feature in CLIP encapsulates instance-level boundary information. Leveraging this insight, they develop a pipeline named \"Zip\" for annotation-free instance segmentation. The paper also pioneers a \"classification-first-then-discovery\" paradigm. Through extensive experiments, the proposed method significantly outperforms its baselines.", "questions": "1. **Semantic-Aware Initialization**: Could you provide a more in-depth explanation of the Semantic-Aware Initialization mentioned on page 17? Specifically, I'm unclear about how the parameters for K-means clustering are determined.\n\n2. **Visualization in Figure 2**: The differentiation between gray and orange in Figure 2 could be clearer. Enhancing the contrast by making the background color more transparent might help in better distinguishing the two.\n\n3. **Ambiguous Object Representation**: How does the middle-level clustering handle images where the concept of an object is ambiguous? For instance, in scenarios like a box with a person's image on one side or stacked blocks. In such cases, it's debatable whether the person's image should be considered a separate instance or if each block in the stack should be individually segmented.\n\n4. **Handling Occlusions**: How does your method address situations where an object is partially occluded and appears as two separate parts? For example, if a dog is behind a tree, would your system recognize both parts as belonging to the same object?\n\n5. **Locating Middle-Level Features**: I'm intrigued about the process of pinpointing the middle-level feature. Is the method you've employed reproducible across different training weights? For instance, would the results be consistent if you were to use the openCLIP model (or the official CLIP model, if you've already utilized openCLIP)?\n\n6. **Exploring DINO/DINOv2 Features**: Have you considered using the middle-level features from DINO or DINOv2? Given that MaskCut can discern between closely packed objects using DINO features, I wonder if DINO/DINOv2 might also possess a middle-level feature adept at boundary extraction.\n\n7. **Resolution of Middle-Level Feature**: The resolution of your middle-level feature appears to be quite high. Could you specify the image input size and the dimensions of your middle-level feature?\n\n8. **Self-Training Visualization**: It might be beneficial to include a comprehensive visual representation of the self-training process. Relying solely on textual differences compared to CutLER and directing readers to refer to CutLER might not be the most user-friendly approach.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The utilization of CLIP's middle-level feature to generate instance proposals presents a compelling and novel approach. This stands in stark contrast to traditional RPN-based or embedding-based proposal generation methods, offering a fresh perspective in the realm of annotation-free instance segmentation. It also provide comprehensive experiment results to prove its leading performance.", "weaknesses": "1. **Figure 1's Layout**: The sequencing in Figure 1 is somewhat perplexing. A more intuitive order, such as DINO, Prev. SOTA, SAM, Clustering, and then Ours, might enhance clarity. Additionally, specifying the exact method when referencing \"Prev. SOTA\" would provide better context to the readers.\n\n2. **Figure 2's Clarity**: The visualization in Figure 2 appears cluttered, making it challenging to discern the boundaries in images labeled A through D. Simplifying or enhancing the contrast could make the distinctions more evident.\n\n3. **Understanding Figure 3**: Figure 3 is somewhat intricate, especially when trying to comprehend the roles of \"activation\" and \"boundary.\" A more detailed caption or a supplementary explanation might aid in understanding.\n\n4. **Discussion on DINO and SAM**: While it's essential to provide context, the extensive space dedicated to discussing the limitations of DINO and SAM might be excessive. Streamlining this section could make the paper more concise and focused on the primary contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cMaye1qZlF", "reviewer": "Reviewer_WVZD", "summary": "The authors analyze the weakness of the previous computer vision foundation models. By discovering the instance-level understanding of CLIP, the authors then utlize it to propose a novel training-free method to address the issues by clustering the activation maps from CLIP. Zip demonstrates significant performance improvements compared to the previous state-of-the-art on the COCO dataset across various settings.", "questions": "- Q1. The work is super interesting, but it takes me a long time to understand how the clustering works.\n   - What do the colors represented in \"Clustering\" in Fig.3?\n   - How does the clustering algorithm draw the boundaries in Fig.2 and Fig.3?\n   - Taking \"Semantic Clues\" in Fig. 3 as example, what will it look like after applying Eq. 5?  I assume that there will be a single box surrounded all the green parts? How can instances be seperated after that?\n   - The authors stated that after obtaining the feature maps for each category, it will then run the clustering algorithm. Please clarify how to run the algorithm and why it will work.\n- Q2. How stable is the clustering algorithm? Will the results change drastically if optimal hyperparameters are not found? For example, which intermediate layer should be used? Is there any experiment on how the choice of $K$ will affect the results? Are there any guidelines? Please also provide some insights for Eq.7.\n- Q3. Is it appropriate to fully adopt the setting from CutLER? To my understanding, CutLER does not involve any annotation during training. While Zip also does not use annotation from COCO, but it did exploit the language information from CLIP. \n- Q4. Is it possible to compare Zip to other open-vocabulary works, such as Openseed[1]? I am aware that Zip is annotation-free, and the authors have stated that it will be a future work to utilize annotations at Section J. However, the work would be more convincing and useful if it is possible to somehow utilize annotations from COCO since the raw performance of Zip is still significantly behind Mask-RCNN.\n\nI belive it is an interesting and promising work. However, the explanation of clustering is not clear enough. I am willing to raise my score if the authors can reasonably address my concerns and questions.\n\n[1] Zhang, Hao, et al. \"A simple framework for open-vocabulary segmentation and detection.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The idea is novel, inspiring and useful.\n2. A detailed analysis of the issues in previous works is provided.\n3. Significant improvements over previous state-of-the-art on various COCO settings.\n4. The authors are aware and thoroughly investigate the weaknesses and limitations of the work.", "weaknesses": "1. It is challenging to follow the explanation of how the clustering works. It would be helpful if the authors could include some figures to illustrate this process more clearly.\n2. The robustness of the clustering algorithm is concerning when applying to other datasets or settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6mZaPecx5I", "reviewer": "Reviewer_5mwu", "summary": "The authors propose a method called ZIP, which\ncombines CLip and SAM in a novel classification-first-then-discovery pipeline, enabling annotation-free,\n complex-scene-capable, open-vocabulary object detection and instance segmentation.\n\nZip establishes state-of-the-art performance in various settings, including training-free, self-training, \nand label-efficient finetuning.\n\nThe overall idea is based on a discovery that CLIP can provide a highly\nbeneficial and strong instance-level boundary prior in the clustering results of its\nparticular intermediate layer.\n\nThe proposed method combines the CLIP classification score with the SAM localization score using a geometric mean. \nThis refinement boosts the confidence scores of masks, achieving a slightly higher precision.", "questions": "Page 2 - the line:\n....higher precision yet not satisfaction enough.\nmay be corrected (English).\n\nYou state - \"classification-first-then-discovery pipeline\"; in page 3:\nWhat if the reverse is done ? Edges in images, as boundaries of objects, may hypothetically lie\non Decision boundaries between binary classes. So discover the edges first may be a better approach, rather than\nrelying on classification to provide those edges ?\n\nDino (also Clip?) uses transformers (ViT) , right ?\n- is that the reason for the need for large computational resource to solve your problem,\nwhen you combine the two?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Good ZSL model proposed- combining CLIP and SAM, and giving semantics meaning to edges.\n\nLarge amount of experimental results, exhibiting superiority with SoA, ablation studies etc\nprovided for benchmark datasets.\nIllustrations of results are vivid and eye-catching -= say in Fig. 10.", "weaknesses": "The statement used - The devil is in the object boundaries;\nis it not in line with a part work, which hypothesized - \"all you need are priors\"?\n\nAlthough the  analytics presented in Eqns. (1) - (7) are clear, unable to find much of novelty in them.\neg Cosine similarity, boundary score etc.\n\nWhat about inference ? How many GPUs are required, and what is the timing then ?\nI could not locate that information in document.\n\nA generic opinion:\nThe recent trend of using a large GPU cluster to implement detectors/segmentors is something of a concern, I believe.\nIts like exploiting massive computing power to solve a problem. Either use of a large set of activation/attention heads, \\\nor for pre-training tasks in self-supervision, meta-learning are the main reasons of using so. What is the limit now ? \nThe focus is shifting from devising novel learning algorithms to use of larger to mega-clusters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gEgOqNhbtA", "reviewer": "Reviewer_CLaV", "summary": "This paper proposed the annotation-free instance segmentation method using vision foundation models (i.e., SAM and CLIP). They found that SAM shows high recall rates but relatively low precision. Also, DINO is suitable to obtain salient regions but is not suitable to obtain instance-level masks. To address the challenges, they discovered that clustering the features of CLIP's specific middle layer can be effectively used for adequate prompts for SAM. The class of each instance mask is classified using the CLIP. Consequently, the proposed method outperforms the previous state-of-the-art annotation-free instance segmentation methods. In addition, it shows a competitive performance to existing open vocabulary object detection methods without any annotations.", "questions": "Due to some concerns in the Weakness part, my initial rating is borderline.\nAfter a discussion with the authors and other reviewers, I will finalize the rating.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "## 1. Great motivation and findings.\n\nI strongly agree with the motivation of this work.\nNamely, SAM itself may not be suitable for instance segmentation on COCO due to the high recall and low precision rates. \nAlso, DINO may be suitable for salient object detection, but not for instance segmentation.\nMotivated by the limitations of vision foundation models, this paper designed a new pipeline for annotation-free instance segmentation using fine-grained features from CLIP and SAM.\n\nIn addition, the proposed classification-first-then-discovery pipeline is convincing in resolving CLIP's misclassifying issue.\n\n## 2. Well-structured and well-written paper\n\nI enjoyed reading this paper because the motivation is clear and understandable, the proposed pipeline is well-explained with proper illustrations.\n\n## 3. Outstanding performance\n\nCombined with CLIP and SAM, the proposed method outperforms the existing methods by a large margin on COCO dataset.\nEach proposed component is well-ablated.", "weaknesses": "## 1. Limited technical novelty\n\nI feel that leveraging the features on particular middle layers of CLIP and applying K-Means clustering on CLIP's features are not technically novel but well-engineered.\n\nOf course, the proposed annotation-free instance segmentation pipeline is interesting.\nThis paper seems to be an engineering paper that addresses how to effectively use CLIP and SAM and is not far from an academic paper.\n\n## 2. Behind the outstanding performance.\n\nI think the key feature of the proposed method is how to properly cluster objects in an image (because we regard that SAM can segment anything when bounding box prompts are properly introduced).\nAt first, I was surprised by the outstanding performance of the proposed method on COCO 2017 dataset.\nHowever, there is a question that the outstanding performance was due to the well-aligned semantic domain between the features of CLIP and the COCO dataset; the clustered object boundary from CLIP may be well-aligned to the ground-truth object region in COCO dataset.\nI wonder if the target dataset was human part segmentation or medical segmentation, the CLIP features are still valid for annotation-free instance segmentation.\nIt would be great if this paper discuss it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)