Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels

Authors: heeseong  shin
chaehyun  kim
sunghwan  hong
seokju  cho
anurag  arnab
paul hongsuck seo
seungryong  kim
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:45:24

Abstract: Large-scale vision-language models like CLIP have demonstrated impressive open-vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which require understanding where the objects are located. In this work, we propose a novel method, MCLIP, to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where, which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. MCLIP shows significant performance improvements over CLIP and competitive results compared to caption-supervised methods in open-vocabulary semantic segmentation.

Main paper: 
 introduction: 
 Semantic segmentation is a fundamental task in computer vision where the goal is to identify class labels for each pixel within the given image. However, segmentation datasets often require extensive human effort to obtain densely-annotated semantic labels, limiting their scalability. In this regard, recent advances in large-scale pre-trained vision-language models, e.g. CLIP [15] and ALIGN [16] , We visualize the masks generated by SAM [13] and by clustering image features from DINO [14] . Although such models can freely generate fine-grained masks, the resulting masks can be too small or incomplete to have semantic meaning. To address this over-segmentation issue, we employ online clustering [18] of the masks into semantically meaningful groups defined globally for given images.have facilitated open-vocabulary semantic segmentation [1, 3, 2, 17, 4, 6] , which aims to generalize semantic segmentation into unbounded range of classes. Despite showing remarkable generalization capabilities, they still require pixel-level semantic labels for leveraging the image-level pre-trained vision-language models for semantic segmentation.Recently, several studies [11, 12, 7, 8] have pioneered open-vocabulary semantic segmentation without densely-annotated semantic labels. These studies often utilize image-level semantic labels, such as image captions, to enhance the pre-trained vision-language models like CLIP for semantic segmentation. However, image captions typically provide information about what is in the image, but without where it is. Since CLIP is already effective in recognizing what the objects are, this causes models to only implicitly learn object locations, leading to sub-optimal performance or requiring millions of image-caption pairs to compensate for this weak supervision [8, 7] . Instead, we focus on informing CLIP about where objects are located to address the missing information.In this study, we propose a novel approach to achieve open-vocabulary semantic segmentation without leveraging semantic labels, but through guiding the pre-trained vision-language models, such as CLIP, on where to look. We leverage recent vision foundation models (VFMs), such as DINO [14] and SAM [13] , to partition images into fine-grained regions to indicate where to look. Consequently, we explore methods to effectively leverage these masks for fine-tuning the image encoder of CLIP.In contrast to existing works that leverage semantic labels [6, 19, 7] , we do not have any captions or class names that can be fed to the text encoder of CLIP. To leverage its knowledge, we devise a method that employs prompt learning [20, 21] on the text encoder of CLIP to construct learnable classes. Setting the learnable classes as a centroid, we propose applying the online clustering algorithm [18, 22] along the given masks to gather them into semantically meaningful groups, as shown in Fig. 2 . We keep these learnable classes global across the entire images, which guides the learnable classes to contain the general semantic concepts. Despite the absence of semantic labels, our method is able to jointly leverage the image encoder and text encoder of CLIP during training, successfully achieving dense open-vocabulary recognition.Our framework, called PixelCLIP, achieves significant improvements to CLIP, on average of +16.2 mIoU in open-vocabulary semantic segmentation. Moreover, despite not using any semantic labels, PixelCLIP shows competitive performance in comparison to image-level supervised methods using captions [7, 9, 10] , demonstrating the effectiveness of unlabeled masks for. We further show the effectiveness of PixelCLIP for classifying masks from various open-vocabulary segmentation models, which can be simply done by replacing the CLIP within existing methods. We also provide extensive ablation studies to validate our choices, with a detailed analysis of our method.We summarize our contribution as follows:• We propose a novel formulation of learning from images without semantic label for openvocabulary semantic segmentation by leveraging masks generated from DINO and SAM to fine-tune vision-language models. • We propose to globally cluster semantically similar masks by employing an online clustering algorithm, while learning class prompts for representing semantic clusters. • We demonstrate significant gains in open-vocabulary semantic segmentation, even surpassing methods leveraging image-level semantic labels, and provide thorough ablation studies with analysis to validate our framework.2 Related Work 
open-vocabulary semantic segmentation: 
 Open-vocabulary semantic segmentation [2, 23] aims to label each pixel within an image into an unbounded range of classes. In this regard, recent works [1, 17, 2, 6, 24] aim to generalize to classes unseen during training through leveraging pre-trained vision-language models, such as CLIP [15] . Despite their remarkable performance, they leverage per-pixel semantic labels during their training, which requires expensive cost to annotate. Instead, we focus on the weakly-supervised setup, where the goal is to zero-shot transfer to segmentation task without densely-annotated class labels [11, 12, 25, 5, 26, 7, 27] , utilizing image-level labels as supervision or even no labels at all.In this regard, recent studies [11, 12, 25, 5] leverage image caption as supervision. GroupViT [11] and ViL-Seg [12] are pioneering works for identifying groups or clusters emerging from captions.Along with the advance of vision-language models, SegCLIP [9] and TCL [7] leverage pre-trained CLIP and learn additional decoder modules to learn dense vision-language alignment. PACL [8] learns additional embedding layers to enhance the patch-level alignment in vision-language models and SAM-CLIP [10] attempts to merge SAM [13] and CLIP [15] into a unified model by additionally leveraging unlabeled mask data from SAM. Apart from these approaches, we avoid employing any semantic labels [26, 28] , but leverage vision foundation models to obtain masks as a source for supervision to fine-tune the CLIP image encoder for achieving open-vocabulary semantic segmentation.Recent large-scale pre-trained vision-language models have shown its effectiveness for jointly understanding images and language [29, 15, 16] . Notably, CLIP [15] , trained with web-scale image-caption pairs, has been widely popularized for transferring its open-vocabulary recognition capabilities to various downstream tasks [26, 30, 31, 32] . However, despite its success in image-level tasks like image classification, CLIP tends to struggle in dense prediction tasks [17, 26, 6] , such as object detection and semantic segmentation. This originates from CLIP being trained from image-level supervision being captions, hence exhibits bias towards the global image rather than fine-grained regions within the image [17] . While non-learnable approaches, such as MaskCLIP [26] show improvements by slightly modifying the architecture, CLIP still shows limited capabilities in dense predictions in comparison to its global understanding.To address this, OWL-ViT [33] directly fine-tunes pre-trained vision and text encoders to downstream open-vocabulary detection task, and CAT-Seg [6] introduces a cost aggregation scheme for finetuning the encoders of CLIP for semantic segmentation. Alternatively, ZegCLIP [19] and Xu et al. [3] implement prompt tuning [21, 20] for tuning the image and text encoders of CLIP. Instead of fine-tuning the full model, they learn prompt tokens that serve as a global prefix for the encoders of CLIP. While such methods show remarkable results from fine-tuning the encoders of CLIP for dense downstream tasks, they require densely annotated detection and segmentation data for training.With the advent of large-scale learning enabled by scalable vision backbone architectures [34, 35] and vast amounts of data, diverse vision foundation models are emerging in the field of computer vision. In this regard, self-supervised methods [36, 37, 38, 39] have demonstrated the effectiveness of its rich visual representations for various downstream tasks. Especially, DINO [14] exerted strengths in fine-grained semantic recognition [40, 41] , making it highly effective for object detection and image segmentation. Moreover, DINO features have also been demonstrated for yielding fine-grained masks within the image through applying the k-means clustering with its features [42, 43, 44] .On the other hand, the segment anything model (SAM) [13] has demonstrated its capability for generating fine-grained, high-quality segmentation masks for any object in an image. Through its selfannotation pipeline, SAM has collected an unprecedented amount of mask annotation for achieving its capabilities. While we can freely leverage SAM to obtain detailed masks in any given image, we mainly utilize the pre-computed masks within the collected dataset, SA-1B. Both DINO and SAM, however, yield unlabeled masks without semantic labels as both models are also trained without semantic labels, presenting a challenge for leveraging their masks for achieving dense vision-language recognition. ℇ ! " ℇ ! 𝑓 ! 𝑓 " 𝑓 # ℇ $ 𝒟 ℇ ! ℇ $ ̅ 𝑓 ! 𝑀 !# & 𝑀 𝑀 𝐼frozen parameters M masked pooling C cosine similarity learnable parameters  
methodology: 
 In this section, we first establish our problem formulation of learning dense vision-language alignment from images paired with masks, generated from vision foundation models. Next, we discuss the challenges of leveraging masks as supervision for fine-tuning the image encoder of CLIP and finally, present our methodology of semantic clustering of masks to address the challenges.Given an input image I ∈ R H×W ×3 , open-vocabulary semantic segmentation [6, 7] aims to label each pixel within an image with classes given in free-form text. As a training signal, semantic labels offer a set of S textual descriptions for a semantic class T = {T i } S i=1 related to I. This can be directly utilized with the CLIP text encoder E L (•) to obtain text features f T = E L (T ) ∈ R S×d , where d is the hidden dimension. Dense image features f I = E V (I) ∈ R h×w×d , where h × w is the output feature resolution, are then extracted. We finally obtain dense image-text similarity map M IT ∈ R h×w×S :EQUATIONThis can be interpreted as soft binary masks predicted from image and text features of CLIP, and be supervised with binary mask loss L mask in a pixel-level manner to fine-tune CLIP [6] .In this work, we do not have any access to T , but are only given unlabeled masks M ∈ R H×W ×N , where N denotes the number of masks for the given image I. Hence, we devise methods to predict masks by incorporating M into CLIP features. We aim to fine-tune the CLIP image encoder E V (•) through leveraging unlabeled masks M as supervision. Since M is generated from vision foundation models, e.g. DINO or SAM, this presents us with the challenge of not having any semantic labels.In order to integrate masks into CLIP, a straightforward approach would be employing the masks M with the CLIP image feature map f I to obtain per-mask CLIP features. While there could be various methods to extract regional CLIP features [26, 45, 5] , we apply mask pooling over f I to obtain maskEQUATIONThis allows us to supervise the model with a binary mask loss L mask for fine-tuning CLIP with given image I and unlabeled masks M . In practice, since M IM has the same resolution as the feature map from the CLIP image encoder f I , we employ a light-weight decoder D to mitigate the resolution gap between M IM and M , as shown in Fig. 3 . This can be written as D : R h×w → R h ′ ×w ′ , where h ′ × w ′ is resolution for the upsampled mask. Therefore, the output of the model can be updated as M = D(M ).Upon using mask pooled CLIP image features f M to predict M IM , however, we find the masks generated from DINO and SAM to often over-segment the image, resulting in too small or incomplete masks as seen in Fig. 2 . This would require CLIP to forcefully discriminate regions that are semantically similar, impeding the training process.In this regard, we propose to group semantically similar masks into clusters and predict based on the clusters rather than individual masks. Moreover, we aim to define this cluster globally, which is shared across the entire training process rather than for each image or iteration. This would be analogous to constructing pixel-level semantic labels, where a fixed set of classes defined over the dataset is equivalent to each cluster. However, the difference is that there is no pre-defined set of classes that we can define the clusters with. While we could heuristically pre-define such classes, we describe our learnable method for globally clustering masks into semantically meaningful groups.Online clustering via learnable class prompts. To globally cluster masks into semantic categories, we propose representing these clusters using CLIP text features as centroids for clustering mask features. Given that the CLIP text encoder is trained with a broad understanding of natural language semantics, we expect these clusters to capture meaningful semantics by leveraging its comprehensive pre-trained knowledge. In this regard, we take a learnable approach, where each cluster is defined by class-specific learnable prompts fed into the CLIP text encoder. Unlike existing prompt learning methods, which typically focus on learning a task-specific prefix [20, 21, 3] , we aim to learn prompt tokens that represent each class. For instance, in the sentence "A photo of an object", traditional prompting methods would learn the tokens for the "A photo of a" prefix, whereas our method focuses on learning the token for the "object." Specifically, given the number of clusters k, we can define prompt tokens as C ∈ R k×l×de , where l is the token length of the prompt and d e is the dimension of the token embeddings. From this, we can utilize the CLIP text encoder E L (•) to obtain a set of class features f C = E L (P * , C) ∈ R k×d in the form of CLIP text features, where P * is a fixed template for the CLIP text encoder, such as "A photo of a {} in the scene." While we could assign each mask f M with f C in a winner-takes-all manner, we desire the classes to encode general semantics across all images. Therefore, we assume that we can equally divide m masks within a minibatch [18, 14] , into k clusters given a sufficient amount of masks.Consequently, we aim to find an assignment Q ∈ R k×m + based on the image-text similarity between the mask pooled features f M and the class text features, which can be defined as:EQUATIONwhere F M is the set of all m mask features f M within the minibatch, and 1 k denotes the k-dimensional vector of ones. H is the entropy function,H(Q) = -ij Q ij logQ ij with ε as a hyperparameter.The solution Q from Eq. 3 is an assignment matrix defining which of the k clusters each m mask should belong to, hence ε determines the smoothness of this mapping Q by scaling the entropy regularization from H. The equipartition constraint,Q ⊤ 1 k = 1 m 1 m , Q1 m = 1 k 1 k encourages theclass features f C to be selected at least m/k times on average, allowing to learn general concepts represented by the masks within the dataset. In practice, with the soft assignment relaxation [46] , Q can be solved as follows:EQUATIONwhere u ∈ R k , v ∈ R m denote renormalization vectors, which can be efficiently computed by Sinkhorn-Knopp algorithm [46] .Finally, we can re-write the prediction of our model to be a cosine-similarity map between f I and f C :EQUATIONthereby predicting masks for f C (i) being the i-th class feature from f C , which we have obtained from clustering mask pooled features f M . Accordingly, ground truth masks M are also clustered according to Q by converting it into hard assignment with the argmax operator [47, 22] . This can be written as M ∈ R k×H×W where Mi is the union of masks assigned into the cluster represented by i-th learned class f C (i).Momentum encoder for integrating mask features. Since we jointly optimize the CLIP image encoder E V (•) as well as the learnable class feature f C , we may experience instability during our training process, or forgetting of the pre-trained knowledge [48] . To stabilize the training, we keep a momentum encoder [39, 38] for obtaining f M as seen in Fig. 3 . Therefore, we update f M as f M = MaskPool( ĒV (I), M ), where ĒV is the momentum encoder of the CLIP image encoder, updated with momentum γ. This can be denoted asθ ′ V ← γθ ′ V + (1 -γ)θ V , where θ V ′ , θ V are model parameters of ĒV and E V , respectively. 
implementation details: 
 For training, we employ per-pixel binary cross-entropy loss as L mask to jointly train all of the components [6] . For all our experiments, we use a single text prompt "A photo of {} in the scene" for P * , including for our learnable class prompts while training and for inference, we apply prompt ensemble strategy [30] with 7 additional prompts originally curated from CLIP [15] . We train our model on SA-1B [13] dataset, where we randomly sample 5% of the images. We train for 10000 iterations with a batch size of 48 for all experiments. For experiments using masks from DINO, we obtain masks with k-means clustering where we set k = 16. For experiments using masks from SAM, we use the unlabeled mask annotation in the SA-1B dataset. Without specification, we report results on ConvNeXt-B [49] backbone with mask annotation from SAM, which takes approximately 6 hours to train with 4 NVIDIA A6000 GPUs. We provide more details in the supplementary materials.Following Cha et al. [7] , we evaluate our model on zero-shot transfer to semantic segmentation on the validation sets of COCO-Stuff [50] , ADE-20K [51] , PASCAL-Context [52] , PASCAL VOC [53] , and CityScapes [54] . For CLIP [15] , we apply MaskCLIP [26] for ViT backbone for extracting image features, and remove the global pooling layer for OpenCLIP [55] with ConvNeXt [49] backbone. We note that we do not apply any post-processing to the predictions and for the compared methods. For the evaluation metric, we employ the mean Intersection over Union (mIoU).Open-vocabulary semantic segmentation. We provide results for quantitative comparisons in Tab. 1. We first compare with CLIP, and demonstrate remarkable gains in all benchmarks, bringing in an average of +16.2 mIoU improvement. Since we do not have comparable baselines without leveraging semantic labels, we further provide a comparison with image-level supervised methods [7, 9, 10] . Surprisingly, PixelCLIP surpasses TCL [7] and SegCLIP [9] in all benchmarks while using only a fraction of the images without semantic labels. Furthermore, we show competitive performance compared to SAM-CLIP, which uses not only 40 million image-level semantic labels, but also leverages the SA-1B dataset on a similar scale to our framework. Zero-shot mask classification. We provide results for evaluating mask classification in Tab. 2. We consider ZegFormer [1] and FC-CLIP [24] as baselines since they first predict masks, then employ CLIP as a zero-shot mask classifier within their framework, and also provide results with ground-truth masks to simulate having oracle mask predictions. For all methods, we apply masked pooling to CLIP image feature map to classify masks. For ZegFormer [1] and FC-CLIP [24] , reported results are only from the zero-shot prediction branch to solely ablate our gains. We highlight that PixelCLIP can be readily applied to existing frameworks that leverage CLIP as a zero-shot mask classifier, and bring instantaneous improvements by simply replacing the model and weights of CLIP.Qualitative results. We provide qualitative results for open-vocabulary semantic segmentation in Fig. 4 compared with results from CLIP, highlighting the dense open-vocabulary recognition capabilities of our framework. We further provide qualitative results in the supplementary materials.In Tab. 3, we show ablation studies on open-vocabulary semantic segmentation to validate our design choices. We report results without prompt ensembling for ablations, and also report results from OpenCLIP [55] as a baseline.Component analysis. In Tab. 3 (a), we provide results for ablating our key components. Notably, we observe that without global semantic clustering of masks, the framework collapses and loses the pre-trained knowledge of CLIP. This validates the challenge presented by leveraging unlabeled masks and demonstrates the crucial role of our proposed clustering approach. Moreover, we observe constant improvements over all datasets with our learnable class prompt, proving our approach of leveraging the text encoder of CLIP to define the clusters in the form of prompt learning. We also observe constant gains with the momentum encoder for extracting mask pooled features f M . Number of clusters. In Tab. 3 (b), we compare the results of the variants of the proposed method by varying the number of clusters k. We find that scaling k does not necessarily guarantee performance boosting, but it generally improves until k is set to 64 and tends to degrade as k grows. Considering that with an extremely large number for k, we can assign each of the masks to individual clusters(e.g. 1 billion for SA-1B.) This scenario would virtually be identical to not having semantic clustering as (d) Effects of utilizing learnable classes. We compare our method of learnable class prompts to having fixed set of classes from COCO-Stuff [50] .seen in Table . 3 (a), and progressively growing k would slowly converge to this scenario. We further provide analysis in 4.5, studying the different aspects from varying k.Length to represent learnable class prompts. Tab. 3 (c) compares the effects of varying the length of the learnable class prompt, l. We find that l = 1 shows lower scores in comparison to other lengths. We can interpret this as only describing a class with a single word, whereas having multiple words would better describe the depicted class. However, for l = 4 and larger, we find that increasing l does not result in a gain of performance, hence, we adopt l = 4 as default.Effects of learnable prompt token. Finally, we compare PixelCLIP to having a pre-defined set of classes instead of using learnable prompt tokens. Specifically, we use 171 classes from COCO-Stuff [50] , and do not apply online clustering for assignment when utilizing classes from COCO-Stuff, as it already yields text features with semantic meanings. We find apparent improvements over all the datasets as shown in Tab. 3 (d). We speculate that since the classes defined in COCO-Stuff are heuristically chosen, it is hard to ideally encompass various semantics and concepts that may appear in images, hence restricting the perception of the model to the finite set of classes. Since the learned prompts should act as implicit class names, we visualize the results from inference with learned class prompts in Fig. 5 (c-d ). Although both k = 64 and k = 128 show similar performance when evaluated, we observe that the prompts have learned more fine-grained semantics for k = 128. We generally observe human parts to be well distinguished; this could come from the SA-1B dataset, as there are numerous images with fine-grained masks representing human parts as annotations. We can observe that with k = 128, as the prompt learns more diverse semantics, we observe more accurately mapped classes. However, we still see predictions with large disparity to the actual ground truth. We leave a more in-depth analysis of the learned classes for future investigation. 
conclusion: 
 In this paper, we introduced PixelCLIP, a framework for leveraging unlabeled images and masks for fine-tuning the pre-trained vision-language models for open-vocabulary semantic segmentation. To address the unique challenges posed by incorporating unlabeled masks generated by vision foundation models into our framework, we propose global semantic clustering of the masks, with learnable class prompts to represent each cluster. We demonstrated PixelCLIP to show remarkable improvements to CLIP and its applicability to existing methods, providing instantaneous improvements, as well as surpassing methods that leverage image-level semantic labels such as image captions. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We provide exhaustive details in Section 4.1 in the main paper, as well as further details in Section A in the supplementary materials for reproducing the experimental results. We are committed to releasing our code upon acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide exhaustive details in Section 4.1 in the main paper, as well as further details in Section A in the supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We do not report error bars, but we fix the random seed to minimize the stochasticity for all of our experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We report the resources and training time of our method in Section 4.1.Guidelines:• The answer NA means that the paper does not include experiments. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss broader impacts in Section F in the supplementary material. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks.Answer: [NA] Justification: We do not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 In contrast to existing methods utilizing (a) pixel-level semantic labels [1, 2, 3, 4, 5, 6] or (b) image-level semantic labels [7, 8, 9, 10, 11, 12] , we leverage unlabeled masks as supervision, which can be freely generated from vision foundation models such as SAM [13] and DINO [14] . 
 
 Appendix: In Tab. 5, we show results with prompt ensembling being applied during only training, only inference, and both. We report results with ViT-B/16 using SA-1B masks as supervision. Although prompt ensembling does bring slight gains when enabled during training, the computation for optimizing learnable class prompts scales along with the number of prompts used, increasing the training time and the memory consumption. On the other hand, applying prompt ensembling during inference only adds negligible cost as they can be computed once and be cached, but shows much significant gains compared to when applied training. Therefore, we adopt prompt ensemlbing only during inference, but noticing that the performance can be maximized with better prompts during training. In this regard we can better results with better prompt design or a learnable prefix to accompany the learnable class prompts, which we leave for future investigation. In Tab. 6 (a), we show results for varying γ for the momentum update. While having the momentum encoder generally shows improvements, we find γ = 0.999 to show the best results for updating the momentum encoder.In Tab. 6 (b), we show results for training with mask annotation from COCO-Stuff [50] . For COCO-Stuff, we remove the ground truth class labels and utilize them as unlabeled masks, and other hyperparameters are set identically with k = 64. Although the masks from COCO-Stuff show better results across all datasets, we highlight that the SA-1B [13] dataset mostly consists of automatically generated masks from SAM, whereas COCO-Stuff has human annotated masks from expert annotators. We provide qualitative results of visualization on ADE-20K [51] , PASCAL-Context [52] in Fig. 8 and Fig. 9 .In Fig. 7 , we show visualization on COCO-Stuff by classifying the image features with our learned class prompts for varying k. From the first and the second row, we can observe that with larger numbers of k, different parts of human are segmented into fine-grained regions whereas k = 64 has On the other hand, we also observe cases where a small number of k struggles to differentiate visual concepts in the last row, where the animals are partially grouped with k = 64 and show better groups for k = 128 and k = 256. This could indicate that with only a small number of clusters, several fine-grained visual concepts that may not be seen often in the dataset to be grouped as a whole, whereas independent clusters could be assigned with a larger number of k, allowing fine-grained recognition of semantics.Although we aim to fine-tune the image encoder of CLIP for adapting to dense predictions, we initialize the mask encoder within our framework with pre-trained weights of CLIP, which yields poor results for classifying masks when applying mask pooling to its features. Consequently, the noisy mask features in the earlier stage of training may result in sub-optimal performance. While there could be alternative methods to extract per-mask CLIP image features, we consider mask pooling to be sufficient to show meaningful improvements to CLIP and consider such exploration for future directions.Our framework facilitates open-vocabulary semantic segmentation through leveraging visionlanguage models, hence the recognition capabilities of our method rely on the pre-trained knowledge of the vision-language models. Considering that large-scale pre-trained vision-language models [15, 16, 55] leverage web-crawled data within its training, the models may exhibit wrongful behaviors from bias or corrupted data from the internet which calls for future research to address.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: All reviewers found the article interesting, exploring the use of unlabeled masks for building an open vocabulary segmentation model. In general, reviewers found that the results and analyses support the proposed approach, consistently outperforming CLIP/MaskCLIP and methods using caption as supervision.

The AC went through the paper and the discussion and agreed with the reviewers' assessment. The article shows how we can learn a competitive open-vocabulary segmentation model without semantic labels is feasible, and this can be of interest to a large portion of the community, especially considering the new perspective under which the task has been addressed.

During the discussion, reviewers suggested several analyses and comparisons provided by the authors (e.g., MaskCLIP+, prompts, the impact of data/scalability, and centroids, among others). The authors should add these additional analyses and discussions in the final version of the manuscript and/or the appendix, as promised.

Reviews: review: paper_summary: paper_summary: This paper proposes to enhance the semantic segmentation performance of the pretrained CLIP model using unlabeled images and pseudo segmentation masks generated with vision foundation models such as SAM and DINO. Specially, the pseudo masks are acquired via a online feature clustering algorithm. Experiments on standard benchmarks demonstrate the superior performance over CLIP and competitive results to existing open-vocabulary semantic segmentation methods.
strength_weakness: strength_weakness: 1. How is the proposed method in comparison with MaskCLIP+? Figure 4 shows the visual comparisons between the proposed method and MaskCLIP, as the Sec. 4.2 mentioned “For evaluating CLIP [42, 10], we apply MaskCLIP [61] to extract CLIP image features”. It would be interesting to compare the proposed method with MaskCLIP+ which distills MaskCLIP to train more advanced segmentation model. 
2. Some related works should be discussed and compared to the proposed method, for example, Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only. ICCV 2023.
questions: questions: See the weakness section.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes to learn an open-vocabulary semantic segmentation model with only unlabeled images and pretrained foundation models, such as SAM and DINO. The intuition is that CLIP model already knows what is in the image, so we only need to teach CLIP where the object is. It first uses pretrained DINO to generate pseudo masks and then exploits a online clustering method to group the part segments into valid object masks. It also proposes learnable class embeddings to solve the problem of lacking ground-truth text labels. Compared with baselines, the proposed method achieves decent improvements.
strength_weakness: strength_weakness: - Compared with other methods  [ 50 , 30 , 43 , 51 ] leveraging image caption as supervision, this paper actually uses stronger DINO generated pseudo masks to train the segmentation model. Furthermore, it even uses the SAM masks during experiments (In Table 2). Regrading this, I think this method includes strong segmentation prior into the training, making the comparison unfair.

- Compared with methods using similar VFM, such as SAM-CLIP, the proposed method performs much worse. I understand it is not an apple-to-apple comparison due to the differences in training data. More specially, this paper doesn't use captions. However, caption is also very easy to get with pretrained caption models. If simply adding caption could boost the performance so much, why should we stick to a setting without caption?
questions: questions: - Why does Table 1 appear before Table 2? Tabel 1 looks more like an ablation study to me.
limitations: limitations: See weakness
paper_summary: paper_summary: The authors introduce a novel unsupervised formulation of open vocabulary semantic segmentation, adapting a pre-trained vision-language model (CLIP) to the task via distillation from vision-only segmentation models (i.e., SAM, DINO). To use the language encoder without constraining fine-tuning to the set of classes of the dataset, they propose to apply online clustering at the dataset level and to learn class-specific feature prototypes. The method is compared against state-of-the-art approaches trained with and without supervision, and the results validate the approach as a practical direction for training open vocabulary models for semantic segmentation.
strength_weakness: strength_weakness: - While I understand the rationale behind performing clustering at pixel level on the entire dataset, I am not sure of the scalability of the approach. This probably explains why the authors only use 5% of the SA-1B dataset. It would be helpful to quantify the costs of performing the online clustering at training time.
- Since the method could potentially suffer scalability issues, it would be interesting to understand performance when trained unsupervised in-distribution, reporting both in- and out-of-distribution performance, e.g., MCLIP trained on VOC and tested on Context.
- The reasoning behind the selection process for the prompt (i.e., "a photo of a {} in the scene") is unclear. This is a minor issue, but the selection should probably be justified. While I would expect the model to learn to ignore the prefixes and suffixes, it would be interesting to understand how the model performs with other prompts.
- Using CLIP as a baseline for the ablation studies may be misleading due to the mismatch between its application and open vocabulary semantic segmentation
questions: questions: - What are the costs of performing online clustering at the dataset level during training?
- What is the performance of MCLIP when trained on the benchmark datasets and tested in- and out-of-distribution? How does it compare against some baseline methods?
- What is the model performance when changing the template at inference time? What is the performance without the template? What about training without the template? Does performance improve with prompt ensembling (i.e., similar to what CLIP does with N prompts that average to get better class centroids)?
limitations: limitations: - A potential limitation is the one I reported above, i.e., scalability issues due to the global online clustering procedure.
paper_summary: paper_summary: This paper present MCLIP, which aims to finetune CLIP using image and mask data without semantic labels. The goal is to make its open-vocabulary recognition ability to be adapted to position-sensitive semantic segmentation tasks. MCLIP chooses to use SAM or feature clusters of DINO to obtain masks, which are class-agnostic but sometimes too small to have consistent semantics for finetuning. In addition, in order to achieve a trade-off between training stability and avoiding catastrophic forgetting, clustering uses the EMA version of finetuned image encoder.
strength_weakness: strength_weakness: - What role does the text encoder play in the clustering process?

The key to the success of clustering lies in the existence of unlabeled data in a certain grouping relationship in the space. The mask representations f_M can be considered to meet such conditions. If k learnable object prompts are initialized randomly, then k text representations f_C are randomly scattered in the text space initially, unable to provide the information needed for clustering. If k learnable object prompts are initialized by predefined categories, then this should violate the experimental setup without semantic labels. To illustrate this problem, the authors need to provide an explanation and add a comparative experiment: What results can be obtained just by clustering with f_M?

- Why use clustering technology to solve the problem of too fine initial mask granularity?

Firstly, previous work (GroupViT, CVPR'22 and SegCLIP, ICML'23) has used a grouping block designed based on the idea of clustering in the process of fine-tuning CLIP. In contrast, the clustering used in MCLIP provides better masks initially than the image patches used in previous work and the clustering process of MCLIP is parameter-free. It may be necessary to conduct ablation experiments to prove which differences bring positive results. But overall, technically, the use of clustering may be not novel.

- In order to obtain class-agnostic masks, there is a class of experimental settings called open set semantic
segmentation (O3S, arXiv preprint arXiv:2307.02003, 2023) . Can the ”SAM initialization + clustering” scheme proposed by MCLIP obtain competitive performance in the O3S experimental setting?

- Table 1 shows that using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP in the previous method improves the performance of the previous method’s zero-shot mask classification. Taking FC-CLIP as an example, MCLIP improves its zero-shot mask classification performance on ADE20K by 2.7. But compared to the final performance of FC-CLIP with two branches integrated, it is still 4.0 lower. So, in the case of keeping the complete inference strategy of the previous method, how much improvement can be achieved by using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP?

- In the ablation experiment in Table 3a, does ’w/o Semantic Clustering’ mean fine-tuning with SAM-initialized masks directly without clustering? If so, why is the performance so poor? Without using clustering, it is equivalent to the number of cluster k is greater than the number of masks initialized by SAM on all images. However, the conclusion obtained from table 3b is that with the increase of k, the
performance tends to saturation, rather than decreasing to the level of ”w/o Semantic Clustering”.
questions: questions: I think the authors need to further explain the novelty of this paper and give more analysis on the experimental results as listed in the weaknesses part.
limitations: limitations: Limitations are provided. It seems that there is no potential negative societal impact.

Comments: title: Final rating
I've raised my score from 4 to 5
comment: Thanks for the detailed response! I'll raise my initial rate to weak accept.
I appreciate the responses from the reviewers. Though most of my concerns have been solved, I still think the novelty of this paper cannot reach the level of an accept. So, I tend to keep my original rating unchanged.
I thank the authors for their reply, which answers my doubts and questions. I appreciate that they executed all the experiments I proposed. Scaling the dataset size seems to (unsurprisingly) improve model performance. Also, a better template for the input query or a template ensemble has a positive impact. For the first, I believe the work would benefit from some experiments showing how performance increases with an increase in the number of train samples. I expect the performance to improve until the compute-optimal point is reached [1].

Overall, I am glad my suggestions improved the method's performance, however, this also gives the impression the work was not carried out very rigorously. In any case, I am satisfied with the authors' rebuttal and confirm my initial positive rating.

[1] Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv (2022).
Thanks for the rebuttal! I've raised my score from 4 to 5.