Title: Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
Venue: NeurIPS 2024
Accepted: yes
Keywords: semantic segmentation

open-vocabulary segmentation

vision-language models

fine-tuning foundation models
Abstract: Large-scale vision-language models like CLIP have demonstrated impressive open-vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which require understanding where the objects are located. In this work, we propose a novel method, PixelCLIP, to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where, which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to caption-supervised methods in open-vocabulary semantic segmentation.

Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels

Abstract

Large-scale vision-language models like CLIP have demonstrated impressive openvocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which additionally require understanding where the objects are located. In this work, we propose a novel method, PixelCLIP , to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where , which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to captionsupervised methods in open-vocabulary semantic segmentation. Project page is available at https://cvlab-kaist.github.io/PixelCLIP

(b) image-level semantic labels

'photo of a {

corgi

} running

in a {

grass

} field'

image

(a) pixel-level semantic labels

model

tree

dog

grass

image

model

image

model

(c)

without

semantic labels (ours)

mask generation

VFM

(e.g. SAM, DINO)

global semantic clustering

[CLASS

1

]

[CLASS

2

]

[CLASS

3

]

learnable classes

1 Introduction

Semantic segmentation is a fundamental task in computer vision where the goal is to identify class labels for each pixel within the given image. However, segmentation datasets often require extensive human effort to obtain densely-annotated semantic labels, limiting their scalability. In this regard, recent advances in large-scale pre-trained vision-language models, e.g. CLIP [15] and ALIGN [16],

clustered SAM masks

DINO masks

clustered DINO masks

SAM masks

have facilitated open-vocabulary semantic segmentation [1, 3, 2, 17, 4, 6], which aims to generalize semantic segmentation into unbounded range of classes. Despite showing remarkable generalization capabilities, they still require pixel-level semantic labels for leveraging the image-level pre-trained vision-language models for semantic segmentation.

Recently, several studies [11, 12, 7, 8] have pioneered open-vocabulary semantic segmentation without densely-annotated semantic labels. These studies often utilize image-level semantic labels, such as image captions, to enhance the pre-trained vision-language models like CLIP for semantic segmentation. However, image captions typically provide information about what is in the image, but without where it is. Since CLIP is already effective in recognizing what the objects are, this causes models to only implicitly learn object locations, leading to sub-optimal performance or requiring millions of image-caption pairs to compensate for this weak supervision [8, 7]. Instead, we focus on informing CLIP about where objects are located to address the missing information.

In this study, we propose a novel approach to achieve open-vocabulary semantic segmentation without leveraging semantic labels, but through guiding the pre-trained vision-language models, such as CLIP, on where to look. We leverage recent vision foundation models (VFMs), such as DINO [14] and SAM [13], to partition images into fine-grained regions to indicate where to look. Consequently, we explore methods to effectively leverage these masks for fine-tuning the image encoder of CLIP.

In contrast to existing works that leverage semantic labels [6, 19, 7], we do not have any captions or class names that can be fed to the text encoder of CLIP. To leverage its knowledge, we devise a method that employs prompt learning [20, 21] on the text encoder of CLIP to construct learnable classes . Setting the learnable classes as a centroid, we propose applying the online clustering algorithm [18, 22] along the given masks to gather them into semantically meaningful groups, as shown in Fig. 2. We keep these learnable classes global across the entire images, which guides the learnable classes to contain the general semantic concepts. Despite the absence of semantic labels, our method is able to jointly leverage the image encoder and text encoder of CLIP during training, successfully achieving dense open-vocabulary recognition.

Our framework, called PixelCLIP, achieves significant improvements to CLIP, on average of +16 . 2 mIoU in open-vocabulary semantic segmentation. Moreover, despite not using any semantic labels, PixelCLIP shows competitive performance in comparison to image-level supervised methods using captions [7, 9, 10], demonstrating the effectiveness of unlabeled masks for. We further show the effectiveness of PixelCLIP for classifying masks from various open-vocabulary segmentation models, which can be simply done by replacing the CLIP within existing methods. We also provide extensive ablation studies to validate our choices, with a detailed analysis of our method.

We summarize our contribution as follows:

We propose a novel formulation of learning from images without semantic label for openvocabulary semantic segmentation by leveraging masks generated from DINO and SAM to fine-tune vision-language models.

We propose to globally cluster semantically similar masks by employing an online clustering algorithm, while learning class prompts for representing semantic clusters.

Wedemonstrate significant gains in open-vocabulary semantic segmentation, even surpassing methods leveraging image-level semantic labels, and provide thorough ablation studies with analysis to validate our framework.

2 Related Work

2.1 Open-vocabulary semantic segmentation

Open-vocabulary semantic segmentation [2, 23] aims to label each pixel within an image into an unbounded range of classes. In this regard, recent works [1, 17, 2, 6, 24] aim to generalize to classes unseen during training through leveraging pre-trained vision-language models, such as CLIP [15]. Despite their remarkable performance, they leverage per-pixel semantic labels during their training, which requires expensive cost to annotate. Instead, we focus on the weakly-supervised setup, where the goal is to zero-shot transfer to segmentation task without densely-annotated class labels [11, 12, 25, 5, 26, 7, 27], utilizing image-level labels as supervision or even no labels at all.

In this regard, recent studies [11, 12, 25, 5] leverage image caption as supervision. GroupViT [11] and ViL-Seg [12] are pioneering works for identifying groups or clusters emerging from captions. Along with the advance of vision-language models, SegCLIP [9] and TCL [7] leverage pre-trained CLIP and learn additional decoder modules to learn dense vision-language alignment. PACL [8] learns additional embedding layers to enhance the patch-level alignment in vision-language models and SAM-CLIP [10] attempts to merge SAM [13] and CLIP [15] into a unified model by additionally leveraging unlabeled mask data from SAM. Apart from these approaches, we avoid employing any semantic labels [26, 28], but leverage vision foundation models to obtain masks as a source for supervision to fine-tune the CLIP image encoder for achieving open-vocabulary semantic segmentation.

2.2 Fine-tuning vision-language models for dense prediction

Recent large-scale pre-trained vision-language models have shown its effectiveness for jointly understanding images and language [29, 15, 16]. Notably, CLIP [15], trained with web-scale image-caption pairs, has been widely popularized for transferring its open-vocabulary recognition capabilities to various downstream tasks [26, 30, 31, 32]. However, despite its success in image-level tasks like image classification, CLIP tends to struggle in dense prediction tasks [17, 26, 6], such as object detection and semantic segmentation. This originates from CLIP being trained from image-level supervision being captions, hence exhibits bias towards the global image rather than fine-grained regions within the image [17]. While non-learnable approaches, such as MaskCLIP [26] show improvements by slightly modifying the architecture, CLIP still shows limited capabilities in dense predictions in comparison to its global understanding.

To address this, OWL-ViT [33] directly fine-tunes pre-trained vision and text encoders to downstream open-vocabulary detection task, and CAT-Seg [6] introduces a cost aggregation scheme for finetuning the encoders of CLIP for semantic segmentation. Alternatively, ZegCLIP [19] and Xu et al. [3] implement prompt tuning [21, 20] for tuning the image and text encoders of CLIP. Instead of fine-tuning the full model, they learn prompt tokens that serve as a global prefix for the encoders of CLIP. While such methods show remarkable results from fine-tuning the encoders of CLIP for dense downstream tasks, they require densely annotated detection and segmentation data for training.

2.3 Vision foundation models

With the advent of large-scale learning enabled by scalable vision backbone architectures [34, 35] and vast amounts of data, diverse vision foundation models are emerging in the field of computer vision. In this regard, self-supervised methods [36, 37, 38, 39] have demonstrated the effectiveness of its rich visual representations for various downstream tasks. Especially, DINO [14] exerted strengths in fine-grained semantic recognition [40, 41], making it highly effective for object detection and image segmentation. Moreover, DINO features have also been demonstrated for yielding fine-grained masks within the image through applying the k -means clustering with its features [42, 43, 44].

On the other hand, the segment anything model (SAM) [13] has demonstrated its capability for generating fine-grained, high-quality segmentation masks for any object in an image. Through its selfannotation pipeline, SAM has collected an unprecedented amount of mask annotation for achieving its capabilities. While we can freely leverage SAM to obtain detailed masks in any given image, we mainly utilize the pre-computed masks within the collected dataset, SA-1B. Both DINO and SAM, however, yield unlabeled masks without semantic labels as both models are also trained without semantic labels, presenting a challenge for leveraging their masks for achieving dense vision-language recognition.

̅

image

encoder

momentum

update

text

encoder

image feature

image feature

mask

decoder

A photo of [

C

1

] …

A photo of [

C

i

] …

A photo of [

C

k

] …

…

…

learnable

class prompts

similarity map

class features

online cluster assignment

masks

image

assigned class feature

binary

mask loss

image

encoder

mask feature

…

…

C

M

mask prediction

image

encoder

text

encoder

image feature

open-vocabulary prediction

image

text features

A photo of 'animal' in the scene

A photo of 'tree' in the scene

A photo of 'grass' in the scene

C

training with image-mask pairs

open-vocabulary inference

ℇ

!

"

ℇ

!

𝑓

!

𝑓

"

𝑓

#

ℇ

$

𝒟

ℇ

!

ℇ

$

𝑓

!

𝑀

!#

& 𝑀

𝑀

𝐼

frozen parameters

M

masked pooling

C

cosine similarity

learnable parameters

3 Methodology

In this section, we first establish our problem formulation of learning dense vision-language alignment from images paired with masks, generated from vision foundation models. Next, we discuss the challenges of leveraging masks as supervision for fine-tuning the image encoder of CLIP and finally, present our methodology of semantic clustering of masks to address the challenges.

3.1 Preliminaries

Given an input image I ∈ R H × W × 3 , open-vocabulary semantic segmentation [6, 7] aims to label each pixel within an image with classes given in free-form text. As a training signal, semantic labels offer a set of S textual descriptions for a semantic class T = { T i } S i =1 related to I . This can be directly utilized with the CLIP text encoder E L ( · ) to obtain text features f T = E L ( T ) ∈ R S × d , where d is the hidden dimension. Dense image features f I = E V ( I ) ∈ R h × w × d , where h × w is the output feature resolution, are then extracted. We finally obtain dense image-text similarity map M IT ∈ R h × w × S :

This can be interpreted as soft binary masks predicted from image and text features of CLIP, and be supervised with binary mask loss L mask in a pixel-level manner to fine-tune CLIP [6].

3.2 Integrating masks into CLIP features

In this work, we do not have any access to T , but are only given unlabeled masks M ∈ R H × W × N , where N denotes the number of masks for the given image I . Hence, we devise methods to predict masks by incorporating M into CLIP features. We aim to fine-tune the CLIP image encoder E V ( · ) through leveraging unlabeled masks M as supervision. Since M is generated from vision foundation models, e.g. DINO or SAM, this presents us with the challenge of not having any semantic labels.

In order to integrate masks into CLIP, a straightforward approach would be employing the masks M with the CLIP image feature map f I to obtain per-mask CLIP features. While there could be various methods to extract regional CLIP features [26, 45, 5], we apply mask pooling over f I to obtain mask

pooled features f M = MaskPool( f I , M ) ∈ R N × d . Consequently, we can leverage f M to obtain imagemask similarity map denoted M IM ∈ R h × w × N :

This allows us to supervise the model with a binary mask loss L mask for fine-tuning CLIP with given image I and unlabeled masks M . In practice, since M IM has the same resolution as the feature map from the CLIP image encoder f I , we employ a light-weight decoder D to mitigate the resolution gap between M IM and M , as shown in Fig. 3. This can be written as D : R h × w → R h ′ × w ′ , where h ′ × w ′ is resolution for the upsampled mask. Therefore, the output of the model can be updated as ˜ M = D ( M ) .

3.3 Semantic clustering of masks

Upon using mask pooled CLIP image features f M to predict M IM , however, we find the masks generated from DINO and SAM to often over-segment the image, resulting in too small or incomplete masks as seen in Fig. 2. This would require CLIP to forcefully discriminate regions that are semantically similar, impeding the training process.

In this regard, we propose to group semantically similar masks into clusters and predict based on the clusters rather than individual masks. Moreover, we aim to define this cluster globally , which is shared across the entire training process rather than for each image or iteration. This would be analogous to constructing pixel-level semantic labels, where a fixed set of classes defined over the dataset is equivalent to each cluster. However, the difference is that there is no pre-defined set of classes that we can define the clusters with. While we could heuristically pre-define such classes, we describe our learnable method for globally clustering masks into semantically meaningful groups.

Online clustering via learnable class prompts. To globally cluster masks into semantic categories, we propose representing these clusters using CLIP text features as centroids for clustering mask features. Given that the CLIP text encoder is trained with a broad understanding of natural language semantics, we expect these clusters to capture meaningful semantics by leveraging its comprehensive pre-trained knowledge. In this regard, we take a learnable approach, where each cluster is defined by class-specific learnable prompts fed into the CLIP text encoder. Unlike existing prompt learning methods, which typically focus on learning a task-specific prefix [20, 21, 3], we aim to learn prompt tokens that represent each class. For instance, in the sentence 'A photo of an object' , traditional prompting methods would learn the tokens for the 'A photo of a' prefix, whereas our method focuses on learning the token for the 'object.'

Specifically, given the number of clusters k , we can define prompt tokens as C ∈ R k × l × d e , where l is the token length of the prompt and d e is the dimension of the token embeddings. From this, we can utilize the CLIP text encoder E L ( · ) to obtain a set of class features f C = E L ( P ∗ , C ) ∈ R k × d in the form of CLIP text features, where P ∗ is a fixed template for the CLIP text encoder, such as 'A photo of a {} in the scene." While we could assign each mask f M with f C in a winner-takes-all manner, we desire the classes to encode general semantics across all images. Therefore, we assume that we can equally divide m masks within a minibatch [18, 14], into k clusters given a sufficient amount of masks.

Consequently, we aim to find an assignment Q ∈ R k × m + based on the image-text similarity between the mask pooled features f M and the class text features, which can be defined as:

where F M is the set of all m mask features f M within the minibatch, and ✶ k denotes the k -dimensional vector of ones. H is the entropy function, H ( Q ) = -∑ ij Q ij log Q ij with ε as a hyperparameter. The solution Q from Eq. 3 is an assignment matrix defining which of the k clusters each m mask should belong to, hence ε determines the smoothness of this mapping Q by scaling the entropy regularization from H . The equipartition constraint, Q ⊤ ✶ k = 1 m ✶ m , Q ✶ m = 1 k ✶ k encourages the class features f C to be selected at least m/k times on average, allowing to learn general concepts represented by the masks within the dataset. In practice, with the soft assignment relaxation [46], Q

can be solved as follows:

where u ∈ R k , v ∈ R m denote renormalization vectors, which can be efficiently computed by Sinkhorn-Knopp algorithm [46].

Finally, we can re-write the prediction of our model to be a cosine-similarity map between f I and f C :

thereby predicting masks for f C ( i ) being the i -th class feature from f C , which we have obtained from clustering mask pooled features f M . Accordingly, ground truth masks M are also clustered according to Q by converting it into hard assignment with the argmax operator [47, 22]. This can be written as ¯ M ∈ R k × H × W where ¯ M i is the union of masks assigned into the cluster represented by i -th learned class f C ( i ) .

Momentum encoder for integrating mask features. Since we jointly optimize the CLIP image encoder E V ( · ) as well as the learnable class feature f C , we may experience instability during our training process, or forgetting of the pre-trained knowledge [48]. To stabilize the training, we keep a momentum encoder [39, 38] for obtaining f M as seen in Fig. 3. Therefore, we update f M as f M = MaskPool( ¯ E V ( I ) , M ) , where ¯ E V is the momentum encoder of the CLIP image encoder, updated with momentum γ . This can be denoted as θ ′ ¯ V ← γθ ′ ¯ V +(1 -γ ) θ V , where θ ¯ V ′ , θ V are model parameters of ¯ E V and E V , respectively.

4 Experiments

4.1 Implementation details

For training, we employ per-pixel binary cross-entropy loss as L mask to jointly train all of the components [6]. For all our experiments, we use a single text prompt 'A photo of {} in the scene' for P ∗ , including for our learnable class prompts while training and for inference, we apply prompt ensemble strategy [30] with 7 additional prompts originally curated from CLIP [15]. We train our model on SA-1B [13] dataset, where we randomly sample 5% of the images. We train for 10000 iterations with a batch size of 48 for all experiments. For experiments using masks from DINO, we obtain masks with k -means clustering where we set k = 16 . For experiments using masks from SAM, we use the unlabeled mask annotation in the SA-1B dataset. Without specification, we report results on ConvNeXt-B [49] backbone with mask annotation from SAM, which takes approximately 6 hours to train with 4 NVIDIA A6000 GPUs. We provide more details in the supplementary materials.

4.2 Experimental setting

Following Cha et al. [7], we evaluate our model on zero-shot transfer to semantic segmentation on the validation sets of COCO-Stuff [50], ADE-20K [51], PASCAL-Context [52], PASCAL VOC [53], and CityScapes [54]. For CLIP [15], we apply MaskCLIP [26] for ViT backbone for extracting image features, and remove the global pooling layer for OpenCLIP [55] with ConvNeXt [49] backbone. We note that we do not apply any post-processing to the predictions and for the compared methods. For the evaluation metric, we employ the mean Intersection over Union (mIoU).

4.3 Results

Open-vocabulary semantic segmentation. We provide results for quantitative comparisons in Tab. 1. We first compare with CLIP, and demonstrate remarkable gains in all benchmarks, bringing in an average of +16.2 mIoU improvement. Since we do not have comparable baselines without leveraging semantic labels, we further provide a comparison with image-level supervised methods [7, 9, 10]. Surprisingly, PixelCLIP surpasses TCL [7] and SegCLIP [9] in all benchmarks while using only a fraction of the images without semantic labels. Furthermore, we show competitive performance compared to SAM-CLIP, which uses not only 40 million image-level semantic labels, but also leverages the SA-1B dataset on a similar scale to our framework.

Zero-shot mask classification. We provide results for evaluating mask classification in Tab. 2. We consider ZegFormer [1] and FC-CLIP [24] as baselines since they first predict masks, then employ CLIP as a zero-shot mask classifier within their framework, and also provide results with ground-truth masks to simulate having oracle mask predictions. For all methods, we apply masked pooling to CLIP image feature map to classify masks. For ZegFormer [1] and FC-CLIP [24], reported results are only from the zero-shot prediction branch to solely ablate our gains. We highlight that PixelCLIP can be readily applied to existing frameworks that leverage CLIP as a zero-shot mask classifier, and bring instantaneous improvements by simply replacing the model and weights of CLIP.

Qualitative results. We provide qualitative results for open-vocabulary semantic segmentation in Fig. 4 compared with results from CLIP, highlighting the dense open-vocabulary recognition capabilities of our framework. We further provide qualitative results in the supplementary materials.

4.4 Ablation studies

In Tab. 3, we show ablation studies on open-vocabulary semantic segmentation to validate our design choices. We report results without prompt ensembling for ablations, and also report results from OpenCLIP [55] as a baseline.

Component analysis. In Tab. 3 (a), we provide results for ablating our key components. Notably, we observe that without global semantic clustering of masks, the framework collapses and loses the pre-trained knowledge of CLIP. This validates the challenge presented by leveraging unlabeled masks and demonstrates the crucial role of our proposed clustering approach. Moreover, we observe constant improvements over all datasets with our learnable class prompt, proving our approach of leveraging the text encoder of CLIP to define the clusters in the form of prompt learning. We also observe constant gains with the momentum encoder for extracting mask pooled features f M .

Number of clusters. In Tab. 3 (b), we compare the results of the variants of the proposed method by varying the number of clusters k . We find that scaling k does not necessarily guarantee performance boosting, but it generally improves until k is set to 64 and tends to degrade as k grows. Considering that with an extremely large number for k , we can assign each of the masks to individual clusters( e.g. 1 billion for SA-1B.) This scenario would virtually be identical to not having semantic clustering as

ceiling

sconce

doue

sky

step

sconce

ceiling

wall

door

windowpane

blind

skyscraper

mirro

skyscraper

mirror

wall

tray

windowpane

sink

shower

sink

step

cabinet

bathtub

cabinet

bathtub

shower

building

canopy

tree

countertop.

Sianboard

windowpane

step

wall

windowpane

curtain

windowpane

curtain

painting

wall

poster

wall

poster

persontray

wall

painting

pillow

person

microwave

pillow

plaything

bed

a

step

floor

blanket

stool

floor

blanket

stool

desk

(a) Ours

(b) CLIP

(a) Component analysis. We validate the core components of our framework by ablating each components. Notably, global clustering of masks shows its importance for facilitating the framework.

(c) Length of learnable prompt token. For varying l , we find that l = 4 shows best overall performance.

(b) Number of clusters. For varying k , we find that scaling k larger than 64 does not show much improvements, while k = 32 also show competitive results.

(d) Effects of utilizing learnable classes. We compare our method of learnable class prompts to having fixed set of classes from COCO-Stuff [50].

seen in Table. 3 (a), and progressively growing k would slowly converge to this scenario. We further provide analysis in 4.5, studying the different aspects from varying k .

Length to represent learnable class prompts. Tab. 3 (c) compares the effects of varying the length of the learnable class prompt, l . We find that l = 1 shows lower scores in comparison to other lengths. We can interpret this as only describing a class with a single word, whereas having multiple words would better describe the depicted class. However, for l = 4 and larger, we find that increasing l does not result in a gain of performance, hence, we adopt l = 4 as default.

Effects of learnable prompt token. Finally, we compare PixelCLIP to having a pre-defined set of classes instead of using learnable prompt tokens. Specifically, we use 171 classes from COCOStuff [50], and do not apply online clustering for assignment when utilizing classes from COCO-Stuff, as it already yields text features with semantic meanings. We find apparent improvements over all the datasets as shown in Tab. 3 (d). We speculate that since the classes defined in COCO-Stuff are heuristically chosen, it is hard to ideally encompass various semantics and concepts that may appear in images, hence restricting the perception of the model to the finite set of classes.

(a)

𝑘 = 64

(b)

𝑘 = 128

(c)

𝑘 = 64

(d)

𝑘 = 128

learned

init

COCO-St.

4.5 Analysis

Learnable class prompt. We further analyze the learned class prompt in Fig. 5 (a-b) with t -SNE visualization on the text features encoded from the learned class prompts, as well as text features obtained from class names of COCO-Stuff. Since we initialize the class prompt tokens as random tokens, we observe that they are in a skewed distribution in the initial state. However, the learned prompts show that they are well-dispersed among the text features from COCO-Stuff, indicating that the class prompts have well-learned diverse semantic concepts within the text features. We observe well-distributed features both for k = 64 and k = 128 .

Since the learned prompts should act as implicit class names, we visualize the results from inference with learned class prompts in Fig. 5 (c-d). Although both k = 64 and k = 128 show similar performance when evaluated, we observe that the prompts have learned more fine-grained semantics for k = 128 . We generally observe human parts to be well distinguished; this could come from the SA-1B dataset, as there are numerous images with fine-grained masks representing human parts as annotations.

mountain

mountain

sky

escalator

ship

house

escalator

grass

bridge

table

grass

truck

bag

arcade mac

road

toilet

sidewalk

(a) ! = 64

(b) ! = 128

Interpreting learned classes. Considering the learned class prompts represent semantic concepts, we further study the learned embeddings by mapping each class embeddings to class names in COCO-Stuff with the highest cosine-similarity score. Fig. 6 shows results when we first inference the image features with learned class prompts, then map the results with the closest COCO classes. We can observe that with k = 128 , as the prompt learns more diverse semantics, we observe more accurately mapped classes. However, we still see predictions with large disparity to the actual ground truth. We leave a more in-depth analysis of the learned classes for future investigation.

5 Conclusion

In this paper, we introduced PixelCLIP, a framework for leveraging unlabeled images and masks for fine-tuning the pre-trained vision-language models for open-vocabulary semantic segmentation. To address the unique challenges posed by incorporating unlabeled masks generated by vision foundation models into our framework, we propose global semantic clustering of the masks, with learnable class prompts to represent each cluster. We demonstrated PixelCLIP to show remarkable improvements to CLIP and its applicability to existing methods, providing instantaneous improvements, as well as surpassing methods that leverage image-level semantic labels such as image captions.

Acknowledgement. This research was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019II190075, RS-2024-00509279, RS-2020-II201819, RS-2024-00398115, Research on the reliability and coherence of outcomes produced by Generative AI) and the Culture, Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism (RS-2024-00348469, RS-2023-00266509), and National Research Foundation of Korea (RS-2024-00346597).

References

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11583-11592, 2022.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , pages 540-557. Springer, 2022.

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX , pages 736-753. Springer, 2022.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2945-2954, 2023.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2955-2966, 2023.

Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation, 2024.

Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11165-11174, 2023.

Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19413-19423, 2023.

Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In International Conference on Machine Learning , pages 23033-23044. PMLR, 2023.

Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. arXiv preprint arXiv:2310.15308 , 2023.

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18134-18144, 2022.

Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Openworld semantic segmentation via contrasting and clustering vision-language embedding. In European Conference on Computer Vision , pages 275-292. Springer, 2022.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:2210.04150 , 2022.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

Ziqin Zhou, Bowen Zhang, Yinjie Lei, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. arXiv preprint arXiv:2212.03588 , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.05371 , 2019.

Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems , 32, 2019.

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems , 36, 2024.

Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5571-5584, 2023.

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV , pages 696-712. Springer, 2022.

Nir Zabari and Yedid Hoshen. Semantic segmentation in-the-wild without seeing any segmentation examples. arXiv preprint arXiv:2112.03185 , 2021.

Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Mohamed Elhoseiny, and Sean Chang Culatana. Exploring open-vocabulary semantic segmentation without human labels. arXiv preprint arXiv:2306.00450 , 2023.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 , 2021.

Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2085-2094, 2021.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 , 2021.

Matthias Minderer, Alexey Gritsenko, Maxim Neumann Austin Stone, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection with vision transformers. ECCV , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 , 2(3):4, 2021.

Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems , 36, 2024.

Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR , 2022.

Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14543-14553, June 2022.

Yin Zhaoyun, Wang Pichao, Wang Fan, Xu Xianzhe, Zhang Hanling, Li Hao, and Jin Rong. Transfgu: A top-down approach to fine-grained unsupervised semantic segmentation. In European Conference on Computer Vision , pages 73-89. Springer, 2022.

Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskclip. arXiv preprint arXiv:2208.08984 , 2022.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2582-2593, 2022.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7959-7971, 2022.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11976-11986, 2022.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1209-1218, 2018.

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302-321, 2019.

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 891-898, 2014.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-308, 2009.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2818-2829, 2023.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning openvocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2935-2944, 2023.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755. Springer, 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github.com/facebookresearch/detectron2 , 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Sehban Omer. fast-pytorch-kmeans, September 2020.

Terrance DeVries. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4268-4277, 2022.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343 , 2023.

Appendix

A Further Implementation Details

We set γ = 0 . 999 , input resolution as H = W = 640 , which results in h = w = 20 , and set h ′ = w ′ = 80 for ConvNeXt [49] backbones. For ViT [62] backbones, we set H = W = 320 , which also results in h = w = 20 . For global clustering, we set ε = 1 for ConvNeXt backbones and ε = 0 . 01 for ViT backbones. We implement our work using PyTorch [63] and Detectron2 [64]. AdamW [65] optimizer is used with a learning rate of 2 · 10 -4 for the decoder, 2 · 10 -5 for the prompt tokens and 2 · 10 -6 for CLIP, with weight decay set to 10 -4 . Prompt tokens are initialized as random word tokens with l = 4 , and k = 64 as default. We use GPU implementation [66] of k -means clustering for our experiments with DINO masks. For E V , we apply CutOut [67] and color augmentations [68] during training. For the prompt ensemble strategy during inference, we use the prompts curated originally from CLIP [15] in their repository, which results in total of 8 text prompt as follows:

B Additional Experiments

B.1 Results on Different Backbones

In Tab. 4, we show results for PixelCLIP when applied to different backbones. We note that since the ViT backbone has a larger output feature resolution scale compared to ConvNeXt models, we set the input image resolution to match the output feature resolution, and report results without prompt ensembling. In general, we observe noticeable gains across all backbones, with CLIP ViTB/16 outperforming ConvNeXt-B on several datasets. Through testing with various pre-trained CLIP models, we demonstrate that our method can effectively fine-tune CLIP for dense prediction regardless of the backbone architecture.

B.2 Analysis on Prompt Ensembling

In Tab. 5, we show results with prompt ensembling being applied during only training, only inference, and both. We report results with ViT-B/16 using SA-1B masks as supervision. Although prompt ensembling does bring slight gains when enabled during training, the computation for optimizing learnable class prompts scales along with the number of prompts used, increasing the training time and the memory consumption. On the other hand, applying prompt ensembling during inference only adds negligible cost as they can be computed once and be cached, but shows much significant gains compared to when applied training. Therefore, we adopt prompt ensemlbing only during inference, but noticing that the performance can be maximized with better prompts during training. In this regard we can better results with better prompt design or a learnable prefix to accompany the learnable class prompts, which we leave for future investigation.

B.3 Additional Ablation Studies

(a) Varying momentum γ . We show additional results for varying γ for the momentum update.

(b) Different training dataset. We show results for leveraging ground-truth masks from COCO-Stuff while removing its class labels.

B.3.1 Ablation on the momentum update rate γ

In Tab. 6 (a), we show results for varying γ for the momentum update. While having the momentum encoder generally shows improvements, we find γ = 0 . 999 to show the best results for updating the momentum encoder.

B.3.2 Ablation on training dataset

In Tab. 6 (b), we show results for training with mask annotation from COCO-Stuff [50]. For COCO-Stuff, we remove the ground truth class labels and utilize them as unlabeled masks, and other hyperparameters are set identically with k = 64 . Although the masks from COCO-Stuff show better results across all datasets, we highlight that the SA-1B [13] dataset mostly consists of automatically generated masks from SAM, whereas COCO-Stuff has human annotated masks from expert annotators.

(a)

𝑘 = 64

(b)

𝑘 = 128

(c)

𝑘 = 256

(a) Ours

(b) CLIP

land)

windowp

mountain

ceiling

hw

SAY

mountain

ceiling

wall

hill

field

wall

signboard

ball

bridge

house

lght

dmirror

water

water

step

floor

water

case

ceiling

land

ceiling]

step

wall

tower

wall

building

tree

lant

lamp

armcha

windowpane

pillow

bed

painting

sconce

building

flag

三区

tee

canopy

Lamo

armchair

windowpane

blind

Dillow

painting

paa

sconce

bench

plant

flower

bench

rug

canopy

(cabinet.

grass

floor

grass

loor

cushio

C Additional Qualitative Results

We provide qualitative results of visualization on ADE-20K [51], PASCAL-Context [52] in Fig. 8 and Fig. 9.

D Additional Visualization

In Fig. 7, we show visualization on COCO-Stuff by classifying the image features with our learned class prompts for varying k . From the first and the second row, we can observe that with larger numbers of k , different parts of human are segmented into fine-grained regions whereas k = 64 has

(a) Ours

(b) CLIP

mountain

punob

mountain

sky

tree

ground

sky

persd

cloth

poOM

tree

tree

plate

tence

boat

boat

water

water

ceiling

Light

ceiling

light

building

building

tree

window

sign

window

bus

wall

wall

snow

snow

sofa

floor

punob

(a) Ours (b) CLIP more coarse regions. Especially for k = 256 , in the second row, we observe the glasses, hair, and hands all classified into different classes with our learned prompt.

On the other hand, we also observe cases where a small number of k struggles to differentiate visual concepts in the last row, where the animals are partially grouped with k = 64 and show better groups for k = 128 and k = 256 . This could indicate that with only a small number of clusters, several fine-grained visual concepts that may not be seen often in the dataset to be grouped as a whole, whereas independent clusters could be assigned with a larger number of k , allowing fine-grained recognition of semantics.

E Limitations

Although we aim to fine-tune the image encoder of CLIP for adapting to dense predictions, we initialize the mask encoder within our framework with pre-trained weights of CLIP, which yields poor results for classifying masks when applying mask pooling to its features. Consequently, the noisy mask features in the earlier stage of training may result in sub-optimal performance. While there could be alternative methods to extract per-mask CLIP image features, we consider mask pooling to be sufficient to show meaningful improvements to CLIP and consider such exploration for future directions.

F Broader Impact

Our framework facilitates open-vocabulary semantic segmentation through leveraging visionlanguage models, hence the recognition capabilities of our method rely on the pre-trained knowledge of the vision-language models. Considering that large-scale pre-trained vision-language models [15, 16, 55] leverage web-crawled data within its training, the models may exhibit wrongful behaviors from bias or corrupted data from the internet which calls for future research to address.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We verify our claims in L54-L59 experimentally in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in Section E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not involve theory assumption and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide exhaustive details in Section 4.1 in the main paper, as well as further details in Section A in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We provide exhaustive details in Section 4.1 in the main paper, as well as further details in Section A in the supplementary materials for reproducing the experimental results. We are committed to releasing our code upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide exhaustive details in Section 4.1 in the main paper, as well as further details in Section A in the supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not report error bars, but we fix the random seed to minimize the stochasticity for all of our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the resources and training time of our method in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Anonymity is kept as shown in the first page.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impacts in Section F in the supplementary material.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have used only publicly available datasets. We have cited the original authors and respected the respective licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper present MCLIP, which aims to finetune CLIP using image and mask data without semantic labels. The goal is to make its open-vocabulary recognition ability to be adapted to position-sensitive semantic segmentation tasks. MCLIP chooses to use SAM or feature clusters of DINO to obtain masks, which are class-agnostic but sometimes too small to have consistent semantics for finetuning. In addition, in order to achieve a trade-off between training stability and avoiding catastrophic forgetting, clustering uses the EMA version of finetuned image encoder.\n\nSTRENGTHS:\n- The idea of this paper is interesting. Solving open-vocabulary semantic segmentation without semantic labels is an interesting topic.\n\n- The experimental results show that the preformance of the proposed approach is good.\n\n- Ablation studies and analysis are provided to help better understand of the role of each component and hyper-parameter.\n\nWEAKNESSES:\n- What role does the text encoder play in the clustering process?\n\nThe key to the success of clustering lies in the existence of unlabeled data in a certain grouping relationship in the space. The mask representations f_M can be considered to meet such conditions. If k learnable object prompts are initialized randomly, then k text representations f_C are randomly scattered in the text space initially, unable to provide the information needed for clustering. If k learnable object prompts are initialized by predefined categories, then this should violate the experimental setup without semantic labels. To illustrate this problem, the authors need to provide an explanation and add a comparative experiment: What results can be obtained just by clustering with f_M?\n\n- Why use clustering technology to solve the problem of too fine initial mask granularity?\n\nFirstly, previous work (GroupViT, CVPR'22 and SegCLIP, ICML'23) has used a grouping block designed based on the idea of clustering in the process of fine-tuning CLIP. In contrast, the clustering used in MCLIP provides better masks initially than the image patches used in previous work and the clustering process of MCLIP is parameter-free. It may be necessary to conduct ablation experiments to prove which differences bring positive results. But overall, technically, the use of clustering may be not novel.\n\n- In order to obtain class-agnostic masks, there is a class of experimental settings called open set semantic\nsegmentation (O3S, arXiv preprint arXiv:2307.02003, 2023) . Can the ”SAM initialization + clustering” scheme proposed by MCLIP obtain competitive performance in the O3S experimental setting?\n\n- Table 1 shows that using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP in the previous method improves the performance of the previous method’s zero-shot mask classification. Taking FC-CLIP as an example, MCLIP improves its zero-shot mask classification performance on ADE20K by 2.7. But compared to the final performance of FC-CLIP with two branches integrated, it is still 4.0 lower. So, in the case of keeping the complete inference strategy of the previous method, how much improvement can be achieved by using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP?\n\n- In the ablation experiment in Table 3a, does ’w/o Semantic Clustering’ mean fine-tuning with SAM-initialized masks directly without clustering? If so, why is the performance so poor? Without using clustering, it is equivalent to the number of cluster k is greater than the number of masks initialized by SAM on all images. However, the conclusion obtained from table 3b is that with the increase of k, the\nperformance tends to saturation, rather than decreasing to the level of ”w/o Semantic Clustering”.\n\nQUESTIONS:\nI think the authors need to further explain the novelty of this paper and give more analysis on the experimental results as listed in the weaknesses part.\n\nLIMITATIONS:\nLimitations are provided. It seems that there is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel unsupervised formulation of open vocabulary semantic segmentation, adapting a pre-trained vision-language model (CLIP) to the task via distillation from vision-only segmentation models (i.e., SAM, DINO). To use the language encoder without constraining fine-tuning to the set of classes of the dataset, they propose to apply online clustering at the dataset level and to learn class-specific feature prototypes. The method is compared against state-of-the-art approaches trained with and without supervision, and the results validate the approach as a practical direction for training open vocabulary models for semantic segmentation.\n\nSTRENGTHS:\n- The work is well-presented and curated, and the motivation is clear and sound.\n- The approach finetunes models pre-trained for a different purpose, and employs supervision from other pre-trained models, effectively reusing knowledge efficiently. Distillation from pre-trained models also removes the need for data supervision, potentially permitting scale training for open vocabulary semantic segmentation to billions of samples.\n- The method is tested against various baseline and on different benchmark datasets.\n- The model components are evaluated independently in the ablation study, helping to uncover the individual contribution to the final picture.\n\nWEAKNESSES:\n- While I understand the rationale behind performing clustering at pixel level on the entire dataset, I am not sure of the scalability of the approach. This probably explains why the authors only use 5% of the SA-1B dataset. It would be helpful to quantify the costs of performing the online clustering at training time.\n- Since the method could potentially suffer scalability issues, it would be interesting to understand performance when trained unsupervised in-distribution, reporting both in- and out-of-distribution performance, e.g., MCLIP trained on VOC and tested on Context.\n- The reasoning behind the selection process for the prompt (i.e., \"a photo of a {} in the scene\") is unclear. This is a minor issue, but the selection should probably be justified. While I would expect the model to learn to ignore the prefixes and suffixes, it would be interesting to understand how the model performs with other prompts.\n- Using CLIP as a baseline for the ablation studies may be misleading due to the mismatch between its application and open vocabulary semantic segmentation\n\nQUESTIONS:\n- What are the costs of performing online clustering at the dataset level during training?\n- What is the performance of MCLIP when trained on the benchmark datasets and tested in- and out-of-distribution? How does it compare against some baseline methods?\n- What is the model performance when changing the template at inference time? What is the performance without the template? What about training without the template? Does performance improve with prompt ensembling (i.e., similar to what CLIP does with N prompts that average to get better class centroids)?\n\nLIMITATIONS:\n- A potential limitation is the one I reported above, i.e., scalability issues due to the global online clustering procedure.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to learn an open-vocabulary semantic segmentation model with only unlabeled images and pretrained foundation models, such as SAM and DINO. The intuition is that CLIP model already knows what is in the image, so we only need to teach CLIP where the object is. It first uses pretrained DINO to generate pseudo masks and then exploits a online clustering method to group the part segments into valid object masks. It also proposes learnable class embeddings to solve the problem of lacking ground-truth text labels. Compared with baselines, the proposed method achieves decent improvements.\n\nSTRENGTHS:\n+ This paper proposes a solution to train an open-vocabulary model only with unlabeled images, i.e., without masks nor captions. \n\n+ Three technical contributions to make the solution happen: (1). use DINO to generate pseudo masks (2). group part masks into objects (3). user learnable embeddings to substitute text captions.\n\nWEAKNESSES:\n- Compared with other methods  [ 50 , 30 , 43 , 51 ] leveraging image caption as supervision, this paper actually uses stronger DINO generated pseudo masks to train the segmentation model. Furthermore, it even uses the SAM masks during experiments (In Table 2). Regrading this, I think this method includes strong segmentation prior into the training, making the comparison unfair.\n\n- Compared with methods using similar VFM, such as SAM-CLIP, the proposed method performs much worse. I understand it is not an apple-to-apple comparison due to the differences in training data. More specially, this paper doesn't use captions. However, caption is also very easy to get with pretrained caption models. If simply adding caption could boost the performance so much, why should we stick to a setting without caption?\n\nQUESTIONS:\n- Why does Table 1 appear before Table 2? Tabel 1 looks more like an ablation study to me.\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper proposes to enhance the semantic segmentation performance of the pretrained CLIP model using unlabeled images and pseudo segmentation masks generated with vision foundation models such as SAM and DINO. Specially, the pseudo masks are acquired via a online feature clustering algorithm. Experiments on standard benchmarks demonstrate the superior performance over CLIP and competitive results to existing open-vocabulary semantic segmentation methods.

STRENGTHS:
1. It is a neat idea to leverage unlabeled masks  as supervision generated from foundation models (e.g. SAM, DINO) for the open-vocabulary semantic segmentation task. 
2. The paper is well-written and easy to follow. 
3. The ablation study demonstrates the effectiveness of some design choices such as momentum update of the image encoder, online cluster assignment, and learning class prompts. Performance on the standard benchmarks also show the competitive results in comparison with existing baseline methods.

WEAKNESSES:
1. How is the proposed method in comparison with MaskCLIP+? Figure 4 shows the visual comparisons between the proposed method and MaskCLIP, as the Sec. 4.2 mentioned “For evaluating CLIP [42, 10], we apply MaskCLIP [61] to extract CLIP image features”. It would be interesting to compare the proposed method with MaskCLIP+ which distills MaskCLIP to train more advanced segmentation model. 
2. Some related works should be discussed and compared to the proposed method, for example, Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only. ICCV 2023.

QUESTIONS:
See the weakness section.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15