Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels
Authors: Heeseong Shin
Chaehyun Kim
Sunghwan Hong
Seokju Cho
Anurag Arnab
Paul Hongsuck Seo
Seungryong Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: semantic segmentation
open-vocabulary segmentation
vision-language models
fine-tuning foundation models
Primary area: machine_vision
TL;DR: We fine-tune CLIP for open-vocabulary semantic segmentation only from unlabeled masks without any semantic information, generated from SAM and DINO.

Abstract: Large-scale vision-language models like CLIP have demonstrated impressive open-vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which require understanding where the objects are located. In this work, we propose a novel method, PixelCLIP, to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where, which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to caption-supervised methods in open-vocabulary semantic segmentation.

Towards Open- Vecablocyyy Semantic Segmentation Without Semantic Labels
Heeseong Shin'
Chaehyun Kim
Sunghwan Hong"
Seokju Cho
Anurag Arnab
Paul Hongsuck Seo
Seungryong Kim 1.1
KAISS
Korea University
GGoole Research
(hsshin98, kchyun, seokju.cho, seungryong. kkaist. ac kr
(sung_hwan, phoeoe Skorea. kr.kr² aarnabšgoogle. com"
Abstract
Large-scale lssionllnguaae models like CLIP have demonstrated impressive open- vocabulary capabilities for image-level tasks, excelling in recognizing what objects are present. However, they struggle with pixel-level recognition tasks like semantic segmentation, which additionally require understanding where the objects are located. this work, propose novel method, PixeICLIP, to adapt the CLIP image encoder for pixel-level understanding by guiding the model on where, which is achieved using unlabeled images and masks generated from vision foundation models such as SAM and DINO. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm using learnable class names to acquire general semantic concepts. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to caption- supervised methods in open vocabllcyy semantic segmentation. Project page is available hhttt:t:////........ //cps://ccvlabb kaist. .ensst.ggtttu github. io/Pixe1CLIP
"photo of running
KUASS
(grass) field"
tebal semantic contnts.ccuttringg custerig
model
modell
model
Image
Image
Image (c) without semantic labels (ours)
(8) pisel-level semantic labels
(b) image-level semantic labels
Figure 1: Illustration of different approaches for open-vocabulary semantic segmentation. In
contrast to existing methods utilizing (a) pixel-level semantic labels D
3
4
5
or (b) image-level
semantic labels 7 8 9 10 12). leverage unlabeled masks supervision, which can be freely
generated from vision foundation models such as SAM [3] and DINO 14]
Introduction
Semantic segmentation fundamental task in computer vision where the goal is to identify class labels for each pixel within the given image. However, segmentation datasets often require extensive human effort obtain densely-annotatte semantic labels, limiting their scalability. In this regard, recent advances in large-scale pre-trained vision-language models, e.g. CLIP [15 and ALIGN [16].
"Crresponniig authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SAM masks
clustered SAM masks
DINO masks
clustered DINO masks
Figure Visualization masks from vision foundation models. We visualize the masks generated by SAM [3 and by clustering image features from DINO [14]. Although such models can freely generate fine-grained masks, the resulting masks can be too small or incomplete to have semantic meaning. To address this over-segmentation issue, we employ online clustering [18] of the masks into semantically meaningful groups defined globally for given images.
have facilitated open-vocabulary semantic segmentation 13 2 4], which aims to generalize semantic segmentation into unbounded range of classes. Despite showing remarkable generalization capabilities, they still require pixel-level semantic labels for leveraging the image-level pre-trained vision-language models for semantic segmentation.
Recently, several studies without densely-annotated semantic labels. These studies often utilize image-level semantic labels, such as image captions, to enhance the pre-trained vision-language models like CLIP for semantic segmentation. However, image captions typically provide information about what is in the image, but without where it is. Since CLIP is already effective in recognizing what the objects are, this causes models to only implicitly learn object locations, leading sub-optimal performance or requiring millions of image-caption pairs to compensate for this weak supervision [ [7. Instead, we focus on informing CLIP about where objects are located address the missing information.
7 [ have pioneered veen-vocabulary semantic segmentation
In this study, we propose a novel approach to achieve open- vocabulayy semantic segmentation without leveraging semantic labels, but through guiding the pre-trained vision- -anguage models, such CLIP, on where to look. We leverage recent vision foundation models (VFMs), such as DINO [14] and SAM 133, partition images into fine-grained regions indicate where to look. Consequently, explore methods to effectively leverage these masks for fine-tuning the image encoder of CLIP.
we do not have any captions or class names that can be fed to the text encoder of CLIP, To leverage its knowledge, we devise a method that employs prompt learning 220 21 on the text encoder of CLIP to construct learnable classes. Setting the learnable classes as a centroid, we propose applying the online clustering algorithm [18 [22] along the given masks to gather them into semantically meaningful groups, as shown in Fig. 2 We keep these learnable classes global across the entire images, which guides the learnable classes contain the general semantic concepts. Despite the absence of semantic labels, our method is able to jointly leverage the image encoder and text encoder of CLIP during training, successfully achieving dense open- vocabulayy recognition.
In contrast to existing works that leverage semantic labels 6 19
Our framework, called PixeICLIP, achieves significant improvements to CLIP, on average of +16.2 mloU in open-vocabulary semantic segmentation. Moreover, despite not using any semantic labels, PixeICLIP shows competitive performance in comparison to image-level supervised methods using captions 77 9 10), demonstrating the effecttiennes of unlabeled masks for. We further show the effectiveness of PixeICLIP for classifying masks from various open-vocabulary segmentation models, which can be simply done by replacing the CLIP within existing methods. We also provide extensive ablation studies to validate our choices, with a detailed analysis of our method.
We summarize our contribution as follows:
We propose novel formulation of learning from images without semantic label for open- vocabulary semantic segmentation by leveraging masks generated from DINO and SAM to fine-tune rision-langual models.
We propose to globally cluster semantically similar masks by employing an online clustering algorithm, while learning class prompts for representing semantic clusters.
We demonstrate significant gains in open- vocabulayy semantic segmentation, even surpassing methods leveraging image-level semantic labels, and provide thorough ablation studies with analysis to validate framework.

---

2 Related Work
2.1 Open-vocabulary semantic segmentation
Open-vocabulary semantic segmentation 2 23 aims to label each pixel within an image into an unbounded range of classes. In this regard, recent works [ 17
2 6 24) aim generalize to classes unseen during training through leveraging pre-trained isioonlanguaae models, such as CLIP [1]. Despite their remarkable performance, they leverage per-pixel semantic labels during their training, which requires expensive cost to annotate. Instead, we focus on the weakly-supervised setup, where the goal is zero-shot transfer segmentation task without densely-annotated class
labels
25 5 26 7 27], utilizzig image-level labels as supervision or even no labels at all.
12 25 leverage image caption supervision. GroupVi1 [T] and ViL-Seg [2] are pioncering works for identifying groups or clusters emerging from captions. Along with the advance of vision-language models, SegCLIP [ and TCL. 7 leverage pre-trained CLIP and learn additional decoder modules to learn dense vision-language alignment. PACL g learns additional embedding layers enhance the patch-level alignment in lision-language models and SAM-CLIP [I0] attempts to merge SAM [13] and CLIP [15] into a unified model by additionally leveraging unlabeled mask data from SAM. Apart from these approaches, we avoid employing any semantic labels 266 (88), but leverage vision foundation models to obtain masks as a source for super- vision fine-tune the CLIP image encoder for achieving open-vocabulary semantic segmentation.
In this regard, recent studies
2.2 Fine-tuning vision-language models for dense prediction
Recent large-scale pre-trained vision- Asioonlanguage models have shown its effectiveness for jointly under- standing images and language [29] [5 [6]. Notably, CLIP [1), trained with web-scale image-caption
pairs, has been widely popularized for transferring its open- open-vocabulary recognition capabilities to various downstream tasks (26
However, despite its success in image-level tasks like image classification, CLIP tends to struggle in dense prediction tasks [17 26 6). such as object detection and semantic segmentation. This originates from CLIP being trained from image- supervision being captions, hence exhibits bias towards the global image rather than fine-grained regions within the image (17] While non-leamable approaches, such as MaskCLII (26 show improvements by slightly modifying the architecture, CLIP still shows limited capabilities in dense predictions in comparison to its global understanding.
30
31 32
To address this, OWL-VIT 33 directly fine-tunes pre-trained vision and text encoders to downstream open-vocabulary detection task, and CAT-Seg [6] introduces cost aggregation scheme for fine- tuning the encoders of CLIP for semantic segmentation. Alternatively. ZegCLIP [9 and Xu et al. [3] implement prompt tuning 21 220 for tuning the image and text encoders of CLIP. Instead of fine-tuning the full model, they learn prompt tokens that serve as a global prefix for the encoders of CLIP. While such methods show remarkable results from fine-tuning the encoders of CLIP for dense downstream tasks, they require densely annotated detection and segmentation data for training.
2.3 Vision foundation models
With the advent of large-scale learning enabled by scalable vision backbone architectures 34 35 and vast amounts of data, diverse vision foundation models are emerging in the field of computer have demonstrated the effectiveness of its rich visual representations for various downstream tasks. Especially, DINO [44 exerted strengths in fine-grained semantic recognition 44 41). making it highly effective for object detection and
vision. In this regard, self-supervised methods [36 37
38
39
image segmentation. Moreover, DINO features have also been demonstrated for yielding fine- grained
masks within the image through applying the l--maans clustering with its features 42 44).
On the other hand, the segment anything model (SAM) [3] has demonstrated its capability for generating fine- -raiindd high-quality segmentation masks for any object in an image. Through its self- annotation pipeline, SAM has collected an unprecedented amount of mask annotation for achieving its capabilities. While we can freely leverage SAM to obtain detailed masks in any given image, we mainly utilize the pre-computed masks within the collected dataset, SA-1B. Both DINO and SAM, however, yield unlabeled masks without semantic labels as both models are also trained without semantic labels, presenting a challenge for leveraging their masks for achieving dense lasnunnlanguuge recognition.
3

---

cosine similarity
masked pooling
learnable parameters
parameters
image
mask decoder
binary mask loss
encoder By
fi image feature
M similarity map
M prediction
image
O assigned class feature
momentum
update
Sinkhorn-Knopp
photo of C+
image encoder
0 fc class features
text encoder E₁
photo of C
fR mask feature
Ey
fi feature
photo of Cc
M masks
online cluster assignment
learnable class prompts
training with image-mask pairs
open-vocabulary inference
photo "animal"
image
text
encoder
000
encoder
"tree"
Ey
features
photo "cress"
image
image
open-vocabulary prediction
Figure 3: Illustration of our overall framework. We provide illustration of PixeICLIP, utilizing unlabeled images and masks for fine-tuning the image encoder of CLIP, enabling open-vocabulary semantic segmentation. We note that the momentum image encoder and the mask decoder are only leveraged during training, and inference is only done with image and text encoders of CLIP.
3
Methodology
In this section, we first establish our problem formulation of learning dense vision- laiguul alignment from images paired with masks, generated from vision foundation models. Next, we discuss the challenges of leveraging masks supervision for fine-tuning the image encoder of CLIP and finally, present our methodology of semantic clustering of masks address the challenges.
3.1 Preliminaries
Given an input image HH each pixel within an image with classes given in free-form text. As a training signal, semantic labels offer a set of S textual descriptions for a semantic class 7 {Ti) related I. This can be directly utilized with the CLIP text encoder (-)(-) obtain text features JT & (T) RSSx where d is the hidden dimension. Dense image features fi (1) xxx where h x w is the output feature resolution, are then extracted. We finally obtain dense image-text similarity map Mrr E Rxxxxx
open-vocabulary semantic segmentation 6 aims to label
fr(2,0) fr(n) [lf 9) fr
MIr y,
(1)
This can be interpreted as soft binary masks predicted from image and text features of CLIP, and be supervised with binary mask loss Fmask in a pixel-level manner to fine-tune CLIP [6]
3.2 Integrating masks into CLIP features
In this work, we do not have any access to T. but are only given unlabeled masks E HXWNN where N denotes the number of masks for the given image /. Hence, devise methods to predict masks by incorporating M into CLIP features. We aim to fine-tune the CLIP image encoder (-) through leveraging unlabeled masks M supervision. Since M is generated from vision foundation models, egg. DINO or SAM, this presents us with the challenge of not having any semantic labels.
In order to integrate masks into CLIP, a straightforward approach would be employing the masks M with the CLIP image feature map fi to obtain per-mask CLIP features. While there could be various methods to extract regional CLIP features [26]] 45 5]. we apply mask pooling over fr to obtain mask

---

pooled features fM MaskPool MM) E R Consequently, we can leverage fM to obtain image-mask similarity map denoted MIM E :RixxxN;
MI,(I,y, f1(2,9) v) fa)(n) [ff(a, y) y)ll
(2)
This allows us supervise the model with a binary mask loss Lmakk for fine-tuning CLIP with given image / and unlabeled masks M. In practice, since MrM has the same resolution as the feature map from the CLIP image encoder Sr. we employ a light-weight decoder D mitigate the resolution gap between MrM and M, shown in Fig. 3 This can be written as D kx + g xwe' where h' x w' is resolution for the upsampled mask. Therefore, the output of the model can be updated as M = D(M).
3.3 Semantic clustering of masks
Upon using mask pooled CLIP image features far to predict MIM, however, find the masks generated from DINO and SAM to often over-segment the image, resulting in too small or incomplete masks as seen in Fig. 2 This would require CLIP to forcefully discriminate regions that are semantically similar, impeding the training process.
In this regard, propose to group semantically similar masks into clusters and predict based on the clusters rather than individual masks. Moreover, we aim to define this cluster globally, which is shared across the entire training process rather than for each image iteration. This would be analogous to constructing pixel-level semantic labels, where a fixed set of classes defined over the dataset is equivalent each cluster. However, the difference is that there is no pre-defined set of classes that we can define the clusters with. While we could heuristically pre-define such classes, we describe our learnable method for globally clustering masks into semantically meaningful groups.
Online clustering via learnable class prompts. To globally cluster masks into semantic categories, propose representing these clusters using CLIP text features centroids for clustering mask features. Given that the CLIP text encoder is trained with a broad understanding of natural language semanties, we expect these clusters to capture meaningful semantics by leveraging its comprchensive pre-trained knowledge. In this regard, we take Iearnable approach, where each cluster is defined by class-specific learnable prompts fed into the CLIP text encoder. Unlike existing prompt learning methods, which typically focus on learning task-specific prefix 120 21 B. we aim to learn prompt tokens that represent each class. For instance, in the sentence "A photo of an object", traditional prompting methods would learn the tokens for the "A photo of a" prefix, whereas our method focuses on learning the token for the "object."
Specifically, given the number of clusters k, we can define prompt tokens as C E RixxIxd, where / the token length of the prompt and d. is the dimension of the token embeddings. From this, can utilize the CLIP text encoder (-) to obtain a set of class features fc = {p O) E kxdd in the form of CLIP text features, where P* is fixed template for the CLIP text encoder, such as "A photo of a { in the scene." While we could assign each mask fas with fc in a winner-takes-all manner, we desire the classes to encode general semantics across all images. Therefore, we assume that we can equally divide m masks within a minibatch [18 14], into k clusters given a sufficient amount of masks.
Consequently, we aim to find an assignment Q E TR based on the image-text similarity between the mask pooled features JM and the class text features, which can be defined as:
max Tr(Q Tr(QT + +H((),, s.t. Q 11=== = 1m Q1m= QEO
(3)
where FM is the set all m mask features fM within the minibatch, and 1 denotes the k-dimeensional vector of ones. H is the entropy function, HQQ) E.j QiylogQi; with E as a hyperparameter. The solution Q from Eq. 3 is an assignment matrix defining which of the k clusters each m mask
should belong to, hence E determines the smoothness of this mapping Q by scaling the entropy regularization from H. The equipartition constraint, T 11, Q1 [1 encourages the
class features fc to selected least m/k times on average, allowing to learn general concepts represented by the masks within the dataset. In practice, with the soft assignment relaxation [46). Q
5

---

can be solved as follows:
FMJC E
(u) exp
diag(t),
where R v R denote renormalization vectors, which can be efficiently computed by Sinkhorn-Knopp algorithm [46].
Finally, we can re-write the prediction of our model to be a cosine-similarity map between /1 and fc:
MIC(F.U v, = [1(2,1) v) Sc( 1ff( y) ffll
(
thereby predicting masks for fc(i) being the i-th class feature from Sc. which we have obtained from clustering mask pooled features fai- Accordingly, ground truth masks M are also clustered according to Q by converting it into hard assignment with the argmax operator [47] [2]. This can be written as E xx where M, is the union of masks assigned into the cluster represented by i-th learned class fc(
Momentum encoder for integrating mask features. Since we jointly optimize the CLIP image encoder &y (.) as well as the learnable class feature fc, we may experience instability during our training process, or forgetting of the pre-trained knowledge 48] To stabilize the training, we keep a momentum encoder (39 38 for obtaining JM as seen in Fig- 3 Therefore, we update f'M as
fas Moolp M), where εy is the momentum encoder of the CLIP image encoder,
updated with momentum y. This can denoted as 8/ 766 + (1 7)yy, where 0p, By are model parameters of &y and εv, respectively.
4 Experiments
4.1
Implementation details
For training, employ per-pixel binary cross-entropy loss Lmask jointly train all of the components [6]. For all our experiments, we use a single text prompt "A photo of O in the scene" for Pt, including for our learnable class prompts while training and for inference, we apply prompt ensemble strategy (30 with 7 additional prompts originally curated from CLIP [15). We train our model on SA-IB 13 dataset, where we randomly sample 5% of the images. We train for 10000 iterations with batch size of 48 for all experiments. For experiments using masks from DINO, we obtain masks with k-means clustering where we A 16. For experiments using masks from SAM, we use the unlabeled mask annotation in the SA-1B dataset. Without specification, we report results on ConvNeXt-B [49 backbone with mask annotation from SAM, which takes approximately 6 hours to train with 4 NVIDIA A6000 GPUs. We provide more details in the supplementary materials.
4.2 Experimental setting
Following Cha et al. 77,, we evaluate our model on zero-shot transfer to semantic segmentation on
the validation sets of COCO-Stuti [50]. ADE-20K [51]. PASCAL-Conter [52), PASCAL VOC [59].
and CityScapes [54]. For CLIP [3]. we apply MaskCLIP (26 for ViT backbone for extracting image features, and remove the global pooling layer for OpenCLIP [55] with ConvNeXt [49] backbone. We
note that we do not apply any post-processing to the predictions and for the compared methods. For
the evaluation metric, we employ the mean Intersection over Union (mloU).
4.3 Results
Open-vocabulate semantic segmentation. We provide results for quantitative comparisons in Tab. [ We first compare with CLIP, and demonstrate remarkable gains in all benchmarks, bringing in an average of +16.2 mloU improvement. Since we do not have comparable baselines without leveraging semantic labels, we further provide comparison with image-level supervised methods 7 9, [0]. Surprisingly, PixeICLIP surpasses TCL. [7) and SegCLIP (9 in all benchmarks while using only a fraction the images without semantic labels. Furthermore, we show competitive performance compared to SAM-CLIP, which uses not only 40 million image-level semantic labels, but also leverages the SA-1B dataset on a similar scale to our framework.

---

Table 1: Quantitative comparison on open-vocabulary semantic segmentation. We compare in open- -oeebooaabl semantic segmentation with vision-language models, as well as image-level supervised methods. *; Images were seen during training. 1 Masks from SA-IB [33 were used.
Method
Training Dutaset
Backhome
Additoonl
Labels
VFM
cOCo-St.
ADE-150
Evaluation Datasel
Contect
CityScapes
VOC
GooupViT [1]]
CC12M [56] YFCCISM 577
VIT-S/16
15.3
9.2
23:4
11.1
79.7
CLIPpy M4
HQTTP-134M 75
VIT-B/16
13.5
52.2
OVSemmeiiee
CCAM
VIT-/16
DINO
53.8
WIT-100M
16.5
13.2
OpenCLIP 55
LAION-2B
Com-NeXI-B
12.8
13.1
16.5
16.2
34.8
Traniin with additional jnwige-lenel semuntic lubels
SegCLIP 9
COCO [60) CC12M [6
WIT-6116
Captions Captions Captions
CLIP
26.5
24.7
52.6
TC1
OC3M, CC3M,CCI2M CC12M
VIT-R/16
CLIP
19.6
14.9
30.3
25.1
77.5
SAM-CLIP
Merged-41M
[D]
VIT-R/16
CLI
17.1
Training additional somandi
ZeroSeg
[8
ImageNet-1K [1]
VIT-B/16
CLIP
20.2
20:4
40.8
VIT-B/16
CLIP DINO
22.2
17.4
34.3
22.9
83.8
PixeCCUPP
5% SA-1B
ConwNeXi-B
VIT-B/16
CLIP. SAM!
CLIP DINO
23.6
20.2
19.4
18.7
37.9
32.7
27.2
85.9
62.9
ComwNeXi-B
CLIP. SAM!
21.4
20.3
35.4
34.8
67.2
Table 2: Quantitative comparison on zero-shot mask classification. We compare the results
for mask classification using ground truth masks and generated masks from ZegFormer [[] and
FC-CLIP [24]. To evaluate zero-shot mask classification from CLIP, we report the results from the zero-shot branch for both methods.
Evaluation Dataset Context
VLM
Method
Backbone
COCO-St.
ADB-150
CityScapes
VOC
OpenCLIP 55
Zegformer Zeglormer
ConvNeXt-B
15.3
19.1
24.7
26.5
51.8 71.5 (+19.7)
PixeICIIP (Ours)
ComyNeXt-B
23.9 (+8.6)
21.5 (+2.4)
38.5 (+13.8)
34.2 (+7.7)
OpenCLIP SS
FC-CLIP 24
ConvNeXt-L
37.3
27.4
42.8
35.8
91.4
PixcICLIP (Ours)
FC-CLIP 24
ConvNeXI-L
46.8 (+9.5)
30.1 (+2.7)
52.2 (49.4)
48.1 (+12.3)
90.7 (-0.7)
OpenCLIP [55]
Ground Truth
ConvNeXi-B ConvNeXi-B
23.8
30.2
31.4
32.8
68.3
PixcICLIP (Ours)
Ground Truth
34.2 (+10.4)
34.6 (+4.4)
51.2 (+18.4)
41.4 (+8.6)
85.4 (+17.1)
Zero-shot mask classification. We provide results for evaluating mask classification in Tab. We consider ZegFormer I and FC-CLIP (4 as baselines since they first predict masks, then employ CLIP zero-shot mask classifier within their framework, and also provide results with ground -ruth masks to simulate having oracle mask predictions. For methods, we apply masked pooling to CLIP image feature map to classify masks. For ZegFormer [ and FC-CLIP 244. reported results are only from the zero-shot prediction branch to solely ablate our gains. We highlight that PixeICLIP can be readily applied to existing frameworks that leverage CLIP as a zero-shot mask classifier, and bring instantaneous improvements by simply replacing the model and weights of CLIP.
Qualitative results. We provide qualitative results for open- open-vocabulary semantic segmentation in Fig. 4 compared with results from CLIP, highlighting the dense open-vocabulary recognition capabilities of our framework. We further provide qualitative results in the supplementary materials.
4.4 Ablation studies
In T.b. we show ablation studies open-vocabulary semantic segmentation to validate our design choices. We report results without prompt ensembling for ablations, and also report results from baseline.
OpenCLIP
55
Component analysis. In Tab. 3 (a), we provide results for ablating our key components. Notably, we observe that without global semantic clustering masks, the framework collapses and loses the pre-trained knowledge of CLIP. This validates the challenge presented by leveraging unlabeled masks and demonstrates the crucial role of our proposed clustering approach. Moreover, we observe constant improvements over all datasets with learnable class prompt, proving our approach of leveraging the text encoder of CLIP to define the clusters in the form of prompt learning. We also observe constant gains with the momentum encoder for extracting mask pooled features fM-
Number of clusters. In Tab. (b), we compare the results of the variants of the proposed method by varying the number clusters We find that scaling & does not necessarily guarantee performance boosting, but it generally improves until A is set to 64 and tends to degrade as k grows. Considering that with an extremely large number for k; we can assign each the masks to individual clusters(e.gg 1 billion for SA-IB.) This scenario would virtually be identical to not having semantic clustering as
7

---

5O
(a) Ours
(b) CLIP
Figure 4: Comparison between PIXelCLIP and CLIP. We provide qualitative comparison on ADE-
20K [51] dataset with PixeICLIP and CLIP. We demonstrate the dense visual recognition capabilities achieved from fine-tuning CLIP, whereas CLIP shows results with significant noise.
Table 3: Ablation studies. We show results on open-vocabulary semantic segmentation for validating our design choices. We also report results from OpenCLIP [55] as baseline in the results.
Component
coco
ADE-150
Evaluztion Dutaset
Context
CityScapes
VOC
COCO
ADE-150
Evalcation Dataset
Contest
CityScapes
VOC
Bascline
Baseline
12.8
13.1
16.5
16.2
34.8
33.0
31.3
60.5
w/o Semantic Clustering
0.8
2.1
4.2
64
21.1
20.2
34.2
33.2
66.0
wio CLIP Texi Encuder
17.9
18.5
29.9
28.9
53.5
128
21.0
20.3
33.5
30.1
64.1
wlo Class Prompt
18.2
18.8
30.1
28.1
54.4
256
21.3
20.4
33.6
30.0
64.
wio Momencum
18.5
28.8
27.2
55.2
20.2
32.7
29.8
62.7
(a) Component analysis. Wc validate the core components of our framework by ablating cach components. Notably, global clustering masks shows its importance for facili- tating the framework.
(b) Number of clusters. For varying k, we find that scaling à larger than 64 does show much
improvements, while 32 also show competi- tive results.
Evaluation Dataset Context
Evaluation Dataset Context
COCO
ADE-150
CityScapes
VOC
TeXL.
COCO
ADE-150
CityScapes
VOC
Baseline
12.8
13.1
16.5
16.2
34.8
Baseline
12.8
13.1
16.5
16.2
34.8
20.2
19.7
32.7
30.8
64.5
Ours COCO
21.1
20.2
34.2
33,2
66.0
21.1
20.2
34,2
33.2
66.0
19.5
17.7
30.0
24.8
63.3
10
33.2
20
19.9
19.7
32.6
33.8
62.8
(d) Effects of utilizing learnable classes. We com- pare our method of learnable class prompts to having fixed of classes from COCO-Stufi (50).
(c) Length of learnable prompt token. For varying 1. we find that shows best overall performance.
seen in Table. (a), and progressively growing ls would slowly converge to this scenario. We further provide analysis in 4.5 studying the different aspects from varying k.
Length to represent learnable class prompts. Tab. 3 (C) compares the effects of varying the length of the learnable class prompt, I. We find that / shows lower scores in comparison to other lengths. We can interpret this as only describing class with a single word, whereas having multiple words would better describe the depicted class. However, for I and larger, we find that increasing does not result in a gain of performance, hence, we adopt / = 4 as default.
Effects of learnable prompt token. Finally, we compare PixeICLIP to having pre-defined set of classes instead using learnable prompt tokens. Specifically, we use 171 classes from COCO- Stuff [50], and do not apply online clustering for assignment when utilizing classes from COCO-Stie Stuff, as it already yields text features with semantic meanings. We find apparent improvements over all the datasets as shown in Tab. (d). We speculate that since the classes defined in COCO-Stuf' are heuristically chosen, is hard to ideally encompass various semantics and concepts that may appear in images, hence restricting the perception of the model to the finite set of classes.

---

(B)
(b) 128
(cl
(d) 128
Figure 5: Visualization of learned class prompts. We visualize the text features from our learned class prompts, as well as text features from classnames of COCO-Stuff with t-SNE visualization in (a-b). We also visualize images inferenced with the learned class prompts in (c-d).
4.5 Analysis
Learnable class prompt. We further analyze the learned class prompt in Fig. (a-b) with I-SNE visualization the text features encoded from the learned class prompts, well as text features obtained from class names COCO- Stuff.. Since we initialize the class prompt tokens as random tokens, observe that they in skewed distribution in the initial state. However, the learned prompts show that they are well-dispersed among the text features from COCO-Stufi indicating that the class prompts have well-learned diverse semantic concepts within the text features. We observe well-distributed features both for = 64 and k = 128.
Since the learned prompts should act implicit class names, we visualize the results from inference with learned class prompts in Fig. 5 (c-d). Although both = 64 and k 128 show similar performance when evaluated, we observe that the prompts have learned more fine-grained semantics
for & = 128. We generally observe human parts be well distinguished: this could come from SA-1B dataset, there numerous images with fine- ane-grained masks representing human parts as annotations.
TO JAMaIn
ator
house
escalator ttr
(a) k 64
(b) 128
Figure 6: Visualization of interpreting learned text prompt. We provide visualization on results for predicting with learned class prompts, then mapping the results to classes in the dataset with the highest similarity to the prompt.
Interpreting learned classes. Considering the learned class prompts represent semantic concepts, we further study the learned embeddings by mapping each class embeddings to class names in COCO-Stuff with the highest cosine-similarity score. Fig. shows results when first inference the image features with learned class prompts, then map the results with the closest COCO classes. We can observe that with & 128, as the prompt learns more diverse semantics. we observe more accurately mapped classes, However, we still see predictions with large disparity to the actual ground truth. We leave more in-depth analysis of the learned classes for future investigation.
Conclusion
In this paper, we introduced PixeICLIP, framework for leveraging unlabeled images and masks for fine-tuning the pre-trained lisouallanguage models for open-voca vocabulaay semantic segmentation. To address the unique challenges posed by incorporating unlabeled masks generated by vision foundation models into our framework, we propose global semantic clustering of the masks, with learnable class prompts to represent each cluster. We demonstrated PixeICLIP show remarkable improvements to CLIP and its applicability to existing methods, providing instantaneous improvements, as well as surpassing methods that leverage image-level semantic labels such image captions.

---

This research was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea governmett (MSIT) (RS-2019- 11190075, RSS-0024-00509229, RS-2020-11201 819, RS-2024-00398 15, Research on the reliability and coherence of outcomes produced by Generative Al) and the Culture, Sports, and Tourism R&D) Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism (RS- 2224--04444,, R8-2023-00266509, and National Research Foundation of Korea (RS-2024-00346597).
Acknowledgement.
References
[1] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583-11592, 2022.
[2] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 540-557. Springer, 2022.
[3] Mengde Xu, Zheng Zhang. Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 736-753. Springer, 2022.
[4] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for opcauuocybylary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2945-2954, 2023.
[5] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulas panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955-2966 2023.
[6] Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seun- gryong Kim. Cat-seg: Cost aggregation for open- open-vocabulary semantic segmentation, 2024.
[7] Junbum Cha, Jonghwan Mun, and B Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165-111 11174, 2023.
[8] Jishnu Mukhoti, Tsung-Ya Yu Lin, Omid Poursaced, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
pages
[9] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In International Conference on Machine Learning, pages 23033 23044.. PMLR, 2023.
[10] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. arXiv preprint arXiv:2310. 15308, 2023.
[11] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144 2022.
[12] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open- world semantic segmentation via contrasting and clustering lasion- embedding. In European Conference on Computer Vision, pages 275-292. Springer, 2022.
10

---

[13] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Men Lo, al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
[14] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7774-88632 PMLR, 2021.
[16] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and usion-language representation learning with noisy text supervision. International Conference on Machine Learning, pages 9904-4916 PMLR, 2021.
[17] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint 10Xiv2210.00150, 2022.
[18] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:99 13:9912-9922 2020.
[19] Ziqin Zhou, Bowen Zhang, Yinjie Lei, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. arXiv preprint arXiv:2212.03588 2022.
[20] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16816-16822 2022.
[21] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009))2337-2448, 2022.
[22] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.0537/, 2019.
[23] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
[24] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chen. Convolutions die hard: ehee-loccbbulryy segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36, 2024.
(25) Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping contrastive vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5571-5584, 2023.
[26] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, pages 12. Springer, 2022.
[27] Nir Zabari and Yedid Hoshen. Semantic segmentation in-the-wild without seeing any segmenta- tion examples. arXiv preprint arXiv:2112.031155. 2021.
[28] Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Mohamed Elhoseiny, and Sean Chang Culatana. Exploring open-vocabulary semantic segmentation without human labels. arXiv preprint arXiv:2306.0044 2023.
[29] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for lision-and-language tasks. Advances in neural information processing systems, 32, 2019.

---

[30] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2021.
[31] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen- -r, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2085-2094, 2021.
[32] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: reference-free evaluation metric for image captioning, arXiv preprint arXiv:2104.087/8, 2021.
[33] Matthias Minderer, Alexey Gritsenko, Maxim Neumann Austin Stone, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang. Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detection with vision transformers. ECCV, 2022.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[35] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[36] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
[37] Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
[38] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 2020.
[39] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[40] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.058844 2(3):4, 2021.
[41] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances Neural Information Processing Systems, 36, 2024.
[42] Luke Jelas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral meth- ods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022.
[43] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufrey- daz. Self-supervised transformers for unsupervised object discovery using normalized cut. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 14543-14553, June 2022.
[44] Yin Zhaoyun, Wang Pichao, Wang Fan, Xu Xianzhe, Zhang Hanling, Li Hao, and Jin Rong. Transfgu: A top-down approach to fine-grained unsupervised semantic segmentation. In European Conference on Computer Vision, pages 73-89. Springer, 2022.
[45] Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskelip. arXiv preprint arXiv:2208.089% 2022.
12

---

[46] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems. 26, 2013.
[47] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1582-2593, 2022.
[48] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[49] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11976- 1198,, 2022.
[50] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1209-1218, 2018.
[51] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127:302-321, 2019.
[52] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gry Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 89) 199- 2014.
[53] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-308, 2009.
[54] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[55] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818 2829, 2023.
[56] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, 2021.
[57] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[58] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2935-2944, 2023.
[59] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[60] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, pages 740-755. Springer, 2014.
13

---

[61] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[62] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
[63] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[64] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://g thub. wwtp:///ccccooerrerrrrrtettttttt 2019.
[65] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint drXiv:1711.05101, 2017.
[66] Schban Omer. fast-pytorch sapytorch-mmann. kmeans, September 2020.
[67] Terrance DeVries. Improved regularization of convolutional neural networks with cutout, arXiv preprint arXiv: /708. 04552, 2017.
[68] Lihe Yang, Wei Zhuo, Lei Qi. Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi- supersuped semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4268 4277, 2022.
[69] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, arXiv preprint arXiv:2303. 2023.
14

---

Appendix
A Further Implementation Details
We set 7 = 0.999, imput resolution as H = W = 640, which results
w = 20, and set h' w' = 80 for ConvNeXt [49] backbones. For ViT [62 backbones, we set H W 320, which also results in = w = 20. For global clustering, we set = 1 for ConvNeXt backbones and 0.01 for ViT backbones. We implement
work using PyTorch (63 and Detectron2 [64]. AdamW [65] optimizer is used with learning rate of 2 10 for the decoder, 2 2000 for the prompt tokens and 10-6 for CLIP, with weight decay
10 Prompt tokens initialized as random word tokens with = 4, and k = 64 as default. We use GPU implementation [66] of k-means clustering for our experiments with DINO masks. For Ey, apply CutOut 67 and color augmentations [68] during training. For the prompt ensemble strategy during inference, we use the prompts curated originally from CLIP [15] in their repository, which results in total of 8 text prompt as follows:
"itap of a 0.",
bad photo the 0.",
"a origami O.",
"a photo of the large 0.",
a video game.", "art of the 0.",
"a photo of the small 0.",
photo of a 0 in the scene".
B Additional Experiments
Table 4: Quantitative results with various backbones. We show results on opea-vocybulary semantic segmentation of various CLIP backbones with the addition of ViT-B from SigLIP (69 and ConvNeXt-L.
Evaluation Dataset Context
Method
Backbone
COCO-St.
ADE-150
CityScupes
VOC
IP 69
VIT-R/16
62
12.4
11.8
18.3
19.2
46.8
PixeICIIP (Ours)
ViT-B/16
20.0 (+7.6)
19.2 (+7.4)
33.1 (+14.8)
31.6 (+12.4)
72.3 (+25.5)
CLIP 26
ViT-B/16 62
16.5
13.2
25.6
14.9
73.9
PixeICLIP (Ours)
ViT-B/16
62
21.4 (+4.9)
16.7 (+3.5)
34,9 (+9.3)
23.8 (+8.9)
83.1 (+9.2)
OpenCLIP 55
ConvNeXi-B 49
12.8
13.1
16.5
16.2
34.8
PixeICIIP (Ours)
ConvNeXt-B
49
21.1 (+8.3)
20.2 (+7.1)
34,2 (+17.7)
33.2 (+17.0)
66.0 (+31.2)
OpenCI.IP
ConyNeXt-I.
16.9
15.2
22.9
17.1
57.2
PixeICIIP (Ours)
ConwNeXt-I.
24.8 (+7.9)
22.6 (+7.4)
(+16.5)
34.3 (+17.2)
78.9 (+21.7)
B.1 Results on Different Backbones
In Tab, 4 we show results for PixeICLIP when applied to different backbones. We note that since the backbone has larger output feature resolution scale compared ConvNeXt models, we set the input image resolution to match the output feature resolution, and report results without prompt ensembling. In general, we observe noticeable gains across backbones, with CLIP ViT- B/16 outperforming Con-NeXt-B on several datasets. Through testing with various pre-trained CLIP models, we demonstrate that our method can effectively fine-tune CL.IP for dense prediction regardless of the backbone architecture.
15

---

Table 5: Additional experiments on prompt ensembling. We show results on open- vocabulary semantic segmentation with prompt ensembling being used during only training, only inference, or both. The default setting of prompt ensembling only being used during inference is highlighted in gray.
Prompt Ensembling Training Inference
Evaluation Datasel Context
COCO-St.
ADE-150
CityScapes
VOC
21.4
16.7
34.9
23.8
83.1
23.6 (+2.2)
18.7 (+2.0)
37.9 (+3.0)
27.2 (+3.4)
85.9 (+2.8)
21.6 (+0.2)
17.1 (+0.4)
35.1 (+0.2)
24.9 (+1.1)
82.9 (-0.2)
23.7 (+2.3)
19.2 (+2.5)
37.9 (+3.0)
28.1 (+4.3)
85.5 (+2.4)
B.2 Analysis on Prompt Ensembling
In Tab. we show results with prompt ensembling being applied during only training, only inference, and both. We report results with VÝT-B/16 using SA-IB masks as supervision. Although prompt ensembling does bring slight gains when enabled during training, the computation for optimizing learnable class prompts scales along with the number of prompts used, increasing the training time and the memory consumption. On the other hand, applying prompt ensembling during inference only adds negligible cost as they can be computed once and be cached, but shows much significant gains compared to when applied training. Therefore, but noticing that the performance can be maximized with better prompts during training. In this regard we can better results with better prompt design learnable prefix to accompany the learnable class prompts, which we leave for future investigation.
adopt prompt ensemlbing only during inference,
B.3 Additional Ablation Studies
Table 6: Additional ablation studies. We show results open-vocabulary semantic segmentation with a larger number of clusters and different training datasets.
Evaluation
COCO
ADE-150
Context
CityScapes
VOC
Datasct
COCO
ADE-150
Evaluntion Dataset
Context
CityScapes
VOC
Baseline 55
12.8
13.1
16.5
16.2
34,8
Baseline 53
12.8
13.1
16.5
16.2
348
32.5
COCO-SL SA-1B
0.999
21.1
20.2
34.2
33.2
66.0
13
21.1
20.2
34.2
33.2
66.0
0 0.9999
20.4
19.7
32.0
29.9
63.0
(b) Different training dataset. show results for leveraging ground-truth masks from COCO-Stuff while removing its class labels.
(a) Varying momentum 7. We show additional re- sults for varying for the momentum update.
B.3.1 Ablation on the momentum update rate y
In Tab, 6 (a), we show results for varying for the momentum update. While having the momentum encoder generally shows improvements, find y = 0.999 show the best results for updating the momentum encoder.
B.3.2 Ablation on training dataset
In Tab. 6 (b), we show results for training with mask annotation from COCO-Stuff [50]. For COCO-Siuff, we remove the ground truth class labels and utilize them as unlabeled masks, and other hyperparameters are set identically with & 64. Although the masks from COCO-Stuff show better results across all datasets, we highlight that the SA-1B [3] dataset mostly consists of automatically generated masks from SAM, whereas COCO-Stuff has human annotated masks from expert annotators.
16

---

Ia) 64
(b) k 128
(c) k = 256
Figure 7: Visualization on COCO-Stuff with learned class prompts. We provide results with learned classes with different & up 256.
(a) Ours
(b) CLIP
Figure 8: Visualization on ADE-20k We provide qualitative comparison ADE-20K [51]
C Additional Qualitative Results
We provide qualitative results of visualization on ADE-20K 51. PASCAL-Context
52 in Fig
and Fig.
D Additional Visualization
In Fig. we show visualization on COCO-Stufi by classifying the image features with our learned class prompts for varying k. From the first and the second row, we can observe that with larger numbers of k, different parts of human are segmented into tne-grained regions whereas & 64 has
17

---

HOUPTA
(a) Ours
(b) CLIP
Figure 9: Visualization on PASCAL-Context. We provide qualitative comparison on PASCAL- Context 52
more coarse regions. Especially for 1: = 256, in the second row, we observe the glasses, hair, and hands all classified into different classes with learned prompt.
On the other hand, we also observe cases where a small number of struggles to differentiate visual concepts in the last row, where the animals are partially grouped with k: = 64 and show better groups for l 128 and k 256. This could indicate that with only a small number of clusters, several fine-grained visual concepts that may not be seen often in the dataset to be grouped as whole, whereas independent clusters could be assigned with a larger number of k, allowing fine-grained recognition of semantics.
E
Limitations
Although we aim to fine-tune the image encoder of CLIP for adapting to dense predictions, we initialize the mask encoder within our framework with pre-trained weights of CLIP, which yields poor results for classifying masks when applying mask pooling to its features. Consequently, the noisy mask features in the earlier stage of training may result sub-optimal performance. While there could be alternative methods to extract per- mask CLIP image features, we consider mask pooling to be sufficient to show meaningful improvements CLIP and consider such exploration for future directions.
F Broader Impact
Our framework facilitates open-vocabulary semantie segmentation through leveraging vision- language models, hence the recognition capabilities of method rely the pre-trained knowl- edge of the vision-language models. Considering that large-scale pre-trained vision-language mod- els [1 16 55 leverage web-crawled data within its training, the models may exhibit wrongful behaviors from bias or corrupted data from the internet which calls for future research to address.
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We verify our claims in L54-L59 experimentally in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section E.
19

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not involve theory assumption and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide exhaustive details in Section 4.1 in the main paper, as well as further details in Section A in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We provide exhaustive details in Section 4.1 the main paper, as well as further details in Section A in the supplementary materials for reproducing the experimental results. We are committed to releasing our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/nnips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 7/nips. wwwpcccccccc/.....c....co..eetle
wwtppp/guuides///ppppppppopppaep guwdee/eee//eee CooeeSumisssii sssssoooicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide exhaustive details in Section 4.1 in the main paper, as well as further details in Section A in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars, but we fix the random seed to minimize the stochasticity for all of our experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the resources and training time of our method in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Anonymity is kept shown in the first page.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Section F in the supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
23

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have used only publicly available datasets. We have cited the original authors and respected the respective licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pww.::s...aaseiiiii...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?

---

Answer:
[NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "hwlqQTqVdB", "reviewer": "Reviewer_Dx6v", "summary": "This paper proposes to enhance the semantic segmentation performance of the pretrained CLIP model using unlabeled images and pseudo segmentation masks generated with vision foundation models such as SAM and DINO. Specially, the pseudo masks are acquired via a online feature clustering algorithm. Experiments on standard benchmarks demonstrate the superior performance over CLIP and competitive results to existing open-vocabulary semantic segmentation methods.", "questions": "See the weakness section.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is a neat idea to leverage unlabeled masks  as supervision generated from foundation models (e.g. SAM, DINO) for the open-vocabulary semantic segmentation task. \n2. The paper is well-written and easy to follow. \n3. The ablation study demonstrates the effectiveness of some design choices such as momentum update of the image encoder, online cluster assignment, and learning class prompts. Performance on the standard benchmarks also show the competitive results in comparison with existing baseline methods.", "weaknesses": "1. How is the proposed method in comparison with MaskCLIP+? Figure 4 shows the visual comparisons between the proposed method and MaskCLIP, as the Sec. 4.2 mentioned “For evaluating CLIP [42, 10], we apply MaskCLIP [61] to extract CLIP image features”. It would be interesting to compare the proposed method with MaskCLIP+ which distills MaskCLIP to train more advanced segmentation model. \n2. Some related works should be discussed and compared to the proposed method, for example, Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only. ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Dx6v for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below:\n> **Comparison with MaskCLIP+**\n> \n\n|  | COCO-St. | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- |\n| MaskCLIP+[61] | 18.0 | - | 31.1 | - | - |\n| ZeroSeg[a] | 20.2 | - | 20.4 | - | 40.8 |\n| MCLIP (ours) | 21.4 | 16.7 | 34.9 | 23.8 | 83.1 |\n\n*Table A. Additional comparison on open-vocabulary semantic segmentation with other methods.*\n\nWe provide comparison with MaskCLIP+ in Table A, with results from ViT-B/16 CLIP backbone. Despite MaskCLIP+ having an advanced decoder for segmentation and requiring training on the target dataset, we demonstrate that MCLIP outperforms MaskCLIP+ in both COCO-Stuff and PC-59, demonstrating the effectiveness of our approach.\n\n> **Some related works should be discussed and compared**\n> \n\nWe thank the reviewer for pointing out related works with our work. We provide results from ZeroSeg[a] as mentioned by the reviewer in Table A, and we will add discussions and comparison to ZeroSeg and MaskCLIP+ to the paper.\n\nReferences:\n\n[a] Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only. J. Chen et al., ICCV 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KF443L9Ekx", "reviewer": "Reviewer_rHvB", "summary": "This paper proposes to learn an open-vocabulary semantic segmentation model with only unlabeled images and pretrained foundation models, such as SAM and DINO. The intuition is that CLIP model already knows what is in the image, so we only need to teach CLIP where the object is. It first uses pretrained DINO to generate pseudo masks and then exploits a online clustering method to group the part segments into valid object masks. It also proposes learnable class embeddings to solve the problem of lacking ground-truth text labels. Compared with baselines, the proposed method achieves decent improvements.", "questions": "- Why does Table 1 appear before Table 2? Tabel 1 looks more like an ablation study to me.", "limitations": "See weakness", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ This paper proposes a solution to train an open-vocabulary model only with unlabeled images, i.e., without masks nor captions. \n\n+ Three technical contributions to make the solution happen: (1). use DINO to generate pseudo masks (2). group part masks into objects (3). user learnable embeddings to substitute text captions.", "weaknesses": "- Compared with other methods  [ 50 , 30 , 43 , 51 ] leveraging image caption as supervision, this paper actually uses stronger DINO generated pseudo masks to train the segmentation model. Furthermore, it even uses the SAM masks during experiments (In Table 2). Regrading this, I think this method includes strong segmentation prior into the training, making the comparison unfair.\n\n- Compared with methods using similar VFM, such as SAM-CLIP, the proposed method performs much worse. I understand it is not an apple-to-apple comparison due to the differences in training data. More specially, this paper doesn't use captions. However, caption is also very easy to get with pretrained caption models. If simply adding caption could boost the performance so much, why should we stick to a setting without caption?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer rHvB for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below:\n> **Compared with methods using similar VFM, such as SAM-CLIP, the proposed method performs much worse.**\n> \n\n|  | COCO-Stuff | COCO-Object | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- | --- |\n| GroupViT[50] | 15.3 | 27.5 | 9.2 | 23.4 | 11.1 | 79.7 |\n| SAM-CLIP[43] | - | 31.5 | 17.1 | 29.2 | - | 60.6 |\n| MCLIP (DINO) | 20.2 | 41.9 | 15.6 | 31.3 | 19.2 | 80.0 |\n| MCLIP (SA-1B) | 21.4 | 43.3 | 16.7 | 34.9 | 23.8 | 83.1 |\n\n*Table B. Additional comparison on open-vocabulary semantic segmentation including COCO-Object, evaluated on 80 instance classes of COCO-Stuff.*\n\nThank you. We would first like to point out that SAM-CLIP actually reported results on COCO-Object and not COCO-Stuff, which we have verified with the authors of SAM-CLIP. We therefore present the corrected comparisons in Table B, which we will also include in the revision. We can see that our proposed MCLIP does indeed outperform SAM-CLIP on COCO-Object, consistent with PC-59 and VOC, and is only marginally behind by 0.4 points on ADE. We achieve this strong performance despite SAM-CLIP additionally using 40M image-text pairs, showing the benefits of our approach.\n\n> **If simply adding caption could boost the performance so much, why should we stick to a setting without caption?**\n> \n\nWe would like to highlight the main contribution to be our exploration for effectively leveraging unlabeled masks, which has also been acknowledged by other reviewers to be a “neat idea” (Dx6v), “motivation is clear and sound” (kepm) and “interesting” (83hz), hence focus on masks instead. The framework can be potentially enhanced by incorporating captions along with unlabeled masks, which we anticipate for future exploration.\n\n> **Compared with other methods [ 50 , 30 , 43 , 51 ] leveraging image caption as supervision, this paper actually uses stronger DINO generated pseudo masks**\n> \n\nWe emphasize that we do not need human labels within our training as DINO is trained in a self-supervised manner with only images. In contrast, the mentioned methods leverage human-annotated captions in addition to images, which makes it hard to consider that our method is leveraging stronger supervision. Furthermore, we also highlight that SAM is leveraged to demonstrate scenarios with higher quality unlabeled masks, and MCLIP still shows strong performance with DINO generated masks.\n\n> **Table 1 looks more like an ablation study to me.**\n> \n\nWe thank the reviewer for the suggestion, and will adjust the tables accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hK1wlTfasZ", "reviewer": "Reviewer_kepm", "summary": "The authors introduce a novel unsupervised formulation of open vocabulary semantic segmentation, adapting a pre-trained vision-language model (CLIP) to the task via distillation from vision-only segmentation models (i.e., SAM, DINO). To use the language encoder without constraining fine-tuning to the set of classes of the dataset, they propose to apply online clustering at the dataset level and to learn class-specific feature prototypes. The method is compared against state-of-the-art approaches trained with and without supervision, and the results validate the approach as a practical direction for training open vocabulary models for semantic segmentation.", "questions": "- What are the costs of performing online clustering at the dataset level during training?\n- What is the performance of MCLIP when trained on the benchmark datasets and tested in- and out-of-distribution? How does it compare against some baseline methods?\n- What is the model performance when changing the template at inference time? What is the performance without the template? What about training without the template? Does performance improve with prompt ensembling (i.e., similar to what CLIP does with N prompts that average to get better class centroids)?", "limitations": "- A potential limitation is the one I reported above, i.e., scalability issues due to the global online clustering procedure.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work is well-presented and curated, and the motivation is clear and sound.\n- The approach finetunes models pre-trained for a different purpose, and employs supervision from other pre-trained models, effectively reusing knowledge efficiently. Distillation from pre-trained models also removes the need for data supervision, potentially permitting scale training for open vocabulary semantic segmentation to billions of samples.\n- The method is tested against various baseline and on different benchmark datasets.\n- The model components are evaluated independently in the ablation study, helping to uncover the individual contribution to the final picture.", "weaknesses": "- While I understand the rationale behind performing clustering at pixel level on the entire dataset, I am not sure of the scalability of the approach. This probably explains why the authors only use 5% of the SA-1B dataset. It would be helpful to quantify the costs of performing the online clustering at training time.\n- Since the method could potentially suffer scalability issues, it would be interesting to understand performance when trained unsupervised in-distribution, reporting both in- and out-of-distribution performance, e.g., MCLIP trained on VOC and tested on Context.\n- The reasoning behind the selection process for the prompt (i.e., \"a photo of a {} in the scene\") is unclear. This is a minor issue, but the selection should probably be justified. While I would expect the model to learn to ignore the prefixes and suffixes, it would be interesting to understand how the model performs with other prompts.\n- Using CLIP as a baseline for the ablation studies may be misleading due to the mismatch between its application and open vocabulary semantic segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer kepm for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below: \n\n> **It would be helpful to quantify the costs of performing the online clustering at training time. What are the costs of performing online clustering at the dataset level during training?**\n> \n\n| Training dataset | COCO | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- |\n| 5% SA-1B | 21.4 | 16.7 | 34.9 | 23.8 | 83.1 |\n| 10% SA-1B | 21.6 | 16.9 | 36.1 | 23.9 | 83.9 |\n\n*Table C. Results on open-vocabulary semantic segmentation with additional training data.*\n\nWe compute the cluster assignment through the Sinkhorn-Knopp algorithm [13], which only adds around 1ms in our training every iteration, as it can be performed on GPU. We opted for a 5% split of SA-1B due to the large scale of the full dataset, which is similar in scale with SAM-CLIP [43]. To study the scalability of our approach, we provide results when doubling the amount of training data in Table C, showing that we do observe modest improvements from more training data. We will add this experiment to the revision.\n\n\n> **What is the performance of MCLIP when trained on the benchmark datasets and tested in- and out-of-distribution? How does it compare against some baseline methods?**\n> \n\n|  | COCO-Stuff | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- |\n| MaskCLIP[61] | 16.5 | 13.2 | 23.4 | 11.1 | 79.7 |\n| MaskCLIP+[61] | 18.0* | - | 31.1* | - | - |\n| MCLIP (COCO+DINO) | 20.0* | 15.5 | 30.1 | 18.5 | 79.4 |\n| MCLIP (COCO+SAM) | 21.7* | 16.6 | 34.2 | 23.1 | 82.4 |\n\n*Table D. Results on open-vocabulary semantic segmentation for in- and out-of-distribution analysis. \\*: indicates in-distribution results where the training splits of the datasets were seen.*\n\nWe thank the reviewer for providing an interesting point for discussion. To study in- and out-of-distribution, we provide “in-distribution” results by training MCLIP with COCO-Stuff images, and “out-of-distribution” results by zero-shot evaluating on other datasets. For comparison, we provide MaskCLIP as “out-of-distribution” and MaskCLIP+ suggested by reviewer Dx6v as “in-distribution” baselines, where MaskCLIP+ is trained with images from the target dataset COCO-Stuff and PC-59 respectively. We observe that our MCLIP outperforms MaskCLIP+ in COCO-Stuff, but the in-distribution performance of MaskCLIP+ outperforms MCLIP in PC-59 with DINO masks. However, when incorporating stronger SAM masks, MCLIP can improve largely, outperforming MaskCLIP+ despite being out-of-distribution for PC-59. We will add this experiment and discussion to the revision.\n\n> **What is the model performance when changing the template at inference time? What is the performance without the template? What about training without the template?**\n> \n\n| Training(↓)/Inference(→) | {} | itap of a {} | a photo of a {} | a photo of a {} in the scene |\n| --- | --- | --- | --- | --- |\n| {} | 35.4 | 38.7 | 38.0 | 36.6 |\n| itap of a {} | 35.4 | 39.2 | 38.1 | 36.3 |\n| a photo of a {} | 35.3 | 38.6 | 37.8 | 36.4 |\n| a photo of a {} in the scene | 35.1 | 38.6 | 37.8 | 36.0 |\n\n*Table E. Results on open-vocabulary semantic segmentation with different prompts. Each row show results from different prompts in training, while each column is in inference. For brevity, we report the mean over 5 benchmarks from Table D. “itap” is a common abbreviation of “I took a picture”*\n\nWe thank the reviewer for suggesting ablation with different prompts. We provide results when trained with no prompt(“{}”)  and with different prompts in Table E. We initially used “a photo of a {} in the scene” following other methods [11, 14, 21] but surprisingly, the prompt “itap of a {}” shows significant improvements for both training and inference. Given that “itap of a {}” is one of the well-performing prompts originally curated from CLIP, we speculate the results to reflect the preference of prompts from CLIP. We will add the results and the discussions, and re-conduct our experiments with better prompts.\n\n> **Does performance improve with prompt ensembling (i.e., similar to what CLIP does with N prompts that average to get better class centroids)?**\n> \n\n| Training | Inference | COCO-Stuff | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- | --- |\n| No ensemble | No ensemble | 21.4 | 16.7 | 34.9 | 23.8 | 83.1 |\n|  | Ensemble | 23.6 (+2.2) | 18.7 (+2) | 37.9(+3.0) | 27.2(+3.4) | 85.9(+2.8) |\n| Ensemble | No ensemble | 21.6(+0.2) | 17.1(+0.4) | 35.1(+0.2) | 24.9(+1.1) | 82.9(-0.2) |\n|  | Ensemble | 23.7(+2.3) | 19.2(+2.5) | 37.9(+3.0) | 28.1(+4.3) | 85.5(+2.4) |\n\n*Table F. Results on open-vocabulary semantic segmentation with prompt ensembling strategy.*\n\nWe provide results for ensembling in Table F, where we ensemble the default prompt “a photo of a {} in the scene” with 7 additional prompts originally curated in CLIP. We observe that not only can prompt ensembling largely boost the performance during inference time, it also shows additional gains when applied during training. We thank the reviewer for the valuable suggestion, and we will add the results and discussion to the paper.\n\n> **Using CLIP as a baseline for the ablation studies may be misleading due to the mismatch between its application and open vocabulary semantic segmentation**\n> \n\nWe clarify that the baseline results from CLIP in the ablation studies are in fact obtained from applying MaskCLIP [61], which slightly modifies CLIP for open-vocabulary semantic segmentation. We apologize for the confusion, and we will revise the baseline as MaskCLIP in the ablations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s9dNMRWPRX", "reviewer": "Reviewer_83hz", "summary": "This paper present MCLIP, which aims to finetune CLIP using image and mask data without semantic labels. The goal is to make its open-vocabulary recognition ability to be adapted to position-sensitive semantic segmentation tasks. MCLIP chooses to use SAM or feature clusters of DINO to obtain masks, which are class-agnostic but sometimes too small to have consistent semantics for finetuning. In addition, in order to achieve a trade-off between training stability and avoiding catastrophic forgetting, clustering uses the EMA version of finetuned image encoder.", "questions": "I think the authors need to further explain the novelty of this paper and give more analysis on the experimental results as listed in the weaknesses part.", "limitations": "Limitations are provided. It seems that there is no potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of this paper is interesting. Solving open-vocabulary semantic segmentation without semantic labels is an interesting topic.\n\n- The experimental results show that the preformance of the proposed approach is good.\n\n- Ablation studies and analysis are provided to help better understand of the role of each component and hyper-parameter.", "weaknesses": "- What role does the text encoder play in the clustering process?\n\nThe key to the success of clustering lies in the existence of unlabeled data in a certain grouping relationship in the space. The mask representations f_M can be considered to meet such conditions. If k learnable object prompts are initialized randomly, then k text representations f_C are randomly scattered in the text space initially, unable to provide the information needed for clustering. If k learnable object prompts are initialized by predefined categories, then this should violate the experimental setup without semantic labels. To illustrate this problem, the authors need to provide an explanation and add a comparative experiment: What results can be obtained just by clustering with f_M?\n\n- Why use clustering technology to solve the problem of too fine initial mask granularity?\n\nFirstly, previous work (GroupViT, CVPR'22 and SegCLIP, ICML'23) has used a grouping block designed based on the idea of clustering in the process of fine-tuning CLIP. In contrast, the clustering used in MCLIP provides better masks initially than the image patches used in previous work and the clustering process of MCLIP is parameter-free. It may be necessary to conduct ablation experiments to prove which differences bring positive results. But overall, technically, the use of clustering may be not novel.\n\n- In order to obtain class-agnostic masks, there is a class of experimental settings called open set semantic\nsegmentation (O3S, arXiv preprint arXiv:2307.02003, 2023) . Can the ”SAM initialization + clustering” scheme proposed by MCLIP obtain competitive performance in the O3S experimental setting?\n\n- Table 1 shows that using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP in the previous method improves the performance of the previous method’s zero-shot mask classification. Taking FC-CLIP as an example, MCLIP improves its zero-shot mask classification performance on ADE20K by 2.7. But compared to the final performance of FC-CLIP with two branches integrated, it is still 4.0 lower. So, in the case of keeping the complete inference strategy of the previous method, how much improvement can be achieved by using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP?\n\n- In the ablation experiment in Table 3a, does ’w/o Semantic Clustering’ mean fine-tuning with SAM-initialized masks directly without clustering? If so, why is the performance so poor? Without using clustering, it is equivalent to the number of cluster k is greater than the number of masks initialized by SAM on all images. However, the conclusion obtained from table 3b is that with the increase of k, the\nperformance tends to saturation, rather than decreasing to the level of ”w/o Semantic Clustering”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 83hz for the remarks on the paper and the considerate review. We address the comments and questions from the reviewer below:\n\n> **What role does the text encoder play in the clustering process?**\n> \n\n|  | COCO | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- |\n| without Text enc. | 17.9 | 18.5 | 29.9 | 28.9 | 53.5 |\n| with Text enc. | 21.2 | 20.2 | 34.2 | 33.2 | 66.0 |\n\n*Table G. Ablation on the usage of text encoder for clustering during training.*\n\nWe use the text encoder to match the inference strategy where classes are passed through the text encoder, as well as for leveraging its pre-trained knowledge to aid the semantic clustering process. The reviewer is correct in that the clustering process is possible without the text encoder by initializing $f_C$ as random vectors. We show such ablations in Table G where we verify that although the clustering is possible without the text encoder, it largely benefits with the text encoder. We thank the reviewer for pointing out an important factor for ablation, and we will add the results and discussions to the paper.\n\n> **Why use clustering technology to solve the problem of too fine initial mask granularity?**\n> \n\nWe would like to highlight that our clustering is done at a global level: we cluster masks across all images in the batch. GroupViT and SegCLIP in contrast cluster at a local level: they cluster regions within a single image using the caption for that image. Therefore, it is not obvious how we can use the GroupVit or SegCLIP grouping blocks in our framework. We will update the Related Work in our revision with this discussion. We emphasize that although the use of clustering is not novel, to the best of our knowledge,  our use of clustering across images for unsupervised representation learning is.\n\n> **Can the ”SAM initialization + clustering” scheme proposed by MCLIP obtain competitive performance in the O3S experimental setting?**\n> \n\n|  | Fold-0 | Fold-1 | Fold-2 | Fold-3 | mean |\n| --- | --- | --- | --- | --- | --- |\n| ZS3[3] | 18.8 | 20.1 | 24.8 | 20.5 | 21.1 |\n| LSeg[b] | 22.1 | 25.1 | 24.9 | 21.5 | 23.4 |\n| Fusioner[c] | 23.6 | 28.2 | 26.2 | 24.1 | 25.5 |\n| Yang et al.[d] | 26.5 | 30.8 | 26.3 | 24.1 | 26.9 |\n| MCLIP | 25.1 | 25.5 | 22.1 | 24.1 | 24.2 |\n\n*Table H. Results on COCO-20i under Z/FS setting from Yang et al.[d] Following LSeg[b], we used ‘others’ for the background class and prompt ensembling was used during inference.*\n\nTo validate the performance in O3S setting, we directly evaluate MCLIP trained on SA-1B on COCO-20i without training with any COCO images or annotations. Interestingly, we find MCLIP to show reasonable performance, surpassing LSeg despite not being trained on other folds of the COCO-20i dataset. We thank the reviewer for suggesting the evaluation, and we will add it to our revision.\n\n> **So, in the case of keeping the complete inference strategy of the previous method, how much improvement can be achieved by using MCLIP’s fine-tuned CLIP instead of the Frozen CLIP?**\n> \n\n|  | COCO-Panoptic | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- |\n| FC-CLIP | 63.7 | 34.1 | 58.4 | 56.2 | 95.4 |\n|  + MCLIP | 64.9 | 34.1 | 59.2 | 56.8 | 95.6 |\n\nTable I. Results from FC-CLIP with the complete inference strategy\n\nWe report the results with the full pipeline from FC-CLIP, when incorporating the “out-of-vocabulary” branch with MCLIP in Table I. We notice that due to the fusion strategy, the gains from the “out-vocabulary” branch are not completely transferred to the fused scores, but observe improvements over all datasets. We will add these results to the revision.\n\n> **Why is the performance so poor without Semantic Clustering? The conclusion obtained is that with the increase of k, the performance tends to saturation rather than decreasing to the level of ”w/o Semantic Clustering”.**\n> \n\n| Number of clusters, k | COCO | ADE-150 | PC-59 | Cityscapes | VOC |\n| --- | --- | --- | --- | --- | --- |\n| 64 | 21.1 | 20.2 | 34.2 | 33.2 | 66.0 |\n| 128 | 21.0 | 20.3 | 33.5 | 30.1 | 64.1 |\n| 256 | 21.3 | 20.4 | 33.6 | 30.0 | 64.1 |\n| 512 | 21.2 | 20.2 | 32.7 | 29.8 | 62.7 |\n\n*Table J. Additional results for increasing the number of clusters k. Performance starts decreasing after k=256*\n\nWe clarify that ‘w/o Semantic Clustering’ refers to supervising CLIP directly with fine-grained masks from SAM, i.e. f_C=f_M, which is to the reviewer’s understanding. This would enforce CLIP to distinguish small regions within the image, despite having near identical semantic meanings. We find this to conflict with the coarse, semantic understanding of CLIP when fine-tuning, eventually losing the pre-trained knowledge of CLIP, i.e. catastrophic forgetting, which is critical when fine-tuning foundation models[11, 43, 65].\n\nFurthermore, as the reviewer pointed out, ‘w/o Semantic Clustering’ would virtually be k=50M, (5% of SA-1B), and we agree that further increasing k should start to decline the performance, eventually reaching 50M. To verify this, we additionally provide results with k=512 in Table J which is the maximum number of k within our GPU memory constraints, and observe the performance decreasing for all datasets. We greatly thank the reviewer for providing the insight, and we will add the discussions to the revision.\n\nReferences:\n\n[b] Language-driven Semantic Segmentation. B. Li et al., ICLR 2022\n\n[c] Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models, C. Ma et al., BMVC 2022\n\n[d] Multi-Modal Prototypes for Open-World Semantic Segmentation, Y. Yang et al., IJCV 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the remarks on the paper, as well as their considerate reviews. Especially, we appreciate the thoughtful comments on the idea and motivation for leveraging unlabeled masks (**Dx6v, kepm, 83hz**), paper being well-written and curated (**Dx6v, kepm**), the proposed solution and technical contributions (**rHvB**), solid experiments and ablations (**Dx6v, kepm, 83hz**), as well as our approach effectively reusing knowledge from other pre-trained models (**kepm**).

Furthermore, we thank the reviewers for sharing insights and providing constructive feedback, which we have responded below. We hope our response adequately addresses the concerns, and we believe that the addition of the discussions will greatly improve the paper.

Meta-review: All reviewers found the article interesting, exploring the use of unlabeled masks for building an open vocabulary segmentation model. In general, reviewers found that the results and analyses support the proposed approach, consistently outperforming CLIP/MaskCLIP and methods using caption as supervision.

The AC went through the paper and the discussion and agreed with the reviewers' assessment. The article shows how we can learn a competitive open-vocabulary segmentation model without semantic labels is feasible, and this can be of interest to a large portion of the community, especially considering the new perspective under which the task has been addressed.

During the discussion, reviewers suggested several analyses and comparisons provided by the authors (e.g., MaskCLIP+, prompts, the impact of data/scalability, and centroids, among others). The authors should add these additional analyses and discussions in the final version of the manuscript and/or the appendix, as promised.
Decision: Accept (poster)