Learning Mask-aware CLIP Representations for Zero-Shot Segmentation
Authors: Siyu Jiao
Yunchao Wei
Yaowei Wang
Yao Zhao
Humphrey Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: Zero-Shot Segmentation
Open-Vocabulary Segmentation
Fine-tuning

Abstract: Recently, pre-trained vision-language models have been increasingly used to tackle the challenging zero-shot segmentation task. Typical solutions follow the paradigm of first generating mask proposals and then adopting CLIP to classify them. To maintain the CLIP's zero-shot transferability, previous practices favour to freeze CLIP during training. However, in the paper, we reveal that CLIP is insensitive to different mask proposals and tends to produce similar predictions for various mask proposals of the same image. This insensitivity results in numerous false positives when classifying mask proposals. This issue mainly relates to the fact that CLIP is trained with image-level supervision. To alleviate this issue, we propose a simple yet effective method, named Mask-aware Fine-tuning (MAFT). Specifically,  Image-Proposals CLIP Encoder (IP-CLIP Encoder) is proposed to handle arbitrary numbers of image and mask proposals simultaneously. Then, *mask-aware loss* and *self-distillation loss* are designed to fine-tune IP-CLIP Encoder, ensuring CLIP is responsive to different mask proposals while not sacrificing transferability. In this way, mask-aware representations can be easily learned to make the true positives stand out. Notably, our solution can seamlessly plug into most existing methods without introducing any new parameters during the fine-tuning process. We conduct extensive experiments on the popular zero-shot benchmarks. With MAFT, the performance of the state-of-the-art methods is promoted by a large margin: 50.4\% (+ 8.2\%) on COCO, 81.8\% (+ 3.2\%) on Pascal-VOC, and 8.7\% (+4.3\%) on ADE20K in terms of mIoU for unseen classes. Codes will be provided for reproducibility. Code is available at https://github.com/jiaosiyu1999/MAFT.git .

Learning Mask-aware CLIP Representations for Zero-Shot Segmentation
Siyu Jiao 1,2,3 J,,3 Yunchao We...
Yaowei Wang
Yao Zhao 1,2 Humphrey Shi
Institute of Information Science, Beijing Jiaotong University
Peng Cheng Laboratory
3 Beijing Key Laboratory of Advanced Information Science and Network Georgia Institute of Technology Picsart Al Research jiaosiyu99@bj edu.
Abstract
Recently, pre-trained vision-language models have been increasingly used tackle the challenging zero-shot segmentation task. Typical solutions follow the paradigm first generating mask proposals and then adopting CLIP classify them. To maintain the CLIP's zero-shot transferability, previous practices favour to freeze CLIP during training. However, the paper, we reveal that CLIP is insensitive to different mask proposals and tends to produce similar predictions for various mask proposals of the same image. This insensitivity results numerous false positives when classifying mask proposals. This issue mainly relates to the fact that CLIP is trained with image-level supervision. To alleviate this issue, we propose a simple yet effective method, named Mask-aware Fine-tuning (MAFT). Specifically, Image-Proposal CLIP Encoder (IP-CLIP Encoder) is proposed handle arbitrary numbers of image and mask proposals simultaneously. Then, mask-aware loss and self-distillation loss are designed to fine-tune IP-CLIP Encoder, ensuring CLIP is responsive to different mask proposals while not sacrificing transferabilityy this way. mask- mask- representations can be easily learned to make the true positives stand out. Notably, solution seamlessly plug into most existing methods without introducing any new parameters during the fine-tuning process. We conduct extensive experiments MAFT, the performance of the state-of-the-art methods is promoted by large margin: 50.4% (+ 8.2%) on COCO, 81.8% (+ 3.2%) on Pascal-VOC, and 8.7% (+4.3%) on ADE20K terms of mloU for unseen classes. Code is available at httpb//ooom/aaaiiyyy99//AA/M///
the popular zero-shot benchmarks. With
Introduction
Semantic segmentation, of the most widely researched topics in computer vision, has achieved remarkable success [3; 39; 15; 16] with the development of deep learning techniques [14]. However, traditional segmentation models are only capable of segmenting few predefined categories within a closed vocabulary [7; 2; 22; 21], which is much smaller than the number of categories used by humans describe the real world. Therefore, zero-shot segmentation [31; 1; 10; 13] is introduced to segment objects using arbitrary categories described by texts.
Recently, large-scale visual- language pre-training models (e.g. CLIP [28] and ALIGN [17]) have shown impressive transferability in recognizing novel categories, leading to their increased adoption
Work done during an internship at Picsart Al Research (PAIR).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Image
propusal of swaw
propusal of niver
Psi proposal of swean river
swaan: 99, 49%
Sowan: 97.011%
*wan: 54%
solid: 0.05%
Water:: 0.85%
water: 0,03%
person: 0.04%
bost: 0.17%
buat: 0.02%
pillow: 0.01%
river: 0.02%
0.01%
plastic:
0.01%
swan: 99,50%
Ariver: 285
A swan: 74 00%
person: 6.10%
water: 22.54%
river: 12.38%
orange:
8e8: 1.32%
sex: 10.50%
hums: 0.02%
rock: 0.69%
carrot: 0.02%
Stone: 0.24%
phasiic: 0,25%
predictions by frezen CLIP
predictions by mask-aware CLIP
Figure I: Comparison between the frozen CLIP and our mask-aware CLIP for proposal classification. Regions of proposals are highlighted with green. The frozen CLIP classifies p1, P2, and P3 swan class and produces similar predictions, although these proposals contain different regions of the image. After the MAFT, the mask-aware CLIP can produce proper scores for different proposals.
for tackling the challenging zero-shot segmentation task [6; 33; 20; 27J. A mainstream solution follows the "frozen CLIP" paradigm, which executes the zero-shot segmentation with two steps: 1) first employing a Proposal Generator to produce class-agnostic mask proposals and 2) then leveraging frozen pre-trained CLIP to classify each mask proposal via similarity matching in the aligned image-text feature space. While acceptable results are obtained, we reveal that these approaches overlook crucial issue, i.e. the frozen CLIP is insensitive to different mask proposals and tends to produce similar predictions for various proposals of the same image.
To better illustrate the above-mentioned issue, show several examples in Fig. 1. We use Mask- Former [5] to generate a series of mask proposals and select three typical ones. When using frozen CLIP for classification, we observe that correctly classifies the high-quality swan proposal pi- However, for the other two proposals and pa. which respectively contain only shape information of swan and both regions of swan and river, the frozen CLIP produces similar predictions compared to P1. This is reasonable since CLIP is trained by image-text pairs, making it insensitive to pixel- level information (e.g. background noise), and resulting numerous false positives. Based on the above observations, we consider that an expected CLIP for zero-shot segmentation task should 1) be sensitive different mask proposals, 2) not compromise its original transferability on novel classes.
To this end, we introduce Mask-aware CLIP Fine-tuning method (dubbed MAFT). To make CLIP sensitive to different mask proposals, we devise an Image-Proposals CLIP Encoder (IP-CLIP Encoder), which utilizes mask proposals perform masked Multihead Attention [5; 4]. This design enables the model to handle arbitrary numbers of images and proposals simultaneously. The mask- aware loss is proposed to minimise the distance between the IoU score of mask proposals and the classification score of IP-CLIP Encoder, prompting IP-CLIP Encoder to differentiate various proposals. Besides, to preserve CLIP's zero-shot transferability, we utilize frozen CLIP as a teacher network to facilitate fine-tuning. This is achieved by aligning the outputs of the frozen CLIP and IP- CLIP Encoder through self-distillation loss. By performing MAFT, several advantages are provided: 1) Fine-tuning is efficient since only a few mask proposals need to be classified. 2) Compared to pixel-level fine-tuning, mask-aware fine-tuning hardly alters the structure of CLIP itself, preserving its maximum transferability. 3) Mask-aware fine-tuning of CLIP is released from the segmentation module, making it plug-and-play and applicable to any "frozen CLIP" approaches. As shown in Fig. 1, the mask-aware CLIP can well distinguish different proposals and provide proper classification scores for both seen (river) and unseen (swan) classes.
We evaluate our MAFT on three commonly used zero-shot segmentation benchmarks: COCO- Stuff [2], Pascal-VOC [7], and ADE20K [40]. Extensive experiments show that MAFT works well with various zero-shot segmentation methods. In particular, by plugging MAFT, the of the-ar art approach FreeSeg [27] achieves superior performance on COCO- (42.2% + 50.4%), Pascal-VOC (78.6% + 81.8%) 8%) and ADE20K (4.4% + 8.7%) in terms of mIoU of unseen classes open-vocabulary setting, where MAFT enhances the performance of A-847 [40], A-150 [40], PC-459 [24], PC-59 [24] and PAS-20 [7] datasets by +3.0% +1 +1.2%, +6.4% + +19.1 % and +4.4%, respectively. Notably, our approach outperforms the freezing CLIP counterpart and establishes new state-of-the-ee the results on all datasets.
Furthermore, we conduct experiments

---

2 Related Work
Zero-Shot Segmentation [29] is established to break the restriction of categories and perform segmentation on unseen classes. Earlier works SPNet [31] learn a joint pixel and vocabulary concept embedding space, ZS5 [1] utilizes generative model generate pixel-level features based on word embeddings of unseen classes, CaGNet [10] incorporates context information for better feature generation. Recent approaches take the advent of large-scale risual-language models (e.g. CLIP [28] and ALIGN [17]) to leverage rich alignment features from image-text pairs. [34] uses CLIP to generate pseudo-labels for single-image segmentation. STRICT [25] obtains pixel-level pseudo-labels from CLIP for unlabeled pixels and proposes a self-training strategy to capture latent information on unseen classes. L.Seg [8] trains CNN model compute per-pixel image embeddings and use CLIP text embeddings as a classifier. [32] employs contrastive supervision to learn segmentation masks
from text.
Concurrently, recent works [6; 33; 27; 20; 9] follow the "frozen CLIP" paradigm for zero-shot segmentation, they first generate series of mask proposals and then utilize CLIP [28] ALIGN [17] to classify them. ZSSeg and OVSeg [33; 20] train CLIP adapters to boost performance. FreeSeg[27] simultaneously uses semantic, instance, and panoptic labels and performs fusion training. OpenSeg[9]
takes extra images with image-level supervision (e.g. captions) to scale up training data.
Pre-trained model fine-tuning is widely used for transferring pre-trained knowledge to downstream tasks, e.g. segmentation. However, this strategy may not work well for data-limited tasks like few-shot learning and zero-shot learning due to the daunting overfitting problem. To address this problem and transfer pre-trained knowledge to data-limited tasks, [43; 42; 12; 33; 20; 27] propose to learn text prompts or image prompts by using (a few) annotated images from target dataset. SVF [30] fine-tunes only a few parameters in the pre-trained image encoder to adapt pre-trained knowledge to few-shot segmentation. [38; 37] use contrastive learning to avoid catastrophic forgetting. Alternatively, many outstanding approaches in data-limited tasks [23; 35; 36; 6; 33] choose to freeze the parameters of pre-trained models to maintain the transferability.
Specific to the task of zero-shot/ open-vocabulary segmentation, mainstream approaches use frozen CLIP to avoid overfitting. Recently, MaskCIII [41] conducts adequate experiments to fine-tune CLIP for open-vocabulary segmentation but has failed. While this attempt is meaningful and appreciated. is believed that the failure is due to the large domain gap between pixel-level and image-level tasks. This motivates us further research fine-tuning CLIP to be mask-aware (region-level task).
3 Preliminary
Problem Setting. Zero-shot segmentation aims at training a segmentation model capable of segment- ing novel objects using text descriptions. Given two category sets Cseen and Cunscen respectively, where Creen and Gunseen are disjoint terms of object categories (Cseen O Cunseen = 0). The model is trained on Cseen and directly tested on both Cseen and Canseen- Typically, C Cseen and Cunscen are described with semantic words (e.g. sheep, grass).
Revisiting the "frozen CLIP" paradigm. The "frozen CLIP" approaches [6; 33; 27; 20] execute zero-shot segmentation in two steps: mask proposals generation and mask proposals classification. In the first step. these approaches train a Proposal Generator to generate N class-agnostic mask proposals (denoting as M, M E R xxx, ) and their corresponding classification scores (denoting as AP, AP E N...all))))). MaskFormer [5] and Fask2Former [4] are generally used the Proposal Generator since the Hungarian matching [19] in the training process makes the mask proposals strongly generalizable. In the second step, N suitable sub-images (Luut) are obtained by merging N mask proposals and the input image. Luub is then fed into the CLIP Image Encoder to obtain the image embedding (E'). Meanwhile, text embedding (E") is generated by a CLIP Text Encoder. The classification score (AA A
R NxC) predicted by CLIP is calculated as:
Aç Softmax(
), [1. 21,
(1)
Dizo exp(: ? se (E"',EE)
-T-E" EE where r the temperature hyper-parameter se. FF/", FT = |
represents the cosine similarity
between E. and F.!. C is the number of classes, with [Ceeen U Cunseen| during inference. Noting that CLIP is frozen when training to avoid overfitting.
[Ceen during training and C
3

---

CLIP (teacher net)
Self-distiltaion Loss
Mark Musk-aware Lows
IoU sco'e S
IP CLIP Encader
E layers
Lyyes
Tranforee layer
Truit Train forneer layer
Lrun oomtir layer
Tranf layer
Emn Prod
A
Conv. & Daiten
Ernd
CLIP Text Encoder
Eisd
Train only Train & Test
attention bias B N (w)
Proposal Generator
Tunod
Frozen
score Soot *
Frusk pruposele M
N K.n.
Figure 2: Overview of the Mask-Aware Fine-tuning (MAFT). In IP-CLIP Encoder, modify the CLIP Image Encoder, and apply the mask proposals as attention bias in Multihead Attention from the T,th layer. The final projection unit is an MLP module used for reshaping the channels of Fats w.o. M denotes IP-CLIP Encoder processes image without utilizing mask proposals (M). Mask-aware Loss is designed train CLIP to be mask-aware, while Self-distillation Loss is designed to maintain the transferability. Only the IP-CLIP Encoder is trained (orange part), the Proposal Generator and the CLIP Text Encoder are frozen (blue part).
To further enhance the reliability of A the classification score of the Proposal Generator (AP) is ensembled with A since AP is more reliable on seen classes. This ensemble operation is wildly used "frozen CLIP" approaches. The pipeline of "frozen CL.IP", as well as the merge and ensemble operations, are described in detail in the Appendix.
Although "frozen CLIP" approaches have achieved promising results, it clear that directly adopting an image-level pre-trained CLIP for proposal classification can be suboptimal. A frozen CLIP usually produces numerous false positives, and the merge operation may destroy the context information of an input image. In view of this, we rethink the paradigm of the frozen CLIP and explore new solution for proposal classification.
4 Methodology
We introduce Mask-Aware Fine-tuning (MAFT), a method for learning mask-aware CLIP represen- tations. Within MAFT, we first propose the handle images with any number of mask proposals simultaneously (Sec. 4.1). Then, mask-aware loss and self-distillation loss are introduced to fine-tune the IP-CLIP Encoder and make distinguishable for different mask proposals while maintaining transferability (Sec. 4.2). The complete diagram of the MAFT is shown in Fig. 2, we use the ViT-B/16 CLIP model for illustration.
CLIP Encoder (IP-CLIP Encoder) to
4.1 Image-Proposal CLIP Encoder (IP-CLIP Encoder)
IP-CLIP Encoder aims to process arbitrary numbers of images and mask proposals simultaneously. We draw inspiration from MaskFormer [4; 5], which uses attention-masks in Multihead Attention and provides the fiexibility for accepting any number of queries and features of different masked regions. Accordingly, we apply mask proposals as attention iitention-maaks in Multihead Attention and designate independent classification queries for each mask proposal.
In the IP-CLIP Encoder shown in Fig.
denote the features propagate between Transformer layers FS where = [1, 22.. We can express F as E [Pa.i Fooet) [R(+hw)xd here 1 represents a class-embedding vector (Fi)), hw represents the number of the flattened image features (Fr ..tt To obtain the classifications of all mask proposals simultaneously, N times, where N is the number of mask proposals, denoting the repeated class eaasdddnb vectors ds- We can express the modified features (F**) as Fi [F F Feate (NN +hw)
repeat FX at layer L

---

Propagation of F", where = [I.. We consider that CLIP's classification sigmificantly relies on context information. In the first L Transformer layers, the propagation of Fr the same in standard CLIP. Specifically, Fds, utilizes cross-attention with all pixels within F featt effectively retaining the context information.
In the subsequent 12 L Transformer layers, the propagation of f can be partitioned into two parts: the propagation of F and the propagation of Fi feaat
Propagation of Fals. We use [n] and [nn] to represent the position n in File and M, where n = [1, 2...N. is expected Fri,n) [n] computes Multihead Attention for the positions where M[n]= 1 and itself. To achieve this, we construct an attention bias E (NNx(thh)) as follows:
0, if M(.)) =1 Bun -00, if (a.)) =
M [[(N,N;; (Mat
here I(N, N) denotes Nth order identity matrix, F.at(.) denotes the flatten operation. M is an intermediate variable for better representation. Therefore, a masked Multihead Attention is used for propagating Fir
((+1) Softmaxx Que Fig, Key + B) Val(F
Vd
where Que(-), Key(-), and Val(.) denote linear projections, the hidden dimension of F Notably, We omit the MLP Layer and Layer Normalizations in Transformer layers simplify the representation Eq. 3 and Eq.
Propagation of fear A standard Multihead Attention is used for propagating F)
Que(F) Key(F) (ear))
Fitl Softmax(
fear
Val(Fjjaa)
Vd
Therefore, for any given mask proposal M[n], the corresponding class-embedding [ng[n] only performs Multihead Attention with Feet where M[n]= and Fig. [n]. The propagation of Fj feat remains undisturbed by attention-masks Compared with the frozen CLIP, IP-CLIP Encoder leverages context information effectively and reduces computational costs.
4.2 Objective
IP-CLIP Encoder with CLIP pre-trained parameters remains challenging in distinguishing differennt mask proposals, e-g., when the proposals contain more background regions than foreground objects, IP-CLIP may tend to classify them into the foreground categories. To overcome this limitation, introduce mask-aware loss and self-distillation loss to fine-tune the IP-CLIP Encoder to be mask-aware without sacrificing transferability.
We conduct the mask-aware loss function (Lma) on AS. The goal is to assign high scores to high- quality proposals and low scores to low-quality proposals in A Concretely, we use Intersection over Union (IoU) score obtained from ground-truth and align it with the A to prompt CLIP to become mask-aware. Assuming there are k classes in ground-trutte we can generate & binary maps of ground-truth and calculate the IOU score (Stor) with N mask proposals. We identify discrepancy between the maximum values A and Stov. The maximum value of A tends to approach whereas the maximum value of StoU ranges from 0.75 0.99. This inconsistency can hinder the alignment between these two metrics. Therefore, we introduced a min-max normalization technique for Stot as follows:
SIOU min(S max mar((S1ar) min(Snntt STOU E KNN
LoU
(5)
Meanwhile, we select & pre-existing classes in A (A A SmoothL1 Loss to align with Sporm Therefore, Lma can be formulated as follows:
I KKNN, and employ
Lma (ASetec: STor"") SmoothLl STor""
(6)
5

---

Table 1: Comparison with state-of- of-the-art methods in zero-shot segmentation. mloU and m denote the ml) of seen classes and unseen classes.
COCO-Stuff
Pascal-VOO VOC
ADEZUK
Method
mloU
mloU"
hloU
mloU
mloU*
hloU
mloU"
mlo
hloU
SP
34.6
26.9
30.3
77.8
25.8
38.8
2S5[1]
34,9
10.6
16.2
78.0
21.2
33.3
10)
35.6
13.4
19.5
78.6
30.3
43.7
STRICT]25]
35.3
30.3
32.6
82.7
35.6
73.3
FogFore
36.7
36.2
36.4
90.1
70.6
79.2
17.4
5.1
7,9
ZogForeer MAFT
36.4
40.1
38.1
91.5
80.7 +.001
85.7
16,6
7.0
9.8
ZS [eg[33]
40.4
36.5
"3.3
86.6
59.7
69.4
18.0
4,5
7.2
ZSSeg +MAFT
40.6
40.1 13.6
40.3 12.0
88.4
66.2 6.5.5
75.7 16.8
18.9 10.9
6.7 12.2
9.9
42.4
42.2
42.3
91.9
78.6
84.7
22.3
4,4
7.3
FreeSeg *MAFT
43.3 +0.8
50.4
+5.2
46.5
91.4 -0.5
81.8
4-3.2
86.3
+116
21.4
-0.8
8.7
+1.
12.4
-5.1
Table 2: Results on representative methods [6; 33; 27] with/without MAFT. Here we remove the ensemble operation, and only maintain CLIP classifier results.
COCO-Stuff
Pascal-VOC mloUr
ADE20K
Method
mloU"
mloU
hloU
mioU!
mioU"
mioU
hToU
ZegFormed
18.5
23.0
20.5
81.4
768
79.0
5.1
2.6
ZegFormer +MAFT
35.1 138.
31.6
126
33,3 .12.7
87.6
15.
79.9 1s.
14..
15.8
10.5
7.0
14.4
10.8
33
20.6
27.4"
23.6
82.0
71.2"
7622"
599
28
7SSeg +MAFT
36.1 +15.5
35.9 +5.3
87.1 +5.3
76.1 +..
81.2 45.0
17.2-113
+12
10.2 +6.3.
FreeSeg(22]
223
29.3
25.3
87.4
74.7
80.5
63
28
FreeSeg +MAFT
40.1 +37.
49.7 +20.4
419.1
90.4 +3.0
84.7 +30.0
47.
-16.
45.9
12.2 45.3
SmoothLl(z.g)- [0.5 (x y) ix 1 yl 0.5, otherwise
In addition to Lma. we also introduce a self-distillation loss Lasa to maintain CLIP's transferability and alleviate overfitting Cseen- Within Ldis. we use frozen CLIP the leacher net, the IP-CLIP as the student net for self-distillation. The predictions of the frozen CLIP and IP-CLIP are expected to be the same when mask is included. Denoting the output of the frozen CLIP Ar, and the output of the fine-tuned IP-CLIP without masks as Ag. We use SmoothL1 Loss to minimize the difference follows:
Cais Ar) SmoothLI(As,,
(8)
is important to note that when processing an image through IP-CLIP without mask proposals, the resulting As a matrix with dimensions cx1 Therefore, the final loss function can be formulated as: L = Lm + ACdis. where we set the constant À to in our experiments. The mask-aware fine-tuning process is efficient we only perform few iterations (less than I epoch).
Experiments
5.1 Setting
Dataset. We first follow [1; 11; 26; 6; 33] to conduct experiments on three popular zero-shot segmentation benchmarks, Pascal- COCO-Stuff and ADE20K, to evaluate our method. Then, we evaluate MAFT on the open-vocabulary setting [20; 33), i.e,, training on COCO-Stuff and testing on ADE20K (A-847, A-150), Pascal-Conters (PC-459, PC-59), and Pascal-VOC (PAS-20). More details of the dataset settings are provided in the Appendix.
Evaluation Metrics. To quantitatively evaluate the performance, we follow standard practice [1; 31; 10; 25; 6; 33; 27], adopt mean Intersection over Union (mloU) to respectively evaluate the performance for classes (IoU*) and unseen classes (IoU"). We also employ the harmonic mean IoU (hloU) among the seen and unseen classes to measure comprehensive performance.
Methods. Three representative methods are used to verify the generality of MAFT. We unify the three methods into the same framework, with methods using ResNet101 as the backbone of Proposal Generator and ViT-B/16 CLIP model for fair comparison.

---

Table Comparison with ofte--o-tthe-are methods on the open-vocabulary setting mloU is used to evaluate the performance. denotes additional training data is used.
A-847
A-150
PC-459
PC-59
PAS-20
[P
24.3
18.3
7SSeg[
19.4
38.3
LSeg+[8]
2.5
13.0
5.2
36.0
59.0
OVSeg[20]
7.1
24.8
11.0
53.3
92.6
OpenSeg* [9]
8.8
28.6
12.2
48.2
72,2
FreeSeg[2] 27]
7.1
17.9
6.4
34.4
85.6
FreeSeg +MAFT
10.1
43.0
29.1
431.2
12.8 46..4
53.5
+-1.11
90.0 444.4
ZegFormer (CVPR 2022) [6] is an early adopter of the "frozen CLIP" paradigm. It uses Farker as Proposal Generator and employs an ensemble operation to improve confidence of the results.
ZSSeg (ECCV 2022) [33] uses MaskFormer Proposal Generator and introduces learnable prompts to improve classification accuracy, which significantly affects the subsequent methods. ZSSeg also adopts self- self-training strategy, this strategy is excluded from all methods for a fair comparison.
FreeSeg (CVPR 2023) [27] represents the state-of-the-art method, unifies semantic, instance, and panoptic segmentation tasks and uses annotations from all three tasks for fusion training. We retrain FreeSeg with only the semantic annotations to ensure faimess.
Implementation details. We employ ResNet101 as backbone of the Proposal Generator and ViT- B/16 CLIP model. The training process consists of stages. For the first stage, we follow the official code of FegForeer, ZSSeg and FreeSeg for model training. For the second stage, we fine-tune IP-CLIP Encoder with MAFT. We take the batch size of 16 and set CLIP input image size 4800 The optimizer is Adam W with a learning rate of 0.00001 and weight decay of 0,00001. The number of training iterations set to 100 for Pascal-VOC, 1000 for COCO-Stuff and 5000 for ADE20K.
5.2 Comparisons with State-of-the-art Methods
In this section, three representative methods are used [6; 33; 27] evaluate the effectiveness of MAFT. We compare three representative methods with MAFT and frozen CLIP. Additionally, we compare the results with previous state-of-the-an methods [31; 10; 25].
Comparisons in the zero-shot setting, In Tab. 1, MAFT remarkably improves the performance. MAFT promotes the state-of-the-art performance by 8.2% COCO, 3.2% on Pascal, and +4.3% on ADE20K in terms of mloU for unseen classes. It is important to note that the results for seen classes are mainly based A rather than A due the ensemble operation in [6; 33; 27] (Details in Sec. 3). Therefore, the effect of MAFT on the seen classes is relatively insignificant.
Comparisons without ensemble strategy. To better showcase the performance gains from MAFT, we removed the ensemble operation [6; 33; 27] and presented the results in Tab. 2. It can be seen that the performance of different methods is significantly improved after applying MAFT. In particular, the ofate-of-the-ar method FreeSeg achieves hloU improvements of 19.1%, 7.0%, and 8.3% COCO, VOC2012 and ADE20K datasets.
Comparisons in the open-vocabulary setting. We further evaluated the transferability of MAFT in the open-vocabulary setting [20; 33], using FreeSeg baseline for comparison. Results are shown Tab. 3. Compared with OVSeg [20] and OpenSeg [9], FreeSeg achieves suboptimal performance. However, the proposed MAFT enhances the performance of A-847, A-150, PC-459, PC-59 and PAS-20 by 3.0%, 1.2%, 6.4%, 19.1% 9.1% and 4.4%, and outperforms OpenSeg on all five datasets.
5.3 Ablation Study
We conduct ablation studies on various choices of designs of our MAFT to show their contribution to the final results in Tab. 4. FreeSeg is used as the baseline model and ensemble operation is removed.
Component-wis ablations. To understand the effect of each component the MAFT, including the IL-C Encoder and the fine-tuning strategy (Cmas Late), start with standard FreeSeg and
7

---

Table Ablations on COCO dataset. GFLOPs (a) is used to measure the computation of CLIP Image Encoder. The best results are highlighted with red, and the default settings are highlighted with gray background.
(a) Ablation on components of MAFT. fi denotes the mask aware fine tining
(b) Ablation on mask-aware loss Lms- Lais removed.
mloU"
mloU
hloU
GFLOPs
mloU
mlol."
hlloU
FreeSeg
29.3
1127.0
Li
38.6
45.8 +16.5
41.8 +10.5
IP-CLIP
29.4 47.1
36.2 +6.
32.4 +7.1
53.4
La
+17.7
45.8 +18.
42.7 +17a
[[ (Cma)
39.9 +17.6
47.1 +17.
43.1 +17a
53.4
Smooth KL
39,9 +17e
47.1 +178
43.1 +17a
ft (Com Cais)
40.1-10
49.7+20.4
19.0
53.4
40.9
41.8 +12.
41.3 +10.0
(c) Ablation of the training iterations
(d) Ablation the frozen unils in CLIP
(0) Ablation of the start mask attention layer L
mioU"
mloU
mioU"
mloU
hloU
mloU
mloU
hloU
500 iters
37.7
47.0
None
40.6
44.7
42.5
39.3
46.4
42.6
iters
40.0
+
40.7
44.7
39.2
41.1
+ pos.
44.9
39.5
iters
41.4
+ mmp
48.7
iters
41.5
come..
49.7
40.0
44.3
5k iters
42.0
45.7
pro).
40.2
49.1
44.2
10
39.9
45.7
42.6
progressively add each design. (Tab. 4a). FreeSeg uses frozen CLIP and yields inferior performance due to CLIP's mask-unaware property (1*t row). Then, IP-CLIP Encoder obtains rich context information and greatly reduces the omputational costs, resulting in improvement of 7.1 % on seen classes and 6.9% on unseen classes. However, mask-aware is not accomplished at this point. Using only Lma for fine-tuning CLIP produces decent performance (the 3rd result). The introduction of Ldis (the 4th result) maintains transferability while learning mask-award representations, which further enhances the performance on unseen classes by 2.6%.
Effect of different Lma- Mask-aware Loss Lm an essential component of MAFT, In Tab. 4b, we investigate how different loss functions (I11, 1.2, SmoothL1 and K Loss) impact performance, here we remove Liie for analysis. Results show SmoothT1 Loss boosts performance on Cunseen to 47.1% ++17.8), K. Loss provides +12.5% improvement on Ceeen. but only +11.8% on Cunseen manifesting LL Loss compromises the model of transferability comparing with Smoothh1 Loss.
Training iterations. Tab. 4c examines the impact of training iterations. Increasing the number of iterations leads gradual improvement of IoU but it also results in significant overfitting on unseen classes. Therefore, we choose to fine-tune 1k iterations to maximize the zero-shot ability.
Frozen units in CLIP. We also explore the impact of fine-tuning units within IP-CLIP Encoder. As illustrated in Fig. 2, IP-CLIP Encoder comprises convolution layers (dubbed as conv.), class embedding (cls.), Transformer layers, final projection (proj.) and positional embedding (pos., shown in Fig. 2). We start with fine-tuning the entire IP-CLIP Encoder, and then freezing each unit sequentially, as specified in Tab, 4d. We only freeze M LP in the Transformer layers (dubbed as mll)). Compared with fine-tuning the entire IP-CLIP Encoder, the performance of mloU" is improved by 5.0% when freezing conv., cls., pos. and mlp.
Start mask attention layer. Tab, 4e presents the results of the start mask attention layer (L). We observe significant improvement in the performance of unseen classes by +3.4% when the value of increases from 0 to 8. This could be attributed to the fact that starting masked Multihead Attention later enables Fir to gain more context information. However, the performance significantly drops when I, 10 (from 49.7% to 45.7%), which may be due to the loss of mask-aware property.
5.4 Extending MAFT with SAM
We explore using the Segment Anything Model [18] (SAM) as the proposal generator. We evaluate the performance with SAM-H using original CLIP (dubbed SAM) or mask-aware fine-tuned CLIP (dubbed SAM + MAFT). In fact, SAM can be seamlessly integrated into our framework as the proposal generator. The results are shown in Tab. 5. Experiments conducted under both zero-shot setting and open-vocabulary setting.
It can be observed that SAM + MAFT obtains significant improvement over SAM under both settings. Besides, SAM + MAFT also surpasses FreeSeg + MAFT on all benchmarks. Particularly, in the zero-shot setting Pascal-VOOC) SAM + MAFT outperforms FreeSeg + MAFT by 6.8% in
8

---

Table 5: Comparison with SAM. We use SAM-H as the proposal generator.
(a) Results in zero-shot segmentation.
Pascul- -OCC
COCO-Stuff
mioU
mloU"
hloU
mIoU
mloU*
mloU
hloU
mloU
SAM
85.1
86.7
85.9
85,5
43.1
43.3
43.2
42.1
SAM MAFT
91.0
46.9
88.6
89.8
90.4
43.4 40.3
51.5
47.1
44.1 +2.0
(b) Results open vocabulayy segmentation.
A-847
A-150
PC-459
PC-59
PAS-20
SAM SAM + MAFT
9.0
21.3
7.8
33.7
87.5
12.7 +3.7
33.0 -11.7
16.2 48.4
59.0 425.3
92.7 4.5.2
terms of mloU". This enhancement can be attributed to the stronger generalization capabilities of SAM for unseen classes.
5.5
Extending MAFT with more Vision-Language Models
Table 6: Comparison with more SsiionLLanguage Models.
backbone
A-847
A-150
PC-459
PC-59
PAS-20
OVSeg [20]
9.0
29,6
12.4
55.7
94.5
FreeSeg [27]
ViT-L
8.5
21.0
7.6
33.8
86.4
FreeSeg MAFT
12.1 +3.6
32.0
-11.0
15.7
+8.1
58.5 +24.7
92.1 +5.7
FreeSeg [27]
5,3
15.5
5.4
28.2
87.1
Res50
FreeSeg MAFT
8.4 +3.1
27.0
-11.5
9.9
+4.5
50.8
+22.6
89.0 +1.9
In order to demonstrate the efficacy and robustness of MAFT, we conduct experiments using stronger (CLIP-ViT-L) and ResNet-based (CLIP-Res50) Vision-Languag Models. The open-vocabulary results are shown in Tab. 6, we also include the results of OVSeg with CLIP-ViT-L for comparison.
CLIP-VIT-L. According to Tab. 6, FreeSeg with a standard CLIP- L model (dubbed FreeSeg) still can not achieve satisfactory results. However, by integrating our MAFT (dubbed FreeSeg + MAFT), the segmentation results are remarkably enhanced, thus establishing new state-of-the-art benchmarks.
CLIP-Res50. Our MAFT easily adapted into ResNet-based models. Specifically, we modified the AttentionPool24 unit within CLIP-R50 Image Encoder. The mask proposals are introduced as attention bias (B) in Multihead Attention, with Fels being repeated N times. Notably in CLIP-R50, Fas is obtained via vooobllAvrrggel Pooling performing on Fjeat- The results are presented in Tab. 6. The performance on all datasets is improved by large margin. FreeSeg + MAFT with CLIP-R50 achieves competitive results with some CLIP- ELIP-VIT-B- ViT-B-basse B-based methods according to Tab. 3.
5.6 Qualitative Study
Visualizations of typical proposals. Fig. 3 shows frozen CLIP and mask-aware CLIP classifications of typical proposals, including high-quality proposals of foreground (p1, P4). high-quality proposals of background (ps, Ps). a proposal with background noise (p2), and a proposal containing part of the foreground (pş). The proposal regions highlighted in green yellow.
Several observations can be obtained: (1) The frozen CLIP provides good predictions for pi and p4- (2) The frozen CLIP assigns p2 as cal and p5 horse, with scores even higher than Pı, p4, indicating the frozen CL.IP cannot distinguish proposals containing information on the same objects. (3) The frozen CLIP fails to give correct predictions for p3 and P6, which may be due the lack of context information. (4) Our mask-aware CLIP gives good predictions for high-quality proposals (p1. P3. P4. Pc) and provides suitable predictions for p2 and ps-
Fig. 4. Some segmentation results of FreeSeg contain background noise (e.g. the 1st & 2nd row, 3rd column) or contain only part of the objects (3nd row, 3rd column). In ADE20K-847 dataset, too many classes may lead to the anticipated results (last row, 3rd column) with the frozen CLIP. Using mask-aware CLIP learn mask-aware representations can significantly improve these segmentation results, as evident from the last column.
Qualitative analysis. We show some visual examples
More visual samples are shown in the Appendix.

---

P1
P2
P3
cat 50 7356 musse: 8. 74% ne: L.12%
cal: 99.5%
cat 51, 3..4%%
chair 40.15%
chai:: 22 62%
souch: 70.22%
genam: 45%%
chair 5999
umads: ADGG cor 13 13,29%
cat: 14,05%
cteir: 22 37%
predictions by Trouen CUP
prndictions by miskaaare CUP
mouse: 2.00%
mouse: 11.10%
couch: 7,39%
fumnure: 5.9%%
solid: 10%6
cog 1.00%
conch: 5.80%
Samitone: 12 79
curtain: 8o%
bench 10%
pollow:
dage:
person:
pillow 4.31%
pesson 0.15%
1
P
lunc:
hueae:
stato
cow
sround:
gpoundl
0.42% fiddl 0.42%
teville 0.79%
4.87%
0.1%
chothex 0.231%
metal: 0.60%
smaadtrn 05%
hads: 12%
Figure 3: Visualizations of typical proposals & top 5 A by frozen CLIP and mask-aware CLIP
VOC2012
bird
pottedplant
COCO-Stuff
cardboard
Furniture
ADE20K 150
building
tree
sky
lkyseraper
ADE20K-844 -877
building
anterns
sky
mount
tramcar
kkyerrppt
brick
Image
GT
FreeSeg
FreeSeg MAFT
Figure 4: Qualitative results. The models trained with COCO-Sture and directly tested VOC2012, COCO, and ADE20K.
6 Conclusion
In this paper, we rethink the "frozen CLIP" paradigm in zero-shot segmentation and propose Mask- Aware Fine-Tune (MAFT) for fine-tuning CLIP. Firstly, IP-CLIP Encoder is proposed handle images with any number of mask proposals. Then, Lma and Lais are designed for fine-tuning CLIP be mask-aware without sacrificing its transferability. MAFT plug-and-pla, and can applied to any "frozen CLIP" approach. Extensive experiments well demonstrate the performance of various zero-shot segmentation methods is improved by plugging MAFT.
Limitations. Our MAFT introduces CLIP fine-tining framework to the research of zero-shot segmentation. However, the classification ability for novel classes is still limited by pre-trained lasion-language models. How to further narrow this limitation is our future research focus.
Acknowledgment. This work was supported in part by the National Key R & D Program of China (No.20217ZD0112100)
10

---

References
[1] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
[2] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition, pages 1209-1218, 2018.
[3] Liang- Shan-CChihh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence. (((4)::34-848, 2017.
[4] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing. Mask2former for video instance segmentation. arXiv preprint 12..:::2122....... 2021.
[5] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances Neural Information Processing Systems, 34, 2021.
[6] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11982,1592,,, 2022.
[7] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-136, 2015.
[8] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv: /2. /2143, 2021.
[9] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. In Computer ECCinn-ECVVA 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 540-557. Springer, 2022.
[10] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia,, pages 1921-1929, 2020.
[11] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1921-1929, 2020.
[12] Zixian Guo, Bowen Dong, Zhilong Ji, Jinfeng Bai. Yiwen Guo, and Wangmeng Zuo. Texts as images in prompt tuning for multi-label image recognition. arXiv preprint arXiv:2211.12739, 2022.
[13] Kunyang Han. Yong Liu, Jun Hao Liew, Henghui Ding, Yunchao Wei, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, et al. Global knowledge calibration for fast vocabuaayy segmentation. In Proceedings the IEEE/CVF International Conference Computer Vision, 2023.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[15] Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 603-612, 2019.
[16] Zilong Huang, Yunchao Wei, Xinggang Wang, Wenyu Liu, Thomas S Huang, and Humphrey Shi. Alignseg: Feature-aligned segmentation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):550-5577 2021.

---

[17] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen. Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and ision-languedn representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 49164 PMIR, 2021.
[18] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0264. 2023.
[19] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97 1955.
[20] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- ppen-vocabulary semantic segmentation with mask-adapted 2022.
clip. arXiv preprint arXiv:2210.0411
[21] Jiaxu Miao, Xiaohan Wang. Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21033-21043, 2022.
[22] Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guangrui Li, and Yi Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4133-4143, 2021.
[23] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmen- tation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1941-6952, 2021.
[24] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Cry Cho, Seong-Whai Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 891-898, 2014.
[25] Giuseppe Pastore, Fabio Cermelli, Yonggin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. A closer look at self-training for zero-label semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2693-2702, 2021.
[26] Giuseppe Pastore, Fabio Cermelli, Yonggin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. A closer look at self-training for zero-label semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2693-2702, 2021.
[27] Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open- open-vocabulary image segmentation. arXiv preprint 2303. 2023.
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 1748-87655 PMLR, 2021.
[29] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.034// 2017.
[30] Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng. Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning. arXiv preprint arXiv:2206.06122. 2022.
[31] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256-828 02668226,200 2019.
12

---

[32] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. CVPR, 2022.
[33] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Computer ECion-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 736-753. Springer, 2022.
[34] Nir Zabari and Yedid Hoshen. Semantic segmentation in-the-wild without seeing any segmenta- tion examples. arXiv preprint arXiv:21 2021.
[35] Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle- consistent transformer. Advances in Neural Information Processing Systems, 34:21984-2199,,
2021.
[36] Gengwei Zhang, Shant Navasardyan, Ling Chen, Yao Zhao, Yunchao Wei, Honghui Shi, al. Mask matching transformer for few-shot segmentation. Advances Neural Information Processing Systems, 35:823-836, 2022.
[37] Zekang Zhang, Guangyu Gao, Zhiyuan Fang, Jianbo Jiao, and Yunchao Wei. Mining unseen classes via regional objectness: A simple baseline for incremental segmentation. Advances in Neural Information Processing Systems, 45:24340-2443 2022.
[38] Zekang Zhang, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, and Yunchao Wei. Coinseg: Contrast inter-and intra-class representations for incremental segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[39] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1881-28995 2017.
[40] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 633 603,641,222 2017.
[41] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Com- puter Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedinggs, Purt XXVIII, pages 696-712. Springer, 2022.
[42] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lasion-languuae models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[43] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009):2337-2348 2022.
13

---

Appendix
Here introduce technical details of the "frozen CLIP" approaches in Sec. A. The dataset settings are shown in Sec. B. Moreover, we provide additional experiments in Sec. C, and additional qualitative results in Sec. D.
A Technical details of the "frozen CLIP" approaches
Train only
Test only
cls. score AP
Loss
Train & Test
seg. Loss
Proposal Generator
maak propomil xima
img. embed
CLIP Image Encoder
Laua
Xxa
emsemble
Adaptive predetined Pronapt
M
merge
sheep
CLIP
Cuns Cund Xd 00:0
els score A
Trained
text embed E
Frozen
Figure 5: Overview of the "decoupling paraaliiing.aaimmmmm..
Fig. 5 presents an overview of the "frozen CLIP" approach. During training, a standard MaskFormer or Mask2Former is used as Proposal Generator generate N mask proposals (M, M e XXxxx))) and classification score (AP, AP E RR- During testing, the input image is merged with M obtain N sub-images sLuu Leub E Nxhx(e) These sub-images are fed into frozen CLIP get the CLIP classification score (A?, A Nx/Cenn (Csnaeen)))))))) Here Cseen and Cunseen represent set of seen classes and unseen classes. An ensemble operation is used to ensemble A" and A for the final prediction. The merge and the ensemble operations will be introduced in detail in following:
Merge operation. To generate appropriate sub-images based on mask proposals, [6] presents three different merge operations: 1) mask, 2) crop, 3) mask & crop. Through experimentation, they demonstrate that the mask & crop option yields the best results. Fig. provides an example of these operations. It's worth noting that all sub-images are resized to h x ú; here h and i typically take a value of 224, which is the default input size of CLIP Image Encoder. Although acceptable results can be obtained with the merge operation, it involves repeatedly feeding images into CLIP, which leads to significant computational redundancy.
Ensemble operation. Comparatively, A" provides higher confidence classification scores for the seen classes, and A provides higher confidence classification scores for the unseen classes. Therefore, an ensemble of AP and A achieves better results. The ensemble operation can be formulated as:
A(c) [P'(c)> (c) C Ceeen
(c(c)>
C Cunscen
here a geometry mean of AP and A is caleulated (dubbed Å), and the contribution of both classification scores is balanced by X. As per literature (6; 33; 27], X usually takes values from 0.6 to 0.8. Therefore, the final output (O, multiplication: = Å" M. With the ensemble operation, the classification results of seen classes primarily depend on AP, whereas the classification results of unseen classes mainly rely on A
can be obtained by matrix
B Dataset
We follow [l; 11; 26; 6; 33] to conduct experiments on three benchmarks of the popular zero-shot setting, Pascal-VOC. COCO-Stuff and ADE20K, to evaluate the performance of MAFT. Additionally, we evaluate MAFT on the open-vocabulary setting [20; 33), iee,, training on COCO-Stuff and testing on ADE20K, Pascal-Context, and PaScal-VOC.
14

---

M
(1) mask
(2) crop
(3) mask & crop
Figure 6: Comparison among three merge operations.
COCO-Stuff: COCO-Stuff is large-scale semantic segmentation dataset that includes 171 classes. For the zero-shot setting [6; 33; 27], it is divided into 156 seen classes for training and 15 unseen classes for testing. For the open-vocabulary setting, all 171 classes are used for training.
Pascal-VOC: There are 10582 images for training and 1,449 images for testing. For the zero-shot setting, Pascal-VOC is split into 15 seen classes and 5 unseen classes. For the open-vocabulary setting, all classes are used for evaluation (dubbed as PAS-20).
ADE20K: ADE20K contains 25k images for training and 2k images for validation. For the zero-shot setting, follow [6] choose 847 classes present in both training and validation sets, and split them into 572 seen and 275 unseen classes. For the open-vocabulary setting, we use settings of ADE20K: 150 classes (dubbed as A-150) and 847 classes (dubbed as A-847).
Pascal-Context is an extensive dataset of Pascal-VOC 2010. Two versions are used for open-vocabulary setting, one with 59 frequently used classes (dubbed as PC-59) and another with the whole 459 classes (dubbed as PC-459).
C Additional experiments
C.1 Analysis of the Upper Bound of MAFT
Table 7: Upper Bound analysis.
COCO-Stuff
mIoU"
mIoU"
hIoU
mIoU
MAFT Upper Bound
50.4
46.5
43.9
77.2
82.1
79.6
77.6
Considering the mask-aware loss may be limited if the quality of proposals is too bad, conducted an evaluation of the Upper Bound while using Mask2Former as the proposal generator. The results are presented in Tab. 7. Specificallyy, we replace A by SIOU (IoU score between binary ground-truth masks and proposals) during inference, and multiply proposals with SIOU to obtain the segmentation result. This result can be regarded as the Upper Bound for the given proposals. Notably, the Upper Bound achieves satisfactory results (77.6 % mloU), indicating Mask2Former is capable of providing high-quality proposals in most cases. Additionally, there is still a large gap between the current performance and the Upper Bound (sz 30% mloU), which suggests that our MAFT has enormous potential for improvement, whereas we have achieved ofate-of-the-art performance.
C.2 Analysis of the Self-Training (ST) strategy
Several previous approaches [25; 33; 41] adopt the Self-Training (ST) strategy to enhance perfor- mance. we conduct experiments to investigate the application of ST into our method. Specifically, we use the existing FreeSeg MAFT model to generate pseudo-labels for unseen classes on the training data, and then re-train FreeSeg with the lacudo-labels. Results are shown in Tab. 8.
15

---

Table 8: Analysis of the self-training (ST) strategy.
VOS
COCO-Stuff
mloU
mloU
hloU
mloU
mloU"
mloU"
hloU
mloU
MAFT
91.4
81.8
86.3
89.0
43.3
50.4
46.5
43.9
MAFT ST
90.0 -14
86.3
44.8
88.1 41.8
89.1 40.1
44.1
40.8
55.2 4.4.8
49.0 42.6
45.0 +1.1
The improvement of ST on the unseen category is significant (Pascal: 81.8 % + 86.3%. COCO: 50.4% - 55.2%% in terms of mloU". However, it's essential to highlight the applicability of ST is limited by crucial requirement: unseen classes need to be obtained during training. This requirement poses significant limitations on generalizing ST to various scenarios, e.g., open-vocabulary settings, since images of unseen classes may not be obtained during training.
D Visualization
We provide more qualitative results, including typical proposals and top-5 A (Fig. 7), as well as examples of models trained COCO and text A-847 (Fig. 8), A-150 (Fig. 9), PC-459 (Fig. 10), PC-59 (Fig. 11), Pascal- VOC (Fig. 12), and COCO-Stuf((Fgg, 13).
Typical Proposals and Top-5 A Fig. 7 shows frozen CLIP and mask-aware CLIP classifications of typical proposals. the 2nd column, we provide high-quality proposals of thing classes. Both the frozen CLIP and mask-aware CLIP provide high classification scores for the correct classes. In the 3rd column, we provide proposals that only contain part of the objects (row 1-3), and proposals containing more than class (row 4). The mask-aware CLIP provides more proper results compared to the frozen CLIP. In the 4th column, we provide some high-quality background proposals. The frozen CLIP typically gives incorrect predictions, but the mask-aware CLIP assigns high scores for the correct classes.
Qualitative Analysis. Fig. 8,9,10,11,12,13 show segmentation results on Pascal-VOS COCO- Stuff, ADE20K. In Pascal-VOO VOC dataset (Fig. 12), which only contains 20 thing classes, the FreeSeg + MAFT model tends to assign background regions to the similar thing classes, egg., "train" row "pottedplant" in row3-4. "boat" in row 8. In A-847, A-150, PC-459, PC-59 and COCO-Stuff datasets, both seen classes and unseen classes exist in the input images, the FreeSeg + MAFT model generates better segmentation results compared to FreeSeg.
16

---

siplane: 91 1,0% walzndrup 2.2%
airplane: 95, 77s
airpine: 75 5.
airplzze: 18.2%
zirplane: 95 2% kale 1.01%
sky: 95 2016
Top5 prllicton by frozen CLIP
metal: 0 78%
kile: 1.62%
mutal: 13.58%
cloude 3 3.069%
prolistions mark eware CLIP
cuiling: 0.56%,
sinctural 0..7%
skic 1,0%,
strustus:: 6.0%
sky: 58%
gouandd 0.27%
strctiri: 0.4%
skyr 8,29% person:
knife:
ground: 2.00%
structural 0.21% Bi: 0.20%
nirplaner 2..7%
structural:
person:
cow:
sheap: 50,62%
4.7256
0.63%
vegetable.0. 0. 17%
sheap: 4,07%%
sheep: 0.53%
fichl: 0.44%
shaqgs 0.62%
3.04%
acbra: 2.03%
0.49%
lapsop:
laptups 20% keyboand: 3.30% playingfichl: 3,0%
luptog: byptop 18% keyboard:
deske 42,
kayhurnd 19%% 0.09% mewsve: 0.84%
diningtabler
clutk: 1.0%% 2 glose: 2.17%
3.81% cubiint: 3,73%,
2.4856
1.74%
bontle:
15%g
cups 77%,
7.73%
diningsble 12.85%
diningtable: 0.76%6 desk-stuff 0.52%,
1.20% 0.57%
sandwicch: 3.12%
3.30%
0.0%%
12.0%
wall wrall-stene: 2.51%
0.26%%
plantic: 0.93%
plantic 44%
wolid-other. 2.49%
Figure 7: Visualizations of typical proposals & top A by frozen CLIP and mask-aware CLIP. The correct classes are highlighted in red.
17

---

133 PUTO
gasoune SLIPO
CrENO DIVCITA
trreelign Srreet
traffic ligtt traffic signal,
(ONO) IBLERTUUS
Auto autommbblle mathhe mulc
CAr. HURO Wi Lomobrie,
EUTO. curting keft:
roNd, TOuR
migeonhole
bulling. edfiic
say CaSE showcase viree
ddewak pavement
Edewaak
Image
GT
FreeSeg
FreeSeg+ MAFT
Figure 8: Qualitative results A-847, using 847 class names in ADE20K generate text embed- dings.
18

---

Image
GT
FreeSeg
FreeSeg+ MAFT
Figure 9: Qualitative results A-150, using 150 class names in ADE20K generate text embed- dings.
19

---

CARMESA SALOON
LOON
SALOON
Image
GT
FreeSeg
FreeSeg+
Figure 10: Qualitative results on PC-459, using 459 class names in Pascal-Context generate text embeddings.
20

---

Plero
Plamm
Phoro
ARONA
ARONA HOTEL
ARŌNA GRAN HOTEL
ARÓNA
GRAN HOTEL
GRAN LITEL
VCS
Image
GT
FreeSeg
MAFF
Figure 11: Qualitative results on PC-59, using class names in Cocal--Context to generate text embeddings.
21

---

Image
GT
FreeSeg
FreeSeg MAAFT
Figure 12: Qualitative results Pascal-VOCC using 20 class names in Pascal-VOC VOC to generate text embeddings.
22

---

J.P.Morgan
J.P.Morann
M.P.Mrr
J.P.orran
YEARS
Image
GT
FreeSeg
FreeSeg+MAFT
Figure 13: Qualitative results on COCO. using 171 class names in COCO-Stuff to generate text embeddings.
23


Reviews:
[{"review_id": "YhCacCADOu", "reviewer": "Reviewer_fJQP", "summary": "The general goal of the paper is to leverage CLIP for zero-shot segmentation. For this, in contrast to prior work, an CLIP-inspired IP-CLIP encoder is trained to enable mask-level encodings for segmentation. The core of the approach is the so-called IP-encoder that uses as input a frozen mask generator, as well as two losses, a mask-aware loss and a distillation loss", "questions": "Here are my main questions about writing\n\n1) A^c is defined to be of dimension NxC (line 119) - with N the number of mask proposals\nin the self-distillation loss however, figure 2 is showing a cx1 dimensional vector?\nin any case I am confused as standard CLIP (here used as teacher) would not generate a NxC dimensional \nmap - and thus A^C_{fro} in equation (7) does not seem to make sense to me - can you please explain?\n\n2) related to that: in figure 2 it seems that the final projection from IP-CLIP encoder is used without biases? (w.o. B) - that seems quite obscure to me actually - what is meant? \n\n3) the mask-aware loss seems sensible on a high-level - but the details are not really motivated well. E.g. the reason for normalizing the IoU scores is unclear (equation 5)  - similarly the reason behind the exact formulation of equation 6 remain unclear. \n\n4) the paper uses L layers prior to condition on the masks, and 12-L after that. While there is an experimental ablation about this why would that make sense intuitively?\n\nif the authors can clarify these points I will consider upgrading my score. \n\ndetail:\n- line 125 \"wildly\" -> \"widely\"\n\n\npost rebuttal\nthanks for addressing my questions - I have upgraded mu review as mentioned in my initial review. Please make sure that the improvements are promised are implemented - thanks", "limitations": "ok", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The general idea is good and on a high-level the components (IP-CLIP encoder, mask aware loss and CLIP distillation) make sense to me\n\nThe reported results are good across three prior baselines and different datasets", "weaknesses": "In my view the paper is not well written and important details are either not well motivated or even unclear (see below)\n\nThe paper reads to a large extend like an engineering paper with a few changes here and there to adapt to the task as hand. I would assume that to be not so interesting for the majority of NeurIPS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer fJQP\n\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. Explanation of $A^c \\in\\mathbb{R}^{C \\times N}$, $A^c_{IP}$$\\in\\mathbb{R}^{C \\times 1}$ and $A^{c}_{fro}$$\\in\\mathbb{R}^{C \\times 1}$**\n\nIn our framework,  $A^c \\in\\mathbb{R}^{C \\times N}$ is used for computing the *mask-aware loss*. $A^c_{IP} \\in \\mathbb{R}^{C \\times 1}$ and $A^c_{fro} \\in \\mathbb{R}^{C \\times 1}$ are generated only when computing the *self-distillation loss*. Specifically, during the process denoted by the red arrows in Fig. 2, the image propagates into both IP-CLIP and frozen CLIP without incorporating any mask proposals, resulting in  $A^c_{IP}$ and $A^c_{fro}$ respectively. To execute this process at the code level, we invoke the `forward` function of the frozen CLIP and load the parameters of the IP-CLIP to obtain $A^c_{IP}$. Therefore, $A^c_{IP}$ and $A^c_{fro}$ are both vectors of dimensions $C\\times 1$. We utilize the $\\mathrm{smoothL1}$ in Eq. 7 to enforce equivalence between $A^c_{IP}$ and $A^c_{fro}$.\n\n**Q2. Explanation of the final projection unit and** *w.o.* $B$\n\nFirst, the final projection unit within CLIP is essentially an MLP module, adept at reshaping the channels of $F_{feat}$ and $F_{cls}$. Notably, the final projection is one component unit in CLIP, and it operates independently of attention bias ($B$). \nBesides, the notation \"*w.o.* $B$ \" in Fig.2 relates to the process for generating $A^{c}_{IP}$ as described in **Q1**. During this process, the image is directly input to IP-CLIP without incorporating any mask proposal. *i.e.*, this process does not involve attention bias and *repeat* operation. \n\n\n**Q3. Explanation of the *mask-aware loss***\n\nThe alignment between $A^c_{select}$ and $S_{IoU}$ is a fundamental goal of the *mask-aware loss*. We observe that in most cases, the maximum value of $A^c_{select}$ tends to approach 1, whereas the maximum value of $S_{IoU}$ ranges from 0.75 to 0.99. This inconsistency can hinder the convergence of the *mask-aware loss*. To address this issue, we introduced a normalization technique for $S_{IoU}$ using Eq. 5. This normalization technique enhances the smoothness of the loss and effectively reduces training variance. Notably, normalization techniques within the loss function have been widely applied in prior works [1*, 2*]. In addition, we experimentally demonstrate the effect of this normalization technique in the subsequent table.\n\nEffect of normalization technique under zero-shot setting (COCO-Stuff):\n| |mIoU$^{s}$ | mIoU$^{u}$ | hIoU| mIoU\n| -------- | -------- | -------- |-------- |- |\n| FreeSeg + MAFT | 43.3     | 50.4     |46.5| 43.9\n| FreeSeg + MAFT (without norm) | 42.1    | 48.7   |45.2| 42.7\n\nEq. 6 provides the mathematical representation of the $\\mathrm{smoothL1}$ function. The primary goal of the proposed *mask-aware loss* is to improve CLIP's classification ability of proposals. Reflecting within the loss function, *i.e.*, aligning $A^c_{select}$ and $S_{IoU}$ by $\\mathrm{smoothL1}$. \n\n[1*] Mask Matching Transformer for Few-Shot Segmentation, NeurIPS 2022\n\n[2*] Few-shot semantic segmentation via mask aggregation, arxiv 2022\n\n\n**Q4. Explanation of starting mask attention from layer $L$**\n\nWe consider that CLIP's classification significantly relies on context information. If attention bias is introduced right from the first transformer layer, $F_{cls}$ will only interact with the foreground pixels in $F_{feat}$. In this scenario, the loss of context information would be inevitable.\nTherefore, in the initial $L^{th}$ transformer layers of IP-CLIP, we maintain the same design as the original CLIP. $F_{cls}$ utilizes cross-attention with all pixels in $F_{feat}$, thereby retaining the crucial context information. In the subsequent $12-L^{th}$ layers, we introduce the attention bias within MultiHead Attention, with mask-aware fine-tuning to learn the mask-aware representation.\n\n**Q5. Assuming that the paper is not so interesting for the majority of NeurIPS**\n\nIt's important to note that zero-shot segmentation constitutes a compelling task within the realm of NeurIPS, as evident from the pertinent works published over recent years [1*, 2*, 3*, 4*]. \nWe achieve noteworthy performance enhancements in both zero-shot and open-vocabulary settings by appropriate fine-tuning of the CLIP Image Encoder. We emphasize that while our approach is simple, it remains impactful. It introduces a solid and innovative solution that can seamlessly integrate with zero-shot and open-vocabulary methods (*e.g.*, Zegformer, ZSSeg, FreeSeg).\n\n[1*] ReCo: Retrieve and Co-segment for Zero-shot Transfer (NeurIPS 2022) \n\n[2*] The Emergence of Objectness: Learning Zero-shot Segmentation from Videos (NeurIPS 2021)\n \n[3*] Consistent Structural Relation Learning for Zero-Shot Segmentation (NeurIPS 2020) \n\n[4*] Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (NeurIPS 2020) \n"}, {"review_id": "ttU6N86Vkp", "reviewer": "Reviewer_q7Qn", "summary": "The paper proposes a mask-aware fine-tuning method to address challenges faced by frozen-CLIP-based zero-shot segmentation methods. It addresses the problem of CLIP being insensitive to different mask proposals and tending to produce similar predictions regardless of the variation in proposals. The proposed IP-CLIP successfully assigns appropriate scores to different proposals, unlike the frozen CLIP that exhibits similar scores. Instead of processing each mask individually, the proposed modified CLIP considers all mask proposals simultaneously, thereby reducing computational costs. The experimental results consistently demonstrate that the proposed method outperforms the baselines by a significant margin.", "questions": "- Including experiments with other backbones and proposal generators would enhance the comprehensiveness of the paper\n", "limitations": "The limitations are briefly discussed in the paper, while the societal impact is not addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": " - The proposed method is designed as a plug-and-play approach, making it applicable to any frozen CLIP-based method.\n- The proposed method consistently improves the performance of baseline methods, including SegFormer, ZSSeg, and FreeSeg, by substantial margins, particularly on unseen classes.\n- The method significantly reduces the computational requirements of CLIP in FreeSeg, and the effectiveness of the proposed mask-aware loss and IP-CLIP is demonstrated through ablation studies.\n", "weaknesses": "- The starting point of the mask attention layer L is determined by a user-defined hyperparameter. The proposed method specifically employs ViT-B/16 as the backbone in the paper. However, if a different backbone is utilized, the selection of this hyperparameter would necessitate a hyperparameter search.\n\n- The notation presented in the paper would be better if it were simplified and clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer q7Qn\n\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. Experiments with other backbones and proposal generators**\n\nTo enhance the comprehensiveness of the paper, we extend our experimentation with more Vision-Language Models (CLIP-ViT-L and CLIP-R50) and more proposal generators (SAM-H).\n\n* **Using CLIP-ViT-L for proposal classification**  \nWe conduct experiments using a stronger Vision-Language Model, CLIP-ViT-L, and the results in cross-dataset setting are shown below. We also include the results of OvSeg with CLIP-ViT-L for comparison. FreeSeg with a standard CLIP-ViT-L model (dubbed `FreeSeg`) still can not achieve satisfactory results. However, by integrating our MAFT (dubbed `FreeSeg+MAFT`), the segmentation results are remarkably enhanced, thus establishing new state-of-the-art benchmarks.\n\n|(ViT-L) | A-847 | A-150 | PC-459 | PC-597 | PAS-20 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| OvSeg      | 9.0     | 29.6     |   12.4  | 55.7     | 94.5     |\n| Freeseg    | 8.5     | 21.0     |7.6     | 33.8     | 86.4     |\n| Freeseg + MAFT | 12.1 (+3.6)    | 32.0 (+11.0)    |15.7 (+8.1)   | 58.5 (+25.3)    | 92.1 (+5.7)   |\n\n* **Using CLIP-R50 for proposal classification**  \nWe also extend our MAFT to CLIP-R50, a Vision-Language Model based on ResNet. Specifically, we modified the $\\mathrm{AttentionPool2d}$ unit within CLIP-R50 Image Encoder. The mask proposals are introduced as attention bias ($B$ ) in Multihead Attention, with $F_{cls}$ being repeated N times. Notably in CLIP-R50, $F_{cls}$ is obtained via performing $\\mathrm{Global Average Pooling}$ on $F_{feat}$. The results are presented below, the performance on all 5 datasets is improved by a large margin. `Freeseg+MAFT` with CLIP-R50 achieves competitive results with some CLIP-ViT-B-based methods such as OvSeg and OpenSeg, according to Tab. 3 in the paper.\n\n|(R50) | A-847 | A-150 | PC-459 | PC-597 | PAS-20 |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| Freeseg    |  5.3    | 15.5    |5.4     | 28.2     | 78.1     |\n| Freeseg + MAFT | 8.4 (+3.1)    | 27.0  (+11.5)   |9.9 (+4.5)    | 50.8  (+22.6)    | 89.0  (+10.9)   |\n\n\n\n* **Using Segment Anything Model (SAM-H) as proposal generator**  \nFurthermore, we explore using SAM-H as proposal generator. Compared to the Mask2Former trained on COCO-Stuff, SAM-H can produce superior-quality mask proposals. According to **General Response-Q1**, The successful collaboration of SAM-H with our MAFT approach (dubbed `SAM+MAFT`) has yielded remarkable performance enhancements.\n\nIn a word, the aforementioned experiments demonstrate the efficacy and robustness of MAFT across different Vision-Language Models and different proposal generators. We will include these experiments in the final version.\n\n\n**Q2. Concerns about defining the hyperparameter $L$**\n\nIt is indeed a consideration when utilizing new backbones, the value of $L$ needs to be re-determined. However, it's worth noting that considering the short training duration of our MAFT (as detailed in **Line 221** of the paper), the cost of defining $L$ should be acceptable. To provide further insight, we present the training time for MAFT on different datasets using 4 RTX A6000 GPUs. *e.g.*, on the commonly used COCO-Stuff dataset, MAFT only takes about 20 minutes for training. \n\n||Traning iterations| Traning times | \n| -------- | -------- | -------- |\n| Pascal-VOC  | 100    | 2 minutes     |  \n| COCO-Stuff  | 1000    | 20 minutes     |\n| ADE20K      | 5000     | 100 minutes     |\n \n**Q3. Simplify and clarify the notations**\n\nThanks for your kind suggestions, we will optimize and clarify the notations in the final version. *e.g.*, $A^c_{IP}$ and $A^c_{fro}$ might be confused with $A^c$. We will simplify them with $A_{S}$ (classification score of the student net) and $A_{T}$ (classification score of the teacher net), while showcasing them in Fig. 2. We believe the simplification and clarification will contribute to the understanding of this work."}, {"review_id": "s46Ds0jTiE", "reviewer": "Reviewer_uMA5", "summary": "This paper mainly discusses how to use pre-trained CLIP to solve zero-shot segmentation task, and proposes a new method called Mask-aware Fine-tuning (MAFT) to address the issue of significant false positives in CLIP's classification of mask proposals. Specifically, the paper introduces an Image-Proposals CLIP Encoder (IP-CLIP Encoder) to handle any number of images and mask proposals simultaneously, and designs mask-aware loss and self-distillation loss to fine-tune the IP-CLIP Encoder, ensuring that CLIP responds to different mask proposals without sacrificing its transferability.", "questions": "Same as weakness.", "limitations": "The paper has a description of some limitations.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces an Image-Proposals CLIP Encoder (IP-CLIP Encoder) that is sensitive to different mask proposals.\n\n2. This paper includes mask-aware loss and self-distillation loss to fine-tune the IP-CLIP Encoder without sacrificing its transferability.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The ability to handle any number of mask proposals is not unique to this method and has already been a feature of previous methods such as ZegFormer.\n\n2. The main effect of this method comes from the mask-aware loss, which utilizes mask proposals as prior knowledge to obtain more accurate prediction probabilities from the cls score map. Therefore, the effectiveness of this loss function is limited by the quality of the mask proposals, which limits the innovation of this paper.\n\n3. In terms of experiments, it is necessary to conduct experiments on the updated methods such as \"Scaling Open-Vocabulary Image Segmentation with Image-Level Labels\"(ECCV2022) where the performance of it has already surpassed this method on VOC and COCO.\n\n4. Why is Table 2's benchmark experiment conducted under the setting of using only CLIP classifier?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer uMA5\n\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. The ability to handle any number of mask proposals is not unique.**\n\nThere may be some misunderstandings about the contribution of our MAFT. Although there are a few approaches proposed to handle any number of mask proposals, the way our MAFT employs the mask proposals is significantly different from these \"frozen CLIP\" methods and is more effective and efficient.  We would like to re-emphasize the difference as follows. \n\n* The \"frozen CLIP\" approaches (*e.g.*, Zegformer, ZSSeg and FreeSeg) use a frozen CLIP for mask proposal classification. Specifically, they first use a $\\mathrm{Merge}$ operation on the input image, creating $N$ sub-images ($I_{sub}$), where $N$ (defaulted to 100) is the number of mask proposals.This $\\mathrm{Merge}$ operation is detailed at **Line 12** of the supplementary material. Then, $I_{sub}$ undergo classification with a frozen CLIP. This approach causes significant computational redundancy and the $\\mathrm{Merge}$ operation loses image context information.  \n* MAFT, on the other hand, innovates by modifying CLIP. It achieves simultaneous classification for all proposals with the image single pass through CLIP. Specifically, we modify CLIP by repeating $F_{cls}$ $N$ times and using $N$ mask proposals as attention bias in MultiHead Attention. This efficient design eliminates the need for the aforementioned $\\mathrm{Merge}$ operation, reducing the computational cost and the image's context information is retained.\n\nTo highlight the computational efficiency of MAFT, we provide GFLOPs analyses in Tab. 4(a). And we also evaluate the inference time (in terms of FPS) in the following table. All models are based on the CLIP-ViTB/16 and tested with one single RTX A6000 GPU for a fair comparison.\n\n|(FPS)|Zegformer|ZSSeg|FreeSeg\n|-|-|-|-\n|frozen CLIP|11.9|12.1|9.3\n|IP-CLIP|27.6|29.1| 25.5\n\n**Q2. Concerns about the quality of the mask proposals may limit the innovation**\n\nWe agree that the *mask-aware loss* may be limited if the quality of proposals is too bad. However, we experimentally demonstrate that using Mask2Former as proposal generator can already provide high-quality proposals. In detail, we replace $A^{c}$ by $S_{IoU}$ (IoU score between binary groundtruth masks and proposals) during inference, and multiply proposals and $S_{IoU}$ to obtain the segmentation result. This result can be regarded as the Upper bound for the given proposals. Notably, the `Upper bound` achieves satisfactory results (77.6 mIoU), indicating Mask2Former can provide high-quality proposals in most cases. Additionally, there is still a large gap between current performance and the Upper bound (≈ 30% mIoU), which suggests that our MAFT has enormous potential for improvement whereas we have achieved state-of-the-art performance.\n\nZero-shot setting (COCO-Stuff):\n||mIoU$^{s}$|mIoU$^{u}$|hIoU|mIoU\n|-|-|-|-|-\n|FreeSeg + MAFT| 43.3|50.4|46.5|43.9\n|Upper bound|77.2|82.1|79.6|77.6\n\nBesides, we also explore using stronger proposal generators, *e.g.* SAM-H, as detailed in **General Response-Q1**. The results showcase that the MAFT approach achieves remarkable performance improvements when leveraging superior-quality proposals.\n\nWe hope these experiments and analyses can demonstrate the efficacy and potential of our approach. Your concerns have guided us to delve deeper in our approach, and we are confident that our method offers a valuable contribution to the field.\n\n**Q3. Suboptimal performance comparing with OpenSeg**\n\nWe have compared our method with \"Scaling Open-Vocabulary Image Segmentation with Image-Level Labels\" (dubbed OpenSeg*) in Tab. 3. Although OpenSeg* additionally uses Image-level labeled data for training, our method outperforms it on all five datasets (A-847, A-150, PC-459, PC-59, PAS-20).\n\nWe also investigate new work and find that MaskCLIP+[1*] achieves better performance under zero-shot setting compared to ours. The performance gain in MaskCLIP+ comes mainly from the Self-Training (ST) strategy, which can be easily applied into our method. Specifically, we use the existing FreeSeg+MAFT model to generate pseudo-labels for unseen classes on the training data, and then re-train FreeSeg and MAFT with the pseudo-labels.\n\n|Pascal-VOC|mIoU$^{s}$|mIoU$^{u}$|hIoU|mIoU\n|-|-|-|-|-\n|FreeSeg + MAFT|91.4|81.8|86.3|89.0\n|FreeSeg + MAFT + ST|**90.0**(-1.4)|**86.3**(+4.5)|**88.1**(+1.8)|**89.1**(+0.1)\n|MaskCLIP+|88.8|86.1|87.4|88.1\n\n|COCO-Stuff|mIoU$^{s}$|mIoU$^{u}$|hIoU|mIoU\n|-|-|-|-|-\n|FreeSeg + MAFT|43.3|50.4|46.5| 43.9\n|FreeSeg + MAFT + ST|**44.1**(+0.8)|**55.2**(+4.8)|**49.0**(+2.5)|**45.0**+(1.1)\n|MaskCLIP+|38.1|54.7|45.0| 39.6\n\nThe improvement of ST on the unseen category is significant (Pascal: 81.8 $\\rightarrow$ 86.3, COCO: 50.4 $\\rightarrow$ 55.2) in terms of mIoU$^{u}$. However, it's essential to highlight the applicability of ST is limited by a crucial requirement: **unseen classes need to be obtained during training**. This requirement poses significant limitations on generalizing ST to general scenarios, *e.g.* open-vocabulary settings and cross-dataset settings, since images of unseen classes may not be obtained when training. Therefore, we do not recommend the use of ST, as mentioned in **Line 211**.\n\n[1*] Extract Free Dense Labels from CLIP (ECCV22)\n\n**Q4. Reasons for using only CLIP classifier in Table 2**\n\nPrevious works like Zegformer, ZSSeg and FreeSeg $\\mathrm{ensemble}$ $A^{c}$ (CLIP classification score) and $A^{p}$ (proposals generator classification score). The supplementary materials introduce the $\\mathrm{ensemble}$ operation in detail. However, our method only optimizes $A^{c}$ through the mask-aware fine-tuning process. So the setting of using only CLIP classifier in Tab. 2 provides a more direct and meaningful evaluation of the advancements achieved by MAFT."}, {"review_id": "5x9yDr3yvU", "reviewer": "Reviewer_XVP7", "summary": "This paper proposes a new topic and method for training a mask-aware CLIP, which could serve as a core component for open-vocabulary segmentation. The designed structure could be used as a flexible plug-in but brings significant improvements for existing methods on various benchmarks.", "questions": "See weakness", "limitations": "yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This topic is promising.  Previous methods decouple open-vocabulary segmentation into class-agnostic segmentation and CLIP-guided recognition. However,  most of them fail to use CLIP effectively,  I think training a mask-aware CLIP is an ideal way to deal with this problem.  \n\n2. The model design is reasonable, using a mask2former-style network and tasks the masks to perform masked attention sounds reasonable.\n\n3. The experiment results are great with significant improvement. It would be an ideal solution for various open-vocabulary segmentation tasks incorporating strong class-agnostic segmentation models like SAM. \n\n4. The paper is clearly presented.  ", "weaknesses": "The experiment setting is unsatisfactory, which only tackles zero-shot semantic segmentation. \nAs this topic and idea are good,  I expect the authors to extend the method into open-vocabulary panoptic settings, and use some large datasets for training. Currently, the datasets used are small. it is hard to distill universal knowledge from CLIP.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer XVP7\n\nThank you so much for acknowledging the strength of our method. We have carefully considered your constructive and insightful comments and here are the answers to your concerns.\n\n**Q1. Extending the method into open-vocabulary panoptic settings**\n\nFirst of all, we fully agree with you that MAFT could be applied to more expansive tasks, *e.g.*, open-vocabulary segmentation, rather than tackling zero-shot semantic segmentation only. Therefore, we have expanded our experimentation to open-vocabulary semantic segmentation, termed the \"cross-dataset\" setting within the paper. The results are shown in Tab. 3. The results substantiate our method's efficacy, as it outperforms existing approaches, thereby establishing a new state-of-the-art benchmark of open-vocabulary semantic segmentation.\n\nFurthermore, we start an initial exploration into open-vocabulary panoptic segmentation. We first train a FreeSeg model with COCO-Panoptic labels as the baseline. Then, we use the proposed IP-CLIP that has been fine-tuned solely with semantic labels for proposal classification. We experiment with both CLIP-ViT-B and CLIP-ViT-L, results are presented as follows:\n\n\nOpen-vocabulary panoptic (PQ), instance (mAP) and semantic (mIoU) segmentation results with CLIP-ViT-B:\n||PQ| mAP | mIoU\n| -------- | -------- | -------- |--- |\n| FreeSeg   | 10.5    |4.7 |18.2  \n| FreeSeg + MAFT  | 17.1 (+6.6)   | 6.6 (+1.9)  | 27.6 (+9.4) \n\nOpen-vocabulary panoptic (PQ), instance (mAP) and semantic (mIoU) segmentation results with CLIP-ViT-L:\n||PQ| mAP | mIoU\n| -------- | -------- | -------- |--- |\n| FreeSeg   | 11.9    |7.1 |20.3  \n| FreeSeg + MAFT  | 18.9 (+7.0)   | 9.2 (+2.1)  | 30.9 (+10.1) \n\nThe results show the effectiveness of MAFT across different CLIP models in the open-vocabulary panoptic setting. For a comprehensive analysis of CLIP-ViT-L, please refer to the **Response to Reviewer q7Qn-Q1**. Notably, only semantic-level labels are used during the mask-aware fine-tuning process. It is possible to adapt *mask-aware loss* to panoptic-level labels, and would be an interesting future work.\n\n**Q2. Using some large datasets for training**\n\nIn the standard open-vocabulary segmentation scenario, COCO serves for model training, while additional segmentation datasets, *e.g.*, ADE20K, PASCAL, and Cityscapes, are employed for testing\n\nIntuitively, utilizing larger datasets for training can lead to better performance. With this in mind, we are actively exploring further enriching our training pipeline by including datasets such as LVIS and Open Images. These datasets offer a wealth of diverse visual data, thus potentially enhancing the model's generalization capability and transferability.\n\n\n**Q3. Incorporating strong class-agnostic segmentation models like SAM**\n\nThanks for your kind suggestions, we use SAM to instead of Mask2Former for proposal generation, resulting in notable performance enhancements for both open-vocabulary semantic segmentation and zero-shot segmentation. Please see **General Response-Q1** for details."}]

Author rebuttal:
# General Response

**Q1. Extending the method with SAM**

As some Reviewers suggest us use stronger proposal generators, *e.g.*, Segment Anything Model (SAM). We explore the performance with SAM-H using an original CLIP (dubbed `SAM+oriCLIP`) or a mask-aware fine-tuned CLIP (dubbed `SAM+MAFT`). In fact, SAM can be seamlessly integrated into our framework as the proposal generator. To provide a comprehensive comparison, we also include the results of `FreeSeg+MAFT`. Experiments are conducted under both zero-shot setting and cross-dataset setting. 

Zero-shot setting (Pascal-VOC): 
|  |mIoU$^{s}$ | mIoU$^{u}$ | hIoU| mIoU
| -------- | -------- | -------- |-------- |- |
| SAM + oriCLIP | 85.1  | 86.7 | 85.9 | 85.5
| SAM + MAFT | 91.0     | **88.6**    |**89.8**|**90.4**
| FreeSeg + MAFT | **91.4**     | 81.8     |86.3| 89.0

Zero-shot setting (COCO-Stuff): 
|  |mIoU$^{s}$ | mIoU$^{u}$ | hIoU| mIoU
| -------- | -------- | -------- |-------- |- |
| SAM + oriCLIP | 43.1    | 43.3   |43.2| 42.1
| SAM + MAFT | **43.4**    | **51.5**   |**47.1**| **44.1**
| FreeSeg + MAFT | 43.3     | 50.4     |46.5| 43.9

Cross-dataset setting: 
|  | A-847 | A-150 | PC-459 | PC-597 | PAS-20 |
| -------- | -------- | -------- |-------- |- |- |
| SAM + oriCLIP | 9.0     | 21.3     |7.8     | 33.7      | 87.5    |
| SAM + MAFT | **12.7**     | **33.0**     |**16.2**     | **59.0**      | **92.7**    |
| Freeseg + MAFT | 10.1     | 29.1     |12.8     | 53.5      | 90.0     |


It can be observed that `SAM+MAFT` obtains significant improvement over `SAM+oriCLIP` under both settings. Besides, `SAM+MAFT` also surpasses `Freeseg+MAFT` on all benchmarks. Particularly, in the zero-shot setting (Pascal-VOC), `SAM+MAFT` outperforms `FreeSeg+MAFT` by 6.8\% in terms of mIoU$^{u}$. This enhancement can be attributed to the stronger generalization capabilities of SAM for unseen classes. We will add these results in our final version. 


Meta-review: The paper addresses the problem of zero-shot segmentation by leveraging CLIP via training an encoder to enable mask-level encodings. The proposed approach leverages mask-aware loss and a distillation loss to fine-tune the image-proposals CLIP encoder (IP-CLIP), thereby enabling responses to different mask proposals without significantly degrading the transferrability. The manuscript received ratings of borderline accept,  weak accept, borderline reject and accept. While the reviewers appreciated the idea and effectiveness of the proposed approach, they also raised some concerns such as, writing issues and selection of hyper-parameters. Authors responded to reviewer's queries by submitting a rebuttal. The rebuttal largely addressed the initial concerns raised by the reviewers. Given that three reviewers are generally positive about the paper as well as author's rebuttal, the recommendation is accept. Authors are strongly encouraged to take into account the suggestions of the reviewers as well as changes acknowledged in the rebuttal when preparing the final manuscript.
Decision: Accept (poster)