Title: Learning Mask-aware CLIP Representations for Zero-Shot Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Zero-Shot Segmentation

Open-Vocabulary Segmentation

Fine-tuning
Abstract: Recently, pre-trained vision-language models have been increasingly used to tackle the challenging zero-shot segmentation task. Typical solutions follow the paradigm of first generating mask proposals and then adopting CLIP to classify them. To maintain the CLIP's zero-shot transferability, previous practices favour to freeze CLIP during training. However, in the paper, we reveal that CLIP is insensitive to different mask proposals and tends to produce similar predictions for various mask proposals of the same image. This insensitivity results in numerous false positives when classifying mask proposals. This issue mainly relates to the fact that CLIP is trained with image-level supervision. To alleviate this issue, we propose a simple yet effective method, named Mask-aware Fine-tuning (MAFT). Specifically,  Image-Proposals CLIP Encoder (IP-CLIP Encoder) is proposed to handle arbitrary numbers of image and mask proposals simultaneously. Then, *mask-aware loss* and *self-distillation loss* are designed to fine-tune IP-CLIP Encoder, ensuring CLIP is responsive to different mask proposals while not sacrificing transferability. In this way, mask-aware representations can be easily learned to make the true positives stand out. Notably, our solution can seamlessly plug into most existing methods without introducing any new parameters during the fine-tuning process. We conduct extensive experiments on the popular zero-shot benchmarks. With MAFT, the performance of the state-of-the-art methods is promoted by a large margin: 50.4\% (+ 8.2\%) on COCO, 81.8\% (+ 3.2\%) on Pascal-VOC, and 8.7\% (+4.3\%) on ADE20K in terms of mIoU for unseen classes. Codes will be provided for reproducibility. Code is available at https://github.com/jiaosiyu1999/MAFT.git .

Learning Mask-aware CLIP Representations for Zero-Shot Segmentation

Abstract

Recently, pre-trained vision-language models have been increasingly used to tackle the challenging zero-shot segmentation task. Typical solutions follow the paradigm of first generating mask proposals and then adopting CLIP to classify them. To maintain the CLIP's zero-shot transferability, previous practices favour to freeze CLIP during training. However, in the paper, we reveal that CLIP is insensitive to different mask proposals and tends to produce similar predictions for various mask proposals of the same image. This insensitivity results in numerous false positives when classifying mask proposals. This issue mainly relates to the fact that CLIP is trained with image-level supervision. To alleviate this issue, we propose a simple yet effective method, named Mask-aware Fine-tuning (MAFT). Specifically, Image-Proposals CLIP Encoder (IP-CLIP Encoder) is proposed to handle arbitrary numbers of image and mask proposals simultaneously. Then, mask-aware loss and self-distillation loss are designed to fine-tune IP-CLIP Encoder, ensuring CLIP is responsive to different mask proposals while not sacrificing transferability. In this way, mask-aware representations can be easily learned to make the true positives stand out. Notably, our solution can seamlessly plug into most existing methods without introducing any new parameters during the fine-tuning process. We conduct extensive experiments on the popular zero-shot benchmarks. With MAFT, the performance of the state-of-the-art methods is promoted by a large margin: 50.4% (+ 8.2%) on COCO, 81.8% (+ 3.2%) on Pascal-VOC, and 8.7% (+4.3%) on ADE20K in terms of mIoU for unseen classes. Code is available at github.com/jiaosiyu1999/MAFT.git.

1 Introduction

Semantic segmentation, one of the most widely researched topics in computer vision, has achieved remarkable success [3; 39; 15; 16] with the development of deep learning techniques [14]. However, traditional segmentation models are only capable of segmenting a few predefined categories within a closed vocabulary [7; 2; 22; 21], which is much smaller than the number of categories used by humans to describe the real world. Therefore, zero-shot segmentation [31; 1; 10; 13] is introduced to segment objects using arbitrary categories described by texts.

Recently, large-scale visual-language pre-training models ( e.g. CLIP [28] and ALIGN [17]) have shown impressive transferability in recognizing novel categories, leading to their increased adoption

Image

person: 0.10%

orange: 0.03%

horse: 0.02%

carrot: 0.02%

solid: 0.05%

person: 0.04%

pillow: 0.01%

towel : 0.01%

river: 12.38%

water: 10.50%

sea: 0.74%

plastic: 0.25%

water: 0.03%

boat: 0.02%

river: 0.02%

person: 0.01%

GLYPH(cmap:d835)3: proposal of  swan & river

GLYPH(cmap:d835)

2: proposal of  river

GLYPH(cmap:d835)1: proposal of  swan

river: 73.28%

water: 22.74%

sea: 1.32%

rock: 0.69%

stone: 0.24%

water: 0.85%

boat: 0.17%

river: 0.17%

plastic: 0.12%

predictions by frozen CLIP

predictions by mask-aware CLIP

for tackling the challenging zero-shot segmentation task [6; 33; 20; 27]. A mainstream solution follows the "frozen CLIP" paradigm, which executes the zero-shot segmentation with two steps: 1) first employing a Proposal Generator to produce class-agnostic mask proposals and 2) then leveraging a frozen pre-trained CLIP to classify each mask proposal via similarity matching in the aligned image-text feature space. While acceptable results are obtained, we reveal that these approaches overlook a crucial issue, i.e. the frozen CLIP is insensitive to different mask proposals and tends to produce similar predictions for various proposals of the same image.

To better illustrate the above-mentioned issue, we show several examples in Fig. 1. We use MaskFormer [5] to generate a series of mask proposals and select three typical ones. When using frozen CLIP for classification, we observe that it correctly classifies the high-quality swan proposal p 1 . However, for the other two proposals p 2 and p 3 , which respectively contain only shape information of swan and both regions of swan and river , the frozen CLIP produces similar predictions compared to p 1 . This is reasonable since CLIP is trained by image-text pairs, making it insensitive to pixellevel information ( e.g. background noise), and resulting in numerous false positives. Based on the above observations, we consider that an expected CLIP for zero-shot segmentation task should 1) be sensitive to different mask proposals, 2) not compromise its original transferability on novel classes.

To this end, we introduce a Mask-aware CLIP Fine-tuning method (dubbed MAFT). To make CLIP sensitive to different mask proposals, we devise an Image-Proposals CLIP Encoder (IP-CLIP Encoder), which utilizes mask proposals to perform masked Multihead Attention [5; 4]. This design enables the model to handle arbitrary numbers of images and proposals simultaneously. The maskaware loss is proposed to minimise the distance between the IoU score of mask proposals and the classification score of IP-CLIP Encoder, prompting IP-CLIP Encoder to differentiate various proposals. Besides, to preserve CLIP's zero-shot transferability, we utilize a frozen CLIP as a teacher network to facilitate fine-tuning. This is achieved by aligning the outputs of the frozen CLIP and IPCLIP Encoder through self-distillation loss . By performing MAFT, several advantages are provided: 1) Fine-tuning is efficient since only a few mask proposals need to be classified. 2) Compared to pixel-level fine-tuning, mask-aware fine-tuning hardly alters the structure of CLIP itself, preserving its maximum transferability. 3) Mask-aware fine-tuning of CLIP is released from the segmentation module, making it plug-and-play and applicable to any "frozen CLIP" approaches. As shown in Fig. 1, the mask-aware CLIP can well distinguish different proposals and provide proper classification scores for both seen ( river ) and unseen ( swan ) classes.

We evaluate our MAFT on three commonly used zero-shot segmentation benchmarks: COCOStuff [2], Pascal-VOC [7], and ADE20K [40]. Extensive experiments show that MAFT works well with various zero-shot segmentation methods. In particular, by plugging MAFT, the state-ofthe-art approach FreeSeg [27] achieves superior performance on COCO-Stuff (42.2% → 50.4%), Pascal-VOC (78.6% → 81.8%) and ADE20K (4.4% → 8.7%) in terms of mIoU of unseen classes. Furthermore, we conduct experiments in a open-vocabulary setting, where MAFT enhances the performance of A-847 [40], A-150 [40], PC-459 [24], PC-59 [24] and PAS-20 [7] datasets by +3.0%, +11.2%, +6.4%, +19.1% and +4.4%, respectively. Notably, our approach outperforms the freezing CLIP counterpart and establishes new state-of-the-art results on all datasets.

2 Related Work

Zero-Shot Segmentation [29] is established to break the restriction of categories and perform segmentation on unseen classes. Earlier works SPNet [31] learn a joint pixel and vocabulary concept embedding space, ZS5 [1] utilizes a generative model to generate pixel-level features based on word embeddings of unseen classes, CaGNet [10] incorporates context information for better feature generation. Recent approaches take the advent of large-scale visual-language models ( e.g. CLIP [28] and ALIGN [17]) to leverage rich alignment features from image-text pairs. [34] uses CLIP to generate pseudo-labels for single-image segmentation. STRICT [25] obtains pixel-level pseudo-labels from CLIP for unlabeled pixels and proposes a self-training strategy to capture latent information on unseen classes. LSeg [8] trains a CNN model to compute per-pixel image embeddings and use CLIP text embeddings as a classifier. [32] employs contrastive supervision to learn segmentation masks from text.

Concurrently, recent works [6; 33; 27; 20; 9] follow the "frozen CLIP" paradigm for zero-shot segmentation, they first generate a series of mask proposals and then utilize CLIP [28] or ALIGN [17] to classify them. ZSSeg and OVSeg [33; 20] train CLIP adapters to boost performance. FreeSeg[27] simultaneously uses semantic, instance, and panoptic labels and performs fusion training. OpenSeg[9] takes extra images with image-level supervision ( e.g. captions) to scale up training data.

Pre-trained model fine-tuning is widely used for transferring pre-trained knowledge to downstream tasks, e.g. segmentation. However, this strategy may not work well for data-limited tasks like few-shot learning and zero-shot learning due to the daunting overfitting problem. To address this problem and transfer pre-trained knowledge to data-limited tasks, [43; 42; 12; 33; 20; 27] propose to learn text prompts or image prompts by using (a few) annotated images from target dataset. SVF [30] fine-tunes only a few parameters in the pre-trained image encoder to adapt pre-trained knowledge to few-shot segmentation. [38; 37] use contrastive learning to avoid catastrophic forgetting. Alternatively, many outstanding approaches in data-limited tasks [23; 35; 36; 6; 33] choose to freeze the parameters of pre-trained models to maintain the transferability.

Specific to the task of zero-shot/ open-vocabulary segmentation, mainstream approaches use frozen CLIP to avoid overfitting. Recently, MaskCLIP [41] conducts adequate experiments to fine-tune CLIP for open-vocabulary segmentation but has failed. While this attempt is meaningful and appreciated, it is believed that the failure is due to the large domain gap between pixel-level and image-level tasks. This motivates us further research fine-tuning CLIP to be mask-aware (region-level task).

3 Preliminary

Problem Setting. Zero-shot segmentation aims at training a segmentation model capable of segmenting novel objects using text descriptions. Given two category sets C seen and C unseen respectively, where C seen and C unseen are disjoint in terms of object categories ( C seen ∩ C unseen = ∅ ). The model is trained on C seen and directly tested on both C seen and C unseen . Typically, C seen and C unseen are described with semantic words ( e.g. sheep, grass).

Revisiting the "frozen CLIP" paradigm. The "frozen CLIP" approaches [6; 33; 27; 20] execute zero-shot segmentation in two steps: mask proposals generation and mask proposals classification. In the first step, these approaches train a Proposal Generator to generate N class-agnostic mask proposals (denoting as M , M ∈ R N × H × W ) and their corresponding classification scores (denoting as A p , A p ∈ R N ×| C seen | ). MaskFormer [5] and Mask2Former [4] are generally used as the Proposal Generator since the Hungarian matching [19] in the training process makes the mask proposals strongly generalizable. In the second step, N suitable sub-images ( I sub ) are obtained by merging N mask proposals and the input image. I sub is then fed into the CLIP Image Encoder to obtain the image embedding ( E I ). Meanwhile, text embedding ( E T ) is generated by a CLIP Text Encoder. The classification score ( A c , A c ∈ R N × C ) predicted by CLIP is calculated as:

where τ is the temperature hyper-parameter. s c ( E T i , E I ) = E T i · E I | E T i || E I | represents the cosine similarity between E T i and E I . C is the number of classes, with C = | C seen | during training and C = | C seen ∪ C unseen | during inference. Noting that CLIP is frozen when training to avoid overfitting.

repeat

N

attention bias

IP-CLIP Encoder

cls. score

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

layers

Self-distillation Loss

Conv. &  flatten

final proj.

GLYPH(cmap:d835)

. . .

Tranformer layer

Tranformer layer

12 - GLYPH(cmap:d835)

layers

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

. . .

.

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

IoU score  GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc70)GLYPH(cmap:d835)

. . .

Tranformer layer

Tranformer layer

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

. . .

.

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

. . .

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

CLIP (teacher net)

Proposal Generator

IoU score  GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc70)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

mask proposals  M

×

×

(1 + ℎGLYPH(cmap:d835))

×

GLYPH(cmap:d835)

(GLYPH(cmap:d835)

+ℎGLYPH(cmap:d835))

×

GLYPH(cmap:d835)

GLYPH(cmap:d835)

×

GLYPH(cmap:d835)

GLYPH(cmap:d835)

×

GLYPH(cmap:d835)

GLYPH(cmap:d835)

×

1

GLYPH(cmap:d835)

×

(GLYPH(cmap:d835)

+ℎGLYPH(cmap:d835))

Tuned

Frozen

Train & Test

Train only

GT binary masks

×

×

grass

sheep

CLIP Text Encoder

GLYPH(cmap:d835)

×

1

Mask-aware Loss

GLYPH(cmap:d835)

×

GLYPH(cmap:d835)

w.o.

M

GLYPH(cmap:d835)

ℎ

GLYPH(cmap:d835)

GLYPH(cmap:d835)

ℎ

GLYPH(cmap:d835)

To further enhance the reliability of A c , the classification score of the Proposal Generator ( A p ) is ensembled with A c since A p is more reliable on seen classes. This ensemble operation is wildly used in "frozen CLIP" approaches. The pipeline of "frozen CLIP", as well as the merge and ensemble operations, are described in detail in the Appendix.

Although "frozen CLIP" approaches have achieved promising results, it is clear that directly adopting an image-level pre-trained CLIP for proposal classification can be suboptimal. A frozen CLIP usually produces numerous false positives, and the merge operation may destroy the context information of an input image. In view of this, we rethink the paradigm of the frozen CLIP and explore a new solution for proposal classification.

4 Methodology

We introduce Mask-Aware Fine-tuning (MAFT), a method for learning mask-aware CLIP representations. Within MAFT, we first propose the Image-Proposal CLIP Encoder (IP-CLIP Encoder) to handle images with any number of mask proposals simultaneously (Sec. 4.1). Then, mask-aware loss and self-distillation loss are introduced to fine-tune the IP-CLIP Encoder and make it distinguishable for different mask proposals while maintaining transferability (Sec. 4.2). The complete diagram of the MAFT is shown in Fig. 2, we use the ViT-B/16 CLIP model for illustration.

4.1 Image-Proposal CLIP Encoder (IP-CLIP Encoder)

IP-CLIP Encoder aims to process arbitrary numbers of images and mask proposals simultaneously. We draw inspiration from MaskFormer [4; 5], which uses attention-masks in Multihead Attention and provides the flexibility for accepting any number of queries and features of different masked regions. Accordingly, we apply mask proposals as attention-masks in Multihead Attention and designate independent classification queries for each mask proposal.

In the IP-CLIP Encoder shown in Fig. 2, we denote the features propagate between Transformer layers as F i , where i = [1 , 2 ... 12] . We can express F i as F i = [ F i cls ; F i feat ] , ∈ R (1+ hw ) × d , here 1 represents a class-embedding vector ( F i cls ), hw represents the number of the flattened image features ( F i feat ). To obtain the classifications of all mask proposals simultaneously, we repeat F i cls at layer L N times, where N is the number of mask proposals, denoting the repeated class-embedding vectors as F i ∗ cls . We can express the modified features ( F i ∗ ) as F i ∗ = [ F i ∗ cls ; F i feat ] , ∈ R ( N + hw ) × d .

Propagation of F i , where i = [1 , 2 , ...L ] . We consider that CLIP's classification significantly relies on context information. In the first L Transformer layers, the propagation of F i is the same as in standard CLIP. Specifically, F i cls utilizes cross-attention with all pixels within F i feat , effectively retaining the context information.

In the subsequent 12 -L Transformer layers, the propagation of F i ∗ can be partitioned into two parts: the propagation of F i ∗ cls and the propagation of F i feat .

Propagation of F i ∗ cls . We use F i ∗ cls [ n ] and M [ n ] to represent the position n in F i ∗ cls and M , where n = [1 , 2 ...N ] . It is expected F i ∗ cls [ n ] computes Multihead Attention for the positions where M [ n ] = 1 and itself. To achieve this, we construct an attention bias B ∈ R N × ( N + hw ) as follows:

here I( N,N ) denotes N th order identity matrix, Flat ( · ) denotes the flatten operation. ˆ M is an intermediate variable for better representation. Therefore, a masked Multihead Attention is used for propagating F i ∗ cls :

where Que( · ) , Key( · ) , and Val( · ) denote linear projections, d is the hidden dimension of F i ∗ . Notably, We omit the MLP Layer and Layer Normalizations in Transformer layers to simplify the representation in Eq. 3 and Eq. 4.

Propagation of F i feat . A standard Multihead Attention is used for propagating F i feat :

Therefore, for any given mask proposal M [ n ], the corresponding class-embedding F i ∗ cls [ n ] only performs Multihead Attention with F i feat where M [ n ] = 1 and F i ∗ cls [ n ]. The propagation of F i feat remains undisturbed by attention-masks. Compared with the frozen CLIP, IP-CLIP Encoder leverages context information effectively and reduces computational costs.

4.2 Objective

IP-CLIP Encoder with CLIP pre-trained parameters remains challenging in distinguishing different mask proposals, e.g. , when the proposals contain more background regions than foreground objects, IP-CLIP may tend to classify them into the foreground categories. To overcome this limitation, we introduce mask-aware loss and self-distillation loss to fine-tune the IP-CLIP Encoder to be mask-aware without sacrificing transferability.

We conduct the mask-aware loss function ( L ma ) on A c . The goal is to assign high scores to highquality proposals and low scores to low-quality proposals in A c . Concretely, we use the Intersection over Union (IoU) score obtained from ground-truth and align it with the A c to prompt CLIP to become mask-aware. Assuming there are k classes in ground-truth, we can generate k binary maps of ground-truth and calculate the IOU score ( S IoU ) with N mask proposals. We identify a discrepancy between the maximum values of A c and S IoU . The maximum value of A c tends to approach 1, whereas the maximum value of S IoU ranges from 0.75 to 0.99. This inconsistency can hinder the alignment between these two metrics. Therefore, we introduced a min-max normalization technique for S IoU as follows:

Meanwhile, we select k pre-existing classes in A c ( A c select , A c select ∈ R K × N ), and employ SmoothL 1 Loss to align it with S norm IoU . Therefore, L ma can be formulated as follows:

In addition to L ma , we also introduce a self-distillation loss L dis to maintain CLIP's transferability and alleviate overfitting on C seen . Within L dis , we use a frozen CLIP as the teacher net, the IP-CLIP as the student net for self-distillation. The predictions of the frozen CLIP and IP-CLIP are expected to be the same when no mask is included. Denoting the output of the frozen CLIP as A T , and the output of the fine-tuned IP-CLIP without masks as A S . We use SmoothL 1 Loss to minimize the difference as follows:

It is important to note that when processing an image through IP-CLIP without mask proposals, the resulting A S is a matrix with dimensions R C × 1 . Therefore, the final loss function can be formulated as: L = L ma + λ L dis , where we set the constant λ to 1 in our experiments. The mask-aware fine-tuning process is efficient as we only perform a few iterations (less than 1 epoch).

5 Experiments

5.1 Setting

Dataset. We first follow [1; 11; 26; 6; 33] to conduct experiments on three popular zero-shot segmentation benchmarks, Pascal-VOC, COCO-Stuff and ADE20K, to evaluate our method. Then, we evaluate MAFT on the open-vocabulary setting [20; 33], i.e. , training on COCO-Stuff and testing on ADE20K (A-847, A-150), Pascal-Context (PC-459, PC-59), and Pascal-VOC (PAS-20). More details of the dataset settings are provided in the Appendix.

Evaluation Metrics. To quantitatively evaluate the performance, we follow standard practice [1; 31; 10; 25; 6; 33; 27], adopt mean Intersection over Union (mIoU) to respectively evaluate the performance for seen classes (IoU s ) and unseen classes (IoU u ). We also employ the harmonic mean IoU (hIoU) among the seen and unseen classes to measure comprehensive performance.

Methods. Three representative methods are used to verify the generality of MAFT. We unify the three methods into the same framework, with all methods using ResNet101 as the backbone of Proposal Generator and ViT-B/16 CLIP model for a fair comparison.

ZegFormer (CVPR 2022) [6] is an early adopter of the "frozen CLIP" paradigm. It uses MaskFormer as Proposal Generator and employs an ensemble operation to improve the confidence of the results.

ZSSeg (ECCV 2022) [33] uses MaskFormer as Proposal Generator and introduces learnable prompts to improve classification accuracy, which significantly affects the subsequent methods. ZSSeg also adopts a self-training strategy, this strategy is excluded from all methods for a fair comparison.

FreeSeg (CVPR 2023) [27] represents the state-of-the-art method, unifies semantic, instance, and panoptic segmentation tasks and uses annotations from all three tasks for fusion training. We retrain FreeSeg with only the semantic annotations to ensure fairness.

Implementation details. We employ ResNet101 as backbone of the Proposal Generator and ViTB/16 CLIP model. The training process consists of two stages. For the first stage, we follow the official code of ZegFormer, ZSSeg and FreeSeg for model training. For the second stage, we fine-tune IP-CLIP Encoder with MAFT. We take the batch size of 16 and set CLIP input image size to 480 × 480. The optimizer is AdamW with a learning rate of 0.00001 and weight decay of 0.00001. The number of training iterations is set to 100 for Pascal-VOC, 1000 for COCO-Stuff and 5000 for ADE20K.

5.2 Comparisons with State-of-the-art Methods

In this section, three representative methods are used [6; 33; 27] to evaluate the effectiveness of MAFT. We compare three representative methods with MAFT and frozen CLIP. Additionally, we compare the results with previous state-of-the-art methods [31; 1; 10; 25].

Comparisons in the zero-shot setting. In Tab. 1, MAFT remarkably improves the performance. MAFT promotes the state-of-the-art performance by + 8.2% on COCO, + 3.2% on Pascal, and +4.3% on ADE20K in terms of mIoU for unseen classes. It is important to note that the results for seen classes are mainly based on A p rather than A c due to the ensemble operation in [6; 33; 27] (Details in Sec. 3). Therefore, the effect of MAFT on the seen classes is relatively insignificant.

Comparisons without ensemble strategy. To better showcase the performance gains from MAFT, we removed the ensemble operation in [6; 33; 27] and presented the results in Tab. 2. It can be seen that the performance of different methods is significantly improved after applying MAFT. In particular, the state-of-the-art method FreeSeg achieves hIoU improvements of 19.1%, 7.0%, and 8.3% on COCO, VOC2012 and ADE20K datasets.

Comparisons in the open-vocabulary setting. We further evaluated the transferability of MAFT in the open-vocabulary setting [20; 33], using FreeSeg as a baseline for comparison. Results are shown in Tab. 3. Compared with OVSeg [20] and OpenSeg [9], FreeSeg achieves suboptimal performance. However, the proposed MAFT enhances the performance of A-847, A-150, PC-459, PC-59 and PAS-20 by 3.0%,11.2%, 6.4%, 19.1% and 4.4%, and outperforms OpenSeg on all five datasets.

5.3 Ablation Study

We conduct ablation studies on various choices of designs of our MAFT to show their contribution to the final results in Tab. 4. FreeSeg is used as the baseline model and ensemble operation is removed.

Component-wise ablations. To understand the effect of each component in the MAFT, including the IP-CLIP Encoder and the fine-tuning strategy ( L ma , L dis ), we start with standard FreeSeg and

(a) Ablation on components of

MAFT

.

ft denotes the mask-aware fine-tining

(b) Ablation on mask-aware loss

L

ma

.

L

dis is removed.

progressively add each design. (Tab. 4a). FreeSeg uses frozen CLIP and yields inferior performance due to CLIP's mask-unaware property ( 1 st row). Then, IP-CLIP Encoder obtains rich context information and greatly reduces the omputational costs, resulting in an improvement of 7.1% on seen classes and 6.9% on unseen classes. However, mask-aware is not accomplished at this point. Using only L ma for fine-tuning CLIP produces decent performance (the 3 rd result). The introduction of L dis (the 4 th result) maintains transferability while learning mask-aware representations, which further enhances the performance on unseen classes by 2.6%.

Effect of different L ma . Mask-aware Loss L ma is an essential component of MAFT. In Tab. 4b, we investigate how different loss functions ( L 1 , L 2 , SmoothL 1 and KL Loss) impact performance, here we remove L dis for analysis. Results show SmoothL 1 Loss boosts performance on C unseen to 47.1% (+17.8%), KL Loss provides +12.5% improvement on C seen , but only +11.8% on C unseen , manifesting KL Loss compromises the model of transferability comparing with SmoothL 1 Loss.

Training iterations. Tab. 4c examines the impact of training iterations. Increasing the number of iterations leads to gradual improvement of IoU s , but it also results in significant overfitting on unseen classes. Therefore, we choose to fine-tune 1k iterations to maximize the zero-shot ability.

Frozen units in CLIP. We also explore the impact of fine-tuning units within IP-CLIP Encoder. As illustrated in Fig. 2, IP-CLIP Encoder comprises convolution layers (dubbed as conv. ), class embedding ( cls. ), Transformer layers, final projection ( proj. ) and positional embedding ( pos. , not shown in Fig. 2). We start with fine-tuning the entire IP-CLIP Encoder, and then freezing each unit sequentially, as specified in Tab. 4d. We only freeze MLP in the Transformer layers (dubbed as mlp ). Compared with fine-tuning the entire IP-CLIP Encoder, the performance of mIoU u is improved by 5.0% when freezing conv. , cls. , pos. and mlp .

Start mask attention layer . Tab. 4e presents the results of the start mask attention layer ( L ). We observe a significant improvement in the performance of unseen classes by +3.4% when the value of L increases from 0 to 8. This could be attributed to the fact that starting masked Multihead Attention later enables F i ∗ cls to gain more context information. However, the performance significantly drops when L = 10 (from 49.7% to 45.7%), which may be due to the loss of mask-aware property.

5.4 Extending MAFT with SAM

We explore using the Segment Anything Model [18] (SAM) as the proposal generator. We evaluate the performance with SAM-H using an original CLIP (dubbed SAM ) or a mask-aware fine-tuned CLIP (dubbed SAM+MAFT ). In fact, SAM can be seamlessly integrated into our framework as the proposal generator. The results are shown in Tab. 5. Experiments are conducted under both zero-shot setting and open-vocabulary setting.

It can be observed that SAM+MAFT obtains significant improvement over SAM under both settings. Besides, SAM+MAFT also surpasses FreeSeg + MAFT on all benchmarks. Particularly, in the zero-shot setting (Pascal-VOC), SAM+MAFT outperforms FreeSeg + MAFT by 6.8% in

terms of mIoU u . This enhancement can be attributed to the stronger generalization capabilities of SAM for unseen classes.

5.5 Extending MAFT with more Vision-Language Models

In order to demonstrate the efficacy and robustness of MAFT, we conduct experiments using stronger (CLIP-ViT-L) and ResNet-based (CLIP-Res50) Vision-Language Models. The open-vocabulary results are shown in Tab. 6, we also include the results of OVSeg with CLIP-ViT-L for comparison.

CLIP-ViT-L. According to Tab. 6, FreeSeg with a standard CLIP-ViT-L model (dubbed FreeSeg ) still can not achieve satisfactory results. However, by integrating our MAFT (dubbed FreeSeg + MAFT ), the segmentation results are remarkably enhanced, thus establishing new state-of-the-art benchmarks.

CLIP-Res50. Our MAFT can easily adapted into ResNet-based models. Specifically, we modified the AttentionPool2d unit within CLIP-R50 Image Encoder. The mask proposals are introduced as attention bias ( B ) in Multihead Attention, with F cls being repeated N times. Notably in CLIP-R50, F cls is obtained via GlobalAveragePooling performing on F feat . The results are presented in Tab. 6. The performance on all 5 datasets is improved by a large margin. FreeSeg + MAFT with CLIP-R50 achieves competitive results with some CLIP-ViT-B-based methods according to Tab. 3.

5.6 Qualitative Study

Visualizations of typical proposals. Fig. 3 shows frozen CLIP and mask-aware CLIP classifications of typical proposals, including high-quality proposals of foreground ( p 1 , p 4 ), high-quality proposals of background ( p 3 , p 6 ), a proposal with background noise ( p 2 ), and a proposal containing part of the foreground ( p 5 ). The proposal regions are highlighted in green or yellow.

Several observations can be obtained: (1) The frozen CLIP provides good predictions for p 1 and p 4 . (2) The frozen CLIP assigns p 2 as cat and p 5 as horse , with scores even higher than p 1 , p 4 , indicating the frozen CLIP cannot distinguish proposals containing information on the same objects. (3) The frozen CLIP fails to give correct predictions for p 3 and p 6 , which may be due to the lack of context information. (4) Our mask-aware CLIP gives good predictions for high-quality proposals ( p 1 , p 3 , p 4 , p 6 ) and provides suitable predictions for p 2 and p 5 .

Qualitative analysis. We show some visual examples in Fig. 4. Some segmentation results of FreeSeg contain background noise ( e.g. the 1 st & 2 nd row, 3 rd column) or contain only part of the objects ( 3 rd row, 3 rd column). In ADE20K-847 dataset, too many classes may lead to the anticipated results (last row, 3 rd column) with the frozen CLIP. Using a mask-aware CLIP to learn mask-aware representations can significantly improve these segmentation results, as evident from the last column.

More visual samples are shown in the Appendix.

horse: 81.29%

cage: 13.46%

zebra: 0.97%

cow: 0.42%

field: 0.42%

horse: 84.73%

cow: 7.47%

person: 0.98%

elephant: 0.96%

cage: 0.61%

horse: 96.22%

giraffe: 0.53%

zebra: 0.40%

person: 0.27%

clothes: 0.23%

horse: 81.23%

cow: 3.20%

person: 2.04%

textile: 0.79%

metal: 0.60%

wood: 11.34%

ground: 6.41%

wall: 4.90%

hill: 4.87%

structural:4.05%

grass: 79.91%

ground: 9.67%

dirt: 2.25%

plant: 1.62%

cat: 50.73%

mouse: 8.74%

tie: 1.12%

solid: 1.10%

pillow: 1.02%

cat: 69.85%

person: 4.48%

mouse: 2.02%

dog: 1.69%

bear: 0.97%

cat: 51.84%

chair: 13.59%

mouse: 11.10%

couch: 3.87%

dog: 2.42%

chair: 40.11%

couch: 29.00%

cat: 13.28%

furniture: 12.7%

person: 1.17%

chair: 22.02%

cat: 14.95%

couch: 7.39%

curtain: 4.86%

pillow: 4.51%

couch: 70.22%

chair: 22.33%

furniture: 5.97%

bench: 0.16%

person: 0.15%

Top5

predictions by

frozen CLIP

Top5

predictions by

mask-aware CLIP

P1

P2

P3

P4

P5

P6

VOC2012

COCO-Stuff

ADE20K-150

ADE20K-847

Image

GT

FreeSeg

FreeSeg + MAFT

pizza

furniture

building

tree

sky

tower

house

hill

skyscraper

building

antenna

sky

street

mount

tramcar

skyscraper

brick

pottedplant

cardboard

bird

erial,transmittingaerial

sky

sky

skyscraper

building,edifice

skyscraped

skyscraper

mountain.mount

building,edifice

building,edifice

ollor

6 Conclusion

In this paper, we rethink the "frozen CLIP" paradigm in zero-shot segmentation and propose MaskAware Fine-Tune (MAFT) for fine-tuning CLIP. Firstly, IP-CLIP Encoder is proposed to handle images with any number of mask proposals. Then, L ma and L dis are designed for fine-tuning CLIP to be mask-aware without sacrificing its transferability. MAFT is plug-and-play and can be applied to any "frozen CLIP" approach. Extensive experiments well demonstrate the performance of various zero-shot segmentation methods is improved by plugging MAFT.

Limitations. Our MAFT introduces a CLIP fine-tining framework to the research of zero-shot segmentation. However, the classification ability for novel classes is still limited by pre-trained vision-language models. How to further narrow this limitation is our future research focus.

Acknowledgment. This work was supported in part by the National Key R & D Program of China (No.2021ZD0112100).

References

Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems , 32, 2019.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1209-1218, 2018.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764 , 2021.

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34, 2021.

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11583-11592, 2022.

Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision , 111:98-136, 2015.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2112.12143 , 2021.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , pages 540-557. Springer, 2022.

Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia , pages 1921-1929, 2020.

Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia , pages 1921-1929, 2020.

Zixian Guo, Bowen Dong, Zhilong Ji, Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo. Texts as images in prompt tuning for multi-label image recognition. arXiv preprint arXiv:2211.12739 , 2022.

Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Yunchao Wei, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, et al. Global knowledge calibration for fast open-vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 603-612, 2019.

Zilong Huang, Yunchao Wei, Xinggang Wang, Wenyu Liu, Thomas S Huang, and Humphrey Shi. Alignseg: Feature-aligned segmentation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(1):550-557, 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:2210.04150 , 2022.

Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21033-21043, 2022.

Jiaxu Miao, Yunchao Wei, Yu Wu, Chen Liang, Guangrui Li, and Yi Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4133-4143, 2021.

Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6941-6952, 2021.

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 891-898, 2014.

Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. A closer look at self-training for zero-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2693-2702, 2021.

Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. A closer look at self-training for zero-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2693-2702, 2021.

Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary image segmentation. arXiv preprint arXiv:2303.17225 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410 , 2017.

Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few-shot segmentation requires few-parameters fine-tuning. arXiv preprint arXiv:2206.06122 , 2022.

Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8256-8265, 2019.

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. CVPR , 2022.

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX , pages 736-753. Springer, 2022.

Nir Zabari and Yedid Hoshen. Semantic segmentation in-the-wild without seeing any segmentation examples. arXiv preprint arXiv:2112.03185 , 2021.

Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycleconsistent transformer. Advances in Neural Information Processing Systems , 34:21984-21996, 2021.

Gengwei Zhang, Shant Navasardyan, Ling Chen, Yao Zhao, Yunchao Wei, Honghui Shi, et al. Mask matching transformer for few-shot segmentation. Advances in Neural Information Processing Systems , 35:823-836, 2022.

Zekang Zhang, Guangyu Gao, Zhiyuan Fang, Jianbo Jiao, and Yunchao Wei. Mining unseen classes via regional objectness: A simple baseline for incremental segmentation. Advances in Neural Information Processing Systems , 35:24340-24353, 2022.

Zekang Zhang, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, and Yunchao Wei. Coinseg: Contrast inter-and intra-class representations for incremental segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2881-2890, 2017.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 633-641, 2017.

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII , pages 696-712. Springer, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Appendix

Here we introduce technical details of the "frozen CLIP" approaches in Sec. A. The dataset settings are shown in Sec. B. Moreover, we provide additional experiments in Sec. C, and additional qualitative results in Sec. D.

A Technical details of the "frozen CLIP" approaches

×

sheep / grass / cow ...

. . .

CLIP

Image Encoder

text embed

GLYPH(cmap:d835)

. . .

CLIP

Text Encoder

img. embed

GLYPH(cmap:d835)

GLYPH(cmap:d835)

M

Adaptive / predefined

Prompt

M

merge

Trained

Frozen

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

cls. score

GLYPH(cmap:d835)

GLYPH(cmap:d835)

E

E

ensemble

mask proposals  M

GLYPH(cmap:d835)

×

ℎ

×

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

×

GLYPH(cmap:d835)

(GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

+GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835))

×

GLYPH(cmap:d835)

GT binary masks

GLYPH(cmap:d835)

×

ℎ

×

GLYPH(cmap:d835)

grass

sheep

cls. Loss

cls. score

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

+GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835))

×

GLYPH(cmap:d835)

Proposal

Generator

Train only

Test only

Train & Test

seg. Loss

GLYPH(cmap:d835)

Fig. 5 presents an overview of the "frozen CLIP" approach. During training , a standard MaskFormer or Mask2Former is used as Proposal Generator to generate N mask proposals ( M , M ∈ R N × h × w ) and classification score ( A p , A p ∈ R N ×| C seen | ). During testing , the input image is merged with M to obtain N sub-images ( I sub , I sub ∈ R N × ˆ h × ˆ w ). These sub-images are fed into a frozen CLIP to get the CLIP classification score ( A c , A c ∈ R N ×| C seen ∪ C unseen | ). Here C seen and C unseen represent a set of seen classes and unseen classes. An ensemble operation is used to ensemble A p and A c for the final prediction. The merge and the ensemble operations will be introduced in detail in following:

Merge operation. To generate appropriate sub-images based on mask proposals, [6] presents three different merge operations: 1) mask, 2) crop, 3) mask & crop. Through experimentation, they demonstrate that the mask & crop option yields the best results. Fig. 6 provides an example of these operations. It's worth noting that all sub-images are resized to ˆ h × ˆ w , here ˆ h and ˆ w typically take a value of 224, which is the default input size of CLIP Image Encoder. Although acceptable results can be obtained with the merge operation, it involves repeatedly feeding images into CLIP, which leads to significant computational redundancy.

Ensemble operation. Comparatively, A p provides higher confidence classification scores for the seen classes, and A c provides higher confidence classification scores for the unseen classes. Therefore, an ensemble of A p and A c achieves better results. The ensemble operation can be formulated as:

here a geometry mean of A p and A c is calculated (dubbed as ˆ A ), and the contribution of both classification scores is balanced by λ . As per literature [6; 33; 27], λ usually takes values from 0.6 to 0.8. Therefore, the final output ( O , O ∈ R | C seen ∪ C unseen |× h × w ) can be obtained by matrix multiplication: O = ˆ A T · M . With the ensemble operation, the classification results of seen classes primarily depend on A p , whereas the classification results of unseen classes mainly rely on A c .

B Dataset

We follow [1; 11; 26; 6; 33] to conduct experiments on three benchmarks of the popular zero-shot setting, Pascal-VOC, COCO-Stuff and ADE20K, to evaluate the performance of MAFT. Additionally, we evaluate MAFT on the open-vocabulary setting [20; 33], i.e. , training on COCO-Stuff and testing on ADE20K, Pascal-Context, and Pascal-VOC.

M

(1) mask

(2) crop

(3) mask & crop

COCO-Stuff : COCO-Stuff is a large-scale semantic segmentation dataset that includes 171 classes. For the zero-shot setting [6; 33; 27], it is divided into 156 seen classes for training and 15 unseen classes for testing. For the open-vocabulary setting, all 171 classes are used for training.

Pascal-VOC : There are 10582 images for training and 1,449 images for testing. For the zero-shot setting, Pascal-VOC is split into 15 seen classes and 5 unseen classes. For the open-vocabulary setting, all 20 classes are used for evaluation (dubbed as PAS-20).

ADE20K : ADE20K contains 25k images for training and 2k images for validation. For the zero-shot setting, we follow [6] to choose 847 classes present in both training and validation sets, and split them into 572 seen and 275 unseen classes. For the open-vocabulary setting, we use two settings of ADE20K: 150 classes (dubbed as A-150) and 847 classes (dubbed as A-847).

Pascal-Context is an extensive dataset of Pascal-VOC 2010. Two versions are used for open-vocabulary setting, one with 59 frequently used classes (dubbed as PC-59) and another with the whole 459 classes (dubbed as PC-459).

C Additional experiments

C.1 Analysis of the Upper Bound of MAFT

Considering the mask-aware loss may be limited if the quality of proposals is too bad, we conducted an evaluation of the Upper Bound while using Mask2Former as the proposal generator. The results are presented in Tab. 7. Specifically, we replace A c by S IoU (IoU score between binary ground-truth masks and proposals) during inference, and multiply proposals with S IoU to obtain the segmentation result. This result can be regarded as the Upper Bound for the given proposals. Notably, the Upper Bound achieves satisfactory results (77.6 % mIoU), indicating Mask2Former is capable of providing high-quality proposals in most cases. Additionally, there is still a large gap between the current performance and the Upper Bound ( ≈ 30% mIoU), which suggests that our MAFT has enormous potential for improvement, whereas we have achieved state-of-the-art performance.

C.2 Analysis of the Self-Training (ST) strategy

Several previous approaches [25; 33; 41] adopt the Self-Training (ST) strategy to enhance performance. we conduct experiments to investigate the application of ST into our method. Specifically, we use the existing FreeSeg + MAFT model to generate pseudo-labels for unseen classes on the training data, and then re-train FreeSeg with the pseudo-labels. Results are shown in Tab. 8.

The improvement of ST on the unseen category is significant (Pascal: 81.8 % → 86.3%, COCO: 50.4% → 55.2%) in terms of mIoU u . However, it's essential to highlight the applicability of ST is limited by a crucial requirement: unseen classes need to be obtained during training. This requirement poses significant limitations on generalizing ST to various scenarios, e.g. , open-vocabulary settings, since images of unseen classes may not be obtained during training.

D Visualization

We provide more qualitative results, including typical proposals and top-5 A c (Fig. 7), as well as examples of models trained on COCO-Stuff and text on A-847 (Fig. 8), A-150 (Fig. 9), PC-459 (Fig. 10), PC-59 (Fig. 11), Pascal-VOC (Fig. 12), and COCO-Stuff(Fig. 13).

Typical Proposals and Top-5 A c . Fig. 7 shows frozen CLIP and mask-aware CLIP classifications of typical proposals. In the 2 nd column, we provide high-quality proposals of thing classes. Both the frozen CLIP and mask-aware CLIP provide high classification scores for the correct classes. In the 3 rd column, we provide proposals that only contain part of the objects (row 1-3), and proposals containing more than 1 class (row 4). The mask-aware CLIP provides more proper results compared to the frozen CLIP. In the 4 th column, we provide some high-quality background proposals. The frozen CLIP typically gives incorrect predictions, but the mask-aware CLIP assigns high scores for the correct classes.

Qualitative Analysis. Fig. 8,9,10,11,12,13 show segmentation results on Pascal-VOC, COCOStuff, ADE20K. In Pascal-VOC dataset (Fig. 12), which only contains 20 thing classes, the FreeSeg + MAFT model tends to assign background regions to the similar thing classes, e.g. , "train" in row 1, "pottedplant" in row3-4. "boat" in row 8. In A-847, A-150, PC-459, PC-59 and COCO-Stuff datasets, both seen classes and unseen classes exist in the input images, the FreeSeg + MAFT model generates better segmentation results compared to FreeSeg .

diningtable: 26.3%

sheep: 62.57%

cow: 34.17%

horse: 0.45%

vegetable:0.19%

person: 0.17%

cow: 50.62%

dog: 4.98%

food: 4.72%

sheep: 4.07%

person: 3.04%

cow: 66.77%

bear: 5.73%

elephant: 3.99%

sheep: 3.25%

zebra: 2.03%

sand: 93.18%

dirt: 2.97%

sea: 0.73%

ground: 0.53%

gravel: 0.49%

cow: 91.77%

horse: 0.93%

sheep: 0.89%

solid: 0.64%

field: 0.44%

cow: 93.85%

food: 1.06%

person: 0.75%

horse: 0.63%

sheep: 0.62%

board: 3.61%

racket: 3.58%

cloth: 3.09%

kite: 2.46%

glove: 2.17%

laptop: 62.11%

desk: 5.86%

platform: 4.59%

tv: 3.81%

cabinet: 3.72%

laptop: 98.25%

keyboard: 0.9%

platform: 0.13%

desk: 0.09%

mcwave: 0.04%

laptop: 63.19%

tv: 10.23%

keyboard: 3.7%

desk: 3.30%

playingfield: 3.0%

laptop: 46.18%

keyboard: 14.68%

remote: 3.47%

tv: 3.22%

desk: 2.48%

desk: 42.38%

table: 23.77%

furniture: 1.81%

counter: 1.74%

airplane: 95.2%

kite: 1.01%

sky: 0.88%

structural: 0.2%

tie: 0.20%

sky: 93.22%

clouds: 3.96%

ground: 0.27%

airplane: 0.22%

solid: 0.21%

airplane: 75.6%

kite: 1.62%

skis: 1.60%

knife: 1.59%

structural: 1.0%

airplane: 18.2%

metal: 13.53%

structural: 6.0%

ground: 2.65%

person: 2.63%

airplane: 91.6%

waterdrop: 2.2%

ceiling: 0.56%

structural: 0.4%

water: 0.40%

airplane: 95.7%

metal: 0.78%

structural: 0.7%

sky: 0.29%

person: 0.25%

Top5

predictions by

mask-aware CLIP

Top5

predictions by

frozen CLIP

table: 61.81%

diningtable: 32.5%

furniture: 0.82%

counter: 0.76%

desk-stuff: 0.52%

table: 35.12%

diningtable: 27.1%

car: 12.85%

hot dog: 12.6%

plastic:3.44%

bottle: 94.96%

cup: 0.89%

water: 0.77%

plastic: 0.46%

person: 0.26%

bottle: 86.52%

water-other: 1.52%

plastic: 1.20%

straw: 1.04%

cup: 0.87%

hot dog: 81.14%

food-other: 7.73%

sandwich: 3.12%

plastic: 0.93%

water: 4.83%

wall: 4.01%

floor: 3.30%

wall-stone: 2.51%

solid-other: 2.49%

iagng

Hup

cup

Image

GT

FreeSeg

FreeSeg+MAFT

antenna,aerial,transmittingaerial

tree

stalls

carport

equuscaballus

nce,fencing

hgaspump.gasoline pump.petn

nce,

fencing

henhouse

kennel

earth,ground

hay

terfaucet,

watertap.t

land,ground,soil

sidewalk,paveme

centralreservation

sky

sky

sky

streetight,streeta

treetraffilightraffcsignl

m,paltrafficlightrafficignalst

ar,auto,automobile,machine,moto

car,auto,automobilemachine,

arauabilantra

road,route

Curb,curbing,kerb

road,route

po

windowpane,win

pigeonhole

building,edifice

building,edifice

building,edifice

laycase,hwcasevitrin

aneofglass,windowglass

idewalk,pavement

sidewalk,pavement

pedestal,plinth,footstall

sidewalk,pavement

uee

Kxs

MMTTITN

Kxys

lerator, wind generator

sky

chimn

building,edifice

valley,vale

hovelhut,huth,hack,hant

mountain,.mo

e

tree

plantraali

nd.ground,soil

grass

mountain, mount

Image

GT

FreeSeg

FreeSeg+MAFT

ceiling

ceiling

sky

sky

wall

sea

wall

sea

sea

raill

chair

seat

floor

path

step,stair

floor

sky

ky

Image

GT

FreeSeg

FreeSeg+MAFT

flagstaff

ife buoy

curtain

cabine

curtain

wall

windd

binetdoor

windowblind

cabinet

wall

windowb

babycarriageies

person

person

s

flagstaff

grass

grass

sky

sky

REDONIONSALOON

mountain

mountain

pole

elephonebooth

sidewalk

sidewalk

pea

Image

GT

FreeSeg

FreeSeg+MAFT

cabinet

PIRATES

PIRATES

sky

sign

sky

ARONA

AROI

ARONA

ARONA

GRAN HOTEL

GRAN HOTEL

GRAN HOTEL

GRANSOOTEL

building

diningtab

building

bottle

bottle

tree

person

water

person

CUD

boat

chair

chair

Image

GT

FreeSeg

FreeSeg+MAFT

sofa

sheep

sheep

pottec

pottedplant

diningtable

hair

diningtable

bop

person]

Image

GT

FreeSeg

FreeSeg+MAFT

POLO

cell phe

YEARS

Peer reviews:
comments: SUMMARY:
This paper proposes a new topic and method for training a mask-aware CLIP, which could serve as a core component for open-vocabulary segmentation. The designed structure could be used as a flexible plug-in but brings significant improvements for existing methods on various benchmarks.

STRENGTHS:
1. This topic is promising.  Previous methods decouple open-vocabulary segmentation into class-agnostic segmentation and CLIP-guided recognition. However,  most of them fail to use CLIP effectively,  I think training a mask-aware CLIP is an ideal way to deal with this problem.  

2. The model design is reasonable, using a mask2former-style network and tasks the masks to perform masked attention sounds reasonable.

3. The experiment results are great with significant improvement. It would be an ideal solution for various open-vocabulary segmentation tasks incorporating strong class-agnostic segmentation models like SAM. 

4. The paper is clearly presented.

WEAKNESSES:
The experiment setting is unsatisfactory, which only tackles zero-shot semantic segmentation. 
As this topic and idea are good,  I expect the authors to extend the method into open-vocabulary panoptic settings, and use some large datasets for training. Currently, the datasets used are small. it is hard to distill universal knowledge from CLIP.

QUESTIONS:
See weakness

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly discusses how to use pre-trained CLIP to solve zero-shot segmentation task, and proposes a new method called Mask-aware Fine-tuning (MAFT) to address the issue of significant false positives in CLIP's classification of mask proposals. Specifically, the paper introduces an Image-Proposals CLIP Encoder (IP-CLIP Encoder) to handle any number of images and mask proposals simultaneously, and designs mask-aware loss and self-distillation loss to fine-tune the IP-CLIP Encoder, ensuring that CLIP responds to different mask proposals without sacrificing its transferability.\n\nSTRENGTHS:\n1. This paper introduces an Image-Proposals CLIP Encoder (IP-CLIP Encoder) that is sensitive to different mask proposals.\n\n2. This paper includes mask-aware loss and self-distillation loss to fine-tune the IP-CLIP Encoder without sacrificing its transferability.\n\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The ability to handle any number of mask proposals is not unique to this method and has already been a feature of previous methods such as ZegFormer.\n\n2. The main effect of this method comes from the mask-aware loss, which utilizes mask proposals as prior knowledge to obtain more accurate prediction probabilities from the cls score map. Therefore, the effectiveness of this loss function is limited by the quality of the mask proposals, which limits the innovation of this paper.\n\n3. In terms of experiments, it is necessary to conduct experiments on the updated methods such as \"Scaling Open-Vocabulary Image Segmentation with Image-Level Labels\"(ECCV2022) where the performance of it has already surpassed this method on VOC and COCO.\n\n4. Why is Table 2's benchmark experiment conducted under the setting of using only CLIP classifier?\n\nQUESTIONS:\nSame as weakness.\n\nLIMITATIONS:\nThe paper has a description of some limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a mask-aware fine-tuning method to address challenges faced by frozen-CLIP-based zero-shot segmentation methods. It addresses the problem of CLIP being insensitive to different mask proposals and tending to produce similar predictions regardless of the variation in proposals. The proposed IP-CLIP successfully assigns appropriate scores to different proposals, unlike the frozen CLIP that exhibits similar scores. Instead of processing each mask individually, the proposed modified CLIP considers all mask proposals simultaneously, thereby reducing computational costs. The experimental results consistently demonstrate that the proposed method outperforms the baselines by a significant margin.\n\nSTRENGTHS:\n- The proposed method is designed as a plug-and-play approach, making it applicable to any frozen CLIP-based method.\n- The proposed method consistently improves the performance of baseline methods, including SegFormer, ZSSeg, and FreeSeg, by substantial margins, particularly on unseen classes.\n- The method significantly reduces the computational requirements of CLIP in FreeSeg, and the effectiveness of the proposed mask-aware loss and IP-CLIP is demonstrated through ablation studies.\n\nWEAKNESSES:\n- The starting point of the mask attention layer L is determined by a user-defined hyperparameter. The proposed method specifically employs ViT-B/16 as the backbone in the paper. However, if a different backbone is utilized, the selection of this hyperparameter would necessitate a hyperparameter search.\n\n- The notation presented in the paper would be better if it were simplified and clarified.\n\nQUESTIONS:\n- Including experiments with other backbones and proposal generators would enhance the comprehensiveness of the paper\n\nLIMITATIONS:\nThe limitations are briefly discussed in the paper, while the societal impact is not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe general goal of the paper is to leverage CLIP for zero-shot segmentation. For this, in contrast to prior work, an CLIP-inspired IP-CLIP encoder is trained to enable mask-level encodings for segmentation. The core of the approach is the so-called IP-encoder that uses as input a frozen mask generator, as well as two losses, a mask-aware loss and a distillation loss\n\nSTRENGTHS:\nThe general idea is good and on a high-level the components (IP-CLIP encoder, mask aware loss and CLIP distillation) make sense to me\n\nThe reported results are good across three prior baselines and different datasets\n\nWEAKNESSES:\nIn my view the paper is not well written and important details are either not well motivated or even unclear (see below)\n\nThe paper reads to a large extend like an engineering paper with a few changes here and there to adapt to the task as hand. I would assume that to be not so interesting for the majority of NeurIPS\n\nQUESTIONS:\nHere are my main questions about writing\n\n1) A^c is defined to be of dimension NxC (line 119) - with N the number of mask proposals\nin the self-distillation loss however, figure 2 is showing a cx1 dimensional vector?\nin any case I am confused as standard CLIP (here used as teacher) would not generate a NxC dimensional \nmap - and thus A^C_{fro} in equation (7) does not seem to make sense to me - can you please explain?\n\n2) related to that: in figure 2 it seems that the final projection from IP-CLIP encoder is used without biases? (w.o. B) - that seems quite obscure to me actually - what is meant? \n\n3) the mask-aware loss seems sensible on a high-level - but the details are not really motivated well. E.g. the reason for normalizing the IoU scores is unclear (equation 5)  - similarly the reason behind the exact formulation of equation 6 remain unclear. \n\n4) the paper uses L layers prior to condition on the masks, and 12-L after that. While there is an experimental ablation about this why would that make sense intuitively?\n\nif the authors can clarify these points I will consider upgrading my score. \n\ndetail:\n- line 125 \"wildly\" -> \"widely\"\n\n\npost rebuttal\nthanks for addressing my questions - I have upgraded mu review as mentioned in my initial review. Please make sure that the improvements are promised are implemented - thanks\n\nLIMITATIONS:\nok", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}