SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Siyu Jiao $^{1,2,3}$ ; Yunchao Wei $^{1,2,3}$ ; Yaowei Wang $^{3}$ ; Yao Zhao $^{1,2,3}$ ; Humphrey Shi $^{4,5}$
$^{1}$  Institute of Information Science, Beijing Jiaotong University
2 Peng Cheng Laboratory
$^{3}$  Beijing Key Laboratory of Advanced Information Science and Network
4 Georgia Institute of Technology 5 Pictsart AI Research
jiaosiyu99@bjtu.edu.cn
Recently, pre-trained vision-language models have been increasingly used to tackle the challenging zero-shot segmentation task. Typical solutions follow the paradigm of first generating mask proposals and then adopting CLIP to classify them. To maintain the CLIP's zero-shot transferability, previous practices favour to freeze CLIP during training. However, in the paper, we reveal that CLIP is insensitive to different mask proposals and tends to produce similar predictions for various mask proposals of the same image. This insensitivity results in numerous false positives when classifying mask proposals. This issue mainly relates to the fact that CLIP is trained with image-level supervision. To alleviate this issue, we propose a simple yet effective method, named Mask-aware Fine-tuning (MAFT). Specifically, Image-Proposals CLIP Encoder (IP-CLIP Encoder) is proposed to handle arbitrary numbers of image and mask proposals simultaneously. Then, mask-aware loss and self-distillation loss are designed to fine-tune IP-CLIP Encoder, ensuring CLIP is responsive to different mask proposals while not sacrificing transferability. In this way, mask-aware representations can be easily learned to make the true positives stand out. Notably, our solution can seamlessly plug into most existing methods without introducing any new parameters during the fine-tuning process. We conduct extensive experiments on the popular zero-shot benchmarks. With MAFT, the performance of the state-of-the-art methods is promoted by a large margin:  $50.4\%$ $(+8.2\%)$  on COCO,  $81.8\%$ $(+3.2\%)$  on Pascal-VOC, and  $8.7\%$ $(+4.3\%)$  on ADE20K in terms of mIoU for unseen classes. Code is available at github.com/jiaosiyu1999/MAFT.git.
Semantic segmentation, one of the most widely researched topics in computer vision, has achieved remarkable success [3; 39; 15; 16] with the development of deep learning techniques [14]. However, traditional segmentation models are only capable of segmenting a few predefined categories within a closed vocabulary [7; 2; 22; 21], which is much smaller than the number of categories used by humans to describe the real world. Therefore, zero-shot segmentation [31; 1; 10; 13] is introduced to segment objects using arbitrary categories described by texts.
Recently, large-scale visual-language pre-training models (e.g. CLIP [28] and ALIGN [17]) have shown impressive transferability in recognizing novel categories, leading to their increased adoption
*Work done during an internship at Picsart AI Research (PAIR).
predictions by frozen CLIP
$p_1$ : proposal of swan
predictions by mask-aware CLIP
Figure 1: Comparison between the frozen CLIP and our mask-aware CLIP for proposal classification. Regions of proposals are highlighted with green. The frozen CLIP classifies  $p_1$ ,  $p_2$ , and  $p_3$  as swan class and produces similar predictions, although these proposals contain different regions of the image. After the MAFT, the mask-aware CLIP can produce proper scores for different proposals.
$p_2$ : proposal of river
$p_3$ : proposal of swan & river
for tackling the challenging zero-shot segmentation task [6; 33; 20; 27]. A mainstream solution follows the "frozen CLIP" paradigm, which executes the zero-shot segmentation with two steps: 1) first employing a Proposal Generator to produce class-agnostic mask proposals and 2) then leveraging a frozen pre-trained CLIP to classify each mask proposal via similarity matching in the aligned image-text feature space. While acceptable results are obtained, we reveal that these approaches overlook a crucial issue, i.e. the frozen CLIP is insensitive to different mask proposals and tends to produce similar predictions for various proposals of the same image.
To better illustrate the above-mentioned issue, we show several examples in Fig. 1. We use Mask-Former [5] to generate a series of mask proposals and select three typical ones. When using frozen CLIP for classification, we observe that it correctly classifies the high-quality swan proposal  $p_1$ . However, for the other two proposals  $p_2$  and  $p_3$ , which respectively contain only shape information of swan and both regions of swan and river, the frozen CLIP produces similar predictions compared to  $p_1$ . This is reasonable since CLIP is trained by image-text pairs, making it insensitive to pixel-level information (e.g. background noise), and resulting in numerous false positives. Based on the above observations, we consider that an expected CLIP for zero-shot segmentation task should 1) be sensitive to different mask proposals, 2) not compromise its original transferability on novel classes.
To this end, we introduce a Mask-aware CLIP Fine-tuning method (dubbed MAFT). To make CLIP sensitive to different mask proposals, we devise an Image-Proposals CLIP Encoder (IP-CLIP Encoder), which utilizes mask proposals to perform masked Multihead Attention [5; 4]. This design enables the model to handle arbitrary numbers of images and proposals simultaneously. The mask-aware loss is proposed to minimise the distance between the IoU score of mask proposals and the classification score of IP-CLIP Encoder, prompting IP-CLIP Encoder to differentiate various proposals. Besides, to preserve CLIP's zero-shot transferability, we utilize a frozen CLIP as a teacher network to facilitate fine-tuning. This is achieved by aligning the outputs of the frozen CLIP and IP-CLIP Encoder through self-distillation loss. By performing MAFT, several advantages are provided: 1) Fine-tuning is efficient since only a few mask proposals need to be classified. 2) Compared to pixel-level fine-tuning, mask-aware fine-tuning hardly alters the structure of CLIP itself, preserving its maximum transferability. 3) Mask-aware fine-tuning of CLIP is released from the segmentation module, making it plug-and-play and applicable to any "frozen CLIP" approaches. As shown in Fig. 1, the mask-aware CLIP can well distinguish different proposals and provide proper classification scores for both seen (river) and unseen (swan) classes.
We evaluate our MAFT on three commonly used zero-shot segmentation benchmarks: COCO-Stuff [2], Pascal-VOC [7], and ADE20K [40]. Extensive experiments show that MAFT works well with various zero-shot segmentation methods. In particular, by plugging MAFT, the state-of-the-art approach FreeSeg [27] achieves superior performance on COCO-Stuff  $(42.2\% \rightarrow 50.4\%)$ , Pascal-VOC  $(78.6\% \rightarrow 81.8\%)$  and ADE20K  $(4.4\% \rightarrow 8.7\%)$  in terms of mIoU of unseen classes. Furthermore, we conduct experiments in an open-vocabulary setting, where MAFT enhances the performance of A-847 [40], A-150 [40], PC-459 [24], PC-59 [24] and PAS-20 [7] datasets by  $+3.0\%$ ,  $+11.2\%$ ,  $+6.4\%$ ,  $+19.1\%$  and  $+4.4\%$ , respectively. Notably, our approach outperforms the freezing CLIP counterpart and establishes new state-of-the-art results on all datasets.
Zero-Shot Segmentation [29] is established to break the restriction of categories and perform segmentation on unseen classes. Earlier works SPNet [31] learn a joint pixel and vocabulary concept embedding space, ZS5 [1] utilizes a generative model to generate pixel-level features based on word embeddings of unseen classes, CaGNet [10] incorporates context information for better feature generation. Recent approaches take the advent of large-scale visual-language models (e.g. CLIP [28] and ALIGN [17]) to leverage rich alignment features from image-text pairs. [34] uses CLIP to generate pseudo-labels for single-image segmentation. STRICT [25] obtains pixel-level pseudo-labels from CLIP for unlabeled pixels and proposes a self-training strategy to capture latent information on unseen classes. LSeg [8] trains a CNN model to compute per-pixel image embeddings and use CLIP text embeddings as a classifier. [32] employs contrastive supervision to learn segmentation masks from text.
Concurrently, recent works [6; 33; 27; 20; 9] follow the "frozen CLIP" paradigm for zero-shot segmentation, they first generate a series of mask proposals and then utilize CLIP [28] or ALIGN [17] to classify them. ZSSeg and OVSeg [33; 20] train CLIP adapters to boost performance. FreeSeg[27] simultaneously uses semantic, instance, and panoptic labels and performs fusion training. OpenSeg[9] takes extra images with image-level supervision (e.g. captions) to scale up training data.
Pre-trained model fine-tuning is widely used for transferring pre-trained knowledge to downstream tasks, e.g. segmentation. However, this strategy may not work well for data-limited tasks like few-shot learning and zero-shot learning due to the daunting overfitting problem. To address this problem and transfer pre-trained knowledge to data-limited tasks, [43; 42; 12; 33; 20; 27] propose to learn text prompts or image prompts by using (a few) annotated images from target dataset. SVF [30] fine-tunes only a few parameters in the pre-trained image encoder to adapt pre-trained knowledge to few-shot segmentation. [38; 37] use contrastive learning to avoid catastrophic forgetting. Alternatively, many outstanding approaches in data-limited tasks [23; 35; 36; 6; 33] choose to freeze the parameters of pre-trained models to maintain the transferability.
Specific to the task of zero-shot/ open-vocabulary segmentation, mainstream approaches use frozen CLIP to avoid overfitting. Recently, MaskCLIP [41] conducts adequate experiments to fine-tune CLIP for open-vocabulary segmentation but has failed. While this attempt is meaningful and appreciated, it is believed that the failure is due to the large domain gap between pixel-level and image-level tasks. This motivates us further research fine-tuning CLIP to be mask-aware (region-level task).
Problem Setting. Zero-shot segmentation aims at training a segmentation model capable of segmenting novel objects using text descriptions. Given two category sets  $C_{seen}$  and  $C_{unseen}$  respectively, where  $C_{seen}$  and  $C_{unseen}$  are disjoint in terms of object categories ( $C_{seen} \cap C_{unseen} = \emptyset$ ). The model is trained on  $C_{seen}$  and directly tested on both  $C_{seen}$  and  $C_{unseen}$ . Typically,  $C_{seen}$  and  $C_{unseen}$  are described with semantic words (e.g. sheep, grass).
Revisiting the "frozen CLIP" paradigm. The "frozen CLIP" approaches [6; 33; 27; 20] execute zero-shot segmentation in two steps: mask proposals generation and mask proposals classification. In the first step, these approaches train a Proposal Generator to generate  $N$  class-agnostic mask proposals (denoting as  $M$ ,  $M \in \mathbb{R}^{N \times H \times W}$ ) and their corresponding classification scores (denoting as  $A^p$ ,  $A^p \in \mathbb{R}^{N \times |C_{seen}|}$ ). MaskFormer [5] and Mask2Former [4] are generally used as the Proposal Generator since the Hungarian matching [19] in the training process makes the mask proposals strongly generalizable. In the second step,  $N$  suitable sub-images ( $I_{sub}$ ) are obtained by merging  $N$  mask proposals and the input image.  $I_{sub}$  is then fed into the CLIP Image Encoder to obtain the image embedding  $(E^I)$ . Meanwhile, text embedding  $(E^T)$  is generated by a CLIP Text Encoder. The classification score  $(A^c, A^c \in \mathbb{R}^{N \times C})$  predicted by CLIP is calculated as:
$$
A _ {i} ^ {c} = \operatorname {S o f t m a x} \left(\frac {\exp \left(\frac {1}{\tau} s _ {c} \left(E _ {i} ^ {T} , E ^ {I}\right)\right)}{\sum_ {i = 0} ^ {C} \exp \left(\frac {1}{\tau} s _ {c} \left(E _ {i} ^ {T} , E ^ {I}\right)\right)}, i = [ 1, 2, \dots C ] \right. \tag {1}
$$
where  $\tau$  is the temperature hyper-parameter.  $s_c(E_i^T,E^I) = \frac{E_i^T\cdot E^I}{|E_i^T||E^I|}$  represents the cosine similarity between  $E_{i}^{T}$  and  $E^{I}$ .  $C$  is the number of classes, with  $C = |C_{seen}|$  during training and  $C = |C_{seen}\cup C_{unseen}|$  during inference. Noting that CLIP is frozen when training to avoid overfitting.
Figure 2: Overview of the Mask-Aware Fine-tuning (MAFT). In IP-CLIP Encoder, we modify the CLIP Image Encoder, and apply the mask proposals as attention bias in Multihead Attention from the  $L^{th}$  layer. The final projection unit is an MLP module used for reshaping the channels of  $F_{cls}$ . w.o.  $M$  denotes IP-CLIP Encoder processes image without utilizing mask proposals ( $M$ ). Mask-aware Loss is designed to train CLIP to be mask-aware, while Self-distillation Loss is designed to maintain the transferability. Only the IP-CLIP Encoder is trained (orange part), the Proposal Generator and the CLIP Text Encoder are frozen (blue part).
To further enhance the reliability of  $A^c$ , the classification score of the Proposal Generator  $(A^p)$  is ensembled with  $A^c$  since  $A^p$  is more reliable on seen classes. This ensemble operation is wildly used in "frozen CLIP" approaches. The pipeline of "frozen CLIP", as well as the merge and ensemble operations, are described in detail in the Appendix.
Although "frozen CLIP" approaches have achieved promising results, it is clear that directly adopting an image-level pre-trained CLIP for proposal classification can be suboptimal. A frozen CLIP usually produces numerous false positives, and the merge operation may destroy the context information of an input image. In view of this, we rethink the paradigm of the frozen CLIP and explore a new solution for proposal classification.
We introduce Mask-Aware Fine-tuning (MAFT), a method for learning mask-aware CLIP representations. Within MAFT, we first propose the Image-Proposal CLIP Encoder (IP-CLIP Encoder) to handle images with any number of mask proposals simultaneously (Sec. 4.1). Then, mask-aware loss and self-distillation loss are introduced to fine-tune the IP-CLIP Encoder and make it distinguishable for different mask proposals while maintaining transferability (Sec. 4.2). The complete diagram of the MAFT is shown in Fig. 2, we use the ViT-B/16 CLIP model for illustration.
4.1 Image-Proposal CLIP Encoder (IP-CLIP Encoder)
IP-CLIP Encoder aims to process arbitrary numbers of images and mask proposals simultaneously. We draw inspiration from MaskFormer [4; 5], which uses attention-masks in Multihead Attention and provides the flexibility for accepting any number of queries and features of different masked regions. Accordingly, we apply mask proposals as attention-masks in Multihead Attention and designate independent classification queries for each mask proposal.
In the IP-CLIP Encoder shown in Fig. 2, we denote the features propagate between Transformer layers as  $F^i$ , where  $i = [1,2\dots 12]$ . We can express  $F^i$  as  $F^i = [F_{cls}^i; F_{feat}^i], \in \mathbb{R}^{(1 + hw)\times d}$ , here  $1$  represents a class-embedding vector ( $F_{cls}^i$ ),  $hw$  represents the number of the flattened image features ( $F_{feat}^i$ ). To obtain the classifications of all mask proposals simultaneously, we repeat  $F_{cls}^i$  at layer  $L$ $N$  times, where  $N$  is the number of mask proposals, denoting the repeated class-embedding vectors as  $F_{cls}^{i*}$ . We can express the modified features ( $F^{i*}$ ) as  $F^{i*} = [F_{cls}^{i*}; F_{feat}^{i*}]$ ,  $\in \mathbb{R}^{(N + hw)\times d}$ .
Propagation of  $F^i$ , where  $i = [1,2,\dots L]$ . We consider that CLIP's classification significantly relies on context information. In the first  $L$  Transformer layers, the propagation of  $F^i$  is the same as in standard CLIP. Specifically,  $F_{cls}^i$  utilizes cross-attention with all pixels within  $F_{feat}^i$ , effectively retaining the context information.
In the subsequent  $12 - L$  Transformer layers, the propagation of  $F^{i*}$  can be partitioned into two parts: the propagation of  $F_{cls}^{i*}$  and the propagation of  $F_{feat}^{i}$ .
Propagation of  $F_{cls}^{i*}$ . We use  $F_{cls}^{i*}[n]$  and  $M[n]$  to represent the position  $n$  in  $F_{cls}^{i*}$  and  $M$ , where  $n = [1,2\dots N]$ . It is expected  $F_{cls}^{i*}[n]$  computes Multihead Attention for the positions where  $M[n] = 1$  and itself. To achieve this, we construct an attention bias  $B\in \mathbb{R}^{N\times (N + hw)}$  as follows:
$$
B _ {(i, j)} = \left\{ \begin{array}{c} 0, \text {i f} \hat {M} _ {(i, j)} = 1 \\ - \infty , \text {i f} \hat {M} _ {(i, j)} = 0 \end{array} , \quad \hat {M} = [ I (N, N); \operatorname {F l a t} (M) ] \right. \tag {2}
$$
here  $\mathrm{I}(N,N)$  denotes  $N^{th}$  order identity matrix,  $\mathrm{Flat}(\cdot)$  denotes the flatten operation.  $\hat{M}$  is an intermediate variable for better representation. Therefore, a masked Multihead Attention is used for propagating  $F_{cls}^{i*}$ :
$$
F _ {c l s} ^ {(i + 1) *} = \operatorname {S o f t m a x} \left(\frac {\operatorname {Q u e} \left(F _ {c l s} ^ {i *}\right) \operatorname {K e y} \left(F ^ {i *}\right) ^ {T}}{\sqrt {d}} + B\right) \operatorname {V a l} \left(F ^ {i *}\right) \tag {3}
$$
where  $\mathrm{Que}(\cdot)$ ,  $\mathrm{Key}(\cdot)$ , and  $\mathrm{Val}(\cdot)$  denote linear projections,  $d$  is the hidden dimension of  $F^{i*}$ . Notably, We omit the MLP Layer and Layer Normalizations in Transformer layers to simplify the representation in Eq. 3 and Eq. 4.
Propagation of  $F_{\text{feat}}^i$ . A standard Multihead Attention is used for propagating  $F_{\text{feat}}^i$ :
$$
F _ {f e a t} ^ {i + 1} = \operatorname {S o f t m a x} \left(\frac {\operatorname {Q u e} \left(F _ {f e a t} ^ {i}\right) \operatorname {K e y} \left(F _ {f e a t} ^ {i}\right) ^ {T}}{\sqrt {d}}\right) \operatorname {V a l} \left(F _ {f e a t} ^ {i}\right) \tag {4}
$$
Therefore, for any given mask proposal  $M[n]$ , the corresponding class-embedding  $F_{cls}^{i*}[n]$  only performs Multihead Attention with  $F_{feat}^{i}$  where  $M[n] = 1$  and  $F_{cls}^{i*}[n]$ . The propagation of  $F_{feat}^{i}$  remains undisturbed by attention-masks. Compared with the frozen CLIP, IP-CLIP Encoder leverages context information effectively and reduces computational costs.
4.2 Objective
IP-CLIP Encoder with CLIP pre-trained parameters remains challenging in distinguishing different mask proposals, e.g., when the proposals contain more background regions than foreground objects, IP-CLIP may tend to classify them into the foreground categories. To overcome this limitation, we introduce mask-aware loss and self-distillation loss to fine-tune the IP-CLIP Encoder to be mask-aware without sacrificing transferability.
We conduct the mask-aware loss function  $(\mathcal{L}_{ma})$  on  $A^c$ . The goal is to assign high scores to high-quality proposals and low scores to low-quality proposals in  $A^c$ . Concretely, we use the Intersection over Union (IoU) score obtained from ground-truth and align it with the  $A^c$  to prompt CLIP to become mask-aware. Assuming there are  $k$  classes in ground-truth, we can generate  $k$  binary maps of ground-truth and calculate the IOU score  $(S_{IoU})$  with  $N$  mask proposals. We identify a discrepancy between the maximum values of  $A^c$  and  $S_{IoU}$ . The maximum value of  $A^c$  tends to approach 1, whereas the maximum value of  $S_{IoU}$  ranges from 0.75 to 0.99. This inconsistency can hinder the alignment between these two metrics. Therefore, we introduced a min-max normalization technique for  $S_{IoU}$  as follows:
$$
S _ {I o U} ^ {\text {n o r m}} = \frac {S _ {I o U} - \operatorname* {m i n} \left(S _ {I o U}\right)}{\operatorname* {m a x} \left(S _ {I o U}\right) - \operatorname* {m i n} \left(S _ {I o U}\right)}, S _ {I o U} \in \mathbb {R} ^ {K \times N} \tag {5}
$$
Meanwhile, we select  $k$  pre-existing classes in  $A^c$  ( $A_{select}^c, A_{select}^c \in \mathbb{R}^{K \times N}$ ), and employ SmoothL1 Loss to align it with  $S_{IoU}^{norm}$ . Therefore,  $\mathcal{L}_{ma}$  can be formulated as follows:
$$
\mathcal {L} _ {m a} \left(A _ {\text {s e l e c t}} ^ {c}, S _ {I o U} ^ {\text {n o r m}}\right) = \operatorname {S m o o t h L 1} \left(A _ {\text {s e l e c t}} ^ {c}, S _ {I o U} ^ {\text {n o r m}}\right) \tag {6}
$$
Table 1: Comparison with state-of-the-art methods in zero-shot segmentation. mIoU $^s$  and mIoU $^u$  denote the mIoU(\%) of seen classes and unseen classes.
Method	COCO-Stuff	Pascal-VOC	ADE20K
mIoUs	mIoUu	hIoU	mIoUs	mIoUu	hIoU	mIoUs	mIoUu	hIoU
SPNet[31]	34.6	26.9	30.3	77.8	25.8	38.8	-	-	-
ZS5[1]	34.9	10.6	16.2	78.0	21.2	33.3	-	-	-
CaGNet[10]	35.6	13.4	19.5	78.6	30.3	43.7	-	-	-
STRICT[25]	35.3	30.3	32.6	82.7	35.6	73.3	-	-	-
ZegFormer[6]	36.7	36.2	36.4	90.1	70.6	79.2	17.4	5.1	7.9
ZegFormer +MAFT	36.4 -0.3	40.1 +3.9	38.1 +1.7	91.5 +1.4	80.7 +10.1	85.7 +6.5	16.6 -0.8	7.0 +1.9	9.8 +1.9
ZSSeg[33]	40.4	36.5	38.3	86.6	59.7	69.4	18.0	4.5	7.2
ZSSeg +MAFT	40.6 +0.2	40.1 +3.6	40.3 +2.0	88.4 +1.8	66.2 +6.5	75.7 +6.3	18.9 +0.9	6.7 +2.2	9.9 +2.7
FreeSeg[27]	42.4	42.2	42.3	91.9	78.6	84.7	22.3	4.4	7.3
FreeSeg +MAFT	43.3 +0.9	50.4 +8.2	46.5 +4.2	91.4 -0.5	81.8 +3.2	86.3 +1.6	21.4 -0.9	8.7 +4.3	12.4 +5.1
Table 2: Results on representative methods [6; 33; 27] with/without MAFT. Here we remove the ensemble operation, and only maintain CLIP classifier results.
Method	COCO-Stuff	Pascal-VOC	ADE20K
mIoUs	mIoUu	hIoU	mIoUs	mIoUu	hIoU	mIoUs	mIoUu	hIoU
ZegFormer[6]	18.5	23.0	20.5	81.4	76.8	79.0	5.1	2.6	3.5
ZegFormer+MAFT	35.1 +16.6	31.6 +7.6	33.3 +12.7	87.6 +6.2	79.9 +3.1	83.5 +4.5	15.8 +10.8	7.0 +4.4	9.8 +6.3
ZSSeg[33]	20.6	27.4	23.6	82.0	71.2	76.2	5.9	2.8	3.9
ZSSeg+MAFT	36.1 +15.5	35.9 +8.3	36.0 +12.4	87.1 +5.1	76.1 +4.9	81.2 +5.0	17.2 +11.3	7.2 +4.4	10.2 +6.3
FreeSeg[27]	22.3	29.3	25.3	87.4	74.7	80.5	6.5	2.8	3.9
FreeSeg+MAFT	40.1 +17.8	49.7 +20.4	44.4 +19.1	90.4 +3.0	84.7 +10.0	87.5 +7.0	21.3 +14.8	8.7 +5.9	12.2 +8.3
$$
\operatorname {S m o o t h L 1} (x, y) = \left\{ \begin{array}{l l} 0. 5 \cdot (x - y) ^ {2}, & \text {i f} | x - y | <   1 \\ | x - y | - 0. 5, & \text {o t h e r w i s e} \end{array} \right. \tag {7}
$$
In addition to  $\mathcal{L}_{ma}$ , we also introduce a self-distillation loss  $\mathcal{L}_{dis}$  to maintain CLIP's transferability and alleviate overfitting on  $C_{seen}$ . Within  $\mathcal{L}_{dis}$ , we use a frozen CLIP as the teacher net, the IP-CLIP as the student net for self-distillation. The predictions of the frozen CLIP and IP-CLIP are expected to be the same when no mask is included. Denoting the output of the frozen CLIP as  $A_{T}$ , and the output of the fine-tuned IP-CLIP without masks as  $A_{S}$ . We use SmoothL1 Loss to minimize the difference as follows:
$$
\mathcal {L} _ {\text {d i s}} \left(A _ {S}, A _ {T}\right) = \operatorname {S m o o t h L 1} \left(A _ {S}, A _ {T}\right) \tag {8}
$$
It is important to note that when processing an image through IP-CLIP without mask proposals, the resulting  $A_{S}$  is a matrix with dimensions  $\mathbb{R}^{C\times 1}$ . Therefore, the final loss function can be formulated as:  $\mathcal{L} = \mathcal{L}_{ma} + \lambda \mathcal{L}_{dis}$ , where we set the constant  $\lambda$  to 1 in our experiments. The mask-aware fine-tuning process is efficient as we only perform a few iterations (less than 1 epoch).
5.1 Setting
Dataset. We first follow [1; 11; 26; 6; 33] to conduct experiments on three popular zero-shot segmentation benchmarks, Pascal-VOC, COCO-Stuff and ADE20K, to evaluate our method. Then, we evaluate MAFT on the open-vocabulary setting [20; 33], i.e., training on COCO-Stuff and testing on ADE20K (A-847, A-150), Pascal-Context (PC-459, PC-59), and Pascal-VOC (PAS-20). More details of the dataset settings are provided in the Appendix.
Evaluation Metrics. To quantitatively evaluate the performance, we follow standard practice [1; 31; 10; 25; 6; 33; 27], adopt mean Intersection over Union (mIoU) to respectively evaluate the performance for seen classes  $(\mathrm{IoU}^s)$  and unseen classes  $(\mathrm{IoU}^u)$ . We also employ the harmonic mean IoU (hIoU) among the seen and unseen classes to measure comprehensive performance.