SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Methods. Three representative methods are used to verify the generality of MAFT. We unify the three methods into the same framework, with all methods using ResNet101 as the backbone of Proposal Generator and ViT-B/16 CLIP model for a fair comparison.
Table 3: Comparison with state-of-the-art methods on the open-vocabulary setting. mIoU is used to evaluate the performance. * denotes additional training data is used.
A-847	A-150	PC-459	PC-59	PAS-20
SPNet[31]	-	-	-	24.3	18.3
ZSSeg[1]	-	-	-	19.4	38.3
LSeg+[8]	2.5	13.0	5.2	36.0	59.0
OVSeg[20]	7.1	24.8	11.0	53.3	92.6
OpenSeg* [9]	8.8	28.6	12.2	48.2	72.2
FreeSeg[27]	7.1	17.9	6.4	34.4	85.6
FreeSeg +MAFT	10.1 +3.0	29.1 +11.2	12.8 +6.4	53.5 +19.1	90.0 +4.4
- ZegFormer (CVPR 2022) [6] is an early adopter of the "frozen CLIP" paradigm. It uses MaskFormer as Proposal Generator and employs an ensemble operation to improve the confidence of the results.
- ZSSeg (ECCV 2022) [33] uses MaskFormer as Proposal Generator and introduces learnable prompts to improve classification accuracy, which significantly affects the subsequent methods. ZSSeg also adopts a self-training strategy, this strategy is excluded from all methods for a fair comparison.
- FreeSeg (CVPR 2023) [27] represents the state-of-the-art method, unifies semantic, instance, and panoptic segmentation tasks and uses annotations from all three tasks for fusion training. We retrain FreeSeg with only the semantic annotations to ensure fairness.
Implementation details. We employ ResNet101 as backbone of the Proposal Generator and ViT-B/16 CLIP model. The training process consists of two stages. For the first stage, we follow the official code of ZegFormer, ZSSeg and FreeSeg for model training. For the second stage, we fine-tune IP-CLIP Encoder with MAFT. We take the batch size of 16 and set CLIP input image size to  $480 \times 480$ . The optimizer is AdamW with a learning rate of 0.00001 and weight decay of 0.00001. The number of training iterations is set to 100 for Pascal-VOC, 1000 for COCO-Stuff and 5000 for ADE20K.
5.2 Comparisons with State-of-the-art Methods
In this section, three representative methods are used [6; 33; 27] to evaluate the effectiveness of MAFT. We compare three representative methods with MAFT and frozen CLIP. Additionally, we compare the results with previous state-of-the-art methods [31; 1; 10; 25].
Comparisons in the zero-shot setting. In Tab. 1, MAFT remarkably improves the performance. MAFT promotes the state-of-the-art performance by  $+8.2\%$  on COCO,  $+3.2\%$  on Pascal, and  $+4.3\%$  on ADE20K in terms of mIoU for unseen classes. It is important to note that the results for seen classes are mainly based on  $A^p$  rather than  $A^c$  due to the ensemble operation in [6; 33; 27] (Details in Sec. 3). Therefore, the effect of MAFT on the seen classes is relatively insignificant.
Comparisons without ensemble strategy. To better showcase the performance gains from MAFT, we removed the ensemble operation in [6; 33; 27] and presented the results in Tab. 2. It can be seen that the performance of different methods is significantly improved after applying MAFT. In particular, the state-of-the-art method FreeSeg achieves hIoU improvements of  $19.1\%$ ,  $7.0\%$ , and  $8.3\%$  on COCO, VOC2012 and ADE20K datasets.
Comparisons in the open-vocabulary setting. We further evaluated the transferability of MAFT in the open-vocabulary setting [20; 33], using FreeSeg as a baseline for comparison. Results are shown in Tab. 3. Compared with OVSeg [20] and OpenSeg [9], FreeSeg achieves suboptimal performance. However, the proposed MAFT enhances the performance of A-847, A-150, PC-459, PC-59 and PAS-20 by  $3.0\%$ ,  $11.2\%$ ,  $6.4\%$ ,  $19.1\%$  and  $4.4\%$ , and outperforms OpenSeg on all five datasets.
We conduct ablation studies on various choices of designs of our MAFT to show their contribution to the final results in Tab. 4. FreeSeg is used as the baseline model and ensemble operation is removed.
Component-wise ablations. To understand the effect of each component in the MAFT, including the IP-CLIP Encoder and the fine-tuning strategy  $(\mathcal{L}_{ma},\mathcal{L}_{dis})$ , we start with standard FreeSeg and
Table 4: Ablations on COCO dataset. GFLOPs in (a) is used to measure the computation of CLIP Image Encoder. The best results are highlighted with red, and the default settings are highlighted with gray background.
(a) Ablation on components of MAFT.  $f t$  denotes the mask-aware fine-tining
mIoUu	mIoUu	hIoU	GFLOPs
FreeSeg	22.3	29.3	25.3	1127.0
+ IP-CLIP	29.4 +7.1	36.2 +6.9	32.4 +7.1	53.4
+ ft (Lma)	39.9 +17.6	47.1 +17.8	43.1 +17.8	53.4
+ ft (Lma + Ldis)	40.1 +17.8	49.7 +20.4	44.4 +19.0	53.4
(b) Ablation on mask-aware loss  $\mathcal{L}_{ma}$ .  $\mathcal{L}_{dis}$  is removed.
mIoUs	mIoUu	hIoU
L1	38.6 +16.3	45.8 +16.5	41.8 +16.5
L2	40.0 +17.7	45.8 +16.5	42.7 +17.4
SmoothL1	39.9 +17.6	47.1 +17.8	43.1 +17.8
KL	40.9 +18.6	41.8 +12.5	41.3 +16.0
(c) Ablation of the training iterations
mIoU8	mIoU14
500 iters	37.7	47.0
1k iters	40.0	49.7
2k iters	41.1	47.6
3k iters	41.4	46.5
4k iters	41.5	46.1
5k iters	42.0	45.7
(d) Ablation of the frozen units in CLIP
mIoUs	mIoUu	hIoU
None	40.6	44.7	42.5
+cls.	40.7	44.7	42.7
+pos.	40.6	44.9	42.8
+mlp	40.3	48.7	44.1
+conv.	40.0	49.7	44.3
+proj.	40.2	49.1	44.2
(e) Ablation of the start mask attention layer  $L$
mIoUs	mIoUu	hIoU
0	39.3	46.4	42.6
2	39.2	46.4	42.5
4	39.5	46.6	42.6
6	40.0	47.8	43.6
8	40.0	49.7	44.3
10	39.9	45.7	42.6
progressively add each design. (Tab. 4a). FreeSeg uses frozen CLIP and yields inferior performance due to CLIP's mask-unaware property ( $1^{st}$  row). Then, IP-CLIP Encoder obtains rich context information and greatly reduces the computational costs, resulting in an improvement of  $7.1\%$  on seen classes and  $6.9\%$  on unseen classes. However, mask-aware is not accomplished at this point. Using only  $\mathcal{L}_{ma}$  for fine-tuning CLIP produces decent performance (the  $3^{rd}$  result). The introduction of  $\mathcal{L}_{dis}$  (the  $4^{th}$  result) maintains transferability while learning mask-aware representations, which further enhances the performance on unseen classes by  $2.6\%$ .
Effect of different  $\mathcal{L}_{ma}$ . Mask-aware Loss  $\mathcal{L}_{ma}$  is an essential component of MAFT. In Tab. 4b, we investigate how different loss functions (L1, L2, SmoothL1 and KL Loss) impact performance, here we remove  $\mathcal{L}_{dis}$  for analysis. Results show SmoothL1 Loss boosts performance on  $C_{\text {unseen }}$  to  $47.1\% (+17.8\%)$ , KL Loss provides  $+12.5\%$  improvement on  $C_{\text {seen }}$  but only  $+11.8\%$  on  $C_{\text {unseen }}$ , manifesting KL Loss compromises the model of transferability comparing with SmoothL1 Loss.
Training iterations. Tab. 4c examines the impact of training iterations. Increasing the number of iterations leads to gradual improvement of  $\mathrm{IoU}^s$ , but it also results in significant overfitting on unseen classes. Therefore, we choose to fine-tune  $1k$  iterations to maximize the zero-shot ability.
Frozen units in CLIP. We also explore the impact of fine-tuning units within IP-CLIP Encoder. As illustrated in Fig. 2, IP-CLIP Encoder comprises convolution layers (dubbed as conv.), class embedding (cls.), Transformer layers, final projection (proj.) and positional embedding (pos., not shown in Fig. 2). We start with fine-tuning the entire IP-CLIP Encoder, and then freezing each unit sequentially, as specified in Tab. 4d. We only freeze MLP in the Transformer layers (dubbed as mlp). Compared with fine-tuning the entire IP-CLIP Encoder, the performance of  $\mathrm{mIoU}^u$  is improved by  $5.0\%$  when freezing conv.,cls.,pos. and mlp.
Start mask attention layer. Tab. 4e presents the results of the start mask attention layer  $(L)$ . We observe a significant improvement in the performance of unseen classes by  $+3.4\%$  when the value of  $L$  increases from 0 to 8. This could be attributed to the fact that starting masked Multihead Attention later enables  $F_{cls}^{i*}$  to gain more context information. However, the performance significantly drops when  $L = 10$  (from  $49.7\%$  to  $45.7\%$ ), which may be due to the loss of mask-aware property.
5.4 Extending MAFT with SAM
We explore using the Segment Anything Model [18] (SAM) as the proposal generator. We evaluate the performance with SAM-H using an original CLIP (dubbed SAM) or a mask-aware fine-tuned CLIP (dubbed SAM + MAFT). In fact, SAM can be seamlessly integrated into our framework as the proposal generator. The results are shown in Tab. 5. Experiments are conducted under both zero-shot setting and open-vocabulary setting.
It can be observed that SAM + MAFT obtains significant improvement over SAM under both settings. Besides, SAM + MAFT also surpasses FreeSeg + MAFT on all benchmarks. Particularly, in the zero-shot setting (Pascal-VOC), SAM + MAFT outperforms FreeSeg + MAFT by  $6.8\%$  in
Table 5: Comparison with SAM. We use SAM-H as the proposal generator.
(a) Results in zero-shot segmentation.
Pascal-VOC	COCO-Stuff
mIoUs	mIoUu	hIoU	mIoU	mIoUs	mIoUu	hIoU	mIoU
SAM	85.1	86.7	85.9	85.5	43.1	43.3	43.2	42.1
SAM + MAFT	91.0 +5.9	88.6 +1.9	89.8 +3.9	90.4 +4.9	43.4 +0.3	51.5 +8.2	47.1 +3.9	44.1 +2.0
(b) Results in open-vocabulary segmentation.
A-847	A-150	PC-459	PC-59	PAS-20
SAM	9.0	21.3	7.8	33.7	87.5
SAM + MAFT	12.7 +3.7	33.0 +11.7	16.2 +8.4	59.0 +25.3	92.7 +5.2
terms of  $\mathrm{mIoU}^u$ . This enhancement can be attributed to the stronger generalization capabilities of SAM for unseen classes.
5.5 Extending MAFT with more Vision-Language Models
Table 6: Comparison with more Vision-Language Models.
backbone	A-847	A-150	PC-459	PC-59	PAS-20
OVSeg [20]	9.0	29.6	12.4	55.7	94.5
FreeSeg [27]	ViT-L	8.5	21.0	7.6	33.8	86.4
FreeSeg + MAFT	12.1 +3.6	32.0 +11.0	15.7 +8.1	58.5 +24.7	92.1 +5.7
FreeSeg [27]	Res50	5.3	15.5	5.4	28.2	87.1
FreeSeg + MAFT	8.4 +3.1	27.0 +11.5	9.9 +4.5	50.8 +22.6	89.0 +1.9
In order to demonstrate the efficacy and robustness of MAFT, we conduct experiments using stronger (CLIP-ViT-L) and ResNet-based (CLIP-Res50) Vision-Language Models. The open-vocabulary results are shown in Tab. 6, we also include the results of OVSeg with CLIP-ViT-L for comparison.
CLIP-ViT-L. According to Tab. 6, FreeSeg with a standard CLIP-ViT-L model (dubbed FreeSeg) still can not achieve satisfactory results. However, by integrating our MAFT (dubbed FreeSeg + MAFT), the segmentation results are remarkably enhanced, thus establishing new state-of-the-art benchmarks.
CLIP-Res50. Our MAFT can easily adapted into ResNet-based models. Specifically, we modified the AttentionPool2d unit within CLIP-R50 Image Encoder. The mask proposals are introduced as attention bias  $(B)$  in Multihead Attention, with  $F_{cls}$  being repeated N times. Notably in CLIP-R50,  $F_{cls}$  is obtained via GlobalAveragePooling performing on  $F_{feat}$ . The results are presented in Tab. 6. The performance on all 5 datasets is improved by a large margin. FreeSeg + MAFT with CLIP-R50 achieves competitive results with some CLIP-ViT-B-based methods according to Tab. 3.
5.6 Qualitative Study
Visualizations of typical proposals. Fig. 3 shows frozen CLIP and mask-aware CLIP classifications of typical proposals, including high-quality proposals of foreground  $(p_1, p_4)$ , high-quality proposals of background  $(p_3, p_6)$ , a proposal with background noise  $(p_2)$ , and a proposal containing part of the foreground  $(p_5)$ . The proposal regions are highlighted in green or yellow.
Several observations can be obtained: (1) The frozen CLIP provides good predictions for  $p_1$  and  $p_4$ . (2) The frozen CLIP assigns  $p_2$  as cat and  $p_5$  as horse, with scores even higher than  $p_1$ ,  $p_4$ , indicating the frozen CLIP cannot distinguish proposals containing information on the same objects. (3) The frozen CLIP fails to give correct predictions for  $p_3$  and  $p_6$ , which may be due to the lack of context information. (4) Our mask-aware CLIP gives good predictions for high-quality proposals ( $p_1$ ,  $p_3$ ,  $p_4$ ,  $p_6$ ) and provides suitable predictions for  $p_2$  and  $p_5$ .
Qualitative analysis. We show some visual examples in Fig. 4. Some segmentation results of FreeSeg contain background noise (e.g. the  $1^{st}$  &  $2^{nd}$  row,  $3^{rd}$  column) or contain only part of the objects ( $3^{rd}$  row,  $3^{rd}$  column). In ADE20K-847 dataset, too many classes may lead to the anticipated results (last row,  $3^{rd}$  column) with the frozen CLIP. Using a mask-aware CLIP to learn mask-aware representations can significantly improve these segmentation results, as evident from the last column.
More visual samples are shown in the Appendix.
cat:  $50.73\%$   
mouse:  $8.74\%$   
tie:  $1.12\%$   
solid:  $1.10\%$   
pillow:  $1.02\%$
Top5 predictions by mask-aware CLIP
cat:  $69.85\%$  person:  $4.48\%$  mouse:  $2.02\%$  dog:  $1.69\%$  bear:  $0.97\%$
cat:  $51.84\%$   
chair:  $13.59\%$   
mouse:  $11.10\%$   
couch:  $3.87\%$   
dog:  $2.42\%$
chair:  $40.11\%$   
cochuc:  $29.00\%$   
cat:  $13.28\%$   
furniture:  $12.7\%$   
person:  $1.17\%$
chair:  $22.02\%$  coch:  $14.95\%$  couch:  $7.39\%$  ft curtain:  $4.86\%$  b pillow:  $4.51\%$  pe
couch:  $70.22\%$   
chair:  $22.33\%$   
furniture:  $5.97\%$   
bench:  $0.16\%$   
person:  $0.15\%$
Top5 predictions by frozen CLIP
horse: 81.29%  
cage: 13.46%  
zebra: 0.97%  
cow: 0.42%  
field: 0.42%
horse:  $84.73\%$   
cow:  $7.47\%$   
person:  $0.98\%$   
elephant:  $0.96\%$   
cage:  $0.61\%$
horse:  $96.22\%$  horse:  $81.23\%$  giraffe:  $0.53\%$  cow:  $3.20\%$  zebra:  $0.40\%$  person:  $2.04\%$  person:  $0.27\%$  textile:  $0.79\%$  clothes:  $0.23\%$  metal:  $0.60\%$
wood:  $11.34\%$   
ground:  $6.41\%$   
wall:  $4.90\%$   
hill:  $4.87\%$   
structural:  $4.05\%$
grass:  $79.91\%$   
ground:  $6.97\%$   
dirt:  $2.25\%$   
plant:  $1.62\%$   
bush:  $1.12\%$
Figure 3: Visualizations of typical proposals & top  $5A^{c}$  by frozen CLIP and mask-aware CLIP.
Figure 4: Qualitative results. The models are trained with COCO-Stuff and directly tested on VOC2012, COCO, and ADE20K.
In this paper, we rethink the "frozen CLIP" paradigm in zero-shot segmentation and propose Mask-Aware Fine-Tune (MAFT) for fine-tuning CLIP. Firstly, IP-CLIP Encoder is proposed to handle images with any number of mask proposals. Then,  $\mathcal{L}_{ma}$  and  $\mathcal{L}_{dis}$  are designed for fine-tuning CLIP to be mask-aware without sacrificing its transferability. MAFT is plug-and-play and can be applied to any "frozen CLIP" approach. Extensive experiments well demonstrate the performance of various zero-shot segmentation methods is improved by plugging MAFT.
Limitations. Our MAFT introduces a CLIP fine-tuning framework to the research of zero-shot segmentation. However, the classification ability for novel classes is still limited by pre-trained vision-language models. How to further narrow this limitation is our future research focus.
Acknowledgment. This work was supported in part by the National Key R & D Program of China (No.2021ZD0112100).
[32] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. CVPR, 2022.
[33] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 736-753. Springer, 2022.
[34] Nir Zabari and Yedid Hoshen. Semantic segmentation in-the-wild without seeing any segmentation examples. arXiv preprint arXiv:2112.03185, 2021.
[35] Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems, 34:21984-21996, 2021.
[36] Gengwei Zhang, Shant Navasardyan, Ling Chen, Yao Zhao, Yunchao Wei, Honghui Shi, et al. Mask matching transformer for few-shot segmentation. Advances in Neural Information Processing Systems, 35:823-836, 2022.
[37] Zekang Zhang, Guangyu Gao, Zhiyuan Fang, Jianbo Jiao, and Yunchao Wei. Mining unseen classes via regional objectness: A simple baseline for incremental segmentation. Advances in Neural Information Processing Systems, 35:24340-24353, 2022.
[38] Zekang Zhang, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, and Yunchao Wei. Coinseg: Contrast inter-and intra-class representations for incremental segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[39] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2881-2890, 2017.
[40] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 633-641, 2017.
[41] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pages 696-712. Springer, 2022.
[42] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[43] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
Here we introduce technical details of the "frozen CLIP" approaches in Sec. A. The dataset settings are shown in Sec. B. Moreover, we provide additional experiments in Sec. C, and additional qualitative results in Sec. D.
A Technical details of the "frozen CLIP" approaches
Figure 5: Overview of the "decoupling-paradigm".
Fig. 5 presents an overview of the "frozen CLIP" approach. During training, a standard MaskFormer or Mask2Former is used as Proposal Generator to generate  $N$  mask proposals  $(M,M\in \mathbb{R}^{N\times h\times w})$  and classification score  $(A^p,A^p\in \mathbb{R}^{N\times |C_{seen}|})$ . During testing, the input image is merged with  $M$  to obtain  $N$  sub-images  $(I_{sub},I_{sub}\in \mathbb{R}^{N\times \hat{h}\times \hat{w}})$ . These sub-images are fed into a frozen CLIP to get the CLIP classification score  $(A^c,A^c\in \mathbb{R}^{N\times |C_{seen}\cup C_{unseen}|})$ . Here  $C_{seen}$  and  $C_{unseen}$  represent a set of seen classes and unseen classes. An ensemble operation is used to ensemble  $A^p$  and  $A^c$  for the final prediction. The merge and the ensemble operations will be introduced in detail in following:
Merge operation. To generate appropriate sub-images based on mask proposals, [6] presents three different merge operations: 1) mask, 2) crop, 3) mask & crop. Through experimentation, they demonstrate that the mask & crop option yields the best results. Fig. 6 provides an example of these operations. It's worth noting that all sub-images are resized to  $\hat{h} \times \hat{w}$ , here  $\hat{h}$  and  $\hat{w}$  typically take a value of 224, which is the default input size of CLIP Image Encoder. Although acceptable results can be obtained with the merge operation, it involves repeatedly feeding images into CLIP, which leads to significant computational redundancy.
**Ensemble operation.** Comparatively,  $A^p$  provides higher confidence classification scores for the seen classes, and  $A^c$  provides higher confidence classification scores for the unseen classes. Therefore, an ensemble of  $A^p$  and  $A^c$  achieves better results. The ensemble operation can be formulated as:
$$
\hat {A} (c) = \left\{ \begin{array}{l l} A ^ {p} (c) ^ {\lambda} \cdot A ^ {c} (c) ^ {(1 - \lambda)}, & c \in C ^ {\text {s e e n}} \\ A ^ {c} (c) ^ {\lambda} & , c \in C ^ {\text {u n s e e n}} \end{array} \right. \tag {9}
$$
here a geometry mean of  $A^p$  and  $A^c$  is calculated (dubbed as  $\hat{A}$ ), and the contribution of both classification scores is balanced by  $\lambda$ . As per literature [6; 33; 27],  $\lambda$  usually takes values from 0.6 to 0.8. Therefore, the final output  $(O, O \in \mathbb{R}^{[C_{seen} \cup C_{unseen}] \times h \times w})$  can be obtained by matrix multiplication:  $O = \hat{A}^T \cdot M$ . With the ensemble operation, the classification results of seen classes primarily depend on  $A^p$ , whereas the classification results of unseen classes mainly rely on  $A^c$ .
We follow [1; 11; 26; 6; 33] to conduct experiments on three benchmarks of the popular zero-shot setting, Pascal-VOC, COCO-Stuff and ADE20K, to evaluate the performance of MAFT. Additionally, we evaluate MAFT on the open-vocabulary setting [20; 33], i.e., training on COCO-Stuff and testing on ADE20K, Pascal-Context, and Pascal-VOC.
Figure 6: Comparison among three merge operations.
- COCO-Stuff: COCO-Stuff is a large-scale semantic segmentation dataset that includes 171 classes. For the zero-shot setting [6; 33; 27], it is divided into 156 seen classes for training and 15 unseen classes for testing. For the open-vocabulary setting, all 171 classes are used for training.
- Pascal-VOC: There are 10582 images for training and 1,449 images for testing. For the zero-shot setting, Pascal-VOC is split into 15 seen classes and 5 unseen classes. For the open-vocabulary setting, all 20 classes are used for evaluation (dubbed as PAS-20).
- ADE20K: ADE20K contains 25k images for training and 2k images for validation. For the zero-shot setting, we follow [6] to choose 847 classes present in both training and validation sets, and split them into 572 seen and 275 unseen classes. For the open-vocabulary setting, we use two settings of ADE20K: 150 classes (dubbed as A-150) and 847 classes (dubbed as A-847).
- Pascal-Context is an extensive dataset of Pascal-VOC 2010. Two versions are used for open-vocabulary setting, one with 59 frequently used classes (dubbed as PC-59) and another with the whole 459 classes (dubbed as PC-459).
C.1 Analysis of the Upper Bound of MAFT
Table 7: Upper Bound analysis.
COCO-Stuff
mIoUs	mIoUu	hIoU	mIoU
MAFT	43.3	50.4	46.5	43.9
Upper Bound	77.2	82.1	79.6	77.6
Considering the mask-aware loss may be limited if the quality of proposals is too bad, we conducted an evaluation of the Upper Bound while using Mask2Former as the proposal generator. The results are presented in Tab. 7. Specifically, we replace  $A^c$  by  $S_{IoU}$  (IoU score between binary ground-truth masks and proposals) during inference, and multiply proposals with  $S_{IoU}$  to obtain the segmentation result. This result can be regarded as the Upper Bound for the given proposals. Notably, the Upper Bound achieves satisfactory results (77.6% mIoU), indicating Mask2Former is capable of providing high-quality proposals in most cases. Additionally, there is still a large gap between the current performance and the Upper Bound ( $\approx 30\%$  mIoU), which suggests that our MAFT has enormous potential for improvement, whereas we have achieved state-of-the-art performance.
C.2 Analysis of the Self-Training (ST) strategy
Several previous approaches [25; 33; 41] adopt the Self-Training (ST) strategy to enhance performance. we conduct experiments to investigate the application of ST into our method. Specifically, we use the existing FreeSeg  $^+$  MAFT model to generate pseudo-labels for unseen classes on the training data, and then re-train FreeSeg with the pseudo-labels. Results are shown in Tab. 8.
Table 8: Analysis of the self-training (ST) strategy.
Pascal-VOC	COCO-Stuff
mIoUs	mIoUu	hIoU	mIoU	mIoUs	mIoUu	hIoU	mIoU
MAFT	91.4	81.8	86.3	89.0	43.3	50.4	46.5	43.9
MAFT + ST	90.0 -1.4	86.3 +4.5	88.1 +1.8	89.1 +0.1	44.1 +0.8	55.2 +4.8	49.0 +2.5	45.0 +1.1