SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation with Grounding PQ and Grounding mIoU It is worth emphasizing that despite the absence of grounding loss [33, 97, 29, 89] during training, our model exhibits exceptional grounding segmentation capabilities. Tab. 7 presents the grounding PQ and grounding mIoU scores of FC-CLIP, following the evaluation methodology outlined in [29]. In this evaluation, we exclusively employ ground-truth classes as text query inputs to assess the effectiveness of concept grounding. Compared to OpenSeg [29], FC-CLIP achieves a substantial performance improvement, with notable enhancements of  $+11.6$ ,  $+9.1$ ,  $+13.1$ , and  $+17.7$  on A-847, PC-459, A-150, and PC-59, respectively. Even when compared to OpenSeg trained with the Localized Narrative dataset [65], which enables training on a significantly larger vocabulary, FC-CLIP still surpasses it with improvements of  $+8.0$ ,  $+2.2$ ,  $+8.6$  and  $+13.4$  on A-847, PC-459, A-150 and PC-59, respectively, underscoring the grounding proficiency of FC-CLIP.
Table 7: Grounding segmentation performance. The proposed FC-CLIP also demonstrates state-of-the-art performances on grounding segmentation. MV: Mapillary Vistas
method	grounding PQ	grounding mIoU
ADE20K	Cityscapes	MV	A-847	PC-459	A-150	PC-59	PAS-21	PAS-20
ALIGN [40, 29]	-	-	-	17.8	21.8	25.7	34.2	-	-
ALIGN w/ proposal [40, 29]	-	-	-	17.3	19.7	25.3	32.0	-	-
LSeg+ [48, 29]	-	-	-	10.5	17.1	30.8	56.7	-	-
OpenSeg [29]	-	-	-	21.8	32.1	41.0	57.2	-	-
OpenSeg [29] w/ L. Narr	-	-	-	25.4	39.0	45.5	61.5	-	-
FC-CLIP (ours)	38.4	48.1	21.5	33.4	41.2	54.1	74.9	88.7	98.5
Table 8: Ensemble methods comparison with zero-shot evaluation (PQ) on ADE20K. Our method is robust to different ensemble methods (arithmetic and geometric). The results show that it is preferable to bias towards using the in-vocabulary classifier for seen classes and the out-of-vocabulary classifier for unseen classes. Our final setting  $(\alpha = 0.4, \beta = 0.8)$  is labeled in gray
method	arithmetic	geometric
(α = 0.0,β = 0.0)	17.8	17.8
(α = 1.0,β = 1.0)	21.9	21.9
(α = 0.0,β = 1.0)	25.3	25.3
(α = 1.0,β = 0.0)	17.5	17.5
(α = 0.5,β = 0.5)	25.0	25.3
(α = 0.5,β = 0.6)	25.6	26.4
(α = 0.5,β = 0.7)	25.5	26.7
(α = 0.5,β = 0.8)	25.4	26.6
(α = 0.4,β = 0.6)	25.1	25.6
(α = 0.4,β = 0.7)	25.6	26.4
(α = 0.4,β = 0.8)	25.6	26.8
(α = 0.4,β = 0.9)	25.4	25.8
Table 9: Quantitative results of ViT-based CLIP and CNN-based CLIP when input size (denoted as "res") varies for panoptic segmentation on COCO and ADE20K. All results are obtained by applying CLIP directly as a mask classifier with the same mask proposals from ODISE [89]
CLIP backbone	COCO PQ @res	ADE20K PQ @res
224	448	672	896	1120	224	448	672	896	1120
ViT-L/14	19.3	22.5	20.6	18.5	14.9	11.9	13.7	12.6	11.6	9.1
ConvNeXt-L	17.3	23.5	27.0	28.6	29.3	9.3	12.8	14.8	16.0	15.9
Ensemble In-Vocabulary and Out-of-Vocabulary Classifiers In Tab. 8, we present experiments conducted to evaluate the impact of ensemble methods and ensemble parameters on the performance of the in-vocabulary and out-of-vocabulary classifiers. Specifically, we examine two ensemble methods: arithmetic and geometric. The arithmetic method involves a linear combination of the in-vocabulary classifier and the out-of-vocabulary classifier, while the geometric method is defined as shown in Equation (7) of main paper. It is worth noting that FC-CLIP exhibits robustness to different ensemble methods, with both methods displaying a consistent trend within the explored hyper-parameter ranges. However, the geometric ensemble consistently outperforms the arithmetic ensemble by a slight margin. Additionally, we observe that preference is given to values of  $\alpha \leq 0.5$  and  $\beta \geq 0.5$ , which biases the model towards using the in-vocabulary classifier for seen classes and the out-of-vocabulary classifier for unseen classes. We also explore extreme cases, including  $\alpha = 0.0$  and  $\beta = 0.0$  (i.e., exclusively utilizing the in-vocabulary classifier for every class),  $\alpha = 1.0$  and  $\beta = 1.0$  (i.e., exclusively utilizing the out-of-vocabulary classifier for every class),  $\alpha = 0.0$  and  $\beta = 1.0$  (i.e., using the in-vocabulary classifier for seen classes and the out-of-vocabulary classifier for unseen classes), and  $\alpha = 1.0$  and  $\beta = 0.0$  (i.e., using the out-of-vocabulary classifier for seen classes and the in-vocabulary classifier for unseen classes). The results align with our observations that it is preferable to bias towards the in-vocabulary classifier for seen classes and the out-of-vocabulary classifier for unseen classes.
Table 10: Quantitative results of ViT-based CLIP and CNN-based CLIP when input size (denoted as "res") varies for ImageNet-1k classification.
CLIP backbone	Accuracy @res
224	336	448	560	672	784	896
ViT-L/14	75.3	74.3	71.3	67.5	63.1	58.5	53.9
ConvNeXt-L	75.1	77.1	76.8	74.2	69.8	65.6	58.4
Table 11: Open-vocabulary segmentation performance with different backbones and segmentation frameworks. All models are trained on COCO and tested on the other datasets in a zero-shot manner. MV: Mapillary Vistas. \*: kMaX-DeepLab with multi-scale deformable attention [103]
method	backbone	panoptic datasets (PQ)	semantic datasets (mIoU)
ADE	Cityscapes	MV	A-847	PC-459	PC-59	PAS-21
FC-CLIP	R50 [35, 69]	17.9	40.3	15.9	7.1	12.9	50.5	75.9
FC-CLIP	R101 [35, 69]	19.1	40.9	16.7	7.7	12.3	48.9	77.6
FC-CLIP	R50×4 [69]	21.8	42.2	17.4	8.7	13.1	54.0	79.0
FC-CLIP	R50×16 [69]	22.5	42.0	17.8	10.3	15.7	56.4	80.7
FC-CLIP	R50×64 [69]	22.8	42.7	18.2	10.8	16.2	55.7	80.3
FC-CLIP w/ kMaX	ConvNeXt-L [58, 38]	24.5	43.0	17.0	11.4	15.0	57.4	84.7
FC-CLIP w/ kMaX*	ConvNeXt-L [58, 38]	26.4	40.2	17.4	13.6	17.5	57.1	81.2
FC-CLIP	ConvNeXt-L [58, 38]	26.8	44.0	18.2	14.8	18.2	58.4	81.8
Quantitative ViT-based CLIP vs. CNN-based CLIP when Input Size Scales Training our model solely with ViT-based CLIP, without any additional modifications [101, 17, 91, 21], is infeasible. Furthermore, applying ViT to large input sizes is computationally expensive. Therefore, to evaluate the effects of using ViT- or CNN-based CLIP in our framework, we incorporate them into our out-of-vocabulary classifier, which is performed only during inference. To ensure a fair comparison, we use the same mask proposals and disable the geometric ensemble scheme. We also perform experiment on the ImageNet [73] benchmark to ensure a comprehensive comaprisn. In Tab. 9 and Tab. 10, we conduct an ablation study to analyze the impact of different input resolutions for CLIP models. We consider both ViT-based (ViT-L/14) and CNN-based (ConvNeXt-L) CLIP models. By employing them as zero-shot classifiers and varying the input resolutions, we observe that CNN-based CLIP demonstrates superior generalization ability as the input size scales up. Specifically, we observe that the ViT-L/14 CLIP has a higher PQ and Accuracy at a lower resolution (i.e., input size 224), but suffers from a higher resolution, which leads existing two-stage methods [90, 52, 25, 91, 89] to adopt different input resolutions for mask generator and classifier branches. On the contrary, FC-CLIP provides a simple solution by adopting a CNN-based CLIP that generalizes well to different input sizes.
FC-CLIP with Different Backbones and Different Segmentation Frameworks Though we majorly report FC-CLIP results with ConvNeXt-L [58, 69] backbone in Mask2Former [20] framework. We note that FC-CLIP can be easily incorporated with different backbones and segmentation frameworks. Specifically, we experiment FC-CLIP with different backbones (e.g., ResNet [35]) and different segmentation architecture (e.g., kMaX-DeepLab [94]). As shown in Tab. 11, FC-CLIP demonstrates superior performance across different backbones and frameworks.
Visualization We provide visualization on ADE20K val set in Fig. 5.
7 Datasets Information and Licenses
The datasets we used for training and/or testing FC-CLIP are described as follows.
COCO: We train FC-CLIP on COCO data with panoptic annotation [54]. We follow the 2017 splits which include  $118k$  images for train split and  $5k$  images for val split. If not specified, we train our model on the COCO train split and report results on val set of various datasets.
License: Creative Commons Attribution 4.0 License
URL: https://cocodataset.org/#home
Figure 5: Visualization examples of FC-CLIP on ADE20K val set. FC-CLIP is trained on COCO panoptic training set and zero-shot evaluated on ADE20K validation set.
ADE20k: ADE20k [100] covers a wide range of indoor and outdoor scenes, with  $2k$  val images. We evaluate FC-CLIP on both the version with 847 classes (A-847) and the more widely-used version with 150 frequent categories (A-150).
License: Creative Commons BSD-3 License
URL: https://groups.csail.mit.edu/vision/datasets/ADE20K/
Cityscapes: Cityscapes [22] focuses on semantic understanding of urban street scenes. We use the fine data includes 500 images for validation set.
License: This dataset is made freely available to academic and non-academic entities for non-commercial purposes such as academic research, teaching, scientific publications, or personal experimentation.
URL: https://www.cityscapes-dataset.com/
Mapillary Vistas: Mapillary Vistas [64] is a large-scale traffic-related dataset, including  $2k$  images for validation purposes.
License: Creative Commons Attribution NonCommercial Share Alike (CC BY-NC-SA) license
URL: https://www.mapillary.com/dataset/vistas
Pascal Context: Pascal Context [63] covers a wide variety of indoor and outdoor scenes and includes  $5k$  val images. We evaluate FC-CLIP on both its full version (PC-459) with 459 classes and the more common version (PC-59) with 59 classes.
URL: https://www.cs.stanford.edu/~roozbeh/pascal-context/
Pascal VOC: Pascal VOC [27] contains  $1.5k$  val images with 20 foreground classes and 1 background class. Due to the ambiguity in definition of "background", we assign the background class to the pixels predicted as PC-59 categories that are not in Pascal VOC following [29], which leads to PAS-21. We also evaluate the model with background class excluded, which leads to PAS-20.
URL: http://hostrobots.ox.ac.uk/pascal/VOC/