SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first benchmark CoCu with state-of-the-art zero-shot methods [51, 43] and evaluate its effectiveness. Specifically, we follow prior work [43] and pre-train CoCu over the combination of C3, C12, and Y14 datasets. Tab. 1 reports zero-shot segmentation results. Besides GroupViT as the baseline method, we also compare the advanced MaskCLIP [51]), which directly leverages the frozen CLIP model for segmentation prediction without pre-training. In addition, for a comprehensive comparison, we list the performance of other advanced methods including 1) fully-supervised method [40] that provides Oracle's performance, 2) self-supervised methods [20, 6] that pre-train models with unlabeled data and fine-tuning models over segmentation datasets. Detailed implementations of the comparing methods could be found in the appendix.
As shown in Tab. 1, MaskCLIP achieves limited segmentation performance, primarily due to CLIP being trained with image-level supervision and thus falling short in precise pixel-level predictions. GroupViT achieves better performance than MaskCLIP, but still limited by insufficient supervision from language side in pre-training. On the contrary, our CoCu achieves the best segmentation performance over all eight benchmarks, surpassing GroupViT by large margins on average. This indicates the necessity of bridging semantic gaps in language-supervised semantic segmentation and the effectiveness of our design.
Table 1: Performance of different zero-shot methods for semantic segmentation. Abbreviations of benchmarks, from left to right: Pascal VOC [15], Pascal Context [30], Microsoft COCO [5], ImageNet-S [17], Cityscapes [12], and ADE20K [50]. BS denotes pre-training batch size, while LC represents local consistency [2] in mask prediction.  $\dagger$  denotes our re-implementation. CoCu consistently achieves the best performance across all benchmarks.
Method	Pretrain Data	Supervision	LC	BS	Backbone	PVOC	PCON	COCO	IN50	IN300	CITY	ADE	STUF	AVG
DeiT [40]	IN-1K	full	-	ViT-S	53.0	35.9	-	-	-	-	-	-	-
MoCo [20]	IN-1K	self	-	-	34.3	21.3	-	-	-	-	-	-	-
DINO [6]	IN-1K	self	-	-	39.1	20.4	-	-	-	-	-	-	-
MoCo [20]	C12,Y14	self	-	-	36.1	23.0	-	-	-	-	-	-	-
DINO [6]	C12,Y14	self	-	-	37.6	22.8	-	-	-	-	-	-	-
MaskCLIP [51]	-	N.A.	✓	-	ResNet-50	41.5	18.5	10.5	13.8	7.9	18.8	8.3	10.2	15.0
MaskCLIP [51]	-	N.A.	✓	-	ViT-B/16	49.5	21.7	13.6	25.9	11.7	19.8	9.5	12.5	20.5
GroupViT [43]	C3,C12,Y14	text	4,096	ViT-S	52.4	22.3	24.3	44.3	23.5	15.8	10.4	13.0	25.7
GroupViT† [43]	C3,C12,Y14	text	1,024	ViT-S	43.8	19.3	19.6	37.8	17.2	17.2	10.4	13.6	22.4
CoCu (ours)	C3,C12,Y14	text	1,024	ViT-S	49.7	22.8	22.0	46.7	24.7	21.9	12.0	14.9	26.8
GroupViT† [43]	C3,C12,Y14	text	✓	1,024	ViT-S	45.4	19.9	20.3	39.2	17.7	17.6	10.6	13.9	23.1
CoCu (ours)	C3,C12,Y14	text	✓	1,024	ViT-S	51.4	23.6	22.7	48.8	25.5	22.1	12.3	15.2	27.7
We further evaluate the robustness of CoCu with different pre-train data. Specifically, we pre-train GroupViT and CoCu over CC3M and CC12M, respectively. We also sub-sample half of image-text pairs from CC12M (denoted as  $C12^{*}$ ) for pre-training. Tab. 2 shows the experimental results. We can observe consistent yet significant performance gains on eight benchmarks. The improvement by bridging semantic gap is thus robust and not affected by pre-training size.
<sup>3</sup>https://github.com/criteo/autoaiss.git
Table 2: Zero-shot semantic segmentation performance with different pre-training data. CoCu consistently outperforms the baseline method GroupViT across all benchmarks, demonstrating its effectiveness in bridging semantic gaps and achieving significant improvements.
Method	Pretrain Data	PVOC (21)	PCON (60)	COCO (81)	IN50 (51)	IN300 (301)	CITY (19)	ADE (150)	STUF (171)	AVG
GroupViT† [43]	C3	15.5	10.4	6.5	10.2	2.9	8.1	4.4	7.7	8.2
CoCu	C3	30.6 (+15.1%)	13.9 (+3.5%)	10.8 (+4.3%)	19.3 (+9.1%)	7.3 (+4.4%)	8.2 (+0.1%)	6.1 (+1.7%)	8.5 (+0.8%)	13.1 (+4.9%)
GroupViT† [43]	C12*	32.9	13.3	12.9	27.9	12.4	10.7	5.6	8.6	15.5
CoCu	C12*	34.1 (+1.4%)	16.4 (+3.1%)	17.0 (+4.1%)	33.0 (+5.1%)	17.3 (+4.9%)	11.8 (+1.1%)	8.1 (+2.5%)	9.5 (+0.9%)	18.4 (+2.9%)
GroupViT† [43]	C3,C12*	36.5	15.9	16.2	33.5	14.0	12.4	7.0	10.5	18.2
CoCu	C3,C12*	38.1 (+1.6%)	19.2 (+3.3%)	20.1 (+3.9%)	35.8 (+2.3%)	18.9 (+4.9%)	14.7 (+2.3%)	9.5 (+2.5%)	11.4 (+0.9%)	21.0 (+2.8%)
GroupViT† [43]	C12	37.5	18.0	18.3	35.7	16.6	13.5	9.1	13.1	20.2
CoCu	C12	40.9 (+3.4%)	21.2 (+3.2%)	20.3 (+2.0%)	40.0 (+4.3%)	19.4 (+2.8%)	15.0 (+1.5%)	11.1 (+2.0%)	13.6 (+0.5%)	22.7 (+2.5%)
Figure 3: CoCu enhances training convergence. (a) The training loss curves of GroupViT and CoCu demonstrate that CoCu significantly accelerates pre-training convergence. (b) CoCu achieves superior binary segmentation results (second row) compared to GroupViT (first row) for the concept of "grass," which is missing in the caption, using an example image captioned as "a red fox drinking water." Best viewed in color.
4.3 CoCu helps convergence
Loss Curve. In Figure 3 (a), we compare the pre-training loss curves of GroupViT and our proposed method CoCu. We can see that CoCu exhibits a notably faster convergence rate, primarily attributed to the inclusion of curated semantic concepts for each image, resulting in more effective contrastive learning. Additionally, CoCu achieves a lower minimum loss by extracting significantly richer language concepts from image data. This enriches the training process by incorporating more identified image regions and ultimately learning representations that better align with the training data.
Qualitative Comparison. We also present qualitative results that demonstrate the effectiveness of CoCu. In Figure 3 (b), we show binary segmentation results of GroupViT (first row) and CoCu (second row) over an example image with the caption "a red fox drinking water." Our focus is on the concept of "grass," which is missing in the caption. We compare the visual responses of models trained using these two methods at different checkpoints. Both methods improve progressively during training. However, GroupViT fails to correctly localize the region of "grass" due to the lack of direct supervision from the language side. In contrast, CoCu bridges the semantic gap by accurately capturing and localizing "grass," encoding it in representations during pre-training. Consequently, it achieves significantly better segmentation results under zero-shot context.
Figure 4 displays the activation maps of GroupViT and CoCu for different concepts as text inputs that do not appear in the corresponding captions. These maps further demonstrate the superiority of CoCu in language-supervised learning. In all presented images, GroupViT incorrectly activates corresponding regions based on the given text inputs (e.g., activating the "sky" region with a text input of "person" for the first image). In contrast, CoCu enables the segmentor to have the highest activations on visually relevant regions indicated by the text. This suggests that segmentors derived from our method have a better capability to discriminate various visual concepts. More convergence results can be found in the appendix.
Figure 4: Visualization of activation heatmaps. GroupViT fails to activate on corresponding visual regions for concepts not represented in captions, while CoCu exhibits significantly better localization. High activation is shown as red, and low activation is displayed as blue. Best viewed in color.
Table 3: Ablation study of CoCu. We conduct an ablation study on each designed modules. Zero-shot transfer performance on semantic segmentation results are reported, averaged across eight evaluation datasets. "Naïve ranking" refers to solely using cosine similarity between visual and textual representations (encoded by CLIP) as concept-to-image relevancy. "Naïve sampling" denotes selecting textual concepts based solely on relevancy before pre-training.
Model	Expansion	Ranking	Sampling	Average mIoU(%)
lang-driven	vision-driven	naïve text-to-vision-guided	naïve cluster-guided
Baseline [43]	8.2
#1	✓	✓	✓	9.9 (1.7 ↑)
#2	✓	✓	✓	10.3 (2.1 ↑)
#3	✓	✓	✓	12.4 (4.2 ↑)
#4 (Full CoCu)	✓	✓	13.1 (4.9 ↑)
Ablation Study. We further assess the effectiveness of each module in CoCu, which includes vision-driven expansion, text-to-image-guided ranking and cluster-guided sampling. Specifically, we pre-train five models with the combination of these modules or their alternative strategies, namely: 1) Baseline model of GroupViT, which is pre-trained without involving concept curation. 2) Model #1, which utilizes language-driven expansion, naive ranking, and naive sampling (Caption Curation in Sec. 3.2). 3) Model #2, which replaces language-driven expansion with vision-driven expansion on top of Model #1. 4) Model #3, which incorporates text-to-image-guided ranking on top of Model #2. And 4) the full CoCu Model #4, which combines vision-driven expansion, text-to-image-guided ranking, and cluster-guided sampling in pre-training. We report the average segmentation performance of these models across the eight datasets used previously (as shown in Table 1 and Table 2). Detailed illustrations of implementations are provided in appendix.
As Tab. 3 shows, the simplest strategy of language-driven in model #1 improves average mIoU by  $1.7\%$ , which comes from stronger vision-language correlation in pre-training data enhanced by direct text retrieval. Next, replacing direct text retrieval with vision-driven expansion in model #2 brings an additional performance boost, highlighting its significance in capturing unbiased semantics. Furthermore, incorporating text-to-vision-guided ranking in Model #3 brings another noticeable performance gain, underscoring the importance of measuring concept-to-image relevancy. Finally, we upgrade the sampling strategy from the naive one that solely relies on relevancy to cluster-guided sampling, and build model #4 with the full CoCu, which provides more diverse semantic information in each pre-training step, ultimately leading to the best zero-shot transfer performance for semantic segmentation.
Zero-Shot Classification. In addition to its application in zero-shot segmentation, CoCu can also be used to improve zero-shot classification. Following the previous study [43], we evaluate CoCu and compare it with GroupViT on the ImageNet-1K dataset [13]. As shown in Table 4, CoCu exhibits significant performance gains over GroupViT, demonstrating its superiority in bridging semantic gaps across tasks and achieving improved zero-shot classification results.
Table 4: Zero-shot classification on ImageNet-1K. Acc@1 and Acc@5 denote top-1 and top-5 accuracy, respectively.
Method	Pre-training data	Zero-shot
Acc@1(%)	Acc@5(%)
GroupViT	C12	34.9	63.3
CoCu (ours)	C12	38.4 (4.5 ↑)	68.6 (5.3 ↑)
GroupViT	C3,C12,Y14	36.8	66.8
CoCu (ours)	C3,C12,Y14	43.0 (6.2 ↑)	73.7 (6.9 ↑)
In this paper, we identify the issue of semantic gap in language-supervised semantic segmentation and explore how to bridge semantic gaps effectively. To achieve this, we design Concept Curation, a novel pipeline that resolves the issue by three consecutive stages: vision-driven expansion, text-to-vision-guided ranking and cluster-guided sampling. Extensive experiments demonstrate the superiority of our method for boosting language-supervised semantic segmentation across a bundle of pre-training sets and evaluation benchmarks. Looking ahead, we hope to extend the idea of concept curation to other computer vision tasks, including object detection and instance segmentation.
This project is funded by the Ministry of Education Singapore, under the Tier-2 project scheme with a project number MOE-T2EP20220-0003.