SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yun Xing $^{1}$  Jian Kang $^{1}$  Aoran Xiao $^{1}$  Jiahao Nie $^{1}$  Ling Shao $^{2}$  Shijian Lu $^{1*}$
<sup>1</sup>Nanyang Technological University
$^{2}$ UCAS-Terminus AI Lab, UCAS, China
Vision-Language Pre-training has demonstrated its remarkable zero-shot recognition ability and potential to learn generalizable visual representations from language supervision. Taking a step ahead, language-supervised semantic segmentation enables spatial localization of textual inputs by learning pixel grouping solely from image-text pairs. Nevertheless, the state-of-the-art suffers from clear semantic gaps between visual and textual modality: plenty of visual concepts appeared in images are missing in their paired captions. Such semantic misalignment circulates in pre-training, leading to inferior zero-shot performance in dense predictions due to insufficient visual concepts captured in textual representations. To close such semantic gap, we propose Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, we establish a concept archive that maintains potential visually-matched concepts with our proposed vision-driven expansion and text-to-vision-guided ranking. Relevant concepts can thus be identified via cluster-guided sampling and fed into pre-training, thereby bridging the gap between visual and textual semantics. Extensive experiments over a broad suite of 8 segmentation benchmarks show that CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin, suggesting the value of bridging semantic gap in pretraining data. Code is available at https://github.com/xing0047/rewrite.
Vision-Language Pre-training [34, 21, 1, 10], which aims to learn visual representations directly from natural language supervision, has endowed existing recognition systems with superior generality and open-vocabulary understanding capability. As a representative, CLIP [34] performs contrastive language-image pre-training on 400M web-crawled image-text pairs, whereby the learnt models may effortlessly transfer to a wide spectrum of classification tasks in a zero-shot manner. Motivated by the breakthrough, recent studies [43, 35, 44] extend the supervision paradigm to semantic segmentation, enabling spatial localization of textual queries in images and pixel grouping with solely supervision from image-text pairs. Distinct from conventional semantic segmentation, the language-supervised paradigm obviates the need for costly manual pixel-level annotation and enables million-level pretraining scale with much less effort.
Despite the progresses [43, 35] in language-supervised semantic segmentation, the pre-training stage still suffers heavily from clear semantic gap between visual and textual modality. In image-text pairs used for pre-training, it is ubiquitous that visual concepts appeared in images are missing in the corresponding textual captions. This happens largely because captions merely describe salient concepts that are worthy of mention [16, 25], while naturally forgo full semantic coverage of images
Figure 1: Cross-modal semantic gap is prevalent in web-crawled image-text pairs. As in (a), the caption text often captures certain salient visual concepts only in the paired image but misses many others (i.e., 'person', 'grass', and 'sky') that are also useful in image-text modeling. Leveraging CLIP [34], more useful visual concepts could be captured via image-to-text retrieval, but the retrieved captions usually suffer from the semantic bias as in (b) (i.e., 'person' recovered but 'grass' and 'sky' still missing). Our proposed Concept Curation (CoCu) bridges the cross-modal semantic gap effectively by vision-driven expansion, text-to-vision-guided ranking and cluster-guided sampling while avoiding the negative effect by semantic bias, as illustrated in (c). Best viewed in color.
(Fig. 1 (a)). Under the presence of clear cross-modal semantic gap in image-text pairs, the pre-training stage of language-supervised segmentation is found to be harder to converge, leading to inferior zero-shot performance on downstream tasks (more details are elaborated in Section 4.3).
This work explores to bridge semantic gaps in language-supervised semantic segmentation. For each image in the pre-training data, the goal is to recover the missing visual concepts in its paired caption for more comprehensive image-text modeling. With the rich vision-language correlations in off-the-shelf foundation models such as CLIP [34], a straight solution is to retrieve the missing concepts from the text captions of pre-training data. However, such retrieved captions suffer from the semantic bias illustrated in Fig. 1 (b) (e.g., "person" recovered but "grass" and "sky" still missing). The root cause lies with the original text captions in foundation model pre-training, which only capture salient concepts appeared in the paired images. Hence, the retrieved captions and concepts still suffer from clear cross-modal semantic gap.
We propose Concept Curation (CoCu), a novel pipeline that side-steps the negative effect by semantic bias $^2$  while exploiting vision-language foundation models for semantic segmentation. CoCu consists of three sequential stages: 1) vision-driven expansion that constructs concept archive via cross-image retrieval; 2) text-to-vision-guided ranking that scores the retrieved concepts according to their assigned relevancies; and 3) cluster-guided sampling that exploits semantic diversity beyond the relevancy scores for concept ranking (Fig. 1 (c)). We perform pre-training from scratch on the segmentation backbone of [43] and evaluate zero-shot transfer over 8 widely adopted segmentation benchmarks. The experiments show that the proposed CoCu improves the baseline as well as the state-of-the-art consistently by large margins, indicating the necessity of closing the semantic gap and the effectiveness of our designs in concept archiving and concept ranking.
In summary, the contributions of this work are three-fold. First, we identify the issue of semantic gap in language-supervised semantic segmentation, and demonstrate the effectiveness of mining more relevant visual concepts for segmentation model pre-training. Second, we design Concept Curation (CoCu), a novel pipeline that constructs concept archives to expand and identify relevant
2to clarify, we refer to semantic gap as a problem in web-crawled image-text pairs and semantic bias as an issue in pre-trained vision-language model.
visual concepts from pre-training data, mitigating the semantic gap between textual and visual modalities effectively. Third, extensive experiments show that the proposed pipeline achieves superior zero-shot transfer performance and outperforms the state-of-the-art across 8 segmentation benchmarks consistently by large margins.
Semantic Segmentation. Partitioning an image into semantic regions, also known as semantic segmentation, has been widely studied due to myriad real-world applications such as video surveillance and autonomous driving. It has been explored along different directions, e.g., by designing different network architectures [29, 9, 11, 42], constructing benchmarks of diverse scenes and categories [12, 50, 5, 17], etc. However, collecting per-category dense annotations for supervised training is notoriously labor-intensive, which impedes the upscaling of semantic vocabulary greatly. Different from conventional semantic segmentation, segmentation from language supervision [43, 28, 7, 32, 35, 44] relieves the burden of mask annotation by leveraging image-text pairs available on the Internet. Beyond that, it can handle arbitrary new semantics thanks to the language supervision paradigm, making it feasible to learn generalizable segmentation models.
Vision-Language Pre-training. Recently, Vision-Language Pre-training has become a predominant trend by learning visual representations from natural language supervision [34, 21, 24, 47, 1, 31, 10, 49]. By matching billion-scale image-text pairs via contrast, the learnt representations can be seamlessly transferred to various downstream classification tasks in a zero-shot manner. As a representative, CLIP [34] can match the performance of supervised baselines on ImageNet [13], meanwhile obtain competitive performance over plethora of downstream tasks without accessing any target data. The same learning paradigm has recently been explored for the task of semantic segmentation by hierarchical grouping [43], supervision mining [35, 44, 7, 32], etc. Nevertheless, state-of-the-art language-supervised segmentation is held back by the cross-modal semantic gap between textual and visual pre-training data. Instead of relaxing the strict one-to-one correspondence in vanilla contrastive learning [35], we mitigate the semantic gap by automated curation of relevant visual concepts through concept expanding and concept ranking.
Open-Vocabulary Semantic Segmentation. Open-Vocabulary Semantic Segmentation has been studied extensively and most existing work can be broadly grouped into three categories. Mix Supervision: the first category follows a zero-shot manner [4, 23] which aims to segment new classes by learning from densely annotated seen classes in the pre-training data. Recently, several studies [18, 22, 46, 19, 26, 52, 45, 48] introduce language supervision to enhance the generality of the learnt zero-shot models. These approaches require no data annotation from new classes, but still rely on dense annotations from seen classes during the pre-training stage. No Supervision: the second category follows a training-free approach [51, 38, 37] which explores the segmentation potential of frozen vision-language models (VLMs) [51, 38, 37] to predict segmentation masks. However, most VLMs are trained with image-level supervision which restricts their capability on pixel/region-level predictions in semantic segmentation. Language Supervision: the third category follows a pure-language-supervision paradigm [43, 28, 32, 35, 7, 44] which aims to learn pixel grouping from solely image-text pairs. Our work follows the third approach. Different from existing studies, we identify the semantic gap in pre-training image and text data and design concept curation that mitigates the semantic gap with clearly improved semantic segmentation performance, more details to be described in the ensuing subsections.
With clear semantic gaps between visual and textual concepts in pre-training data as illustrated in Fig. 1 (a), one naive solution is to employ given image as query to retrieve related captions and derive missing textual concepts as described in Sec. 3.2. However, the naive solution suffers from clear semantic bias as most VLM-retrieved captions contain salient concepts only as illustrated in Fig. 1 (b). We thus further design vision-guided expansion, text-to-image-guided ranking and cluster-guided sampling for better mitigation of the semantic gap as presented in Fig. 1 (c) and Sec. 3.3.
3.1 Revisiting GroupViT
Segmentation Architecture. We use GroupViT [43] as the segmentation backbone for pre-training. Assume a batch of image-text pairs  $\{(x^I, x^T)\}_{i=1}^B$ , where  $x^I$  and  $x^T$  denotes an image and its paired caption, respectively. For the vision flow, a grouping-aware transformer  $\mathcal{F}_s^I$  encodes image  $x^I$  as G segment tokens  $z_{seg}^I = \{z_{seg_g}^I, g = 1, \dots, G\} \in \mathbb{R}^{G \times d}$ , where each segment token  $z_{seg_g}^I \in \mathbb{R}^d$  encodes an arbitrary-shaped region in image  $x^I$ .
Image-Caption Contrastive Loss. To perform pre-training, the segment tokens  $Z_{seg}^{I}$  are merged via average pooling, producing a global representation  $z_{seg}^{I} \in \mathbb{R}^{d}$  that captures all the visual concepts appeared in image  $x^{I}$ . Meanwhile, the paired caption  $x^{T}$  is encoded to  $z^{T} \in \mathbb{R}^{d}$  by a text encoder  $\mathcal{F}^{T}$ . The visual embedding  $z_{seg}^{I}$  and textual embedding  $z^{T}$  are mapped to the same space by separate linear projectors. The segmentator is then learnt from language supervision by the standard contrastive objective InfoNCE [33], which is defined as:
$$
\mathcal {L} _ {I \rightarrow T} = - \frac {1}{B} \sum_ {i = 1} ^ {B} \log \frac {\exp \left(z _ {i} ^ {I} \cdot z _ {i} ^ {T} / \tau\right)}{\sum_ {j = 1} ^ {B} \exp \left(z _ {i} ^ {I} \cdot z _ {j} ^ {T} / \tau\right)} \tag {1}
$$
$$
\mathcal {L} _ {T \rightarrow I} = - \frac {1}{B} \sum_ {i = 1} ^ {B} \log \frac {\exp \left(z _ {i} ^ {T} \cdot z _ {i} ^ {I} / \tau\right)}{\sum_ {j = 1} ^ {B} \exp \left(z _ {i} ^ {T} \cdot z _ {j} ^ {I} / \tau\right)} \tag {2}
$$
where  $\tau$  is a learnable parameter initialized with 0.07 [34] and the  $z^I\cdot z^T$  computes the cross-modal cosine similarity.
Multi-Label Loss. Beyond learning segmentation from raw caption  $x^T$ , GroupViT [43] further introduces  $L$  extra text labels  $\{x^{T_l}, l = 1, \dots, L\}$  by prompting the extracted concepts  $\{c_l, l = 1, \dots, L\}$  with handcrafted templates [34] (e.g., "a photo of a {concept}"). The  $L$  text labels are fed to the same text encoder  $\mathcal{F}^T$  to obtain textual representations of  $\{z^{T_l}, l = 1, \dots, L\}$ . The language supervision by multi-label loss is thus defined as:
$$
\mathcal {L} _ {I \rightarrow \{T _ {l} \} _ {l = 1} ^ {L}} = - \frac {1}{B} \sum_ {i = 1} ^ {B} \log \frac {\sum_ {l = 1} ^ {L} \exp \left(z _ {i} ^ {I} \cdot z _ {i} ^ {T _ {l}} / \tau\right)}{\sum_ {l = 1} ^ {L} \sum_ {j = 1} ^ {B} \exp \left(z _ {i} ^ {I} \cdot z _ {j} ^ {T _ {l}} / \tau\right)} \tag {3}
$$
$$
\mathcal {L} _ {\{T _ {l} \} _ {l = 1} ^ {L} \rightarrow I} = - \frac {1}{L B} \sum_ {l = 1} ^ {L} \sum_ {i = 1} ^ {B} \log \frac {\exp \left(z _ {i} ^ {T _ {l}} \cdot z _ {i} ^ {I} / \tau\right)}{\sum_ {j = 1} ^ {B} \exp \left(z _ {i} ^ {T _ {l}} \cdot z _ {j} ^ {I} / \tau\right)} \tag {4}
$$
The overall training objective of learning segmentation from language supervision in [43] is defined as:
$$
\mathcal {L} = \mathcal {L} _ {I \leftrightarrow T} + \mathcal {L} _ {\{T _ {l} \} _ {l = 1} ^ {L} \leftrightarrow I} \tag {5}
$$
Discussion. We use the exact same training objective as in GroupViT [43] to learn segmentation from language supervision. For each pre-training image  $x^{I}$ , the multi-label loss enhances the contrastive learning [43] with  $L$  extra positive pairs and  $L(B - 1)$  extra negative pairs ( $B$  denotes batch size used in pre-training). However, we highlight that the simple concept prompting does not expand the textual concepts much. The cross-modal semantic gap still exists and circulates in pre-training, which holds back the training convergence and degrades the zero-shot transfer.
3.2 Naive Solution
Caption Curation. The web-crawled image-text pairs are often noisy with imprecise and even irrelevant text descriptions [41]. In addition, many visual concepts (especially those inconspicuous in the background) in images are often missing in the corresponding text descriptions. Both factors lead to clear semantic gaps between web-crawled images and texts. With super-rich image-text correlations in pre-trained VLMs such as CLIP [34], a straight solution, which we term by caption curation, is to apply  $x^{I}$  as query to retrieve  $L$  extra captions  $\{x^{T_l}, l = 1,\dots,L\}$  from pre-training data. The semantic gaps between visual and textual modality could thus be mitigated by identifying relevant concepts from the retrieved captions.
Figure 2: Illustration of vision-driven expansion (above) and text-to-image-guided ranking (below) in CoCu. To compensate for missing semantics, vision-driven expansion establishes an archive of potential matched concepts through image-to-image retrieval, while text-to-vision-guided ranking scores retrieved concepts based on assigned relevancy. The textual concepts can later be identified in pre-training by sampling. In the figure, images with a blue border  $\square$  are retrieved via expanded concepts (marked as blue) using their paired captions, while images with a red border  $\square$  represent images for curation (as anchor). Best viewed in color.
Semantic Bias. Though caption curation expands visual concepts effectively, retrieved captions often suffer from clear semantic bias: VLMs tend to retrieve salient concepts but miss many inconspicuous ones that are also useful in image description. Consequently, visual concepts  $C^I = \{c_l, l = 1, \dots, M^I\}$  appeared in an image  $x^I$  are usually clearly more than textual concepts  $C^T = \{c_l, l = 1, \dots, M^T\}$  extracted from  $\{x^T, x^{T_1}, \dots, x^{T_L}\}$  (i.e.,  $M^I > M^T$ ). The root cause of the semantic bias lies with the loose correlation between the visual and textual pre-training data of VLMs, where most captions just capture partial visual concepts appeared in the paired images [25]. The semantic bias thus impedes convergence and effectiveness of language-supervised training without language supervision available for those non-described image regions.
3.3 Concept Curation
To bridge semantic gaps in image-text pairs, we propose Concept Curation (CoCu) to rewrite caption semantics with the help of a pre-trained vision-language model. Consequently, CoCu finds more concept candidates that are aligned to images and compensate for missing semantics in captions. In pre-training, a multi-modal segmentor matches images and visual-enriched captions by contrastive objectives mentioned in Sec. 3.1, encoding better vision-language alignment in its representations. Details of CoCu are described as below.
Vision-driven Expansion. For an image-text pair  $(x^{I}, x^{T})$ , the goal of the vision-driven expansion is to build an archive of textual concepts  $C^{T} = \{c_{m}, m = 1, \dots, M\}$  that are potentially matched with  $x^{I}$  as illustrated in Fig. 2. Instead of acquiring  $C^{T}$  via direct text retrieval as in caption curation, we resort to cross-image retrieval to achieve the expansion. Concretely,  $N$  image-text pairs  $P = \{(x_{i}^{I}, x_{i}^{T}), i = 1, \dots, N\}$  are automatically selected from the pre-training data, where  $\{x_{i}^{I}, i = 1, \dots, N\}$  are  $N$  captioned images whose visual features match the best with that of  $x^{I}$  (all encoded by CLIP).  $C^{T}$  can thus be derived by extracting textual concepts from the captions
$\{x_i^T, i = 1, \dots, N\}$  of the  $N$  best matched images. Compared with the caption curation that retrieves  $L$  descriptions  $\{x_i^T, i = 1, \dots, L\}$ , vision-driven expansion exploits all visual information in images instead of biased caption texts (mostly describes salient visual concepts only), which helps restore more relevant textual concepts. In addition, it builds an extra image set  $\{x_i^I, i = 1, \dots, N\}$  that plays a pivotal role in the upcoming stages.
Text-to-Vision-Guided Ranking. For each textual concept  $c_{m}$  in the concept archive  $C^T$ , we assign a score  $s_{c_m}$  to represent its relevancy to image  $x^{I}$ . A naive solution to  $p_{m}$  is to compute the cosine similarity between the visual representation of  $x^{I}$  and textual representation of  $t_m$  encoded by CLIP [34], which is simply defined as:
$$
s _ {c _ {m}} ^ {a} = f \left(x ^ {I}, t _ {m}\right) \tag {6}
$$
where  $t_m$  is derived from visual concept  $c_m$  via prompt engineering [34, 43]. Note direct text retrieval could easily get biased towards salient visual concepts here, imposing low relevance scores for other text concepts in concept archive. Beyond  $f(x^I, t_m)$ , we also design a non-biased metric to capture the relevancy between image  $x^I$  and concept  $c_m$ . Specifically, for the  $N$  retrieved image-text pairs  $P$ , we first extract a subset  $P_G$  (blue box/text in Fig. 2 (below)), whose caption of each image-text pair contains the textual concept  $c_m$ . The non-biased metric is thus defined with  $x^I$  (red box) and image-text pairs  $P_G = \{(x_i^I, x_i^T), i = 1, \dots, N'\}$  as follows:
$$
s _ {c _ {m}} ^ {b} = \frac {\left(1 + N ^ {\prime}\right) f \left(t _ {m} , x ^ {I}\right)}{f \left(t _ {m} , x ^ {I}\right) + \sum_ {i = 1} ^ {N ^ {\prime}} f \left(t _ {m} , x _ {i} ^ {I}\right)} \tag {7}
$$
The given term functions as follows: 1) lower relevancy between image  $x^{I}$  and irrelevant concepts (e.g., 'horse' in Fig. 2); 2) enhance relevancy between  $x^{I}$  and inconspicuous concepts (e.g., 'grass'). Instead of computing relevancies of  $x^{I}$  to all textual concepts in a single run, we consider one  $c_{m}$  at a time and measure its relevancy by comparing  $f(x^{I}, t_{m})$  with  $\{f(x_{i}^{I}, t_{m}), i = 1, \dots, N'\}$ . The idea behind this is simple: 1) comparing the responses of images  $\{x^{I}, x_{1}^{I}, \dots, x_{N'}^{I}\}$  to the same textual concept  $c_{m}$ ; 2) high relevancy is given if response of  $x^{I}$  to textual concept  $c_{m}$  is comparably high and vice versa. Take the visual concept 'grass' in Fig. 2 (below) as an example. The  $t_{m}$  ('a photo of grass') causes  $x^{I}$  to rank considerably higher than image captioned with  $c_{m}$  ('grass'). In this case, we should be fairly confident to pass the concept  $c_{m}$  to  $x^{I}$ . In conclusion, the relevancy score is simply defined as:
$$
s _ {c _ {m}} = s _ {c _ {m}} ^ {a} + s _ {c _ {m}} ^ {b} \tag {8}
$$
we perform ranking according to computed relevancies  $\{s_{c_m}, m = 1, \dots, M\}$ , which represents chances identified by later sampling.
Cluster-guided Sampling. The pre-training can thus be empowered by including the expanded and ranked textual concepts which as selected by sampling  $L$  textual concepts according to their computed relevancies as in [43]. However, selection with relevancy alone is often short of semantic diversity in the selected text concepts. Instead of directly selecting  $L$  concepts from the ranked archive  $C^T$ , we partition  $C^T$  into  $L$  semantic clusters based on their textual representations and sample one textual concept from each semantic cluster. The cluster-guided sampling has two clear benefits: 1) it includes more diverse semantics in each single training step; 2) it keeps good consistency with the expression of visual concepts, more details to be discussed in Sec. 4.4 and appendix.
Training Detail. We follow the prior study [43] and conduct pre-training on three publicly available image-text datasets: CC3M (C3) [36], CC12M (C12) [8], YFCC14M (Y14) [39]. For fair comparison, we use the same GroupViT [43] as the visual encoder, which is built upon ViT-S backbone [14, 40] and learnt from scratch. We set the global batch size for contrastive learning as 1,024 and use 4 Tesla V100 GPUs to carry out pre-training for all experiments. Consistent with [43], we set the initial learning rate to 0.0016. The pre-training undergoes 30 epochs, with a linear warmup for the first 2 epochs and a cosine schedule for the remaining epochs.  $L$  is set to 3. In our ablations and discussions, we report the performance of models pre-trained on CC3M.
Implementation of curation. The curation pipeline utilizes clip-retrieval [3], a utility that enables efficient computation of CLIP embeddings and fast indexing for retrieval. We employ the CLIP ViT-B/16 [34] model for image/text inference and concept curation. For efficient semantic searching, we build indexing systems using autofaiss  $^{3}$ . It is worth mentioning that alternative systems can also be used for implementing the curation process.
Evaluation. We benchmark zero-shot transfer performance of CoCu on the validation splits of eight different datasets that cover a myriad of scenes and category sets, including Pascal VOC [15], Pascal Context [30], COCO [27], ImageNet-S-50, ImageNet-S-300 [17], COCO Stuff [5], Cityscapes [12], and ADE20K [50]. For the first five datasets, we follow [43] and evaluate foreground classes by thresholding the similarity between visual and textual embeddings. For other datasets, we evaluate both foreground and background classes. More details are given in the appendix.
4.2 Comparison with the state-of-the-art