Title: Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: language-supervised semantic segmentation

vision-language pre-training
Abstract: Vision-Language Pre-training has demonstrated its remarkable zero-shot recognition ability and potential to learn generalizable visual representations from languagesupervision. Taking a step ahead, language-supervised semantic segmentation enables spatial localization of textual inputs by learning pixel grouping solely from image-text pairs. Nevertheless, the state-of-the-art suffers from a clear semantic gap between visual and textual modalities: plenty of visual concepts appeared in images are missing in their paired captions. Such semantic misalignment circulates in pre-training, leading to inferior zero-shot performance in dense predictions due to insufficient visual concepts captured in textual representations. To close such semantic gap, we propose Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, we establish a concept archive that maintains potential visually-matched concepts with our proposed vision-driven expansion and text-to-vision-guided ranking. Relevant concepts can thus be identified via cluster-guided sampling and fed into pre-training, thereby bridging the gap between visual and textual semantics. Extensive experiments over a broad suite of 8 segmentation benchmarks show that CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin, suggesting the value of closing semantic gap in pre-training data.

Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation

Abstract

Vision-Language Pre-training has demonstrated its remarkable zero-shot recognition ability and potential to learn generalizable visual representations from language supervision. Taking a step ahead, language-supervised semantic segmentation enables spatial localization of textual inputs by learning pixel grouping solely from image-text pairs. Nevertheless, the state-of-the-art suffers from clear semantic gaps between visual and textual modality: plenty of visual concepts appeared in images are missing in their paired captions. Such semantic misalignment circulates in pre-training, leading to inferior zero-shot performance in dense predictions due to insufficient visual concepts captured in textual representations. To close such semantic gap , we propose Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, we establish a concept archive that maintains potential visually-matched concepts with our proposed vision-driven expansion and text-to-vision-guided ranking . Relevant concepts can thus be identified via cluster-guided sampling and fed into pre-training, thereby bridging the gap between visual and textual semantics. Extensive experiments over a broad suite of 8 segmentation benchmarks show that CoCu achieves superb zeroshot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin, suggesting the value of bridging semantic gap in pretraining data. Code is available at https://github.com/xing0047/rewrite .

1 Introduction

Vision-Language Pre-training [34, 21, 1, 10], which aims to learn visual representations directly from natural language supervision, has endowed existing recognition systems with superior generality and open-vocabulary understanding capability. As a representative, CLIP [34] performs contrastive language-image pre-training on 400M web-crawled image-text pairs, whereby the learnt models may effortlessly transfer to a wide spectrum of classification tasks in a zero-shot manner. Motivated by the breakthrough, recent studies [43, 35, 44] extend the supervision paradigm to semantic segmentation, enabling spatial localization of textual queries in images and pixel grouping with solely supervision from image-text pairs. Distinct from conventional semantic segmentation, the language-supervised paradigm obviates the need for costly manual pixel-level annotation and enables million-level pretraining scale with much less effort.

Despite the progresses [43, 35] in language-supervised semantic segmentation, the pre-training stage still suffers heavily from clear semantic gap between visual and textual modality. In image-text pairs used for pre-training, it is ubiquitous that visual concepts appeared in images are missing in the corresponding textual captions. This happens largely because captions merely describe salient concepts that are worthy of mention [16, 25], while naturally forgo full semantic coverage of images

∗

alpacas

grass

man

sky

person

man

grass

sky

lawn

horse

camel

Image

Prompted Texts

field

alpacas

llama

Image

Caption

Another one

of my male

alpacas.

(c) Overview of Concept Curation

(a) Semantic Gap

(b) Semantic Bias

Image

A gardener

who is at

work.

Standing with

Person

pictured while

travelling.

…

Retrieved Captions

vision-driven

expansion

text-to-vision-

guided ranking

cluster-guided

sampling

Concept Archive

(Ordered)

person

man

grass

sky

lawn

horse

camel

field

alpacas

llama

Prompt

Vision

Encoder

Text

Encoder

Baseline

Ours

Supervision Signal

Encoding

Text Token

Group Token

alpacas

alpacas

person

sky

grass

alpacas

person

sky

grass

llama

person

gardener

image-to-text

retrieval

semantic

bias

alpacas

(Fig. 1 (a)). Under the presence of clear cross-modal semantic gap in image-text pairs, the pre-training stage of language-supervised segmentation is found to be harder to converge, leading to inferior zero-shot performance on downstream tasks (more details are elaborated in Section 4.3).

This work explores to bridge semantic gaps in language-supervised semantic segmentation. For each image in the pre-training data, the goal is to recover the missing visual concepts in its paired caption for more comprehensive image-text modeling. With the rich vision-language correlations in off-the-shelf foundation models such as CLIP [34], a straight solution is to retrieve the missing concepts from the text captions of pre-training data. However, such retrieved captions suffer from the semantic bias illustrated in Fig. 1 (b) (e.g., 'person' recovered but 'grass' and 'sky' still missing). The root cause lies with the original text captions in foundation model pre-training, which only capture salient concepts appeared in the paired images. Hence, the retrieved captions and concepts still suffer from clear cross-modal semantic gap .

We propose Concept Curation (CoCu), a novel pipeline that side-steps the negative effect by semantic bias 2 while exploiting vision-language foundation models for semantic segmentation. CoCu consists of three sequential stages: 1) vision-driven expansion that constructs concept archive via crossimage retrieval; 2) text-to-vision-guided ranking that scores the retrieved concepts according to their assigned relevancies; and 3) cluster-guided sampling that exploits semantic diversity beyond the relevancy scores for concept ranking (Fig. 1 (c)). We perform pre-training from scratch on the segmentation backbone of [43] and evaluate zero-shot transfer over 8 widely adopted segmentation benchmarks. The experiments show that the proposed CoCu improves the baseline as well as the state-of-the-art consistently by large margins, indicating the necessity of closing the semantic gap and the effectiveness of our designs in concept archiving and concept ranking.

In summary, the contributions of this work are three-fold. First , we identify the issue of semantic gap in language-supervised semantic segmentation, and demonstrate the effectiveness of mining more relevant visual concepts for segmentation model pre-training. Second , we design Concept Curation (CoCu), a novel pipeline that constructs concept archives to expand and identify relevant

visual concepts from pre-training data, mitigating the semantic gap between textual and visual modalities effectly. Third , extensive experiments show that the proposed pipeline achieves superior zero-shot transfer performance and outperforms the state-of-the-art across 8 segmentation benchmarks consistently by large margins.

2 Related Works

Semantic Segmentation. Partitioning an image into semantic regions, also known as semantic segmentation, has been widely studied due to myriad real-world applications such as video surveillance and autonomous driving. It has been explored along different directions, e.g., by designing different network architectures [29, 9, 11, 42], constructing benchmarks of diverse scenes and categories [12, 50, 5, 17], etc. However, collecting per-category dense annotations for supervised training is notoriously labor-intensive, which impedes the upscaling of semantic vocabulary greatly. Different from conventional semantic segmentation, segmentation from language supervision [43, 28, 7, 32, 35, 44] relieves the burden of mask annotation by leveraging image-text pairs available on the Internet. Beyond that, it can handle arbitrary new semantics thanks to the language supervision paradigm, making it feasible to learn generalizable segmentation models.

Vision-Language Pre-training. Recently, Vision-Language Pre-training has become a predominant trend by learning visual representations from natural language supervision [34, 21, 24, 47, 1, 31, 10, 49]. By matching billion-scale image-text pairs via contrast, the learnt representations can be seamlessly transferred to various downstream classification tasks in a zero-shot manner. As a representative, CLIP [34] can match the performance of supervised baselines on ImageNet [13], meanwhile obtain competitive performance over plethora of downstream tasks without accessing any target data. The same learning paradigm has recently been explored for the task of semantic segmentation by hierarchical grouping [43], supervision mining [35, 44, 7, 32], etc. Nevertheless, state-of-the-art language-supervised segmentation is held back by the cross-modal semantic gap between textual and visual pre-training data. Instead of relaxing the strict one-to-one correspondence in vanilla contrastive learning [35], we mitigate the semantic gap by automated curation of relevant visual concepts through concept expanding and concept ranking.

Open-Vocabulary Semantic Segmentation. Open-Vocabulary Semantic Segmentation has been studied extensively and most existing work can be broadly grouped into three categories. Mix Supervision: the first category follows a zero-shot manner [4, 23] which aims to segment new classes by learning from densely annotated seen classes in the pre-training data. Recently, several studies [18, 22, 46, 19, 26, 52, 45, 48] introduce language supervision to enhance the generality of the learnt zero-shot models. These approaches require no data annotation from new classes, but still rely on dense annotations from seen classes during the pre-training stage. No Supervision : the second category follows a training-free approach [51, 38, 37] which explores the segmentation potential of frozen vision-language models (VLMs) [51, 38, 37] to predict segmentation masks. However, most VLMs are trained with image-level supervision which restricts their capability on pixel/region-level predictions in semantic segmentation. Language Supervision: the third category follows a pure-language-supervision paradigm [43, 28, 32, 35, 7, 44] which aims to learn pixel grouping from solely image-text pairs. Our work follows the third approach. Different from existing studies, we identify the semantic gap in pre-training image and text data and design concept curation that mitigates the semantic gap with clearly improved semantic segmentation performance, more details to be described in the ensuing subsections.

3 Methodology

With clear semantic gaps between visual and textual concepts in pre-training data as illustrated in Fig. 1 (a), one naïve solution is to employ given image as query to retrieve related captions and derive missing textual concepts as described in Sec. 3.2. However, the naïve solution suffers from clear semantic bias as most VLM-retrieved captions contain salient concepts only as illustrated in Fig. 1 (b). We thus further design vision-guided expansion , text-to-image-guided ranking and cluster-guided sampling for better mitigation of the semantic gap as presented in Fig. 1 (c) and Sec. 3.3.

3.1 Revisiting GroupViT

Segmentation Architecture. We use GroupViT [43] as the segmentation backbone for pre-training. Assume a batch of image-text pairs { ( x I , x T ) } B i =1 , where x I amd x T denotes an image and its paired caption, respectively. For the vision flow, a grouping-aware transformer F I s encodes image x I as G segment tokens z I seg = { z I seg g , g = 1 , ..., G } ∈ R G × d , where each segment token z I seg g ∈ R d encodes an arbitrary-shaped region in image x I .

Image-Caption Contrastive Loss. To perform pre-training, the segment tokens Z I seg are merged via average pooling, producing a global representation z I seg ∈ R d that captures all the visual concepts appeared in image x I . Meanwhile, the paired caption x T is encoded to z T ∈ R d by a text encoder F T . The visual embedding z I seg and textual embedding z T are mapped to the same space by separate linear projectors. The segmentator is then learnt from language supervision by the standard contrastive objective InfoNCE [33], which is defined as:

where τ is a learnable parameter initialized with 0.07 [34] and the z I · z T computes the cross-modal cosine similarity.

Multi-Label Loss . Beyond learning segmentation from raw caption x T , GroupViT [43] further introduces L extra text labels { x T l , l = 1 , ..., L } by prompting the extracted concepts { c l , l = 1 , ..., L } with handcrafted templates [34] (e.g., 'a photo of a {concept}'). The L text labels are fed to the same text encoder F T to obtain textual representations of { z T l , l = 1 , ..., L } . The language supervision by multi-label loss is thus defined as:

The overall training objective of learning segmentation from language supervision in [43] is defined as:

Discussion. We use the exact same training objective as in GroupViT [43] to learn segmentation from language supervision. For each pre-training image x I , the multi-label loss enhances the contrastive learning [43] with L extra positive pairs and L ( B -1) extra negative pairs ( B denotes batch size used in pre-training). However, we highlight that the simple concept prompting does not expand the textual concepts much. The cross-modal semantic gap still exists and circulates in pre-training, which holds back the training convergence and degrades the zero-shot transfer.

3.2 Naïve Solution

Caption Curation. The web-crawled image-text pairs are often noisy with imprecise and even irrelevant text descriptions [41]. In addition, many visual concepts (especially those inconspicuous in the background) in images are often missing in the corresponding text descriptions. Both factors lead to clear semantic gaps between web-crawled images and texts. With super-rich image-text correlations in pre-trained VLMs such as CLIP [34], a straight solution, which we term by caption curation , is to apply x I as query to retrieve L extra captions { x T l , l = 1 , ..., L } from pre-training data. The semantic gaps between visual and textual modality could thus be mitigated by identifying relevant concepts from the retrieved captions.

CLIP Image

Encoder

Concept Archive

CLIP Text

Encoder

Prompted Texts

Another one of my

male

alpacas

.

Image-Text

Pair

Image-to-Image Retrieval

(within ALL DATA)

Text-to-Image Retrieval

a photo of

person

a photo of

camel

a photo of

grass

a photo of

horse

…

Concept

Score

…

person

camel

grass

horse

Concept Archive

(ordered)

I found you an

alpaca

! ;)

person.

Another one of

my male

alpacas

.

Camel

under

the blue

sky

.

Another one of my

male

alpacas

.

A

man

who

rides a

mule

.

A white

camel

is

looking around.

A

horse

grazes

on the

lawn

.

A

horse

grazes

on the

lawn

.

Alpaca

resting

in the

grass.

Another one of

my male

alpacas

.

A

man

who

rides a

mule

.

I found you an

alpaca

! ;)

person.

Alpaca

resting

in the

grass.

…

…

…

…

Person

hugs the

llama

.

Person

hugs the

llama

.

Person

hugs the

llama

.

…

person

man

grass

sky

lawn

horse

camel

field

alpacas

llama

person

man

grass

sky

lawn

horse

camel

field

alpacas

llama

…

…

I found you an

alpaca

! ;)

person.

Alpaca

resting

in the

grass.

Alpaca

in a

field

during the day.

Camel

under

the blue

sky

.

A

horse

grazes

on the

lawn

.

A white

camel

is

looking around.

Person

hugs the

llama

.

A

man

who

rides a

mule

.

…

Semantic Bias. Though caption curation expands visual concepts effectively, retrieved captions often suffer from clear semantic bias : VLMs tend to retrieve salient concepts but miss many inconspicuous ones that are also useful in image description. Consequently, visual concepts C I = { c l , l = 1 , ..., M I } appeared in an image x I are usually clearly more than textual concepts C T = { c l , l = 1 , ..., M T } extracted from { x T , x T 1 , ..., x T L } (i.e., M I > M T ). The root cause of the semantic bias lies with the loose correlation between the visual and textual pre-training data of VLMs, where most captions just capture partial visual concepts appeared in the paired images [25]. The semantic bias thus impedes convergence and effectiveness of language-supervised training without language supervision available for those non-described image regions.

3.3 Concept Curation

To bridge semantic gaps in image-text pairs, we propose Concept Curation (CoCu) to rewrite caption semantics with the help of a pre-trained vision-language model. Consequently, CoCu finds more concept candidates that are aligned to images and compensate for missing semantics in captions. In pre-training, a multi-modal segmentor matches images and visual-enriched captions by contrastive objectives mentioned in Sec. 3.1, encoding better vision-language alignment in its representations. Details of CoCu are described as below.

Vision-driven Expansion. For an image-text pair ( x I , x T ) , the goal of the vision-driven expansion is to build an archive of textual concepts C T = { c m , m = 1 , ..., M } that are potentially matched with x I as illustrated in Fig. 2. Instead of acquiring C T via direct text retrieval as in caption curation , we resort to cross-image retrieval to achieve the expansion. Concretely, N image-text pairs P = { ( x I i , x T i ) , i = 1 , ..., N } are automatically selected from the pre-training data, where { x I i , i = 1 , ..., N } are N captioned images whose visual features match the best with that of x I (all encoded by CLIP). C T can thus be derived by extracting textual concepts from the captions

{ x T i , i = 1 , ..., N } of the N best matched images. Compared with the caption curation that retrieves L descriptions { x T i , i = 1 , ..., L } , vision-driven expansion exploits all visual information in images instead of biased caption texts (mostly describes salient visual concepts only), which helps restore more relevant textual concepts. In addition, it builds an extra image set { x I i , i = 1 , ..., N } that plays a pivotal role in the upcoming stages.

Text-to-Vision-Guided Ranking. For each textual concept c m in the concept archive C T , we assign a score s c m to represent its relevancy to image x I . A naïve solution to p m is to compute the cosine similarity between the visual representation of x I and textual representation of t m encoded by CLIP [34], which is simply defined as:

where t m is derived from visual concept c m via prompt engineering [34, 43]. Note direct text retrieval could easily get biased towards salient visual concepts here, imposing low relevance scores for other text concepts in concept archive . Beyond f ( x I , t m ) , we also design a non-biased metric to capture the relevancy between image x I and concept c m . Specifically, for the N retrieved image-text pairs P , we first extract a subset P G (blue box/text in Fig. 2 (below)), whose caption of each image-text pair contains the textual concept c m . The non-biased metric is thus defined with x I (red box) and image-text pairs P G = { ( x I i , x T i ) , i = 1 , ..., N ′ } as follows:

The given term functions as follows: 1) lower relevancy between image x I and irrelevant concepts (e.g., 'horse' in Fig. 2); 2) enhance relevancy between x I and inconspicuous concepts (e.g., 'grass'). Instead of computing relevancies of x I to all textual concepts in a single run, we consider one c m at a time and measure its relevancy by comparing f ( x I , t m ) with { f ( x I i , t m ) , i = 1 , ..., N ′ } . The idea behind this is simple: 1) comparing the responses of images { x I , x I 1 , ..., x I N ′ } to the same textual concept c m ; 2) high relevancy is given if response of x I to textual concept c m is comparably high and vice versa. Take the visual concept 'grass' in Fig. 2 (below) as an example. The t m ('a photo of grass') causes x I to rank considerably higher than image captioned with c m ('grass'). In this case, we should be fairly confident to pass the concept c m to x I . In conclusion, the relevancy score is simply defined as:

we perform ranking according to computed relevancies { s c m , m = 1 , ..., M } , which represents chances identified by later sampling.

Cluster-guided Sampling. The pre-training can thus be empowered by including the expanded and ranked textual concepts which as selected by sampling L textual concepts according to their computed relevancies as in [43]. However, selection with relevancy alone is often short of semantic diversity in the selected text concepts. Instead of directly selecting L concepts from the ranked archive C T , we partition C T into L semantic clusters based on their textual representations and sample one textual concept from each semantic cluster. The cluster-guided sampling has two clear benefits: 1) it includes more diverse semantics in each single training step; 2) it keeps good consistency with the expression of visual concepts, more details to be discussed in Sec. 4.4 and appendix.

4 Experiments

4.1 Experimental Setup

Training Detail . We follow the prior study [43] and conduct pre-training on three publicly available image-text datasets: CC3M (C3) [36], CC12M (C12) [8], YFCC14M (Y14) [39]. For fair comparison, we use the same GroupViT [43] as the visual encoder, which is built upon ViT-S backbone [14, 40] and learnt from scratch. We set the global batch size for contrastive learning as 1,024 and use 4 Tesla V100 GPUs to carry out pre-training for all experiments. Consistent with [43], we set the initial learning rate to 0.0016. The pre-training undergoes 30 epochs, with a linear warmup for the first 2 epochs and a cosine schedule for the remaining epochs. L is set to 3. In our ablations and discussions, we report the performance of models pre-trained on CC3M.

Implementation of curation .The curation pipeline utilizes clip-retrieval [3], a utility that enables efficient computation of CLIP embeddings and fast indexing for retrieval. We employ the CLIP ViT-B/16 [34] model for image/text inference and concept curation. For efficient semantic searching, we build indexing systems using autofaiss 3 . It is worth mentioning that alternative systems can also be used for implementing the curation process.

Evaluation . We benchmark zero-shot transfer performance of CoCu on the validation splits of eight different datasets that cover a myriad of scenes and category sets, including Pascal VOC [15], Pascal Context [30], COCO [27], ImageNet-S-50, ImageNet-S-300 [17], COCO Stuff [5], Cityscapes [12], and ADE20K [50]. For the first five datasets, we follow [43] and evaluate foreground classes by thresholding the similarity between visual and textual embeddings. For other datasets, we evaluate both foreground and background classes. More details are given in the appendix.

4.2 Comparison with the state-of-the-art

We first benchmark CoCu with state-of-the-art zero-shot methods [51, 43] and evaluate its effectiveness. Specifically, we follow prior work [43] and pre-train CoCu over the combination of C3, C12, and Y14 datasets. Tab. 1 reports zero-shot segmentation results. Besides GroupViT as the baseline method, we also compare the advanced MaskCLIP [51]), which directly leverages the frozen CLIP model for segmentation prediction without pre-training. In addition, for a comprehensive comparison, we list the performance of other advanced methods including 1) fully-supervised method [40] that provides Oracle's performance, 2) self-supervised methods [20, 6] that pre-train models with unlabeled data and fine-tuning models over segmentation datasets. Detailed implementations of the comparing methods could be found in the appendix.

As shown in Tab. 1, MaskCLIP achieves limited segmentation performance, primarily due to CLIP being trained with image-level supervision and thus falling short in precise pixel-level predictions. GroupViT achieves better performance than MaskCLIP, but still limited by insufficient supervision from language side in pre-training. On the contrary, our CoCu achieves the best segmentation performance over all eight benchmarks, surpassing GroupViT by large margins on average. This indicates the necessity of bridging semantic gaps in language-supervised semantic segmentation and the effectiveness of our design.

We further evaluate the robustness of CoCu with different pre-train data. Spcifically, we pre-train GroupViT and CoCu over CC3M and CC12M, respectively. We also sub-sample half of image-text pairs from CC12M (denoted as C12 ∗ ) for pre-training. Tab. 2 shows the experimental results. We can observe consistent yet significant performance gains on eight benchmarks. The improvement by bridging semantic gap is thus robust and not affected by pre-training size.

epoch 4

epoch 9

epoch 14

epoch 19

training

hours

0

3

6

9

12

15

18

6

9

10

12

14

loss

GroupViT

CoCu (ours)

(a) training convergence

16

(b) segmentation results of

grass

Caption:

a red fox drinking water.

GroupViT

CoCu (ours)

4.3 CoCu helps convergence

Loss Curve. In Figure 3 (a), we compare the pre-training loss curves of GroupViT and our proposed method CoCu. We can see that CoCu exhibits a notably faster convergence rate, primarily attributed to the inclusion of curated semantic concepts for each image, resulting in more effective contrastive learning. Additionally, CoCu achieves a lower minimum loss by extracting significantly richer language concepts from image data. This enriches the training process by incorporating more identified image regions and ultimately learning representations that better align with the training data.

Qualitative Comparison . We also present qualitative results that demonstrate the effectiveness of CoCu. In Figure 3 (b), we show binary segmentation results of GroupViT (first row) and CoCu (second row) over an example image with the caption "a red fox drinking water." Our focus is on the concept of "grass," which is missing in the caption. We compare the visual responses of models trained using these two methods at different checkpoints. Both methods improve progressively during training. However, GroupViT fails to correctly localize the region of "grass" due to the lack of direct supervision from the language side. In contrast, CoCu bridges the semantic gap by accurately capturing and localizing "grass," encoding it in representations during pre-training. Consequently, it achieves significantly better segmentation results under zero-shot context.

Figure 4 displays the activation maps of GroupViT and CoCu for different concepts as text inputs that do not appear in the corresponding captions. These maps further demonstrate the superiority of CoCu in language-supervised learning. In all presented images, GroupViT incorrectly activates corresponding regions based on the given text inputs (e.g., activating the "sky" region with a text input of "person" for the first image). In contrast, CoCu enables the segmentor to have the highest activations on visually relevant regions indicated by the text. This suggests that segmentors derived from our method have a better capability to discriminate various visual concepts. More convergence results can be found in the appendix.

plate

Caption:

Cooked pasta on wooden table

.

person

Caption:

Alpacas Kisses is a new business that brings alpacas to you

milk

Caption:

Baked powdered sugar donuts whole wheat.

CoCu (Ours)

GroupViT

high

low

GroupViT

GroupViT

CoCu (Ours)

CoCu (Ours)

4.4 Analysis

Ablation Study. We further assess the effectiveness of each module in CoCu, which includes vision-driven expansion , text-to-image-guided ranking and cluster-guided sampling . Specifically, we pre-train five models with the combination of these modules or their alternative strategies, namely: 1) Baseline model of GroupViT, which is pre-trained without involving concept curation. 2) Model #1, which utilizes language-driven expansion, naïve ranking, and naïve sampling (Caption Curation in Sec. 3.2). 3) Model #2, which replaces language-driven expansion with vision-driven expansion on top of Model #1. 4) Model #3, which incorporates text-to-image-guided ranking on top of Model #2. And 4) the full CoCu Model #4, which combines vision-driven expansion, text-to-image-guided ranking, and cluster-guided sampling in pre-training. We report the average segmentation performance of these models across the eight datasets used previously (as shown in Table 1 and Table 2). Detailed illustrations of implementations are provided in appendix.

As Tab. 3 shows, the simplest strategy of language-driven in model #1 improves average mIoU by 1 . 7% , which comes from stronger vision-language correlation in pre-training data enhanced by direct text retrieval. Next, replacing direct text retrieval with vision-driven expansion in model #2 brings an additional performance boost, highlighting its significance in capturing unbiased semantics. Furthermore, incorporating text-to-vision-guided ranking in Model #3 brings another noticeable performance gain, underscoring the importance of measuring concept-to-image relevancy. Finally, we upgrade the sampling strategy from the naïve one that solely relies on relevancy to cluster-guided sampling , and build model #4 with the full CoCu, which provides more diverse semantic information in each pre-training step, ultimately leading to the best zero-shot transfer performance for semantic segmentation.

Zero-Shot Classification . In addition to its application in zero-shot segmentation, CoCu can also be used to improve zero-shot classification. Following the previous study [43], we evaluate CoCu and compare it with GroupViT on the ImageNet-1K dataset [13]. As shown in Table 4, CoCu exhibits significant performance gains over GroupViT, demonstrating its superiority in bridging semantic gaps across tasks and achieving improved zero-shot classification results.

5 Conclusion

In this paper, we identify the issue of semantic gap in language-supervised semantic segmentation and explore how to bridge semantic gaps effectively. To achieve this, we design Concept Curation, a novel pipeline that resolves the issue by three consecutive stages: vision-driven expansion , text-to-visionguided ranking and cluster-guided sampling . Extensive experiments demonstrate the superiority of our method for boosting language-supervised semantic segmentation across a bundle of pre-training sets and evaluation benchmarks. Looking ahead, we hope to extend the idea of concept curation to other computer vision tasks, including object detection and instance segmentation.

Acknowledgments and Disclosure of Funding

This project is funded by the Ministry of Education Singapore, under the Tier-2 project scheme with a project number MOE-T2EP20220-0003.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022.

Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4253-4262, 2020.

Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. GitHub, 2022.

Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems , 32, 2019.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1209-1218, 2018.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for openworld semantic segmentation from only image-text pairs. arXiv preprint arXiv:2212.00785 , 2022.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3558-3568, 2021.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual languageimage model. arXiv preprint arXiv:2209.06794 , 2022.

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34:17864-17875, 2021.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, UweFranke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-308, 2009.

Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1473-1482, 2015.

Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2112.12143 , 2021.

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , pages 540-557. Springer, 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546 , 2022.

Peike Li, Yunchao Wei, and Yi Yang. Consistent structural relation learning for zero-shot segmentation. Advances in Neural Information Processing Systems , 33:10317-10327, 2020.

Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208 , 2021.

Yehao Li, Yingwei Pan, Ting Yao, and Tao Mei. Comprehending and ordering semantics for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17990-17999, 2022.

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:2210.04150 , 2022.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open-world semantic segmentation via contrasting and clustering vision-language embedding. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX , pages 275-292. Springer, 2022.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 891-898, 2014.

Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets languageimage pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 529-544. Springer, 2022.

Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. arXiv preprint arXiv:2212.04994 , 2022.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. arXiv preprint arXiv:2302.10307 , 2023.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

Gyungin Shin, Weidi Xie, and Samuel Albanie. Namedmask: Distilling segmenters from complementary foundation models. arXiv preprint arXiv:2209.11228 , 2022.

Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. arXiv preprint arXiv:2206.07045 , 2022.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Tianren Gao, Joseph E. Gonzalez, and Peter Vajda. Data efficient language-supervised zero-shot recognition with optimal transport distillation. In International Conference on Learning Representations , 2022.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems , 34:12077-12090, 2021.

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18134-18144, 2022.

Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary semantic segmentation models from natural language supervision. arXiv preprint arXiv:2301.09121 , 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for openvocabulary semantic segmentation. arXiv preprint arXiv:2302.12242 , 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX , pages 736-753. Springer, 2022.

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 , 2021.

Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.08131 , 2023.

Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. arXiv preprint arXiv:2304.00685 , 2023.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633-641, 2017.

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII , pages 696-712. Springer, 2022.

Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270 , 2022.

Peer reviews:
comments: SUMMARY:
This paper first points out the semantic gap problem of the existing language-supervised semantic segmentation method. It shows that not all visual elements are included in the corresponding language annotations. Then the paper proposes Concept Curation (CoCu), which includes Vision-driven Expansion, Text-to-Vision-Guided Ranking, and Cluster-guided Sampling strategies to solve the semantic gap problem. The experiments show the effectiveness of the CoCu.

STRENGTHS:
1. The semantic gap problem sounds very reasonable and significant for language-supervised semantic segmentation.
2. The proposed method CoCu could alleviate the semantic gap problem to some extent.
3. The experiments and ablation study are comprehensive.

WEAKNESSES:
1. The authors should summarize the Sec. 2.3 at the beginning or the end of this section. For example, in general, Sec. 2.3 provides a method to find better class candidates for the loss in Sec. 2.1. Otherwise it confuses the readers about the ultimate goal of Sec. 2.3.
2. The overall pipeline of CoCu is a kind of dataset pre-processing method, and it is complicated. Is CoCu done online or offline? If online, how much time does it cost to find the final class candidates?

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel data curation/argumentation process, named Concept Curation (CoCu), for language-supervised semantic segmentation. In the setting of language-supervised semantic segmentation (e.g. GroupViT), the network is trained with image-text contrastive loss on large-scale image-text pairs. Authors identified several issues in the vanilla data distribution of the original contrastive learning, e.g. semantic gap, and semantic bias. The proposed CoCu mitigates these issues and improves over prior work GroupViT under a controlled setting, showing a faster convergence rate and high accuracy.\n\nSTRENGTHS:\n1. In the quantitative evaluation, authors re-implemented GroupViT and compare CoCu and GroupViT under the controlled experiment setting, i.e. 1024 batch size. The authors also report the accuracies on additional evaluation datasets, IN50, IN300, Cityscapes, etc. CoCu outperforms GroupViT on all the datasets by a margin. Although CoCu doesn't achieve state-of-the-art results on some datasets, it doesn't degrade the effectiveness of the method. \n2. The proposed method is well-motivated. The web-crawled image-text pairs are indeed quite noisy. And GroupViT is also known to be bad at segmenting background classes like grass and sky. Moreover, from the visualization in Figure 3(b), the CoCu shows that could focus on background grasses instead of the foreground fox. \n3. In the Table 2 ablation study, when only trained on CC3M, CoCu improves over GroupViT by a large margin on Pascal VOC. It is a very interesting result and justifies that CoCu speeds up the convergence.\n\nWEAKNESSES:\n1. Insufficient qualitative comparison. Since one major claim of CoCu is that, compared with vanilla contrastive loss used in GroupViT, the proposed dataset curation mitigates semantic cap and semantic bias issues. So besides quantitative evaluation metric mIoU, more visualizations compared with GroupViT are expected. I would suggest authors add more visualizations in the supplementary materials if there is no space left in the main submission. \n2. In the ablation table 3, the authors studied the effects of different components of CoCu. But the average mIoU may not be an insightful metric. Since different datasets have very different category vocabulary, authors may try to include a detailed ablation study table in the supplementary material and elaborate more on the results.\n\nQUESTIONS:\n1. The boundary between the semantic gap and semantic bias is not that clear. At least from Figure 1, it's hard to tell the major differences between them. Maybe authors could define them more rigorously and provide more examples about them. \n2. How long does it take to process the dataset? It would be informative for other readers who want to reproduce or follow up on the curation process. \n3. In Table 2, CoCu also showed significant improvement on ImageNet50 and ImageNet300. But for these two datasets, the vocabularies are mainly foreground, it would be very interesting to show why CoCu could improve on these datasets as well.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper targets learning unsupervised semantic segmentation from image-text pairs. The authors specifically address the issue of training data quality in previous method Group-ViT and propose an approach to enhance the captions with additional visual concepts through an automated pipeline. The paper demonstrates the effectiveness of this data filtering pipeline by evaluating it on official segmentation benchmarks.

STRENGTHS:
- The semantic gap problem the authors studied is intresting and meaning full.
- The writing is good and the paper is easy to understand.

WEAKNESSES:
The proposed data filtering pipeline relies on the CLIP model for collecting visually similar samples. However, the CLIP model has been trained on a much larger scale, with hundreds of millions of image-text pairs, whereas the training data used in this paper is relatively smaller. Does it work by distilling the CLIP model? This raises concerns about **the effectiveness of the pipeline when scaling up to larger training data sizes**. It would be helpful to explore **alternative self-supervised methods such as DINO, MAE, or Group-ViT itself to replace the CLIP model** in the pipeline and evaluate its performance.

QUESTIONS:
See weakness.

LIMITATIONS:
NaN
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent VLMs suffer from a noticeable semantic gap between visual and textual modalities since many visual concepts present in images are easily missed in their paired captions. This work proposes Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, CoCu establishes a concept archive that maintains potential visually-matched concepts using vision-driven expansion and text-to-vision-guided ranking. This approach enables the identification of relevant concepts through cluster-guided sampling, which are then fed into the pre-training process. As a result, CoCu bridges the gap between visual and textual semantics. Extensive experiments conducted on eight segmentation benchmarks demonstrate that CoCu achieves exceptional zero-shot transfer performance and significantly enhances the language-supervised segmentation baseline by a substantial margin. These results underscore the importance of closing the semantic gap in pre-training data. The code for CoCu will be made available to the research community.\n\nSTRENGTHS:\n1. Overall, the paper is well-written, and the study of this work has a good motivation.\n2. The mechanism of CoCu is intuitive and easy to follow.\n3. CoCu demonstrates significant empirical results: it outperforms GroupViT (re-implemented baseline) by 4.6% mIoU in average on eight popular semantic segmentation benchmarks.\n4. CoCu also yield good qualitative results, with more accurate and smooth masks than its baseline.\n\nWEAKNESSES:\n1. Despite the fact that CoCu indeed enriches visual concepts during VL pre-training, the cost seems to be heavy. During training, a CLIP model is employed to perform the key components of CoCu such as vision-driven expansion and text-to-vision guided ranking, which introduces a lot of additional computation. Thus, the comparison to your baselines such as GroupViT might not be strictly fair. A detailed comparison of training time or FLOPS should be presented to support the effectiveness of CoCu.\n\n2. My bigger concern lies in how CoCu relies on the pre-trained CLIP, i.e., if using a weaker CLIP model for CoCu, how will the performance change? If CoCu is not sensitive to the CLIP's performance, you can directly use GroupViT to perform vision-driven expansion and text-to-image-guided ranking, so that no additional parameters will be introduced. As shown in Table 4, GroupViT obtains acceptable zero-shot classification results. However, if CoCu highly relies on a strong CLIP, your contributions might also be challenged.\n\nQUESTIONS:\n1. In table 1, are both CoCu and GroupViT trained for 30 epochs?\n2. With the concern of Weakness 1, I am also wondering what if you change the x-axis from #epochs to actual training time in Figure 3? Will CoCu still converge faster than GroupViT?\n\nLIMITATIONS:\nAs CoCu basically follows the architecture of GroupViT, it has the same limitation that for each image, the number of semantic concepts segmented by CoCu/GroupViT has a maximum of the number of group tokens fed to the vision encoder. Thus, given a high-resolution image with many semantic regions, these methods might fail to generate accurate segmentation masks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}