Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation
Authors: Yun Xing
Jian Kang
Aoran Xiao
Jiahao Nie
Ling Shao
Shijian Lu
Venue: NeurIPS 2023 poster (2023)
Keywords: language-supervised semantic segmentation
vision-language pre-training

Abstract: Vision-Language Pre-training has demonstrated its remarkable zero-shot recognition ability and potential to learn generalizable visual representations from languagesupervision. Taking a step ahead, language-supervised semantic segmentation enables spatial localization of textual inputs by learning pixel grouping solely from image-text pairs. Nevertheless, the state-of-the-art suffers from a clear semantic gap between visual and textual modalities: plenty of visual concepts appeared in images are missing in their paired captions. Such semantic misalignment circulates in pre-training, leading to inferior zero-shot performance in dense predictions due to insufficient visual concepts captured in textual representations. To close such semantic gap, we propose Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, we establish a concept archive that maintains potential visually-matched concepts with our proposed vision-driven expansion and text-to-vision-guided ranking. Relevant concepts can thus be identified via cluster-guided sampling and fed into pre-training, thereby bridging the gap between visual and textual semantics. Extensive experiments over a broad suite of 8 segmentation benchmarks show that CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin, suggesting the value of closing semantic gap in pre-training data.

Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation
Yun Xing
Jian Kang'
Aoran Xiao
Jiahao Nie Ling Shao?
Shijian Lu
Nanyang Technological University
2 UCAS-Terminus AI Lab, UCAS, China
Abstract
Vision-Language Pre-training has demonstrated its remarkable zero-shot recogni- tion ability and potential learn generalizable visual representations from language supervision. Taking a step ahead, language-supervise semantic segmentation en- ables spatial localization of textual inputs by learning pixel grouping solely from image-text pairs. Nevertheless, the state-of-the-arr suffers from clear semantic gaps between visual and textual modality: plenty of visual concepts appeared in images are missing their paired captions. Such semantic misalignment circulates in pre-training, leading to inferior zero-shot performance in dense predictions due to insufficient visual concepts captured in textual representations. To close such se- mantic gap, we propose Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, establish cept archive that maintains potential risually-matched matched concepts with our proposed vision-driven expansion and text-lo- tisittovviion---e vision-guided ranking. Relevant concepts thus be identified via cluster-guided sampling and fed into pre-training, thereby bridging the gap between visual and textual semantics. Extensive experiments over a broad suite segmentation benchmarks show that CoCu achieves superb zero- segmentation
shot transfer performance and greatly boosts language-supervisg baseline by large margin, suggesting the value of bridging semantic gap in pre- training data. Code available at https: hhttps://githni //gtttbttttt/////// commxiigg004/reewrita
Introduction
10, which aims learn visual representations directly from natural language supervision. has endowed existing recognition systems with superior generality and open-vocabulary understanding capability. As representative, CLIP 34 performs contrastive language-image pre-training on 400M web-crawled image-text pairs, whereby the learnt models may effortlessly transfer to wide spectrum of classification tasks in a zero-shot manner. Motivated by the extend the supervision paradigm to semantic segmentation, enabling spatial localization textual queries images and pixel grouping with solely supervision from image-text pairs. Distinct from conventional semantic segmentation, the language-supervised paradigm obviates the need for costly manual pixel-level annotation and enables million-level pre- training scale with much less effort.
Vision-Langusal Pre-training 34 21
breakthrough, recent studies 43 35
Despite the progresses 443 35) in language- superviied semantic segmentation, the pre-training stage still suffers heavily from clear semantic gap between visual and textual modality. In image-text pairs used for pre-training, is ubiquitous that visual concepts appeared images are missing in the corresponding textual captions. This happens largely because captions merely describe salient concepts that are worthy of mention [16] 25 while naturally forgo full semantic coverage of images
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(n) Semantic Gap
(O) Oversfew of Concept Curation
Imuge
Cancept Archive
Image
Caption
aljacas
hocw
aqpactt
graty
olpucas *-
Anuther one of mule olpacer
vhion-drinoe exponsion
graus Hlass
sky
person Bold
lowa
(b) Semantii
Erooddrr
Excading
Receleved Captions
Grerap Vokee
-it-----tiion- guided ranking
insage de dext retricnal
gurdener work.
Fioct Tokker
Image
Suporisise Saxnd
Cancept Archive (Ordered)
semontic hius
Staneling with Haraar
Esoaly
Restire
aijanas
Peros pietored while tonelling
persun
gardener
Prompted
gras
alpaces
Haraa
alpanas
porsiss
person
Prospi
goass
claster-gwhied
sky
alpocas
mun
5ky
vongaling
contol
korre
Figure 1: Cross- semantic gap is prevalent in web-crawled image-text pairs. As in (a), the caption text often captures certain salient visual concepts only in the paired image but misses many others (i..., 'person'. "grass', and 'sky") that are also useful in image-text modeling. Leveraging CLIP (34, more useful visual concepts could be captured via image-to-text retrieval, but the retrieved captions usually suffer from the semantic bias in (b) (i.e., 'person' recovered but "grass' and 'sky' still missing). Our proposed Concept Curation (CoCu) bridges the cross- modal semantic gap effectively by dsione expansion, text-to- text-to-vision- wision-guided ranking and cluster- cluster-guided sampling while avoiding the negative effect by semantic bias, as illustrated in (c). Best viewed in color.
(Fig. D (a)) Under the presence of clear cross-modal semantic gap in image-text pairs, the pre-training stage of inguage-supervisse segmentation is found to be harder to converge, leading to inferior zero-shot performance on downstream tasks (more details are elaborated in Section 4.3
This work explores to bridge semantic gaps in language-supervised semantic segmentation. For each image in the pre- pre-training data, the goal is to recover the missing visual concepts in its paired caption for more comprehensive image-text modeling. With the rich vision-language correlations in off-the-siie shelf foundation models such as CLIP (4, straight solution is retrieve the missing concepts from the text captions of pre- rre-training data. However, such retrieved captions suffer from the semantic bias illustrated in Fig. (b) (e-g.. "person" recovered but "grass" and "sky" still missing). The root cause lies with the original text captions in foundation model pre-training, which only capture salient concepts appeared in the paired images. Hence, the retrieved captions and concepts still suffer from clear cross-modal semantic gap.
We propose Concept Curation (CoCu), novel pipeline that side-steps the negative effect by semantic bias [ while exploiting vision-language foundation models for semantic segmentation. CoCu consists of three sequential stages: 1) vision-dre driven expansion that constructs concept archive via cross- image retrieval; 2) text-to-vision- -oiiiiionnguudedd ranking that scores the retrieved concepts according to their assigned relevancies; and 3) cluster- Sluserr-uided sampling that exploits semantic diversity beyond the relevancy scores for concept ranking (Fig. (c)). We perform pre-training from scratch on the segmentation backbone of 43] and evaluate zero-shot transfer over 8 widely adopted segmentation benchmarks. The experiments show that the proposed CoCu improves the baseline as well as the -tate-of-the-art consistently by large margins, indicating the necessity of closing the semantic gap and the effectiveness of our designs in concept archiving and concept ranking.
In summary, the contributions of this work three-fold. First, we identify the issue of semantic gap in ranguage-supervisee semantic segmentation, and demonstrate the effectiveness of mining more relevant visual concepts for segmentation model pre- gre-training. Second, we design Concept Curation (CoCu), a novel pipeline that constructs concept archives to expand and identify relevant
2to clarify, we refer to semantic gap as a problem in web-crawled image-text pairs and semantic bias as an issue in pre-trained uision-languee model.
2

---

visual concepts from pre-training data, mitigating the semantic gap between textual and visual modalities effectly. Third, extensive experiments show that the proposed pipeline achieves superior zero-shot transfer performance and outperforms the state-of-the-arr across 8 segmentation benchmarks consistently by large margins.
2
Related Works
Semantic Segmentation. Partitioning an image into semantic regions, also known as seman- tic segmentation, has been widely studied due to myriad real-world applications such video surveillance and autonomous driving. has been explored along different directions, e.g., by de- signing different network architectures
(29 9 42 constructing benchmarks of diverse scenes and categories [12 50 5 17]. etc. However, collecting per-category dense annotations for super- which impedes the upscaling of semantic vocabulary greatly. Different from conventional semantic segmentation, segmentation from language supervi- relieves the burden of mask annotation by leveraging image-text pairs available on the Internet. Beyond that, it can handle arbitrary new semantics thanks to the language supervision paradigm, making it feasible to learn generalizable segmentation models.
vised training is notoriously
sion 43 28 32
35 44
Vision-Language Pre-training. Recently, Vision-Language Pre-training has become a predominant trend by learning visual representations from natural language supervision (34 21 24
47 [1
10 49]. By matching billion-scale image-text pairs via contrast, the learnt representations can be seamlessly transferred to various downstream classification tasks in zero-shot manner. As representative, CLIP (34] can match the performance of supervised baselines on ImageNet [[3, meanwhile obtain competitive performance
plethora of downstream tasks without accessing any target data. The same learning paradigm has recently been explored for the task semantic segmentation by hierarchical grouping [4). supervision mining [35 44 7 32] etc. Nevertheless, state-of-the-arr
segmentation held back by the cross-modal semantic gap
between textual and visual pre- training data. Instead of relaxing the strict one-to-one correspondence in vanilla contrastive learning [35], we mitigate the semantic gap by automated curation of relevant visual concepts through concept expanding and concept ranking.
Bpen-Vocabulary Semantic Segmentation. vvea-Vocabylary Semantic Segmentation has been studied extensively and most existing work can be broadly grouped into three categories. Mix Supervision: the first category follows a zero-shot manner [4] [23] which aims to segment new classes by lcarning from densely annotated seen classes in the pre- pre-training data. Recently, several 48 introduce language supervision to enhance the generality of the learnt zero-shot models. These approaches require data annotation from new classes, but still rely on dense annotations from seen classes during the pre-training stage. No Supervision: the second category follows training- free approach [51] 38 37
studies [18 22 46
26
52
45
which explores the segmentation
38 37 to predict segmentation masks. However, most VLMs are trained with image-level supervision which restricts their capability on pixel/region-level predictions in semantic segmentation. I Language Supervision: the third category
potential of frozen vision-language models (VLMs) [51]
7 44 which aims to learn pixel grouping from solely image-text pairs, Our work follows the third approach. Different from existing studies, we identify the semantic gap pre-training image and text data and design concept curation that mitigates the semantic gap with clearly improved semantic segmentation performance, more details to described in the ensuing subsections.
follows a pure-language- languuggs--upprrvisiin paradigm 43
28
35
3 Methodology
With clear semantic gaps between visual and textual concepts in pre-training data as illustrated in Fig. [ (a). one naive solution is to employ given image query to retrieve related captions and derive missing textual concepts as described Sec. 3.2 However, the naïve solution suffers from clear semantic bias most VLM--retrieedd captions contain salient concepts only as illustrated in Fig.
(b). We thus further design vision- guided expansion, eext-t--maage-ue guided ranking and cluster-guided sampling for better mitigation of the semantic gap presented in Fig. (c) and Sec. 3.3
3

---

3.1 Revisiting GroupVit VIT
Segmentation Architecture. We use GroupViT ViT [43] as the segmentation backbone for pre-training. Assume a batch of image-text pairs {(r', x } where x' amd x" denotes an image and its paired caption, respectively. For the vision flow, a grouping- aware transformer F encodes image x' as G segment tokens >šeg = fzseg, . = 1.....G. G} E RGxd where each segment token 2šeg, E IRd encodes an arbitrary-shaped region image x'.
Image-Caption Contrastive Loss. To perform pre-training, the segment tokens Zzeg are merged via average pooling, producing global representation <seg _. that captures all the visual concepts appeared in image x'. Meanwhile, the paired caption x² is encoded to 2 E F by text encoder Fr. The visual embedding =šeg and textual embedding x" are mapped to the same space by separate linear projectors. The segmentator then learnt from language supervision by the standard contrastive objective InfoNCE 33 which is defined as:
exp(2 -2f
L1
log
(I)
B
B j=1 exp(2 (2) 22/
exp(= /T) log
CT
(2)
B
B Σ exp(= 22//7)
where 7 is a learnable parameter initialized with 0.07 34 and the z'.z" computes the cross-modal
cosine similarity.
Multi-Label Loss. Beyond learning segmentation from raw caption TT, GroupViT 43 further
introduces L. extra text labels {x:"), 1, ..... L} by prompting the extracted concepts {a, }} with handcrafted templates 34 (e.g., "a photo of a [concept]" ")) The L text labels are fed 1, 1,... L}. The language
to the same text encoder F to obtain textual representations of {2 supervision by multi-label loss is thus defined as:
1
Σ -
exp( 2 2, /T)
log
(3)
B
1 exp (2f :2 /T)
==1
L
B
exp T1 12 (T) ΣB exp
log
(4)
LB
1=1 ==1
The overall training objective of learning segmentation from language supervision in as:
43 is defindd
== L + LITHL
(5)
Discussion. We use the exact same training objective in GroupViT [43 learn segmentation from language supervision. For each pre-training image x', the multi-label loss enhances the contrastive learning [43] with L extra positive pairs and L(B 1) extra negative pairs (B denotes batch size used in pre-training). However, we highlight that the simple concept prompting does not expand the textual concepts much. The cross- modal semantic gap still exists and circulates in pre-training, which holds back the training convergence and degrades the zero-shot transfer.
3.2 Naïve Solution
The web-crawled image-text pairs are often noisy with imprecise and even irrelevant text descriptions [1] In addition, many visual concepts (especially those inconspicuous in the background) in images are often missing in the corresponding text descriptions. Both factors lead to clear semantic gaps between web-crawled images and texts. With super-rich image-text correlations in pre-trained VLMs such as CLIP (34,, a straight solution, which we term by cuption
Caption Curation.
curation, is to apply x! as query to retrieve I. extra captions {x 1,..., L.} from pre-training data. The semantic gaps between visual and textual modality could thus be mitigated by identifying relevant concepts from the retrieved captions.

---

Imagestollas Image Retieaal (witie DATA)
Concepe Areblee
Teat
Pair
Encoder
Jasdd Caunel aude perses
Peraa hio the catrra
canot
1931
appcaar
dasing
Testerinage Berioval
Prompied Tress
CLIP
Norr
NOA
of
pervee
Howe
uplttoof
uadd
(1Att Alpace rostingeessl bea see of grons
Conrept (ondera))
of
gort
of Ainu
Jonr
Poran Auso tee (aais
Anven
malt.
Auno
Auno
Figure 2: Illustration dision-driver expansion (above) and text-to-image soannngg-guieedd ranking (below) in CoCu. To compensate for missing semantics, vision-drived expansion establishes an archive of potential matched concepts through image-to-imal retrieval, while rext-to-vision-e vision-guided ranking scores retrieved concepts based on assigned relevancy. The textual concepts can later be identified pre-training by sampling. In the figure, images with a blue border D are retrieved via expanded concepts (marked as blue) using their paired captions, while images with red border represent images for curation (as anchor). Best viewed in color.
Semantic Bias. Though caption curation expands visual concepts effectively, retrieved captions often suffer from clear semantic bias: VLMs tend to retrieve salient concepts but miss many inconspicuous ones that are also useful in image description. Consequently, visual concepts C¹ = {cr, | 1..., M appeared in an image x are usually clearly more than textual concepts C {cl, ( 1,... }.. extracted from {x 2²'; "".,,"',,, (i.e., M > M²). The root cause of the semantic bias lies with the loose correlation between the visual and textual pre-training data VLMs, where most captions just capture partial visual concepts appeared in the paired images [25]. The semantic bias thus impedes convergence and effectiveness of ianguage-superiis training without language supervision available for those non-described image regions.
3.3 Concept Curation
To bridge semantic gaps in image-text pairs, we propose Concept Curation (CoCu) rewrite caption semantics with the help of a pre-trained vision-language model. Consequently, CoCu finds more concept candidates that are aligned images and compensate for missing semantics in captions. In pre-training, a multi-modal segmentor matches images and visual-enm -ericcheerccee captions by contrastive objectives mentioned in Sec. 3.1. encoding better langugee alignment in its representations. Details of CoCu are described as below.
Vision-driven Expansion. For image-text pair (1:², .TT)) ), the goal of the vision-driven expansion to build an archive of textual concepts CT = {Cm, m 1...., M) that are potentially matched with x illustrated in Fig. 2 Instead of acquiring CT via direct text retrieval in caption curation, we resort to cross-image retrieval to achieve the expansion. Concretely, N image-text 1,..,NN N} are automatically selected from the pre-training data, where {ø/ 1,....N) NN are N captioned images whose visual features match the best with that of x' (all encoded by CLIP). CT can thus be derived by extracting textual concepts from the captions
pairs P {(=}, o
5

---

i=..i=_...,, of the NN best matched images. Compared with the caption curation that retrieves I. descriptions {x." i= 1,.,,, I.), vision-driven expansion exploits all visual information in images instead of biased caption texts (mostly describes salient visual concepts only), which helps restore more relevant textual concepts. In addition, it builds an extra image set { i i= L,..,, N that plays a pivotal role in the upcoming stages.
Text-to- -i--t---isin--uuiiee Ranking. For each textual concept Cm in the concept archive CT, we assign score Scm. to represent its relevancy image x². A naïve solution to pm is to compute the cosine similarity between the visual representation of x' and textual representation of t.m encoded by CLIP 34,, which is simply defined as:
cm f (m)
(6)
where im is derived from visual concept Cm via prompt engineering [4] 43 Note direct text retrieval could easily get biased towards salient visual concepts here, imposing low relevance scores for other text concepts in concept archive. Beyond f(x', fm), we also design a non-biased metric to capture the relevancy between image x!' and concept Cm- Specifically, for the N retrieved image-text pairs P. we first extract subset Pa (blue box/text in Fig. [ig..[](oe (below)), whose caption of each image-text pair contains the textual concept Cm. The non-biased metric is thus defined with x' (red box) and image-text pairs Pa
{(at/ , xt ) i I,.., N
} as follows:
(1 f x')
Ci
f(tm: x²) + Ei= f(tm; x()
The given term functions as follows: 1) lower relevancy between image x' and irrelevant concepts (e.g., 'horse' in Fig. 2 2) enhance relevancy between x' and inconspicuous concepts (e.g., 'grass"). Instead of computing relevancies of x' all textual concepts in single run, we consider one Cm. at time and measure its relevancy by comparing f(x', fm) with (f (m), i i= L,..,,,.'...... N'). The idea behind this is simple: 1) comparing the responses of images far ,.,,.., ...... x 'yy' } to the same textual concept Cm: 2) high relevancy is given if response of x to textual concept Cm is comparably high and vice versa. Take the visual concept 'grass' in Fig. F (below) as an example. The tm ("a photo of grass') causes x'' to rank considerably higher than image captioned with cm ('grass"). In this case, we should be fairly confident to pass the concept Cm to x². In conclusion, the relevancy score is simply defined as:
(8)
we perform ranking according to computed relevancies {Sem m I,..., M). which represents chances identified by later sampling.
Cluster-guided Sampling. The pre-training can thus be empowered by including the expanded and ranked textual concepts which selected by sampling L textual concepts according to their computed relevancies as in [43 However, selection with relevancy alone is often short of semantic diversity in the selected text concepts. Instead of directly selecting T. concepts from the ranked archive C., partition CT into T. semantic clusters based on their textual representations and sample one textual concept from each semantic cluster. The cluster-guided sampling has two clear benefits: 1) it includes more diverse semantics in each single training step; 2) it keeps good consistency with the expression of visual concepts, more details to be discussed in Sec. 4n4 appendix.
4 Experiments
4.1 Experimental Setup
Training Detail. We follow the prior study +3 and conduct pre-training on three publicly available
image-text datasets: CC3M (C3) 36). CC12M (C12) 8, YFCC14M (Y14) (39]. For fair comparison, we use the same Group VIT 43 as the visual encoder, which is built upon ViT-S backbone [14 40) and learnt from scratch. We set the global batch size for contrastive learning as 1,024 and use 4 Tesla V100 GPUs to carry out pre-training for all experiments. Consistent with 43 we set the initial learning rate to 0.0016. The pre-training undergoes 30 epochs, with a linear warmup for the first 2 epochs and a cosine schedule for the remaining epochs. is set to In our ablations and discussions, we report the performance of models praited on CC3M.

---

Implementation of curation. The curation pipeline utilizes reip-retrieval B3, utility that enables efficient computation of CLIP embeddings and fast indexing for retrieval. We employ the CLIP ViT-B/16 [34] model for image/text inference and concept curation. For efficient semantic searching. we build indexing systems using autofaiss It is worth mentioning that alternative systems can also be used for implementing the curation process.
Evaluation. We benchmark zero-shot transfer performance of CoCu the validation splits of eight different datasets that cover a myriad of scenes and category sets, including Pascal VOC 15 Pascal Context 30 COCO [27], ImageNet-S
ImageNet-S- [7]. COCO Stuff [3), Cityscapes [2].
and ADE20K [50] For the first five datasets, we follow 43 and evaluate foreground classes by
thresholding the similarity between visual and textual embeddings. For other datasets, we evaluate both foreground and background classes. More details are given in the appendix.
4.2 Comparison with the state-of-the-art
We first benchmark CoCu with sfate-of-the-art zero-shot methods 51 43
and evaluate its effective- ness. Specifically, we follow prior work [43] and pre-train CoCu over the combination of C3, C12, and Y14 datasets. Tab. reports zero-shot segmentation results. Besides GroupViT the baseline method, we also compare the advanced MaskCLIP [SI]), which directly leverages the frozen CLIP model for segmentation prediction without pre-training. addition, for comprehensive compari- son, we list the performance of other advanced methods including 1) fully-supervised method [40] that provides Oracle's performance, 2) self-supervised methods [20] 6 that pre-train models with unlabeled data and fine-tuning models over segmentation datasets. Detailed implementations of the comparing methods could be found in the appendix.
As shown in Tab. I MaskCLIP achieves limited segmentation performance, primarily due to CLIP being trained with image-level supervision and thus falling short in precise pixel-level predictions. Group ViT achieves better performance than MaskCLIP, but still limited by insufficient supervision from language side in pre-training. performance over all eight benchmarks, surpassing Gio by large margins on average. This indicates the necessity of bridging semantic gaps in sunguage-supervise semantic segmentation and the effectiveness of our design.
On the contrary, our CoCu achieves the best segmentation
Table 1: Performance of different zero-shot methods for semantic segmentation. Abbreviations of benchmarks, from left to right: Pascal VOC [5]. Pascal Context [00, Microsoft COCO [5],
ImageNet-S (17], Cityscapes [12,, and ADE20K [50]. BS denotes pre-training batch size, while
LC represents local consistency (2] mask prediction. t denotes our ce-implementation. CoCu consistently achieves the best performance across all benchmarks.
Mathod
Protrain Data
Supersision
LC
BS
Backhome
PVOC
PCON
COCO
IN50
IN30
CITY
ADE
STUF
AVG
DeTT
IN-IK
VIT-S
53.0
scl
21.3
stlf
39.1
20.4
M0
CILY14
s:l
DINO
C12
self
37.6
22.8
MasCLIP 51
N.A.
RooNer-50
41.5
18.5
10.5
1388
79
18.8
8.3
10.3
15.0
MasKCLIP
51
N.A.
ViT-Br16
49.5
21.7
13.6
25.9
11.7
19.8
9.5
12.5
20.5
GoupViT 15
C3,12Y144
4,196
VITES
52.4
22.3
24.3
44.3
23.5
15.8
104
13,0
25.2
GrmpVi''
C.CI2Y14
S6X1
(suss)
C3C12Y14
12X1
VIES
49.7
22.5
46.7
24.7
12.0
14.9
GimpViT' 03
CLCI2Y14 C3C12Y14
8511
L,024
VTS
45.4
19.9
20.3
39.2
17.7
17.6
10.6
13.9
23.1
CoCa (ours)
SEX1
L,024
WIS
51.4
23.6
22.7
48.5
25.5
21.1
12.3
15.2
27.7
We further evaluate the robustness of CoCu with different pre-train data. Speifically, we pre-train GiT and CoCu over CC3M and CC12M, respectively. We also sub-sample half of image-text pairs from CC12M (denoted as C12") for pre-training. Tab. ssoows the experimental results. We can observe consistent yet significant performance gains on eight benchmarks. The improvement by bridging semantic gap is thus robust and not affected by pre-training size.

---

Table 2: Zero-shot semantic segmentation performance with different pre-training data. CoCu consistently outperforms the baseline method GroupViT across all benchmarks, demonstrating its effectiveness in bridging semantic gaps and achieving significant improvements.
Mechol
Pretrain Date
(VIC(20)
POON (4m
(9C0
IN58
18300 (201)
CITV (19)
ADR
STUP(177)
AYG
Grig
4
C3
150
10.4
6.5
10.2
2.
S.1
4.
77
5.2
CoCo
C3
16.8)
(166)
(1..8))
(.39()
520 0.153
(10.8))
Giog
C12'
820
18.8
12.
27.9
124
10.7
5.6
5.6
26.5
C12'
(.49)
8.16)
(..18)
(1 5.19)
(1208)
9.5 (10.00)
18:4 (2.05)
Goaghi
20.1 (.999)
(5.8
( 2.088)
GrigVi
C12
07.5
18.0
35.77
16.5
19.5
13.1
20.2
CoCu
C12
(13,69))
(1.2()
(1-451-
19.4 (+2.888)
16.00 (41.8))
(1200)
(42.6)))))
(e) training cunvergence
(b) segmentation results
1nos
spoch
epoch
epoch
spoch
GroupVIt' VI
CoCu (vurs)
31-
Coure
9-
training
hours
Captiotc fox dhieddng water.
Figure 3: CoCu enhances training convergence. (a) The training loss curves of Group VIT and CoCu demonstrate that CoCu significantly accelerates pre-training convergence. (b) CoCu achieves superior binary segmentation results (second row) compared to Group ViT (first row) for the concept of "grass," which is missing in the caption, using an example image captioned as "a red fox drinking water." Best viewed in color.
4.3 CoCu helps convergence
Loss Curve. In Figure 3 (a), we compare the pre-training loss curves of Group and our proposed method CoCu. We can see that CoCu exhibits notably faster convergence rate, primarily attributed to the inclusion curated semantic concepts for each image, resulting in more effective contrastive learning. Additionally, CoCu achieves lower minimum loss by extracting significantly richer language concepts from image data. This enriches the training process by incorporating more identified image regions and ultimately learning representations that better align with the training data.
Qualitative Comparison. We also present qualitative results that demonstrate the effectiveness of CoCu. In Figure (b), we show binary segmentation results of Group (first row) and CoCu (second row) over an example image with the caption "a red fox drinking water." Our focus is on the concept of "ggrass"" which is missing in the caption. We compare the visual responses of models trained using these two methods at different checkpoints. Both methods improve progressively during training. However, Group ViT fails to correctly localize the region of "grass" due to the lack of direct supervision from the language side. In contrast, CoCu bridges the semantic gap by accurately capturing and localizing "grass," encoding it in representations during pre-training. Consequently, achieves significantly better segmentation results under zero-shot context.
Figure 4 displays the activation maps of GroupViT and CoCu for different concepts as text inputs that do not appear in the corresponding captions. These maps further demonstrate the superiority learning. In all presented images, Group ViT incorrectly activates
of CoCu in sanguage-supervin corresponding regions based on the given text inputs (e.g., activating the "sky" region with a text input of "person" for the first image). In contrast, CoCu enables the segmentor to have the highest activations on visually relevant regions indicated by the text. This suggests that segmentors derived from our method have better capability to discriminate various visual concepts. More convergence results can be found in the appendix.
8

---

GT
CaCh (Ours)
Gropp
(oCa
GraupYil
CaCu (Ours)
Jor
Texl Imputt perans
Tall Iapec
Text Eegest place Captine: Coolen pore tonder sahir.
Captine: Alpacar Fioma tnd burteee
duroen vha orago alocar to
Capilan: powdown rage abace visle veiax.
Figure 4:
Visualization of activation heatmaps. Group ViT fails to activate on corresponding visual regions for concepts not represented in captions, while CoCu exhibits significantly better localization. High activation is shown as red, and low activation is displayed as blue. Best viewed in color.
Table 3:
Ablation study of CoCu. We conduct an ablation study on each designed modules. Zero-shot transfer performance on semantic segmentation results are reported, averaged across eight evaluation datasets. "Naäve ranking" refers to solely using cosine similarity between visual and textual representations (encoded by CLIP) as concept-to-image relevancy. "Naïve sampling" denotes selecting textual concepts based solely on relevancy before pre-training.
Expansion
Ranking
Sampling
Average mloU(%)
Model
lang-driven
vislon-driven
nuive text to vision guided
naive claster cluster-guided
Baseline 4
8.2
91.7 (1.7 t)
#2
10.3 (2.1 t)
12.4 (4.2 f)
CoCa)
13.1 (4.9 1)
4.4 Analysis
Ablation Study. We further assess the effectiveness of each module in CoCu, which includes vision-driven expansion, vea-to--iagg-guuidee ranking and cluster-guided sampling. Specifically, pre-train five models with the combination of these modules or their alternative strategies, namely: 1) Baseline model of Group which pre-trained without involving concept curation. 2) Model #1, which utilizes language-driven expansion, naïve ranking, and naīve sampling (Caption Curation in Sec. 3.2) Model #2, which replaces language-driven expansion with vision-driven expansion on top of Model #1. 4) Model #3, which incorporates tes-tto-maaegguuee
ranking on top of Model #2. And 4) the full CoCu Model #4, which combines dision-drive expansion, sott-to-mmge-guu guided ranking, and cluster- cluster-guided sampling in pre- training. We report the average segmentation performance of these models across the eight datasets used previously (as shown in [ and Table 2 Detailed illustrations of implementations are provided in appendix.
As Tab. 3 shows, the simplest strategy of language-driven model #1 improves average mloU by 1.7%. which comes from stronger vision-language correlation in pre-training data enhanced by direct text retrieval. Next, replacing direct text retrieval with vision-driven expansion in model #2 brings an additional performance boost, highlighting its significance in capturing unbiased semantics. Furthermore, incorporating text-to-viis ti-vision-- guided ranking Model #3 brings another noticeable performance gain, underscoring the importance of measuring concept-to to---iai-ige relevancy. Finally, we upgrade the sampling strategy from the naïve one that solely relies on relevancy to cluster- cusster-guiedd sampling, and build model #4 with the full CoCu, which provides more diverse semantic information each pre-training step, ultimately leading to the best zero-shot transfer performance for semantic segmentation.
Zero-Shot Classification. In addition its ap- plication in zero-shot segmentation, CoCu can also be used to improve zero-shot classification. Following the previous study [43, we evaluate CoCu and compare it with GroupViT on the 1
Table 4: Zero-shot classification on ImageNet IK. Acc@1 and Acc@5 denote top-1 and top-5 accuracy, respectively.
Pre-training data
Zero-shot
Method
ImageNet- KK dataset [13]. As shown in Table CoCu exhibits significant performance gains
Acc@ ((%)
Acc@5(%)
GroupVIT
C12
34.9
over Gro,p demonstrating its superiority in bridging semantic gaps across tasks and achiev- ing improved zero-shot classification results.
CoCu (ours)
C12
38.4
68.6 (5.3
Group
C3,C12.114
36.8
CoCu cours)
C3,C12,Y14
43,0 (6.2
73.7 (6.9

---

5 Conclusion
In this paper, we identify the issue of semantic gap in language-supervised semantic segmentation and explore how to bridge semantic gaps effectively. To achieve this, we design Concept Curation, a novel pipeline that resolves the issue by three consecutive stages: vision-driven expansion, text-to-vision- guided ranking and cluster-guide sampling. Extensive experiments demonstrate the superiority of our method for boosting unguage-Supervised semantic segmentation across a bundle of pre-training sets and evaluation benchmarks. Looking ahead, we hope extend the idea of concept curation to other computer vision tasks, including object detection and instance segmentation.
Acknowledgments and Disclosure of Funding
This project funded by the Ministry of Education Singapore, under the Tier-2 project scheme with a project number MOF--THP20220-0000..
References
[1] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[2] Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[3] Romain Beaumont. Clip retrieval: Easily compute clip embeddingg and build a clip retrieval system with them. GitHub, 2022.
[4] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
[5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209-1218, 2018.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 1650-96600 2021.
[7] Junhum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open- world semantic segmentation from only image-text pairs. arXiv preprint arXiv:22 2022.
[8] Soravit Changpinyo, Piyush Sharma. Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
[9] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[10] Xi Chen, Xiao Wang, Soravit Changpinyo, Piergiovanni, Piotr Padlewski, Daniel Salz, Sehastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language- image model. arXiv preprint XXX::2209.06694 2022.
[11] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-17887 17875, 2021.
[12] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3223,, 2016.
[13] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255
leee, 2009.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 190.9,992, 2020.
[15] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journnall of computer vision. 88:303 308, 2009.
10

---

[16] Hao Fang, Saurabh Cupta, Forrest landola, Rupesh K Srivastava, Li Deng. Piotr Dollár, Jianfeng Gao. Xiaodong Ile, Margaret Mitchell, John C Platt, al. From captions visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1473-1482, 2015.
[17] Shanghua Gao, Zhong-Yu Li, Ming-Hssun Yang. Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analyxis and Machine Intelligence, 2022.
[18] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-W Yi Lin, Open-vncabulary image segmentation. arXiv preprint arXiv:21 12143,1133322 2021.
[19] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin, Scaling open-vocahulary image segmentation with image-level lahels. In Computer VisionEECC 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Procediiggs, Part XXXVI, pages 540-557. Springer, 2022.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[21] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoe Le, Yun Hun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[22] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- semantic segmentation. arXiv preprint arXiv:2201.03444 2022.
[23] Peike Li, Yunchao Wei, and Yi Yang. Consistent structural relation learning for zero-shot segmentation. Advances in Neural Information Processing Systems, 33: 13:10777-03327, 2020.
[24] Yangguang Li, Feng Ling, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.0525
2021.
[25] Yehao Li, Yingwei Pan, Ting Yao, and Tao Mei. Comprehending and ordering semantics for image captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17990-17999, 2022.
[26] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Mareulescu. Open-wocabulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:2210. 04150, 2022.
[27] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. T Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ViC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[28] Quande Liu, Youpeng Wen, Jianhua Han, Chumjing Xu, Hang Xu, and Xiaodan Liang. Open-world semantic segmentation via contrasting and clustering vision-language embedding. In Computer Vision- ECCV 2022: 17th Earopean Conference, Tel Aviy, Israel. October 23-27, 2022, Proceedings, Part XX, pages 275-292. Springer, 2022.
[29] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440. 2015.
[30] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-W Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEFE conference computer vision and pottern recognition, pages 891-898, 2014.
[31] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xic. Slip: Self-supervision meets language- image pre-training. In Computer Vision-FCCC FCCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 229-544. Springer, 2022.
[32] Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaced, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. arXiv preprint arXiv:2212.0441 044994 2022.
[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03777 2018
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.

---

[35] Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering lex--super segmentation masks via multi-view semantic consistency. arXiv preprint arXiv:2302.10307, 2023.
[36] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernmmed image alt- dataset for automatic image captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2018.
[37] Gyungin Shin, Weidi Xie, and Samuel Alhanie. Namedmask: Distilling segmenters from complementary foundation models. arXiv preprint arXiv:2209. 2022.
[38] Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. arXiv preprint arXiv:226.00000
2022.
[39] Bart Thomce, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59(2):44-73 73, 2016.
[40] Hugo Touvron, Matthieu Cord, Mathhij Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347- 10347-10357. PMLR, 2021.
[41] Bichen Wu, Ruizhe Cheng, Peizhao Zhang. Tianren Gao, Joseph E. Gonzalez, and Peter Vajda. Data efficient language-supervisee zero-shot recognition with optimal transport distillation. In International Conference on Learning Representations, 2022.
[42] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarcz, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 10077-12900,, 2021.
[43] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang- Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144 2022.
[44] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng. Yi Wang. Yu Qiao, and Weidi Xie. Learning opea-vocabulary semantic segmentation models from natural language supervision. arXiv preprint arXiv:2301.090 091211,, 2023.
[45] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. arXiv preprint arXiv:2302. 12242, 2023.
[46] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for vocabulary semantic segmentation with pre-trained MsionnEECV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXIX, pages 736-753. Springer, 2022.
model. In Computer
[47] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive inguage-image pre-training. arXiv preprint arXiv:2111.07783, 2021.
[48] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lci Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint 2333v:2303.081 03, 2023.
[49] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. arXiv preprint arXiv:2304.0060 00085,00000000,.... 2023.
[50] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 633-641, 2017.
[51] Chong Zhou, Chen Change Loy. and Bo Dai. Extract free dense labels from clip. Computer Vision- ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pages 696-712. Springer, 2022.
[52] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang. Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:22 /200, 2022.
12


Reviews:
[{"review_id": "esG5vZbI1t", "reviewer": "Reviewer_e4vq", "summary": "Current VLMs suffer from a noticeable semantic gap between visual and textual modalities since many visual concepts present in images are easily missed in their paired captions. This work proposes Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, CoCu establishes a concept archive that maintains potential visually-matched concepts using vision-driven expansion and text-to-vision-guided ranking. This approach enables the identification of relevant concepts through cluster-guided sampling, which are then fed into the pre-training process. As a result, CoCu bridges the gap between visual and textual semantics. Extensive experiments conducted on eight segmentation benchmarks demonstrate that CoCu achieves exceptional zero-shot transfer performance and significantly enhances the language-supervised segmentation baseline by a substantial margin. These results underscore the importance of closing the semantic gap in pre-training data. The code for CoCu will be made available to the research community.", "questions": "1. In table 1, are both CoCu and GroupViT trained for 30 epochs?\n2. With the concern of Weakness 1, I am also wondering what if you change the x-axis from #epochs to actual training time in Figure 3? Will CoCu still converge faster than GroupViT?", "limitations": "As CoCu basically follows the architecture of GroupViT, it has the same limitation that for each image, the number of semantic concepts segmented by CoCu/GroupViT has a maximum of the number of group tokens fed to the vision encoder. Thus, given a high-resolution image with many semantic regions, these methods might fail to generate accurate segmentation masks.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Overall, the paper is well-written, and the study of this work has a good motivation.\n2. The mechanism of CoCu is intuitive and easy to follow.\n3. CoCu demonstrates significant empirical results: it outperforms GroupViT (re-implemented baseline) by 4.6% mIoU in average on eight popular semantic segmentation benchmarks.\n4. CoCu also yield good qualitative results, with more accurate and smooth masks than its baseline.", "weaknesses": "1. Despite the fact that CoCu indeed enriches visual concepts during VL pre-training, the cost seems to be heavy. During training, a CLIP model is employed to perform the key components of CoCu such as vision-driven expansion and text-to-vision guided ranking, which introduces a lot of additional computation. Thus, the comparison to your baselines such as GroupViT might not be strictly fair. A detailed comparison of training time or FLOPS should be presented to support the effectiveness of CoCu.\n\n2. My bigger concern lies in how CoCu relies on the pre-trained CLIP, i.e., if using a weaker CLIP model for CoCu, how will the performance change? If CoCu is not sensitive to the CLIP's performance, you can directly use GroupViT to perform vision-driven expansion and text-to-image-guided ranking, so that no additional parameters will be introduced. As shown in Table 4, GroupViT obtains acceptable zero-shot classification results. However, if CoCu highly relies on a strong CLIP, your contributions might also be challenged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We appreciate the value you see in our innovative motivation and competitive experimental results. Below, please find clarifications regarding your concerns below.\n\n**Q1: In Table 1, are both CoCu and GroupViT trained for 30 epochs?**\n\nYes, in manuscript Table 1, pre-training configurations of GroupViT [43] and CoCu remain the same to ensure a fair comparison.\n\n**Q2: Change the x-axis from #epochs to actual training time in Figure 3?**\n\nWe would clarify that CoCu is implemented in an offline manner, and its does not introduce much overhead in processing. Take the experiment on CC3M [36] dataset as an example. CoCu takes 1.6 hours in total as detailed in the table below, and the rest pre-training is similar to the pre-training in GroupViT [43] which takes around 20 hours. Hence, CoCu introduces around 8% computational overhead. We provide the time cost of each operation in Table below.\n\n|**Method**|**Step**|**Computation**|**Operation**|**Time Cost (hrs)**|**Average mIoU (%)**|\n|:-----|:-----|:-----|:-----|:-----|:-----|\n| GroupViT [43] | 1 (final step) | online | pre-train | 20.0 | 8.2 |\n| CoCu | 1 | offline | inference | 0.3 |  |\n| CoCu | 2 | offline | build index | 0.1 |  |\n| CoCu | 3 | offline | curation | 1.2 |  |\n| CoCu | 4 (final step) | online | pre-train | 20.0 | 13.1 (+4.9) |\n\nDespite the 8% additional offline time cost, CoCu brings remarkable performance gains in comparison to the baseline GroupViT [43] (+4.9% over 8 benchmarks). This supports the significance of bridging semantic gaps in language-supervised segmentation, and offers flexibility while tackling different tasks with different requirements.\n\nAs suggested, we will update a new loss curve that uses actual time as x-axis in manuscript Figure 3. The time usage will include both offline operations and online pre-training.\n\n**Q3: For curation model, replace CLIP with a weaker CLIP model and GroupViT?**\n\nTechnically, CoCu could leverage any vision-language model to perform curation, bridge semantic gaps and facilitate language-supervised segmentation. However, we would highlight that a strong curation model (e.g., CLIP [32]) is not a necessity to achieve decent performance gains in downstream datasets.\n\nTo verify this, we follow your suggestion and replace CLIP model (CLIP ViT-B/16) we used in manuscript with a weaker CLIP model (CLIP ViT-B/32) to perform CoCu, followed by pre-training a segmentor on CC3M [36]. The experiment shows that CoCu with CLIP ViT-B/32 brings comparable performance gains over baseline (8.2%) as compared to that of using CLIP ViT-B/16 (12.8% v.s. 13.1%). This demonstrates that CoCu can work with different CLIP models with consistent performance gains.\n\nTo ensure fairness to baseline method [43], we further replace CLIP ViT-B/16 with GroupViT [43] as our curation model to perform CoCu. As presented in Table below, performing CoCu with GroupViT still brings clear improvement (+3.9%) over baseline method. This indicates the significant performance gain is largely attributed to our novel pipeline design, though employing the powerful CLIP further boosts the performance. Please note that the retrieved results vary depending on different language-vision models, leading to fluctuating pre-training and segmentation results. Nevertheless, the enhancements across all models remain noteworthy, underscoring our pipeline’s inherent capacity for generalization and improvement.\n\n| **Curation Model** | **Backbone** | **PVOC** | **PCON** | **COCO** | **IN-S-50** | **IN-S-300** | **CITY** | **ADE** | **STUF** | **Average** |\n|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|\n| - | - | 15.5 | 10.4 | 6.5 | 10.2 | 2.9 | 8.1 | 4.4 | 7.7 | 8.2 |\n| GroupViT [43] (newly added) | ViT-S | 26.1 | 12.3 | 10.4 | 17.8 | 6.9 | 7.6 | 6.8 | 8.7 | 12.1 (+3.9) |\n| CLIP [32] (newly added) | ViT-B/32 | 27.4 | 14.8 | 10.6 | 16.7 | 6.3 | 10.4 | 6.3 | 9.9 | 12.8 (+4.6) |\n| CLIP [32] | ViT-B/16 | 30.6 | 13.9 | 10.8 | 19.3 | 7.3 | 8.2 | 6.1 | 8.5 | 13.1 (+4.9) |\n\n\n\n\n"}, {"review_id": "oPW52Z10D8", "reviewer": "Reviewer_griQ", "summary": "This paper targets learning unsupervised semantic segmentation from image-text pairs. The authors specifically address the issue of training data quality in previous method Group-ViT and propose an approach to enhance the captions with additional visual concepts through an automated pipeline. The paper demonstrates the effectiveness of this data filtering pipeline by evaluating it on official segmentation benchmarks.", "questions": "See weakness.", "limitations": "NaN", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The semantic gap problem the authors studied is intresting and meaning full.\n- The writing is good and the paper is easy to understand.", "weaknesses": "The proposed data filtering pipeline relies on the CLIP model for collecting visually similar samples. However, the CLIP model has been trained on a much larger scale, with hundreds of millions of image-text pairs, whereas the training data used in this paper is relatively smaller. Does it work by distilling the CLIP model? This raises concerns about **the effectiveness of the pipeline when scaling up to larger training data sizes**. It would be helpful to explore **alternative self-supervised methods such as DINO, MAE, or Group-ViT itself to replace the CLIP model** in the pipeline and evaluate its performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable suggestion. Below please find our clarification regarding your concern.\n\n**Q1: Does it work by distilling CLIP Model? Replace CLIP with GroupViT?**\n\nNo, CoCu does not rely on vast knowledge encoded in CLIP [32] but more on our novel pipeline design. To verify this, we replace CLIP with GroupViT as our curation model to perform CoCu. The quantitative comparison with our baseline is presented in the Table below. As presented, performing CoCu with GroupViT brings decent performance gains (+3.9%) over the baseline method, which is comparable with that of performing CoCu by CLIP (+4.9%). Hence, the performance gain is largely attributed to our CoCu design, though employing the powerful CLIP further boosts the performance\n\n| **Curation Model** | **Backbone** | **PVOC** | **PCON** | **COCO** | **IN-S-50** | **IN-S-300** | **CITY** | **ADE** | **STUF** | **Average** |\n|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|\n| - | - | 15.5 | 10.4 | 6.5 | 10.2 | 2.9 | 8.1 | 4.4 | 7.7 | 8.2 |\n| GroupViT [43] (newly added) | ViT-S | 26.1 | 12.3 | 10.4 | 17.8 | 6.9 | 7.6 | 6.8 | 8.7 | 12.1 (+3.9) |\n| CLIP [32] | ViT-B/16 | 30.6 | 13.9 | 10.8 | 19.3 | 7.3 | 8.2 | 6.1 | 8.5 | 13.1 (+4.9) |\n\n**Q2: Scaling up to larger training size?**\n\nWe did not perform large-scale experiments due to resource limitations. For the pre-training scale in our manuscript Table 1, it takes around 9 days with Tesla V100 GPUs and the global batch size of 1024. Increasing pre-training scales further will lead to much longer training time. Nevertheless, Table 2 of the manuscript shows how CoCu behaves under various pre-training scales, indicating that CoCu's superior performance is quite tolerant to the pre-training scale.\n\n**Q3: Replace CLIP with Self-Supervised Models?**\n\nWe should clarify that CoCu is not applicable to DINO or MAE as CoCu takes texts as inputs, which is not supported by self-supervised models that handle image inputs solely. We surely agree that it is very meaningful to explore what role self-supervised models could play in bridging semantic gaps, and we will study it in the future work."}, {"review_id": "iFhM4ydbAi", "reviewer": "Reviewer_iruH", "summary": "This paper proposed a novel data curation/argumentation process, named Concept Curation (CoCu), for language-supervised semantic segmentation. In the setting of language-supervised semantic segmentation (e.g. GroupViT), the network is trained with image-text contrastive loss on large-scale image-text pairs. Authors identified several issues in the vanilla data distribution of the original contrastive learning, e.g. semantic gap, and semantic bias. The proposed CoCu mitigates these issues and improves over prior work GroupViT under a controlled setting, showing a faster convergence rate and high accuracy. ", "questions": "1. The boundary between the semantic gap and semantic bias is not that clear. At least from Figure 1, it's hard to tell the major differences between them. Maybe authors could define them more rigorously and provide more examples about them. \n2. How long does it take to process the dataset? It would be informative for other readers who want to reproduce or follow up on the curation process. \n3. In Table 2, CoCu also showed significant improvement on ImageNet50 and ImageNet300. But for these two datasets, the vocabularies are mainly foreground, it would be very interesting to show why CoCu could improve on these datasets as well. ", "limitations": "Yes. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. In the quantitative evaluation, authors re-implemented GroupViT and compare CoCu and GroupViT under the controlled experiment setting, i.e. 1024 batch size. The authors also report the accuracies on additional evaluation datasets, IN50, IN300, Cityscapes, etc. CoCu outperforms GroupViT on all the datasets by a margin. Although CoCu doesn't achieve state-of-the-art results on some datasets, it doesn't degrade the effectiveness of the method. \n2. The proposed method is well-motivated. The web-crawled image-text pairs are indeed quite noisy. And GroupViT is also known to be bad at segmenting background classes like grass and sky. Moreover, from the visualization in Figure 3(b), the CoCu shows that could focus on background grasses instead of the foreground fox. \n3. In the Table 2 ablation study, when only trained on CC3M, CoCu improves over GroupViT by a large margin on Pascal VOC. It is a very interesting result and justifies that CoCu speeds up the convergence. ", "weaknesses": "1. Insufficient qualitative comparison. Since one major claim of CoCu is that, compared with vanilla contrastive loss used in GroupViT, the proposed dataset curation mitigates semantic cap and semantic bias issues. So besides quantitative evaluation metric mIoU, more visualizations compared with GroupViT are expected. I would suggest authors add more visualizations in the supplementary materials if there is no space left in the main submission. \n2. In the ablation table 3, the authors studied the effects of different components of CoCu. But the average mIoU may not be an insightful metric. Since different datasets have very different category vocabulary, authors may try to include a detailed ablation study table in the supplementary material and elaborate more on the results. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your value of our efforts and helpful suggestions. Please check our clarification below regarding your concerns. \n\n**Q1: More qualitative comparison.**\n\nPlease refer to the Appendix Figure 2 for additional qualitative comparisons (heatmap). The figure shows that CoCu learns visual concepts (especially not captured by captions) better than the baseline GroupViT (as presented in manuscript Figure 4). As suggested, we will provide more examples presenting better convergence to missing concepts in updated Appendix.\n\n**Q2: Ablation study table.**\n\nPlease find a more detailed ablation study table below for different evaluation datasets, where \"baseline\", \"#1\", \"#2\", \"#3\" and \"#4\" refer to the same experiments as in manuscript Table 3. As suggested, we will include this table into the Appendix for reference.\n\n|**Model** | **PVOC** | **PCON** | **COCO** | **IN-S-50** | **IN-S-300** | **CITY** | **ADE** | **STUF** | **Average** |\n|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|\n| Baseline | 15.5 | 10.4 | 6.5 | 10.2 | 2.9 | 8.1 | 4.4 | 7.7 | 8.2 |\n| #1 | 20.7 | 10.4 | 8.2 | 13.9 | 4.6 | 8.3 | 4.9 | 8.0 | 9.9 |\n| #2 | 22.8 | 11.4 | 8.6 | 13.5 | 5.1 | 8.3 | 5.8 | 7.0 | 10.3 | \n| #3 | 29.2 | 13.0 | 10.2 | 18.8 | 6.3 | 7.5 | 5.9 | 8.3 | 12.4 |\n| #4 | 30.6 | 13.9 | 10.8 | 19.3 | 7.3 | 8.2 | 6.1 | 8.5 | 13.1 |\n\n**Q3: Difference between semantic gap and semantic bias?**\n\nWe further clarify these two concepts as suggested.\n\nWe refer to semantic gap as a problem underlying in **pre-training data**. In web-crawled image-text pairs, it is prevalent that texts (i.e., captions) do not capture comprehensive visual concepts in their paired images, yielding semantic gap between the paired texts and images.\n\nIn comparison, we refer to **semantic bias** as a problem in **pre-trained vision-language models**. We notice that pre-trained vision-language models often retrieves salient visual concepts in images when taking images as inputs and attempting to retrieve texts, hence demonstrating certain bias toward salient visual concepts.\n\nWe conjecture that semantic bias in large-scale vision-language models originates from the semantic gap in pre-training data they used to pre-train the large-scale models. For semantic bias in CLIP models, semantic gap lies in the noisy WIT400M image-text pairs.\n\nWe will further define and clarify these two concepts in the updated manuscript.\n\n**Q4: Time cost to process the dataset?**\n\nFor reference, processing CC3M [36] dataset with our method takes roughly 1.6 hours (8% of pre-training time) including fast inferencing images and texts, building search index, and performing concept curation on all image-text pairs. Please refer to the table below for more details on time cost.\n\n|**Method**|**Step**|**Computation**|**Operation**|**Time Cost (hrs)**|**Average mIoU (%)**|\n|:-----|:-----|:-----|:-----|:-----|:-----|\n| GroupViT [43] | 1 (final step) | online | pre-train | 20.0 | 8.2 |\n| CoCu | 1 | offline | inference | 0.3 |  |\n| CoCu | 2 | offline | build index | 0.1 |  |\n| CoCu | 3 | offline | curation | 1.2 |  |\n| CoCu | 4 (final step) | online | pre-train | 20.0 | 13.1 (+4.9) |\n\n**Q5: Why CoCu is also effective for foreground regions?**\n\nIn Figure 3 of the submitted Appendix, we provided qualitative comparisons between the baseline method GroupViT [43] and our CoCu for segmenting ImageNet-S foreground categories. We showcased that **with CoCu pre-training, the segmentator is more robust to  changes of expressions of the same semantics in captions**, e.g., from \"dog\" to \"kuvasz\", demonstrating that CoCu leads to more effective and generalized representation learning over foreground categories.\n\nWe include a new example for segmenting foreground regions in Figure 1 in the attached **\"global-response.pdf''**. Specifically, in this figure, we segment an image captioned *helicopter fly at high altitudes* with both GroupViT [43] (first row) and CoCu (second row) pre-trained models. When we feed different text inputs (i.e., *cloud*, *helicopter* and *mountain*) to both segmentors and visualize their activations, it is clear to see that GroupViT [43] activated on *helicopter* region (highlighted with red box) with all given text inputs (i.e., *helicopter*, *cloud*, *mountain*). In such a case, the pre-trained segmentor struggled to discriminate *helicopter* from other concepts (*cloud*, *mountain*). As a comparison, CoCu successfully captured corresponding regions when given different text inputs. This is because the  pre-training in the baseline GroupVIT captures insufficient textual concepts in its visual representations, thus is easily confused by text inputs. In contrast, CoCu segmentor learns significantly more comprehensive concepts in pre-training by bridging semantic gaps and is more robust to variations of semantic context, thus localizing each text input more accurately. This further showcased that **our CoCu facilitates discriminating different contexts in the same images**, leading to better foreground segmentation on ImageNet-S-50 and ImageNet-S-300. We thank you for your valuable comments and will include this analysis in the updated appendix.\n\n"}, {"review_id": "64Tjcs3RxL", "reviewer": "Reviewer_wvpf", "summary": "This paper first points out the semantic gap problem of the existing language-supervised semantic segmentation method. It shows that not all visual elements are included in the corresponding language annotations. Then the paper proposes Concept Curation (CoCu), which includes Vision-driven Expansion, Text-to-Vision-Guided Ranking, and Cluster-guided Sampling strategies to solve the semantic gap problem. The experiments show the effectiveness of the CoCu.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The semantic gap problem sounds very reasonable and significant for language-supervised semantic segmentation.\n2. The proposed method CoCu could alleviate the semantic gap problem to some extent.\n3. The experiments and ablation study are comprehensive.", "weaknesses": "1. The authors should summarize the Sec. 2.3 at the beginning or the end of this section. For example, in general, Sec. 2.3 provides a method to find better class candidates for the loss in Sec. 2.1. Otherwise it confuses the readers about the ultimate goal of Sec. 2.3.\n2. The overall pipeline of CoCu is a kind of dataset pre-processing method, and it is complicated. Is CoCu done online or offline? If online, how much time does it cost to find the final class candidates?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for your valuable suggestions and response as follows for your concern.\n\n**Q1: Writing in Sec. 3.3?**\n\nThanks for pointing it out. As suggested, we will clarify the motivation of our method at the beginning of Section 3.3 as well as its relation with Section 3.1.\n\n**Q2: Online or Offline? Time Cost?**\n\nWe implement CoCu in an offline manner so there is no additional online time cost as compared to baseline method. For offline computation, performing Concept Curation on CC3M [36] dataset roughly takes 1.6 hours (which is nearly 8% of pre-training time following our setting) but brings +4.9% performance gains. Please refer to a table below for more details on time cost. \n\n|**Method**|**Step**|**Computation**|**Operation**|**Time Cost (hrs)**|**Average mIoU (%)**|\n|:-----|:-----|:-----|:-----|:-----|:-----|\n| GroupViT [43] | 1 (final step) | online | pre-train | 20.0 | 8.2 |\n| CoCu | 1 | offline | inference | 0.3 |  |\n| CoCu | 2 | offline | build index | 0.1 |  |\n| CoCu | 3 | offline | curation | 1.2 |  |\n| CoCu | 4 (final step) | online | pre-train | 20.0 | 13.1 (+4.9) |"}]

Author rebuttal:
Please check global_response.pdf for our new Figures.

Meta-review: The paper received 6/6/5/6 ratings. Reviewers acknowledge the problem of filling semantic gaps for language supervised segmentation is interesting, well motivated, straightforward and the paper is well presented. While some concerns such as how the CLIP model affects the performance of the method were raised, those were successfully addressed through rebuttal. The AC recommends to accept the paper.
Decision: Accept (poster)