Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Models
Authors: Monika Wysoczańska
Antonin Vobecky
Amaia Cardiel
Tomasz Trzcinski
Renaud Marlet
Andrei Bursuc
Oriane Siméoni
Venue: Accepted by TMLR ()

Abstract: Recent CLIP-like Vision-Language Models (VLMs), pre-trained on large amounts of image-text pairs to align both modalities with a simple contrastive objective, have paved the way to open-vocabulary semantic segmentation. Given an arbitrary set of textual queries, image pixels are assigned the closest query in feature space. However, this works well when a user exhaustively lists all possible visual concepts in an image that contrast against each other for the assignment. This corresponds to the current evaluation setup in the literature, which relies on having access to a list of in-domain relevant concepts, typically classes of a benchmark dataset. Here, we consider the more challenging (and realistic) scenario of segmenting a single concept, given a textual prompt and nothing else. To achieve good results, besides contrasting with the generic “background” text, we propose two different approaches to automatically generate, at test time, query-specific textual contrastive concepts. We do so by leveraging the distribution of texts in the VLM’s training set or crafted LLM prompts. We also propose a metric designed to evaluate this scenario and show the relevance of our approach on commonly used datasets.

Published in Transactions on Machine Learning Research (05/2025)
Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Models
Monika Wysoczaúska', Antonin Vobecky2.3. Amaia Cardiel??7 Tomasz Trzciński Renaud Marlet2.6, Andrei Bursuc", Oriane Siméoni²
'Warsaw University of Technology valeo. CIIRC CTU Praguet *FEE CTU Prague TTooloox 6LIGM, École des Ponts et Chaussées, IP Paris, CNRS, France 'Univessiéé Grenoble Alpes
Reviewed on OpenReview: ottps://ppaeerree...efffommmmmmm
Abstract
Recent CLIP-like Vision-Language Models (VLMs), pre-trained on large amounts of image- text pairs align both modalities with simple contrastive objective, have paved the way to open-vocabulary semantic segmentation. Given an arbitrary set of textual queries, image pixels are assigned the closest query in feature space. However, this works well when user exhaustively lists all possible visual concepts in an image that contrast. against each other for the assignment. This corresponds to the current evaluation sctup in the litorature, which relies on having access to a list of in-domain relevant concepls, typically classes of benchmark dataset. Here, we consider the more challenging (and realistic) scenario of segmenting single concept, given textual prompt and nothing clse. To achieve good results, besides contrasting with the generic "background" text, propose two different approaches to automaticalle ically generate, at test. time, query-specific textual contrastive concepts. We do so by leveraging the distribution of text in the VLM's training set or crafted LLM prompts. We also propose a metric designed to evaluate this scenario and show the relevance our approach on commonly used datasets.
1 Introduction
Vision-language models (VLMs) such as CLIP Radford et al. (2021) are trained to align text and global image representations. Recently, VLMs have been proposed for denser tasks Zhou et al. (2022 Ghiasi et al. 2022
Li et al. (2022). This includes the challenging pixel-level task of open-vocabulary semantic segmentation
(OVSS), which consists of segmenting arbitrary visual concepts in images, i.e., visual entities such as objects, stuff (c.g., grass), visual phenomena (c.g., sky). To that end, several methods exploit frozen CLIP model
with additional operations Zhou et al. (2022): Bousselhan el al. (2021); |Wyoozańska et al. (2024b a),
fine-tune the model with specific losses Xu et al. (2022) Ranasinghe et al. (2023) Cha et al.
(0023 Luo
et al. )023); Mukhoti et al. (2023).
Most OVSS methods label each pixel with the most probable prompt (or query) among finite set prompts provided as input, contrasting concepts with each other. This works well for benchmarks that provide large
and nearly exhaustive list of things that can be found in the dataset images, such as ADE20K Zhou al.)
(2099) COCO-Sture Caesar et al. (2018). However, when given limited list of queries, these methods are bound to occasionally suffer from hallucinations Wysoczañska et al. (022bb Miller et al. (2024). Tn
particular, common setups do not handle cases where only a single concept is queried Cha et al.
2023
Xu
el al. 2022), which results in classifying all pixels using the same concept.
"ooresponding author: monika. wysoczaneka dokt@pw...o.o.. edu.pl
1 CZech Technical University in Prague, Czech Institute Informatirs, Robotics and Cyhernetics

---

Published in Transactions on Machine Learning Research (05/2025)
Image
GT
query
query background"* CC
"background"
corgi
cat
Query prompts
car
bus
boat
water
Figure 1: Illustration of our proposed open-world scenario and benefits of contrastive concepts (CC). We investigate open-world spendworld segmentation, where only one (or a few) visual concepts to be segmented (2nd column), while all concepts that can occur in an image are unknown. Contrasting the query with "background" allows us to obtain a coarse segmentation Ranasinghe et al. (003): Wysoczańska el. al. (004bb (3rd column), but is not enough to catch all pixels not corresponding to the query when they are related or co-occur frequently in the VLM training set. Our antoomatically-gneeraae
contrastive concepts (CC) (4th
column) help separate and disentangle pixels of the query (right column, generated CC in text boxes), therefore achieving better segmentation.
To catch such hallucinations, a common strategy consists of using extra class labeled 'background' intended to capture pixels that do not correspond to any visual concept being queried. This extra class is already object-centric datasets, such as Pascal VOC Everingham et al. (2012). provides an easy, generic concept to be used as a negative query, i.e., to be used to contrast with actual (positive) queries but be discarded from the final segmentation. However, the notion of background is not well defined it is context-dependent, therefore providing suboptimal contrasts. This strategy also fails when queried concept (e.g., "tree") falls in the learned background (which commonly encompasses trees).
In this work, we consider a practical and realistic OVSS task in which only one or few arbitrary concepts are to be segmented, leaving out the remaining pixels without prior knowledge of other concepts that may occur in an image. We name this setup open-world! I Given a query, instead of assuming access to a dataset-specific set of classes (a closed-world sctup), we propose to automatically suggest contrastive concepts useful better localize the queried concept, although they can later be ignored. In particular, we focus on predicting concepts likely to co-occur with the queried concept, e.g., "water" for the query "boat" (as visible in Fig. [ thus leading to better segment boundaries when prompted together.
Moreover, we argue that this scenario needs to be evaluated lo understand the limitations of open-vocabulary segmentation methods better. We therefore propose a new metric to measure such an ability, namely loU-single, which considers one query prompt at a time and thus does not rely on the knowledge potential domain classes.
We distinguish setup from open-world/ open-set setting known from literature Wu et 2024), where segmentation model identifies novel classes and marks them "unknown". Here, consider the task of open-world gpen-vocabulary segmentation, thus considering only OVSS models, where the goal is segment queried concepts unknown test time and leave the remaining pixels an image with class. For the full name our setup, we thus consider open-wone open- veen-vocnyylary segmentation but keep open-world throughout the rest of the work for brevity.
2

---

Published in Transactions on Machine Learning Research (05/2025)
To summarize, our contributions are follows:
We introduce the notion of test-time contrastive concepts and discuss the importance of contrastive concepts in open-vocabulary semantic segmentation.
We analyze the usage of "background" as test-time contrastive concept, which has been accepted but not discussed so far.
We propose a new singlo-query ovaluation setup for open-world semantic segmentation that does not. rely on domain knowledge. We also propose a new metric to evaluate the grounding of visual concepts.
We propose two different methods to generate test-time contrastive concepts automatically and show that our approaches consistently improve the results of 7 different popular OVSS methods or backbones.
2 Related work
VLMs trained on web-collected data to produce aligned image-text representations Radford et al. (2021); Jia et al. (2021); Zhai et al. (2023) had major impact open-vocabulary perception tasks and opened up new avenues for research and practical applications. While CLIP can be used off-the-shelf for image classification in different settings, il does not produce dense
Open-vocabulary semantic segmentation.
pixel-level features and predictions, due to its final global attentive-pooling Zhou et al. (2022); Jatavallabhula ct al. (023). To mitigate this and produce dense image-text features, several methods fine-tune CLIP
with dense supervision Cho et al. (2024); Xu el al. (2023b Zheng Ding (2023); ;; Wu el al.
2023) Other
approaches devise new CL.IP-like models trained from scratch using a pooling compatible with segmentation.
Their supervision comes from large datasets annotated with coarse captions Ghiasi al. (2022); Ranasinghe
et al. (0023; ; Liang et al.
(2023
; Xu et al. (0022): Liu et al. (0022] Xu et al. (2023a) Mukhoti et al. 2023
Cha et al. 2023)) object masks Rao et al. [2022]; Ghiasi et al. [2022] Ding et al. (2022) or pixel labels
Li et al. (2022); Liang et al. (202)). However, when models are fine-tuned, they face feature degradation
Satavallabhui el al. (202)), or require long training cycles on large amounts of images when trained from
scratch.
CLIP densification mothods have emerged as a low-cost altornative to produce pixel-level image-text features while keeping CLIP frozen Zhou et al. (2022); Wysoczańska et al. (202-aa; Jatavallabhula et al. 2023 ;
Abdelreheem et al. (2023) Wysoczaíska et al. (2024b) Bousselham et al.
(024). The seminal MaskCLIP
Zhou et al. (2022) mimics the global pooling layer of CLIP with a 1 x 1 conv layer. The aggregation of fcatures
from multiple views and crops Abdelreheem et al. 2023) Kerr et al.
(0023) Wysoczarisk et al. 2024a] |;;
Jatavallabhula et al. (202)) also leads to dense features, yet with the additional cost. of multiple forward
passes. Some methods Shhin et al. 2022 2023); Karazija et al. (2023) rely on codebooks of visual prototypes
per concept, including per-dataset negative prototypes Karazija al. (2023). or leverage self-self-atteniine to create groups of similar lokens [Bouseeham et al. (2024). The recent CLIP-
Wysoczańska et. al. (2024b) improves MaskCLIP features with limited computational overhead thanks to guided pooling strategy that leverages the correlation information from DINO features Caron et al. (0021).
Prompt augmentation. Prompt engineering is a common practice for adapting Large Language Models (LLMs) to different language tasks Kojima et al. (2022) without updating parameters. This strategy of carefully selecting task-specific prompts also improves the performance of VLMs. For instance, in the original CLIP work Raaffrd et al. (2021), dataset-specific prompt templates, e.g., "a photo of the nice {...}" were devised towards improving zero-shot prediction performance. Although effective, manual prompting can be laborious task, as templates must be adapted per dataset and sufficiently general to apply to all classes. Afterwards, different automated strategies were subsequently explored, e.g., scoring and ensembling predictions from multiple prompts Allingham et al. (2023). Prompts can also be augmented by exploiting semanntc relations between concepts defined in WordNet, Fellbaum (1998) to generate new coarse/ /fne-grained Ge et al. (2023) or synonym Lin et al. (2023) prompts. LLMs can be used as a knowledge base to produce rich visual descriptions adapted for each class starting from simple class names Pratt al. (2023); Menon &
Vondrick (2023). Prompt features can be learned by considering visual co-occurrences Gupta et al. (2019), a connection between training and test distríbutions Xiao et al. (2024), mining important features for the VLM
3

---

Published in Transactions on Machine Learning Research (05/2025)
Esfandiarpoor et al. (2024) or by test-time tuning on a sample Shu et al. 2022 Most of these strategies have been designed and evaluated for the image classification task, and their generalization and scalability for semantic segmentation are not always trivial. Here, we aim to obtain better prompts for semantic segmentation to separate queried object pixels from their background. We do this automatically without supervision and without changing the parameters of either the text encoder or the image encoder, leveraging statistics from VLM training data or LLM-basod knowledge.
Dealing with contrastive concepts in OVSS. Our contrastive concept discovery is tightly related to background handling in the context of open-vocabulary semantic segmentation, since the standard benchmark datasets for this task, originally designed for supervised learning, use background to describe unlabeled pixels, for example, to cover concepts outside of the dataset vocabulary. There are three main types of approaches
to address this problem. The first one is to threshold uncertain predictions Cha et al. (2023): ; Bousselbam
ct al. 20224) Xu et al. (2022) with a given probability value Xu et al. (2022) Bousselham et al. (024) or clip
similarities Cha et. al. )023)- The second group of methods leverages the object-centric nature of certain
datasets by defining background through visual saliency Wysoczańska et al. (2024a b). Finally, a significant body of work addresses the same issue by defining dataset-level concepts either by adding handcrafted names of concepts to the background definition Lin et al. (2024): Yu et al. (2023); [Ranasinhhe et al. (2023): Cho ct al. (2024) or by extracting visual negative prototypes with large diffusion model Karazija et al. (2023). In contrast, in this work, aim for automatic discovery of contrastive concepts without prior access to the vocabulary used to annotale the dataset.
Visual grounding is the task of localizing within images specific objects from text descriptions. The major instances of visual grounding tasks are referring segmentation that produce pixel-level predictions for one Hu et al. (2016): Ding et al.] (2023) : Wang et al. (2022) or multiple target objects Liu et al. (2023) given a text description, and referring expression comprehension Chen et al. (2018) Deng et al. (2021) Liao et al. 2020); Liu et al. (2024) that detects objects. Similarly to referring segmentation, we aim to segment specific user-defined objects. In contrast, we do not use supervision to align textual descriptions with object masks
and do not focus on text-described relations between objects and mine contrastive concepts to disentangle target objects from the background.
3 Open-world open-vocabulary segmentation with test-time contrastive concepts
We consider the following segmentation task: given an image and a set of textual queries characterizing different visual concepts, the goal is to label all pixels in the image corresponding to each concept, leaving out unrelated pixels, if any. Moreover, we want to do so without any prior knowledge of the concepts that could be prompted at the test time. We do not only want to be open-vocabulary in terms of the choice of words for querying, but we also want to be open-world, not specialized in given domain or set of categories. For evaluation purposes, segmenting a specific dataset thus shall not assume anything about the dataset, such knowledge of represented classes.
3.1 Introducing the use of test-time contrastive concepts
Closed-world vs open-world open-vocabulary semantic segmentation. Even when it is open- vocabulary, traditional semantic segmentation is closed-world in the following sense. Given an RGB image IER#*WX3 and set of textual querics q Q, semantic segmentation produces a map Seloww : (I....H x [1..WW) Q, where each image pixel has to be assigned one of the queries as label. In contrast, open-world segmentation considers the queries. The segmentation map, in this case, is then Sopenw : (1...H) x (1...W) + QU(L). For instance, to label a boat, it is enough to ask for the "boat" segment; other pixels (sky, sea, sand, rocks, trees, swimmers, etc.) are expected to be labeled 1 and thus ignored.
additional dummy label 'I' to represent any visual concept that is different from
In the following, we show how to use any ooca-locyy segmenter in an open-world fashion. We only assume that the segmenter uses CLIP-like architecture with text encoder, noted Φr(+), used to extract textual features (g( d for any query g, where d is the feature dimension. Patch-level features Φv (T) E hxx
4

---

Published in Transactions on Machine Learning Research (05/2025)
are generated using the visual encoder, noted (v(+), where h h-H/P, w= w=W/P, and P is the patch size. The cosine similarities wetween each query feature and patch feature are then used as logits when upsampling to obtain pixel-leve predictions. It yields a closed-world segmentation, given our definition above.
From such segmentation, open-world segmentation could be derived by assigning a pixel (or patch) to a query if the cosine similarity between the visual and query embedding is above a given threshold. However, practice, it has been commonly observed that the CLIP space is not easily separable Miller et al. (2024), thus making the definition of such threshold difficult without overfitting the query datasets [oousseham et al. 2024
Cha et al. 2023). We further discuss the separability of CLIP patch features in Appendix C.3
Train-time contrastive concepts. Cues to separate visual concepts without supervision primarily come from data where these concepts occur separately and are described in their captions. If some concepts always co-occur, they are harder to be told apart. This applics in particular to OVSS models trained only from captioned images rather than from dense information. Sharing a caption pushes their embedding to align on a common textual feature, which in turn tends to bring the visual embeddings closer together. Still, such frequently co-occurring visual concepts can often be separated in a closed-world setting: pixels (or patches) are then just mapped to the query with which they align the most. However, a problem arises if visual concept of query q can be mistaken for another visual concept present in the image but not queried (e-g., querying "boat" but not "water" as in Fig.
Test-time contrastive concepts. To address this problem, we propose to use one or more additional textual queries of visual concepts that are likely to contrast well with q. For example, when querying "boat", we want to add the query "water". We name such queries test-time comtrastive concepls and nole them CCq- We further propose different solutions to automatically generate CCy, and such without assuming prior access to the image domain. Given prompt queries {q} U UCC,, we perform closed-world segmentation and assign to the dummy label 1 any patches that are labeled CC..
Multi-query segmentation. This principle can be generalized to several simultaneous queries Q, with [ > 1, considering the union of their contrastive concepts CCq = UyeQ CC2. Open-world multi-query segmentation consists in segmenting Q QU CCQ, and ignoring pixels not assigned to the querics in Q, as in the single-query case. However, some queries in Q may already contrast with each other, which puts them in competition with the set of contrastive concepts CCq and could lead to their elimination when pixels labeled in CCq are discarded. To prevent it, we propose to exclude contrastive concepts CCQ that are too similar to queries Q, c.g., with a cosine similarity of text features above some threshold 3: CCo = Uy' E CC,, | ΦT(4") (r(g) < B). In the following, for simplicity, we only consider the single-query scenario, where [Q] =1=1 1.
Morcover, to our knowledge, none of the evaluation benchmarks currently used for OVSS allows us mcasure the effectiveness of such CC. We therefore propose variant of the traditional evaluation metric for semantic segmentation and discuss it in detail in Sec. 4.1
3.2 Contrasting with "background" (CCBC)
In recent work Ranasinghe et al. 2023 Wysoczańska et al. 20024a the word "background" has been used to try to capture a generic visual concept to help segment foreground objects, separating them from their background. In our framework, amounts to defining " "background" as a test-time contrastive concept to any query g. In other words, it defines CC ("background").
However, if the word "background" feels natural to us, it is not obvious why it should also make sense in the CLIP space. This formulation is not contextual, meaning that the contrastive concept is not specific to the query, which might be suboptimal. Worse, the "background" samples from which CLIP learned could accidentally include the visual concept of the query, making the query representation close to the background representation and defeating the contrast mechanism.
We investigate the occurrence of "background" in VLM training data to sort it out. First, we use the metadata provided by Udandarao et al. (2024), which describes the representation of four thousand common concepts in
5

---

Published in Transactions on Machine Learning Research (05/2025)
LAION-40U0 Schuhmann et al. (2021 which is subset of the web-crawled LAION-2B dataset Schuhmann et al. 2022 used to train CLIP. In Fig. [a] we plot the frequency of occurrence of "background" among other VOC class names. We observe that "background" is significantly more frequent than all other words, hinting that it is widely available in CLIP training data and in general web-crawled data.
sofa
person
horse
chair
car
boat
bird
bicycle
background
0.3
0.6
0.9
-10-2
(a) Freq. of VOC concepts.
(b) "background" in caption
(c) "in the background" in caption
Figure 2: Statistics about "background" in metadata of web-crawled datasets. (a) Frequency of
some of the concepts from VOC dataset in LAION-400M caption samples. Examples of images in web-crawled data with a caption including the words "background" (b) or "in the background" (c).
Fig. 2b shows images sampled from the LAION dataset with a caption containing "background", We observe that they display a high diversity in colors and textures. Images captioned with "in the background" (Fig. 2c appear more photo-oriented. We believe that the combination a high frequency of the "background" word in the dataset and the diversity of associated images make good generic contrastive concept and hence make BC baseline. However, superior results have been obtained by applying well-desigued tricks to
handle the background Wysoczańska al. (2024a] b): Cha et al. (2023): Bousselham et al. (2024. emphasizing the necessity of applying something more than simply "background"
An option is to define a generic background class list, as done by CLIPpy Ranasinghe el al. 2023 or CAT-Seg Cho et al.) (2024), which adds to the concept "background" fixed list of concepts potentially appearing the background, e.g. "sky", "forest", "building", to be discarded. First, since these visual concepts are intended be discarded, would not be possible query them. Second, such list is defined at the dataset level, making it domain-specifie. As it impossible to exhaustively describe all visual concepts appearing in any "background" (without prior knowledge of the domain dataset), propose generating such complements specifically per query, discussed below.
3.3 Automatic contrastive concepts (CC) generation
To generate contrastive concepts that are query-specific but also domain-agnostic, the only data then leverage are (i) the VLM's training data, (ii) unspecific external data. As we focus on text-based contrasts, we can (i) exploit the large vocabulary of concepts used for VLM training or (ii) generate prompts via an LLM. Finally, as we want good contrasts, we must find hard negatives. These are concepts that surround queries in images. To gather them, we can (i) look for word co-occurrences in training data or (ii) ask an LLM to list such concepts. Sec. 3.3.1 investigates option (i), and Sec. 3.3.2 option (ii) and Fig. presents a high-level overview of both approaches.
3.3.1 Mining co-occurrence-bant contrastive concepts (CC")
As discussed above, ambiguity in segmentation for unsupervised approaches arises from co-occurrences training data. Yet, OVSS does better when prompted to create segments simultancously for co-occurring concepts. To list contrastive concepts specific given query q, we propose thus use the information of co-oceurrence in the VL. training captions. For efficiency, construct offline co-oceurrence dictionary,

---

Published in Transactions on Machine Learning Research (05/2025)
Iwo alternative approaches contrastive concepts (CC)
LLM-based CCs
CC bus, building, train, person, bicycle, road,
input image
LLM prompi
"Generate a list of words fhst surounding""
car
online generation
text queries
CCL
LLM
}
object prompt contrastive concepts
Open-Vocabulary Segmenter
car CC CC
dataset-basee CCs
CC motorbike, woman, truck, light.
offline pre-processing
bus
segmentation
bus, bike, road,
00-occurennce lable
result
online generation
road
car
CC
road
bus
background
Figure Overview of our method. We propose two solutions g one (top-left) based on LLM prompting (CC*) and the second one, CC that relies on the distribution of co-occurring concepts in pre-training dataset of VLM (bottom-leftt. Both methods can be effectively integrated into various open-vocabulary segmentation methods.
generate CC automatically, the first.
built for a large lexicon of textual concepts extracted from the captions. We note CC the co-occurrencce-assel contrastive concepts we extract for a query g based on this lexicon.
Co-occurrence extraction. We consider as lexicon a set of textual concepts T extracted from captions of the VLM training dataset and construct the co-occurrence matrix X x(TI, Concretely, two concepts (i,s) T co-occur if they appear simultaneously in the caption of an image. Xi,j counts the number of times concepts {i,}} co-occur in some images. Next, normalize the symmetric matrix X row-wise by the number of occurrences of concept in the dataset, producing the frequency matrix X. We then consider only concepts with frequent co-oceurrences: for each E T, we select concepts T, = i E T & > 7), for some frequency threshold Selecting only a few contrastive concepts in this way is also consistent with the fact. that we target online segmentation: we need to be mindful of computational costs.
Concept filtering. To improve the quality of selected contrastive concepts T;, we design a simple filtering pipeline. For each target concept. i T (which can be considered a future query), we remove from any concept that might interfere with i and induce false negatives. First, we discard uninformative words captions: ("image", "photo", "picture", "view"). Then, remove abstract concepts, such as "liberty". To do so, ask an LLM whether given word can be visible or not in an image (more details in Appendix D.2p. We also filter out concepts that are too semantically similar to target concept i, e.g., such that their cosine similarity with @i(1) is more than a threshold 8. We also consider an alternative approach to filtering, which uses the structured ontology WordNet Fellbaum (1998) to remove the CCs that possibly interfere with q. However, our experiments, which are discussed Appendix C.5 show that our proposed filtering mechanisms based dataset statistics are more effective.
Generalization to arbitrary concepts. So far, we discussed how select contrastive concepts CC, for target concept i T. Now, when we given arbitrary textual query q, to make the generation of
7

---

Published in Transactions on Machine Learning Research (05/2025)
contrastive concepts truly open-vocabulary, we first find in the CLIP space the nearest neighbor i of q in T and then use for y the contrastive concepts of i: CC2 = CCP.
3.3.2 Prompting an LLM to generate contrastive concepts (CC*)
Instead of extracting contrastive concepts from the VLM training set, here investigate another strategy, generating them using an LLM. For a given text query 9, we ask an LLM to directly generate contrastive concepts CC without the need for subsequent filtering. To that end, we design a prompt that excludes potential synonyms, meronyms (e.g.. "wing" for "plane"), or possible contents (e.g., "wine" for "bottle"). We present a shorter version of the prompt Fig. and include the complete version in Appendix D.2
You helpful Al assistant with visual abilities. Given input object want you generate list of words related to objects that can be surrounding input object O in an image to help me perform semantic segmentation.
Figure 4: An abbreviated version of the prompt we use to generate CC
Using an LLM has the benefit of producing specific contrastive concepts CCg for any target query g, without returning to a fixed and practically limited lexicon.
4 Evaluation
4.1
Evaluating open-world segmentation
We discuss here our evaluation protocols and present our new metric ToU-single specifically designed to evaluate open-world segmentation.
Evaluation datasets. We conduct our experiments on six datasets widely used for the task of zero-shot semantic segmentation Cha et al. (2023), fully-annotated COCO-Stuff Cacsar et al. (2018), Cityscapes Cordts ct al. 2016) and ADE20K Zhou et al. (2019) and object-centric VOC Everingham et al. (2012). COCO-Object Caesar al. (2018) and Context Mottaghi et al. (2014), when considering "background" pixels. We treat the input images following the protocol of Cha et al. (2023 which detail in Appendix
Our IoU-single metric. To better evaluate the ability of method to localize a visual concept when given no other information, we propose the ToU-single metric. It modifies the classic IoU by considering cach concept independently and then averaging. Concretely, we individually segment each class annotated in the dataset for the considered image, thus with |Q] = The loU-single is then the average of each IoU with the corresponding ground-trune -tuun class segment. We illustrate this metric in Fig. 5 and provide its pseudo-code in Appendix A.3 If dataset contains a background class, we do not consider it in the mIoU calculation.
Classic mIoU evaluation. We also evaluate the impact of using our CC in the classic mIoU scenario on the datasets that consider "background" a class, i.e., VOC and COCO-Object. We prompl once all dataset classes together with their CCs, using multiple-quety strategy discussed in 3.1 We then assign pixels that fall into any of the CCs to "background", ensuring that none of the concepts competes with the dataset queries. It allows us to verify if our CCs can act as background without hurting the performance on foreground classes.
4.2 Evaluated methods
Test-time contrastive concepts. For CC generation, use the statistics gathered by Udandarao et al. (0244) for four thousand common concepts in the LAION-40M datasct, which is subset of LAION-2B Schuhmann el al. (2022) and which is used to train CLIP Radford el al. (2021). We filter contrastive concepts using a low co-occurrence threshold y = 0.01 and a high CLIP similarity threshold sccnario, we use threshold = 0.9 account for possible similaritics between one query and contrastive concepts close the other queries. We discuss the selection of these values in Appendix C.1 To generate CC the recent Mixtral-8x7B--nnst model Jiang et. al. 2024). More details about the set be found
0.8. In the classic mloU
8

---

Published in Transactions on Machine Learning Research (05/2025)
A
single -IIoU Targets
A
Targets
B
mIoU
A
A
C
Predictions
Predictions
A
A
A
C
input prompts:
A, B, C
complements of A
complements
complements of C
dataset-driven mIoU
single-mIoU
Figure Illustration of IoU-single metric. We show the difference with the standard mloU metric (dataset-driven mloU), where all the concepts present image considered at once. On the contrary, our loU-single considers each of the present concepts separately to measure the single-class segmentation ability of open-vocabulary semantic segmenters.
in Appendix D.1 alongside our designed prompts in Appendix D.2 we include "background" in all CC's: CC < ("background" U UCC and CC + ("background)) UCC
In our experiments, unless stated otherwise,
Baselines. To evaluate the impact of using contrastive concepts, experiment 6 popular stato-of- the-art methods, one of which (MaskCLIP) uses 3 different backbones, thus resulting in 8 different segmenters, which we believe represent the current OVSS landscape. Concretely, we study two training-fres methods
that directly exploit the CLIP backbone, namely MaskCLIP Zhou et al. (2022) and GEM Bousselham et al. (2024), where MaskCLIP may exploit different OpenCLIP backbones Ilhacoo et al. (2021 pre-trained either on LATON Schuhmann et al. (2022), MetaCTIIP Xu et al. (2024), or by default on the original Open training data [Radfodd al. (2021). We also include TCL Cha et al. (2023). CLIP-DINOiser Wysoczańska et al. (2024b) and supervised methods: CAT-Seg Cho et al. 2024] and SAN Xu et al. (002bb) Details on the
evaluation ion protocol, including background handling strategies, can be found in Appendix methods use CLIP ViT-B/16.
All compared
4.3 Contrastive concepts generation results
We first present in Tab. D results obtained with our IoU-single metric 3 datasets, namely ADE20K, Cityscapes and VOC. We compare results when using different CC's proposed in this work. We also include results when having access to privileged information (Cc*"), i.e. the list of concepts present in images as given by the evaluation dataset. More results can be found in Appendix Tab.
"Background" is not enough. We start by analyzing the overall impact of our proposed CCs. In all cases, we observe significant improvement when using contrastive concepts CC and CCL compared to the CC Even for object-centric centric VOC where BC already provides a strong baseline, our proposed CC generation mothods bring significant gains ranging from 0.7 to 16.7 points. Interestingly, test-time CCs also work well for supervised CAT-Seg, showing that our method is beneficial for open-vocabulary segmenters with all levels of supervision.

---

Published in Transactions on Machine Learning Research (05/2025)
CLIP training data
VOC
Cityscapes
ADE20k
Method
CC
CC
cC
CC
CC CC
CC
CCRG
CC
CC
CCP!
Mas
OpenAl
44.2
52.2
53.4
15.0
22.5
22.0
30.6
20.2
23.5
25.2
29.8
DINOiser
LAION-2B
59.3
63.1
64.7
23.2
30.6
27.3
36.0
28.9
29.7
31.6
35.5
TCL
TCL's
52.9*
52.6*
53.6*
9.8
26.3
22.0
29.7
14.9*
25.9
26.5
32.6
GEM
MIIP
48.6*
61.3*
64.6*
*
14.5*
21.5
14.6
20.6
21.5*
26.3
29.1
33.0
SAN
OpenAl
50.2
73.4
69.5
19.9
37.6
32.0
44.2
24.5
35.2
35.1
42.8
CAT-Seg
OpenAI
52.8
69.5
67.7
25.7
38.4
39.7
46.8
Table 1: Benefits of CC measured in IoU-single. indicates that the method's original background
handling is applied, if any, and provided it gives the best results. Note that CAT-Seg input resolution is 640x640, whereas it is 448x448 for all the other methods. We note CC the unrealistic setup where we have access to all of the dataset classes and use them as systematic contrastive concepts (except for VOC, as its annotations do not cover all pixels). Please note that CC is our basclinc.
cc generalize better to domain-specifie datasets. For both VOC and ADE20K, the co-occurrence- based CC outperforms most of the time the LLM-based CC with a margin ranging from 0.6 to 2.8 points. However, this trend does not hold for Cityscapes, where CC gives the best results for all methods. In particular, Cityscapes a dataset of urban driving scenes that contains images depicting a few recurring concepts. This may suggest that LLMs can produce better results than CC for such domain-specific tasks. We also note that CC generally produces fewer CCs, but we do not observe a correlation between segmentation performance and ICC |, as shown in Appendix C.2
Test-time concepts are different from train-time concepts. We also observe that rrsults overall do not exceed 50% mloU. The segmentation quality might thus be limited by the VLM capacity or by a mismatch between the dataset classes and the training data. Well-designed prompt engineering could help address this issue Roth et al. (2023) and improve segmentation results.
Classic mloU evaluation. Additionally, in Tab. 2 we present results with the standard mIoU for MaskCLIP (with LAION-2B backbone) and GEM. We report results with various contrastive concepts (CC) and the original background handling strategy when applicable. We observe that in all cases, the results with CC and CC² are better than baseline CC We also notice that for GEM the results are better than when applying the background handling strategy originally proposed in [ousselham et al. (2024). This shows that integrating our contrastive concepts does not hurt or can even improve performance in the classic mIoU setup. We provide more results in Tab. in Appendix.
Method
Bkg.
Object
VOC
Coaa
17.8
35.1
MasKCLIP
Cet
CCP
25.1
46.1
threshold
27.4
46.6
GEM
CC'
35.7
60.0
CCP
35.5
Table 2: mIoU results.
4.4 Ablation studies
CCD concept filtering. In Tab. 3a we analyze the impact of the different filtering steps discussed in Sec. 3.3.1 on the challenging ADE20K dataset. We observe that each step boosts results by removing noisy or detrimental concepts. The largest gain obtained when filtering highly similar ('sem.sim.) concepts. We also note that the improvement is consistent for all methods. We report the performance without the co-occurrence thresholding (w/o 'co-oce.) and observe significant. degradation. More experiments in Appendix C.5 suggest that antology-based filtering (e.g., using WordNet) does not help and can even be harmful.
Adding "background" to CCL In Tab. 3b we study the influence of adding the word "background" the set of contrastive concepts Cc generated with the LLM. We observe that it always beneficial, in most cases with little gain, except ADF20k where the gain is up to 2.2 ToU-single pts.
10

---

Published in Transactions on Machine Learning Research (05/2025)
C0-
no abs.
sem. sim.
Mask
TCL DINO
Cityscapes w/o
ADE20k
MaskCLIP w/ CLIP
VOC
occ.
CLIP
iser
Method
w/
w/o w/
training set
CC
cc CCD
20.2
22.4
23.9
MaskCLIP
22.3
22.5
22.5 23.5
20.9
23.2
25.5
DINOiser
30.3
30.6
27.5 29.7
LAION-2B
47.9
51.8
53.8
18.4
20.0
26.3
TCL
26.0
26.2
25.4 26.3
OpenAl
44.2
52.2
53.4
25.2
26.0
31.6
GEM
21.3 21.4
25.7 26.1
MetaCLIP
46.8
50.6 50.0
(a) Impact of filtering in CCP on (b) Adding "background" not (c) Impact of pre-training dataset. ADE20K (%loU-single).
to our LLM-based CC
on VOC (%loU-single).
Table 3: Ablation studies. (a) The impact of filtering steps: 'co-oce.' is the co-occurrence- based filtering: 'no abs.' is the removal of abstract concepts; 'sem. sim.' is the semantic-similarity filtering. (b) Relevance of adding "background" to CC (c) Varying the pre-training dataset.
Impact of the pre-training dataset. Tab. 3c shows the results of MaskCLIP with different datasets used to train CLIP. We observe that using CCD always gives boost over using "background" alone (CCBG) across all pre-training datasets, including the highly-curated MetaCLIP. However, notice that for MetaCLIP, cc gives even better results, suggesting that leveraging LLMs can also be more profitable with backbones pre-trained on carefully curated datasets.
4.5 Qualitative results
In Fig. we present qualitative examples when using different contrastive concepts proposed in this work. We compare Cc and CC with ground truth (GT) and basclinc CCBG, For both CC and CC we present the output segmentation mask for the queried concept logether with its contrastive concepts (noted all) as well the single queried concept (noted single), where CCs are discarded. We observe that the output masks accurate, removing the noise from related concepts, c.g. "tree" for the
produced by methods
bird or "sofa" for the "bed".
GT
CCBa
CC (single)
CCD (all)
ceL (single)
ca (all)
sufa
pillow
furriture
miphtstand
ceiling
Wall
wall
floor
trea
poster
aeroplane
airport
airport
Figure 6: Qualitative results. We show segmentation examples from ADE20K (1st and 2nd row) and Context (3rd and 4th row), with segments produced by CLIP-DINOiser. For CCD and CCL, we additionally show the joint segmentation of all contrastive classes (all).
11

---

Published in Transactions on Machine Learning Research (05/2025)
MaskCLII
CLIP-DINOiser
CC
(single)
cC
(all)
CCD (single)
CCD
(all)
table
time
cup
q: mostin iic T:
pastry
Bark
Fard
G cavalier dog
for query concepts beyond T. The closest neighbor to query is presented below each example (grey row).
Figure 7: In the wild examples. We visualize results for MaskCLIP and
Generalization to arbitrary concepts. Fig. 7 presents results when prompting queries that are not included in the subset of concepts T extracted from the VLM training dataset, such as "muffin" or "cavalier" (a dog breed). We show the closest neighbor for the query q below each example and visualize masks for both MaskCLIP and CLIP-DINOiser. We observe that the CC generation method leveraging statistics from pro-training datasots is also robust examples outside the co-occurrence dictionary by accurately mapping g to its closest concept in T, e.g., mapping "cavalier" to "dog".
5 Conclusion
In this work, identify limitations of the current evaluation setup for open-vocabulary semantic segmentation tasks, which are inherited from closo-world evaluation benchmarks. To bridge the gap between closed- and open-world selups, we propose the single-class segmentation scenario. We study the limitations current state-of-the-art models when we assume no prior access to in-domain classes and propose to automatically discover contrastive concepts CC that are useful to better localize any queried concept. To do so, we propose two methods leveraging either the distribution of co-occurrences in the VLM's training set or an LLM generate such CC. Our results show the generalizability of our proposed method across several setups.
Broader Impact Statement. In this work, we leverage statistics extracted from the training set of CLIP. While Vision-Language Models offer powerful capabilities for visual understanding, their reliance on large-scale internet-scraped datasets introduces significant. risks and ethical concerns. These models perpetuate and amplify societal biases present in their training data. Researchers and practitioners must, therefore, carefully consider these ethical implications when developing and deploying VLM-based systems, implementing mitigation strategies, and being transparent about the limitations and potential risks of their applications.
Acknowledgments
This work was supported by the National Centre of Science (Poland) Grant No. 2022/45/B/ST6//022 and by the grant from NVIDIA providing one RTX A5000 24GB used for this project. We would also like thank the authors of Udandarao et. al. (2024) for sharing their metadata.
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Ahmed Abdelrcheem, Ivan Skorokhodov, Maks Ovsjanikov, and Peter Wonka. SATR: Zero-shot semantic segmentation of 3D shapes. In ICCV, 2023.
James Urquhart Allingham, Jie Ren, Michael W Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Takshninarayanan. simple zero-shot prompt weighting technique to improve prompt. ensembling in text-image models. In ICML, 2023.
Walid Boussclham, Felix Petersen, Vittorio Ferrari, and Hilde Kuchne. Grounding everything: Emerging localization properties in vision-language transformers. In CVPR, 2024.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In CVPR, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging propertics in self-supervised vision transformers. In ICCV, 2021.
Junbum Cha, Jonghwan Mun, and Byungscok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo. Real-time referring expression comprehension by single-stage grounding network. arXiv preprint arXiv:1812.084 084.222,... 2018
Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. CAT-Seg: Cost aggregation for open-vocabulary semantic segmentation. In CVPR, 2024.
MMSegmentation Contributors. MMSogmentation:
OpenMMLab semantic segmentation toolbox and
benchmark, 2020.
Marius Cordts, Mohamed Omran, Scbastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding, In CVPR, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Foi. ImageNet: A largo-scale hierarchical image database. In CVPR, 2009.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. TransVG: End-to-end visual grounding with transformers. In ICCV, 2021.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. In NeurIPS Datasets and Benchmarks, 2021.
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. VLT: Vision-language transformer and query generation for referring segmentation. TPAMI, 2023.
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR, 2022.
Reza Esfandiarpoor, Cristina Menghini, and Stephen H Bach. If CLIP could talk: Understanding vision- language model representations through their preferred concopt descriptions. In EMNLP, 2024.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results, 2012.
Christiane Fellbaum. WordNet: An electronic lerical database. MIT press, 1998.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Takshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robustness of multi-modal models. In CVPR, 2023.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling voca-uacyy image segmentation with image-level labels. In ECCV, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Alunad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Tanmay Gupta, Alexander Schwing, and Derek Hoiem. ViCo: Word embeddings from visual co-occurrences. In ICCV, 2019.
Ronghang Hu, Mareus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In ECCV, 2016.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongsook Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, July 2021.
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, et al. ConceptFusson: Open-set multimodal 3D mapping. In RSS, 2023.
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulmier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint 2xXi:2240..4088, 2024.
Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. In ECCV, 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LERF: Language embedded radiance fields. In ICCV, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NewrIPS, 2022.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In ICLR, 2022.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted CLIP. In CVPR, 2023.
Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. A real-time cross-modality correlation filtering method for referring expression comprehension. In CVPR, 2020.
Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofci Hc. CLIP is also an efficient segmenter: text-driven approach for weakly supervised semantic segmentation. In CVPR, 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Yuqi Lin, Minghao Chen, Kaipeng Zhang, Hengjia Li, Mingming Li, Zheng Yang, Dongqin Lv, Binbin Lin, Haileng Tiu, and Deng Cai. TagCLIP: A local-to-global framework to enhance open-vocabulary multi-label classification of CLIP without training. In AAAI, 2024.
Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized referring expression segmentation. In CVPR, 2023.
Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open-world semantic segmentation via contrasting and clustering vision-language embodding. In ECCV, 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-sel object detection. In ECCV, 2024.
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. SegCLIP: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In ICML, 2023.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
Dimity Miller, Niko Sünderhauf, Alex Kenna, and Keita Mason. Open-set recognition in the age of vision- language models. In ECCV, 2024.
Roozbeh Mottaghi, Xianjic Chen, Xiaobai Liu, Nam-Gyu Cho, Scong- Scong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. Tn CVPR, 2014.
Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In CVPR, 2023.
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does a platypus look like. In ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. In ICCV, 2023.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. DenseCI.IP: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In ICCV, 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtere 400 million imago-text pairs. In NeurIPSW, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Claylon Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev, LAION-5B: An open large-scale dataset for training next generation image-text modols. In NewrIPS Datasets and Benchmarks, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Gyungin Shin, Weidi Xie, and Samuel Albanic, ReCo: Retrieve and co-segment for zero-shot transfer. In NewrIPS, 2022.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Gyungin Shin, Weidi Xie, and Samuel Albanie. NamedMask: Distilling segmenters from complementary foundation models. In CVPRW, 2023.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurIPS, 2022.
Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín Vobecký, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Obscrving the background to discover objects. In CVPR, 2023.
Gemma Team, Morgane Ríviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. Xiviv preprinl arXiv:2408. 0004400.001188 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint ar Xiv:25503. 19786, 2025.
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H. S. Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No "zero-shot' without exponential data: Pretraining concept frequency determines multimodal model performance. In DPFM at ICLR, 2024.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. CRIS: CLIP-driven referring image segmentation. In CVPR, 2022.
Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, ct al. Towards open vocabulary lcarning: A survey. T-PAMI, 2024.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Lin, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprint Xr1i:22310.01403, 2023.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Lo, and Ross Girshick. Delectron2. //stth/.cco/cccooooeeeeer/tttoop 2019.
https
Monika Wysoczaríkaa, Michael Ramamonjisoa, Tomasz Trzcinski, and Oriane Simeoni. CLIP-DIY: CLIP dense inference yields open-vocabulary semantic segmentation for-frec. In WACV, 2024a.
Monika Wysoczaúska, Oriane Siméoni, Michaël Ramamonjison, Andrei Bursue, Tomasz Trzciński, and Patrick Pérez. CLIP-DINOiser: Teaching CLIP a few DINO tricks. In ECCV, 2024b.
Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengcai Liao, and Cees GM Snoek. Any-shift prompting for generalization over distributions. In CVPR, 2024.
Hu Xu, Saining Xie, Yao Huang Xiaoqing Ellen Tan, Russell Howes, Shang- Wen Li Vasu Sharma, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2024.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. GiTupViT: Semantic segmentation emerges from text supervision. In CVPR, 2022.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary semantic segmentation models from natural language supervision. In CVPR, 2023a.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary scmantic segmentation. In CVPR, 2023b.
Xin Xu, Tianyi Xiong, Zheng Ding, and Zhuowen Tu. Masqclip for open-vocabulary universal image segmentation. In ICCV, 2023c.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Clae Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional CL.IP. In NeurIPS, 2023.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In TCCV, 2023.
Zhuowen Tu Zheng Ding, Jieke Wang. Open-vocabulary universal image segmentation with maskelip. In ICML, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding scenes through the ADE20K dataset. IJCV, 2019.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. In ECCV, 2022.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Appendix
In this appendix,
we start by providing details on the evaluation in Sec. evaluation protocol (Sec. A.1 approaches lo the background handling of the considered baselines (Sec. A.2). and details of the IoU-Single metric (Sec. A.3
In Sec. we present additional results inchuding classic mloU results (Sec. B.1 and further quantitative (Soc. B.1 and qualitative (Soc. B.3 results. We also discuss failure cases (Sec. B.2
Sec.
presents an additional analysis of our method, particularly: hyperparameter selection (Sec.
(Sec.
C.2 performance vs. the number of contrastive concepts when considering CC and CC (Sec.
C.3
CLIP's patch-level separability and how our method addresses this issue, (Sec. C.4) alternative
to CC scenario based on the sigmoid operation, and our experiments with filtering based on WordNet.
ontology (Sec. C.5).
In Sec. we provide details about LLM and the used prompts, together with examples of LLM- generated contrastive concepts.
Finally, in Sec. E we present an efficiency analysis of the proposed approach regarding the computation cost of generating the contrastive concepts and of employing them at segmentation time.
A Details on the evaluation
A.1 Evaluation protocol
Our experiments follow the evaluation protocol Cha et al. [2023] We use MMSegmentation implementation Contributors (2020) with sliding window strategy and resize input images to have shorter side of 448. In the case of CAT-Seg, we retain the original model framework and integrate IoU-single into Detectron Wu et al. (2019). We also use its evaluation protocol, meaning that the input images differ from other evaluated methods, i.e., with an input image size of 640x640. Regarding the text prompts, we keep the native prompting of each method to stay as close as possible the methods.
A.2 Background handling of baselines
We detail here the different strategics employed in the methods that we evaluate to handle the background.
TCL Cha al. (202)) applies thresholding and considers pixels with maximal logit < 0.5 to be in the background, where the logits are the cosine similarities of the visual embedding with the embedding of queries.
GEM Bousselham et al. (2024) applies a background handling strategy only for Pascal VOC. It only predicts the foreground classes. The background is obtained by thresholding the noftmax-normalized similarity between the patch tokens and the text embedding of each class name. The threshold is fixed (set to 0.85). In our experiments with VOC, we explore the performance of GEM both with and without background handling and report each time a better score. For other datasets than VOC, we apply only our methods.
MaskCLIP Zhou el. al. (0022) does not use any dedicated mechanism for background. Therefore, we do not report the original setup for it.
CLIP-DINOiser Wysoczańska et al. (2024b) leverages a foreground/ bba.hunddddbkggoudddd saliency strategy which focuses on foreground pixcls. In that case, the foreground/ Barground/bbcckroomnt /backrroudddd defined following FOUND Siméoni et al. (2023), which focuses on objectness and mainly discards pixels corresponding to stuff-like classes, which might also be of interest.
18

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm 1: ToU-single
input I input image: e #xx
Y ground-truth annotations I: gt E N#xWx1
T ground-truth text labels
CC dictionary of contrastive concepts per query
model segmenter producing pixel-level predictions given text queries
output: mean loU-single, mloU score for single-query scenario for given image procedure IoUsingle(I,Y)
// Get unique classes from Y glats + -unique(Y) scores (
for E glcts do
9 f- T.
// Texl prompls include query q and controstive concepls of
+ qUCC
1/ Get model predictions for given prompt set
5 4 model (4,
1/ Get binarized version of predicted mask
9 4 binarize(j, i)
// Get ground troth binary mask for gt class? i
1 binarize(Y, i)
// Record corresponding IoU
scores +- scores U IoU(Ú.
end for return mean(scores)
CAT-Seg
Cho et al. 2024 does not apply any background handling strategy. Instead, for VOC they create a list of potential background classes and use them as "dummy" classes. This approach is closest to what we propose. In practice, for the VOC dataset, the authors use class names from the Context datasel, extension of VOC with +40 class names.
SAN Xu et al. (023b) does not design any background handling strategy and does not evaluate datasets with 'backgroudd' class.
A.3 About the loU-single metric
We present a pseudo-code of our metric in Algorithm
B Additional results
B.1 More quantitative results
State-of-the-art results under classic mloU. In Tab. 4 report the results under the classic mloU metric for selected state-of-the-art methods on open-vocabulary semantic sogmentation. For each of the methods, we detail the specific background handling techniques (if any), the CL.P backbone used as well as additional datasets used for training.
Extending the dataset vocabulary with our generated contrastive concepts does not hurt the overall perfor- mance under a normal setup when all dataset labels are considered prompts. For GEM and MaskCLIP observe significant improvements over their original sctups VOC. This holds for both contrastive concept generation methods C and CcL_ Looking at the results of CLIP-DINOiser, we observe that saliency is still more effective in the object-centric scenario.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Background Type of
CLIP
Training dataset
Dataset Context Object
Methods
handling
CC
backbone
VOC
GioupViT
threshold
Ø
scratch
CC12M+RedCaps
18.7
27.5
50.4
CLIP-DIY
saliency
Ø
LATON-2B
19.7
31.0
59.9
TCL
threshold
Ø
OpenAI
CC12M+CC3M
24.3
30.4
51.2
MaskCLIP
Ø
OpenAI
21.1
15.5
29.3
MaskCLIP
Ø
Ø
LAION-2B
22.9
16.4
32.9
MaskCLIP" (*keys)
Ø
Ø
LAION-2B
24.0
21.6
41.3
CLIP-DINOiser
Ø
0
LAION-2B ImageNett (1k im.)
32.4
29.9
53.7
GEM
Ø
0
MetaCLIP
46.8
saliency
Ø
LAION-2B ImageNet (1k im.
34.8
62.1
CC
CC
LAION-2B ImageNet Net (1k im.
32.4
29.5
54.0
CLIP-DINOiser
CC
CC
LAION-2B ImageNet [1k im.
31.3
35.0
60.8
CC
CC
LAION-2B ImageNNe Net (1k im.)
31.8
33.3
60.4
CC
BC
LAION-2B
23.6
17.8
35.1
MaskCLIP
CC
CCL
LAION-2B
22.5
25.9
46.2
CC
CCD
LAION-2B
23.2
25.1
46.4
GEM
threshold
Ø
MetaCLIP
33.4*
27.4*
46.6*
GEM
CC
CC
MetaCLIP
31.6
35.7
60.0
GEM
CC
CCD
MetaCLIP
32.1
35.5
60.5
Table 4: Results with standard mloU metric when employing different contrastive concept generation strategies. denotes implementation, 'f' denotes results from TCL Cha et al.
(2023) and 'MaskCL.IP
(+keys)' denotes keys refinement proposed in the original paper Zhou et al. (2022). Training datasets include
CC12M Changpinyo el al. 2021)) RedCaps Desai et al. (2021). ImageNel, Deng el. al. 2009 CC3M Sharma
et al. 2018).
20

---

Published in Transactions on Machine Learning Research (05/2025)
Method
CLIP dataset
Original
CCPI
CCBG CCL
CCD
VOC
LAION-2B
49.9
47.9
51.8
53.6
MaskCLIP
OpenAl
47.1
44.2
52.2
53.4
MetaCLIP
47.9
46.6
50.6
50.1
CLIP-DINOiser
LAION-2B
63.8*
61.0
59.3
63.1
64.7
TCL
TCL's
52.9*
53.0"
52.9*
52.6*
53.6*
GEM
MeII
48.6*
61.3*
64.6*
CAT-Seg
OpenAl
52.8
69.5
67.7
Cityscapes
LAION-2B
32.2
16.2
27.2
24.0
MaskCLIP
OpenAl
30.6
15.0
22.5
22.0
MetaCLIP
30.0
13.6
24.6
23.3
CLIP-DINOiser
LAION-2B
20.8
36.0
23.2
30.6
27.3
TCL
TCL's
18.6*
29.7
9.8
26.3
22.0
GEM
MetaCLIP
20.6
14.5*
21.5
14.6
COCO-Stuff
LAION-2B
34.1
26.4
28.8
29.5
MaskCLIP
OpenAT
33.6
24.1
28.4
28.8
MetaCLIP
34.0
25.8
28.1
28.1
CLIP-DINOiser
LAION-2B
28.0*
35.3
32.4
33.9
34.4
TCL
TCL's
25.0*
34.7
17.4
29.5
30.6
GEM
MetaCLIP
38.3
22.9*
32.2
33.6
ADE20k
LAION-2B
33.2
22.7
26.8
27.8
MaskCLIP
OpenAI
29.8
20.2
23.5
25.2
MetaCLIP
32.1
21.5
24.7
26.0
CLIP-DINOiser
LAION-2B
28.8*
35.3
28.9
29.7
31.6
TCL
TCL's
14.8*
32.6
14.9*
25.9
26.5
GEM
MetaCLIP
33.0
21.5
26.3
29.1
CAT-Seg
OpenAl
46.8
25.7
38.4
39.7
COCO-Object
LAION-2B
32.1
27.7
33.7
32.9
MaskCLIP
OpenAI
31.3
24.3
34.5
33.3
MetaCLIP
30.9
27.4
32.2
31.1
CLIP-DINOiser
LAION-2B
38.8* 37.1* *
38.9
35.5
41.6
39.9
TCL
TCL's
38.1
37.2*
38.1* *
37.2*
GEM
MetaCLIP
31.4
39.7
40.1
Pascal Context
LAION-2B
40.5
34.4
35.2
37.4
MaskCLIP
OpenAI
41.1
32.9
34.7
36.8
MetaCLIP
41.1
32.6
34.2
35.8
CLIP-DINOiser
LAION-2B
33.9*
45.8
41.5
41.6
44.2
TCL
TCL's
29.7*
41.7
29.7*
36.8
38.2
GEM
MetaCLIP
26.9
40.1
42.1
Table 5: Results all datasets with our loU-single metric defined in Sec. the original background handling gives the best results.
4.1
denotes the result when
21

---

Published in Transactions on Machine Learning Research (05/2025)
More open-wae open-world evaluation results. Tab. 5 extends Tab. and completes the results obtained with the ToU-single on all the datasets that considered.
B.2 Failure case analysis
We present some failure cases of our approach in Fig. 8 Preciscly, we show examples of CLIP-DINOiser when one of the generation methods fails. The first example (first. row), CC suggests "blanket" for "bed", which typically covers the query concept. One of the potential improvements would be to instruct an LLM to ignore potentially occluding objects. In the second row, both methods fail to provide "floor" to contrast with "rug". We notice that CC tend to be more oriented towards objects, opposed stuff-like classes. We also observe that in the example, small part of painting on the wall is segmented as "rug'. This suggests that CCs might not give complete set of. Finally, in the third example, both methods fail to generate "person" to contrast with "bedclothes'. However, CC includes "pyjamas", which results in a better segmentation overall. lmage-conddtioned generation (e.g., with VLMs) could be a candidate solution to this problem, but we leave it for future work.
GT
CCD (single)
CCD (all)
CC (single)
CC (all)
dog
wall
toy
bed
bed
blanket
wall
pattern
rug
sofa
couch
furniture
fable
rug
rug
bedclothes
sheet
sofa
bedclothes
bed
pillow
bedclothes
pajamas
furniture
Figure 8: Failure cases of our method. We show examples of CLIP-DINOise when one of the methods fails to generate accurate CC. In the first example cC suggests "blanket" for "bed" which typically covers the query concept. In the second row, both methods fail to provide "floor" to contrast with "rug". Finally, in the third example, both methods fail to generate "person' to contrast with "bedclothes', however, CC suggest "pyjamas', which results in better sogmentation.
B.3 More qualitative results
More qualitative results are provided in Fig. comparing CCD to CCL
C Additional analyses
C.1 Hyperparameter selection
This section discusses the selection of hyperparameters for our CC generation. For the frequency threshold y and the cosine similarity threshold 6, we randomly select 100 images from the training set of the ADE20K dataset and report IoU-single on this subset which we observed was enough select the values. We report in Tab. 6 parameter study of both hyperparameters and mark in grey selected values, iee, 7 0.01 and
22

---

Published in Transactions on Machine Learning Research (05/2025)
GT
CCBG
CC (single)
CCD (all)
CC (single)
CC (all)
skyacrapes boat bed
woman
pillow
blanket
Toy
coast
city
harbour
dock
weterr
Water
M
cloud
Figure 9: More qualitative results of CLJPPDINNOser with different CC. Here we focus cases where CC and cc give different results. For "boat" (2nd row), CC gives a better result providing a good CC ("dock") ) On the other hand, for "skyscraper" (3rd row), CC yields slightly better results suggesting "sky" and not "cloud". Note that this last example, CCBG' completely fails, possibly due to difficult (uncommon) angle of view.
values of
values of 8
Method
CLIP tr. data
0.001
0.005
0.01
0.015
0.02
0.95
0.9 0.85
0.8
0.75
OpenAI
24.4
26.0
24.8
24.4
23.2
19.9
21.0 23.0
24.4
22.8
MaskCLIP
Laion2B
25.8
27.8
27.4
26.0
25.4
23.0
24.1
26.4
27.4
24.6
MetaCLIP
22.0
24.1
24.4
23.8
23.4
22.7
23.7
25.9
27.2
23.7
DINOiser
Laion2B
24.4
27.2
27.9
27.9
27.7
23.5
24.6 26.4
27.9
26.9
Table 6: Parameter study of and 8. Selection (marked in grey) of the hyperparameters and 6 with ToU-single on 100 randomly-sclccted images in ADE20k training datasct.
Method
CLIP training data
1.0
0.95
0.9
0.85
0.8
OpenAI
26.0
40.4
41.1
39.1
32.1
MaskCLIP
Laion2B
35.3 43.7
44.0
44.6
42.2
MetaCLIP
24.4 39.1
40.3
34.3
30.6
DINOiser
Laion2B
51.3 57.8
58.6
58.8
55.2
TCL
TCL's
37.2
47.6
47.7
47.1
47.7
Table 7: Sclection of 3 with classic mloU on 100 randomly-selected images in the VOC training datasct.
Results are reported for CCL.
8 0.8. For 7, observe that values y 0.005 too low, most likely introducing too much noise in selected contrastive concepts.
Tab. 7 presents a parameter study of the cosine similarity of lext queries 8 in multi-query segmentation. Here, we randomly select 100 images from the VOC training set and report classic mloU for different 8 values. We select 0.9 because it gives the best result for most methods. We also note that controlling the similarity between query concepts and contrastive concepts in the multiple-query scenario is necessary. Not including this step (see results for 1.0) greatly degrades performance.
23

---

Published in Transactions on Machine Learning Research (05/2025)
C.2 Average number of contrastive concepts performance
We present in Fig. 10 scalterplol of performance vs the number of contrastive concepts when considering CC (Fig. 10 a)) and Cc (Fig. (0bb)) The points correspond to the loU-single scores class obtained with CLIP-DINOiS on all the datasets we evaluate. We do not observe a strong correlation between the number contrastive concepts and performance, although there is a small mode of around 20 concepts when using CC
We also observe that, on average, |CCP| > |CC4)
80
loU-single (%)
60
IoU -iU-innlle (%)
60
40
40
20
20
10
15
20
[CC
20
40
|CC
(a) CC
(b) CCP
Figure 10: Number of CC performance. We compare the number of CC against the performance of CLIP- DIN-ser for each class used in evaluations (considering all datasets). Performance is reported with per class ToU-single %.
C.3 On separability of CLIP patch- patch-features
6
8
4
6
x106
2
2
0
-0.5
0.5
-1
-0.5
0.5
Cosine similarity
Cosine similarity
(a) VOC
(b) ADE20K
Figure 11: Distribution of maximum patch similarities with text prompts. We plot histograms for 100 images VOC (a) and ADE20K (b) of patch similarities in MaskCLIP.
In Fig. we present an analysis of the patch-level CLIP space using MaskCLIP features. The figure shows histograms of patch-level maximum text similarities (in cosine similarity) across 100 randomly sampled images from VOC (a) and ADE20k (b). We notice an overall concentration of cosine similarity scores in [0.1,0.3), suggesting that the feature space not easily separable.
To illustrate how our approach overcomes this issue present in Fig. 12 t-SNE analysis over patch features from an image of VOC datasct for the q "bird". We plot the result of classification for each separate set of CCs. We highlight in orange the patches that belong to the ground truth mask of class "bird'. We observe that BC already helps to separate the space background concepts from "bird' patches. However, we notice that only with CCL or CCD we can separate one visible cluster left, possibly belonging to the patch features of branch in the image, by providing 'branch' in the case of CC or 'tree' in CC Both of our proposed methods improve the final segmentation result.
24

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Input image
CC
CC
Ground
bird
background
bird
C
CCL
Ground truth label: bird
Ground truth label: bird
sky bird branch
sky
bird
background
background
(b)
Figure 12: t-SNE analysis of patch features for different CC of image "bird". We present patch features with their predicted closest text embedding coded in color. Text embeddings are corresponding CC of = "bird". We also mark the ground truth labels in orange. The sample from VOC dataset.
25

---

Published in Transactions on Machine Learning Research (05/2025)
thresholded sigmoid ... CC
0.6
single-IoUU%
0.4
0.2
0.52
0.54
0.56
0.58
0.6
threshold
Figure 13: Sigmoid experiments. We replace softmax with sigmoid applied on individual patch-to-query prompt similarities. We show the variation of single-ooU% wrt. the threshold that is applied after sigmoid to decide on a positive vs. "background" class. To get the thresholds, we find the minimum and maximum values of the features after sigmoid and linearly sample 30 values in this range. We can see that the result is sensitive to the threshold value and does not reach the baseline of CCBG.
C.4 Replacing CC with sigmoid operation
Trying separate query from its background using binary criterion natural alternative direction consider, and this could be implemented with sigmoid.
We test using a sigmoid on CLIP similarity scores. We show the results of an experiment with CLIP-DINOiser on VOC in Fig. 13 We make the following observations: (1) none of the thresholds allow us to reach the performance of CC and (2) the performance very sensitive to the threshold value. We believe this is because the CLIP space not easily separable, as discussed in Appendix C.3
C.5 Ontology-based filtering with WordNet
Method
MaskCLIP TCL DINOiser
CC
25.2
26.0
31.6
CC + WordNet
25.2
26.4
26.3
CCD + WordNet sem. sim.
21.0
23.4
25.8
Table 8: Ontology-based (WordNet) filtering out synonyms, meronyms, hyponyms and hypernyms (at depth 1) from CCD Results are reported on ADE20K, as /loU-single.
Here, we discuss our experiments using the WordNet ontology Fellbaum (1998) for CC filtering. We extract synonyms, meronyms, hyponyms, and hypernyms for each query concept in-depth in the WordNet ontology. From the results Tab. g we observe that adding such filtering on top of our semontic similarity filtering brings little no improvement, suggesting that semantic filtering removes most of the contrastive concepts that interfere with a query concept. Furthermore, replacing semanlic similarily with WordNet-based filtering yields significantly worse results than proposed CC
D Prompting the LLM
In this section, provide details about the LLM and the prompts used.
26

---

Published in Transactions on Machine Learning Research (05/2025)
D.1 The LLM model
We use the recent Mixtral--x77B-Instrre model Jiang et al. (2024), sparse mixture of experts model (SMoE), finetuned for instruction following and released by Mistral Al. More precisely, we rely on the v0.1 version of its open weights available via the Hugging Face transformers library. We run the LLM in 4-bit precision with flash attention to speedup inference.
D.2 The prompts used for contrastive concepts
We provide in Fig. 14 the prompt used to generate the contrastive concepts CC and in Fig. 15 the prompt used to predict whether a concept can seen an image or not in order to filter CC
In these prompts, we indicate the inserted input text {q}. We follow Mixtral-8x7B Instruct's prompt template. In particular, use <<>> the beginning of the string (BOS) special token, as well (INST) and 
 string markers to be set around the instructions.
For the generation of CC we also integrate a light post-processing step, ensuring that all generated lists have a unified format, with coma separation. We do not apply any filtering or cleaning step to the L.M-generated results.
<>> INST] You are a helpful AI assistant with visual abilitics.
Given an input object O, want you to generate a list. of words related to objects that can be surrounding input object O in an image help me perform semantic segmentation.
For example:
If the input object is 'fork', you can generate list of words such as '("bottle,, "knife', "table', 'napkin', "bread']:
* If the input object is 'child', you can generate a list of words such as ['toy", 'drawing", 'bed', 'room', "playground"]:
You should not generate synonyms of input object O, nor parts of input object O.
Generate list of objects surrounding the input object {g} without any synonym parts, nor content of it. Answer with list of words. No explanation.
Answer: [/NST]
Figure 14: Prompt for Cc contrastive concept generation.
<>>
[NST] Please specify whether {ạ} is something that one can see.
Reply with 'yes' or 'no' only. No explanation.
Answer: 

Figure 15: Prompt for CC visibility prediction.
D.3 Example of generated CCL
We present Tab. the example of CCL generated for Cityscapes datasct. We provide CC for each query q in separate rows.
D.4 Part removal via L.M-prrmpting
We also explore the possibility of removing suggested contrastive concepts that can be parts of query concepts. Note that in CCL, we explicitly do it in the prompt itself (Fig. 17
Fig. 16 presents of such examples when
27

---

Published in Transactions on Machine Learning Research (05/2025)
Query g
CC₁
road sidewalk building wall
building, tree, car, pedestrian, sky, streetlight, sidewalk, bicycle, parked car, traffic sign building, street, car, tree, people, bike, road, park, sky, lane
sky, tree, road, car, park, people, lane, fence, house, field door, window, floor, ceiling, painting, light, chair, table, carpet, curtain
fence
grass, tree, house, car, path, post, gate, field, flowers, animals
pole
building, wire, tree, street, sky, fence, cable, road, banner, light road, car, building, pedestrian, sky, streetlight, traffic sign, parking meter road, street, pole, vehicle, building, sky, pedestrian, curb, lane, light soil, tree, grass, water, animal, fence, field, sky, rock, sun tree, sky, building, road, mountain, river, field, fence, vehicle, person tree, building, cloud, sun, bird, airplane, mountain, sea, sunset, cityscape
traffic light
traffic sign
vegetation
terrain
sky
person
bike, road, car, tree, building, park, cityscape, nature, animal, sports equipment bicycle, road, nature, park
rider
car
road, tree, building, person, parking road, car, building, tree, parking road, tree, building, sky, person, car, traffic light, bicycle, parking meter, street sign track, grass, sky, building, platform, tree, sign, person, car, road road, person, bike, car, traffic, building, nature, parking, city, scenery road, tree, person, park, building, grass, basket, helmet, traffic, path
truck
bus
train
motorcycle
bicycle
Table Example of LLM-generated cc for Cityscapes.
removing "whecl" from the CC query "bicycle" gives a slight improvement for MaskCLIP segmentation. However, we do not notice a particular improvement the case of other segmentation methods since, typically, they refine the masks or feature maps to include localization priors. For example, in Fig. 16 the second row presents the same example for CLIP-DINOiser (DINOiser), where the improvement is marginal. Finally, quantitative improvement when applying part removal filtering entire datasets.
observe little Therefore, we do not include it in our final method.
GT
CCD
CCD
CCD -parts (single)
CCD -parts (all)
(single)
(all)
MaskCLIP
Maycle
bicycle
CLIP- DINOiser
style
slyle
Afool
street
bicycla
Figure 16: Part removal. We consider an example from Pascal Context with q= bicycle. We show the segmentation masks produced by MIskCLIP and CLIP-DINOise for CCD as well as for CCD when parts of objects are removed (CCD parts).
28

---

Published in Transactions on Machine Learning Research (05/2025)
<S>
[NST]] You are helpful Al assistant with visual abilities.
Given imput object O, want you to generate list of words that are parts of an object O.
For example:
* If the input object is 'rabbit', you can generate a list of words such as "I" "'paw", "tail", "fur", "cars', 'muzzle"].
* If the input object 'building', you can generate a list of words such as ''door", 'window', 'wall", 'hall", 'Bloor]].
Generate list of parts of the imput object {g). Answer with list of words. Do not give any word that is not part of the input object. No explanation.
Answer: [/NSST]
Figure 17: Prompt for part prediction.
E
Efficiency analysis
We first discuss the computational cost of generation and then the cost of employing generated CC at segmentation time.
E.1 Computational cost of cc
We use the HuggingFace implementation of Mixtral--xx7BBlnssrueet--00 through the transformers library. Using 4-bit quantization and flash attention on an A100 GPU, the LLM requires 25.5 GB of GPU memory. The average inference time roquired generate a complete list of contrastive concepts for a given input query (averaged over 20 Pascal VOC queries) 5.4
We also note that new competing LLMs, of comparable smaller sizes than Mixtral 8x7B, are regularly being released, such as Llama 3 8B Instruct Grattafiori et al. (0224) (Apr 2024), or Gemma-2 9B Instruct Team et al. (2024) (Jun 2024), Gemma-3 4B Team et al. (2025) (Mar 2025). For these LLMs, the GPU memory requirements are more than 3.5x smaller, and the generation time more than 3.5x faster.
E.2 Computational cost of CC
Offline cost. In order to obtain a matrix of co-occureecess of concepts the main cost lies in the construction of the co-occurrence matrix X where we iterate over 400M samples of LAION. However, we only need to go through captions, and not images, and we do it once and offline. Finally, this can be efficiently implemented by leveraging modern libraries for multiprocessing.
Online cost. At runtime, the generation of contrastive concepts is fast. We provide below runtimes of all the online steps required for CC extraction, computed on machine equipped with Intel(R) i7 CPU and Nvidia RTX A5000 GPU:
Computing the CLIP embodding for query g: 24.4 ms.
Mapping of query to the closest concept in 7: 0.001 ms.
Retrieving CC of the closest concept in T: marginal cost (look-up table).
E.3
Segmentation efficiency
The computational cost of employing our proposod methods strictly depends on the OVSS. We present Tab. 10 comparison of runtimes between BG and CC with CLIP-DINOiser. We split the comparison into two sub-processes, text embedding extraction, and segmentation, where typically the former stable
29

---

Published in Transactions on Machine Learning Research (05/2025)
Method
CC
CCD
Text Embeddings extraction
49.2 + 0.7 ms
519.5 1.1 ms
Segmentation
22.1 + 0.2
22.3 = 0.2 ms
Table 10: Runtime comparison between CC BC and CCD.
across different OVSS methods. We observe that the main difference in runtimes stems from the embedding
extraction phase, due to the different number of text prompts, that is, [{g) CCBG|= + 2, while Kg) UCC 21 on average. (We discuss the average number of CC and CC in Appendix C.2) However, note that this extraction time could be effectively reduced with caching mechanisms and an increase in memory consumption.
For the segmentation forward pass, we report the runtime of a single forward pass on images of size 448 x 448 when using pre-extracted text embeddings for final segmentation. We observe a negligible increase in the runtime between CC and CC
30


Reviews:
[{"review_id": "ugqVvjB5Xw", "reviewer": "Reviewer_AYvS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Motivation. The motivation for this submission is clear and straightforward.  \nSignificance. The evaluation of using background in existing approaches is problematic, in such case the paper has its own significance in this field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses evaluations of open-vocabulary segmentation, a task aiming to segment any text given a multi-modal model. Specifically, authors broadly investigate the role of background in evaluation prompts, which is widely adopted in existing open-vocabulary segmentation approaches. To achieve this, they propose contrastive concept, an automatic solution that searches for an alternative to background. They explore mining from a large-scale dataset and generation from LLM, both of which appear remarkable performance gain, as compared to using background as a contrastive concept. Authors test their approach over six open-vocabulary segmentation models and three benchmarks, and ablate their modules thoroughly.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The overall submission is complete from my view, no requested changes so far.", "broader_impact_concerns": "Authors have included a broader impact concerns in section Conclusion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "smxrYpeZDI", "reviewer": "Reviewer_cF1G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n\n- The paper is overall well written and easy to follow.\n- The paper addresses an interesting problem of solving the in-effectiveness of the \"background category\" in OVSS. The authors propose a simple-yet-effective method of generating contrastive concepts at test time.\n- The experimental results are comprehensive and convincing, with comparisons against various baselines. The authors also include several ablation studies to validate their design choices.\n\n#### Weaknesses\n\n- Lack reference for several relevant related works [1-3].\n\n  [1] Xu, Xin, et al. \"Masqclip for open-vocabulary universal image segmentation.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.\n\n  [2] Ding, Zheng, et al. \"Open-vocabulary universal image segmentation with maskclip.\" In International Conference on Machine Learning, 2023.\n\n  [3] Wu, Size, et al. \"Clipself: Vision transformer distills itself for open-vocabulary dense prediction.\" *arXiv preprint arXiv:2310.01403* (2023).\n\n- Lack of efficiency analysis. The proposed strategies of generating contrastive concepts  introduce additional computational cost.\n\n  - For $CC^L$, the authors used Mixtral-8x7B-Instruct for contrastive concept generation, which is much bigger than the CLIP models. This requires substantially more computational resources, potentially limiting the practical applications of the proposed method. Could the authors provided a detailed analysis on the computational efficiency of $CC^L$ (e.g. forward pass time, FLOPs)?\n  - For $CC^D$, how many contrastive concepts are used on average? Since passing new text prompts into the CLIP-text encoder leads to additional computational cost, it will be beneficial to provide a comparison of FLOPs between $CC^D$and the $CC^{BG}$ baseline.\n\n- Need additional clarification on the IoU-single metric. While Fig 5 show examples of the standard mIoU and IoU-single, the exact differences are unclear. Does IoU-single simply allow for overlapping of different categories? Why is it necessary to introduce the new metric?\n\n- Limited evaluation on multi-query segmentation. While the authors discuss the extension of their method to multi-query segmentation in Sec. 3.1, the reviewer does not see clear experimental results to validate this claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a test-time approach for open-vocabulary semantic segmentation (OVSS) by generating contrastive concepts for the target category. Specifically, the authors provide two methods for generating contrastive concepts: 1) using a LLM to predict the contrastive concept; 2) mining co-occurence-based contrastive concepts from VLM training captions. They also introduce a new metric for evaluating single-category segmentation performance. Experimental results across various backbones and datasets validate the effectiveness of the propose method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide a more comprehensive discussion on related work.\n- Provide some efficiency analysis on the proposed method.\n- Clarify the IoU-single metric by explicitly explaining how it differs from mIoU and providing justification for its introduction.\n- Provide additional illustration and experiments on how to extend their method to multiple queries (e.g. 1/2 query categories are available at test time).\n\nPlease kindly refer to the weaknesses sections for more details.", "broader_impact_concerns": "Not much.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PyUiZDYjyO", "reviewer": "Reviewer_9eo4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1、The issue investigated in this paper is a practical one. Previous methods perform effectively when a user comprehensively lists all possible visual concepts in an image and makes comparisons among them for assignment. Nevertheless, in a truly open-vocabulary scenario, we are unaware of the concept lists. Therefore, this study offers deeper insights into the open-vocabulary task.\n\n2、Sufficient ablation experiments and visualizations have demonstrated the effectiveness of the method.\n\nWeaknesses:\n\n1、Regarding the prompt given to the large language model, have you ever tried to directly let the LLM output the objects and the positions of their surrounding targets (masks might be a bit difficult, but how about bounding boxes?) and then assist with the subsequent segmentation.\n\n2、I think the point that \"Background is not enough\" is really worth studying. The CLIP-based method combined with softmax makes it impossible to input arbitrary categories. The results vary due to different word lists. So, have you ever tried using a sigmoid-based CLIP, such as SigCLIP, to address this issue?\n\n3、Is there an evaluation metric to prove that the LLM has truly generated the concepts that should be present in this image? Are there any quantitative statistical analyses?\n\n4、Figure 3 contains too little information and needs to be beautified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The previous open-vocabulary methods work well when a user exhaustively lists all possible visual concepts in an image that contrast against each other for the assignment. This is not realistic, so the peper presents a novel scenario which only gives the textual prompt and nothing else. The method automatically generate query-specific textual contrastive concepts at test time. Experiments show the effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Modify and beautify the images, and increase the information content of the images, especially for Figure 3.\n\nThe above experiments on the sigmoid-based CLIP can, to a certain extent, solve the problem of the inability to preset the word list. Please discuss this.\n\nEvaluate whether the LLM can find all the concepts that should be in the image, and assess the impact of the quality of the concepts generated by the LLM on the results of panoramic segmentation.", "broader_impact_concerns": "No Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews were already positive about the work, appreciating the interesting problem it tackles (AYvS, cF1G, 9eo4), the experimental results (cF1G, 9eo4), and the presentation (cF1G). However, they also raised some concerns, with cF1G pointing to missing references, lack of efficiency analyses (cF1G), and limited evaluation on the multi-query setting, while 9eo4 on prompting, LLM generated concepts evaluation, and analyses of multiple CLIP backbones.

The updated manuscript and response were deemed satisfactory, with all reviewers finding their main concerns addressed. As a final modification, the discussion, table, and numbers on efficiency (provided in the response to cF1G) should be included in the final version of the manuscript, even in the appendix.
Decision: Accept (poster)