Test-time Contrastive Concepts for Open-World Semantic Segmentation

## Abstract

Abstract Recent VLMs, pre-trained on large amounts of image-text pairs to align both modalities, have opened the way to open-vocabulary semantic segmentation. Given an arbitrary set of textual queries, image regions are assigned the closest query in feature space. However, the usual setup expects the user to list all possible visual concepts that may occur in the image, typically all classes of benchmark datasets, that act as negatives to each other. We consider here the more challenging scenario of segmenting a single concept, given a textual prompt and nothing else. To achieve good results, besides contrasting with the generic “background” text, we study different ways to generate query-specific test-time contrastive textual concepts, which leverage either the distribution of text in the VLM’s training set or crafted LLM prompts. We show the relevance of our approach using a new, specific metric.

## 1 Introduction

Figure: Figure 1: Illustration of the benefit of our contrastive concepts $\mathcal{CC}$. We investigate open-world, open-vocabulary segmentation, where only one or a few visual concepts to segment are known, as opposed to all classes that can possibly occur in the image. Contrasting the query with “background” helps [, ] (3^rd column). However, this may not suffice to catch all pixels *not* corresponding to the query if they are related or co-occur frequently in the VLM training set. Our automatically-generated *contrastive concepts* $\mathcal{CC}$ help to separate and disentangle pixels of the query (right column).
Refer to caption: extracted/5713349/figures/examples/cat_doc.jpg

Vision-language models (VLMs) [] are trained to create and align text and image representations, i.e., the representation of an image is made close to the representation of text expressing the image contents. VLMs such as CLIP [] and its successors deliver impressive zero-shot performance on various open-vocabulary image-level tasks, such as image classification [, , ], image retrieval [] and image captioning [, ]. However, transferring such ability at the pixel level is not straightforward.

A particularly challenging pixel-level task is the open-vocabulary semantic segmentation (OVSS) of *visual concepts* in images, i.e. visual entities such as objects, stuff (e.g. grass), or visual phenomena (e.g. sky). Various methods have been proposed to that end, exploiting a frozen CLIP model with well-designed additional operations [, , , ], or fine-tuning the model with particular losses [, , , , ]. Better results can be obtained using annotations, such as classes or class-agnostic masks [, , ]. Yet, we focus here on strategies which do not rely on any type of manual supervision, therefore avoiding any specialization to annotated domains.

Despite the remaining gap in segmentation quality compared to fully supervised training
on specific datasets, the reported performance of these open-vocabulary segmenters is appealing. However, the current benchmarking strategy to evaluate them [] does not exactly measure pure, out-of-context, open-vocabulary performance. The fact is that evaluation is mainly based on existing, classical semantic segmentation datasets organized around a limited set of classes [, , , ].
More importantly, to enable dense segmentation, the evaluated algorithms have access to a complete list of all visual concepts in the dataset. While this may seem natural for a fully dense task, it is actually key to part of the reported segmentation performances. The reason is that visual concepts that co-occur in a VLM’s training set can be mistaken for each other at test time as they tend to have close representations. For instance, “boat” can be hard to separate from “water”, leading to coarse segment boundaries when prompted alone.
Now if all visual concepts possibly present in an image are known from the dataset, it is easy to contrast them and assign pixels to the closest concept in feature space.

In this work, we study the impact of contrasting textual prompts at test time. To that end, we consider the harder and maybe more realistic task of segmenting one or several visual concepts given as text (positives), without any given knowledge of all other classes that may occur in the image (negatives).

Various methods tried to address this problem.
One of the most straightforward ways to separate background from foreground, i.e., from the queried visual concept(s), is to
introduce a similarity threshold [, ].
This can be however hard to set and is, at least, dataset-dependent if not image-dependent.
Using an *objectness* map helps separate the queried visual concept from the background [], but fails when the queried object is in the background.
Leveraging rich image features from DINO [] improves object delineation and filters out likely background [], but lacks priors to further disentangle foreground from background.
This ambiguity can be partly tackled by generating visual prototypes for given textual categories, but at the cost of expensive test-time diffusions [].

Another common strategy, in this case, is to consider an extra class labelled “background”, intended to catch all pixels not corresponding to the queried visual concepts. It provides an easy generic and default negative, in particular for object-centric evaluation datasets, such as Pascal VOC []. The notion of background is however not well defined as it is context-dependent. It may also not be present with the expected distribution in the training set. Besides, a failure case is when a queried visual concept (e.g., “tree”) falls in the learned background (which commonly encompasses trees).

In this work, rather than assuming access to a dataset-specific set of classes, we propose instead to automatically produce visual concepts that are useful to better localize a queried concept.
We also propose a new metric to evaluate such an ability, namely IoU-single, which considers one query prompt at a time and thus does not rely on the knowledge of potential domain classes.
We show that leveraging test-time contrastive concepts
allows us to reach better results on our metric and achieve on-par results when using the classic mIoU metric.
To summarize, our contributions are as follows:

- •
We investigate a new evaluation setup for open-world, open-vocabulary semantic segmentation, which does not rely on any domain knowledge.
- •
We introduce the notion of test-time contrastive concepts and discuss the interest of generated contrastive concepts to improve open-vocabulary image segmentation.
- •
We discuss the usage of “background” as a contrastive concept, which has been so far accepted but not discussed.
- •
We propose a new metric to evaluate the grounding of visual concepts and use it to benchmark multiple state-of-the-art methods across several commonly used datasets.

## 2 Related work

##### Open-vocabulary semantic segmentation.

VLMs trained on web-collected data to produce aligned image-text representations [, , ] had a major impact on open-vocabulary perception and opened up new avenues for research and practical applications. While CLIP can be used *off-the-shelf* for image classification in different settings, a major shortcoming is its inability to produce dense pixel-level features and predictions due to its final global attentive-pooling [, ]. To mitigate this and produce dense image-text features, several approaches finetune CLIP with dense supervision or devise new CLIP-like models trained from scratch with segmentation-compatible pooling on large datasets annotated with coarse captions [, , , , , , , ], object masks
[, , ] or pixel labels [, ]. However, such models either face feature degradation [] when finetuning or need long training cycles on huge amounts of images when trained from scratch.

CLIP densification methods arise as a low-cost alternative for producing pixel-level image-text features while keeping CLIP frozen [, , , , , ]. The seminal MaskCLIP [] mimics the global pooling layer of CLIP with a $1\times 1$ conv layer. The aggregation of features from multiple views and crops [, , , ] also leads to dense features, yet with the additional cost of multiple forward passes. Some methods [, , ] rely on codebooks of visual prototypes per concept, including per-dataset negative prototypes [], or leverage self-self attention to create groups of similar tokens []. The recent CLIP-DINOiser [] improves MaskCLIP features with limited computational overhead thanks to a guided pooling strategy which leverages correlation information from DINO features [].

##### Prompt augmentation.

Prompt engineering is a common practice for adapting Large Language Models (LLMs) to different language tasks [] without updating parameters. This strategy of carefully selecting task-specific prompts also improves the performance of VLMs.
For instance, in the original CLIP work [], dataset-specific prompt templates, e.g., “a photo of the nice {}“ were devised towards improving zero-shot prediction performance. Although effective, manual prompting can be a laborious task as templates must be adapted per dataset and must be sufficiently general to apply to all classes, limiting fine-grained accuracy. Different automated strategies have been subsequently explored,
e.g., scoring and ensembling predictions from multiple prompts [].
Prompts can also be augmented by exploiting semantic relations
between concepts defined in WordNet [] to generate new coarse/fine-grained [] or synonym [] prompts. LLMs can be used as a knowledge base to produce rich visual descriptions adapted for each class starting from simple class names [, ]. Prompt features can be learned by taking into account visual co-occurences [], a connection between training and test distributions [], mining important features for the VLM [] or by test-time tuning on a sample []. Most of these strategies have been designed and evaluated for the image classification task and their generalization and scalability for semantic segmentation is not always trivial.
Here we aim to obtain better prompts for semantic segmentation to
separate queried object pixels from their background.
We do this automatically without changing the parameters of either the text encoder or the image encoder, by leveraging statistics from VLM training data or LLM-based knowledge.

##### Dealing with contrastive concepts in OVSS.

Our contrastive concept discovery is tightly related to *background handling* in the context of open-vocabulary semantic segmentation since the standard benchmark datasets for this task, originally designed for supervised learning, use *background*
to describe unlabeled pixels, e.g., covering concepts outside of the dataset vocabulary. There are three main types of approaches to address this issue. The first one is to threshold the uncertain predictions [, , ] with a given probability value [, ] or
clip similarities [].
The second group of methods leverages the object-centric nature of certain datasets by defining background through visual saliency, leveraging an external foreground/background segmentation model to discard the background [, ].
Finally, a significant body of work addresses the same issue by defining dataset-level concepts either by adding handcrafted names of concepts to the background definition [, , , ] or by extracting visual *negative prototypes* with a large diffusion model [].
In contrast, in this work, we aim for automatic discovery of contrastive concepts without any prior access to the vocabulary used for the annotation of the dataset.

##### Visual grounding.

is the task of localizing within images specific objects from text descriptions.
The major instances of visual grounding tasks are *referring segmentation* that produce pixel-level predictions for one [, , ] or multiple target objects [] given a text description, and *referring expression comprehension* [, , , ] that detects objects.
Similarly to referring segmentation, we aim to segment precisely specific objects defined by the user. In contrast, we do not use supervision to align textual descriptions with object masks, do not focus on text-described relations between objects and
mine contrastive concepts to disentangle target objects from the background.

## 3 Improving segmentation with test-time contrastive concepts

We consider here the following segmentation task: given an image and a set of textual queries characterizing different visual concepts, the goal is to label all pixels in the image corresponding to each concept, leaving out unrelated pixels, if any. *Our work is to study how existing segmentation methods may support this task.* In particular, we are interested in the ability to generalize well to unseen datasets []. That is, we not only want to be open-vocabulary regarding the choice of words for querying, we also want to be open-world, i.e., not specialized to a given set of categories, implicitly or explicitly (e.g., when evaluating on a specific dataset).

### 3.1 Closed-world vs open-world open-vocabulary semantic segmentation

Even if open-vocabulary, traditional semantic segmentation is *close-world* in the following sense. Given an RGB image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ and a set of textual queries $q\in Q$, semantic segmentation yields a map $\mathbf{S}_{\text{closew}}:H\times W\mapsto Q$, where each image pixel has to be assigned one of the queries as label.

In contrast, *open-world* segmentation considers an extra label ‘$\bot$’ to represent any visual concept that is different from the queries. The segmentation map in this case is then $\mathbf{S}_{\text{openw}}:H\times W\mapsto Q\cup\{\bot\}$. For instance, to label a boat, it is enough to ask for the “boat” segment; other pixels (sky, sea, sand, rocks, trees, swimmers, etc.) are to be labelled ‘$\bot$’.

Our proposals for open-world segmentation are described below. Meanwhile, in our comparative study, we use the following close-world open-vocabulary segmentation framework. A CLIP-like textual encoder noted $\phi_{\text{T}}(\cdot)$ is used to extract textual features $\phi_{\text{T}}(q)\in\mathbb{R}^{d}$ for each query $q$, where $d$ is the feature dimension. Different segmentation methods then have their own ways of generating patch-level features $\phi_{\text{V}}(\mathbf{I})\in\mathbb{R}^{w\times h\times d}$, where $\phi_{\text{V}}(\cdot)$ denotes a CLIP-like visual encoder, $w=W/P$, $h=H/P$, and $P$ is the patch size. We then compute the cosine similarities between each of the query features and the patch feature (thus after L2-normalization), which we use as logits when upsampling to finally obtain pixel-level predictions.

### 3.2 Test-time Contrastive Concepts ( 𝒞 ⁢ 𝒞 𝒞 𝒞 \mathcal{CC} caligraphic_C caligraphic_C )

We mentioned in the introduction the ambiguity of representation between different visual concepts due to co-occurrences at training time. One of the keys to separating such concepts in a segmentation scenario is to *contrast* them at test time.

If ambiguous visual concepts are part of the queries $Q$, they are “naturally” separated at segmentation time thanks to the closest-feature principle. However, a problem arises if a visual concept in a query $q\in Q$ can be mistaken with another visual concept present in the image but not being queried (e.g., querying “boat” but not ”sea”). In this case, we propose to use one or more additional textual queries of visual concepts that are likely to contrast well with $q$. We name such concepts *contrastive concepts* and note them $\mathcal{CC}_{q}$.
Although this formulation generalizes to several queries $Q$, with contrastive concepts $\mathcal{CC}_{Q}$, we focus on $|Q|=1$ in the following.
In this context, to segment a query $q$, given contrastive concepts $\mathcal{CC}_{q}$, we simply segment $\{q\}\cup\mathcal{CC}_{q}$ in the close-world segmentation framework, and then remap to label ‘$\bot$’ the pixels assigned to $\mathcal{CC}_{q}$ to ignore them.

In recent works [, , ], the word “background” has been used as a generic contrastive concept to help separate foreground objects from their background. We note this setup $\mathcal{CC}^{BG}$$=\{\text{``background''}\}$ and discuss in the next section why such a simple word contrasts well with benchmarked classes.

### 3.3 Why does 𝒞 ⁢ 𝒞 B ⁢ G 𝒞 superscript 𝒞 𝐵 𝐺 \mathcal{CC}^{BG} caligraphic_C caligraphic_C start_POSTSUPERSCRIPT italic_B italic_G end_POSTSUPERSCRIPT = { “background” } absent “background” =\{\text{``background''}\} = { “background” } work?

Figure: (a) Freq. of VOC concepts.
Refer to caption: extracted/5713349/figures/background-study/background_1.jpg

In this section we study why using the single word “background” as contrastive concept, i.e., $\mathcal{CC}^{BG}$$=\{\text{``background''}\}$, achieves good results [, ].
Notably, if the word feels natural to us, it is not obvious why it should work in CLIP space. In fact, this formulation is not contextual in that the contrastive concept is not specific to the query, which might be suboptimal. Offering a single concept to contrast with could also be suboptimal. Worse, the “background” samples that CLIP learned from could accidentally include the visual concept of the query, which could make the query representation close to the background representation and defeat the contrast mechanism.

We investigate the occurrence of “background” in VLM training data.
First, we use the metadata provided by the authors of [], which describe the representation of four thousand common concepts in LAION-400M [] (a subset of the web-crawled LAION-2B dataset [] used to train CLIP). We plot in Fig. 2(a) the frequency of occurrence of “background”, compared to the occurrence of class names of VOC. We observe that the word is significantly more frequent than all others, hinting that the word is widely available not only in CLIP training data but also in web-crawled data in general.

We also show in Fig. 2(b) images sampled from the LAION training set that have a caption containing “background”. We observe that they display high diversity in colours and textures. Images captioned with “in the background” (Fig. 2(c)) appear to be more photo-oriented and are also highly diverse. We believe that the combination of a high frequency of the “background” word in the dataset and the diversity of associated images make it a good contrastive concept. However state-of-the-art results have been obtained by leveraging well-designed tricks [, , , ] to handle the background, showing the need to something more than just “background” as a contrastive concept.

### 3.4 Mining co-occurence-based contrastive concepts 𝒞 ⁢ 𝒞 D 𝒞 superscript 𝒞 𝐷 \mathcal{CC}^{D} caligraphic_C caligraphic_C start_POSTSUPERSCRIPT italic_D end_POSTSUPERSCRIPT

We have just discussed using the word “background” as a contrastive concept, which appears frequently, but not systematically, in VLM training data. However, we argue here that it is possible to *mine* better and more precise contrastive concepts from the VLM training data, which are specific to a query $q$.
To that end, we propose to use *co-occurrence* information to find contrastive concepts $\mathcal{CC}^{D}_{q}$, which are typically described in captions alongside the query concept $q$. This approach relies on the offline construction of a *co-occurrence dictionary*, which is built for a large list of concepts $\mathcal{T}$ extracted from the training data.

##### Co-occurence extraction.

We consider a set of textual concepts $\mathcal{T}$ extracted from the captions of the VLM training dataset and construct the co-occurrence matrix $X\in\mathbb{N}^{|\mathcal{T}|\times|\mathcal{T}|}$. Concretely, two concepts $\{i,j\}\subset\mathcal{T}$ co-occur if they appear simultaneously in the caption of an image. $X_{i,j}$ counts the number of times concepts $\{i,j\}$ co-occur in some images. Last, we normalize the symmetric matrix $X$ row-wise by the number of occurrences of concept $i$ in the dataset, producing the frequency matrix $\hat{X}$. We then consider only concepts with frequent co-occurrences: for each $i\in\mathcal{T}$, we select concepts $\mathcal{T}_{i}=\{j\in\mathcal{T}\mid\hat{X}_{i,j}>\gamma\}$, for some frequency threshold $\gamma$ ($=0.01$ in all our experiments).

##### Concept filtering.

To improve the quality of selected contrastive concept $\mathcal{T}_{i}$, we design a simple filtering pipeline. For each target concept $i\in\mathcal{T}$ (which may be thought of as a future query), we remove from $\mathcal{T}_{i}$ any concept that might interfere with $i$ and induce false negatives. First, we discard uninformative words that appear in captions: $\{$“image”, “photo”, “picture”, “view”$\}$. Then, we remove *abstract* concepts, such as “liberty”. To do so, we ask an LLM whether a given word can be visible or not in an image (more details in Appendix Sec. 6.3.2). We also filter out concepts that are too semantically similar to target concept $i$, e.g., such that their cosine similarity with $\phi_{\text{T}}(i)$ is more than a threshold $\delta$ ($=0.8$ in all experiments).

##### Generalization to arbitrary concepts.

We discussed so far how to select contrastive concepts $\mathcal{CC}^{D}_{i}$ for a target concept $i\in\mathcal{T}$. Now when we are given an arbitrary textual query $q$, to make the generation of contrastive concepts truly open-vocabulary, we first find in CLIP space the nearest neighbour $i$ of $q$ in $\mathcal{T}$ and then use for $q$ the contrastive concepts of $i$: $\mathcal{CC}^{D}_{q}$ = $\mathcal{CC}^{D}_{i}$.

### 3.5 Prompting an LLM to generate contrastive concepts 𝒞 ⁢ 𝒞 L 𝒞 superscript 𝒞 𝐿 \mathcal{CC}^{L} caligraphic_C caligraphic_C start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT

Instead of extracting contrastive concepts from the VLM training set, we investigate here another strategy, which is generating them using an LLM.
For a given text query $q$, we ask an LLM to directly generate contrastive concepts $\mathcal{CC}^{L}_{q}$, without the need for subsequent filtering. To that end, we design a prompt that tries to avoid producing words that are synonyms, meronyms (e.g., “wing” for “plane”) or possible contents (e.g., “wine” for “bottle”). This prompt is partially reproduced below (more details in the Appendix Sec. 6.3.2).

> Given an input object O, I want you to generate a list of words related to objects that can be surrounding input object O in an image to help me perform semantic segmentation. [… In-context examples …] You should not generate synonyms of input object O, nor parts of input object O. Generate a list of objects surrounding the input object $\mathcal{T}$ without any synonym nor parts, nor content of it. Answer with a list of words.

Using an LLM has the benefit of allowing us to produce specific contrastive concepts $\mathcal{CC}_{q}$ for any target query $q$. However, the cost to run an LLM, even a small one (in our case a Mixtral-8x7B Instruct), is high and unlikely to fit on small devices.

### 3.6 Generalizing to several concept queries

The above discussion is formulated for a single concept query $Q=\{q\}$. To generalize to any number of simultaneous queries $Q$, with $|Q|>1$, we consider the union of contrastive concepts $\mathcal{CC}_{q}$ for each query $q$,
however removing contrastive concepts that are too similar to queries, i.e., with a cosine similarity above $\beta$: $\mathcal{CC}_{Q}=\bigcup_{q\in Q}\{q^{\prime}\in\mathcal{CC}_{q}\mid\phi_{
\text{T}}(q^{\prime})\cdot\phi_{\text{T}}(q)\leq\beta\}$. Open-world multi-query segmentation is then just segmenting $Q\cup\mathcal{CC}_{Q}$ and ignoring pixels not assigned to queries, as in the single query case.

## 4 Experiments

In this section, we present our experimental setup (Sec. 4.1) and the background handling strategies of baselines (Sec. 4.2). Then, in Sec. 4.3, we discuss our finding, ablate our method in Sec. 4.4, and produce visualizations in Sec. 4.5.

### 4.1 Experimental setup

##### Baselines.

In our study, we consider several state-of-the-art methods for OVSS.
We study two training-free methods, which directly exploit CLIP-backbone, namely GEM [] and MaskCLIP []. We investigate the generalization of our contrastive concepts generation when considering the application of MaskCLIP to different OpenCLIP [] backbones pre-trained either on
LAION [], MetaCLIP [], or WIT400M [] dataset.
All compared methods use a *frozen* CLIP ViT-B/16.
We further discuss the background handling strategies in the Appendix Sec. 6.1.1.

##### 𝒞 ⁢ 𝒞 𝒞 𝒞 \mathcal{CC} caligraphic_C caligraphic_C generation.

For $\mathcal{CC}^{D}$generation, we use the metadata extracted by []. In particular, we use the gathered statistics for four thousand common concepts in the LAION-400M dataset, which is a subset of LAION-2B [] and used to train CLIP []. We filter contrastive concepts using a low co-occurrence threshold $\gamma=0.01$ and a high CLIP similarity threshold $\delta=0.8$. In multiple queries scenario, we use a threshold $\beta=0.9$ to account for possible similarities between one query and contrastive concepts corresponding to other queries from a set. We discuss the values selection in Appendix Sec. 6.2.1. When generating $\mathcal{CC}^{L}$, we use the recent Mixtral-8x7B-Instruct model []. More details about the setup can be found in Appendix Sec. 6.3.1 alongside our designed prompts in Appendix Sec. 6.3.2.
In our experiments, if not stated otherwise we use $\mathcal{CC}^{D}_{t}\leftarrow\{\text{``background''}\}\cup\mathcal{CC}^{D}_{t}$ and $\mathcal{CC}^{L}_{t}\leftarrow\{\text{``background''}\}\cup\mathcal{CC}^{L}_{t}$.

##### Evaluation datasets.

We conduct our experiments on six datasets widely used for the task of zero-shot semantic segmentation [], namely VOC [], Context [], COCO-Stuff [], COCO-Object [],
Cityscapes [], and ADE20K []. The object-centric datasets VOC and COCO Object have typically been evaluated considering or not the “background” class.
Additionally to our new metric described below, we evaluate results with the standard mIoU metric.
We also follow the protocol of [] detailed in Appendix Sec. 6.1.

##### Our IoU-single metric.

To better evaluate the capabilities of methods to localize a visual concept *when provided with no other information*, we propose IoU-single metric. It consists in using the classic mIoU, but this time considering one concept at a time. Indeed, we iterate over the depicted classes for each dataset image and provide each existing class one at a time, therefore always with $|\mathcal{Q}|=1$. We give a more intuitive illustration of our metric in Appendix Fig. 5.
If a dataset contains a *background* class, we do not consider it in the mIoU calculation.

### 4.2 Lack of generalization of the baselines

Before presenting the benefit of using co-occurrent concepts, we first discuss the lack of generalizability of the current background handling strategies used in recent works [, , ].

[Uncaptioned image]: extracted/5713349/figures/ablations/mountain+deser.png

##### Limitations of the objectness.

The authors of CLIP-DINOiser [] propose a foreground/background saliency generation to handle the segmentation of the “background”.
However, such saliency segmentation strategy is derived from FOUND [], which aims at discovering the *foreground objects* and therefore discards any pixel that “falls in the background” in their definition, but which might depict the queried concepts. The image in the inset presents such a failure case: the mountain (in green) is discarded because falling in the background.

##### Sensitive thresholds.

On the other hand,
we have observed that the thresholding approach of TCL, which simply consists of keeping pixels with a similarity to the query concept above a certain threshold, can hurt performances when the number of queried concepts increases.
In GEM, the thresholding is applied on the softmax probabilities therefore not applicable in the case of a single queried concept.

### 4.3 Contrastive concepts generation results

We present in Tab. 1 results obtained with our metric IoU-single on three different datasets, namely ADE20k, Cityscapes, and VOC. We compare results when using various types $\mathcal{CC}$ proposed in this work.

Moreover, we consider an *oracle*, which uses the in-domain dataset classes as contrastive concepts and
gives us a possible upper bound. We do not report it on VOC as its classes do not cover the ‘stuff’ pixels. We additionally report results on three more datasets in Appendix Tab. 7.

**Table 1: Results with IoU-single. We present results on VOC, Cityscapes and ADE20k with our metric IoU-single (Sec. 4.1). We note with ‘*’ when the original background handling is applied (provided it gives the best results). We note "BG" when using $\mathcal{CC}^{D}$. We report results with an ‘oracle’ corresponding to using the in-domain classes of the specific datasets (except VOC, which are too sparse and give bad performances).**
|  | CLIP | VOC | Cityscapes | ADE20k |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | training data | ‘BG’ | $\mathcal{CC}^{L}$ | $\mathcal{CC}^{D}$ | oracle | ‘BG’ | $\mathcal{CC}^{L}$ | $\mathcal{CC}^{D}$ | oracle | ‘BG’ | $\mathcal{CC}^{L}$ | $\mathcal{CC}^{D}$ |
| MaskCLIP [] | WIT400M | 44.2 | 52.2 | 53.4 | 30.6 | 15.0 | 22.5 | 22.1 | 29.8 | 20.2 | 23.5 | 25.2 |
| DINOiser [] | Laion2B | 59.4 | 63.2 | 64.8 | 36.0 | 23.0 | 30.7 | 27.4 | 35.3 | 27.1 | 29.5 | 31.5 |
| TCL [] | [] | 52.9* | 52.6* | 54.5* | 29.7 | 9.8 | 26.2 | 22.0 | 32.6 | 14.9* | 25.9 | 26.5 |
| GEM [] | MetaCLIP | 48.7* | 59.9 | 60.6 | 20.6 | 21.2* | 21.4 | 14.4 | 33.0 | 23.4* | 26.1 | 28.9 |

**(a) Impact of the filtering in $\mathcal{CC}^{D}$.**
| Method | M.CLIP | TCL | DINOi | GEM |
| --- | --- | --- | --- | --- |
| no filtering | 20.0 | 22.4 | 23.9 | 22.7 |
| *+ abstract* | 20.9 | 23.2 | 25.5 | 23.6 |
| *+ sem. similar* | 25.2 | 26.0 | 31.5 | 28.9 |

In all cases, we observe significant improvement when using generated contrastive concepts w.r.t. the “background” baseline. For both VOC and ADE20k datasets, the co-occurrence-based $\mathcal{CC}^{D}$outperforms LLM-based $\mathcal{CC}^{L}$, with a margin varying from 0.6 pt on ADE20k with TCL to 2.8 pts with GEM.
Interestingly, $\mathcal{CC}^{L}$gives the best results on Cityscapes for all methods. This may suggest that LLM can yield better results for more domain-specific tasks.

**Table 3: Results w/ mIoU.**
| Method | Bkg. | $\mathcal{CC}$ | Object | VOC |
| --- | --- | --- | --- | --- |
| M.CLIP | $\mathcal{CC}$ | $\mathcal{CC}^{BG}$ | 17.8 | 35.1 |
| $\mathcal{CC}$ | $\mathcal{CC}^{L}$ | 25.9 | 46.2 |  |
| $\mathcal{CC}$ | $\mathcal{CC}^{D}$ | 25.0 | 46.5 |  |
| DINOi | sal.[] |  | 34.8 | 62.1 |
| $\mathcal{CC}$ | $\mathcal{CC}^{BG}$ | 29.5 | 54.0 |  |
| $\mathcal{CC}$ | $\mathcal{CC}^{L}$ | 35.0 | 60.7 |  |
| $\mathcal{CC}$ | $\mathcal{CC}^{D}$ | 34.0 | 61.5 |  |

Finally, the *oracle* results show that there is still room for improvement for most of the methods. Yet, the *oracle* results also point out that, in the ideal scenario of having access to all the classes present on an image, there exists a clear limitation in the CLIP space that can only be pushed forward by improving text prompting rather than segmentation techniques themselves.

Additionally, we present results with the standard mIoU in Tab. 3 for MaskCLIP (‘M.CLIP’) and CLIP-DINOiser (‘DINOi’) both with Laion2B backbone. We study different contrastive concepts ($\mathcal{CC}$) and background handling strategies (Bkg.). We observe that in all cases the results produced using $\mathcal{CC}^{D}$and $\mathcal{CC}^{L}$are on par with results obtained with the saliency (noted ‘sal.[]’). We show more results in Appendix (Tab. 6).

### 4.4 Ablation studies

In this section, we discuss different ablation studies presented in Tab. 2.

##### 𝒞 ⁢ 𝒞 D 𝒞 superscript 𝒞 𝐷 \mathcal{CC}^{D} caligraphic_C caligraphic_C start_POSTSUPERSCRIPT italic_D end_POSTSUPERSCRIPT filtering.

In Tab. 2(a), we analyze the impact of the different filtering steps discussed in 3.4 on the ADE20k dataset under IoU-single. We observe that each step helps improve results by removing *noisy concepts*. The largest boost is observed when filtering the *abstract concepts* (“abstract”) and highly *semantic similarity* (“sem similar”). We also note that the improvement is consistent for all the methods.

##### Adding “background”.

In Tab. 2(b), we study the influence of adding “background”, noted “$\cup$‘BG”’, to the set of contrastive concepts $\mathcal{CC}^{L}$generated with the LLM.
We observe that the overall impact of the “background” is not significant, yet still yields better performance up to 2 IoU-single pts on ADE20k.

##### Generalizability to other pre-training datasets.

Tab. 2(c) shows the results of MaskCLIP with different pre-training datasets used to train CLIP. We observe that using $\mathcal{CC}^{D}$always gives a boost over using “background” alone across all pre-training datasets, including highly-curated MetaCLIP.
However, we notice that for MetaCLIP, $\mathcal{CC}^{L}$gives better results suggesting that leveraging LLMs can yield better results for backbones with carefully curated pre-training datasets.

### 4.5 Qualitative results

We present in Fig. 3 some qualitative examples when using different contrastive concepts concepts proposed in this work. We compare $\mathcal{CC}^{L}$and $\mathcal{CC}^{D}$with the ground truth (GT) and baseline $\mathcal{CC}^{BG}$. For both $\mathcal{CC}^{L}$and $\mathcal{CC}^{D}$, we present the output segmentation mask for the single queried concept (noted *pred*) as well as the overall semantic map produced with all contrastive concepts (denoted *all*). We observe that the output masks produced by our methods are more accurate and indeed remove the noise from related concepts, e.g. “tree” for the bird or “sofa” for the “bed”.

Figure: Figure 3: Qualitative results. We show the examples from ADE20K (top row) and Context (bottom row) and the segmentation masks produced by CLIP-DINOiser. For $\mathcal{CC}^{D}$and $\mathcal{CC}^{L}$, we additionally show the segmentation of contrastive classes (all).
Refer to caption: extracted/5713349/figures/qualitative/ade/1172/ADE_val_00001172_bg_bed_gt_blend.png

##### Generalization to arbitrary concepts.

Figure: Figure 4: In the wild examples. We visualize results for MaskCLIP and CLIP-DINOiser for query concepts beyond $\mathcal{T}$. The closest neighbour to a query is presented below each example (grey row).
Refer to caption: extracted/5713349/figures/examples/maskclip_muffin_laion_muffin_blend.png

We present in Fig. 4 results when prompting queries which are not included in
the subset of concepts $\mathcal{T}$ (extracted from the VLM training dataset), such as “muffin” or “cavalier” (dog breed). We show the closest neighbour for the query $q$ below each example and visualize masks for MaskCLIP as well as CLIP-DINOiser. We observe that the $\mathcal{CC}^{D}$generation method leveraging statistics from pre-training datasets is also robust to examples outside of vocabulary by accurately mapping $q$ to its closest concept in $\mathcal{T}$, e.g., “cavalier” mapped to “dog”.

## 5 Conclusions and limitations

##### Conclusions.

In this work, we identify limitations of the current evaluation setup for open-vocabulary semantic segmentation tasks which are inherited
from close-world evaluation benchmarks.
To bridge the gap between close- and open-world setups, we propose the single-class segmentation scenario. We study the limitations of current state-of-the-art models when we assume no prior access to in-domain classes and propose to automatically discover contrastive concepts $\mathcal{CC}$ that are useful to better localize any queried concept.
To do so, we employ two different strategies
and leverage either the distribution of co-occurrences in the VLM’s training set or use an LLM to generate such $\mathcal{CC}$. Our results show the generalizability of our proposed method across several setups. Our analysis reveals some of the other shortcomings of the currently accepted evaluation protocol for OVSS, i.e., limitations of using class names as they are for concept queries, which we leave for future work.

##### Limitations.

In this work, we leverage statistics extracted from the training set of CLIP. Despite its known data-collection problems, CLIP trained with LAION [] is still widely adopted. Therefore, we also base our analysis on LAION dataset by leveraging the thorough work of [] and their metadata.

## Acknowledgments

This work was supported by the National Centre of Science (Poland) Grant No. 2022/45/B/ST6/02817 and by the grant from NVIDIA providing one RTX A5000 24GB used for this project. We would also like to thank the authors of [] for sharing their metadata.

## 6 Appendix

### 6.1 Evaluation protocol

In all experiments, we follow the evaluation protocol of []. We use MMSegmentation [] implementation with a sliding window strategy and resizing input images to have a shorter side of 448.
We also do not perform text expansions of the class names and
use only the standard ImageNet prompts following [, , ].

#### 6.1.1 Background handling of baselines

We detail here the different strategies employed in [, , ] to handle the background.
TCL [] applies thresholding and considers pixels with maximal logits $\leq 0.5$ to be in the background.
GEM [] applies a background handling strategy, by thresholding the normalized logits, only for the VOC dataset and does not consider background otherwise. Therefore, we explore GEM performance both with and without background handling and report each time the better score.
MaskCLIP [] does not use any dedicated mechanism for background therefore we do not report the original setup for it. CLIP-DINOiser [] leverages a foreground/background saliency strategy which focuses on foreground pixels. In that case, the foreground/background is defined following FOUND [] which focuses on objectness and mainly discards pixels corresponding to ‘stuff’ that we might want.

#### 6.1.2 About the metric

Figure: Figure 5: Illustration of our metric.
Refer to caption: x4.png

We present in Fig. 5 an illustration of our proposed metric IoU-single. We illustrate the difference between the standard mIoU metric (dataset-driven mIoU), where all the concepts present on an image are considered at once. On the contrary, our IoU-single considered each of the present concepts separately to measure the single-class segmentation abilities of current open vocabulary semantic segmentation methods.

### 6.2 More quantitative results

#### 6.2.1 Hyperparameter selection

**Table 4: Parameter study of $\gamma$ and $\delta$. Selection (marked in grey) of the hyperparameters $\gamma$ and $\delta$ with IoU-single on 100 images randomly selected in ADE20k training dataset.**
|  |  | values of $\gamma$ | values of $\delta$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | CLIP tr. data | 0.001 | 0.005 | 0.01 | 0.015 | 0.02 | 0.95 | 0.9 | 0.85 | 0.8 | 0.75 |
| MaskCLIP [] | WIT400M | 24.4 | 26.0 | 24.8 | 24.4 | 23.2 | 19.9 | 21.0 | 23.0 | 24.4 | 22.8 |
| Laion2B | 25.8 | 27.8 | 27.4 | 26.0 | 25.4 | 23.0 | 24.1 | 26.4 | 27.4 | 24.6 |  |
| MetaCLIP | 22.0 | 24.1 | 24.4 | 23.8 | 23.4 | 22.7 | 23.7 | 25.9 | 27.2 | 23.7 |  |
| DINOiser [] | Laion2B | 24.4 | 27.2 | 27.9 | 27.9 | 27.7 | 23.5 | 24.6 | 26.4 | 27.9 | 26.9 |

In this section, we discuss the hyperparameter selection for our $\mathcal{CC}$ generation. For $\gamma$ and $\delta$, we randomly select 100 images from the training set of the ADE20K dataset and report IoU-single on this subset — which we observe was enough to select the values. We report in Tab. 4 a parameter study of both hyperparameters and mark in grey selected values, i.e., $\gamma=0.01$ and $\delta=0.8$.
For $\gamma$, we observe that values $\gamma<0.005$ are too low, most likely introducing too much noise in selected contrastive concepts.

We also present a parameter study for $\beta$ in Tab. 5. Here, we randomly select 100 images of VOC training set and report classic mIoU for different $\beta$ values. We select $\beta=0.9$ as it gives the best result for most methods. We also note that controlling the similarity between query concepts and contrastive concepts in the multiple-query scenario is necessary. Not including this step (see results for $\beta=1.0$) highly degrades performance.

**Table 5: Selection of $\beta$ with classic mIoU on 100 images randomly selected in VOC training dataset. We report results when using $\mathcal{CC}^{L}$.**
| Method | CLIP tr. data | 1.0 | 0.95 | 0.9 | 0.85 | 0.8 |
| --- | --- | --- | --- | --- | --- | --- |
| MaskCLIP [] | WIT400M | 26.0 | 40.4 | 41.1 | 39.1 | 32.1 |
| Laion2B | 35.3 | 43.7 | 44.0 | 44.6 | 42.2 |  |
| MetaCLIP | 24.4 | 39.1 | 40.3 | 34.3 | 30.6 |  |
| DINOiser [] | Laion2B | 51.3 | 57.8 | 58.6 | 58.8 | 55.2 |
| TCL | [] | 37.2 | 47.6 | 47.7 | 47.1 | 47.7 |

#### 6.2.2 State-of-the-art results under classic mIoU

**Table 6: Results with standard mIoU metric when employing different contrastive concept generation strategies.**
|  | Background | Type of | CLIP | Training | Dataset |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | handling | $\mathcal{CC}$ | backbone | dataset | Context | Object | VOC |
| GroupViT [] | threshold | $\varnothing$ | scratch | CC12M []+RedCaps [] | 18.7 | 27.5 | 50.4 |
| CLIP-DIY []^∗ | saliency-[] | $\varnothing$ | Laion2B | - | 19.7 | 31.0 | 59.9 |
| TCL [] | threshold | $\varnothing$ | WIT400M | CC12M []+CC3M [] | 24.3 | 30.4 | 51.2 |
| MaskCLIP [] [] | $\varnothing$ | $\varnothing$ | WIT400M | - | 21.1 | 15.5 | 29.3 |
| MaskCLIP^∗ | $\varnothing$ | $\varnothing$ | Laion2B | - | 22.9 | 16.4 | 32.9 |
| MaskCLIP^∗ ${\dagger}$ | $\varnothing$ | $\varnothing$ | Laion2B | - | 24.0 | 21.6 | 41.3 |
| CLIP-DINOiser [] | $\varnothing$ | $\varnothing$ | Laion2B | ImageNet [] (1k im.) | 32.4 | 29.9 | 53.7 |
| GEM [] | $\varnothing$ | $\varnothing$ | MetaCLIP | - | - | - | 46.8 |
| CLIP-DINOiser [] | saliency[] | $\varnothing$ | Laion2B | ImageNet [] (1k im.) | – | 34.8 | 62.1 |
| $\mathcal{CC}$ | $\mathcal{CC}^{BG}$ | Laion2B | ImageNet [] (1k im.) | 32.4 | 29.5 | 54.0 |  |
| $\mathcal{CC}$ | $\mathcal{CC}^{L}$ | Laion2B | ImageNet [] (1k im.) | 31.3 | 35.0 | 60.7 |  |
| $\mathcal{CC}$ | $\mathcal{CC}^{D}$ | Laion2B | ImageNet [] (1k im.) | 31.8 | 39.9 | 61.5 |  |
| MaskCLIP | $\mathcal{CC}$ | $\mathcal{CC}^{BG}$ | Laion2B | - | 23.6 | 17.8 | 35.1 |
| $\mathcal{CC}$ | $\mathcal{CC}^{L}$ | Laion2B | - | 22.5 | 25.9 | 46.2 |  |
| $\mathcal{CC}$ | $\mathcal{CC}^{D}$ | Laion2B | - | 23.2 | 25.0 | 46.5 |  |
| GEM | threshold | $\varnothing$ | MetaCLIP | - | 33.4 | 27.4* | 46.6* |
| GEM | $\mathcal{CC}$ | $\mathcal{CC}^{L}$ | MetaCLIP | - | 31.8 | 34.9 | 58.0 |
| GEM | $\mathcal{CC}$ | $\mathcal{CC}^{D}$ | MetaCLIP | - | 32.4 | 34.9 | 60.6 |

We report in Tab. 6 the results under the classic mIoU metric for selected state-of-the-art methods on open-vocabulary semantic segmentation. For each of the methods we detail the specific background handling techniques (if applied), CLIP backbone used as well as additional datasets used for the training. We notice that extending the dataset vocabulary with our generated contrastive concepts does not hurt the overall performance under a normal setup when all dataset labels are considered as prompts. For GEM and MaskCLIP we observe significant improvements over their original setups for VOC dataset. This holds for both contrastive concept generation methods $\mathcal{CC}^{D}$and $\mathcal{CC}^{L}$. Looking at the results of CLIP-DINOiser we observe that saliency is still more effective in the object-centric scenario.

**Table 7: Results on VOC with our metric IoU-single defined in Sec. 4.1. ‘*’ denotes the result when the original background handling gives the best results. "w/ BG" denotes using only “background” as the contrastive concept.**
| Method | CLIP dataset | Original | ‘oracle’ | w/ ‘BG’ | w/ $\mathcal{CC}^{L}$ | w/ $\mathcal{CC}^{D}$ |
| --- | --- | --- | --- | --- | --- | --- |
| VOC21 |  |  |  |  |  |  |
| MaskCLIP [] | Laion2B | – | 49.9 | 47.9 | 51.8 | 53.8 |
| WIT400M | – | 47.1 | 44.2 | 52.2 | 53.4 |  |
| MetaCLIP | – | 47.9 | 46.8 | 50.6 | 50.0 |  |
| DINOiser [] | Laion2B | 63.8* | 61.2 | 59.4 | 63.2 | 64.8 |
| TCL [] | [] | 52.9* | 53.0* | 52.9* | 52.6* | 54.5* |
| GEM | MetaCLIP | – | – | 48.7* | 59.9 | 60.0 |
| Cityscapes |  |  |  |  |  |  |
| MaskCLIP [] | Laion2B | – | 32.2 | 16.2 | 27.2 | 24.0 |
| WIT400M | – | 30.6 | 15.0 | 22.5 | 22.1 |  |
| MetaCLIP | – | 30.0 | 13.6 | 24.9 | 23.2 |  |
| DINOiser [] | Laion2B | 20.8* | 36.0 | 23.0 | 30.7 | 27.4 |
| TCL [] | [] | 18.6* | 29.7 | 9.8 | 26.2 | 22.0 |
| GEM | MetaCLIP | – | 20.6 | 21.2* | 21.4 | 14.4 |
| COCO-Stuff |  |  |  |  |  |  |
| MaskCLIP [] | Laion2B | – | 34.1 | 26.4 | 28.8 | 29.5 |
| WIT400M | – | 33.6 | 24.1 | 28.5 | 28.8 |  |
| MetaCLIP | – | 34.0 | 25.8 | 27.9 | 27.9 |  |
| DINOiser [] | Laion2B | 28.0* | 35.3 | 32.2 | 33.4 | 34.1 |
| TCL [] | [] | 25.0* | 34.7 | 17.4 | 29.5 | 30.6 |
| GEM | MetaCLIP | – | 38.3 | 22.9* | 32.2 | 33.6 |
| ADE20k |  |  |  |  |  |  |
| MaskCLIP [] | Laion2B | – | 33.2 | 22.7 | 26.8 | 27.8 |
| WIT400M | – | 29.8 | 20.2 | 23.5 | 25.2 |  |
| MetaCLIP | – | 32.1 | 21.5 | 24.7 | 26.0 |  |
| DINOiser [] | Laion2B | 28.8* | 35.3 | 27.1 | 29.5 | 31.5 |
| TCL [] | [] | 14.8* | 32.6 | 14.9* | 25.9 | 26.5 |
| GEM | MetaCLIP | – | 33.0 | 23.4* | 26.1 | 28.9 |
| COCO Object |  |  |  |  |  |  |
| MaskCLIP [] | Laion2B | – | 32.1 | 27.8 | 33.7 | 32.9 |
| WIT400M | – | 31.3 | 24.3 | 34.4 | 33.3 |  |
| MetaCLIP | – | 30.9 | 27.3 | 32.0 | 31.0 |  |
| DINOiser [] | Laion2B | 38.8* | 38.9 | 35.2 | 40.8 | 39.6 |
| TCL [] | [] | 37.2* | 37.9 | 37.1* | 38.1* | 37.2* |
| GEM | MetaCLIP | – | – | 31.6 | 39.0 | 40.1 |
| Pascal Context |  |  |  |  |  |  |
| MaskCLIP [] | Laion2B | – | 40.5 | 34.4 | 35.2 | 37.4 |
| WIT400M | – | 41.1 | 32.9 | 33.7 | 36.8 |  |
| MetaCLIP | – | 41.1 | 32.6 | 34.1 | 35.7 |  |
| DINOiser [] | Laion2B | 33.9* | 45.8 | 41.3 | 41.3 | 44.0 |
| TCL [] | [] | 29.7* | 41.7 | 29.7* | 36.8 | 38.2 |
| GEM | MetaCLIP | – | – | 27.0 | 40.1 | 42.0 |

#### 6.2.3 More results

Tab. 7 presents results obtained with the metric IoU-single on all datasets studied.

### 6.3 Prompting the LLM

We provide in this section more details about the LLM and the prompts used.

#### 6.3.1 The LLM model

We use the recent Mixtral-8x7B-Instruct model [] a sparse mixture of experts model (SMoE), finetuned for instruction following, released by Mistral AI. In particular, we rely on the v0.1 version of its open weights available via Hugging Face transformers library. We run the LLM in 4-bit precision with flash attention to speedup inference.

#### 6.3.2 The prompts

We provide below the prompts used to generate the contrastive concepts $\mathcal{CC}^{L}$as well as those used to predict whether a concept can be seen in an image or not in order to filter $\mathcal{CC}^{D}$.

In the following prompts, we indicate the inserted input text as $\{q\}$. We follow Mixtral-8x7B Instruct’s prompt template. In particular, we use 
 as the beginning of the string (BOS) special token, as well as 
 and 
 as string markers to be set around the instructions.

For the generation of $\mathcal{CC}^{L}$, we also integrate
a light post-processing step, ensuring that all generated lists have a unified format with coma separation. We do not apply a filtering or cleaning step to the LLM-generated results.

##### Contrastive concepts generation.

> 
 
 You are a helpful AI assistant with visual abilities. Given an input object O, I want you to generate a list of words related to objects that can be surrounding input object O in an image to help me perform semantic segmentation. For example: * If the input object is ’fork’, you can generate a list of words such as ’["bottle", "knife", "table", "napkin", "bread"]’. * If the input object is ’child’, you can generate a list of words such as ’["toy", "drawing", "bed", "room", "playground"]’. You should not generate synonyms of input object O, nor parts of input object O. Generate a list of objects surrounding the input object $\{q\}$ without any synonym nor parts, nor content of it. Answer with a list of words. No explanation. Answer: 


##### Visible or not prediction.

> 
 
 Please specify whether $\{q\}$ is something that one can see. Reply with ’yes’ or ’no’ only. No explanation. Answer: 


### 6.4 Parts removal via LLM-prompting

Figure: Figure 6: Parts removal. We show an example of $q=$bicycle from Pascal Context and the segmentation masks produced by MaskCLIP and CLIP-DINOiser for $\mathcal{CC}^{D}$as well as for $\mathcal{CC}^{D}$when parts of objects are removed ($\mathcal{CC}^{D}$$-$ parts).
Refer to caption: extracted/5713349/figures/examples/2008_008337_llm_bicycle_gt_blend.png

We also explore the possibility of removing suggested contrastive concepts that can be *parts* of query concepts. Note that in $\mathcal{CC}^{L}$, we explicitly do it in the prompt itself. Fig. 6 presents one of such examples when removing “wheel” from the $\mathcal{CC}^{D}$of query “bicycle” gives a slight improvement for MaskCLIP segmentation. However, we do not notice a particular improvement in the case of other segmentation methods, since typically they refine the masks or feature maps to include localization priors. For example, in Fig. 6, the second row presents the same example for CLIP-DINOiser (DINOiser), where the improvement is marginal. Finally, we observe little or no quantitative improvement when applying part removal filtering on entire datasets. Therefore, we do not include it in our final method.

For part prediction, we use the following prompt:

> 
 
 You are a helpful AI assistant with visual abilities. Given an input object O, I want you to generate a list of words that are parts of an object O. For example: * If the input object is ’rabbit’, you can generate a list of words such as ’["paw", "tail", "fur", "ears", "muzzle"]’. * If the input object is ’building’, you can generate a list of words such as ’["door", "window", "wall", "hall", "floor"]’. Generate a list of parts of the input object $\{q\}$. Answer with a list of words. Do not give any word that is not a part of the input object. No explanation. Answer: 


### 6.5 Example of generated 𝒞 ⁢ 𝒞 L 𝒞 superscript 𝒞 𝐿 \mathcal{CC}^{L} caligraphic_C caligraphic_C start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT

We provide in Tab. 8 an exemplary result of $\mathcal{CC}^{L}$for Cityscapes dataset.

**Table 8: Example of LLM-generated $\mathcal{CC}^{L}$for Cityscapes.**
| Query $t$ | $\mathcal{CC}^{L}$ |
| --- | --- |
| road | building, tree, car, pedestrian, sky, streetlight, sidewalk, bicycle, parked car, traffic sign |
| sidewalk | building, street, car, tree, people, bike, road, park, sky, lane |
| building | sky, tree, road, car, park, people, lane, fence, house, field |
| wall | door, window, floor, ceiling, painting, light, chair, table, carpet, curtain |
| fence | grass, tree, house, car, path, post, gate, field, flowers, animals |
| pole | building, wire, tree, street, sky, fence, cable, road, banner, light |
| traffic light | road, car, building, pedestrian, sky, streetlight, traffic sign, parking meter |
| traffic sign | road, street, pole, vehicle, building, sky, pedestrian, curb, lane, light |
| vegetation | soil, tree, grass, water, animal, fence, field, sky, rock, sun |
| terrain | tree, sky, building, road, mountain, river, field, fence, vehicle, person |
| sky | tree, building, cloud, sun, bird, airplane, mountain, sea, sunset, cityscape |
| person | bike, road, car, tree, building, park, cityscape, nature, animal, sports equipment |
| rider | bicycle, road, nature, park |
| car | road, tree, building, person, parking |
| truck | road, car, building, tree, parking |
| bus | road, tree, building, sky, person, car, traffic light, bicycle, parking meter, street sign |
| train | track, grass, sky, building, platform, tree, sign, person, car, road |
| motorcycle | road, person, bike, car, traffic, building, nature, parking, city, scenery |
| bicycle | road, tree, person, park, building, grass, basket, helmet, traffic, path |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness part.
soundness: 3
strengths: 1. Automatic discovery of comparison concepts is valuable because artificially defining comparison concepts requires professional knowledge and a manual preset knowledge base in advance.
2. I agree that the concept of "background" is confusing in segmentation but is not well discussed. Therefore, the article's exploration of "background" is valuable. This paper discussed the CLIP-based method, in which the pixel is scored by cosine similarity, and the threshold filters the background. However, thresholds are difficult to set and often overfit specific datasets, which is a real pain point.
3. Experiments on multiple datasets and verification of multiple methods prove the effectiveness of the proposed method.
summary: The paper discussed the 'background' concept in open-world semantic segmentation and proposed a method to suppress background with contrastive concepts when segmenting a specific category in test time. In addition, a new single-query evaluation setup for open-world semantic segmentation that does not rely on any domain knowledge is proposed, and a new metric is designed to evaluate the grounding of visual concepts. I think the "background" is confusing in segmentation but is not well-discussed and well-defined. It is valuable to study this problem.
weaknesses: 1. When using LLM for contrastive concepts extraction, do you consider generating corresponding contrastive concepts for each image (such as inputting the image along with the category name), and whether this is better than just asking with text? (For example, if you type a boat and LLM will tell you water, person, beach), but if a boat is stranded on the beach and you enter the image together, LLM(or VLM) will only output 'beach', which is more targeted when you divide it later.
2. How will this method be handled when the same pixels (or region) belong to the whole and parts (such as person and arm) at the same time? Because for ‘arm’, the rest region of the ’person‘ is the background. Can you give an example to describe the process briefly?
2. Can the background class be circumvented if the similarity is calculated using ‘sigmoid’ instead of ‘softmax’? You can have a try.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Compared to the results of CC^{L}, the performance gains achieved by using CC^{D} appear minimal. Could this imply that simple language model-generated concepts might be sufficient for open-world segmentation? For example, in Table 1, CC^{L} can get even better results than CC^{D} on cityscapes. If this is true, then it seems like that we only need to have "hard-negative" mining to make the open-set segmentation model work?
soundness: 2
strengths: The most interesting part of this paper, in my opinion, is its deep dive into how current open-vocabulary segmentation models still lean on fixed, closed-set categories during evaluation. This is a real issue if we want these models to work effectively in real-world settings, where they’ll face plenty of unknown objects and only the object of interest is provided in the prompt. 

Beside that, the paper brings some interesting contributions to the table:

1. [Test-Time Contrastive Concept Generation]: The authors introduce a fresh approach that automatically creates contrastive concepts during testing to boost segmentation in open-world scenarios. This context-specific contrast helps separate objects better without needing pre-set information.

2. [Background Handling Analysis]: They also dig into the common practice of treating "background" as a one-size-fits-all contrastive concept, showing its drawbacks and making a strong case for using concepts tailored to each query for more accurate results.

3. [New Metric for Open-World Segmentation]: To better measure performance, they propose IoU-single, a metric that assesses each concept’s segmentation independently, giving a clearer view of how well models handle open-world data.

4. [Extensive Benchmarking]: Lastly, they put their method to the test across various datasets, showing it consistently beats current open-vocabulary segmentation models, proving it’s robust and ready for broader applications.
summary: This paper introduces a method to open-world semantic segmentation using test-time contrastive concepts (CC), addressing the challenge of segmenting specific objects in images with minimal prior information. Current Vision-Language Models (VLMs), like CLIP, assign each pixel to one of many known visual concepts. However, this method struggles when only one or a few objects are specified, especially if relevant context (e.g., related background objects) is unknown. To solve this, the authors propose generating contrastive concepts at test time, using either VLM training data or large language models (LLMs), to improve segmentation without needing extensive dataset-specific knowledge. This approach represents a step towards effective open-world segmentation by dynamically enhancing contrast without altering VLM structures or needing extensive dataset-specific adaptations. Authors evaluated the model performance on various popular benchmarks and observed consistent performance gains.
weaknesses: 1. [Problem setting] As noted in the strengths, evaluating the model's performance using only a textual prompt is a valuable contribution and can address issues faced by many existing models. However, this approach overlaps significantly with the objectives of referring segmentation and expression localization models, which aim to segment or localize objects based on a text prompt. It would be beneficial to include comparisons with existing referring segmentation or detection models to provide a clearer benchmark. While this was briefly acknowledged in the related work section, I didn't find any quantitative or qualitative comparisons. Additionally, it would be necessary to compare with works like [1], which also supports segmentation/detection of multiple instances using a single text prompt.

2. [Re-define some existing concepts] This paper labels its setting as "open-world," a term already widely used in publications, such as [2]. Redefining this established concept might lead to confusion. It may help to clarify or differentiate this setting further to avoid conflicting terminology.

3. [Paper presentation] Some tables are unclear and difficult to interpret without the context provided in the paper. For example, Table 1 lacks a clear baseline for comparison—it appears to only show different CC variants without a standard baseline to measure the gains introduced by the proposed CC method. After reading the experimental section, I understood that CC^{BG} is the baseline, but its labeling as a variant rather than a baseline is confusing. The evaluation metric is also missing from the caption, which adds to the ambiguity.


Minor Issues: There are some minor typos and grammatical errors. For example, on line 53, "without any a priori knowledge" should be "without any prior knowledge."

[1] Liu, Chang, Henghui Ding, and Xudong Jiang. "Gres: Generalized referring expression segmentation." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23592-23601. 2023.

[2] Sodano, Matteo, Federico Magistri, Lucas Nunes, Jens Behley, and Cyrill Stachniss. "Open-World Semantic Segmentation Including Class Similarity." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3184-3194. 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In what scenarios will the LLM-based method outperform the co-occurrence-based method? Additionally, how can one determine which method is optimal for a particular dataset?
2. Is the proposed method also suitable for multi-domain evaluation, similar to CAT-Seg?
3. Please provide some examples where the proposed method fails.
soundness: 3
strengths: 1. Promising results with extensive experiments.
2. A new problem in OVSS has been identified, and a new setting of open-world open-vocabulary semantic segmentation is posed.
3. The method is simple to implement.
summary: This study identified a significant issue with existing OVSS methods, which is the necessity of providing an exhaustive list of all potential visual concepts for an image to be segmented. To address this, the authors proposed a more practical scenario where only a single concept is segmented based solely on a textual prompt. To improve performance, they introduced two methods for expanding contrastive concepts: co-occurrence-based and LLM-based. Experimental results across multiple datasets demonstrate the effectiveness of their proposed methods.
weaknesses: 1. The problem addressed in the study lacks in-depth analysis. While the authors mention that CLIP's feature space is not easily separable, citing Miller et al.'s work, it should be noted that Miller et al.'s focus was primarily on classification and object detection tasks. Therefore, it is uncertain whether their conclusions are applicable to the image segmentation task. To provide a more comprehensive analysis, the authors should delve deeper into the feature space for the image segmentation task, comparing scenarios with and without the use of contrastive concepts.
2. The straightforward descriptions of the proposed method make it challenging to comprehend. To enhance clarity, the authors should incorporate specific examples or a conceptual diagram to illustrate their approach.
3. Comparing the proposed method directly to baselines like CAT-Seg may not be entirely fair. These baselines could potentially be enhanced through simple modifications, such as randomly selecting a subset of class names for each image during the training phase. Such adjustments could make them more suitable for the proposed setting, which involves a single query. Therefore, it would be prudent to consider such potential improvements before making direct comparisons.
4. The comparison to the classical method SAN is missing: Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2945–2954, 2023
5. To demonstrate the effectiveness of selecting/generating contrastive concepts for each query, the following two experiments are suggested to add:
    * use the full set of textual concepts $\mathcal{T}$
    * use LLM to generate many concepts (say 500) for a certain dataset
At both the training and test stages, the full set of concepts can be suplied.

Meta-review: This paper introduces a test-time method for open-vocabulary semantic segmentation (OVSS) that suppresses false positive segments using contrastive concepts (CC) when segmenting a specific category in test time. The authors propose generating contrastive concepts at test time using either VLM training data or large language models (LLMs), which can be applied to off-the-shelf OVSS methods to improve performance. In addition, a new single-query evaluation setup and a new metric are also proposed to evaluate the grounding of visual concepts. 

While all reviewers appreciated the authors’ investigation into the issue of requiring pre-set information on all potential visual concepts in OVSS, they also raised concerns about the limited technical contribution, lack of in-depth analysis, missing comparisons, and unclear presentations, giving borderline scores. After discussion, some issues were addressed, but no reviewers strongly supported this paper, retaining borderline scores. The main issue, among others, is the limited technical contribution; reviewer dLMb pointed out that enabling the sigmoid-based classification would be more reasonable and generalizable in open scenarios rather than using the proposed test-time trick, and reviewer ZBpw had concerns that relying on LLM or the information of VLM training set may not be generalizable in many practical downstream applications. After carefully reading the manuscript, the reviews, and the discussions, AC agreed that the technical contribution of this paper is limited; the proposed method suppresses false positive predictions as post-processing in test time using train-set information or LLM rather than making the predictor itself more robust to background clutter. In particular, such background suppression/filtering tricks have been suggested earlier, e.g., in (Wysoczanska et al., 2024a;b), and the proposed method does not significantly outperform them; e.g., the saliency method (Wysoczanska et al., 2024b) shows comparable or better results in Table 2 and Table 7. It is unclear what the merit of the proposed method is compared to such alternatives.

Decision: Reject