A Simple Framework for Open-Vocabulary Zero-Shot Segmentation

## Abstract

Abstract Zero-shot classification capabilities naturally arise in models trained within a vision-language contrastive framework. Despite their classification prowess, these models struggle in dense tasks like zero-shot open-vocabulary segmentation. This deficiency is often attributed to the absence of localization cues in captions and the intertwined nature of the learning process, which encompasses both image representation learning and cross-modality alignment. To tackle these issues, we propose SimZSS , a Sim ple framework for open-vocabulary Z ero- S hot S egmentation. The method is founded on two key principles: i) leveraging frozen vision-only models that exhibit spatial awareness while exclusively aligning the text encoder and ii) exploiting the discrete nature of text and linguistic knowledge to pinpoint local concepts within captions. By capitalizing on the quality of the visual representations, our method requires only image-caption pairs datasets and adapts to both small curated and large-scale noisy datasets. When trained on COCO Captions across 8 GPUs, SimZSS achieves state-of-the-art results on 7 out of 8 benchmark datasets in less than 15 minutes.

## 1 Introduction

Semantic segmentation stands as a cornerstone task within the realm of computer vision, playing a pivotal role in various applications ranging from autonomous driving to medical image analysis. However, its widespread adoption and scalability are hindered by the inherent label-intensive nature of traditional methods, demanding copious amounts of fine-grained annotations for training. Furthermore, traditional approaches, e.g.,  Strudel et al. (); Cheng et al. () are typically closed-vocabulary, meaning they only work for a pre-defined set of categories and generalize poorly to unseen classes. Self-supervised learning paradigms (Caron et al., ; Zhou et al., ; He et al., ; Balazevic et al., ; Lebailly et al., ), which use pretext tasks to learn discriminative representations from data itself, offer a promising solution to alleviate the annotation burden. Representations obtained in this manner are typically clustered semantically, potentially even at a fine-grained
level (Balažević et al., ; Lebailly et al., ; Stegmüller et al., ) and, as such, yield excellent performance in various applications. Nonetheless, a degree of labeling is still necessary, whether for finetuning or constructing the support set of a $k$-NN classifier, for instance.

Recently, vision-language contrastive learning has proven to be a simple and effective approach for transforming web-scale amounts of noisy image-caption pairs into zero-shot classification capabilities (Radford et al., ; Jia et al., ). Nonetheless, the substantial
computational and data requirements of these methods pose significant challenges. To address these limitations, LiT (Zhai et al., ) proposed aligning a text encoder with a pretrained and frozen pure vision model. By decoupling the image representation learning process from the vision-language alignment process, this methodology enhances both computational and data efficiency while also improving performance in zero-shot classification tasks. Orthogonal to this, various studies (Hamilton et al., ; Siméoni et al., ; Wang et al., ; Zhang et al., ; Siméoni et al., ) have observed and capitalized on the spatial awareness of self-supervised pretrained vision transformers (ViTs). These works have demonstrated remarkable capabilities for dense downstream tasks with minimal or even zero trainable parameters (Balazevic et al., ). This evidence suggests that self-supervised vision transformers may just be one text encoder away from becoming open-vocabulary segmenters. We investigate this hypothesis and propose a simple framework for open-vocabulary zero-shot segmentation. In essence, we leverage linguistic knowledge to identify concepts in captions and build upon the quality of a frozen vision tower to retrieve corresponding concepts in images.

Our main contributions are as follows:

- (i)
SimZSS is designed to be simple, with minimal hyperparameters, making it highly compatible with both small curated datasets and large-scale noisy datasets.
- (ii)
The proposed framework is robust, supporting various pretrained vision towers, and accommodating both supervised and self-supervised pretraining of the vision backbone.
- (iii)
By decoupling visual representation learning from cross-modality concept-level alignment, our proposed framework, SimZSS achieves high computational and data efficiency.

## 2 Related works

##### Open-vocabulary learning.

Traditionally, computer vision methods operate under the closed-vocabulary hypothesis. This assumption presumes that all object categories a model is expected to classify, detect, or segment at test time are already known and labeled during training. This presents significant challenges due to the extensive labeling required and the limited generalizability of the resulting models. Open-vocabulary learning aims to eliminate these limitations. Notably, this is a more challenging objective than the one targeted by self-supervised visual representation learning (Chen & He, ; Caron et al., ; Li et al., ; Grill et al., ; Oquab et al., ), which assumes the availability of labels at test time. To address this additional constraint, a text encoder is typically trained jointly with the vision tower to maximize vision-language alignment on large amounts of image-caption pairs via contrastive learning (Radford et al., ; Jia et al., ). Subsequent studies have either focused on improving the cross-modality alignment pretext task (Alayrac et al., ; Yu et al., ; Tschannen et al., ) or on improving the computational efficiency (Zhai et al., ; Li et al., ). However, while these models excel at classification, their performance on dense downstream tasks is usually subpar.

##### Open-vocabulary segmentation.

The shortcomings of open-vocabulary methods on dense downstream tasks have been the focus of significant research efforts. A recurring approach in the literature is to use dense annotations for learning segmentation and to leverage a pretrained text encoder to provide the weights of a generalizable classifier (Li et al., ; Ghiasi et al., ; Liang et al., ; Ding et al., ; Zhou et al., ; Xu et al., ; Ma et al., ; Yu et al., ). While these methods demonstrate excellent zero-shot segmentation performance, they only partially alleviate the pixel-level annotation requirements.

To overcome this limitation, approaches that do not require segmentation masks have been proposed. PACL (Mukhoti et al., ) demonstrates the effectiveness of training only the projections at the interface of the two modalities and bootstrapping patch-level cross-modal similarities. Similarly, TCL (Cha et al., ) leverages existing fine-grained similarities to jointly learn text-grounded masks and impose contrastive region-text alignment based on the obtained masks. Alternatively, GroupViT Xu et al. () proposes a specialized architecture that performs hierarchical grouping/pooling of visual tokens based on their similarity with learnable query tokens. Another line of work showed the benefits of integrating a consistency objective between augmented and/or masked views of the input image (Dong et al., ; Ren et al., ) in the context of vision-language contrastive learning. Related to that, ReCo (Shin et al., ) reported improved localization capabilities by exploiting the co-occurrence of objects across multiple images. Finally, Wysoczańska et al. (; ); Rewatbowornwong et al. () combine the vision-language alignment capabilities of CLIP (Radford et al., ) with the spatial-awareness of self-supervised vision transformers, e.g., DINO (Caron et al., ) to develop open-vocabulary zero-shot segmenter.

Figure: Figure 2: Overview of SimZSS. On the text side (a.), each concept in the caption is represented using a trainable text encoder. On the vision side (c.), visual representations of each concept are obtained via a similarity-based pooling of the visual tokens. These visual concept representations are then projected onto a linear classifier, with weights derived from the text concept representations of the current batch. Cross-modality consistency is enforced using cross-entropy loss (b.).
Refer to caption: x2.png

## 3 Method

### 3.1 Preliminaries

We first provide a brief primer on the terminology used throughout the paper.

##### Dense representation.

Transformers encode input signals as a sequence of tokens, and we refer to the corresponding output sequence as the dense representation. For vision transformers, this corresponds to a tensor $\bm{z}_{v}\in\mathbb{R}^{n_{v}\times d_{v}}$, where $d_{v}$ is the dimension of the representation space and $n_{v}$ is the number of patches in the input image. For text input, the dense representation $\bm{z}_{t}\in\mathbb{R}^{n_{t}\times d_{t}}$ is a sequence of $d_{t}$-dimensional tokens. The number of tokens, $n_{t}$, is determined by the byte-pair encoding algorithm (Gage, ).

##### Local representation.

We refer to the result of indexing a dense representation with a sequence index $i$ as a local representation. This translates to $\bm{z}_{t}^{i}\in\mathbb{R}^{d_{t}}$ for textual input signals and $\bm{z}_{v}^{i}\in\mathbb{R}^{d_{v}}$ for visual input signals.

##### Concept representation.

Aggregating local representations corresponding to the $l^{th}$ semantic concept in the input signal yields a concept representation. For visual input, this is denoted as $\mathbf{c}_{v}^{l}\in\mathbb{R}^{d_{v}}$, whereas for textual input, we use $\mathbf{c}_{t}^{l}\in\mathbb{R}^{d_{t}}$.

##### Global representation.

A representation of the entire input signal is referred to as the global representation. Specialized tokens are typically used for this purpose. In vision transformers, the global representation $\bar{\bm{z}}_{v}\in\mathbb{R}^{d_{v}}$ is the [CLS] token, while for a text encoder $\bar{\bm{z}}_{t}\in\mathbb{R}^{d_{t}}$ denotes the [EOS] token.

### 3.2 Vision & language concept identification

At a conceptual level, enforcing cross-modality consistency faces a significant challenge due to the complexity of identifying and matching concepts across modalities. In a nutshell, our proposed solution leverages the discrete nature of textual data, enabling the straightforward identification of concepts within captions. Subsequently, we retrieve associated visual concepts, conditioned on those identified within the text, effectively circumventing the complexity of cross-modality concept matching.

#### 3.2.1 Textual concept identification

Given an image-caption pair $(\bm{x}_{v},\bm{x}_{t})$, we utilize the text modality to identify semantic concepts. Language adheres to structured grammatical rules, allowing us to leverage linguistic knowledge to pinpoint the main subjects and objects within a sentence. Since captions are intended to describe images, the primary subjects and objects in the captions are likely to correspond to visual elements in the images. To achieve this, we employ a part-of-speech (POS) tagger, which automatically classifies words according to their grammatical roles. In our context, noun phrases (NPs) are of particular interest, as they generally encapsulate the main subjects and objects in the sentences, providing a direct link to the visual concepts depicted in the images. However, captions can be quite noisy, so further filtering is necessary to discard NPs that are unlikely to appear in the images. To address this, we refine a noun phrase into a concept by restricting it to the noun and its first compound. We then discard any concept that is absent in a predefined bank of concepts.

Once the concepts in the caption have been identified, it remains to obtain their equivalent in the representation space. This is accomplished by tracking the token indices spanned by each concept. Denoted as $\mathcal{S}_{l}$, this set represents the indices associated with the $l^{th}$ concept. The concept representation $\mathbf{c}_{t}^{l}\in\mathbb{R}^{d_{t}}$ is then derived by averaging the local representation at the corresponding indices:

$$ $\mathbf{c}_{t}^{l}=\frac{1}{|\mathcal{S}_{l}|}\sum_{i\in\mathcal{S}_{l}}\bm{z} _{t}^{i}$ (1) $$

The $l^{th}$ textual concepts $\mathbf{c}_{t}^{l}\in\mathbb{R}^{d_{t}}$ is then mapped to the visual space via a linear projection $g:\mathbb{R}^{d_{t}}\rightarrow\mathbb{R}^{d_{v}}$:

$$ $\tilde{\mathbf{c}}^{l}_{t}=g\left(\mathbf{c}_{t}^{l}\right)$ (2) $$

In the remainder of this paper, we also refer to $\tilde{\mathbf{c}}^{t}_{t}$ as a text concept.

#### 3.2.2 Visual concept identification

Upon identifying the $l^{th}$ text concept $\tilde{\mathbf{c}}^{l}_{t}\in\mathbb{R}^{d_{v}}$, we use it to query the dense visual representation and obtain the corresponding visual concept $\mathbf{c}_{v}^{l}\in\mathbb{R}^{d_{v}}$. To do this, we first embed the image $\bm{x}_{v}$ using a vision encoder $f_{v}$, which outputs the dense visual representation $\bm{z}_{v}\in\mathbb{R}^{n_{v}\times d_{v}}$ and a global visual representation $\bar{\bm{z}}_{v}\in\mathbb{R}^{d_{v}}$. We then compute the similarities between each local visual representation and the query text concept:

$$ $\mathbf{s}=\texttt{softmax}\left(\frac{\bm{z}_{v}\tilde{\mathbf{c}}_{t}^{l}}{ \tau}\right)$ (3) $$

where $\tau$ is a temperature parameter that regulates the sharpness of the similarity (see Tab. 7). Consequently, the $l^{th}$ visual concept is obtained by bootstrapping existing cross-modality similarities, i.e., via similarity-based pooling (see Fig. 2):

$$ $\mathbf{c}_{v}=\bm{z}_{v}^{\top}\mathbf{s}$ (4) $$

This process is performed in parallel for all concepts from the $b$ captions in the batch, resulting in $\tilde{b}$ cross-modality concept pairs for which consistency can be encouraged.

### 3.3 Cross-modality consistency

In Section 3.2, we outline a methodology for identifying pairs of local concepts across textual and visual modalities. This method leverages established similarities and thus may benefit from additional supervision at the beginning of training, as discussed in Section 3.3.1.

#### 3.3.1 Global consistency

We now discuss the global consistency objective, which ensures similarities between images and captions. To this end, let $\bar{\bm{Z}}_{v}\in\mathbb{R}^{b\times d_{v}}$ be the matrix containing all global visual representations $\bar{\bm{z}}_{v}$ within a batch, and let $\bar{\bm{Z}}_{t}\in\mathbb{R}^{b\times d_{t}}$ be its equivalent in the text modality. After projecting the text concepts to the visual space, we can compute the global cross-modality similarity:

$$ $\bar{\mathbf{s}}=g\left(\bar{\bm{Z}}_{t}\right)\bar{\bm{Z}}_{v}^{\top}$ (5) $$

The learning objective is to maximize the similarity of paired entries and minimize that of unpaired ones:

$$ $\mathcal{L}_{g}=-\frac{1}{2b}\sum_{i}\log\left(\frac{\exp\left(\bar{\mathbf{s} }_{ii}\right)}{\sum_{j}\exp\left(\bar{\mathbf{s}}_{ij}\right)}\right)-\frac{1} {2b}\sum_{j}\log\left(\frac{\exp\left(\bar{\mathbf{s}}_{jj}\right)}{\sum_{i} \exp\left(\bar{\mathbf{s}}_{ij}\right)}\right)$ (6) $$

Overall the global consistency objective is identical to the one used in CLIP (Radford et al., ).

#### 3.3.2 Concept-level consistency

After obtaining pairs of vision-language concepts, an intuitive approach to enforce consistency at the concept level is to use a contrastive objective, akin to the one described in Section 3.3.1, but applied between concepts. Empirically, we found that this approach did not yield the desired performance improvements on dense downstream tasks. At a global level, images and captions represent complex scenes, supporting the hypothesis that only $b$ positive image-caption pairs exist among the $b^{2}$ possible ones in a batch. Conversely, concepts typically encode individual objects that are likely to occur multiple times within a batch. This suggests that an instance-level objective is ill-suited for our setting. Therefore, we opt for a semantic-level objective.

Let’s define $\mathbf{C}_{t}\in\mathbb{R}^{\tilde{b}\times d_{t}}$ representing the set of all $\tilde{b}$ text concepts in a batch and $\mathbf{C}_{v}\in\mathbb{R}^{\tilde{b}\times d_{v}}$ as its counterpart in the visual space. Thanks to the discrete nature of text, it is straightforward to keep track of concepts occurring in a batch, with each unique concept assigned a specific index, conveniently stored in $\mathbf{q}\in\{0,1,...,k-1\}^{\tilde{b}}$ (where $k$ denotes the number of unique concepts in a batch). The weights $\mathbf{h}\in\mathbb{R}^{k\times d_{v}}$ of a linear classifier can be computed by summing the representations of identical concepts:

$$ $\mathbf{h}_{i}=\sum_{j}\mathds{1}_{\{\mathbf{q}_{j}=i\}}g\left(\mathbf{C}_{t} \right)_{j}$ (7) $$

After $l2$-normalizing the columns of both $\mathbf{h}$ and $\mathbf{C}_{v}$, we project one onto the other to derive a probability distribution over the visual concepts:

$$ $\mathbf{p}=\underset{k}{\texttt{softmax}}\left(\mathbf{C}_{v}\mathbf{h}^{\top}\right)$ (8) $$

It follows that the cross-entropy loss can be used to ensure consistency between the query text concept and the retrieved visual concept:

$$ $\mathcal{L}_{l}=\frac{1}{\tilde{b}}\sum_{i}\sum_{j}-\mathds{1}_{\{\mathbf{q}_{ i}=j\}}\log\left(\mathbf{p}_{ij}\right)$ (9) $$

The overall objective of SimZSS, denoted as $\mathcal{L}_{\text{tot}}$, is a weighted sum of the global and local consistency objectives:

$$ $\mathcal{L}_{\text{tot}}=\mathcal{L}_{g}+\lambda\mathcal{L}_{l}$ (10) $$

where $\lambda$ is a weighting parameter whose impact is ablated in Table 7.

## 4 Experiments

In this section, we investigate the properties of SimZSS through various experiments. Details of the experimental setup are provided in Appendix A, and additional experiments can be found in Appendix B.

### 4.1 Zero-shot segmentation of foreground

**Table 1: Zero-shot foreground segmentation. Pixel-wise predictions are obtained by projecting patch representations onto pre-computed text embeddings of the class names, followed by up-sampling. The mIoU scores are reported across five standard segmentation datasets. $\dagger$ refers to our reproduction using DINOv2 pretrained vision backbones. The remaining results are as reported in Wysoczańska et al. ().**
| Method | Params | Params |  | Pascal VOC |  | Pascal Context |  | COCO-Stuff |  | Cityscapes |  | ADE20K |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ReCo (Shin et al., ) | 313M | 0 |  | 57.7 |  | 22.3 |  | 14.8 |  | 21.1 |  | 11.2 |
| GroupViT (Xu et al., ) | 0 | 55M |  | 79.7 |  | 23.4 |  | 15.3 |  | 11.1 |  | 9.2 |
| TCL (Cha et al., ) | 156M | 21M |  | 77.5 |  | 30.3 |  | 19.6 |  | 23.1 |  | 14.9 |
| MaskCLIP (Dong et al., ) | 291M | 0 |  | 74.9 |  | 26.4 |  | 16.4 |  | 12.6 |  | 9.8 |
| OVDiff (Karazija et al., ) | 1,226M | 0 |  | 81.7 |  | 33.7 |  | - |  | - |  | 14.9 |
| LiT^† (Zhai et al., ) (ViT-B + LAION-400M) | 94M | 63M |  | 80.5 |  | 31.8 |  | 23.3 |  | 24.7 |  | 18.7 |
| LiT^† (Zhai et al., ) (ViT-B + COCO Captions) | 94M | 63M |  | 86.1 |  | 35.5 |  | 25.6 |  | 25.8 |  | 18.1 |
| CLIP-DNOiser (Wysoczańska et al., ) | - | - |  | 80.9 |  | 35.9 |  | 24.6 |  | 31.7 |  | 20.0 |
| Ours |  |  |  |  |  |  |  |  |  |  |  |  |
| SimZSS (ViT-B + LAION-400M) | 94M | 63M |  | 85.1 |  | 34.2 |  | 24.9 |  | 27.8 |  | 19.6 |
| SimZSS (ViT-S + COCO Captions) | 21M | 40M |  | 87.2 |  | 37.3 |  | 23.8 |  | 29.2 |  | 17.9 |
| SimZSS (ViT-B + COCO Captions) | 94M | 63M |  | 90.3 |  | 43.1 |  | 29.0 |  | 33.0 |  | 21.8 |

We validate the proposed method on a pixel-level zero-shot segmentation task. In this scenario, the model relies solely on textual class descriptions to classify image pixels. However, accurately representing the background class can be challenging due to dataset-specific properties not fully captured by text descriptions. As such, we follow previous works (Wysoczańska et al., ; Cha et al., ) and first evaluate without the background class.

Figure: Figure 3: Vision-language alignment of text concepts and dense visual representations. Concepts present in the image are embedded independently by the text encoder and then projected onto the representations of each patch within the image. The images are processed at a resolution of $896\times 896$ pixels, corresponding to $4\times$ the training resolution. The alignment is performed on LAION-400M using a ViT-B/14 as the vision tower.
Refer to caption: x3.png

We follow the MMSegmentation (Contributors, ) implementation of Cha et al. (). Specifically, images are resized to have a shorter side of 448 pixels, and inference is performed with a sliding window using a stride of 224 pixels. ImageNet templates from Radford et al. () are used to contextualize each class name before obtaining their textual representation. Finally, the predictions obtained by projecting patch representations onto class names, are up-sampled using bilinear interpolation to restore the predictions to the original image size. We report the mIoU scores across five standard datasets, namely Pascal VOC (Everingham et al., ), Pascal Context (Mottaghi et al., ), COCO-Stuff (Caesar et al., ), Cityscapes (Cordts et al., ) and ADE20K (Zhou et al., ). For datasets that encompass the background class, the corresponding annotations and pixels are simply ignored. For the sake of clarity, we report the results without any post-processing techniques such as Pixel-Adaptive
Mask Refinement (PAMR) (Araslanov & Roth, ).

In Table 1, we observe that our method using a ViT-S/14 as the vision tower is already competitive and that shifting to a larger vision backbone yields state-of-the-art results on all datasets when trained on COCO Captions. In particular, it can be seen that the proposed pipeline significantly outperforms models trained under the same conditions with LiT (Zhai et al., ). This suggests that the reported performance is not solely attributable to the freezing of the vision tower or DINOv2 (Oquab et al., ) visual pretraining. The results obtained with LAION-400M indicate that SimZSS is compatible with large-scale uncurated datasets, exhibiting overall strong performance and improvements compared to models trained with LiT in the same setting. However, it is also apparent that both LiT and SimZSS seem to benefit more from curation than scale in the segmentation tasks. This trend contrasts with findings in classification tasks, as shown in Table 3.

### 4.2 Zero-shot segmentation

**Table 2: Zero-shot segmentation. Pixel-wise predictions of the foreground classes are obtained by projecting patch representations onto pre-computed text embeddings of the class names, followed by up-sampling. Predictions that fall below a predetermined threshold are assigned to the background class. The mIoU scores are reported across three standard segmentation datasets. $\dagger$ refers to our reproduction using DINOv2 pretrained vision backbones. The remaining results are as reported in Wysoczańska et al. ().**
| Method | Params | Params |  | Pascal Context |  | COCO-Object |  | Pascal VOC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ReCo (Shin et al., ) | 313M | 0 |  | 19.9 |  | 15.7 |  | 25.1 |
| OVDiff (Karazija et al., ) | 1,226M | 0 |  | 30.1 |  | 34.8 |  | 67.1 |
| GroupViT (Xu et al., ) | 0 | 55M |  | 18.7 |  | 27.5 |  | 50.4 |
| ZeroSeg (Chen et al., ) | - | - |  | 21.8 |  | 22.1 |  | 42.9 |
| SegCLIP (Luo et al., ) | - | - |  | 24.7 |  | 26.5 |  | 52.6 |
| TCL (Cha et al., ) | 156M | 21M |  | 24.3 |  | 30.4 |  | 51.2 |
| CLIPpy (Ranasinghe et al., ) | - | - |  | - |  | 32.0 |  | 52.2 |
| OVSegmentor (Xu et al., ) | - | - |  | 20.4 |  | 25.1 |  | 53.8 |
| CLIP-DIY (Wysoczańska et al., ) | - | - |  | 19.7 |  | 31.0 |  | 59.9 |
| MaskCLIP (Dong et al., ) | 291M | 0 |  | 23.6 |  | 20.6 |  | 38.8 |
| CLIP-DNOiser (Wysoczańska et al., ) | - | - |  | 32.4 |  | 34.8 |  | 62.1 |
| LiT^† (Zhai et al., ) (ViT-B + LAION-400M) | 94M | 63M |  | 29.6 |  | 38.3 |  | 48.1 |
| LiT^† (Zhai et al., ) (ViT-B + LAION-400M) | 94M | 63M |  | 31.5 |  | 39.5 |  | 51.4 |
| Ours |  |  |  |  |  |  |  |  |
| SimZSS (ViT-B + LAION-400M) | 94M | 63M |  | 31.1 |  | 38.1 |  | 48.6 |
| SimZSS (ViT-S + COCO Captions) | 23M | 40M |  | 32.8 |  | 39.5 |  | 55.5 |
| SimZSS (ViT-B + COCO Captions ) | 94M | 63M |  | 37.2 |  | 43.5 |  | 58.4 |

In a second scenario, we explore a zero-shot segmentation task including background class. Similar to Cha et al. (), we do not rely on the textual representation to predict the background class, but rather on the confidence levels of the predictions on the remaining classes. More precisely, we assign a given pixel to the background class if the highest confidence score among the other class predictions falls below a dataset and model-specific threshold. The remaining implementation details are identical to the above-described setting. The mIoU scores are reported on three datasets, namely Pascal Context (Mottaghi et al., ), COCO-Object (Caesar et al., ) and Pascal VOC (Everingham et al., ).

In Table 2, we observe similar trends as in the scenario without the background class (see Tab. 1). Reported results are not as unequivocal as in the former evaluation. This is in part due to the crudeness of the background detection mechanism, which contrasts with the more sophisticated approaches used by some of the competing baselines. For instance, CLIP-DNOiser (Wysoczańska et al., ) relies on FOUND (Siméoni et al., ), whereas OVDiff (Karazija et al., ) uses different background prototypes for each class. Once again, training on COCO Captions leads to improved performance. When training on LAION-400M, a concept is identified in less than 15% of the samples on average. As a result, concepts rarely co-occur, making it sufficient to detect the concept without localizing it in the image to minimize loss. Therefore, the model is not trained to be less confident in the background. Exploring the possibility of enlarging the list of concepts could provide further insights.

### 4.3 Zero-shot classification

**Table 3: Zero-shot classification on ImageNet-1k. Image-level predictions are obtained by projecting the image [CLS] token onto pre-computed text embeddings of class names. Accuracy is reported for various visual pretraining and vision-language alignment methods. $\dagger$ refers to our reproduction using DINOv2 pretrained vision backbones. The remaining results are as reported in Zhai et al. ().**
| Method | Visual pretraining | Backbone | Pretraining dataset | Alignment dataset | Labels |  | Top-1 accuracy |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LiT (Zhai et al., ) | MoCo-v3 (Chen et al., ) | ViT-B/16 | ImageNet-1k | CC12M+YFCC100M | ✗ |  | 55.4 |
| LiT (Zhai et al., ) | DINOv1 (Caron et al., ) | ViT-B/16 | ImageNet-1k | CC12M+YFCC100M | ✗ |  | 55.5 |
| LiT (Zhai et al., ) | AugReg (Steiner et al., ) | ViT-B/16 | ImageNet-21k | CC12M+YFCC100M | ✓ |  | 55.9 |
| LiT^† (Zhai et al., ) | DINOv2 (Oquab et al., ) | ViT-B/14 | LVD-142M | COCO Captions | ✗ |  | 22.6 |
| LiT^† (Zhai et al., ) | DINOv2 (Oquab et al., ) | ViT-B/14 | LVD-142M | LAION-400M | ✗ |  | 63.6 |
| Ours |  |  |  |  |  |  |  |
| SimZSS | DINOv2 (Oquab et al., ) | ViT-B/14 | LVD-142M | COCO Captions | ✗ |  | 24.3 |
| SimZSS | DINOv2 (Oquab et al., ) | ViT-B/14 | LVD-142M | LAION-400M | ✗ |  | 64.1 |

Figure: Figure 4: Zero-shot segmentation performance as a function of the number of processed image-caption pairs in LAION-400M. The left plot shows the mIoU percentages for different datasets, while the right plot shows the relative performance percentages. Each data point represents the result of running a vision-language alignment from scratch using SimZSS; these are not training curves.
Refer to caption: x4.png

The proposed method derives from LiT (Zhai et al., ), noted for its excellent zero-shot classification performance on ImageNet-1k (Deng et al., ) among other benchmarks. Although our approach is tailored for dense downstream tasks, we investigate whether SimZSS exhibits zero-shot classification capabilities or if the introduced local consistency objective breaks this property. We follow the evaluation protocol from OpenCLIP (Ilharco et al., ). Class names from ImageNet-1k are contextualized with the corresponding templates from CLIP (Radford et al., ) and embedded through the text encoder. On the vision side, images are passed through the vision encoder, and the global representation is used to obtain class predictions via projection onto the class embeddings. The accuracy of the prediction is reported.

The results reported in Table 3 reveal that SimZSS not only preserves classification capabilities comparable to LiT but also exhibits slight improvements in the low-data regime. It is worth mentioning that it takes CLIP $32$ epochs on LAION-400M to train a ViT-B/16 to reach a zero-shot accuracy of $67.0$ (see Ilharco et al. (), Tab. 18). Overall, while pretraining on COCO Captions excels for segmentation tasks, zero-shot classification benefits from larger datasets. This suggests that a pretraining on LAION-400M followed by finetuning on COCO Captions could optimize the benefits of both datasets.

### 4.4 Scalability of SimZSS

An important property of vision-language pretraining methods is their capacity to leverage large datasets effectively, which necessitates them to be i) compatible with noisy image-caption pairs, and ii) computationally efficient.

We first verify in Tables 9 and 8 that SimZSS-specific operations incur only moderate memory overhead and negligible time overhead compared to LiT. Considering this, along with the data efficiency of the proposed approach, SimZSS is orders of magnitude cheaper to train than CLIP. Secondly, we escalate the number of image-caption pairs from LAION-400M during training and report the performance on zero-shot segmentation at the end of each training phase. The results depicted in Figure 4 underscore the scalability of our method SimZSS and suggest potential gains from even larger datasets, such as LAION-5B (Schuhmann et al., ), or increased training epochs.

### 4.5 Bank of concepts

**Table 4: Ablation study on the impact of concept sets. Training is conducted on COCO Captions with and without class names from Pascal VOC included in the concept set, using a ViT-B/14 vision backbone. Performance on the zero-shot segmentation task is then compared based on achieved mIoU scores. $\dagger$ refers to our reproduction using DINOv2 pretrained vision backbones.**
|  |  |  | Without background |  | With background |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Pascal VOC concepts |  | Pascal VOC | Pascal Context | COCO-Stuff | Cityscapes | ADE20K |  | Pascal Context | COCO-Object | Pascal VOC |
| LiT^† (Zhai et al., ) | ✗ |  | 86.1 | 35.5 | 25.6 | 25.8 | 18.1 |  | 31.5 | 39.5 | 51.4 |
| SimZSS | ✗ |  | 88.1 | 42.9 | 28.7 | 32.9 | 22.6 |  | 36.6 | 42.4 | 53.4 |
| SimZSS | ✓ |  | 90.3 | 43.1 | 29.0 | 33.0 | 21.8 |  | 37.2 | 43.5 | 58.4 |

At its core, SimZSS relies on identifying noun phrases in captions and filtering them through a bank of concepts. This filtering ensures that the identified noun phrases are valid candidates for visual concepts and can be used to query the images. A potential question that arises is whether the segmentation improvements of the proposed method are limited to the classes included in the bank of concepts or if they generalize to any class. To investigate this, we train SimZSS with the class names from Pascal VOC (Everingham et al., ) removed from the bank of concepts and report the resulting zero-shot segmentation mIoU scores across all downstream datasets.

Most importantly,  Table 4 shows that SimZSS outperforms LiT on Pascal VOC both with and without background. This invalidates the hypothesis that only classes contained in the bank of concepts benefit from the proposed local consistency objective. Furthermore, the performance on other downstream datasets only slightly deviates from the one obtained using the complete list of concepts, which further supports our claim, considering that the labels from Pascal VOC account for approximately $22\%$, $26\%$, and $33\%$ of the labels from COCO Object (Caesar et al., ), Cityscapes (Cordts et al., ), and Pascal Context (Mottaghi et al., ), respectively. Overall, these results suggest that our method effectively trains the text encoder to align with local features.

Alternatively,  Figure 1 suggests that the vision tower, here a ViT-B/14 pretrained with DINOv2 (Oquab et al., ), could handle more concepts than those present in the concept bank. Indeed, some visual concepts, such as bike (third row, columns 1-5), appear well-segmented in the image but are not highlighted in the caption, meaning the concept is absent from the concept bank, and thus, no constraint is enforced. This claim is further supported by the fine-grained vision-language alignment depicted in Figure 3. The visualization further shows that, even though concepts such as forest, chef, water buffalo, or bike are not in the concept bank, the text concepts and the corresponding patches appear well aligned.

## 5 Conclusion

We introduce SimZSS, a simple framework to endow pretrained pure vision models with open-vocabulary segmentation capabilities. This approach is versatile: it can accommodate various backbones, pretraining methods, and datasets, irrespective of their scale and degree of curation. Overall, SimZSS is both a computationally and data-efficient method that yields state-of-the-art results across standard zero-shot segmentation benchmarks.

##### Aknowledgement.

This research is supported by the Personalized Health and Related Technologies (PHRT), grant number 2021/344. Additional funding is provided by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (Grant Agreement No. 101021347). This work is supported by a grant from the Swiss National Supercomputing Centre (CSCS) on the Swiss share of the LUMI system under project ID 606.

## Appendix

## Appendix A Experimental setup

### A.1 Pretraining

#### A.1.1 Pretraining datasets

We train our models on two distinct datasets: COCO Captions (Lin et al., ; Chen et al., ) and LAION-400M (Schuhmann et al., ). The former comprises 600K image-caption pairs with high-quality human-generated captions, while the latter is derived from LAION-5B (Schuhmann et al., ) consisting of image-caption pairs with vision-language cosine similarity exceeding 0.3, as determined by a pretrained CLIP model (Radford et al., ). These datasets represent opposing ends of the spectrum in terms of curation and scale.

#### A.1.2 Networks architectures

Vision transformers (Dosovitskiy et al., ) are used to obtain image representations. More precisely, we experiment with ViT-S/14 and ViT-B/14 pretrained with DINOv2 (Oquab et al., ) on LVD-142M. A ViT-B/16 pretrained with AugReg (Steiner et al., ) on ImageNet-21k is also tested. The architecture of the text transformers is identical to the one used in CLIP, and its weights are randomly initialized. Overall, the only architectural difference w.r.t. CLIP is the removal of the learnable linear layer that maps visual representations to the cross-modal representation space i.e., we project textual representations directly onto the visual space rather than projecting both the textual and visual representations onto an intermediary space.

#### A.1.3 Optimization

For COCO Captions, we conduct training over 10 epochs using a global batchsize of 16,384. We incorporate a warm-up strategy spanning 10% of the training steps, linearly ramping up the learning rate until it reaches its peak value, chosen from the set {8e-5, 3e-5, 8e-6, 3e-6}(^1^11scaled by batchsize / 256). Subsequently, we employ a cosine decay schedule for the remaining steps. Similarly, for LAION-400M, we train for 1 epoch with a global batchsize of 32,768, and we set the learning rate from the options {3e-5, 8e-6, 3e-6, 8e-7, 3e-7}. The remaining optimization settings align with those of OpenCLIP (Ilharco et al., ).

#### A.1.4 Text concepts

We use the en_core_web_trf model from SpaCy (Honnibal et al., ) as part-of-speech tagger to identify noun phrases. The concept bank is obtained as the union of the class names from Pascal VOC (Everingham et al., ), Pascal Context (Mottaghi et al., ), COCO-Stuff (Caesar et al., ), Cityscapes (Cordts et al., ) and ADE20K (Zhou et al., ). This results in $574$ concepts.

## Appendix B Additional experiments

### B.1 Impact of the vision tower

**Table 5: Ablation study on the vision backbone for zero-shot foreground segmentation. We investigate the effect of the vision backbone’s size and pretraining method on the downstream task. The impact of using artifact-free models is also evaluated (with/without DVT (Yang et al., )). We report the mIoU scores across five standard datasets. $\dagger$ refers to our reproduction using DINOv2 pretrained vision backbones.**
| Method | Pretraining | Backbone | DVT |  | Pascal VOC |  | Pascal Context |  | COCO-Stuff |  | Cityscapes |  | ADE20K |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Supervised |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LiT^† (Zhai et al., ) | AugReg | ViT-B/16 | ✗ |  | 74.1 |  | 18.3 |  | 13.3 |  | 15.6 |  | 10.2 |
| SimZSS | AugReg | ViT-B/16 | ✗ |  | 81.2 |  | 23.3 |  | 16.1 |  | 18.9 |  | 12.6 |
| Self-supervised |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LiT^† (Zhai et al., ) | DINOv2 | ViT-S/14 | ✗ |  | 84.0 |  | 32.8 |  | 22.0 |  | 24.0 |  | 16.3 |
| SimZSS | DINOv2 | ViT-S/14 | ✗ |  | 87.2 |  | 37.3 |  | 23.8 |  | 29.2 |  | 17.9 |
| LiT^† (Zhai et al., ) | DINOv2 | ViT-B/14 | ✗ |  | 86.3 |  | 34.1 |  | 24.6 |  | 25.8 |  | 17.1 |
| SimZSS | DINOv2 | ViT-B/14 | ✗ |  | 87.4 |  | 41.1 |  | 26.6 |  | 31.6 |  | 19.9 |
| LiT^† (Zhai et al., ) | DINOv2 | ViT-B/14 | ✓ |  | 86.1 |  | 35.5 |  | 25.6 |  | 25.8 |  | 18.1 |
| SimZSS | DINOv2 | ViT-B/14 | ✓ |  | 90.3 |  | 43.1 |  | 29.0 |  | 33.0 |  | 21.8 |

The underlying assumption of SimZSS is that the visual representations do not require further training, and only the vision-language alignment must be learned. As such, the quality of the vision backbone is pivotal to the overall performance of SimZSS.

**Table 6: Ablation study on the vision backbone for zero-shot segmentation. We investigate the effect of the vision backbone’s size, pretraining method, and patch size on the downstream task. The impact of using artifact-free models is also evaluated (with/without DVT (Yang et al., )). We report the mIoU scores on three standard datasets.**
| Method | Pretraining | Backbone | DVT |  | Pascal Context |  | COCO-Object |  | Pascal VOC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Supervised |  |  |  |  |  |  |  |  |  |
| LiT (Zhai et al., ) | AugReg | ViT-B/16 | ✗ |  | 17.8 |  | 26.1 |  | 32.2 |
| SimZSS | AugReg | ViT-B/16 | ✗ |  | 21.6 |  | 29.7 |  | 35.9 |
| Self-supervised |  |  |  |  |  |  |  |  |  |
| LiT (Zhai et al., ) | DINOv2 | ViT-S/14 | ✗ |  | 29.9 |  | 36.5 |  | 49.6 |
| SimZSS | DINOv2 | ViT-S/14 | ✗ |  | 32.8 |  | 39.5 |  | 55.5 |
| LiT (Zhai et al., ) | DINOv2 | ViT-B/14 | ✗ |  | 30.5 |  | 28.7 |  | 49.8 |
| SimZSS | DINOv2 | ViT-B/14 | ✗ |  | 35.7 |  | 40.5 |  | 57.5 |
| LiT (Zhai et al., ) | DINOv2 | ViT-B/14 | ✓ |  | 31.5 |  | 39.5 |  | 51.4 |
| SimZSS | DINOv2 | ViT-B/14 | ✓ |  | 37.2 |  | 43.5 |  | 58.4 |

In Tables 5 and 6, we explore the performance of various pretrained vision towers in zero-shot segmentation tasks. Our observations indicate that SimZSS is versatile and can integrate diverse models and pretraining approaches effectively. As expected, models with superior performance on pure vision tasks also show enhanced results when used with our method, as well as with LiT. Most importantly, regardless of the vision backbone employed, SimZSS consistently surpasses LiT. Additionally, employing denoised ViTs (Yang et al., ), which exhibit better semantic feature correlation, leads to improved segmentation. This outcome is anticipated since our approach assumes that features corresponding to semantically related regions are highly correlated (see Sec. 3.2).

### B.2 SimZSS -specific hyperparameters

**Table 7: Ablation study on the impact of the loss weight $\mathbf{\lambda}$ and the temperature parameter $\mathbf{\tau}$. Training is performed on COCO Captions using a ViT-B/14 vision backbone. The performance is evaluated through a comparison of mIoU scores on the zero-shot segmentation task.**
|  |  |  | Without background |  | With background |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Loss weight | Temperature |  | Pascal VOC | Pascal Context | COCO-Stuff | Cityscapes | ADE20K |  | Pascal Context | COCO-Object | Pascal VOC |
| $\lambda=0.00$ | - |  | 86.1 | 35.5 | 25.6 | 25.8 | 18.1 |  | 31.5 | 39.5 | 51.4 |
| $\lambda=0.01$ | $\tau=0.1$ |  | 89.8 | 40.2 | 27.6 | 29.7 | 19.6 |  | 34.9 | 42.1 | 54.9 |
| $\lambda=0.02$ | $\tau=0.1$ |  | 89.8 | 41.9 | 28.2 | 30.9 | 20.4 |  | 36.1 | 42.3 | 57.1 |
| $\lambda=0.05$ | $\tau=0.1$ |  | 90.3 | 43.1 | 29.0 | 33.0 | 21.8 |  | 37.2 | 43.5 | 58.4 |
| $\lambda=0.10$ | $\tau=0.1$ |  | 90.1 | 41.8 | 28.6 | 33.6 | 22.1 |  | 36.1 | 43.1 | 58.5 |
| $\lambda=0.05$ | $\tau=0.01$ |  | 89.2 | 42.3 | 28.8 | 33.1 | 21.6 |  | 36.6 | 43.2 | 57.0 |
| $\lambda=0.05$ | $\tau=0.04$ |  | 89.7 | 42.7 | 28.7 | 34.3 | 22.1 |  | 36.7 | 42.9 | 57.0 |
| $\lambda=0.05$ | $\tau=0.07$ |  | 90.0 | 42.9 | 28.9 | 33.4 | 21.3 |  | 37.0 | 43.1 | 58.8 |
| $\lambda=0.05$ | $\tau=0.10$ |  | 90.3 | 43.1 | 29.0 | 33.0 | 21.8 |  | 37.2 | 43.5 | 58.4 |
| $\lambda=0.05$ | $\tau=0.40$ |  | 88.7 | 40.0 | 27.6 | 28.2 | 19.5 |  | 34.4 | 42.1 | 52.6 |

Thanks to its simplicity, SimZSS has few hyperparameters, with the primary ones being $\tau$, which regulates the temperature used to query the dense visual representation, and $\lambda$, which modulates the contribution of the dense loss to the overall objective (see Eqs. 10 and 4). After a grid search on $\lambda$, $\tau$, and the learning rate, we find that the best-performing setting for training on COCO Captions is $(\lambda=0.05,\tau=0.1)$. In Table 7, we report the resulting zero-shot segmentation performance when varying $\lambda$ and $\tau$ around this optimal point.

It can be observed that the setting with $\lambda=0$, corresponding to LiT (Zhai et al., ), performs the worst. Regarding $\tau$, SimZSS works best with low temperatures. This is not surprising, as in this setting, fewer patches contribute to the representation of the visual concepts, aligning more closely with the downstream evaluations.

### B.3 High-level profiling & runtime comparison

**Table 8: High-level Profiling of SimZSS. We report the absolute and relative times of the various operations performed in SimZSS. Experiments are conducted on a single node with 4x AMD MI250x GPUs (2 compute dies per GPU, i.e., $\texttt{worldsize}=8$) with a memory usage of 38GB per compute die. The backbone used is ViT-B/14, and the batch size is set to 1024 per compute die, totaling 8192. SimZSS-specific operations are highlighted.**
| Description | Operation | Absolute time per iteration [ms] |  | Relative time [%] |
| --- | --- | --- | --- | --- |
| Forward pass vision | $f_{v}(\cdot)$ | 1435.0 |  | 69.3 |
| Forward pass text | $f_{t}(\cdot)$ | 194.0 |  | 9.3 |
| Vision concept extraction | Equations 4 and 3 | 12.0 |  | 0.6 |
| Features gathering | - | 22.5 |  | 1.1 |
| Global consistency | Equation 6 | 1.0 |  | 0.05 |
| Concept-level consistency | Equation 9 | 0.9 |  | 0.04 |
| Weights update | backpropagation | 404.4 |  | 19.5 |
| Total |  | 2069.8 |  | 100 |

**Table 9: Computational and memory efficiency. The efficiency of SimZSS is compared to that of related methods, i.e., LiT and CLIP. When feasible, we report results using the local training batch size; otherwise, the largest power of 2 that fits into memory is utilized. The reported values are obtained on a single node equipped with 4x AMD MI250x (2 compute die per GPU, i.e., $\texttt{worldsize}=8$).**
| Method | Batchsize per compute die | Memory per compute die [GB] | Time per step [ms] | Throughput [img/s] |
| --- | --- | --- | --- | --- |
| CLIP | 256 | $\sim 40$ | 1196.0 | 1712 |
| LiT | 1024 | $\sim 27$ | 2049.2 | 3997 |
| SimZSS | 1024 | $\sim 38$ | 2069.8 | 3957 |

In Table 8, we present a profiling of the operations performed during a training step of SimZSS. Overall, SimZSS-specific operations account for less than $1\%$ of the total runtime.  Table 9 further confirms that SimZSS runtime is comparable to that of LiT (Zhai et al., ) and adds modest memory overhead.

## Appendix C Evaluation datasets

##### Pascal VOC 2012

The Pascal VOC dataset (Everingham et al., ) contains 20 classes with semantic segmentation annotations. The training set consists of 1,464 images, while the validation set includes 1,449 images. An additional background class is provided.

##### Pascal Context

The Pascal Context dataset (Mottaghi et al., ) extends the Pascal VOC dataset by providing detailed annotations for entire scenes. It includes 60 classes (including a background class) of semantic segmentation annotations. The training set contains approximately 4,998 images, while the validation set includes around 5,105 images.

##### COCO-Stuff

COCO-Stuff (Caesar et al., ) is an extension of COCO (Lin et al., ), providing pixel-wise annotations for 80 things classes and 91 stuff categories. The annotations are exhaustive, i.e., no pixel remains unlabeled. It includes over 164K images for training and 20K images for validation, covering a wide range of scenes and objects.

##### COCO-Object

COCO-Object uses the same set of images as the above-described COCO-Stuff, but only contains labels for the “things” categories. An additional background label covers the remaining pixels.

##### ADE20K

The ADE20K dataset (Zhou et al., ) comprises a diverse set of images with annotations for 150 semantic categories. The training set includes 20,210 images, and the validation set consists of 2,000 images.

##### Cityscapes

The Cityscapes dataset (Cordts et al., ) contains high-resolution images of urban street scenes, annotated for 30 classes. The training set includes 2,975 images, and the validation set includes 500 images captured from 50 cities for semantic segmentation and urban scene understanding.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: What is the motivation of the proposed method?
What is the reason of the good performance? I think it is because the large training dataset.
soundness: 2
strengths: The experimental performance is good.
summary: This work proposes a simple framework for open-vocabulary Zero-Shot Segmentation with two key principles of leveraging frozen
vision-only models and exploiting the discrete nature of text and linguistic knowledge.
weaknesses: 1. The novelty and contribution is limited. First, I prefer simple method. I believe that the simple method is more valuable for applications and research. However, simple method requires more deep analysis and insights. Unfortunately, this work only presents a simple method without any insights. The method is designed without motivation or explanation. This paper looks like a experiment report, rather than a research paper. For a good research paper, the authors need to tell new insights, rather than just propose a model and conduct some experiments.
2. The comparison is unfair. The proposed method is trained on COCO Captions (Lin et al., 2014; Chen et al., 2015) and LAION-400M (Schuhmann et al., 2021) and the image encoder is pretrained on LVD-142M dataset, as shown in L764-L774. However, other methods are not trained on these large-scale dataset. Even trained on these large datasets, the performance improvement is limited and the proposed method even performs worse than previous methods on Pascal VOC dataset. For example, the proposed method SimZSS trained on LAION-400M achieves 48.6 on Pascal VOC, which is much lower than CLIP-DINOiser (Wysoczanska et al., 2023), CLIP-DIY (Wysoczanska et al., 2024), OVSegmentor (Xu et al., 2023a), OVDiff (Karazija et al., 2023), CLIPpy (Ranasinghe et al., 2023) and TCL (Cha et al., 2023).

Overall, this work propose a simple method without motivation and experimental analysis for the working mechanism. The good performance is mainly derived from its large training dataset.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Since the visual encoder is frozen, the quality of the segmentation masks heavily depends on the encoder's ability to recognize spatial positions. Would the performance improve if the visual encoder were fully or partially fine-tuned?
2. In an extreme scenario, how significantly would the test performance be affected if the class names in the training set and test set are entirely different?
soundness: 3
strengths: 1. The paper is easy to understand and follow.
2. The method is straightforward, easy to implement, and can be readily adapted to various backbones and training datasets.
3. The method can be trained without the supervision of semantic masks, reducing the burden of annotations.
4. The motivation for proposing a concept-level objective is clear and more suitable compared to a contrastive objective in scenarios where concepts encode individual objects that are likely to occur multiple times within a batch.
summary: The paper introduces SimZSS, a straightforward yet effective framework for zero-shot semantic segmentation. This framework aims to align vision encoders, which exhibit spatial awareness, with textual descriptions. SimZSS leverages text concept representations to extract corresponding visual representations based on similarity metrics. It ensures cross-modality consistency by employing different objectives at the sample and concept levels. Notably, the method requires minimal hyperparameters and does not rely on the supervision of semantic masks. Additionally, SimZSS can be easily adapted to various backbones and datasets. Overall, the method achieves state-of-the-art results across standard zero-shot segmentation benchmarks.
weaknesses: 1. It is unclear how the final segmentation masks are generated during inference. Is there a similarity threshold used to determine the class names to which visual tokens belong? If so, how does the performance vary with different threshold settings?
2. There is a lack of analysis explaining why SimZSS outperforms other zero-shot semantic segmentation methods.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: - Given that the predefined concept bank seems to impact performance, how does the model perform on datasets with a much broader set of concepts, such as OpenImagesV7, which covers over 5000 classes? Has the model been evaluated on classes outside of the concept bank, and if so, what are the results?

- Have you evaluated the recognition capabilities of the trained model on other classification datasets beyond ImageNet? Including evaluations on a wider range of datasets could provide deeper insights into how the method affects recognition performance.

-  How does your model compare with training-free approaches?

- Could you discuss the potential trade-off between improved localization and any degradation in recognition performance? Understanding this trade-off would help assess the overall impact of your method on different tasks.
soundness: 4
strengths: - The paper is well-written and easy to follow. The authors provide clear explanations of their methodology. I would like to highlight the quality of the figures and tables in a visually pleasing way that enhances the understanding of the content.

- While the problem of localization in vision-language models is not novel, the proposed approach offers a novel perspective. By freezing the vision backbone and only training the text encoder, the authors leverage pretrained self-supervised models effectively, resulting in efficient training.

- The method demonstrates state-of-the-art performance on multiple zero-shot segmentation benchmarks.
summary: The paper introduces SimZSS, a simple framework that enables open-vocabulary zero-shot segmentation by leveraging pretrained vision-only models within a vision-language contrastive learning paradigm. Addressing the challenge of poor localization in dense tasks, SimZSS decouples visual representation learning from cross-modality alignment by utilizing frozen, spatially aware vision models like DINOv2 and focusing on training only the text encoder. The framework identifies local concepts within captions using linguistic cues and aligns them with corresponding visual concepts in images through similarity-based pooling.
weaknesses: 1. It is surprising that the main paper lacks essential details about the training data and the pretrained models used. Given that the paper is only 9 pages (below the 10-page limit), including this information in the main text is necessary.

2. 
(a) One major limitation is the use of a predefined concept bamk. The authors claim that it does not impact the breadth of the concept the model can localize(sec. 4.4). However, in Table 4, removing PascalVOC classes from the concept bank decreases the performance. Therefore, it seems that the breath of the concept bank is a crucial component. Also, the authors could evaluate on datasets with much diverse set of concepts e.g. OpenImagesV7 (which covers more than 5000 classes), this way the breadth of the vocabulary learned by the model can effectively be assessed. 

(b) Moreover, to identify the recognition capabilities of the trained model, I would expect the authors to evaluate the performance of more classification datasets than just ImageNet. More specifically, it is common practice I think to evaluate on the set of 38 classification datasets (https://github.com/LAION-AI/CLIP_benchmark). Also, table 3 should include the performance of at least the OpenCLIP-ViT/16 trained on LAION-400M as the training dataset is the same, whose zero-shot accuracy on ImageNet is 67.05 vs 64.1 for your ViT-B/14 which indicates that your method hurts the overall recognition capability of the final model. This tradeoff between localization and recognition should be discussed in the paper.

(c)  Since the concept bank is derived from the union of class names from the segmentation datasets used for evaluation, the framework may be biased towards these classes. This design choice limits the model's ability to generalize to a truly open vocabulary, as it may not effectively localize concepts outside the predefined bank.
 
3. The paper misses important related work, particularly training-free approaches that tackle poor localization in VLMs from a different angle. For instance, methods like [a][b][c][d] achieve competitive performance without requiring additional training and can detect concepts not covered by a predefined concept bank. Including comparisons with such methods would provide a more comprehensive evaluation.

References:
[a] Lan, M., Chen, C., Ke, Y., Wang, X., Feng, L. and Zhang, W., 2024. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. arXiv preprint arXiv:2408.04883.

[b] Li, Y., Wang, H., Duan, Y. and Li, X., 2023. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653.

[c] Bousselham, W., Petersen, F., Ferrari, V. and Kuehne, H., 2024. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3828-3837).

[d] Zhou, C., Loy, C.C. and Dai, B., 2022, October. Extract free dense labels from clip. In European Conference on Computer Vision (pp. 696-712). Cham: Springer Nature Switzerland.

[e] Benenson, R. and Ferrari, V., 2022. From colouring-in to pointillism: revisiting semantic segmentation supervision. arXiv preprint arXiv:2210.14142.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - What are the current limitations of the concept bank used? Would a larger bank lead to better generalization? What are the complexities in scaling the concept bank to, e.g., WordNet or other taxonomies?
- As mining concepts from the captions provide promising results, I wonder how well the approach would work on more noisy captions, e.g., from web-scale datasets. How sensitive is SimZSS to such noise? Are there any techniques that could be employed to improve the robustness of the generation of the concept bank?
soundness: 3
strengths: - The authors propose a simple yet effective approach for the task that exploits pre-trained vision models.
- The method is both data- and compute-efficient, as it does not require long training periods and needs captioned images for training.
- The method achieves state-of-the-art results on seven benchmark datasets.
- The method takes inspiration from LiT and extends it for the segmentation task while maintaining its classification capabilities.
summary: The work proposed SimZSS, a simple and efficient framework for open-vocabulary zero-shot semantic segmentation. The work builds on top of a frozen pre-trained vision-only model and aligns a text encoder to achieve cross-modality concept-level alignment. Despite its simplicity, SimZSS achieves state-of-the-art results on seven out of eight benchmarks.
weaknesses: - The work relies on a bank of pre-defined concepts to pre-process the captions and extract concepts to segment, which may represent a limitation in some situations. For example, the concept bank may not cover the diversity of potential objects in the image.
- While performance is competitive when using ViT-B, scaling the model backbone to a ViT-L model does not improve performance.

Meta-review: The paper presents a framework that aligns vision and text representations using a pretrained visual encoder, by incorporating the discrete nature of text modality and cross-modal consistency. It received three positive evaluations (two 6s and an 8) and one negative rating (5). The authors' rebuttal effectively maintained the positive scores and slightly improved the negative score to 5. Although there is still a minor concern from the negative reviewer regarding the motivation behind the study, this issue appears relatively minor. The authors have successfully addressed most other concerns. Consequently, the AC recommends acceptance of the paper.

Decision: Accept (Poster)