A Simple Framework for Open-Vocabulary Zero-Shot Segmentation
Authors: Thomas Stegmüller
Tim Lebailly
Nikola Đukić
Behzad Bozorgtabar
Tinne Tuytelaars
Jean-Philippe Thiran
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-language models
Zero-shot segmentation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We densely align vision and language for open-vocabulary zero-shot segmentation.

Abstract: Zero-shot classification capabilities naturally arise in models trained within a vision-language contrastive framework. Despite their classification prowess, these models struggle in dense tasks like zero-shot open-vocabulary segmentation. This deficiency is often attributed to the absence of localization cues in captions and the intertwined nature of the learning process, which encompasses both image/text representation learning and cross-modality alignment. To tackle these issues, we propose SimZSS, a $\textbf{Sim}$ple framework for open-vocabulary $\textbf{Z}$ero-$\textbf{S}$hot $\textbf{S}$egmentation. The method is founded on two key principles: i) leveraging frozen vision-only models that exhibit spatial awareness while exclusively aligning the text encoder and ii) exploiting the discrete nature of text and linguistic knowledge to pinpoint local concepts within captions. By capitalizing on the quality of the visual representations, our method requires only image-caption pair datasets and adapts to both small curated and large-scale noisy datasets. When trained on COCO Captions across 8 GPUs, SimZSS achieves state-of-the-art results on 7 out of 8 benchmark datasets in less than 15 minutes. Our code and pretrained models are publicly available at https://github.com/tileb1/simzss.

Published as a conference paper at ICLR 2025
A SIMPLE FRAMEWORK FOR OPEN-VOCABULAN ZERO-SHOT SEGMENTATION
Thomas Stegmüller' Behzad Bozorgtabar
Tim Lebailly
Nikola Dukić² Jean-Philippe Thiran CHUV
Tinne Tuytelaars
EPFL
KU Leuven
(firstnamm); [Lastname)@@epl..co
2 ?(Firstname). (lastname) @esat. kuleuven.
ABSTRACT
Zero-shot classification capabilities naturally arise in models trained within vision-language contrastive framework. Despite their classification prowess, these models struggle in dense tasks like zero-shot open-vocabulary segmentation. This deficiency is often attributed to the absence localization cues in captions and the intertwined nature of the learning process, which encompasses both im- age/text representation learning and cross-modality alignment. To tackle these issues, we propose SimZSS, a Simple framework for open-vocabulary Zero-Sho Segmentation. The method is founded on two key principles: i) leveraging frozen vision-only models that exhibit spatial awareness while exclusively aligning the text encoder and iř) exploiting the discrete nature of text and linguistic knowl- edge to pinpoint local concepts within captions. By capitalizing on the quality of the visual representations, and adapts to both small curated and large-scale noisy datasets. When trained on COCO Captions across GPUs, SimZSS achieves state-of-the benchmark datasets in less than 15 minutes. Our code and pretrained models are publicly available https: /tttt/giittttt /tttttt.t.uggggo com/tilebe / simzss.
method requires only image-caption pair datasets
results on 7 out
INTRODUCTION
Semantic segmentation stands as cornerstone task within the realm of computer vision, playing a pivotal role in various applications ranging from autonomous driving to medical image analysis.
giratee standing next to zebra nex: Foreet
PUppy PAR toy place of piaza in 1*x soath
Evo sabra stand 43 elephant, and hezc vatue budelo
dog laying on whits abeet
sidewalk
bes1ding
starding
Figure 1: Visualization of the patch-level representations and text concepts in the RGB color space. PCA is used to map the dense representation space. The three-dimensional representations (color) of the text concepts from the concept bank and the corresponding caption are obtained and shown for each image. Each row includes original image and dense feature visualization at different resolutions. These include the training resolution (16 x 16) and higher resolutions (2x, 4x, and 8x).
single image into dine--immensionl
denotes equal contribution.

---

Published as a conference paper at ICLR 2025
However, its widespread adoption and scalability are hindered by the inherent label-intensive ture of traditional methods, demanding copious amounts of fine- line-grained annotations for training. Furthermore, traditional approaches, e.g., [Strudel et al.] (202I): Cheng et al. (0022) are typically closed-vocabulary. meaning they only work for a pre-defined set of categories and generalize poorly to unseen classes. Self-supervised learning paradigms (Caron et al.; 202T Zhou et al.) 2022b He et al. 2020; Balazevic et al.) [024; Lebailly et al.) 2024), which use pretext tasks to learn discrimi- native representations from data itself, offer a promising solution to alleviate the annotation burden.
Representations obtained in this manner are typically clustered semantically, potentially even at a fine-grained level (Balazevic et al.) 2024, Lebailly et al.) 20224, [Stegmüller al.) (023) and, as such, yield excellent performance in various applications. Nonetheless, degree of labeling is still necessary, whether for finetuning or constructing the support set of a k-NN classifier, for instance.
Recently, vision-language contrastive learning has proven to be simple and effective approach for transforming web-scale amounts of noisy image-caption pairs into zero-shot classification ca- pabilities (Radford et al. 2021: Jia et al., 2021). Nonetheless, the substantial computational and data requirements of these methods pose significant challenges. Furthermore, the performance of these methods on finer-grained downstream tasks, such as zero-shot segmentation tends to be subpar (see Tables and We argue that this is not only the consequence of a lack of local lision-language alignment but also hint that the learned visual representations therein might be suboptimal. Based on this same observation, LiT (Zhai et al. 2022) proposed aligning a text encoder with a pretrained and frozen pure vision model. By decoupling the image representation learning process from the vision- laiiugl alignment process, this methodology enhances both computational and data effi- ciency while also improving performance in zero-shot classification tasks. The strong performance of LiT suggests that achieving zero-shot classification or segmentation relies on two key require- ments:
1. The vision encoder must be capable of clustering images or groups of pixels with shared semantics.
2. The text encoder must be sufficiently aligned with the visual modality to accurately label these clusters.
The moderate performance of frozen vision encoder pretrained with CLIP on pure vision tasks (see Appendix A.3) combined with the stellar results of LiT support that the second requirement is significantly easier to accomplish than the first one and that the first requirement can be addressed
independently of the second. Fortunately, various studies (Hamiltnn et al. 2022; Siiéónii et al.,
2021 Wang et al. [2023] Zhang et al. [0233; Siméoni et al., (023) have observed and capitalized on the spatial awareness of sell-supervised pretrained vision transformers (ViTs). These works have demonstrated remarkable capabilities for dense downstream tasks with minimal or even zero trainable parameters (Balazevic et al., 2024). This evidence suggests that self-supervised vision transformers may just be one text encoder away from becoming open-vocabulary segmenters. We investigate this hypothesis and propose a simple framework for open-vocabulary zero-shot segmen- capitalize a pretrained and frozen vision encoder that exhibits excellent
tation. In essence, fine-grained semantic coherence and focus solely on aligning a text encoder to it. To effectively enforce a localized consistency objective, we leverage linguistic knowledge to identify concepts in captions and build on the quality of the frozen vision encoder to retrieve the corresponding concepts in images.
Our main contributions are as follows:
(i) SimZSS is designed to be simple, with minimal hyperparameters, making it highly com- patible with both small curated datasets and large-scale noisy datasets.
(ii) The proposed framework is robust, supporting various pretrained vision towers, and ac- commodating both supervised and self-supervised pretraining of the vision backbone.
(iii) By decoupling visual representation learning from cross modality concept-level alignment, our proposed framework, SimZSS achieves high computational and data efficiency.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Open-vocab open-vocabulary learning. Traditionally, computer vision methods operate under the closed- vocabulary hypothesis. This assumption presumes that all object categories a model is expected to classify, detect, segment at the test time already known and labeled during training. This presents significant challenges due to the extensive labeling required and the limited generalizability of the resulting models. Open-vocabulary learning aims eliminate these limitations. Notably, this is a more challenging objective than the one targeted by self-superviied visual representation learning (Chen & He| 2021| Caron al. [021] Li et al. 2021 Grill al. 2020 Oquab al. 2023), which assumes the availability of labels at test time. To address this additional constraint, a text encoder is typically trained jointly with the vision tower to maximize vision-langue laniunl alignment on large amounts of image-caption pairs via contrastive learning (Radford et al., [021]] Dia et al.)
2021]. Subsequent studies have either focused on improving the cross- moddlity alignment pretext task (Alayrac et al. [0222] Yu et al. 2022; schannen al.) (202)) on improving the computational efficiency (Zhai et al., 2022) Li et al. 2023b). However, while these models excel at classification, their performance on dense downstream tasks is usually subpar.
Open- vocabularyby segmentation. The shortcomings open-vocabulary methods on dense down- stream tasks have been the focus of significant research efforts. A recurring approach in the literature to use dense annotations for learning segmentation and to leverage a pretrained text encoder to provide the weights of a generalizable classifier (Li et al. 2022, Ghiasi et al. 2022 Liang et al.)
2023 Ding et al. 2022; Zhou et al.) 2022a; Xu et al./ 2023b Ma et al. 2022 Yu et al. 2024). While these methods demonstrate excellent zero-shot segmentation performance, they only partially
alleviate the pixel-level annotation requirements.
To overcome this limitation, approaches that do not require segmentation masks have been proposed. PACL (Mukhoti et al., 2023) demonstrates the effectiveness of training only the projections at the interface of the two modalities and bootstrapping patch-level cross-modal similarities. Similarly, TCL (Cha et al. 2023) leverages existing fine-grained similarities to jointly learn text-grounded masks and impose contrastive region-text alignment based on the obtained masks. Alternatively, GiT Xu et al. (2022) proposes a specialized architecture that performs hierarchical group- ing/pooling of visual tokens based on their similarity with learnable query tokens. Another line of work showed the benefits of integrating a consistency objective between augmented and/or masked views of the input image (Dong et al. 20233 Ren al., (023) in the context of rision-language contrastive learning. Recent works (Shin et al. 2022; (Sun et al. 2024) also showed the possibility of achieving excellent zero-shot segmentation capabilities without any additional training. Finally, Wysoczańska et al. (2024) (023); Rewatbowornwong et al.) (2023) combine the vision-language alignment capabilities of CLIP (Radford et al. (021)) with the spatial- of self-supervised vision transformers, e.g., DINO (Caron al. 202 to develop open-vocabulary zero-shot seg-
menter.
text concect extraction
cross cross modclify vision concept extraction consstency
doe snd Eog grees
dog and Co on gisse
concept Idantification
text
encoder
Toxt repaesee fotlon
11000 CldN Ba
Via top wusrrottoo on
Figure 2: Overview of SimZSS. On the text side (a.), each concept in the caption is represented using a trainable text encoder. On the vision side (c.), visual representations of each concept are obtained via similarity- based pooling of the visual tokens. These visual concept representations are then projected onto a linear classifier, with weights derived from the text concept representations of the current batch. Cross-modality modality consistency is enforced using cross-entropy loss (b.).
3

---

Published as a conference paper at ICLR 2025
3 METHOD
3.1 PRELIMINARIES
We first provide a brief primer on the terminology used throughout the paper.
Dense representation. Transformers encode an input signal as a sequence of tokens, and we refer to the corresponding output sequence the dense representation. For vision transformers, this corresponds to a tensor 2, E Ru..dd, where de is the dimension of the representation space and ny is the number of patches in the input image. For text input, the dense representation E Rd a sequence of dy-dimensional tokens.
xd, is
Local representation. We refer to the result of indexing a dense representation with a sequence index i local representation. This translates 2 Rd for textual input signals and z e Rd. for visual input signals.
Concept representation. Aggregating local representations corresponding to the 1th semantic concept in the input signal yields a concept representation. For visual input, this is denoted as c, E Red,, whereas for textual input, we use cf E Rdd.
Global representation. A representation of the entire input signal referred to as the global representation. Specialized tokens typically used for this purpose. In vision transformers, the global representation 2, E Rd is the [CLS] token, while for a text encoder 2, E Rd denotes the [EOS] ] token.
3.2 VISION LANGUAGE CONCEPT IDENTIFICATION
At a conceptual level, enforcing cross-modality consistency faces significant challenge due to the complexity of identifying and matching concepts across modalities. In a nutshell, our proposed solution leverages the discrete nature of textual data, enabling the straightforward identification of concepts within captions. Subsequently, we retrieve associated visual concepts, conditioned on those identified within the text, effectively circumventing the complexity of cross-modality concept matching.
3.2.1 TEXTUAL CONCEPT IDENTIFICATION
Given image-caption pair (I,, v.), utilize the text modality to identify semantic concepts. Language adheres to structured grammatical rules, allowing us to leverage linguistic knowledge to pinpoint the main subjects and objects within sentence. Since captions intended describe images, the primary subjects and objects in the captions are likely to correspond to visual elements in the images. Therefore, we employ a part-of-speech (POS) tagger, which automatically classifies words according to their grammatical roles. In our context, noun phrases (NPs) are of particular interest, as they generally encapsulate the main subjects and objects in the sentences, providing direct link to the visual concepts depicted in the images. However, captions can be quite noisy, so further filtering necessary discard NPs that are unlikely to appear the images. To address this, we refine noun phrase into a concept by restricting it to the noun and its first compound, We then discard any concept that is absent in a predefined bank of concepts.
Once the concepts in the caption have been identified, it remains to obtain their equivalent in the representation space. This accomplished by tracking the token indices spanned by each concept. Denoted Sj. this set represents the indices associated with the (th concept. The concept represen- tation of e IR is then derived by averaging the local representation at the corresponding indices:
c:
St
iES.
4

---

Published as a conference paper at ICLR 2025
The 1th textual concepts cj Rd is then mapped to the visual space via a linear projection g -4 + Rd:
= (c))
In the remainder of this paper, we also refer to of text concept.
3.2.2 VISUAL CONCEPT IDENTIFICATION
Upon identifying the 1th text concept ée E RRd,, we use it to query the dense visual representation and obtain the corresponding visual concept c, E Rd.. To do this, we first embed the image 2y using a vision encoder fu. which outputs the dense visual representation representation &. E R We then compute the similarities between each local visual representation and the query text concept:
E IRa.xdd and a global visual
softman
where т is a temperature parameter that regulates the sharpness the similarity distribution (see Tab. 9). Consequently, the 7t visual concept is obtained by bootstrapping existing cross- modality similarities, i.e., via similarity-based pooling (see Fig.
C₂
x, s
(4)
This process performed in parallel for all concepts from the captions in the batch, resulting in eross-modality concept pairs for which consistency be encouraged.
3.3
CROSS-MODALITY CONSISTENCY
we outline a methodology for identifying pairs local concepts across textual and visual modalities. This method leverages existing similarities and thus may benefit from additional supervision at the beginning of training, as discussed in Section 3.3.I
In Section 3.2
3.3.1 GLOBAL CONSISTENCY
We now discuss the global consistency objective, which ensures similarities between images and captions. To this end, let Zu R6xd, be the matrix containing all global visual representations ž, within a batch, and let Z, E Rixxd be its equivalent in the text modality. After projecting the text concepts the visual space,
compute the global cross-modality similarity:
8 g (Z.) Z
(5)
The learning objective is maximize the similarity of paired entries and minimize that of unpaired ones:
exp (5x)
exp ($1))
-
log
log
(6)
2b
Σn exp (5:))
2b
Σ, exp (5:y)
Overall the global consistency objective is identical to the one used in CLIP (Radford al. 202
3.3.2 CONCEPT-LEVER CONSISTENCY
After obtaining pairs of vision-language concepts, an intuitive approach to enforce consistency at the concept level is to use contrastive objective, akin to the one described in Section 3.3.1 but applied between concepts from each modality. Empirically, we found that this approach did not yield the desired performance improvements on dense downstream tasks. At a global level, images and captions represent complex scenes, supporting the hypothesis that only b positive image-caption pairs exist among the 6² possible ones in a batch. Conversely, concepts typically encode individual objects that are likely to occur multiple times within a batch. This suggests that an instance-level objective is ill-suited for our setting. Therefore, we opt for a semantic-lever objective.
Let's 's define C, E R bxd, representing the set of all b text concepts in batch and C, E gixd, as its counterpart in the visual space. Thanks the discrete nature of text, it straightforward keep track of concepts occurring in a batch, with each unique concept assigned specific index, conve- niently stored in q E (0, 1, .... k- -18 (where l denotes the number of unique concepts in batch).
5

---

Published as a conference paper at ICLR 2025
The weights h Rexde of linear classifier can be computed by summing the representations of identical concepts:
h; 1lag
After 12-normalizing the columns of both h and Cv, we project onto the other derive probability distribution:
= softmax (C,h²)
(8)
follows that the cross-entropy loss can be used to ensure consistency between the query text concept and the retrieved visual concept:
= ΣΣ - (q,=)} log (Pij)
L
(9)
The overall objective of SimZSS, denoted Lus, is weighted sum of the global and local consis-
tency objectives:
Lix Cg AL:
(10)
where À is weighting parameter whose impact is ablated in Table 9
4
EXPERIMENTS
In this section, we investigate the properties of SimZSS through various experiments. Additional experiments can be found in Appendix
4.1
EXPERIMENTAL SETUP
4.1.1
PRETRAINING DATASETS
We train our models on two distinct datasets: COCO Captions (Lin et al. 2014 Chen et al. 2015) and LAION-400M (Schuhmann et al. 2021). The former comprises 600K image-caption pairs with high-qualit, numan-generatte captions, while the latter is derived from LAION-5B (Schuhmann ct al. 2022) consisting of image-caption pairs with languaean cosine similarity exceeding 0.3, determined by pretrained CLIP model (Radford et al. 2021). These datasets represent opposing ends the spectrum in terms of curation and scale.
4.1.2 NETWORKS ARCHITECTURES
Vision transformers (Dosovitskiy al., 2020 are used to obtain image representations. More pre- cisely, we experiment with VIT-S/14 and ViT-B/14 pretrained with DINOv2 (Oquab et al.) 2023) on LVD--42M. ViT-B/16 pretrained with AugReg (Steiner et al. 2022) on ImageNet-2TK is also tested. The architecture of the text transformers identical to the used in CLIP. and its weights are randomly initialized. Overall, the only architectural difference w.r.t. CLIP is the removal of the learnable linear layer that maps visual representations to the cross-modal representation space ie., we project textual representations directly onto the visual space rather than projecting both the textual and visual representations onto an intermediary space.
4.1.3 OPTIMIZATION
For COCO Captions, we conduct training over 4M processed samples (~ 6.6 epochs) using global batchsize of 16,384. We incorporate warm-up strategy spanning 10% of training steps, lin- early ramping up the learning rate until it reaches its peak value, chosen from the set (8e-5, 3e-5, 8e-6, 6, 3e-6 H'I Subsequently, we employ a cosine decay schedule for the remaining steps. Similarly, for LAION-400M,
train for epoch with global batchsize of 32,768, and set the learning rate from the options (3e-5, 8e-6, 3e-6, 8e-7, 3e-7). The remaining optimization settings align with those of OpenCLIP (Ilharco et al. 2021).
scaled by batchsize 256

---

Published as a conference paper at ICLR 2025
4.1.4 TEXT CONCEPTS
We use the en_ wwwp_eee_ett__eeeeeeeeeeeeeeeeee model from SpaCy (Honnibal et al. 2020) as part-of-speed tag- ger to identify noun phrases. The concept bank obtained as the union of the class names from Pascal VOC (Everingham et al. at, 2012). Pascal Context (Mottaghi et al. 2014). COCO-Stuff (Caesar et al. 2018), Cityscapes (Cordts et 20T6) and ADE20K (Zhou et al. 2017). This results 574 concepts.
4.2 ZERO-SHOP SHOT SEGMENTATION OF FOREGROUND
Table I: Zero-shot foreground segmentation. Pixel-wise predictions are obtained by project- ing patch representations onto pre-computed text embeddings of the class names, followed by up- sampling. The mloU scores reported across five standard segmentation datasets. f refers to reproduction using DINOv2 pretrained vision backbones. The remaining results are as reported in Wysoczańska et al. (2023)
COCO-Stulf
Cityscapes
Miscellancoos
Shig
Group 10 F022
35M
23,4
15.3
11.3
21M
30.3
19.6
23.1
26.4
16.4
12.6
DVD
Kannej 23063
33.7
CL
DiNDD 1M5000e000 1023
80.9
35.9
24,6
31.7
20.0
LAION 500
CLIP Radfonl et [021 (VT--)
157M
35.
7.7
4.2
1.8
2.0
Ln7 Zhai di N022 (VT)B)
94M
63M
80.5
51.8
23.3
24,7
18.7
SimZ33
94M
63M
85.1
34.2
24.9
27.8
19.6
COCO Captions
L27 Zlaieral,
2022 (VT-B)
63M
35.5
25.6
25.8
40M
57.3
29.2
Sin255
63M
33.0
We validate the proposed method on pixel-level zero-shot segmentation task. In this scenario, the model relies solely on textual class descriptions to classify image pixels. However, accurately repre- senting the background class can be challenging due dataset-specific properties not fully cap- tured by text descriptions. As such, we follow previous works (Wysoczañska et al. 2023 (0023:Co Cha et al.) 2023) and first evaluate without the background class. We follow the MMSegmentation (Contrib-) utors 2020) implementation of Cha et al. (2023). Specifically, images are resized to have a shorter side 448 pixels, and inference performed with sliding window of 448 x 448 and a stride of 224 pixels. ImageNet templates from Radford et al. (2021) are used to contextualize each class name before obtaining their textual representation. Finally, the predictions obtained by projecting patch representations onto class names, are up-sampled using bilinear interpolation to restore the predic- tions to the original image size. We report the mloU scores across five standard datasets, namely Pascal VOC (Everingham et al. 2012), Pascal Context (Mottaghi al.] 2014), COCO-Stuff (Cae-) sar et al., 2018), Cityscapes (Cordts et al., 2016) and ADE20K (Zhou et al. 2017]. For datasets that encompass the background class, the corresponding annotations and pixels are simply ig- nored. For the sake of clarity, we report the results without any post-processing techniques such as Pixel-Adaptive Mask Refinement (PAMR) (Araslanov & Roth 2020).
In Table D we observe that our method using a ViT-S/14 as the vision tower is already competitive and that shifting to a larger vision backbone, namely ViT-B/14 14, yields state-of-the-art results on all datasets when trained on COCO Captions. In particular, it can be seen that the proposed pipeline significantly outperforms models pretrained with LiT (Zhai et al. 2022) under the same conditions. This suggests that the reported performance is not solely attributable to the freezing of the vision tower or DINOv2 (Oquab et al. 2023) visual pretraining. The results obtained with LAION-400M indicate that SimZSS is compatible with large-scale uncurated datasets, exhibiting overall strong performance and improvements compared to models trained with LiT in the same setting. However, it is also apparent that both LiT and SimZSS seem to benefit more from curation than scale for segmentation. This trend contrasts with findings classification tasks, as shown in Table
Finally, we present the performance of a CLIP-based model trained with ViT-B/16 vision back- bone on the LAION-400M dataset for 32 epochs (Tlharco et al., 202T). The results highlight that, while CLIP excels image-level tasks such as zero-shot classification, it demonstrates limitations
7

---

Published as a conference paper at ICLR 2025
sidewaik
Leash
zebra
elephant
butfll
plate
donat p1226
gizade
ZOOZ2
forest
IIS COM bara
ch okkn
Cher
cnite
Figure 3: Vision-language alignment of text concepts and dense visual representations. Con- cepts present in the image are embedded independently by the text encoder and then projected onto the representations of each patch within the image. The images are processed resolution of 896 x 896 pixels, corresponding to 4x the training resolution. The alignment is performed on LAION-400M using ViT-B/14 as the vision tower.
Table 2: Zero-shot segmentation. Pixel-wise predictions of the foreground classes obtained by projecting patch representations onto pre-computed text embeddings of the class names, followed by up-sampling. Predictions that fall below predetermined threshold are assigned to the background class. The mloU scores are reported across three standard segmentation datasets. refers to our reproduction using DINOv2 pretrained vision backbones. The remaining results reported in
Wysoczańska et al. (2033).
Method
Params
Params
Pascal Context
COCO-Object
Pascal VOC
Miscellanenus
ReCo (hhnn al. 2022
313M
19.9
15.7
25.1
OVDin Karazziaa ml.
2023
1,226M 0
0
30.1
34.8
67.1
GroupV/ et
55M
18.7
27.5
50.4
ZeroSeg (Chiit
2003
21.8
22.1
42.9
SegCLIP (LD 20753
24.7
26.5
52.6
TCL (Cha 2025
156M
21M
24.3
30.4
51.2
CLIP/Y [Kaniiingge
2026
32.0
52.2
OVSegneetor Duu 201450
20.4
25.1
53.8
CLIP-DIY WWstezziis 1e
2024
19.7
31.0
59.9
MiskCLIP ofog 2015)
291M
23.6
20.6
38.8
CLIP-DINOner Wystczumsad el 2023
32.4
348
62.1
LATON-400M
CLIP RRdford et al 2021 (VIT-B)
0
157M
6.9
5.2
11.1
LiT Zhas et al. 2022
(VT-B)
94M
63M
29.6
38.3
48.1
Sim2Ss (V1-B)
94M
63M
31.1
38.1
48.6
COCO Captions
Zhai al. 2022 (VI-B)
94M
63M
31,5
39.5
51.4
SimZSS
23M
40M
32.8
39.5
55.5
SimZSS (VIT-B)
94M
63M
37.2
43.5
58.4
in achieving fine-grained cross-modal alignment. In Appendix A.3
we address this limitation and provide evidence suggesting that is not solely attributable to vision-language misalignment.
4.3 ZERO-SHOT SEGMENTATION
In second scenario, we explore a zero-shot segmentation task including background class. Sim- ilar to Cha al. (2023), we do not rely the textual representation predict the background
8

---

Published as a conference paper at ICLR 2025
Table 3: Zero-shot classification. Image-level predictions are obtained by projecting the image [CLS] token onto pre-computed text embeddings of class names. Accuracy is reported for various visual pretraining and lasgun- alignment methods. refers to our reproduction using DI- NOv2 pretrained vision backbones. The remaining results are as reported in Zhai et al. (022))
Method
Visui preirainng
Backbone
Preiraining datanet
Allgiment dutanet
Allguemtent somple
Labrk
IraggrNet-1 Ik
Averag
LTT Ziriet
NYN
Mi [berettal
2011
V/T
IraggNNen-1 3k IniegeNen:1k
CCIEM-YECCCOM CFCMIYP
211k
DINOY2
V/15//4
COCO LAION-400S LAHON-100
26.4
DINO
V/1
LND- 142M
403M 128
37.5
C1.1
VIEV116
67.0
47.2
Oher
Sin7es
DINOS?
Dquabatal 2073
VITRVIA
IND-147M IND-
COCO Capiies
261
Sia255
DINOY2
V/T
LA00N-400M
41.3
class, but rather the confidence levels of the predictions the remaining classes. More pre- cisely, we assign given pixel to the background class if the highest confidence score among the other class predictions falls below a dataset and model-specific threshold. The remaining im- plementation details are identical to the above-described setting. The mloU scores are reported on three datasets, namely Pascal Context (Mottaghi et al., 2014), COCo-Object (Caesar et al., 2018) and Pascal VOC (Everingham et al. 2012).
In Table we observe similar trends as in the scenario without the ba ck ckground class (see Tab. [ Reported results are not as unequivocal as in the former evaluation. This is in part due to the crudeness of the background detection mechanism, which contrasts with the more sophisticated approaches used by some of the competing baselines. For instance, CLIP-DINOIS
(Wysoczañska (et al. 2023) relies on FOUND (Siméoni et al.] 2023), whereas OVDiff (Krazzja et al. (023) uses different background prototypes for each class. Once again, training on COCO Captions leads to improved performance. When training on LAION-400M, a concept from the bank of concepts is identified in less than 15% of the samples on average. As result, concepts rarely co-occur in single sample, making it sufficient to detect the concept without localizing it in the image to minimize the loss. Therefore, the model is not trained to be less confident the background.
4.4 ZERO-SHOP CLASSIFICATION
The proposed method derives from LiT (Zhai et al. 2022 noted for its excellent zero-shot classi- fication performance on ImageNet-1k 1k (Deng al.] 2009) among other benchmarks. Although our approach is tailored for dense downstream tasks, investigate whether SimZSS exhibits zero-shot classification capabilities or if the introduced local consistency objective breaks this property. We follow the evaluation protocol from OpenCLIP (Ilaarco et al. (021). Class names are contextualized with the corresponding templates from CLIP (Radford et al. (202)) and embedded through the text encoder. On the vision side, images are passed through the vision encoder, and the global represen- tation is used to obtain class predictions via projection onto the class embeddings. We evaluate on 38 datasets and report the average top-1 accuracy in Table 3 The accuracy on ImageNet-Ik k is also reported specifically. The complete zero-shot classification results on the 38 datasets are depicted in Figure
The results reported in Table 3 reveal that SimZSS not only preserves classification capabilities comparable LiT but also exhibits slight improvements the low-data regime, namely when train- ing on COCO Captions. It is worth mentioning that the performance of CLIP after 1 epoch is approximately 43% (as depicted here). Overall, while pretraining on COCO Captions excels for
Table 4: Training without the bank of concepts. The impact of the concept bank is evaluated by training SimZSS on COCO Captions without it, that is, the concept-level objective is enforced for all noun phrases in the captions. The training is conducted with ViT-B/14 pretrained with DINOv2. f indicates results reproduced using DINOv2 pretrained vision backbones.
Without backgromnd
backgronnd COCD-Bbjer
Method
Bank
Poscal VDC
Precal Context
COCO-Stuff
Citysaaps
ADEXOK
Pasczil Contest
Pascil VOC
1,T" (2is at al. 2012 Sim2ss SinZss
86.
35.5
25.6
25.8
18.1
31.5
30.5
$1.4
$9.2
41.1
27.6
3122
20.0
35.2
42,6
56.8
90.3
43.1
29,0
33.0
21.8
37.2
43.5
56.4

---

Published as a conference paper at ICLR 2025
segmentation tasks, zero-shot classification benefits from larger datasets. This suggests that pre- training on LAION-400M followed by finetuning on COCO Captions could optimize the benefits of both datasets.
4.5 BANK OF CONCEPTS
At its core, SimZSS relies on identifying noun phrases within captions and filtering them through bank of concepts. This filtering ensures that the identified noun phrases are valid candidates for visual concepts and can be used to query the images. A key question that arises is whether the segmentation improvements of the proposed method are confined the classes within the concept bank or if they generalize to other classes as well. To test this hypothesis, we trained SimZSS on COCO Captions without the concept bank.
As expected, the scores reported in Table 4 are slightly lower compared to using the concept bank. However, the results consistently outperform the baseline LiT, which focuses solely on enforcing consistency between caption and image representations. Moreover, we observe that the performance of SimZSS without the concept bank closer to the version with the concept bank than LiT. This demonstrates that SimZSS's performance is driven more by the consistency objective between noun phrases and localized image regions than by the concept bank itself. Considering the results achieved conclude that SimZSS yields great
by concurrent basclines on the same tasks (see Tabs. and 2) and performance compared to existing methods, irrespective of whether a concept bank is used, which shows that concept does not need appear in the concept bank to be properly localized at test time.
Alternatively, Figure suggests that the vision tower, here ViT-B/14 pretrained with DI- NOv2 (Oquab et al., 2023). could handle more concepts than those present in the concept bank. Indeed, some visual concepts, such bi ke (third row, columns 1-5), appear well-segmented in the image but are not highlighted in the caption, meaning the concept is absent from the concept bank, and thus, no constraint is enforced. This claim further supported by the fine-grained vision- language alignment depicted in Figure The visualization further shows that, even though concepts such as forest, chef, water buffalo or bike are not in the concept bank, the text concepts and the corresponding patches appear well aligned.
5 CONCLUSION
We introduce SimZSS, a simple framework to endow pretrained pure vision models with open- vocabulary segmentation capabilities. This approach is versatile: it accommodate various back- bone sizes, pretraining methods, and datasets, irrespective of their scale and degree curation. Overall, SimZSS is both computationally and data-efficient method that yields excellent results across standard zero-shot segmentation benchmarks while preserving efficiency at inference time.
Reproducibility Statement We will release code and pretrained models the public upon acceptance. The release will include a README file with detailed instructions for setting up the environment and running the training. All datasets used in our work publicly available.
Acknowledgement This project is funded by the Personalized Health and Related Technologies (PHRT), grant number 2021/344, well as the European Research Council (ERC) under the Euro- pean Union's 's Horizon 2020 research and innovation program (Grant Agreement No. 101021347). This work is also partially supported by funding from the Flemish Government under the "Onder- zoeksprogramma Artificiele Intelligentie (Al) Vlaanderen". This work was supported by a grant from the Swiss National Supercomputing Centre (CSCS) on the Swiss share of the LUMI system under project ID 606.
REFERENCES
lean-Baptise Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716
23736, 2022.
10

---

Published as a conference paper at ICLR 2025
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Pro- ceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4253- 2020.
Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelovié, and Olivier Henaff. To- wards in-context scene understanding. Advances in Neural Information Processing Systems, 36, 2024.
Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding every- thing: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3828-3837, 2024.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedinggs of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11165-11174, 2023.
Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Culatana, and Mohamed Elhoseiny. Exploring open-vocabulary semantic seg- mentation from clip vision encoder distillation only. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 699-710, 2023.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750,15758, 2021.
Xinlei Chen, Hao Fang, Tsung-Ys Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 00325, 2015.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self suf--sueervieed vision transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp- 9640-9649, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-12999 1209, 2022.
MMSegmennation Contributors. MMSSgmmentttion: Openmmlab semantic segmentation toolbox and benchmark. htttp::/ http:/ccooopppppeeeooeeee mml ab/ //mmma/mmmmmmmttt..ooooo 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009, doi: 10. 101./C/VR.209902006633
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11583-11592, 2022.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Maskelip: Masked self- distillation advances contrastive language-iman pretraining. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10995-110055.0 June 2023.

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Reprexentations, 2020.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.paseal- 2012.
The
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo. Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to sule-supprvised learning, 2020.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Un- supervised semantic segmentation by distilling feature correspondences. In International Confer- ence on Learning Representations, 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial- strength natural language processing in python, 2020.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https: //tpss///cc... org 10. 5281/ Izerodo. 5143773
Chao Jia, Yinfei Yang, Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero- shot open- open-vocabulary segmentation. arXiv preprint arXiv:2306.09316, 2023.
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open- vocabulayy segmentation. In ECCV, 2024.
Tim Lebailly, Thomas Stegmüller, Behzad Bozorgtabar, Jean-Philippe Thiran, and Tinne Tuyte- laars. Cribo: Self-supervised learning via cross-image object-level bootstrapping. In The Twelfth International Conference Learning Representations, 2024.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022. URL
htttps://openreviee. htt.rarrwweeeeeetttooouuueeeeee Beee//oorummlGGrriijjaa
net
Chunyuan Li, Jianwei Yang. Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. In Inter- national Conference Learning Representations, 2021.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ing image pre-traine training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023a.
12

---

Published as a conference paper at ICLR 2025
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Com- 2023b.
puter Vision and Pattern Recognition, pp.
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05665. 2023c.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. speablocayulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7061-7070, June 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong Hc, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. International Conference PMLR, 2023.
on Machine Learning, pp.
Chaofan Ma. Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulary semantic segmentation with frozen vision-language models. In 33rd British Machine Vision Conference (BMVC). BMVA Press, 2022. URL https: wttp///wwcc22cc///////////// wwmmmppiiiin inf. wpw.ppp...........p..... de 0045. pdf
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic seg- mentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898 -988 2014.
Jishnu Mukhoti, Tsung-Yo Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19413-19423, 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Shang-Wen Li, Wojeiech Galuba. Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve. Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Valentinos Pariza, Mohammadreza Salchi, and Yuki Asano. Hummingbird evaluation for vision en- coders, 4 2024a. URL
com/
Valentinos Pariza, Mohammadreza Salehi, Gertjan Burghouts, Francesco Locatello, and Yuki M Asano. Neco: Improving dinov2's spatial representations in 19 gpu hours with patch neighbor consistency, arXiv preprini arXiv:2408. 1054, 2024b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. In Proceedings of the JEEE/VVF International Conference on Computer Vision pp. 5571-5584, 2023.
Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supe segmentation masks via multi-view semantic consistency. In The Eleventh International Conference on Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2025
Pitchaporn Rewatbowornwong Nattanat Chatthee, Ekapol Chuangsuwanich, and Supasorn Suwa- janakorn. Zero-guide grro-guidance segmentation using zero segment labels. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 1162-1172, 2023.
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114. 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2522 2022.
Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot trans- fer. Advances in Neural Information Processing Systems, 35:33754-33377 2022.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-super transformers and no labels. In BMVC 2021-32nd British Machine Vision Conference, 2021.
Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín Vobecký, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Observing the background to discover objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3176-3186, 2023.
Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, and Jean lean-Philippe Thiran. Croc: Cross-view online clustering for dense visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7000-7009, June 2023.
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7262-7272, 2021.
Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip mn: Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13171-13182, 2024.
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances in Neural Information Processing Systems, 36, 2024.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: inference. In European Conference Computer Vision, pp. 315-332. Springer, 2025.
Rethinking self-attention for dense vision-language
Haoqi Wang. Tong Zhang, and Mathieu Salzmann. Sinder: Repairing the singular defects of dinov2. arXiv preprint arXiv:2407. 240.:2220.666 /6826, 2024.
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li. Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. IEEE transactions on pattern analysis and machine intelligence, 2023.
Monika Wysoczańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzciński, and Patrick Pérez. Clip-dinoiser: Teaching clip few dino tricks for open-vocabulary semantic segmentation. arxiv, 2023.
Monika Wysoczańska, Michaël Ramamonjisoa, Tomasz Trzcińskii, and Oriane Siméoni. Clip-diy: Clip dense inference yields vocabuacyy semantic segmentation for-free. In Proceedings of the IEFF/CVF Winter Conference Applications of Computer Vision, pp. 1403-1413, 2024.
14

---

Published as a conference paper at ICLR 2025
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18134-18144, 2022.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2935-2944. 2023a.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2945-2954, 2023b.
Jiawei Yang, Katie Z Luo, Jiefeng Li, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denois- ing vision transformers. arXiv preprint arXiv:2401.0280 022557 2024.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seycdhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learn- ing Research, 2022.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18123-18133 18133, 2022.
Xinyu Zhang, Yuting Wang, and Abdeslam Boularias. Detect every thing with few examples, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattemn recognition, pp. 633-641, 2017.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 696-712. Springer, 2022a.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representa- tions (ICLR), 2022b.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL EXPERIMENTS
A.1 IMPACT OF THE VISION TOWER
The underlying assumption of SimZSS is that the visual representations do not require further train- ing, and only the vision-language alignment must be learned. As such, the quality of the vision backbone pivotal to the overall performance of SimZSS.
In Tables and segmentation tasks. Our observations indicate that SimZSS is versatile and can integrate diverse models and pretraining approaches effectively. As expected, models with superior performance pure vision tasks also show enhanced results when used with our method, as well as with LiT. Most importantly, regardless of the vision backbone employed, SimZSS consistently surpasses LiT. Ad- ditionally, employing denoised ViTs (Yang et al., 2024), which exhibit better semantic feature cor- relation, leads to improved segmentation. This outcome is anticipated since our approach assumes that features corresponding to semantically related regions are highly correlated (see Sec. 3.2).
explore the performance of various pretrained vision towers in zero-shot
shown in Tables 5 and 6 using a ViT-L/14 as the vision backbone does not improve overall per- formance compared to the ViT-B/14, whether for LiT (Zhai al.] 2022) or SimZSS. This outcome cannot be attributed to overfitting due to the increased number of trainable parameters in the text encoder. Replacing the text encoder with the architecture tailored for the ViT-B/14 did not result in any performance gains. The results for the ViT-L/14 in Tables a were obtained using average pooling of the dense visual representation. This approach necessary because observed that the [CLS] token in this model was poorly aligned with the remaining tokens, making the training of both LiTand SimZSS unsuccessful. These findings align with the lower results reported by Pariza et al.] (2024a) when using frozen features from DINOv2's ViT-L/14 for dense downstream tasks, as well as with the growing body of literature on the denoising of DINOv2 models (Yang et al. 2024 Wang et al. 2024).
A.2 COMPARISON WITH TRAINING-FREE APPROACHES
We compare SimZSS with training-free approaches on the zero-shot segmentation benchmark. The results in Table 7 show that these methods, particularly ProxyCLIP (Lan et al.) 2024), can achieve strong performance without requiring any training. Morcover, these methods retain CLIP's zero-shot classification capabilities.
While SimZSS does require training, the cost is manageable it only takes 15 minutes on 8 GPUs (for training on COCO Captions) and offers several advantages. SimZSS enables the use of smaller models, such as ViT-S/14, can be tailored to specific domains, and is generally less computationally intensive at inference time, which can effectively amortize the training cost.
Table 5: Ablation study the vision backbone for zero-shot foreground segmentation. We investigate the effect of the vision backbone's size and pretraining method on the downstream task. The impact using artifact-free models is also evaluated (with/without DVT (Yang et al. 2024)). We report the mIoU scores across five standard datasets. i refers to our reproduction using DINOV2 pretrained vision backbones.
Pretraining
Context
cOCO-Stuff
Cityscapes
Supersieed
LIT' Zhi etal. 2022
AugReg
WTB8/16
18.3
13.3
15.6
SimZSs
AugReg
Self-suwmisn!
11T Thii otal. by23
DINOv2
VIT-S/14
84.0
92.8
22.0
24.0
16.3
SimZSs
DINOV2
VIT-S/14
$7.3
Zhi etal.
DINO+2
VIT-8/14
86.3
34.1
24.6
25.8
17.1
Sim7
DINOW2
VIT.R/14
87.4
41.1
26.6
31.6
19.9
1:T
Zhi et
2013
DINO+2
VIT-
86.1
35.5
25.6
25.8
18.1
VIT-8/14
43.1
1/7'
Zhii et 2022
DINOV2
VIT-1/144
81.5
33.5
25.1
26.0
17.6
Sim2Sx
DINOV2
VIT-1/14
83.9
36.4
24.5
31.1
18.8
16

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study on the vision backbone for zero-shot segmentation. We investigate the effect of the vision backbone's size, pretraining method, and patch size on the downstream task. The
impact of using artifact-free models is also evaluated (with/without DVT YYngg et al. 2024)). We report the mloU scores three standard datasets.
Method
Pretraining
Backbone
DVT
Pascal Context
COCO-Object
Pascal VOC
Supervised
LT Zhui Zhui ela Sim258
2022
AugReg
VIT-B/16
17.8
26.1
32.2
AugReg
VIT-W16
X
21.6
29.7
35.9
Self superviedd
LiT Zhai et al.
2022
DINOv2
VIT-S/14
X
29.9
36.5
49.6
Sim2SS
DINOv2
VIT-S/14
32.8
39.5
55.5
Zhui et
2022
DINOv2
ViT-B/14
30.5
28.7
49.8
SimZSS
DINOv2
VIT-B/14
35.7
40.5
57.5
LiT Zhui et ul.
2022
DINOv2
VIT-B/14
31.5
39.5
51.4
Sim2SS
DINOV2
VIT-B/14
43.5
58.4
1i1 Zhai 2022
DINOV2
ViT-1/14
29.2
33.2
42.1
SimZSS
DINOV2
VIT-L/14
31.9
34.6
46.6
Table 7: Comparison with training-free approaches. Training-free approaches are compared SimZSS trained on COCO Captions with different vision backbones pre-trained with DINOv2. Results for the training-free approaches are reported as in Lan et al. (2024).
Without backgraand
With background
Method
Backhone
Pascal VOC
Piecal Contest
CUCU Statt
Uhysccpes
ADE2IK
Contert
CUOO--Uyect
Piscal
WT--Bac:
ViT-Busc
54.2
32.2
PresyClute
VIT
38.1
VIB
19.4
12.4
11.
73
11.
73
21.1
SCLIP Walr
Willaage
69.1
22.3
17.6
186
10.9
223
25.0
43.5
PhoaaC(iP [221 11.2004
VIT Logge
83.1
57.7
25,6
46.1
224
34.5
39.2
00.5
SimZSs
87.2
37.3
213
2922
17.9
128
75.5
55.5
Smn2S
VIT-Bacc
96.3
43.1
29.0
33.0
213
37.2
43.5
38.4
Sin255
VIT Large
63.9
36.4
24.5
31,1
193
31.9
34.6
46.5
A.3 IDENTIFYING BOTTLENECKS
To pinpoint potential areas for improvement, we take close look the performance of SimZSS in the context of zero-shot segmentation, directly comparing it to other segmentation methods on the same datasets and backbone. To dissect the influence of vision-language alignment on overall performance, strip out the language model's class name embeddings and replace them with visual embeddings. These visual embeddings, derived from the training set, are computed by average pooling of the corresponding local representations for each class.
In Table 8 we report the results of the above-described experiment using the ViT-B/14 from DI- NOv2. For datasets that include background class, explore two strategies: treating the background as a distinct class with its own concept, and üi) classifying a given patch or pixel as background when the confidence of the prediction across all foreground classes low.
The results support the hypothesis that the background is, general, too heterogeneous to be well captured by single concept, even in the absence of modality gap. We observe that the few- shots strategy generally outperforms the zero-shot one. The high variability of the difference in mloU scores between two strategies across different datasets indicates that the modality gap is not the sole reason for the observed discrepancies. In the context of zero-shot segmentation, the con- cepts are common all datasets, while in few-shot segmentation, they are dataset-speeific. This is well-reflected by the similarity of the two strategies on COCO-Stuff and COCO-Object, which can be attributed to the alignment of the pretraining dataset (COCO Captions) with the downstream datasets. Overall, this suggests that the image-caption pairs dataset is stronger limiting factor than the modality gap itself. Leveraging image-captioning methods (Li et al. 2023a Liu et al.) (2024) to generate captions for target datasets could prove beneficial. The performance achieved through dense neighot retrieval or linear segmentation using the same vision backbone on the same datasets indicates that there is still plenty of room for improvement.
Conversely, Table 10 from DINOv2 (Oquab et al. 2023) demonstrates significantly superior per- formance on the linear segmentation downstream evaluation across all their models, including ViT.
17

---

Published as a conference paper at ICLR 2025
Table 8: Few-shot vs. zero-shot segmentation comparison. We compare the zero-shot segmenta- tion performance of SimZSS with an alternative approach that constructs bank of visual concepts from the training set rather than relying on embedded class names (i.e., textual concepts). For refer- ence, we also include the performance of linear segmentation and dense nearest-neighbor retrieval, as reported Pariza et al. (2024b).
Withont background
With background
Method
Background
Pascal VOC
Puscal Context
COCO Stuff
Cityscages
ADEZUK
Pascal Contest
COCO Object
Pascal VOC
Donse L-NN
Concept
38.7
76.2
Linear
Concept
57.6
40.3
78.3
79.8
Tew-sbots
Concept
9144
51.0
31.3
49.7
31,7
43.7
27.7
$1.6
Few-shots
Contdeace
91.4
Table 9: Ablation study on the impact of the loss weight À and the temperature parameter т. Training performed COCO Captions using a ViT-B/14 vision backbone. The performance is evaluated through a comparison of mloU scores on the zero-shot segmentation task.
Without background
With background
Loss weight
Temperature
Piscal VOC
Pascal Conteat
COCD-Stutt
Cityscapes
ADEZUK
Pascal Coneett
COCO--Obeer
Pascal VUC
À 0.00
86.1
35.5
25.6
25.8
18.1
31.5
39.5
$1.4
0.1
39.8
27.6
29.7
42.1
0.1
41.9
42.3
90.1
41.8
43.1
A 0.06
7- 0.00
892
42.3
28.8
33.1
2166
43.2
$7.0
À 0005
0.04
19.7
42.7
25.7
34.3
22.1
36.7
42.9
57.0
À 0.06
0.07
90.0
42.0
78.0
31.4
21.3
370
43.1
58.5
A 0.08
0.10
90.3
43.1
29.0
33.0
21.3
37.2
43.5
58.4
A 0.06
0.40
88.7
40.0
27.6
28.2
19.5
34.4
42.1
52.6
S/14, compared to OpenCLIP ViT-G/14 (Ilharco et al. 2021). This underscores the limitations the cross-modal contrastive objective in learning effective visual representations. Moreover, it suggests that methods relying on pretrained CLIP-based models face a lower performance ceiling compared our approach. Indeed, zero-shot segmentation can be viewed as pixel- patch-level classification using a frozen linear layer initialized with the embeddings of textual labels.
A.4 SIMZSS-SPECIFG HYPERPARAMETE
Thanks to its simplicity, SimZSS has few hyperparameters, with the primary ones being т, which regulates the temperature used query the dense visual representation, and which modulates the contribution of the dense loss to the overall objective (see Eqs. (4) and (10). After grid search on X, т, and the learning rate, we find that the jest-performing setting for training on COCO Captions is (X 0.05, r = 0.1). In Table 9 when varying À and around this optimal point.
we report the resulting zero-shot segmentation performance
It can be observed that the setting with À = O, corresponding to LiT (Zhai et al. 2022), performs the worst. Regarding т. SimZSS works best with low temperatures. This is not surprising, as this setting, fewer patches contribute to the representation of the visual concepts, aligning more closely with the downstream evaluations.
A.5 SCALABILITY SIMZSS
An important property of vision-language pretraining methods is their capacity leverage large datasets effectively, which necessitates them to be i) compatible with noisy image-caption pairs, and ii) computationally efficient.
We first verify in Tables 12 and 13 that SimZSS-speciffs operations incur only moderate mem- ory overhead and negligible time overhead compared to LIT. Considering this, along with the data efficiency of the proposed approach, SimZSS is orders of magnitude cheaper to train than CLIP. Sec- ondly, we escalate the number of image-caption pairs from LAION-4000 during training and report the performance in Figure Junderscore the scalability our method SimZSS and suggest potential gains from even larger datasets, such LAION-5B (Schuhmann al. 2022 increased training epochs.
zero-shot segmentation at the end of each training phase. The results depicted
18

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study on the impact of the training resolution. Training conducted on COCO Captions with an image resolution of either 224 x 224 or 448 x 448 pixels using a ViT-B/14 vision backbone pretrained with DINOv2. Performance on the zero-shot segmentation task is then compared based on achieved mloU scores. refers to our reproduction using DINOv2 pretrained vision backbones.
Without background
With backgruand
Method
Resobution
VOC
Cityscaes
Paccal
Object
Lit' Zhai ot 2002
468 448
54.2
33.9
25.5
24.9
17.3
31.1
39.7
50.5
Simn5Ss
448 448
$7.2
40.2
27.9
29.4
20.4
25.2
43.2
56.5
1ast
Zhai at al. 2023
224 235
6..
85.5
25.6
25.8
1811
31.5
59.5
51A
Simz
224 220
90.3
43.1
2001
33,0
21.8
37.2
43.5
58.4
A.6 IMAGE RESOLUTION
Segmentation methods typically benefit from higher image resolutions during training. To assess whether this holds for SimZSS, we conducted training on COCO Captions, increasing the image resolution from 224 224 pixels to higher resolution images, 448 x 448 pixels. For that experiment, the vision backbone utilized was ViT-B/14 pretrained with DINOv2. Apart from the increased resolution, all other training procedures remain unchanged.
As shown in Table 10 neither LiT nor SimZSS exhibited any performance gains when trained with higher-resolution images. This can likely be attributed to the fact that higher resolutions reduce the receptive field of each patch, leading poorer semantic alignment between local representations and the global image representation, typically captured by the [CLS] token. In LiT Zhai et al. (2022,, this misalignment hampers zero-shot segmentation performance, it weakens the alignment between local visual features and the global textual representation, which is fundamental to the task. The same trend is observed with SimZSS, which builds upon the knowledge acquired from the alignment of global textual and visual representations.
We further explore the effect of using higher resolutions during evaluation. After training with an input resolution of 224 x 224, we assess zero-shot segmentation performance using target size for the shorter side of either 448 or 896. As shown in Table TI increasing the evaluation resolution proves beneficial primarily for the Cityscapes (Cordts et al. (2066) dataset. This is expected, given that Cityscapes consists of high-resolution images that undergo substantial downscaling, leading to greater potential loss of information.
A.7 HIGH-LEVEL PROFILING & RUNTIME COMPARISON
In Table P2 we present a profiling of the operations performed during training step of SimZSS. Overall, SimZ.S -speciffie operations account for less than 1% of the total runtime. Table 13 further confirms that SimZSS runtime is comparable to that of LiT (Zhai et al.) (022) and adds modest memory overhead.
Zaro chr--hhet-ae coggeentatinn nnation mieu [S]
Pasral
ADE20K
Pasral
Citysrapes
COCO-Stul
Average
Relative performance [96]
40-
20-
4x
4x'10' 4x
4x
4x'10'
4x100
dimage-caption pairs processed
Pimsge-caption pairs processed
Figure Zero-shot segmentation performance as function the number of processed image- caption pairs in LAION-400M The left plot shows the mloU percentages for different datasets, while the right plot shows the relative performance percentages. Each data point represents the result of running lanion-language alignment from scratch using SimZSS; these not training curves.
19

---

Published as a conference paper at ICLR 2025
Table 11: Ablation study on the impact of the evaluation resolution. Training is conducted on
COCO Captions with an image resolution of 224 x 224 pixels using ViT-B/14 vision backbone pretrained with DINOv2. Zero-shot segmentation performance is reported for images resized shorter side of either 448 896 pixels. i refers to our reproduction using DINOv2 pretrained vision backbones.
barkgreond
barkground
Method
Shorter side
Citysapes
ADE2OK
et
2022
648
M.
253
18.1
31.5
19.5
51.4
Zlai et
2022
806
79.7
34.0
23.8
30.4
20.6
31.0
37.2
49.2
Sunzes
896
86.1
28.9
36.1
21.7
37.7
41.7
59,9
Table 12: High-level profiling of SimZSS. We report the absolute and relative times of the vari- ous operations performed in SimZSS. Experiments are conducted on single node with 4x AMD MI250x GPUs (2 compute dies per GPU, i.e., worldsi ze 8) with memory usage of 38GB compute die. The backbone used is ViT-B/14, and the batch size is set to 1024 per compute die,
totaling 8192. SimZSS sppecffc operations are highlighted.
Description
Operation
Absolute time per iteration [ms]
Relative time [%
Forward pass vision
fo(-)
1435.0
Forward pass
f((-)
194.0
Vision concept extraction
Equations
12.0
Features gathering
22.5
1.1
Global consistency
Equation 6
1.0
0.05
Concept-level consistency
Equation
0.9
0.04
Weights update
backpropagat
404.4
19.5
Total
2069.8
100
B EVALUATION DATASETS
Pascal VOC 2012 The Pascal VOC dataset (Everingham et al., 2010) contains 20 classes with
semantic segmentation annotations. The training set consists of 1,464 images, while the validation set includes 1,449 images. An additional background class is provided.
Pascal Context The Pascal Context dataset (Mottaghi dataset by providing detailed annotations for entire scenes. It includes 60 classes (including background class) of semantic segmentation annotations. The training set contains approximately 4,998 images, while the validation set includes around 5,105 images.
2014) extends the Pascal VOC
COCO-Stuff COCO-Stuff (Caesar et al. 2018) is an extension of COCO (Lin et al., 2014), pro- viding pixel-wise annotations for 80 things classes and 91 stuff categories. The annotations exhaustive, i.e., no pixel remains unlabeled. images for validation, covering a wide range of scenes and objects.
includes over 164K images for training and 20K
COCO-Object COCO-Object uses the same set of images as the above-described COCO-Stut Stuff, only contains labels for the "things" categories. An additional background label covers the remaining pixels.
ADE20K The ADE20K dataset (Zhou et al., 2017) comprises a diverse set of images with anno- tations for 150 semantic categories. The training set includes 20,210 images, and the validation set consists of 2,000 images.
The Cityscapes dataset (Cordts al. 2016) contains high-resolution images of urban street scenes, annotated for 30 classes. The training set includes 2,975 images, and the validation set includes 500 images captured from 50 cities for semantic segmentation and urban scene under- standing.
Cityscapes
20

---

Published as a conference paper at ICLR 2025
wds/cars
wds/country211
Simzss -LAIO-10OM
COCO Captions
Simzss COCO Captions w/ bank
Simzss COCD Captions bank
wds/fgvc_aircene aircraft
wds/gtsbb
wd5/imagenet-a
wd5//mageneto
wds/imagenet
vdsllmaaeeet1k
wd wdslimaggeet__kee sketch
wdxfimmggeeev22
wds/mnist-
wds/objectnet
wdstrenderedsste
s/st11100
wds/sun397
wdstvoc2007
wdd/vtab/cca Itech101
wds//tab//cffr110
wdsivtabjjcfar100
wds/vtab/clewr http:aab/clerr_clleestoooo claeeeeeooeeec___iiiieeo
wds/vtaa/cleer_ccom count_at
wddsviaabidiaabsii__eeeet hetpraaaiiiiieiiopaaeheeeo
wds/vtabbdmiaab
wdd/vtab/ddsprrtts Laweeees__eeeeeeeeeeeeeeeeeeeeee orientation
wesivtaa/dspritts__lbed position
wdssstabbddsprites_labe label position
wds/vtab/dtd
wds/vtab/euossat
wds/vtab//lowwrrs
wds/vvta//kiitii Beciiiitiicccleeeee clasest vehicle, distance-
hds//aab/ccmmm
wds/vlab/pets-
wds/vtab/resisc4
wds/vtab/smmllinrbb. baaab_labblaalluttth azimuth
wds/ttab/smallnorbb httpl//oorbbbbbeeeeeeeeeeee
wdd/vttb/svhn
20
40
60
80
100
Zero-Shot Classification Accuracy
Figure 5: Zero-shot classification benchmark. We report the top-1 accuracy of SimZSS with and without the concept bank 38 evaluation datasets when trained on COCO Captions and LAION- 400M. Additional comparison with LiT is provided.
21

---

Published as a conference paper at ICLR 2025
Table 13: Computational and memory efficiency. The efficiency SimZSS compared that of related methods, i.e. LiT and CLIP. When feasible, we report results using the local training batch size; otherwise, the largest power of that fits into memory is utilized. The reported val- ues are obtained on single node equipped with 4x AMD MI250x (2 compute die per GPU, i.e., worldsize 8).
Method
Batchsize per compute die
Memory per compute die [GB]
Time step [ms]
Throughput [img/s]
CLIP
256
40
1196.0
1712
LIT
1024
~ 27
2049 2
3997
SimZSS
1024
38
2069.8
3957
22


Reviews:
[{"review_id": "kDecrRhtzO", "reviewer": "Reviewer_t3xp", "summary": "This work proposes a simple framework for open-vocabulary Zero-Shot Segmentation with two key principles of leveraging frozen\nvision-only models and exploiting the discrete nature of text and linguistic knowledge.", "questions": "What is the motivation of the proposed method?\nWhat is the reason of the good performance? I think it is because the large training dataset.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experimental performance is good.", "weaknesses": "1. The novelty and contribution is limited. First, I prefer simple method. I believe that the simple method is more valuable for applications and research. However, simple method requires more deep analysis and insights. Unfortunately, this work only presents a simple method without any insights. The method is designed without motivation or explanation. This paper looks like a experiment report, rather than a research paper. For a good research paper, the authors need to tell new insights, rather than just propose a model and conduct some experiments.\n2. The comparison is unfair. The proposed method is trained on COCO Captions (Lin et al., 2014; Chen et al., 2015) and LAION-400M (Schuhmann et al., 2021) and the image encoder is pretrained on LVD-142M dataset, as shown in L764-L774. However, other methods are not trained on these large-scale dataset. Even trained on these large datasets, the performance improvement is limited and the proposed method even performs worse than previous methods on Pascal VOC dataset. For example, the proposed method SimZSS trained on LAION-400M achieves 48.6 on Pascal VOC, which is much lower than CLIP-DINOiser (Wysoczanska et al., 2023), CLIP-DIY (Wysoczanska et al., 2024), OVSegmentor (Xu et al., 2023a), OVDiff (Karazija et al., 2023), CLIPpy (Ranasinghe et al., 2023) and TCL (Cha et al., 2023).\n\nOverall, this work propose a simple method without motivation and experimental analysis for the working mechanism. The good performance is mainly derived from its large training dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ye1JsGLOmS", "reviewer": "Reviewer_88HB", "summary": "The paper introduces SimZSS, a straightforward yet effective framework for zero-shot semantic segmentation. This framework aims to align vision encoders, which exhibit spatial awareness, with textual descriptions. SimZSS leverages text concept representations to extract corresponding visual representations based on similarity metrics. It ensures cross-modality consistency by employing different objectives at the sample and concept levels. Notably, the method requires minimal hyperparameters and does not rely on the supervision of semantic masks. Additionally, SimZSS can be easily adapted to various backbones and datasets. Overall, the method achieves state-of-the-art results across standard zero-shot segmentation benchmarks.", "questions": "1. Since the visual encoder is frozen, the quality of the segmentation masks heavily depends on the encoder's ability to recognize spatial positions. Would the performance improve if the visual encoder were fully or partially fine-tuned?\n2. In an extreme scenario, how significantly would the test performance be affected if the class names in the training set and test set are entirely different?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to understand and follow.\n2. The method is straightforward, easy to implement, and can be readily adapted to various backbones and training datasets.\n3. The method can be trained without the supervision of semantic masks, reducing the burden of annotations.\n4. The motivation for proposing a concept-level objective is clear and more suitable compared to a contrastive objective in scenarios where concepts encode individual objects that are likely to occur multiple times within a batch.", "weaknesses": "1. It is unclear how the final segmentation masks are generated during inference. Is there a similarity threshold used to determine the class names to which visual tokens belong? If so, how does the performance vary with different threshold settings?\n2. There is a lack of analysis explaining why SimZSS outperforms other zero-shot semantic segmentation methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZWmt5eXbqH", "reviewer": "Reviewer_wnrD", "summary": "The paper introduces SimZSS, a simple framework that enables open-vocabulary zero-shot segmentation by leveraging pretrained vision-only models within a vision-language contrastive learning paradigm. Addressing the challenge of poor localization in dense tasks, SimZSS decouples visual representation learning from cross-modality alignment by utilizing frozen, spatially aware vision models like DINOv2 and focusing on training only the text encoder. The framework identifies local concepts within captions using linguistic cues and aligns them with corresponding visual concepts in images through similarity-based pooling.", "questions": "- Given that the predefined concept bank seems to impact performance, how does the model perform on datasets with a much broader set of concepts, such as OpenImagesV7, which covers over 5000 classes? Has the model been evaluated on classes outside of the concept bank, and if so, what are the results?\n\n- Have you evaluated the recognition capabilities of the trained model on other classification datasets beyond ImageNet? Including evaluations on a wider range of datasets could provide deeper insights into how the method affects recognition performance.\n\n-  How does your model compare with training-free approaches?\n\n- Could you discuss the potential trade-off between improved localization and any degradation in recognition performance? Understanding this trade-off would help assess the overall impact of your method on different tasks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The authors provide clear explanations of their methodology. I would like to highlight the quality of the figures and tables in a visually pleasing way that enhances the understanding of the content.\n\n- While the problem of localization in vision-language models is not novel, the proposed approach offers a novel perspective. By freezing the vision backbone and only training the text encoder, the authors leverage pretrained self-supervised models effectively, resulting in efficient training.\n\n- The method demonstrates state-of-the-art performance on multiple zero-shot segmentation benchmarks.", "weaknesses": "1. It is surprising that the main paper lacks essential details about the training data and the pretrained models used. Given that the paper is only 9 pages (below the 10-page limit), including this information in the main text is necessary.\n\n2. \n(a) One major limitation is the use of a predefined concept bamk. The authors claim that it does not impact the breadth of the concept the model can localize(sec. 4.4). However, in Table 4, removing PascalVOC classes from the concept bank decreases the performance. Therefore, it seems that the breath of the concept bank is a crucial component. Also, the authors could evaluate on datasets with much diverse set of concepts e.g. OpenImagesV7 (which covers more than 5000 classes), this way the breadth of the vocabulary learned by the model can effectively be assessed. \n\n(b) Moreover, to identify the recognition capabilities of the trained model, I would expect the authors to evaluate the performance of more classification datasets than just ImageNet. More specifically, it is common practice I think to evaluate on the set of 38 classification datasets (https://github.com/LAION-AI/CLIP_benchmark). Also, table 3 should include the performance of at least the OpenCLIP-ViT/16 trained on LAION-400M as the training dataset is the same, whose zero-shot accuracy on ImageNet is 67.05 vs 64.1 for your ViT-B/14 which indicates that your method hurts the overall recognition capability of the final model. This tradeoff between localization and recognition should be discussed in the paper.\n\n(c)  Since the concept bank is derived from the union of class names from the segmentation datasets used for evaluation, the framework may be biased towards these classes. This design choice limits the model's ability to generalize to a truly open vocabulary, as it may not effectively localize concepts outside the predefined bank.\n \n3. The paper misses important related work, particularly training-free approaches that tackle poor localization in VLMs from a different angle. For instance, methods like [a][b][c][d] achieve competitive performance without requiring additional training and can detect concepts not covered by a predefined concept bank. Including comparisons with such methods would provide a more comprehensive evaluation.\n\nReferences:\n[a] Lan, M., Chen, C., Ke, Y., Wang, X., Feng, L. and Zhang, W., 2024. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. arXiv preprint arXiv:2408.04883.\n\n[b] Li, Y., Wang, H., Duan, Y. and Li, X., 2023. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653.\n\n[c] Bousselham, W., Petersen, F., Ferrari, V. and Kuehne, H., 2024. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3828-3837).\n\n[d] Zhou, C., Loy, C.C. and Dai, B., 2022, October. Extract free dense labels from clip. In European Conference on Computer Vision (pp. 696-712). Cham: Springer Nature Switzerland.\n\n[e] Benenson, R. and Ferrari, V., 2022. From colouring-in to pointillism: revisiting semantic segmentation supervision. arXiv preprint arXiv:2210.14142.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sY5HlmteEb", "reviewer": "Reviewer_enbH", "summary": "The work proposed SimZSS, a simple and efficient framework for open-vocabulary zero-shot semantic segmentation. The work builds on top of a frozen pre-trained vision-only model and aligns a text encoder to achieve cross-modality concept-level alignment. Despite its simplicity, SimZSS achieves state-of-the-art results on seven out of eight benchmarks.", "questions": "- What are the current limitations of the concept bank used? Would a larger bank lead to better generalization? What are the complexities in scaling the concept bank to, e.g., WordNet or other taxonomies?\n- As mining concepts from the captions provide promising results, I wonder how well the approach would work on more noisy captions, e.g., from web-scale datasets. How sensitive is SimZSS to such noise? Are there any techniques that could be employed to improve the robustness of the generation of the concept bank?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a simple yet effective approach for the task that exploits pre-trained vision models.\n- The method is both data- and compute-efficient, as it does not require long training periods and needs captioned images for training.\n- The method achieves state-of-the-art results on seven benchmark datasets.\n- The method takes inspiration from LiT and extends it for the segmentation task while maintaining its classification capabilities.", "weaknesses": "- The work relies on a bank of pre-defined concepts to pre-process the captions and extract concepts to segment, which may represent a limitation in some situations. For example, the concept bank may not cover the diversity of potential objects in the image.\n- While performance is competitive when using ViT-B, scaling the model backbone to a ViT-L model does not improve performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)