Class Distribution-induced Attention Map for Open-vocabulary Semantic Segmentations
Authors: Dong Un Kang
Hayeon Kim
Se Young Chun
Venue: ICLR 2025 Poster (2025)
Keywords: Vision Language Model
Dense Localization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Refining the attention map of a vision-language model using the similarity of class distributions across patches improves open-vocabulary semantic segmentation without additional training.

Abstract: Open-vocabulary semantic segmentation is a challenging task that assigns seen or unseen class labels to individual pixels. While recent works with vision-language models (VLMs) have shown promising results in zero-shot semantic segmentation, they still struggle to accurately localize class-related objects. In this work, we argue that CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object. Then, we propose Class Distribution-induced Attention Map, dubbed CDAM, that is generated by the Jensen-Shannon divergence between class distributions of two patches that belong to the same (class) object. This CDAM can be used for open-vocabulary semantic segmentation by integrating it into the final layer of CLIP to enhance the capability to accurately localize desired classes. Our class distribution-induced attention scheme can easily work with multi-scale image patches as well as augmented text prompts for further enhancing attention maps. By exploiting class distribution, we also propose robust entropy-based background thresholding for the inference of semantic segmentation. Interestingly, the core idea of our proposed method does not conflict with other prior arts in zero-shot semantic segmentation, thus can be synergetically used together, yielding substantial improvements in performance across popular semantic segmentation benchmarks.

Published as a conference paper at ICLR 2025
CLASS DISTRIBUTION-INDUCHR ATTENTION A MAP FOR OPEN-VOCABUU
SEMANTIC SEGMENTATIONS
Dong Un Kang', Hayeon Kim', Se Young Chun 122 'Department of ECE, 'INMC & IPAI, Scoul National University fokrtnskfk23, khy5630, sychunJ ( @nuu. ac. kr
ABSTRACT
Open- vhen-vocabulary semantic segmentation is a challenging task that assigns seen unseen class labels to individual pixels. While recent works with vision-language models (VLMs) have shown promising results in zero-shot semantic segmentation, they still struggle to accurately localize class-related objects. In this work, we argue that CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object. Then, we propose Class Distribution-indueee Attention Map, dubbed CDAM, that is generated by the Jensen-Shanmer divergence between class distributions of two patches that belong to the same (class) object. This CDAM can be used for open-vocabulary semantic segmentation by integrating it into the final layer of CLIP to enhance the capability to accurately localize desired classes. Our class distribution-inducer attention scheme can easily work with multi-scale image patches as well as aug- mented text prompts for further enhancing attention maps. By exploiting class distribution, we also propose robust entropy-based background thresholding for the inference semantic segmentation. Interestingly, the core idea our proposed method does not conflict with other prior arts in zero-shot semantic segmentation, thus can be synergetically used together, yielding substantial improvements performance across popular semantic segmentation benchmarks. Code is available at httpeeyeowoogituu.oooooocooaaaae
INTRODUCTION
Open-vocabulary semantic segmentation aims to assign correct semantic labels in an open set of classes to each pixel of given image. Classical semantic segmentation that assigns labels in closed set of pre-defined classes and is trained in a supervised manner has achieved remarkable
progress (Long al. 2015 Noh et 2013 Chen et al. 2017 2018 Xie et al. 2021 Yuan et al.. 2020 Zhao et al. (017b). However, the limited number of classes and laborious pixel- level human
annotation have restricted the model's ability to recognize numerous seen and unseen classes in real-world settings. Open-vocabulary semantic segmentation is emerging as promising approach for real-world applications since allows the segmentation model to assign novel class labels at
inference (Ghiasi et al., [022 Ding et al. [2022 Xian et al. 2019:; Bucher et al. 2019 Gu et al. 2020
Li et al. 2021 Liu et al. 2022: Zhao et al. 7a Xu et al. 2022b; Zhou et al.) 2022] Cha et al.
2023
Xu al. 2022a Luo et al. 2023 Shin al. 2022
Ren al. 2023 Ranasinghe et al. 2023
Recent advances of vision-language models (VLMs) such as ALIGN (Jia et al. 2021) and CLIP RRad
ford et al. 2021) have shed light the problem of zero-shot open- vocabulary semantic segmentation for novel classes. Prior arts aimed to enhance the localization capabilities of pre-trained CLIPP models for achieving great performance by 1) enhancing local alignment of VL.Ms between region visual
and textual features with contrastive learning (Xu et al., 2022a [uo et al.) 2023 Cha et al., 2023). 2) modifying the last attention layer of CLIP without retraining (Zhou et al. 2022). or 3) leveraging
self-self attention mechanisms such as query-query or key-key feature interactions within the attention
map (Li al. 2023 Bousselham et al. 2024 Wang et al. 2023 Lan al. 2024). However, despite these ad udvancement,, prior arts still struggle with accurately localizing target objects within images.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
In this work, we argue that CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object. Then, based on these observations, propose Class cetttributiiniinnuuee Attention Map, dubbed CDAM, that is generated by the Jensen-Shannon (JS) divergence between class distributions of two patches that belong to the same (class) object for the last attention layer of CLIPP. Specifically, starting from the noisy class predictions, we measure the similarity of class distributions between each patch and all other patches in the image. Since patches belonging to the same object class exhibit highly correlated class distributions, these similarities are high for same class patches and low for different class patches. By exploiting this property, these similarity scores are used to construct an attention map that refines the attention mechanism in CLIP's final layer, ensuring that attention is focused more effectively on relevant regions. Since attention map implies the significance of relevant features across different patches in the self-attention mechanism of vision transformer (ViT) (Dosovitskiy et al. 2020), our CDAM assigns high attention weights to the patches belonging to the same object class while allocating low attention weights to the patches from different object classes. This CDAM can be used for zero-shot semantic segmentation by integrating the semantic information of class-relevant patches into the final layer of CLIP to enhance the capability to accurately localize desired classes without requiring additional training or dense annotations.
Moreover, our CDAM can easily work with multi-scale image patches, augmented text prompts and entropy-based background thresholding for further enhancing semantic segmentation. CDAM with multi-scale image patches generates multiple CDAMs various spatial scales and merges them to achieve improved spatial consistency of attention maps. CDAM with augmented text prompts such as attribute classes of common objects (e.g., color and super-category) can strengthen the class distribution similarity between patches belonging to the same target class by leveraging a wider range of features to expand text class categories for enhancing attention maps. Lastly, by exploiting class distribution, we propose robust entropy-based background thresholding technique to effectively extract foreground classes from background for the inference of semantic segmentation.
Interestingly, the core idea of our proposed method does not conflict with other prior arts in zero- shot semantic segmentation and thus can be synergetically and scamlessly integrated into them for further enhanced performance. Our proposed CDAM substantially outperformed prior arts in zero-shot average mloU (mean Intersection-over Unioo) across several widely used benchmarks. The contributions of our work are summarized as:
Proposing class distribution- indlucedd attention map (CDAM) that yields higher weights to class-relevant patches to enhance localization capability by exploiting robust class distribu- tion over noisy class prediction for the patches of each object class.
Proposing CDAM with multi-scale image patches, augmented text prompts, and entropy- based background thresholding for further improving the CDAM.
Demonstrating that our CDAM remarkably outperformed prior arts on CLIP-based training- free zero-shot semantic segmentation over diverse benchmark datasets.
RELATED WORKS
2.1 OPEN- OPEN-VOCABULARY SEMANTIC SEGMENTATION WITH VISION- MODEL
Due to the recent success of large-scale vision-language models, researchers have shown a growing interest in open-voayy semantic segmentation. Prior works can be broadly categorized into two approaches: First approaches leverage annotated datasets containing examples of the seen classes Ghiass et al. 2022 Xian et al. 2019 Bucher et al. 2019 Gu et al.) 2020 Li al. 2021] Liu et al. 2022, Xu et al. 20226 2023b Jiao et al. 2023 Ge et al., 2025), and evaluates performance
using m for both seen and unseen classes. While some recent works (Xu al.) 2023b] Jiao et al. 2023 Ge et al. 2025) also utilize frozen CLIP features, they differ from our approach as they rely on supervised training. Specifically, they train additional mask proposal networks for object segmentation and to improve the classification of generated mask proposals. Second approaches attempt segmentation without any class- saass-s annotations and measures mloU exclusively for unseen classes. These approaches train the vision-language models with image-text paired datasets based on weak supervision. Group ViT (Xu et al. 2022a) constructs the hierarchical grouping structure of transformer for localing the image regions. TCL. (Cha et al. 2023) addresses suffering the train-test.
2

---

Published as a conference paper at ICLR 2025
Segmentattio result
Frozen Model
Existing Method
Input
Proposed Method
Entropy-baaee based tresholding
Class Distribution-induee Attention Map
f
Similanit
Atthcaam
frids
Atsention Layer
Unasifrate
Jassen Shatnon
Dovergence
Min
2a-1
sullrss
Attr
111
CDAM ganeration
Taggst Clases
Augnested Prompts
Similezit map
al jo
irame
(0-1)
all
Text Encoder
Image Fieender
NN
Example class distribution
Class distribution for each patch
CDAM for specific patch
Class Text
ATP
Image
CDAM (ours)
(b) CDAM generation
Figure 1: The overall pipeline of our proposed CDAM. During inference, the class distribution- induced attention map (CDAM) is constructed by measuring the distance between the class distribu- tions of each patch in the initial similarity map S. The CDAM is then integrated with the last attention layer of CLIP, highlighting the class-specific regions in the input image. CDAM with multi-scale image patches and augmented text prompts can further enhance the quality of attention map. Next, regions based on the entropy.
we dynamically adjust the threshold value for guegrrund-backgeen
discrepancy for region-text alignment through training grounded mask decoder. SegCLIP (Luo et al.
2023) proposed semantic group module with several weakly-supervisee losses. Recent training- methods, like CLIPSurgery (Li et al.) 2023), SCLIIP (Wang et al. (023) and GEM (Bousselham et al.) 2024). use self-self attention mechanisms, specifically query-query, key-key or value-value attention, to capture similar characteristic patches in the attention map. In contrast, our CDAM generates an attention map based on image-text feature similarity from class distribution. Consequently, our CDAM originates from the initial noisy predictions of existing methods, and our approach can be easily integrated into other training- ffree methods enhance their localization capabilities.
2.2 BACKGROUND SUBTRACTION
In image processing, separating foreground objects from the background is crucial for many applica- tions. Thresholding provides simple and effective technique to achieve this by classifying pixels with intensity values below chosen threshold background and those above as foreground. In
grayscale images, various thresholding approaches are categorized based on the type of information they use, including: histogram shape information (Rosenfeld & De La Torre 1983), measurement space clustering (Otsu et al.) 1975 Sezan 1990) Olivo 1994), histogram entropy information (Li &
Lee 1993 Pal 1996), image attribute information (Tsai 1985). spatial information (Pal & Pal 1989)
and local characteristics (Sauvola & Pietikäinen 2000). However, in open-vocabulary semantic
segmentation, the background class is considered "unknown" and distinct from the foreground classes with specific labels. This makes background subtraction more challenging compared to traditional image processing methods. this paper, we dynamically adjust the threshold value for discrimination of the background region considering the entropy of the class distribution.
METHODS
The key contribution of our proposed method is to enhance the localization ability of large-scale rision-language models for open-vocabulat semantic segmentation without additional training and annotations. Starting with initial, potentially inaccurate prediction, introduce the Class crtpbbuttoo-innuuee Attention Map (CDAM), an approach that emphasizes the attention weight on patch regions relevant to specific classes within the attention map of last attention layer. Additionally, we propose a entropy-based background thresholding technique that adaptively distinguishes between foreground and background regions. The overall pipeline of CDAM is illustrated in Fig.
3

---

Published as a conference paper at ICLR 2025
P3
Noisy Class prediction
Highly Correlated Class Distribution
P3
P1
gmax sofa
Sp1
Pys(Sr Prs(Sr111Sp2) 0. 0.0153
Same class!
P2
P2
aT gmaaax chair
Sp2
ilat slaltto
Dys(Sr [1sp)) 0.0576
Different class!
P3
gmax pa) sofa
Sy3
PssSsa11Sss)) 0621 Different class!
Ground Truth Sofa P1.P2.PS: Patches
MaskCLIP
chair
sofn
Figure 2: Similarity of class distributions between patches. From the noisy prediction of MaskCLIP (Zhou et al., (2022). we explore the similarity of class distributions between patches. Sp, represents the class distribution the position of patch Pi within the similarity map S. Although the segmented classes differ, the similarity of the class distribution of patches between true positive (P1) and false negative (P2) is more similar than between true positive (P1) and false positive (P3). The distance of class distribution is measured by JS divergence, DJs (pll4).
3.1 LIMITATION OF SEMANTIC SEGMENTATION WITH VISION-LANGIUS MODEL
CLIP (Radford et al, 202T> is a pre-trained VLM with 400 million curated image-text paired dataset. In the self-attention mechanism of its wansformer-basee image encoder, the attention map reflects the relationships among visual tokens. The attention weight is computed using the similarity between pairs of query and key embeddings within each attention layer. Taking the flatten feature map æ E IR xD where N denotes the number of tokens and D refers to the embedding dimension, the attention map is formulated as
Attn( Q, K) Softmax (QK?)QQK))) (VD)EEE
where the query and key embeddings, Q = xW and K = WW. are obtained using the projec- tion matrices Wg and Wa E RDDD respectively. Thus, the output of the self-attention will be (Q, KVV where V = wW and W, E DxD The attention map, the output Attn(::),, is crucial for capturing long-range semantic dependencies between patch tokens in image recognition. However, due to image-level texts for pre-training, CLIP is not usually applicable for dense prediction tasks like semantic segmentation where precise localization of target classes is essential.
To mitigate the limitation of CLIP, the attention map should be adjusted to weigh more on class- relevant region for each patch token. MaskCLIP (Zhou et al. 2022) extends the pre-trained CLIP model to perform dense predictions by minimally modifying the last layer of the image encoder. Specifically, the original attention map is replaced by the identity matrix I N removing the query and key embedding layers in the self-attention of the last layer and thus the output of the self-attention in the last layer is V. However, using the identity matrix the last attention map may overly emphasize the patch embedding itself and neglects information from class-relevant neighboring patches, thus leading inaccurate segmentation quality.
3.2 CLASS DISTRIBUTION-INDUEE ATTENTION MAP
Our approach aims enhance the quality of the attention map of the last layer of CLIP in Eq. for CLIP-based open- vecabuocyyl semantic segmentation methods.
3.2.1 HIGHLY CORRELATED CLASS DISTRIBUTIONS Vs. NOISY CLASS PREDICTIONS
The class distribution learned by the pre-trained model includes rich information about recognition patterns (Hinton et al. 2015). We conjecture that the pre-trained CLIP model implicitly captures knowledge about target classes through its class distribution. We chose prior art on CLIP based semantic segmentation, MaskCI.IP (Zhou et al.] 2022). and carefully observed the output of it for toy example with "sofa' by selecting two patches (PI, P2) that belong to it and one patch (P3) that does not as illustrated in Fig. 2 MaskCLIP yielded locally noisy 'sofa' output with the maximum likelihood class prediction per patch (P2, P3), but surprisingly also yielded highly correlated class distributions between the same class patches (P1, P2) and somewhat uncorrelated across the different class patches (P1, P3), thus confirming our conjecture. To support these observation, we use several based training-free methods baseline model and extend the analysis with benchmark datasets.
4

---

Published as a conference paper at ICLR 2025
For given image, one patch Piarget was randomly selected and then two patches P.n and Pout were randomly selected from the target class region and the rest of the region, respectively. Then, measure (1) the probability that class prediction in Ptorget is correct and (2) the probability that distribution similarity between Ptarget and Pin. is higher than distribution similarity between Ptarget and Pout The results show that while CLIIP-based baseline methods exhibit relatively low accuracy in class predictions for Parrget (e.g.. 67.0% and 70.8% on VOC21, and 33.6% and 37.5% on COCO-Obj for SCLIP (Wang et al. 2023) and GEM (Bousselham al. 2024). respectively), they perform significantly better in identifying class distribution similarity between patches of the same object class (e.g.. 78.9% and 79.4% on VOC21, and 75.4% and 74.2% on COCO-Obj) Obj). More detailed results are provided in the supplementary materials. These similarity and dissimilarity can be measured by the Jensen- -hannnnn (IS) divergence and thus can be incorporated potentially for more precise attention weights for the patches belonging to the same object.
3.2.2 CLASS DISTRIBUTION-INDUCCO ATTENTION MAP
Here we propose class intrrbuttioninduuce attention map, dubbed CDAM, that utilizes the class distributions from most CLIP-based semantic segmentation methods with dense predictions. Firstly, the dense visual features and text features are extracted for the input image 2 and the text prompts for each class name of the target objects 2T,els using the image and text encoders & and &T as fr (z)) E IR (N)xx and fr.el = ET(TT,els) E RCxd respectively, where and refer to the number of target classes and the projected output space dimension. respectively. Then, we proposed to measure the distance between class distributions using the divergence Dus( [l:), a finite and symmetric metric. Min-max max normalization was applied before and after softmax operation, but was omitted for simplicity. Thus, our class distribution-induees attention map is formulated as:
AtlnCDAM Softmax (11 DJs(S))) D)s((S(s"))//
where S = (pl fT) (N-1)xC is the similarity map between dense visual features and text features, denotes the cosine similarity and the temperature t controls the softness of attention such as vD in the original self-attention layer, Eq. 11. Our CDAM was designed to weigh more class-relevant patches with the source patch.
3.2.3 REFINEMENTS OF CLASS DISTRIBUTION- ATTENTION MAP
CDAM with multi-scale image patches. Our class distribution conjecture is valid over different space scales. Thus, we propose the multi-scale structure of CDAMs by constructing CDAMs with wwwnsampling-upsampiin at different scales in the set of scaling factors M and aggregating them as:
Attnus
AttncDAM,
M
ME
where AttDcDAM.. Upm [Softmax (11 DDsmDnmm [S] Dn... [ST)))/TT] and Upm.(1 and Dnm denote the upsampling and downsampling operations at the scale m, respectively. This multi-scale CDAM Atinms helped to refine CDAMs with enhanced spatial consistency.
CDAM with augmented text prompts. We propose to incorporate the names of attributes classes (e.g., yellow, fabric, striped) and super-category (e.g., animal, indoor, food) of common objects. dubbed augmented text prompts (ATP), to enrich the representation of implicit knowl- edge within class distribution. PACO (Ramanathan et al.] (023) provides 59 attribute classes for common objects. encompassing properties like color, pattern, material and transparency. Addi- tionally, COCO-Stuff (Caesar et al. 2018) and MSCOCO (Lin et al. (014)) offer 12 and 15 super-categories, respectively. We filtered out the overlapping classes and removed vague terms like 'others'. Then, we used 80 augmented text prompts, &T,TTP, and add them to the origi- nal target class text prompts. Therefore, we extract the text features of augmented text prompts fr [fT.atsi fT,ATP) = ( (TTT.ATT) not for inferencing with it. These additional classes in text prompts contribute to enhancing the similarity of class distributions among class-relevans patches.
(CC+80)xd only for constructing CDAM,
3.2.4 OPEN- VOCABULARY SEMANTIC SEGMENTATION WITH CDAM
The overall inference process is visualized in Fig. I First, we generate the initial similarity map from dense prediction of existing methods using CLIP, such as MaskCIIP (Zhou al. 2022 and
5

---

Published as a conference paper at ICLR 2025
SCLIP (Wang et al. )023). The CLIP consists of L attention layers. Second, construct the CDAM with multi-scale image patches Ast by measuring the similarity of class distribution within the initial similarity map S. Finally, we incopporate this localized attention map, A Attums, into the last attention layer of CLIP to compute the final similarity map S. We reuse the latent features from the 1th attention layer of CLIP, w(4-1), as value features. Note that augmented text prompts are not used for computing final similarity map S. The visualized examples of class distribution-induee attention maps are shown in the supplementary materials.
3.3 ENTROPY-BR BATREDY-BASED BACKGROUND THRESHOLDING
For segmenting the background class that excludes all target classes, a class probability thresholding approach has been commonly used in class prediction with the probability be for foreground patches and 0 for background patches with the default threshold value Thr = 0.5. In real cases, however, one must adjust the threshold value by scaling a in the optimal threshold a Thraatttt considering the uncertainty for belonging to the both foreground and background. However, it is challenging finding an optimal a for the whole image. Our proposed CDAM allows us exploit class distribution per patch by coormation-thhoredee measures.
Here we propose an entropy-based background thresholding method that dynamically adjusts thresh- old values. For the similarity map S, the entropy of (S)) = 1 Sogg where S is the probability of the äth class. Then, we can conjecture that the foreground that belongs to an object class yields low entropy due to high confidence while the background yield high entropy due to high uncertainty. Then, define the center entropy value H(S) that is the average of the maximum and minimum values of H (S) over all image patches. Thus, our entropy-based background thresholding is formulated
Threat-bg Thh HH(S) center
where Thremthn represents the entropy-based background threshold value and refers to hy- perparameter. Highly correlated class distribution also helps to determine a stable threshold over existing methods based noisy class prediction. We empirically verified the effectiveness of our entropy-based background thresholding by comparing with existing thresholding methods in image processing in the supplementary material.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate our CDAM method on the three widely used benchmark datasets that include a background class: PASCAL VOC (Everingham et al., 2010), PASCAL Context (Mottaghi et al. 2014) and COCO-Objer (Lin et al. 2014). All three datasets include a background class, which is separate from the foreground classes. These datasets have 20, 59, and 80 foreground classes, respectively. The validation sets contain 1 1449, 5105, and 5000 images, respectively. We also use three additional benchmark datasets that do not include background class: CityScapes (Cordts et al. 2016), ADE20K (Zhou et al. 2017). and COCO-Stuff (Lin et al. 2014). which have 19, 150, and I7I classes, respectively.
Unified evaluation protocol. We follow the unified evaluation protocol by TCL (Cha in vocabuloyy semantic segmentation. This protocol ensures no access to target data before evaluation. It prohibits sataset-specific hyperparameter tuning or tricks like expanding or rephrasing class names. We fixed the background thresholding hyperparameters across datasets. To ensure fair comparison, we reproduced existing CL.PP-based training-fres methods, including MIs (Zhou al., 2022), SCLIP (Wang et al.) 2023), CaR (Sun et al. 2024), GEM (Bousselham et al.) (024) and Clear (Lan et al. 2024), following the unified protocol and eliminating renaming tricks. The reproduction details are described in supplementary material.
2023
Implementation details. In our CDAM model, we utilize the CLIP ViT/B- model from Open- CLIP (Radford al. 2021). trained on the LAION dataset (Schuhmann et al. 2022). The input image is resized to 224 x 224 pixels, and the patch size is set to 16 x 16 pixels. Following the experimental settings of GroupViT (Xu et al. 2022a), resize input images to have the shorter

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with state-of-the-art methods on benchmark datasets with background
class. We evaluate the open-vocabular semantic segmentation methods on VOC21 (Everingham al., 2010p, Context60 (Mottaghi et al.) 2014) and COCO-Obj (Lin al. 2014). SD stands for Stable Diffusion (Rombach et al., 2022) and we marked f for the reproduced methods by following the unified evaluation protocol (Cha et al. 2023) and removing renaming tricks. For each dataset, we highlighted the best performance in bold and underlined the second-best performance. Performance improvements by CDAM are indicated parentheses. The evaluation is based on mloU (%).
Method
Pre-trained
Extra Training
VOC21
Contex160
COCO-Obi
Avg.
Model
weakdy-supert supervised methods with additional training dataser
GTT Xu et 2022a
Scratch
50.4
18.7
27.5
32.2
CLIPpy Ranasinghe et
2023
CLIP
52.2
32.0
ViewCo Ren al. 2023
Scratch
52.4
23.0
23.5
33.0
SegCLIP (Lao
2015
CLIP
52.6
24.7
26.5
34.6
OVsegmentor [Xu
2023g
DINO
53.8
20.4
25.1
33.1
TCL Cha
2023
CLIP
51.2
24.3
30.4
35.3
PACL Mukholi
2023
CLIP
72.3
50.1
Visaal prototype géneration methods for each concept
CLIP
OVDiff Karazija
2023
+SD+DINO
67.1
34.8
44.0
CLIP
FreeDA Barsellotti et al. 2024
+SD+DINO
X
55.4
38.3
37.4
43.7
CLiP-bused training-fi free methods
CLIPSurgery Li 2023
CLIP
X
29.3
CLIP-DIY (Wysczzaiska at.
2024
CLIP+DINO
X
59.0
30.4
CaR Sun 2024
CLIP
59.4
25.0
33.2
39.2
MaskCLIP Zhou 2022
CLIP
33.1
23.3
24.8
27.1
MaskCLIPP+DDM
(+22.8)
(+7.2)
34.3 (49.5)
(+13.1)
SCLIPI WWag et al.
2023
CLIP
50.5
31.3
35.9
SCLIP+CDAM
CLIP
(+8.5)
30.4 (+4.5)
34.5 (+3.0)
41.3 (+5.4)
ClearCLIP (Lan et al
2024
CLIP
X 50,7
27.8
33.0
37.2
CleaCLIP+CDAM
CLIP
X
57.6 (+6.9)
29.8 (+2.0)
(+1.5)
40.6 (+3.4)
GEM1 (Bousselham et al.
2024
CLIP
X
52.1
28.1
33.8
38.0
GEM+CDAM
CLIP
X
58.7 (46.6)
30.6 (+2.5)
35.2 (.1)4)
41.5 (+3.5)
side of 448 pixels and employ the Intersection-ovee--niee (mloU) metric, which is generally used for evaluating semantic segmentation performance. To ensure fair comparison, Fixel-Adapte Mask Refinement (PAMR) (Araslanov & Roth 2020) post-processing were not applied any of the evaluated methods. The temperature 7 and the modulation of entropy are set to 0.1 and 2.5, respectively. The set of scaling factor M {0.25, 0.37, 0.5, 0.63, 0.75, 0.87, 1.0].
4.2 COMPARISON WITH STATE-OF-THE ART METHODS
Datasets with background class. As shown in the Table 1 we compare our CDAM with existing apen-vocabulary semantic segmentation methods on VOC21, Context60 and COCO-Obj, including background class. First, superiy-uperriiedd methods requires large-scale image-text paired training datasets such as CC12M (Changpinyo et al.) 2021) and YFCC (Thomee et al. 2016). Additionally. the results of specific methods, including OV Oegementor (Xuetal, 2023a), SegCLiP (Luo al.) 2023). and
ViewCo (Ren et al.) 2023). were obtained by tuning dataset-specific hyperparameters for background thresholding, significantly improving the segmentation performance. contrast, our CDAM enhances the localization ability of existing methods using a pre-trained CLIP model during inference and adaptively adjusts the thresholding value. As result, our method, even when incorporated with MaskCLiP (Zhou et (022),, outperforms weakly-supervin except for PACL (Mukhoti et al. 2023). Second, we compare our proposed method with training- free methods. Our CDAM effectively enhances the localization ability of existing methods, significantly achieving a performance improvement over VOC21, Contex160, and COCO-Obj. Using a single model CLIP, our method with GEM (Bousselham et al. 2024) performs better than visual prototype generation methods that use three sarge-scale foundation models, specifically OVDiff (Karazija et al. 2023) Context60 and COCO-Obj, and FreeDA (Barsellotti et al.) 2024) VOC21. While CaR (Sun et al. 2024) requires high computational costs and CLIP-DIY Wysoczanska et al. 2024) employs additional pre-trained background extractor, FOUND (Siméoni al. 2023).
methods on all benchmark datasets
surpass
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison with state-of-the-art methods on benchmark datasets without background
class. We evaluate the vocabbocyy semantic segmentation methods on COCO-Stuff (Lin et al. 2014). CityScapes (Cordts et al. 2016) and ADE20K (Zhou et al. 2017). We marked for the reproduced methods. For each dataset, we highlighted the best performance in bold and underlined second-best performance, Performance improvements by CDAM are indicated parentheses.
The evaluation is based on mloU (%).
Pre-trained Model
Extra Training
Method
COCO-Stf
CityScapes
ADE20K
Avg.
weally-superveee methods with additional training dataset
GiT (uu et al. 2022a
Scratch
15.3
11.1
9.2
11.9
CLIPpy Raaasinnhe
2023
CLIP
13.5
ScgCLIP (Luo et al. 2023
CLIP
8.7
TCL
Cha et al. 2023
CLIP
19.6
23.1
14.9
19.2
PACL (Mukhoti al.
2023
CLIP
38.8
31.4
Visual protolype generolion methods for each concept
CLIP
FreeDA Barsellotti al. 2024
+SD+DINO
27.8
36.7
22.4
29.0
CLIP based training-rree methods
MaskCLIPI Zhou al.
2022
CLIP
16.5
23.8
12.2
17.5
MastCLIPPCDDM
CLIP
24.5 (+8.0)
27.6 (+3.8)
17.8 (+5.6)
23.3 (+5.8)
SCLIPi (Wang al. 2023
CLIP
X
21.1
19.7
14.6
18.5
SCLIP+CDAM
CLIP
24.5 (+3.4)
24.6 (44.9)
(42.6)
22.1 (+3.6)
ClearCLIPI (Lan et 2024
23,9
16.6
20.4
ClcarCLI+CCDAM
CLIP
X
24.6 (+0.7)
21.7 (+0.9)
17.1 (+0.5)
21.1 (+0.7)
GEM! Bousselham et al. 2024
CLIP
23.7
21.2
15.7
20.2
GEM+CDAM
CLIP
24.8 (+1.1)
23.7(+1.5)
17.2 (41.5)
21.9 (+1.7)
them in averaged zero-shot performance without additional pre-trained models and with minimal computational costs. Notably, prior works such as CaR (Sun et al. (2024) or CLIP-DIY (Wysoczanski al. 2024) are structurally incompatible with our CDAM, as they classify regions using CLS tokens, whereas CDAM relies on local visual tokens.
Datasets without background class. We further evaluate the effectiveness of our CDAM by incor- porating it with existing methods on benchmark datasets that do not include a background class. Entropy-based background thresholding, a component of CDAM, was originally designed to handle real-world scenarios with background classes. To evaluate performance without backgrounds, we disable the entropy-based background thresholding in CDAM. Table we demonstrated that our CDAM synergetically and consistently enhance the performance of existing CLIP-basee training-fre free methods. Notably, MaskCLIP (Zhou et al.) 2022) with CDAM outperformed weakly-supervised and training-fres free methods, except for PACL (Mukhoti et al. 2023) that requires substantially large-scale training datasets, and FreeDA (Barsellotti
al., (024) that employs large foundation models. Without the background class, evaluation focuses solely target classes, minimizing the impact of background prediction errors. Our method reduces false positives in both target and background re- gions, with entropy- based thresholding improving results when the background is evaluated. However, these gains are less impactful, observed smaller improvements in Table compared to Table
4.3 ABLATION STUDY AND ANALYSIS
To validate the importance of each component in our CDAM model, we conducted ablation studies. We employed MaskCI.IP (Zhou et al. (002). SCLIP (Wang al. 2023). ClearCLIP (Lan al.
2024). and GEM (Bousselham et al. 2024) as baseline models.
Class iitrributioninndueed attention map (AttnCDAM and Attons). We first investigate the effec- tiveness of a class distribution-induced attention map for open- veccbvocayulary semantic segmentation. Applying our proposed AttnCdAM significantly improves performance across all baseline models. Notably, AttnCDAM achieves this improvement without any additional training and annotations. More- over, considering the spatial consistency based on CDAM with multi-scau image patches, Attnus. leads to substantial performance boost.
8

---

Published as a conference paper at ICLR 2025
Table Ablation study on components of our CDAM. The proposed components of our CDAM consistently improve the semantic segmentation performance on VOC21 (Everingham et al. 2010
dataset. ATP refers to the augmented text prompts. The evaluation based on ml))(%))
Bascline
AtlDCADMM
Attnms
ATP
Thraat-
MaskCLIP
SCLIP
ClcarCLIP
GEM
33.1
50.5
50.7
52.1
50.1
55.0
52.1
54.7
/
53.7
56.9
55.8
56.5
/
54.7
57.2
56.0
56.9
/
55.9
59.0
57.6
58.7
Augmented text prompts (ATP). Next, we leverage the 80 augmented text prompts to enrich the representation of the class distribution, leading to consistent performance improvement. Incorporating augmented text prompts leads to a higher similarity between the class distributions of class-relevant patches and enhances the localized class distribution-induced attention map. This suggests that our CDAM can effectively integrate the text information of the target class and additional context into the attention process.
Entropy-based background thresholding (Thrent-bg). Finally, we apply the entropy- back- ground thresholding which adjusts threshold based the entropy. Without dataset-specifie hyper- parameter tuning, Threat significantly improves the performance with several baseline methods. suggests that even though Threat-b is developed through the empirical experiments, affects performance improvement over all baseline models.
To analyze (I) inference time computation cost and (2) the metries for measuring similarity of class distribution for our CDAM, we conducted additional experiments. The detailed results and analysis are provided in the supplementary materials.
Inference time. As shown in Table and our CDAM introduces minimal computational overhead, with an increase of at most 34 ms, which remains suitable for real-time applications. Notably, CDAM achieves approximately 200 times faster inference compared to CaR (Sun et al. 2024) on the COCO- Obj dataset. Moreover, the majority the increased inference time is attributed to the Attn and ATP modules, primarily due to the computational burden of calculating JS divergence.
Correlation metric. The JS divergence suitable for constructing our CDAM due its symmetry and bounded range, providing better robustness in generating attention maps compared to metrics such as KL divergence (asymmetric and unbounded) and Wasserstein distance (symmetric but unbounded). Empirically, Table demonttratess that JS divergence achieved the best average performance across several baseline methods. For these reasons, JS divergence was selected for our CDAM.
4.4 QUALITATIVE RESULTS
Effect of CDAM components. In Fig. we qualitatively visualize examples of segmentation results by adding components of CDAM. Without relying on annotations or training, our proposed (AltncDAM) can effectively reduce the noisy class prediction of baseline model (MaskCLIP (Zhou et al. 2022)) and localize the target classes within an image. The quality of the segmentation results further enhanced by two factors: (I) improved spatial consistency achieved through CDAM with multi-scale image patches, and (2) enhanced similarity of class distributions between class-relevant patches by incorporating the augmented text prompts (ATP).
Qualitative segmentation results. As shown in Fig. 4 we qualitatively demonstrate the improvements localizing the target object using the proposed CDAM with Attims- Compared to the initial predictions from existing CLIP--bsed training- free methods, integrating approach reduces noisy predictions, resulting in smoother and more accurate outcomes. Remarkably, as we observed in Fig. 2 our method is able to generate clean and accurate attention maps even from noisy predictions. Our generated CDAM effectively highlights patches within same objects class like trains and sheep, significantly enhancing segmentation quality by making predictions more precise and less noisy. Notably, we observed that our attention map can capture highly detailed features. For example, it can distinguish elements such the train's doors or the fence in front of the sheep in the input images.

---

Published as a conference paper at ICLR 2025
Image
Ground Truth
Bascline
w/ ADMM
gt
CDAM (Ours)
aeroplane
car
grass
motorbike
A
teddy
bear
keyboard
Figure 3: Qualitative effectiveness of CDAM components. Each component of CDAM consistently improves the quality of the semantic segmentation, leading less noisy predictions and improved localization accuracy. We use MaskCLIP (Zhou et al. 2022) as baseline model.
MaskCLIP
MaskCLIP CDAM
MaskCLIP
MaskCLIP
CDAM
Input image
SCLIP
SCLIP CDAM
Iaput image
SCLIP
SCLIP CDAM
ClearCLIP
ClearCIIP
CleaCLIP
CleaCCIIP
GEM
GEM CDAM
GEM
GEM CDAM
Figure 4: Qualitative segmentation results CDAM from inaccurate initial predictions. Our proposed CDAM demonstrated its ability to generate high-quality attention maps (Attmys) even when starting from inaccurate predictions provided by prior methods. This capability led to significantly reduced noise in the final predictions of CDAM. Notably, CDAM captures fine-grained details present within images, such as doors train and fence in front of sheep.
CONCLUSION
This paper proposes CDAM, training-free approach for open-vocabulary semantic segmentation. We exploit the observation of highly correlated class distribution between class-relevant patches, even when the object within an image are inaccurately segmented. Based on this, we construct a localized, class distribution-induced attention map during inference. This map allocates higher attention weights to patches likely to belong to the same object class. Furthermore, the proposed entropy-based background thresholding adjusts the threshold value dynamically, thereby improving discrimination for foreground and background patches. CDAM can be seamlessly integrated into other training-ffe free methods using CLIP, and we demonstrate its effectiveness through evaluations on benchmark datasets, achieving substantial improvements in semantic segmentation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by Institute of Information & communications Technology Plan- ning Evaluation (IITP) grant funded by the Korea govemmen((MSTT) INO.RS-2021-121
Artificial Intelligence Graduate School Program (Seoul National University)l. ). the National Re- search Foundation of Korea(NRF) grant funded by the Korea govemmeen(MSSIT) (No. NRF- (022M3C1A302202111 AI-Bio Research Grant through Seoul National University and Samsung Electronics Co., Ltd (IO22100K) 1004-02674-01). Also, the authors acknowledged the financial support from the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Scoul National University.
REFERENCES
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4253 4263,42622200 2020.
Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Training- free open-vocabulary segmentation with offline diffusion-augmentte prototype generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 3689-3698 2024.
Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuchne. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1828-3837, 2024.
Anton D Brink and Neil E Pendock. Minimum cross-entropy threshold selection. Pattern Recognition, 29(1):179-188, 1996.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes context. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 1209-1218, 2018
Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate -xo-ngr mask for open-world world semantic segmentation from only image-text pairs. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11165-11174, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-33 3566,, 2021.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-88) 848, 2017.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings the European Conference on Computer Vision, pp. 801-818, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 3213-3223, 2016.
Jian Ding, Nan Xue, Gui-Song Xia. and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11583-11553 1592, 2022.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 303-338, 2010.
Jiannan Ge, Lingxi Xie. Hongtao Xie, Pandeng Li, Xiaopeng Zhang, Yongdong Zhang, and Qi Tian. Alignzeg: Mitigating objective misalignment for zero-shot semantic segmentation. In Proceedings of the European Conference on Computer Vision, pp. 142-161. Springer, 2025.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open vecabuocyyulary image segmentation with image-level labels. In Proceedings of the European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 1921-1929, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint wrXiv:1503.0253: 2015.
Liang-Kai Huang and Mao-Jiun J Wang. Image thresholding by minimizing the measures of fuzziness. Puttern Recognition, 28(1):41-,,, 1995.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and iision-languse representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. Advances in Neural Information Processing Systems, 36:35631-35633 2023.
Jagat Narain Kapur, Prasanna K Sahoo, and Andrew KC Wong. A new method for gray-level picture thresholding using the entropy of the histogram. Computer Vision, Graphics, and Image Processing, 29(3):273-28) 1985.
Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot vocabulayy segmentation. arXiv preprint Stti::2300.0931166 2023.
Kwanyoung Kim, Yujin Oh, and Jong Chul Ye. Otseg: Multi-prompt sinkhorn attention for zero-shot semantic segmentation. In Proceedings of the European Conference on Computer Vision, pp. 200-217. Springer, 2024.
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vasion-language inference. arXiv preprint 1xXiv:2407.12442, 2024.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2021.
Chun Hung Li and CK Lee. Minimum cross entropy thresholding. Pattern Recognition, 26(4): 617-625, 1993.
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.056633 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference Computer Vision, pp. 740-755. Springer, 2014.
12

---

Published as a conference paper at ICLR 2025
Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open Open-world world semantic segmentation via contrasting and clustering -aiiual embedding. Proceedings of the European Conference Computer Vision, pp.
Springer, 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3431-3440, 2015.
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In International Conference on Machine Learning, pp. 23033 23044. PMLR, 2023.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 891-898, 2014.
Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
19413-19423, 2023.
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1520-1528, 2015.
Jean-Christophe Olivo. Automatic threshold selection using the wavelet transform. CVGIP: Graphical Models and Image Processing. 56(3):205-218, 1994.
Nobuyuki Otsu et A threshold selection method from gray-level histograms. Automatica, 11 (285-296):23-222 27, 1975.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. Advances Neural Information Processing Systems, 32, 2019.
Nikhil R Pal. On minimum cross-entropy thresholding. Pattern Recognition, 29(4):575-580 1996.
Nikhil R Pal and Sankar K Pal. Entropic thresholding. Signal processing, 16(2):97-108, 1989.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7141-7151, 2023.
Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. 2023 ieee. In Proceedings of the IEEE/CVF International Conference on Computer Vision, volume 1, pp. 3, 2023.
Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency, arXiv preprint arXiv:2302. 10307, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Azriel Rosenfeld and Pilar De La Tore. Histogram concavity analysis as an aid threshold selection. IEEE Transactions Systems, Man, and Cybernetics, (2)2231-235, 1983.
13

---

Published as a conference paper at ICLR 2025
Jaakko Sauvola and Matti Pietikäinen. Adaptive document image binarization. Pattern Recognition, 33(2):225-23 236, 2000.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25295 2022.
M Ibrahim Sezan. A peak detection algorithm and its application to histogram- based image data reduction. Computer Vision, Graphics, and Image Processing, 49(1):36-55 1990.
Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Retrieve and co-segment for zero-shot transfer. Advances in Neural Information Processing Systems, 35:33754-33767, 2022.
Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín Vobecký, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Observing the background to discover objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3176-3186, 2023.
Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip as mn: Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13171-13182 2024.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data multimedia research. Communications of the ACM, 59(2)644-73, 2016.
Wen-Hsiand Tsai. prreeeripr thresolding: A new approach. Computer Vision, Graphics and Image Processing, 29(3):377-393, 1985.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. arXiv preprint arXiv:2312.01597, 2023.
Monika Wysoczańska, Michaël Ramamonjisoa, Tomasz Trzciński, and Oriane Siméoni. Clip-diy: Clip dense inference yields open-vocabulary semantic segmentation for-free. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1403-1413, 2024.
Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schicle, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8256-8266 2019.
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 14717777700000, 2021.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 18134-18144, 2022a.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng. Yi Wang, Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2935-2944. 2023a.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-uocab semantic segmentation with pre- preitrained rision-languae model. In Proceedings of the European Conference on Computer Vision, pp. 736-753. Springer, 2022b.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2945-2955 2023b.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextur representations for semantic segmentation. In Proceedings of the European Conference on Computer Vision, pp. 173-190. Springer, 2020.
14

---

Published as a conference paper at ICLR 2025
Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2002-2010, 2017a.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2881-2890, 2017b.
Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 633-641, 2017.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In Proceedings of the European Conference on Computer Vision, pp. 696-712. Springer, 2022.
Ziqin Zhou, Yinjie Lei, Bowen Zhang, Linggiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11175-11185, 2023.
A DETAILS OF AUGMENTED TEXT PROMPTS
We utilize augmented text prompts, including attribute classes and supercategories of common objects. From PACO (Ramanathan al. 2023), we obtained the 59 attribute classes, which consist of color (31 classes), pattern (8 classes), material (16 classes), and transparency (4 classes). For general text prompts, we filtered out the four vague classes: "others(color)". "others(pattern marking)" "others(material)" and "others(transpaeenc))
Next, from COCO-Stufi (Caesar et al., 2018) and MSCOCO (Lin al. 2014). we obtain the and 15 supercategories, respectively. We remove the overlapped class "furniture" and "food". Thus, we utilize the 80 augmented text prompts (55 attribute classes and 25 supercategories) and the list of augmented text prompts is described in Table.
Table List of augmented text prompts. We list the augmented text prompts used in CDAM, including attribute classes and supercategories.
Source
List class names
black, light blue, blue, dark blue, light brown, brown, dark brown, light green, green, dark green, light grey, grey, dark grey, light orange, orange, dark orange, light pink, pink. dark pink, light purple, purple, durk purple, light red, red, dark red. white. light yellow, yellow, dark yellow, plain, striped, dotted, checkered, woven. studded, perforated, floral, logo, text, stone, wood, rattan, fabric, crochet, wool, leather, velvet, metal, paper, plastic, glass, ceramic, opaque. translucent, transpurent person, vehicle, outdoor, animal, accessory. sports, kitchen, food, furniture, clec- tronic, appliance, indoor, water, ground, solid, sky, structural, building, textile, window, floor, ceiling, wall, rawmaterial, plant
Attribute class
PACO
Supercategory
COCO-Stuff
MSCOCO
B DETAILS OF COMPUTATIONAL COSTS
Analysis of inference time for CDAM. We analyzed the computational complexity of our CDAM by
measuring its inference time (in seconds per image). Specifically, we conducted two experiments: (1) a comparison of inference time with several baseline methods in Table and (2) an analysis of the
inference time for each component of our CDAM in Table 6 All measurements were performed on an NVIDIA A 110 GPU. Note that, due to the shared GPU server environment, the reported inference costs may be higher than those observed on a dedicated local GPU setup.
As shown in Table 5] our proposed CDAM introduces increase in inference time, approximately 0.026 seconds per image for VOC21, 0.032 seconds per image for Context60, and 0.034 seconds per image for COCO-Obj compared to baseline methods. While CDAM requires additional computational cost, it remains feasible for real-tife timi applications. For example, the inference time of our method (e.g., 0.051 seconds for COCO-Obj with MaskCI.IP (Zhou et al. 2022) + CDAM) is significantly
15

---

Published as a conference paper at ICLR 2025
lower than other methods like CaR (Sun al., (024) (12.270 seconds for COCO-Obj) and CLIP- DIY (Wysoezańska et al. 2024) (0.559 seconds for COCO-Obj). These results demonstrate that our approach achieves balance between computational efficiency and enhanced functionalityy.
Table 5: Inference time comparison (in seconds per image) of baseline methods with our CDAM. Despite introducing minimal computational overhead, CDAM remains feasible for real- time applications, especially when compared to computationally intensive methods like CaR and CLIP-DIY.
Methods
VOC21
Context60
COCO-Obj
CaR (Sun et al. 2024)
3.497
9.340
12.270
CLIP-DIY Wysoczanska et al.
2024
0.520
0,559
MaskCLIP Zhou al. 2022)
0.017
0.017
0.017
MaskCLIP+
0.043 (+0.026)
0.049 (+0.032)
0.051 (+0.034)
SCLIP (Wang et al.
2023
0.018
0.018
0.018
SCLIP+CDAM
0.044 (+0.026)
0.050 (+0.032)
0.052 (+0.034)
ClearCI.IP (Lan et al.
2024)
0.017
0.018
0.018
ClearCLIP+CDAM
0.044 (+0.027)
0.050 (+0.032)
0.051 (+0.033)
GEM [Bousselham et al.
2024
0.026
0.026
0.026 sec
GEM+CDAM
0.052 (+0.026)
0.059 (+0.033)
0.060 (+0.034)
As shown Table 6 the majority of the additional computational cost CDAM arises from its multi-scale image patches (Attnys) and augmented text prompts (ATP). The computational overhead increases with the number of scales (m) for Attnys and the number of classes for ATP, primarily due to the increased computational burden of calculating the Jensen-Shannon (IS) divergence. Nevertheless, the overall inference time remains within a feasible range, ensuring that CDAM is still practical for real-time applications.
C VISUALIZATION OF CLASS DISTRIBUTION-INDUCHH ATTENTION MAP
Fig. 5 visualizes examples of class distribution-induued attention maps, highlighting how the attention weight is emphasized on the relss-relevant patch for each source patch.
dog
chair
chair
donut
donut
teddy
da
bear
banana
dog
chair
donut
teddy
.teddy
cat
cat
bear
bear
banana
banana
Figure 5: We visualized examples class distribbutioniinduedd attention maps (CDAM) from several source patches. This demonstrates that our proposed CDAM can localize regions of the target class within the attention map.
D MODEL. DETAILS OF COMPARISON METHODS
For a fair comparison, we provide detailed architectural information and an additional training dataset for all methods, detailed in Table Importantly, proposed CDAM can be seamlessly integrated with existing Cased training-fret free methods, utilizing their CLIP encoder without the need for retraining.
16

---

Published as a conference paper at ICLR 2025
Table 6: Inference time (in seconds per image) for each component of CDAM.The baseline model is set as MaskCLIP, as the additional overhead introduced by CDAM is consistent across other baseline methods.
Bascline ALlDCDAM
Atinss
ATP
ent
VOC21
Contex160
COCO-Obj
0.017
0.017
0.017
0.020
0.022
0.024
/
0.032
0.038
0.040
0.043
0.049
0.051
0.043
0.049
0.051
Table 7: Detailed information on comparison methods for open-vocabulary semantic segmen- tation. provide detailed information regarding (1) visual encoder architecture, (2) pre-trained model, and (3) additional training datasets.
Method
Encoder
Model
Additional training dataset
methods with additional training dataset
GiT (Xu et al. 2022a)
ViT/S-16
Scratch
CCI5M-RedCaps 12M
CLIPpy Ranasinghe et al.
2023
VIT/B-16
CLIP
HQITP- 134M
ViewCo Ren et al. 2023
VIT/S-16
Scratch
CCI2M+YFCCISM
SegCLIP (Luo
2023
VIT/B-16
CLIP
CC3M-COC0400K
segemeentor (Xu et al.
2023a
V1T/B-16
DINO
CC4M
TCL (Cha et al. 2023
VIT/B-16
CLIP
CC15M
PACL Mukhoti al.
2023
VIT/B-16
CLIP
CC3M+CC12M-YFCCISN
visual prototype generation geneeratonnnnootto methods for each concept
OVDiff Karazija al.
2023
ViT/B-16
CLIP4SD+DINO
FreeDA Barsellotti et al.
2024
ViT/L-14
CLIP+SD+DINO
X
CLIP- training-tree methods
CLIPSurgery (Li ct al. 2023
V6
CLIP
X
CLIP-DIY (Wysoczanske ka ctal.
2024
VIT/B-16
CLIP+DINO
CaR (Sun al. 2024
VIT/L-14
CLIP
MIskCLIP (Zhou et al. 2022
VIT/B-16
CLIP
X
SCLIPi Wang et al.. 2023
VIT/B-16
CLIP
ClcarCLiP (Lan ct 2024
VIT/B-16
CLIP
GEM Bousselhan et al.
2024
VIT/B-16
CLIP
E ADDITIONAL. EXPERIMENTS
Effectiveness of JS divergence. To assess the effectiveness of JS divergence relative other metrics like KL divergence and Wasserstein distance (WS), we conducted an ablation study using several baseline methods, as presented in Table The results demonstrate that JS divergence performs comparably to KL divergence, consistently achieving equal or slightly better performance.
Discussion the rationale for using JS divergence. It is reasonable for measuring the similarity of class distributions to have the properties of symmetricity, meaning it should remain the same regardless of input order, and permutation invariance, meaning should remain the same regardless of class order. The former is satisfied by the Wasserstein distance (WS) and JS divergence, while the latter is satisfied by the KI. and JS divergences. Note that not straightforward to properly define metrics among classes for the problem of semantic segmentation, making the WS distance unsuitable for our CDAM. Morcover, divergence-brl based metrics such as KL and JS divergences are known to be more sensitive small changes over WS distance (Ozair et al. 2019), which may be advantageous for our CDAM. Thus, JS divergence appears to have favorable properties for measuring the distance between class distributions, which is consistent with experimental validation in Table
Analysis of CDAM. As shown in Table we verified the effectiveness of our Atto on VOC21 (Ev eringham al. 2010), Context60 (Mottaghi al. 2014) and COCO-Obj (Lin et al. 2014) datasets.
17

---

Published as a conference paper at ICLR 2025
Table Ablation study of similarity metries for measuring the distance of class distributions over patches. WS refers to the Wasserstein distance. The evaluation is based on mloU (%).
Methods
Metric VOC21
Context60
COCO-Obj
Avg.
MaskCLIP
Zhou et al.
2022 + CDAM
KL div.
55.7
30.4
34.3
40.1
JS div.
55.9
30.5
34.3
40.2
WS
53.0
26.7
28.5
36.1
SCLIP Wang et al.
2023 +CDAM
KL div.
58.8
30.4
34.6
41.3
IS div.
59.0
30.4
34.5
41.3
WS
57.2
29.0
31.4
39.2
ClearCLIP (Lan et al.
2024 +CDAM
KL div.
57.4
29.5
34.3
40.4
JS div.
57.6
29.8
34.5
40.6
WS
56.9
28.6
33.4
39.6
GEM (Bousselham et al.
2024 CDAM
KL div.
58.9
30.5
35.1
41.5
div.
58.7
30.6
35.2
41.5
WS
58.4
29.5
34.0
40.6
Low uncertainty of segmentation for both foreground and background regions corresponds to low and high entropy of the class distribution, respectively. Compared other types of attention maps, proposed A Attnms effectively reduce the uncertainty by widening the gap between Hiack and H This significantly affects the segmentation performance and demonstrates the enhanced localization ability of our Attoms.
Table 9: Uncertainty in boreground-backgneen regions. By varying the types of attention maps in the last layer, we compute the mean entropy of the class distribution in the foreground region fore), and background region (Hheek), respectively. Diff refers to the absolute difference between Hback and H fore (Hback - H fore). Reducing uncertainty (Diff 1) of our proposed CDAM with multi-scale image patches leads significantly improved segmentation performance.
VOC21
Context60
COCO-Obj
Attn. Map
Diff
mloU
Diff
mloU 1
Diff
mloU 1
Random
-0.11
8.5
-0.10
4.7
-0.01
4.82
Identity
0.40
33.1
0.00
23.3
0.35
24.8
Attnms (ours)
1.02
54.7
0.25
27.5
1.06
34.2
Reproduction details of existing methods. For a fair comparison among training-free CLIP based methods, we reproduced the prior works, MaskCLIP (Zhou et al. 2022), SCLIP (Wang et al. 2023) CaR (Sun al. 2024), GEM (Bousselham al. 2024 and ClearCLIP (Lan et al. 2024). by following the unified evaluation protocol and eliminating the renaming tricks. remove the expanded the target class names (e.g., "person in shirt", "human" for "person" class Then, we selected the results demonstrating the highest overall performance from among those obtained using various hrckground threshold values 10. 0.2, 0.3, 0.4, 0.5). An example of the reproduced results is provided in Table IO] Based on the results in Table I0 we set the Thraefuuk value of SCLIP (Wang et al. (2023) to 0.2 for all benchmark datasets to avoid dataset-specifie parameters. For all benchmark datasets, inference was conducted with sliding window size of 448 and a stride of 224.
Analysis of entropy-basee background threholsding, Before implementing the entropy-based background thresholding method, we aim to demonstrate its effectiveness. We achieve this by comparing various thresholding strategies with respect to the optimal threshold value that maximizes the mloU score for given dataset. To achieve this, we first employ the Pocket algorithm to identify the threshold that maximizes the mloU value for each image. We then calculate the Pearson correlation coefficient between this optimal threshold and the results obtained using different thresholding techniques. Since Pearson correlation measures the strength the linear relationship between two variables, evaluate which method produces threshold value most closely aligned with the optimal threshold. The results are presented Table [T] As evident from the table, our proposed method consistently demonstrates high correlation value with the optimal threshold across most
18

---

Published as a conference paper at ICLR 2025
Table 10: Reproduced results of SCLIP (aag et al., 2023) with varying background threshold values. We evaluated the performance of SCLIP (Wang et al., 2023) with background threshold values ranging from 0.1 0.5. The best overall performance was achieved with Thrae set to 0.2, and we therefore selected the results obtained at this threshold for the unified evaluatino protocol (Chal
al. 2023 The evaluation based on mloU (%).
Thrdefault
VOC21
Context60
COCO-Obj
Avg.
0.1
37.8
30.5
29.9
32.7
0.2
50.5
25.8
31.3
35,9
0.3
54.4
20.3
28.1
34.3
0.4
51.8
15.7
23.4
30.3
0.5
46.6
11.7
18.4
25.6
datasets. This indicates that there are strong agreement between the optimal threshold and the thresholds generated by our method.
Conversely, other thresholding techniques exhibit either weak or inconsistent correlation with the optimal threshold. Moreover, some techniques might show high correlation specific datasets, they show downgraded performance in different datasets. In contrast, our method delivers consistently high correlation, highlighting its suitability for this task. This superior performance likely stems from the inherent differences between the target tasks addressed by various methods. Traditional thresholding techniques are primarily designed for grayscale images, which often contain well-defined foreground objects and have higher dimensionality (e.g., 12xx512 pixels). Our task, however, presents a greater challenge: thresholding the maximum probability value in images with significantly smaller size (e.g., 28x28 pixels) and uncertain object boundaries. Due to these distinct characteristics, traditional grayscale image thresholding methods are not well-suited for determining the optimal threshold in our specific scenario.
Table 11: Correlation between optimal threshold and thresholding techniques. At first, we searched the optimal threshold that maximizes the mloU score using ground truth. Subsequently, we compare the Pearson correlation coefficient between this optimal threshold and the results obtained using different thresholding strategies. As a result, our proposed method achieves the highest average correlation with the optimal threshold over three benchmark datasets.
Method
VOC21
Context60
COCO-Obj
Avg.
Otsu (Otsu et al.
1975
0.397
0.203
0.189
0.263
Lloyd Rosenteld & De La Torre 1983
0.393
0.206
0.189
0.262
Kittler (Sezan
1990)
0.235
0.135
0.025
0.129
Li [Li & Lee
1993)
0.280
0.207
0.055
0.181
Kapur (Kapur et al. 1985
0.047
-0.026
0.117
0.046
Pal (Pal 1996)
1996)
0.051
-0.023
0.119
0.049
Brink (Brink & Pendock 1996)
0.147
0.058
-0.129
0.025
Huang (Huang & Wang 1995
0.230
0.239
0.049
0.172
Threat-bg (Ours)
0.477
0.175
0.321
0.325
Tuning of background thresholding value Thr Thraefautt for each dataset. For each benchmark dataset, we fine-tuned the background thresholding value, Thraefautt and disabled entropy-based background thresholding in our CDAM. As shown in Table [2] tuning the background thresholding value substantially improves performance compared to the fixed Thr used in the main paper. While existing training-free free methods achieve high performance through tuning Thr our proposed CDAM further enhances performance improvements. Notably, CDAM with MaskCLIP (Zhou et al. 2022) surpasses all existing methods when tuning Thraefaul
Ablation studies for hyperparameter of CDAM. For the temperature т, the modulation of entropy and the set of scaling factor M, we conducted an ablation study by varying their values. The baseline model is MaskCLIP (Zhou al. 2022 and applied proposed CDAM with various
19

---

Published as a conference paper at ICLR 2025
Table 12: Tuning background thresholding value Thrdef for CLIP-based training-free methods with CDAM. We measured the performance of existing training-free methods by varying Thraetaul from 0.1 0.6. Then, we report the highest performance of all methods on three benchmark datasets. We marked for the reproduced methods. Performance improvements by CDAM are indicated in parentheses. The evaluation is based on mloU (%).
Methods
VOC21
Context60
COCO-Obj
Avg.
MaskCLIP Zhou et al.
2022
42.9
23.3
24.8
30.3
MaskCLIP+CDAM
55.0 (+12.1)
33.6 (+10.3)
(+9.6)
41.0 (+8.3)
SCLIPt (Wang et al.
2023)
54.4
30.5
31.3
38.7
SCLIP+DDAM
57.5 (+3.1)
33.5 (+3.0)
34.9 (+3.6)
42.0 (+3.3)
ClearCLIPI (Lan al.
2024)
51.8
32.3
33.1
39.1
ClearCLIPCCDAM
56.3 (+4.5)
33.0 (+0.7)
34.6 (+1.5)
41.3 (+2.2)
GEM [Bousselham et al.
2024)
53,8
32.4
34.1
40.1
GEM+CDAM
57.3 (43.5)
33.6 (+1.2)
35.6 (+1.5)
42.2 (42.1)
hyperparameters.Be Based on the results shown in Table 13 (0.25, 0.37, 0.5, 0.63, 0.75, 0.87, 1.0), respectively.
14 15
we set 7, a and M to 0.1, 2.5 and
Table 13: Ablation study temperature for generating the class distribattonn-iddueed attention map.
VOC21
Context60
COCO-Obj
Avg.
0.05
53.3
29.6
33.0
38.6
0.10 (Ours)
55.9
30.5
34.3
40.2
0.15
55.6
29.7
33.6
39.6
0.20
52.5
28.1
31.1
37.2
Table 14: Ablation study modulation parameter for entropy-based background threshold- ing.
VOC21
Context60
COCO-Obj
Avg.
1.0
44.1
33.5
28.6
35.4
1.5
50.5
33.3
31.8
38.5
2.5 (Ours)
55.9
30.5
34.3
40.2
3.5
48.3
25.2
34.0
35.8
Table 15: Ablation study set of scaling factor M for the CDAM with multi-scale image patches.
Set of scaling factor M
VOC21
Context60
COCO-Obj
Avg.
(0.3333370.00066,,775577...0.000 (Ours) (0.50.0.63.00.5...88....00)))))) (0.75,0.87.1.00) (1.00) (0.25.0.50.0.75.1.00)
55.9
30.5
34.3
40.2
55.8
30.4
34.4
40.2
55.5
30.2
34.2
40.0
53.1
27.7
32.1
37.6
55.7
30.4
34.1
40.1
Ablation studies for different patch size. To explore the impact of different patch sizes proposed CDAM, we conducted benchmark experiments using CLIP ViT/B-32, which has a patch size of 32. Note that the results presented in the main paper used CLIP VIT/B-16, which uses patch
20

---

Published as a conference paper at ICLR 2025
size of 16. As shown in the Table 16 these results demonstrate that proposed CDAM effectively enhances the performance of CLIP-based training- free methods, even when using CLIP models with different and larger patch sizes. It also shows that larger patch size (32) typically results in decreased performance compared to smaller patch size (16) in Table] T of the main paper. This suggests that small patch size of 16 is advantageous over large patch size of 32 for segmentation due its ability to provide better spatial resolution. In this experiment, the modulation of entropy, a, is set to 2.0.
Table 16: Ablation study with CLIP ViT/B-32 for exploring different patch sizes. We evaluated the open-vocabulary semantic segmentation performance of existing training free methods. We marked for the reproduced methods. Performance improvements achieved by CDAM are indicated parentheses. The evaluation metric used is mloU (%).
Methods
VOC21
Context60
COCO-Obj
Avg.
MaskCLIPI Zhou et al.
2022)
29.5
8.1
11.5
16.4
MasACLIP+CDAN
50.1 (+20.6)
27.6 (+19.5)
27.8 (+16.3)
35.2 (+18.8)
SCLIPi WWag et al.
2023
38.0
24.1
25.1
29.1
SCLIP+DDAM
51.6 (+13.6)
25.7 (+1.6)
27.6 (+2.5)
35.0 (+5.9)
ClearCLIPi Lan al.
2024)
47.6
23.3
27.3
32.7
CleArCLIP+CDAM
51.4 (+3.8)
27.5 (+4.2)
28.4 (+1.1)
35.8 (+3.1)
GEM [Bousselham et al. 2024)
52.1
28,1
33.8
38.0
GEM+CDAM
55.9 (+3.8)
32.4 (+4.3)
34.4 (+0.6)
40.9 (+2.9)
Supporting our motivation with several baseline methods. As mentioned in Section 3.2. Table 17 provides statistical evidence to support our claim "CLII--based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object".
The experimental setup is follows: For given image, randomly selected Ptarget within the target class region and also randomly selected P.,. and Ptarget from the the target class region and the rest of the region, respectively. We then measured (1) the probability that the class prediction for Ptarget is correct and (2) the probability that the distribution similarity between Ptarget and P.,. is higher than the distribution similarity between Ptarget and Pout-
Table 17: Accuracy comparison of class predictions and similarity of class distributions with several CLIP--ased training-free methods across datasets. Similarity of class distribution is measured using JS divergence.
Baseline
VOC21
Context60
COCO-Obj
Avg.
MaskCLIP Zhou et al.
2022
Class Prediction
56.1 + 1.17
38.4 + 0.23
27.4
43.0
Sim of Class Dist
70.9 1 0.44
73.1 0.34
69.8
71.0
SCLIP Wang al. 2023
Class Prediction
67.0 0.49
41.8 0.31
33.6
47.4
Sim of Class Dist
78.9 0.26
72.0 7.30
7..4
75.5
ClearCLIP Lan et al.
2024
Class Prediction 70.3 0.51
42.7 0.23
36.4
49.9
Sim of Class Dist 76.0 0.58
70.5 0.58
71.7 0.48
72.3
GEM Bousselham et al. 2024
Class Prediction
1.12
42.4 0.38
37.5
50.0
Sim of Class Dist
79.4 + 0.89
71.2 + 0.34
74.2 + 0.20
74.8
Analysis of performance discrepancy between SCLIP and ClearCLIP with CDAM. In Ta- ble and 2 of the main paper, the performance of ClearCLIP (Lan et al. 2024) surpasses that of SCLTP [Wang al. 2023). However, when combined with CDAM, SCLIP outperforms ClearCLII. This discrepancy arises from ClearCLIP's design choice remove residual connections suppress noisy class predictions, which becomes less effective when used alongside CDAM. The absence of residual connections in ClearCLIP, combined with the lack of additional training, likely hinders its ability to fully utilize CLIP's implicit knowledge of class distributions. Consequently. the synergy between CDAM and CLearCLIP is reduced, resulting in smaller performance gains relative SCLIP.
Applying our CDAM on musk-annotation based supervision methods. We conducted additional experiments to validate the effectiveness of our CDAM by incorporating it into existing methods that
21

---

Published as a conference paper at ICLR 2025
mask annotations for specific classes in supervised manner. Specifically, applied CDAM to mask-annotation based methods and report the results in Table [8 As demonstrated, our CDAM did not degrade the performance of seen classes optimized by training- based methods with mask annotation such as ZegCLIP (Zhou et al. 2023) and OTSSe (Kim et al. 2024). Instead, it consistently improved performance for unseen classes.
Table 18: Results of open-vocabulary semantic segmentation methods supervised using mask annotations with our CDAM on VOC 2012. hloU refers to harmonic mean IoU, Performance improvements by CDAM are indicated in parentheses.
Method
mIoU (Seen)
mIoU (Unseen)
hloU
ZegCLIP Zhou al.
2023
91.8
77.9
84.3
ZegCLIP+CDAM
91.8 (+0.0)
78.4 (+0.5)
84.6)+
OTSeg (Kim al. 2024
93.3
81.8
87.2
OTSeg+CDAM
93.4 (+0.1)
82.2 (+0.4)
87.4 (+0.2)
F QUALITATIVE RESULTS
We visualize additional qualitative segmentation results in Fig. 6 It demonstrates that accurate localization ability of our CDAM for semantic segmentation. Furthermore, in Fig. 7 we present more examples of class distribution-induced attention maps. It indicates that without the necessity of training and annotation, can construct the well-localized attention map based the similarity of class distribution between patches within an image.
G LIMITATIONS
Leveraging the knowledge of pre-trained vision-language models, we construct class distribution- induced attention map based on class distribution of target classes. However, the quality of the localized attention map is dependent on the diversity and number of target classes used. This dependence can limit the representation of CLIP's implicit knowledge within the class distribution. Specifically, CLIP's s limitations in class diversity and image-text representation hinder segmentation performance, particularly for rare classes. CLiP-based training-free methods inherit these constraints, as they lack additional training. While CDAM improves the performance of these approaches, it cannot overcome CLIP's s fundamental limitations. Addressing such challenges may require training- based strategies like domain- specific prompt tuning advanced multi-moda models with better class diversity and image-tex text alignment. However, fully resolving out-of -itoo--distribution (OOD) issues or achieving complete class coverage remains challenging without extensive training, which is beyond the scope of this work.
Entropy- based background thresholding, being empirically designed, often struggles in handling com- plex and challenging test images. The primary difficulty lies in its inability to determine patch-specific threshold values effectively in complex scenes. In simpler scenarios, such as when single target class object is prominently located at the center of the image, a single threshold value may suffice. However, as scene complexity increases, local thresholding becomes necessary, which introduces ad- ditional challenges. To address these limitations and improve the robustness of background detection, data-driven methods like FOUND (Siméoni et al. 2023) offer a potential solution. FOUND proposed an unsupervised framework to effectively extract background regions. However, its dependence on saliency-base foreground detection, without incorporating knowledge of target classes, limits its adaptability for -pcalvocabulary semantic segmentation. While data-driven approaches show promise, achieving universally robust and adaptable background thresholding remains a significant challenge.
22

---

Published as a conference paper at ICLR 2025
GT
CDAM
GT
CDAM
CDAM
Figure 6: Qualitative results of open-vocabulary semantic segmentation using our CDAM with MaskCLIP.
23

---

Published as a conference paper at ICLR 2025
Initial prediction
GT
Initial prediction
GT
Initial prediction
GT
CDAM
CDAM
CDAM
potted plant
person
watch
Cat
bottle
Initial prediction
GT
Initial prediction
GT
Initial prediction
GT
CDAM
CDAM
CDAM
chair
tv monitor
dog
plate
car
train
Figure 7: Additional examples of class distribution attention maps (CDAM) from the initial prediction of MaskCLIP
24


Reviews:
[{"review_id": "wBbrsRf3o7", "reviewer": "Reviewer_cFei", "summary": "The noisy patch-level prediction is rectified by class distribution-induced attention in zero-shot open-vocabulary semantic segmentation in this paper. The motivation is clear. Experimental results show the effectiveness of the proposed idea cooperating with several state-of-the-art approaches and significant performance improvement. The paper is well-written and the figures are easy to follow.", "questions": "1. It's better to give the mIoU both for seen and unseen classes.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is good enough for me as the clear motivation and consistent performance gain on several public semantic segmentation datasets when integrating the proposed Class Distribution-induced Attention Map to different SOTA methods. \n2. The motivation is clear and illustrated well in Fig. 1.", "weaknesses": "1. In Table 1, even though CaR is a heavily computational method and CLIP-DIY uses an extra background extractor,  the proposed CDAM is not integrated into CaR and CLIP-DIY, and the best performance is not achieved on VOC21.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MRAORDoZz5", "reviewer": "Reviewer_874d", "summary": "Current CLIP-based Open-Vocabulary Semantic Segmentation (OVSS) methods face limitations due to noisy patch-wise class predictions and highly correlated class distributions for each object. To address these issues, the authors propose a Class Distribution-induced Attention Map (CDAM), generated using the Jensen-Shannon divergence between class distributions of two patches from the same object, to enhance focus on class-relevant regions without additional training. The authors also introduce enhancements such as multi-scale image patches, augmented text prompts, and entropy-based background thresholding to further improve CDAM. Comprehensive experiments demonstrate that CDAM improves multiple OVSS methods across several datasets, with ablation studies validating the effectiveness of each component.", "questions": "1. Similar to Table 3, an additional ablation study on CDAM components using another baseline model would be beneficial.\n2. Since CDAM relies on distributions between patches, an ablation study on the impact of different patch sizes would be beneficial.\n3. A detailed analysis of the computational complexity of the CDAM would be helpful. Additional experiments related to runtime or efficiency when constructing CDAM would add value.\n4. Comparing Table 1 and Table 2, there is a noticeable difference in CDAM’s improvement with/without the background class. The authors should provide more explanation for these differences to clarify their impact on performance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is good, with CDAM introducing a novel approach to leverage class distributions for enhancing OVSS performance, offering a new pathway for training-free segmentation improvements.\n2. The paper is well-written, with clear explanations of the methodology, experimental settings and results.\n3. CDAM is training-free, and the authors provide comprehensive quantitative results demonstrating the effectiveness of each component.", "weaknesses": "Writing Suggestions\n1. The authors state that “CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object.” Is this conclusion based on statistical analysis, or is it an observation from limited examples? Providing more statistical evidence would make this argument more convincing.\n2. In the third paragraph of Section 1 (lines 51 to 60), the authors should provide additional details on how CDAM is constructed and, more importantly, explain why it is effective. Focusing on why it works would strengthen this section.\n3. The order of Figure 1 and Figure 2 should be swapped, as Figure 2 is referenced before Figure 1 (lines 125 to 126).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7U2nyfi0l", "reviewer": "Reviewer_ehJ5", "summary": "This paper proposes a Class Distribution-induced Attention Map (CDAM) to enhance the capability of CLIP features in representing different categories for open-vocabulary semantic segmentation. The proposed method can be easily embedded into various approaches to boost their performance. Additionally, the authors introduce an entropy-based background thresholding technique for semantic segmentation inference to facilitate the extraction of foreground classes. The experiments demonstrate the effectiveness of the proposed methods.", "questions": "My primary concern is that the motivation for this work lacks sufficient support, and there is a lack of necessary explanation for some of the techniques used in the proposed method, as outlined in the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, with a clear and accessible presentation.\n2. The authors conducted comprehensive experiments that effectively highlight the superiority of the proposed method.", "weaknesses": "1. The authors claim that CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object, but this lacks necessary validation. Although an analysis of CLIP and MaskCLIP is provided in the methods section to support this claim, the analysis is not sufficiently general. MaskCLIP is an earlier work, and more recent research in the field may have addressed patch-wise noisy class predictions. Thus, the argument may not be robust, and the paper's novelty compared to related works remains debatable.\n\n2. Jensen-Shannon divergence is a key technique used in this paper. However, there is insufficient discussion on the necessity of using this method and why alternative techniques would be inadequate.\n\n3. The paper lacks discussion of other relevant methods. Methods such as [1], [2], and [3] involve reusing the CLIP [CLS] token and optimizing the feature space, which could enhance CLIP's performance in region recognition. It remains unclear whether these methods could also address the issues raised in this paper.\n\n4. In Table 2, the performance of ClearCLIP is significantly higher than that of SCLIP, as reported in the original ClearCLIP results. However, after incorporating CDAM, SCLIP outperforms ClearCLIP. The reason for this performance discrepancy requires further explanation.\n\n[1] Side Adapter Network for Open-Vocabulary Semantic Segmentation\n\n[2] Learning Mask-aware CLIP Representations for Zero-Shot Segmentation CLIP-Adapted Region-to-Text Learning\n\n[3] AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYvZyT45HZ", "reviewer": "Reviewer_6UEY", "summary": "This manuscript is dealing with Open-vocabulary semantic segmentation aiming to improve the labeling of  individual pixels with both seen and unseen classes, with a focus on localization and background separation. This work leverages class distribution comparisons between patches of the same object to improve localization. By integrating their approach (CDAM) into CLIP’s final layer, the model’s ability to focus on desired classes is enhanced. CDAM also supports multi-scale image patches and augmented text prompts, improving segmentation accuracy and enabling entropy-based background thresholding. The presented results show some performance improvements on standard benchmarks.", "questions": "The questions below are summarizing the weaknesses I've highlighted above. \n1. How does the complexity introduced by the Jensen-Shannon divergence calculation and multi-scale patch analysis impact the model’s runtime and feasibility in real-time applications?\n2. To what extent do CLIP’s limitations in class diversity and image-text representation influence the segmentation results of CDAM?\n3, What specific challenges could arise when applying the heuristic entropy-based background thresholding in complex scenes with ambiguous background features? Are there data-driven approaches that could replace the heuristic thresholding method to improve robustness and adaptability?\n4.  Why did the authors focus their analysis primarily on MaskCLIP, and what advantages or disadvantages does this bring in evaluating CDAM’s effectiveness? How would CDAM’s performance and computational requirements compare if implemented on more recent segmentation models like CaR or others?\n5. How might the CDAM approach be adapted to address challenges in recognizing rare or fine-grained classes that are crucial in open-vocabulary segmentation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea of the manuscript is rather simple and straight-forward. Nevertheless, there are some things to be highlighted: \n1. By using class distribution similarities the approach aims at enhancing the localization of objects in open-vocabulary segmentation  addressing up to a point the problem of other methods when coping with the patch-wise noise in class predictions. A more reliable attention map is statistically obtained by looking at the Jensen-Shannon divergence between patch pairs belonging to the same class..\n2. Although not new, it is good that the proposed approach is able to deal with multi-scale patches and augmented prompts making it a more versatile framework able to more precisely capture class distinctions. Along the same lines (not really new) is the entropy-based background thresholding enhancing the segmentation performance by providing a cleaner separation of relevant classes from the background.\n3. Probably the most important contribution is the compatibility with other zero-shot segmentation approaches allowing the approach to be integrated with existing models, potentially compounding improvements without much redundancy. This adaptability offers an approach that complements, rather than competes with, prior work, making it suitable for further extensions.", "weaknesses": "There are several issues that need to be clarified. \n1. Increased Computational Complexity. There is little discussion regarding the complexity overhear introduced, especially with the Jensen-Shannon divergence calculation and multi-scale patch analysis that might require significant computational resources, which could limit its scalability in real-time or resource-constrained applications.\n2. There is an inherent dependency on the CLIP Model inheriting the CLIP’s limitations in terms of class diversity, image-text representation, and the specificity of semantic segmentation. Any inherent biases or limitations in the CLIP model could be amplified or remain unresolved in this framework.\n3. The background thresholding approach is rather heuristic (also indicated by the authors) and this brings uncertainty regarding its robustness especially in the presence of highly complex scenes with ambiguous background features. Accurately setting thresholds for diverse and dynamic backgrounds could be challenging and might require extensive tuning for different environments.\n4. It is unclear why most of the detailed analysis has been done on top of MaskCLIP and not on some of the newer approaches. I understand that the improvement is larger when compared to MaskCLIP but one would have expected to see the analysis on the more performing approaches. Some of the details are missing or the authors are treated them superficially. For example when doing the analysis of the results in Table 1 they simply indicate that the best performing approach, i.e., CaR requires high computational costs but it is unclear why this is indeed a problem given that CDAM is supposed to be added on top of it. \n5. There is a no insight into generalization on rare or fine-grained classes. The approach emphasizes improvement in localization but may not specifically address challenges in recognizing rare or highly similar fine-grained classes, a common difficulty in open-vocabulary segmentation.\n6. Although CDAM is designed to work alongside existing zero-shot methods, effectively combining this technique with other methods might be challenging in practice. There is practically no discussion highlighting for example whether this compatibility would likely require additional tuning and could complicate model training, implementation, and maintenance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F16Ltp9gNJ", "reviewer": "Reviewer_sGKo", "summary": "This paper introduces CDAM for open-vocabulary semantic segmentation that improves object localization by utilizing class distribution similarities within patches. It employs Jensen-Shannon divergence to create an attention map that boosts CLIP's segmentation accuracy without extra training. CDAM also uses multi-scale patches and entropy-based thresholding for enhanced performance, outperforming other methods on segmentation benchmarks.", "questions": "1. How to explain the rationality of using two hyper-parameters α and Thr_default to control the background thresholding in Formula 4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper improves the localization of objects within images by leveraging class distribution similarities.\n\n2. The entropy-based background thresholding adapts dynamically to different images, which helps in accurately separating the foreground from the background in segmentation tasks.", "weaknesses": "1. There is little explanation why the Jensen-Shannon divergence is suitable for semantic segmentation. The rationality of this method should be explained clearly.\n\n2. There are some confusing aspects in the use of symbols in this paper, such as the final similarity map S in line 251 and the class distribution S in line 266; The image representation is unclear, for example, the meaning of S_p1 in Figure 1 has not been mentioned yet in the paper.\n\n3. The JS divergence is used in the paper to obtain Attn_CDAM.  How is the effect of using KL divergence and other measurement methods.\n\n4. What is the inference time after adding CDAM to other methods.\n\n5. The word 'food' appears twice in the Supercategory in Tab 4. An additional ‘()’ appeared in line 330", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)