Class Distribution-induced Attention Map for Open-vocabulary Semantic Segmentations

Authors: Dong Un Kang, Hayeon Kim, Se Young Chun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Vision Language Model, Dense Localization

Abstract:
Open-vocabulary semantic segmentation is a challenging task that assigns seen or unseen class labels to individual pixels. While recent works with vision-language models (VLMs) have shown promising results in zero-shot semantic segmentation, they still struggle to accurately localize class-related objects. In this work, we argue that CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object. Then, we propose Class Distribution-induced Attention Map, dubbed CDAM, that is generated by the Jensen-Shannon divergence between class distributions of two patches that belong to the same (class) object. This CDAM can be used for open-vocabulary semantic segmentation by integrating it into the final layer of CLIP to enhance the capability to accurately localize desired classes. Our class distribution-induced attention scheme can easily work with multi-scale image patches as well as augmented text prompts for further enhancing attention maps. By exploiting class distribution, we also propose robust entropy-based background thresholding for the inference of semantic segmentation. Interestingly, the core idea of our proposed method does not conflict with other prior arts in zero-shot semantic segmentation, thus can be synergetically used together, yielding substantial improvements in performance across popular semantic segmentation benchmarks.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729504848970

thread_type:
official_review

<thread_object>
creator: reviewer_sgko

<content>
summary:
This paper introduces CDAM for open-vocabulary semantic segmentation that improves object localization by utilizing class distribution similarities within patches. It employs Jensen-Shannon divergence to create an attention map that boosts CLIP's segmentation accuracy without extra training. CDAM also uses multi-scale patches and entropy-based thresholding for enhanced performance, outperforming other methods on segmentation benchmarks.

strengths:
1. This paper improves the localization of objects within images by leveraging class distribution similarities.

2. The entropy-based background thresholding adapts dynamically to different images, which helps in accurately separating the foreground from the background in segmentation tasks.

weaknesses:
1. There is little explanation why the Jensen-Shannon divergence is suitable for semantic segmentation. The rationality of this method should be explained clearly.

2. There are some confusing aspects in the use of symbols in this paper, such as the final similarity map S in line 251 and the class distribution S in line 266; The image representation is unclear, for example, the meaning of S_p1 in Figure 1 has not been mentioned yet in the paper.

3. The JS divergence is used in the paper to obtain Attn_CDAM.  How is the effect of using KL divergence and other measurement methods.

4. What is the inference time after adding CDAM to other methods.

5. The word 'food' appears twice in the Supercategory in Tab 4. An additional ‘()’ appeared in line 330

questions:
1. How to explain the rationality of using two hyper-parameters α and Thr_default to control the background thresholding in Formula 4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730427956162

thread_type:
official_review

<thread_object>
creator: reviewer_6uey

<content>
summary:
This manuscript is dealing with Open-vocabulary semantic segmentation aiming to improve the labeling of  individual pixels with both seen and unseen classes, with a focus on localization and background separation. This work leverages class distribution comparisons between patches of the same object to improve localization. By integrating their approach (CDAM) into CLIP’s final layer, the model’s ability to focus on desired classes is enhanced. CDAM also supports multi-scale image patches and augmented text prompts, improving segmentation accuracy and enabling entropy-based background thresholding. The presented results show some performance improvements on standard benchmarks.

strengths:
The idea of the manuscript is rather simple and straight-forward. Nevertheless, there are some things to be highlighted: 
1. By using class distribution similarities the approach aims at enhancing the localization of objects in open-vocabulary segmentation  addressing up to a point the problem of other methods when coping with the patch-wise noise in class predictions. A more reliable attention map is statistically obtained by looking at the Jensen-Shannon divergence between patch pairs belonging to the same class..
2. Although not new, it is good that the proposed approach is able to deal with multi-scale patches and augmented prompts making it a more versatile framework able to more precisely capture class distinctions. Along the same lines (not really new) is the entropy-based background thresholding enhancing the segmentation performance by providing a cleaner separation of relevant classes from the background.
3. Probably the most important contribution is the compatibility with other zero-shot segmentation approaches allowing the approach to be integrated with existing models, potentially compounding improvements without much redundancy. This adaptability offers an approach that complements, rather than competes with, prior work, making it suitable for further extensions.

weaknesses:
There are several issues that need to be clarified. 
1. Increased Computational Complexity. There is little discussion regarding the complexity overhear introduced, especially with the Jensen-Shannon divergence calculation and multi-scale patch analysis that might require significant computational resources, which could limit its scalability in real-time or resource-constrained applications.
2. There is an inherent dependency on the CLIP Model inheriting the CLIP’s limitations in terms of class diversity, image-text representation, and the specificity of semantic segmentation. Any inherent biases or limitations in the CLIP model could be amplified or remain unresolved in this framework.
3. The background thresholding approach is rather heuristic (also indicated by the authors) and this brings uncertainty regarding its robustness especially in the presence of highly complex scenes with ambiguous background features. Accurately setting thresholds for diverse and dynamic backgrounds could be challenging and might require extensive tuning for different environments.
4. It is unclear why most of the detailed analysis has been done on top of MaskCLIP and not on some of the newer approaches. I understand that the improvement is larger when compared to MaskCLIP but one would have expected to see the analysis on the more performing approaches. Some of the details are missing or the authors are treated them superficially. For example when doing the analysis of the results in Table 1 they simply indicate that the best performing approach, i.e., CaR requires high computational costs but it is unclear why this is indeed a problem given that CDAM is supposed to be added on top of it. 
5. There is a no insight into generalization on rare or fine-grained classes. The approach emphasizes improvement in localization but may not specifically address challenges in recognizing rare or highly similar fine-grained classes, a common difficulty in open-vocabulary segmentation.
6. Although CDAM is designed to work alongside existing zero-shot methods, effectively combining this technique with other methods might be challenging in practice. There is practically no discussion highlighting for example whether this compatibility would likely require additional tuning and could complicate model training, implementation, and maintenance.

questions:
The questions below are summarizing the weaknesses I've highlighted above. 
1. How does the complexity introduced by the Jensen-Shannon divergence calculation and multi-scale patch analysis impact the model’s runtime and feasibility in real-time applications?
2. To what extent do CLIP’s limitations in class diversity and image-text representation influence the segmentation results of CDAM?
3, What specific challenges could arise when applying the heuristic entropy-based background thresholding in complex scenes with ambiguous background features? Are there data-driven approaches that could replace the heuristic thresholding method to improve robustness and adaptability?
4.  Why did the authors focus their analysis primarily on MaskCLIP, and what advantages or disadvantages does this bring in evaluating CDAM’s effectiveness? How would CDAM’s performance and computational requirements compare if implemented on more recent segmentation models like CaR or others?
5. How might the CDAM approach be adapted to address challenges in recognizing rare or fine-grained classes that are crucial in open-vocabulary segmentation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484249545

thread_type:
official_review

<thread_object>
creator: reviewer_ehj5

<content>
summary:
This paper proposes a Class Distribution-induced Attention Map (CDAM) to enhance the capability of CLIP features in representing different categories for open-vocabulary semantic segmentation. The proposed method can be easily embedded into various approaches to boost their performance. Additionally, the authors introduce an entropy-based background thresholding technique for semantic segmentation inference to facilitate the extraction of foreground classes. The experiments demonstrate the effectiveness of the proposed methods.

strengths:
1. The paper is well-written, with a clear and accessible presentation.
2. The authors conducted comprehensive experiments that effectively highlight the superiority of the proposed method.

weaknesses:
1. The authors claim that CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object, but this lacks necessary validation. Although an analysis of CLIP and MaskCLIP is provided in the methods section to support this claim, the analysis is not sufficiently general. MaskCLIP is an earlier work, and more recent research in the field may have addressed patch-wise noisy class predictions. Thus, the argument may not be robust, and the paper's novelty compared to related works remains debatable.

2. Jensen-Shannon divergence is a key technique used in this paper. However, there is insufficient discussion on the necessity of using this method and why alternative techniques would be inadequate.

3. The paper lacks discussion of other relevant methods. Methods such as [1], [2], and [3] involve reusing the CLIP [CLS] token and optimizing the feature space, which could enhance CLIP's performance in region recognition. It remains unclear whether these methods could also address the issues raised in this paper.

4. In Table 2, the performance of ClearCLIP is significantly higher than that of SCLIP, as reported in the original ClearCLIP results. However, after incorporating CDAM, SCLIP outperforms ClearCLIP. The reason for this performance discrepancy requires further explanation.

[1] Side Adapter Network for Open-Vocabulary Semantic Segmentation

[2] Learning Mask-aware CLIP Representations for Zero-Shot Segmentation CLIP-Adapted Region-to-Text Learning

[3] AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation

questions:
My primary concern is that the motivation for this work lacks sufficient support, and there is a lack of necessary explanation for some of the techniques used in the proposed method, as outlined in the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730524889019

thread_type:
official_review

<thread_object>
creator: reviewer_874d

<content>
summary:
Current CLIP-based Open-Vocabulary Semantic Segmentation (OVSS) methods face limitations due to noisy patch-wise class predictions and highly correlated class distributions for each object. To address these issues, the authors propose a Class Distribution-induced Attention Map (CDAM), generated using the Jensen-Shannon divergence between class distributions of two patches from the same object, to enhance focus on class-relevant regions without additional training. The authors also introduce enhancements such as multi-scale image patches, augmented text prompts, and entropy-based background thresholding to further improve CDAM. Comprehensive experiments demonstrate that CDAM improves multiple OVSS methods across several datasets, with ablation studies validating the effectiveness of each component.

strengths:
1. The motivation is good, with CDAM introducing a novel approach to leverage class distributions for enhancing OVSS performance, offering a new pathway for training-free segmentation improvements.
2. The paper is well-written, with clear explanations of the methodology, experimental settings and results.
3. CDAM is training-free, and the authors provide comprehensive quantitative results demonstrating the effectiveness of each component.

weaknesses:
Writing Suggestions
1. The authors state that “CLIP-based prior works yield patch-wise noisy class predictions while having highly correlated class distributions for each object.” Is this conclusion based on statistical analysis, or is it an observation from limited examples? Providing more statistical evidence would make this argument more convincing.
2. In the third paragraph of Section 1 (lines 51 to 60), the authors should provide additional details on how CDAM is constructed and, more importantly, explain why it is effective. Focusing on why it works would strengthen this section.
3. The order of Figure 1 and Figure 2 should be swapped, as Figure 2 is referenced before Figure 1 (lines 125 to 126).

questions:
1. Similar to Table 3, an additional ablation study on CDAM components using another baseline model would be beneficial.
2. Since CDAM relies on distributions between patches, an ablation study on the impact of different patch sizes would be beneficial.
3. A detailed analysis of the computational complexity of the CDAM would be helpful. Additional experiments related to runtime or efficiency when constructing CDAM would add value.
4. Comparing Table 1 and Table 2, there is a noticeable difference in CDAM’s improvement with/without the background class. The authors should provide more explanation for these differences to clarify their impact on performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730815437237

thread_type:
official_review

<thread_object>
creator: reviewer_cfei

<content>
summary:
The noisy patch-level prediction is rectified by class distribution-induced attention in zero-shot open-vocabulary semantic segmentation in this paper. The motivation is clear. Experimental results show the effectiveness of the proposed idea cooperating with several state-of-the-art approaches and significant performance improvement. The paper is well-written and the figures are easy to follow.

strengths:
1. The paper is good enough for me as the clear motivation and consistent performance gain on several public semantic segmentation datasets when integrating the proposed Class Distribution-induced Attention Map to different SOTA methods. 
2. The motivation is clear and illustrated well in Fig. 1.

weaknesses:
1. In Table 1, even though CaR is a heavily computational method and CLIP-DIY uses an extra background extractor,  the proposed CDAM is not integrated into CaR and CLIP-DIY, and the best performance is not achieved on VOC21.

questions:
1. It's better to give the mIoU both for seen and unseen classes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the problem of open-vocabulary semantic segmentation, aiming to address the limitations due to noisy patch-wise class predictions in existing CLIP-based approaches. To this end, the authors propose the class-distribution-induced attention map, generated using the Jensen-Shannon divergence between class distributions of two patches from the same object, to enhance focus on class-relevant regions without additional training. They also introduce enhancements such as multi-scale image patches, augmented text prompts, and entropy-based background thresholding to improve the performance further. Experimental results show the effectiveness of the proposed idea, which cooperates with several state-of-the-art approaches and results in significant performance improvement. 
All reviewers appreciated the clear motivation, reasonable idea, and comprehensive analysis/experiments. The main concerns raised by reviewers were limited novelty, unclear exposition, and missing discussion/comparisons. The authors’ detailed rebuttal addressed most of them, resulting in unanimous acceptance at the end of the discussion. AC thus recommends acceptance.