SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation

Authors: Pengfei Chen, Lingxi Xie, Xinyue Huo, Xuehui Yu, XIAOPENG ZHANG, Yingfei Sun, Zhenjun Han, Qi Tian
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Versatile Segmentation, Composable Prompts, Open-vocabulary

Abstract:
The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches, and then merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities. Codes are released on https://github.com/ucas-vg/SAM-CP.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730027709259

thread_type:
official_review

<thread_object>
creator: reviewer_3kql

<content>
summary:
**Objective**

SAM-CP proposes a method to further refine the output of the Segment Anything Model (SAM). The aim is to transform SAM’s original, class-agnostic, variance-scale patches into two more informative segmentation types: Semantic Segmentation and Instance Segmentation.

**Proposed Framework**

The training framework operates as follows:

- Input: The framework takes the initial class-agnostic segments (CA-segments) from SAM, along with category text.
- Processing:
SAM Patches: The SAM patches are first processed through self-attention.
Queries: Queries (which can be target categories in text or instance queries) are processed through self-attention, then used to refine the features by performing cross-attention with the SAM patches.
Affinity Calculation: The resulting  N  patch features and  M  query features are used to calculate affinity for instance grouping.
Semantic Classification: Finally, the learned features are compared with CLIP text input features for semantic classification.

The entire pipeline is trained in a fully supervised, end-to-end manner.

**Results**

Experiments show that this framework achieves state-of-the-art (SOTA) results when trained on one dataset and tested on another.

strengths:
1. I believe this task is important. Transforming the class-agnostic SAM into a usable semantic and instance segmentation is a valuable pursuit. This directly connects to the question of how to leverage the advance of large-scale vision foundation models into practical application.

2. The proposed method demonstrates strong performance on commonly used open-world segmentation frameworks, achieving state-of-the-art (SOTA) results when trained on one dataset and tested on another. This includes benchmarks such as: COCO → ADE20K; ADE20K → COCO; COCO → Cityscapes

3. The overall experinement and results presentation is good.

weaknesses:
# At a higher level:
The advantage of SAM lies not only in its ability to segment an entire image but also in its prompt-based segmentation, which introduces more accurate masks.
I think this method uses SAM only as an initial “superpixel” generator, which somewhat overlook SAM’s full potential.

# Generalization:
The proposed method requires training on specific datasets to learn how to merge patches, which seems to be not very generalizable.
This approach may not translate well from natural images to medical images, or even from natural images with large object segments to tasks requiring finer, more detailed segmentation. Is this the case in this setting. Would like to hear author feedback on this point. 

# Splitting and Merging:
The proposed method only focuses on merging, which to some degree simplifies the problem. I believe splitting is equally important as merging, though it seems it is not within the scope of this method. 

# Comparison with Other Methods:
Most of the results are compared with pre-SAM methods. I wonder how this method compares with other SAM-based semantic segmentation/instance segmentation methods, such as Grounding SAM and SEEM etc. It would be interesting to see a comparison with other SAM-based methods and discuss the advantage of such approach.

questions:
1. At line 150, the authors state that the two prompts can be iteratively called when finer-level segmentation is required. I’m not entirely sure I understand this; my impression is that granularity is determined by SAM’s results. Could the authors clarify this point?

2. How do the authors view this approach in comparison to SAM-based generalized segmentation? Most other approaches appear to be training-free and straightforward to use. It would be interesting to discuss the advantages that SAM-CP offers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730110629612

thread_type:
official_review

<thread_object>
creator: reviewer_pkrw

<content>
summary:
The paper presents SAM-CP that extends SAM by incorporating composable prompts to improve segmentation. SAM-CP utilizes Type-I prompts to align SAM-generated patches with text labels and Type-II prompts to merge patches belonging to the same instance based on the text information. Experimental results show that SAM-CP performs well in open-vocabulary segmentation and shows some improvement in semantic discriminativity across various segmentation types.

strengths:
1. This work presents a method for improving SAM for versatile segmentation.

2. SAM-CP considers segmentation tasks from a novel perspective, introducing two types of composable prompts: Type-I for matching SAM patches with text labels and Type-II for merging patches belonging to the same instance.

3. The dynamic mechanism offers novel design insights for handling cross-attention computations.

4. Generally, the paper is well-written.

5. The figures and tables are informative.

weaknesses:
1. Prompt I classifies each SAM-segmented patch, potentially leading to ambiguities. For example, without contextual information, it’s challenging to determine if a patch representing clothing on a person should be classified as part of the person or as an independent clothing item. The proposed CP lacks a clear strategy to handle the inherent ambiguity in segmentation without surrounding context.

2. The performance gains are not significant, such as mIoU for the COCO->ADE20K and COCO->Cityscapes settings in Table 1.

3. Table 2's comparison may be unfair due to the use of SAM, which is trained on a significantly larger scale. This complicates attributing improvements directly to the proposed method.

4. Although SAM-CP's speed heavily relies on SAM, the paper lacks a clear analysis of the computational efficiency of the proposed non-SAM modules. It is crucial to examine whether these modules actually enhance efficiency.

5. Other Issues:
- Figure 1's many colors and mixed visual elements are hard to read.
- The description of patches needs greater specificity. For example, as vaguely mentioned in Line 139, it's not specified if the patches are segments from SAM's segment-everything mode or just features from SAM's backbone.

questions:
1. I would like to know if SAM-CP has effective strategies for addressing the aforementioned issues of segmentation ambiguity?

2. Although the speed is influenced by SAM's computational efficiency, can you provide a detailed analysis of the speed of non-SAM modules and compare it with other SAM-based methods? Alternatively, could replacing SAM with a lightweight variant like EfficientSAM provide sufficient performance and computational efficiency for comparison with non-SAM methods?

3. Does SAM-CP offer any advantages in training time compared to other SAM-based methods?

4. Since SAM's segmentation results may not always be precise, the Affinity matrix used by the Unified Affinity Decoder could potentially accumulate errors. How is this issue addressed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730515113362

thread_type:
official_review

<thread_object>
creator: reviewer_suuh

<content>
summary:
This paper proposes a framework that combines SAM and CLIP for open-vocabulary universal segmentation.

strengths:
1. The proposed model keeps the zero-shot abilities of CLIP. This is supported by experiments on open-vocabulary segmentations.
2. The ablation studies are sufficient and well explain the design choice of loss functions.
3. This method is easy to follow, with many intuitive figures and visualizations.

weaknesses:
1. The experiments on open-vocabulary segmentations (Table 1) are not well supportive. SAM-CP combines SAM and CLIP but is only comparable to FC-CLIP that is built with CLIP only. Also, the experiments on closed-vocabulary segmentations (Table 2) do not best address the potential of combining SAM and CLIP since it is worse than CLIP-only X-Decoder and supervised SOTA MaskDINO.
2. The advantages of the proposed framework are not well discussed or presented. The performances are only comparable to previous SOTA. This paper should look for more perspectives to highlight its advantages, e.g. efficiency, part segmentation (also see Question 1), few-shot capabilities, out-of-domain generalizations, etc. 
3. It could be more convincing if this paper can compare to other SAM-based methods on the same settings and show its advantages. For example, [1] and [2] also work on open-vocabulary segmentation with SAM.
4. As already discussed and illustrated in the paper, SAM-CP fails at small objects since there are no prompts on small objects. The efficiency is limited by SAM.

[1] Open-Vocabulary SAM. ECCV 2024

[2] Segment Everything Everywhere All at Once. NeurIPS 2023

questions:
1. SAM can generate fine-grained masks for part, e.g. head, hand, body. Does SAM-CP have the potential to work on open-vocabulary part segmentation? Quantitative experiments and comparisons to related works like HIPIE would enhance the applicability of this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691299784

thread_type:
official_review

<thread_object>
creator: reviewer_xyh9

<content>
summary:
The paper introduces SAM-CP, which integrates composable prompts for versatile segmentation. It utilizes two levels of queries—semantic and instance—to assign unlabeled patches from SAM with appropriate labels.

In rebuttal, the authors have presented additional experimental results that address many of my concerns. As a result, I am inclined to increase my rating to borderline accept.

strengths:
- The overall architecture designs are well-grounded. The two types of prompts for semantic and instance understanding are intuitively designed to support different segmentation tasks.

- SAM-CP models the two types of prompts with separated queries for better efficiency. The quantitative results on open-vocabulary segmentation also demonstrate the effectiveness of this query-based approach.

- Thorough analysis is provided for the advantages and limitations of the proposed method.

weaknesses:
- The running time might be a potential issue due to the two-level prompt design (especially if an iterative approach is used). A comparison of running time performance with the original SAM and other segmentation models would be informative.

- Experiments: The current baselines do not include models based on SAM. Therefore, it's unclear whether the performance improvements come from the composable prompts or the internal capacity of SAM. Adding baselines building on SAM that perform instance [1] and semantic [2] segmentation can be incorporated to further demonstrate the effectiveness of the proposed composable prompts.

- The figures are low-resolution and notably blurry. Replacing them with high-resolution versions would improve readability and presentation quality.

- The performance of SAM-CP on closed-domain segmentation is not superior over other baselines.

[1] Prompt Segment Anything (https://github.com/RockeyCoss/Prompt-Segment-Anything)
[2] Semantic Segment Anything (https://github.com/fudan-zvg/Semantic-Segment-Anything)

questions:
The design of SAM-CP relies solely on SAM's final predictions, disregarding intermediate features from SAM that could potentially enhance the decoder's performance. It would be valuable to include discussions on this aspect and to investigate whether integrating SAM's intermediate features with the proposed decoder could yield improved results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732449752112

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thanks to the reviewers for reading our paper and sharing valuable comments with us.

In this paper, we presented a versatile segmentation approach, SAM-CP, that designs two composable prompts upon the vision foundation model, SAM. By composing the prompts, one can perform semantic, instance, and panoptic segmentation. We developed a sophisticated and efficient training method and showed the ability of SAM-CP in both open-domain and closed-domain segmentation tasks.

In the original review, reviewers are generally content with the submission. The strengths are summarized as follows.
1. The overall architecture designs are well-grounded (**xYh9**). The paper is well-written (**pKRW**)
2.  SAM-CP considers segmentation tasks from a novel perspective(**pKRW**). Transforming the class-agnostic SAM into a usable semantic and instance segmentation is a valuable pursuit(**3KqL**), directly connecting to the question of how to leverage the advance of large-scale vision foundation models into practical application.  This task is important(**3KqL**). 
3.  The method is is easy to follow(**SuUH**), with many intuitive figures, tabels and visualizations(**SuUH**, **pKRW**, **3KqL**). The two types of prompts for semantic and instance understanding are intuitively designed (**xYh9**) to support different segmentation tasks. The dynamic mechanism offers novel design insights for handling cross-attention computations(**pKRW**). 
4. The quantitative results on open-vocabulary segmentation also demonstrate the effectiveness of this query-based approach(**xYh9**, **SuUH**) and achieve state-of-the-art performance (**3KqL**) under multiple datesets. The overall experiment and results presentation is good(**3KqL**). The ablation studies are sufficient and well explain the design choice of loss functions(**SuUH**).
5. Thorough analysis is provided for the advantages and limitations of the proposed method.(**xYh9**)


 In the rebuttal to each reviewer, we responded to their concerns with texts and supplementary experiments. The pictures are in the main PDF (rebuttal section, page 24-25).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733297828603

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the rebuttal and author-reviewer discussion

comment:
Dear PC, SAC, AC, and Reviewers,

Thanks again for your time in handling our submission. During the past weeks, we had thorough discussions with the reviewers and they also gave us valuable feedback that will further improve our work. After the rebuttal:

* Reviewer **pKRW** said that *most of his/her concerns have been resolved* and decided to keep the original rating (6).
* Reviewers **SuUH** said that *the additional experiments and clarifications have addressed most of his/her concerns* and decided to raise the score to 6.
* Reviewer **3KqL** said that *the detailed response* has *addressed most of his/her concerns* and decided to raise the score to 6. Specifically, he/she was satisfied with our approach in (1) *generalizing across different granularities when the model is jointly trained on part-level and general datasets* and (2) *finer segmentation can be achieved by interactively using two types of prompts in combination with SAM*.

In summary, all three reviewers who responded after the rebuttal **converged to a score of 6**. It is a pity that Reviewer **xYh9** did not respond to our rebuttal. Some of his/her questions were also raised by other reviewers and have already been dealt with and acknowledged by other reviewers during the rebuttal process.  Should he/she have further comments during the AC-reviewer discussion phase, we are more than happy to accept the suggestions and revise our paper accordingly.

We sincerely thank you for your time and advice and look forward to sharing our work with the community.

Best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents a simple approach (SAM-CP) to establish two types of composable prompts beyond SAM and then compose them for versatile segmentation.