SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation

## Abstract

Abstract The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP 1 1 1 This work was done when Pengfei Chen ¡chenpengfei20@mails.ucas.ac.cn¿
was an intern at Huawei Inc. * Corresponding author: Zhenjun Han ¡hanzhj@ucas.ac.cn¿. , a simple approach that establishes two types of c omposable p rompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches and merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities.

## 1 Introduction

The past decade has witnessed a rapid development of vision-aware foundation models [, , , , ]. These models apply to a series of visual recognition tasks and serve as a building block for multimodal (e.g., vision-language) understanding. Recently, a powerful foundation model named the Segment Anything model (SAM) [] has attracted a lot of attention. Pre-trained on a large corpus of images, SAM shows an impressive ability to group image pixels into patches and generalizes across various vision domains (e.g., medical images [, , , ], camouflaged images [, ], thermal images [], etc.) as well as different downstream scenarios, (e.g., image editing [, ], 3D recognition [, , ], object tracking [, ], etc.).

Despite its success, there still exist major challenges in applying SAM to semantic-aware segmentation tasks including semantic, instance, or panoptic segmentation. We notice two lines of research in this direction. The first line (e.g., Grounded-SAM []) heavily relied on a standalone model (e.g., DINO [] or Grounding-DINO []) to generate proposals and SAM was only used for refinement. This weakens the function of SAM as a foundation model. The second line (e.g., SSAM [], Semantic-SAM [], SAM-CLIP []) tried to assign a semantic label to each patch produced by SAM. However, in many scenarios, SAM may over-segment an instance into sub-patches, making it difficult to determine which patches belong to the same instance.

Figure: Figure 1: An illustration of how SAM-CP works at the idea level. Given an image and the patches produced by SAM, we first execute Prompt I to find the patches corresponding to any text label (in either closed or open domains), and then, if necessary, execute Prompt II to group the patches within each class into instances. In the upper part, the height of each bar corresponds to the probability that a patch belongs to a text label; in the lower part, two patches are connected by a solid line if they belong to the same instance. This figure is best viewed in color.
Refer to caption: x1.png

This paper presents a novel approach named SAM-CP where ‘CP’ stands for composable prompts. Different from the existing methods, we establish two types of prompts beyond the patches produced by SAM. The idea is illustrated in Figure 1. When a semantic class (in text) is given, the model needs to determine: (i) Prompt I: whether a SAM patch aligns with the text label, and (ii) Prompt II: whether two patches belong to the same instance of the corresponding category. Once the model learns to process these two prompts, one can apply a simple traversal algorithm over the SAM patches for semantic segmentation (based on the first prompt) and instance segmentation (adding the second prompt), and further compose them for panoptic segmentation.

A naive implementation of SAM-CP suffers super-linear computational complexity in enumerating class-patch pairs (Prompt I) and patch-patch pairs (Prompt II). To accelerate it, we establish a unified affinity framework, as shown in Figure 2. It involves a query-based mechanism where two types of queries (semantic and instance) are established, and the features extracted from SAM patches are taken as keys. Both the queries and keys are fed into a vision transformer; throughout forward propagation, each query merges the keys with high affinity with it. At the end of affinity propagation, the keys contained in each query form the desired entity (a semantic region or an instance). This implementation allows us to carry out the training procedure efficiently in GPUs.

We train SAM-CP on COCO [] and ADE20K [] and evaluate it on these two datasets as well as Cityscapes [] for both open-vocabulary and closed-domain segmentation. Since SAM-CP is trained to understand text labels, it can easily adapt to unseen classes with a CLIP [] text encoder. Extensive experiments demonstrate SAM-CP’s ability to cover semantic, instance, and panoptic segmentation with a single model. In particular, it reports state-of-the-art accuracy in open-vocabulary segmentation. Qualitative studies demonstrate that SAM-CP improves the semantic discriminativity of SAM’s features. Our research offers a new methodology to equip vision foundation models (e.g., SAM) with a solid and flexible ability for semantic recognition. We expect the proposed approach to gain stronger and more versatile abilities (e.g., for part segmentation) in the future, with the vision foundation models being upgraded and becoming more robust.

## 2 Related Work

In recent years, both the CV and MLP communities have witnessed a rapid development of foundation models. In particular, the vision foundation models have largely evolved from being simply pre-trained for image classification [, , ] to incorporating multimodal information [, , , ] and/or being pre-trained to deal with different tasks [, , ]. They improve the accuracy of various downstream visual recognition tasks including detection, segmentation, etc.

Recently, SAM [] appeared as a foundation model for versatile segmentation. Pre-trained on a large corpus with billions of instances, SAM can segment an image into a set of basic patches. An important advantage of SAM lies in its ability to recognize images in different domains, yet a disadvantage lies in the lack of semantic labels on each patch. The community has been trying to adapt SAM to various scenarios, including adapting it on other image data (e.g., medical images [, , , ], camouflaged images [, ], remote sensing images [, , ], etc.), raising it to segment 3D objects [, , ], and using it as the pre-processing step of image editing [, ]. Among these efforts, one of the most challenging topics is to assign the SAM patches with semantic labels. Existing works involved calling other foundation models (e.g., CLIP []) for image tagging [, , ], applying SAM as a refinement stage after other detection and/or segmentation models [], and generating other variants. However, in many scenarios, SAM may over-segment basic semantic units into sub-patches, which increases our burden of segmentation for specific purposes.

This paper focuses on establishing basic prompts beyond the segmentation results of SAM for versatile segmentation. This is related to a series of query-based algorithms for visual recognition, such as DETR [] and the subsequent variants [, , , , ]. Meanwhile, we compute the affinity to determine the relationship between semantic units (e.g., queries and objects), which is related to a few prior work [, , ] that tried to compute the affinity between pixels and objects to perform segmentation. The idea was also inspired by ViRReq [], a recent work that proposed decomposing complex visual recognition tasks into elementary units to ease annotation and optimization. Further, motivated by the previous open-vocabulary panoptic segmentation methods [, , ], we leverage a CLIP-based classifier to equip the model with an open-domain recognition ability.

## 3 Our Approach

### 3.1 Overview: Composite Prompts for Segmentation

The overall design of our approach is illustrated in Figure 1. The core idea is to establish two types of prompts beyond SAM [], a recent vision foundation model that extracts patches from an input image as potential instances. By composing the output of different prompts, the SAM patches are labelled and/or combined into semantic regions and/or instances, and thus versatile segmentation tasks can be performed. We name the approach SAM-CP, where ‘CP’ stands for composable prompts.

Mathematically, let the input image be $\mathbf{I}$. SAM extracts a number of patches, $\mathcal{P}=\{\mathbf{P}_{1},\mathbf{P}_{2},\ldots,\mathbf{P}_{N}\}$, where $N$ is the number of patches and $\mathbf{P}_{n}$ is the $n$-th patch which is represented as a binary mask of the same shape as the input image. Although SAM is robust across various vision domains, it does not offer a semantic label to each patch and, sometimes, an instance (e.g., a person) may be over-segmented into multiple patches. We design the following two types of prompts.

- •
Prompt I – semantic labeling. Given a text label $\mathbf{T}$ and one patch $\mathbf{P}$, judge if $\mathbf{P}$ can be classified as $\mathbf{T}$.
- •
Prompt II – instance merging. Given a text label $\mathbf{T}$ and two patches $\mathbf{P}_{1}$ and $\mathbf{P}_{2}$ classified as $\mathbf{T}$, judge if $\mathbf{P}_{1}$ and $\mathbf{P}_{2}$ belong to the same instance of $\mathbf{T}$.

A wide range of segmentation tasks can be accomplished by composing the above two prompts. Firstly, note that the regular semantic segmentation only involves Prompt I (assigning a label to each patch), and instance segmentation is enabled by adding Prompt II (merging over-segmented patches into an instance). Additionally, the two prompts can be iteratively called when segmentation is required at a finer level, e.g., Prompt I for classifying a region into sub-classes, Prompt II for segmenting an instance into parts, etc.

Figure: Figure 2: The unified affinity framework as an efficient implementation of SAM-CP. The input image with SAM patches is fed into a patch encoder. Type-I and Type-II prompts appear as two sets of queries. Affinity values are computed and the SAM patches are merged according to the affinity values. Semantic and instance level supervision are added to the merged patches. The purple arrows are present only in the inference stage of open-vocabulary segmentation. Best viewed in color.
Refer to caption: x2.png

### 3.2 Efficient Training with a Unified Affinity Framework

A straightforward implementation of SAM-CP involves executing Prompt I for each patch and then Prompt II for each pair of patches, following which the patches belonging to the same instance can be merged. However, this naive pipeline incurs unsatisfying efficiency because the number of Prompt II to be executed is $O(N^{2})$ where $N$ can be hundreds for a regular image. Moreover, the merging procedure requires serial operation and can inevitably encounter conflict in the inference stage (e.g., $\mathbf{P}_{1}$ and $\mathbf{P}_{2}$, $\mathbf{P}_{1}$ and $\mathbf{P}_{3}$ are considered to be the same instance but $\mathbf{P}_{2}$ and $\mathbf{P}_{3}$ are not).

To accelerate the procedure, we design an equivalent but more efficient mechanism named the unified affinity framework. We initialize a set of queries for potential units (i.e., semantic regions and instances) and set all SAM patches as keys. We perform affinity propagation between the queries and keys, gradually merging units with high-affinity scores into a larger unit. With this modified mechanism, the over-segmented patches are merged on the fly and no further post-processing is required (i.e., at the end of affinity propagation, the survived units naturally form the output). We illustrate the procedure in Figure 2 and elaborate on the individual modules as follows.

#### 3.2.1 Patch Encoder

We first extract visual features from the patches. For each patch $\mathbf{P}_{n}$, we apply a regular backbone (e.g., ResNet50 [] or Swin-L []) equipped with an RoIAlign [] operator to obtain a basic feature vector $\tilde{\mathbf{f}}_{n}$. We also design a MaskRoI operator to extract more accurate visual features by masking out the background areas. All these features are propagated through a multi-layer perceptron (MLP) and fed into $\omega$ multi-head self-attention layers, where $\omega$ is set to 6 throughout this paper. The feature vector corresponding to $\mathbf{P}_{n}$ is denoted as $\mathbf{f}_{n}$.

#### 3.2.2 Unified Affinity Decoder

This is the core module that performs affinity propagation and merges patches into super-patches. We will elaborate on three key elements below, namely, (1) a set of semantic and instance queries, (2) the algorithm of affinity propagation, and (3) the label assignment mechanism (Section 3.2.3).

Queries.
The queries are of a similar form as in DETR []. Differently, we establish two types of queries for semantic and instance segmentation, respectively. (1) For each text label $\mathbf{T}_{c}$ where $c\in\{1,2,\ldots,C\}$ is the class index, we use the language branch of a vision-language model (e.g., CLIP []) to convert it into a query vector, $\mathbf{e}_{c}^{\mathrm{S}}$, where the superscript ‘$\mathrm{S}$’ stands for ‘semantic’. (2) We also create $N$ instance queries (i.e., assuming that each patch may correspond to an instance, abbr. patch-as-query, PasQ) and initialize them using the visual features and position embeddings of the patches, i.e., $\mathbf{e}_{n}^{\mathrm{I}}\equiv\mathbf{f}_{n}$, where the superscript ‘$\mathrm{I}$’ stands for ‘instance’. In what follows, we denote both types of queries as $\mathbf{Q}_{m}$, $m=1,2,\ldots,M$ where $M$ is the number of queries.

Affinity.
The affinity is mathematically defined as a matrix $\mathbf{A}$ sized $M\times N$. Each entry of $\mathbf{A}$, $A_{m,n}$, denotes the probability that the patch $\mathbf{P}_{n}$ belongs to the query $\mathbf{Q}_{m}$. Initially, we set all entries of $\mathbf{A}$ to $1$. Then, in each affinity propagation layer (please see below for details), the query vectors (denoted as Q) and the patch features (denoted as K and V) are fed into a multi-head cross-attention module to update the query vectors for subsequent classification. There are three key modules here. (1) The affinity matrix $\mathbf{A}$ serves as a dynamic mask after binary operation in cross-attention, which we call dynamic cross-attention (DCA), to extract the feature from high-affinity patches. (2) We insert a module named affinity refinement (AR) to update the affinity matrix $\mathbf{A}$ using the cosine similarity between Q and K. (3) To enhance the query feature, we apply a Query Enhancement (QE) mechanism to fuse the query’s feature with the RoI features of its high-affinity regions. The details of DCA, AR, and QE are described in Appendix A.1. As shown in the ablation (Section 4.4), all these designs contribute to the segmentation accuracy.

#### 3.2.3 Label Assignment and Supervision

Each query, regardless of its type (semantic or instance), is expected to occupy a set of (one or more) patches and be assigned a class label. So, two sources of supervision are required, which come from the semantic labels and instance IDs, respectively.
Each query is supervised by both signals.

Semantic-level supervision.
We first build a vision-language classifier upon the semantic queries. Following GLIP [], the score $S_{m,c}^{\mathrm{cls}}$ of the $m$-th query at the $c$-th class is determined by $\hat{\mathbf{Q}}_{m}$ and $\hat{\mathbf{e}}_{c}$, the quantities produced by linearly normalizing $\mathbf{Q}_{m}$ and $\mathbf{e}_{c}$ into $[0,1]$. The classification loss [] $\mathcal{L}_{\mathrm{cls}}$ is computed upon $S_{m,c}$ in Equation (1).

$$ $S_{m,c}^{\mathrm{cls}}=\frac{1}{s}\cdot\hat{\mathbf{Q}}_{m}^{\top}\cdot\hat{ \mathbf{e}}_{c}+b,\quad\mathcal{L}_{\mathrm{cls}}=\frac{1}{M}\sum_{c=1}^{C} \sum_{m=1}^{M}\mathrm{FL}\big{(}\sigma(S_{m,c}^{\mathrm{cls}}),\mathbb{I}_{[c_ {m}^{\star}=c]}\big{)}.$ (1) $$

In $S_{m,c}^{\mathrm{cls}}$, $s$ is a learnable scaling factor, and $b$ is a bias parameter. In $\mathcal{L}_{\mathrm{cls}}$, $\mathrm{FL}(\cdot,\cdot)$ is the focal loss, $\sigma(\cdot)$ is the sigmoid activation function, $c_{m}^{\star}$ is the ground-truth class label for the $m$-th query (we explain how to compute $c_{m}^{\star}$ in the supplementary material), and $\mathbb{I}_{[\cdot]}$ the indicator function which takes $1$ if the statement is true and 0 0 otherwise.

Instance-level supervision.
At the end of affinity propagation, each instance query corresponds to a binary segmentation mask. Let the ground truth contain $K$ instances. We first establish a matching matrix $\mathbf{G}$ (sized $K\times N$) by computing the box-level IoP (intersection-over-patch) and mask-level IoP values between each pair of predicted and true instances; they are considered matched if both IoP values are greater than a pre-defined hyper-parameter $\tau$, i.e., $G_{k,n}=\mathbb{I}_{[\min(\mathrm{IoP}_{\mathrm{box}},\mathrm{IoP}_{\mathrm{
mask}})>\tau]}$, where $\tau=0.8$ throughout this paper. If no patches are assigned to an object, the patches with an IoU value of at least $0.5$ will be chosen as candidates of low-quality matching. Based on the matching matrix $\mathbf{G}$, we compute the ground-truth affinity matrix $\mathbf{B}$ (sized $M\times N$, same as $\mathbf{A}$). For each $m$, we first determine if any ground-truth instance (indexed $k_{m}$) matches the $m$-th query (see the next part for details). If yes, $\mathbf{B}_{m}=\mathbf{G}_{k_{m}}$ (i.e., the $k_{m}$-th row of $\mathbf{G}$ is copied to the $m$-th row of $\mathbf{B}$); otherwise, $\mathbf{B}_{m}\equiv\mathbf{0}$. Then, following [, ], we compute the mask focal loss $\mathcal{L}_{\mathrm{mfl}}$ and the Dice loss $\mathcal{L}_{\mathrm{dice}}$:

$$ $\mathcal{L}_{\mathrm{mfl}}=\frac{1}{M^{\ast}}\sum_{m=1}^{M}\frac{\varepsilon_{ m}}{\max(|\mathbf{B}_{m}|_{0},1)}\cdot\sum_{n=1}^{N}\mathrm{FL}(A_{m,n},B_{m,n }),\quad\mathcal{L}_{\mathrm{dice}}=\frac{1}{M^{\ast}}\sum_{m=1}^{M} \varepsilon_{m}\cdot\mathrm{Dice}(\mathbf{A}_{m},\mathbf{B}_{m}),\\$ (2) $$

where $\mathrm{FL}(\cdot,\cdot)$ and $\mathrm{Dice}(\cdot,\cdot)$ are the focal loss and Dice loss, respectively, $M^{\ast}$ is the number of positive query embeddings in the image, $|\mathbf{B}_{m}|_{0}$ is the number of non-zero entries in $\mathbf{B}_{m}$ (i.e., the number of patches that are assigned to the $m$-th query), and $\varepsilon_{m}\in\{0,1\}$ is a weight indicating whether to take the $m$-th query into consideration.

Determining $\mathbf{B}_{m}$ and $\varepsilon_{m}$ for each query. This procedure is different between the types of queries.

- •
For a semantic (Type-I) query, $\mathbf{Q}_{c}$ ($c\in\{1,2,\ldots,C\}$), we first check if the $c$-th class appears in the image. If not, we have $\mathbf{B}_{c}\equiv\mathbf{0}$ and $\varepsilon_{c}=0$. Otherwise, the $c$-th class (as a unique semantic region) must appear in the ground-truth ‘instance’ set; let the index be $k_{c}$, thus $\mathbf{B}_{c}=\mathbf{G}_{k_{c}}$. We set $\varepsilon_{m}$ to $1$ for the positive embeddings and 0 0 otherwise.
- •
For an instance (Type-II) queries, we follow the DETR series [, , ] to apply the Hungarian algorithm to find the best matches between the queries and the ground-truth instances. Differently, we compute the matching cost using more metrics, including the classification loss (cls), the mask focal loss (mfl), the Dice loss (dice), the bounding-box cost (bbox), and the gIoU cost (giou). We will show in experiments that all these components contribute to better segmentation results. After the matching is done, we obtain the index $k_{m}$ for the $m$-th query ($k_{m}$ can be null, in which situation the query is ignored), and assign $\mathbf{B}_{m}=\mathbf{G}_{k_{m}}$. We set $\varepsilon_{m}$ to $1$ for the positive embeddings and 0 0 otherwise.

The overall loss function. The overall loss is defined as $\mathcal{L}_{\mathrm{all}}=\lambda_{\mathrm{cls}}\cdot\mathcal{L}_{\mathrm{cls
}}+\lambda_{\mathrm{mfl}}\cdot\mathcal{L}_{\mathrm{mfl}}+\lambda_{\mathrm{dice
}}\cdot\mathcal{L}_{\mathrm{dice}}$, where the loss coefficients are $\lambda_{\mathrm{cls}}=2$, $\lambda_{\mathrm{mfl}}=1$, and $\lambda_{\mathrm{dice}}=1$. We apply the denoising strategy in DINO [] to improve the training performance. The details are provided in Appendix A.3.

### 3.3 Inference

The inference procedure varies slightly between closed-set and open-vocabulary segmentation. In a closed domain, the logit $\mathbf{S}^{\mathrm{cls}}$ in Equation (1) is used for classification. The normalized and quantized affinity matrix $\mathbf{A}$ is used for patch merging. For semantic segmentation, we refer to the rows corresponding to $C$ classes and merge all patches surpassing a pre-defined threshold. For instance segmentation, we look up the non-empty instance rows of $\mathbf{A}$, each of which corresponds to an instance. Panoptic segmentation is achieved by combining semantic and instance segmentation results.
In an open domain, we complement the logit matrix $\mathbf{S}^{\mathrm{cls}}$ (sized $M\times C$) with a CLIP-based logit matrix, $\mathbf{S}^{\mathrm{CLIP}}$. The remaining part is the same as in a closed domain. To calculate $\mathrm{S}^{\mathrm{CLIP}}$, we follow FC-CLIP [] to extract CLIP features using mask pooling on the predicted masks. Then, the feature $\mathbf{S}^{\mathrm{CLIP}}$ is obtained by calculating the similarity between the CLIP feature and $\hat{\mathbf{e}}_{c}$. The final class score is computed as $\mathbf{S}^{\mathrm{ov}}=(\sigma(\mathbf{S}^{\mathrm{cls}}))^{(1-\kappa)}+(
\mathrm{softmax}(\mathbf{S}^{\mathrm{CLIP}}))^{\kappa}$, where $\kappa=0.4$ is a coefficient balancing the closed-domain and open-vocabulary class scores.

## 4 Experiments

**Table 1: Accuracy (%) of Open-vocabulary panoptic segmentation (in PQ, SQ and RQ), instance segmentation (in AP) and semantic segmentation (in mIoU). CN-L means ConvNext-L.**
| Method | Backbone | COCO$\rightarrow$ADE20K | ADE20K$\rightarrow$COCO | COCO$\rightarrow$Cityscapes |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PQ | SQ | RQ | AP | mIoU | PQ | SQ | RQ | AP | mIoU | PQ | AP | mIoU |  |  |
| MaskCLIP [] | VIT-L | 15.1 | 70.5 | 19.2 | 6.0 | 23.7 | – | – | – | – | – | – | – | – |
| FreeSeg [] | VIT-B | 16.3 | 71.8 | 21.6 | 6.5 | 24.6 | 21.7 | 72.0 | 21.6 | 6.6 | 21.7 | – | – | – |
| ODISE [] | VIT-H | 23.3 | 74.4 | 27.9 | 13.0 | 29.2 | 25.0 | 79.4 | 30.4 | – | – | 23.9 | – | – |
| OPSNet [] | VIT-L | 19.0 | 52.4 | 23.0 | – | – | – | – | – | – | – | 41.5 | – | – |
| MaskQCLIP [] | VIT-L | 23.3 | – | – | – | 30.4 | – | – | – | – | – | – | – | – |
| X-Decoder [] | Focal-L | 21.8 | – | – | 13.1 | 29.6 | – | – | – | – | – | 38.1 | 24.9 | 52.0 |
| FCCLIP [] | CN-L | 26.8 | 71.5 | 32.3 | 16.8 | 34.1 | 27.0 | 78.0 | 32.9 | – | – | 44.0 | 26.8 | 56.2 |
| SAM-CP | CN-L | 27.2 | 77.7 | 32.9 | 17.0 | 31.8 | 28.6 | 78.4 | 34.5 | 21.9 | 34.3 | 41.0 | 29.3 | 47.9 |

**Table 2: Accuracy (%) of panoptic segmentation (in PQ), instance segmentation (in AP), and semantic segmentation (in mIoU) on the COCO-Panoptic and ADE20K datasets. The segmentation (seg.) style ‘SAM*’ means that we use fixed segmentation results of SAM without any refinement including regression (‘reg.’) or segmentation (‘seg.’) refinement.**
| Method | Backbone | Seg. Style | COCO | ADE20K |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Epoch | PQ | AP^det | AP | mIoU | Epoch | PQ | AP^det | AP | mIoU |  |  |  |
| DETR[] | R50 | reg.+seg. | 50+25e | – | – | 31.1 | – | – | – | – | – | – |
| MaskFormer [] | R50 | seg. | 300e | 46.5 | – | 33.9 | 57.8 | 128e | 34.7 | – | – | – |
| Mask2Former [] | R50 | seg. | 50e | 51.5 | – | 41.7 | 61.7 | 128e | 39.7 | – | 26.4 | 47.7 |
| Mask2Former [] | Swin-L | seg. | 100e | 57.8 | – | 48.6 | 67.4 | 128e | 48.1 | – | 34.2 | 56.1 |
| Mask DINO [] | R50 | reg.+seg. | 50e | 53.0 | 48.8 | 44.3 | 60.6 | – | – | – | – | – |
| Mask DINO [] | Swin-L | reg.+seg. | 50e | 58.3 | 56.2 | 50.6 | 67.3 | 128e | – | – | – | 56.6 |
| X-Decoder [] | Focal-T | seg. | 50e | 52.6 | – | 41.3 | 62.4 | 128e | 41.6 | – | 27.7 | 51.0 |
| X-Decoder [] | Focal-L | seg. | 50e | 56.9 | – | 46.7 | 67.5 | 128e | 49.6 | – | 35.8 | 58.1 |
| SAM-CP | R50 | SAM* | 36e | 48.6 | 46.1 | 41.7 | 55.6 | 128e | 38.5 | 28.7 | 25.1 | 42.4 |
| SAM-CP | Swin-L | SAM* | 36e | 52.7 | 50.4 | 45.2 | 61.8 | 128e | 44.4 | 34.6 | 30.3 | 49.4 |

### 4.1 Settings

Datasets and evaluation metrics.
We train SAM-CP on the COCO-Panoptic [] and ADE20K [] datasets, and evaluate the models on either closed-domain or open-vocabulary segmentation (with cross-dataset validation and Cityscapes used as test data). COCO-Panoptic (the 2017 version) has 118K training and 5K validation images with $80$ ‘thing’ and $53$ ‘stuff’ categories. We report the instance segmentation results with the standard AP metric on the $80$ ‘thing’ categories. For semantic segmentation, we report the mIoU for all ($80+53$) categories. For panoptic segmentation, the PQ metric is computed for all categories and the ‘thing’ and ‘stuff’ subsets individually. ADE20K contains 20,210 images. We use the $150$ most common object categories, including $100$ ‘thing’ and $50$ ‘stuff’ categories. Cityscapes is a street view dataset with 8 ‘thing’ categories and 11 ‘stuff’ categories. We inherit the same metric from COCO to ADE20K and Cityscapes datasets.

Implementation details.
The proposal masks are generated by SAM with $48$ grid points along each axis of the input image. To verify our idea better, we use SAM with VIT-H to generate better patches. For open-vocabulary segmentation, we use a frozen CLIP image encoder (for a fair comparison, we use the same architecture (ConvNext-L) as the previous best method, FCCLIP []) as the backbone and equip it with FPN []. For closed-domain segmentation, we establish SAM-CP on ResNet50 (R50) [] and Swin-L []. We use the implementation of the MMDetection [] (v3.0) library. We use $8$ Tesla-V100 GPUs ($4$/$2$ images per GPU) for open-vocabulary/closed-domain experiments.
The data augmentation strategy follows the DETR series. An AdamW optimizer [] with a basic learning rate of $0.0002$. See Appendix B for further details.

### 4.2 Quantitative Results

Open-vocabulary segmentation. Results are summarized in Table 1.
In both COCO$\rightarrow$ADE20K and ADE20K$\rightarrow$COCO, SAM-CP surpasses FCCLIP [], the previous state-of-the-art method, in terms of PQ, SQ, RQ for panoptic segmentation, and AP for instance segmentation. In particular, SAM-CP not only reports competitive SQ (for high-quality segmentation), but also achieves a better tradeoff between PQ and RQ; in COCO$\rightarrow$Cityscapes, SAM-CP achieves a $40.6\%$ PQ, a $29.3\%$ AP and a $47.5\%$ mIoU for versatile segmentation, which brings the best Instance Segmentation performance. We owe the excellent results to the efficient mechanism that combines SAM and CLIP, two open-world foundation models for open-vocabulary segmentation.

Closed-domain segmentation.
Results are summarized in Table 2. In COCO-Panoptic, with the ResNet-50 backbone, SAM-CP achieves a $48.6\%$ PQ, $41.7\%$ AP, and $55.6\%$ mIoU; with a stronger backbone, Swin-L, SAM-CP reports higher segmentation accuracy, with a $52.7\%$ PQ, $45.2\%$ AP, and $61.7\%$ mIoU. In ADE20K, SAM-CP achieves a $38.5\%$ PQ, a $25.1\%$ AP, and a $42.4\%$ mIoU on ResNet-50, and a $44.4\%$ PQ, a $30.3\%$ AP, and a $49.4\%$ mIoU on Swin-L. An interesting comparison comes from MaskFormer [] where SAM-CP reports higher PQ and AP but a lower mIoU, which implies its advantageous performance in instance-level recognition. These numbers demonstrate the effectiveness of our methodology, i.e., establishing composable prompts beyond vision foundation models. We will delve into the limitation of SAM-CP in Section 4.5 and explain why this novel mechanism falls short in closed-domain segmentation.

Summary. As a new methodology for versatile segmentation, SAM-CP shows a unified pipeline and promising performance over three popular benchmarks. In particular, SAM-CP demonstrates state-of-the-art performance in the open domain. We look forward to the future when stronger foundation models are available and further boost the accuracy of SAM-CP.

Figure: Figure 3: A qualitative study of how SAM-CP works. Each row displays an example. The leftmost column shows the input image with SAM patches; the middle and right parts show the semantic and instance segmentation results, respectively. We use the t-SNE algorithm to project the learned visual features (by SAM-CP; please refer to Figure 4 for the difference from the features of SAM) in a 2D coordinate system. The points with the same color belong to the same semantic class or the same instance (according to the ground truth). This figure is best viewed in color.
Refer to caption: x3.png

Figure: Figure 4: The t-SNE visualization upon the visual features of SAM and SAM-CP. Due to the limited space, only semantic segmentation results are displayed. The points with the same color belong to the same semantic class (according to the ground truth). This figure is best viewed in color.
Refer to caption: x4.png

### 4.3 Qualitative Studies

We show that SAM-CP learns discriminative visual features beyond SAM. We first show how SAM-CP accomplishes the entire segmentation procedure in Figure 3. From the t-SNE visualization maps, one can see that the features extracted from the SAM patches form clusters that correspond to different semantic classes. Additionally, when instance segmentation is required, the specific cluster can be further partitioned into sub-clusters that correspond to different instances. This aligns with the high-level idea shown in Figure 1, and SAM-CP accomplishes the goal efficiently.

We further compare the visual features learned by SAM and SAM-CP in Figure 4. Not surprisingly, the features extracted by SAM are not semantically discriminative, with samples from different semantic classes overlaying in feature space. Such features are clearly improper for visual recognition purposes. SAM-CP shows much better discriminativity, aligning with the display in Figure 3.

### 4.4 Ablative Studies

We study the effectiveness of the design principles and individual modules via ablative studies. In the open vocabulary, we report the results of COCO$\rightarrow$ADE20K with a frozen CLIP encoder (ConvNext-L) and a $1\times$ schedule (i.e., $12$ epochs). In the closed domain, experiments are performed on the COCO dataset using a ResNet-50 backbone with a $1\times$ schedule.

**Table 3: Accuracy (%) in open and closed domains with different loss terms and matching strategies.**
| Loss | Label Assignment | Closed-domain (COCO) | Open-domain (COCO$\rightarrow$ADE20K) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PQ | AP^det | AP | mIoU | PQ | SQ | RQ | AP | mIoU |  |  |
| all | all | 47.0 | 45.8 | 41.4 | 54.2 | 27.2 | 77.7 | 32.9 | 17.0 | 31.8 |
| w/o $\mathcal{L}_{\mathrm{mfl}}$ | w/o $\mathrm{mfl}$ | 0.0 | 3.5 | 0.0 | 0.0 | 0.6 | 22.0 | 0.9 | 0.0 | 3.4 |
| w/o $\mathcal{L}_{\mathrm{dice}}$ | w/o $\mathrm{dice}$ | 41.3 | 35.1 | 34.3 | 48.3 | 23.8 | 73.4 | 29.1 | 15.8 | 28.6 |
| all | w/o $\mathrm{mfl}$ | 42.8 | 44.0 | 39.8 | 51.4 | 26.5 | 78.2 | 32.3 | 17.2 | 31.6 |
| all | w/o $\mathrm{dice}$ | 45.3 | 44.8 | 40.6 | 53.7 | 26.6 | 76.6 | 32.4 | 16.7 | 31.5 |
| all | w/o $\mathrm{box}$ & $\mathrm{giou}$ | 45.5 | 44.0 | 40.7 | 53.9 | 25.9 | 76.1 | 31.6 | 16.4 | 30.5 |

Loss and label assignment strategy.
Table 3 shows how different loss functions and label assignment strategies impact the performance. One can see that the mask focal loss is essential, without which the model runs into failure. The dice loss also contributes, especially for instance detection and segmentation. Regarding label assignment, the experiments show clear benefits in introducing more metrics into the weight term to improve the results of bipartite matching.

**Table 4: Accuracy (%) in open and closed domains with different modules in the SAM-CP framework.**
| DCA | AR | MaskRoI | QE | BG | Closed-domain (COCO) | Open-domain (COCO$\rightarrow$ADE20K) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PQ | AP^det | AP | mIoU | PQ | SQ | RQ | AP | mIoU |  |  |  |  |  |
|  | ✓ | ✓ | ✓ | ✓ | 45.4 | 45.6 | 41.1 | 51.8 | 26.6 | 76.9 | 32.5 | 16.6 | 31.7 |
| ✓ |  | ✓ | ✓ | ✓ | 43.5 | 44.0 | 39.9 | 51.1 | 25.8 | 76.8 | 31.3 | 16.3 | 30.5 |
| ✓ | ✓ |  | ✓ | ✓ | 44.1 | 45.3 | 40.6 | 51.1 | 25.6 | 74.4 | 31.1 | 16.5 | 30.3 |
| ✓ | ✓ | ✓ |  | ✓ | 44.8 | 44.5 | 40.5 | 51.6 | 26.5 | 75.7 | 32.1 |  | 31.4 |
| ✓ | ✓ | ✓ | ✓ |  | 45.2 | 45.4 | 41.3 | 52.6 | 25.5 | 75.7 | 31.2 | 16.1 | 30.3 |
| ✓ | ✓ | ✓ | ✓ | ✓ | 47.0 | 45.8 | 41.4 | 54.2 | 27.2 | 77.7 | 32.9 | 17.0 | 31.8 |

**Table 5: Accuracy (%) on COCO (on R50, $12$ epochs) with different strategies of dynamic cross-attention (DCA), where ‘patch-level’ is the best option, ‘PE’ denotes the patch encoder.**
| Strategy | PQ | AP^det | AP | mIoU |
| --- | --- | --- | --- | --- |
| patch-level | 47.0 | 45.8 | 41.4 | 54.2 |
| patch-level, w/o PE | 45.8 | 45.0 | 40.6 | 51.7 |
| image-level | 46.2 | 45.6 | 40.9 | 52.3 |

Module-level ablation.
There are five components that are helpful to produce better segmentation results, namely, (1) the dynamic cross-attention (DCA) mechanism used for local feature extraction in the decoder, (2) the affinity refinement (AF) strategy by adding a prediction before the sigmoid function in each stage, and (3) the MaskRoI operator which masks out the background region for more accurate visual feature extraction. (4) the query enhancement (QE) which add RoI feature to query embedding. (5) the self-affinity for negative queries to keep the ’segment anyting’ ability. Table 4 summarizes the ablation on these five components. Each of them contributes individually and they combined to boost the baseline by at least $1.0\%$ in all the reported metrics.

The design of DCA.
Among the above modules, DCA requires further investigation. We report the performance of three DCA options, differing from each other in whether cross-attention is computed at the patch level or image level, and whether the patch encoder is used. As shown in Table 6, the patch-level cross-attention with a patch encoder works the best, implying that we can extract sufficient visual features from the SAM’s output which often contains hundreds of patches.

Classifier for open-vocabulary segmentation.
Table 6 shows the impact of classifiers for open-vocabulary segmentation. With a single closed-set (learnable) and CLIP (frozen) classifier, SAM-CP reports a $17.5\%$ PQ and a $16.9\%$ PQ, respectively. After the classifiers are fused, the closed-set and open-vocabulary scores are balanced, resulting in a much higher $27.2\%$ PQ. Interestingly, the closed-set and CLIP classifiers are better at instance and semantic segmentation, respectively, and the fused classifier excels in both scenarios, with a larger gain for instance segmentation.

The matching threshold and mechanism.
We study the matching threshold and mechanism described in Section 3.2.3, which is key to affinity propagation. We ablate the threshold $\tau$ and the choice of whether to use both the box-level and mask-level IoP in Table 10. We find that $\tau=0.8$ is a proper threshold.
On the other hand, removing either the box-level or mask-level IoP results in a clear accuracy drop in visual recognition, implying the importance of improving the recall of patch merging.
In addition, we have added low-quality matching, which will give an object a high IoU patch when it fails to match a positive patch, this will bring performance gain. Therefore, a more accurate mechanism may improve the overall segmentation accuracy, which we leave as future work.

Adding extra proposals. We add the proposals extracted by the pre-trained Mask DINO (MD) model into the pool of candidate patches. Details are provided in Appendix B.1. Table 10 shows improved segmentation results, inspiring us that SAM does not generate ideal patches, especially for closed-set segmentation, and we look forward to stronger vision foundation models in the future.

Figure: Figure 5: A showcase of our results on COCO for closed domain. In each group, from left to right: input, SAM patches, panoptic segmentation by SAM-CP (ours). This figure is best viewed in color.
Refer to caption: x5.png

**Table 7: Accuracy (%) on COCO (with R50, $12$ epochs) with different definitions of parts in the training stage.**
| $\tau$ | PQ | AP^det | AP | mIoU |
| --- | --- | --- | --- | --- |
| 0.9 | 43.2 | 42.8 | 38.2 | 49.9 |
| 0.8 | 45.0 | 45.4 | 40.7 | 52.6 |
| 0.7 | 43.3 | 43.3 | 39.9 | 50.3 |
| 0.5 | 42.0 | 34.3 | 37.2 | 48.9 |
| 0.8 (w/o $\mathrm{IoP}_{\mathrm{box}}$) | 42.1 | 31.7 | 35.4 | 48.4 |
| 0.8 (w/o $\mathrm{IoP}_{\mathrm{mask}}$) | 39.7 | 42.3 | 32.5 | 44.1 |
| 0.8 (w/low quality match) | 47.0 | 45.8 | 41.4 | 54.2 |

Average coefficient for open-vocabulary segmentation. In Table 10, we ablate the average coefficient $\kappa$ defined in Section 3.3 on the setting of COCO$\rightarrow$ADE20K. The results show that $0.4$ is the best option in the geometric average between the closed-set and CLIP classification scores.

### 4.5 Advantages and Limitations

SAM-CP inherits a clear advantage from SAM in the generalized ability across different visual domains. SAM-CP achieves this by decomposing low-level pixel grouping (offered by SAM) from high-level semantic recognition (offered by the composable prompts). In the meantime, the imperfection of SAM limits the segmentation accuracy of SAM-CP, especially in the closed-set vision benchmarks. We showcase this point in Table 10 where the patches generated by SAM are compared to those generated by Mask DINO [], a state-of-the-art segmentation model. We find that SAM suffers a higher missing rate (e.g., one cannot find a proposal with an IoU larger than $0.5$ for $16.7\%$ instances; the rate is not significantly smaller even if we refer to the ground-truth masks to merge some proposals) than Mask DINO (where the rate is only $10.1\%$). In other words, SAM failed to find some objects or incorrectly merged two or more objects into one patch, and SAM-CP cannot save such loss. This contributes to the deficit in the segmentation accuracy (e.g., AP or mIoU) compared to Mask DINO. We display some typical examples in Appendix A.4. Additionally, the inference speed of SAM-CP is bound by that of SAM; once a more efficient vision foundation model is available, our framework can be seamlessly transplanted and achieve faster inference.

## 5 Conclusions

In this paper, we propose SAM-CP, a novel approach that equips SAM with semantic and instance segmentation abilities. At the core of SAM lies two composable prompts, which determine (1) whether a SAM patch aligns with a text label and (2) whether two SAM patches belong to the same instance, respectively. The idea is implemented using a unified affinity framework for efficient training and inference. We show both quantitative and qualitative results with panoptic segmentation on COCO, ADE20K and Cityscapes for both open-vocabulary and closed-domain segmentation.
Our study offers a new methodology to make use of the vision foundation models such as SAM.

## Appendix A Details of Our Approach

### A.1 Affinity Computation

MaskRoI.
To extract patch-level features, we use the MaskRoI operator to mask out the background areas. Specifically, we first extract the RoI features using the RoIAlign operator on the minimum outer rectangle $\mathbf{P}^{\mathrm{rt}}_{n}$ of the patch $\mathbf{P}_{n}$. Then, we transfer the binary mask by rescaling $\mathbf{P}_{n}$ into the same size (width and height) as the RoI features (i.e., the down-sampled size of $\mathbf{P}^{\mathrm{rt}}_{n}$). Finally, the RoI features are multiplied by the region mask to obtain the MaskRoI features.

Dynamic cross-attention.
Rather than performing global computation, we use dynamic cross-attention (DCA) to guide the cross-attention operator in focusing on the local features. This is motivated by deformable attention []. The cross-attention operator follows the same structure of multi-head attention [], and the matrix $\mathbf{A}$ serves as the dynamic attention mask. We binarize $\mathbf{A}$ by setting the value of each entry to $1$ if the value is smaller than the threshold (e.g., $0.5$) which means that the area is masked and ignored in computing the cross-attention. As described in Section 3.2.2, the query vectors (as Q) and the patch features (as both K and V) are fed into a multi-head cross-attention with the dynamic attention mask.

Affinity refinement.
Affinity refinement (AF) is used to update the affinity matrix $\mathbf{A}$ in a coarse-to-fine manner. We first calculate the similarity $\hat{\mathbf{A}}$ between the query vectors (Q) and patch features (K), which will be described in Algorithm 1. The affinity matrix $\mathbf{A}$ is obtained by conducting an element-wise sigmoid activation $\sigma(\cdot)$ on $\hat{\mathbf{A}}$. For the first stage, the $\hat{\mathbf{A}}$ is obtained by calculating similarity; in the subsequent stages, the $\hat{\mathbf{A}}$ is obtained by stacking the similarity of the previous stages, which is called affinity refinement.

Query enhancement.
The query enhancement (QE) mechanism aims to fuse the query’s features with the RoI features of its high-affinity regions. Each query will predict the affinity values and merge the patches according to the values as the mask prediction. The minimum bounding rectangle of the estimated mask will be used to extract the RoI feature. In order to save GPU memory and speed up calculations, we directly merge the minimum bounding rectangles of the patches. Then, the RoI features and the updated query features (mentioned in Section A.1) are averaged as the new query features for query enhancement.

Figure: Algorithm 1 Affinity Similarity Calculation

### A.2 Supervision

Ground-truth class labels.
Let $c_{m}^{\star}$ be the ground-truth class label for the $m$-th query. For a Type-I prompt, the $m$-th query corresponds to the $c_{m}^{\star}$-th category label. Hence, the objective of the $m$-th query is set to be $c_{m}^{\star}$ if the image has the $c_{m}^{\star}$-th category label, and ‘negative’ otherwise. For a Type-II prompt, $c_{m}^{\star}$ comes from the label assignment of the Hungarian matching algorithm. Hence, the objective of the $m$-th query is set to be $c_{m}^{\star}$ if it is a positive (matched) query, and ‘negative’ otherwise.

Affinity matrix labels.
We use Figure 6 to illustrate how to obtain the affinity matrix label, $\mathbf{B}$. The left figure represents the label assignment between the queries and ground truth; the different colors indicate different ground-truth units, where ‘1’ means the query and the ground truth are matched and ‘0’ means not. In the middle figure, $\mathbf{G}$ means whether each patch is part of the ground-truth unit, where ‘1’ means yes and ‘0’ means no. The colors are consistent with those in the left figure. Based on $\mathbf{G}$, we compute the ground-truth affinity matrix $\mathbf{B}$ (sized $M\times N$, same as $\mathbf{A}$), shown in the right figure. For the $m$-th query that matches the $k$-th ground truth in label assignment (left column), the $k$-th row of $\mathbf{G}$ is copied to the $m$-th row of $\mathbf{B}$ (with the same color); otherwise, $\mathbf{B}_{m}\equiv\mathbf{0}$.

Figure: Figure 6: The illustration of how to get the ground-truth affinity matrix $\mathbf{B}$.
Refer to caption: x6.png

### A.3 The Denoising Strategy

We follow DINO [] to enhance the stability of optimization. We increase some queries generated by adding some noise for the ground-truth instances and supervise them using the corresponding ground truth in the training stage. The denoising strategy can make the training more stable because one ground truth may be matched to one query in this epoch but be matched to another in the next epoch, which is unfavourable for optimization.

For each instance, we shake the bounding box of the object as the position embedding and provide incorrect category features as the query vectors in a certain proportion, which is the same as DINO. Then, the denoising queries are concatenated to the original queries for prediction. Note that the original queries cannot see these queries in self-attention because the denoising queries will be removed in the inference stage. Then, in loss calculation, the denoising queries do not participate in label assignment and are directly supervised by the corresponding class and affinity matrix labels. We refer the readers to the original DINO paper for further details.

Figure: Figure 7: SAM-CP can group the parts to the whole for objects which do not have complete candidate patches. For the first line, SAM segment the pair of ski boards separately. But if our requirement is to treat a pair of skis as one instance, only SAM-CP can put them together. For the second line, in this complex scenario, SAM segments these fragments separately but does not provide a complete motorcycle mask. SAM-CP can stitch these fragments together as fully as possible according to the requirements.
Refer to caption: x7.png

Figure: Figure 8: The examples for the limitation of SAM patches. We have summarized the above four situations, and SAM itself has not segmented the target. Our method’s premise assumption is to trust SAM’s segmentation quality, so the above issues with SAM will limit our final performance. These limitations are also worth further investigation.
Refer to caption: x8.png

### A.4 Examples for Advantages and Limitations

Advantages.
We select the objects (labelled with yellow bounding boxes) that do not match any patches with an IoU greater than $0.5$. In Figure 7, we show examples of objects that do not have complete patches from SAM, while SAM-CP can group the parts to the whole as the object representation. The first line shows a case where SAM fails to predict the whole pair of skis because they are separated. SAM-CP can group them into one instance, which aligns with the definition in COCO. In the second line, the motorcycle is divided into many messy units in SAM, but SAM-CP can group them into a whole instance.

Limitations. We also give some examples of objects that are missed by SAM (or have a small IoU with SAM patches). We summarize four scenarios in Figure 8: (a) objects that are very small, (b) objects whose colors are similar to the background, (c) objects that are partly occluded, and (d) object which has cluttered components. Situation (a) is the most serious because the grid-based sampling of SAM is insufficient to find all small targets, and sometimes the image resolution is not high enough for precise prediction. Situation (b) can cause SAM to segment incorrect patches due to indistinguishable low-level patterns (e.g., texture, color, etc.). Situation (c) shows that SAM cannot segment well if the object is occluded by other objects, and situation (d) shows that if the object consists of too many cluttered components, SAM cannot segment them very well. This situation limits the quality of SAM patches and, consequently, results in a gap between the accuracy of SAM-CP and the state-of-the-art segmentation methods.

## Appendix B Details of Implementation.

### B.1 Settings for COCO.

Data preparation.
We use the same data augmentation strategy in DETR-based detector [, , ]. We randomly flip the image, resize the image ranging from $1333\times 480$ to $1333\times 800$, randomly crop the image, and randomly resize the image again. In the inference stage, we resize the image to $1333\times 800$ without any other augmentation. For SAM proposal generation, we use the everything mode in SAM with $48$ points per side, a $0.8$ IoU threshold, a $0.9$ stability score threshold, and a $0.7$ NMS threshold.

Settings for the model.
We use loss weights of $2.0$, $1.0$, and $1.0$ for $\mathcal{L}_{\mathrm{cls}}$, $\mathcal{L}_{\mathrm{mfl}}$, and $\mathcal{L}_{\mathrm{dice}}$, and metric weights of $2.0$, $1.0$, $1.0$, $1.0$, and $1.0$ for the cls, mfl, dice, bbox, and giou terms in the Hungarian matching algorithm. The batch size is $32$, where we use $8$ Tesla-V100 GPUs ($4$ images per GPU) for the R50 experiments and $32$ GPUs ($1$ image per GPU) for the Swin-L experiments. The learning rate is set to be $10^{-5}$ at the beginning and is multiplied by $0.1$ after the $8$/$16$/$40$-th epoch when the total number of epochs is $12$/$24$/$50$. The threshold $\tau$ in Section 3.2.3 is set to be $0.8$. The number of patch encoders and unified affinity decoders are both set to be $6$. The memory usage of each GPU is about 20G for R50 experiments, and training time is about $22$ hours for $12$ epoch experiments with $8$ GPUS. Similar computing resources are needed in further studies.

Add patches from Mask DINO.
We add the patches from Mask DINO [] to show that the upper bound of SAM patches limits our performance. If the quality of patches generated by the foundation model gets better in the future, the segmentation accuracy of SAM-CP will further increase. We saved the segmentation results of Mask DINO and deleted the label result and confidence score. Then, the segmentation results are viewed as the extra proposal and added to the SAM patches.

### B.2 Settings for ADE20K.

Data preparation. We use the same data augmentation strategy in Mask2former [] in ADE20K dataset. The image is randomly flipped and resized in the range of $[320,1280]$. Then, the image is randomly cropped by $2560\times 640$. In the inference stage, the image is resized to $2560\times 640$. The hyper-parameters for SAM patch generation are the same as those for the COCO dataset.

Setting for the model.
We use loss weights of $4.0$, $1.0$, and $1.0$ for $\mathcal{L}_{\mathrm{cls}}$, $\mathcal{L}_{\mathrm{mfl}}$, and $\mathcal{L}_{\mathrm{dice}}$, and metric weights of $4.0$, $1.0$, $1.0$, $1.0$, and $1.0$ for the cls, mfl, dice, bbox, and giou terms in the Hungarian matching algorithm. The batch size is $16$ with a basic learning rate of $2\times 10^{-4}$. The total number of iterations is $160\mathrm{K}$, and the learning rate will be multiplied by $0.1$ after $135\mathrm{K}$ iterations. Other settings remain unchanged as in the COCO experiments. The memory usage of each GPU is about 20G for R50 experiments, and training time is about 55 hours for $160\mathrm{K}$ iterations experiments ($128$ epochs) with 8 GPUS.

### B.3 Settings for Open-Vocabulary Segmentation.

Data preparation. The image pre-processing operations for training and inference on COCO and ADE20K are the same as the closed-domain experiments. When inferencing on Cityscapes, the images are resized to $2048\times 1024$. When generating SAM patches, because most of the images in Cityscapes are $2048\times 1024$, it is not suitable to resize the image to a square, so we crop the images on the long side and then feed them into SAM.

Setting for the model.
The training hyper-parameters are the same as those in close-domain experiments. Only the trainable backbone is replaced by a frozen CLIP image encoder. In the inference stage, the parameter $\kappa$ described in Section 3.3 is set as 0.4. In addition, we follow FCCLIP [] to use the prompt engineering when inference. Each label can derive multiple entries, and we chose the highest score as the classification score for this label.

### B.4 Codes and Datasets.

We provide our codes in [https://github.com/paperrepo123/SAM-CP](https://github.com/paperrepo123/SAM-CP). After the acceptance of our paper, we will make our codes publically available.

The datasets are public datasets; their links are provided here:

- •
COCO: [https://cocodataset.org/](https://cocodataset.org/)
- •
ADE20K: [http://groups.csail.mit.edu/vision/datasets/ADE20K/](http://groups.csail.mit.edu/vision/datasets/ADE20K/)
- •
Cityscapes: [https://www.cityscapes-dataset.com/](https://www.cityscapes-dataset.com/)

## Appendix C More Visualization Results

### C.1 More t-SNE Visualization

Figure 9 shows more t-SNE visualizations of our approach. The features of patches belong to different semantic/instance groups to different clusters in SAM-CP. Not that, in lines 6–10, the different instances with the same label (’train’, ’elephant’, ’person’, ’horse’, and ’umbrella’) cluster together to the same semantic area but separate to different instances in the t-SNE feature space.

### C.2 More Segmentation Examples

We show more segmentation results in Figure 10. Columns 1 and 2 are the original images and SAM patches. Columns 3 and 4 are the ground truth and prediction of instance segmentation. Columns 5 and 6 are the ground truth and prediction of panoptic segmentation.

## Appendix D Broader Impacts and Safeguards

Broader Impacts. In our view, our research may not have a negative societal impact. Firstly, the comparative methods and datasets used in our research are publicly available and do not make biased decisions that could unfairly impact specific groups. Secondly, our method does not fall under generative models; we aim to enhance the multi-task segmentation capabilities of models. There are no direct pathways for our model to be applied in any negative manner. Therefore, our research does not pose any negative societal impact.

Safeguards. We have not released any data or models with a high risk of misuse. The proposed model is trained on benchmark datasets such as COCO, ADE20K, and Cityscapes. These datasets are publicly available, widely used in the field of computer vision, and have undergone comprehensive safety risk assessments.
In this paper, we clearly specify the sources of the datasets and code used and provide appropriate links in the references section. Upon completion of the review process, we will make the code and data of this work publicly available to the community.
This research does not involve any human subjects experiments or studies, nor does it involve crowdsourced experiments or human subjects research. All experiments were conducted using code and GPU servers.

Figure: Figure 9: More visualization of a qualitative study on how SAM-CP work .
Refer to caption: x9.png

Figure: Figure 10: More visualizations of segmentation prediction. Columns 3 and 4 are the ground truth and prediction of instance segmentation. Columns 5 and 6 are the ground truth and prediction of panoptic segmentation.
Refer to caption: x10.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: The design of SAM-CP relies solely on SAM's final predictions, disregarding intermediate features from SAM that could potentially enhance the decoder's performance. It would be valuable to include discussions on this aspect and to investigate whether integrating SAM's intermediate features with the proposed decoder could yield improved results.
soundness: 3
strengths: - The overall architecture designs are well-grounded. The two types of prompts for semantic and instance understanding are intuitively designed to support different segmentation tasks.

- SAM-CP models the two types of prompts with separated queries for better efficiency. The quantitative results on open-vocabulary segmentation also demonstrate the effectiveness of this query-based approach.

- Thorough analysis is provided for the advantages and limitations of the proposed method.
summary: The paper introduces SAM-CP, which integrates composable prompts for versatile segmentation. It utilizes two levels of queries—semantic and instance—to assign unlabeled patches from SAM with appropriate labels.

In rebuttal, the authors have presented additional experimental results that address many of my concerns. As a result, I am inclined to increase my rating to borderline accept.
weaknesses: - The running time might be a potential issue due to the two-level prompt design (especially if an iterative approach is used). A comparison of running time performance with the original SAM and other segmentation models would be informative.

- Experiments: The current baselines do not include models based on SAM. Therefore, it's unclear whether the performance improvements come from the composable prompts or the internal capacity of SAM. Adding baselines building on SAM that perform instance [1] and semantic [2] segmentation can be incorporated to further demonstrate the effectiveness of the proposed composable prompts.

- The figures are low-resolution and notably blurry. Replacing them with high-resolution versions would improve readability and presentation quality.

- The performance of SAM-CP on closed-domain segmentation is not superior over other baselines.

[1] Prompt Segment Anything (https://github.com/RockeyCoss/Prompt-Segment-Anything)
[2] Semantic Segment Anything (https://github.com/fudan-zvg/Semantic-Segment-Anything)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. SAM can generate fine-grained masks for part, e.g. head, hand, body. Does SAM-CP have the potential to work on open-vocabulary part segmentation? Quantitative experiments and comparisons to related works like HIPIE would enhance the applicability of this work.
soundness: 3
strengths: 1. The proposed model keeps the zero-shot abilities of CLIP. This is supported by experiments on open-vocabulary segmentations.
2. The ablation studies are sufficient and well explain the design choice of loss functions.
3. This method is easy to follow, with many intuitive figures and visualizations.
summary: This paper proposes a framework that combines SAM and CLIP for open-vocabulary universal segmentation.
weaknesses: 1. The experiments on open-vocabulary segmentations (Table 1) are not well supportive. SAM-CP combines SAM and CLIP but is only comparable to FC-CLIP that is built with CLIP only. Also, the experiments on closed-vocabulary segmentations (Table 2) do not best address the potential of combining SAM and CLIP since it is worse than CLIP-only X-Decoder and supervised SOTA MaskDINO.
2. The advantages of the proposed framework are not well discussed or presented. The performances are only comparable to previous SOTA. This paper should look for more perspectives to highlight its advantages, e.g. efficiency, part segmentation (also see Question 1), few-shot capabilities, out-of-domain generalizations, etc. 
3. It could be more convincing if this paper can compare to other SAM-based methods on the same settings and show its advantages. For example, [1] and [2] also work on open-vocabulary segmentation with SAM.
4. As already discussed and illustrated in the paper, SAM-CP fails at small objects since there are no prompts on small objects. The efficiency is limited by SAM.

[1] Open-Vocabulary SAM. ECCV 2024

[2] Segment Everything Everywhere All at Once. NeurIPS 2023

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I would like to know if SAM-CP has effective strategies for addressing the aforementioned issues of segmentation ambiguity?

2. Although the speed is influenced by SAM's computational efficiency, can you provide a detailed analysis of the speed of non-SAM modules and compare it with other SAM-based methods? Alternatively, could replacing SAM with a lightweight variant like EfficientSAM provide sufficient performance and computational efficiency for comparison with non-SAM methods?

3. Does SAM-CP offer any advantages in training time compared to other SAM-based methods?

4. Since SAM's segmentation results may not always be precise, the Affinity matrix used by the Unified Affinity Decoder could potentially accumulate errors. How is this issue addressed?
soundness: 3
strengths: 1. This work presents a method for improving SAM for versatile segmentation.

2. SAM-CP considers segmentation tasks from a novel perspective, introducing two types of composable prompts: Type-I for matching SAM patches with text labels and Type-II for merging patches belonging to the same instance.

3. The dynamic mechanism offers novel design insights for handling cross-attention computations.

4. Generally, the paper is well-written.

5. The figures and tables are informative.
summary: The paper presents SAM-CP that extends SAM by incorporating composable prompts to improve segmentation. SAM-CP utilizes Type-I prompts to align SAM-generated patches with text labels and Type-II prompts to merge patches belonging to the same instance based on the text information. Experimental results show that SAM-CP performs well in open-vocabulary segmentation and shows some improvement in semantic discriminativity across various segmentation types.
weaknesses: 1. Prompt I classifies each SAM-segmented patch, potentially leading to ambiguities. For example, without contextual information, it’s challenging to determine if a patch representing clothing on a person should be classified as part of the person or as an independent clothing item. The proposed CP lacks a clear strategy to handle the inherent ambiguity in segmentation without surrounding context.

2. The performance gains are not significant, such as mIoU for the COCO->ADE20K and COCO->Cityscapes settings in Table 1.

3. Table 2's comparison may be unfair due to the use of SAM, which is trained on a significantly larger scale. This complicates attributing improvements directly to the proposed method.

4. Although SAM-CP's speed heavily relies on SAM, the paper lacks a clear analysis of the computational efficiency of the proposed non-SAM modules. It is crucial to examine whether these modules actually enhance efficiency.

5. Other Issues:
- Figure 1's many colors and mixed visual elements are hard to read.
- The description of patches needs greater specificity. For example, as vaguely mentioned in Line 139, it's not specified if the patches are segments from SAM's segment-everything mode or just features from SAM's backbone.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. At line 150, the authors state that the two prompts can be iteratively called when finer-level segmentation is required. I’m not entirely sure I understand this; my impression is that granularity is determined by SAM’s results. Could the authors clarify this point?

2. How do the authors view this approach in comparison to SAM-based generalized segmentation? Most other approaches appear to be training-free and straightforward to use. It would be interesting to discuss the advantages that SAM-CP offers.
soundness: 3
strengths: 1. I believe this task is important. Transforming the class-agnostic SAM into a usable semantic and instance segmentation is a valuable pursuit. This directly connects to the question of how to leverage the advance of large-scale vision foundation models into practical application.

2. The proposed method demonstrates strong performance on commonly used open-world segmentation frameworks, achieving state-of-the-art (SOTA) results when trained on one dataset and tested on another. This includes benchmarks such as: COCO → ADE20K; ADE20K → COCO; COCO → Cityscapes

3. The overall experinement and results presentation is good.
summary: **Objective**

SAM-CP proposes a method to further refine the output of the Segment Anything Model (SAM). The aim is to transform SAM’s original, class-agnostic, variance-scale patches into two more informative segmentation types: Semantic Segmentation and Instance Segmentation.

**Proposed Framework**

The training framework operates as follows:

- Input: The framework takes the initial class-agnostic segments (CA-segments) from SAM, along with category text.
- Processing:
SAM Patches: The SAM patches are first processed through self-attention.
Queries: Queries (which can be target categories in text or instance queries) are processed through self-attention, then used to refine the features by performing cross-attention with the SAM patches.
Affinity Calculation: The resulting  N  patch features and  M  query features are used to calculate affinity for instance grouping.
Semantic Classification: Finally, the learned features are compared with CLIP text input features for semantic classification.

The entire pipeline is trained in a fully supervised, end-to-end manner.

**Results**

Experiments show that this framework achieves state-of-the-art (SOTA) results when trained on one dataset and tested on another.
weaknesses: # At a higher level:
The advantage of SAM lies not only in its ability to segment an entire image but also in its prompt-based segmentation, which introduces more accurate masks.
I think this method uses SAM only as an initial “superpixel” generator, which somewhat overlook SAM’s full potential.

# Generalization:
The proposed method requires training on specific datasets to learn how to merge patches, which seems to be not very generalizable.
This approach may not translate well from natural images to medical images, or even from natural images with large object segments to tasks requiring finer, more detailed segmentation. Is this the case in this setting. Would like to hear author feedback on this point. 

# Splitting and Merging:
The proposed method only focuses on merging, which to some degree simplifies the problem. I believe splitting is equally important as merging, though it seems it is not within the scope of this method. 

# Comparison with Other Methods:
Most of the results are compared with pre-SAM methods. I wonder how this method compares with other SAM-based semantic segmentation/instance segmentation methods, such as Grounding SAM and SEEM etc. It would be interesting to see a comparison with other SAM-based methods and discuss the advantage of such approach.

Meta-review: This work presents a simple approach (SAM-CP) to establish two types of composable prompts beyond SAM and then compose them for versatile segmentation.

Decision: Accept (Poster)