SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation
Authors: Pengfei Chen
Lingxi Xie
Xinyue Huo
Xuehui Yu
XIAOPENG ZHANG
Yingfei Sun
Zhenjun Han
Qi Tian
Venue: ICLR 2025 Poster (2025)
Keywords: Versatile Segmentation
Composable Prompts
Open-vocabulary
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We design a new paradigm of vision perception: two composable prompts based on the vision foundation model SAM to achieve versatile segmentation tasks in both open and closed domains.

Abstract: The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches, and then merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities. Codes are released on https://github.com/ucas-vg/SAM-CP.

Published as a conference paper at ICLR 2025
SAM-CP: MARRYING SAM WITH COMPOSABLE PROMPTS FOR VERSATILE SEGMENTATION
Pengfei Chen 1.2-
Lingxi Xie?
Xinyue Huo? Xuehui Yu!
Xiaopeng Zhang?
Yingfei Sun' Zhenjun Han
Qi Tian
1 University of Chinese Academy of Sciences
Huawei Inc.
ABSTRACT
The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether SAM patch aligns with a text label, and the Type-Il prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches, and then merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves the state-of-the-ar performance in open- vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities. Codes are released on gi github.... com/ ucas - vvg SAM CP
INTRODUCTION
The past decade has witnessed a rapid development of vision-aware foundation models Radford eet al. (2021):[ He et al. (2022); Wang et al.] (2023c] 2021); Liu et al. (202T). These models apply series of visual recognition tasks and serve as building block for multimodal (e.g., vision- language) understanding. Recently, a powerful foundation model named the Segment Anything model (SAM) Kirillov al. (2023) has attracted lot of attention. Pre-trained on large corpus of images, SAM shows an impressive ability to group image pixels into patches and generalizes across various vision domains (e.g., medical images He et al. (2023); Ma et al. (2024); (Hu & Li (2023); Roy et al.
(2023), camouflaged images Chen et al. b): Tang et (2023b) thermal images Chen & Bail
(0023 etc.) as well as different downstream scenarios, (e.g., image editing Yu et al. (2023c); Xie
et al. 2023)) 3D recognition Shvets et al. (2024); Wang et al. (2023b); Cen et al. (2023)
object
tracking Cheng et al. (2023); Yang et al. (202)), etc.).
Despite its success, there still exist major challenges in applying SAM semantic-aware segmentation tasks including semantic, instance, or panoptic segmentation. We notice two lines of research in this direction. The first line (e.g., Grounded-SAM Ren et al. (2024)) heavily relied standalone model (e.g., DINO Zhang et al. (2023a) or Stoundding-DING Liu et al. (2024)
) to generate proposals and SAM was only used for refinement. This weakens the function of SAM
foundation model. The second line (e.g., SSAM Chen et al. (2023a), Semantic-SAM Li et al. (2023a), SAM-CLIP Wang & Vasu 2023 ) tried to assign a semantic label to each patch produced by SAM. However, in many scenarios, SAM may over-segment an instance into sub-patches, making it difficult to determine which patches belong to the same instance.
This paper presents a novel approach named SAM-CP where 'CP' stands for composable prompts. Different from the existing methods, we establish two types of prompts beyond the patches produced by SAM. The idea is illustrated in Figure]] [ When a semantic class (in text) is given, the model
"This work was done when he was an intern at Huawei Inc, chenpengfei20@mnm mails. ucas.ac.cn 1 'Correponndng Author, hanzhj @ucas.ac.en

---

Published as a conference paper at ICLR 2025
Semantic Affinity
Labels
Affinity Patches
Vision Foundation Model (e.g. SAM)
4 1
kite
(D...
0.5
sky
sky
28...
person
(B. S. 18)
Prompt
(semantic labeling)
Prompt (instance merging)
Labels
Instance
Affinity Patches
By Prreerea
person-1
person
person-2
S.
Text Label
Instance Relationship
An illustration of how SAM-CP works at the idea level. Given an image and the patches produced by SAM, we first execute Prompt I to find the patches corresponding to any text label (in either closed or open domains), and then, if necessary, execute Prompt II to group the patches within each class into instances. In the upper part, the height of each bar corresponds to the probability that a patch belongs to text label (yellow, green, blue and red for 'sand', 'kite', 'sky' and 'person'); ); in the lower part, two patches are connected by solid line if they belong to the same instance (purple for 'person-1' I' and orange for 'person-2'). This figure is best viewed color
Figure 1:
needs to determine: (i) Prompt I: whether a SAM patch aligns with the text label, and (ii) Prompt II: whether two patches belong to the same instance of the corresponding category. Once the model learns to process these two prompts, one can apply simple traversal algorithm over the SAM patches for semantic segmentation (based on the first prompt) and instance segmentation (adding the second prompt), and further compose them for panoptic segmentation.
naive implementation of SAM-CP suffers super-linear computational complexity in enumerating class-patch pairs (Prompt and patch-patch pairs (Prompt II). To accelerate it, we establish a unified affinity framework, as shown in Figure 2 It involves a query-based mechanism where two types of queries (semantic and instance) are established, and the features extracted from SAM patches are taken as keys. Both the queries and keys are fed into vision transformer; throughout forward propagation, each query merges the keys with high affinity with it. At the end of affinity propagation, the keys contained in each query form the desired entity (a semantic region or an instance). This implementation allows us to carry out the training procedure efficiently in GPUs.
We train SAM-CP on COCO Lin & Maire (2044) and ADE20K Zhou et al. (211)) and evaluate on these two datasets as well as Cityscapes Cordts et al. (2016) for both open-vocabulan and closed-domain segmentation. Since SAM-CP is trained to understand text labels, it can easily adapt to unseen classes with a CLIP Radford et al. (02I)) text encoder. Extensive experiments demonstrate SAM-CP's ability to cover semantic, instance, and panoptic segmentation with single model. In particular, it reports state-of fatte-of-the--ee the-art accuracy in ppcabvocayylary segmentation. Qualitative studies demonstrate that SAM-CP improves the semantic discriminativity of SAM's features. Our research offers a new methodology to equip vision foundation models (e.g., SAM) with a solid and flexible ability for semantic recognition. We expect the proposed approach to gain stronger and more versatile abilities in the future, with the vision foundation models being upgraded and becoming more robust.
2 RELATED WORK
In recent years, both the CV and NLP communities have witnessed a rapid development of foundation models. In particular, the vision foundation models have largely evolved from being simply pre- trained for image classification Dosoviiskki et al.] (202T) Liu et al. (202T) T) Wang et al.) (202T)
p to
incorporating multimodal information Radford et al. (202 Fang al. (2023 Li ct al. (202cc); Alayrac et al. (2022) and/or being pre-trate traineiied to deal with different tasks Huang et al. [2023) Yu et al.
(2023b); Wang et al. (202cc). They improve the accuracy of various downstream visual recognition tasks including detection, segmentation, etc.
Recently, SAM Kirillov et al. (2023) appeared foundation model for versatile segmentation. Pre-trained on a large corpus with billions of instances, SAM can segment an image into a set of basic patches without tuning. An important advantage of SAM lies in its ability to recognize images
2

---

Published as a conference paper at ICLR 2025
in different domains, yet disadvantage lies in the lack of semantic labels each patch. The community has been trying to adapt SAM to various scenarios, including adapting it on other image
data (e.g., medical images [e et al. (2023); ; Ma et al. (2024); Hu & Li (2023); Roy et al. (2023).
camouflaged images Chen et al. 20 lang et al. (023b)) remote sensing images Chen et al.
(2024a): /Zhang et al. (2023c); Wang et al. (2023a), etc.), raising it segment 3D objects (Shvets itt al. (2024); Wang et al. (2023b); Cen et al. (202)) and using it as the pre-processing step of image editing Yu al. (2023c); Xie et al. (2023). ). Among these efforts, one of the most challenging topics to assign the SAM patches with semantic labels. Existing works involved calling other foundation models (e.g.. CLIP Radford et al. (202T)) ) for image tagging Chen et al. (2023a): [Li al. (2023a); Wang & Vasu (202)); Wei et al. (2024). applying SAM as a refinement stage after other detection and/or segmentation models Ren et al. (2024), and generating other variants. However, in many scenarios, SAM may over-segment basic semantic units into sub-patches, which increases our burden of segmentation for specific purposes.
This paper focuses establishing basic prompts beyond the segmentation results of SAM for versatile segmentation. This is related to a series of query-based algorithms for visual recognition, such DETR Carion et al.] (2020) and the subsequent variants Zhu et al. (2021); Li et al. (2022a); Liu et al. ) Zhang et al. (2023a); Liu et al. (2024). Mcanwhile, we compute the affinity to determine the relationship between semantic units (e.g.. queries and objects), which is related to a few prior work Ahnn & Kwak (2018): Yu et al. (020); Liu et al.] (2017) that tried to compute the affinity between pixels and objects to perform segmentation. The idea was also inspired by ViRReq Tang et al. (2023a), recent work that proposed decomposing complex visual recognition tasks into elementary units to ease annotation and optimization. Further, motivated by the previous open-domain panoptic segmentation methods Yu et al. (2023a); Xu et al. (202a)) : Chen et al. (2023c). we leverage a CLIP-based classifier to equip the model with an open-domain recognition ability.
3
OUR APPROACH
3.1
OVERVIEW: COMPOSITE PROMPTS FOR SEGMENTATION
The overall design of our approach is illustrated in Figure The core idea is to establish two types of prompts beyond SAM Kirillvv et al. 2002), a recent vision foundation model that extracts patches from an input image as potential instances. By composing the output of different prompts, the SAM patches are labelled and/or combined into semantic regions and/or instances, and thus versatile segmentation tasks can be performed. We name the approach SAM-CP, where 'CP' stands for composable prompts.
Mathematically, let the input image be I. SAM extracts a number of patches, P = (P P2...., PN.). under regment-everythin represented as a binary mask of the same shape as the input image. Although SAM is robust across various vision domains, it does not offer a category to each patch and, sometimes, instance (e.g., person) may be over-segmented into multiple patches. We design the following two types of prompts.
mode. N is the number of patches and P, is the n-th patch which is
Prompt I semantic labeling. Given a text label T and one patch P. judge if P can be classified as T.
Prompt II - instance merging. Given a text label T and two patches P1 and P2 classified as T. judge if P, and P2 belong to the same instance of T.
A wide range of segmentation tasks can be accomplished by composing the above two prompts. Firstly, note that the regular semantic segmentation only involves Prompt I (assigning a label to each patch), and instance segmentation is enabled by adding Prompt II (merging over-segmented patches into an instance). Additionally, the two prompts can be iteratively called when segmentation is required at finer level, e.g., Prompt for classifying a region into sub-classes, Prompt II for segmenting an instance into parts, etc.
3.2 EFFICIENT TRAINING WITH A UNIFIED AFFINITY FRAMEWORK
straightforward implementation of SAM-CP involves executing Prompt I for each patch and then Prompt II for each pair of patches, following which the patches belonging to the same instance can be merged. However, this naive pipeline incurs unsatisfying efficiency because the number of Prompt II to be executed is O(N2) where N can be hundreds for regular image. Moreover, the merging procedure requires serial operation and can inevitably encounter conflict in the inference stage (e.g., P, and Pz, P, and P3 are considered to be the same instance but P2 and P3 are not).
3

---

Published as a conference paper at ICLR 2025
Categgry Text
MLP
Culogory Assian
17
Encoder (e.g: CLIP)
Sup.
Calculato Similarit
Patch Encoder N
Cis
Score
Unified Affinity Decoder
Pe(Py Pg.).PP))
(c+0)
C)
Categery
Score
Cli
Attention
Mult--ead Selt-
Dynamic Multi-Head
MUW
Cross-Altantioo
Mark Pooling
SAN pacces
Predioste Mack
@ 1R.
Calculate Affinityy
(Prompt ar
Sup.
Albotty Assgg
Figure 2: The unified affinity framework as efficient implementation of SAM-CP. The input image with SAM patches is fed into a patch encoder. Type-I and Type-ll prompts appear as two sets of queries. Affinity values are computed and the SAM patches are merged according to the affinity values. Semantic and instance level supervision are added to the merged patches. The purple arrows are present only in the inference stage open-vocabulary segmentation. Best viewed in color
To accelerate the procedure, we design an equivalent but more efficient mechanism named the unified affinity framework. We initialize set of queries for potential units (i.e., semantic regions and instances) and set all SAM patches as keys. We perform affinity propagation between the queries and keys, gradually merging units with high-affinity scores into larger unit. With this modified mechanism, the over-segmented patches are merged on the fly and no further post-processing is required (i.e., at the end of affinity propagation, the survived units naturally form the output). We illustrate the procedure in Figure 2 and elaborate on the individual modules as follows.
3.2. PATCH ENCODER
We first extract visual features from the patches. For each patch Pa. we apply a regular backbone (e.g.. ResNet50 He al. (2016) LLin et al. (2021)) equipped with an RolAlign He al. (017)) operator to obtain a basic feature vector f,. We also design MaskRol operator extract more accurate visual features by masking out the background areas. All these features are propagated through multi-layer perceptron (MLP) and fed into multi-head self-attention layers, where is set to 6 throughout this paper. The feature vector corresponding to P,, is denoted as fn
3.2.2 UNIFIED AFFINITY DECODER
This is the core module that performs affinity propagation and merges patches into super-patches. We will elaborate on three key elements below, namely, (1) set of semantic and instance queries, (2) the algorithm of affinity propagation, and (3) the label assignment mechanism (Section 3.2.3).
The queries of similar form in DETR Carion al. (2020). Differently, establish two types of queries for semantic and instance segmentation, respectively. (1) For each text label Te where e E (1.2..... C} is the class index, model (e.g., CLIP Radford et al. 2202)) to convert it into query vector, e,. where the superscript SS stands for 'semantic" (2) We also create N instance queries (i.e., assuming that each patch may correspond to an instance, abbr patch-as-query, PasQ) and initialize them using the visual features and position embeddings the patches, i.e., e f,, where the superscript 'I' stands for 'instance'. We denote both types of queries as Qm. m = 1, 2..... M where M is the number of queries.
Queries.
the language branch of vision-language
Affinity. The affinity is mathematically defined denotes the probability that the patch P,, belongs to the query Qm Initially. we set all entries of A to Then, in each affinity propagation layer (please see below for details), the query vectors (denoted as Q) and the patch features (denoted as K and V) are fed into multi-head cross-attention module to update the query vectors for subsequent classification. There are three key modules here. (1) The affinity matrix A serves dynamic mask after binary operation in cross-attention. which we call dynamic cross-ates aosss-tttnttoon (DCA), to extract the feature from high-affinity patches. (2) We insert a module named affinity refinement (AR) to update the affinity matrix A using the cosine similarity between Q and K. (3) To enhance the query feature, we apply a Query Enhancement (QE) mechanism to fuse the query's 's feature with the Rol features of its high-affinity regions. The details of DCA, AR,
matrix A sized M x N. Each entry of A, Am.n-
4

---

Published as a conference paper at ICLR 2025
and QE are described in Appendix A.1 As shown in the ablation (Section 4.5 all these designs contribute to the segmentation accuracy.
3.2.3 LABEL. ASSIGNMENT AND SUPERVISION
Each query, regardless of its type (semantic or instance), is expected to occupy a set of (one or more) patches and be assigned class label. So, two sources of supervision are required, which come from the semantic labels and instance IDs, respectively. Each query is supervised by both signals.
Semantic-level supervision. We first build vision-language classifier upon the semantic queries. Following GLIP Li et al. (2022b), the score Syh, of the m-th query at the c-th class is determined by Qm and 62. the quantities produced by linearly normalizing Qm and ea into [0, 1]. The classification
loss Lin et al. (2017b) Lets is computed upon Sm.c in Equation [
M
1 Q.Q e + b, Lels 3
Scls m, 5
Σ
FL Ifc me)=
(I)
In Sole 8 is a learnable scaling factor, and is a bias parameter. In Cels, FL(:,.) is the focal loss, (.)) is the sigmoid activation function, ei, is the ground-truth class label for the m-th query (we explain how to compute o in the Appendix (A.2). and Ii the indicator function which takes 1 if the statement is true and O otherwise. Note that, it we only perform Type-I prompts, the category text is not necessary, and only need to perform binary prediction for each text query to judge whether the category cxists in the image. This mechanism was designed for Type-II prompts. An important technical contribution of our work is to unify Type-I and Type-II prompts into one framework so that the training and inference costs are reduced.
Instance-level supervision. At the end of affinity propagation, each instance query corresponds to a binary segmentation mask. Let the ground truth contain K instances. We first establish a matching matrix G (sized K x N) by computing the box-level ToP (intersection-over-piaee ToP values between each pair of predicted and true instances; they are considered matched if both IoP values are greater than a pre-defined hyper-parameter т, i.e., Gk,n = [[min[ooP box loP m where 0.8 throughout this paper. If no patches assigned to an object, the patches with an IoU value of at least 0.5 will be chosen as candidates of low-quality matching. Based on the matching matrix G, we compute the ground- truth affinity matrix B (sized M N, same A). For each m, we first determine if any ground-trute truth instance (indexed km matches the m-th query (see the next part for details). If yes, B,, Gk.., (i.e., the kim row of G is copied to the m-th row of B); otherwise, Bm = 0. Then, following Cheng et al. (0022) Li et al. (023b), we compute the mask focal loss Cmi and the Dice loss Laice:
and mask-level
M Σ
M
Cmil
1
Em
Laice
Σ M
1
Em Dice Am, Bm).
M max [mm m
1)
R=1
(2)
where FL(:)) and Dice(;--) are the focal loss and Dice loss, respectively, M is the number of positive query embeddings in the image, [Bm lo is the number of non-zero entries in Bn (i.e., the number of patches that are assigned to the m-th query), and e. {0, 1} is weight indicating whether to take the m-th query into consideration.
Determining Bn, and Em for each query. This procedure is different between the types of queries.
For semantic (Type-I) query, Qu (c E (1, C}), we first check if the e-th class appears in the image. If not, we have B. and E. = O. Otherwise, the e-th class (as a unique semantic region) must appear in the ground-truth 'instance" set; let the index be ke, thus B. Gk.- We set Em to for the positive embeddings and ( otherwise.
For an instance (Type-II) queries, we follow the DETR series Carion et al. (020)): Zhu al. (202T): Zhang et al. (2023a) to apply the Hungarian algorithm to find the best matches between the queries and the ground-truth instances. Differently, we compute the matching cost using more metrics, including the classification loss (cls), the mask focal loss (mfl), the Dice loss (dice), the bounding-boo box cost (bbox), and the gloU cost (giou). We will show in experiments that all these components contribute better segmentation results. After the matching is done, obtain the index ki for the m-th query (km, can be null, in which situation the query is ignored), and assign Bm Gk.,. We set Sm to for the positive embeddings and 0 otherwise.
5

---

Published as a conference paper at ICLR 2025
COCO-ADERD
ADE2OK -COCO
COCO ACCO--Cyycepes
Method
Backbone
PQ
SQ
RQ
AP
mloUi
PQ
SQ
RQ
AP
mloUi
AP
PQ
mloU
MaskC
VIT-L
70.3
19.2
23.7
FreeSeggkerr bas
VIT
16.3
71.8
21.6
6.5
24.6
21.7
72.0
21.6
6.6
21.7
OPSNe Dasaa
VII-L
52.4
23.0
41.5
OpenSee1 Zhayes
Focal L
19.7
15.0
29.0
41.4
HIPIE Wew mill
VII-H
29.0
X
Decisler Seper
Focal-L
21.8
13.1
29.6
24.9
38.1
$2.0
MaskOCLir Kact
VII-L
30.4
ODISE/kea bis
VIT-I
23.3
74.4
27.9
13.0
29.2
25.0
79.4
30.4
23.9
CLIPS Wosti
VII-L
30.1
FruzenSee
DORR
CN-L
25.9
16.4
34.4
28.4
45.8
56.8
FCCLIP
CN-L
32.3
34.1
78.0
32.9
26.8
44,0
SAM-CP
CN-L
27.2
17.7
32.9
17.0
31.8
28.6
78.4
34,5
21.9
34.3
29.3
41.0
47.9
Table 1: Accuracy (%) of Open-vocabulary panoptic segmentation (in PQ, SQ and RQ). instance segmentation (in AP) and semantic segmentation (in mloU). CN-L means Cex-Next-L.
Method
Backhone
Seg. Style
Epoch
PQ
COCO
Ap
AP
mlooU
Epoch
PQ
AD3E2KK
Ap
AP
mloU
R50
reg. #seg.
50- 15e
31.1
MaskFormeer
2007
R50
300c
46.5
33.9
57.8
128c
34.7
Mask2Former
Clesse
R50
seg.
50c
41.7
128e
39.7
26.4
Mask2Tormer
Chas
362
Swin-L
100l
57.8
48.6
67.4
128e
48.1
34.2
56.1
Mask DINO
R50
reg.
50c
53.0
48.8
44,3
Mask DINO
Swin-L
reg. -seg
50e
58.3
56.2
50.6
67.3
128c
56.6
X-Decoder
Focal-T
50c
52.6
41.3
62.4
128c
41.6
27.7
51.0
X-Decoder Meneal
Focal-T.
seg.
50e
46.7
128e
49.6
35.8
SAM-CP
R30
SAMP
36c
48.6
46.1
41.7
55.6
128c
38.5
28.7
25.1
42.4
SAM-CP
Swin-L
50.4
45.2
128e
34.6
30.3
Table 2: Accuracy (%) of panoptic segmentation (in PQ), instance segmentation (in AP), and semantic segmentation (in mloU) on the COCO- PoCC--anoopic and ADE20K datasets. The segmentation (seg.) style 'SAM*' means that we use fixed segmentation results of SAM without any refinement including regression ('reg.') )) or segmentation ('seg.') refinement.
The overall loss function. The overall loss defined as Call = dela Cels + Amit Cmil + Adice Ldice. where the loss coefficients Acts 2, Amat 1, and λdice = 1. We apply the denoising strategy in DINO Zhang et al. (2023a) to improve the training performance.
4
EXPERIMENTS
4.1 INFERENCE
The inference procedure varies slightly between closed-set and opea-vocay segmentation. In closed domain, the logit Sels in Equation I is used for classification. The normalized and quantized affinity matrix A is used for patch merging. For semantic segmentation, we refer to the rows corresponding to C' classes and merge all patches surpassing a pre-defined threshold. For instance segmentation, we look up the non-empty instance rows of A, each of which corresponds to an instance. Panoptic segmentation is achieved by combining semantic and instance segmentation results. In open domain, we complement the logit matrix Sela (sized M C) with Cl.PP-hased logit matrix, S SCLIP
The remaining part is the same as in a closed domain. To calculate SCLIP we follow FC-CLIP Yu et al. (2023a) extract CLIP features using mask pooling on the predicted masks. Then, the feature S SULL is obtained by calculating the similarity between the CLIP feature and é.. The final class score is computed as S = (a (o(lss)) (( + (softmax (SCLPP) ))" where k = 0.4 is coefficient balancing the closed-domain and open-vocabulary class scores.
Datasets and evaluation metrics. We train SAM-CP on the COCO-Panoptic Lin & Maire (2014) and ADE20K Zhhou et al.) (2017) datasets, and evaluate the models on either closed-domain or open-vocabulary segmentation (with cross-dataset validation and Cityscapes used as test data). COcO-Panoptic (the 2017 version) has 118K training and 5K validation images with 80 'thing' and 53 'stuff" categories. We report the instance segmentation results with the standard AP metric on the 80 'thing' categories. For semantic segmentation, we report the mloU for all (80 53) categories. For panoptic segmentation, the PQ metric is computed for all categories and the 'thing' and 'stuff" subsets individually. ADE20K contains 20,210 images. We use the 150 most common object categories, including 100 'thing" and 50 'stuff" categories. Cityscapes is street-vice view dataset with 8 'thing' and 11 'stuff" categorics. We inherit the same metric from COCO to ADE20K and Cityscapes datasets.
4.2 SETTINGS
Implementation details. The proposal masks are generated by SAM with 48 grid points along each axis of the input image. To verify our idea better, we use SAM with VIT-H to generate better patches.

---

Published as a conference paper at ICLR 2025
E-SNE
ESNE
bird
Water
grass
free
N:
ESN
river
elepher
slephard
Figure 3: A qualitative study of how SAM-CP works. Each row displays example. The leftmost column shows the input image with SAM patches; the middle and right parts show the semantic and instance segmentation results, respectively. We use the t-SNE algorithm to project the learned visual features (by SAM-CP; please refer to Figure I for the difference from the features of SAM) in a 2D coordinate system. The points with the same color belong to the same semantic class the same instance (according to the ground truth). This figure is best viewed in color
I-SNE
L-SNE
petrs
CINDAL
Figure 4: The t-SNE visualization upon the visual features of SAM and SAM-CP. Due to the limited space, only semantic segmentation results are displayed. The points with the same color belong to the same semantic class (according to the ground truth). This figure is best viewed in color:
For open- vocabulaay segmentation, we use a frozen CLIP image encoder (for a fair comparison, we use the same architecture (ConvNext-L) as the previous best method, FCCLIP Yu et al. (2023a)) as the backbone and equip it with FPN Lin et al. (2017a). For closed-domain segmentation, we establish SAM-CP ResNet50 (R50) He al. 2016) and Swin-L et al. (202T). We use the implementation of the MMDetection Chen et al. (2019) (v3.0) library. We use 8 Tesla- V V100 GPUs (4/2 images per GPU) for open-vocabullryylloee---ooo domaan experiments. The data augmentation strategy follows the DETR series. An AdamW optimizer Loshchilov & Hutter (0019) with a basic learning rate of 0.0002. See Appendix B for further details.
4.3 QUANTITATIVE RESULTS
Open-vocabulary segmentation. Results are summarized in Table T In both COCO and ADE20K->COCK FrozenSeg Chen et al. (2024b) (SAM-based), the previous state-of-the-an methods, in terms of PQ. SQ. RQ for panoptic segmentation, and AP for instance segmentation. In particular, SAM-CP not only reports competitive SQ (for high-quality segmentation), but also achieves a better tradeoff between PQ and RQ; in COCO Cityscapes, SAM-CP achieves 40.6% PQ, 29.3% AP and 47.5% mloU for versatile segmentation, which brings the best Instance Segmentation performance. We owe the excellent results to the efficient mechanism that combines SAM and CLIP, open-world foundation models for opca-vocabylary segmentation.
SAM-CP surpasses FCCLIP Yu et al. (2023a) (non-SAM-basees and
Closed-domain segmentation. Results are summarized in Table 2 In COCO-Panoptic with the ResNet-50 backbone, SAM-CP achieves a 48.6% 6% PQ, 41.7% AP, and 55.6% mloU; with a stronger backbone, Swin-L, SAM-CP reports higher segmentation accuracy, with a 52.7% PQ, 45.2% AP, and 61.7% mloU. In ADE20K, SAM-CP achieves a 38.5% PQ, a 25.1% AP, and a 42.4% mloU on ResNet-50. and 44.4% PQ, 30.3% AP, and 49.4% mloU on Swin-L. An interesting comparison comes from MaskFormer Cheng et al. (202T) where SAM-CP reports higher PQ and AP but lower mloU, which implies its advantageous performance instance-level recognition. These numbers demonstrate the effectivvnees of our methodology, i.e., establishing composable prompts beyond vision foundation models. We will delve into the limitation SAM-CP in Section (.6 and explain why this novel mechanism falls short in closed-domain segmentation.
Summary. As new methodology for versatile segmentation, SAM-CP shows a unified pipeline and promising performance over three popular benchmarks. In particular, SAM-CP demonstrates ffute-of-the-e performance in the open domain. We look forward to the future when stronger foundation models are available and further boost the accuracy of SAM-CP.
7

---

Published as a conference paper at ICLR 2025
Closed -ossad-domain (COCO)
Open-domain (COCO~ADE20RS
Loss
Label Assignment
PQ
Apde:
AP
mloU
PQ
SQ
RQ
AP
mloU
all
47.0
45.8
41.4
54.2
27.2
77.7
32.9
17.0
31.8
w/o Lmit
w/o mifl
0.0
3.5
0.0
0.0
0.6
22.0
0.9
0.0
3.4
w/o Ldice
w/o dice
41.3
35.1
34.3
48.3
23.8
73.4
29.1
15.8
28.6
all
w/o mff
42.8
44.0
39.8
51.4
26.5
78.2
32.3
17.2
31.6
all
w/o dice
45.3
44.8
40.6
53.7
26.6
76.6
32.4
16.7
31.5
all
w/o box & giou
45.5
44.0
40.7
53.9
25.9
76.1
31.6
16.4
30.5
Table 3: Accuracy (%) in open and closed domains with different loss terms and matching strategies.
Closed-domain (COCO)
Open-domain
(COCO-+44
+DD22K))
DCA
AR
MaskRol
QE
BG
PQ
Ap
AP
mIoU
PQ
SQ
RQ
AP
mloU
45,4
45.6
41.1
51.8
26.6
76.9
32.5
16.6
31.7
43.5
44.0
39.9
51.1
25.8
76.8
31.3
16.3
30.5
v
44.1
45.3
40.6
51.1
25.6
74.4
31.1
16.5
30.3
v
44.8
44.5
40.5
51.6
26.5
75.7
32.1
16.5
31.4
45.2
45.4
41.3
52.6
25.5
75.7
31.2
16.1
30.3
47.0
45.8
41.4
54.2
27.2
77.7
32.9
17.0
31.8
Table 4: Accuracy (%) in open and closed domains with different modules in the SAM-CP framework.
Strategy 47.0
PQ
Ap
AP
mIoU
Learnable CLIP
PQ
PQth
PQ*
AP
mloU
patch-level
45,8
41.4
$4.2
17.5
15.7
21.1
11.9
19.1
patch-level, w/o PE
45.8
45.0
40.6
51.7
16.9
14.1
22.6
22.5
image-level 46.2
45.6
40.9
52.3
27.2
27.0 27.7
17.0
31.8
Table 5: Accuracy (%) on COCO (on R50, 12
Table
Accuracy (%) on the setting of COCO->ADE2K (on ConvNext-L, 12 epochs) with different classifier types for open domain.
epochs) with different strategies of dynamic cross- attention (DCA), where 'patch-level" is the best option, 'PE' denotes the patch encoder.
'Learnable' means classifier trained on COCO.
4.4 QUALITATIVE STUDIES
We show that SAM-CP learns discriminative visual features beyond SAM. We first show how SAM- CP accomplishes the entire segmentation procedure in Figure 3 From the t-SNE visualization maps, one
that the features extracted from the SAM patches form clusters that correspond to different semantic classes. Additionally, when instance segmentation required, the specific cluster be further partitioned into sub-clusters that correspond different instances. This aligns with the high-level idea shown in Figure D and SAM-CP accomplishes the goal efficiently.
We further compare the visual features learned by SAM and SAM-CP in Figure 4 Not surprisingly, the features extracted by SAM are not semantically discriminative, with samples from different semantic classes overlaying in feature space. Such features are clearly improper for visual recognition purposes. SAM-CP shows much better discriminativity. aligning with the display in Figure
4.5 ABLATIVE STUDIES
We study the effectiveness of the design principles and individual modules via ablative studies. In the open vocabulary, we report the results of COCO-ADEE20 with frozen CLIP encoder (ConvNext-L) and a 1 schedule (i.e., 12 epochs). In the closed domain, experiments are performed on the COCO dataset using a ResNet-50 backbone with 1 schedule.
Loss and label assignment strategy. Fable how different loss functions and label assignment strategies impact the performance. One can see that the mask focal loss is essential, without which the model runs into failure. The dice loss also contributes, especially for instance detection and segmentation. Regarding label assignment, the experiments show clear benefits in introducing more metrics into the weight term to improve the results of bipartite matching.
Module-Ievel ablation. There are five components that are helpful to produce better segmentation results, namely, (1) the dynamic cross-attention (DCA) mechanism used for local feature extraction in the decoder, (2) the affinity refinement (AF) strategy by adding a prediction before the sigmoid function in each stage, and (3) the MaskRol operator which masks out the background region for more accurate visual feature extraction. (4) the query enhancement (QE) which add Rol feature to query embedding. (5) the self-affinity for negative queries to keep the 'segment anyting' ability.
8

---

Published as a conference paper at ICLR 2025
Proposal types
PQ
Ap
AP
mloU
K
0.0
0.3
0.4
0.5
0.8
1.0
SAM#
48.6
46.1
41.7
55.6
PQ
17.5
26.9
27.2
26.2
22.9
16.9
SAM MD
51.4
51.6
45.8
57.3
Table 8: Accuracy (%) on the setting of Table 7: Accuracy (%) COCO (with R50, 36 COCOO>DDE20R with different coefficients, R. 0.4 is chosen the default setting.
epochs) by adding proposals.
Method
1
50.
50.3
2Rg 4
MRo.s
MR 4
Method
MAP
APho
AF%s
Mask DINO
76.3
83.0
4.2%
10.1%.
32.3%
w.o. general
13.7
30.5
10.6
SAM
71.1
79.5
8.9%
16.7%
39.3%
general
13.6
30.1
10.5
SAM-Merging
73.3
81.2
9.0%
15.0%
33.3%
Table 10: Accuracy (%) of part seg- Table 9: A comparison between the mIoU and missing rates with mentation Pascal Part dataset. respect to different IoUs for COCO (val2017) instance segmen-The first row shows results SAM- tation. Here, mloU is the IoU between the highest-loU proposal CP trained only on part datasets, and the ground truth, and ml. means we only calculate the while the second shows those of mIoU for the instances that match the best proposal with an IoU SAM-CP trained on both part and higher than 0.5. MR, denotes the proportion of instances that general datasets.
have no matched proposal with an loU higher than x.
Method
SAM - based
SAM/ / Don-SAM time (sfimg)
total time (s/mg)
PQ
AP
mloU
FCCLIP Yu 2023a
no
070.24
D,24
25.2
16.4
32.6
Frozen Seg Chen él at 2024b
yes
3.41 /1.07
16.4
34,4
SAM-CP
yes
3.41 /0.21
3.62
27.2
17.0
31.8
SAM2-CP
yes
1.61 /0.18
1.79
27.9
18.0
32.6
Table 11: Inference time comparison. 'SAM/ non-SAM' means the SAM and non-SAM modules.
Table summarizes the ablation on these five components. Each of them contributes individually and they combined to boost the baseline by at least 1.0% in all the reported metrics.
The design of DCA. Among the above modules, DCA requires further investigation. We report the performance of three DCA options, differing from each other in whether cross-attention is computed at the patch level or image level, and whether the patch encoder is used. As shown in Table 5 the patch-level tross-attentios with patch encoder works the best, implying that we extract sufficient visual features from the SAM's output which often contains hundreds of patches.
Classifier for open-vocabulary segmentation. Table 6 shows the impact of classifiers for open- vocabulary segmentation. With a single closed-set (learnable) and CLIP (frozen) classifier, SAM-CP reports 17.5% PQ and 16.9% PQ, respectively. After the classifiers and open-vocabulary scores are balanced, resulting in a much higher 27.2% PQ. Interestingly, the closed-set and CLIP classifiers are better at instance and semantic segmentation, respectively, and the fused classifier excels in both scenarios, with a larger gain for instance segmentation.
fused, the closed-set
Adding extra proposals. We add the proposals extracted by the pre-trained Mask DINO (MD) model into the pool of candidate patches. Details are provided in Appendix B.] Table 7 shows improved segmentation results, inspiring us that SAM does not generate ideal patches.
Average coefficient for vocabbacyyy segmentation. In Table S we ablate the average coefficient The results show that 0.4 is the best
defined in Section 4. on the setting of option in the geometric average between the closed-set and CLIP classification scores.
Time comparison. We chose two SOTA methods to compare their time performance within our vocabuacayy setting in Table 11. All experiments were carried out using an RTX 4090. Compared with SAM-based FrozenSeg, our method has less time in non-SAM modules. When replacing SAM with SAM2, the speed becomes faster while the performance is higher.
Part segmentation. We conduct part segmentation on the Pascal Part datasets Chen et al. (0014). featuring 4465 images with 93 part categories. Table 10 presents the performance, and Figure offers visualizations. The results' first and second lines stem from models trained solely on part categories and on both part and general ones, respectively. The model demonstrates stable and reliable part- segmentation performance across different segmentation types. SAM relies on low-level cues like color instead of semanties for predictions, as seen in its tendency separate arm cuffs from skin. Our BAM-based approach inherits these drawbacks, but future tuning on this data may improve results. Additionally, we crop the original image using the instance segmentation map or bounding box, enlarge the sub-image for extra SAM segmentation to obtain more parts, shown in Figure 7 and iteratively use prompt I and prompt II for finer semantic and instance segmentation.

---

Published as a conference paper at ICLR 2025
Figure 5: Direct classification for Figure 6: Dynamic prompts for look- look- Figure 7: Interactive SAM parts and "parts of the whole'. ing for small objects.
calling yields finer results.
Figure 8: The visualization of part and general instance segmentation. The result is obtained with one model and text labels of different granularities. On the left are the GTs, and on the right are the results.
4.6 DISCUSSION
Advantages. SAM-CP inherits clear advantage from SAM in the generalized ability across different visual domains. SAM-CP achieves this by decomposing low-level pixel grouping (offered by SAM) from high-level semantic recognition (offered by the composable prompts). In the meantime, the imperfection of SAM limits the segmentation accuracy of SAM-CP, especially in the closed-set vision benchmarks. We showcase this point in Table 9 where the patches generated by SAM are compared to those generated by Mask DINO Li et al. (2023b), a state-of-the-an segmentation model. We find that SAM suffers a higher missing rate (e.g.. one cannot find a proposal with an IoU larger than 0.5 for 16.7% instances; the rate is not significantly smaller even if we refer to the ground-truie masks to merge some proposals) than Mask DINO (where the rate only 10.1%).
Classification mechanism. SAM-CP does not directly classify each patch but uses query to conduct cross-attention to all high-affinity patches and then fuses their features as the query feature for classification. This enables the model to use richer and more complete information for semantic classification. F shows an example. Three patches are present, corresponding to dog's 's torso, tail, and leg. Both 'cat' and "dog" labels are in the semantic space, but it is difficult for each of the three patches alone to classify itself as 'dog' or 'cat'. The model, instead of assigning a high affinity between class label ('dog' or 'cat') and any patch, assigns high affinity between these three patches (note that, no matter what the class is, 'dog' or 'cat', they belong to the same instance). After they have been merged, the visual features are often sufficient for classification ('dog' 'cat'). In the example, the 'dog's 's leg' patch gets an ambiguous classification score (dog: 0.4844, cat: 0,5156) initially, but when it is merged with the other patches (as a 'dog'). the score becomes distinguishable.
Limitations. In other words, SAM failed to find some objects or incorrectly merged two or more objects into one patch, and SAM-CP cannot save such loss. This contributes the deficit in the segmentation accuracy (e.g., AP or mloU) compared to Mask DINO. We display some typical examples in Appendix A.3] This issue can be alleviated by dynamically adding denser point prompts to the regions (e.g. small objects). We show a case in Figure [6] where small targets were found with this simple mechanism. Additionally, the inference speed of SAM-CP bound by that of SAM; once more efficient vision foundation model is available, our framework can be scamlessly transplanted.
CONCLUSIONS
In this paper, we propose SAM-CP, a novel approach that equips SAM with semantic and instance segmentation abilities. At the core of SAM lies two composable prompts, which determine (1) whether a SAM patch aligns with a text label and (2) whether two SAM patches belong to the same instance, respectively. The idea is implemented using a unified affinity framework for efficient training and inference. We show both quantitative and qualitative results with panoptic segmentation on COCO, ADE20K and Cityscapes for both open-vocabulary and closed-domain segmentation. Our study offers a new methodology to make use of the vision foundation models such as SAM.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGE
This work was supported in part by the Key Deployment Program of the Chinese Academy of Sciences, China under Grant KGFZD145-23-18 and the Strategic Priority Research Program of Chinese Academy of Sciences under Grant EIXA310103. In addition, we would like to thank Zhaoyang Wei for his technical support during the article- experiment process and the rebuttal phase.
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR, 2018.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, and Antoine et al.Micch. Flamingo: visual language model for few-shot learning. In NeurIPS, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Jan- Michael Frahmm Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. In NeurlPS, 2023.
Jiaqi Chen, Zeyu Yang, and Li Zhang. Semantic segment anything. https://github. com/
Fudan-zve Sengntteinnniiii c- Segment-Aeeee AnythntttAtttiii 2023a.
Junzhang Chen and Xiangzhi Bai. Learning "segment anything" in thermal infrared images through knowledge distillation with a large scale dataset SATIR. CoRR, abs/2304.07969, 2023.
Kai Chen, Jiaqi Wang, and Jiangmiao et Pang. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint (30Kv::990607701 2019.
Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model. IEEE Trans. Geosci. Remote. Sens., 2024a.
Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam fails segment anything? saanig--ss---aaaerr Adapting sam in underperformed scenes: Camouflage, shadow, and more. arXiv preprint arX1v:2304.09148, 20236.
Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, and Hengshuang Zhao. Open-vocabulary panoptic segmentation with embedding modulation. In ICCV, 2023c.
Xi Chen, Haosen Yang, Sheng Jin, Xiatian Zhu, and Hongxun Yao. Frozenseg: Harmonizing frozen foundation models for npe-vvoabularry segmentation. In CVPR, 2024b.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan L. Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, 2014.
Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, 2021.
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.
Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything. CoRR, habs/2305.06558. 2023.
Marius Cordts, Mohamed Omran, Sebastian Ramos. Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
11

---

Published as a conference paper at ICLR 2025
Zheng Ding, Jicke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. In IMCL, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
Kaiming He, Xiangyu Zhang, and Shaoqing Ren al.. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, and Piotr Dollár et al.. Mask R R-CNN. In ICCV, 2017.
Kaiming He, Xinlei Chen, and Saining Xie et al.. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Sheng He, Rina Bao, Jingpeng Li, P Ellen Grant, and Yangming Ou. Accuracy of begment-anyhinng model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324 2023.
Chuanfei Hu and Xinde Li. When SAM meets medical images: An investigation of segment anything model (SAM) on multi-phase liver tumor segmentation. CoRR, htw/2330.085006 2023.
Tianyu Huang, Bowen Dong, Yunhan Yang. Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In ICCV, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. ICCV, 2023.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, and Lei Zhang. DN-DETR: accelerate DETR training by introducing query denoising. In CVPR, 2022a.
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. CoRR, abs/2307.0477 04767, 2023a.
Feng Li, Hao Zhang, Huaizhe Xu. Shilong Liu, Lei Zhang, Lionel M. Ni, and Heung- Yeung Shum. Mask DINO: towards A unified transformer-base framework for object detection and segmentation. In CVPR, 2023b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023c.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, 2022b.
Tsung- Yi Lin and Michael et aaiMeire. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, and Ross B. Girshick et al.. Feature pyramid networks for object detection. In CVPR, 2017a.
Tsung-Y Yi Lin, Priya Goyal, and Ross B. Girshick et al.. Focal loss for dense object detection. In ICCV, 2017b.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: dynamic anchor boxes are better queries for DETR. In ICLR, 2022.
Shilong Liu, Zhaoyang Zeng, and Tianhe et al.Ren. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024.
12

---

Published as a conference paper at ICLR 2025
Sifei Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming- Ming-Hsuan Yang, and Jan Kautz. Learning affinity via spatial propagation networks. In NeurIPS, 2017.
Ze Liu, Yutong Lin, and Yue Cao et al.. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 2024.
Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Yuxi Ren, Xuefeng Xiao, Yitong Wang. Rui Wang. Shilei Wen, Xin Pan, and Xingang Wang. Freeseg: Unified, universal and open-vocabulary image segmentation. In CVPR, 2023.
Alec Radford, Jong Wook Kim, and Chris et al. aallacy Learning transferable visual models from natural language supervision. In ICMI., 2021.
Tianhe Ren, Shilong Liu, and Ailing Zeng et al.. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Saikat Roy, Tassilo Wald, and Gregor Köhler et al.. SAM.MD: zero-shot medical image segmentation capabilities of the segment anything model. CoRR, abs/2304.05396, 2023.
Mykhailo Shvets, Dongxu Zhao, Mare Niethammer, Roni Sengupta, and Alexander C. Berg. Joint depth prediction and semantic segmentation with multi-view SAM. In WACV, 2024.
Chufeng Tang, Lingxi Xie, Xiaopeng Zhang, Xiaolin Hu, and Qi Tian. Visual recognition by request. In CVPR, 2023a.
Lv Tang, Haoke Xiao, and Bo Li. Can SAM segment anything? when SAM meets camouflaged object detection. CORR, abs/2304.04 04709, 2023b.
Di Wang. Jing Zhang, Bo Du, Minqiang Xu, Lin Liu, Dacheng Tao, and Liangpei Zhang. SAMRS: scaling-up remote sensing segmentation dataset with segment anything model. In NeurlPS, 2023a.
Haoxiang Wang and Pavan Kumar Anasosalu et Vasu. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. arXiv preprint arXiv:23100 10. 15308, 2023.
Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, et al. Sam-med3d. arXiv preprint arXiv:23100 15161 2023b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In CVPR, 2021.
Wenhui Wang, Hangbo Bao, and Li et al. al.Dong. Image as a foreign language: Beit pretraining for all vision and vision- languue tasks. In CVPR, 2023c.
Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor
Darrell. Hierarchical vocebblcyy universal image segmentation. In NeurlPS, 2023d.
Zhaoyang Wei, Pengfei Chen, Xuchui Yu, Guorong Li, Jianbin Jiao, and Zhenjun Han. Semantic- aware sam for point-prompted instance segmentation. In CVPR, 2024.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. In ICLR, 2024.
Defeng Xie, Ruichen Wang, and Jian Ma et al.. Edit everything: A text-guided generative system for images editing. CoRR, abs/2304. 10.00.140000 14006, 2023.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang. and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023a.
13

---

Published as a conference paper at ICLR 2025
Xin Xu, Tianyi Xiong, Zheng Ding, and Zhuowen Tu. Masqelip for open vocabulayy universal image segmentation. In ICCV, 2023b.
Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos. CoRR, abs/2304. 1966., 2023.
Changqian Yu, Jingho Wang. Changxin Gao, Gang Yu, Chunhua Shen, and Nong Sang. Context prior for scene segmentation. In CVPR, 2020.
Liang-Chich Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chich Chen. Convolutions die hard:
Open-vocabulary segmentation with single frozen convolutional CLIP. In NeurlPS, 2023a.
Qihang Yu, Xiaohui Shen, and Liang-Chieh Chen. Towards open-ended visual recognition with large language model. In arxiv: 23/ 1.08400, 2023b.
Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. CoRR, abs/2304.06790, 2023c.
Hao Zhang, Feng Li, and Shilong Liu al.. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. ICLR, 2023a.
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In ICCV, 2023b.
Jielu Zhang. Zhongliang Zhou, and Gengchen Mai et al.. Text2seg: Remote sensing image semantic segmentation via text-guided visual foundation models. CoRR, abs/2304.10597, 2023c.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In ICLR, 2021.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In CVPR, 2023.
14

---

Published as a conference paper at ICLR 2025
A DETAILS OF OUR APPROACH
A.1 AFFINITY COMPUTATION
MaskRoI. To extract patch-level features, we use the MaskRol operator to mask out the background areas. Specifically, we first extract the Rol features using the RoIAlign operator on the minimum outer rectangle P of the patch P.. Then, we transfer the binary mask by rescaling P,, into the same size (width and height) as the Rol features (i.e., the down-sampled size of P. Finally, the Rol features are multiplied by the region mask to obtain the MaskRol features.
Dynamic cross-attention. Rather than performing global computation, attention (DCA) to guide the cross-attention operator in focusing on the local features. This is motivated by deformable attention Zhu et al. (2021). The cross-attention operator follows the same structure multi-head attention Carion et al. 2020), and the matrix serves as the dynamic attention mask. We binarize A by setting the value of each entry to 1 if the value is smaller than the threshold (e.g., 0.5) which means that the area is masked and ignored in computing the cross-attention. As described in Section 3.2.2, the query vectors (as Q) and the patch features (as both and V) fed into a multi-head cross-attentie with the dynamic attention mask.
dynamic cross-
Affinity refinement. Affinity refinement (AF) used update the affinity matrix in a coarse- to-fine manner. We first calculate the similarity Á between the query vectors (Q) and patch features (K), which will be described in Algorithm [ The affinity matrix A is obtained by conducting an element- wise sigmoid activation (-) on A. For the first stage, the Á is obtained by calculating similarity; in the subsequent stages, the Å is obtained by stacking the similarity of the previous stages, which is called affinity refinement.
Query enhancement. The query enhancement (QE) mechanism aims fuse the query's features with the Rol features of its high-affinity regions. Each query will predict the affinity values and merge the patches according to the values as the mask prediction. The minimum bounding rectangle of the estimated mask will be used to extract the Rol feature. In order to save GPU memory and speed up calculations, directly merge the minimum bounding rectangles of the patches, Then, the Rol features and the updated query features (mentioned in Section A.1 are averaged as the new query features for query enhancement.
A.2 SUPERVISION
Ground-truth class labels. Let e, be the ground-truth class label for the m-th query. For Type-I prompt, the mnth query corresponds to the et -th category label. Hence, the objective of the m-th
Algorithm 1 Affinity Similarity Calculation
Input: Query vectors Patch features K, Head number n). Stage number w.
Output: Affinity similarity A. Note: Q RM XD E NxD where M and N is the number of Q and x. D is the feature dimension, which a multiple ry. s RR. bo E R by E R& are the learnable scaling factor and bias parameters to initialize the score to 0.01 for the focal loss.
1: <- (O):
2: K (K);
3: Reshape Q to Mx (D/v)) and transpose Q to (DXMx
4: Reshape N xxn((/))) and transpose to (D/(D/
À
E 2
VD/1
6: Á + MLP(Å) E MIXMxN
Reshape A R MxN.
8: À 1. !.A + b. where b = b, K bu:
9: A. < Á
10: ifw > then
11: A + + -
12: end if
15

---

Published as a conference paper at ICLR 2025
GT
patch
patch
GTX
P
3
PN
P1
P3
PN
Neewual Laeel
Q
GTC
Affinity matrix label
Q
-
0
2
1
AM
0
Q2
0
maaooa G
3
0
Q
0
1
Q
0
GT
Q.
Qu
1
query
query
Qg
bird-2
t-SNE
t-SNE
Q22
bird-1
Q5
GT₃ GT2
GT,
Qg Q
Category Assignment: Hungarian Matching
GT,=bird-1, GT,=bird-2, GT,=water
Qg. Q₂ are mismatched
Figure 9: The illustration of how to get the ground-truth affinity matrix B. The left is GT&&Q, the middle GT&&P and the right is Q&P. Line is the t-SNE visualization of category assignment.
query is set to be en the image has the et category label, and 'negative' otherwise. For a Type-Il prompt, eh, comes from the label assignment of the Hungarian matching algorithm. The matrix comes from the classification cost, mask focal loss cost, and dice cost. Hence, the objective of the m-th query is set to be cr, if it is positive (matched) query, and 'negative' otherwise.
Affinity matrix labels. We use Figure to illustrate how to obtain the affinity matrix label, B. The left figure represents the label assignment between the queries and ground truth; the different colors indicate different ground-truth units, where '1 means the query and the ground truth are matched and '0' means not. In the middle figure, G means whether each patch part of the ground-truth unit, where '1' means yes and 'O' means no. The colors are consistent with those in the left figure. Based on G, we compute the ground-trus affinity matrix B (sized M N, same as A), shown the right figure. For the m-th query that matches the k-th ground truth in label assignment (left column), the k-th row of G copied to the m-th row of B (with the same color); otherwise, B,, 0.
The bottom half of Figure 9 shows: (1) The left part is the category assignment processing. We use the Hungarian Matching to achieve one-to-one matching between queries and GTs. The matrix comes from the classification cost, mask focal loss cost, and dice cost. (2) The middle part is the t-SNE visualization between GTs (GT,, GT2, ...) and patches. After the category assign procedure, some queries are assigned to the GT's while others are not matched. (3) the right part is the t-SNE visualization which shows the relationship between patches and queries (Qi, ...).
A.3 EXAMPLES FOR ADVANTAGES AND LIMITATIONS
Advantages. We select the objects (labelled with yellow bounding boxes) that do not match any patches with an loU greater than 0.5. In Figure TO we show examples of objects that do not have complete patches from SAM. while SAM- CP can group the parts to the whole as the object representation. The first line shows a case where SAM fails to predict the whole pair of skis because they are separated. SAM-CP can group them into one instance, which aligns with the definition in COCO. In the second line, the motorcycle is divided into many messy units in SAM, but SAM-CP can group them into a whole instance.
Limitations.
We also give some examples of objects that are missed by SAM (or have a small IoU
with SAM patches). We summarize four scenarios in Figure IT (a) objects that are very small, (b) objects whose colors similar to the background, (c) objects that are partly occluded, and (d) object which has cluttered components. Situation (a) is the most serious because the grid-based sampling of SAM is insufficient to find all small targets, and sometimes the image resolution is not high enough for
16

---

Published as a conference paper at ICLR 2025
bus: 12 36.7
motorcycle
p
Figure 10: SAM-CP can group the parts to the whole for objects which do not have complete candidate patches. For the first line, SAM segment the pair of ski boards separately. But if our requirement is to treat pair of skis as one instance, only SAM-CP can put them together. For the second line, this complex scenario, SAM segments these fragments separately but does not provide complete motorcycle mask. SAM-CP can stitch these fragments together as fully as possible according to the requirements.
precise prediction. Situation (b) can cause SAM to segment incorrect patches due to indistinguishable low-level patterns (e.g., texture, color, etc.). Situation (c) shows that SAM cannot segment well if the object is occluded by other objects, and situation (d) shows that if the object consists of too many cluttered components, SAM cannot segment them very well. This situation limits the quality of SAM patches and, consequently, results in gap between the accuracy of SAM-CP and the state-of-the-art segmentation methods.
B DETAILS OF IMPLEMENTATION.
B.1 SETTINGS FOR COCO.
Data preparation. We the same data augmentation strategy in DETR-based detector Carion et al. (2020); Zhu et al. (202T); Zhang et al. (202a)). We randomly flip the image, resize the image ranging from 1333 480 1333 x 800, randomly crop the image, and randomly resize the image again. In the inference stage, we resize the image to 1333 x 800 without any other augmentation. For SAM proposal generation, IoU threshold, 0.9 stability score threshold, and 0.7 NMS threshold.
the everything mode in SAM with 48 points per side, 0.8
Settings for the model. We loss weights of 2.0, 1.0, and 1.0 for Cets. Cuft. and Ldice. and metric weights of 2.0, 1.0, 1.0, 1.0, and 1.0 for the cls, mfl, dice, bbox, and giou terms in the Hungarian matching algorithm. The batch size is 32, where we use 8 Tesla-V100 GPUs images per GPU) for the R50 experiments and 32 GPUs (1 image per GPU) for the Swin-L experiments. The learning rate is set to be 10-5 the beginning and is multiplied by 0.1 after the 8/16/40-th epoch when the total number of epochs is 12/24/50. The threshold in Section 3.2.3 is set to be 0.8. The number of patch encoders and unified affinity decoders are both set to be The memory usage of each GPU about 20G for R50 experiments, and training time is about 22 hours for 12 epoch experiments with 8 GPUS. Similar computing resources are needed in further studies.
Add patches from Mask DINO. We add the patches from Mask DINO Li al. 2023b) show that the upper bound of SAM patches limits our performance. If the quality of patches generated by the foundation model gets better in the future, the segmentation accuracy of SAM-CP will further increase. We saved the segmentation results of Mask DINO and deleted the label result and confidence score. Then, the segmentation results are viewed as the extra proposal and added the SAM patches.
17

---

Published as a conference paper at ICLR 2025
(a) bjjcts whose scales small
(b) bjject whose colors are similar to the background
(c) Object which is occluded
(d) Object which has cluttered components
The examples for the limitation of SAM patches, We have summarized the above four situations, and SAM itself has segmented the target. Our method's s premise assumption trust SAM's segmentation quality, so the above issues with SAM will limit our final performance. These limitations are also worth further investigation.
Figure 11:
B.2 SETTINGS FOR ADE20K.
Data preparation. We use the same data augmentation strategy in Mask2former Cheng al. (2022) in ADE20K dataset. The image is randomly flipped and resized in the range of [320, 1280). Then, the image is randomly cropped by 2560 640. In the inference stage, the image resized 2560 640. The
for SAM patch generation are the same as those for the COCO dataset.
Setting for the model. We use loss weights 4.0, 1.0. and 1.0 for Lolas Cunft. and Cdice. and metric weights of 4.0, 1.0, 1.0, 1.0, and 1.0 for the cls, mil, dice, bbox, and giou terms in the Hungarian matching algorithm. The batch size is 16 with a basic learning rate of 2 10-4 The total number of iterations is 160K, and the learning rate will be multiplied by 0.1 after 135K iterations. Other settings remain unchanged as the COCO experiments. The memory usage of each GPU about 20G for R50 experiments, and training time is about 55 hours for 160K iterations experiments (128 epochs) with GPUS.
B.3 SETTINGS FOR OPEN- VOCABULARY SECMENTATION.
Data preparation. The image pre-processing operations for training and inference on COCO and ADE20K are the same as the closed-domain experiments. When inferencing Cityscapes, the images are resized 2048 x 1024. When generating SAM patches, because most of the images in Cityscapes are 2048 1024, it is not suitable to resize the image to square, on the long side and then feed them into SAM.
crop the images
Setting for the model. The training typer-parameters are the same as those in close-domain experiments. Only the trainable backbone is replaced by a frozen CLIP image encoder. In the inference stage, the parameter described in Section 4.] is set 0.4. In addition, follow FCCLIP Yu et al. (2023a) to use the prompt engineering when inference. Each label can derive multiple entries, and chose the highest score as the classification score for this label.
18

---

Published as a conference paper at ICLR 2025
PQ
Apdet
AP
mloU
0.9
43.2
42.8
38.2
49.9
0.8
45.0
45.4
40.7
52.6
0.7
43.3
43.3
39.9
50.3
0.5
42.0
34.3
37.2
48.9
0.8 (w/o loxbox)
42.1
31.7
35.4
48.4
0.8 (w/o IoP mask)
39.7
42.3
32.5
44.1
0.8 (w/low quality match)
47.0
45.8
41.4
54.2
Table 12: Accuracy (%) on COCO (with R50, 12 epochs) with different definitions of parts in the training stage.
B.4 CODES AND DATASETS.
We provide codes in code.zip file in the Supplementary Material. After the acceptance of our paper, we will make our codes publically available.
The datasets are public datasets; their links provided here:
COCC/Ittpss//occodtaaet..oo
ADE20K:
C MORE EXPERIMENTS
C.I MORE ABLATION STUDIES
The matching threshold and mechanism. We study the matching threshold and mechanism described in Section 3.2.3 which is key affinity propagation. We ablate the threshold 7 and the choice of whether to use both the box-level and mask-level IoP in Table [2 We find that т = 0.8 is a proper threshold. On the other hand, removing either the box-level or mask-level IoP results in clear accuracy drop in visual recognition, implying the importance of improving the recall of patch merging. In addition, patch when it fails to match positive patch, this will bring performance gain. Therefore, more accurate mechanism may improve the overall segmentation accuracy, which we leave as future work.
have added low-quality matching, which will give an object a high IoU
C.2 MORE COMPARISON
Comparison with SAM-based method. Then, we compared our method with OVSAM [I] and prompt segment anything [3] on the COCO dataset evaluate the instance segmentation ability on Table I All other methods need pre-trained detector on COCO, and the detection results are used as the prompt for SAM. But our localization only relies on the original SAM, without tuning on COCO. The results show our performance is the best:
Method
Backbone
Detector
MAP
AP50
OVSAM [a]
R50
FRCNN
35,8
55.6
OVSAM [a]
Swin
Detic
36.7
57.2
Prompt segment anything [b]
Res50
H --Deormmable DETRRtricks
41.5
SAM CP(ours)
R50
41.7
59.4
Table 13: Instance segmentation comparison with SAM-based method.
The comparison between method and SEEM (Table 14) the COCO dataset is listed as follows. With the same amount backbone, we achieved higher performance. We also added comparison with Semantic SAM (Table 15). Even though it has point prompts in inference,
19

---

Published as a conference paper at ICLR 2025
method outperforms them by lot, which fully demonstrates that method is superior other SAM-based methods.
Method
Backbone
PQ
mAP
mloU
SEEM [c]
VIT
52.0
43.5
60.2
SAM CP(ours)
Swin
52.7
45.2
61.8
Table 14: Comparison of panoptic segmentation metrics with SEEM.
Method
Backbone
Prompt(testing)
mloU
Semantic SAM [d]
Swin L
point
57.0
Semantic SAM [d] (reproduce)
Swin I.
point
55.1
SAM CP(ou)))
Swin L
None
61.8
Table 15: Comparison semantic segmentation with different SAM-based methods.
By iteratively calling our composable prompts, SAM-CP achieves the instance and semantic segmen- tation based on the vision foundation model SAM without mask tuning. For SAM2-CP in the first table of this question, we substitute SAM with SAM2 [5] in SAM-CP. SAM2 faster than SAM. We utilize the CP trained on SAM-CP and transfer the weights to SAM2-CP without re-train, with the performance being 27.9 PQ. This indicates that our method is modular and can be extended to different patch generation methods (whether stronger faster). Although the current 3AM-based method is more time-consuming than the non-SAM-basee method, with the improvement of the segmentation accuracy and effectiveness of the SAM-based foundational model, the new bottom-up perception paradigm designed based on its high generalization ability is valuable, which is also one of our main motivations.
[a] Open- Vocabulary SAM. ECCV 2024
[b] Prompt segment anything, http::/:/w...toooooo.oc////////a
[c] Segment Everything Everywhere All at Once. NeurIPS 2023
[d] Semantic-SAM: Segment and Recognize Anything at Any Granularity, arXiv:2307.04767
Traing time comparison. addition, SAM-CP only needs 12 epochs of training, while 50 epochs required for FCCLIP and FrozenSeg.
D MORE VISUALIZATION RESULTS
D.1 VISUALIZATION DOMAANN EXPERIMENTS
For out-of-domain generalization, we conducted experiments on camouflage object segmentation and provided some visualizations in Figure 12
D.2
MORE T-SNE VISUALIZATION
Figure 1 shows more t-SNE visualizations of our approach. The features of patches belong to different semantic/instance groups different clusters in SAM-CP, Not that, in lines 6-10, the different instances with the same label ('train', 'elephant', "person', 'horse', and 'umbrella') cluster together to the same semantic area but separate to different instances in the t-SNE feature space.
D.3 MORE SEGMENTATION EXAMPLES
We show segmentation results in Figure 14 Columns 1 and 2 are the original images and SAM patches. Columns 3 and 4 are the ground truth and prediction of instance segmentation. Columns and 6 are the ground truth and prediction of panoptic segmentation.
20

---

Published as a conference paper at ICLR 2025
Figure 12: The visualization of camouflage object segmentation CODIOK to show the out-of- domain generalization. Left is the GTs and right the results.
E BROADER IMPACTS AND SAFEGUARDS
Broader Impacts. our view, our research may not have a negative societal impact. Firstly, the comparative methods and datasets used in our research are publicly available and do not make biased decisions that could unfairly impact specific groups. Secondly, our method does not fall under generative models; we aim to enhance the multi-task segmentation capabilities of models. There no direct pathways for our model to be applied in any negative manner. Therefore, our research does not pose any negative societal impact.
Safeguards. We have not released any data or models with a high risk of misuse. The proposed model is trained on benchmark datasets such COCO, ADE20K, and Cityscapes. These datasets publicly available, widely used in the field of computer vision, and have undergone comprehensive safety risk assessments. In this paper, and provide appropriate links in the references section. Upon completion of the review process, we will make the code and data of this work publicly available the community. This research does involve any human subjects experiments or studies, nor does it involve crowdsourced experiments or human subjects research. All experiments were conducted using codes and GPU servers.
clearly specify the sources of the datasets and code used
21

---

Published as a conference paper at ICLR 2025
Closes
Figure 13: More visualization of qualitative study on how SAM-CP works.
22

---

Published as a conference paper at ICLR 2025
LAR SOL
Derson 50
SOTT
Figure 14: More visualizations segmentation prediction. Columns and are the ground truth and prediction of instance segmentation. Columns 5 and 6 the ground truth and prediction of panoptic segmentation.
23


Reviews:
[{"review_id": "T7tiA1AMxQ", "reviewer": "Reviewer_xYh9", "summary": "The paper introduces SAM-CP, which integrates composable prompts for versatile segmentation. It utilizes two levels of queries—semantic and instance—to assign unlabeled patches from SAM with appropriate labels.\n\nIn rebuttal, the authors have presented additional experimental results that address many of my concerns. As a result, I am inclined to increase my rating to borderline accept.", "questions": "The design of SAM-CP relies solely on SAM's final predictions, disregarding intermediate features from SAM that could potentially enhance the decoder's performance. It would be valuable to include discussions on this aspect and to investigate whether integrating SAM's intermediate features with the proposed decoder could yield improved results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The overall architecture designs are well-grounded. The two types of prompts for semantic and instance understanding are intuitively designed to support different segmentation tasks.\n\n- SAM-CP models the two types of prompts with separated queries for better efficiency. The quantitative results on open-vocabulary segmentation also demonstrate the effectiveness of this query-based approach.\n\n- Thorough analysis is provided for the advantages and limitations of the proposed method.", "weaknesses": "- The running time might be a potential issue due to the two-level prompt design (especially if an iterative approach is used). A comparison of running time performance with the original SAM and other segmentation models would be informative.\n\n- Experiments: The current baselines do not include models based on SAM. Therefore, it's unclear whether the performance improvements come from the composable prompts or the internal capacity of SAM. Adding baselines building on SAM that perform instance [1] and semantic [2] segmentation can be incorporated to further demonstrate the effectiveness of the proposed composable prompts.\n\n- The figures are low-resolution and notably blurry. Replacing them with high-resolution versions would improve readability and presentation quality.\n\n- The performance of SAM-CP on closed-domain segmentation is not superior over other baselines.\n\n[1] Prompt Segment Anything (https://github.com/RockeyCoss/Prompt-Segment-Anything)\n[2] Semantic Segment Anything (https://github.com/fudan-zvg/Semantic-Segment-Anything)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5PAZNxzuy9", "reviewer": "Reviewer_SuUH", "summary": "This paper proposes a framework that combines SAM and CLIP for open-vocabulary universal segmentation.", "questions": "1. SAM can generate fine-grained masks for part, e.g. head, hand, body. Does SAM-CP have the potential to work on open-vocabulary part segmentation? Quantitative experiments and comparisons to related works like HIPIE would enhance the applicability of this work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed model keeps the zero-shot abilities of CLIP. This is supported by experiments on open-vocabulary segmentations.\n2. The ablation studies are sufficient and well explain the design choice of loss functions.\n3. This method is easy to follow, with many intuitive figures and visualizations.", "weaknesses": "1. The experiments on open-vocabulary segmentations (Table 1) are not well supportive. SAM-CP combines SAM and CLIP but is only comparable to FC-CLIP that is built with CLIP only. Also, the experiments on closed-vocabulary segmentations (Table 2) do not best address the potential of combining SAM and CLIP since it is worse than CLIP-only X-Decoder and supervised SOTA MaskDINO.\n2. The advantages of the proposed framework are not well discussed or presented. The performances are only comparable to previous SOTA. This paper should look for more perspectives to highlight its advantages, e.g. efficiency, part segmentation (also see Question 1), few-shot capabilities, out-of-domain generalizations, etc. \n3. It could be more convincing if this paper can compare to other SAM-based methods on the same settings and show its advantages. For example, [1] and [2] also work on open-vocabulary segmentation with SAM.\n4. As already discussed and illustrated in the paper, SAM-CP fails at small objects since there are no prompts on small objects. The efficiency is limited by SAM.\n\n[1] Open-Vocabulary SAM. ECCV 2024\n\n[2] Segment Everything Everywhere All at Once. NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zTEz8rv4zB", "reviewer": "Reviewer_pKRW", "summary": "The paper presents SAM-CP that extends SAM by incorporating composable prompts to improve segmentation. SAM-CP utilizes Type-I prompts to align SAM-generated patches with text labels and Type-II prompts to merge patches belonging to the same instance based on the text information. Experimental results show that SAM-CP performs well in open-vocabulary segmentation and shows some improvement in semantic discriminativity across various segmentation types.", "questions": "1. I would like to know if SAM-CP has effective strategies for addressing the aforementioned issues of segmentation ambiguity?\n\n2. Although the speed is influenced by SAM's computational efficiency, can you provide a detailed analysis of the speed of non-SAM modules and compare it with other SAM-based methods? Alternatively, could replacing SAM with a lightweight variant like EfficientSAM provide sufficient performance and computational efficiency for comparison with non-SAM methods?\n\n3. Does SAM-CP offer any advantages in training time compared to other SAM-based methods?\n\n4. Since SAM's segmentation results may not always be precise, the Affinity matrix used by the Unified Affinity Decoder could potentially accumulate errors. How is this issue addressed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work presents a method for improving SAM for versatile segmentation.\n\n2. SAM-CP considers segmentation tasks from a novel perspective, introducing two types of composable prompts: Type-I for matching SAM patches with text labels and Type-II for merging patches belonging to the same instance.\n\n3. The dynamic mechanism offers novel design insights for handling cross-attention computations.\n\n4. Generally, the paper is well-written.\n\n5. The figures and tables are informative.", "weaknesses": "1. Prompt I classifies each SAM-segmented patch, potentially leading to ambiguities. For example, without contextual information, it’s challenging to determine if a patch representing clothing on a person should be classified as part of the person or as an independent clothing item. The proposed CP lacks a clear strategy to handle the inherent ambiguity in segmentation without surrounding context.\n\n2. The performance gains are not significant, such as mIoU for the COCO->ADE20K and COCO->Cityscapes settings in Table 1.\n\n3. Table 2's comparison may be unfair due to the use of SAM, which is trained on a significantly larger scale. This complicates attributing improvements directly to the proposed method.\n\n4. Although SAM-CP's speed heavily relies on SAM, the paper lacks a clear analysis of the computational efficiency of the proposed non-SAM modules. It is crucial to examine whether these modules actually enhance efficiency.\n\n5. Other Issues:\n- Figure 1's many colors and mixed visual elements are hard to read.\n- The description of patches needs greater specificity. For example, as vaguely mentioned in Line 139, it's not specified if the patches are segments from SAM's segment-everything mode or just features from SAM's backbone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1DdUibIXaY", "reviewer": "Reviewer_3KqL", "summary": "**Objective**\n\nSAM-CP proposes a method to further refine the output of the Segment Anything Model (SAM). The aim is to transform SAM’s original, class-agnostic, variance-scale patches into two more informative segmentation types: Semantic Segmentation and Instance Segmentation.\n\n**Proposed Framework**\n\nThe training framework operates as follows:\n\n- Input: The framework takes the initial class-agnostic segments (CA-segments) from SAM, along with category text.\n- Processing:\nSAM Patches: The SAM patches are first processed through self-attention.\nQueries: Queries (which can be target categories in text or instance queries) are processed through self-attention, then used to refine the features by performing cross-attention with the SAM patches.\nAffinity Calculation: The resulting  N  patch features and  M  query features are used to calculate affinity for instance grouping.\nSemantic Classification: Finally, the learned features are compared with CLIP text input features for semantic classification.\n\nThe entire pipeline is trained in a fully supervised, end-to-end manner.\n\n**Results**\n\nExperiments show that this framework achieves state-of-the-art (SOTA) results when trained on one dataset and tested on another.", "questions": "1. At line 150, the authors state that the two prompts can be iteratively called when finer-level segmentation is required. I’m not entirely sure I understand this; my impression is that granularity is determined by SAM’s results. Could the authors clarify this point?\n\n2. How do the authors view this approach in comparison to SAM-based generalized segmentation? Most other approaches appear to be training-free and straightforward to use. It would be interesting to discuss the advantages that SAM-CP offers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I believe this task is important. Transforming the class-agnostic SAM into a usable semantic and instance segmentation is a valuable pursuit. This directly connects to the question of how to leverage the advance of large-scale vision foundation models into practical application.\n\n2. The proposed method demonstrates strong performance on commonly used open-world segmentation frameworks, achieving state-of-the-art (SOTA) results when trained on one dataset and tested on another. This includes benchmarks such as: COCO → ADE20K; ADE20K → COCO; COCO → Cityscapes\n\n3. The overall experinement and results presentation is good.", "weaknesses": "# At a higher level:\nThe advantage of SAM lies not only in its ability to segment an entire image but also in its prompt-based segmentation, which introduces more accurate masks.\nI think this method uses SAM only as an initial “superpixel” generator, which somewhat overlook SAM’s full potential.\n\n# Generalization:\nThe proposed method requires training on specific datasets to learn how to merge patches, which seems to be not very generalizable.\nThis approach may not translate well from natural images to medical images, or even from natural images with large object segments to tasks requiring finer, more detailed segmentation. Is this the case in this setting. Would like to hear author feedback on this point. \n\n# Splitting and Merging:\nThe proposed method only focuses on merging, which to some degree simplifies the problem. I believe splitting is equally important as merging, though it seems it is not within the scope of this method. \n\n# Comparison with Other Methods:\nMost of the results are compared with pre-SAM methods. I wonder how this method compares with other SAM-based semantic segmentation/instance segmentation methods, such as Grounding SAM and SEEM etc. It would be interesting to see a comparison with other SAM-based methods and discuss the advantage of such approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)