Unlocking the Power of SAM 2 for Few-Shot Segmentation
Authors: Qianxiong Xu
Lanyun Zhu
Xuanyi Liu
Guosheng Lin
Cheng Long
Ziyue Li
Rui Zhao
Venue: ICML 2025 poster (2025)
Keywords: Few-Shot Segmentation
Segment Anything Model 2
Video
Intra-class Gap
Primary area: applications->computer_vision

Abstract: Few-Shot Segmentation (FSS) aims to learn class-agnostic segmentation on few classes to segment arbitrary classes, but at the risk of overfitting. To address this, some methods use the well-learned knowledge of foundation models (e.g., SAM) to simplify the learning process. Recently, SAM 2 has extended SAM by supporting video segmentation, whose class-agnostic matching ability is useful to FSS. A simple idea is to encode support foreground (FG) features as memory, with which query FG features are matched and fused. Unfortunately, the FG objects in different frames of SAM 2's video data are always the same identity, while those in FSS are different identities, i.e., the matching step is incompatible. Therefore, we design Pseudo Prompt Generator to encode pseudo query memory, matching with query features in a compatible way. However, the memories can never be as accurate as the real ones, i.e., they are likely to contain incomplete query FG, and some unexpected query background (BG) features, leading to wrong segmentation. Hence, we further design Iterative Memory Refinement to fuse more query FG features into the memory, and devise a Support-Calibrated Memory Attention to suppress the unexpected query BG features in memory. Extensive experiments have been conducted on PASCAL-5$^i$ and COCO-20$^i$ to validate the effectiveness of our design, e.g., the 1-shot mIoU can be 4.2\% better than the best baseline.

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Qianxiong Xu
Lanyun Zhu?
Xuanyi Liu Guosheng Lin
Cheng Long'
Ziyue Li
Rui Zhao
Abstract
Few-Shot Segmentation (FSS) aims to learn class- agnostic segmentation few classes to segment arbitrary classes, but at the risk of overfitting. To address this, some methods use the well-learned knowledge of foundation models (e.g., SAM) to simplify the learning process. Recently, SAM 2 has extended SAM by supporting video segmen- tation, whose class-agnostic matching ability useful FSS. A simple idea is encode sup- port foreground (FG) features memory, with which query FG features are matched and fused. Unfortunately, the FG objects in different frames of SAM 2's video data are always the same iden- tity, while those in FSS are different identities, i.e. the matching step incompatible. Therefore, we design Pseudo Prompt Generator to encode pseudo query memory, matching with query fea- tures in compatible way. However, the mem- ories can never be as accurate the real ones, i.e., they are likely to contain incomplete query FG, and some unexpected query background (BG) features, leading to wrong segmentation. Hence, we further design Iterative Memory Refinement fuse more query FG features into the memory, and Memory Attention
(a) Video
(b) Naive SAM for FSS
Support
upport Memary
Memory Encoder
Support Mask
Similar
Lisslar
Incompetiole
Memory Attention
F rame #100
Query
1 uery Feat
Our Main Idea
Support
Query M Memory
Pseudo Prompt Generator
Memory Encoder
Support
Query Prior
Compatiole
Memory Attantion
More similar
Query
Query Feat
(d) Iterative Memory Refinement
(e) Support-Calibrated Memory Attention
Delect and Suppress 8G
Prior
Disc Prior
Refined Prior
Refined Prior (C&V)
GT
Figure I. Illustrations of (a) video data, (b) simple use of SAM
2, (c) main idea. (d) prior masks and our Iterative Memory Refinement, and (c) our Support-Calibrated Memory Attention, In (a), SAM 2's learned knowledge is same-objects matching. In (b), the objects in FSS are different, posing challenges to use SAM 2's knowledge. (c), we generate prior masks and encode them as pscudo query memories to enable same-objects matching, In (d) and (e), we use priors visualize memories, and fuse more query FG, while suppress the unexpected query BG features in memory.
devise a Support-Calibraan to suppress the unexpected query BG features in memory. Extensive experiments have been con- ducted on PASCAL-5' and COCO-20' to validate the effectiveness our design, e.g., the 1-shot mloU can be 4.2% better than the best baseline.
seen significant advancements (Long et al., 2015). Unfortu- nately, most of them depend on extensive pixel-wise annota- tions, requiring massive time and human efforts. Besides, they cannot generalize to novel classes that were not present during training, struggling to segment arbitrary classes.
1. Introduction
Semantic segmentation is fundamental vision task that involves assigning class labels to each pixel in an image, which is essential for detailed scene understanding, and has
SS-Lab, Nanyang Technological University "Singapore Univer- sity of Technology and Design 'Peking University 'University of Cologne TeenseTime Research. Correspondence Guosheng Lin <gslin@ ntu.edu.sg>, Cheng Long <c.long @ @ntu.cdu.sg>.
Few-Shot Segmentation (FSS) (Shaban et al., 2017) has been introduced to address these issues, which can segment arbitrary classes by referring to few annotated samples with the target class. The rationale lies in similarity- feature matching, i.e., find matched objects a query image, whose features are similar to that of annotated support sam- ples. FSS would learn such pattern from some base classes,
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
and apply it to segment novel classes without re-train or fine-tune, making it cheaper and generalizable solution.
strong, these BG features may lead wrong segmenta- tion of query BG objects. Therefore, we follow (Xu et al., 2024a) to suppress them and additionally generate discrimi- native (Disc) prior masks and memories. As shown in Fig- ure ((d) and Figure 5(b), existing FG priors usually include more complete FG but activate more BG regions, and Disc priors have less complete FG yet activate less BG regions.
Existing methods roughly include prototypical (Tian et al., 2020) and attention-buas -bateddn- methods (Xu et al., 2023). Proto- typical methods compress support foreground (FG) features into a few prototypes, and segment query image via fea- ture comparison (Wang et al., 2019) or fusion (Zhang et al., 2019b). Attention methods (Zhang et al., 2021) employs cross attention to measure the feature similarity between query and support FG pixels, dynamically fuse the latter's s features to the former. However, novel classes are unknown until testing, thus, these models can easily get overfitting on base classes, hindering the use of more parameters to learn better class-agnostic segmentation knowledge.
As Disc memory includes incomplete query FG and few query BG features, hindering FG-FG matching, we further design Iterative Memory Refinement (IMR) and Support- Calibrated Memory Attention (SCMA) modules as fol- lows. As displayed in Figure (d)) we design IMR to complement query FG features from FG memory to Disc memory, where the support information is used to restrain the propagation of query BG features. Hence, Disc memory would have more complete query FG features, making the matching with FG objects easier. After IMR, Disc memory may include some BG features, which (1) are initially con- tained in Disc memory, and (2) are fused from FG memory. Therefore, we further devise a SCMA (Figure (())) to cali- brate the cross attention scores between query features and the memory, where the BGs in memory will be detected and their scores will be suppressed. Thus, query features are fused with less BG features, leading to accurate predictions.
To tackle this issue, recent advancements (Sun et al., 2024; Zhang et al., 2024) deploy foundation models (e.g.. DI- NOv2 (Oquab et al., 2023) and SAM (Kirillov et al., 2023)) to benefit from their fine-grained image features or the promptable image segmentation capability, making the learning of FSS easier. Recently, SAM 2 (Ravi et al., 2024) extends SAM by enabling promptable video segmentation, and has shown remarkable FG-FG matching ability among video frames, which may benefit FSS.
As shown in Figure (bb), SAM 2 can be directly applied by encoding support features (frame #1) and its mask (prompt) as support memory, used to dynamically match and fuse query FG features (frame #2). However, the FG-FG match- ing of FSS is incompatible with that of SAM 2, as the train- ing videos (Figure 1(a)) of SAM 2 always have uniform FG objects (e.g., human) across frames, yet those in query and support images in FSS are always different (e.g., ships). As an object is essentially more similar to itself rather than oth- ers, the FG-FG matching in videos is naturally easier than that in FSS data, i.e FSS needs to re-learn these parameters to facilitate FG-FG matching and fusion between differ- ent objects. As result, SAM 2's robust parameters will degenerate back to narrow ones, i.e., overfit base classes.
Our contributions include: (1) We are the first to adapt SAM 2 for FSS, benefiting from its remarkable matching ability. We start with a simple way, and identify the incompatible matching. (2) We design PPG to encode pseudo query memories, enabling compatible FG-FG matching, aligned with SAM 2's well-learned parameters. (3) Pseudo query memories always include incomplete FG and unexpected BG features, so we design IMR to enrich query FG fea- tures, and SCMA to suppress the matching and fusion with query BG features. (4) Extensive experiments have been conducted to validate the effectiveness of our design. We set new state-of-the-are on PASCAL-5' and COCO-20', e.g. the 1-shot mloU scores are 81.0% and 62.3%, respectively.
In this paper, we present Few-Shot Segment Anything Model (FSSAM) to appropriately use the matching abil- ity of SAM 2, i.e., convert the matching between differ- ent objects (support&query) to that between same objects (query). The question is how to construct prompts for en- coding pseudo query memories. As leanning-agnostic prior masks (Tian et al.. 2020) can roughly locate query FGs by measuring feature similarities between query and sup- port FG pixels, we design Pseudo Prompt Generator (PPG) (Figure 1(c)) to generate and take prior masks as pscudo mask prompts. However, existing prior masks, af- fected by feature ambiguity (Xu et al., 2024a), will acti- vate many query background (BG) regions (Figure ((d), i... the encoded memories will include many query BG features. Since the matching ability of SAM 2 is quite
2. Related Work
Classical FSS. Based on the way of utilizing support sam- ples, existing methods can be categorized into prototypi- cal (Shaban al., 2017: Wang et al., 2019; Tian et al., 2020; Fan al., 2022; Wang et al., 2024a), attention-based (Zhang et al., 2019a; 2021: Wang et al., 2020; Hu et al., 2019; Xu
et al., 2023; Hong et al., 2022; Xiong et al., 2022; Shi
et al., 2022; Liu et al., 2023a; Park et al., 2024; Iqbal et al., 2022; Peng et al., 2023; Wang et al., 2023a; Xu et al., 2024a), mamba-based (Xu ct al., 2024b), and text-based methods (Yang et al., 2023; Zhu et al., 2024; Chen et al., 2024; Wang et al., 2024b). Prototypical methods usually extract few prototypes from support features, then either measure pixel-wise feature similarity (Fan et al., 2022) or
2

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Parameter
Frozen
Image ncoder
Memory Encoder
Fs
Parameter
Finetune
Dise Memg
Memr
Mems
Pseudo Prompt Generator (PPG)
Mrs
FGGBG Prototypes
ReLU
PQ
pise
Iterative Memory Refinement (IMR)
Mq
GAP
CosSNom
R Refined Memq Plac
Mask Decoder
Pd
Support-Calibraed Memory Attention (SCMA)
Image ncoder
Fe
lg
Figure Overview FSSAM, which includes: (1) Pseudo Prompt Generator generates a pair FG (with more complete FG more wrongly activated BG regions) and Disc priors (with less complete FG yet less wrongly activated BG regions) for encoding pseudo query memories M FG and Mem Dixe, (2) Iterative Memory Refinement aims to iteratively complement FG features from M Memq
FG
Mem2 and (3) Support-CCllirrtee Memory Attention helps suppress the unexpected BG features in refined Mem Disc.
fuse (Tian et al., 2020) with query features to segment FG objects. Nevertheless, the compression of support features to prototypes would inevitably result in information loss, so attention-based methods propose take advantages of cross attention (Vaswani et al., 2017) to dynamically fuse each query pixel with uncompressed support FG features. As the computational complexity of attention is quadratic to feature sizes, HMNet (Xu et al., 2024b) design a cross attention-like mamba (Gu & Dao, 2023) to take the role of attention, but keep the complexity linear. These methods uniformly learn segmentation knowledge from a few base classes, and can easily get overfitting on them, failing to well segment arbitrary novel classes.
3. Problem Definition
The objective of FSS is to segment objects from arbitrary classes, using few annotated support samples of each class. The training and testing datasets, denoted as Draai and Drest- contain disjoint sets of classes, respectively represented as Crase and Covel, where Cbasc 7Cnovel 0. This ensures that no classes in Dirain appear in Dueat making FSS challenging problem of generalizing to unseen classes. To achieve this, FSS employs episodic training, where both Drain and Drest are divided into a series of episodes. Each episode in a k-shot setting comprises support set (7g, MY), n==1 query set (lq, MQ), all related to particular target class. Here, Iğ and M9 denote the n-th support image and its binary mask, while IQ and Mq represent the query image and its corresponding mask. During training, the model learns use the support S to guide segmentation for classes in Coase- In testing, the model applies this learned segmentation pattern to previously unseen classes in Cnovel-
) and
Foundation-based FSS. Some methods (Liu et al., 2023b;
Sun et al., 2024; Chang et al., 2024; Zhang et al., 2024)
have proposed incorporate foundation models (e.g.. DI- NOv2 (Oquab al., 2023), SAM (Kirillov al., 2023)) into FSS, using their well learned knowledge to simplify the learning of FSS. Particularly, SAM has shown excellent alass-agnostic ability of promptable image segmentation, e.g., if any prompts like points or bounding boxes are pro- vided, the relevant objects can be well segmented. Some efforts have been made to leverage such ability, e.g., VRP- SAM (Sun et al., 2024) designs Visual Reference Prompt Encoder to generate prompt embeddings for SAM. Besides, GF-SAM (Zhang et al., 2024) leverages graph analysis to ex- tract point prompts for SAM. Recently, SAM 2 (Ravi et al., 2024) has extended SAM by enabling promptable video segmentation, whose FG-FG matching ability is excellent, so we propose to leverage such ability in this paper.
IQ
4. Methodology
As shown in Figure we design FSSAM use SAM 2's excellent matching ability facilitate the segmentation of query FGs. Firstly, query and support images IQ and 7g are forwarded to image encoder to extract features F2Q and Fs. Meanwhile, they are forwarded to Pseudo Prompt Generator (Section 4.1) to obtain FG and discriminative (Disc) prior masks PFC and PDise, Then, support features Fs and support mask prompt Ms are encoded as support memory Memg via memory encoder. Similarly, p PPG and P Diss
3

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
P
Cosine 6M
BG
GAP
Suppress (Eq. (8))
Age
le
Memo
Prog
Age
Memory Refinement [Eq. (9))
Refined Mem Pisc
Mg
Cosine GN
Pe
GAP
Memn
Pros
Ags
Refined
Memb
plese
PQ
Figure 3. Details of Iterative Memory Refinement (IMR). IMR refines Disc memory Mema Mem Disc by measuring its similarity with memory M Mem EG and incorporating sufficient query FG features from the latter into the former. Mcanwhile, support memory M Mems (only FG) is used prevent from fusing many BG features from Mems FG so the refined Disc memory M Memq while still include few BG features. IMR supports iterative refinement.
Disc can have more complete FG,
are used to encode query features Fe as pseudo query mem- ories Memq and Mem After that, Iterative Memory Refinement (Section 4.2) iteratively refines MemQ Disc com- plementing FG features. As M Mem always contains BG features, we devise a Support-Calibrated (Section 4.3) to suppress the unexpected BG features in MemQ Disc during memory attention, thus, query features Fe can be appropriately fused as Fo. decoded by mask decoder to obtain the final predictions Me.
Then, we follow AENet (Xu et al., 2024a) to compress FDINO into FG and BG prototypes via global average pool- ing. After that, two normalized cosine similarities, PFG and Q are measured between prototypes and F FDINO Finally, Disc prior mask Poise is obtained via a subtraction from PFG PBG and the negative values are set as
Calibrated Memory Attention
Proş GAP(FPINO, ,M))
(2)
P5 = Reshape TQINO, (
4.1. Pseudo Prompt Generator
Pe Disse Norm(ReL.U(PP00 - PPC))
Motivation. Recall that the straightforward way to adapt SAM 2 for FSS is to take support image Is as the first frame, use the annotated mask Mg to encode its features F's as support memory Mems, which is then propagated into the second frame, i.e., query features Fq, via memory attention. Nevertheless, SAM is trained abundant video data, where the FG objects in different frames are always the same. Instead, the FG objects in FSS always correspond to visually-different identities, denoted as intra-class gap (Fan et al., 2022; Xu et al., 2023). Therefore, SAM 2's FG-FG matching ability is incompatible with that FSS. Inspired by the FG locating ability of prior masks (Tian et al., 2020; Xu et al., 2023; 2024a), we design Pseudo Prompt Generator (PPG) alternatively encode query features FQ pseudo query memories, so the FG-FG matching and fusion can be compatible with the trained ones.
where E (FG, BG). GAP(.) means global average pool- ing, Norm(-) is min-max normalization. p&G and P&G show the probabilities of query pixel being FG or BG, and Po PDise means whether a pixel is more likely to be FG than BG, appearing to be more discriminative prior masks. Some examples are included in Figure 5(b) and Appendix D.2.1.
Memory Encoding, After obtaining prior masks, query and support features Fq and Fs will be encoded as memories:
M Mems = ME(Fs, Ms)
Mem FG ME(Feg P
(5)
M Memo MemDisc.
Disc ME Po prise)
where MB(-) SAM 2's memory encoder.
Extension to k-shot. Each support sample is used to gen- erate pair of FG and Dise prior masks, averaged PFG and Po Besides, support samples will be encoded into k support memories Memg accordingly.
Pseudo Prompt Generation. As shown in Figure 2, we follow existing works (Liu et al., 2023b; Zhang al., 2024; Chang et al., 2024) to deploy DINOv2 (Oquab al., 2023), a powerful vision foundation model trained with discrimina- tive self- celf-supervised learning paradigm, to generate discrimi- native query and support features as:
4.2. Iterative Memory Refinement
Motivation. Both PFC and Po Pisse have been encoded into pseudo query memories, however, as shown in Figure 3,
F8 DDINO- DIN) FDINO DINOv2(Ts)

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
PeG always has more complete FG but activates more BG regions, while Pa always has less complete FG yet ac- tivates less BG regions. Note that (1) if there are more complete FG features in memory, the FG-FG matching in memory attention will naturally be easier; (2) SAM 2's matching ability is quite strong, so the unexpected BG fea- tures can easily lead to wrong segmentation of BG objects. As Poise has much less BG than P0 we take M Memq Disc as the main memory, and selectively fuse extra FG features from Mem FG During this process, Memg is used to re- strain the incorporation of redundant BG features.
where MR(-) includes Equation (6) to Equation (9), and is iteration times. In general, with the increase of n, the refined memory will have more complete FG but more BG features. Some visual impacts are depicted in Figure S(c).
Extension to k-shot. Each of k support memories will calculate a similarity, averaged as Ags.
Memory Complexity. As cosine similarities are measured between prototypes (each with 1 pixel) and FG memory (N HW pixels), the cost is On (k + N) under k-shot setting. refine n times, where n < N and k < N.
Memory Refinement. As shown in Figure 3. we first com- press Mem Disc and Memg into prototypes Propisc and Pros. Then, we measure the cosine similarities between prototypes and M M
4.3. Support- Memory Attention
FG
as:
Ms
Cosine SNorm
Agg (1 x N)
X M Matrix Mulppicatioo
Prod Disc = GAP(Mem Disc Peiiig
GAP
Pro;
E lemmontwwise Sum
Moms
Pros = GAP(Mems, Ms)
BG S Suppress
AQq = Reshape(Norm((os(( FG .ProQ Disc))
Men
Of
Safmax
Aqs =
(wannCCossseeeeee Memo
FG Pros)))
Atlen GNorm
Ago
where Aqq E #x and Ags E H represent the
OY N)
Fg
Fg
overall similarities of Mem Diec and Mems to Mem FC
As there exist BG features in M Mem Disc, Aqq appears to have relatively larger similarities to BG (e.g.. non-ship) re- gions, i.e., directly taking this score to refine M MemDien Disc will take in too many unexpected BG features. As M eemg con- tains pure FG features, we can observe Aqs has relatively smaller scores on query BG regions. Therefore, we further propose BG Suppress mechanism to filter Aqq, whose essence is preserving those regions that are similar to both Mem Diec (FG&BG) and M Memg (only FG):
Figure 4. Illustrations of Support- Caalib--a Memory Attention (SCMA). We only show cross attention in this figure. When per- forming cross attention between F4g and Mem (FG&BG), the irrelevant memory (BG) will be suppressed by Mems (FG).
Motivation. Remind that our goal is to refine Mem Disc, trying to (1) make its FG features more complete, and (2) mitigate the side-effects of entangled BG features. In Sec- tion 4.2, IMR only focuses on the first point, with the second point untouched, i.e., IMR cannot drop initially-included included BG features but might fuse more BG features into Disc mem- ory. Hence, we further devise Support-Calibraate Memory Attention (SCMA) address this issue. The main idea is preventing query features FQ from fusing those memory features that are less likely to be FG.
Aqq ReLU(AAQ + (Aqs - 1))
(8)
where Ago E [0, 1| is regarded as the weight to fuse (1) Memb Disc and Mem FG (2) P@ and PFG:
M Memo
FG + (1 111-ARm AQQ). Memb Disc Disc Aqq - -Mem
Support- Calibrated Cross Attention. As shown in Fig- ure 4, query features Fq will be projected as Q E NxC. where N HW and C is hidden dimension. Similarly, Mem is projected as Kss E V and Mem Disc is pro- jected as KQQ E NxC and Vo E RNCC for feature fusion:
phise = Aqq PhC + - AQo) pein.
(9) where- is wise multiplication. In Figure 3, the refined prior mask Poise (1) has more complete FG regions than Poise and (2) has less BG regions than P5C i.c., the pseudo query prompts and memories get better.
Q Proj(FQ, Oo), Kcs = Proj(Mems, Ok), Kiq Peej
Disc VQ= Proj(MemQ Ov)
Diec ①x)
Iterative Mechanism. We further propose iteratively refine M Mmmiiee Disc and prise to incorporate more FG features from M em which can be expressed as:
(11)
where θ represent the parameters for projection. Note that Kgq and K.s share the same projection parameters Øx-
M MemBis Disc
Disc pDisc_ = MR(Mem Disc, PP Memb. FG Mems,, Ms,n)
P.Dise
Following SAM 2, standard cross attention (i.e., scaled dot product) is conducted between XCQ and Q, to obtain the attention score Ago E N Then, a global average
(10)
5

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Table 1. Quantitative comparisons with state state-of-the of-thh-arts arts on PASCAL-S'. "5t" denotes the mloU score of the i-th fold, "Mean" is the averaged mloU score of 4 folds, "FB-IoU" is averaged from folds. Bold values show the best performance.
I-shot
5-shot 5
Method
59
5
Mean
FB-IoU
Mcan
FB-IoU
Clavsical F3S
ABCNet (CVPR'23) 233) (Wang al., 2023h)
68.8
73.4
62.3
59.5
66.0
76.0
71.7
74.2
65.4
67.0
69.6
80,0
SCCAN (ICCV 23) (Xu al., 2023)
68.3
72.5
66.8
59.8
66.8
77.7
72.3
74.1
69.1
65.6
70.3
81.8
RiFeNet (A))I (Bao 2024)
68.4
59.4
69.4
64.2
69.6
MIANet (CVPR 23) (Yang et al., 2023)
68.5
75.8
67.5
63.2
68.7
79,5
70.2
77.4
70.0
68.8
71.6
82.2
HDMNet (C)PR223) (Peng ct al., 2023)
71.0
75.4
68.9
62.1
69.4
71.3
76.2
71.3
68.5
71.8
(AAA1244) (Wang et al., 2024a)
71.1
75.5
67.0
64.5
69.5
74.7
78.0
75.3
70.8
74.7
AENet 24))))22) et al., 2024a)
72.2
75.5
68.5
63.1
69.8
80.8
74.2
76.5
74.8
70.6
74.1
84.5
AMNet (NIPS'23) (Wang al., 2023a)
71.1
75.9
69.7
63.7
70.1
73.2
77.8
73.2
68.7
73.2
HMNet (NIPS'24 24) (Xu ct al., 2024b)
72.2
75.4
70.0
63.9
70.4
81.6
74.0
77.2
74.1
70.5
73.9
84.4
Foumdation-baxed FSS
Matcher (ICLR'20 (Liu et al., 20236)
67.7
70.7
66.9
67.0
68.1
71.4
77.5
74.1
72.8
74.0
VRP SAM (CVPR'24) 24) (Sun et al. 2024)
73.9
78.3
70.6
65.1
71.9
GF-SAM (NIPS'24 24y (Zhang et 2024)
71.1
75.7
69.2
73.3
72.1
81.5
86.3
79.7
82.9
82.6
FounFSS (Arxi'24) (4) (Chang al., 2024)
76.5
81.3
72.1
77.4
76.8
79.5
84.8
75.8
82.5
80.7
FSSAM (Ours)
81.6
84,9
81.6
76.0
81.0
89.4
84.1
88.5
83,8
85.0
85.4
91.9
pooling operation is carried out on Mems to compress its FG memory features into Pros E RIIC As the FG objects in query and support images are different, and cosine simi- larity is a much looser similarity operator (Xu et al., 2023) than scaled dot product, we calculate cosine similarity be- tween Pros and Kg encourage better FG-FG matching, and obtain similarity score Asq E [11,1]
5. Experiments
5.1. Experiment Setup
Datasets. We evaluate FSSAM two benchmarks, in- cluding PASCAL-5' (Shaban et al., 2017) and COCO- 20 (Nguyen & Todorovic, 2019). PASCAL-5* includes 20 classes, built upon the PASCAL VOC 2012 (Ever- ingham et al., 2010) with additional annotations from SDS (Hariharan al., 2014). COCO-20', derived from the MSCOCO (Lin et al., 2014), is more challenging, compris- ing 80 classes. Both datasets are divided into disjoint folds for -ross-validation, with each fold containing 5 classes for PASCAL-5 and 20 classes for COCO-20* In each iteration, 3 folds are used for training, and the remaining fold is used for testing. Following existing works, we randomly sample 1,000 episodes for testing by default, while the results for er- ror bars evaluation and different number of testing episodes are included in Appendix C. and Appendix C.2.
2 KQ di
Prog @ Kg
Agq
Asq
(12)
Prosl & [Kol++e
where @ means matrix multiplication, de denotes the hid- den dimension of K., and is small constant to avoid O. Next, both Aqq and Asq will be normalized into [0, 1]. BG Suppress is then performed to suppress the attention scores of irrelevant BG pixels in Aqo as:
1/20 Norm Norm(Aqq) (Norm (Asq)- 1)
(13)
Aqq Aqq + a- NQoo
Evaluation Metrics. Mean intersection over union (mloU) IoU (FB-IoU) are utilized.
where a is a scaling factor and is empirically set as 10, AlqQ means preserving those entries with negative values. Then, Aqq is normalized by softmax and used to aggregate features from Memgise Disc and projected as Fe. which will be fused with input features Fo via skip connection:
and Implementation Details. Please refer to Appendix B.
5.2. Comparisons with State-of-the-Arts
The quantitative comparisons between our FSSAM and pre- vious state-of-the-ard are shown in Table and Table 2 for PASCAL-5 and COCO-20'. Specifically, we select some classical FSS methods, as well as more recent foundation- based methods for comparisons. Following existing works, the comparisons are conducted under both 1-shot and 5-shot settings. We could draw the following conclusions from the tables: (1) Foundation-basee methods consistently behave better than classical methods, showing the necessity of intro- ducing foundation models to take over some of FSS models' duties, e.g., the matching ability can thus be less likely to
Fq- FQ + Proj(Softmax(Aqq) & VQ. tow))
(14)
where Oout used for output projection. The visual im- pacts of SCMA is shown in Figure 6.
Extension to k-shot. The k support memories will be used to measure & cosine similarities, and averaged to be Asq-
Memory Complexity. Remind that the memory cost of original cross attention has already been O(N2), and the extra cost of our design O(kN) under shot setting.
6

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Table 2. Quantitative comparisons with state-of-thh of-the-artee arts on COCO-20 2000--20..00 "20"" denotes the mloU score of the i-th fold, "Mean" is the averaged mloU score of 4 folds, "FB-IoU" is averaged from folds. Bold values show the best performance.
I-shot 20
5-shot 20
Method
20
20
Mean
FB-IoU
20
20
Mcan
FB-IoU
Clavsical FSS
ABCNet (CVPR'23) (Wang al., 2023h)
42.3
46.2
46.0
42.0
44.1
69.9
45.5
51.7
52.6
46.4
49.1
72.7
RiFeNet (4)AI'24) (Bao et al., 2024)
39.1
47.2
44,6
45.4
44.1
44,3
52.4
49.3
48.4
48.6
SCCAN (ICCV'23) 23) (Xu al., 2023)
49.6
45.6
69.9
57.2
59.2
52.1
53.9
74,2
MIANet (CVPR 23) (Yang et 2023)
42.5
53.0
47.8
474
47.7
71.5
45.8
58.2
51.3
51.9
51.7
73.1
PAM PAM (AAAI'24) (Wang et al., 2024a)
44.1
55.0
46.5
48.5
48.5
48.1
60.8
54.8
51.9
53.9
(BCCY (Xu al., 2024a)
43.1
56.0
50.3
48.4
49.4
73.6
51.7
61.9
57.9
55.3
56.7
76.5
HDMNee (CVPR'23) (Peng et al., 2023)
43.8
55.3
51.6
49.4
50.0
72.2
50.6
61.6
55.7
56.0
56.0
77.7
AMNet (NIPS'23) (Wang nl., 2023a)
44.9
55.8
52.7
50.6
51.0
72.9
52.0
61.9
57.4
57.9
57.3
78.8
HMNet (NIPS'24 24) (Xu ct al., 2024b)
45.5
58.7
52.9
51.4
52.1
74.5
53.4
64.6
60.8
56.8
58.9
77.6
Foundation-baxed FSS
Matcher (ICLR'22 (Liu et al., 20236)
52.7
53.5
52.6
52.1
52.7
60.1
62.7
60.9
59.2
60.7
FounFSS (Arxiv/24) (2) (Chang al. 2024)
56.0
61.3
57.9
58.8
58.5
61.4
69.4
65.9
64.9
65.4
GF-SAM (NIPS'24) 24) (Zhang et 2024)
56.6
61.4
59.6
57.1
58.7
67.1
69.4
66.0
64.8
66.8
VRP-SAM (CVPR'22 24) (Sun al., 2024)
56.8
61.0
64.2
59.7
60.4
FSSAM (Ours)
59.9
65.6
62.1
61.6
62.3
77.3
68.6
74.0
64.5
69.9
69.3
82,9
incompatible matching. When we deploy PPG to enable compatible same-objects matching, the score can be greatly improved from 69.4% to 78.2%, showing the reasonability of our main idea. Nevertheless, the generated prior masks al- ways have less FG yet more BG regions than the real masks. When we use IMR to introduce more query FG features into the memory, the mloU can be further improved by 1.4%. Mcanwhile, if we deploy SCMA to suppress the unexpected BG features during cross attention, the score is 79.0%. Once IMR and SCMA are both utilized, the final score can reach 81.0 11.0% setting new state-of-the-arts
Table 3. Component-wise ablation study. "PPG", "IMR" and "SCMA" are Pseudo Prompt Generator, Iterative Memory Refine- ment and Support- Calbbbrte Memory Attention, respectively.
PPG
IMR
SCMA
5²
5²
Mean
FB-JoU
71.6
59.9
69.4
80.3
79.0
82.1
78.4
73.2
78.2
87.3
79.4
75.2
79.6
88.3
80.0
82.8
79.2
74.1
79.0
87,8
81.6
74.9
81.6
76.0
81.0
89.4
overfit the base classes. Besides, their performance gap appears to be larger on COCO-20' dataset, and we attribute this to the fact that the samples of COCO-20' are much more complicated than those of PASCAL-5', including tiny objects, complex BG, etc. Therefore, models need to learn more complex knowledge to handle them, while the risk of overfitting also increases. (2) FSSAM can surpass other baselines by large margins, setting new state-of-the-arts. For example, the --shot mloU score is 4.2% better than that of the best baseline FounFSS, and the FB-IoU score can reach 89.4%. Under 5-shot setting, the gap is more prominent (e.g., 85.4% v.s. 80.7%), demonstrating its effectiveness.
Qualitative Results and Pseudo Prompts. To have clearer understanding of FSSAM, we we jointly visualize some qualitative results of PPG, IMR and final predictions in Fig- ure 5, where: (1) In (b), PPG can generate reasonable FG and Disc priors, coarsely locating the target FG objects, but they each has some problems. For example, FG priors contain much more BG regions, as SAM has quite strong matching ability, the obtained predictions are likely have unexpected BG regions. As for Disc priors, although there are much less activated BG, the FG regions also become less. (2) In (c), IMR can help to incorporate FG regions from FG prior to Disc prior, struggling to fuse less BG regions. Take the second row as an example, the orange rectangle includes some non-cow animals, which are wrongly considered as cows in FG prior. During IMR, Disc prior gradually acti- vate more FG regions, while the orange box keeps correctly inactivated, leading to correct predictions.
5.3. Ablation Study
If not explicitly specified, the experiments are conducted on PASCAL-5', under I-shot setting. Kindly remind that more experiments are included in Appendix.
Component-wise Ablation Study. To validate the effective- ness of each design, the detailed component- wine ablation studies are provided in Table 3. We start from pure SAM 2-S (46 M) model with the most straightforward idea (Fig- ure 1(b) and Appendix D.I), and the averaged mloU score is 69.4%, which cannot surpass some classical FSS methods. As explained earlier, we attribute this to the existence of
We vary the iteration times n of IMR in the final model, and show their effects in Figure 5(c) and Table 4. can observed: (1) with the increase of n, the refined Disc memory and prior mask can be supplemented with more FG features and regions, (2) if they are initially entangled with BG, IMR cannot help filter them, and (3)
Parameter Study on IMR.

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Support (a) Inputs and Labels
Query
FG Prior (b) Pseudo Mask Prompts
Disc Prior
Output (d) FSSAM
(c) Iterative Memory Refinement
Figure 5. Qualitative illustrations of query and support samples, (b) pseudo priors (mask prompts), (c) iterative memory refinement. and (d) outputs. We plot some rectangles to highlight some FG and BG areas. In (b), FG prior appears to have more complete FG but more wrongly activated BG regions, while BG prior has less complete FG yet less wrongly activated BG regions. In (c), the FG regions of FG prior can be propagated to Disc prior. With more iterations, more FGs are fused into Disc prior, but also fused with more BG regions.
Visualizations of SCMA. An example is depicted in Fig- ure 6, where the second column represent the refined query pseudo memory (prior) of IMR, the last two columns repre- sent the cross attention scores between the sampled query FG pixel and the pseudo memory. We could observe: (1) The refined memory usually contain unexpected BG fea- tures, e.g., wheels. (2) The original memory attention (MA) cannot filter these unexpected BGs, leading to the wrong segmentation. (3) Our designed SCMA can suppress the attention scores of these positions, with the help of support memory. Kindly note that SCMA can reduce the scores be- tween query FG pixels and unexpected BG pixels (in mem- ory) by 41.5%, the details are included in Appendix C.4.
Table 4. Parameter study on Iterative Memory Refinement (IMR).
#lterations
5
5a
Mean
FB-IoU
83.4
79.2
73.9
88.2
81.2
84.4
80.4
75.1
80.3
88.9
81.6
84.9
81.6
76.0
81.0
89.4
81.5
84.6
81.0
76.3
80.8
89.0
some BG regions of FG prior might also be propagated into Disc prior instead. Therefore, trade-offs should be made between introducing more query FG and more query BG features. Table 4 demonstrates that using more iterations can boost the performance at first, e.g., the mloU score improved from 79.3% to 81.0%, yet further refinement cannot improve the score anymore, as the incorporation of more BGs hinders the FG-FG matching and fusion. Hence, set iteration times as in this paper.
6. Conclusion
In this paper, we incorporate SAM 2 into FSS to leverage its well-learned FG-FG matching ability. The simple way is encoding support features as memory, used to match and enhance query features. However, the matching of SAM 2 belongs to same-objects matching. while the FG objects in query and support different. Therefore, design PPG to generate pseudo query memories, making such matching compatible. Furthermore, we design IMR to supplement this memory with more query FG features, and devise a SCMA to mitigate the side-effects unexpected BG features in pseudo memory. Extensive experiments have been conducted to validate the effectiveness of design.
Score of Figure 6. Visual impacts of SCMA. The last two columns are the cross attention scores of the blue star. Orange rectangles represent some challenging areas that should be classified as BG.
Query Image & Mask
Refined Prior Memory
Score of
8

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Acknowledgements
Hariharan, B., Arbeláez, P., Girshick, R., and Malik, J. Simultaneous detection and segmentation. In European conference on computer vision, pp. 297-312. Springer, 2014.
This study is supported under the RIE2020 Industry Align- ment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
Hong, S., Cho, S., Nam, J., Lin, S., and Kim, S. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In European Conference on Com- puter Vision, pp. 108-126. Springer, 2022.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hu, T., Yang, P., Zhang, C., Yu, G., Mu, Y., and Snoek, C. G. Attention -baied multi-context guiding for few- shot semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8441- 8448, 2019.
References
Iqbal, E., Safarov, S., and Bang, S. Msanet: Multi-similarity and attention guidance for boosting few-shot segmenta- tion. arXiv preprint
Bao, X., Qin, J., Sun, S., Wang, X., and Zheng, Y. Rele- vant intrinsic feature enhancement network for few-shot semantic segmentation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 765-773, 2024.
(tl::22000096661 2022.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Chang, S., Zhang, L., and Lu, H. High-performance few- shot segmentation with foundation models: An empirical study, arXiv preprint urXivv2409.066: 06300, 2024.
Lin, T.-Y. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755 Springer, 2014.
Chen, S., Meng, F., Wu, C., Wei, H., Zhang, R., Wu, Q. Xu, L., and Li, H. No re-train, more gain: Upgrading back- bones with diffusion model for few-shot segmentation. arXiv preprint ar 2rXiv:2407.1 200.2/6872.1..... /6/22 2024.
Liu, H., Peng, P., Chen, T., Wang, Q. Yao, Y., and Hua, X.-S. Fecanet: Boosting few-shot semantic segmenta- tion with feature-enhanced context-aware network. IEEE Transactions on Multimedia, 2023a.
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., and Yuille, A. Detect what you can: Detecting and repre- senting objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1971-1978, 2014.
Liu, Y., Zhu, M., Li, H., Chen, H., Wang, X., and Shen, C. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.1 13310, 2023b.
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. International journal of computer vision, 88 (2):303 338.003-33222 2010.
Liu, Y., Jing, C., Li, H., Zhu, M., Chen, H., Wang, X., and Shen, C. A simple image segmentation framework via in-context examples. arXiv preprint arXiv:2410.048.4 2024.
Fan, Q., Pei, W., Tai, Y.W.,, and Tang, C.-K. Self-support few-shot semantic segmentation. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIX, pp. 701- 719. Springer, 2022.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 1220v22312.00052, 2023.
Milletari, F., Navab, N., and Ahmadi, S.-A. V-net: Fully con- volutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-5711 IEEE, 2016.
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364 2019.
Morabia, K., Arora, J., and Vijaykumar, T. Attention -ased joint detection of object and semantic part. arXiv preprint arXiv:2007.02419 2020.

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Nguyen, K. and Todorovie, S. Feature weighting and boost- ing for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 622-622222 2221,,1,220 2019.
Wang, J., Li, J., Chen, C., Zhang, Y., Shen, H., and Zhang, T. Adaptive fss: novel few-shot segmentation framework via prototype enhancement. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 5463- 5471, 2024a.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fermandez, P., Haziza, D., Massa, F., EI- Nouby, A., al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Wang, J., Zhang, B., Pang, J., Chen, H.,, and Liu, W. Rethink- ing prior information generation with clip for few-shot segmentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3941-3951, 2024b.
Park, S., Lee, S., Hyun, S., Seong, H. S., and Heo, J.- P. Task-disruptive background suppression for few-shot segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 38. pp. 4442 02222444,,222 2024.
Wang, K., Liew, J. H., Zou, Y., Zhou, D., and Feng. J. Panet: Few-shot image semantic segmentation with prototype alignment. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9197-9206, 2019.
Peng, B., Tian, Z., Wu, X., Wang, C., Liu, S., Su, J., and Jia, J. Hierarchical dense correlation distillation for few- shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23641-236511 2023.
Wang, Y., Luo, N., and Zhang, T. Focus on query: Ad- versarial mining transformer for few-shot segmentation. Advances in Neural Information Processing Systems, 36: 3 31524-31542, 2023a.
Wang, Y., Sun, R., and Zhang, T. Rethinking the correlation in few-shot segmentation: A buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7183-7192, 2023b.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00711 007144 2024.
Xiong, Z., Li, H., and Zhu, X. X. Doubly deformable aggre- gation of covariance matrices for few-shot segmentation. In Computer Vision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel. October 23-27, 2022, Proceedings. Part XX, pp. 133-150. Springer, 2022.
Shaban, A.,
Bansal, S., Liu, Z., Essa, I., and Boots, B. One- shot learning for semantic segmentation. arXiv preprint arXiv: 110917799934 2017.
Shi, X., Wei, D., Zhang, Y., Lu, D., Ning, M., Chen, J., Ma, K., and Zheng, Y. Dense cross-query-and-supoor attention weighted mask aggregation for few-shot seg- mentation. In European Conference on Computer Vision pp. 151-168. Springer, 2022.
Xu, Q.. Zhao, W., Lin, G., and Long, C. Self-calibrated cross attention network for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 655-665, 2023.
Sun, Y., Chen, J., Zhang, S., Zhang, X., Chen, Q., Zhang, G., Ding, E., Wang, J., and Li, Z. Vrp-sam: Sam with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 225544, 2024.
Xu, Q., Lin, G., Loy, C. C., Long, C.,, Li, Z.., and Zhao, R. Eliminating feature ambiguity for few-shot segmentation. In European Conference on Computer Vision, pp. 416- 433. Springer, 2024a.
Xu, Q., Liu, X., Zhu, L., Lin, G., Long, C., Li, Z., and Zhao, R. Hybrid mamba for few-shot segmentation. Advances in Neural Information Processing Systems, 37:73858- 73883, 2024b.
Tian, Z., Zhao, H., Shu, M., Yang, Z., Li, R., and Jia, Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 2020.
Yang. Y., Chen, Q. Feng, Y., and Huang, T. Mianet: Ag- gregating unbiased instance and general information for few-shot semantic segmentation. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pp. 7131-7140, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wang, H., Zhang, X., Hu, Y., Yang, Y., Cao, X., and Zhen, X. Few-shot semantic segmentation with democratic attention networks. In European Conference on Computer Vision, pp. 730-746. Springer, 2020.
Zhang, A., Gao, G., Jiao, J., Liu, C., and Wei, Y. Bridge the points: Graph- based few-shot segment anything se- mantically. Advances in Neural Information Processing Systems, 37:33232-33266 2024.
10

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Zhang, C., Lin, G., Liu, F., Guo, J., Wu, Q., and Yao, R. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9587-9595, 2019a.
Zhang, C.,, Lin, G., Liu, F., Yao, R., and Shen, C. Canet: Class-agnostic segmentation networks with iterative re- finement and attentive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5217-5226, 2019b.
Zhang, G., Kang, G., Yang, Y., and Wei, Y. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems, 34:21984- 21996, 2021.
Zhu, L., Chen, T., Ji, D., Ye, J., and Liu, J. Llafs: When large language models meet fhw-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3065-3075. 2024.

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
A. Code
Table S. Error bars evaluation on PASCAL-S' and COCO-20 datasets. The number of testing episodes is 1,000. The selected random seeds are {0. 1, 2, 321]. "Mean" is the averaged mloU score of folds, "F-i" means the i-th fold.
The source code is available at our Github repository https://gitte com/ Sam1224/ 2222/2///....///
B. Additional Details
Datuset
Seed
F-O
F-1
F-2
F-3
Mean
Fi-IoU
B.1. Implementation Details
0
81.6
85.7
79.9
77.6
81.2
89.4
81.4
83.3
79.9
77.3
80.5
88.8
82.1
85.4
80.8
78.8
81.8
89.7
All models are trained with 4 NVIDIA V100 (32G) GPUs, and tested with 1 V100 GPU. Remind that our FSSAM is built upon SAM 2, without introducing additional learn- able parameters. We deploy AdamW to optimize SAM 2's memory encoder, memory attention and mask decoder, with other components frozen. The learning rate is initialized as 0.001, and decayed with a polynomial scheduler (Tian et al., 2020). We follow SCCAN (Xu et al., 2023) to perform data augmentation, and adopt Dice loss (Milletari et al., 2016) for training fine-tuning), During training. the images are randomly cropped as 512 patches, and the testing predictions will be resized back to the original shape for metric calculation. The batch size is set as 8, with 2 samples distributed to each GPU. Following existing works ( (Wang et al., 2023a), the training epochs are set as 300 and 75 for PASCAL-5' and COCO-20'. We employ SAM 2-S (46 M) and DINOv2-B (86 M) for the main experiments. and study different model sizes in Appendix C.3.
PASCAL-
81.2
84.1
79.8
78.3
80.9
89.2
321
81.6
84,9
81.6
76.0
81.0
89,4
Mean
81.6
84.7
80.4
77.6
81.1
89.3
Std
0.3
1.0
0.8
1.1
0.5
0.3
0
$7.5
67.1
62.3
60.5
61.9
78.2
61.8
63.1
62.6
59.7
61.8
77.6
55.8
63.7
63.6
60.3
78.0
COCO-200
56.0
64.8
60.3
60.4
60.4
76,9
321
59.9
65.6
62,1
61.6
62.3
77.3
Mean
58.2
64.9
61.1
61.2
61.3
77.6
Std
2.6
1.5
0.9
0.5
background, etc. Therefore, with the change of random seeds, the sampled 1,000 episodes could have quite different testing difficulties, c.g., the mloU and FB-IoU scores are 61.3+0.9 and 77.6 0.55, where the standard deviation values appear to be consistently larger than those in PASCAL-5', Hence, we study different episode number next.
C.2. Different Number of Testing Episodes
C. Additional Experiments
Table 6. Performance COCO-20' with different number of test- ing episodes in {1.000, 4,000, 10,000, 20,0003. The random seed is fixed as 321. "Mean" is the averaged mIoU score of folds, "20"" denotes the mIoU score of the i-th fold.
In this section, we conduct experiments to validate the ef- fectiveness of FSSAM, including error bars evaluation (Ap- pendix C. the performance and efficiency with different testing episodes (Appendix C.2), different sizes of SAM 2 and DINOv2 (Appendix C.3), study the quantitative im- pacts of SCMA (Appendix C.4). provide more quantitative comparisons (Appendix C.5), emphasize the necessity of
#Test
20
20'
20
20
Mean
FB-IoU
Time (s)
1,000
59.9
65.6
62.1
61.6
62.3
77.3
353.2
4,000
58.6
64.1
65.3
61.5
62.4
77.6
1433.5
10,000
56.7
64.8
64.6
60.6
61.7
77.5
3579.2
fine-tuning (Appendix C.6), and compare our parameter number with baselines (Appendix C.7).
20,000
57.2
64.6
63.1
60.9
61.5
77.4
7006.1
Mean
58.1
64.8
63.8
61.2
62.0
77.5
3093.0
C.1. Error Bars Evaluation
As mentioned earlier, COCO-20 is a challenging dataset, the testing results of 1,000 randomly sampled testing episodes may not be convincing enough. Therefore, we change the number of testing episodes as {1,000, 4,000, 10,000, 20,000}, and present the testing results in Table 6 to study the impacts of different episode number. Meanwhile, we also measure the average time cost for testing each fold. We use SAM 2-S (46 M) and DINOv2-B (86 M) for the experiments. The testings are executed on single NVIDIA V V100 with 32 GB onboard memory. From the table, draw the following conclusions: (1) Testing with different number of episodes do not have huge impacts on the ob- tained scores, e.g., the mloU score of testing 1,000 episodes is 62.3%, while the average mloU score of testing 1,000, 4,000, 10,000 and 20,000 episodes is 62.0%, where there
We conduct error bars evaluation in Table 5 to examine the robustness towards randomness. The random seeds are selected from (321 (default), 0, 1, 2, 3}, and each seed would be responsible for sampling 1,000 testing episodes from PASCAL-5' and COCO-20', i.e., the query and sup- port samples will vary, with the altering of random seeds. It could be observed from the table: (1) In PASCAL-5', the mIoU and FB-IoU scores, averaged from 5 random seeds, are + and 0..,, similar to the reported values (in main paper) 81.0 and 89.3, when the seed is set as 321. This could demonstrate that our proposed FSSAM is robust; (2) COCO-20* is much more challenging than PASCAL-S', e.g., it includes multiple objects, small objects, complex
12

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
P Parameter Frozen
Image Encoder
Ms
Memory Encoder
P Parameter F Finetune
Mems
Is
Fss
Image Encoder
Memory
Mask Decoder
Attention
Fq
Fe
Ma
Figure 7. Overview of the simple idea to use SAM for FSS. First of all, query and support features Fe and F's are extracted by the image encoder. Then, support features Fs will be encoded as support memory Mems, taking support mask Ms as mask prompt. After that, query features FQ are matched and fused with support memory Mems in memory attention to obtain the fused query features Fo. Finally, Fo is decoded by mask decoder to obtain the predictions Mę-
merely a gap of 0.3%, showing the stability and superiority of the model; (2) As the deployment of foundation models would introduce more parameters, the amount of calcula- tions would be larger, i.c., the efficiency would be inevitably reduced. According to the table, although we deploy large foundation models (with 46+86=132 M parameters in total), the time cost is still affordable, e.g., the FPS is around 2.8.
When SAM 2-S (46 M) is deployed, upgrading DINOv2 from B (86 M) to L (300 M) cannot boost the performance. Particularly, the scores on fold 5 and 52 are not as good as those with DINOv2-B (86 M). After delving into the visualizations of these two folds, we observe DINOv2-B (86 M) can generate better prior masks for them. Although DINOv2-L (300 M) can improve the score by 1.5% on fold 5th the average mIoU score cannot surpass that of DINOv2- B (86 M), yet the computational cost is much larger (86 M - 300 M). (2) When DINOv2 is fixed as B (86 M), up- grading SAM 2 from S (46 M) to L (224.4 M) can witness an mloU improvement of 0.5%, showing the advantages of introducing more parameters. Nevertheless, the parameter number would be approximately times larger.
C.3. Different Sizes of SAM 2 and DINOv2
Table 7. Study different model sizes of SAM 2 and DINOv2 on PASCAL-5'' The value in (-) means the number of parameters (M), e.g., for SAM 2, S (46) means SAM 2-S has 46 M parameters. "Mean" is the averaged mloU score of 4 folds, "5"" denotes the
mloU score of the i-th fold.
After making trade-offs between the performance and the computational cost, we deploy SAM 2-S (46 M) and DINOv2-B (86 M) in this paper.
S
DINOv2
5
5*
5²
53
Mean
FB-IoU
B (86)
81.6
84.9
81.6
76.0
81.0
89.4
(46)
L (300)
81.9
82,5
79.4
78.5
80.6
88.9
C.4. Quantitative Impacts of SCMA
(80.8)
B (86)
79.8
84.8
79.0
75.8
79.9
88.3
We include visual example in the main paper, and further quantify the impacts of SCMA in Table 8. Kindly remind that the main idea of SCMA is the BG Suppress mechanism, which selectively suppresses the cross attention scores of those unexpected BG pixels (in memory) with a bias.
L. (300)
80.2
82.8
77.9
78.2
79.8
88.7
B (86)
82.0
85.1
80.3
78.7
81.5
89.7
1. (224.4)
I. (300)
82.1
82.6
79.1
80.6
81.1
89.5
We further study the impacts of SAM 2's and DINOv2's sizes on the performance and obtain the results in Table 7.
Note that SCMA is devised from memory attention (MA), and they both comprise 4 attention layers, each of which further includes a pair of self and cross attentions. Self atten- tions only operate on query features, while cross attentions aim to dynamically match and fuse query features (Q) with the pseudo query memory (K&V).
Specifically, we select 3 out of versions of SAM 2, includ- ing small (S), base plus (B) and large (L), comprising 46 M. 80.8 M and 224.4 M parameters, respectively. For DINOv2, choose base (B) and large (L), which have 86 M and 300 M parameters. According to the table, we could find: (1) It is not the case that the foundation models are the larger, the better. For example, remind that the only responsibil- ity of DINOv2 is to generate query prior masks, acting as pseudo mask prompts to encode pseudo query memories.
Both query features and the pseudo query memory have the same N HHW pixels. After cross attention, the score has a size of N x N, where the two dimensions refer to
13

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Table Quantitative Impacts of SCMA. SCMA includes 4 consecutive attention layers, each has a pair of self and eross attentions. Cross attention is used to match and fuse query features with the pseudo query memory. The values of "MA" and "SCMA" are the weefore-sfttmay average cross attention scores of each query FG pixel with the unexpected BG pixels in memory. "Mean" the averaged values of 4 folds, denoess denotes the mloU score of the i-th fold.
5
5
52
5
Mcan SCMA
Layer
MA
SCMA
Gap
MA
SCMA
Gap
MA
SCMA
Gap
MA
SCMA
Gap
MA
GAP
1
-5.9
-8.2
38.4%
4.6
-6.3
36.5%
-5.1
-7.2
40.2%
-4.7
-6.3
33.4%:
-5.1
-7.0
37.3%
2
-4.8
-7.1
48.0%
-4.8
-6.3
32.4%
-5.1
-6.9
36.0%
-4.9
-6.4
30.8%
-4.9
-6.7
36.8%
-4.4
-7.0
60.2%
4.6
-6.6
43.2%
-3.7
-6.0
61.6%
-3.9
-6.0
52.6%
-4.1
-6.4
54.0%
4
-6.6
-9.0
37.1%
-.2
-6.0
42.4%
-4.8
-7.0
43.6%
-4.4
-6.1
38.1%
-5.0
-7.0
40.0%
Mcan
-5.4
-7.8
44.5%
-4.6
-6.3
38.5%
-4.7
-6.8
44.2%
-4.5
-6.2
38.1%
-4.8
-6.8
41.5%
Table Quantitative comparisons on IVIS-92', "Mean" the averaged values of 10 folds, "92'"" denotes the mloU score the i-th fold.
1-shot 92"
Method
92a
92
92a
92a
92
92b
92"
92a
92"
Mean
FB -oU
Matcher (Liu et al., 2023b)
31.4
30.9
33.7
38.1
30.5
32,5
35.9
34.2
33.0
66.2 27.8
29.7
33.0
SINE (Liu et al., 2024)
28.3
31.0
31.9
34.6
30.0
31.9
32.2
33.7
30.6
31.2
63.5
FSSAM (Ours)
34.7
37.8
37.2
41.1
33,9
38.1
40.6
38.9
36,9
33.8
37.3
68.4
features and memory, respectively. Then, we use the query ground truth label Mo to preserve FG pixels in the first dimension as NrG x N. After that, we subtract the query label Mo from the refined Disc prior mask Peine and multiply with the second dimension to preserve the entries with positive values, represented as NFG NBG. This reduced score means the similarities of each query FG pixel to the unexpected BG pixels in the pseudo query memory. The values are the smaller the better, so query FG pixels would not fuse too many unexpected BG features from the memory, leading to more accurate segmentation results.
et al., 2020; Liu et al., 2023b) (PASCAL VOC (Everingham et al., 2010) with body part annotations (Chen et al., 2014)), and the results are presented in Table 9 and Table 10, respec- tively. We select both Matcher and SINE (Liu et al., 2024) for comparisons, where SINE trained with the whole COCO (80 classes) and directly tested on LVIS-92'. Simi- larly, take trained FSSAM on COCO-209 (trained with 60 classes, under 1-shot setting) to perform evaluation on both LVIS-92" and PASCAL-Part, We can observe that our FSSAM can show excellent generalizability (COCO- 20 + LVIS-92', and COCO-20' + PASCAL-Part..
As presented in Table 8, the provided scores have not been normalized by softmax, because those after softmax will be quite small, being difficult to see the impacts of SCMA. We could observe that the overall average gap between the reduced cross attention scores of MA and SCMA are 41.5%, showing the effectiveness of SCMA to mitigate the side- effects of unexpected BG features in pseudo query memory.
C.6. Necessity of Fine-tuning
Table Experiment on the necessity of fine-tuning.
I-shot 5²
Method
5"
5'
5
Mcan
SAM 2 w/o FT SAM 2 w/ FT
49.1
43.7
51.1
35,0
44.7
C.5. More Quantitative Comparisons
71.8
74.4
71.6
59.9
69.4
FSSAM w/o FT FSSAM FT
61.9
59.8
61.0
51.4
58.5
81.6
74.9
81.6
76.0
81.0
Table 10. Quantitative comparisons on PASCAL-Part.
We study whether fine-tuning (FT) is required when using SAM 2 for FSS, and present the results in Table 11, We can observe fine-tuning is necessary:
1-shot
Method
F-0
F-1
F-2
F-3
Mean
(I) Original SAM 2's memory attention is trained for same-object matching. while the matching of FSS is between different query and support FG objects, which we call it incompatible FG-FG matching. As we can observe from the table, with or without mme-tuning show prominent performance gap; (2) FSSAM can consistently outperform SAM 2 by large margins, since we design modules to resolve the issue. However, FSSAM
Matcher (Liu al., 2023b)
37.1
32.4
33.7
38.1
42.9
FSSAM (Ours)
34.7
37.8
37.2
41.1
46.4
Following Matcher (Liu et al., 2023b), we evaluate the per- formance of the designed FSSAM on two more challenging datasets, including LVIS-92² (Liu et al., 2023b) (derived from LVIS (Gupta et al., 2019)) and PASCAL-Part (Morabia
14

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
still requires fine-tuning, because the pseudo mask prompt is inaccurate, covering incomplete FG regions and unexpected BG regions, which is different from SAM 2's mask prompt.
by the mask decoder obtain predictions. As explained ear- lier, the support-query matching here would be incompatible with that of SAM 2, where the former is different-objects matching, and the latter is same-objects matching.
C.7. Comparisons Parameter Number
D.2. More Visualizations
Table 12. Compare with existing baselines in terms of parameter number. "#Total" and "#Learnable" denote the total number of parameters and the learnable parameter number in million (M).
We include more visualizations about pseudo prompts in Appendix D.2.1, and IMR in Appendix D.2.2.
D.2.1. PSEUDO PROMPTS
Method
#Total
#Leamuble
mloU
HDMNet (Peng et al., 2023)
51
69.4
We select more testing episodes and depict the input query and support samples, as well as the prior masks (i.e., pseudo mask prompts) and final outputs in Figure 8 and Figure 9, validating: (1) FG prior has more complete FG but much more unexpected BG regions than Disc prior; (2) Although Disc prior masks contain less unexpected query BG regions, so the encoded pseudo Disc memory would have less BG features, they have incomplete FG features, posing chal- lenges to segment complete query FG objects. For exam- ple, in the first column of Figure 9, the tole of the ship is wrongly classified as BG objects, because these regions are inactivated in Disc prior: (3) Our method, benefiting from powerful foundation models SAM 2 and DINOv2, can achieve very good results.
AMNet (Wang et al., 2023a)
54
7
70.1
HMNett (Xu et al., 2024b)
62
15
70.4
Matcher (Liu et al., 2023b)
941
0
68.1
VRP-SAM (Sun et al., 2024)
666
2
71.9
GF-SAM (Zhang al., 2024)
941
72.1
FounFSS (Chang al., 2024)
87
1
76.8
FSSAM (Ours)
132
11
81
To further show the computational burden of FSSAM, we select some methods and summarize their parameter num- ber, learnable parameter number, and the 1-shot mloU on PASCAL-5' in Table 12. The first 3 rows refer to classical FSS methods that use ResNet50 as the pretrained back- bone, and the remaining methods refer to foundation-based FSS methods that use DINOV2 and/or SAM. For our final- ized model, we use DINOv2-B (86M) and SAM 2-S (46M) without extra parameters (kindly remind our proposed mod- ules are parameter-free), and fine-tune part of SAM 2's parameters. It can be observed from the table: (1) Among foundation-based FSS methods, our parameter number is much smaller than most of them, while our performance is consistently much better; (2) Compared to classical FSS methods, though we use more parameters, the difference is not as large as expected, while the performance gap is quite prominent, so we believe the additional cost is worthy.
D.2.2. ITERATIVE MEMORY REFINEMENT
More visual impacts of Iterative Memory Refinement (IMR) are depicted in Figure 10 and Figure 11, and we can learn that: (1) IMR can successfully incorporate FG prior and the corresponding memory's 's FG information into Disc prior and its memory, without propagating too much BG information, as the activated BG regions of the refined Disc prior are consistently less than that of FG prior; (2) IMR cannot help to filter the BG regions initially contained in Disc prior; (3) The worst case of Disc prior after IMR is bounded by FG prior; (4) With the increase of iterations, Dise prior and memory can be fused with more FG information, but also with more unexpected BG information. Therefore, we need to make trade-offs between the incorporated FG and BG information to set the iteration number.
D. Additional Figures
We provide details about the simple use of SAM 2 in Ap- pendix D.I, and more visualizations in Appendix D.2.
D.I. Simple Idea of Using SAM 2
E. Limitation and Future Direction
The details about the simple way for adapting SAM 2 for FSS is illustrated in Figure 7, where the image encoder, memory encoder, memory attention and mask decoder con- stitute to the full SAM 2. The query and support images are forwarded the image encoder to extract query and support features. Then, support features will be encoded support memory, based on the support mask prompt. Memory at- tention consists of 8 consecutive self and cross attentions. During cross attention, the query features will match with support memory, and are dynamically fused with support FG features. Finally, the fused query features are decoded
There is failure case/limitation. Kindly remind our method relies on pseudo mask prompt to resolve the in- compatible FG-FG matching issue. In some very difficult examples (all baselines cannot deal with such cases), e.g., the FG and BG are very similar and are quite difficult to distinguish, the generated pseudo mask prompt may be mis- leading, e.g., the real FG is completely uncovered. This motivates us to further design error correction module, and we leave it as a future direction.
15

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Support
Query
FG Prior
Disc Prior
Output
Figure 8, More visualizations of prior masks (i.e., pseudo mask prompts) and outputs.
Support
Query
FG Prior
Disc Prior
Oupput
Figure 9. More visualizations of prior masks (i.e.. pseudo mask prompts) and outputs.
16

---

Unlocking the Power of SAM 2 for Few-Shot Segmentation
Support
Query
FG Prior
Disc Prior
(a) Inputs and Labels
(b) Pseudo Mask Prompts
(c) Iterative Memory Refinement
Figure 10. More visualizations of Iterative Memory Refinement.
uppoott
Query
FG Prior (b) Pseudo Mask Prompts
Disc P Prior
2
7=3
(a) Inputs and Labels
(c) Iterative Memory Refinement
Figure 11. More visualizations of Iterative Memory Refinement.
17


Reviews:
[{"review_id": "YKsdhLejgq", "reviewer": "Reviewer_n9Lm", "summary": "The paper introduces the Few-Shot Segment Anything Model (FSSAM), a novel method that leverages the powerful matching capabilities of SAM 2 to enhance few-shot segmentation tasks. The authors address the challenge of adapting SAM 2's same-object matching ability to the different-object matching required in few-shot segmentation by proposing the Pseudo Prompt Generator (PPG), which generates pseudo query memories using prior masks to enable compatible matching between support and query features. To further refine these memories and suppress background noise, the paper introduces the Iterative Memory Refinement (IMR) module, which iteratively incorporates more complete foreground features, and the Support-Calibrated Memory Attention (SCMA) module, which suppresses unexpected background features during the attention process. Extensive experiments on PASCAL-5i and COCO-20i benchmarks demonstrate state-of-the-art performance, with significant improvements in the averaged mIoU and FB-IoU compared to previous methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Computational burden and learnable parameters.\n\nThanks for your valuable suggestion! We agree it would be better to include the **parameter number** for further comparisons. We select some methods and summarize their parameter number, learnable parameter number, as well as the 1-shot mIoU scores on PASCAL-5$^i$ as follows:\n\n|Method|#Params (M)|#Learnable Params (M)|mIoU|\n|-|-|-|-|\n|HDMNet|51|4|69.4|\n|AMNet|54|7|70.1|\n|HMNet|62|15|70.4|\n|Matcher|941|0|68.1|\n|VRP-SAM|666|2|71.9|\n|GF-SAM|941|0|72.1|\n|FounFSS|87|1|76.8|\n|Ours|132|11|81.0|\n\nThe first 3 rows correspond to classical FSS methods that use ResNet50 as the pretrained backbone, and the remaining methods refer to foundation-based FSS methods that use DINOv2 and/or SAM. For our finalized model, we use DINOv2-B (86M) and SAM 2-S (46M) without extra parameters (kindly remind our proposed modules are parameter-free), and fine-tune part of SAM 2's parameters. It can be observed: (1) Among foundation-based FSS methods, our parameter number is much smaller than most of them, while our performance is consistently much better; (2) Compared to classical FSS methods, though we use more parameters, the difference is not as large as expected, while the performance gap is quite prominent, so we believe the additional cost is worthy.\n\nFor **computational complexity**, the designed modules will introduce additional **linear complexity** to the original foundation model, which have already been described in \"Memory Complexity\" of Section 4.2 and 4.3.\n\nTherefore, the computational burden of our FSSAM is reasonable and acceptable.\n\n> Comparison with versions of these methods that incorporate learnable parameters.\n\nThis comment locates in \"Methods And Evaluation Criteria\", sorry but we are confused about \"**different versions**\", do you mean **using SAM 2 and DINOv2 with different sizes** or something else?\n\nIf our understanding is correct:\n1. FSSAM is built upon SAM 2 and DINOv2 **without additional parameters**, the designed modules are **parameter-free**, so the variants in Table 3 have same (learnable) parameters.\n2. All parameters of DINOv2 are frozen. For SAM 2, we **fine-tune its memory encoder, memory attention and mask decoder**.\n3. Different SAM 2 mainly differ in image encoders, and **the learnable parameters of FSSAM are uniformly 11M**, regardless of which size is used.\n4. We have studied the impacts of different sizes in Table 7. For your convenience, we summarize some statistics as follows.\n\n|SAM 2|DINOv2|#Params (M)|#Learnable Params (M)|mIoU|\n|-|-|-|-|-|\n|S|B|132|11|81.0|\n|S|L|346|11|80.6|\n|B|B|167|11|79.9|\n|B|L|381|11|79.8|\n|L|B|310|11|81.5|\n|L|L|524|11|81.1|\n\nAfter making trade-offs between computational burden and performance, we use SAM 2-S (46M) and DINOv2-B (86M). For reasons why using larger SAM 2 and DINOv2 cannot guarantee performance gain, please refer to our reponses to **Larger backbone cannot guarantee consistent improvement** of **Reviewer pnf2**.\n\n> Same symbol $A_{QQ}$ in different modules.\n\nThanks for this comment. They uniformly refer to the mutual similarities between two features, but we do agree it would be much better to differentiate them in different modules, e.g., additionally include module names as superscripts.\n\n> Inconsistency between Figure 2 and detailed description in Section 4.\n\nThanks for pointing out this issue, we will modify them accordingly.\n\n> Confusing relationship between IMR in Figure 2 and Figure 3.\n\nWe guess that the confusion comes from the inconsistent outputs of IMR modules in Figure 2 and 3, we will unify them in a newer version.\n\n> In \"Parameter study on IMR\", performance decreases when the iteration n is 4, yet it increases in fold 5$^3$.\n\nThe test classes of fold 5$^3$ comprise of potted plant, sheep, sofa, train, and tv/monitor, and we find that the samples of this fold are more challenging than those in other 3 folds (e.g., the performance of fold 5$^3$ is consistently the worst among all folds for all methods in Table 1).  Specifically, many samples from this fold include: (1) multiple tiny objects, and (2) complex background (BG), making it hard to distinguish FG and BG. Therefore, the initially generated prior masks in fold 5$^3$ CANNOT cover sufficient FG regions, and **requires larger n** in IMR to **complete FG regions**.\n\nKindly remind with the increase of n, more unexpected BG regions will also be activated, acting as some noises to prevent from boosting the performance. Therefore, n should not be too large, otherwise, the performance (e.g., that of fold 5$^0$, 5$^1$ and 5$^2$) will decrease.\n\nIn Table 4, we conduct experiments to verify the impacts different iteration number, and we uniformly set it as 3 for all folds currently. We think a better way is **adaptively determining the iteration number n for each sample**, and we will leave it as a future direction.\n\n> Discrepancy between the data in Table 1 and 4.\n\nThanks for your careful checking! We will correct the value in Table 1.", "claims_and_evidence": "The claims made in the submission are generally well-supported by clear and convincing evidence. The authors provide a comprehensive approach to adapting SAM 2 for few-shot segmentation tasks and demonstrate its effectiveness through both quantitative and qualitative results.", "methods_and_evaluation_criteria": "The proposed methods (PPG, IMR, SCMA) and evaluation criteria (mIoU, FB-IoU on PASCAL-5i and COCO-20i) are relevant and appropriate for the problem of few-shot segmentation. They address key challenges in adapting SAM 2 to this task and provide a comprehensive evaluation of the model's performance. The methods are well-supported by both quantitative results and qualitative visualizations, making them suitable for the application at hand. \nHowever, it would be beneficial to include a comparison with versions of these methods that incorporate learnable parameters. This would help demonstrate the impact of learnable parameters on the model's adaptability and performance, and further validate the robustness of the proposed approach in different scenarios.", "theoretical_claims": "Two A_QQ appear in Figure 3, and in conjunction with Equation (8), do these instances carry different meanings? Additionally, the correlation between Figure 2, \"Overview of FSSAM,\" and the detailed descriptions of each module in the Methodology section appears somewhat inconsistent. More specifically, the relationship between the illustration of IMR in Figure 3 and the overview presented in Figure 2 seems somewhat confusing.", "experimental_designs_or_analyses": "The selection of datasets, evaluation metrics, and ablation studies offers a comprehensive assessment of the proposed method's performance. \nHowever, in the \"Parameter Study on IMR\" analysis, the performance decreases when the iteration n is equal to 4, yet it increases in 5^3. Is there any further analysis?", "relation_to_broader_scientific_literature": "The paper leverages recent advancements in foundation models, pseudo prompt generation, iterative refinement, and attention mechanisms to address the challenges of adapting SAM 2 for few-shot segmentation. The comprehensive evaluation and ablation studies further validate the effectiveness of the proposed methods.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper leverages the strengths of SAM 2 to address the challenges of few-shot segmentation. The key contributions include the Pseudo Prompt Generator (PPG) for creating compatible pseudo query memories, the Iterative Memory Refinement (IMR) module for enhancing foreground features, and the Support-Calibrated Memory Attention (SCMA) to suppress background noise during segmentation.\nWhile the experiments are comprehensive and demonstrate performance improvements, there is room for deeper analysis. Specifically, further investigation into the impact of computational trade-offs, and the influence of learnable parameters could provide additional insights and strengthen the overall robustness of the proposed method.", "other_comments_or_suggestions": "There appears to be a discrepancy between the data in Table 4, where the value for n=3 in 5^3 is 76.0, and the corresponding value in Table 1 is 75.9.\nUsing the same symbol AQQ for different quantities can indeed lead to confusion. To improve clarity, it would be advisable to use distinct symbols in Equation (13).", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "71h6TD4I4a", "reviewer": "Reviewer_pnf2", "summary": "This paper presents FSSAM, which leverages SAM2 for few-shot segmentation. \nFSSAM designs a Pseudo Prompt Generator to generate pseudo query memories, an Iterative Memory Refinement to iteratively refine pseudo query memories, and a Support-Calibrated Memory Attention to suppress background noise. \nExtensive experiments demonstrate state-of-the-art performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Larger number of parameters than other methods and result in additional computational cost.\n\nThanks for this comment. Our parameter number is actually much smaller than most of foundation-based FSS methods. We select some methods and summarize their parameter number, learnable parameter number, as well as the 1-shot mIoU scores on PASCAL-5$^i$ as follows:\n\n|Method|#Params (M)|#Learnable Params (M)|mIoU|\n|-|-|-|-|\n|HDMNet|51|4|69.4|\n|AMNet|54|7|70.1|\n|HMNet|62|15|70.4|\n|Matcher|941|0|68.1|\n|VRP-SAM|666|2|71.9|\n|GF-SAM|941|0|72.1|\n|FounFSS|87|1|76.8|\n|Ours|132|11|81.0|\n\nThe first 3 rows correspond to classical FSS methods that use ResNet50 as the pretrained backbone, and the remaining methods refer to foundation-based FSS methods that use DINOv2 and/or SAM. For our finalized model, we use DINOv2-B (86M) and SAM 2-S (46M) without extra parameters (kindly remind our proposed modules are parameter-free), and fine-tune part of SAM 2's parameters.\n\nIt can be observed from the table:\n1. Among foundation-based FSS methods, our parameter number is much smaller than most of them, while our performance is consistently much better.\n2. Compared to classical FSS methods, though we use more parameters, the difference is not as large as expected, while the performance gap is quite prominent, so we believe the additional cost is worthy.\n\nFor **computational complexity**, the designed modules will introduce additional **linear complexity** to the original foundation model, which have already been described in \"Memory Complexity\" of Section 4.2 and 4.3.\n\nTherefore, the computational burden of our FSSAM is reasonable and acceptable.\n\n> Substantive discussion about the most important finding. Is the main finding that SAM 2 can be used for FSS, or can a memory-based video segmentation model be used for FSS through the proposed module?\n\nOur main finding is **memory-based video segmentation model can be used for FSS through the proposed modules**, and we focus on one of the most representative and powerful models, i.e., SAM 2.\n\nIn Table 1, baselines Matcher, VRP-SAM and FG-SAM uniformly deploy SAM-L for FSS, where SAM-L (641M) is much larger than the one we deployed, i.e., SAM 2-S (46M).\n\nAs shown in the first row of component-wise ablation study in Table 3, naive SAM 2-S cannot outperform any of these baselines, and we attribute it to the facts that (1) our SAM 2-S (46M) is much smaller than their SAM-L (641M), and (2) there exist an *incompatible matching issue* (as introduced in Section 1).\n\nAfter using our designed modules to resolve this issue, our FSSAM can use much fewer parameters to outperform these foundation-based FSS methods by large margins, showing the effectiveness of our design, serving as our main finding.\n\n> Larger backbone cannot guarantee consistent improvement.\n\nWe would like to make the following notes to Table 7:\n1. We study 3 versions of SAM 2, including S (46M), B (80.8M) and L (224.4M). According to the official Github of SAM 2, *SAM 2-B (80.8M) originally CANNOT outperform SAM 2-S (46M) in 2 out of 3 datasets*, so it's reasonable that adapting SAM 2-B (80.8M) for FSS cannot show comparable performance as SAM 2-S (46M). When we remove SAM 2-B (80.8M), the improvement is stable, e.g., the mIoU is increased from 81.0 (SAM 2-S, DINOv2-B) to 81.5 (SAM 2-L, DINOv2-B).\n2. DINOv2 is only responsible for generating cosine similarity-based pseudo mask prompt, **whose features will not be directly used in other modules**, thus upgrading DINOv2 from from B (86M) to L (300M) cannot guarantee better performance, i.e., the pseudo mask prompt generated by DINOv2-B is good enough.\n\nIn summary, using larger backbone (SAM 2) can guarantee improvement.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n- The idea of employing SAM2's Foreground-Foreground matching ability in video segmentation to measure feature similarities between query and support in few-shot segmentation is interesting. \n- FSSAM designs several effective modules to make SAM2 adapt to few-shot segmentation. Extensive quantitive and qualitative analysis enhances its interpretability.\n- The proposed method outperforms the existing SOTA on two benchmarks.\n\nWeaknesses\n- FSSAM uses SAM2 and DINOv2, which may have a larger number of parameters than other methods and result in additional computational cost.\n- It's an interesting work but lacks substantive discussion. What was the most important finding in this work? Is the main finding that SAM2 can be used for FSS, or can a memory-based video segmentation model be used for FSS through the proposed module?\n- The previous approaches usually obtain better results using a larger model. Table 7 in the appendix explores the impact of different backbones on performance. It can be found that using a larger backbone does not bring consistent improvement. Does it indicate that the method has poor scalability?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tCutWy2umx", "reviewer": "Reviewer_HtbM", "summary": "This paper leverages SAM2’s strong matching ability to do the few-shot segmentation. Considering the matching of SMA2 is for sam2-object matching, the paper introduces Pseudo Prompt Generator (PPG) to generate pseudo query memories, and further design Iterative Memory Refinement (IMR) to supplement this memory with more query FG features, and devise a SupportCalibrated Memory Attention (SCMA) to mitigate the side-effects of unexpected BG features in pseudo memory.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Evaluation on LVIS-92$^i$.\n\nConducting evaluation on LVIS-92$^i$ (920 classes) can show the **excellent generalizability** of FSSAM. We select both Matcher and SINE for comparisons. Since SINE is trained with COCO (80 classes) and directly test on LVIS-92$^i$, we directly take our trained 1-shot FSSAM on COCO-20$^0$ (60 classes) for fair evaluation. Following baselines, each fold comprises 2,300 testing episodes, and the 1-shot results are as follows:\n\n|Method|Matcher|SINE|Ours|\n|-|-|-|-|\n|92$^0$|31.4|28.3|**34.7**|\n|92$^1$|30.9|31.0|**37.8**|\n|92$^2$|33.7|31.9|**37.2**|\n|92$^3$|38.1|34.6|**41.1**|\n|92$^4$|30.5|30.0|**33.9**|\n|92$^5$|32.5|31.9|**38.1**|\n|92$^6$|35.9|32.2|**40.6**|\n|92$^7$|34.2|33.7|**38.9**|\n|92$^8$|33.0|30.6|**36.9**|\n|92$^9$|29.7|27.8|**33.8**|\n|Mean|33.0|31.2|**37.3**|\n|FB-IoU|66.2|63.5|**68.4**|\n\nWe can observe our FSSAM consistently outperform Matcher and SINE in all folds, showing **excellent generalizability** (COCO-20$^i$ $\\to$ LVIS-92$^i$). We will include these results in a newer version.\n\n> (Major concern) Compare with SINE [2] on PASCAL-5$^i$ and COCO-20$^i$.\n\nSINE CANNOT be fairly compared on these datasets:\n1. FSS models are trained on some **base classes**, then tested on **unseen classes**, i.e., the testing samples are **out-domain** ones.\n2. In Section 4.2 of SINE, authors mention \"SINE is trained with all data of COCO\". Specifically, COCO and PASCAL have 80 and 20 classes, and COCO's classes include PASCAL's classes. Since SINE is trained on whole COCO, **it has learned to deal with all classes of PASCAL and COCO during training**, i.e., the testing samples are **in-domain** ones. Since SINE has been trained with the test classes (in each fold of FSS), its scores will naturally be higher than FSS methods. Unless we remove COCO from SINE's training set, SINE **can never be fairly compared on PASCAL-5$^i$ and COCO-20$^i$**.\n3. That being said, fair comparisons can be conducted on LVIS-92$^i$, and the results are included in previous question, where our method surpasses SINE by **6.1\\% mIoU**, under the consistent **out-domain** setting of \"training on COCO and directly testing on LVIS-92$^i$\".\n\n> Failure cases and limitations.\n\nThere is one failure case/limitation. Kindly remind our method relies on **pseudo mask prompt** to resolve the *incompatible FG-FG matching issue*. In some very difficult examples (all baselines cannot deal with such cases), e.g., the FG and BG are very similar and are quite difficult to distinguish, the generated pseudo mask prompt may be *misleading*, e.g., *the real FG is completely uncovered*. This motivates us to further design an error correction module, and we leave it as a future direction. We will include some failure cases in a newer version.\n\n> Insufficient visualization.\n\nThanks for this suggestion! We will include some baselines' results for comparisons in a newer version.\n\n> Can iteration number (in IMR) be adaptive?\n\nThe iteration number can theoretically be adaptive, yet there exist some challenges, and we uniformly fix it as 3 (see Table 4). Let's recall (1) the impact of IMR, and (2) when to use more iterations.\n\nFor (1), IMR will **make the incomplete FG regions in pseudo mask prompt complete**, but at risk of *introducing unexpected BGs (noises)*. Generally, using more iterations can make FG regions more complete, but introduce more noises, so trade-offs should be made to determine the iteration number.\n\nFor (2), we dive into the test samples, and find that **difficult samples, containing either multiple FG objects or complex BG, require larger iteration number**. The initial pseudo mask prompt of these difficult samples can **only cover limited FG regions**, so more iterations are required to make FG regions complete (to be a better prompt).\n\nHence, whether the iteration number can be adaptive depends on if such difficult samples can be automatically identified or not. Unfortunately, determining the difficulty of each case is not trivial, i.e., we cannot either *measure the number of FG objects in query images* or *judge whether query FG and BG are easy to distinguish or not*, and we leave it as a future direction.\n\n> Can we use point prompt?\n\nYes, but it's not a good choice compared to pseudo mask prompt, since:\n1. Although we can find some points (in query) with largest similarities to support FG features as point prompts, the overall framework is not easy to optimize, i.e., it's hard to determine if a point is the best candidate or not.\n2. 1 point prompt can only correspond to 1 entity. When there are multiple FG objects in a query image, e.g., 20 people, it's very difficult to automatically find 1 point for each people. If any people is not assigned with point prompt, he will be classified as BG.\n3. Instead, pseudo mask prompt can be regarded as a special case of point prompt, which can (1) make optimization easier, and (2) include sufficient points (to cover each FG object).", "claims_and_evidence": "The claims are clear and convincing.", "methods_and_evaluation_criteria": "The paper presents results on COCO and PASCAL, which are standard FSS datasets. However, it would be valuable to also evaluate on LVIS-92$^i$[1], which is a more challenging benchmark for evaluating the generalization of a model across datasets based on LVIS.\n\n[1] Liu, Yang, et al. \"Matcher: Segment anything with one shot using all-purpose feature matching.\" arXiv preprint arXiv:2305.13310 (2023).", "theoretical_claims": "The equations look good to me.", "experimental_designs_or_analyses": "The experimental designs and analyses (for the ablation study) look good to me. Please see the weakness for other issues", "relation_to_broader_scientific_literature": "The paper explores the potential of SAM2 for few-shot segmentation, leveraging its strong matching capability and extending it beyond object-specific matching to include class-level matching across different identities.", "essential_references_not_discussed": "The paper didn’t discuss a very related paper: [NeurIPS'24] A Simple Image Segmentation Framework via In-Context Examples. And the evaluation result couldn’t beat the results in SINE [2]. (see Table 1 in SINE)\n\n[2] Liu, Yang, et al. \"A Simple Image Segmentation Framework via In-Context Examples.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems.", "other_strengths_and_weaknesses": "**Strengths**  \n- Effectively leverages SAM 2’s memory bank matching for few-shot segmentation.  \n- Comprehensive ablation studies validate each proposed module.\n\n**Weaknesses**  \n- The pipeline is relatively complex, and the performance seems to fall behind SINE, which is simpler and can perform more tasks. (this is my major concern).\n- The paper lacks a discussion of failure cases and potential limitations.\n- The visualization results are insufficient. While most of them are effective—such as the comparisons between FG Prior and Disc Prior, as well as the memory refinement across different iterations—the model's results are not compared with segmentation results from other prior models (as well as the original sam2 model's result). Please include these additional visualizations.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- As the paper points out, the number of memory iterations should be traded off for better performance. Is there a way to find the optimized iteration for each case adaptively?\n- This work primarily focuses on refining the mask memory. Given the strong prompting capability of SAM2, would it be possible to generate prompt points on the target object region and use these prompts in the mask decoder to obtain the final output?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MX2fO2HXgj", "reviewer": "Reviewer_ZVAB", "summary": "This paper utilizes SAM 2 and DINO-v2 to solve the few-shot segmentation problem. The authors first point out that the class-agnostic matching ability of SAM 2 is useful for few-shot segmentation, but SAM 2 focuses too much on the identity of objects, which makes it unsuitable for FSS. To address this issue, the authors propose Pseudo Prompt Generator to generate pseudo query memory and further optimize it using Iterative Memory Refinement and Support-Calibrated Memory Attention. The experimental results show that these methods achieve good performance on PASCAL-5i and COCO-20i datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Evaluation on LVIS-92$^i$.\n\nThanks for this precious suggestion, conducting evaluation on LVIS-92$^i$ that includes 920 classes can indeed show the **excellent generalizability** of our method. We select both Matcher and SINE (A Simple Image Segmentation Framework via In-Context Examples, NeurIPS'24) for comparisons. Since SINE is trained with COCO (80 classes) and directly test on LVIS-92$^i$, we directly take our trained FSSAM on COCO-20$^0$ (trained with 60 classes, under 1-shot setting) to perform evaluation on LVIS-92$^i$. Following Matcher and SINE, each fold comprises 2,300 testing episodes, and the 1-shot results are shown as follows:\n\n|Method|Matcher|SINE|Ours|\n|-|-|-|-|\n|92$^0$|31.4|28.3|**34.7**|\n|92$^1$|30.9|31.0|**37.8**|\n|92$^2$|33.7|31.9|**37.2**|\n|92$^3$|38.1|34.6|**41.1**|\n|92$^4$|30.5|30.0|**33.9**|\n|92$^5$|32.5|31.9|**38.1**|\n|92$^6$|35.9|32.2|**40.6**|\n|92$^7$|34.2|33.7|**38.9**|\n|92$^8$|33.0|30.6|**36.9**|\n|92$^9$|29.7|27.8|**33.8**|\n|Mean|33.0|31.2|**37.3**|\n|FB-IoU|66.2|63.5|**68.4**|\n\nWe can observe our FSSAM consistently outperform Matcher and SINE in all folds, showing **excellent generalizability** (COCO-20$^i$ $\\to$ LVIS-92$^i$). We will include these comparisons in a newer version.\n\n> Is fine-tuning SAM 2 necessary?\n\nYes, fine-tuning is necessary, and the reasons are as follows:\n1. Original SAM 2's memory attention is trained for **same object matching**, while the matching of FSS is between *different query and support FG objects*, which we call it *incompatible FG-FG matching*. As we can observe from the table, with (w/) or without (w/o) fine-tuning (FT) show prominent performance gap.\n2. FSSAM can consistently outperform SAM 2 by large margins, since we design modules to resolve the issue. However, FSSAM still requires fine-tuning, because the pseudo mask prompt is *inaccurate*, e.g., it covers *incomplete FG regions* and *unexpected BG regions*, which is different from SAM 2's mask prompt.\n\n|Method|5$^0$|5$^1$|5$^2$|5$^3$|Mean|\n|-|-|-|-|-|-|\n|SAM 2 w/o FT|49.1|43.7|51.1|35.0|44.7|\n|SAM 2 w/ FT|71.8|74.4|71.6|59.9|69.4|\n|FSSAM w/o FT|61.9|59.8|61.0|51.4|58.5|\n|FSSAM w/ FT|81.6|74.9|81.6|76.0|81.0|\n\nBesides, in our response to your previous question, our FSSAM shows **excellent generalizability**, since our performance on LVIS-92$^i$ (the model is trained on COCO-20$^0$ and directly tested on LVIS-92$^i$) consistently outperform other methods by large margins.\n\n> Typo in Line 31.\n\nThanks for your careful checking! We have corrected this typo.\n\n> Can FSSAM use a trainable resnet instead of DINO-v2? What will happen if FSSAM unfreezes the DINO-v2 backbone and fine-tunes it?\n\nKindly remind DINOv2 is only responsible for generating pseudo mask prompt without learning, so it can be replaced as pretrained ResNet50, and there is no need to make it trainable. When we replace DINOv2 as ResNet50, the 1-shot mIoU on PASCAL-5$^i$ is 74.1, which is still much better than other methods (in Table 1). kindly note that FounFSS also uses DINOv2, and our performance is 4.2\\% better.\n\nFor the second question, we can fine-tune DINOv2, but we need to design an extra learning objective to encourage better pseudo mask prompt, which may raise *inductive bias* issue (i.e., *degrading the generalizability*), since the setting of FSS is **training on some base classes while testing on unseen classes**. Besides, the *training cost would be much heavier*, since DINOv2-B has 86M parameters.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence. The authors provide a comprehensive analysis of the limitations of SAM 2 in few-shot segmentation and demonstrate that the proposed methods can significantly improve the performance of few-shot segmentation tasks through extensive ablation studies and experiments on PASCAL-5i and COCO-20i datasets.", "methods_and_evaluation_criteria": "the proposed methods make sense for the few-shot segmentation problem. The authors introduce a novel approach that leverages the class-agnostic matching ability of SAM 2 to address the limitations of few-shot segmentation. The evaluation criteria are also well-defined, with experiments conducted on PASCAL-5i and COCO-20i datasets to validate the effectiveness of the proposed methods.\n\nThe only limitation is that the proposed methods are not tested on more challenging datasets to evaluate the generalization ability of the models, such as LVIS-92i or cross-domain datasets.", "theoretical_claims": "Not applicable, as the paper does not contain any theoretical claims or proofs.", "experimental_designs_or_analyses": "Yes, the authors follow traditional FSS settings and conduct extensive experiments to validate the effectiveness of the proposed methods. The ablation studies are also well-designed and comprehensive.", "relation_to_broader_scientific_literature": "This paper is related to how to leverage foundation models to boost downstream vision tasks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper's logic is clear, with the authors first pointing out some limitations of SAM 2 in FSS and then proposing several reasonable modules to address these issues. The design of these modules may have an impact on future FSS research.\n2. The experimental results are solid, with FSSAM showing significant improvements over the previous SOTA on PASCAL-5i and COCO-20i.\n3. The ablation studies are comprehensive, demonstrating the effectiveness of each module and exploring the impact of some hyperparameters.\n\nWeaknesses:\nAs I mentioned earlier, my main concern is the generalization ability of the model.\n1. Previous works such as Matcher and GF-SAM achieved good results without additional fine-tuning on COCO, while FSSAM requires fine-tuning of some modules, which may affect the model's generalization ability. Is fine-tuning these modules necessary? If not fine-tuned, how much will the model's performance be affected?\n2. The authors should test the model on more challenging datasets to validate the generalization ability of the model. For example, LVIS-92i, One-shot Part Segmentation, or some cross-domain datasets. (see Matcher, GF-SAM)", "other_comments_or_suggestions": "In line 31, 'but some unexpected query background' seems like 'but' should be 'and'.", "questions_for_authors": "Instead of using DINO-v2, VRP-SAM using a trainable resnet also achieves competitive results. Can FSSAM use a trainable resnet instead of DINO-v2? Or, What will happen if FSSAM unfreezes the DINO-v2 backbone and fine-tunes it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents FSSAM, which extends SAM 2 and DINO-v2 for few-shot segmentation with a new Pseudo Prompt Generator and two tailored modules. The method shows strong performance on standard benchmarks and demonstrates solid generalization to LVIS-92 and PASCAL-Part. The rebuttal effectively clarified technical points and strengthened the case for the paper’s contributions. The reviewers appreciated the paper's novelty and contribution, and the final ratings are WA, WA, WA, and A. Given its ratings and contributions, I recommend acceptance.
Decision: Accept (poster)