SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation
Authors: Claudia Cuttano
Gabriele Trivigno
Giuseppe Averta
Carlo Masone
Venue: NeurIPS 2025 spotlight ()
Keywords: Few shot segmentation
Segment Anything
Promptable Segmentation
Primary area: deep_learning

Abstract: Few-shot segmentation aims to segment unseen categories from just a handful of annotated examples. This requires mechanisms to identify semantically related objects across images and accurately produce masks. We note that Segment Anything 2 (SAM2), with its prompt-and-propagate mechanism, provides strong segmentation capabilities and a built-in feature matching process. However, we show that its representations are entangled with task-specific cues optimized for object tracking, which impairs its use for tasks requiring higher level semantic understanding. Our key insight is that, despite its class-agnostic pretraining, SAM2 already encodes rich semantic structure in its features. We propose SANSA (Semantically AligNed Segment Anything 2), a framework that makes this latent structure explicit, and repurposes SAM2 for few-shot segmentation through minimal task-specific modifications. SANSA achieves state-of-the-art on few-shot segmentation benchmarks designed to assess generalization and outperforms generalist methods in the popular in-context setting. Additionally, it supports  flexible promptable interaction via points, boxes, or scribbles, and remains significantly faster and more compact than prior approaches.  Code at: https://github.com/ClaudiaCuttano/SANSA.

SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation
Claudia Cuttano" Gabriele Trivigno"
Giuseppe Averta
Carlo Masone
Politecnico di Torino
{name. surname Soolammme@pploo.ii.i.
Abstract
Few-shot segmentation aims to segment unseen categories from just handful of annotated examples. This requires mechanisms to identify semantically related ob- jects across images and accurately produce masks. We note that Segment Anything 2 (SAM2), with its prompt-and propaaaaaaaateee mechanism, provides strong segmenta- tion capabilities and a built-in feature matching process. However, we show that its representations are entangled with task-specific cues optimized for object tracking, which impairs its use for tasks requiring higher level semantic understanding. Our key insight is that, despite its class-agnostic pretraining, SAM2 already encodes rich semantic structure in its features. We propose SANSA (Semantically AligNed SegmentAnything 2), a framework that makes this latent structure explicit, and repurposes SAM2 for few-shot segmentation through minimal task task-specific modifi- cations. SANSA achieves state-of tfa on few-shot segmentation benchmarks designed to assess generalization and outperforms generalist methods in the popular in-context setting. Additionally, it supports flexible promptable interaction via points, boxes, scribbles, and remains significantly faster and more compact than prior approaches. Code at: bttps://gt //ggtttt.../.. thub. com/ ClaudiaCuttano/SSS SSNSA.
Introduction
Segmenting images is core problem Computer Vision, yet achieving high-quality results typically requires extensive human effort annotate pixel-level masks. Moreover, conventional semantic segmentation methods [13, 26, 34, 16] struggle to generalize to unseen categories. Inspired by the human ability to recognize novel objects from just few examples, few-shot segmentation (FSS) [38, 69, 41, 59] has emerged as a paradigm that leverages a small set of labeled reference samples to guide the segmentation of target images containing arbitrary, previously unseen classes.
To this end, recent work has turned visual foundation models (VFMs) [7], which offer rich visual representations and strong generalization capabilities [52, 82]. A natural approach [42, 83, 62] is to decouple the few-shot segmentation task into two stages: feature matching followed by promptable segmentation. This is typically achieved by combining DINOv2 [50], known for its strong semantic correspondence capabilities [86, 44, 85, 63], with Segment Anything [35], which excels at producing high-quality segmentation masks [35, 90, 87]. While effective, these modular approaches add computational overhead and require prompt engineering to coordinate multiple VFMS [89].
We observe that Segment Anything 2 (SAM2) [54] offers an alternative paradigm. Designed for framework, where an object is specified via its mask and tracked across frames. To achieve this, SAM2 introduces Memory Attention mechanism to implicitly match features across video frames and propagate masks over time with high spatial precision. While this feature matching is originally intended for object tracking
Video Object Segmentation, it operates as
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A(Frozen SAM2 SOTA): mloU
17.7%
59.1%
Frozen SAM2
10
-15.7%
-10
-26
Low sementic shitt High visual similartty
High semantic shift similarty
-0
SANSA (ours)
Figure I: We evaluate frozen SAM2 on few-shot segmentation tasks on four datasets with varying degrees of semantic variability. On datasets [10, 18] with low semantic shift and high intra-class visual similarity, SAM2 matches even outperforms state-of-the-an APSeg [28]. However, on more challenging datasets like COCO and LVIS, with high se- mantic shift (e.g., cruise ship vs. rowboat), its performance drops significantly, compared with GF-SAM [83]. The bottom row illustrates ex- amples of ground-trun truth masks in both scenarios.
Figure 2: ject instances across diverse images and visual- ize their distribution using the first three prin- cipal components of PCA, mapped to RGB channels. The features appear entangled, with clusters mixing across categories, highlighting the lack of a coherent semantic structure in the original feature space. After adapting the fea- ture space with SANSA, well-defined clusters emerge: semantically similar instances group together, forming coherent structures despite intra-class variation in visual appearance.
We extract SAM2 features from ob-
based on visual similarity, we note that this architecture inherently unifies two capabilities central to FSS within single model: dense feature matching and high-quality mask generation. Building on this analogy, we propose repurposing the prompt-and-prope pnoppaaggee framework to address the FSS task, by reinterpreting the temporal dimension of videos as a collection of semantically related images.
This raises the central question: instead of limiting SAM2 to object tracking, can it generalize beyond visual similarity perform semantic tracking based on shared concepts across images?
Addressing this inquiry means, in turn, to answer whether SAM2 has implicitly learned semantic- aware representations, despite its class agnostic pre-training. To answer this, we conduct toy experiment Fig. I, testing S A2 performance on FSS datasets with different semantic variability. Interestingly, we observe that in low-semantic-shift scenarios, SAM2 achieves comparable or even higher performance with respect to state-of-the-aa methods. However, on challenging datasets with high semantic shift, its performance drops drastically. A straightforward conclusion from these preliminary results would be that SAM2 may not have learned discriminative representations in its class- class-agnostic pretraining, making it unsuited for tasks requiring semantic alignment.
We challenge this interpretation, based on the observation that SAM2 pretraining. focused on instance matching across frames, shares similarities with self- -upervisdd learning frameworks [77, 76, 29, 5], which are known to elicit semantic understanding by enforcing feature invariance across views [11,4]. Given this analogy, we posit that SAM2 does encode semantic information, which is however entangled with instance-specific features optimized for object tracking, reflecting experimental evidence in Fig. 1. If our intuition is true, it implies that i) this structure could be disentangled through lightweight transformations [1. 32, 37], such as adapter modules and ii) it should be learnable from a set of base classes and generalize to unseen categories [57, 17]. To this end, we i) intentionally use one of the simplest adapters in the literature, namely AdaptFormer [31], and ii) verify the generalization hypothesis through extensive experiments in strict few-shot benchmarks.
Building on this insight, we introduce SANSA (Semantically AligNed SegmentAnything 2), and show how to expose SAM2 latent semantic structure, repurposing its Memory Attention mechanism to shift from visual similarity to semantic similarity. The effectiveness of SANSA is illustrated in Fig. 2, where a 3D PCA, computed on unseen classes, reveals the emergent semantic organization of our features. We complement our method with a novel training objective designed exploit S SA22 temporal continuity to convert each target image into pseudo-annotation for subsequent frames.

---

Our contributions are the following:
We are the first to investigate the semantic structure within SAM2. We show that such semantics can be disentangled through bottleneck transformations, enabling a unified approach that reinterprets few-shot segmentation the task of tracking semantic concepts across images;
We validate SANSA through extensive experiments, achieving SOTA performance in strict FSS benchmarks while also outperforming generalist approaches in the 'in-context' scenario. Our experiments reveal that SAM2 encodes coarse-to-fing semantics, from high level concepts (e.g. dog vs. cat, +6.3% on COCO-20") to fine-grained distinctions at category-level (e.g. dalmatian vs. bulldog, +8.3% on LVIS-927 92') and part-level (e.g. hand vs. arm +4.6% on Pascal-Part):
By supporting prompts like points, boxes, or scribbles, our approach enables a wide range of downstream task, such as data annotation without the need for costly pixel-level masks. Finally, by exploiting the tight integration of memory attention and mask decoding in SAM2, we avoid the need for auxiliary models or complex pipelines, setting new SOTA with a framework more than 3x faster than competitors, and 4-5x smaller parameter count.
2 Related works
Few-shot segmentation aims to segment a target image given an annotated reference. Early methods relied on compressed prototype representations [38, 69, 36, 41, 22]. later replaced by attention- and correlation-banl based approaches [68, 84, 47, 46, 30] to better capture pixel-level relationships. More recently, research focused on leveraging the large-scale pretraining and generalization capabilities of vision foundation models [89, 87, 42, 71, 75]. Matcher [42] and GF-SAM [83] utilize a training-fres pipeline: DINOv2 extracts correspondences which are used to prompt SAM for segmentation. Simi- larly, VRP-SAM [62] leverages a frozen SAM with an external encoder for feature matching. Recent works [87, 89, 44] explore the use of single VFM [7] to jointly handle semantic understanding and mask prediction. DiffewS [89] exploit the emergent semantic correspondences in StableDif- fusion [56] to unify the process, while SegIC [44] combines a frozen DINOv2 with a lightweight segmentation decoder. Yet, recent findings [86, 85] suggest that diffusion and DINOv2 features offer complementary but disjoint strengths: the former offers spatial precision but weak semantics, the latter strong semantics but sparse matches. In contrast, we posit that SAM2 unifies both properties: its features possess high spatial granularity and implicitly encode semantic information. We show that this latent semantic structure can be extracted, effectively enabling FSS within a unified architecture.
Semantic correspondences and Foundation Models. Finding correspondences between images is longstanding problem in Computer Vision [6, 43, 33, 55, 49]. While early deep-learning approaches train dedicated models to establish semantic correspondences [33, 55], recent works [86, 63] have shown that VFMs enable generalization across tasks [66, 2. 73, 19]. Among them, DINOv2 [50] and StableDiffusion [56] have demonstrated compelling ability to establish semantic correspondences between images [86, 49, 85]. Recently, Segment Anything 2 [54] established itself as foundation model for Video Object Segmentation. We observe that its pretraining, entailing matching object instances across frames under viewpoint changes and motion blur, closely parallels self-supervised learning frameworks [77. 76, 29] that derive semantic understanding through self-similarity training. However, the extent to which SAM2 embeddings encode (if any) semantic concepts has not been studied yet. Recent applications semantic-shift scenarios (e.g. propagating masks across slices of 3D imagery given support example). However, frozen SAM2 shows poor performance in standard FSS benchmarks requiring higher level semantic understanding, In this work, we shed light on this matter, providing insights into SAM2 feature structure and showing that semantic content can be disentanged from its embeddings.
specialized domains [91, 3] utilize a frozen SAM2 low-
3 Method
We structure our investigation around the following key research questions: (1) Can semantic information be effectively extracted from SAM2 features? (2) Can this extraction occur without impairing the functionality of the SAM2 decoder, thereby maintaining its precise segmentation performance? (3) Finally, if a mapping that enhances semantic structure within SAM M2 features can be learned, does this mapping generalize effectively to unseen classes?
3

---

Target image
Bank T
SDOrE bank
Memory
Z+
Eacoder
Semantic-eware features (Visualized PCA)
B+
Memory Attention
Mask Decoder
F+
Fr Tmanch
AdaptFormee
F
F+
Prompt
1
14 TT
K reference imagee(s) with annotation (mask, point, box or scmbblee
Reference image(s)
Target image
Dense Feature Matching,
High quality Mask Generation
Figure 3: Overview of SANSA: Given annotated reference images and a target image, we construct pseudo-video by concatenating them, then leverage SAM2 streaming pipeline to process reference frames together with their annotations sequentially. We restructure SAM2 feature space to make its latent semantic structure explicit, enabling mask propagation based on semantic similarity from reference to target. The emergent semantic structure visualized by the 3D PCA projection of F.
These questions directly shape the design of our approach. Throughout the rest of the manuscript, we provide empirical evidence supporting affirmative answers to each.
Task Definition. We consider the general k-shot segmentation setting, where the model is given K reference pairs R = (a+f, )=)== each consisting of an image x² RR "R=xxx and its mask annotation R x3 the goal is to predict yr E [0, H| the at E [0, 1 H Given target image semantically aligned with the reference. As shown in segmentation mask of objects de that are Fig. interpret the references and the target sequence of frames in pseudo-video M:
M [ a U (Ir,, Ø),
where only the k reference frames are annotated.
3.1 From Object Tracking to Semantic Tracking with SAM2
Segment Anything Model extends SAM [35] to Promptable Video Object Segmentation. Like its predecessor, SAM2 comprises three main elements: an Image Encoder, Prompt Encoder. and a Mask Decoder. Specifically, given an image z with features F and prompt of E (mask, point, box) the Mask Decoder processes them to produce the segmentation mask j. The key innovation of SAM2 lies in its extension to video domain: masks be propagated across new unannotated frames 3i without additional prompts, thanks to a memory mechanism. As shown in Fig. 3, conceptually decompose SAM2 architecture into two functional components:
Comprising the Memory Encoder, Memory Bank, and Memory Attention, this module establishes dense correspondences across frames. Concretely, given object reference mask S (either predicted or given as prompt), the Memory Encoder constructs its memory representation, by fusing the mask with frame features Fk:
Dense Feature Matching:
I = Fkk + conv_down
This representation is stored in the Memory Bank. Subsequently, the features F, of target unan- notated frames undergo a cross-attention (Memory Attention) that aims at establishing dense correspondences between the current frame and the memory representations from previous frames:
Fr.nttch = Attention
(E) ( I,1),
where It are past representations and (()),(.), V(-) the query, key, and value projections.

---

High-quality Mask Generation: For unannotated frames, the Mask Decoder is tasked with refining the coarse features matches Fr.nh which encode dense correspondences with prior object representations, to produce the segmentation output 1/. (cf Fig. 3).
We propose to repurpose the memory-based feature matching and mask decoding mechanisms, reinterpreting the temporal dimension of videos as a collection of semantically related images. Thus, rather than tracking a specific object across continuous frames, we aim at tracking its semantic class.
We highlight two advantages of this formulation: first, unlike recent approaches [62, 44, 89], our model naturally supports variable k-shots without modifications; second, our solution seamlessly sup- ports promptable FSS, where prompts can take the form af (mask, point, box, scribble) ). removing the reliance on pixel-level annotations, Finally, we note that reference frames are encoded in Memory Bank without undergoing Memory Attention (ef Fig. 3). By avoiding cross-refer,ncing, we ensure predictions for the target image to be invariant to the ordering of reference images.
3.2 SAM2 Feature Adaptation
At the core of the feature matching mechanism lie the learned feature representations, central to the Memory Attention mechanism: the reference object representation I, is constructed from F, (Eq. (2)), and matching is performed via aross-attention between target features Fr and I, (Eq. (3)).
Our goal is repurposing the Memory Attention to shift from instance-level to semantic-level match- ing. To achieve this, we introduce minimal architectural changes and instead focus restructuring the feature space itself. Specifically, we seek to induce a semantic organization of the features, enabling dense correspondences to reflect semantic similarity rather than visual-similarity.
We hypothesize that SAM2 features already encode semantic concepts, albeit entangled with signals specific to tracking, such as instance-level details and spatial biases. If such structure exists, then it should be learnable from set of base classes by training few parameters [1, 32, 57]. Thus, we opt for simple AdaptFormer [14] blocks, although our analysis is not tied to the adaptation method, as we will show in the Experimental Section. We integrate AdaptFormer blocks within the last two layers of the Image Encoder, as these encode higher-level semantic representations. Given down- and up-projection matrices W down Rd.dd
W up E Rd.dd an AdaptFormer block operates token-wise:
A(r) (z W Woown) W aps
(4)
where is ReLu and d < d is the bottleneck dimensionality. The adapted features are summed in residual fashion in the backbone transformer blocks:
Jelf Attention(z),
(5)
MLP (Isetr + Jsell + A(zset);
(6)
The backbone weights are kept frozen and we only train projections W down and W up-
3.3 Training objective
Following standard practice in FSS, we adopt an episodic training paradigm [67, 59, 69, 36). We have access to set of training episodes, each one containing annotated instances from a single class. We denote these episodes as {a', y/ J. where y' E [0, 1| the mask that segments these instances.
Leveraging our model inherent ability to process sequences of variable length, we create challenging training examples by inverting the standard k-shot setup: instead of predicting single target image from multiple references, the model receives a single labeled reference image and is tasked with propagating the concept multiple unlabeled target images. Formally, define the training clip:
Merain = [Ir, a,l 21
(7)
where {Iy, } } is the annotated reference image and { ト are the J/ unlabeled target images. We feed Merain to our model, which propagates the provided concept across target frames to predict the masklet ) Initially, the Memory Bank is populated with the reference representation I,- We propose to condition the prediction for each target frame on the reference as well as previous target predictions, by encoding in the Memory Bank the predicted representation Z/, computed as
5

---

Frozen SAM2
SANSA (ours)
chair
Spoon
dog
person
boat airplane
mo.89
a. PCA visualizations
3D PCA visualizations
Clustering Match visualizations
Figure 4: Semantic structure of feature space. (a) PCA visualization of frozen SAM2 and our SANSA features COCO fold with unseen classes, showing the first two principal components color-coded by class. SAM2 features exhibit weak semantic separability, indicating entanglement with other signals. (b) PCA-based RGB visualization of SANSA features across images with seen and unseen categories, showing consistent semantic mapping. (c) Part-level semantics and cross-image consistency. We cluster features per object and match clusters actoss image pairs via Hungarian Matching. This reveals that SANSA captures fine-grained distinctions (c.g., handlebar vs. wheel), spatial layout (e.g., upper vs. lower wheel), and produces representations that align across images.
in Eq. (2). This design transforms each intermediate frame into pseudo-reference for subsequent frames. This objective discourages overfitting to individual image pairs and encourages robust, semantically grounded correspondences, forcing the model to disentangle semantics from low-level features. We supervise the predicted { د with Binary Cross- pooss-Entropy loss and Dice loss [45].
3.4 Visualizing SANSA Feature Space
In this section we analyze how adaptation reshapes SAM2 feature space. In Fig. 4a, we extract features for SAM2 and SANSA for a set of objects belonging to unseen classes for our trained model, and visualize their first two Principal Components (PCs), labeled by class. Frozen SAM2 features show weak class discriminability. reinforcing the hypothesis that semantic structure entangled with task-specific features tailored for the original tracking objective, explaining SAM2 poor FSS performance. Consequently, the leading PCs reveal mix of semantic and non-semantic signals. In contrast, our SANSA features show clear semantic diseriminability. mapping novel classes into well-defined semantic clusters. In Fig. 4b, we visualize SANSA features using PCA, computed jointly across images, where the first three PCs are mapped to RGB channels. The visualization spans three images containing instances from both unseen (e.g., person, chair, ball) and seen (e.g.. umbrella) categories. The consistent color mapping across instances of the same category highlights strong semantic grouping, mapping similar concepts coherently in feature space despite visual variability.
Finally, in Fig. 4c, shift from coarse category level to fine-grained correspondences the part level. Following [86], we cluster object features with K-Means and match centroids across image pairs via the Hungarian Algorithm, then visualize matched clusters with shared colors to assess semantic consistency. The results show that, despite not being trained with part-level supervision, SANSA features encode lert-level understanding (e.g. hand vs arm, handlebar wheel), spatial layout (e.g. upper vs lower wheel), and produce representations that align across images.
We provide an in-depth study of semantic representations in Appendix B, using PCA, clustering, and linear probing analyze how semantics encoded in SAM2 and made explicit through adaptation.

---

Table I: Striet Few-Shot Segmentation setting, Results for k-shot segmentation on LVIS-92', COCO-20", and FSS-1000. We include both specialist models and approaches based on foundation models, trained and tested on disjoint classes. Training-fri free methods are also reported for reference.
few-shot segmentation
Method
Venue
Backbone
Params
LVIS-92-
COCO-20*
PSS-1000 I-shot 5-shot
I-shot 5-shot
I-shot 5-shot
Training-fre methods
SAM [54]
SAM2-L
224
16.5
26.3
32.2
44.2
73.0
84.3
PerSAM [87]
ICLR'24
SAM-H
641 M
11.5
23.0
71.2
Matcher [42]
ICLR'24
DINOV2-L-AAM-
945 M
33.0
40.0
52.7
60.7
87.0
89.6
GF-SAM [83]
NeurIPS'24
DINOW-LL-SMMH
945 M
35.2
44.2
58.7
66.8
88.0
88.9
Strict k-shot segmentation
AMFormer [72]
23
ResNer101
49 M
51.0
S7.3
HMNet [75]
NeurIPS'24
RN50
39 M
52.1
58.9
VRP-SAM [62]
CVPR'24
RN50 + SAM-II
670
28.3
53,9
87.7
SegIC [44]
ECCV'24
DINOv2-G
1.2
40.5
53.6
88.5
DiffewS [89]
Nc
StableDiffusion
890 M
33.9
43.7
52.2
60.7
90.2
90.6
SANSA
25
SAM2-L
234 M
48.8
53.9
60.2
64.3
91.4
92.1
4 Experiments
Implementation details. We employ SAM2 with Hiera-Large [58] as encoder. AdaptFormer [14] is inserted into the last two blocks, with hidden size set to 0.3 the block channel dimension in the strict few-shot setting and 0.8x in the generalist. SAM2 is frozen, and only the adapters are trained (~10M params the strict case and ~55 in the generalist). We train with AdamW and learning rate 10-1 for 5 epochs (strict) and 20 (generalist), with k=1 (a single annotated reference) and sequence length J=3. The same model evaluated on 1-shot and 5-shot. Full details are Appendix H.
Datasets. COCO-20' [48] is built on MSCOCO [39] and consists of 80 classes split into four folds, each with 20 classes. FSS- 1000 [23] contains 1000 classes, with 520 for training, 240 for validation, and 240 for testing. LVIS-92 [42] is more challenging, selecting 920 classes from LVIS [24], divided
in 10 folds. PASCAL-Part [42] includes four superclasses with 56 object parts across 15 classes. PACO-Part is built from PACO [53] and contains 303 classes, split in four folds.
4.1 Strict Few-Shot Segmentation Setting
We first evaluate our method in strict FSS setting. Following standard protocols [72, 62, 61]. we train on base classes and evaluate on novel classes with k-shots. These experiment address the question of whether the semantic mapping learned on base classes can transfer meaningfully to novel categories.
Few-shot segmentation. In Tab. 1, we compare SANSA, besides specialist models, such as AM- Former [72] and HMNet [75], with the most relevant and recent approaches based on Foundation Models. These include methods that, like ours, leverage single Foundation Model, such as SegIC [44] and D [89], as well as modular two-stage pipelines like VRP-SAM [62]. For completeness, we also report results from generalist training-free methods built on DINOv2 and SAM, including PerSAM [87], Matcher [42], GF-SAM [83], as well as our baseline, i.e. frozen SAM2.
In the one-shot setting, SANSA consistently outperforms all prior methods, demonstrating superior generalization to unseen classes. Specifically, we surpass the best direct competitors SegIC, VRP- SAM and DiffwS by +8.3%, +6.3%, and +1.2% LVIS-92* COCO-20', and FSS-1000, respectively. This performance gap remains consistent also against training free approaches: compared to GF-SAM. SANSA achieves gains of +13.6% +1.5%, and +3.4%. On the challenging LVIS- including 920 fine ime-grained categories (e.g.. buildog, dalmatian), DINO-basee methods (SegIC, GF-SAM, Matcher) tend to underperform, possibly due to DINO overly-semantic features [20, 86] (e.g., grouping distinct breeds under the concept of "dog"). Here, SANSA achieves substantial +8.3% gain, suggesting that SAM2 encodes a latent hierarchical semantic structure, capturing both high-level semantic concepts (dominant in COCO-20") and fine-grat fine-grained distinctions (crucial for LVIS-92').
7

---

VRP-SAM SANSA(OurS)
Params
670M
234M
SANSA
(ours) 234M
point
38.4
53.4
+15.0
mmou (%6
SegiC
scribble
47.3
53.1
+5.8
box
49.7
54.3
+4.6
40
1.20
mask
53.9
60.2
+6.3
GF-SAM
LV15-92 VIS-92'
35
Matcher
390M
scribble
box
point
945M
DiffewS
57MM
VRP-SAM
Figure 6: Promptable few-shot segmentation with SANSA. Top: performance in strict few- shot (COCO-20') using different prompt types compared with VRP-SAM. Bottom: qualitative examples with point, scribble, and box prompts.
Inference Speed (FPS)
Figure 5: Comparison of inference speed and mloU, with bubble size representing #parame- ters. The plot highlights our superior trade-off.
Regarding the 5-shot setting, we note that SegIC and VRP-SAM do not provide an inference pipeline to extend to k-shot. In contrast, SANSA natively models correspondences across multiple reference frames and it outperforms the best competitor, DiffewS, by +10+ +10.2% and +3.6% LVIS-92' and COCO-20', respectively. Compared to training-free models, SANSA shows improvements +9.7%, +3.2%, LVIS-92' and FSS-1000, while suffering -2.5% gap 1OCO-20' w.r.t. GF-SAM. We also report results upgrading SAM-based baselines to SAM2 in Appendix D.
Performance-Efficien Trfde-off. In Fig. 5, analyze the trade-off between model size, inference speed (img/s), and performance. SANSA achieves state-of-the-ar results while being the most lightweight solution. Specifically, it is: i) over 3x faster than the direct competing method (GF-SAM), ii) more compact, introducing only adapter parameters on top of S SAM2 (totaling 234M), and iii substantially more accurate, outperforming GF-SAM by +13.6% mIoU on the challenging LVIS-92* benchmark. Importantly, SANSA keeps the SAM2 architecture entirely frozen, meaning that by storing only the adapter weights, it retains SAM2 state-of-the-art performance on Video Object Segmentation while also achieving sop-tier results on few-shot segmentation within a single model. large-scale annotation study quantifying speed/quality trade-offs is presented Appendix
One-Shot Promptable Segmentation. Our framework keeps SAM2 decoder frozen, maintaining its capability to generate masks from any prompt (i.e. points, scribbles, or boxes). As such, during inference, users segment reference objects by providing simple point, without requiring costly pixel-level masks. Fig. we evaluate the performance with different prompts on strict FSS setting on COCO-20' against VRP- SAM, which also supports such prompts. SANSA shows gains of +15.0%, +4.6%, and +5.8% when using points, scribbles, and boxes as annotations. More importantly, the performance drop from masks point prompts is substantially smaller for SANSA (-6.8% compared to VRP-SAM (-15.5%). We attribute this to the tightly coupled design of our architecture, which maximizes feature reuse by jointly leveraging the representations for prompt encoding, feature matching, and mask decoding. Prompt generation details and qualitative results are Appendix E.
4.2 Generalist In-Context Setting
Following recent few-shot segmentation works [44, 89, 42, 83], we evaluate SANSA in the in-context segmentation setting, where a single generalist model is tested across multiple benchmarks (cf. Tab. 2). Given the lack of standardized training protocol, each method beyond ADE20K and COCO, which are shared across all approaches. We evaluate three configurations of approach: i) minimal setup using only COCO and ADE20K, ii) extended version incorporating LVIS, following the setup of SegIC [44], and iii) a variant including PACO to mitigate object-level bias and reinforce part segmentation capabilities.
explicitly report additional datasets used by
When trained only on COCO and ADE20K coarse categories, SANSA exhibits strong out-of-domain generalization the LVIS benchmark, outperforming DiffewS and SINE by +4.8% and +5.0%, respectively. Moreover, when LVIS included in the training set (matching SegIC in-domain setup), SANSA further improves performance and surpasses SegIC by +5.4%. Despite not being trained at
8

---

Table 2: Performance of SANSA against Generalist In-context models. Excluding training- free approaches, all methods are trained on COCO and ADE20K, and we report additional training datasets for each one. SegIC (*) uses additional supervision via textual meta- meta-promy prompt test time.
few-shot segmentation
par7 seg.
Additional Datasets
LVIS-92
COCO-20*
USS-1000
Pascal Part
PACO
Method
Params
1-shot
5-shot
I-shot 5-shot
1-shot 5-shot
Part
Training-fre free methods
Matcher [42]
n.a.
945M
33.0
40.0
52.7
60.7
87.0
89.6
42.9
34.7
GF-SAM [83]
n.a.
945M
35.2
44.2
58.7
66.8
88.0
88.9
44.5
36.3
Painter [70]
NYUv2
354
10.5
61.7
62.3
30.4
SegGPT [71]
Pascal, PACO
354
18.6
25.4
89.3
35.8
13.5
SINE [40]
Object365, PACO
373
31.2
36.2
DiffewS [89]
Pascal
890
31.4
35.4
71.3
72.2
87.8
88.0
34.0
22.8
SegIC [44]
LVIS
1.2 B
47.8
74.5*
88.4
31.2
18.7
SANSA
250 M
36.2
42.5
76.4
77.1
88.2
89.0
40.4
29.2
SANSA
LVIS
250 M
53.2
58.1
74.7
76.3
89.1
89.7
41.5
31.2
SANSA
LVIS, PACO
250 M
50.3
59.0
75.6
78.6
90.0
91.0
49.1
43.0
the part level, SANSA exhibits strong cross-task generalization capabilities, surpassing generalist baselines by large margin (+7.5% and +8.4% over DiffewS, the best competitor on Pascal-Part and Paco-Part, respectively). including those trained on part segmentation datasets [71, 40]. Our method is only outperformed by training-free models, which, by design, not biased by object-level training. To address this bias, we follow [71, 40] and augment our training set with PACO, leading to +6.7% on Paco-Part (in-domain) and +4.6% Pascal-Part (out-of-domain) against the strong baseline of GF-SAM. Finally, we highlight that, within the generalist model category, SANSA is the most compact solution, with only 250 M parameters.
4.3 Generalization across domains and styles
To explore how SANSA generalizes beyond the scope of standard benchmarks, follow [62, 42] and present qualitative examples drawn from in-the-wild image pairs, shown in Fig. 7. In each case, the model is tasked with few-shot segmentation using reference image from COCO [39] and target image collected from the web, offering a complementary view to benchmark evaluations.
Reference image
Target images
Figure 7: Few-shot segmentation with SANSA in-the-wild. These examples showcase SANSA ability to handle alternative types of challenges not typically covered by standard benchmarks, such as domain shifts (e.g. real-world to cartoon) and style variations (e.g.. photos to sketches).
These examples introduce different challenge compared to traditional benchmarks, focusing on domain and style shifts, such when the target image is cartoon stylized sketch, when the

---

Table 3: Ablation on COCO-20', Left: fine-tuning underperforms adaptation. Middle: simple
adapters yield strong gains, while higher capacity (larger bottlenecks added complexity, e.g..
MONA) hurts generalization. Right: adapting the last two stages suffices to disentangle semantics.
Fine-tuning strategies
Adaptation strategies
Adapter placement
Method
Params
mloU
Method
mIoU
Stages
mIoU
Frozen
0
32.2
LoRA [32]
58.0
None
32.2
From Scratch
224 M
37.1
Adapter [31]
59.4
All
0-3
59.4
Full FT
224 M
51.6
MONA [80]
56.9
Early
0-1
38.7
Decoder FT
4 M
52.1
AdaptFormer [14]
Middle
1-2
57.9
QKV FT
M
55.3
Bottleneck 0.8x channel dim
58.2
Late
2-3
60.2
Backbone FT
210 M
55.2
Bottleneck 0.5x channel dim
59.6
SANSA
10M
60.2
Bottleneck 0.3x channel dim
60.2
visual appearance changes dramatically between the reference and target. These results demonstrate SANSA strong generalization. Importantly, this robustness across domains and styles is inherited from the frozen SAM2. Our method preserves this capability by learning a semantic mapping within the frozen feature space, enabling reliable correspondence even under significant distribution shifts.
4.4 Ablation studies
We investigate three central aspects of our design: adaptation vs. fine-tuning, adapter architecture and capacity, and adapter placement. Results are summarized in Tab. 3.
Why adaptation instead of fine-tuning? central question of our work is how to distill knowledge from a pretrained model (i.e., SAM2) while preserving generalization, key challenge in FSS. To this end, we evaluate fine-tuning strategies targeting the decoder, QKV projections, backbone, and full model, and compare them with inserting adapters into frozen weights. Results show that adaptation outperforms fine-tuning, indicating that it better preserves S SAM2 pretrained priors, shifting the embedding space toward task-relevant semantics without altering the underlying representations.
What makes an adaptation strategy effective? Basic adapters such as LoRA [32], Adapter [31], and AdaptFormer [14] all yield similar gains around 22% mloU over frozen SAM2. The slight gains (~2%) with Adapter and AdaptFormer w.r.t. LoRA suggest that simple non-linearity can refine this structure but not fundamentally reshape it. By contrast, increasing adapter capacity, either by enlarging the bottleneck or using more complex designs such as MONA [80], reduces generalization. These results show that effective adaptation thus requires simplicity and constraint: www-capacity bottlenecked modules best expose SAM2 latent semantics, while excessive capacity tends to overfit.
Where should adapters be placed? Prior works often insert adapters throughout the entire backbone. In our case, we find that adapting only the last two stages is sufficient, as these layers already capture high-level semantic information, which is the focus of our disentanglement objective.
Additional ablations, including backbone scale and training objective, are provided in Appendix D.
Conclusions
In this work, we introduced SANSA, which enhances SAM2 to accept and propagate visual prompt across frames, and showed that its memory attention mechanism can be re-focused towards semantic correspondences by restructuring the feature space. We addressed three fundamental inquiries: (1) we demonstrated that semantic information can be extracted from SAM2 features through lightweight bottleneck transformations, answering initial research question, (2) showed that this can be achieved while keeping SAM2 frozen, thereby preserving its segmentation capabilities, and finally (3) experimentally verified that the leamned semantic mapping generalizes robustly to novel classes, with SOTA performance on strict few-shot benchmarks and against generalist models. Beyond technical contributions, SANSA offers practical advantages: it supports diverse prompts for annotation-efficient applications, and offers a 3x speed increase. Finally, our results suggest that foundation models like SAM2 may contain richer task-adaptable knowledge than their original objectives imply, a direction worthy of future exploration.
10

---

Acknowledgements. Claudia Cuttano was supported by the Sustainable Mobility Center (CNMS) which received funding from the European Union Next Generation EU (Piano Nazionale di Ripresa Resilienza (PNRR), Missione 4 Componente 2 Investimento 1.4 "Potenziamento strutture di ricerca e creazione di "campioni nazionali di R&S" su alcune Key Enabling Technologies") with grant agreement no. EN_00000023.
Gabriele Trivigno, Giuseppe Averta and Carlo Masone were supported by FAIR Future Artificial Intelligence Research which received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) MISSIONE 4 COMPONENTE 2, INVES- TIMENTO 1.3 - D.D. 1555 1 11/10/2022, PE0000001 3). This manuscript reflects only the authors' views and opinions, neither the European Union nor the European Commission can be considered responsible for them.
We acknowledge the CINECA award under the ISCRA initiative, for the availability of high perfor- mance computing resources.
References
[1] A. Aghajanyan, L. Zettlemoyer, and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine- tinettnnig. In Proceedings of International Joint Conference on Natural Language Processing, pages 7319-733177 7328.. Association for Computational Linguistics, 2021. 2,5
[2] M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F, S. Khan. Foundation models defining new era in vision: survey and outlook. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
[3] Y. Bai, Q Yu, B. Yun, D. Jin, Y. Xia, and Y. Wang. FS-MedSAM2: Exploring the Potential of SAM2 for Few-Shot Medical Image Segmentation without Fine-tuning. arXiv preprint arXiv:2409.0499, 2024.
[4] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Varranne-ivvarinnc--ovvaia regularization for self-
supervised learning. Int. Conf. Learn. Represent, 2022.
[5] A. Bardes, Ponce, and Y. LeCun. Vicregl: Self-supervised learning of local visual features, Advances in Neural Information Processing Systems, 35:8799-88100 2022. 2
[6] H. Bay. T. Tuytelaars, and L. Van Gool. Surf: Speeded up robust features. In Computer Vision-ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9. pages 404-417. Springer, 2006. 3
[7] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg. A. Bosselut, E. Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint 10l.2220 2021. 1,3
[8] Boudiaf, Kervadec, Masud, Piantanida, I. Ben Ayed, and Dolz. Few shot segmentation without meta- learnig: A good transductive inference all you need? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1979-139889 2021. 24
[9] T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:187771901,, 2020. 27
[10] S. Candemir. S. Jaeger, K. Palaniappan, J. P. Musco, R. K. Singh, Z. Xue, A. Karargyris, S. Antani, G. Thoma, and McDonald. Lung segmentation chest radiographs using anatomical atlases with nonrigid registration. JEEE transactions on medical imaging, 33(2):577-590 2013. 2
[11] M. Caron, H. Touvron, Misra, H. Jégou, J. Mairal, Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer
vision, pages 9650-9660, 2021. 2, 18
[12] H. Chen, Y. Dong, Z. Lu, Y. Yu, and J. Han. Pixel matching network for cross-domain few-shot segmen- tation. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 978-987, 2024. 23
[13] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848, 2017.
[14] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16666 2022. 5, 7, 10, 19, 26, 27

---

[15] X. Chen, R. Mottaghi, X. Liu. S. Fidler. R. Urtasun, and A. Yuille. Detect what you can: Detecting and representing ubjects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1971-1978, 2014. 28
[16] B. Cheng, L. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-122 1299, 2022. 1
[17] C. Cheng, L.. Song, R. Xue, Wang, H. Sun, Y. Ge, and Y. Shan. Meta-adapter: An online few-shot learner for vision-language model. In NeurlPS, 2023. 2
[18] N. Codella, V. Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, ct al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: 092. 03368, 2019. 2
[19] C. Cuttano, G. Trivigno, G. Rosi, C. Masone, and G. Averta. Samwise: Infusing wisdom in sam2 for text- driven vidco segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3395-3405, 2025. 3
[20] P. Engstler, L. Mclas Kyrrizzi, C. Rupprecht, and I. Laina. Understanding self- supervisdd features for learning unsupervised instance segmentation. arXiv preprint arXiv:2311.14665, 2023.
[21] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303 338, 2010. 28
[22] Q. Fan, W. Pei, Y.-W Tai, and C.-K. Tang. Self-support few-shot semantic segmentation. In European Conference on Computer Vision, pages 701-719. Springer. 2022. 3, 3.26,22 27
[23] H. Gao, J. Xiao, Y. Yin, T. Liu, and J. Shi. A mutually supervised graph attention network for few-shot segmentation: The perspective fully utilizing limited samples. IEEF Transactions on neural networks and learning systems, 35(4):4826-4838, 2022. 7828
[24] A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pages 5356-5364,
2019. 7, 28
[25] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020. 18
[26] K. He. G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017. I
[27] R. He, L. Liu, H. Ye, Q. Tan, B. Ding, L. Cheng. J. Low, L. Bing, and L. Si. On the effectiveness of adapter-baser tuning for pretrained language model adaptation. ACI, 2021.
[28] W. He. Y. Zhang, W. Zhuo, L. Shen, J. Yang, S. Deng, and L. Sun. Apseg: nuto-prompt network for cross-domain few-shot semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 227722,,,, 2024, 2
[29] O. Hénaff, S. Koppula, J.-B. Alayrac, A. Van den Oord, O. Vinyals, and J. Carreira. Efficient visual pretraining with contrastive detection. In Proceedingy of the IEEE/CVF International Conference on Computer Vision, pages 10086-10096, 2021. 2,
[30] S. Hong, S. Cho, J. Nam, S. Lin, and S. Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In European Conference on Computer Vision, pages 108-126. Springer, 2022. 3, 27
[31] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and
S. Gelly. Parameter-<fffiient transfer learning for NI.P. In International conference on machine learning, pages 2790-2799, PMLR, 2019. 2. 10, 19
[32] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Li, S. Wang. L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. Inf. Conf. Learn. Represent. 1(2)3, 2022. 2, 5, 10, 19
[33] S. Kim, Min, B. Ham, S. Jcon, Lin, and K. Sohn. Fess: Fully convolutional sclf-similarity for dense semantic correspondence, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6560-6569, 2017. 3
[34] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár. Panoptic segmentation. in 2019 iece. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9396-9405, 2018.
[35] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitchead, A. C. Berg. W.-Y. Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer 2023. 1,4
Vision, pages
12

---

[36] C. Lang, G. Cheng, B. Tu, and J. Han. Learning what not to segment: A new perspective on few-shot segmentation. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition. pages 8057-8067, 2022. 3, 5, 26, 27
[37] C. Li, H. Farkhoor, R. Liu, and J. Yosinski. Mcasuring the intrinsic dimension of objective landscapes. In Int. Conf. Learn. Represent., 2018. 2
[38] G. Li, V. Jampani, I.. Sevilla-Lara, D. Sun, J. Kim, and J. Kim. Adaptive prototype learning and allocation for few-shot segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 8334-8343, 8344,,3332222 2021. 1, 3, 26, 27
[39] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitmick. Microsoft coco: Common objects in context. In Computer vision-ES Esion-ECCV& 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pages 740-755. Springer. 2014. 7, 9, 28
[40] Y. Liu, C. Jing, H. Li, M. Zhu, H. Chen, X. Wang, and C. Shen, A simple image segmentation framework via in-context examples. arXiv preprint (rXiv:2410.04842) 2024, 9, 28
[41] Y. Liu, N. Liu, Q. Cao, X. Yao, J. Han, and L. Shao. Learning non ton-ttrget knowledge for few-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pages 11573-11583 11582, 2022. 3, 26, 27
[42] Y. Liu, M. Zhu, H. Li, H. Chen, X. Wang, and C. Shen. Matcher: Segment anything with shot using all-purpose feature matching. arXiv preprint arXiv:2305.13310. 2023. 7, 9, 21, 23. 24, 26, 27. 28
[43] D. G. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
[44] L. Meng, S. Lan, H. Li, J. M. Alvarez, Z. Wu, and Y-G. Jiang- Segie: Unleashing the emergent corresoondence for in-context segmentation. In European Conference Computer Vision, pages 203 220.
Springer, 2024. 3.
9, 20, 22, 23, 27, 28
[45] F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. Iece, 2016.
[46] J. Min, D. Kang, and M. Cho. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6941-6952, October 2021. 3, 23, 27
[47] S. Moon, S. S. Sohn, H. Zhou, S. Yoon, V. Pavlovic, M. H. Khan, and M. Kapadia. Msi: Maximize support- set information for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19266-19276, 2023. 3, 27
[48] K. Nguyen and S. Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 622-631, 2019. 7,
[49] D. Ofri-Amar, M. Geyer, Kasten, and Dekel. Neural congealing: Aligning images a joint semantic atlas. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
19403-19412, 2023. 3
[50] M. Oquab, T. Darcet, Moutakanni, Vo, M. Szafraniec, Khalidov, Fernandez, D. Haziza, F. Massa, A. El-Nouby. et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719/ 2023. 1, 3, 18
[51] F. Perazzi, J. Tont-, B. Mililliams, L. Van Gool, M. Gross, and A. Sorkine- Sorkine-Homunng A benchmark datasct and evaluation methodology for vidco object segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 724-732, 2016. 26
[52) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748 8763. PmLR, 2021. 1
[53] V. Ramanathan, A. Kalia, V. Petrovic, Y. Wen, B. Zheng, B. Guo, R. Wang, A. Marquez, R. Kovvuri,
A. Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEF/CV Conference Computer Vision and Pattern Recognition. pages 7141-7151, 2023. 7
[54] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and vidcos. arXiv preprint arXiv: (tf::2400.00714 2024. 1, 3, 7, 16, 20, 21
[55] I. Rocco, R. Arandjelovié, and Sivic. End-to-end weakly-supervised semantic alignment. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 6917-692 2018. 3
[56] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
13

---

[57] A. A. Rusu, D. Rao, J. Sygnowski. O. Vinyals. R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. arXiv preprint urXiv:1807.05960 2018. 2,5
[58] C. Ryali, Y.-T. Hu, D. Bolya, C. Wei, H. Fan, P.-Y. Huang, V. Aggarwal, A. Chowdhury, O. Poursaced.
J. Hoffman, et al. Hiera: A hierarchical vision transformer without the hells-and-whistles In International
conference machine learning, pages 29441-29454. PMLR, 2023. 7, 26, 27
[59] A. Shaban. Bansal, Z. Liu, Essa, and B. Boots. One-shot learning for semantic segmentation. arXiv
preprint ar 1XX.::700.0044
2017. 1,5
[60] A. Shaker, S. T. Wasim, M. Danelljan, S. Khan, M.-H. Yang. and F. S. Khan. Efficient video object segmentation via modulated cross-attention memory. arXiv preprint arXiv:2403. 2024. 19, 20
[61] X. Shi, D. Wei, Y. Zhang, D. Lu, M. Ning, J. Chen, K. Ma, and Y. Zheng. Dense
attention weighted mask aggregation for few-shot segmentation. In European Conference on Computer
Vision, pages 151-168. Springer, 2022. 7
[62] Y. Sun, J. Chen, S. Zhang, X. Zhang, Q. Chen, G. Zhang. E. Ding, J. Wang, and Z. Li. VRP-SAM: SAM with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 23565 23574, 2024. 1, 3.5, 5, 7, 9, 20, 21, 24, 26
[63] L. Tang, M. Jia, Q. Wang, C. P. Phoo, and B. Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1 136--138, 2023. 1,3
[64] T. Taniai, S. N. Sinha, and Y. Sato. Joint recovery of dense correspondence and cosegmentation in two images. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4246 4255, 2016. 26
[65] H. Touvron, M. Cord, A, El-Nouby, J. Verbeek, and H. Jégou. Three things everyone should know about vision transformers. In Europeun Conference Computer Vision, pages 497-515. Springer, 2022. 19
[66] N. Tumanyan, O. Bar-Tal, S. Bagon, and T. Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10748- 10757, 2022. 3
[67] O. Vinyals, C. Blundell, T. Lillicrap. D. Wierstra, al. Matching networks for shot learning. Advances in neural information processing systems, 29, 2016.
[68] H. Wang, X. Zhang, Y. Hu, Y. Yang, X. Cao, and X. Zhen. Few-shot semantic segmentation with democratic attention networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 730-746. Springer, 2020. 3, 27
[69] K. Wang. J. H. Liew, Y. Zou, D. Zhou, and J. Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In proceedings of the TEEF/CVF international conference on computer vision, pages 9197-9206, 2019. 3, 5, 26, 27
[70] X. Wang. W. Wang, Y. Cao, C. Shen, and Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830 (0000683,2200 2023. 2,
[71] X. Wang, X. Zhang, Y. Cao, W. Wang, C. Shen, and T. Huang. Seggpt: Towards segmenting everything in context. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 1130-1140,
2023. 3, 9. 27, 28
[72] Y. Wang. N. Luo, and T. Zhang. Focus query: Adversarial mining transformer for few-shot segmentation. Advances in neural information processing systems, 36:31524 31542, 2023. 7.23
[73] 7. Wei, I.. Chen, Y. Jin, X. Ma, T. Liu, P. Ling, B. Wang, H. Chen, and J. Zheng. Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and patiern recognition, pages 28619-28630, 2024. 3
(74] Q Xu, G. Lin, C. C. Loy, C. Long, Z. Li, and R. Zhao. Eliminating feature ambiguity for fow-shot segmentation. In European Conference on Computer Vision, pages 116-433. Springer, 2024. 23, 24
[75] Q Xu, X. Liu, L. Zhu, G. Lin, C. Long. Z. Li, and R. Zhao. Hybrid mamba for few-shot segmentation. Advances in Neural Information Processing Systems, 37:73858-7555 738883 2025. 3, 7, 23, 24
[76] C. Yang, Z. Wu, B. Zhou, and S. Lin. Instance localization for self- ceffssuperiieed detection pretraining, In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3987-3996.
2021. 2,3
[77] J. Yang, K. Zhang, Z. Cui, J. Su, J. Luo, and X. Wei. Inscon: Instance consistency feature representation via self-supervised learning, arXiv preprint M2Xiv:2203.07088 2022. 2.3
[78] Z. Yang, Y. Wei, and Y. Yang. Associating objects with transformers for video object segmentation. Advances in Neural Information Processing Systems, 34:249,-2502, 2021. 19,
[79] Z. Yang and Y. Yang. Decoupling features in hierarchical propagation for video object segmentation. Advances in Neural Information Processing Systems,
2022. 19, 20
14

---

[80] D. Yin, L. Hu. B. Li, Y. Zhang, and X. Yang. 5%0 100%: Breaking performance shackles of full fine-tuning visual recognition tasks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20071-20081, 2025.
[81] J. Yu, Lin, J. Yang. X. Shen, X. Lu, and T. S. Huang, Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference computer vision, pages 4471-4480, 2019. 24
[82] Y. Yuan. On the power of foundation models. In International Conference on Machine Learning, pages 40519-40530. PMLR, 2023.
[83] Zhang, G. Gao, J. Jiao, H. Liu, and Y. Wei. Bridge the Points: Graph-based Few-shot Segment Anything Semantically, arXiv preprint arXiv:2410.06964 2024. 1, 2, 3, 7, 8, 9, 20, 21, 22, 23, 24, 26
[84] G. Zhang, G. Kang, Y. Yang. and Y. Wei. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems, 34:21984-21996, 2021. 3,27
[85] J. Zhang, C. Hermman,, J. Hur, E. Chen, V. Jampani, D. Sun, and M.-H. Yang. Telling left from right: semantic correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3076-3085, 2024. 1,3
Identifying geometry
[86] J. Zhang, C. Herrmann, J. Hur, L. Polania Cabrera, V. Jampani, D. Sun, and M.-H. Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36:45533 45547, 2023. 3, 6, 26
[87] R. Zhang. Z. Jiang, Z. Guo, S. Yan. Pan, X. Ma, H. Dong, P. Gao, and H. Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305. 03048, 2023. 2,
[88] B. Zhou, Zhao, Puig. Xiao, S. Fidler, Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302 321, 2019. 28
[89] M. Zhu, Y. Liu, Z. Luo, C. Jing. H. Chen, G. Xu, X. Wang, and C. Shen. Unleashing the potential of the diffusson model in few-shot semantic segmentation. arXiv preprint arXiv:2410.02366, 2024. 3, 7, 8.
9, 23,
[90] X. Zou, J. Yang, H. Zhang, F. Li, L. J. Wang. L. Wang, J. Gao, and Y. J. Lec. Segment everything everywhere all once. Advances in neural information processing systems, 36:1 36:19769-11777 2023. 1, 24
[91] H. Zu, J. Ge, H. Xiao, J. Xie, Z. Zhou, Y. Meng, J. Ni, J. Niu, L. Zhang, L.. Ni, et al. Rethinking Few-Shot Medical Image Segmentation by SAM2: A Training-Free Framework with Augmentative Prompting and Dynamic Matching. arXiv preprint ar0iv:2003.00 04826, 2025. 3
15

---

Appendix
Table of Contents:
A: Discussion
B Emergence of Semantic Representations
$C: Scalable and Fast Annotation
SD Baselines and Ablation Studies
SE Additional Datasets and Qualitative Comparison
S Exploratory Analysis: Negative Prompts
SF: Exploring Dependence on Spatial Continuity in SAM2
SH: Implementation and Reproducibility
A Discussion
A.1 Limitations and Future Works
By keeping the Segment Anything 2 (SAM2) [54] model entirely frozen and storing only adapter weights, our SANSA preserves SAM2 state-of-the- performance on Video Object Segmentation (VOS), while enabling few-shot segmentation within unified pipeline. As method builds directly upon SAM2, it also inherits its design assumptions and limitations. In particular, while SAM2 supports tracking and segmenting multiple objects across video, it does so by processing each object independently. This behavior is standard in few-shot segmentation, where each object is typically queried and segmented in isolation using a separate prompt. Nevertheless, future work could explore leveraging contextual relationships between multiple objects, which may improve segmentation accuracy through cross-instance interactions and enable multi-class few-shot segmentation.
A.2 Societal Impacts
Our method leverages the publicly available, pre-trained weights of Segment Anything (SAM2), thus requiring minimal additional computational resources. This significantly limits the environmental impact commonly associated with training large foundation models from scratch. By providing an analysis of the underlying mechanisms of foundation models like SAM2, contribute understanding of their operation and potential applications beyond conventional tasks. We believe foster broader awareness and acceptance of foundation models
deeper
this knowledge dissemination both within and outside the scientific community.
Moreover, our approach facilitates the efficient annotation of large-scale datasets with minimal human effort (see Section C). Given that data annotation represents one of the primary bottlenecks and cost drivers in deploying vision models in real-world scenarios, method could bring benefit in reducing the impact of computational burdens associated with dataset curation.
However, we recognize potential risks inherent to few-shot learning methods, including SANSA. Specifically, its capability for rapid adaptation with limited data could exploited ways that raise ethical and privacy concern. For example, SANSA might be applied in surveillance systems to identify individuals without their consent, posing risks personal privacy and civil liberties.
B Emergence of Semantic Representations
B.1 Analyzing Semantic Structure via Principal Component Decompasition
Our hypothesis is that SAM2 features do encode semantic information, albeit entangled with low- level signals, whereas our adapted SANSA features disentangle this semantic content to make it explicit. To validate this hypothesis, we analyze the feature spaces of both models via clustering- based experiment, reported in Fig. 8. Specifically, we evaluate how well semantic clusters form in lower-dimensional subspaces obtained PCA, comparing frozen SAM SAM2 to our adapted SANSA features. To this end, extract object-level features from fold 0 of COCO-20', which contains
16

---

a. Accuray wrt #PCA components
b. SAM2 Features
chair
1.0
Cumulative k-means Accuracy
spoon
0.8
dog
0.6
person
C. SANSA Features
boat
airplane
0.2
Frozen SAM2
SANSA (ours)
mouse
100
200
300
400
Principal Components
Figure 8: Semantic information concentration across principal components. (a) We evaluate how semantic information is distributed across the principal components of the feature space. For each number retained components (x-axis), we compute class centroids on COCO-20' training embeddings and assign test embeddings to the nearest centroid. The y-axis shows the relative classification accuracy, normalized by the accuracy at full dimensionality. indicates that semantic information is concentrated in the leading components; a gradual rise (SAM2) suggests it is entangled with other signals. (b) 2D PCA projection of frozen SAM2 features, colored by class. Semantic structure is weak, with significant overlap across classes. (c) 2D PCA projection of SANSA features, showing compact and well-separated clusters aligned with semantic categories.
steep rise (SANSA)
categories unseen by SANSA (trained the remaining folds). We then analyze the semantic organization of these features through a clustering- assignment task combined with Principal Component Analysis (PCA):
1. We apply PCA to the extracted features, varying the number of principal components (PCs) retained, from 2 up to 450, capturing 99% of the total variance.
2. For each reduced dimeesdodiim representation, class centroids are computed on the training set features using k-means clustering.
3. During testing, embeddings from the test set are assigned the nearest centroid based on minimum Euclidean distance.
4. The assignment accuracy per class is computed using the ground-truth labels.
Plot (a) reports the relative assignment accuracy (y-axis) function of the number of PCs (x-axis). This accuracy is normalized by the overall accuracy obtained with the full feature space (i.e., no dimensionality reduction) for each method, allowing direct comparison of how semantic information concentrates across principal components independently absolute accuracy differences. The results reveal marked difference between SAM2 and SANSA. The blue curve (SAM2) shows a gradual increase in accuracy as more PCs are included, indicating that semantic and non-semantic signals are mixed across many leading components. Conversely, the orange curve (SANSA) exhibits sharp rise with only a few PCs, revealing that semantic information is concentrated and explicitly encoded in the leading principal components of the adapted feature space.
Plots (b) and (c) provide a visual illustration of this phenomenon by projecting the features onto the first two principal components and coloring them by class label. The frozen SAM2 features, in plot (b), show weak class discriminability, confirming that their semantic structure entangled with task-specific features optimized for SAM2 original tracking objective. Conversely, SANSA features, plot (c), form compact, well-separated clusters, indicating that semantic information dominates the main axes of variation in the adapted feature space.
17

---

B.2 Semantic Discriminability and Downstream Transferability
The previous analysis revealed that in the adapted SANSA feature space, semantic information is predominantly encoded. This raises a natural follow-up question: to what extent this semantic information was already encoded the original, frozen SAM2 features, and how easily can be accessed for downstream tasks?
To investigate this, we present a two-fold analysis in Fig. 9, comparing two models: i) frozen SAM2, and ii) our SANSA trained on folds 3, of COCO-20'. We conduct two experiments on fold 0, ie. unseen classes for model:
1. Linear probing: following standard practice in representation learning [25, 50, 11], conduct linear probing experiment to assess semantic discriminability in the two feature spaces. We extract features from both models and train pixel-level linear classifier on the training of COCO-20* fold O. Note that both models are frozen and only a single linear layer is trained for the task of semantic segmentation. Evaluation performed on the corresponding validation set. A high mean Intersection- -oeeeeccctoooover--eee Union (mIoU) indicates that semantic signals are present in the feature space and can be recovered by simple linear projection.
2. Few-Shot Segmentation: on the same COCO-20' fold, we test the performance of frozen SAM2 and SANSA on unseen classes in the downstream task of few-shot segmentation.
On the left side of the plot, the results indicate that SAM2 achieves 65.8% mIoU with a simple linear probe, confirming that they encode a non-trivial amount of semantic information that is linearly accessible, Interestingly, when repeating the same experiment using our SANSA features, we observe only modest improvement to 69.2% mloU (+3.4%).
Semantic Discriminability vs. Downstream Performance
69.2
f +3,4
65.8
57.5
miou (%)
50-
+28.6
30-
Frozen SAM2
SANSA (ours)
28.9
Linear Probing
Few-Shot Segmentation
(Semantic Segmentation)
Figure 9: Left: Linear probing for Semantic Segmentation on a fold of COCO-20" comparing frozen SAM2 and SANSA (trained on a disjoint class set). Right: Few-shot segmentation performance on the same set of data. Despite poor performance on few-shot segmentation, a simple linear layer (ef linear probing) can learn to discriminate semantic-classes from SAM2 frozen features. These results, together, suggest the presence of semantic structure, albeit entangled with other signals.
On the right, FSS results show that SAM2 achieves 28.9% mIoU, while SANSA reaches 57.5%, with substantial +28.6% improvement. When considered together, these results highlight key insight: although SAM2 features already encode meaningful semantic cues, their utility in downstream tasks is limited by the way this information is entangled with low-level and instance-specific signals. The relatively modest gain in linear probing performance (+3.4%) compared to the substantial improvement in few-shot segmentation (+28.6%) suggests that SANSA primarily reorganizes the latent structure into an explicitly semantic form, enabling downstream tasks that require semantic-level understanding.
18

---

B.3 Extracting Semantics Without Compromising SAM2 Capabilities
In the previous sections, we investigate how SANSA exposes the latent semantic structure embedded in frozen SAM2 features, which was our main goal. Moreover, an important principle of our design was to extract this semantic structure without sacrificing SAM2 Promptable Segmentation capabilities or its state-of-the-ar performance on Video Object Segmentation. Consequently, our choice was to insert AdaptFormer blocks [14] into the last two layers of the frozen Image Encoder, which introduce residual bottleneck projections, reorganizing the feature space without altering any of the A22 weights.
For comparison, we evaluate several alternative adaptation strategies: i) fine-tuning only the decoder, tii fine-tuning the Query, Key, and Value (QKV) projections within the attention layers [65], and iii) full backbone fine-tuning. Decoder fine-tuning updates around 4 million parameters, QKV fine-tuning modifies approximately 50 million parameters, and full backbone fine-tuning involves over 210 million parameters.
Tab. 4 reports results under two settings: out-of-domain, where models are trained subset of COCO-20' folds and evaluated on unseen classes (i.e., the strict few-shot segmentation setting, which our primary focus), and in-domain, where models are trained on all categories.
Table 4: Comparison of adaptation strategies. We compare various strategies for adapting SAM2 in terms of generalization to unseen classes lout-of-domain) and performance on seen classes (in- domain). SANSA achieves the best out-of-domain mIoU with minimal parameter overhead, while preserving the original SAM2 weights.
COCO-20
Trainable Parameters
SAM2 Weights
Adaptation Strategy
in-domain out-of-domain
Update
Decoder Finetuning
63.7
52.1
4 M
QKV-only Finetuning
73.5
55.3
50 M
Backbone Finetuning
74.8
55,2
210 M
AdaptFormer (SANSA)
72.0
60.2
10 M
X
The table highlights the advantages of our approach. Tunnigg the decoder provides sub-optimal results both in- and outmoi-domain, as performances are constrained by the limited semantic structure of frozen features. Finetuning the backbone, or the attention layers, is effective in producing semantic aware features, as shown by the in-domain performance. However, such fine-tuning, despite its greater capacity which boosts in-domain performance, leads to overfitting to training categories, as shown by the large performance drop on unseen classes.
On the other hand, adapters restrict updates a low-dimensional bottleneck [32, 31], forcing the model to prioritize generalizable transformations over spurious in-domain correlations [27]. demonstrating more consistent and stable performance across both seen and unseen classes.
Finally, by maintaining the SAM2 backbone and decoder entirely frozen, we ensure that the original model core capabilities, such as Promptable Segmentation and Video Object Segmentation, remain uncompromised. This is critical to deploying SANSA in practical settings, obtaining a model that fully supports both geometric and visual prompts, by storing only the adapter weights.
B.4 Exploring Implicit Semantics in Traditional Video Object Segmentation Methods
Our hypothesis is that the class- agnostic training objective of Video Object Segmentation encourages the learning of semantically meaningful representations, even without category supervision. While our main analysis focused on SAM2, which benefits from its large-scale pretraining, we investigate whether similar semantic understanding properties arise in traditional VOS trackers trained at smaller scale.
To this end, we evaluate three representative VOS methods, AOT [78], DeAOT [79], and MAVOS [60], alongside SAM2, on the COCO-20 1-shot segmentation benchmark. As shown Tab. 5, all models exhibit relatively weak performance when directly transferred to few-shot segmentation. We then apply our adaptation to each of these models, and evaluate them in the striet FSS setting on COCO-20', i.e. focusing on their ability to generalize to previously unseen classes. The results show
19

---

consistent behavior: despite their different training scales and capacities, all VOS models demonstrate a large performance gap between their frozen and adapted versions. For instance, MAVOS improves from 27.1% to 52.3% mloU after adaptation, while SAM2 with Hiera-B improves from 28.0 % to 55.4%. The gap between these methods and SAM2-B is attributable to its broader generalization capabilities, stemming from its diverse and large-scale pretraining.
Table 5: Evaluation of traditional Video Object Segmentation methods and SAM2 with and without SANSA adaptation. We report mloU (%) on COCO-20' 1-shot segmentation.
Method
Backbone
#Params
Frozen
+SANSA
AOT [78]
Swin-B
70 M
25.0
48.5
DeAOT [79]
Swin-B
26.9
51.0
MAVOS [60]
Swin-B
96 M
27.1
52.3
SAM2 [54]
Hiera-B
86 M
28.0
55.4
SAM2 [54]
Hiera-L
234 M
32.2
60.2
This consistent behavior across models suggests that the VOS training paradigm inherently encourages the emergence of transferable semantic representations. In summary, our findings support the view that such semantic structure an implicit property of the VOS objective itself, and that it can be effectively made explicit, and successfully repurposed, for downstream semantic tasks.
C Scalable and Fast Annotation
While the previous sections have established the strong segmentation performance of SANSA across benchmarks, particularly compelling advantage of our approach is its ability to facilitate fast and scalable annotation in practical, real-world scenarios. In many applications, such as constructing new datasets adapting models to novel domains, rapid annotation of large image collections without the need for fine-tuning on test categories is essential.
To rigorously evaluate this capability, we simulate large-scale annotation task on both the COCO-20* and LVIS-92' validation sets, mimicking a realistic deployment scenario in which new concepts must be annotated the fly. For both datasets, we designate fold 0 as the test split, containing categories that are held out during training. Models are trained on the remaining folds. On COCO-20" we randomly sample 20 reference images, one per category in fold O, and use them to guide the segmentation of 10,000 target images sampled from the same fold. On LVIS-92 which contains a larger and more fine- meneggrined label space, similarly sample 92 reference classes from fold 0 and use these to segment 10,000 target images. In both cases, the reference and target sets are fixed across all methods ensure fair comparison. Moreover, for all the methods, reference image features computed once and cached, avoiding repeated backbone forward passes at inference time.
Table 6: Evaluation of large-scale annotation efficiency on COCO-20' and IVIS-92 datasets. We report mloU and wall-clock annotation time (minutes) required to segment 10,000 target images using reference examples from unseen categories fold O. Reference features are precomputed and reused for fair comparison. Experiments conducted on an NVIDIA RTX 4090. SANSA achieves the best trade-off' between accuracy and speed, demonstrating strong generalization and efficiency.
Fold
Method
Total Annotation Time
COCO mloU (%) f
LVIS mloU (%)
VRP-SAM [62]
54.8
34.9
38 min
SegIC [44]
55.1
46.7
77 min
GF-SAM [83]
59.7
38.4
58 min
SANSA (ours)
61.2
55.4
16 min
We benchmark SANSA against recent state-of-the-art few-shot segmentation methods, including GF-SAM [83], SegIC [44], and VRP-SAM [62], by measuring the total wall-clock time required to
20

---

segment all 10,000 images and evaluating segmentation quality via mean cnterreectiooovvv-nniio (mloU) against ground truth masks. Results are summarized in Tab.
SANSA achieves the best trade-off between speed and accuracy across both COCO and LVIS. In particular, it completes annotation 2-5x faster than competing approaches while maintaining the highest mloU. On LVIS, where the label space is significantly more complex, the gains are more pronounced: SANSA outperforms all baselines by a large margin (+8.7 mIoU over the best competitor), highlighting its strong generalization ability and annotation efficiency.
D Baselines and Ablation Studies
D.1 Evaluating the Impact of SAM2 on Prior SAM-Based Pipelines
We analyze the effect of replacing SAM with SAM2 in existing Segment Anything- based methods, namely Matcher [42], GF-SAM [83], and VRP-SAM [62]. These methods follow modular two-stage design: an external backbone (e.g., DINOV2 for Matcher and GF-SAM, ResNet-50 for VRP-SAM) is used perform feature matching, and the matched features are then used generate geometric prompts that are passed to SAM, which operates at the image level to predict the final segmentation mask. We re-evaluate these approaches by replacing SAM with SAM2 in the final segmentation step, while leaving the matching backbone unchanged. Note that all these methods adopt the largest version of SAM (SAM-H), hence replace it with the largest SAM2 version (SAM2-L). As shown in Tab. 7, the performance differences are negligible. This is consistent with findings from the SAM2 paper [54], which shows that SAM2-L yields only little gains over SAM-H in image-level promptable segmentation tasks. In other words, given the same prompts the segmentation quality is essentially unchanged, as both SAM and SAM2 excel converting prompt geometry into accurate masks.
Table 7: Performance comparison Segment Anything-base methods replacing SAM with SAM2. These methods prompt SAM at image-level, hence SAM2 additional capabilities (e.g. mask propagation) play no role this setting. As a result, SAM2 provides only minimal gains for existing two-stage methods (Matcher [42], GF-SAM [83], VRP-SAM [62]).
Feature Matching
Segmentation Model
mIoU (%)
Method
Backbone
SAM SAM2
23.0
PerSAM [87]
23.6
SAM
52.7
Matcher [42]
DINOv2
SAM2
52.4
SAM
53.9
VRP-SAM [62]
ResNet50
SAM2
54.3
SAM
58.7
GF-SAM [83]
DINOv2
SAM2
58.9
SANSA (ours)
SAM2
60.2
These findings highlight the fundamental difference between SANSA and previous SAM-based approaches. While prior methods rely on external backhones for matching and treat SAM as a image- level decoder, our architecture tightly integrates all components into a unified pipeline. By leveraging SAM2 not only for mask prediction but also for feature matching and prompt encoding, SANSA ensures more effective feature reuse, leading stronger performance and improved computational efficiency.
D.2 Experiments with Different Backbones
In this section, we evaluate the performance of different SAM2 Encoder variants at varying scales. Table 8 presents comparison of model parameters, inference speed (Frame Per Second), and mloU across three versions of the model (Tiny, Base, and Large), with Frame Per Second (img/s) values computed NVIDIA RTX 4090.
21

---

Table Comparison of model parameters, inference speed (FPS), and mloU for different scales of the SAM2 encoder (Tiny, Base, Large) using Hiera. FPS are measured on an NVIDIA RTX 4090.
SANSA
Total Parameters
Trainable Parameters
Frames Per Second
mloU (%)
Encoder
(img/s)
Tiny
40 M
1.3 M
50
51.1
Base
86 M
3.3 M
31
55.4
Large
234 M
10 M
15
60.2
All variants of SANSA offer a favorable trade-off between efficiency and accuracy, with consistent gains in mloU as model capacity increases. Notably, the Large model achieves the highest mloU of 60.2%, while still maintaining a practical inference speed FPS.
D.3 Effect of the Training Objective
We next study the impact of our sequential conditioning objective. When J = 1, training reduces to the standard few-shot segmentation objective, where each target is predicted only from the reference image, without propagation. In contrast, we propagate predictions across multiple unlabeled targets (J 1), encouraging the model to extract semantic structure that generalizes beyond individual pairs.
Table 9 shows that sequential conditioning consistently improves performance over the baseline with
J 1. We observe the largest gains with J 3-4, after which results plateau.
Table 9: Ablation training sequence length J. Using J > 1 improves the standard few-shot objective (J = 1). Gains saturate at J = 3-4, which we use as default.
J
COCO-20' (I-shot)
LVIS-92' (I-shot)
(no propagation)
57.0
44.7
2
58.9
47.2
(ours)
60.2
49.6
4
59.8
49.6
59.5
49.8
7
60.0
49.4
At inference, we emphasize that sequential conditioning is not used: each target is segmented inde- pendently given the annotated reference(s), following the standard few-shot segmentation protocol.
E
Additional Datasets and Qualitative Comparison
E.1
Qualitative Comparison with Prior Methods
Fig. 10 showcases the performance of SANSA compared to GF-SAM [83] and SegIC [44] on samples from the LVIS-92' benchmark. Each row visualizes a reference image with an annotated mask and the corresponding target image. We report predictions from the best-performing baselines (GF-SAM [83] and SegIC [44]), our approach (SANSA), and the ground truth segmentation. For clarity, name of the queried object class is also provided (note, however, that none of the models have access to the class label during inference).
These examples illustrate SANSA ability to resolve fine-grained distinctions and spatial part- awareness, addressing some of the core challenges of LVIS -222, where semantically close categories and partial object prompts are common. For instance, in the example where the reference depicts lamb, SANSA able distinguish from the semantically similar sheep. Similarly, it correctly seg- ments gazelle while avoiding confusion with adjacent zebras. In another case, SANSA successfully discriminates between a plush monkey and visually similar plush horse, despite both being soft toys with overlapping colors and textures. Moreover, SANSA exhibits part-level understanding. When prompted with a reference showing 1-shirt, GF-SAM oversegments the full person, while SegIC narrowly restricts to clothing regions. In contrast, SANSA accurately segments the t-shirt alone.
22

---

Class name
Reference
Target
GF-SAM
SegiC
SANSA
Ground Truth
Lamb
Gazelle
Monkey
Tongs
T-shirt
School bus
Figure 10: Qualitative comparison on the LVIS-92' benchmark. For each example, we show: the annotated reference, the target image, predictions from GF-SAM and SegIC, our SANSA prediction, and the ground truth. The queried class name is shown for clarity, but is not used by any model.
These qualitative results complement our quantitative findings, further supporting our hypothesis that SANSA bridges high-level category understanding with fine-grained spatial precision.
E.2 Additional Experiments: Pascal-5 and COCO Pascal
While main comparisons follow recent works [44, 89, 42] and focus the benchmarks of COCO-20', LVIS-92', and FSS-1000, we additionally report results on Pascal-5' for completeness.
Table 10: Evaluation on Pascal-5' and Pascal We report the mean Intersection-over- Union (mloU) for each fold and the average across folds. All methods are evaluated in a strict few-shot setting, including both the standard Pascal-5' benchmark and the distribution-shie setting (COCO-PPacal)) where models are trained on COCO-20* and tested on Pascal-5'. t indicates training-free methods.
Pascal-5
COCO Paseal
Method
fold0
fold1
fold2 fold3
fold0
fold1
fold2 fold3 mean
HSNet [46] (ICCV22
67.3
72.3
62.0
63.1
66.2
47.0
65.2
67.1
77.1
64.1
VAT [30] [BCCV122]
70.0
72.5
64.8
64.2
67.9
68.3
64.9
67.5
79.8
65.1
AMFormer [72] INIPS 231
71.3
76.7
70.7
63.9
70.7
AENet [74] [BCCV124]
72.2
75.5
68.5
63.1
69.8
HMNet [75] [NIPS:24]
72.2
75.4
70.0
63.9
70.4
PMNet [12] [WACV124]
71.3
72.4
66.9
61.9
68.1
71.0
72.3
66.6
63.8
68.4
Matcher [42] [[CLR224]
67.7
70.7
66.9
67.0
68.1
67.7
70.7
66.9
67.0
68.1
GF-SAM [83] [NIPS'24]
71.1
75.7
69.2
73.3
72.1
71.1
75.7
69.2
73.3
72.1
SANSA (ours)
78.1
80.0
68.3
66.1
73.1
67.1
70.9
73.5
78.6
72.5
23

---

These experiments are conducted in the strict few-shot setting and include both the standard evaluation on Pascal-5', and the distribution-shie -hift setting, COCO-+Pascal where models trained on COCO-20' are evaluated on Pascal-5' with folds repurposed to avoid any class overlap, as proposed originally in [8]. This setup aims to assess the robustness of models shifts in data distribution, condition often encountered in practical applications where training and deployment domains may differ. In Tab. 10, we compare with recent specialist few-shot methods (e.g., HMNet [75], AENet [74]) and training-fres ffree methods employing two foundation models in cascade (i.e. Matcher [42], GF-SAM [83]. For the latter, we report identical results in both evaluation settings, as these models are frozen and evaluated directly Pascal-5' without any training. The results confirm the competitive results of SANSA across benchmarks.
E.3 Promptable Segmentation: Prompt Generation Process and Qualitative Results
We denote the type of annotation as of E {mask, point, box, scribble]. Following [62, 90], point-based reference prompts are generated by randomly sampling between 1 and 20 points from the ground truth (GT). Scribble-based prompts are obtained, consistently with [62, 90], by using the free-form training mask generation algorithm proposed in [81], producing I to 20 scribbles per sample. Box- box-d prompts are derived by extracting object bounding boxes from the GT annotations.
In Fig. 11, we show the performance of SANSA in one-shot promptable segmentation. SANSA yields consistent and robust results across various prompts, including points, boxes, and scribbles.
Figure 11: Qualitative results for one-shot few-shot promptable segmentation. The figure illustrates segmentation results using different types of prompts. The first row shows examples where the reference object is annotated with a point, the second row uses bounding box annotations, and the third row employs scribble-based annotations. Examples are extracted from COCO-20'.
F Exploratory Analysis: Negative Prompts
Segment Anything supports the use negative prompts in interactive scenarios. These prompts, such as negative clicks on the image, allow users to correct mistakes by explicitly indicating regions that should not belong to the predicted mask. Since our method builds on SAM2, natural to ask whether this mechanism can be extended in the few-shot segmentation setting.
We conducted a small exploratory study investigate this idea. We consider two ways of extending negative prompting:
Geometric negatives: providing negative clicks on the target image at inference time, as in the interactive setting. This setup does not require any retraining and directly uses the mechanism already implemented in SAM2.
24

---

Semantic negatives: providing additional examples of visually similar but semantically different categories alongside the reference image, to serve as contrastive context (e.g. including a photo of a horse to help disambiguate the target class zebra). We experimented with two simple variants:
Training-free: the negative reference is added to the Memory Bank with an empty mask. Training-based small MLP maps the average feature of negative object into a learnable negative token, similar to SAM2 handling of negative clicks.
The results in Table 11 indicate that both geometric and semantic negative prompts can provide measurable benefits in few-shot segmentation. However, this should be regarded as an exploratory direction: in particular, semantic negative prompting currently lacks standardized baselines and evaluation protocols, but represents promising extension of the few-shot setting for future work.
Table 11: Exploratory results with negative prompts on COCO-20'. Geometric negatives act as test-time corrections. Semantic negatives provide contrastive context.
Method
mloU
SANSA (baseline)
60.2
Geometric Negative Prompts
+1 click
60.8
+3 clicks
62.9
+5 clicks
64.5
Semantic Negative Prompts
Training-fie
61.2
Training- based
63.0
F.1 Part Segmentation: Qualitative Results
In one-shot part segmentation (Fig. 12), SANSA demonstrates strong abilities retrieve fine- grained object parts, highlighting the effectiveness of our method in learning part-level representations.
Figure 12: Qualitative results for one-shot part segmentation. Examples from Pascal-Pait
G
Exploring Dependence on Spatial Continuity in SAM2
The SAM2 model targets is tracking in temporally coherent videos. However, its architecture imposes prior spatial continuity. As detailed in Eq. (3), masks are propagated across frames through
25

---

a gross-attenting between features, which allows all-to-all matching. Spatial coherence is thus not structurally enforced, but rather an emergent bias arising from pretraining on videos, encoded in the feature space. Our core finding is that this feature space contains latent semantic structure, entangled with spatial and appearance-related cues. Our adaptation disentangles these signals to isolate semantic information, enabling matching and segmentation across spatially independent inputs.
natural question is how this implicit spatial continuity bias affects SAM2 behavior in practice. To investigate this, we adopt the spatial coherence metric of [86], which computes the average first-order gradient of the displacement field induced by feature matching. Low values indicate smooth,
correspondences, while high values reflect noisy or fragmented matches. We evaluate three conditions with progressively weaker spatial relationships across image pairs:
(i) Consecutive video frames from DAVIS [51], where both temporal and spatial continuity are preserved.
(ii) Flipped DAVIS frames, where the second image is mirrored, disrupting spatial alignment while preserving object identity.
(iii) Independent images from TSS [64], with no spatial or temporal coherence.
As shown in Table 12, SAM2 performs well natural videos as expected, but its coherence degrades when spatial alignment perturbed and drops further on independent images. contrast, SANSA maintains stable coherence across all scenarios, showing its ability to abstract semantic correspondences without relying on spatial continuity.
Table 12: Spatial coherence across varying continuity conditions. Lower is better.
Continuity
Setting
SAM2 1
SANSA +
A Appearance
Spatial
(i) Video frames (DAVIS)
V
1.5
1.7
(ii) Flipped frames (DAVIS)
V
X
5.5
2.1
(iii) Independent images (TSS)
X
X
9.1
4.1
H Implementation and Reproducibility
H.1 Implementation details
We provide full implementation details to support reproducibility.
We adopt SANSA with Hiera-Large [58] as the visual encoder. Hiera is a hierarchical Transformer with channel dimensions [144, 288, 576, 1 152]. AdaptFormer [14] adapters are inserted into the last
two hierarchical blocks of the encoder (layers 9-48), corresponding to channel dimensions 1 1152 and 576. In the strict few-shot setting, the adapter hidden size is set to 0.3 the block channel dimension, yielding hidden sizes of 346 (for 1152) and 173 (for 576). This configuration introduces about 10M trainable parameters, while all SAM2 weights remain frozen. In the generalist setting, the adapter hidden size is increased to 0.8 x the block channel dimension, which expands the adapters and raises the total number of trainable parameters to about 25M Training in both settings is performed with Adam W (learning rate 10 10-4) and gradient clipping to 1.0. We train for 5 epochs in the strict setting and 20 epochs in the generalist setting, with training sequence length of J=3. Supervision combines Binary Cross Entropy loss and Dice loss, equally weighted (1.0), applied to the predicted segmentation mask and ground truth. We train with batch size 32 8 A100 GPUs.
For clarity, all relevant training and architecture details are summarized in Tab. 13.
H.2 Clarification of Few-Shot Segmentation Evaluation Settings
To avoid ambiguity in terminology and evaluation protocols, we first clarify the conventions followed in our work. In traditional few-shot segmentation literature [38, 69, 36, 41, 22], the annotated image typically referred to as the support image, while the unannotated image to be segmented is called the query image. However, more recent works [62, 42, 83] have shifted terminology, denoting the
26

---

Table 13: Summary of architecture and training hyperparameters.
Component
Value / Description
Base model
SAM2 (frozen)
Visual encoder
Hiera-Large [58]
Adapter type
AdaptFormer [14]
Adapted layers
Last two blocks (layers 9-48)
Adapter hidden size
0.3x (strict) / 0.8 (generalist) of block dim.
Trainable parameters
10M (strict) / ~-20 (generalist)
Optimizer
Adam
Learning rate
10-4
Gradient clipping
1.0
Batch size
32
Training epochs
(strict) / 20 (generalist)
Training sequence length
J = 3
Loss function
BCE (1.0) + Dice (1.0)
annotated image as the reference, and the image to be segmented as the target. We adopt this updated nomenclature throughout our work.
Beyond terminology, few-shot segmentation is commonly evaluated under two distinct settings: the strict few-shot setting and the more recent generalist in-context setting. We describe both below to clarify the assumptions and evaluation scope of our method.
Striet Few-Shot Segmentation. Early methods in few-shot segmentation [38, 69, 36, 41, 22, 68, 84, 47, 46, 30] adopt a meta-lcarning framework, where the goal is to segment novel object classes given only few annotated examples. Training proceeds episodically: each episode samples a subset of classes from a predefined training set, along with labeled reference images and corresponding unlabeled target images. The model must then predict segmentation masks for target images that contain instances of the same classes shown in the support set. A key assumption of this setup is the strict separation between training (seen) and evaluation (unseen) classes. This constraint ensures that performance reflects a model ability to generalize to entirely novel categories. To emphasize this evaluation protocol, setting has recently been referred [89, 44] as stricl few-shot segmentation.
Generalist in-context Setting, The notion of in-context learning was popularized in natural language processing by GPT-3 [9], where models perform new tasks by conditioning on sequence of input- output examples, without parameter updates. Rather than being explicitly trained for single task, these models learn to infer the intended behavior directly from the prompt structure, enabling flexible adaptation to wide range of downstream tasks. Inspired by this idea, recent vision models such as SegGPT [71] and Painter [70] reformulate segmentation as a conditional image generation problem. These models cast segmentation
type of image-to-image translation inpainting: prompts consist of paired inputs and output masks provided as images, and the model completes the target segmentation given these visual exemplars. This formulation allows generalization across broad set of tasks, such as semantic, instance, and part segmentation, without retraining.
Within few-shot segmentation literature, Matcher [42] builds on this paradigm and proposes the in-context setting, where the model is prompted with examples and conditioned to produce the desired output. This formulation enables single generalist model to address wide spectrum of segmentation tasks, including few-shot segmentation, few-shot part segmentation, and video object segmentation. Several recent methods, including DiffewS [89], SegIC [44], and our SANSA, adopt this generalist perspective. These approaches operate a more flexible evaluation regime, where prompts may include both seen and unseen classes, relaxing the disjoint class constraint of the strict setting.
H.3 Datasets
For strict few-shot segmentation, separate models are trained on COCO-20', LVTS-92 and FSS-1000 following their respective k-fold splits or fixed partitions. In contrast, the generalist in-context setting
27

---

involves training a single model under three progressively more comprehensive configurations: minimal setup with ADE20K and COCO; an extended setup including LVIS following SegIC [44]; and further extended configuration incorporating PACO, following 40), to mitigate object-level bias and improve part segmentation. Below, we provide brief descriptions of each dataset used in our experiments.
COCO-20' [48] is constructed from the MSCOCO dataset [39], containing 80 object categorics split into four disjoint folds. Each fold includes 60 training classes and 20 test classes. This dataset provides a challenging setting due to its large-scale nature and diverse object appearances.
LVIS-92', introduced by Matcher [42], is derived from the LVIS dataset [24] and presents more challenging few-shot segmentation benchmark, emphasizing long-tail distributions. consists of 920 classes that appear least two images, split into folds for cross-validation. This dataset introduces significant variations in object appearances and class distributions.
FSS-1000 [23] is large-scale few-shot segmentation dataset containing 1,000 classes with pixel-wise annotations, divided into 520 training, 240 validation, and 240 test classes. Unlike LVIS-92* and COCO-20', which use folds, FSS-1000 follows a fixed class partitioning.
PASCAL-Part, introduced by Matcher [42], is based on PASCAL VOC 2010 [21] and its part annotations released [15] and provides fine-grained object part annotations. It consists of four superclasses: animals, indoor objects, persons, and vehicles. The dataset contains 56 object parts across 15 categories. Given its focus on object parts rather than whole objects, this dataset presents a challenging segmentation task.
PACO-Part, derived from PACO, which provides part annotations for 75 object categories. Matcher [42] proposes k-fold split of the 456 object part classes, from which 303 classes with at least two samples are retained. The dataset split into four folds, each containing 76 object parts. Similar PASCAL-Part, PACO-Part is designed for evaluating one-shot part segmentation models, but includes larger and more diverse set of categories.
ADE20K [88] is large-scale semantic segmentation dataset containing 20,210 images annotated with 150 semantic categories at the pixel level. It includes diverse range of indoor and outdoor scenes, covering natural landscapes, urban environments, and everyday objects.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. We clearly state, shortly in the abstract, and more extensively in the introduction, which are our intended contributions and which is the scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We report Limitations section in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
we do not include novel theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, in the paper, and in the Appendix, we report information on dataset, training protocol and model to reproduce all experiments. Upon acceptance we will release code and trained models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our code and instructions are publicly released on GitHub. The data we use for the experiments are all from open-access datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cctpublic//eeeccccccccccccccecct mabbic/ggiids///// /Coo//eoeeeeeeeee wwtalubmisssionioo seiissolisiyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We clearly described the training and test details the experimental section and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Like the previous works we follow, we do not report error bars. The overhead of retraining with different random seeds is significant. If computational resources permit, we will supplement this in future works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In training details of appendix we detail our hardware setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:: /tttur:///eurpps cc hec//pulic//Bee c/Ethi itt.
Answer:
[Yes]
Justification:
We follow NeurlPS code of ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss the potential positive and negative impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper credited and their license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets at this time.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our work does not involve LLMs all.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /tt////erssp.///e/ cc/Conffeeecces/2225 /LLM) for what should or should be described.
35


Reviews:
[{"review_id": "KnBsTdNfyZ", "reviewer": "Reviewer_vipR", "summary": "This work aims to solve Few-shot semantic segmentation by leveraging SAM-2 built in feature matching matching process. It argues that SAM2 encodes rich semantic structure, however representations are entangled with object tracking cues and cannot be naively extended for few-shot semantic segmentation. It proposed SANSA framework that learns a simple adaptor on top of SAM features that repurposes SAM-2 for few-shot semantic segmentation with minimal samples. SANSA achieves state-of-the-art results on standard benchmarks and remains faster and more compact than existing approaches.", "questions": "I have two major questions - \n1. Do we really need masks? With SAM2 strong segmentation ability one doesnt need fully fine-grained mask for the few shot samples.\n2. Is the performance gain due to stronger SAM2? Prior works like VRP-SAM are based on older SAM and hence maybe limited in terms of performance.", "limitations": "One limitation can be - \n\nThe work only segments single object classes and maynot be straightforward to extend to multi object segmentation?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths - \n\n1. The idea of extending SAM-2 which is based on video object segmentation for few shot segmentation is interesting and novel. While it is not straightforward to extend SAM-2 for few shot segmentation, it does have inbuilt feature matching property which is exploited by this work.   \n \n2. The feature visualisation in Fig 4 showing better and separated semantic class features further improves the claim of better features with proposed approach. It even shows better part segmentation when the work performs k-means clustering the object features.   \n\n3. The work is well written and easy to follow. It achieves state of the art performance on standard benchmarks (tab 1) with complete ablations and also compares performance with in-context learning methods (tab 2).\n\n\nWeaknesses - \n\n1. Performance with just prompts/ Do we really need masks? - The proposed work uses segmentation masks for training in few shot manner. However SAM-2 can accept just prompts such as points and generate nice segmentation masks. How does the performance change when just prompts are provided in few shot samples (lets say just points instead of masks). Having full binary masks doesnt seem to be necessary.  \n \n2. Integration of negative prompt - Did the work also trying integration of negative prompts? The negative prompt might further improve representation quality for desired object and overall improve performance.  \n\n3. Performance gain due to stronger SAM2? Prior works like VRP-SAM, PerSAM are based on SAM (previous version of SAM-2) which supposedly is inferior. Its unclear if the performance gain is based on stronger SAM-2 or by the proposed method.   \n\n4. Single class segmentation? Is the work limited to segmenting a single class? Can it segment multiple class objects in a single image? This should be discussed in the work.", "was_revised": true, "final_justification": "Thank you so much for the rebuttal! It answers all my concerns. And I propose a score of 5. I really like the analysis of how SAM2's inherent feature matching can be extended to few shot segmentation. The work is well motivated and a nice read with sufficient experiments and ablations.", "rebuttal": "We thank the Reviewer for the thoughtful and positive feedback. We are glad that the proposed solution was found to be both **interesting** and **novel**, and that the **feature analysis**, **comprehensive ablations**, and **state-of-the-art results**, across standard benchmarks and in-context settings, were recognized as strengths of the work. We also sincerely value the constructive questions raised. In the following, we did our best to provide clarifications and additional analyses to address these points as clearly and thoroughly as possible.\n\n--------------\n\n1.  \n> **W1,Q1:** *\"Do we really need masks? With SAM2 strong segmentation ability one doesnt need fully fine-grained mask for the few shot samples.  How does the performance change when just prompts are provided in few shot samples (lets say just points instead of masks)?*\n\nWe thank the reviewer for this valuable question. We fully agree that reducing annotation effort is desirable in practical applications. Motivated by this, our main paper includes an evaluation of SANSA under different prompt types (*points*, *boxes*, and even *scribbles*), reported in **Figure 6 (top table), reproduced below**. Implementation details are provided in **§F.3 of the Supplementary**, along with extensive qualitative results in **Figure 5 of the Supplementary Material**.\n\nAs shown below, SANSA achieves **state-of-the-art performance across all prompt types**, significantly outperforming VRP-SAM, also targeting *promptable* few-shot segmentation, by **+15.0%** with points, **+5.8%** with scribbles, and **+4.6%** with boxes. Furthermore, SANSA exhibits strong robustness, with a performance drop of –6.7% from full-mask to point supervision, compared to –15.5% for VRP-SAM. \n\nWe also highlight that several **recent works** [35, 67, 71] built on SAM **cannot operate with point- or box-level prompts**, as they **rely on external encoders** (*e.g.*, DINOv2) for feature matching, and use SAM to produce the segmentation mask given the matched coordinate locations. VRP-SAM purposefully trains an additional decoder to support such prompts. In contrast, SANSA is the first to directly adapt the SAM2 feature space so that *the same features* can drive both feature matching and segmentation, enabling to **fully repurpose** SAM2 promptability for few-shot segmentation with low-effort prompts (L255-L264).\n\n\n**Table from Figure 6 (top table) of the Main paper**\n| Method        | Params | Point | Scribble | Box  | Mask |\n|---------------|--------|:-----:|:--------:|:----:|:----:|\n| VRP-SAM       | 670M   | 38.4  | 47.3     | 49.7 | 53.9 |\n| **SANSA**  | **234M**   | **53.4 (+15.0)** | **53.1 (+5.8)** | **54.3 (+4.6)** | **60.1 (+6.2)** |\n\n-----------------\n\n2.  \n> **W3,Q2:** *Is the performance gain due to stronger SAM2? Prior works like VRP-SAM are based on older SAM and hence maybe limited in terms of performance.*\n\nWe agree with the Reviewer on the importance of proper baselines. We address this point in **Table 4 of Section §E of the Supplementary Material, reproduced below**, where we isolate the effect of upgrading from SAM to SAM2 in recent Segment Anything-based methods. Recent state-of-the-art [21,20,42] models adopt a modular design: they use **external encoders for feature matching** (e.g. DINOv2), and treat **SAM as a promptable segmentation decoder** operating at the **image level**.\nAs shown in Table 1, replacing SAM with SAM2 results in minimal performance gains (< 2% mIoU). This is consistent with results reported in the original SAM2 paper [27], which was designed for *video* segmentation and reports similarly small improvements in promptable *image-level* segmentation.\n\nThese findings demonstrate that our performance improvements do not arise from simply using SAM2. Instead, the key novelty lies in **how SAM2 is used**: our method adapts the feature space, revealing a previously **unexplored semantic structure** that supports **both matching and segmentation** within a unified framework.\n\n\n**Table 4 from Section §E of the Supplementary Material**, † Newly added for rebuttal\n| Method          | Feature Matching Backbone | Segmentation Model | COCO-20i mIoU(%) | LVIS-92i  mIoU(%)  | \n|-----------------|----------------------------|--------------------|-------------------|-------------------|\n| PerSAM    | –                          | SAM / SAM2         | 23.0 / 23.6 †       | 11.5 / 13.2 † |\n| Matcher    | DINOv2                     | SAM / SAM2         | 52.7 / 52.4       |  33.0 / 33.6 † |\n| VRP-SAM    | ResNet-50                  | SAM / SAM2         | 53.9 / 54.3       |  28.3 / 28.9 † |\n| GF-SAM    | DINOv2                     | SAM / SAM2         | 58.7 / 58.9       |  35.2 / 37.0 † |\n| **SANSA (ours)**| –                          | **SAM2**            | **60.1**          |  **49.8** |\n\n------------------------------\n\n3.    \n> **W3, L1:** *Is the work limited to segmenting a single class?*\n\nWe thank the reviewer for raising this point which gives us the opportunity to further clarify the discussion provided in Section §A.1 of the Supplementary. Our model **can perform multi-class segmentation**. Indeed, SANSA builds upon SAM2, which already supports tracking and segmentation of multiple objects by storing multiple object embeddings in the Memory Bank. In the same way, our framework naturally handles multiple categories by storing separate reference representations, predicting each class mask independently, and aggregating them into a final segmentation map [27]. \n\nThat said, the current setup is **shared by all existing methods** [53, 35, 67], where categories are **segmented independently**, without leveraging potential relationships between them. In the Section §A.1 of the Supplementary, we highlight this as a **broader limitation of the current few-shot segmentation paradigm**. This paradigm is **effective for evaluating per-class generalization**; however, exploring how to incorporate cross-category reasoning could be a promising direction for future research.\n\n--------\n\n4. \n> **W2**: *Did the work also trying integration of negative prompts?*\n\nWe thank the reviewer for this insightful suggestion. While our main experiments do **not** include negative prompts, so **as to ensure a fair comparison** with prior works, we find this direction **highly compelling** and took the **opportunity to explore it further**. The key challenge lies in adapting the notion of *negative prompting*, originally designed for interactive segmentation, to the setting of few-shot segmentation.\n\nDelving into this analysis, a key distinction is that negative prompts in the SAM2 paper are geometric cues (*e.g.*, clicks) provided at test time to correct segmentation mistakes. Inspired by the reviewer suggestion, we propose to broaden the concept of negative prompting by introducing **semantic negative prompts**, namely, annotated examples of visually similar but semantically different categories, provided alongside the reference as contrastive context  (*e.g.*, including a photo of a *horse* to help disambiguate the target class *zebra*).\n\nWe thus explored two main strategies: geometric negative prompts (for test-time correction), and semantic negative prompts (for improved reference conditioning). Our experiments include both training-free and lightweight training-based variants, and are summarized below:\n\n- (Training Free) **Geometric Negative Prompts**: In this setting, negative clicks are provided at test time on the target image, mimicking interactive correction. Since SANSA reuses SAM2 prompt encoder, we can directly leverage SAM2 built-in negative point mechanism.\n- **Semantic Negative Prompts**: We then explore a novel interpretation of negative prompting in FSS, providing an annotated reference image of a semantically different (but potentially confusing) category. We implement it in two variants:\n    - *Training Free Negative Prompts*: We encode the negative reference into SAM2 memory bank, assigning it an empty mask. This signals to the model that this region contains no object matching the current task.\n    - *Training-based Negative Prompts*: We extend SAM2 prompt encoder with a small MLP that encodes the average feature of a negative object into a learnable negative prompt token, similar to SAM2 handling of negative clicks.\n\n\n**Table R1: Geometric vs. Semantic Negative Prompts**\n| Method                                 | mIoU |\n|----------------------------------------|:----:|\n| **SANSA (baseline)**                   | 60.1 |\n| **Geometric Negative Prompt**          |      |\n| &nbsp;&nbsp;&nbsp;&nbsp;• +1 click     | 60.8 |\n| &nbsp;&nbsp;&nbsp;&nbsp;• +3 clicks    | 62.9 |\n| &nbsp;&nbsp;&nbsp;&nbsp;• +5 clicks    | 64.5 |\n| **Semantic Negative Prompt**           |      |\n| &nbsp;&nbsp;&nbsp;&nbsp;• Training-Free Negative Prompts      | 61.2 |\n| &nbsp;&nbsp;&nbsp;&nbsp;• Training-Based Negative Prompts | 63.0 |\n\n\n\nWe believe this analysis to be highly interesting and potentially impactful. However, we emphasize that it opens a **broad set of questions**, particularly for semantic negative prompting, for which **standardized baselines are currently lacking**. We plan to explore these directions more rigorously in **future works**.\n\n---\n\nOnce again, we sincerely thank the Reviewer for their time and careful consideration. We hope our responses have provided the necessary clarifications and addressed all points raised."}, {"review_id": "ICykzWfunu", "reviewer": "Reviewer_6sqR", "summary": "This paper proposes a framework to unleash the hidden semantic capabilities of SAM2 for few-shot segmentation. A lightweight adapter module is applied after the image encoder to reinterpret the temporal dimension as semantically related features. The results set a new state-of-the-art in strict few-shot segmentation across multiple benchmarks.", "questions": "Instead of feature adaptation from the frozen original SAM2 weights, why not retrain a few-shot learning with the whole weights using the proposed architecture?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.The paper is well written with clear method illustration, feature analysis, and result presentations. \n2.The motivation to adapt original SAM2 for few-shot segmentation is interesting and technically sound. The paper shows a clear focus and boundary on few-shot learning with fair claims.\n3.The experiment design is comprehensive with promising results. \n\nWeaknesses:\nThe proposed framework is technically modified from SAM2. As SAM2 is designed for object tracking given videos with spatial relationships, there lacks a theoretical analysis and insights of the adaptation from continual frames to spatially independent pseudo-videos in the feature space. Therefore limits the methodological contribution is limited.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns in Weakness about lacking theoretical analysis and insights, as well as in Questions about the comparison to the method training from scratch. Therefore I raised the score to 5.", "rebuttal": "We thank the Reviewer for the positive and encouraging feedback. We are glad that the **motivation** to adapt SAM2 to few-shot segmentation was found to be both **interesting** and **technically sound**, and the experiments to be **comprehensive**. We also appreciate the positive remarks on the **clarity of the analysis and presentation**. We hope that the additional insights and empirical evidence provided below further strengthen the paper and address the remaining concerns.\n\n---\n\n1.  \n> **Q1:** *Instead of feature adaptation, why not retrain the model with the whole weights?*\n\nThis is an interesting point, which directly relates to a fundamental aspect of few-shot segmentation (FSS). The core challenge in FSS is **generalization**: the ability to transfer to novel, unseen categories from only a handful of labeled examples. For this reason, recent state-of-the-art FSS methods [35, 37, 67, 72] rely on **frozen** Vision Foundation Models (VFMs) such as DINOv2 and Stable Diffusion. While these models provide rich, transferable features acquired through large-scale pretraining, it has been shown that in low-shot settings [A, B, C] **retraining or fine-tuning** leads to **degrading the pre-trained representations** and ultimately poor generalization.\n\nWe explicitly analyze this point in **Section §C.3 of the Supplementary Material**, where we compare our *adaptation* with several *fine-tuning* strategies, including decoder-only, QKV-only, and full-backbone tuning. These strategies update between 4M and 210M parameters, yet all fall short of our proposed adaptation.\n\nIn response to the reviewer suggestion, we additionally **include two further baselines**: *i)* full fine-tuning of the entire model, starting from SAM2 pretrained weights; and *ii)* training the SAM2 architecture from ImageNet initialization. Both configurations yield substantially lower performance. This **confirms the theoretical expectation** that, in few-shot regimes, *i)* full fine-tuning degrades the generalization acquired during pretraining due to overfitting on limited categories, while *ii)* training from a generic initialization lacks the semantic priors necessary to generalize to novel classes.\n\n**Table 1 from Section §C.3 of the Supplementary Material**, † Newly added for rebuttal\n| Adaptation Strategy           | Train Params | COCO-20i mIoU (%) |\n|-------------------------------|------------|-------------------|\n| Training from Scratch†        | 224M       | 37.1              |\n| Full Finetuning†   | 224M       | 51.6              |\n| Decoder Finetuning           | 4M         | 52.1              |\n| QKV-only Finetuning          | 50M        | 55.3              |\n| Full Backbone Finetuning     | 210M       | 55.2              |\n| **SANSA (Ours)**             | **10M**    | **60.1**          |\n\n\n[A] \"Fine-tuning can cripple your foundation model; preserving features may be the solution.\" TMLR2024  \n[B] \"Fine-tuning can distort pretrained features and underperform out-of-distribution.\" ICLR2022  \n[C] \"Ranpac: Random projections and pre-trained models for continual learning.\" NeurIPS2023  \n\n---\n\n2.  \n> **W1**: As SAM2 is designed for object tracking given videos with spatial relationships, there lacks a theoretical analysis and insights of the adaptation from continual frames to spatially independent pseudo-videos in the feature space. \n\n\nWe thank the Reviewer for the insightful comment, which raises an important point about the adaptation of SAM2 to spatially independent inputs in the few-shot segmentation scenario.\n\nAlthough SAM2 is designed for tracking in temporally coherent videos, its architecture does **not impose any prior on spatial continuity**. In fact, as detailed in Eq. (2), masks are propagated through the video based on a cross-attention between frame features, that allows all-to-all matching. Therefore, spatial coherence is not structurally enforced, but rather an emergent **bias** that **arises from pretraining** on videos, encoded in the **feature space**. \nOur core finding is identifying, within such feature space, **the presence of latent semantic structure** (*cf* Linear probing, Section §C.2 Supp) entangled with spatial and appearance-related information (2D PCA, Section 3.4 Main). Our adaptation disentangles these signals to isolate the semantic information, enabling matching and segmentation across **spatially independent** inputs.\n\nWe agree that providing in-depth analyses and insights on the effect of this adaptation is important to ground our results. Indeed, **Section 3.4** of the main paper is devoted to study how this **adaptation reshapes the feature space**, by showing the principal component analysis of the feature space before and after the disentanglement. Please note that, due to space constraints, a more **thorough investigation** with additional insights and theoretical justifications is presented in the Supplementary (**Section  §C: Emergence of Semantic Representations**). \n\nWhile our analysis primarily focused on the **emergence of semantic representations**, we appreciate the Reviewer suggestion to further examine the **effect of adaptation** from **continual frames to spatially independent pseudo-videos** in the feature space. To this end, we have performed a new analysis adopting the **spatial coherence** metric from [70], which computes the average first-order gradient of the displacement field produced by matching: low values indicate smooth, object-consistent matches, while high values reflect noisy or fragmented correspondences. Through the lens of this metric, we investigate the extent to which SAM2 relies on spatial correlations in the data, and how our adaptation mitigates this dependence. We consider three settings, with progressively looser spatial relationships across image pairs:\n\n* i) **Video frames**: pairs of consecutive frames from DAVIS [D], where both temporal and spatial continuity are preserved, *e.g.*, a Labrador appears in the bottom left in the first frame and slightly to the right in the second.\n* ii) **Flipped frames**: same DAVIS frame pairs, but the second image is horizontally flipped, disrupting spatial alignment while preserving object identity, *e.g.* the Labrador appears in the same pose but mirrored, now on the opposite side.\n* iii) **Independent images**: pairs of unrelated images from the same category, sampled from TSS [E], with no spatial or temporal coherence, *e.g.* the first image shows a Labrador, and the second a Dalmatian.\n\n**Table R2 Spatial Coherence across varying continuity conditions**  \n*(Lower is better)*\n\n| Setting                           | Appearance Continuity | Spatial Continuity | SAM2 ↓ | SANSA ↓ |\n|----------------------------------|:-------------------:|:------------------:|:------:|:-------:|\n| *i)* Video frames (DAVIS)         | ✓                   | ✓                  | **1.5**| 1.7     |\n| *ii)* Flipped frames (DAVIS)      | ✓                   | ✗                  | 5.5    | **2.1** |\n| *iii)* Independent images (TSS)   | ✗                   | ✗                  | 9.1    | **4.1** |\n\n\nResults show that SAM2, as expected, performs well under *i)*, degrades under *ii)*, and struggles under *iii)*, confirming that its coherence partially relies on a continuity in the video frames. In contrast, SANSA maintains coherence across all scenarios, showing its **ability to abstract semantic** correspondences without relying on spatial continuity. \n\n[D] \"A benchmark dataset and evaluation methodology for video object segmentation.\" CVPR16  \n[E] \"Joint recovery of dense correspondence and cosegmentation in two images.\" CVPR16\n\n---\n\n\n\nWe sincerely thank the Reviewer for these insightful suggestions. We will incorporate the new analysis into Supplementary Section §C, which focuses on the theoretical study of the adapted feature space. We believe this addition offers a valuable complementary perspective and further deepens the conceptual understanding of our approach."}, {"review_id": "1Jdpp5s6mE", "reviewer": "Reviewer_eWJd", "summary": "This paper proposes SANSA, a method that adapts the frozen SAM2 model for few-shot segmentation by inserting AdaptFormer blocks into the last two layers of SAM2's image encoder. The authors argue that SAM2 features already contain semantic information that is entangled with low-level signals, and their adaptation method disentangles this to make semantic content explicit. The approach achieves strong performance on multiple benchmarks while preserving SAM2's original capabilities and requiring only 10M additional trainable parameters.", "questions": "During the rebuttal stage, please answer the following question: I could raise my score if the authors addressed my concerns.\n1. Could you provide systematic ablation studies comparing different adapter placement strategies (e.g., early layers, middle layers, all layers) to justify the \"last two layers\" choice?\n2. What insights can you offer about why AdaptFormer works explicitly better than other adaptation methods for disentangling semantic information in SAM2?\n3. How does your method fundamentally differ from standard fine-tuning approaches beyond the efficiency perspective?\n4. Could you demonstrate that the semantic disentanglement is specific to your approach rather than a general property of any adaptation of SAM2 features?", "limitations": "Yes, the authors acknowledge limitations in Section A.1, noting that their method inherits the design assumptions and limitations of SAM2.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The performance of the proposed method is quite good. It achieves state-of-the-art results on COCO-20i, LVIS-92i, and FSS-1000 with significant efficiency gains.\n2. The authors provide a thorough investigation of the semantic structure through PCA decomposition and linear probing experiments.\n3. This paper also provides detailed feature visualization and analysis, which clarifies the overall contribution of the paper.\n\nWeaknesses:\n1. Novelty is rather low. The core contribution is essentially applying AdaptFormer (an existing technique) to SAM2 (an existing model), which is a relatively straightforward engineering solution without significant algorithmic novelty. The author should more clearly claim their main contributions.\n2. The manuscript occasionally suffers from dense prose, unclear transitions (e.g., Sections 3.1 and 3.2), and long paragraphs. This makes it challenging for readers less familiar with SAM2 or FSS to follow.\n3. The paper lacks thorough ablation studies on adapter placement strategies, architectures, or hyperparameters. The choice of \"last two layers\" appears somewhat arbitrary without systematic justification. Also, there is limited theoretical insight into why this specific adaptation strategy works or how it fundamentally differs from other fine-tuning approaches beyond empirical observations", "was_revised": true, "final_justification": "All my concerns were solved after reading the reviewers' replies. I'm grateful for the authors' reclaim, and I understand the paper better now, so I will raise my rating to 4. After reading the final remarks, I finally raised my rating to 5 and appreciate the quality of this paper.", "rebuttal": "We thank the Reviewer for the constructive feedback. We are glad to see that the **strong performance**, **efficiency gains**, and our **in-depth analysis**, including PCA decomposition, linear probing, and feature visualizations, were appreciated. In the following, we address the concerns raised with new experiments, expanded theoretical insights, and a clearer articulation of our contributions. We hope these additions clarify the novelty and significance of our work.\n\n---\n1.\n> **W1:** *The core contribution is essentially applying AdaptFormer to SAM2: the authors should more clearly claim their main contributions.*\n\nWe thank the Reviewer for the opportunity to clarify the core contributions of our work. The novelty lies in a key insight: SAM2, though trained for class-agnostic object tracking, encodes a latent semantic structure (L51-L56) that can be disentangled and repurposed for few-shot segmentation (FSS).\n- From a **scientific** standpoint, we are the **first** to **identify**, **analyze**, and **validate** the presence of **semantic structure in SAM2 features**. We show in Supp. §C.2 that a simple linear probe on frozen features recovers class-level information, providing evidence that semantics are already embedded and do **not need to be learned**. Based on this, we propose a new perspective on few-shot segmentation: instead of relying on separate models (DINOv2 for **matching** and SAM for **segmentation**), we show that SAM2 alone can perform **both**, enabling a unified and streamlined solution. More broadly, this **repositions SAM2** as a standalone foundation for a wider class of pixel-level semantic tasks. \n- From a **technical** standpoint, we *i)* repurpose SAM2 memory mechanism by feeding it pseudo-videos, reinterpreting the temporal dimension as a collection of semantically related images. We *ii)* introduce a novel training protocol that promotes concept propagation across *multiple* unlabeled frames, encouraging the emergence of robust semantic correspondence. We *iii)* adapt the feature space to shift from low-level visual similarity to class-level alignment. Notably, the **architectural simplicity of our design is a consequence of our hypothesis**: if semantic structure had to be introduced through architectural modifications, it would undermine our claim that such structure already exists in SAM2. \n- From a **practical** standpoint, SANSA achieves SOTA results across FSS benchmarks, in-context learning scenarios, and part segmentation tasks. It does so while being 3× faster and up to 5× smaller than prior methods. Crucially, our model supports low-effort prompts (*e.g.*, points), making it well-suited for real-world applications such as data annotation.\n\nWe will revise the manuscript to better frame and position our contributions.\n\n---\n2.\n> **Q2:** *Why does AdaptFormer work explicitly better than other adaptation methods? **Q4:** Is the semantic disentanglement specific to your approach rather or a general property of any adaptation of SAM2 features?*\n\nBuilding on the previous point, our results are **not meant to imply that semantic disentanglement arises specifically because of AdaptFormer**. \nWe emphasize that our goal is **not** to engineer a dedicated adapter (L56-L60), but to investigate the representational structure in SAM2: **if semantic disentanglement depended on the design of the adapter, it would suggest that the semantics are induced by adaptation,** rather than being already embedded in the pretrained model.\n\nIn this context, we detail what makes an adaptation strategy effective, expanding on results reported in the paper with additional experiments in Tab. R3.\n- As shown in Tab. 4 (main paper), basic adaptation strategies (*i.e.* LoRA, Adapter, AdaptFormer) yield similarly **strong improvements** of +25% w.r.t. frozen SAM2 (32.2%).\n- Importantly, LoRA is a linear adapter. The slight gains (~2%) observed with Adapter and AdaptFormer, both of which introduce a single ReLU non-linearity, suggest that simple **non-linear transformations** can **help** refine this structure, but do **not fundamentally reshape it**.\n- In Tab. 5, we study the effect of **adapter capacity**. Reducing the bottleneck dimension from 768 → 576 → 384 leads to progressively better generalization (58.2% → 59.6% → 60.1%). This indicates that **smaller adapters** are **better** at capturing the relevant structure in the pretrained features, **avoiding overfitting** to spurious correlations.\n- To further test this hypothesis, we **introduce a new experiment** with a recent SOTA adapter, **MONA** [A], which include depthwise convolutions and LayerNorm to increase functional complexity. Despite this, MONA underperforms (56.9%), suggesting that added complexity leads to overfitting rather than better representations.\n\nIn summary, **semantic disentanglement is not unique to AdaptFormer, but also does not arise from arbitrary adaptation**. Our results suggest that effective adaptation requires **simplicity and constraint**: low-capacity, bottlenecked modules are best suited to uncovering SAM2 latent semantics. In contrast, more expressive adapters tend to overfit.\n\n**Table R3 Finetune vs Adaptation.** † Newly added for rebuttal\n|||Train Params (M)|mIoU (%)|\n|-|-|-|-|\n|**Frozen**|-|0|32.2|\n|**Finetuning (FT)**|Train from Scratch †|224|37.1|\n||Full FT †|224|51.6|\n||Decoder FT|4|52.1|\n||QKV-only FT|50|55.3|\n||Backbone FT|210|55.2|\n|**Adaptation**|LoRA|18|58.0|\n||Adapter|20|59.4|\n||Mona†|26|56.9|\n||AdaptFormer (**SANSA**)|**10**|**60.1**|\n||├─ Bottleneck: 768,576,384|26,16,10|58.2, 59.6, **60.1**|\n\n[A] \"5%> 100%: Breaking performance shackles of full fine-tuning on visual recognition tasks.\" CVPR2025.\n\n---\n3.\n> **Q3:** *How does SANSA differ from standard fine-tuning approaches beyond the efficiency perspective?* **W3:** *Theoretical insight into why adaptation strategy works or how it differs from other fine-tuning?*\n\nOur choice to keep SAM2 frozen is **not motivated by efficiency**, but by the need for **generalization**, which is the core challenge in few-shot segmentation. All recent state-of-the-art methods [35, 67, 72, 37] do *not* finetune  pretrained Vision Foundation Models to avoid overfitting when learning from limited data. SANSA follows this paradigm. As shown in the **Tab. 1 of the Supp.** (Sec. §C.3) and in **Tab. R3 above**, full or partial fine-tuning of SAM2 consistently results in worse generalization. We further added two new experiments (training from scratch and full fine-tuning), which confirm the same trend. \n\nFrom a **representation learning perspective**, SAM2 encodes rich, reusable features that capture both appearance and **latent semantics**. Lightweight bottleneck adapters such as AdaptFormer or LoRA apply **low-rank transformations** that reorient the embedding space toward task-relevant semantics while **preserving** the underlying pretrained structure. This can be viewed as a form of semantic disentanglement under a strong inductive prior, *i.e.* the pretrained model [B, C]. In contrast, directly fine-tuning the model parameters risks erasing these priors, particularly in few-shot or continual learning settings, where **data is limited and distributions shift at test time** [C, D].\nOur empirical findings (Tab. R3: *Finetune vs Adaptation*) align with this interpretation: adaptation successfully leverages and restructure existing structure, thus promoting generalization.\n\n[B] \"Fine-tuning can cripple your foundation model; preserving features may be the solution.\" TMLR2024  \n[C] \"Fine-tuning can distort pretrained features and underperform out-of-distribution.\" ICLR2022  \n[D] \"Ranpac: Random projections and pre-trained models for continual learning.\" NeurIPS2023\n\n---\n4. \n>  **Q1, W3**: *The paper lacks ablations on adapter placement (to justify the \"last two layers\" choice), architectures, or hyperparameters.*\n\nWe thank the reviewer for raising this important point. While prior works typically insert adapters throughout the **entire** network to maximize representational capacity [E,F,G,H], we purposefully limit adaptation to the last 2 hierarchical blocks of the backbone, which are known to encode **high-level semantics**. This design choice stems from **our core objective**: **disentangle semantic structure**. In response to the reviewer suggestion, we conducted a systematic ablation on adapter placement (Tab. R4) to better justify our design choice.\n\n**Table R4. Adapter Placement**  \n|Placement|Stages|mIoU (%)|\n|-|-|-|\n|None|–|32.2|\n|All|0,1,2,3|59.4|\n|Early|0,1|38.7|\n|Middle|1,2|57.9|\n|Late|2,3|**60.1**|\n\nThe ablation confirms that, for **semantic disentanglement**, it is beneficial to adapt the latest stages of the backbone.\n\nThese results complement our ablations on the core design elements, including **Adapter architecture** (Tab. 4, main paper; expanded in Tab. R3), **Adapter capacity** (Tab. 5, main paper), **Adaptation vs. fine-tuning** (Tab. 1, Supp; expanded in Tab. R3). We hope the additional experiment fully addresses the reviewer concern. \n\n[E] \"Side Adapter Network for Open-Vocabulary Semantic Segmentation.\" CVPR2023  \n[F] \"Exploring Temporally-Aware Features for Point Tracking.\" CVPR2025  \n[G] \"AdapterFusion: Non-Destructive Task Composition for Transfer Learning.\" EACL2021  \n[H] \"Vision Transformer Adapter for Dense Predictions.\" ICLR2023  \n\n---\n5. \n> **W2:** *Dense prose; challenging for readers less familiar with SAM2 or FSS.*\n \nWe sincerely thank the reviewer for this helpful observation. We recognize that clarity is essential, particularly when introducing concepts that may be unfamiliar to all readers. We will revise the manuscript in Sections 3.1 and 3.2 by smoothing transitions and simplifying technical descriptions where needed. While other reviewers found the presentation to be clear, we appreciate that individual perspectives vary, and we are committed to ensuring that the final version is accessible to a broader audience through careful revision."}, {"review_id": "Rykl39JkmH", "reviewer": "Reviewer_Cxfs", "summary": "This paper repurposes the SAM2 object tracking framework for the Few-Shot Segmentation (FSS) task. FSS requires the model to segment objects in a query image based on feature matching with a given support image. While SAM2 is inherently capable of instance-level object matching and tracking, it lacks the semantic-level generalization needed for FSS. To address this, the authors introduce an adaptor module based on AdaptFormer, which enables the model to adapt to class-level segmentation tasks. The training objective is designed around support and query sets, and predicts the mask conditioned on its previous prediction. This setup enables more coherent and context-aware segmentation results.", "questions": "Please refer to the Weaknesses part.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is clearly written and easy to follow.\n- The motivation for adapting the SAM2 framework to Few-Shot Segmentation (FSS) is well presented, with an analysis of SAM2’s capabilities. Also, exploiting SAM2 enables segmentation from diverse input prompts.\n- By using an adaptor, the method preserves the original object tracking performance of SAM2 while extending it effectively to FSS.\n- The proposed method outperforms prior approaches in both accuracy and efficiency. Design choices are well supported through ablation studies.\n\n**Weaknesses**\n- Figure 1 lacks an explanation for the qualitative results shown in the bottom part. It is unclear whether these are SAM2's zero-shot outputs or ground-truth segmentations. If they are SAM2 zero-shot results, the rightmost example may be an inappropriate illustration, as performance degradation is expected in such cases.\n- In Figure 2, it is not explained whether the colors carry semantic meaning. If they do, adding a colorbar would improve clarity. For instance, in the right example, SAM2's output does not appear significantly worse, so clarification would help contextualize the comparison.\n- The paper lacks details regarding the training objective part. It is unclear what specific values or configurations were used for hyperparameters such as r and J. Do these values vary depending on whether the model is trained under the 1-shot or 5-shot setting?\n- The construction of the training image set is only briefly mentioned. Was the sampling purely random based on class labels, or were there additional criteria? And, is the order of images within each training set shuffled during training? Given that object size and the number of objects can vary significantly across images, this might impact learning stability and generalization.\n- Relatedly, since the model conditions each query prediction on previous predictions, it is unclear how robust the method is to poor initial masks. If the initial prediction is inaccurate, is there a risk of cascading errors or convergence to suboptimal solutions (e.g., local minima)? An analysis or discussion of such failure cases would strengthen the paper.", "was_revised": true, "final_justification": "The authors have successfully addressed all of my concerns. While the application is limited to few-shot segmentation, the paper presents a clear motivation that is well aligned with the proposed method. Overall, I believe this is a solid and well-executed work. Therefore I raise my score to 5.", "rebuttal": "We sincerely thank the Reviewer for the positive and constructive feedback. We are encouraged to see appreciation for the **clarity** of the paper, the **motivation** behind adapting SAM2 to the few-shot segmentation setting, the **flexibility** to support diverse prompts, and the strong **results** achieved in both **performance and efficiency**.\nBelow, we address the remaining concerns in detail, providing clarifications and additional analyses. We hope these additions further strengthen the paper and improve its presentation.\n\n----------\n1. \n\n> **W1:** *Unclear whether qualitative results shown in the bottom of Figure 1 are SAM2 zero-shot outputs or ground-truth segmentations.*\n\nWe appreciate the Reviewer observation and clarify that the bottom part of Figure 1 shows **ground-truth masks**. These examples are meant to visually contextualize the two types of scenarios discussed in our toy experiment: *low semantic shift / high visual similarity* (e.g., medical scans, left) and *high semantic shift / low visual similarity* (e.g., cruise ship vs. rowboat, right). The red masks simply indicate the object of interest. We will revise the caption accordingly.\n\nMore broadly, the right part of the figure motivates our core research question: *Can the semantic capabilities observed in other vision foundation models (VFMs) also be found in SAM2?* A growing line of research [21, 37, 67, 69, 70] has demonstrated that semantic structure is encoded in the features of **frozen** VFMs such as DINOv2 and Stable Diffusion. We conducted this toy experiment to **test** whether **similar semantic properties** could be observed in the frozen SAM2. The aim of Figure 1 is thus to set the stage for the rest of the paper: to highlight that, **unlike in other VFMs, semantic structure in SAM2 does not naturally emerge in zero-shot**.\n\n------------\n2. \n\n> **W2:** *In Figure 2 (PCA), it is not explained whether the colors carry semantic meaning.*\n\nWe thank the Reviewer for the helpful observation and agree that clarifying the caption would improve interpretability. \nIn Figure 2 we apply Principal Component Analysis (PCA) to the extracted feature vectors and map the first three principal components (PC1, PC2, PC3) to the red, green, and blue channels, respectively. Consequently, each point RGB color encodes its position in the 3D PCA-projected space. This approach is widely used to **analyze the global structure** of high-dimensional embeddings [43, 54, 70].\nImportantly, the colors do *not* correspond to ground-truth class labels and should not be interpreted categorically. Rather, they reflect spatial proximity in the projected space: points with similar colors lie close in the 3D PCA space. This allows us to **qualitatively assess what types of variation are captured by the leading axes**. In the case of frozen SAM2, although some hue variation exists (*e.g.*, surfboards appear slightly violet and people greenish-blue), the overall color distribution lacks clear separation across the four object categories (*dog, frisbee, person, surfboard*). This suggests that the principal components capture a mixture of low-level or instance-specific signals, rather than pure semantics. In contrast, **SANSA** yields clearly distinct and consistent colors (*e.g.*, orange, yellow, violet, light blue), indicating that **semantic structure dominates the main axes of variability.**\n\nA more comprehensive evaluation is reported in **Figure 4a**, where we use a 2D PCA projection with ground-truth color labels to visualize **class-level clusters**.\n\n---\n3.\n \n> **W3:** *Details regarding the training: specific values for hyperparameters such as r and J. Do these values depends on whether the model is trained under the 1-shot or 5-shot setting?*\n\nWe thank the reviewer for the comment and are happy to clarify. We use a *training sequence* length of **J=3**, meaning each episode consists of one annotated reference followed by three unlabeled target image. The number of annotated references during *training* is fixed to **r=1**. At training, these values **do not change**. At *inference time*, we simply provide either one (r=1) or five (r=5) annotated examples based on the benchmark protocol (as in Table 1 of main paper). This unified training strategy enables **seamless support for variable r-shot inference** (*e.g.*, 1-shot or 5-shot), without retraining or architectural modifications.\n\nDetails on the training configuration, including the fixed value J=3, are included in **Section §G.1 Implementation Details of the Supplementary Material**. We appreciate the suggestion and will revise the text to also clearly state that r=1 is used during training.\n\n----------------------------------------------\n\n4. \n\n> **W5:** *If the initial prediction is inaccurate, is there a risk of cascading errors or convergence to suboptimal solutions (e.g., local minima)?*\n\nWe appreciate the reviewer thoughtful question. The sequential conditioning mechanism is used **exclusively during training to promote robust and generalizable representations**. By requiring the model to propagate semantic information across multiple unlabeled targets, this objective discourages overfitting to individual image pairs and encourages the emergence of shared, semantic structure, **forcing the network to disentangle high-level concepts from low-level visual cues**. \n\n\nTo directly address the concern about potential cascading errors or instability from inaccurate early predictions, we perform an ablation on the training sequence length J, which defines how many unlabeled target frames are segmented after the annotated reference. When J=1, no sequential conditioning or mask propagation occurs. If our formulation were prone to cascading errors, we would expect performance to degrade for larger J. Instead, we observe that all settings with J>1 consistently outperform the baseline with J=1, **demonstrating the stability and benefits of our training strategy**.\n\n**Table R5: Ablation on Mask Propagation**, † Newly added for rebuttal.\n| \\( J \\) | COCO-20i (1-shot) | LVIS-92i (1-shot) |\n|--------|-------------------|-------------------|\n| 1 (no mask propagation)      | 57.0             | 44.7             |\n| 2      | 58.9 †            | 47.2 †            |\n| 3      | **60.1**         | **49.8**        |\n| 4      | 59.8 †            | 49.6 †            |\n| 5      | 59.5 †            | 49.8 †            |\n| 7      | 60.0 †            | 49.4 †            |\n| 10      | 59.9 †            | 49.5 †            |\n\nThese results provide strong empirical evidence that our sequential training formulation is **not vulnerable to cascading errors**, but instead enhances the quality and generality of the learned features.\n\nLastly, we clarify that this sequential conditioning mechanism is used only during training. At **inference time**, our method follows the standard few-shot segmentation protocol: each target image is segmented independently, using only the annotated reference(s), without mask propagation or reliance on previous predictions. \nWe thank the reviewer for this insightful observation and will incorporate the ablation and its analysis in the revised manuscript.\n\n\n-------------------------\n\n5. \n\n> **W4:** *The construction of the training image set is only briefly mentioned.*\n\nWe appreciate the Reviewer suggestion to clarify the training data construction process. We followed the **standard protocol adopted in few-shot segmentation literature** [16, 17, 20, 39, 40]. For each training episode, a class is randomly selected from the training set, and images containing at least one instance of that class are sampled uniformly at random. No filtering is applied based on object size, instance count, or other properties. The order of images within episodes is shuffled during training.\nSince no special constraints are introduced, we did not emphasize this aspect in the main paper. However, we agree that adding this clarification can improve completeness, and will revise the manuscript accordingly.\n\n---\n\nWe once again thank the Reviewer for the time dedicated to evaluating our work and for the constructive questions and observations raised. We hope our responses have clarified all points and helped improve the overall quality of the paper."}]

Author rebuttal:

Meta-review: This paper exploits the latent semantic structure within SAM2 for few-shot segmentation. 

While the reviewers like the idea and results in general, reviewers raised concerns about novelty, theoretical depth, and fairness of comparisons. In particular
- Reviewer eWJd felt the contribution was incremental (applying AdaptFormer to SAM2), and requested more systematic ablations (adapter placement, hyperparameters)
- Reviewer 6sqR commented that there lack of a theoretical analysis about the adaptation of SAM2 to spatially independent inputs
- Reviewer vipR questioned whether performance gains were mainly due to stronger SAM2 compared to prior SAM-based methods 

The rebuttal convincingly addressed these through additional experiments and clarifications. Why not higher scores: 
- Even after the rebuttal clarified the novelty (semantic structure hidden in SAM2), the perception that this was more an adaptation than a fundamentally new algorithm limited enthusiasm.

Reviewers converged on 5 as the appropriate score. The consensus is that SANSA is a valuable and timely contribution that will be of interest to both the segmentation and foundation model communities.
Decision: Accept (spotlight)