Title: SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Few shot segmentation

Segment Anything

Promptable Segmentation
Abstract: Few-shot segmentation aims to segment unseen categories from just a handful of annotated examples. This requires mechanisms to identify semantically related objects across images and accurately produce masks. We note that Segment Anything 2 (SAM2), with its prompt-and-propagate mechanism, provides strong segmentation capabilities and a built-in feature matching process. However, we show that its representations are entangled with task-specific cues optimized for object tracking, which impairs its use for tasks requiring higher level semantic understanding. Our key insight is that, despite its class-agnostic pretraining, SAM2 already encodes rich semantic structure in its features. We propose SANSA (Semantically AligNed Segment Anything 2), a framework that makes this latent structure explicit, and repurposes SAM2 for few-shot segmentation through minimal task-specific modifications. SANSA achieves state-of-the-art on few-shot segmentation benchmarks designed to assess generalization and outperforms generalist methods in the popular in-context setting. Additionally, it supports  flexible promptable interaction via points, boxes, or scribbles, and remains significantly faster and more compact than prior approaches.  Code at: https://github.com/ClaudiaCuttano/SANSA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper repurposes the SAM2 object tracking framework for the Few-Shot Segmentation (FSS) task. FSS requires the model to segment objects in a query image based on feature matching with a given support image. While SAM2 is inherently capable of instance-level object matching and tracking, it lacks the semantic-level generalization needed for FSS. To address this, the authors introduce an adaptor module based on AdaptFormer, which enables the model to adapt to class-level segmentation tasks. The training objective is designed around support and query sets, and predicts the mask conditioned on its previous prediction. This setup enables more coherent and context-aware segmentation results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is clearly written and easy to follow.\n- The motivation for adapting the SAM2 framework to Few-Shot Segmentation (FSS) is well presented, with an analysis of SAM2’s capabilities. Also, exploiting SAM2 enables segmentation from diverse input prompts.\n- By using an adaptor, the method preserves the original object tracking performance of SAM2 while extending it effectively to FSS.\n- The proposed method outperforms prior approaches in both accuracy and efficiency. Design choices are well supported through ablation studies.\n\n**Weaknesses**\n- Figure 1 lacks an explanation for the qualitative results shown in the bottom part. It is unclear whether these are SAM2's zero-shot outputs or ground-truth segmentations. If they are SAM2 zero-shot results, the rightmost example may be an inappropriate illustration, as performance degradation is expected in such cases.\n- In Figure 2, it is not explained whether the colors carry semantic meaning. If they do, adding a colorbar would improve clarity. For instance, in the right example, SAM2's output does not appear significantly worse, so clarification would help contextualize the comparison.\n- The paper lacks details regarding the training objective part. It is unclear what specific values or configurations were used for hyperparameters such as r and J. Do these values vary depending on whether the model is trained under the 1-shot or 5-shot setting?\n- The construction of the training image set is only briefly mentioned. Was the sampling purely random based on class labels, or were there additional criteria? And, is the order of images within each training set shuffled during training? Given that object size and the number of objects can vary significantly across images, this might impact learning stability and generalization.\n- Relatedly, since the model conditions each query prediction on previous predictions, it is unclear how robust the method is to poor initial masks. If the initial prediction is inaccurate, is there a risk of cascading errors or convergence to suboptimal solutions (e.g., local minima)? An analysis or discussion of such failure cases would strengthen the paper.\n\nQUESTIONS:\nPlease refer to the Weaknesses part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SANSA, a method that adapts the frozen SAM2 model for few-shot segmentation by inserting AdaptFormer blocks into the last two layers of SAM2's image encoder. The authors argue that SAM2 features already contain semantic information that is entangled with low-level signals, and their adaptation method disentangles this to make semantic content explicit. The approach achieves strong performance on multiple benchmarks while preserving SAM2's original capabilities and requiring only 10M additional trainable parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The performance of the proposed method is quite good. It achieves state-of-the-art results on COCO-20i, LVIS-92i, and FSS-1000 with significant efficiency gains.\n2. The authors provide a thorough investigation of the semantic structure through PCA decomposition and linear probing experiments.\n3. This paper also provides detailed feature visualization and analysis, which clarifies the overall contribution of the paper.\n\nWeaknesses:\n1. Novelty is rather low. The core contribution is essentially applying AdaptFormer (an existing technique) to SAM2 (an existing model), which is a relatively straightforward engineering solution without significant algorithmic novelty. The author should more clearly claim their main contributions.\n2. The manuscript occasionally suffers from dense prose, unclear transitions (e.g., Sections 3.1 and 3.2), and long paragraphs. This makes it challenging for readers less familiar with SAM2 or FSS to follow.\n3. The paper lacks thorough ablation studies on adapter placement strategies, architectures, or hyperparameters. The choice of \"last two layers\" appears somewhat arbitrary without systematic justification. Also, there is limited theoretical insight into why this specific adaptation strategy works or how it fundamentally differs from other fine-tuning approaches beyond empirical observations\n\nQUESTIONS:\nDuring the rebuttal stage, please answer the following question: I could raise my score if the authors addressed my concerns.\n1. Could you provide systematic ablation studies comparing different adapter placement strategies (e.g., early layers, middle layers, all layers) to justify the \"last two layers\" choice?\n2. What insights can you offer about why AdaptFormer works explicitly better than other adaptation methods for disentangling semantic information in SAM2?\n3. How does your method fundamentally differ from standard fine-tuning approaches beyond the efficiency perspective?\n4. Could you demonstrate that the semantic disentanglement is specific to your approach rather than a general property of any adaptation of SAM2 features?\n\nLIMITATIONS:\nYes, the authors acknowledge limitations in Section A.1, noting that their method inherits the design assumptions and limitations of SAM2.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes a framework to unleash the hidden semantic capabilities of SAM2 for few-shot segmentation. A lightweight adapter module is applied after the image encoder to reinterpret the temporal dimension as semantically related features. The results set a new state-of-the-art in strict few-shot segmentation across multiple benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
1.The paper is well written with clear method illustration, feature analysis, and result presentations. 
2.The motivation to adapt original SAM2 for few-shot segmentation is interesting and technically sound. The paper shows a clear focus and boundary on few-shot learning with fair claims.
3.The experiment design is comprehensive with promising results. 

Weaknesses:
The proposed framework is technically modified from SAM2. As SAM2 is designed for object tracking given videos with spatial relationships, there lacks a theoretical analysis and insights of the adaptation from continual frames to spatially independent pseudo-videos in the feature space. Therefore limits the methodological contribution is limited.

QUESTIONS:
Instead of feature adaptation from the frozen original SAM2 weights, why not retrain a few-shot learning with the whole weights using the proposed architecture?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to solve Few-shot semantic segmentation by leveraging SAM-2 built in feature matching matching process. It argues that SAM2 encodes rich semantic structure, however representations are entangled with object tracking cues and cannot be naively extended for few-shot semantic segmentation. It proposed SANSA framework that learns a simple adaptor on top of SAM features that repurposes SAM-2 for few-shot semantic segmentation with minimal samples. SANSA achieves state-of-the-art results on standard benchmarks and remains faster and more compact than existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths - \n\n1. The idea of extending SAM-2 which is based on video object segmentation for few shot segmentation is interesting and novel. While it is not straightforward to extend SAM-2 for few shot segmentation, it does have inbuilt feature matching property which is exploited by this work.   \n \n2. The feature visualisation in Fig 4 showing better and separated semantic class features further improves the claim of better features with proposed approach. It even shows better part segmentation when the work performs k-means clustering the object features.   \n\n3. The work is well written and easy to follow. It achieves state of the art performance on standard benchmarks (tab 1) with complete ablations and also compares performance with in-context learning methods (tab 2).\n\n\nWeaknesses - \n\n1. Performance with just prompts/ Do we really need masks? - The proposed work uses segmentation masks for training in few shot manner. However SAM-2 can accept just prompts such as points and generate nice segmentation masks. How does the performance change when just prompts are provided in few shot samples (lets say just points instead of masks). Having full binary masks doesnt seem to be necessary.  \n \n2. Integration of negative prompt - Did the work also trying integration of negative prompts? The negative prompt might further improve representation quality for desired object and overall improve performance.  \n\n3. Performance gain due to stronger SAM2? Prior works like VRP-SAM, PerSAM are based on SAM (previous version of SAM-2) which supposedly is inferior. Its unclear if the performance gain is based on stronger SAM-2 or by the proposed method.   \n\n4. Single class segmentation? Is the work limited to segmenting a single class? Can it segment multiple class objects in a single image? This should be discussed in the work.\n\nQUESTIONS:\nI have two major questions - \n1. Do we really need masks? With SAM2 strong segmentation ability one doesnt need fully fine-grained mask for the few shot samples.\n2. Is the performance gain due to stronger SAM2? Prior works like VRP-SAM are based on older SAM and hence maybe limited in terms of performance.\n\nLIMITATIONS:\nOne limitation can be - \n\nThe work only segments single object classes and maynot be straightforward to extend to multi object segmentation?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}