A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models
Authors: Reda Bensaid
Vincent Gripon
François Leduc-Primeau
Lukas Mauch
Ghouthi BOUKLI HACENE
Fabien Cardinaux
Venue: Accepted by TMLR ()

Abstract: Few-shot semantic segmentation (FSS) is a crucial challenge in computer vision, driving extensive research into a diverse range of methods, from advanced meta-learning techniques to simple transfer learning baselines. With the emergence of vision foundation models (VFM) serving as generalist feature extractors, we seek to explore the adaptation of these models for FSS.
While current FSS benchmarks focus on adapting pre-trained models to new tasks with few images, they emphasize in-domain generalization, making them less suitable for VFM trained on large-scale web datasets. To address this, we propose a novel realistic benchmark with a simple and straightforward adaptation process tailored for this task. Using this benchmark, we conduct a comprehensive comparative analysis of  prominent VFM and semantic segmentation models. To evaluate their effectiveness, we leverage various adaption methods, ranging from linear probing to parameter efficient fine-tuning (PEFT) and full fine-tuning.
Our findings show that models designed for segmentation can be outperformed by self-supervised (SSL) models. On the other hand, while PEFT methods yields competitive performance, they provide little discrepancy in the obtained results compared to other methods, highlighting the critical role of the feature extractor in determining results.
To our knowledge, this is the first study on the adaptation of VFM for FSS.

Published in Transactions Machine Learning Research (05/2025)
A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models
Reda Bensaid
reda. bew..essrddd.mm--llaaiiiuu...eee
IMT Atlantique, Brest, France Polytechnique Montréal, Canada
Vincent Gripon
1MT Atlantique, Brest, Fronce
François Ceduc-Primeat Polytechnique Montrial, Canada
Lukas Mauch
Lukas, Mauckss.aucckssoy..ccomm
Sony Europe, B. V. Stuttgart Laboratory 1, Germany
Ghouthi Boukli Hacene
Ghouthi Chouth.BBokkHHoceeeBBoy..cccc
Sony Europe, B. V. Shullgarl Laboralory 1, Germany
Mila, Montreal, Conada
Fabien Cardinaux
Fabien. Cawdinoarr@songy.coo
Sony Europe, B. V. Stutigart Laboratory 1, Germany
Reviewed OpenReview: kttps: // openrevsew net/ forum? id= 5EXrH2N333
Abstract
Few-shot semantic sogmentation (FSS) is crucial challenge in computer vision, driving extensive research into a diverse range of methods, from advanced meta-learning tech- niques to simple transfer learning baselines. With the emergence of vision foundation models (VFM) serving as generalist feature extractors, seek to cxplore the adaptation of these models for FSS. While current FSS benchmarks focus on adapting pre-trained models to new tasks with few images, they emphasize in-domain generalization, mak- ing them less suitable for VFM trained on large-scale web datasets. To address this, we propose a novel realistic benchmark with a simple and straightffrward adaptation process tailored for this task. Using this benchmark, parative analysis of prominent VFM and semantic sogmentation models. To evaluate their effectiveness, we leverage various adaption methods, ranging from linear probing to parameter efficient fine-tuning (PEFT) and full fine-tuning. Our findings show that models designed for segmentation can be outperformed by self-supervised (SSL) models. On the other hand, while PEFT mothods yields competitive performance, they provide little discrepancy in the obtained results compared to other methods, highlighting the critical role of the feature extractor in determining results. To our knowledge, this the first study on the adaptation of VFM for FSS.
conduct comprehensive com-
1 Introduction
Semantic segmentation, the task of pixel-level object classification within images, plays pivotal role in computer vision applications (Minace et al. (022). This precise level of image understanding facilitates numerous applications across various domains, such autonomous driving (Ha et al. (017), seismic im-
agery analysis (Civitarese et al. 2019), aerial imagery (Tang al. 2023) and medical imaging (Petitjean)
et al. 2015 Wang et al. 2019) where precise image analysis is crucial.
Code to be released at: http:///gttaaoocco///eeeaaaaaats

---

Published in Transactions Machine Learning Research (05/2025)
Catalano & Matteucci 2023) has emerged as 2017). This paradigm seeks to train adapt models to previously unseen object classes with minimal labeled data. The typical benchmarks for FSS involve datasets split into four folds with different classes: three are used for training or creating training episodes, while the fourth fold is roserved for tosting or generating few-shot tasks, all containing an equal number of classes (Shaban et al. 2017: Nguyen & Todorovic 2019).
In recent years, few-shot semantic segmentation (FSS)
challenging yet crucial area of research (Shaban et al.
Recently, Vision Foundation Models (VFM) have emerged as pivotal advancement in computer vi- sion. These models are extensively pre-trained on vast amounts of data and then fine-tuned for specific downstream lasks. Unlike traditional models trained from scratch general-purpose representations, which can be transferred to a wide range of tasks and domains. The key advantage of VFM lies in their ability to capture complex patterns and semantic relationships present in diverse datascts, making them highly effective for tasks requiring cross-domain generalization. This has led us to explore the potential of adapting VFMs for FSS.
lask-specific data, VFM learn rich,
Although PASCAL-5' and COCO-20' serve standard benchmarks for FSS (Catalano & Matteucei 2023), they exhibit several limitations and do not align well with the capabilities of VFMs, which, we argue, allow for more general, efficient, and realistic use cases. More specifically, current benchmarks have the following limitations:
They focus mainly on in-domain generalization, where models are trained on specific classes and then evaluated on fow-shot tasks from the remaining classes within the same dataset. VFMs, by contrast, can generalize to new data without. the need to observe distinct yet similar distributions from the same dalaset beforehand. As such, VFMs can be deployed in more diverse conditions.
They assume that the classes of interest were not seen during pretraining of the VFM, which is unverifiable when it comes to foundation models, and also a restrictive setting when it comes to actual applications.
They only employ binary masks during training phases, where a single class is annolated as foreground and all other pixels are treated as background in support image This approach, adapted from few-shot classification, enables controlled construction of few-shot episodes by ensuring fixed number of samples per class. However, it introduces inconsistencies, such as the same class being labeled as foreground in one episode and background in another, which can hinder learning (Yang et al. 2021). Additionally, it limits the use of multi-class context during training, which is often present in realistic scenes
They exhibit balanced class distríbutions when constructing few-shot tasks. This design sim- plifics performance evaluation, but does not reflect the natural statistics of real-world semantic segmentation datasets, where certain classes occur far more frequently than others. As result, benchmarks with artificially balanced class sampling may misrepresent the challenges models face in realistic deployment scenarios.
In this paper, we introduce a new realistic benchmark meant to improve the following aspects:
Our benchmark docs not roly on training or fine-tuning on a subset of classes before evaluating on the rest, Instead, we directly perform few-shot adaptation on the target task, without prior exposure to the datasel. This design belter rellects real-world deployment scenarios of VFMSs which are expected to generalize to novel tasks with minimal supervision and without task- specific pretraining.
stronger emphasis on adaptation rather than requiring the model generalize to unseen classes in few-shot tasks TT which aligns better with the goal of VFMs (Bommasaui et al. 2021 Section 4.3).
"While existing FSS benchmarks do support multi-class segmentation in the final query images, they often rely on binary masks during the training phase for each class the N-way K-shot sctting.
'The recent Generalized Few-shot Semantic Segmentation (GFSS) benchmark (Tian al. 2022 supports multi-class segmentation. However, this benchmark relies on the existence of base clases, which is incompatible with our benchmark as stated above.
*Hecent benchmarks have started to address cross-domain few-shot segmentation. However, these approaches often involve training or meta-training on one dataset and then testing on another, which still differs from our objective.

---

Published in Transactions Machine Learning Research (05/2025)
DINOv2
SogFormer
iBOT SAM
100
ResNet
CLIP
mloU
MAE
60
40
20
Cityscapes
COCO
PPD
Figure 1: mIoU of various pretrained models when adapted to 3 different 1-shot segmentation tasks (only best performance achieved out of 5 tested methods is displayed).
Our approach uses fully labeled lew-shot examples, allowing for multi-class supervision. This setup enables richer contextual understanding, avoids semantic inconsistencies during training, and supports more complex few-shot scenarios. It is also compatible with standard semantic segmentation datasots and aligns with application domains, such as autonomous driving and remote sensing where full-scene understanding is typically required (Cagri Kaymak & Uçar 2018
Peng et al. 2022 Holder & Shafique 2022).
Class balance is not strictly enforced, instead, our benchmark reflects the natural class distri- bution present in the selected datasets. For instance, in datasets like Cityscapes, frequently occurring classes such as "car" or "road" appear in most images, while other classes are compara- tively rare. Preserving this imbalance provides a more realistic evaluation of model performance, particularly in terms of handling dominant: and rare classes under practical conditions.
The primary objective of our research to investigate which vision foundation model and adaptation method yields the most elfective few-shot semantic segmentation pipeline. We the first, knowledge, to systematically test the adaptation capabilities of renowned foundation models to this specilic challenge. To this end, we have introduced a novel benchmark, constructed using three widely- recognized semantic segmentation datasets, and systematically evaluated multiple adaptation methods. The five foundation models under consideration are DINOv2 (Oquab et al.] 2023), Segment Anything (SAM) [Krillov al.] (203)), CLIP (Radford et al.) [2021), Masked AutoEncoder (MAE) (He et al.) 2021 and iBOT (Zhou et al. 2022al. Additionally; we have carried out extensive experiments to better understand the elementary contribution of foundation models design (e.g. type architecture, size, training dataset) and the effect of the adaptation methods onto the obtained performance.
Our findings indicate that, surprisingly, models designed for segmentation are consistently outperformed by a self-supervisod learning (SSL) model, DINOv2, across various datasets and adaptation meth- ods. While Segment Anything (SAM) generally provides competitive results, it underperforms datasets (Figure D Additionally, our study indicates that parameter-effiienn fine-tuning (PEFT) meth- ods yield highly competitive performance, although the performance differential with other methods not substantial. This underscores the critical role of feature extractors in model performance. We antic- ipate that our research will assist in selecting appropriate solutions for few-shot semantic segmentation tasks and contribute to the rigorous benchmarking of this evolving field.
some
Related Work
In this section, we discuss related work the literature few-shot classification and FSS.
2.1
Few-Shot Classification
Few-shot classification initially focused on in-domain tasks, with benchmarks involving dataset splitting into training, validation, and fake few-shot task generation for evaluation (Ravi & Larochelle 2017 Snell
2017 Vinyals et al. 2016 [iin et. al. (001)). The introduction of Meta-dataset (Triantafillou let al.
al.]
2020 marked shift towards the cross-domain setting, where the dataset used for model training dilfers from the lasks evaluated later, aligning with more realistic scenarios. More recently, the field has embraced the of foundation models, adapting them to few-shot tasks. Notably, benchmarks such those in [Zoou et al. 2022b) explore the application of CLIP (Radford et al. 2021 to 11 downstream
tasks, and [CNOo al. (021)) few-shot downstream Lasks
2023

---

Published in Transactions Machine Learning Research (05/2025)
Methodologically, diverse approaches have emerged, ranging from hallucination- based Li et al 2020
Haribaran & Girshick) 2017] to mela-learning {Finn et al. 2017 Munkhdalai et al. 2018 Zhang et al. Munkhdalai & Yu, 2017]. Straightforward mothods have also demonstrated ellectiveness (Li et al.
2021
2021
Tian et al. 2020: Li et 2022 Recent surveys, such as (Luo et al. 2023), suggest that optimal approaches often involve simple finctuning atop competitive pretrained models, augmented with an additional linear layer. This observation motivates methods introduced in our paper.
2.2
Semantic Segmentation and Few-Shot Semantic Segmentation
Fow-shot semantic segmentation has been an active area of research for several years. Early works, such (Shaban et al. 2017) and (Nguyen & Todorovic, 2019 introduced benchmarks like PASCAL-5 and COCO-20', which share similarities with the in-domain benchmarks commonly used in few-shot. classification.
Subsequent studies leveraged pretrained models on ImageNet (Yang et al. 2021 Hong et al. 2022 Wn
(ct al. 2021 Okazawa 2022), drawing parallels to the Mota-datasct approach few-shot classification. Despite this progression, these methods continued to rely on the earlier benchmarks, limiting their applicability to broader scenarios.
More recently, efforts have shifted toward cross-domain few-shot semantic segmentation (Wang et al, Nie el. al. 2024: Lei et al. 2022). These approaches often depend on mela-training and assume that images from the target domain originate from distribution distinct from the training dataset. However, this assumption may not hold in scenarios involving foundation models, where domain boundaries are less clearly defined.
2022
Recent works focused adapting vision foundation models to FSS (Shuai al. 2023 Liu et al. 2024
Sun et al.] 2024 Meng et. al. 2024 Zhu el al. 2024 Zhang el al. 2024). However, these studies mainly explore specific model adaptations, without offering broad comparative analysis of their effectiveness as feature extractors. Our work provides benchmark and comprchensive comparative analysis of these models alongside various adaptation mothods, making our contribution orthogonal to existing rescarch.
Additionally, many existing methods focus on binary labeling for the few-shot setting, where only single object of interest is annotated in the support images, and all other objects are treated as background. This simplification can hinder effective training, particularly tasks requiring more muanced multi-class segmentation.
3
Benchmark
3.1
Task Formulation
In accordance with the established terminology of few-shot classification (Snell et al. 2017). our bench- mark revolves around the concept of support set S and a query set. Q. The support set S comprises a limited number of images, each accompanied by their corresponding, fully labeled, ground truth seg- mentation. These support images are employed for the training or calibration of generic semantic segmentation model. In contrast, the query set Q serves as the evaluation dataset on which we compute the mean intersection-ovve--nnion (mloU). In practical applications, is typically not available and utilized exclusively for benchmarking purposes.
To compute the mIoU, we calculate the Intersection-ovee Union (IoU) for each class and then average these values:
TP,
mIoU
TP: + FP;+
where n represents the total number of classes or objects, TP; is the true positive count for class i, FP; the false positive count for class i, and FN, is the false negative count for class i. One noteworthy distinction from few-shot classification is that individual images typically contain instances from various classes interest. As such, it is challenging lo define tasks in the k-shot manner, where each class associated with exactly l elements the support
In our proposed framework, we introduce more practical approach. We define a k-shot segmentation task one in which the support contains at least instances of each class. Using our proposed

---

Published in Transactions Machine Learning Research (05/2025)
sampling protocol described in the next subsection, a scenario involving a total of classes, k-shot lask mandates the inclusion of precisely nk unique images within the support set.
3.2 Benchmark Generation
To construct few-shot tasks from readily available semantic segmentation datasets, employ the fol- lowing sampling procedure:
1. Initialize an empty support set, denoted as S
2. For each class (excl. the background class) within the chosen dataset D, compile list of all training images that contain at least one instance of that class, resulting in D. {image image contains instances of class i).
3. Following arbitrary order of classes, we randomly select images from D, without replacement to include in the support set S. We ensure that images are only selected once.
If there are not enough remaining images for specific class i, we return to Step 1. important to note that this reset condition did not arise during our experiments.
4 Datasets
In constructing our proposed benchmark, we have selected three prominent semantic segmentation datasets, each offering unique characteristics:
Cityscapes (Cordts et al., 2016 2015) is a large scale dataset on semantic understanding of urban scene streets. It contains 2,975 images for training, 500 for validation and 1,525 for testing. It annotated using 19 classes (such as "car", "road", "building") and has input resolution 1024x2048.
COCO (Microsoft. Common Objects Context) (Lin et al., 2014) is a large scale dataset used for multiple tasks including semantic segmentation. The 2017 release contains 118k images for training, 5k for validation and 41k for lesting. It is annotated using 80 classes (such as "person", "bicycle", "elephant"). Images in COCO often exhibit varying input resolutions, commonly falling between 400 and 640 pixels for both width and height.
PPD (PLANT PHENOTYPING DATASETS) (Minervini et al., 2016: Hanno Scharr; Sotirios A. Tsaftaris) is small dataset used for multiple tasks including plant segmentation (fore- ground to background) and leaf segmentation (multi-instance seginentttion). We focus in our work on the plant segmentation task, where the dataset contains 783 images and 2 classes pnreground-backkroonn segmentation). Typical input resolutions for PPD images hover around 500x500. Contrary to the two other datasets, we considered the background to be a class of own for benchmark generation, meaning that I-shot problem would contain support images.
The inclusion of these three datasets ensures a diverse evaluation framework, spanning different levels of difficulty and data distributions. From simpler binary segmentation tasks, such as foreground-bcckground segmentation in PPD. to complex multi-class problems like COCO with 80 classes, this benchmark en- ables a comprehensive assessment of model performance across varied scenarios. Foreground bacoondd-- sogmentation, exemplified by PPD, holds significant. relevance in domains like medical imaging (Dumitru & Peteleaza 2023) and road extraction (Aich et al. 2018). Despite its smaller scale, PPD demonstrates the importance of addressing practical, real-world segmentation challenges that demand robust and efficient solutions. This approach aligns with practices in few-shot classification, such as the Coop benchmark, which utilized 11 publicly available datasets (Zhou et al. 2022b
5 Backbones and Adaptation Methods
In this section, we introduce the methodologies that form the basis of our comparative study. Drawing from existing literature on both fho- classification (Luo et al. 22023) and few-shot semantic seg- mentation (Catalano & MMtteucci] 2023 we have identified that the majority of methods rely on a combination of two fundamental components.
The first essential component pretrained model, typically a transformer (Vaswani et al. 2017 or convnet. It's important note that while these models are not specifically designed for semantic

---

Published in Transactions Machine Learning Research (05/2025)
segmentation, they bring substantial amount of valuable knowledge that can be leveraged in this conlext.
The second critical component is an adaptation method, a mechanism devisod to tailor the knowlodge encoded within the pretrained model for effective utilization in the specific task of interest.
Next, we provide comprehensive details on the specific models and adaptation methods that constitute the experimental foundation of our study.
5.1 Pretrained Models
In our study, investigate five pretrained models, each offering unique characteristies and advantages (see also summary of the models and their main components in Table 10::
MAE (Masked AutoEncoder) (He et al., 2021) model pretrained on ImageNet-1K Dong jet al. 2009) using a self-supervised technique involving masking portions of input images and recon- structing them using partial tokens from the unmasked segments. It comprises a Vision Transformer (VVT) (Dosovitskiy al.] 2 component. that maps observed image regions into latent representations and a decoder for image reconstruction. Previous research (He et al.) 2021) has demonstrated MAE's effectiveness in transfer learning for segmentation tasks, particularly in semantic segmentation.
SAM (Segment Anything Model) (Kirillov et al., 2023) is a recent foundation model for segmen- tation, pretrained on the SA-1B datasct, which includes 11 million images and 1 billion segmentation masks. SAM exhibits robust generalization capabilities across a broad range of segmentation tasks. It utilizes Vision Transformer (ViT) (Dosovitskiy et al. (021)) initialized with a MAE (He ct al., (021), prompt encoder, and a mask decoder. SAM's image enooder generates emberddings from input images, which are subsequently employed by the mask decoder in conjunction with the prompt encoder. While SAM's mask decoder outputs class-agnostic masks intended for segmenting instances or objects at var- ious granularilics, our specific task focuses exclusively on the encoder's role as feature extractor and does not require prompts.
iOTT (Zhou et al., 2022a) is a self-supervised learning model based on both masked image modeling and knowledge distillation. It comprises VIT trained either on ImageNet-1K (Deng al. 2009 or on ImageNet-222k and has show great. generalization capabilities on multiple downstream tasks. In our study, use the VIT trained on Imagenet- 1K since it has better downstream tasks performance in [Zhou et al. 2022a).
DINOv2 (Qquab et al., 2023 a loundational vision model trained using the self-supervised tech- nique (Caron et al. 2021) called DINO and iBOT (Zhou et al., 2022a This methods consists in having two networks with the same architecture, namely the teacher and the student. The student parameters are learned by minimizing loss measuring the discrepancy between teacher and student outputs learning "local to global" correspondence where the student. processes global and local views of the image while the teacher only processes the global views. The teacher is built with an exponential moving average of past iteration of the student. The model comprises VIT trained on diverse range of datasets, including notable semantic segmentation datascts like Cityscapes (Cordts et al.) (2016), ADE20k (Zhou let al.] 2017 2019), and Pascal VOC 2012 [Everingham et al.). DINOv2 has previously been evaluated on (not few-shot) semantic segmentation datasets in (Oquab et al. 2023). utilizing the frozen backbone and a simple segmentation decoder.
CLIP (Contrastive Language-Image Pre-training) (Radford et al. 2021 is model that con- sists of vision encoder and a text encoder. These encoders are jointly trained using contrastive loss to align embeddings images with their corresponding captions. The model is pretrained on private dataset of 400 million (image, text) pairs and has gained significant attention for its utility in zero-shot classification on downstream tasks. In our study, we exclusively leverage the ViT version of the visual encoder.
FCN (Fully Convolutional Network) (Long et al., 2015) is a conventional convolutional architec- lure extensively used for semantic segmennatiin tasks. It consists of an encoder and a decoder, both which are composed of convolutional layers. In configuration, the encoder is based on ResNet50 (He) let al. 2016) and is pretrained on a subset of COCO, specifically using the 20 classes shared with the Pascal VOC dataset. We incorporate FCN as convolution-only benchmark for comparison.

---

Published in Transactions Machine Learning Rosearch (05/2025)
Sugport
Adaptation
Support Image
Iarih Norts
Support
Outgut
Pre-truised
Frente
Query Sex Image
Query
Ore
Inference
Figure 2:
The pipeline of the Linear method: we freeze the backbone and train the shallow decoder on the support set, and then we test the encoder + on the query set.
SegFormer (iie et al., 2021) a semantic segmentation model based on a hierarchical Trans- former (Vaswani et al.] (0177) encoder paired with simple, lightweight MLP decoder. In our study, we use a SegFormer trained on ADE20K (Zhou et al. (001)) dataset, specifically utilizing only the ViT encoder. This model was selected for its recent advancements semantic segmentation, serving as a benchmark for comparison in our study,
5.2 Methods
We then consider five adaptation methods in study, ranging from linear probing methods to partial fine-tuning and full fine- mue-tuning methods
Linear: common approach (Oquab al. 2023 Strudel al. 2021 consists in freezing the pretrained backbone and adding segmentation head, called decoder, consisting of batch normalization layer
followed by a convolutional 1x1 layer. Figure 2 illustrates the overall pipeline for this linear adaptation method.
Multilayer: It has been shown in (Oquab et al. 2023) and Xic et al. 2021 that using a concatenation of multiple layers from the pretrained model can boost performance downstream tasks. We thus propose to use linear method on the concatenation of the last blocks of the pretrained model.
Singular Value Fine-tuning (Sun al., 2022) (SVF): The SVF method decomposes parameters into three matrices using Singular Value Decomposition (SVD) and fine-tunes only the singular values diagonal matrix while keeping the rest frozen. In (Sun et al. 2022). this approach demonstrated promising results in few-shot semantic segmentation tasks, showcasing its ability specialize the pretrained model without overfitting, In our study, we initially apply the linear method and then procced to fine-tune the singular values of the encoder alongside the decoder parameters.
Low-Rank Adaptation (Hu et al., 2022) (LoRA)): In this approach, we maintain the pretrained backbone's frozen state and introduce low-rank trainable adapters, as described in (Hu et al. 2022). Similarly to the SVF method, LoRA enables tuning only a limited number of parameters. Our procedure begins with training the decoder using the lincar method and then introduces LoRA adapters to the encoder. Subsequently, we fine-tune both the adapters and decoder parameters.
Fine-tuning:
The fine-tuning method, straightforward yet effective approach, involves fine-tuning the entire pretrained model. This method has demonstrated state-of-the-art performance in few-shot classification (Luo et al. (0023). However, it is susceptible to overfitting due to the limited number of training samples compared to the typically extensive parameter pool in the considered encoders.
We do not consider meta beaanng methods in our study, as these approaches rely on base classes to facilitate rapid adaptation to new classes, such as in episodic training. In contrast, our benchmark directly begins training on the support set.

---

Published in Transactions Machine Learning Research (05/2025)
6 Experiments
6.1 Training Procedure
To emsure the reproducibility of our resulls, we conducted all experiments with three fixed random seeds and computed the mean intersection-over-uiio across the datascts, we averaged the results per run corresponding to a seed and thus having the same sampled support set, and then averaged these three means to compute their mean and standard deviation. For preprocessing, we followed the approach outlined in (Gao) 2023). This involved random horizontal flipping, random scaling of the shorter side within the range of [400, 1600] while preserving the aspect ratio, and random cropping 2020) operations, including auto contrast, equalize, rotate, color, contrast, brightness, and sharpness. Subsequently, all images were resized to 1024x1024 resolution, except for CLIP and MAE, which accept. input sizes of 224x224. We acknowledge that the resolution variation, depending on the considered model, may introduce some analysis challenges but was an unavoidable aspect of our setup. For the learning rate, we employed the Polynomial learning rate schoduler with a power of 0.9 for the linear and multilayer mothods. Specifically, we used a learning rate of 0.2 for Cityscapes, 0.05 for COCO, and 0.001 for PPD. For fine-tuing methods (SVF, LoRA, and fine-tune), we did a grid search across models and datasets with values including 10-2, 10-3, 10-4, 10-5, and 10-6 to determine the one yielding the best average performance for 1-shot. Once identified, we applied this optimal learning rate to experiments involving more shots, primarily due to computational constraints. Linear and multilayer methods were trained with batch size of 4. For the Fine-tuning, SVF, and LoRA methods, the training phase consists of two stages. First, we train the decoder using the linear method. Then, we perform fine-tuning on both the decoder and the adaplers (or all parameters the case of fine-tuning) with a reduced batch size of 2 to address memory constraints. Training duration extended to 200 epochs for Cityscapes and PPD, and 100 epochs for COCO. The query set consisted of the standard validation sets for Cityscapes and COCO, For PPD, given the absence of conventional split, we performed a fixed 50/50 random split into training and validation sets. Our models were trained on a combination of NVIDIA A6000, NVIDIA RTX3090 and NVIDIA A100 GPUs. Runtime varied across datasets and models, ranging from approximately 30 minutes per run for MAF, CLIP, and ResNet. on Cityscapes approximately hours per run for DINOv2 on COCO. For the size of the models, we chose the base version of the vision transformers, leading to a 89M parameters model for SAM and 86M for DINOv2, CLIP and MAE. For the FCN we chose ResNet50 as an encoder, leading to 23M parameters model.
(mIoU) across these runs. For the results averaged
size of 1024x1024. We applied subset of RandAug (Cubuu et al.
6.2 What Works The Best?
First, we aim to identify the optimal combination of pretrained models and adaptation methods based on mloU. Table presents the average performance on the three considered datasets for all combinations of pretrained models and adaptation methods, along with their standard deviation over 3 runs (6.1 in the context of {1, 2,5, 5, 10)-shots. Several conclusions can be drawn:
DINOv2 consistently outperforms all other models across various settings. This superiority is especially notable Cityscapes and COCO, where DINOy2 surpasses other models by significant margin (see Appendix Tables 3 4
for the detailed results). Surprisingly, it outperforms supervised models pro-trained on other semantic segmentation datasots, showing the potential superiority of VFM adaptation over cross-domain generalization. However, since Cityscapes is included in DINOv2's pretraining data, albeit. without segmenlation masks, some degree of data leakage may be present. As stated the introduction, our primary objective is to assess the adaptation capabilities of off-the-shelf VFM rather than their gencralization abilities. Recent studies (Udandarao et al. 2024) have begun exploring this issue, emphasizing the need for further research into the influence of pretraining data distribution on downstream tasks.
SAM encoder exhibits strong performance on average. outperforms CLIP, MAE, and ResNet on both Cityscapes and PPD. However, it yields comparatively poor results on COCO (refer to Appendix Tables 3
5 6). This difference is likely attributed to our decision to retain the SAM image encoder while discarding the original mask decoder A.3
The multilayer method outperforms the linear approach for most models, indicating that in- cluding carlier feature maps, which are more locally focused (Raghu et al. 2021 Dosovitskiy

---

Published in Transactions Machine Learning Research (05/2025)
Table 1: mloU averaged over our three considered datascts in k-shot semantic segmentation for k 2,5, 10). Bold numbers correspond to the best performance across extractors and underline mumbers to the best performance across adaptation methods.
Linear
Multilayer
SVF
LoRA
Fine-tuning
SAM
40.66=09.00
43.36=02.52
41.84116.83
43.10:503.35
42.35=
DINOv2
54.78=03.09
53.51=08.10
57.77=04.20
57.67=02.50
57.21=04.04
1-shot
iBOT
41.28=20.87
41.24=01.80
43.044:03.31
43.10=:02.67
43. 08.791
CLIP
35.88+10.90
39.57=07:29
10.90+13.99
38.23+04.50
38. 68.694 10.20
MAE
35.50+10.16
36.65+00.38
37.76+008088
36.37+00.22
36.46++ 10.28
RCN-ResNet50
37.78=04.17
44.08=02.16
43.40+08.63
41.45=03.70
39.63+06.49
SegFormer
45.60=02.40
48.96=01.90
50.92+02.41
49.93#01.23
51.16+01.83
SAM
42.30=13.72
46.78=01.61
46.56101.96
45.78+02.94
45.73:02.16
DINOv2
61.20:100.29
59.21
64.10100.83
63.73+01.28
64.35+00.33
2-shot
iBOT
45.60++01.55
46.20+01.02
47.25
47.81+02.54
47.91+01.55
CLIP
39.08=00.45
43.46=00.70 70
45.59+01.94
42.72=00.97
35. 07.18
MAE
36.90=04.15
38.73=02.39
40.36+03.88
40.37::04.27
39. 09.
1CN-ResNet50
40.98=00.96
47.36=00.85
08.66100.92
46.35+02.36
08.81+00.32
SegFormer
50.79++01.2
55.58+01.00
54.622+0188
52.87+01.52
55.48++00.29
SAM
46.55++00.82
50.23+00.21
00.23
51.57+000.24
52.244-00.29
DINOv2
66.47=00.60
65.89=00.63
68.64100.88
67.54100.20
08.82101.40
5-sliot
iBOT
50.82=:00.38
51.08=00.16
03.12100.16
04.67
54..77+00.29
CLIP
46.30=100.29
48.67=00.61
51.56
49.00=100.39
43. 00.76
MAE
44.36=+00.23
05.02=+00.15
46.20+00.92
17.02:100.16
00.34
R0N-ResNe550
47.02:100.45
49.70=10.27
51.58+00.03
51.40:00.28
51.51
SegFormer
56.85=100.06
00.93
09.08
58.30:00.22
60.20
SAM
47.79=00.39
54.26=00.11
51.41:00.29
54.20=:00.37
55. 00.40
DINOv2
69.86-100.23
69.95=00.12
69.97-00.69
71.07+01.35 69.25:00.25
10-shou
iBOT
54.41
54.51=00.26
55.17
58.00+:00.58
58.59+
CLIP
50.14=00.41
52.49=00.41
00.77
51.92::0.411
47 47.62+00..38
MAE
47.56=00041
47.97=00.28
06.73
42.94+03.07
48. 00.30
48 18.699000.99
51.98=00.22
53.04400.31
53.02+00.25
52.86+00.55
SegFormer
59.53+t00.52
64.08=100.1
60.99100077
60.211:0.29
62.46-000.56
al. 202 provides the decoder with greater segmentation flexibility. Furthermore, it exhibits a lower standard deviation, indicating greater stability across the different runs of our proposed benchmark.
Despite variations the number of trainable parameters (less than 1% for LoRA, around 2% for SVF, and 100% for finetune), the finctuning methods (SVF, LoRA, and Full Fino-tuning) yield comparable results between them highlighting the importance of using efficient methods.
Fine-tuning scales better than LoRA and SVF and gives better results on average with 10 shots, this due the fact that we suffer less from over-fitting when having more shots.
6.3 Individual Factors
In the next series of experiments, we study the effect of various elements that could influence the perfor- mance of the models. Namely, these include the model size, the underlying architecture, the pretrained dataset, the pretraining method and the adaptation method, To isolate the effects of each component, we compare foundation models that differ primarily one of these aspects, although minor variations may exist between the models. By studying these individual components, sock to provide understanding of what drives model performance. (Other individual factors such as input resolution and registers are discussed in the Appendix A.5).
granular

---

Published in Transactions Machine Learning Research (05/2025)
6.3.1 Model Size.
demonsteeess that a larger model size does not necessarily translate a big boost of performance. Specifically, for SAM and DINOv2 with the linear method, the base model (ViT-B) yields, on average, competitive performance, despite having 3 times less parameters than the large (ViT-L.) models. In the case of ResNet, moving from ResNet-50 (23M parameters) to ResNet-101 (44M parameters) shows performance boast, though the increase is modest.
For the LoRA mothod, fine-tuning of subset of parameters of the encoder yields better performance with bigger models. These observations partially accounts for the performance discrepancy between SAM, DINOv2, and ResNet50, highlighting the influence of model size differences.
For tiny models like SegFormer-BB0, which has 3.7M parameters, there is a noliceable drop in perfor- mance, indicating that models adapted for device deployment struggle with poor adaptation performance. Enhancing the adaptation of these models remains area for future rescarch.
Table 2: Effect of different components on mloU. Bold numbers indicate the best performance among the compared models for a specific component.
Linear
LoRA
Cityseupes
COCO
PPD
Cityscupes
COCO
PPD
SAM (WT-B)
35.72:00.50
03.13=00.09
83. 13.14+10.36
38.50:+00012
05.85=00.15 13
84.91+10.11
SAM (VIT-L)
35.00:101.29
03.43=00.07
72.95=09.05
40.70:1:0:45
07.31=00.37
79.14=10.75
SAM (VIT-I))
34.64:101.36
03.38=00.11
83.34104.02
40.78:100.57
07.46=:00229
85.73:08.42
zandel size
Seg Bormoeee-BBi B0
30.97 00 .88
09.44
72.35--07.02
32.194
09.70+00..0 00.41
87.65 +1..35
Seg
40.55:+03.40
18.37=01.13
77.88+00.82
02.34++0.20
19.64=00.49
87.82=03.78
DINOv2 (Vit-S)
01.16:122.39
20 20.62-00.33 38
87, 07.99-105.58
44.75:100:71
22. 02.72200.71 71
86. 86.552166.0 06.08
DINOv2 (Vit-B)
48.80.101.88 882
00.38
92.31 1.0.84
54.35.101 .55
28.90 0133
199.77106.44
DINOv2 (Vit-L)
49.36::0.61
23.50=00.33
87. 87.87=07.30
54..9:100.64
31.54=00.81
87. 87.46=07.97
PCN-ResNet50 PCN-RosNet100
32.39.101.70
10.08-00.33
70.86105.99
35.74101.41
11.71 24
76.91 10.11110..9
32.97 01.76
00.16 10.94
69.70 I 12.98
29.38 10.003
12.23
00.57
81.79 11.40
DINOv1
29.03:101.08
05. 00..2200039
10.14
32.53:100:17
06 06.89400044
82. 02.990-109.0
training datnaet
DINOv2
01.82
23.24
00.38
92.31
54.35 01..5
28.99
01.33
89.67 06.43
CLIP
23.83:101.1
11.32=00.31
72.49+10.77
27.95++01.22
13.62=00.78
73.13+11.70
OpenCLIP
21.01=02.28
11.64=00.40
71.83=099885
22.73:101.58
14.34=00.06
71.55=09.44
MAE (INTK)
23.63 1.3..000 10.89
08.54
00.46
74.34 1. 10.88
26.10 1 100.26
10.13 ..0042
72.87 L 12.83
MAE (IG-3B)
21.73-101 01.00
00.16 05.76
75.18 : 09.26
23.59
10.21
07.84 00.33
80.05 : 07.44
DINOvi (VIT-S)
26.61:100.53
04.86=00.29
11.
29.90:1:0.995
05.82=00.34
79.00
DINOv1 (ResNet-50)
23.51=01.:0
03.42=00.05
58, 00.60
26.98:101.22
03.78=00.00
17.35=10.50
architecture
CLIP (ViT-B)
23.83100.551
11.32 01.31
72.49-1100 10.78
27.95-101.62
13.62:1 10..78
73.13
CLIP (Ro)Net-101)
20.22 01.57
02.61
00:06
69.00 1 06.03
08.96 00.11
00.91
00.03
67.40
OPenCLIP(VIT-B)
21.01:102.28
11.64=00.4f
71.855099.88 85
22,73:101.55
14.34:100.06
71.55:100.12
29.29 2,80
11.30 (00.02
81.05 1. 08.30
38.52 01.14
17.05 00.20
84.94 09.72
training method
DINOVI (KD)
29.03::0100
(M.92=00.39
82. 10.114::000 10.04
32.53:+00..7
06.89=00.444
82.99
MAE (MIM)
23.63 00.89
08.54 000.6
74.34 1 10.88
26.10 00.26
10.13 100.42
72.87 12.83
(BOT (KD + MIM)
29.46 01.77
09.31 00.68
:5.88
$2.54-101.43
11.27:#000 000.9
85.48 +07.80
6.3.2 Training Dataset.
To evaluate the impact of training dataset characteristics on model performance, we conducted a com- parative analysis of DINOv1 and DINOv2. DINOv1 was trained on the standard ImageNet-1K datasct, while DINOv2 utilized a more extensive dataset of 142 million curated and uncurated images. Addition- ally, it is noteworthy that the models differ their loss functions; however, our primary focus remains on the influence of dataset size and composition.
As illustrated in Tableg] DINOv2 siguificantly out outperforms DINOv1, particularly demonst superior results the COCO dataset. This suggests that larger training datasct may cnhance performance in few-shot semantic segmentation tasks. Nonetheless, this performance boost could also be influenced by other factors differentiating DINOv1 from DINOv2.
10

---

Published in Transactions Machine Learning Research (05/2025)
Furthermore, we compared CLIP, trained on 400 million (image, text) pairs, with OpenCLIP (Cherti jet al. 2023) trained on the LAION-5B (Schuhmann et al, 2022) dataset comprising 5 billion (image, text) pairs, and MAE trained on ImageNet-11K against another MAE variant trained on the Instagram-33 dataset (Singh et al. 2023p. which includes 3 billion images. According Table 2 for MAE and CLIP, an increase in training datasct size docs not consistently rosult in better performance and can sometimes lead to performance decline on certain datasets. Overall, the benefits of larger pre-training datasets remain ambiguous, indicating that the advantages of scaling up the training data not universally guaranteed.
6.3.3 Architecture.
To assess the influence of model architecture, we analyzed the performance differences between DINOvl (ViT-S) and its RosNet-50 variant, alongside comparing the CLIP (ViT-B) version against the CLIP ResNet-50 version. It is noteworthy that CLIP, the ViT-B model larger than its ResNet counterpart, an aspect previously discussed in terms of model size impact. According to Table 2 ViT architectures demonstrate clear performance advantage over their ResNet equivalents. This trend underscores the polential superiority of VIT models in adapling lo various lasks. For OpenCLIP, the ConvNext-L outperforms the VTT-B, this can be explained by the fact that we used a resolution of 1120 for ConvNext as in Yu ct al. 2023 and that CLIP only uses a resolution of 224 as discussed in A.5
6.3.4 Training Method.
To investigate the effect of different. self-supervised learning (SSL) methods model performance, we conducted comparison between knowledge distillation (KD) and masked image modeling (MIM) models. Despite our initial expectations that MIM models would surpass KD models (Park et al. 2023). the comparison in Table 2 reveals that DINOv1 (KD) surpasses MAE (MIM) on the Cityscapes and PPD datasets, and MAE outperforms DINOv1 on COCO. This mixed outcome indicatos that there no definitive answer as which SSL training approach offers the best adaptability for FSS tasks. However, iBOT, which integrates both KD) and MIM techniques, shows superior performance over both models, leveraging the strengths of both approaches.
7 Conclusion
We have introduced a novel benchmark specificlly designed to assess the adaptation of Vision Foundation Models (VFM) Few-Shot Semantic Segmentation (FSS). Our evaluation of multiple VFMs across this new benchmark has highlighted several key arcas for further exploration DINOv2 emerges as the top-performing backbone model, consistently outperforming other foundation models. While SAM exhibits strong overall results, it prescnts unexpected limitations certain datasets. Surprisingly, experiments underscore the competitive advantages of straightforward and efficient. methods such as simple linear segmentation heads, challenging the necessity of complex procedures?
This work not only presents a comparison of techniques but also more realistic and practical benchmark. We hope this benchmark will enhance the development few-shot semantic segmentation methods and offer robust framework for future comparisons.
References
Shubhra Aich, William van der Kamp, and Ian Stavness. Semantic binary segmentation using convolu- lional networks without decoders. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1824, 2018. doi: 10.1109/CVPRW 2218.0000..0
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-effinn efficient tine-tnnng for transformer-based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavi- cencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 9, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10.18653/ vv/ w22.2222222.oo..... URL http::: http:s:://allmmthoooy.oooo wrg.://////2/// acl-short.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creol, Jared

---

Published in Transactions Machine Learning Research (05/2025)
Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea. Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsumori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hau, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Gooff Kooling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ledhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitrion, Joon Sung Park, Chris Piech, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong. Yusuf H. Roohami, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michi- hiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang, On the opportunities and risks of foundation models. ArXin, 2021. URL htttps:/crffi //tp:////////mm///e wtw.ffsssanfor....moom www.e/aaatteeeprrttttoo
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Nico Catalano and Matteo Matteucei. Few shot semantic segmentation: a review of methodologies and open challenges, 04 2023.
Shoufa Chen, Chongjian GE, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances Nearul Information Processing Systems, volume 35, pp. 16664-16678. Curran As- sociates, Inc., 2022. URL
nourips cc/paper www.ffllllppppee pttpee//pppprr20///1100. htt.:///0070.0000070005500----r.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsey. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 2818-2829, June 2023.
Daniel Civitarese, Danicla Szwarcman, Emilio Vital Brazil, and Bianca Zadrozny. Semantic segmentation of seismic images, 2019.
Marius Cordus, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes datasct. In CVPR Workshop on The Future of Datasets in Vision, 2015.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Bo- nenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes datasct for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In 2020 TEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3008-3017, 2020. doi: SVPR009/VVRW55008. 2020.00359.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need reg- isters, 2023.
Jin Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrar- chical image database. In 2009 IEEE Conference on Computer Vision and Pottern Recogmition, pp. 248-255, 2009. doi: (VL00//CPP.20000.55
12

---

Published in Transactions Machine Learning Research (05/2025)
Alexey Dosovitakiy, Lucas Beyer, Alexander Kolesmikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- lerthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreil, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Inter- national Conference on Learning Representations, 2021. URL Sttrgs://ppemeeriew.ne///oooo net/forum?id= YicbFaNTTy.
Razvan- Gabriel Dumitru and Darius Peteleaza. Using duck-net for polyp image segmentation. Scientific Reports, 13, 06 2023. doi: 10.1038 00038/441598-225 023-6990-----
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. network.org. wwt.ook..og//caallalee Schallenge/VOCCC,C hwwoo////wwrkssooo/oooooooooaeee
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation deep networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, IC,L'17, pp. 1126-1135. JMLR.org, 2017.
Roland Gao. Rethinking Dilated Convolution for Real-time Semantic Segmentation. In 2023 IEEE/CVP Conference on Computer Vision and Pallern Recognition Workshops (CVPRW), pp. 4675 4684, Los Alamitos, CA, USA, June 2023. IEEE Computer Society. doi: 10.1109 CWtp///VPPP/../../// 202339202222200404400 URL https://doi. wesscoonpteersooceeey..... org/10. 10. 1109/CVVPP992228 2023. 00493
Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: To- wards real-time somantic segmentation for autonomous vehicles with multi-spectral scenes. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5108 5115, 2017. doi: 10.1109 10.1109/IROS... 22....017..200696
Massimo Minervini; Hanno Scharr; Sotirios A. Tsaftaris. URL httpgs//ww..llann-peeoooooop org/ datasets-home
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating fea- tures. pp. 3037-3046, 10 2017. doi: 10.1109/ICCCI 2017.328.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770 778, 2016. doi: MM.1009/CVPR221666
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoen- coders are scalable vision learners. 111607111.00337, 2021.
Christopher Holder and Muhammad Shafique. On efficient real-time semantic segmentation: A survey, 06 2022.
Sunghwan Hong, Scokju Cho, Jisu Nam, Stephen Lin, and Soungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In Europeon Conference on Computer Vision, pp. 108- 126. Springer, 2022.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Lawk adaptation of large language models. In International Conference on Learning Representations, 2022. URL
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Scr-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), 2022.
Shibo Jie and Zhi- Hhi-Hong Deng. Convolutional bypasses are better vision transformer adapters, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yer Lo, Piotr Dollár, and Ross Girshick. Segment anything. 0tti:2334402244.. 2023.
Shuo Lei, Xuchao Zhang, Jianfeng He, Fanglan Chen, Bowen Du, and Chang- Lu. Cross-domain few-shot semantic segmentation. In European Conference on Computer Vision, pp. 73 90. Springer, 2022.

---

Published in Transactions Machine Learning Research (05/2025)
Kai Li, Yulun Zhang, Kumpeng Li, and Yun Fu. Adversarial feature hallucination networks for few-shot learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recogrillon (CVPR), June 2020.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few-shot classification. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9526 9535, October 2021.
Wei-Hong Li, Xialci Liu, and Hakan Bilen. Cross-domain few-shot learning with task-specific adapters. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognitiin (CVPR), pp. 7161-7170, June 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014, Springer International Publishing. ISBN 978-3-319-10602-11
Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching, 2024. URL httpa://arziw.org//aa 1805.13310
Jonathan Long, Evan Shelhamer, and Trevor Darroll. Fully convolutional networks for semantic segmen- tation.. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431- 3440, Los Alamitos, CA, USA, June 2015. IEEE Computer Society. doi: 10. 1099/CYP..2015.220000 URL https://doi. htw.o/ooooeeeccoooooooooooeeeeee org/ 11/g/10..10 1109/CVPR. 201...7918990
Xu Luo, Hao Wu, Ji Zhang, Lianli Gao, Jing Xu, and Jingkuan Song. A closer look at few-shot classification again. In International Conference on Machine Learning. 2023.
Lingchen Meng, Shiyi Lan, Hengduo Li, Jose M. Alvarez, Zuxuan Wu,, and Yu-Gang Jiang. Segie: Unleashing the emergent correspondence for in-context sogmentation, 2024. URL https://arxiv.
org/abs/2311.140
Shervin Minace, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopou- los. Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(():3523-335) 2022. doi: 10.1109/ 10.1109/TPAMI.200 20A1M...022100000
Massimo Minervini, Andreas Fischbach, Hanno Scharr, and Sotirios A. Tsaftaris. Finely-grained anno- laled datasets for imsee plant phenotyping. Paltern Recogvilion Lellers, 81:80 89, 2016. ISSN 0167-8655. doi: https://doi..rg 10. pttp://00000000000..ta..0..1.3 patrec..011.10000....... URL https://www www.seedireccc....... http://sseee/aarrceee///////1111
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Confer ence on Machine Learning Volume 70, ICML'17, pp. 2554-22 2563. org, 2017.
Tsendsuren Mumkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with con- ditionally shifted neurons. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Interna- tional Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3664-3673. PMLR, 10-15 Jul 2018. URL https://prroceedings .ar.iingsss......mmmmmmm.. mlr, pttppeee/00/mmmmaaaaaaaaaaaaaaaa
html
Khoi Nguyen and Sinisa Todorovie. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Jiahao Nie, Yun Xing, Gongjee Zhang, Pei Yan, Aoran Xiao, Yap-Peng Tan, Alex C Kot, and Shijian Lu. Cross-domain few-shot sogmentation via iterative support-query correspondence mining. arXiv preprint or xrXio22401.88007 2024.
Atsuro Okazawa. Interclass prototype relation for few-shot sogmentation. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 362-378, Cham, 2022. Springer Nature Switzcrland. ISBN 978-3-031-19818.2

---

Published in Transactions Machine Learning Research (05/2025)
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacddin El-Nouby, Russell Howes, Pa-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Namuk Park, Wonjac Kim, Byeongho Heo, Tackyung Kim, and Sangdoo Yun. What do supervipe vision transformers learn? In The Eleventh International Conference on Learning Represendations, 2023. URL https::/openreviieee http://wweeettttrortmodaaaaaaaaa
Juncai Peng, Yi Liu, Shiyu Tang, Yuying Hao, Lutao Chu, Guowei Chen, Zewu Wa, Zeyu Chen, Zhiliang Yu, Yuning Du, Qingqing Dang, Baohua Lai, Qiwen Liu, Xiaoguang Hu, Dianhai Yu, and Yanjun Ma. Pp-liteseg: A superior real-time semantic sogmentation model, 04 2022.
Caroline Petitjean, Maria A. Zuluaga, Wenjia Bai, Jean-Nicolas Dacher, Damien Grosgeorge, Jérôme Candron, Su Ruan, Ismail Ben Ayed, M. Jorge Cardoso, Hsiang-Chon Chen, Daniel Jimenez- Carretero, Maria J. Ledesma- Ledesma-Carbayo, Christos Davatzikos, Jimit Doshi, Guray Erus, Oskar M.O. Maier, Cyrus M.S. Nambakhsh, Yangming Ou, Sébastien Ourselin, Chun-War Wei Peng, Nicholas S. Peters, Terry M. Peters, Martin Rajchl, Daniel Rueckert, Andres Santos, Wenzhe Shi, Ching-Wae Wei Wang, Haiyan Wang, and Jing Yuan. Right ventricle seginentation from cardiac mri: A collation study. Medical Image Analysis, 19(1):187-20) 202, 2015. ISSN 1361-8415. doi: /ttp::///oiioog////// 100 1tt.:1011....... media. 10.044 URL https: www. scieecedrree.... com/ ccccaaieeaarrccceeep/////111111/
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18 24 Jul 2021. URL https://prooceedings. mlr. sttess/v139/radfordiie html
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (ods.), Advances in Newral Information Processing Systems, 2021. URL
httpps://openreviev. wttp://wwww.eeeoooooooooonnaaaaa
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Con- ference on Learning Representations, 2017. URL https://openreviir nett/forumii--rJY--Koo
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Thco Coombes, Aarush Katta, Clayton Mullis, Mitcholl Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thérty- sizth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. 2017.
Chen Shuai, Meng Fanman, Zhang Runtong, Qiu Heqian, Li Hongliang, Wu Qingbo, and Xu Linfeng, Visual and textual prior guided mask assomble for few-shot segmentation and beyond, 2023. URL
https://arxiv.orgg wttp://////bbb/////eeeeeeee//// 2308.075383
Mannat
Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Reichtenhofer, Ross Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of mae pre-pretraining for billion-scale pretraining, In ICCV, 2023.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, In Advances in Neurul Information Processing Systems, 2017.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia. Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7262-7272, October 2021.
15

---

Published in Transactions Machine Learning Research (05/2025)
Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang, Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value line-luning: Few-shot segmentation requires lew-parameters fino-tuning. 2022.
Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, and Zechao Li. Vrp-sam: Sam with visual reference prompt, 2024. URL fttpa://arxiv.oom abs/2402.17PB
Maofeng Tang, Konstantinos Georgion, Hairong Qi, Cody Champion, and Marc Bosch. Semantic seg- mentation in aerial imagery using multi-level contrastive learning with local consistency. In 2023 IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pp. 3787-3796, 2023. doi: 10.1109/ 10.1109/WAV566688 21253008882023003773
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: A good embedding is all you need? Andrea Vedaldi, Horst Bischof, Thomas Brox, and Michhel Frahm (eds.), Computer Vision - ECCV 2020, pp. 266 282, Cham, 2020. Springer Internationa Publishing. ISBN 978-9-030-58568-6.
Zhnotao Tian, Xin Lai, Li Jiang, Shu Liu, Michelle Shi, Hengshuang Zhao, and Jiaya Jia Generalized few-shot semantic segmentation, 2022. URL httptt://aaaxi...oott http::///araaiv..rg//////./..0/0 05210
Eleni Triantafilluu, Tyler Zlu, Vincent. Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations (submission), 2020.
Carles Gelada, Kevin Jordan Swersky,
Vishaal Udandarao, Ameya Prabhu, Philip Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. Pre- training concept frequency is predictive of CLIP zero-shot performance. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024. URL htttps://operrvviet
pet//forummid=551122/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, I. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://pproceedingg meurips. pdf
cc/paper
Oriol Vinyals, Charles Blundell, Tim Lillicra,, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pp. 3630-3633 3638, 2016.
Li Wang, Dong Nie, Guanman Li, Élodie Puybareau, Jose Dolz, Qian Zhang, Fan Wang. Jing Xia, Zhengwang Wu, Wei Chen, Kim-Han Thung, Toan Due Bui, Jitae Shin, Guodong Zeng, Guoyan Zheng, Vladimir S. Fonov, Andrew Doyle, Yongchao Xu, Pim Moeskops, Josien P. W. Pluim, Christian Desrosiers, Ismail Bon Ayod, Gerard Sanroma, Oualid M. Benkarim, Adrià Casamitjana, Verónica Vilaplana, Weili Lin, Gang Li, and Dinggang Shen. Benchmark on automatic six-month-old infant brain segmentation algorithms: The iseg- 2017 challenge. IEEE Transactions on Medical Imaging, 38 (9):2219 2230, 2019. doi: N0.110//TM1.201 29917722..
Wenjian Wang, Lijuan Duan, Yuxi Wang, Qing En, Junsong Fan, and Zhaoxiang Zhang. Remember the difference: Cross-domain few-shot semantic segmentation via mela-menory transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7065-7074, June 2022.
Zhonghua Wu, Xiangxi Shi, Guosheng Lin, and Jianfei Cai. Learning meta-class memory for few-shot semantic segmentation. In 2021 JEEE/CVE International Conference on Computer Vision (ICCV), pp. 497-506, 2021. doi: 10.1119/ICCV4400 CBCCV449222002100000
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Sogformer: Simple and efficicnt design for semantic segmentation with transformers. In Neural Information Pro- cessing Systems Caatems(eenn (NeuriPS), 2021.
Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. Mining latent classes for few-shot segmen- tation. In ICCV, 2021.
16

---

Published in Transactions Machine Learning Research (05/2025)
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang Cheh-Chhhhhhhh Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. In NearIPS, 2023.
Anqi Zhang, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, and Yunchao Wei. Bridge the points: Graph- based lew-shot segmentt anything semantically, 2024. URL https://arxiv, org/ abs/ /4100 2211//1111.....44.
Chi Zhang, Henghui Ding, Guosheng Lin, Ruibo Li, Changhu Wang, and Chunhua Shen. Meta Naviga- tor: Search for Good Adaptation Policy for Few-shot Learning In 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 9415 9424, Los Alamitos, CA, USA, October 2021. IEFE Computer Society, doi: 10.1109 10.1109/ICCV (LCCCV892.2209 2112222222.00000 URL huttps://dol www..oaattcccttttrccc.e.....
org/10 10. 1109/ ICCV48922 2021 00930
Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets loRA: Parameter efficient finetuning for segment anything model. In The Twelfth International Conference on Learning Representations, 2024. URL https:
Bolci Zhou, Hang Zhao, Xavier Puig, Sauja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recogrilloon, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tele Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, :27(3):302 321, 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, ibot: Image bert pre-training with online tokenizer. International Conference Learning Representations (ICLR), 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwci Liu. Learning to prompt for vingon-language models. International Journal of Computer Vision (IJCV), 2022b.
Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang, and Chunhua Shen. Unlcashing the potential of the diffusion model in few-shot semantic segmentation, 2024. URL
https://arxiv org/ abe/ /410. 02366
Çağnı Kaymak and Ayşegül Uçar. A brief survey and an application of semantic image segmentation for autonomous driving. In Handbook of Deep Learning Applications, 2018 URL. htttp:://aaa
semanticscholar org/ (oorpusID6697881111 69783131
A Appendix
A.1
Detailed Results for Every Dataset
We can see in Tables 3 i
5
6
the detailed results the 3 datasets separately:
The range of performance
the 3 datasets is different mainly due to the varying size and number of classes. On PPD, we only have 2 classes making the task easier but the training is more unstable due the restricted number of training examples, since we only have 2 images in our support set. This leads to a bigger standard deviation in PPD compared to COCO and Cityscapes since the choice of these images is crucial and the images are different between the 3 folders constituting the dataset.
SAM image encoder performance is poor on COCO compared other models even though it yielded good performance in the other 2 datascts. This is probably due to the fact that we isolated the image encoder (the adaptation of the whole SAM model to prompt free Few-Shot. Semantic Segmentation is left to future work).

---

Published in Transactions Machine Learning Research (05/2025)
Table 3: Detailed results of mloU on our three considered datasets in 1-shot semantic segmentation.
Linear
Multilayer
SVF
LoRA
Fine-tuning
SAM
35.72=+01.0
39.06=:0.99
38.90::0::4
38.50=00. 12
38, 08.14400.40
DINOv2
48.80=:01.22
46.77=02280
51.96:10.37
54.35-101.55
50.27:00.25
Cityscapes
iBOT
29.46++01.72
26.84+02.04
32.16+01..20
32.54+00..43
31.78++01.3)
CLIP
23.83-101.51
16.10-00..0
28.72:+01.06
27.95-1-01.2
27.74-1:11. .38
MAE
23.63:100.89
23.811100.76
25.881100332
26.10-1:00.6
25.07-100.64
(CN-ResNet50
32.39-11.779
34.12=#02.0
36.141:0.119
35.74+01.41
36.17=00.41
SegFormer
10.55-103.40 40
44.22+-03.50
42.52+01.97
42.344+02.20
42.19-002
SAM
03.10-400.09
00.21+
05.31 000.99
05.85-1-00.13
05.16+
DINOv2
23.24-100.33
20.92 10019
28.30 00.77
28.99101.33
28.15--01
00----1.24
COCO
iBOT
09.31=00.68
00..57
10.91
11.27=00.99
11.67+01.00
CLIP
11.32:10.31
14.47=00.299
15.70+00.71
13.62++00778
05.19
MAE
0.54
08.60=#00445
00.52+00.45
10.13+00.42
10.71
FCN-ResNet50
10.08
08.82+-00.88
02.05
11.71+-00.24
11.88++00.35
SegFormer
18.37-101.1
19.69-100.82
20.25
19.64-100.49
21.16:1:0064
SAM
83.14=10.36
84.82=00.43
81.32=17.07
84.94=10.11
83.75#10.01
DINOv2
92.31=01.84
92.85=01.770
93.05+02.57
89.67
92.61+02.10
iBOT
85.06=108.80
87.52++06.23
86.04410.37
85.48+107.80
87.63+
PPD
CLIP
72.49+10.77
78. 06.881
78.28 12.66
13.13+11.70
73.194-06
MAE
14.34--10.85
07.54
76.90
12.87+12.83
73.62 10.59
TCN-ResNethhy
00.96
89.31=01.88
02.01+08.04
10.91
06.84
SegFormer
77.88=00.82
82.98=10.000
88.99405.98
87.82+00378
90.14+:03.72
Table 4: Detailed results of mloU three considered datasets in 2-shot semantic segmentation.
Linear
Multilayer
SVF
LoRA
Fine-tuning
SAM
38.84=00.38
42.71=00.24
40.241:0.18
40.62::00.2
40.63=00.77
DINOv2
56.54=00.75
53.36200.09
58.21#01.00
58.84+01.75
58.60=00.57
Cityseapes
iBOT
33.25-100.62
32.42.100.58
00.60-10.85
36.37
35.29-100.87
CLIP
25.19:101.00
28.96:101.62
00.851100422
30.53:10..19
19.29:104.90
MAE
24.15:100.32
24.88100.51
27.681:0022
27.92
26. 00.332
TCN-ResNet50
36.40=00.57
37. 37.89=00.44
08.98+00.06
38.34+00.22
38.97400.33
SegFormer
00.82
10..1
06.35
47.35
46.96--01.77
SAM
04.45-100.03
000.0
17.92-00022
08.82-100.31
08.99
DINOv2
33.08-100.17
30.01
39.91110.90
39.01101.36
40.09100.68
COCO
iBOT
13.13-100.81
13.13:100.63
00.65=10.75
16.844:0078
06.84+00.88
CLIP
16.03=200.4
18.85=00.67
22.33+000466
20.40
21.85+00.884
MAE
12.20-100.69
00.66
15.30 000.27
15.26-100.34
15.66 00.21
FCN-ResNet50
12.54-100.50
11.44.100.48
13.23.L00.78
13.42.100.81
12.55-100.83
SegFormer
24.23:100.20
25.38:100.25
26.34100.33
25.62+100.7
28. 08.088100.72
SAM
83. 63.62-10..9
88.32-104.81
91.51105.99
87,894109.08
87.56-107.75 .3
DINOv2
93.96=01.44
94.24-02.05
94.17101.68
93.33-02.18
94.35+
iBOT
90.43-1-05.44
93.05 1- 02.63
90.79-106.00
90.23 0-6.42
91.594 04.
PPD
CLIP
10.91
82.58
03.59
17.21
65.92 15.61
MAE
74.36+112.00
79.05-107.61
78.12+11.30
12.92
74.97+
PCN-RosNet50
74.01
92.74102.02
93.76+00200
87.88106.22
94.91
SegFormer
82.74-104.82
90.67-00.62
90.64+104.80
85.66-402.75
91.39--01.59

---

Published in Transactions Machine Learning Research (05/2025)
Table 5: Detailed results of mloU on our three considered datasets in 5-shot semantic segmentation.
Linear
Multilayer
SVF
LoRA
Fine-tuning
SAM
41.15=01.71
44.90=10.74
42.29::00.32
44.38
45, 45.15=00066
DINOv2
59.31=122.0
59.22:102.16
61.22:101.97
61.69+00.37
62.88
Cityscapes
iBOT
36.45-+00.24
36.67=00.22
39.45+00.22
41.05++00.72
40.46++00.47
CLIP
28.70++0120
31.21=01.68
33.88+01.25
33.23
13.40--01.33 .31
MAE
27.49-100.72
28.19:101.01
30.67.100.91
30.92-100.90
29.91-101.0
PCN-ResNet50
37.68=100.97
39.85=200.13
42.42+:00177
42.80+00.45
03.30
SegFormer
18.26+101.13
53.39+00.33
48.00+00075
50.06+000.49
50.00++00.63
SAM
05.92-+00.17
04.01
11.89
14.30++00.45
14.92
DINOv2
44.48-100.64
42.32 1 00.66
49.07--01.3
45.821.01.. 11.80
50.52-100.34
50.52
01.34
COCO
iBOT
20.99=10088
20.85=00.61
00.99
27.62=#00.8
27.28=00.200
CLIP
06.71
29.11=00.67
01.85+
30.71=000.1
00.39
MAE
19.20=:00.52
19.27=00.44
20.67
22.80:100.40
23.39+000.39
FCN-ResNet50
15.42-+00.27
14.23+00.65
15.47+00.23
15.944-00.11 16
14.09+-00.10
SegFormer
31.80-101.20
33.47101.19
34.68101.47
32.87
36.56-101.5
SAM
92.59=01.81
96.28=00.19
95.61200.65
95.82100.55
96. 00.
DINOv2
95.21=00.35
06.13
95.62=00.47
95.11=00.33
93.06=
iBOT
95.02+100.95
95.71+00.255
95.54400.63
95.344100.71
96.57+00.16
PPD
CLIP
83.48++01.12
05.68-00044
08.94
83.07+001.6
85.76+
MAE
06.49-100:42
87.61
36.26101199
87.341100.74
96.73--00.36
87 07.97=00.79
95.02=10.11
96.84=00.15
95.45=:0065
97.13#0.02
SegFormer
90.48=+00.82
95.92+100.67
94.56100000
91.97+00.24
94.03++00.19
Table 6: Detailed results of mloU on our three considered datasets in 10-shot semantic segmentation.
Linear
Multilayer
SVF
LoRA
Finc-tuning
SAM
42.10=01.22
48.41=00.34
42.74+:00.47
46.75=:0.22
47.17=00.68
DINOv2
63.19:00.23
63.64=00.64
63.381000455
68.87100.46
66.24-00.49
Cityscapes
iBOT
30.57
39.22
40.46.10086
43.05
43. 13.7991-1.000
CLIP
32.44101.33
31.83:100..2
36.76101.07
36.50101.11
11.83-110.21
MAE
30.46:100.28
30.59:100.18
02.63:00.75
32.99+00:45
31.85 01..55
R0N-ResNet50
40.63=10066
43.20
45. 03.55
45.36+00.66
45. 15.63+01.33
SegFormer
50.76-100.56
00.67
00.12:000.45
52.811000.03
52.08 00.17
SAM
07.13:000.55
17.81 100.33
15.02
19.36-L00.74
21.32 10.49
DINOv2
50.84-100.43
49.83:100.34
50.84101.23
48.55100066
52.74-01..09
COCO
iBOT
28.23:100.46
28. 000.0
29. 00.88
35.26+101.00
35.20+
CLIP
33.75=000336
36.45-200.53
35.36:00054
35.38+00.33
34. 00,100
MAE
25.38-100.22
25.88-1-00.25
20.96:100.54
08.844- 11.26
26.86- 00.25
FCN-ResNet50
17.04-100.43
16.74.100448
16..1
17.15.100.21
15.55-100.17
SegFormer
36.47-100.9
38.97
37.64110..60
35.59+100.55
40.61
SAM
94.15-100.6
96.57:00.07
96.481100.14
96.49100.13
97.12-100.54
DINOv2
95.55-00.15
96.38-00.24
95.67-00.33
95.32400.41
94.24+02.66
iBOT
95.53-100.00
96.02
96.01+00.11
95.68
96.78 4- 00.
PPD
CLIP
00.55
000.2
10.87
83.37
00..99
MAE
86.84+100.44
87.441100.88
00.59
87.00401.90
07.17
PCN-ResNet50
88.40=101.22
96.01+00.28
97.18100.20
96.34100.16
97.39-100.11
SegFormer
91.35-+-00.13
96.49-100.06
95.21400.15
92.21+-00.30
94.68--00.27

---

Published in Transactions Machine Learning Research (05/2025)
Table 7: Effect of PEFT methods on mloU, bold numbers denote top performance across methods.
SVF
LoRA
VPT
Adaptformer
Convpass
Bitfit
Cityscapes COCO
51.96+011.0 .37
54.35+01.55
51.92+00.56
50.24+01.12 50.44+000.33
51.55-00.558
28.30:100.77
28.99-101.33
30.15:102.10
28.68:101.23 92.87101.477
29.35:101.63
29.19:100.4
PPD
93.05:102.57
80.67106.43
91.251102.1
92.21110.50
92.91-01.50
A.2
Parameter Efficient Fine- Tine-Tuning (PEFT) Methods.
Given the strong performance Parameter-Efficies Fine-Tuning (PEFT) methods on our benchmark, we opted to evaluate various PEFT approaches from multiple families. We examined LoRA and SVF alongside two adapter tuning methods: AdaptFormer (Chen et al.) (022). which integrates lightweight adapter module parallel to the feed-forward network within the encoder blocks and only trains these adapters, and Convpass (Jie & Deng) 2022), which adds convolutional layers to the parallel adapter to incorporate inductive biases and trains only these layers. Additionally, we assessed Visual Prompt Tuning (VPT) (Jia et al. 2022), which introduces lokens at the imput of encoder blocks and trains only these tokens, and partial-based tuning method, BitFit Zaken et al. 2022), which freezes the model weights and fine-tunes only the biases. For computing constraints, we only tosted these methods with DINOv2 on I-shot setting.
The results, presented in Table 7 indicate minimal differences among the PEFT methods, suggesting no definitive superior method. This outcome underscores the critical role of feature extractors in determining the effectiveness of these tuning approaches.
A.3 SAM Underperformance
In this section, we delve into the reasons behind SAM's poor performance on the COCO dataset, exploring two main hypotheses:
Decoder Limitations: The primary suspect for SAM's subpar performance is the dependency of the image encoder on the prompl encoder and the mask decoder. To address this, we replaced the original mask decoder with transformer decoder, as detailed in (Kirillov et al.] 2023), mod- ilying it for few-shot semantic segmentation following the inspiration from (Zhong ct al. 2024 We introduced a constant prompt to the decoder, altering tokens to produce an N-class depth map rather than binary masks, and eliminated the IoU score output branch. We experimented with two mask decoder configurations, one with Conv Transpose module and another where we upscaled the output. through interpolation, while initializing remaining modules with pre- trained weights. Despite these adjustments, this approach was outperformed by a simpler linear decoding method by up to 5% mIoU, likely due to the challenges of training a complex decoder few-shot scenario.
Mask Distribution Bias: Another investigated issuc was the mask distribution bias in COCO towards image centers, unlike the distribution in the SA-IB dataset used for SAM, highlighted in (Kiiillov et al., 2028), where masks are more evenly distributed. We attempted to mitigate this bias with various preprocessing techniques (RandomCrop, CenterCrop, dividing each image into four smaller images); however, these modifications did not. enhance results.
These findings shed light the specific challenges faced by SAM in adapting to the COCO dataset and highlight the complexities involved in optimizing for few-shot semantic segmentation tasks.
A.4 Practical Considerations
Previous values were obtained by selecting the best. learning rates on the validation sets. Instead, here we study the effect of transfering the learning rate from a dataset to another one on the mIoU. The learning rate values we explored are 10-2, 10-3, 10-1, 10-3 and 10-6, Table shows the drop of mloU when we train pair (model, method) dataset A with the best learning rate for datasct B. We that the learning rate is relatively consistent between the datasets and especially between Cityscapes and COCO for a given model and given method. Even though the optimal values are not exactly the

---

Published in Transactions Machine Learning Research (05/2025)
Table 8: Drop in mloU when using the best learning rate corresponding to the dataset on the line (source) when training on the dataset on the column (target), instead using the target validation set to find the best learning rate.
SVF
LoRA
Fine-tuning
Cityscapes
COCO
PPD
Cityscapes
COCO
PPD
Cityscapes
COCO
PPD
CLIP DINOv2 SAM
Cityscapes
0
02.72
22.83
COCO
0
00.94
03.01
PPD
0
0
0
03.38
03.83
Cityscapes
0
0
02.31
15.62
COCO
0
0
0
02.00
0
26.30
PPD
0
00.88
00.88
01.17
01.17
Cityscapes
00.40
00.52
00.52
COCO
0
01.09
00.24
0
0
PPD
01.91
01.91
01.53
0
MAE
Cityscapes
00.19
00.19
00.54
00.54
00.12
COCO
00.28
0
00.87
0
00.65
00.65
PPD
01.41
01.47
0
01.53
ResoNe
Cityscapes
01.07
00.23
00.82
00.56
COCO
00.88
04.07
00.66
00.66
01.55
01.55
PPD
08.46
18.01
07.85
0
12.13
iBOT
Cityscapes
00.37
00.52
09.85
09.85
COCO
0
00.35
0
00.84
00.65
0
PPD
01.97
01.97
01.67
01.67
02.01
SegFormer
Cityscapes
COCO
0
0
0
0
0
PPD
0
0
8ame between these datasets, the variation of performance is low and applying the optimal learning of one dataset on the other does not cause a big drop of performance, so we argue that these values could serve good baseline for new unseen dataset. The higher drops occur when either take the optimal learning rate for PPD or when we Lest an optimal learning rale of Cityscapes or COCO on PPD. This is mainly due to the fact that the support set of PPD significantly smaller than that of the other considered datascts.
A.5
More Individual Factors
A.5.1 Input Resolution Effect.
We explore how input resolution impacts model performance, critical factor given our training setup, Specifically, CLIP and MAE were trained at a resolution of 224x224, while other models were trained with higher resolution of 1024x1024. This discrepancy of CLIP and MAE's positional embeddings. To address this, adapted the positional embeddings matrix for variable resolutions, a common practice for ViT models. Our findings, illustrated in Figure B reveal divergence in performance trends: increasing resolution degrades results for CLIP and MAF. but improves them for DINOv2. This can be attributed to DINOv2's training across multiple resolutions, unlike CLIP and MAE. Consequently, this suggests using 1024x1024 imput images on DINOy2 whereas 224x224 input images CLIP and MAE is fair for performance comparison.
due to the fixed input. size requirements
A.5.2 Registers Effect.
In the study by (Darcet et al. 2023), it is demonstrated that DINOv2, unlike its predecessor DINOv1, presents artifacts within its feature maps. This phenomenon is altributed the presence of tokens with unusually high norms, which are integral to the model's internal computations. To address this issuc, the authors introduced modified version of DINOv2, incorporating additional tokens termed "Registers" during the training phase, which are subsequently omitted during inference. As evidenced Table this

---

Published in Transactions Machine Learning Research (05/2025)
80
70
DINOv2 CLIP
MAE
mloU
60
50
40
30
20
224
512
768
1024
Input Resolution
Figure 3: Impact of the input resolution the obtained mloU.
Table 9: Effect of training with registers on mloU
Linear
LoRA
Citysrapes
COCO
PPD
Cityscapes
COCO
PPD
DINOv2 w/o Registers DINOv2 w/ Registers
48.80:101.88
23.24:+00.88
02.31=01.84
54.35+01.55
28.99++01.33
89.67=006.43
U2.46
27.42 100.72
91.58
54.08 L0U-44
34.02 02.26
102.08
innovative approach significantly enhances DINOv2's performance on the COCO dataset, showcasing potential benefits of Registers during training for FSS tasks.
A.6 Model Comparison Summary
Table 10 summarizes the main components of the different models used in our study, we discuss the importance of each component. (Architecture, Size, Pre-training Dataset, Pre-training Method) in 6.3
A.7
Dataset Comparison Summary
Table D and Figure D provide an overview the three datasets considered in this study. These datasets exhibit varying levels of class diversity. Notably, in the PPD datasot, segmentation masks show significant variation across images, contributing to higher standard deviation in mloU. This variability poses a challenge one-shot learning scenarios, where models must generalize effectively from only two reference images.
A.8 Impact on the Number of Shots
We can see in Fig. that for SVF, the different models scale similarly and that models are still outper- formed by DINOv2 even when more shots are available. We can see Fig, 6 that for DINOv2, the gap between the models tends to be smaller the more we increase the number of shots. Also, full finc-tuning scales better than other methods because we are less prone to overfitting, leading to the conclusion that the better to choose the full inn-tuming method when more shots are available. Yet not crucial since the full fine tuning is time and memory expensive compared to the other methods, and simple linear probing can give good performance with a minor drop in performance.
Table 10: Summary of main model elements.
Architecture
Size
Dataset
Pre-training Method
SAM
ViT
89M
SA-1B 142M image dataset
MAE Prompt Segmentation
DINOv2
VIT
S6M
DINO (SSL with Knowledge Distillation)
IOT
VIT
85M
ImageNet-IK 400M of (image, text) pairs
Masked image modeling knowledge distillation
CLIP
VIT
S6M
image, text embedding alignement
MAE
VIT
86M
ImageNet- 1K
Masking Image reconstruction
FCN-ResNete
CNN
23M
Subset of COCO
Semantic Segmentation
SegFormer
VII
81M
ADE20K
Semantic Segmentation

---

Published in Transactions Machine Learning Research (05/2025)
Cityscapes
COCO
PPD
Figure 4: Examples of images and corresponding segmentation masks from Cityscapes, COCO, and PPD datasets.
80
70
60
mloU
50
40
DINOv2
SAM
CLIP
MAE
30
ResNet iBOT
SegFormer
20
10
Number of shots
Figure 5: Impact of the number of shots mlot, for the SVF method.

---

Published in Transactions Machine Learning Research (05/2025)
Table 11: Overview of the datascts used in our benchmark.
Dataset
Number of Classes
Resolution
Size of Training Set
Cityscapes
19
1024 2048
2,975
COCO
(00000000(((00----)00000 640)
118k
PPD
2
500 x
783
80
70
mloU
60
Lincar SVF
Multilayer
LoRA
Fine-tuning
50
10
Number of shots
Figure 6: Impact of the number of shots on mloU for DINOv2.
A.9 COVID-19 CT Dataset
We conducted additional experiments fourth dataset, medical dataset for COVID-19 CT image segmentation (nttps://www.kar) httpg//e..cc////oooddaaaaaaraaaa
which includes 4 classes. The 1- shot results (mloU), presented below, demonstrate a significant performance drop for Vision Foundation Models (VFMs) trained predominantly with SSL methods natural images when applied to medical images. This gap in performance, caused by the substantial distributional differences between natural and modical images, highlights the advantages of current scmantic sogmentation models, such ResNet-50 and SogFormer, in handling tasks with large domain shifts.
Table 12:
mIoU averaged over our three considered datasets 1-shot semantic segmentation for COVID- 19 CT Dataset. Bold numbers correspond to the best performance across extractors and underline numbers to the best performance across adaptation methods.
Model
Lincar
Multilayer 45.69115. 40
SVF
LoRA
Finetune
SAM
23.20:102.7
32.90
39.28105.29 25
24.49101.38 1.39
DINOv2
34.74+-02.52
10.18-02..3
30.11
34.05+00.76
32.01
CLIP
34.95+001.4
35.50=101117
36.70=02.40
34.71=00.83
26.98+01.66
MAE
23.37:t01.33
24.10++01.5
23.48=01.83
23.92101.75
21.04401.2 .22
FCN-RosNeL50
54.03:001.14 01.277
56.97-00.05
54.71=01.41
57.35
IBOT
23.514:0..19
24.544108.02
21.87+10046
40.28-03.79
21.49
SEGFORMER
44.144:00.22
50.30-101.0
49.35-10.02
47.51101.49
52.25+00.99
A.10
Limitations
While our benchmark introduces novel insights into the adaptation of Vision Foundation Models (VFMs) for few-shot semantic segmentation, several limitations warrant further investigation:
Dataset Representation: Our benchmark relies on limited number of datasets, primacily from urban scenes (Cityscapes), natural images (COCO), and plant phenotyping (PPD). Al- though diverse, this selection does not encompass other important domains, such as satellite imagery or underwater scenes, where few-shot segmentation could be valuable.

---

Published in Transactions Machine Learning Research (05/2025)
The study focuses on specific of pretrained VFMs and adaptation methods. While comprehensive, this excludes polential alternative models, such as specialized medical imaging networks or models pretrained on multimodal datasets, which could perform better in domain-specific settings.
Model Adaptation Scope:
Domain Gap:
The significant drop in performance observed for medical images (COVID- 19 CT dataset) highlights that VFMs trained on natural images are not inherently robust to large domain shifts. Addressing this limitation may require dedicated pretraining or domain adaptation strategies, which outside the scope of this study.
The absence of energy efficiency analysis is limitation of work. However, it is evident that methods like LoRA and SVF are considerably more efficient than full fine-tuning, as they require fewer computational resources and less processing time while maintaining strong performance. A more detailed study on the energy consumplion of different. adaptation techniques would be valuable for future research.
Energy Efficiency:
These limitations present opportunities for future research to enhance the applicability and robustness of VFMs in few-shot semantic segmentation, particularly in challenging, real-world scenarios.
A.11 Negative Societal Impact
VFMs are trained on carefully curated large-scale datasets designed to exclude harmful content and private information, mitigating privacy risks a priori. These datasets are generally less biased than traditional datasets, helping reduce datasct-induced biases in model predictions. However, potential biases and harmful content may still arise, necessitating continuous efforts in debiasing and content filtering. Additionally, while the benchmark we developed has minimal potential for direct malicious applications, the models used in our research could still reflect biases from their pretraining adaptation datasets. Furthermore, by streamlining semantic segmentation and enabling elfective model training with limited labeled data, this approach could inadvertently lower the barrier for applications in surveillance other privacy-sensitive domains. While this accessibility fosters innovation and practical deployment, also highlights the importance of responsible use and potential regulatory considerations in sensitive contexts.
A.12 Object Size Effect mIOU
We analyzed the relationship between object size and segmentation performance by plotting the mloU for the Car class against its pixel area (see Figure 7 Our results show that larger cars tend to achieve higher mIoU scores, while smaller cars exhibit lower mIoU and higher variance.
This trend may not solely reflect better model performance but could be due to geometric bias: larger objects have more imner pixels, which are casier to sogment, whereas smaller objects have higher proportion of boundary pixels, which are harder to segment accurately.
These findings suggest that object size influences segmentation performance, but the apparent advantage for larger objects may stem from intrinsic biases rather than true model generalization.

---

Published in Transactions Machine Learning Research (05/2025)
100
80
00
mloU for Class 'Car'
60
40
20
0 22
0.2
0.4
0.6
0.8
1.2
1.4
1.6
Car Area in Image
-10
Figure 7: Impact of Car Area mloU for the 'Car' class using DINOv2.


Reviews:
[{"review_id": "7bMcJWSD5q", "reviewer": "Reviewer_B7hM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**. The experiments are sufficient and provide empirical evidences for future few-shot segmentation works, which is a good point.\n\n**Weakness**. The paper needs to be more accurate on claiming contributions. Motivations are not strong enough. A few experiments should be involved as well.\n\nFor weaknesses, please refer to **Requested Changes** part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission contributions are summarized as follows.\n\n**Sufficient Experimental Explorations**. The submission provides sufficient empirical results, exploring potentials of multiple foundation models on few-shot segmentation, including CLIP, SAM, DINO, MAE, iBOT. Such explorations could provide a good reference to the future few-shot segmentation studies.\n\n**New Experimental Setups**. Existing few-shot segmentation approaches mainly benchmark on Pascal-5i or COCO-20i. While, this paper crafts few-shot setups on Cityscapes and PPD, which may provide new testbed for FSS that are more challenging.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Motivation\n**(1-A)** The paper made a claim as follows: \"The few-shot examples in our approach are fully labeled, i.e., masks are not necessarily binary. This aspect aligns more closely with contemporary industrial requirements, and can be used with any semantic segmentation\ndataset.\" Authors are suggested to state more clearly on why binary masks are not suited to industrial scenarios, given the fact that most influential papers in this domain apply binary masks.   \n**(1-B)** The submission made a claim as follows: \"Class balance is not assumed, variations are permitted following the distribution of classes within the selected datasets, better reflecting real-world scenarios.\" The class imbalance is undoubtedly not expected in training distribution, which is well-known as long-tail problem. However, for a benchmark, balance is expected to perform a comprehensive evaluation. While this is more natural, authors seem considering \"long-tail\" as a good thing for benchmarking, which is quite confusing.\n\n### Experiments\n**(2-A)** Given that Stable Diffusion (SD) is also an important vision foundation models and a few existing studies have explored on using SD to perform few-shot segmentation [1-2], authors are suggested to include related experiments and discussions in their submission.  \n**(2-B)** The proposed setup has its own significance. One aspect is that data like Cityscapes are diverse in object size. For example, a close-up of car compared to a car far away from camera, where instances of car vary in resolution (a.k.a, size). This might be a good perspective to make this paper more solid. Authors are suggested to include such case studies in this paper.  \n**(2-C)** Notice that authors use resolution of 224 for CLIP and MAE, which may make the few-shot segmentation on Cityscapes quite challenging, as the typical input resolution for such data is 1024. This may lead to false conclusion for design choices when performing few-shot segmentation. Authors are suggested to use OpenCLIP ConvNeXT-L, with input resolution of 1120px as presented in this paper [3].\n\n### Citations\n\n[1] DifFSS: Diffusion Model for Few-Shot Semantic Segmentation.  \n[2] Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation. NeurIPS 2024.  \n[3] Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP. NeurIPS 2023.", "broader_impact_concerns": "No ethical concern for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a1QTjAlPQW", "reviewer": "Reviewer_dmLk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+ The study is well-motivated, given the rising adoption of VFMs in vision tasks.\n\n+ The paper systematically compares five VFMs and five adaptation techniques across three datasets (COCO, Cityscapes, PPD), providing a thorough empirical analysis.\n\n+ Unlike traditional FSS benchmarks that focus on in-domain generalization, this benchmark emphasizes adaptation to new distributions while addressing issues like class imbalance and multi-class labeling.\n\n+ The finding that self-supervised models (DINOv2) outperform segmentation-focused models (SAM, CLIP) is non-trivial and challenges common assumptions about FSS.\n\n+ Ablation studies: The authors provide thorough ablation studies to investigate the impact of dataset size, adaptation strategy, and feature extractor depth.\n\n**Weaknesses**\n\n- Benchmark novelty is somewhat incremental. While the benchmark improves upon existing ones (e.g., PASCAL-5i, COCO-20i), it inherits many design choices from prior FSS benchmarks, making it an evolution rather than a fundamental shift.\n\n- Missing large-scale noisy dataset evaluation: The benchmark is tested on COCO, Cityscapes, and PPD, but does not evaluate generalization on large-scale noisy datasets (e.g., LAION-400M, WIT-400M), which are common in real-world applications.\n\n- Lack of discussion on pretraining dataset overlap: Some VFMs (e.g., CLIP, DINOv2) are trained on massive web-scale datasets. How does the presence of test classes in these pretraining datasets affect the results? This is not explicitly addressed.\n\n- Lack of discussion on meta-learning approaches: Few-shot learning often leverages meta-learning strategies (e.g., ProtoNets, MAML, MetaOptNet). Why were these omitted from the adaptation methods?\n\nAlso I believe that implementation details need more clarity:\n\n- How were learning rates set across layers in full fine-tuning?\n- Were batch norm statistics frozen or updated?\n- How does LoRA compare in terms of runtime vs. accuracy tradeoff?\n- PPD dataset results seem unusually high: DINOv2 achieves ~90% mIoU on PPD, significantly higher than on COCO or Cityscapes. Is this due to dataset bias or overfitting? A discussion is needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new benchmark for evaluating Few-Shot Semantic Segmentation (FSS) performance in the context of Vision Foundation Models (VFMs). The benchmark is designed to overcome limitations in prior FSS benchmarks by:\n\n- Incorporating fully labeled few-shot examples without assuming class balance.\n- Evaluating five foundation models (DINOv2, SAM, CLIP, MAE, iBOT) with five adaptation strategies (linear probing, multilayer feature extraction, singular value fine-tuning (SVF), LoRA, and full fine-tuning).\n- Analyzing how different pretraining strategies (supervised vs. self-supervised) impact FSS performance.\n- Demonstrating that self-supervised models (DINOv2) outperform models explicitly designed for segmentation.\n\nThe findings suggest that feature extractor quality is more important than segmentation-specific design and that parameter-efficient fine-tuning methods like LoRA and SVF can match the performance of full fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe that major revisions required:\n\n- Clarify the potential data leakage issue: Discuss whether pretraining datasets contain test classes, and how this affects model performance.\n- Justify the exclusion of meta-learning approaches: Many few-shot learning benchmarks include methods like ProtoNets and MAML. At least discuss why they were not considered.\n- Expand evaluation to large-scale noisy datasets: Would the proposed benchmark still be reliable on noisier datasets like LAION-400M?\n- Investigate why PPD results are so high: Provide a comparison of dataset complexity to explain why PPD leads to much higher mIoU scores.\n\nI also suggest the authors provide additional implementation details:\n- How were fine-tuning hyperparameters set?\n- Did learning rates vary across layers?\n- Were batch normalization statistics frozen?\n- Expand discussion on adaptation vs. feature extractor quality: The paper finds that DINOv2 outperforms segmentation-specific models. Could this be due to DINOv2’s pretraining dataset rather than its architecture?\nInclude error bars or confidence intervals in Figure 1 to indicate result stability across runs.", "broader_impact_concerns": "- If VFMs have been trained on biased datasets, their performance in few-shot segmentation may carry dataset-induced biases. A discussion on fairness implications should be included.\n\n- If foundation models memorize specific training data, they could pose privacy risks when used in FSS. A short discussion on data leakage risks should be added.\n\n- Many VFMs (e.g., CLIP, DINOv2) are trained on web-scale datasets without explicit human oversight. How do these models handle harmful content or biases in their few-shot predictions?\n\n- Full fine-tuning is computationally expensive compared to LoRA or SVF. The paper should briefly discuss energy efficiency as a practical consideration.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jafGHLRLuc", "reviewer": "Reviewer_gkw8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is well-written and easy to follow.\n2. The paper proposes a new benchmark and introduces the process of constructing the benchmark. Unlike the previous FSS setting that only supports single-class binary masks, this benchmark supports multi-class masks.\n3. The experiments are very comprehensive and detailed. The authors conducted experiments on multiple VFMs, including SAM, MAE, DINOv2, CLIP, and tried various adaptation methods, including linear probing, PEFT, full fine-tuning. They also explored the impact of other factors such as model size.\n\nWeaknesses\n1. Regarding the task setting, it seems that Generalized Few-shot Semantic Segmentation[1] (GFSS) can also support multi-class, and it needs to make a clearer comparison between the setting of this paper and traditional FSS, GFSS in the task formulation.\n[1] Generalized few-shot semantic segmentation. CVPR 2022\n\n2. Recently, there have been some works using VFM for FSS, and it would be better to have a more detailed discussion of these works in the related work.\nFor example, some works use a single VFM such as CLIP, SAM, DINOv2 for FSS, some works combine multiple VFMs for FSS, and some works explore the use of generative VFMs(SD) for FSS.\n\n[2] Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyond (based on CLIP)\n\n[3] Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching\n(DINOv2+SAM) ICLR 2024\n\n[4] VRP-SAM: SAM with visual reference prompt. (based on SAM) CVPR2024\n\n[5] Segic: Unleashing the emergent correspondence for in-context segmentation. (based on DINOV2) ECCV 2024\n\n[6] Unleashing the potential of the diffusion model in few-shot semantic segmentation. (based on SD) Neurips 2024\n\n[7] Bridge the Points: Graph-based Few-shot Segment Anything Semantically. (DINOv2+SAM) Neurips 2024\n\n3. The explanation of the adaptation methods is not clear enough. For example, about LORA, \"Our procedure begins with training the decoder using the linear method and then introduces LoRA adapters to the encoder. Subsequently, we fine-tune both the adapters and decoder parameters.\" Is this means that there are three stages, the first stage is linear probing, the second stage is LoRA adapters to the encoder, and the third stage is fine-tune both the adapters and decoder parameters. Is it reasonable to divide it into three stages? Is it feasible to only perform the third stage?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new benchmark for FSS to evaluate the performance of VFM in FSS tasks. The authors conducted detailed experimental analysis around multiple VFMs (SAM, MAE, DINOv2, CLIP) using this benchmark and tried various adaptation methods, including linear probing, PEFT, full fine-tuning. The authors found that DINOv2 performed the best on this benchmark, even outperforming segmentation-based models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see Weaknesses section for the requested changes.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new benchmark for the task of few-shot semantic segmentation with foundation models. The reviewers appreciated the writing of the paper, new benchmark, sufficient experimental evaluations and analysis, and also the findings of the paper.  The responses of the authors during the rebuttal period clarified most of the concerns raised by the reviewers. But there are still few important concerns that need to be satisfactorily addressed before the paper can be accepted as is. Specifically, the authors need to satisfactorily justify the claims regarding the full labelling in FSS scenario and also benchmarking using imbalanced data. Also, the authors need to clearly highlight the novelty of the proposed benchmark over the existing benchmarks. The final acceptance of the paper depends on satisfactorily addressing these remaining concerns.
Decision: Accept (poster)