Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
Authors: Yabo Liu
Waikeung Wong
Chengliang Liu
Xiaoling Luo
Yong Xu
Jinghua Wang
Venue: ICML 2025 poster (2025)
Keywords: computer vision
transfer learning
domain adaptation
Primary area: applications->computer_vision
TL;DR: We propose a mutual learning framework with dual collaborative networks for source-free SAM adaptation, using dynamic role-switching and feature-based reliability to enhance generalization across target domains.

Abstract: Segment Anything Model (SAM) has demonstrated remarkable zero-shot segmentation capabilities across various visual tasks. However, its performance degrades significantly when deployed in new target domains with substantial distribution shifts. While existing self-training methods based on fixed teacher-student architectures have shown improvements, they struggle to ensure that the teacher network consistently outperforms the student under severe domain shifts. To address this limitation, we propose a novel Collaborative Mutual Learning Framework for source-free SAM adaptation, leveraging dual-networks in a dynamic and cooperative manner. Unlike fixed teacher-student paradigms, our method dynamically assigns the teacher and student roles by evaluating the reliability of each collaborative network in each training iteration. Our framework incorporates a dynamic mutual learning mechanism with three key components: a direct alignment loss for knowledge transfer, a reverse distillation loss to encourage diversity, and a triplet relationship loss to refine feature representations. These components enhance the adaptation capabilities of the collaborative networks, enabling them to generalize effectively to target domains while preserving their pre-trained knowledge. Extensive experiments on diverse target domains demonstrate that our proposed framework achieves state-of-the-art adaptation performance.

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Forrce-Free Domain Transfer
Yabo Liu Waikeung Wong?
Chengliang Liu Xiaoling Luo*
Yong Xu' Jinghua Wang'
Abstract
1. Introduction
The development of deep learning technology has brought remarkable breakthroughs the field of computer vision (He al., 2016; Liu et al., 2022; Redmon al., 2016; Car- ion et al., 2020; Liu et al., 2025; Sun et al., 2024; Huang et al., 2022b;a). Among these advancements, progress in pre-training techniques has been particularly remarkable, with models like Segment Anything Model (SAM) (Kirillov et al., 2023) demonstrating powerful capabilities across var- ious visual perception tusks(Oqual) et al., 2023; Bai et al., 2024). Taking advantage of its pre-training procedure on extensive and diverse datasets, SAM demonstrates remark- able generalization capabilities and delivers high-quality segmentation results. However, SAM often suffers signif- icant performance degradation when deployed in a target domain (i.e., the testing data) that shifts significantly from the source domain (i.e., the training data) (Zhang al., 2024; Chen et al., 2023; 2024). These domain shifts arise due differences in visual characteristics, object distributions, environmental conditions between the pre-training datasets and the target domains. For example, SAM fails to perform well in professional applications, such as medical imaging, camouflaged object detection, and robotic vision. This high- lights the urgent need to adapt SAM to target domains for improved usability in real-world applications.
Segment Anything Model (SAM) has demon- strated remarkable zero-shot segmentation capa- bilities across various visual tasks. However, performance degrades significantly when de- ployed in new target domains with substantial dis- tribution shifts. While existing self-training meth- ods based on fixed teacher-student architectures have shown improvements, they struggle to en- sure that the teacher network consistently outper- forms the student under severe domain shifts. To address this limitation, we propose a novel Collab- orative Mutual Learning Framework for source- free SAM adaptation, leveraging dual-networks in dynamic and cooperative manner. Unlike fixed teacher-student paradigms, our method dynami- cally assigns the teacher and student roles by eval- uating the reliability of each collaborative network in each training iteration. Our framework incorpo- rates dynamic mutual learning mechanism with three key components: direct alignment loss for knowledge transfer, reverse distillation loss to encourage diversity, and a triplet relationship loss to refine feature representations. These compo- nents enhance the adaptation capabilities of the collaborative networks, enabling them to general- ize effectively to target domains while preserving their pre-trained knowledge. Extensive experi- ments on diverse target domains demonstrate that our proposed framework achieves state-of-the-art adaptation performance. Our code is accessible at https///jitbb.commyyyblluunnnunu
It is infeasible to adapt SAM in the traditional framework of unsupervised domain adapation (UDA), as the large scale pre-training dataset could introduce substantial computa- tional burden and make the adaption procedure both ineffi- cient and unscalable. In addition, the source domain data is not always accessible due to privacy concerns, storage limitations, data-sharing restrictions. Source-free domain adaptation (SFDA) has emerged promising solution to address these issues. SFDA enables the adaptation of pre-trained model using only unlabeled target domain data, making it particularly applicable to sensitive fields such as medical imaging and autonomous driving. SFDA of SAM requires the model to learn robust representations of the target domain while avoiding catastrophic forgetting of its pre-trained generalization capabilities.
Harbin Institute of Technology, Shenzhen 'Laboratory for Ar- tifcial Intelligence in Design, Hong Kong "School of Fashion and Textiles, Hong Kong Polytechnic University "Shenzhen University. Correspondence to: Waikeung Wong <calvinwong @ aidlab.hk>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A common approach to SFDA is self-training through teacher-student knowledge distillation. In this paradigm,

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
teacher network generates pseudo-labels for the target domain data, which are then used to supervise the training of a student network (Zhang et al., 2024). This approach assumes that the teacher consistently provides more reli- able predictions than the student. However, this assump- tion often does not hold in practice, especially when the model is deployed a new target domain. Our experiments and analysis reveal that the student network may surpass the teacher network in capturing domain-specific features during training. The fixed knowledge transfer direction in the teacher-student frameworks limits the potential of self-training, ultimately leading to suboptimal performance.
tive networks based on their reliability and encourages mutual learning during training.
We introduce the direct alignment, reverse distillation, and triplet relationship losses to ensure robust and effi- cient adaptation.
Extensive experiments demonstrate the effectiveness of our framework in adapting SAM to challenging tar- get domains, including medical imaging, camouflaged object detection, and robotic vision.
2. Related work
To address these limitations, we propose a novel Collabo- rative Mutual Learning framework that replaces the fixed teacher-student paradigm with a more flexible and dynamic one, inspired by (Zhou et al., 2023a). Two networks col- laborate in equal roles in the proposed method, and they dynamically learn from each other rather than adhere to static teacher or student role. This bidirectional knowledge exchange allows both networks to leverage the strengths of the other to achieve superior performance in the target domain.
2.1. Image Segmentation
Image segmentation is a fundamental problem in the field of visual perception (Cao et al., 2023; Zhou et al., 2023b; Jin et al., 2023; Wu et al., 2024; Zhao & Tao, 2023). Over the years, research in image segmentation has evolved from small, task-specific models to large foundational models. Mask R-CNN (He et al., 2017) is a widely used two-stage instance segmentation model that first detects the regions of foreground instances and then generates their segmenta- tion masks. In contrast, YOLACT (Bolya al., 2019) is single-stage approach that directly outputs segmentation masks from image feature maps. With advancements in transformers and pre-training techniques, large visual mod- els have garnered significant attention from researchers. The Segment Anything Model (SAM) (Kirillov et al., 2023) is the first visual foundational model to achieve remarkable segmentation performance. SAM is capable of handling various types of prompts, such as bounding boxes, points, polygons, and text descriptions, and produces fine-grained segmentation masks. By pre-training on massive and di- verse datasets, SAM exhibits outstanding zero-shot gener- alization ability. making it applicable to a wide range of tasks. Similarly, DINO v2 (Oquab et al., 2023) leverages self-supervised learning to acquire knowledge from large datasets and demonstrates significant performance improve- ments on many downstream tasks. These breakthroughs highlight the growing potential of pre-trained foundational models in advancing segmentation tasks.
A key challenge in implementing mutual learning lies on the determination of the relative reliability of the networks dur- ing training. To address this, we introduce dynamic role assignment mechanism based on a wwwwledge-rreeevinng metric. This metric compares the foreground features of each network with those extracted from the original pre- trained SAM model, and tells how much the pre-trained knowledge is preserved. The network that preserves more pre-trained knowledge is considered more reliable and thus temporarily assumes the guiding role (i.e., teacher), while the other acts as the learner (i.e,, student). This dynamic role assignment ensures effective collaboration and adaptability throughout the training process. addition, we introduce the direct knowledge distillation to align the predictions of the student network with those of the teacher, enabling the transfer of domain-specific knowledge at each training iteration. We also employ the reverse knowledge distillation loss to encourage diversity by slightly deviating the teacher from the student, preventing both networks from collapsing into identical representations. In this way, we achieve a balance between exploration (via reverse distillation) and exploitation (via direct alignment) and ensure the networks maintain complementary strengths. To further improve fea- ture robustness, we propose triplet relationship loss that models the relationships between the teacher, student, and the pre-trained SAM.
2.2. Domain Adaptation
Domain adaptation (DA) is a critical technique for address- ing domain shifts between the source (training) and target (testing) domains. Many methods have been proposed to tackle this problem effectively (Liu et al., 2023b; Chen et al., 2018; Liu et al., 2024b; Chen et al., 2022; Liu et al., 2024a). DANN (Ganin & Lempitsky, 2015) employs generative ad- versarial learning with a Gradient Reversal Layer to achieve significant improvements in unsupervised domain adapta- tion. Recent methods like SIGMA (Li et al., 2022) and CIGAR (Liu et al., 2023a) transform image features into
In summary, our contributions are as follows:
We propose a Collaborative Mutual Learning frame- work for the source-free domain adaptation of SAM.
Our framework dynamically assigns roles to collabora-
2

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
graph space and align category-level graph node features across domains. TENT (Wang al., 2020) minimizes the entropy loss of model predictions during testing to handle domain shifts. Similarly, MLFA (Liu et al., 2024a) ad- dresses object detection problems by modeling and aligning the feature distributions of the source and target domains. In the context of source-free domain adaptation, SHOT (Liang et al., 2021) utilizes self-training with pseudo-labels generated for target domain data, significantly improving performance without requiring source data. DePT (Gao et al., 2022) adopts prompt learning to address source-free domain adaptation by fine-tuning only the visual prompts from the source domain. WDASS (Das et al., 2023) pro- poses a weakly supervised method to tackle unsupervised domain adaptation problems. Wang et al. propose the Con- tinual Test-Time Domain Adaptation (CTTA) framework (Wang et al., 2022), which incrementally adapts source model to continually evolving target domains. Note (Gong et al., 2022) introduces instance-aware batch normalization to address the normalization challenges posed by new target domain samples. RMT (Döbler et al., 2023) employs sym- metric cross-entropy loss within a mean teacher framework to tackle the CTTA problem effectively. For SAM adap- tation, WeSAM (Zhang et al., 2024) introduces a weakly supervised self-training method. However, it employs a fixed teacher-studen architecture, which limits its ability to adapt effectively in challenging target domains.
Each network brings a unique perspective, allowing them to complement one another and collaboratively explore the characteristics of the target domains. 2) Dynamic Role Assignment. Unlike fixed teacher-studen architectures, our method dynamically determines which network acts as the teacher and student at each iteration. This decision is based on the reliability of their real-time predictions. The more reliable network guides the less reliable one, ensuring that the learning process remains adaptive and responsive the changing reliability of predictions. By leveraging these ad- vantages, our mutual learning framework achieves a more effective and robust source-free adaptation of SAM.
4. Preliminary: Segment Anything Model
The Segment Anything Model (SAM) (Kirillov et al., 2023) is a state-of-the-ard foundational model designed for vi- sual perception tasks. Its architecture follows an encoder- decoder framework and comprises three key components: an image encoder Eing. a prompt encoder Eprampt, and a mask decoder Dmack. Given an input image I and asso- ciated prompts Prompt (such as bounding boxes, points, masks, or textual descriptions), the image encoder gen- erates an image feature map F, and the prompt encoder produces prompt embedding P: F =
Eing (1), P E Eprompt (Prompt). The mask decoder then processes both F and P to produce a segmentation score map S: $ = Dmskk (F, P). Finally, a binarization threshold is applied to the score map S to produce the segmentation mask M. Although SAM demonstrates strong performance across various segmentation tasks, its direct application to target domains with significant distribution shifts often results in suboptimal performance (Zhang et al., 2024; Chen et al., 2023; 2024).
3. Motivation
Existing single-model adaptation methods often struggle to balance the preservation of pre-trained knowledge with the adaptation to sprget-specific knowledge. Recent approaches using knowledge distillation, such as teacher-student net- works (Zhang et al., 2024), have shown potential to address this challenge. This method has a significant limitation: the roles of teacher and student networks are fixed through- out the training process, based on the assumption that the teacher network consistently provides more reliable predic- tions than the student network. However, this assumption is often unrealistic when adapting to entirely different target domains. In such cases, both networks may fail to provide accurate predictions at different stages of training, limiting the effectiveness of the adaptation process.
5. Method
In this section, we present our framework for source-free domain adaptation of SAM, which utilizes Collaborative Mutual Learning strategy. We first describe the overall archi- tecture in Sec. 5.1. Then, we detail the Mutual Learning and Dynamic Role Assignment mechanism in Sec. 5.2. Finally, we introduce the optimization objectives in Sec. 5.3.
5.1. Architecture
Our Approach: Collaborative Mutual Learning. To ad- dress these limitations, we propose novel framework based on mutual learning, where two independent networks collab- orate and dynamically learn from each other. This approach overcomes the restrictions of fixed teacher-student meth- ods and offers several key advantages: 1) Collaborative Knowledge Sharing. Mutual learning enables two net- works exchange information during training, leading to more comprehensive understanding of the target domain.
The proposed framework leverages a dual-network architec- ture, as illustrated in Fig. 1. It consists of a frozen SAM and two independent collaborative image encoders, denoted as Eing and 13 Each collaborative encoder is built by extending the SAM image encoder with LoRA-based fine- tuning module for efficient adaptation (Hu et al., 2022). To encourage diversity and avoid symmetrical failures, the collaborative encoders are initialized with slight parameter
3

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
Fa
Aug.
a LoRA
M
Dynamic Assigument
E img Image Encoder
Lyyverse
Dask
B LoRA
Teacher
fa
Mask Decoder
Leriplet
fp
box
points
bygon
student
Idiiect
Eprompt Prompt Encoder
M
Fg
Figure 1. Overview of the proposed Collaborative Mutual Learning framework for source-free domain adaptation of SAM. It consists of the pre-trained SAM and collaborative image encoders ( (Eing and E. Each collaborative image encoder is built from the original SAM image encoder and a LoRA module. Our proposed mutual learning strategy dynamically assigns the teacher and student roles to the cooperative networks during each training iteration. We optimize these cooperative networks using the direct distillation loss, the reverse distillation loss, and the triplet relationship loss.
perturbations from the pre-trained SAM. Specifically, the collaborative encoders are defined as:
By fine-tuning only the collaborative image encoders via LoRA, the framework achieves efficient and domain- specific adaptation while preserving the rich representations of the pre-trained SAM. At the end of training, the better- performing network is selected as the adapted target SAM model.
Ea18 perturb(Esmg) + A Ea/B smg
(1)
iig
where AE represents the low-rank weights introduced by LoRA and perturb(.) denotes the parameter perturba- tion operation. This design enables efficient fine-tuning with minimal additional parameters while retaining the knowl- edge of the pre-trained SAM image encoder.
5.2. Mutual Learning and Dynamic Role Assignment
The core innovation of our framework is the mutual learning mechanism, which dynamically assigns the roles of teacher (T) and student (S) to the two collaborative SAM networks (SAM,, and SAM,). Different from the fixed-role knowl- edge distillation, this dynamic role assignment allows the networks to guide each other alternatively.
During training, each collaborative encoder processes dif- ferently augmented view of the input image I, denoted as Adng and 1 1AA The image augmentations (e.g., random color, brightness, contrast, shadows) expose the collabora- tive encoders to diverse visual patterns, encouraging them to learn complementary representations of the target do- main. The resulting feature maps are: Fa/8 = Emmg (1a/g) ) where Fa Fa/B E hxwxe and h, w, and e denote the height, width, and channel dimensions of the feature map.
A. Reliability Estimation
In the adaptation process, the pre-trained models often for- get their knowledge due to overfitting. This may degrade the reliability of the models. Normally, more knowledge pre- served means higher reliability (Zhang et al., 2024). To this end, we extract the foreground representations from the im- age feature maps and corresponding pseudo-segmentation masks of two collaborative networks (SAM,/a) and a ref- erence network (SAM,, original pre-traine pre-trained SAM). Given the segmentation score map Sa/3/7 and image feature map Fa/8/7 generated by these three networks, the foreground representation fa/3/7 is computed the mean feature tors of pixels within the predicted foreground region:
We adopt the same method as WeSAM (Zhang et al., 2024) to generate prompts (Prompt). Specifically, five positive points within the ground truth mask and five negative points outside are randomly selected point prompts. Polygon prompts generated by fitting coarse polygons around the ground truth masks. The shared mask decoder integrates image features and prompt embeddings to produce seg- mentation score map: Sa/3 = [Damood[Dmaae. (F,,, (Fa/3. P)]. The score map Sa/a a is binarized with a threshold T. to pro- duce the final segmentation mask: Ma/3 0, w These components form two independent collaborative SAM net- works:
Fij 1(5 > T) a/8/7
fa/8/7 =
(3)
Σny 1(5
> T)
SAMajp = Pmask 1 (T42g) Fprompe (ecceempperrrott)) (Prompt)].
where F+s 0/3/1
I represents the feature vector at pixel

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
(i,j), and I(-) is an indicator function that identifies fore- ground pixels based on the threshold T. Given foreground, we assume that the cooperative network that preserves more knowledge produces a more similar feature map to the refer- ence network. So we calculate the cosine similarity between the foreground representation of each collaborative network So or (s) and that of the pre-trained reference SAM (f)):
Reverse distillation
Leeverse
teacher feature
Direct Alignment
Liirect
Sa -ref al [I
S3 -Jref
Triplet Relationship
Pripptt
sima
sim I/ [|
(4)
student feature
Prreserve
Although So remains frozen throughout, it encodes rich and reliable pre- pre-trained knowledge learned from large-scale datasets. The similarity to fa measures how much pre- trained knowledge is preserved during adaptation. Collabo- rative Networks with higher similarity to fa considered more reliable, as their updates are more cautious and less prone to overfitting to noisy or domain-speciffe features. So we consider the network with the higher similarity score to be the teacher, while the other becomes the student:
Lpreseeve
reference feature
Figure 2. The instruction of the loss components in our proposed collaborative mutual learning method.
between two collaborative networks. Different from direct alignment, this loss encourages the teacher network to devi- ate slightly from the predictions of the student, preventing both networks from collapsing into identical representations. The loss also expands the solution space of the teacher net- work, avoiding local minima caused by over-alignment with the student. The reverse distillation loss is defined as:
(T.S) = (a,3) if sima > simg: (B,a) otherwise.
This dynamic assignment mechanism ensures that the roles are updated adaptively in each iteration based on the relia- bility of networks, enabling them to learn from each other
LSAM(( : S:) (T, = (3,o), (8)
effectively.
Liveerse
LSAM Sa. Sa) (T,S) = (a, B).
B. Role Assignment Distillation
Once the teacher and student roles are assigned, knowledge distillation performed to share information between the collaborative networks while preserving their complemen- tary strengths. This is achieved through the direct alignment, the reverse distillation, and the triplet relationship losses.
Similar to (Zhang et al., 2024), to prevent catastrophic for- getting of the pre-trained SAM, we also constrain the out- puts of both collaborative networks to remain close to the reference predictions of SAM. This is achieved by minimiz- ing the SAM loss between each collaborative network and the reference:
Direct A Alignment via Knowledge Distillation. The stu- dent network is trained to align its predictions with those of the teacher network, enabling the transfer of task- aask-specffic knowledge. The direct alignment loss minimizes the differ- ence between their segmentation score maps:
[SAMI( Saiae S.). (T,S) = (3, a), CSAM((E Safe. S.) (T,S) = (a, 9),
L
TTS preeercee
Triplet Relationship Loss. To further enhance the robust- ness of feature learning, we employ triplet relationship loss to model the relationships between the teacher, student, and the reference foreground features. During each training iteration, the foreground representations of the teacher (fr). student (fs), and reference (f-,) are extracted following Eq. (3). The triplet relationship loss defined as:
LSAM(x Sa Sa) [sAM(( Sa,y: S.) (T,S) = (a.
(T,S) (3)
Ldirect
(6)
Similarly to (Zhang et al., 2024) and (Carion et al., 2020),
the SAM loss AMAM consists the focal loss Focal (Ross & Dollár, 2017) and the Dice loss Dice (Milletari et al., 2016):
LSAM(E: (z. a) = Focal(2, de)ach(y)) + Dice(æ, detcchy)))))) (7)
Leriptet max(0, Ilfs f-1 llft fa1 + +), (10)
The detach operation detach(-) ensures that the target score map (from the teacher) does not participate in backpropaga- tion, providing stable supervision for the student network.
where 8 > 0 is a margin hyperparameter. This loss sures that the teacher features remain closer to the reference features while pushing the student features further away. This loss can maintain the feature consistency across the pre-trained SAM, teacher, and student networks. Prevent- ing excessive divergence and overfitting ensures meaningful
Reverse Distillation for Network Diversity. Inspired by (Zhou et al., 2023a; 2022), we introduce a reverse distil- lation loss to avoid over-convergence and ensure diversity
5

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
feature relationships and works synergistically with reverse distillation and direct alignment losses to enable robust opti- mization.
Kvasir-SEG. Kvasir-SEG is medical image segmentation dataset containing a large number of gastrointestinal en- doscopy images. It provides pixel-level segmentation masks for diseased areas, such as gastrointestinal polyps. This dataset is used as the target domain for medical-specific segmentation tasks.
5.3. Model Optimization
The total optimization loss is defined as:
CAMO. CAMO is a dataset for camouflaged object detec- tion. It primarily contains images of camouflaged animals and artificially hidden objects. The dataset includes 1,000 images with pixel-level segmentation masks. We employ CAMO to evaluate the adaptation performance when trans- ferring from natural images to camouflaged scenarios.
Liote! Ldtrent + Aneserrabbreerree + Atripted
(11)
+ l.preserve + (1 + Apreserve) I T. "preseruer
where Areverse. Apresercve; and Atriplet are weights of re- verse distillation, knowledge preserve, and triplet relation- ship losses, respectively. Areverse is set to smaller value, while Apreserve exerts a larger weight on Lpreserve to im- prove the stability of teacher network training. As shown in Fig. 2, the gradients from I.distill, Lypeeerse and Teriplet encourage convergence toward complementary yet distinct networks, while Lpreserve preserve pre-trat rreitraiedd knowledge of original SAM.
CODIOK. CODIOK widely used dataset for camou- flaged object detection. It contains 10,000 images catego- rized into five super-categories and 69 sub-categories. The camouflaged objects include animals, plants, artificial ob- jects, and more. This dataset provides diverse evaluation of methods for detecting camouflaged objects.
OCID. OCID is a dataset designed for object detection and segmentation in indoor scenes, with focus on recogniz- ing objects in high-density stacking scenarios. The images are collected from various indoor environments simulate object stacking and occlusion challenges encountered dur- ing robotic operations. OCID contains over 20,000 images across 89 categories.
6. Experiment
6.1. Datasets and Evaluation
To evaluate the effectiveness of our proposed method, we conduct extensive experiments on four types of downstream segmentation tasks, including natural images, medical im- ages, camouflaged images, and robotic images. These tasks exhibit varying degrees of domain shift from the pre-trained dataset of SAM. We used the Mean Intersection over Union (mloU) as the primary metric to assess adaptation perfor- mance on specific target domains. The results of all compar- ative methods were obtained from (Zhang et al., 2024). To provide a more comprehensive evaluation of each adapta- tion method, we calculated the average mloU, which reflects the mean performance of the method across three types of prompts (box, points, and polygon) used for image segmen- tation. Additionally, we introduced the GAIN metric to quantify the performance improvement of cach adaptation method over the original SAM (source model). There are six benchmark datasets employed in our experiments: COCO
6.2. Implementation Details
The collaborative encoder E is initialized using the pre- trained SAM image encoder with a ViT-B backbone. To prevent symmetrical failures in the collaborative image en- coders, we set the amplitude of the normally distributed random noise perturb to 0.001. During the adaptation process, we freeze the SAM model and utilize LoRA to fine-tune only a small subset of parameters, enabling effi- cient adaptation. We use the Adaptive Moment Estimation (Adam) optimizer (Kingma, 2014) with a learning rate of 0.0001 and weight decay of 0.0001. The model is trained for 20,000 iterations each target domain, with batch size of 1. The hyperparameters Areverse and Xpreserve in the total optimization loss are set to 0.1. To balance the train- ing between the teacher and student networks, Atriptes is set to a small value of 0.01. All experiments are conducted using four NVIDIA A100 GPUs.
(Lin et al., 2014), Pascal VOC (Everingham et al., 2015),
Kvasir-SEG (Jha et al., 2020), CAMO (Le et al., 2019),
COD10K (Fan et al., 2020), and OCID (Suchi et al., 2019).
COCO. COCO is a widely used dataset in computer vision tasks, including object detection, segmentation, keypoint detection, and image captioning. It contains over 330,000 images and 200,000 labeled images. It provides pixel-level instance segmentation labels of 80 categories.
6.3. Comparison with State-Of-The-Are
A Adatation to natural target domains. Tab. I presents the comparison results on downstream target domains with natural images. Our method achieves an average mloU of 71.50 COCO and 76.45 on Pascal VOC, outperforming existing methods. Compared with WeSAM, WDASS, and SHOT, our approach improves the average mIoU on COCO
Pascal VOC. Pascal VOC is dataset composed of natural images, featuring 20 object categories with corresponding bounding box and pixel-level labels. We utilize its 2012 version as the target domain, which includes 11,530 images and 27,450 labeled instances.
6

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
Table
Comparison Results on COCO and Pascal VOC. Source and Target denote models trained with source domain data and
target domain data, respectively. W denotes reproduced results of WeSAM.
COCO
Pascal VOC poly
Method
box
point
poly
Average
GAIN
box
point
Average
GAIN
Source
74.29
55.06
65.64
65,00
69.21
69.21
60.79
66,40
Target
81.50
69.77
73.39
74.89
9.90
81.23
76.98
71.32
76.51
10.11
TENT (Wang et al., 2020)
78.21
52.99
71.51
67.57
2.58
80.24
74.97
65.03
73,41
7.01
SHOT (Liang et al., 2021)
75.18
58.46
69.26
67.63
2.64
79.80
74.26
63.38
72.48
6,08
soft Teacher (Xu et al., 2021)
75.94
43.36
68.27
62.52
-2.47
72.93
56.09
62.20
63.74
-2.66
TRIBE (Su et al., 2024)
77.56
49.56
70.99
66.04
1.05
78.87
69.21
65.39
71.16
4.76
DePT (Gao al., 2022)
71.00
37.35
63.27
57.21
-7.78
74.09
42.99
59.94
59.01
-7.39
WDASS (Das et al., 2023)
77.29
60.55
70.19
69.34
4.35
80.12
76.15
66.98
74.42
8.02
WeSAM (Zhang et al., 2024)
77.32
60.50
70.77
69.53
4.54
80.27
74.15
66.72
73.71
7.31
ours
78.97
63.00
72.54
71.50
6.51
82.90
76.24
70.20
76.45
10.05
Table 2. Comparison Results on CAMO and CODIOK. Source and Target denote the models trained with source domain data and target domain data, respectively.
CAMO
COD10K
Method
box
point
poly
Average
GAIN
box
point
poly
Average
GAIN
Source
62.72
57.43
50.85
57,00
66.32
63.61
40.04
56,66
Target
79.17
77.01
67.12
74.43
17.43
78.06
78.44
64.90
73.80
17.15
TENT (Wang ct al., 2020)
71.24
59.59
60.29
63.71
6.71
69.36
61.94
43.36
58.22
1.57
SHOT (Liang et al., 2021)
71.61
62.78
58.72
64.37
7.37
69.09
65.25
42.38
58.91
2.26
soft Teacher (Xu al., 2021)
62.30
48.64
51.26
54.07
-2.93
66.32
50.04
32.27
49.54
-7.11
TRIBE (Su et al., 2024)
66.00
61.97
60.54
62.84
5.84
67.84
63.62
42.75
58.07
1.42
DePT (Gao et al., 2022)
55.44
33.07
48.63
45.71
-11.29
59.32
34.06
35.51
42.96
-13.69
WDASS (Das et al., 2023)
71.25
63.39
62.29
65.64
8.64
71.42
65.61
43.93
60.32
3.67
WeSAM (Zhang al., 2024)
73.42
65.55
62.90
67.29
10.29
71.93
70.55
45.87
62.78
6.13
ours
74.46
70.21
67.54
70.74
13.74
73.89
72.83
47.27
64.66
8.01
by 1.97, 2. and 3.87, respectively. On Pascal VOC, our method surpasses WeSAM by 2.74 and WDASS by 2.03 in average mloU. These results highlight the effectiveness of our approach in adapting to challenging scenarios.
passing WeSAM by 7.26 and TRIBE by 9.06, demonstrating its clear superiority.
Adaptation to robotic target domains. Tab. lists the results on the OCID dataset. The segmentation results using polygon prompts are significantly better than those obtained with box or point prompts. These results might be caused by the sparse supervision provided by box and point prompts, which often includes multiple overlapping or similar in- stances. Such cases result in ambiguous boundaries and imprecise masks. In contrast, polygon prompts explicitly de- fine instance boundaries, reducing ambiguity and enabling more accurate segmentation. Our method achieves an aver- age mIoU of 84.02, outperforming all competing methods. Compared with WeSAM and WDASS, our approach sur- passes them by 2. 14 and 3.52 in average mloU. Additionally, our method achieves GAIN of 7.16, which is significantly higher than that SHOT (2.71) and TRIBE (9.. These results demonstrate the effectiveness and robustness of our method adapting to the target domain.
Adaptation eamouflaged target domains. As illustrated in Tab. we report the performance of various state-of- the-art methods on the COCO and CODIOK 10K datasets. Our method achieves the highest average mloU of 70.74 on COCO and 64.66 on COD1OK. Specifically, compared to the self-training method WeSAM, our approach delivers substantial improvements of 3.45 and 1.88 in average mloU on CAMO and COD10K, respectively. method also achieved the largest improvement 13.74 on CAMO and 8.01 on CODIOK. Additionally, when com-
terms of GAIN,
pared with other methods like WDASS and SHOT, our method maintains consistent superiority across all metrics, further demonstrating its robustness and versatility.
Adaptation medical target domains. We summarize the comparison results on kvasir-SEG in Tab. 3. Our method achieves an average mIoU of 83.29, significantly outper- forming all competing methods. Compared with WeSAM, TRIBE, and WDASS, our approach improves the average mloU by 7.26, 5.06, and 12.43, respectively. Our method achieves a remarkable GAIN improvement of 17.32, sur-
6.4. Qualitative Results
Fig. presents the qualitative results of our proposed method alongside the comparison methods, SAM and We- SAM. The visualizations demonstrate that our method

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
Image and box prompt
Ground truth
SAM
WeSAM
ours
Figure 3. Qualitative comparison results the CAMO target domain among SAM, WeSAM, and proposed method.
Table Comparison Results on kvasir-SEG. Source and Target denote the models trained with source domain data and target domain data, respectively.
Table S. Performance comparison of our proposed method with different loss components. Direct, Reverse, and Tripler denote the direct alignment distillation loss, reverse distillation loss, and triplet relationship loss, respectively.
kvasir-SEG poly
Method
box
point
Average
GAIN
Direct
Reverse
Triplet
CAMO
CODI0K
Source Target
81.59
62.30
54.03
65.97
68.83
63.21
85.89
77.54
81.64
81.69
15.72
70.22
64.13
TENT
82.47
61.84
62.97
69.09
3.12
69.75
63.75
SHOT
82.30
63.76
61.34
69.13
3.16
70.74
64.66
soft Teacher
84.12
73.53
58.15
71.93
5.96
TRIBE
85.05
73.03
64.61
74.23
8.26
DePT
81.91
52.06
61.55
65.17
-0.80
WDASS
84.01
63.78
64.78
70.86
4.89
WeSAM
85.47
75.23
67.40
76.03
10.06
achieves finer and more precise segmentation performance in the downstream segmentation task.
ours
86.92
76.18
86.78
83.29
17.32
Table Comparison Results OCID, Source and Target denote the models trained with source domain data and target domain data, respectively.
6.5. Ablation Studies
To evaluate the effectiveness of each component in our pro- posed method, conduct ablation studies the adaptation tasks from source to CAMO and CODIOK. We employ the Average mloU as the metric for evaluating their adaptation performance.
OCID poly
Method
box
point
Average
GAIN
Source
86.35
71.41
72.81
76.86
Target
91.24
89.22
79.23
86.56
9.71
TENT
87.77
66.61
77.53
77.30
0.45
Ablation on the reverse distillation loss. Tab. shows the impact of the Reverse Distillation loss on adaptation perfor- mance. By comparing the baseline method with only Direct loss (68.83 CAMO and 63.21 COD10K) to the method with both Direct and Reverse losses (70.22 on CAMO and 64.13 on CODIOK), it is evident that the Reverse loss signif- icantly enhances performance. The improvements 1.39 on CAMO and 0.92 on CODIOE demonstrate that the Re- verse Distillation loss boosts network diversity, enabling the
SHOT
88.06
74.39
76.25
79.57
2.71
soft Teacher
84.98
68.46
73.75
75.73
-1.13
TRIBE
86.77
67.86
76.50
77.04
0.19
DePT
82.00
56.52
70.92
69.81
-7.04
WDASS
87.68
77.13
76.70
80.50
3.65
WeSAM
88.09
80.14
77.41
81.88
5.02
ours
88.07
77.33
86.66
84.02
7.16
8

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source- Fource-Free Domain Transfer
Pascal VOC box
Pascal VOC point
Pascal VOC poly
70-
68-
68
m
mlou
66
mioU
66
64
78
ours (Alpha)
ours (Alpha)
ours (Alpha)
ours (Beta)
62
ours (Beta)
64
ours (Beta)
WeSAM
WeSAM
WeSAM
76
10
20
30
5
10
15
2
8
10
Epoch
Epoch
Epoch
COD10K box
COD10K point
COD10K poly
72
70
46
72
68
miou
mioU
E 66
68
(Alpha)
64
(Alpha)
(Alpha)
66
ours (Beta)
ours (Beta)
42
ours (Beta)
WeSAM
62
WeSAM
WeSAM
10
20
10
15
Epoch
Epoch
Epoch
Figure Segmentation performance curves of method and WeSAM. The experiments conducted in different prompt forms on two target domain datascts. The orange, blue, and light blue curves represent the performances of WcSAM, SAMa. and SAM, in our method, respectively.
model learn more robust and complementary features for better adaptation results.
dynamic teacher-student role switching and collaborative mutual learning in our method lead to faster convergence and superior performance, validating its efficiency and ef- fectiveness over fixed-role approaches.
Ablation the triplet relationship loss. Tab. 5 highlights the contribution of the Triplet Relationship loss to adapta- tion performance. Adding the Triplet Relationship loss to the baseline with Direct loss achieves performance gains from 68.83 to 69.75 on CAMO and from 63.21 to 63.75 on CODIOK. These improvements (0.92 on CAMO and 0.54 on COD10K) show that the Triplet loss effectively cap- tures the triplet relationships, leading enhanced feature representation and improved adaptation performance.
7. Conclusion
We propose a Collaborative Mutual Learning framework for source-free domain adaptation of the Segment Anything Model (SAM). Our approach introduces a dual-network structure and facilitates their interaction through flexi- ble and adaptive training process. During the adaptation process, we estimate the reliability of the two collabora- tive networks and propose Dynamic Role Assignment mechanism to dynamically assign teacher and student roles to the networks. To enhance learning, we incorporate direct aligmment loss, enabling the student network effee- tively learn domain-specific features from the teacher. Ad- ditionally, employ the reverse distillation loss to avoid over-convergence and ensure the diversity of the teacher network. Furthermore, we design triplet relationship loss to encourage robust feature learning while preserving the pre- trained knowledge SAM. Extensive experiments con- ducted across various challenging target domains demon- strate the effectiveness and adaptability of our framework.
6.6. Effectiveness and Efficiency Analysis
Fig. illustrates the mloU performance curves of our pro- posed dual collaborative-networf method (Ours) and the baseline method (WeSAM) on two datasets. Our method exhibits significantly faster performance improvement com- pared to WeSAM on both datasets. The mloU curves of SAM, and S SAMs in our method show alternating improve- ments, demonstrating the mutual learning process. This allows our model achieve higher performance levels in fewer epochs. Furthermore, the final performance of both S SAM,, and SAM, surpasses that of WeSAM, highlight- ing the effectiveness of collaborative mutual learning. The

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
Acknowledgment
Chen, Y., Li, W., Sakaridis, C., Dai, D., and Van Gool, L, Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3339-3348, 2018.
This research is supported by Laboratory for Artificial In- telligence in Design (Project Code: RP3-3) under InnoHK Research Clusters, Hong Kong SAR Government.
Das, A., Xian, Y., Dai, D., and Schiele, B. Weakly- supervised domain adaptive semantic segmentation with prototypical contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15434-15443, 2023.
Impact Statement
This paper presents work whose goal is to enhance the per- formance of foundational models within the target domain. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Döbler, M., Marsden, R. A., and Yang, B. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7704-7714, 2023.
References
Bai, Y., Geng, X., Mangalam, K., Bar, A., Yuille, A. L., Darrell, T., Malik, J., and Efros, A. Sequential model- ing enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22861-22877 2024.
Everingham, M., Eslami, S. A., Van Gool, L.,, Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 1 111:98 136, 2015.
Fan, D.-P., Ji, G.-P., Sun, G., Cheng, M.-M., Shen, J., and Shao, L. Camouflaged object detection. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 2777-2787, 2020.
Bolya, D., Zhou, C., Xiao, F., and Lee. Y. J. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9157-9166, 2019.
Ganin, Y. and Lempitsky, V. Unsupervised domain adap- tation by backpropagation. In ICML, pp. 1189- PMLR, 2015.
Cao, Y., Zhang. H., Lu, X., Chen, Y., Xiao, Z., and Wang, framework for unsupervised domain adaptation semantic segmen- tation. IEEE Transactions on Circuits and Systems for Video Technology, 33(8):3822-3833 2023. doi: 10.1 D0.1199/TCSVTT2223332444422
Y. Adaptive refining-agee aagggggggggaiion-ppraaaat
Gao, Y., Shi, X., Zhu, Y., Wang, H., Tang, Z., Zhou, X., Li, M., and Metaxas, D. N. Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:22 10.04831, 2022.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. European conference on computer vision, pp. 213-229. Springer, 2020.
Gong, T., Jeong. J., Kim, T., Kim, Y., Shin, J., and Lee, S.-J. Note: Robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems, 35:27253-27266 2022.
Chen, C., Li, Zhou, H.-Y., Han, X., Huang, Y., Ding, X., and Yu, Y. Relation matters: Foreground-aware graph- based relational reasoning for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):367733694, 2022.
Hc, K., Zhang, X., Ren, S., and Sun, J. Deep residual lear- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r- cnn. In Proceedings of the IEEE international conference on computer vision, pp. 29612221222 2969, 2017.
Chen, T,, Zhu, L., Deng, C., Cao, R., Wang, Y., Zhang, S., Li, Z., Sun, L., Zang, Y., and Mao, P. Sam-adapter: Adapting segment anything in underperformed scenes. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 3367-3375, 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rand adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openreview net wtrumoormm//mm idd www.mmiiidiieeeeeee..
Chen, T., Lu, A., Zhu, L., Ding, C., Yu, C., Ji, D., Li, Z.,, Sun, I., Mao, P., and Zang, Y. Sam2-adapter: Eval- uating adapting segment anything 2 in downstream tasks: Camouflage, shadow, medical image segmenta- tion, and more, 2024. URL https: / //twxiix.voor///// org/ abs/ 04579.
Huang, C., Liu, Y., Zhang, Z., Liu, C., Wen, J., Xu, Y., and Wang, Y. Hierarchical graph embedded pose regularity learning via spatio-temporal transformer for abnormal
10

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
behavior detection. In Proceedings of the 30th ACM international conference on multimedia, pp. 307-315, 2022a.
Liu, Y., Wang, J., Huang, C., Wang, Y., and Xu, Y. Cigar: Cross-modality graph reasoning for domain adaptive ob- ject detection. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 23776-23786, 2023a.
Huang, C., Wen, J., Xu, Y., Jiang, Q. Yang, J., Wang, Y., and Zhang, D. Self-supervised attentive generative adversarial networks for video anomaly detection. IEEE transactions on neural networks and learning systems, 34 (11):938999444) 9403,,999322220 2022b.
Liu, Y., Wang, J., Xiao, L., Liu, C., Wu, Z., and Xu, Y. Foregroundness-wware task disentanglement and self- paced curriculum learning for domain adaptive object detection. IEEE Transactions on Neural Networks and Learning Systems, 2023b.
Jha, D., Smedsrud, H., Riegler, M. A., Halvorsen, P., De Lange, T., Johansen, D., and Johansen, H. D. Kvasir- seg: A segmented polyp dataset. In MultiMedia model- ing: 26th international conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, proceedings, part II 26, pp. 351-462. Springer, 2020.
Liu, Y.. Wang, J., Huang, C., Wu, Y., Xu. Y., and Cao, X. Mifa: Towards realistic test time adaptive object detection by multi-level feature alignment. IEEE Transactions on Image Processing. 2024a.
Jin, Z., Xie, J., Wu, B., and Shen, L. Weakly super- vised pedestrian segmentation for person re-identification. IEEE Transactions on Circuits and Systems for Video Technology, 33(3):1349-1362, 2023. doi: 10.1109) TCSVT.2022.32210476
Liu, Y., Wang, J., Wang, W., Hu, Y., Wang, Y., and Xu, Y. Crada: Cross domain object detection with cyclic recon- struction and decoupling adaptation. IEEE Transactions on Multimedia, 2024b.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-. et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Milletari, F., Navab, N., and Ahmadi, S.-A. V-net: Fully con- volutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 5-5-55557777 Icee, 2016.
Le, T.-N., Nguyen, T. V., Nie, Z., Tran, M.-T., and Sug- imoto, A. Anabranch network for camouflaged object segmentation. Computer vision and image understanding, 184:45-56, 56, 2019.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199. 2023.
Li, W., Liu, x., and Yuan, Y. Sigma: Semantic-complete graph matching for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 5291-5300, 2022.
Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Liang, J., Hu, D., Wang, Y., He, R., and Feng. J. Source data-absent unsupervised domain adaptation through hy- pothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11); 6602-8617, 2021.
Ross, T.-.. and Dollár, G. Focal loss for dense object detection. In proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2980-2988, 2017.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zimick, C. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Su, Y., Xu, X., and Jia, K. Towards real-world test-time adaptation: Tri-net self-training with balanced normaliza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15126-15135, 2024.
Suchi, M., Patten, T., Fischinger, D., and Vincze, M. Fa- sylabel: A semi aanimautomatic pixel-wise object annotation tool for creating robotic rgb-d datasets. In 2019 Interna- tional Conference on Robotics and Automation (ICRA), pp.
Liu, C., Wen, J., Xu, Y., Zhang, B., Nie, L., and Zhang, M. Reliable representation learning for incomplete multi- view missing multi-label classification. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2025.
IEEF, 2019.

---

Mutual Learning for SAM Adaptation: A Dual Collaborative Network Framework for Source-Free Domain Transfer
Sun, Y., Qin, Y., Li, Y., Peng, D., Peng, X., and Hu, Robust multi-view clustering with noisy correspondence. IEEE Transactions on Knowledge and Data Engineering, 36( 12):9150-9116 2024.
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. arXiv preprint MxXi:2000610722) 2020.
Wang, Q., Fink, O., Van Gool, I.., and Dai, Continual test- time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 7201-7211, 2022.
Wu, J., Fan, H., Li, Z., Liu, G.-H., and Lin, S. Infor- mation transfer in semi-supervised semantic segmen- tation. IEEE Transactions on Circuits and Systems for Video Technology, 34(2):114411855 2024. doi: 10.1 22110///SST.222333999999
Xu, M., Zhang. Z., Hu. H., Wang, J., Wang. L., Wei, F., Bai, X., and Liu, Z. End-to-end semi-supervised ob- ject detection with soft teacher. In Proceedings of the IEEF/CVF international conference computer vision, pp. 3060-3069, 2021.
Zhang. H., Su, Y., Xu, X., and Jia, K. Improving the general- ization of segmentation foundation model under distribu- tion shift via weakly supervised adaptation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23385-23392 2024.
Zhao, L. and Tao, W. Jsnet++: Dynamic filters and point- wise correlation for 3d point cloud instance and semantic segmentation. IEEE Transactions on Circuits and Sys- tems for Video Technology, 10.1 10.1100TTCSV:::22221188877
1844-8677, 2023. doi:
Zhou, L., Ye, M., Zhu, X., Li. S., and Liu, Y. Class dis- criminative adversarial learning for unsupervised domain adaptation. In Proceedings of the 30th ACM international conference on multimedia, pp. 4316- 2022.
Zhou, L., Xiao, S., Yc, M., Zhu, X., and Li, S. Adaptive mu- tual learning for unsupervised domain adaptation. IEEE Transactions on Circuits and Systemsfor Video Technol- ogy, 33(1 (6662266344, 2023a.
Zhou, Q. Feng, Z., Gu, Q., Pang, J., Cheng, G., Lu, X., Shi, J., and Ma, I.. Context-aware mixup for domain adaptive semantic segmentation. IEEE Transactions on Circuits and for Video Technology, 33(2):804-817 2023b. doi: 10.1109/TCSVT222222206677
12


Reviews:
[{"review_id": "exSZn5KsZF", "reviewer": "Reviewer_SWqC", "summary": "The paper introduces a Collaborative Mutual Learning Framework for source-free domain adaptation of the Segment Anything Model. It employs dual networks that alternate roles as teacher and student based on reliability, preserving SAM’s pre-trained knowledge. T Experiments in medical, camouflaged, and robotic domains show state-of-the-art performance, outperforming WeSAM and SHOT in mIoU.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.\n\n---\n\n### **Claims And Evidence: About the motivation**\n\nWe appreciate your detailed understanding of teacher-student networks. Our motivation is not directed at the use of EMA or the exploration role of the student network but rather at the fixed-role assignment in traditional teacher-student frameworks. These methods assume that the teacher consistently provides more reliable predictions than the student, and this fixed-role assumption limits their adaptability.\n\nTo address this, we propose a dynamic role assignment mechanism, where teacher and student roles are reassigned based on real-time reliability, enabling more flexible and adaptive knowledge transfer. Additionally, we emphasize robust knowledge distillation by comparing each network's output with the frozen pre-trained SAM. The network retaining more pre-trained knowledge is assigned as the teacher, ensuring cautious and stable updates. This approach reduces error accumulation and catastrophic forgetting.\n\nWe will revise the motivation section to clarify these points and include references to related CTTA methods to better contextualize our contributions.\n\n### **Experimental Designs Or Analyses: About the threshold $\\mathcal{T}$ in Eq. (3)**\n\nTo set the base value for this threshold, we referred to common practices in SAM applications, where thresholds of 0.5 or 0.7 are often used. To further refine this selection, we conducted cross-validation experiments on domains with significant distribution shifts, specifically the CAMO and COD10K datasets. We present more ablation study experiments on $\\mathcal{T}$ in the following table:\n\n|$\\mathcal{T}$|0.3|0.4|0.5|0.6|0.7|\n|-|-|-|-|-|-|\n|source$\\to$CAMO|66.59|70.36|70.74|70.13|69.92|\n|source$\\to$COD10K|60.12|63.74|64.66|64.85|64.27|\n\nWe observed that 0.5 consistently produced the best or near-optimal results across both datasets, balancing precision and recall. To ensure fairness and consistency across domains, we used 0.5 as the default threshold for all experiments. It generalizes well, achieving significant performance improvements across diverse datasets without domain-specific tuning. We will include these results and their analysis in the revised manuscript to justify the threshold selection and its impact on performance.\n\n### **Q1: How is $f_{\\gamma}$ obtained?**\n\nYes, $f_{\\gamma}$ is directly derived from the frozen pre-trained SAM encoder, without any LoRA modifications. It represents the general knowledge learned during SAM’s large-scale pre-training. By using a frozen reference, we ensure that $f_{\\gamma}$ provides a stable baseline for evaluating how much pre-trained knowledge the collaborative networks retain.\n\n### **Q2: Why does computing similarity with $f_{\\gamma}$ estimate model confidence, especially if $f_{\\gamma}$ is frozen?**\n\nThe similarity to $f_{\\gamma}$ measures how much pre-trained knowledge is preserved during adaptation. Although $f_{\\gamma}$ remains frozen throughout, it encodes rich and reliable pre-trained knowledge learned from large-scale datasets. Using $f_{\\gamma}$ as a reference allows us to measure how much pre-trained knowledge each network retains during adaptation. Networks with higher similarity to $f_{\\gamma}$ are considered more reliable, as their updates are more cautious and less prone to overfitting to noisy, domain-specific features.\n\nThis approach has two advantages:\n\n1. Compared to relying on the teacher network determined in the previous step, which may have already drifted from the pre-trained knowledge, $f_{\\gamma}$ provides a stable and consistent baseline for evaluating reliability. \n\n2. It is more computationally efficient, as it avoids the need for iterative teacher updates. By dynamically assigning the teacher role to the network closer to $f_{\\gamma}$, we ensure robust and cautious knowledge transfer, reducing error accumulation and catastrophic forgetting. \n\nOur experiments validate the effectiveness of this method, demonstrating that it achieves stable and efficient domain adaptation while leveraging the reliability of pre-trained knowledge. We will further elaborate on this in the revised manuscript to clarify its rationale and efficiency.\n\n### **Relation to CTTA methods**\n\nThank you for pointing out the connection to CTTA methods. While CTTA methods address challenges like error accumulation and catastrophic forgetting, our approach introduces key innovations: \n\n1. Dynamic Role Assignment based on real-time reliability, allowing flexible teacher-student role switching; \n\n2. Robust Knowledge Preservation via the frozen pre-trained SAM, preventing overfitting to noisy features; \n\n3. Bidirectional Knowledge Exchange, enabling mutual learning rather than unidirectional transfer. \n\nWe will expand the related work section to highlight these distinctions and include the provided references([1-4]).", "claims_and_evidence": "The paper's motivation (L121-L125) states that previous works using the teacher-student approach assume that the teacher network consistently provides more reliable predictions than the student network. However, I believe this is not the actual assumption. In my opinion, the teacher model is updated by the student model using EMA, which helps better prevent catastrophic forgetting, while the student model's purpose is to explore a broader space to guide the teacher model in the right direction.", "methods_and_evaluation_criteria": "Make sense.", "theoretical_claims": "I think this paper do not contain the theoretical claims.", "experimental_designs_or_analyses": "The proposed method achieves state-of-the-art performance on various benchmarks. However, I would like to know how to set $\\mathcal{T}$ in Eq. (3). I hope the authors can include an ablation study on this parameter. Additionally, does $\\mathcal{T}$ need to be set to different values for different domains?", "relation_to_broader_scientific_literature": "Related to the CTTA setting, which is also a variant of SFDA, many methods in CTTA[1-4] have utilized the Teacher-Student Network. Moreover, the CTTA task also needs to address issues such as error accumulation and catastrophic forgetting, which are also the focus of this paper.\n\n[1] Continual Test-Time Domain Adaptation\n\n[2] Robust mean teacher for continual and gradual test-time adaptation\n\n[3] Exploring sparse visual prompt for cross-domain semantic segmentation\n\n[4] Note: Robust continual test-time adaptation against temporal correlation", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "See below (Questions For Authors*).", "other_comments_or_suggestions": "None", "questions_for_authors": "1. How is $f_{\\gamma}$ obtained? Is it derived from the raw pre-trained SAM encoder without any LoRA modifications?  \n\n2. Why does computing similarity with $f_{\\gamma}$ allow the estimation of model confidence and consequently determine whether A or B should be the teacher? This is particularly unclear if the model corresponding to $f_{\\gamma}$ is frozen.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pPh5uf0FV5", "reviewer": "Reviewer_t7Bb", "summary": "This paper introduces a mutual teaching framework for SAM adaptation in target domains. Compared to self-training framework, the mutual learning enables dynamic assignment of teacher and student roles, leading to more robust and generalized performance. To facilitate adaptation, three loss functions are proposed: direct alignment loss, reverse distillation loss, and triplet relationship loss. Extensive experiments demonstrate the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.\n\n---\n\n### **Claims And Evidence: Clarification on \"Catastrophic Preservation\" in line 256**\n\nWe appreciate your observation regarding the term \"catastrophic preservation\". In the revised manuscript, we will rephrase this term to \"preventing catastrophic forgetting\" to align with standard terminology and clarify its conceptual basis. \n\n### **Experimental 1: Attribution of improvements to the mutual learning framework**\n\nWe conduct an ablation study to evaluate the impact of different role assignment strategies (learning paradigms) in our framework. Experimental results are as follows. These results confirm that the performance improvements are directly attributable to our dynamic role assignment mechanism in mutual learning. \n\n|role assignment strategy|CAMO|COD10K|\n|-|-|-|\n|source(SAM)|57.00|56.66|\n|WeSAM(fixed)|67.29|62.78|\n|random(mutual)|60.97|50.22|\n|reverse(mutual)|53.61|38.32|\n|ours(mutual)|70.74|64.66|\n\n1. **Role Assignment Matters:** Random or reverse assignments disrupt learning, leading to performance drops, while our dynamic role assignment achieves the best results by leveraging real-time reliability evaluation.\n\n2. **Reverse Assignment Impact:** Incorrectly assigning the less reliable model as the teacher severely hampers knowledge transfer, highlighting the importance of proper role allocation.\n\n3. **Dynamic Role Assignment Effectiveness:** Our approach ensures stable collaboration and effective knowledge transfer, significantly outperforming fixed or random assignments.\n\nWe also counted the number of roles exchanged at different training epochs in several adaptation tasks:\n\n|Target|total iteration|1|2|3|4|5|\n|-|-|-|-|-|-|-|\n|CAMO|1972|976|638|416|204|103|\n|OCID|1000|469|491|466|457|197|\n|Kvasir-SEG|836|444|375|305|29|10|\n\nThe results align with our design goal: early in training, frequent switches allow both networks to explore the target domain, while later, more stable role assignments aid convergence.\n\n### **Experimental 2: Performance differences with Prompts (Table 4)**\n\nOCID images have dense objects with severe occlusion and overlap, making segmentation difficult. Sparse supervision from box and point prompts often includes multiple overlapping or similar instances, leading to ambiguous boundaries and imprecise masks. Polygon prompts explicitly describe instance boundaries, reducing ambiguity and enabling more accurate segmentation.\n\nWeSAM’s fixed teacher provides stable guidance, which is beneficial when sparse prompts introduce noise or ambiguity. Our dynamic role assignment can amplify noise in challenging scenarios, leading to suboptimal role allocations. WeSAM may be better tuned for sparse prompts, while our framework emphasizes bidirectional learning.\n\n### **W1: Mutual learning is inherently more robust but comes with higher computational and design complexity, which is a limitation of the proposed method.**\n\nWe conducted experiments comparing the training time and memory usage and performance of WeSAM, a single LoRA adapter, and our dual LoRA adapter framework on the source$\\to$Kvasir-SEG(box) task. The results are as follows:\n\n||WeSAM|single adapter|dual adapter (ours)|\n|-|-|-|-|\n|training time per iteration (s)|**1.8**|1.1|1.3|\n|CUDA memory (MB)|12500|14276|**14906**|\n|mIoU|85.47|-|**86.92**|\n\nOur method achieves a balance between computational efficiency and performance. Compared to the single LoRA adapter, the dual adapter only increases training time by 0.2 seconds per iteration and requires 4.4% more memory, while delivering significantly better results. Importantly, our algorithmic optimizations make our method faster than WeSAM (1.3s vs. 1.8s per iteration) despite incorporating a more advanced mutual learning framework.\n\n### **W2: Insights into loss design motivations**\n\nOur loss functions are designed to enhance the effectiveness of the mutual teaching framework by facilitating knowledge transfer, increasing model diversity, and stabilizing training.\n\n1. **Direct Alignment Loss:** This loss aligns the student network’s predictions with the teacher’s outputs, enabling effective transfer of target domain knowledge from the teacher to the student. It ensures the student learns domain-adapted features from the teacher.\n\n2. **Reverse Distillation Loss:** Unlike direct alignment, this loss increases the divergence between the teacher and student networks, encouraging the teacher to explore more diverse outputs. By providing the teacher with greater optimization freedom, this loss enhances the robustness of the teacher network.\n\n3. **Triplet Relationship Loss:** This loss reinforces the feature relationships among the reference model (pre-trained SAM), the teacher, and the student. By aligning their relative feature representations, ensures consistent and robust training, improving target domain adaptation.", "claims_and_evidence": "Line 256 states, “Similar to (Zhang et al., 2024), to prevent catastrophic preserving of the pre-trained SAM.” Please review this claim carefully, especially the misleading term \"catastrophic preservation\". WeSAM proposes regularization on the student model using an anchor model to prevent model collapse. The proposed preseving loss is to prevent catastrophic forgetting of the pretrained knowledge, which is conceptually different from preventing catastrophic preservation of SAM.", "methods_and_evaluation_criteria": "This paper largely adopts the model architecture, loss design motivations, and experimental settings from the baseline WeSAM.", "theoretical_claims": "There are no specific theoretical claims or corresponding proofs, but the overall model design is reasonable.", "experimental_designs_or_analyses": "1. The experiments do not clearly demonstrate how WeSAM would perform if adapted to a mutual learning framework, making it difficult to attribute the improvements solely to the proposed modifications.\n2. In Table 4, the proposed model performs worse than the baseline WeSAM with box and point prompts, but shows significant improvement with the polygon prompt. The authors should provide an analysis of the reasons behind this difference.", "relation_to_broader_scientific_literature": "1. This paper primarily adopts the model architecture and loss design motivations from the existing work WeSAM, including: 1) the model architecture with a fixed encoder network, prompt encoder, and mask decoder; 2) fine-tuning the LoRA module for efficient SAM adaptation; 3) a teacher-student self-training loss; 4) a regularization loss from the fixed anchor/reference model to prevent model collapse; and 5) a regularization loss in the feature space. Additionally, the specific loss formulations (i.e., the SAM loss) are consistent with existing work, while the triplet loss is derived from metric learning and domain adaptation papers.\n\n2. The mutual teaching framework is based on concepts from many existing semi-supervised learning and domain adaptation papers.\n\n3. The triplet loss is derived from existing metric learning and domain adaptation papers.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-writen and clear. \n2. The experiments are sufficient to evaluate the effectiveness of the proposed model. \n\nWeaknesses:\n1. Model design. From my understanding, the proposed method improves upon the baseline WeSAM by replacing the self-training framework with a mutual teaching framework. However, it is well-known that mutual learning is inherently more robust to noise and better generalizes to diverse target distributions, which naturally leads to improved performance. While this paper inherits these advantages of mutual learning, it also retains some of its limitations, such as high computational and design complexity. \n2. Loss design. The direct alignment loss, reverse distillation loss, and triplet loss are designed based on motivations presented in WeSAM.\nTherefore, the improved performance over WeSAM can mainly be attributed to mutual learning. Beyond this, I find limited insights into the model architecture and loss design motivations.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "Please correct me if I have any misunderstandings.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EGMbtTxTvP", "reviewer": "Reviewer_s4SY", "summary": "This paper addresses the challenge of adapting the Segment Anything Model to new domains with significant distribution shifts through a Collaborative Mutual Learning Framework. The method involves two collaborative networks that dynamically alternate between teacher and student roles based on reliability. They also propose the reverse distillation to train the teacher network in each training iteration. Experiments on six datasets demonstrate the approach's adaptability and superior performance compared to existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.\n\n---\n\n### **W1: Lack of analysis on role switching frequency and its Impact on convergence and performance**\n\nWe conducted an additional quantitative analysis of the role-switching frequency across training iterations. Specifically, we tracked the number of role switches for both networks during training on three datasets. The results show that the switching frequency decreases as training progresses, stabilizing after several epochs. This behavior aligns with our design goal: early in training, frequent switches allow both networks to explore the target domain, while later, more stable role assignments aid convergence.\n\n|Target|total iteration|1|2|3|4|5|\n|-|-|-|-|-|-|-|\n|CAMO|1972|976|638|416|204|103|\n|OCID|1000|469|491|466|457|197|\n|Kvasir-SEG|836|444|375|305|29|10|\n\nWe also evaluated the impact of role switching on performance by comparing source SAM, a fixed-role method (WeSAM), and our dynamic switching framework. The dynamic role assignment consistently outperformed the fixed-role method. These results confirm that dynamic switching is crucial for effective domain adaptation.\n\n|role assignment strategy|CAMO|COD10K|\n|-|-|-|\n|source(SAM)|57.00|56.66|\n|WeSAM(fixed)|67.29|62.78|\n|random(mutual)|60.97|50.22|\n|reverse(mutual)|53.61|38.32|\n|ours(mutual)|70.74|64.66|\n\n### **W2: Lack of visualization of the dynamic Teacher-Student Process**\n\nWe generated a visualization of the reliability scores for both networks over training iterations. This visualization demonstrates how the reliability-based role assignment mechanism dynamically adjusts roles based on the real-time performance of each network. Early in training, roles alternate frequently as both networks adapt to the target domain. Over time, the network with consistently higher reliability assumes the teacher role more often, reflecting a stabilization in the mutual learning process. This visualization will be included in the revised manuscript or open-source platform to illustrate how roles evolve during training and to provide a clearer understanding of the dynamic teacher-student process.\n\n### **W3: Lack of detailed visualizations of segmentation improvements**\n\nWe have prepared detailed visualizations of segmentation results on challenging samples from the CAMO, COD10K, and OCID datasets. These visualizations highlight the improvements achieved by our method, particularly in cases involving occlusion, complex object boundaries, and cluttered scenes. For example, on the OCID dataset, our method produces cleaner and more precise segmentation masks compared to both the baseline SAM and WeSAM, especially when using polygon prompts. These improvements are attributed to the combination of our proposed loss functions, which enhance feature alignment, promote diversity, and ensure robust training. We will include these visualizations in the revised manuscript and provide additional examples on the open-source platform for further clarity.\n\n\n### **Q1: Could you provide more theoretical insights or empirical evidence to justify why reverse distillation enhances diversity?**\n\nThe reverse distillation loss is designed to encourage output diversity by promoting a divergence between the teacher and student networks. We summarize its theoretical and empirical justifications:\n\n**Theoretical Insights**:\n\n1. **Balancing Exploration and Exploitation:** Reverse distillation introduces controlled divergence between the teacher and student, forcing the teacher to explore diverse predictions. This ensures the teacher provides richer and more comprehensive guidance.\n\n2. **Regularization Effect:** It prevents knowledge collapse by maintaining teacher-student diversity, critical for adapting to domains with significant distribution shifts.\n\n3. **Optimization Perspective:** This loss expands the teacher’s solution space, avoiding local minima caused by excessive alignment with the student.\n\n4. **Model Ensemble Parallel:** Similar to ensemble learning, diversity between teacher and student predictions strengthens generalization.\n\n**Empirical Evidence**:\n\nWe conducted an ablation study comparing the performance of our method with and without the reverse distillation loss. On the CAMO dataset, removing the reverse distillation loss led to a 0.99 drop in average mIoU, while on COD10K, the performance dropped by 0.91 average mIoU. \n\n|Direct|Reverse|Triplet|CAMO|COD10K|\n|-|-|-|-|-|\n|✓|||68.83|63.21|\n|✓|✓||70.22|64.13|\n|✓||✓|69.75|63.75|\n|✓|✓|✓|70.74|64.66|\n\nIn summary, reverse distillation enhances diversity, improves robustness, and synergizes with other loss terms to ensure effective knowledge transfer. These insights will be added to the revised manuscript.", "claims_and_evidence": "The claims are backed by strong experimental evidence. The authors demonstrate state-of-the-art results on multiple datasets and provide ablation studies to validate each proposed component.", "methods_and_evaluation_criteria": "The methods are reasonable and align well with the problem's requirements. The use of standard benchmarks like COCO and Pascal VOC provides credibility.", "theoretical_claims": "The paper does not seem to make any complex theoretical claims requiring proof.", "experimental_designs_or_analyses": "The experimental designs are robust. The ablation studies justify the contributions of each loss term, and the comparisons with state-of-the-art methods are thorough.", "relation_to_broader_scientific_literature": "The paper is well-situated within the literature on SAM and source-free domain adaptation. It builds on previous works like WeSAM but innovates by introducing a dynamic mutual learning framework.", "essential_references_not_discussed": "The authors have cited relevant works in SAM adaptation, domain adaptation, and mutual learning.", "other_strengths_and_weaknesses": "Strengths:\n(1) Dynamic teacher-student assignment is an innovative idea.\n(2) The paper provides detailed ablation experiments to validate the contribution of each loss component.\n(3) The introduction of the GAIN metric provides an intuitive measure of performance improvement over the baseline SAM.\n(4) The combination of direct alignment, reverse distillation, and triplet losses is well-designed to balance exploration and exploitation.", "other_comments_or_suggestions": "(1) The framework does not explicitly analyze how often roles are switched or how this impacts convergence and performance.\n(2) The dynamic teacher-student process is not visualized, making it difficult to understand how roles evolve during training.\n(3) The paper lacks detailed visualizations of segmentation improvements, which would strengthen the qualitative evaluation.", "questions_for_authors": "Could you provide more theoretical insights or empirical evidence to justify why reverse distillation enhances diversity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DHVzjF5xBJ", "reviewer": "Reviewer_B25v", "summary": "The paper proposes a Collaborative Mutual Learning Framework for the source-free domain adaptation of the Segment Anything Model. The key innovation lies in dynamically assigning \"teacher\" and \"student\" roles to two collaborative networks based on their reliability during training. The framework introduces three key components direct alignment loss for knowledge transfer, reverse distillation loss for diversity, and triplet relationship loss. Extensive experiments demonstrate superior performance across several challenging target domains, including medical imaging, camouflaged object detection, and robotic vision.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical as well as constructive assessments of our work, and we address the concerns point by point as follows.\n\n---\n\n### **W1: The triplet loss contributes only marginal improvements, raising questions about its necessity.**\n\nThe triplet loss, although contributing marginally to numerical performance, plays a critical theoretical role in stabilizing the training process. It ensures feature consistency between the pre-trained SAM, teacher, and student networks, preventing excessive divergence that could lead to noisy predictions or unstable optimization. From a theoretical perspective: \n\n1. The triplet loss acts as a regularizer, preserving the reference model’s general-purpose knowledge during adaptation. \n\n2. It prevents overfitting by maintaining meaningful feature relationships across networks. \n\n3. Its synergy with reverse distillation and direct alignment losses ensures robust training. \n\nWhile its numerical gains may appear small, its role in stabilizing the mutual learning process is essential. This theoretical significance will be clarified in the revised manuscript.\n\n### **W2: The method is heavily tailored to SAM, and its generalizability to other foundational models is unclear.**\n\nOur framework is not inherently tied to SAM and can be generalized to other foundational models. Theoretical principles such as dynamic teacher-student assignment, reverse distillation, and triplet loss are broadly applicable: \n\n1. **Dynamic Role Assignment:** Reliance on real-time reliability scoring allows adaptation to any pre-trained model. \n\n2. **Reverse Distillation:** Promotes diversity and avoids overfitting, critical for models fine-tuned on new domains. \n\n3. **Triplet Loss:** Ensures consistent alignment with the foundational model while adapting to target domains. \n\nWe will explicitly discuss this generalizability and propose extending the framework to other models as future work.\n\n### **W3: The framework’s sensitivity to hyperparameters (e.g., thresholds for role assignment, loss weights) is not discussed, which could impact reproducibility.**\n\nWe conducted additional experiments analyzing the impact of key hyperparameters.\n\n1. **Threshold for Role Assignment:** The values of $\\mathcal{T}$ around 0.5 consistently yielded the best or second-best results across datasets, confirming robustness to moderate variations.\n\n|$\\mathcal{T}$|0.3|0.4|0.5|0.6|0.7|\n|-|-|-|-|-|-|\n|source$\\to$CAMO|66.59|70.36|70.74|70.13|69.92|\n|source$\\to$COD10K|60.12|63.74|64.66|64.85|64.27|\n\n2. **Loss Weights:** Reverse distillation and direct alignment losses were set to the same weight based on their design purpose. Optimal weights were determined as 0.1 for reverse and direct alignment losses and 0.001 for the triplet loss. Extremely high or low weights resulted in significant performance degradation. These values were validated across datasets and generalized to all experiments.\n\n|$\\lambda_{reverse}$|$\\lambda_{preserve}$|$\\lambda_{triplet}$|CAMO|COD10K|\n|-|-|-|-|-|\n|0.02|0.02|0.001|69.23|63.45|\n|0.1|0.1|0.01|70.74|64.66|\n|0.1|0.1|0.05|70.32|64.37|\n|0.5|0.5|0.1|68.81|62.93|\n\nWe will include these findings in the revised manuscript and supplementary material to enhance reproducibility.\n\n### **W4: The triplet loss formulation could be made more intuitive.**\n\nTo improve clarity, we will revise the explanation of the triplet loss in the manuscript with step-by-step derivations and an intuitive diagram illustrating its role in aligning feature representations between the reference, teacher, and student networks. This will help readers better understand its purpose and implementation.\n\n### **Q1: Could the dynamic role-switching mechanism be extended to more than two networks?**\n\nOur current framework is designed for two networks to simplify the reliability-based role assignment process, as it allows for a straightforward comparison of reliability scores. However, the mechanism can be extended to more than two networks by generalizing the reliability metric to rank multiple networks and dynamically assign roles (e.g., multiple teachers and students). This extension could potentially enhance performance further by leveraging a broader ensemble of models. While we have not implemented this in the current work, we consider it an exciting direction for future research and will discuss this possibility in the revised manuscript.\n\n### **Q2: Should the original SAM data be added to the table in the supplementary material to make the role selection schemes more intuitive?**\n\nWe agree with this suggestion. Adding the original SAM data will provide a clearer baseline for understanding the improvements achieved by our role selection schemes. We will update the supplementary material to include this data and highlight the relative gains of each scheme.\n\n|role assignment strategy|CAMO|COD10K|\n|-|-|-|\n|source(SAM)|57.00|56.66|\n|WeSAM|67.29|62.78|\n|random|60.97|50.22|\n|reverse|53.61|38.32|\n|ours|70.74|64.66|", "claims_and_evidence": "1. The proposed framework achieves state-of-the-art results on multiple benchmarks.\n2. The dynamic teacher-student assignment is validated through performance curves and ablation studies.", "methods_and_evaluation_criteria": "The methods are well-designed, and the evaluation criteria are appropriate for the task. The use of standard datasets like COCO and Pascal VOC ensures comparability with prior work. The inclusion of metrics like mIoU and GAIN is appropriate for assessing segmentation performance.", "theoretical_claims": "There are no complex theoretical claims in this work. The reliability metric and its use in dynamic role assignment are intuitive.", "experimental_designs_or_analyses": "The experiments are comprehensive, covering a wide range of domains and providing ablation studies to validate the contributions of individual components.", "relation_to_broader_scientific_literature": "The paper is well-situated in the domain adaptation literature and builds on prior work like WeSAM and SHOT. However, the connection to broader mutual learning frameworks outside SAM adaptation could be strengthened.", "essential_references_not_discussed": "No major references appear to be missing, but the authors could consider citing additional works on dynamic teacher-student frameworks or mutual learning approaches in other fields to further contextualize the novelty of their method.", "other_strengths_and_weaknesses": "This work presents a range of strengths.\n1. The dynamic teacher-student assignment is innovative and addresses the limitations of fixed teacher-student frameworks.\n2. The paper evaluates the framework on six datasets across diverse domains, demonstrating its versatility and robustness.\n3. The use of LoRA for efficient fine-tuning is a practical and computationally efficient choice.\n4. The framework achieves state-of-the-art performance, consistently outperforming existing methods like WeSAM and SHOT on mIoU and GAIN metrics.\n5. The method is highly relevant for real-world applications, where source data is often inaccessible due to privacy restrictions.\n\nHowever, I am still concerned about the following questions.\n1. The triplet loss contributes only marginal improvements, raising questions about its necessity.\n2. The method is heavily tailored to SAM, and its generalizability to other foundational models is unclear.\n3. The paper does not discuss the sensitivity of the framework to hyperparameters (e.g., thresholds for role assignment, loss weights), which could impact reproducibility.", "other_comments_or_suggestions": "Improve the clarity of certain equations, particularly the triplet loss formulation, which could be made more intuitive.", "questions_for_authors": "1. Could the dynamic role-switching mechanism be extended to more than two networks, or is it inherently limited to a dual-network setup?\n2. The supplementary material shows the performance of different role selection schemes. Should the original SAM data be added to the table, so as to show the various methods more intuitively?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a Collaborative Mutual Learning Framework for source-free domain adaptation of the Segment Anything Model. All four reviewers are positive about this paper, and the rebuttal addresses most of the reviewers' concerns. The AC recommends acceptance.
Decision: Accept (poster)