Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
Authors: Jintao Tong
Ran Ma
Yixiong Zou
Guangyao Chen
Yuhua Li
Ruixuan Li
Venue: ICML 2025 spotlightposter (2025)
Keywords: Decouple
Cross-Domain
Few-Shot
Segmentation
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We find the adapter not only helps the fine-tuning of downstream tasks but also naturally acts as a domain decoupler. Based on this, we proposed a structure-based decoupling method to bridge domain gap, achieving superior performance in CD-FSS.

Abstract: Cross-domain few-shot segmentation (CD-FSS) is proposed to first pre-train the model on a source-domain dataset with sufficient samples, and then transfer the model to target-domain datasets where only a few training samples are available for efficient finetuning. There are majorly two challenges in this task: (1) the domain gap and (2) finetuning with scarce data. To solve these challenges, we revisit the adapter-based methods, and discover an intriguing insight not explored in previous works: the adapter not only helps the fine-tuning of downstream tasks but also naturally serves as a domain information decoupler. Then, we delve into this finding for an interpretation, and we find the model's inherent structure could lead to a natural decoupling of domain information. Building upon this insight, we propose the Domain Feature Navigator (DFN), which is a structure-based decoupler instead of loss-based ones like current works, to capture domain-specific information, thereby directing the model's attention towards domain-agnostic knowledge. Moreover, to prevent the potential excessive overfitting of DFN during the source-domain training, we further design the SAM-SVN method to constrain DFN from learning sample-specific knowledge. On target domains, we freeze the model and fine-tune the DFN to learn knowledge specific to target domains. Extensive experiments demonstrate that our method surpasses the state-of-the-art method in CD-FSS significantly by 2.69% and 4.68% average MIoU in 1-shot and 5-shot scenarios, respectively.

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
Jintao Tong' Ran Ma
Yixiong T
Guangyao Chen
Yuhua Li
Ruixuan Li'
Abstract
Training Sumple
Tosting Sample
Testing Sample
Cross-domain few-shot segmentation (CD-FSS) is proposed pre-train the model source- domain dataset with sufficient samples, and then transfer the model to target-domain datasets where only a few samples are available for ef- ficient fine-tuning. There are majorly two chal- lenges in this task: (1) the domain gap and (2) fine-tuning with scarce data. To solve these chal- lenges, revisit the adapter- based methods, and discover an intriguing insight not explored in previous works: the adapter not only helps the fine-tuning of downstream tasks but also naturally serves as a domain information decoupler. Then. we delve into this finding for interpretation, and find the model's inherent structure could lead a natural decoupling of domain information. Build- ing upon this insight, we propose the Domain Fea- ture Navigator (DFN), which is structure-basee decoupler instead of loss-based ones like current works, to capture domain-specific information, thereby directing the model's attention towards domain-agnn aagnosicc knowledge. Moreover, pre- vent the potential excessive overfitting of DFN during the source-domain training, we further de- sign the SAM-SVN method to constrain DFN from learning sample-specific knowledge. On tar- get domains, we freeze the model and fine-tune the DFN to learn target-specific knowledge spe- cific. Extensive experiments demonstrate that our method surpasses the state-of-the-an method in CD-FSS significantly by 2.69% and 4.68% MIoU in 1-shot and 5-shot scenarios, respectively.
Sounce
Tepet Deawis festing
teels tralniing slage
testing stege
map
Encoder
Lear
feature extractor
desmii sppoolic daconia speeifie kaontedge
others
mp
Decoder
Captire dermuts specilic kanwehedge
map
Encoder
our
mg
Decocirr
Figure 1: Cross-domain few-shot segmentation (CD-FSS) aims to transfer the source-domain- soarre-domai--trinnedd model to tar- get domains for efficient learning with scarce data. By inserting adapters into the common network structure (e.g.. HSNet) for CD-FSS, previous works: adapter naturally serves domain infor- mation decoupler based on its structure instead of training losses, which "grabs" domain information from the en- coder+decoder structure and encourages the model to learn domain aagnosic information on the source domain.
find an insight not explored in
et al., 2015; Zhao et al., 2017; Yuan al., 2020) have driven the rapid progress of large vision models (Dosovitskiy et al., 2020; Kirillov et al., 2023; Zhang et al., 2024), resulting in impressive segmentation task outcomes. However, when applied downstream tasks, these models face significant challenges when there is substantial distributional differ- ence between upstream pretraining data and downstream data, where collecting downstream data may be difficult. To address this issue, the Cross Domain Few-Shot Segmenta- tion (CD-FSS) task (Lei et al., 2022) has been introduced (see Figure 1 top). CD-FSS involves pre-train a model on a source-domain dataset and then adapting to gener- ate pixel-level predictions for unseen categories in target- domain datasets with only few annotated samples, which still remains challenging.
1. Introduction
In recent years, advancements in large-scale annotated datasets and deep neural networks (Chen et al., 2014; Long
'School of Computer Science and Technology, Huazhong Uni- versity of Science and Technology PPeking University, Correspon- dence to: Yixiong Zou <yixiongz hust.edu.cn..
There are majorly two challenges in the CD-FSS task: (1) a huge domain gap between source and target domains, mak- ing difficult for the generalization from the source dataset to the target dataset; and (2) extremely limited target domain
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
data, making it challenging for the model to adapt to the distribution of the novel domain. To address the second challenge, currently, a group of methods based on adapters have been proposed (Houlsby et al., 2019; Mahabadi et al., 2021; Hu et al., 2021), which fixes the backbone network and only finetune the extra appended structures. In this pa- per, we revisit the adapter-based methods, and discover an intriguing insight not explored in prior works: adapters not only help the fine-tuning on downstream tasks but also nat- urally serves as domain information decoupler. This finding indicates that the adapter can address two challenges simultancously: (1) by decoupling the source domain in- formation into a Nomain-agnostio part, which aids in gen- eralizing from the source to the target domain, and (2) by
To the best of our knowledge, we are the first to discover the phenomenon that the adapter naturally serves decoupler, which we then delve into for an interpretation. Building upon this finding and interpretation, we pro- pose the DFN to decouple source domain information into domain-agnostic knowledge and domain-specific one, solely based on the adapter's 's structure and position.
We propose the SAM-SVN to avoid the potential exces- sive overfitting introduced by source-domain training of the DFN, while maintaining the DFN's absorption of do- main information.
Extensive experiments show the effectiveness of our work on four different CD-FSS scenarios. Our model signifi- cantly outperforms the state-of-the-art method.
parameter-cfficient fine-tuning to adapt to downstream data.
In this paper, we first delve into this phenomenon for interpretation. We first conduct experiments to verify which factors determine the adapter serving as a decoupler, includ- ing the adapter's insertion position and structure. We find this phenomenon holds only when adapters are inserted into deeper layers of the backbone network with scratch training and residual connections, without applying any domain- decoupling loss. This indicates the structural design could de-
2. Adapter Naturally Serves as Decoupler: Phenomenon and Interpretation
In this section, we conduct experiments to demonstrate that an adapter can naturally act as a decoupler. Furthermore, we examine which type of adapter is suitable for this role and investigate the underlying reasons. Following the standard CD-FSS setting, we use Pascal (Shaban et al., 2017) as the source dataset and the other four datasets as target datasets.
lead to the inherent capability of Nomain-informatiie coupling, which inspires us to design a structure-base do- main decoupler, instead of the loss-based decoupler adopted by current works (Tzeng et al., 2015; Motiian et al., 2017; Kang et al., 2019; Lu et al., 2022).
2.1. Adapter Decouples Domain Information
The network structure studied in this paper is shown in Fig.2 (top). which consists of a backbone network, an encoder, and a decoder. We choose this structure because it is widely recognized as versatile architecture (Min et al., 2021). We first attempt attach simple adapter (implemented as x 1 convolution) to the backbone with the residual connection, then train them jointly in the source domain.
Based on these findings and interpretations, we propose the Domain Feature Navigator (DFN, Fig. 1), structure- based domain decoupler built on adapters with specific structures and positions. In the source-domain phase, DFN absorbs domain-specific knowledge without any domain- decoupling losses, directing the model's attention toward acquiring donain-agnostic information. Then, during the target-domain phase, we fine-tune the DFN to capture target- specific features. The fusion of domain-specifee features with the model's domain-agnostic features serves to align the feature spaces for each domain.
Then, given ResNet50 (He et al., 2016) as the backbone, we measure the domain similarity by the CKA' (Kornblith et al., 2019; Zou et al., 2022; Tong et al., 2024b; Zou et al., 2024c) similarity (lower values indicate more domain- specciii in- formation) based on the backhone's s stage-4 output before and after attaching the adapter (Fig.2). As shown in Table I (row 1 after training with an adapter attached, the CKA is lower, which means the adapter captures Nomain-specifie information, therefore decreasing the domain similarity.
However, the implicit absorption of domain information could potentially lead to excessive overfitting of source- domain samples, as the learning of domain information can also be understood as a kind of overfitting (to the source do- main), but there are no labels to guide the magnitude of over- fitting like other loss-based decouplers. To further address this problem, we introduce SAM-SVN to constrain the DFN from excessive overfitting in source-domain training. Specif- ically, we tailor the sharpness-awaa minimization (Foret et al., 2020) to apply it to the singular value matrix of the DFN. By doing so, excessive overfitting is avoided but the absorption of domain information is maintained.
Subsequently, we study the change that the adapter brings to the output of the encoder (which is different from the backbone network). We also measure the domain similarity by the CKA similarity through the encoder's 's output with and without the adapter attached to the backbone network (BKB), As shown in Table I (row 3,4), the CKA is higher after the adapter is attached, which implies that the encoder focuses more on the domain-agnostic information.
Without the adapter (Fig. 2 top), the encoder needs si-
To sum up, our primary contributions are as follows:
Seee the Appendix for detailed formulations.
2

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
Stege adlater
Encoder culput
Seare
wh's adagieee
Damala
Target
net
net
Decull
Fiadd Rockhone
Eacoder
Deooler
fixed
fived BKB
fixed BKB
Theeture betrector)
Siatilerity
lager
layer
Sinsfartte
Figure Three different positions for adapters.
adapter
Fncoder cuiput edapter
Seare
Posttoo
BK8 shallooar
RKB deeper
betwecnn
E
enooder
encodde
Terpst
FSS 1600
0,5371
016709
0.9679]
0
0,4788
037333
0,5371
0,0695]
Deentt
Despulabe
0.4147
0.0498
0.3736]
0.0452.
0.3657]
0.0679
04147:
0.0459
Faeel Bus Muose
Enceder
Stndlarity f
Dreanit
Decader
ISIC
0,5266
0.0678
0.9028]
0.0633,
0,4653]
0.07247
0.5266
0.0658
Figure 2: Network structure studied in this paper, we an- alyze stage-4 and encoder outputs to study the absorbed domain information by measuring the domain similarity.
Chest X-my
0.4866
0.0844
0.4699
0.0512.
0.4390
0.036)
0.4865-
0.0539
Table 2: Verify the impact of different insertion points of
the adapter on its decoupling ability by domain similarity.
datascts
FSS-1000
Deepglobe
ISIC
ChestX
(2) is positioned between the fixed, pretrained backbone and subsequent learnable modules. The pretrained backbone is fixed on the source domain, while the adapter is trained from scratch. Such a difference guides the backbone network to extract general features and leads the adapter to learn more from the source domain to capture domain information.
stage-4 (w/o adapter)
0.5371
0.4147
0.5266
0.4865
stage-4 (wd adapter)
0,4605L
0.3426L
0.4459
0.3875L
encoder (wlo BKB adapter)
0.0709
0.0498
0,0678
0.0554
encoder (w/ BKB adapter)
0.07331
0.06797
0.07241
0.0626
Table I:
Domain similarities for the output of stage-4 or encoder between Pascal and target datasets with and without adapter (Fig. 2). The adapter captures domain-specifer infor- mation, guiding the encoder and decoder to Iearn domain- invariant knowledge. BKB: backbone network.
Comparing insertion methods (1) and (2), the distinction that (2) is located in the deeper layers of the fixed backbone, whereas (1) is positioned in the shallower layers. As is well known, the deeper the neural network is, the more semantic information its features can encompass. For cross-domain tasks, the knowledge learned by deeper network layers tends to be more domain-specifie. To verify it, we first visualized feature maps with different examples using insertion method (2), and the results are shown in Fig.4. For each example, the adapter outputs more complex features focus on the objects' profile, such as the eagle's 's wings, fish tail and fins, and the clock face of the clock tower, which means that inserting the adapter into deeper layers enables it to capture features that are more semantic and complex.
multaneously capture both domain-specifie and domain- agnostic information extracted by the feature extractor. Af- ter attaching the adapter (Fig. bottom), the adapter cap- tures domain-specific information, therefore decreasing the BKB's domain similarity. Consequently, the encoder pa- rameters could focus less on the domain-specifer informa- tion and pay more attention to the domain-agnostic one, increasing the domain similarity. In other words, the adapter decouples the domain information from the original model.
2.2. Why adapter can serve as a decoupler?
Image
Feat
Adapter
Image
Adapter
The above phenomenon inspires us to ask: Can all types of adapters decouple features, and why are adapters able to decouple features? In this section, we delve into this phe- nomenon by exploring two factors that determine whether an adapter can be decoupler: position and structure.
2.2. 1. POSITION
We categorize the insertion positions of adapters into three types as shown in Fig 3: (I) shallow layers of the fixed back- bone; (2) deep layers of the fixed backbone; (3) between learnable encoder and decoder. We use CKA to assess if position affects an adapter's 's decoupling ability.
Figure 4: The visualization results of the feature maps with and without the adapter attached to the backbone network.
Furthermore, as shown in Table 3, we evaluate the CKA similarity between features extracted at deep layers and that of the first convolution layer (convl) in the source domain following (Zou et al., 2022). The higher the CKA similarity, the simpler the features are. The row results indicate that the deeper the layer is, the lower the CKA would be. This shows high-level features are more complex, which means it could be more overfitting to data, e.g., domain information. The second row's adapters are attached to the convolutional layer
Following Table 1, if the CKA of the backbone output de- creases and that of the encoder output increases, we can recognize the attached adapter as decoupler. As shown in Table 2, the adapter can serve as a decoupler only when the adapter is inserted into the deeper layers of the backbone.
Comparing the positions (2) and (3), the distinction is that
3

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
backbone
conv1
conv3.4
conv4.6
conv5_3
ing domain information) to be transmitted subsequent modules. In contrast, the serial connection allows only the adapter's domain-specifee features to be transmitted to the subsequent learnable modules.
output
1.0
0.8768
0,8496
0.8164
adapter oulput
1.0
0.8707
0.8364
0.7936
Table 3: Validate the complexity of different layers' features by the similarity with the first-layer feature.
Conclusion for structure: the residual connection explic- itly partitions the domain-specific features learned by the adapter from the general features extracted by the backbone, before jointly feeding them into subsequent learnable struc- tures, thereby promoting subsequent modules to focus on invariant information. Therefore, the residual connection is crucial for an adapter to act as a decoupler, while structural differences do not affect its decoupling capability.
in the first row. Comparing these two rows, we can see that the CKA is lower after the feature passes through the adapter, which means the adapters' features are more complex and could capture more domain-specifie knowledge.
Conclusion for position: (1) The disparity between the backbone network (pretrained, fixed) and the adapter (leam- able, from scratch), enables the adapter to acquire specific features during training. (2) The semantics become richer in deeper layers, for cross-domain tasks, the knowledge learned by deeper network layers tends to be more domain- specific. These two factors enable the adapter to act as decoupler when it is inserted into the deeper layers of the fixed, pretrained backbone network.
Discussion. Inserting the adapter with residual connections deep into the pre-trained and fixed backbone, and before subsequent learnable modules, enables it to act as domain knowledge decoupler. The above findings and interpreta- tions indicate the structural design of the feature extractor can lead to the inherent capability of decoupling, which is different from the decoupling achieved by applying domain losses adopted by current works (Moti- ian et al., 2017; Kang et al., 2019; Lu et al., 2022). This inspires us to design a structure- based decoupler further to make better use of such inherent capability.
2.2.2. STRUCTURE
As shown in Figure 5, we explore two design structures, con- ventional and LoRA, and two connection structures, serial (ser) and residual (res). We adopt CKA to assess if structure affects an adapter's decoupling ability shown in Table 4.
Method
PSS
Deepglobe
ISIC
ChestX
Method
loss fluc.
Baseline
1.43
2.18
2.22
L.77
Baseline
0.298
design structure
connection structure
Baseline adapier
1.76
3.12
2.03
Boscinntad.
0.521
Table 5: Sharpness of loss landscapes measured by the loss fluctuations, where higher sharpness means higher tendency of overfitting. Left: Perturbation by parameter initialization. Right:
DNN
DNNN
(1) conventional
(2) LoRA
(1) serial
(2) residual
Perturbation by Gaussian noises.
Figure 5: Two different design and connection structures.
However, solely the structure- based decoupler can also lead to problems: the implicit absorption of domain informa- tion has no guarantee for its correctness. For example, the adapter generally absorbs deeper and more complex patterns that are more likely to be domain information, but also en- ables the model to overfit specific training samples by such complex patterns, which is harmful for downstream general- ization and adaptation. To validate this risk, we are inspired by (Foret et al., 2020; Zou et al., 2024a) to test the sharp- ness of the model's loss landscapes to verify its tendency of overfitting, as shown in Fig. (Flatten loss landscape) The perturbation is added in two ways: different initialization of the model parameters in Tab. 5 (left) and random Gaussian noise to the model parameters in Tab. 5 (right). We can see both kinds of perturbation lead to increased sharpness of loss landscapes, indicating a higher tendency to overfit training samples. Since absorbing domain information can also be understood as a kind of overfitting (to the source domain), there are no specific criteria to determine whether the overfitting is just for the domain instead of samples,
wro adapter
convensional-rrs
LoRA+res
conventionalseer
stucctre
sage-1
adapter
adupter
enc
adupter
FSS-1000
0.5371
0.0700
0,4005,
0.0733
0.4529.
0.07527
0,4006]
0.0633
Deapglate
0,4147
0.4998
0.3426,
6.679
0.3503,
0,0628
0.3932)
0.415)
I8IC
0.5266
0.0688
0,4459_
0,07245
0.4537_
0,0005
0,4752]
0,0683)
CheeX
D.4865
0.084
0,3875.
6.0626-
0,40211
0.09977
0,43101
0.0506]
Table 4: Assess the impact of different adapter structures on their decoupling capabilities by domain similarity.
Both design structures lead to reduced CKA for features passing through the adapter and increased CKA for features through the encoder. indicating that the adapter learned domain-specific knowledge while the encoder focused on domain-agnostic information. However, changing the con- nection from residual to serial leads to a decrease in the encoder output feature's CKA, making more domain- specific. This suggests that adapter's ability to decouple domain information is independent of its design structure and solely related to its connection structure.
We hold that this is because the residual connection explic- itly separates and integrates the backbone feature (contain- ing general information) and the adapter feature (contain-
"See the Appendix for a detailed explanation of sharpness.
4

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
P
domain-specific
NF
bypecsrraletion pynarid
DEN'S internal siracture
SAM: Flation has landecape
DEN
NIV
domain domninn-gnostic paramelers
MF
C
Cesine Samierity
NFW
feature extracttio
camec
a
F
NFA
contatt chrccdn
NFA
XAM-SVN
Weiighes
Emgal
SVD
SAM
SANS
Figure 6: Overview of our method in I-shot example. After obtaining the pyramid features of support and query images, DFN is introduced to accumulate the domain-specifie knowledge then encouraging the model to learn parameters. Moreover, we propose SAM-SVN to eliminate potential excessive overfitting introduced by source domain training on DFN. The internal structure of DFN and SVN-SAM is highlighted in green.
unlike the loss-based decoupler where the model is guided by the domain labels. Therefore, during source-domain training, we need to constrain the adapter from excessive overfitting (to enhance stability during target-domain fine- tuning), while also preserving its decoupling capability (to address the domain gap).
such conditions as the proposed DFN.
Therefore, we use a feature extractor to extract features and attach DFN the features by residual connection (see Figure.6 green box for DFN's internal structure). Con- cretely, let Na denote the module DFN with parameters a E RCxxxIxX x i.e., N₂ is implemented as convolu- tional operation with 1 x kernels. It maintains the same number of input and output channels. fo denote the fea- ture extractor, a sequence L pairs of intermediate feature maps {(F),F) FF))== is produced by fa. We then mask each support feature map Ff RCi XixxH xW, using the support mask M E {0, 1] #x to discard irrelevant activations for reliable mask prediction:
3. Method
Building upon the insight that adapters can naturally serve as decouplers, we propose DFN, variant of adapters, to simultaneously handle the domain gap and data scareity problem for CD-FSS. As in Fig 6, during the source-domain training, the DFN is jointly trained with the base model, we connect the DFN to low, middle, and high-level features separately to ensure semantic consistency. Moreover, we design SAM-SVN to prevent the risk of excessive overfitting on DFN introduced by source-domain training that hinders downstream generalization and fine-tuning.
= fo( Ff G(M")
(1)
where E RCXHXW is the input tensor, C is Hadamard product, and (1()) is a function that bilinearly interpolates input tensor to the spatial size of the feature map Ff at layer followed by expansion along channel dimension such that G 2 xW + IR Ci xCCxHiiWW
3.1. Problem Definition
Given a source domain D. = (P(X,), Y2) and a target domain D. = (P(X.), Y.), where P(X) represents input data distribution and represents label space. Each class C E D. has sufficient labels while C, D, has only limited labels. Notably, the source domain D. and target domain D, exhibit distinct input data distribution, with their respective label spaces having no intersection, i.c., P(X,)= # P(Xt), Y, n Y2= = 0. The model will be trained on the dataset from the D.. then applied to segment novel classes in the D,.
The final outputs of navigated feature maps are:
(NFT) = ( - NN.
(2)
(3)
where NF; is the feature that has been aligned the target domain feature space after passing through the Na.
For the subsequent hypercorrection pyramid construction, a pair of navigated query features and navigated masked support features at each layer forms a 4D correlation tensor C E Hi WixHixW; using cosine similarity:
3.2. DFN: Domain Feature Navigator
The role of DFN is to absorb domain information to encour- age the model to acquire domain- domain-agnostie information on the source domain, and adapt the model to target domains. Since in section 2 we have concluded under what condition an adapter can be decoupler, term the adapter satisfying
Gim,, n) = ReLU NF) NFF (n) (m)]] (m)]]
(4)
where m, n denote 2D spatial positions of navigated feature maps (i.e. have been decoupled by DFN) NF and NP
5

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
respectively. Then, the (mm,, n) is fed into the 4D convolu- tional pyramid encoder and the 2D convolutional decoder to obtain segmentation results, as shown in Figure 6.
neighborhood defined by an lp ball around matrix sp
In practice, SAM uses p 2 and approximates the inner maximization by a single gradient step, yielding:
3.3. SAM-SVN: Perturbing only the Singular Value of the Domain Feature Navigator
é c <=
(6)
The absorption of domain information by solely the based adapter could lead to problems: if DFN learns overly complex patterns, it may overfit source sam- ples rather than the source domain, represented as increased sharpness against perturbations, as verified in Table 5.
and requiring an additional pared to SGD. The gradient is then re-evaluated at the per- turbed point + e, giving the actual weight update:
-aaekadd pass com-
â USS +)VTT w ー w - VVL(W, â) a イ a - VVL(w, a)
(8)
To handle this problem, we are inspired by the Sharpness- Aware Minimization (SAM) (Foret et al., 2020), which was proposed reduce the sharpness of minima and has been shown to resist overfitting in various settings (Kaddour et al., 2022; Mueller et al., 2023). However, simply resisting the sharpness can also limit the absorption of domain informa- tion, since the learning of domain information can also be understood as a kind of overfitting (to the source domain).
(9)
In this endeavor, SAM is applied to the singular value matrix S obtained through the SVD of DFN. The prediction BCE loss L is calculated using the updated {w, a).
During the target-domain finetuning stage, we fine-tune the DFN to learn domain--pecinf features, the fusion of domain- specific features with the model's domain-agnostic features serves to align the feature spaces for each domain.
To overcome this issue, inspired by SAM-ON (Mueller et al., 2023), which perturbs only the normalization layers, and building on BSP (Chen et al., 2019), which proposed that singular values control the importance of different represen- tations during transfer, we developed a tailored enhancement for the DFN, called SAM-SVN (Fig.6, green box for SAM- SVN). In essence, we first perform singular value decom- position on the DFN, then apply SAM the singular value matrix. Since only the most overfitting-sensithe parameters (singular values) are constrained, the remaining parameters are still capable of absorbing domain information.
4. Experiments
Datasets. We adopt the benchmark proposed by PATNet (Lei et al., 2022) and follow the same data preprocessing pro- cedures. We employ PASCAL (Shaban et al., 2017), which is an extended version of PASCAL VOC 2012 (Evering- ham et al., 2010), as our source-domain dataset for training. We regard FSS-1000 (Li et al., 2020), Deepglobe (Demir et al., 2018), ISIC2018 (Codella et al., 2019; Tschandl et al., 2018), and Chest X-ray (Candemir et al., 2013; Jaeger et al., 2013) target domains for evaluation.
In detail, for a DFN N₂ with C₁ input channels, C. output channels, and a kernel size of K x K, we first fold its weight tensor E x into a matrix a' E RC,xC,k" XC.K? then decompose the obtained matrix by applying SVD with full-rank in subspaces (rank R = min(C. C,K?) ). Thus,
Implementation Details. We employ ResNet-50 (He et al., 2016) pre-trained on ImageNet (Russakovsky et al., 2015) our backbone, with its weights frozen during train- ing, following HSNet (Min et al., 2021). To optimize mem- ory usage and speed up training, we set the spatial sizes of both support and query images to 400 x 400. The model is trained using the Adam (Min et al., 2021) optimizer with learning rate of 1e-3. The hyperparameter p in SAM is set to 0.5. We also implemented our method in the trans- former architecture following FPTrans (Zhang et al., 2022), employing ViT (Dosovitskiy et al., 2020) as the backbone. The fine-tuning of DFN is performed using the Adam op- timizer, with learning rates set at le-3 for FSS-1000, 5e-1 for Deepglobe, 5e-3 for ISIC and Chest X-ray. Each task undergoes a total of 50 iterations.
(5) where U E RCCRR S IR RxR and VT E RRRCKK In our setting, we have K and input channels as same as output channels (i.e., C. C.). The obtained pair of matri- ces VT and U construct two new convolution layers, and S is a diagonal matrix with singular values on the diagonal.
a' = USVV
Let's redirect our attention towards the SAM. We consider a neural network fwr attached with the DFN Nr, as our model Mpp (p {w, a))). The training sample (zz y) consists of input-output pairs are drawn from the source domain D.. Following HSNet (Min et al., 2021), we employ the standard Binary Cross-Entropy (BCE) loss, denoted as L., to train the model M, aiming to minimize L over the training set. Conventional SGD-like optimization methods minimize L by stochastic gradient descent V. SAM aims at additionally minimizing the worst-case sharpness of the training loss in
4.1. Comparison with State-of-the-arts
In Table 6, we compare our method with several state-of-the- art few-shot segmentation methods on benchmark proposed by PATNet (Lei et al., 2022). The results reveal that the performance of existing few-shot semantic segmentation
6

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
FSS-1000
Decpglobe
ISIC
Chest X-ray
Average
Method
Mark
Backbone
I-shot
5-shot
1-shot
5-shot
I-shot
5-shot
1-shot
5-shot
I-shot
5-shot
CaNet (Zhang et al., 2019)
CVPR-19
Res-50
70.67
72.03
22.32
23.07
25.16
28.22
28.35
28.62
36,63
37.99
PANct (Wang ct al., 2019)
ECCV-20
Res-50
69.15
71.68
36.55
45.43
25.29
33,99
57.75
69.31
47.19
55.10
RPMMs (Yang et al., 2020)
ECCV-20
Res-50
65.12
67.06
12.99
13.47
18.02
20.04
30.11
30.82
31.56
32.85
PFENet (Tian et al., 2020)
TPAM1-20
Res-50
70.87
70.52
16.88
18.01
23.50
23.83
27.22
27.57
34.62
34.98
RePRI (Boudiaf et al., 2021)
CVPR-21
Res-50
70.96
74,23
25.03
27.41
23.27
26.23
65.08
65.48
46.09
48.34
HSNet (Min et al., 2021)
ICCV-21
Res-50
77.53
80.99
29.65
35.08
31,20
35.10
51.88
54.36
47.57
51.38
PATNet (Lei et al., 2022)
ECCV-22
Res-50
78.59
81.23
37.89
42.97
41.16
53.58
66.61
70.20
$6,06
61.99
PATNet (Lei et al., 2022)
ECCV-22
ViT-hase
72.03
22.37
44.25
76.43
53.77
RestNet (Huang et al., 2023)
BMVC-23
Res-50
81.53
84.89
22.70
29.90
42.25
51.10
70.43
73.69
54.22
59.89
PerSAM (Zhang al., 2024)
ICLR-24
ViT-buse
60.92
66.53
36,08
40.65
23.27
25.33
29.95
30.05
37.56
40.64
ABCDESS (Herzog, 2024)
CVPR-24
Res-50
74.60
76.20
42,60
45.70
45.70
53,30
79.80
81.40
60.67
64.97
APSeg (Hc et al., 2024)
CVPR-24
ViT-base
79.71
81,90
35.94
39.98
45.43
53.98
84.10
84.50
61.30
65.09
APM (Tong al., 2024b)
NeurIPS-24
Res-50
79.29
81.83
40.86
44.92
41.71
51.76
78.25
82.81
60.03
65.18
DEN (Ours)
Ours
Res-50
80.73
85.80
45,66
47.98
36.30
51.13
85.21
90.34
61.98
68.81
DEN (Ours)
Ours
ViT-buse
82.97
85.72
39.45
47.67
50.36
58.53
83.18
87.14
63.99
69.77
Table 6: MloU of 1-shot and 5-shot results on the CD-FSS benchmark. The best and second-best results are in bold and underlined, respectively. In the appendix E/F, we conduct experiments on hyper-parameters and computational complexity.
Deepglbbe PS--109
DEN
SAM
SVD
1-shot
5-dlac
apply SAM
I-shot
5-shot
47.57
51.38
Enc.+Dcc,+DFF
60.04
66.98
66.59
60.65
67.74
DFN
60.65
67.74
61.98
68.81
SVN
61.98
68.81
Table 7: Ablation study on various designs (left) and apply SAM on different modules (right). Enc: Encoder, Dec: Decoder, SVN: Singular Value matrix of the DFN.
ISIC
Chet Keng
around truth Figure 7: Qualitative results of our model for 1-shot setting.
prodiction
penaad truth
prudiction
methods degrades significantly under domain shifts. Table indicates a significant advancement in cross-domain se- mantic segmentation for both 1-shot and 5-shot tasks using our proposed approach. Specifically, we surpass the per- formance of the state-of-the-art APSeg by 2.69% (1-shot Ave.) and 4.68% (5-shot Ave.), respectively. Furthermore, we showcase qualitative results of the proposed method in 1-way s-ohot segmentation, as depicted in Figure 7.
Figure 8: Our designs reduce the feature's 's domain distance, enhancing the model's 's learning of agnostic knowledge.
plying SAM to different modules, detailed in Tab 7 (right). SAM was applied to DFN, Encoder, and Decoder collec- tively. exclusively to DFN, and on SVN. Results highlight the effectiveness of SAM on SVN. SAM prevents DFN from overfitting to the source domain, aiding fine-tuning but it may affect domain-specific knowledge capture. SAM-SVN strikes a balance, preventing DFN from overfitting to the source domain without compromising its ability to capture domain-speciffi information during training.
4.2. Ablation Study
Effectiveness of each component. We analyze each pro- posed component on the I-shot and 5-shot settings to val- idate the effectiveness of each design, including DFN and SAM-SVN (SAM applied on the Singular Value Matrix of Navigator, i.e., SAM and SVD shown in the table). The results, presented in Table 7 (left), indicate that introduc- ing DFN improved average MIoU by 12.32% for 1-shot and 10.21% 5% for 5-ohot, while adding SAM-SVN further in- creased it by 2.09% and 2.22%, respectively. These results demonstrate that each design in our approach significantly contributes to performance gains. (see appendix for ViT)
Improvement in model generalization. We evaluate our method's ' impact on model output using Maximum Mean Discrepancy (MMD') (Gretton et al., 2012) to measure do- main distance of target datasets with respect to the PASCAL. A lower MMD indicates reduced domain distance, meaning the model produces more agnostic features. Figure 8 shows
SAM perturbation targets. We explored the impact of ap-
"The CKA measure is in the appendix showing similar results.
7

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
ing
backbone
DFN
img
backbone
DEN
--DF
H DFN-GAMM-NN
20
13
0.25
Figure 9: Feature map visualization: (row 1) DFN guides model's s attention to more distinctive features. (row 2) DFN guides the model to perform more accurate segmentation.
Figure 10: After adopting SAM-SVN, the model exhibits re- duced sharpness and enhanced robustness to domain shifts.
Method
FSS. 1000
Deepglobe
ISIC
ChestX
that attaching DFN to the backbone reduces the MMD, and further it decreases after applying SAM-SVN. This sug- gests the model has acquired more agnostic representations, reducing the distance between domains.
Only DEN
1.76
2.33
3.12
2.03
DEN SAM
1.25
2.92
2.85
1.89
DEN SAM-SVN
0.94
1.53
1.68
1.18
Table 8: Train five best checkpoints on the same device and measure I-shot performance fluctuations (best worst).
4.3. DFN Guiding Model Attention
During source-domain training, the DFN absorbs domain- specific knowledge, directing the model's attention toward acquiring domain- agnostic insights. The impact of DFN is demonstrated through feature map visualization (Figure 9). In the first row, without DFN, the model focuses on general features such as the dog's head, limbs, and tail, and the swan's head. After passing through the DFN, the model is guided to pay attention to more distinctive and specific features, such as the dog's s head and tail, and the swan's 's wing. In the second row, the model focuses on various objects, namely the bird and the tree branch. After passing through the DFN, the model is guided to pay attention specifically to the object we want to segment (the bird).
5. Analssis of DFN Usage and Adapter Choice
Impact of DEN's usage manner. We examined different strategies for utilizing DFN: 1) using DFN in source domain training only, removing it in target domain; 2) using DFN during source training, but re- initializing its parameters and re-trained from scratch during target domain finetuning. Ta- ble shows that even when DFN removed in the target domain, it still improves performance over the baseline by guiding the model toward domain-agnostic representa- tions. However, this remains suboptimal due to the lack of adaptation to target-specific information. Re-initializing and retraining DFN during target finetuning, performance varies with the initialization method, yet remains consis- tently strong across all variants. Our approach (i.e., DFN + SAM-SVN) is seen as using DFN to guide the model to focus on domain-agnostic knowledge, while SAM-SVN pro- vides DFN with a reasonable initialization value beneficial for adapting to various domains.
4.4. SAM-SVN Reduces Sharpness and Improve Robustness to Domain Shifts
The motivation behind the SAM-SVN is primarily twofold: 1) to limit the DFN's acquisition of excessive source-specific knowledge; and 2) to avoid hindering the DFN's decoupling ability, as this might undermine the model's learning of invariant knowledge. The SAM-SVN can achieve both goals by reducing the sharpness of the training loss against DFN's parameter, because (1) it can prevent DFN from being vulnerable to domain shifts, and (2) it does not harm the minimization of the training loss.
1-shot Setting
ESS1000
Deepglobe
ISIC
ChestX
Mean
Baseline
77.53
29.65
31.20
$1.88
47.57
DFN (remove in target)
78.16
38.21
34.12
76.92
56.85
DFN (scratch, Kaiming)
79.02
42.53
36.63
82.03
60.05
(scratch, Xavier)
79.83
45.57
34.79
79.46
59.91
DFN (scratch, Gaussian)
78.97
38.75
33.82
78.59
57.53
To verify the reduced sharpness and improved robustness of domain shifts attributed to the SAM-SVN, as shown in Figure 10, we adopted style transfer methods to superim- pose target-domain styles as domain shifts (horizontal axis). The lowered performance drop (vertical axis) validates the flattened loss landscapes and the enhanced robustness to domain shifts. We also measured the impact of SAM-SVN on model's 's performance stability, as shown in Table 8. By constraining DFN from absorbing sample-specific informa- tion (flatten DFN's 's loss landscape) without affecting its ab- sorption of domain- -seeaiff- information. SAM-SVN helps improve model stability (lower performance fluctuation).
DEN (Ours)
80.73
45.66
36.30
85.21
61.98
Table 9: Impact of DFN usage manner on performance.
Impact of adapter choice on sharpness We further mea- sure the impact of different adapter choices on sharpness. Consistent with Figure and Figure 5, we analyze the re- sults based adapter position and structure (res: residual, ser: serial, BKB: backbone, enc-dec: encoder-decoder), The results in Table 10 and Table 11 show that, from the perspec- tive of sharpness, significant changes in loss fluctuations occur only when residual links are satisfied and the position is deep within the backbone (with the adapter structure not
8

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
Position
Baseline
BKB shallower
BKB deeper
Between enc-doc
For the adapter parameters 0g. the gradient is:
Loss flue.
0.398
0,402
0.521
0.405
ac Oc 0g(f(z)) 00, aF(r) 00g
Table 10: Impact of adapter position on loss fluctuations.
(16)
Structure
Baseline
conventional-res
LoRA+res
conventional+ser
Loss fluc. 0.398
0.521
0.533
0.399
Differentiated gradients: Due to the adapter's selective absorption of domain-specific information and the residual adapter structure, gradient flow naturally separates network optimization into two complementary learning objectives. (Detailed derivation in appendix A)
Table 11: Impact of adapter structure on loss fluctuations.
being a determining factor). This indicates that the adapter has captured domain-specific related information, which is consistent with the conclusions drawn in section 2.
7. Related Work
6. Theoretical Analysis for Adapter naturally Serves as Information Decoupler
Few-shot semantic segmentation FSS (Tong et al., 2024a) aims to segment unseen semantic objects in query images with only a few annotated samples. OSLSM (Shaban et al., 2017) contributes to the first two-branch FSS model. Fol- lowing this, PL (Dong & Xing, 2018) introduces a prototype learning paradigm, where predictions leverage the cosine similarity between pixels and prototypes. SG-One (Zhang et al., 2020) adopt the masked average pooling operation to enhance the extraction of support feature. HSNet (Min et al., 2021) employs efficient 4D convolutions on multi-level fea- ture correlations, significantly enhancing performance and serving as the baseline for our work. However, these meth- ods focus solely on segmenting novel classes within the same domain and struggle to generalize to unseen domains. Bridging the huge domain gap between source and target domains with limited labeled data, remains a challenge.
The behavior of adapters as domain information decouplers can be analyzed through the Information Bottleneck (IB)
theory. The IB objective is given by:
LB = 2) - (I)
(10)
where ((:+) denotes mutual information, X is the input, Y is the final output label, Z is the intermediate representation, and / is the trade-off parameter.
Assume input X can be decomposed into dovain-invariant and comminn-specife components X Xin + Xspes then the IB objective becomes:
LB = I(Xinv Xspec; - -I
(11)
Domain-invariant representation learning Domain- invariant representation learning has been widely adopted to address domain shifts. (Tzeng et al., 2015) introduces domain confusion loss and soft label render domains in- distinguishable in feature space, promoting the learning of domain-invann invariant features. (Motiian al., 2017) implements domain alignment and semantic alignment through CCSA loss. DIFEX(Lu et al., 2022) maximizes both invariant and domain-specifee features through additional regularization terms to better leverage invariant characteristics. These methods aim to induce models to acquire invariant knowl- edge through restrictive mechanisms such as losses and regularization. Diverging from these loss-based approaches, our work leverages the inherent properties of adapters to propose structure-basse domain knowledge decoupler.
For encoder-decoder (ED) network with parameters O and a adapter with parameters 0g where the adapter has a limited capacity (i.e., Of > 0g). Due to the lower capacity of the adapter (much smaller than the ED's parameters), tends to absorb primarily the domain- specciic signal, which more informative for the current training objective. This leads to:
I [X Zudapter)
>>
i Zudapter)
(12)
where Zudnpeer is the information processed or represented by the adapter. It promotes gradient flow differentiation. The forward function of the residual adapter structure is:
F(x) f(x) + (x))
(13)
where g(f(x)) is the output of the adapter. The gradient w.r.t. the encoder-decoder parameters 0f is:
8. Conclusion
ac ac 0F(x) of(x)
(14) In this paper, we introduce a novel perspective: adapter natu- rally serve as domain information decoupler. Based on this, we propose the DFN to guide the model's attention towards acquiring comain-agnosti information. Additionally, we introduce the SAM-SVN to prevent the potential excessive overfitting on DFN introduced by source-domain training that hinders the acquisition of domain-specifis knowledge during fine-tuning. Experimental results demonstrate the effectiveness of our approach in bridging domain gaps.
00,
(F() 0f(x) 00,
Expanding the middle term:
0F(x) I+ 0g()(z)) 0f(z)
(15)
0f(z)
where is the identity matrix (from the direct residual path), and the second term is the Jacobian matrix of the adapter function with respect to (x)

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
Acknowledgements
istration. IEEE transactions on medical imaging, 33(2): 577-590, 2013.
This work supported by the National Natural Science Foundation of China under grants 62206102; the National Key Research and Development Program of China un- der grant 2024YFC3307900; the National Natural Science Foundation of China under grants 62436003, 62376103 and 62302184; the National Natural Science Foundation of China under grants 62402015; the Postdocotoral Fel- lowship Program of CPSF under grants G2B20230024; the China Postdoctoral Science Foundation under grant 2024M750100; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Sci- ence and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation is completed in the HPC Platform of Huazhong University of Science and Technology.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv: /4:i::::42..110 1412. 2014.
Chen, X., Wang, S., Long, M., and Wang. J. Transferability vs. discriminability: Batch spectral penalization for ad- versarial domain adaptation. In International conference on machine learning, pp. 1081-1090. PMLR, 2019.
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: 03368, 2019.
Impact Statement
Based our novel perspective that adapters naturally serve as decouplers, we propose structure-based decoupling method to address the CD-FSS problem. This approach can also he applied other areas such as few-shot learning and domain adaptation, since the widespread use of adapters in various fields. The limitation of this work is its oversight of many-shot scenarios, which have significant practical value in real-world applications. Nonetheless, our method is adaptable and can be further optimized to support many-shot scenarios in the future.
Demir, I., Koperski, K., Lindenbaum, D., Pang, G., Huang, J., Basu, S., Hughes, F., Tuia, D., and Raskar, R. Deep- globe 2018: A challenge to parse the earth through satel- lite images. In Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition Workshops, pp- 172-181, 2018
Dong, N. and Xing, E. P. Few-shot semantic segmentation with prototype learning. BMVC, volume 3, 2018.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x16 16 Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
References
words:
An. Z., Sun, G., Liu, Y., Li, R.. Wu. M., Cheng, M.-M., Konukoglu, E., and Belongie, S. Multimodality helps few-shot 3d point cloud semantic segmentation. arXiv preprint arXiv:2410.224 224299 2024a.
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
An, Z., Sun, G., Liu, Y., Liu, F., Wu, Z., Wang, D., Van Gool, L., and Belongie, S. Rethinking fho-shot 3d point cloud semantic segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 3996-4006, 2024b.
Fan, Q., Pei, W., Tai, Y.W.,, and Tang, C.-K. Self-support few-shot semantic segmentation. In European Conference on Computer Vision, pp. 701-719. Springer, 2022.
Boudiaf, M., Kervadec, H., Masud, 7. I., Piantanida, P., Ben Ayed, I., and Dolz, J. Few-shot segmentation with- out meta-learning: A good transductive inference is all you need? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13979- 13988, 2021.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Interna- tional conference on machine learning, pp. 1135. PMLR, 2017.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness- awwre minimization for efficiently improving generalization. arXiv preprint 2xXiv:001000141: 2020.
Candemir, S., Jaeger, S., Palaniappan, K., Musco, J. P., Singh, R. K., Xue, Z., Karargyris, A., Antani, S., Thoma, G., and McDonald, C. J. Lung segmentation in chest radiographs using anatomical atlases with nonrigid reg-
Garcia, V. and Bruna, J. Few-shot learning with graph neural networks. arXiv preprint arXiv: 1.04043, 2017.
10

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. A kemel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1900000 1910.0927. 2019.
Guo, Y., Codella, N. C., Karlinsky, L., Codella, J. V., Smith, J. R., Saenko, K., Rosing, T., and Feris, R. A broader study of cross-domain few-shot learning. In Computer vision-ECCV 2020: 16th European conference, glasgow, UK, August 23-28, 2020, proceedings, part XXVII 16, PP- 124-141. Springer, 2020.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehcad, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4022 2023.
Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., and Malik, Semantic contours from inverse detectors. In 2011 international conference on computer vision, pp. 991- 998. IEEE, 2011.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similar- ity of neural network representations revisited. In Interna- tional Conference on Machine Learning, pp. 3519-35155555 3529. PMLR, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Lei, S., Zhang, X., He, J., Chen, F., Du, B., and Lu, C.- T. Cross-domain few-shot semantic segmentation. In European Conference Computer Vision, pp. 73-90. Springer, 2022.
Li, X., Wei, T., Chen, Y. P., Tai, Y.-W., and Tang, C.-K. Fss- 1000: A 1000-class dataset for few-shot segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2869-2878, 2020.
He, W. Zhang, Y., Zhuo, W., Shen, L., Yang, J., Deng, S., and Sun, I.. Apseg: Auto-prompt network for cross- domain sho-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23762-23772, 2024.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431 3431-3440, 2015.
Herzog, J. Adapt before comparison: A new perspective on cross-domain few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23605-23615,, 2024.
Lu, W., Wang, J., Li, H., Chen, Y., and Xie, X. Domain- invariant feature exploration for domain generalization. arXiv preprint arXiv:2207. 12020, 2022.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Mahabadi, R. K., Ruder, S., Dehghani, M., and Hender- son, J. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint 0rXiv:2106.04489 2021.
Hu, J., Shen, Y., Wallis, P., Allen-Zhtt -Zu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Min, J., Kang, D., and Cho, M. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 6941- 6952, 2021.
Huang, X., Zhu, C., and Chen, W. Restnet: Boosting cross- domain few-shot segmentation with residual transforma- tion network. arXiv preprint arXiv:2308 13469, 2023.
Motiaa,, S., Piccirilli, M., Adjeroh, D. A., and Doretto, G. Unified deep supervised domain adaptation and gen- eralization. In Proceedings of the IEEE international conference on computer vision, pp. 5715-5725, 2017.
Jaeger, S., Karargyris, A., Candemir, S., Folio, I.., Siegel- man, J., Callaghan, F., Xue, Z., Palaniappan, K., Singh, R. K., Antani, S., et al. Automatic tuberculosis screening using chest radiographs. IEEE transactions on medical imaging. 33(2):233-22) 245, 2013.
Mueller, M., Vlaar, T., Rolnick, D., and Hein, M. Normal- ization layers are all that sharpness-aware minimization needs. arXiv preprint arXiv: 040000044222... 2023.
Kaddour, J. Liu, L., Silva, R., and Kusner, M. J. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577-16595, 2022.
Nie, J., Xing, Y., Zhang, G., Yan, P., Xiao, A., Tan, Y.P.,, Kot, A. C., and Lu, S. Cross domain shot segmenta- tion via iterative support-query correspondence mining.

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 3380-3390, 2024.
Yuan, Y., Chen, X., and Wang. J. Object-contextual rep- resentations for semantic segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glas- 8ow, UK. August 23-28, 2020, Proceedings, Part VI 16, pp. 173-190. Springer, 2020.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211-252,2011
Zhang, C., Lin, G., Liu, F., Yao, R., and Shen, C. Canet: Class-agnostic segmentation networks with iterative re- finement and attentive few-shot learning. In Proceedings of the IEEE/CVT conference on computer vision and pat- tem recognition, pp. 5217-5226, 2019.
Shaban, A., Bansal, S., Liu, Z., Essa, I., and Boots, B. One- shot learning for semantic segmentation. arXiv preprint 10X1::1709.034100 2017.
Zhang, J.-W., Sun, Y.. Yang, Y., and Chen, W. Feature- proxy transformer for few-shot segmentation. Advances in neural information processing systems, 35:6575-6588,, 2022.
Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Zhang, R., Jiang, Z., Guo, Z., Yan, S., Pan, J., Dong, H., Qiao, Y., Gao, P., and Li, H. Personalize segment any- thing model with one shot. In The Twelfth International Conference on Learning Representations, 2024.
Tian, Z., Zhao, H., Shu, M., Yang, Z., Li, R., and Jia, J. Prior guided feature enrichment network for few-shot segmentation. IFFE transactions on pattern analysis and machine intelligence, 14(2):0050-1065, 2020.
Zhang, X., Wei, Y., Yang, Y., and Huang, T. S. Sg-one: Sim- ilarity guidance network for one-shot semantic segmenta- tion. IEEE transactions on cybernetics, 50(9):3855-3855, 2020.
Tong. J., Zhou, H., Liu, Y., Hu, Y., and Zou, Y. Dynamic knowledge adapter with probabilistic calibration for gen- eralized few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2781 17790, 2024a.
Zhao, H., Shi, Qi, X., Wang, X., and Jia, J. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881- 2890, 2017
Tong, J., Zou, Y., Li, Y., and Li, R. Lightweight frequency masker for gross-domain few-shot semantic segmentation. Advances in Neural Information Processing Systems, 37: 2024b.
Zou, Y., Zhang, S., Li, Y., and Li, R. Margin-based few-shot class-incremental learning with class-level overfitting mit- igation. Advances in neural information processing sys-
Tschandl, P., Rosendahl, C.,, and Kittler, H. The ham 1 0000) dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
tems, 35:27277-272799 2022.
Zou, Y., Liu, Y., Hu, Y., Li, Y., and Li, R. Flatten long- range loss landscapes for cross-domain few-shot learning, 2024a.
Tzeng, E., Hoffman, J., Darrell, T., and Saenko, K. Si- multancous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on com- puter vision. pp. 4068 0066-4076722 2015.
Zou, Y., Ma, R., Li, Y., and Li, Attention temperature matters in vit-based cross-domain few-shot learning. Ad- vances in Neural Information Processing Systems, 37: (16332-116355 2024b.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Zou, Y., Zhang, S., Zhou, H., Li, Y. and Li, R. Composi- tional few-shot class-incremental learning. arXiv preprint 2xXhv:2405.17022 2024c.
Wang, K., Liew, J. H., Zou, Y., Zhou, D., and Feng, J. Panet: Few-shot image semantic segmentation with prototype alignment. In proceedings of the IEEE/CVF international conference on computer vision, pp. 9197-9206, 2019.
Yang, B., Liu, C., Li, B., Jiao, J., and Ye, Q. Prototype mixture models for few-shot semantic segmentation. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020. Proceedings, Part VIII 16, pp. 763-778. Springer, 2020.
12

---

Appendix for Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
A. Detailed Derivation of Adapter Naturally Serves as Information Decoupler
Step 5: Substituting the adapter's 's Jacobian approximation:
ac dc
-I Pappe] fmv (x)
f
2
00; +
(22)
Let the features fx) from the ED be conceptually decom- posable into domain-invarianl fin (x) and domain-specific fapso (x) components:
00, aF(x)
30;
Step 6: Expanding the terms involving the projection. Since
Pine ~ I - Pspecs we have:
((x) - finv(x) (2) + (x)
(17)
0c dc Pmm
(()((())
(x))
+
(23)
00, 0F(z)
00g
00,
We assume the existence of orthogonal projection opera- tors Piny and Ppee onto conceptual domain-invariant and domain-specific subspaces, respectively, such that finv R Pinv ff (x) and (x)) = P (1), with P.n. Popec = I and Piny TanPPpee spec R O.
Expanding this:
OC OC
Sim (x) Pin
(pppec())
Pine
(24)
00, OF(x)
00,
Step 7: Assuming approximate orthogonality of gradient
Step 1: Decompose the gradient of the ED's features f(x) w.r.t. its parameters f into contributions from changes in invariant and specific feature components:
components. If changes to fi)((x) due to Of (i.c., ddy lie in the invariant subspace, then:
ffu(())
Pine 0im((a) 3m
(25)
0f(x) f
(x)
0f (x)
00,
00,
(18)
00,
00,
If changes to fapec (x) due to Og (i.e.,
lie in the specific subspace (orthogonal to the invariant one), then:
0f
86,
This assumes that changes in f can conceptually lead to changes in both domain-invariant and domain-specific as- pects of the features f(x).
Panv b fxpc(())
~ 0
(26)
30,
Step Adapter learns mainly from domain- specific signals. As motivated by IB (Fq. 12), the adapter y primarily models domain-specifie information. Its parameters 0g are updated to capture domain-specife characteristics:
Substituting Eq.
25 and Eq. 26 into Eq. 24:
Uc OC a fmv
+0| = OF(x) 00,
ac fflv(z) (27)
00, OF(x)
00,
ac Og(s(z)) ac 00,-
This derivation shows that, under these assumptions, the gradient updates to the encoder-decoder parameters 0y are predominantly guided by the of the features. The adapter, by learning to approximate f()), effectively filters or projects out the domain- specific gradient components that would otherwise update 0y. Consequently, the ED network is primarily optimized toward learning domain-invarian representations, while the adapter itself (via Eq. 16) focuses on domain-specific adap- tation. This provides a theoretical foundation for viewing adapters as information decouplers.
(19)
aF(x) a0,
components
Step 3: The encoder-decoder receives gradients influenced by both its direct path and the adapter path. Substituting Eq. 15 and Eq. 18 into Eq. 14:
Oc
Oc
0g(f(x)) Of(x) | [
(fm((() 00,
Sfsecc (x)
I+
00, 0F(z)
00,
(20)
Step 4: Adapter approximates the negative of the projected domain-specific representation. The adapter y learns to model domain-specific aspects. We hypothesize gxf( 2
fpe (z). Using the projection, 9 9(f()) s Ppe (x). Thus, the Jacobian of the adapter function is (assuming Pspee
B. Centered Kernel Alignment (CKA)
CKA (Kornblith et al., 2019) is widely used metric for comparing the similarity between two data representations by normalizing the Hilbert-Schmin Independence Criterion (HSIC). This normalization mitigates the effects of scaling differences between the two representations.
acts as an approximately linear operator or its dependency on ff fx) is negligible for this derivative):
0g(f(z)) 0f(x) R - -Psccc
(21)
13

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
B.1. Dot Product-Based Similarity
C. Adapter Naturally Serves Decoupler
CKA builds on the concept of dot product-based similarity, which relates inter-example similarities to feature similari- ties. Given two representations X Ieddd and Y gaxd the dot product similarity between examples is:
The deeper the neural network is, the more semantic infor- mation its features can encompass, meaning can better represent category information. Different categories exhibit more distinctive semantic information at higher layers. We validate it for CKA measure in paper, here, we further illus- trate this perspective by visualizing feature maps. As shown in Fig. I I, the two examples illustrate that as the depth of the layers increases, the model focuses more on specific objects, extracting features that are more discriminative.
(Xe ), veo( YY T) = tr(xx YY |Y X
(28) The terms XX and YY represent dot products between pairs of examples in X and Y. The left-hand side measures similarity between these structures, while the right-hand side captures feature similarity through the squared dot products.
B.2. Hilbert-Schmidt Independence Criterion (HSIC)
HSIC generalizes dot product-based similarity by measuring dependence between two sets of variables in reproducing kernel Hilbert spaces (RKHS). For centered matrices X and
Y. Equation (1) implies:
1 1)2 tr(XX T YYT) |/oov(XTTTT )
original image
mnssk
fentyr
fent
fectfags
(29)
Figure 11: The deeper the neural network is, the more semantic information its features can encompass.
HSIC extends this to kernel methods, where Kij kz) and Liy (V Bj) are kernel matrices. The empirical HSIC estimator is:
The adapter is network layer positioned deeper than each layer of the backbone network, placing itself at a relatively deeper level while the backhone is relatively shallower. In- tuitively, the adapter serves as a natural boundary line for domain features and always captures a level of semantics higher than what the backbone learns, making more spe- cific. We validate it for CKA measure in paper, here, we further illustrate this perspective by visualizing feature maps at different levels, and the results shown in Figure. 12. For each column, after attaching an adapter, these feature maps contain more specific features like the bird's profile compared with the first row, which means the adapter assim- ilates domain-specific information.
HSIC(K, L)
(n 1)2 KHLH),
(30)
with H being the centering matrix:
- 11T n
(31)
where I, is the identity matrix and 1 is vector of ones. HSIC measures dependence between two distributions and converges to the population value at a rate of 1/ /n.
B.3. Centered Kernel Alignment (CKA)
To address the scaling issues inherent in HSIC, CKA nor- malizes the dependence measure. CKA between two kermel matrices K and defined as:
image
feat
feat nast
feat
HSIC(K,L) L) CKA(K,L) /HSIC(K, K). HSI
(32)
The numerator measures the similarity between the two kernel matrices, while the denominator normalizes this sim- ilarity by accounting for the self-similarities within each representation. This normalization ensures that CKA is in- variant to isotropic scaling, making it robust variations in the magnitudes of the features.
musk
Adapter
Adupter
Adapterigh
Figure 12: The adapter serves as natural boundary line for domain features and always captures a level of semantics higher than what the backbone learns
We extract features from the source and target domains through the feature extractor, treat them as two sets of rep- resentations, and compute the domain similarity using the CKA formula above.
14

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
D. SAM: Flatten Loss Landscape
Dewaglate
For any p > and any distribution 2, with probability 1 over the choice of the training set S D,
L (w)) & max (sw + c)+
(33)
|e I=
5/02 0.04
3000 one
Co) 031 SO1A ORE 5908 302
Learn
1 | (1 V (x)(
2) )
4 log 8 +1(1)
(S/22005
k log
+
n-1
n2 1
3.32
(34)
$1.84
1139
where |S], h is the number of parameters and we assumed (w)) < ,e.~N( (ww + e)]. This condi- tion implies that adding Gaussian perturbations should not reduce the test error, which generally holds for the final solution but not necessarily for all w.
0.02 GEE 0.06 USE D1 03.2 0155 03.6 0L.E
Figure 14: The Learning rate for the fine-tuning of four target datasets.
Source
Target
Saame
Miuieen
Complesity
Tarast
ParamstKi
FLOPIG)
Loss
Laxc
Misiuom
baseline
26174
20.11
Leaned
Wegght
ours
31679
ratio
21
Weights
Weights
Table 12: (Left) The hyper-paramett parameter of the SAM. (Right) The complexity analysis.
Figure 13: Sharpness-Aware Minimization (SAM): flattens the loss landscape of the DFN, limits its absorption of exces- sive source-domain information (prevent overfitting to the source-domain samples), thereby facilitating the fine-tuning of DFN during target-domain stage.
F. The Analysis of Computational Complexity
As shown in Table 12 (right), we present the results of the complexity analysis, demonstrating that our method incurs minimal overhead in terms of parameter count and training time. We use single 4090 GPU for training and testing.
The overview of the main text, we present the process dia- gram of SAM and its effects. Here, we provide further de- tailed explanations, as shown in Figure. 13. During source domain training, the DFN absorbs domain-specifee informa- tion, leading to loss minima that are specific to the source domain, i.e., source minima. When domain shift occurs, the weights learned by the DFN become misaligned on the target domain. If the loss landscape of the DFN is too sharp, fine-tuning to the target minima becomes more diffi cult. Adopting SAM during training on the source domain flattens the loss landscape of the DFN, making it easier to fine-tune to the target minima when transferring the target domain.
The Computational Efficiency of SAM-SVN Our SAM- SVN used only during source domain training and not during fine-tuning or inference, so does not affect the computational efficiency during inference. Regarding ef- ficiency during training in the source domain, although it requires double backpropagation, it is applied only to the sin- gular value matrix of DFN, resulting in negligible additional computation. As shown in Table 13, we demonstrate the computational efficiency of SAM-SVN by measuring the efficiency of the baseline, PATNet (which adopts the same baseline ours), SAM applied to the entire model, SAM applied to DFN, and SAM applied to SVN (the singular value matrix of DFN).
E. The Analysis of Hyper-parameter
Fine-tuning of DFN is performed using the Adam optimizer, with learning rates set le-3 for FSS-1000, 5c-1 for Deep- globe, 5e-3 for ISIC, and Chest -ary Each task undergoes total of 50 iterations. As shown in 14g.14, we evaluated the fine-tuning performance on four target datasets at different learning rates. We also present experiments on the hyper- parameter p for Sharpness-Arr Aware Minimization (SAM) as shown in table 12 (left).
Method
Baseline
PATNet
SAM Whole
SAM-DEN
SAM-SV
FLOPs Increase Ratio
20.11
22.62
26.82
22.68
20.51
12.4%
33.35%
12.77%
1.9%
Table 13: Computational efficiency comparison of different SAM variants.
15

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
G. More Ablation Study Results
existing CD-FSS methods set the batch size I during testing. This is to avoid unfair comparisons that arise from including information from other samples in the batch. How- ever, IFA sets the batch size to 96 during testing. To ensure a fair comparison, we conduct comparisons with IFA using a batch size of 96.
In the main text, we presented the average mloU across four target datasets to demonstrate our design effectiveness. Table 14 provides detailed performance results for each individual target dataset. While the main ablation studies used ResNet-50 as the backbone, we supplement these find- ings with additional experiments using ViT-Base (following FPTrans (Zhang et al., 2022), Table 15).
FSS-1000
Deepglobe
ISIC
Chest X-ray
Method
1-shot
5-shot
1-shot
5-shox
1-shot
5-shox
1-shat
5-shot
IFA
80.1
82.4
50.6
58.8
66.3
69.8
74.0
74.6
PSS-1900
Decpplate
ISIC
X-ray
Ours
82.6
87.9
51.3
71.4
86.1
91.6
DRN
SAM
SVD
1-sbot
S-shor
I-shot
S-hhot
1-chot
5-sbox
I-shot
S-hhot
Table 16: Comparison with IFA under it specific testing setting. (Backbone: Res-50, Batch size: 96)
77.53
90,99
29.05
35.08
31,20
35.10
51,88
54.36
80.67
$4.87
41:29
44.37
34,82
80.97
85.25
42.28
45.52
35.21
90.12
84.14
90.08
80.73
85.80
45.66
47.98
36,30
51,13
85.21
90.34
1. Related Work
Table 14: (ResNet50): Ablation study on various designs, observing the MIoU for I-shot and 5-shot on four datasets.
Few-shot learning Few-shot learning focuses on developing robust representations for novel concepts with limited anno- tated samples (An et al., 2024a;b).. Existing approaches can be broadly categorized into three primary frameworks: met- ric learning methods (Snell et al., 2017; Vinyals et al., 2016), optimization-based methods (Finn et al., 2017), and graph- based methods (Garcia & Bruna, 2017). dross-domain few-shot learning (Guo et al., 2020; Zou et al., 2024b) receiving increasing interest recently, there exist dispari- ties not only in the data distribution but also in the label space between the meta-testing stage and the meta-training stage. A significant challenge in this field remains effec- tively bridging the domain gap between source and target domains when only a few labeled samples are available.
FSS-1300
Despglate
ISIC
Chest X-ray
DRN
SAM
1-thot
S-shor
I-shot
5-llor
1-thoo
5-tbor
I-shot
S-dlot
78,90
81.77
38.29
42.54
81.83
83.86
39.12
06.23
18.95
56.79
81.97
85.32
82.76
84.59
39.37
46.98
50,09
57.51
82.77
86.15
82.97
85.72
39.45
47.67
87.14
Table 15: (ViT-Base): Ablation study on various designs, observing the MIoU for 1-shot and 5-shot settings.
We futher investigate how our method impacts the encoder's output. Using the relative CKA similarity (the ratio of CKA between Pascal and target datasets to Pascal itself, i... CKA/CKApuscul) A higher value indicates a lower domain similarity, meaning the encoder produces more ag- nostic features. Figure 15 reveals that attaching DFN to the backbone increases relative CKA, and further improvement occurs after applying SAM-SVN. This suggests the model has acquired more agnostic representations, reducing the distance between domains.
J. More Dataset Details
Na 54bi-ith
ESS 1860
Deepglate
ISIC21S
Chest Xeey
Figure 16: Example of segmentation for four datasets.
We adopt the benchmark (see Figure. 16) proposed by PAT- Net (Lei et al., 2022) and follow the same data preprocessing procedures as the dataset it employs.
9000 1000
DHC
Clond Srmo
Duraarts
PASCAL-5' (Shaban et al., 2017) is an extended version of PASCAL VOC 2012 (Everingham et al., 2010), incorporat- ing supplementary annotation enhancement details from the SDS dataset (Hariharan et al., 2011). We employ PASCAL- 5' as our source domain for training. Subsequently, we assess the performance of the trained models across four additional datasets.
Figure 15: Our designs increase encoder's domain similarity, enhancing the model's learning of agnostic knowledge.
H. Comparison with More Methods under Their Settings
Because the baseline SSP (Fan et al., 2022) used by the IFA (Nic et al., 2024) aggregates all samples within a batch when computing the foreground and background prototypes,
FSS-1000 (Li et al., 2020) is a natural image dataset for few- shot segmentation, containing 1000 categories, and each
16

---

Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation
category has 10 samples. We follow the official split for semantic segmentation in our experiment and present results on the designated testing set, consisting of 240 classes and 2,400 images. We regard FSS-1000 as a target domain for testing.
Deepglobe (Demir et al., 2018) comprises satellite images with dense pixel-level annotations across 7 categories: ur- ban, agriculture, rangeland, forest, water, barren, and un- known. As ground-truth labels only provided in the training set, we rely on the official training dataset, consist- ing of 803 images, to present our results. We adopt it as our target domain for testing and we take the same processing approach as PATNet.
ISIC2018 (Codella et al., 2019; Tschandl et al., 2018) is designed for skin cancer screening, and comprises lesion images, with each image containing precisely one primary lesion. The dataset is processed and utilized in accordance with the standards set by PATNet. And we regard ISIC2018 as a target domain for testing.
Chest X-ray (Candemir et al., 2013; Jaeger et al., 2013) is an X-ray dataset for Tuberculosis, consisting of 566 images (4020 4892 resolution). These images are derived from 58 Tuberculosis cases and 80 normal cases. To address large image sizes, a common practice involves downsizing to 1024 x 1024 pixels.
17


Reviews:
[{"review_id": "qPqrpHIpSx", "reviewer": "Reviewer_ueDb", "summary": "This paper introduces a novel perspective on using adapters as structural domain decouplers for cross-domain few-shot semantic segmentation (CD-FSS). They introduce the Domain Feature Navigator (DFN), a structure-based decoupler inserted into deeper network layers with residual connections, and SAM-SVN, a sharpness-aware regularization method applied to the singular values of DFN weights to prevent overfitting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Deeper theoretical analysis for “natural decoupling”：\n\nThe behavior of adapters as decouplers can be analyzed through the Information Bottleneck (IB) theory. The IB objective is:\n\n$\\mathcal{L}_{IB} = I(X;Z) - \\beta I(Z;Y)$\n\nwhere $I(\\cdot;\\cdot)$ is mutual information, X is the input, Y is the final output label, $Z$ is the intermediate representation, and $\\beta$ is the balance parameter. Decomposing the input as $X = X_{inv} + X_{spec}$, the IB objective becomes:\n\n$\\mathcal{L}_{IB} = I(X\\_{inv} + X\\_{spec};Z) - \\beta I(Z;Y)$\n\nFor encoder-decoder network (ED) parameters $\\theta_f$ and adapter parameters $\\theta_g$ where $\\theta_f \\gg \\theta_g$, the adapter's information capacity is much smaller than the ED, causing it to selectively absorb source domain-specific information that better optimizes the current training objective, resulting in $I(X_{spec};Z) \\gg I(X_{inv};Z)$, which promotes gradient flow differentiation. In a network $f$ with residual adapter $g$, the forward propagation is: $F(x) = f(x) + g(f(x))$\n\nFor ED parameters $\\theta_f$, the gradient is: $\\frac{\\partial \\mathcal{L}}{\\partial \\theta_f} = \\frac{\\partial \\mathcal{L}}{\\partial F(x)} \\cdot \\frac{\\partial F(x)}{\\partial f(x)} \\cdot \\frac{\\partial f(x)}{\\partial \\theta_f}$, expanding the middle term: $\\frac{\\partial F(x)}{\\partial f(x)} = I + \\frac{\\partial g(f(x))}{\\partial f(x)}$\n\nwhere $I$ is the identity matrix (from the direct residual path), and the second term is the Jacobian matrix of the adapter function.\n\nFor adapter parameters $\\theta_g$, the gradient is proved to be $\\frac{\\partial \\mathcal{L}}{\\partial \\theta_g} = \\frac{\\partial \\mathcal{L}}{\\partial F(x)} \\cdot \\frac{\\partial g(f(x))}{\\partial \\theta_g}$\n\nDifferentiated gradients: Due to the adapter's selective absorption of domain-specific information and the residual adapter structure, gradient flow naturally separates network optimization into two complementary learning objectives.\n\n\n\n## 2. Further justification for perturbing singular values in SAM-SVN:\n\nFor any $\\rho>0$ and any distribution $\\mathscr{D}$, with probability $1-\\delta$ over the choice of the training set $\\mathcal{S}\\sim \\mathscr{D}$, SAM bounds the generalization error as\n$$\nL\\_\\mathscr{D}(\\boldsymbol{w}) \\leq \\max\\_{\\|\\boldsymbol{\\epsilon}\\|\\_2 \\leq \\rho} L\\_\\mathcal{S}(\\boldsymbol{w} + \\boldsymbol{\\epsilon}) + \\sqrt{\\frac{k\\log\\left(1+\\frac{\\|\\boldsymbol{w}\\|\\_2^2}{\\rho^2}\\left(1+\\sqrt{\\frac{\\log(n)}{k}}\\right)^2\\right)+ \\frac{4\\log\\frac{n}{\\delta} + \\tilde{O}(1)}{n-1}}{n-1}}\n$$\nwhere $n=|\\mathcal{S}|$, $k$ is the number of parameters, and we assume $L\\_\\mathscr{D}(\\boldsymbol{w}) \\leq \\mathbb{E}\\_{\\epsilon\\_i \\sim \\mathcal{N}(0,\\rho)}[L\\_\\mathscr{D}(\\boldsymbol{w}+\\boldsymbol{\\epsilon})]$. This condition implies that adding Gaussian perturbations should not reduce the test error, which generally holds for the final solution but not necessarily for all $\\boldsymbol{w}$. $w$ represent the parameters influenced by SAM. Applying SAM to the entire network or the entire DFN introduces excessive perturbations, which can hinder DFN’s ability to capture domain-specific information. SAM-SVN strikes a balance by decomposing $w_{DFN}$ and applying perturbations only to the singular value matrix of DFN. The singular value matrix governs the representation space of DFN, thus limiting perturbations to a reasonable spatial range. This prevents DFN from overfitting to the source domain while maintaining its ability to capture domain-specific information during training. \n\nWe also quantitatively compared the perturbation on the singular values and other weights in Tab.7.\n\n\n\n## 3. Comparisons with IFA use a batch size of 96:\n\nIFA is set to a batch size (bsz) of 96, so we adopt the same setting of bsz=96 for comparison to ensure fairness, as stated in appendix section D. Moreover, we also found using different bsz leads to different performance in IFA. For example, for Deepglobe, the performance is 50.1 when bsz=96, while it is 44.7 when bsz=1.\n\n\n\n## 4. Discussion on GPRN (AAAI'25)：\n\nSince the work had not been published at the time of our submission, we did not include a comparison. We are now offering a discussion on GPRN. GPRN exploits SAM’s generalizability by converting SAM-extracted masks into semantic prompts, aggregating prompt information through graph-based reasoning, and adaptively choosing feedback points. Our approach highlights that adapters naturally function as decouplers. We explore this concept further, proposing a decoupling strategy that is applicable to a variety of models.\n\n\n## 5. Impact statement in page 9:\n\nThank you very much for your kind reminder, but the official guidelines indicate that the impact statement is not subject to page limitations.  Here is the original wording: “Papers must be prepared and submitted as a single file: 8 pages for the main paper, **with unlimited pages for references, the impact statement, and appendices.**\"", "claims_and_evidence": "good", "methods_and_evaluation_criteria": "good", "theoretical_claims": "good", "experimental_designs_or_analyses": "good", "relation_to_broader_scientific_literature": "good", "essential_references_not_discussed": "A recent work on cd-fss is not discussed: SAM-Aware Graph Prompt Reasoning Network for Cross-Domain Few-Shot Segmentation, aaai 2025", "other_strengths_and_weaknesses": "Strengths:\n1. The discovery that adapters structurally decouple domain information (without explicit loss functions) is innovative and challenges existing loss-based domain adaptation paradigms.\n2. SAM-SVN effectively balances domain-specific knowledge absorption and overfitting prevention.\n3. Experiments demonstrate state-of-the-art performance on four benchmarks. Comprehensive ablation studies validate design choices (adapter position, residual connections, SAM-SVN). Experiments span multiple datasets (FSS-1000, DeepGlobe, ISIC, Chest X-ray) and backbones (ResNet-50, ViT), demonstrating robustness.\n\nWeaknesses\n1. The term \"natural decoupling\" lacks intuitive explanation. While experiments show reduced CKA similarity, a deeper theoretical analysis (e.g., information bottleneck principles) could strengthen claims. The rationale for perturbing singular values (vs. other parameters) in SAM-SVN needs further justification.\n2. Comparisons with IFA use different batch sizes (96 vs. 1). Why?\n3. Why is there an impact statement analysis on page 9? Does it exceed the page limit?", "other_comments_or_suggestions": "please see weakness", "questions_for_authors": "please see weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wcAMeaPpbE", "reviewer": "Reviewer_U2qX", "summary": "The paper proposes that adapters naturally serve as domain information decouplers in Cross-Domain Few-Shot Segmentation (CD-FSS) by separating domain-specific and domain-agnostic features. Based on this insight, the authors introduce Domain Feature Navigator (DFN), a structure-based decoupler that captures domain-specific knowledge without requiring explicit domain losses. To prevent DFN from overfitting to source samples, they propose SAM-SVN, which applies Sharpness-Aware Minimization (SAM) on Singular Value Norms (SVN) to constrain overfitting while preserving domain decoupling. The approach is evaluated on FSS-1000, DeepGlobe, ISIC, and Chest X-ray datasets, showing gains over SOTA methods. Ablation studies confirm that DFN improves domain decoupling, while SAM-SVN enhances generalization by reducing loss sharpness. Qualitative results demonstrate that DFN redirects model attention toward domain-invariant features, improving segmentation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Deeper theoretical analysis for “natural decoupling”：\n\nDue to space limitation, please refer to reviewer ueDb's reply 1 for theoretical analysis.\n\n\n## 2. Analysis of trade-off between decoupling and domain knowledge retention:\n\nFor any $\\rho>0$ and any distribution $\\mathscr{D}$, with probability $1-\\delta$ over the choice of the training set $\\mathcal{S}\\sim \\mathscr{D}$, SAM bounds the generalization error as\n$$\nL\\_\\mathscr{D}(\\boldsymbol{w}) \\leq \\max\\_{\\|\\boldsymbol{\\epsilon}\\|\\_2 \\leq \\rho} L\\_\\mathcal{S}(\\boldsymbol{w} + \\boldsymbol{\\epsilon}) + \\sqrt{\\frac{k\\log\\left(1+\\frac{\\|\\boldsymbol{w}\\|\\_2^2}{\\rho^2}\\left(1+\\sqrt{\\frac{\\log(n)}{k}}\\right)^2\\right)+ \\frac{4\\log\\frac{n}{\\delta} + \\tilde{O}(1)}{n-1}}{n-1}}\n$$\nwhere $n=|\\mathcal{S}|$, $k$ is the number of parameters, and we assume $L\\_\\mathscr{D}(\\boldsymbol{w}) \\leq \\mathbb{E}\\_{\\epsilon\\_i \\sim \\mathcal{N}(0,\\rho)}[L\\_\\mathscr{D}(\\boldsymbol{w}+\\boldsymbol{\\epsilon})]$. The trade-off is controlled by $\\rho$. For $\\rho$, a larger value indicates more perturbations are added, emphasizing domain knowledge retention (experiment is in Appendix Tab11 left)；For $w$​, a larger perturbation range signifies a greater focus on knowledge retention (experiment is in Tab7 right). All experiments regarding SAM-SVN are presented in Tab7 left, Fig10, Tab8, and Tab11 left.\n\n\n## 3. Clarity for Figure 1:\n\nIn fact, the feature does not contain domain-agnostic and domain-specific knowledge in equal parts; our diagram merely indicates that the feature includes both specific and agnostic information. Regarding DFN, which is a structure-based decoupler rather than a loss-based one like current approaches, it captures domain-specific information, thereby directing the model’s attention towards domain-agnostic knowledge (described in abstract and discussed in section 2). We will add a brief explanation of DFN in the caption to make it clearer for readers. Thank you very much for your suggestion.\n\n\n\n## 4. More explanations for why cka decrease means \"domain-specific\"\n\nCKA is a metric used to measure domain similarity by comparing the distances between kernel centers of two sets of data. If the kernel centers of the representations extracted by neural networks from the two data sets are closer (resulting in a higher CKA value), it indicates that their data distributions in the feature space are more consistent, meaning their feature representations are more similar and they share more patterns. Conversely, a lower CKA value implies lower domain similarity, with the kernel centers being farther apart in the feature space. Lower domain similarity suggests that the extracted features are less similar, indicating that the two sets of representations share fewer common patterns and have more characteristics specific to their own data distributions (more domain-specific).\n\nTo verify it is the domain gap that influences CKA, we divided the data in the Pascal into 20 groups based on categories and measured the CKA between different groups. We reported the CKA between the two most divergent groups, as well as the mean CKA and standard deviation (std) CKA. The results are as follows: It can be seen that even for the most divergent feature groups within the same dataset, the CKA is above 0.85, indicating the validity of using CKA to measure domain similarity.\n\n|  CKA   | max diff |  mean  |  std   |\n| :----: | :------: | :----: | :----: |\n| Pascal |  0.8656  | 0.8895 | 0.0179 |\n\n\n\n## 5. Reliability of the CKA metric\n\nAbsolute change is used here. We choose CKA as a measurement metric for several reasons: 1) CKA employs kernel center alignment, which eliminates measurement errors caused by extreme feature representations (noise); 2) Compared to other metrics like MMD and cosine similarity, CKA removes the effects of scaling differences; 3) CKA is more sensitive to differences in data distribution, allowing it to more accurately reflect distribution changes (detailed theory in the appendix).\n\nThe discrepancy in change between FSS and DeepGlobe is consistent due to: 1) CKA's sensitivity to data distribution differences; 2) The smaller domain gap between FSS and Pascal (both being natural image datasets) results in smaller changes, whereas the larger domain gap between Deepglobe (remote sensing images) and Pascal leads to greater gains and changes.\n\nAdditional metrics: In the experimental section (Figure 8), we also used MMD to validate our viewpoint, which is consistent with CKA. Furthermore, in the appendix, we employed relative CKA to mitigate the impact of data distribution (Figure 15).", "claims_and_evidence": "The paper provides strong empirical evidence supporting its claims through quantitative results, ablation studies, and qualitative visualizations. The effectiveness of DFN as a domain decoupler is demonstrated through domain similarity CKA analysis.\n\nHowever, some potential weaknesses exist:\n(1) The claim that DFN naturally serves as a domain decoupler is supported by empirical observations (CKA similarity changes) but lacks a theoretical foundation explaining why this occurs.\n\n(2) While SAM-SVN is shown to reduce sharpness and improve generalization, the trade-off between decoupling and domain knowledge retention is not deeply analyzed.", "methods_and_evaluation_criteria": "Yes, both the proposed methods and evaluation criteria are well-aligned in my opinion.", "theoretical_claims": "The paper does not present formal theoretical proofs but instead supports its claims through empirical observations and experimental validation.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental design and analyses, which are generally well-structured and sound.", "relation_to_broader_scientific_literature": "The paper builds on prior work in Cross-Domain Few-Shot Segmentation (CD-FSS), domain adaptation.", "essential_references_not_discussed": "No, just one related work that also found residual connect is important, aligns with the finding of this paper.\n\nWang, Pei, Yijun Li, and Nuno Vasconcelos. \"Rethinking and improving the robustness of image style transfer.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "Figure 1 is hard to understand. So half of feature maps is to learn domain-agnostic knowledge and half for domain-specific knowledge? Exactly half? Any references? What is DFN? Maybe need to clarify in the caption.\n\nLine 95, I can feel what the authors want to deliver, but hope to see more explanations why the decrease means the adapter captures domain-specific information\n\n\nTable 1, is the relative or absolute change can hint something like for FSS-1000, only 0.0024 increase but for Deepglobe, it is almost 0.02.\n\nThe analysis relies on CKA, is it trustworthy? Any other metric to use for reference?", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vFGoVL4FJ7", "reviewer": "Reviewer_iPWG", "summary": "This paper find an interesting phenomenon that a sort of adapters naturally serve as domain-information decoupler for the CDFSS task. By comprehensive experiments, the authors validate the condition that makes adapters to be decouplers. Then, they extend such a natural decoupler by sharpness-aware minimization to build the DFN for CDFSS, which shows state-of-the-art performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Our method can fit different structures：\nOur structure and the APM[1] structure are both based on HSNet, while our ViT structure is based on FPTrans[2]. Additionally, in the appendix, we used the SSP-based architecture for comparison with IFA. Thus, our method can be applied to networks with various architectures.\n\n## 2. The reason for the different performance between ResNet and ViT：\nThe difference is due to the datasets' bias towards local recognition cues or global ones.\n\nWe use KL divergence (DisFB) to measure the similarity between the foreground and background of datasets. A higher DisFB suggests a greater requirement for the model's global resolution capability, with ISIC having the highest DisFB. ResNet, due to its convolution-based design, has local prior characteristics, making it more effective for datasets like DeepGlobe that demand detailed local features. On the other hand, ViT, based on attention mechanism, excels in long-range dependencies and global resolution, offering better performance for datasets like ISIC that rely more on global perception.\n|       | FSS1000 | Deepglobe | ISIC  | ChestX |\n| ----- | :-----: | :-------: | :---: | :----: |\n| DisFB |  0.113  |   0.125   | 0.156 | 0.131  |\n\n## 3. Validating adapter choice impact on sharpness\n\nIn the main text, we measured loss fluctuations by adding Gaussian noise to observe the changes in the sharpness of loss landscapes before and after integrating DFN. Here, we further measure the impact of different adapter choices on sharpness. Consistent with Fig3 and Fig5 in the main text, we analyze based on position and structure (res: residual, ser: serial, BKB: backbone, enc-dec: encoder-decoder):\n\n|  Position  | Baseline | BKB shallower | BKB deeper | Between enc-dec |\n| :--------: | :------: | :-----------: | :--------: | :-------------: |\n| loss fluc. |  0.398   |     0.402     |   0.521    |      0.405      |\n\n| Structure  | Baseline | conventional+res | LoRA+res | conventional+ser |\n| :--------: | :------: | :--------------: | :------: | :--------------: |\n| loss fluc. |  0.398   |      0.521       |  0.533   |      0.399       |\n\nThe results above show that, from the perspective of sharpness, significant changes in loss fluctuations occur only when residual links are satisfied and the position is deep within the backbone (with the adapter structure not being a determining factor). This indicates that the adapter has captured domain-specific related information, which is consistent with the conclusions drawn in the main text.\n\n## 4. Our methods can benefit general few-shot segmentation tasks\n\nWe tested the performance of DFN on the source domain (Pascal). Pascal consists of 20 classes and is set to a 4-fold configuration in the FSS setup. This means training is conducted on 5 classes, while testing is performed on 15 classes that were not seen during the training phase. Due to its ability to enhance the model's adaptation to new domains/classes, it also provides benefits for general few-shot segmentation. After fine-tuning DFN, its positive impact becomes even more pronounced.\n\n| 1shot（Pascal） | Fold0 | Fold1 | Fold2 | Fold3 | Mean |\n| --------------- | :---: | :---: | :---: | :---: | :--: |\n| Baseline        | 64.3  | 70.7  | 60.3  | 60.5  | 64.0 |\n| DFN w/o ft      | 65.2  | 70.9  | 60.8  | 61.3  | 64.6 |\n| DFN w/ ft       | 66.8  | 72.4  | 62.5  | 62.7  | 66.1 |\n\n## 5. More settings to DFN：\nHere, we include more experiments on the DFN training and fine-tuning settings：1）DFN is involved in source domain training but removed in the target domain. 2）DFN is involved in source domain training, but its original parameters are discarded and it learns from scratch in the target domain (using different initialization methods).\n| 1-shot | FSS1000 | Deepglobe | ISIC  | ChestX | Mean |\n| :------: | :-----: | :-------: | :---: | :----: | ----- |\n|baseline|77.53|29.65|31.20|51.88|47.57|\n|DFN (remove in target)|78.16|38.21|34.12|76.92|56.85|\n|DFN(scratch, kaiming init)|79.02|42.53| 36.63 | 82.03  |60.05|\n|DFN(scratch, xavier init)|79.83|45.57| 34.79|79.46|59.91|\n|DFN(random gauss init)|78.97|38.75| 33.82 |78.59|57.53|\n|DFN|80.73|45.66| 36.30|85.21|61.98|\n\n\nFor the setting that DFN is removed in the target domain, DFN captures domain-specific information during source domain training and guides the model to learn domain-agnostic knowledge, resulting in a significant performance improvement compared to the baseline. However, due to a lack of target-specific knowledge, this approach is suboptimal. In the setting that DFN learns from scratch, its performance is influenced by the initialization method, yet it still performs well under various initializations. Our approach (i.e., DFN+SAM-SVN) is seen as using DFN to guide the model to focus on domain-agnostic knowledge in the source domain, while simultaneously providing DFN with a reasonable initialization value beneficial for adapting to various domains. Therefore, it achieves the best performance.", "claims_and_evidence": "The phenomenon that adapters naturally serve as decoupler is novel to me. Most claims are validated in the experiments. I like the writing of this paper to detailly study each component of the phenomenon and methods.\n\nHowever, this paper claims the decoupler phenomenon for CDFSS methods, but it majorly studies the HSNet structure. Nowadays many other structures are available to achieve state-of-the-art performance, such as [1] and [2]. Indeed, HSNet is prevailing, but I wish to see whether this phenomenon and method could fit other structures.\n\n[1] Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation\n[2] Feature-Proxy Transformer for Few-Shot Segmentation", "methods_and_evaluation_criteria": "The proposed method is a kind of adapter found to decouple source-domain information, and a training strategy to resist the overfitting, which are reasonable and novel to me. The performance in table 6 is good compared with current works.\nHowever, in table 6, I see resnet50 and vit-base achieves state-of-the-art performance differently on these four datasets. Similarly, in appendix, table 12 and 13 show that the proposed method contribute differently to resnet and vit. Could you give some insights on these results?", "theoretical_claims": "The theoretical side of this paper is in the sharpness-aware minimization. Experiments validate the influence that DFN brings to the sharpness.\nHowever, do other choice of adapters also influence the sharpness? I suggest adding these experiments to validate from the sharpness side that only the DFN can decouple domain information.", "experimental_designs_or_analyses": "The experiments and analysis are comprehensive and thorough. I appreciate the author to validate each aspect of this problem and design choice.\n\nHowever, I wonder how the proposed method influence the source-domain performance. Since the adapter decouples domain information which is important for the source domain, will it harm the source-domain performance? I know the FSS dataset is close to Pascal, but I wish to see the experiments on Pascal.", "relation_to_broader_scientific_literature": "This work can be applied to medical analysis (e.g., ISIC for skin diseases, chest x-ray for lung diseases), so it will help the scientific study by providing a easy-to-adapt model.", "essential_references_not_discussed": "I think the authors can supplement the related work with some latest work about adapters such as [1], although some of the most famous ones have been studied in section 2.\n\n[1] Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation", "other_strengths_and_weaknesses": "I see in section 2 the authors use CKA to validate the domain similarity, but in section 4 the MMD is used for validation. Although the CKA experiments are included in the appendix, I still suggest keeping the criteria consistent in the paper.", "other_comments_or_suggestions": "Typos: Section D title, whit -> with.", "questions_for_authors": "I wonder what will happen if a trained adapter is removed from network. That is, after the source-domain training, since adapters grab domain information that is harmful for target domains, how is the performance if we directly remove these adapters and only keep the remaining structures and weights for segmentation? Will it be higher than appending the source-domain-trained adapter but not finetuning it? Furthermore, during the target-domain finetuning, how is the performance if we directly use a scratch adapter for finetuning? These experiments can help understand the information captured by the source-domain-trained adapter.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8AbA2zZEqB", "reviewer": "Reviewer_eLxN", "summary": "The paper proposes utilizing an adapter for cross-domain few-shot semantic segmentation. The authors first demonstrate that the adapter naturally serves as a decoupler, and then design a DFN network to decouple source domain information into domain-agnostic and domain-specific components. They also propose SAM-SVN to mitigate potential overfitting issues of DFN on the source samples, achieving good performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. The computational efficiency of SAM-SVN：\n\nOur SAM-SVN is used only during source domain training and not during fine-tuning or inference, so it does not affect the computational efficiency during inference. Regarding efficiency during training in the source domain, although it requires double backpropagation, it is applied only to the singular value matrix of DFN, resulting in negligible additional computation. We demonstrate the computational efficiency of SAM-SVN by measuring the efficiency of the baseline, PATNet (which adopts the same baseline as ours), SAM applied to the entire model, SAM applied to DFN, and SAM applied to SVN (singular value matrix of DFN). The results are shown in the table below:\n\n|                | baseline | PATNet [1] | SAM-Whole | SAM-DFN |  SAM-SVN  |\n| -------------- | :------: | :--------: | :-------: | :-----: | :-------: |\n| FLOPs(G)       |  20.12   |   22.63    |   26.83   |  22.69  |   20.52   |\n| increase ratio |    /     |   12.4%    |  33.35%   | 12.77%  | **1.99%** |\n\n[1] Cross-Domain Few-Shot Semantic Segmentation\n\n## 2. References：\n\nWe promise to cite the listed references in the final version.", "claims_and_evidence": "The claims made in the paper is that the adapter naturally serves as a decoupler for domain-specific information. The authors provide detailed experiments and analysis to support this claim.", "methods_and_evaluation_criteria": "The proposed method is well explained, and the evaluation criteria and datasets used are appropriate for the task.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The main experiments are comprehensive and demonstrate the effectiveness of the proposed method. However, since SAM-SVN requires an additional forward-backward computation that may be computationally expensive, it would be useful to compare the model's computational efficiency with other baselines.", "relation_to_broader_scientific_literature": "The proposed designs complement the broader literature and introduce new designs.", "essential_references_not_discussed": "The following papers are related to the model’s hypercorrelation design and should be cited:\n\n- CVPR 2024, Rethinking Few-shot 3D Point Cloud Semantic Segmentation\n- ICLR 2025, Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation", "other_strengths_and_weaknesses": "The paper is clearly written and easy to follow. The motivations for the design choices are clear and reasonable, and the experiments demonstrate the superior performance achieved by the proposed method.", "other_comments_or_suggestions": "Please see the Questions.", "questions_for_authors": "Since SAM-SVN requires an additional forward-backward computation that may be computationally expensive, could you compare the model's parameter count and computational efficiency with other baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Hi,

Draft has received overall positive reviews, with 2 weak accept and 2 accept.  Dear authors, please update the draft per recommendations and comments of the reviewers.

Congratulations. 

regards

AC
Decision: Accept (spotlight poster)