Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation
Authors: Jintao Tong
Yixiong Zou
Guangyao Chen
Yuhua Li
Ruixuan Li
Venue: ICML 2025 poster (2025)
Keywords: Disentanglement
Composition
Few-shot
Transfer Learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We find a natural decomposition of ViT outputs, which we use as a novel perspective to interpret the feature entangement problem. Based on it, we further propose a method to self-disentangle ViT features and re-compose them for CD-FSS.

Abstract: Cross-Domain Few-Shot Segmentation (CD-FSS) aims to transfer knowledge from a large-scale source-domain dataset to unseen target-domain datasets with limited annotated samples. Current methods typically compare the distance between training and testing samples for mask prediction. However, a problem of feature entanglement exists in this well-adopted method, which binds multiple patterns together and harms the transferability. However, we find an entanglement problem exists in this widely adopted method, which tends to bind source-domain patterns together and make each of them hard to transfer. In this paper, we aim to address this problem for the CD-FSS task. We first find a natural decomposition of the ViT structure, based on which we delve into the entanglement problem for an interpretation. We find the decomposed ViT components are crossly compared between images in distance calculation, where the rational comparisons are entangled with those meaningless ones by their equal importance, leading to the entanglement problem. Based on this interpretation, we further propose to address the entanglement problem by learning to weigh for all comparisons of ViT components, which learn disentangled features and re-compose them for the CD-FSS task, benefiting both the generalization and finetuning. 		Experiments show that our model outperforms the state-of-the-art CD-FSS method by 1.92% and 1.88% in average accuracy under 1-shot and 5-shot settings, respectively.

Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation
Jintao Tong'
Yixiong Z
Guangyao Chen Yuhua Li
Ruixuan Li
Abstract
4
Few-Shot Segmentation (CD-FSS) aims to transfer knowledge from source-domain dataset unseen larget-domain datasets with lim- ited annotations. Current methods typically com- pare the distance between training and testing samples for mask prediction. However, we find an entanglement problem exists in this widely adopted method, which tends to bind source- domain patterns together and make each of them hard to transfer. In this paper, we aim to address this problem for the CD-FSS task. We first find natural decomposition of the ViT structure, based which we delve into the entanglement problem for an interpretation. We find the decomposed VIT components are crossly compared between images in distance calculation, where the rational comparisons are entangled with those meaning- less ones by their equal importance, leading the entanglement problem. Based this inter- pretation, we further propose to address the en- tanglement problem by learning to weigh for all comparisons of ViT components, which learn dis- entangled features and re-compose them for the CD-FSS task, benefiting both the generalization and finetuning. Experiments show that our model outperforms the state-of-the- toffthh-aartt CD-FSS method by 1.92% and 1.88% in average accuracy under 1-shot and 5-shot settings, respectively.
Dientangk
Compolte
Vbualication
prediction
predictoo
(n) Orkers
(h) Onrs
cstepeiic
Disantengled Fentares
(c) Disintongle & Compuoit
Figure 1: (a) A problem of feature entanglement exists in current works, which entangles multiple patterns and reduces the transferability. (b)(c To handle this problem, we find natural decomposition VIT's feature, then analyze the entanglement problem based on this decomposition, and finally propose to self- self-disentangle and re-compose the ViT feature to address this problem for efficient cross-domain transferring and target-domain adaptation.
on abundant labeled data poses a major challenge, espe- cially for dense prediction tasks like semantic segmenta- tion. Cross-Domain Few-shot Semantic Segmentation (CD- FSS) (Shaban et al., 2017; Dong Xing, 2018; Zhang et al., 2020b; Lei et al., 2022) has been introduced to address this issue, enabling predictions for target-domain unseen classes by limited annotated samples, with knowledge transferred from a sufficcinf source domain.
Existing CD-FSS works (Herzog, 2024; Su et al., 2024; He et al., 2024; Tong al., 2024) usual perform segmentation measuring the similarity between the support and query set based on features output by the encoder (Fig. la). How- ever, we find this well-adopted method always leads to the entanglement of multiple patterns' and harms the transfer- ability. For example, in Fig. la, the model tends to entangle the patterns of wings and bodies, i.e., detecting wings and bodies only when these two patterns appear simultaneously. However, if image contains only the wings but the body is different from the training data (e.g., another kind of
1. Introduction
Recent progress in deep neural networks (Long al., 2015; Zhao et al., 2017; Dosovitskiy et al., 2020) has been driven by large-scale annotated datasets. However, the reliance
'School of Computer Science and Technology, Huazhong Uni- versity of Science and Technology, Wuhan, China "Scoool of Com- puter Science, Peking University, Beijing, China. Correspondence to: Yixiong zou <yixiongz hust.edu.cn.>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Represent various attributes such object region, texture, color, and more, we visualize activated object regions.

---

Self- Disenaienttnnllmen and Re-Compasition for Cross-Domain Few-Shot Segmentation
the same feature space
Fusion Weight (AFW) dynamically learn the comparison
weights for efficient adaptation (Fig. 1b).
MSA*
NLP
MS.*
MLP
outpost
To sum up, our primary contributions are as follows:
Layer*
Leyer
To our knowledge, we are the first to analyze the fea- ture entanglement problem from the aspect of the natural decomposition of ViT structures for the CD-FSS task.
outpost
M MALL%MMAA' +MLP) +.... AS 4MLPP
Block
Block!
Rlock
The residual connection and consistent spatial size make the output of ViT components located in the same feature space, which inspires us view the final output of ViT the cumulative composition of all ViT components.
Figure 2:
We interpret the entanglement problem as a result of en- tangling rational comparisons between ViT components with those meaningless ones by their equal importance.
Based on this interpretation, we further propose to self- disentangle and re-compose ViT components for the CD- FSS task with the proposed orthogonal space decoupling module, cross-comparison module, and adaptive fusion weight module, which addresses the entanglement prob- lem by learning to weigh for each comparison, benefiting both the generalization and finetuning for CD-FSS.
bat), the model may fail to capture the wings, leading to segmentation errors. For the CD-FSS task, domain gaps and semantic gaps heavily exist between the source and target datasets. Therefore, transferring entangled patterns is much more difficult than transferring disentangled ones. Inspired by this issue, in this paper, we aim to address the entanglement problem for the CD-FSS task (Fig. lbc).
Extensive experiments show the effectiveness of our work on four different CD-FSS scenarios. Our model signifi- cantly outperforms state-of-the-art methods.
Recent work on ViT interpretabilityy (Gandelsman et al.) shows that the residual connections and the consistent spa- tial size make the output of each ViT component (e.g., MSA, MLP) located in the same feature space. Therefore, we find the final output of a ViT can naturally be seen as a cumula- tive composition of all ViT components (shown in Fig.2 and detailed in Section 2.2). Such structural decomposition of ViT's output inspires us to ask: can the entangled semantic patterns also be decomposed in this way?
2. Delve into Feature Entanglement
In this section, we delve into the causes of feature entangle- ment by decomposing the structure of the ViT output.
2.1. Problem Definition
Cross-domain few-shot semantic segmentation (CD-FSS) aims to transfer knowledge learned from the source domain to unseen target domains with only a few annotated support images. Consider a source domain D, = (X.))%) and a target domain D. = (x))))),, where X' denotes the input distribution and y denotes the label space. The input data distributions of D. and D, are distinct, and their label spaces do not overlap, i.e., X, * Xr V. n no; 0. The model is trained solely on D, and without access the target data, and then applied to segment novel classes in D,.
Based on this inspiration, we first delve into the entangle- ment problem for an interpretation. We find each ViT com- ponent captures distinct semantic patterns, c.g., bodics and wings, while the cumulative composition of these compo- nents implicitly binds all these patterns in ViT's output. By comparing distances between two images, as mainstream of CD-FSS methods, the model essentially combines all pos- sible comparisons between different components equally. Therefore, rational comparisons between patterns (wings vs. wings) are entangled with those meaningless comparisons (bodies vs. wings) by their equal importance, which we interpret cause the feature entanglement problem.
In this work, we adopt the meta-learning episodic manner following (Lei et al., 2022) to train and test our model. Specifically, both the training set from D, and the testing set from D, consist of several episodes. Each episode includes K support samples S = {[E M; (K image-mask pairs) and query Q = { My where I represents the image and M denotes the label. Within each episode, the model is expected to use the support sample (Is, M, M, and the query image I, to predict the query label.
Inspired by this interpretation, we further propose to han- dle the entanglement problem by learning to weigh for all comparisons between ViT components. Specifically, we first self-disentangle VIT's output by extracting features of different ViT components. Then, we introduce an Orthogo- nal Space Decoupling (OSD) module to further reduce the correlation of the disentangled features. Given these dis- entangled component features, we propose a Comparison (CPC) module, where the disentangled pat- terns are compared crossly for the re-composition, based on weights generated by OSD to emphasize the comparison between components with the same position. During the target-domain finetuning, we further introduce the Adaptive
2.2. Structural Decomposition of the VIT Output
ViT architecture. ViT (Dosovitskiy et al., 2020) is resid- ual network built from I. layers, each of which contains meati-head self-attention (MSA) followed by an MLP block. The input I is first split into N novvoverlg image patches. The patches are projected linearly into N d- dimensional vectors, and positional embeddings are added
2

---

Sell- Dell-Disentanglement and Re-Compasition for Poos-Doomain Few-Shot Segmentation
query
Faa. 1088
To Dergllske
Dalt 2013
Chess N.ng
Figure 4: Domain similarities between source- and target- domain features extracted from different layers. A brighter color means higher domain similarity, indicating less over- fitting to the source domain and less feature entanglement.
Figure 3: Visualization of the cross-match between layers, where the same column means the same layer ID and bold lines indicate rational matches.
Target Dataset
FSS-1000
Decpglobe
ISIC
ChestX
to them to create the image tokens 2i (1.......NN Due to the segmentation task, the CLS token is excluded (not included in the following formulas), Formally, the matrix z E dRd,,N, N, with the tokens =f. =g, ..., =& as columns, con- stitutes the initial state of the residual stream. It is updated for L iterations via these two residual steps:
Final Output
0.4288
0.3135
0.2527
0.2856
Layer-wise Avg.
0.6107
0.4988
0,5074
0.6612
Top-12 Avg. 0.8126
0.6441
0.6164
0.7823
Bottom- Avg. 0.1407
0.0130
0.0473
0.0163
Table 1:
Simply shifting cross-matched layers heavily af- fects domain similarities, inspiring us to handle the entan- glement problem by learning the cross-match of layers.
2, = MSA (Z + z Z
MLP'(22)
(1)
Decomposition of the VIT. The residual structure of ViT allows us to express its output as a sum of the direct con- tributions of individual layers of the model. By unrolling Eq. 1 across layers, the image representation VIT(I) can be written as (Both here and in Eq. 1, we ignore a layer- normalization term to simplify derivations):
This implies the output of every layer is compared with all other layers. Since all layers are in the same feature space (Fig. 2), it is feasible to compare even the output of the first layer and the last layer, although the comparison results may be meaningless (Fig.3). However, in Eq. 6, we observe the matching process treats all layers equally, which means even the meaningless comparison between two distinct layers will have a non-trivee non-trivial impact on the final distance.
VIT(1) = -
MSA MSA((Z'-1) + Σ MLP'(2 (2")
(2)
1=1
We ignore here the indirect effects of the output of one layer on another downstream layer, and further simplify the MI.Ps and MSAs into L
As the feature entanglement can be viewed as kind of over- fitting to the source domain, which can be represented as the recognition based on meaningless patterns, such mean- ingless comparisons would lead the model to rely on pat- terns specific to such comparisons, leading to overfitting. However, Eq. 6 entangles these patterns and comparisons together with equally. Therefore, we hypothesize it is the entanglement in the cross-match of different layers that leads to the entanglement in the semantic features.
VIT(/) 2 Layer'
2.3. Analyzing Entanglement by VIT Decomposition
Since most CD-FSS methods are based on distances be- tween support and query set images, we begin our analysis by revisiting the distance comparison. For support-query pair {Ls, Ig). their features extracted by VIT are:
Validation of hy hypothesis. To validate this hypothesis, we use domain similarities between source and target domains to measure the feature entanglement, i.c.. feature entangle- ment leads to overfitting to the source domain, and more overfitting leads to less transferable features across domains, reducing the domain similarity. We follow (Zou al., 2024a) to take the CKA similarity to measure the domain similarity. Specifically,
Z, = VIT(I.), Zy = ViT(I,)
(4)
The similarity score S computed using cosine similarity:
S= Z. Zg/11 ll
(5)
Substituting Eq 3 and Eq 4 into the similarity formula:
S= 2 Σ Lava()
2 Σ Layet)) 2,
(6)
different VIT layers to ex- tract features from each domains, and then compare features from the source domain and target domains to measure the CKA similarity. Since ViT contains 12 layers, this would lead to 12 x 12 CKA values for each source-targe target domain pair. As shown in Fig. 4, the cross-match deviating from the diagonal shows much lower domain similarities, e.g., the
From Eq. 6, can see a cross-match of different layers in the distance calculation:
L. L (Layer) Layer.) (7)
š Layer") Σ (Layer)) = >>
==
Please see Fig. for distinguishing Layer and Block.
Please refer to the appendix A for detailed formulations.
3

---

Self- Belf-Disentanglement and Re-Compasition for Cross-Domain Few-Shot Segmentation
top-right corner which means matching Layer 0 of the target domain and Layer 11 of the source domain. In Table 1, we also calculate the CKA of the comparison between final out- puts (i.e., viewed as the average of Fig. 4) and the layer-wise comparison (i.e., comparing outputs with the same layer ID, the diagonal of Fig. 4). We can see the layer-wise domain similarity is much higher than that of the final output. This verifies that the correct match between layers can lead to higher domain similarity, and therefore less feature entangle- ment. In other words, the decomposed components (Layers) are well-suited for generalization themselves, It is the cross- match of components that leads to feature entanglement.
temn learning. During target-domain finetuning, the Adaptive Fusion Weight (AFW) is introduced to dynamically learn the comparison weights for efficient adaptation.
3.1. Orthogonal Space Decoupling
Since misaligned and correct matches are assigned equal weights during comparison, we need to rectify misaligned matches. This involves two steps: (1) adjusting the seman- tics of each feature and assigning different weights before comparison, and (2) allocating different weights to each sim- ilarity after comparison. Therefore, we propose the OSD module as an explicit global decoupling and weight alloca- tion mechanism. helps semantic disentanglement by ag- gregating feature channels, enforeing orthogonal constraints, and assigning appropriate weights to different patterns.
Moreover, in Fig. 4, we can also observe a small fraction of cross-matches show higher CKA values than the diago- nal (layer-wise) ones. This indicates the patterns captured by each layer are not strictly different from others (Fig. 3), possibly due to the dynamic calculation of the self-attention mechanism (Park& & Kim, 2022). To verify it, in Table 1, we simply shift the match between layers, and the domain similarities are even higher than the layer-wise ones, indi- cating a learnable cross- match may be better than the naive layer-wise match for solving the entanglement problem.
Specifically, given support and query set, a sequence of L pairs of support and query feature maps FFF M is extracted from various ViT layers. Each representation Fi E IRdxNN is reshaped to Fi E RRdxnxn, where n is the patch number and the is channel dimension. These support and query patterns are first concatenated along the channel dimension form complete representation:
F". concat( (F) HF))
(8)
2.4. Discussion and conclusion
where F.... E &R Ldxxxx and * denotes that both the support and query patterns undergo the same operation.
To handle the feature entanglement problem for CD-FSS, we revisit ViT's inherent structure, which provides a natural decomposition of its features. Since all internal features of ViT layers (components) are in the same feature space due to the residual connection, ViT's final output implicitly combines all component features with the same importance. This also leads to the cross-match with equal importance be- tween all ViT components. By taking the domain similarity as a measure of the source-domald overfitting caused by the entanglement problem, we find is the meaningless cross- mathh between VIT components that majorly causes the entanglement problem, where the rational matches are en- tangled with those meaningless ones by their equal weights. Inspired by these, we aim to handle this problem by learning the cross-match weights of components.
Then, these concatenated features are fed into the OSD, where explicit constraints on each pattern channel enable semantic decoupling, while handling weight allocation. The OSD consists of a fully connected layer W.. E Rldxr, a convolutional layer W Worth& E R and a fully con- nected layer Wout E Ld Ld Here, r is low a rank (default set to 8) to save computational resources. The concatenated features are reduced a low-dimensional orthogonal space, applying orthogonal constraints and allocating weights:
Fdown W., W..((F...) (Fron); Forth = Worth ( (Fāown)
where Firth E R Next, we compute the orthogo- nal regularization (Xie et al., 2017) by reshaping Forth to Rrxn" and using it as a loss term to constrain the extracted pattern, promoting their disentanglement:
3. Method
Building on our analysis of feature entanglement, we propose the concept of denfddisentanglmment and re- composition. Our framework is illustrated in Fig. 5. We first extract support and query features from different VIT layers, concatenate them along the channel dimension, and feed them into the Orthogonal Space Decoupling (OSD) module for weight allocation and semantic disentanglement. Subse- quently, the outputs of OSD are input into the Cross-Pattern Comparison (CPC) module, where the disentangled patterns are compared crossly for the re-composition of patterns. For the re-copposiiion, during source-domain training, score maps are composed with weights from OSD for efficient pat-
Torth Forth
F+ orth T
(10)
Finally, we map F Föreh back to the original space and split the concatenated support and query features:
Fop = W onne (FFrenn:
(T)) splil(Fipp)
(11)
During source-domain training, OSD is trained jointly with the encoder. During target-domain fine-tuning, W and Wout are frozen, and we fine-tune the compact Worth-
3.2. Cross- Comparison for Re-Composition
Cross Comparison. Based on feature entanglement caused by misaligned matches and the dynamic nature of
4

---

Self- Sesf-iisettanllmment and Re-Composition for Cross-Domain Few-Shot Segmentation
Framework
Orthogonal Space Decoupling
Cross.Patiern Comparison (CPC)
(OSD)
Paler
Poar
Adaplive
Ld
Fasion
Print
Prier
Projector
Cros-Patterm
Comparioon
1x1 * Conv
Lorth
19 prote
distance
disenturgled features
Disealaugled Feature
Score Map
Score
Map
OSD
Seare
Seare
Map
Map
+ concaterate
Projector
Adaptive Fusion Weight (AFW)
support
Average Weight
Paum
Ld
domain
7200
Fuom
Ldl
Scare Hop
stumatn
target
X
Alaptine
Festo
query
Vision Encoder
learnable training target finetuning
f learmable training
K; only learnable target finetuning
Figure 5: Overview of our method. We extract L support and query features from various blocks. These features are concatenated along the channel dimension and fed into OSD to impose orthogonal constraints for weight allocation and semantic decoupling. Outputs OSD are then fed into the CPC module, where support and query features are compared crossly, yielding score maps. source-domain training, these maps are composed with average weights for efficient pattern learning. In target-domain fine-tuning, the AFW dynamically learns the composition weights for efficient adaption.
VIT, propose the Cross-Pattens Comparison (CPC) mod- ule. After semantic decoupling and weight allocation by the OSD, we use mask average pooling (MAP) (Zhang et al., 2020b) to obtain L sets of foreground prototypes Pfg E IR 4xxxx1xx 1x1 and background prototypes Pog E xxxxxxxx from support features. These disentangled support proto- types and query features are then input into the CPC module, where they cross-compared for re-composition.
for efficient adaption. The reason for not using AFW during training is that it is small parameter matrix of size I,2 x 2 (just 288 for ViT-B), where the "2" corresponds to the com- position weight for Cbg and Cyg- trained jointly with the encoder in the source domain, it is prone to overfitting the source data. As lightweight module, directly introducing it in the target domain allows for flexible adjustment and adaptation based the target domain, resulting in better performance. The formula is as follows:
Specifically, for the query feature sets F E IR Lxdxnxn and the support prototypes [Pog p/ll, we compute the distance through cross paising to obtain the cross-pattern comparison maps (score maps), denote as C:
D1. C(I)
source:
C Custin
(14)
WAPW @ C
target:
Cjusion
(15)
L2
Chalfo distance(F" F", Poglfal; = concat(Crp C1a)
(12)
where C(I) E 2xxxx & indicates the element-wise multi-
where C' is reshaped to R.. 1I2xx2xx00 2xxxx (2 means background and foreground), and the distance can be calculated vari-
plication. The final prediction pred describe:
ous ways; we default to using cosine similarity, while also exploring other distance metrics (see Experiment 4.3):
pred
(Crnnaar((GCFusiooonn )))
(16)
where ((+)) is function that bilinearly interpolates Cfusion to the spatial size of the input image by expanding along the spatial dimension, i.e., R 22xXx + 22xhx Here, h and are the image's 's height and width.
distance.os F9. Pog/ [PP]
(13)
Adaptive Fusion Weight. For ro-costosite the obtained T,2 sets of cross-pattern comparison maps, during source- domain training, the comparison maps C' are composed with average weights for efficient pattern learning; during target- domain mee-tuning, Adaptive Fusion Weight (AFW), which includes background and foreground weight, is introduced
Loss Strategy. During both source-domain training and target-domain fine-tuning tne-tunnng, we employ the standard Binary Cross-Entrops (BCE) loss LECE, with the orthogonal loss Torth from OSD added as a regularization term to promote
5

---

Self- Belf-Disentanglement and Re-Compasition for Cross-Domain Few-Shot Segmentation
FSS-1000
Decpglobe
ISIC
Chest X-ray
Average
Method
Mark
Backhone
I-shot
5-shot
I-shot
5-shot
I-shot
5-shot
1-shot
5-shot
I-shot
5-shot
PANet (Wang al., 2019)
ECCV-20
Res-50
69.15
71.68
36.55
45.43
25.29
33.99
57.75
69.31
47.19
55.10
RPMMs (Yang et al, 2020a)
ECCV-20
Res-50
65.12
67.06
12.99
13.47
18.02
20.04
30.11
30.82
31.56
32.85
PFENet (Tian et al., 2020)
TPAM1-20
Res-50
70.87
70.52
16.88
18.01
23.50
23.83
27.22
27.57
34.62
34,98
RePRI (Boudiaf et al., 2021)
CVPR-21
Res-50
70.96
74.23
25.03
27.41
23.27
26.23
65.08
65.48
46.09
48.34
HSNet (Min et al., 2021)
ICCV-21
Res-50
77.53
80.99
29.65
35.08
31,20
35,10
51.88
54.36
47.57
$1.38
PATNet (Lci ct al., 2022)
ECCV-22
Res-50
78.59
81.23
37.89
42.97
41.16
53.58
66.61
70.20
56.06
61.99
PATNet (Lei et al., 2022)
ECCV-22
ViT-buse
72.03
22.37
44.25
76.43
53.77
PerSAM (Zhang et al., 2024)
ICLR-24
ViT-hase
60.92
66,53
36.08
40.65
23.27
25.33
29.95
30.05
37,56
40.64
APM (Tong et al., 2024)
NeurIPS-24
Res-50
79.29
81.83
40.86
44.92
41.71
51.16
78.25
82.81
60.03
65.18
ABCDFSS (Herzog. 2024)
CVPR-24
Res-50
74.60
76.20
42.60
45.70
45.70
53.30
79.80
81.40
60.67
64.97
DRA (Su al., 2024)
CVPR-24
Res-50
79.05
80,40
41.29
50.12
40.77
48.87
82.35
82.31
60,86
65,42
APSeg (Hc et al., 2024)
CVPR-24
ViT-basc
79.71
81.90
35.94
39.98
45.43
53.98
84.10
84.50
61.30
65.09
SDRC (Ours)
Ours
ViT-hase
80.31
82.55
43.15
46.83
46.57
55.02
82.86
84.79
63.22
67.30
Table 2:
Mean-IoU of 1-shot and 5-shot results on the CD-FSS benchmark. The best and second- results are highlighted in bold and underlined, respectively. The comparison with domain transfer methods is provided in Appendix C.
semantic decoupling, Notably, in target-domain finetuning, we do not access the query data. Instead, treat the support as the query for the calculation of LEC and Lorth We optimize the model using the final loss L:
PSS 990
I LBCCE = BCF(ppeed y)
(17)
L LBCE + XLorth
(18)
where y is the query ground truth in source-domain training and denotes support mask in target-domain fine-tuning, and is a hyperparameter, with a default value of 0.1, that adjusts the weight of the orthogonal loss Lorth.
INCC
4. Experiments
support
query
base Rine
oars
Figure 6: Qualitative results of our model for 1-shot setting. The support labels are highlighted in blue, while the predic- tions and ground truth of query images are presented in red.
4.1. Dataset and Implementation Details
The benchmark proposed by PATNet (Lei et al., 2022) is adopted, following the same data preprocessing procedures as the dataset it employs. PASCAL VOC 2012 (Everingham et al., 2010) with SBD (Hariharan et al., 201 augmentation serves as the training dataset. FSS-1000 (Li et al., 2020), DeepGlobe (Demir et al., 2018), ISIC2018 (Codella et al., 2019; Tschandl et al., 2018), and Chest X-ray (Candemir et al., 2013; Jaeger et al., 2013) are considered as target domains for evaluation. See the Appendix for details.
1-shot and 5-shot settings. Specifically, we surpass the per- formance of the state-of-the-art by 1.92% and 1.88% under I-shot and 5-shot settings, respectively. Notably, APSeg also utilizes ViT as its backbone; however, its parameter count is significantly larger than ours due to its SAM-based encoder-decoder architecture, while we employ an encoder- only structure. Consequently, our method not only surpasses APSeg in performance but also entails substantially lower computational costs. Furthermore, we showcase the qualita- tive results of our method in 1-way 1-shot segmentation, as depicted in Figure 6. These results demonstrate the substan- tial enhancement in generalization ability across significant domain gaps while maintaining a comparable accuracy in the face of similar domain shifts using our method.
Following previous prototype- based work (Dong& & Xing, 2018; Zhang et al., 2020b; Wang et al., 2019), we built lightweight encoder-only baseline that employs ViT- B (Dosovitskiy et al., 2020) pre-trained on ImageNet (Rus- sakovsky et al., 2015) as the backbone network. The hy- perparameter X, which adjusts the weight of the orthogonal loss, is set to 0. and the rank of the OSD module set to 8. For other details, please refer to the appendix.
4.3. Ablation Study
4.2. Comparison with State-of-the-Art Works
Impact of each design. As shown in Table 3, introduc- ing the CPC, which serves as the foundation for the other designs, improved the average mloU by 9.62% and 9.04%
In Table2, we compare our method with existing works, where we achieve a significant improvement under both
6

---

Self- Dell-Disentanglement and Re-Composition for Cross-Domail Few-Shot Segmentation
PSS-1000
Deepglobe
ISIC
Chest X-ray
Average
CPC
AFW
OSD
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
77.80
80.69
33.18
37.39
36.99
41.20
51.54
55.26
49.88
53.64
79.20
81.46
41.71
43.25
42.99
47.97
74.81
78.05
59.50
62.68
79.22
82.02
42.59
45.22
43.11
50.73
80.36
80.36
61.32
65.22
80,05
82.18
41.87
44.68
45.63
52.61
75.45
78.32
60.75
64,45
80.31
82.55
43.15
46.83
46.57
55.02
82.86
84.79
63.22
67.30
Table 3: Detailed ablation study results of our various designs on four target datasets under 1-shot setting and 5-shot setting.
Metric
haseline
ours
Encoder (ViT-B)
AFW
OSD (rank=8)
1-shot
5-shot
I-shot
5-shot
Win
Worth
Wost
Euclidaan
48, 92
53.07
62.49
66.53
Params(K)
8.6
x10*
0.288
6.144
0.064
6.144
Dot
49.18
53.03
62.75
66.58
FMD
50.02
53.23
63.37
67.01
rank
64
32
16
8
4
Cosine 49.88
53.64
63.22
67.30
mloU
62.61
63,43
63.25
63.22
61.73
60.39
Table 4: Impact of the different distance metric for CPC.
Table 6: Analysis the impact of parameters on performance.
for the I-shot and 5-shot settings, respectively. The OSD is an explicit global decoupling module, guiding layers to focus on the patterns emphasized by the current layer and promoting disentanglement. Meanwhile, the AFW adjusts the weights of comparison maps for each pattern based different target domains, resulting in more accurate seg- mentation predictions. These results demonstrate that each design in our approach significantly enhances performance.
costs while ensuring effective global decoupling for differ- ent domains. The AFW is not involved in source-domain training and is directly adopted during the target stage. We also explored the impact of the OSD's rank value on perfor- mance. With rank=8, the performance is slightly lower than 32, but the parameter count is only 4. so we set rank 8.
Re-composition strategy for comparison maps. During source domain training. we composite the comparison maps with average weights. During target domain fine-tuning, the AFW is introduced to adaptively learn the combination weights for efficient adaption. As shown in Table 7, training AFW jointly with the encoder during source-domain train- ing does not achieve better results compared adapting it directly on the target domain. Training AFW in the source domain can lead to weights being biased toward the source data, so we opt for fine-tuning directly on the target domain.
Impact of different distance metric. Our method is highly versatile, as we evaluated it using various distance metrics, including Euclidean distance (Snell al., 2017), cosine similarity (Vinyals et al., 2016), dot product (Chen et al., 2019), and EMD (Zhang et al., 2020a), as shown in Table 4. Regardless of the distance metric, our method consistently outperforms the baseline with significant gains. Notably, under 1-shot setting. EMD is the best distance metric, while cosine similarity performs best under 5-shot setting.
AFW
I-shot
S-shot
Effectiveness of cross-comparison. In Table 5, we report the performance of both the position-wise pattern compari- son and cross-patters comparison, confirming the effective- ness of the cross cross-layer strategy. Due to the dynamic nature of ViT and intra-class variations, features extracted from dif- ferent layers may still form correct matches. Therefore, the CPC effectively compares these patterns for re-composition.
w/ source-domain training
61.01
64,93
w/o source-domain training
63.22
67.30
Table 7: Validation of the training strategy for AFW.
4.4. Seff-Disentanglement and Re-Composition
Disentangle Feature by decomposing ViT structure: To validate our approach, we visualize the features of each layer of ViT-B, as shown in Fig. 7. Since ViT-B has 12 layers, the features grouped in pairs (with 6 layers per row). The results reveal that the extracted features from different layers capture distinct semantic information, focusing on elements such as the fish tail, body, fins and outline. The
w/o AFD
1-shot
5-shot
baseline
49.88
53.64
position-wise comparison
15
59.39
cross-pattern comparison 59.50
62.68
Table 5: Validate the effectiveness of cross-comparison.
PSS-1000
Deepglohe
ISIC
ChestX
OSD
Impact of parameters on performance. As shown in Ta- ble 6, the OSD and the AFW are lightweight modules with minimal parameters. The OSD is trained jointly with the en- coder on the source domain, but on the target dataset, only the W Worth& of OSD is fine-tuned, reducing computational
w/o
w/
w/o
support MI
0.6444
0.6059
0,8509
0.7958
0.8738
0. 0.7800
0.9095
0.6506
query MI
0.6437
0.6051
0,8593
0.7962
0.8712
0,7829
0.9110
0.6517
Table 8: The MI between features; lower values correspond to lower correlations (better disentanglement).
7

---

Self- Dell-Diseetanglement and Re-Compasition for Cross-Domale Do-min Few-Shot Segmentation
query image & mink
sim exampples of the comparton maae for re-ompuiiiinn comppesitin realt
Figure 8: The heatmaps of some examples of cross- comparison maps and the results after re-composition.
Figure 7: Visualization of features extracted from different layers of ViT demonstrates the feasibility of disentangling the entangled patterns by decomposing the VIT structure.
result demonstrates that ViT inherently has the potential for semantic disentanglement and supports the feasibility of our insight: decompose the entangled semantic patterns through structural decomposition of the VIT output.
Further semantic disentanglement by OSD: Mutual in- formation reflects the correlation between features. Lower mutual information indicates weaker correlations, meaning the features are more semantically independent. Therefore, we measure the average mutual information (MI) separately between the support features and between the query features to verify that OSD promotes further semantic disentangle- ment. As shown in Table 8, after applying OSD, the mutual information between both support features and between query features decreases, verifying OSD's s decoupling.
Derppliee
Figure 9: Visualization of AFW four target datasets, which includes background and foreground weight. A brighter color indicates a higher re-composition weight.
performance. The results indicate that the optimal choice of the weight falls within a wide interval, which means the tuning of this ally, we used the same orthogonal loss weight in the Swin Transformer architecture we did in the VIT architecture (Table 17). The performance indicates that our method design is not sensitive to this weight.
is not difficult. Addition-
Pross-nattern Comparison for re-composition: In Fig. 8, we visualize the results of cross-comparison (some samples) and re-composition. Through cross-comparison, the model focuses different regions of the segmented object. These comparison maps are then re-composed, allowing the model to accurately identify the complete segmentation region.
Orth. Loss Weight
0.01
0.05
0.1
0.2
0.5
Re-composition by Adaptive Fusion Weight: We visual- ize the AFW as heatmaps, where brighter colors indicate higher re-oomposttion weights. The result in Fig.9 high- lights that: 1) AFW learns different re-composition weights for each domain; 2) The highest weight learned by AFW is not necessarily at the diagonal of the matrix, suggesting that cross-matching and re-composition are more effective than position-we wise matching. Additionally, we observe an interesting phenomenon: without any constraints imposed on AFW, the learned re-composition weights for foreground and background tend to be mutually exclusive, trend par- ticularly noticeable in the Deepglobe and ISIC datasets.
1-shot Avg mloU
62.59
63.01
63.22
63.18
62.87
Table 9: Impact of orthogonal loss weight on performance.
Impact of Background Prototypes's Number Most cur- rent prototype- baseed methods (e.g., PANet, SSP) utilize single background prototype to model background pat- terns, and have demonstrated good performance in both recent works and our experiments. Indeed, is preferable to consider different background classes for different im- ages. Therefore, shown in Table 10, further introduce clustering to obtain multiple background prototypes (Yang et al., 2020b; Li et al., 2021a). By comparing the single- background prototype to multi-background prototypes, we observe a slight performance improvement. However, the gains are not substantial enough to justify the additional computational overhead of clustering.
5. Analysis of Performance and Efficiency
Orthogonal Loss Weight As shown in Table 9, we val- idated the impact of the weight of the orthogonal loss on

---

Self- Self-Disentanglement and Re-Compasition for Cross-Domain Few-Shot Segmentation
7. Related Work
FSS1000
Deepglobe
ISIC
ChestX
Mean
Single BG protetype
80.31
43.15
46.57
82.86
63,22
Cross-Domain Few-Shot Segmentation (CD-FSS) CD-
Multi BG prototype
80.96
43.53
46.78
83.09
63,59
FSS has received increasing attention recently. PAT- Net (Lei et al., 2022) establishes CD-FSS benchmark and proposes feature transformation layers to map domain- specific features into domain-agnostie ones for fast adap- tion. APSeg (He et al., 2024), based on SAM (Kirillov et al., 2023), introduces a novel auto auto-prompt network for guiding features in cross-domain segmentation. DRA (Su et al., 2024) adopts a compact adapter to align diverse target domain features with the source domain, while ABCDESS (Herzog, 2024) introduces tiny adaptors that learn to refine features at test-time only. APM (Tong et al., 2024) proposes lightweight frequency masker to achieve feature enhancement. These methods focus on optimizing encoders' final features to obtain a better representation. In contrast, our approach focuses on decomposing ViT's final features based on the natural decomposition of ViT's structure, and re-composing them for better comparison.
Table 10: Comparison between single and multiple back- ground prototypes under 1-shot setting.
Computational Efficiency As shown in Table 11, we compared our method with PATNet (Lei et al., 2022), HSNet (Min et al., 2021), and SSP (Fan et al., 2022). Our approach exhibits greater computational efficiency than the other methods, as it does not require additional networks and instead leverages the inherent structure of the ViT for feature separation.
PATNet
HSNet
SSP
Ours
FLOPs (G)
22.63
20.11
18.97
18.86
Table 11: Analysis of computational efficiency.
6. Theoretical Analysis of the Effectiveness of ViT Disentanglement
Feature Disentanglement Learning (FDL) FDL aims to learn an interpretable representation for image variants, which has long been a popular solution for addressing do- main shifts. InfoGAN (Chen et al., 2016) maximizes mutual information to learn disentangled representations in an unsu- pervised manner. Disentangled- VAE (Li et al., 2021b) exca- vate category-distillin information from visual and seman- tic features for generalized zero-shot learning. DFR (Cheng et al., 2023) separates discriminative features from class- irrelevant components. However, these methods require additional complex YAE-Discriminator networks, result- ing in significant computational overhead. In contrast, our approach does not introduce any extra branch networks. In- stead, it leverages feature space consistency across different layers of the ViT to extract distinct patterns, decomposing the entangled semantic patterns through a structural decom- position of the ViT output from a novel perspective.
In cross-domain few-shot segmentation tasks, models need to transfer knowledge from the source domain S with abun- dant annotations to the target domain T with limited data. Let H. represent the hypothesis space of the segmentation model. The upper bound of the generalization error for target domain risk 6T (h) is defined as:
(Th) S (h)
(S,
(19)
where h denotes features extracted by the encoder, (h) is the source domain risk, (x,S, T) represents the H-
divergence (domain gap) between the source and target domains, and is the irreducible ideal joint risk.
Our approach reduces ey (h) through two mechanisms:
1) Adaptive Fusion Weights (AFW): adaptively assigns higher weights to semantically appropriate matches, leading to better alignment (as confirmed by the experiments on the source domain in Answer 3), thereby optimizing source domain output and reducing (h)).
8. Conclusion
In this paper, we analyze and interpret the feature entangle- ment problem from novel aspect of the natural decomposi- tion of VIT. Based on it, we propose self-disentanglementt and re-composition for CD-FSS. Experiments show our effectiveness and achieve a new state-of-the-art in CD-FSS.
2) Domain-luvariant Component Isolation: minimizes (S, T) by isolating ject parts) via:
patterns (e.g., ob-
L
L
L
L
A Akknowledeements
dx(S, T) ~0
wiyd (00 H
(S,T)
S
(S,T), H
1=
This work is supported by the National Key Re- search and Development Program of China under grant 2024YFC3307900; the National Natural Science Founda- tion of China under grants 62206102, 62436003, 62376103 and 62302184; the National Natural Science Foundation of China under grants 62402015; the Postdocotoral Fel- lowship Program of CPSF under grants GZB20230024;
(20)
where (1) denotes the inter-layer domain discrepancy. By leveraging the self-disentangling property and the orthogo- nal constraints from the OSD module, inappropriate matches are learned to have small W3y, reducing the layer- mu- tual information Th,,h?), thereby tightening the boundary.

---

Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation
the China Postdoctoral Science Foundation under grant 2024M7501100; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Sci- ence and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation is completed the HPC Platform of Huazhong University of Science and Technology.
sification. IEEE transactions on neural networks and learning systems, 2023.
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E.,
Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: 2019.
Impact Statement
Demir, I,, Koperski, K., Lindenbaum, D., Pang, G., Huang, J., Basu, S., Hughes, F,, Tuia, D., and Raskar, R. Deep- globe 2018: challenge to parse the earth through satel- lite images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pP. 172-181, 2018
Given current interpretations of VIT structures, we find a natural decomposition exists in features extracted by ViT, which inspires us to propose the concept of self- disentanglement and re-composition of VIT features. Our research can also be applied in other fields such as transfer learning and domain adaptation. What's more, our method also provides interpretable perspective and theoretical foundation for the feature disentanglement from the per- spective of ViT's self-decomposition. Future research will aim to broaden our evaluations to encompass wider range of target domains, enhancing our understanding of their performance in various real-world scenarios.
Deng, Z., Li, J., Guo, Z., Liu, W., Zou, L., and Li, G. Multi- view multi-aspect neural networks for next-basket rec- ommendation. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pp. 1283-1292. 2023.
Dong, N. and Xing, E. P. Few-shot semantic segmentation with prototype learning. BMVC, volume 3, 2018.
References
Boudiaf, M., Kervadec, H., Masud, Z. I., Piantanida, P., Ben Ayed, I., and Dolz, J. Few-shot segmentation with- out meta-learning: A good transductive inference is all you need? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13979- 13988, 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Candemir, S., Jaeger, S., Palaniappan, K., Musco, J. P., Singh, R. K., Xue, 7., Karargyris, A., Antani, S., Thoma, G., and McDonald, C. J. Lung segmentation in chest radiographs using anatomical atlases with nonrigid reg- istration. IEEE transactions on medical imaging, 33(2): 577-590, 2013.
Fan, Q. Pei, W., Tai, Y.-W., and Tang, C.-K. Self-support few-shot semantic segmentation. In European Conference on Computer Vision, pp. 701-719. Springer, 2022.
Cao, Y., Liu, Y., Chen, Z., Shi, G., Wang, W., Zhao, D., and Lu, T. Mmfuser: Multimodal multi-layer feature fuser for fine-grained dision-languag understanding. arXiv preprint 4rXiv:2410.11829 2024.
Gandelsman, Y., Efros, A. A., and Steinhardt, Interpreting clip's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations.
Chen, W.-.. Liu, Y.-C., Kira, Z.. Wang. Y.-. F., and Huang, J.B.. A closer look at few-shot classification. In Interna- lional Conference on Learning Representations, 2019.
Hariharan, B., Arbeláez, P., Bourdev, L., Maji. S., and Malik, J. Semantic contours from inverse detectors. In 201 international conference computer vision, pp. 991- 998. IEEE, 2011.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
He, W., Zhang, Y., Zhuo W., Shen, L.,, Yang, J., Deng, S., and Sun, L. Apseg: Auto-prompt network for cross- domain few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23762-23772, 2024.
Cheng, H., Wang, Li, H., Kot, A. C., and Wen, B. Dis- entangled feature representation for few-shot image clas-
10

---

Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation
Herzog, J. Adapt before comparison: A new perspective cross-domain few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Liu, W. Wang, J., Wang, H., Li, R., Deng, Z., Zhang, Y., and Qiu, Y. D-separation for causal self-explanation. Advances in Neural Information Processing Systems, 36: 16620-43633 2023b.
2024.
Jaeger, S., Karargyris, A., Candemir, S., Folio, L., Siegel- man, J., Callaghan, F., Xue, Z., Palaniappan, K., Singh, R. K., Antani, S., et al. Automatic tuberculosis screening using chest radiographs. IEEE transactions on medical imaging, 33(2):233-245, 2013.
Liu, W. Wang, J., Wang, H., R., Qiu, Y., Zhang, Y., Han, J., and Zou, Y. Decoupled rationalization with asymmetric learning rates: A flexible lipschitz restraint. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1535-1547, 2023c.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C.,, Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015 001164426,,- 2023.
Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H., Zhang, Y., and Li, R. Is the mmi criterion necessary for interpretabil- ity? degenerating non-causal features to plain noise for self-rationalization. Advances in Neural Information Pro- cessing Systems, 37:117636-11765 2024a.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similar- ity of neural network representations revisited. In Interna- tional Conference on Machine Learning, pp. 3519-3529. PMLR. 2019.
Liu, W., Wang, H., Wang, J., Deng. 7., Zhang, Y., Wang, C., and Li, R. Enhancing the rationale-inpu alignment for self-explaining rationalization. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pp. 2218-22211 IEEE, 2024b.
Lei, S., Zhang, X., He, J., Chen, F., Du, B., and Lu, C.- T. Cross-domain few-shot semantic segmentation. In European Conference on Computer Vision, pp. 73-90. Springer, 2022.
Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H., and Li, R. Exploring practical gaps in using cross entropy to implement maximum mutual information criterion for rationalization. Transactions of the Association for Com- putational Linguistics, 13:577-594, 2025a.
Li, G., Jampani, V., Sevilla-Lara, L., Sun, D., Kim, J., and Kim, J. Adaptive prototype learning and allocation for Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 8334 8334-8343, 2021a.
few-shot segmentation.
Liu, Y., Zou, Y., Li, Y., and Li, R. The devil is in low-level features for cross dromi-d few-shot segmentation. arXiv preprint arXiv:2503.2 50, 2025b.
Li, X., Wei, T., Chen, P., Tai, Y.-.., and Tang, C.-K. Fss- 1000: A 1000-class dataset for few-shot segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2869-281 2078-2878808 2020.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Li, X., Xu, Z., Wei, K., and Deng, C. Generalized zero-shot learning via disentangled representation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 1966-1974, 2021b.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. Advances in neural information processing systems, 33:1 1525-11551158 2020.
Liu, W. Wang, H., Wang, J., Li, R., Yue, C., and Zhang, Y. Fr. Folded rationalization with unified encoder. Advances in Neural Information Processing Systems, 35: 6966-6966, 2022.
Long, J., Shelhamer, E., and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Liu, W., Wang. H., Wang, J., X., Zhang, Y., and Qiu, Y. Mgr: Multi-generator based rationalization. In Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume I: Long Papers), pp. 12771-12787 2023a.
Min, J., Kang, D.,, and Cho, M. Hypercorrelation squeeze for sho- segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6941 6952, 2021.

---

Self-Disentanglement and Re-Compasition for Cross-Domain Few-Shot Segmentation
Nie, J.. Xing, Y., Zhang, G., Yan, P., Xiao, A., Tan, Y.-P., Kot, A. C., and Lu, S. Cross-domain few-shot segmenta- tion via iterative support-query correspondence mining. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 3380-3390, 2024.
Xie, D., Xiong, J., and Pu, S. All you need is beyond good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6176- 6185, 2017.
Park, N. and Kim, S. How do vision transformers work? In 10th International Conference on Learning Representa- tions, ICLR 2022, 2022.
Yang. B., Liu, C., Li, Jiao, J., and Ye, Q. Prototype mixture models for few-shot semantic segmentation. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK. August 23-28, 2020, Proceedings. Part VIII 16, pp. 763-778. Springer, 2020a.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211-252, 2015.
Yang, B., Liu, C., Li, B., Jiao, J., and Ye, Q. Prototype mixture models for few-shot semantic segmentation. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16. pp. 763-778. Springerr 2020b.
Seitzer, M., Horn, M., Zadaianchuk, A., Zietlow, D., Xiao, C.-J., He, T., Zhang, Z., Schölkopf, T., B., Brox, T., et al. Bridging the gap to real-world object- centric learning. arXiv preprint arXiv:2209. 14860, 2022.
Zhang, C., Cai, Y., Lin, G., and Shen, C. Deepemd: Few- shot image classification with differentiable earth mover's ' distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12203-12213, 2020a.
Shaban, A., Bansal, S., Liu, Z., Essa, I., and Boots, B. One- shot learning for semantic segmentation. arXiv preprint wiXiv:170.034411 2017.
Zhang, R., Jiang, Z., Guo, Z., Yan, S., Pan, J., Dong, H., Qiao, Y., Gao, P., and Li, H. Personalize segment any- thing model with one shot. In The Twelfih International Conference on Learning Representations, 2024.
Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Su, J., Fan, Q. Pei, W., Lu, G., and Chen, F. Domain- rectifying adapter for cross-domain few-shot segmenta- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24036- 24045, 2024.
Zhang, X., Wei, Y., Yang. Y., and Huang, T. S. Sg-one: Sim- ilarity guidance network for one-shot semantic segmenta- tion. IEEE transactions on cybernetics, 5085))3855538866 2020b.
Zhao, H., Shi, Qi, X., Wang, X., and Jia, Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 2881- 2890, 2017.
Tian, Z., Zhao, H., Shu, M., Yang, Z., Li, R., and Jia, J. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 44(2):1050-1065, 2020.
Zhu, J., Wang, H., and Shi, M. Multi-modal large language model enhanced pscudo 3d perception framework for visual commonsense reasoning. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Tong, J., Zou, Y., Li, Y., and Li, R. Lightweight frequency masker for pross-domain few-shot semantic segmentation. Advances in Neural Information Processing Systems, 37: 96728-96749, 2024.
Zou, Y., Zhang, S., Li, Y., and Li, R. Margin-based few-shot learning with class-level overfitting mit- igation. Advances in neural information processing sys- tems, 35:27267-27279 2022.
Tschandl, P., Rosendahl, C., and Kittler, H. The ham 1000 dataset, a large collection of multi-souree dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1 2.18.
Zou, Y., Liu, Y., Hu, Y., Li, Y., and Li, R. Flatten long-range loss landscapes for cross-domain few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
2024a.
Wang, K., Liew, J. H., Zou, Y., Zhou, D., and Feng, J. Panet: Few-shot image semantic segmentation with prototype alignment. In proceedings of the IEEE/CVF international conference on computer vision, pp. 9197-9206, 2019.
Zou, Y., Zhang, S., Zhou, H., Li, Y., and Li, R. Compo- sitional few-shot dlass-nncrmmentll learning. In Inter- national Conference on Machine Learning, pp. 62964- 62977. PMLR, 2024b.
12

---

Appendix for Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation
A. Centered Kernel Alignment (CKA)
Centered Kernel Alignment (CKA) (Kornblith et al., 2019) a widely used metric for measuring the similarity between two data representations (Zou et al., 2022; 2024b; Liu al., 2025b). It normalizes the Hilbert-Schmidt Independence Criterion (HSIC) mitigate scale differences and ensure more stable similarity measure.
FSS-1000
Decpglobe
HSIC quantifies dependence between two sets of features in a reproducing kernel Hilbert space (RKHS). For centered feature matrices X and Y, Equation (1) gives:
ISIC2018
Chest X-ray
1 (n 11-11: 1)2
XX YY |cov(X Y 12F
(21)
Figure 10:
Examples of images and their corresponding ground truth masks from four target domain datasets.
HSIC extends this to sernel-based methods, where Kij k(x,,2) 22) and Ly = [yt, yÿ) define kernel matrices. The empirical HSIC estimator is:
PASCAL-5' (Shaban et al., 2017) is an extended version of PASCAL VOC 2012 (Everingham al., 2010), incorpo- rating additional annotation details from the SDS dataset (Hariharan et al., 2011). 1). We use PASCAL as the source- domain dataset for model training and then evaluate the performance on four target-domain datasets.
HSIC(K, L)
(n - (KLHH),
(22)
where H is the centering matrix:
H 11 71
(23)
FSS-1000 (Li et al., 2020) is a natural image dataset, en- compassing 1,000 distinct categories, each represented by 10 samples. In this study, we adhere the official dataset split for semantic segmentation and report our results the specified test set, which comprises 240 classes and a total of 2, 2,400 images. FSS-1000 is employed as the target domain for performance evaluation.
Here. I, is the identity matrix, and is a vector of ones. HSIC effectively quantifies statistical dependence and con- verges to its population estimate at rate of 1/ /n.
To address HSIC's sensitivity to scale, CKA introduces nor- malization. The CKA metric between two kernel matrices K and L is:
Deepglobe (Demir et al., 2018) is dataset comprising satellite imagery with dense pixel-level annotations across categories: urban, agriculture, rangeland, forest, ter, barren, and unknown. Since ground-truth labels are only available for the training set, we utilize the official training dataset, which includes 803 images, for evaluation. We adopt Deepglobe as the target domain for evaluation and follow the same processing methodology as PATNet.
HSIC(K, L)
CKA(K,L) HSIIC(K, K) D.SSCICLLL
(24)
The numerator measures the similarity between the two kernels, while the denominator normalizes using self- similarities within each representation. This normalization ensures CKA's invariance to isotropic scaling, making it robust similarity measure for feature representations.
ISIC2018 (Codella et al., 2019; Tschandl et al., 2018) is skin cancer screening dataset, consisting of lesion images where each image contains single primary lesion. The dataset is processed and utilized following the standards established by PATNet. We consider ISIC2018 18 as the target domain for evaluation.
B. More Details about Datasets
We adopt the benchmark established by PATNet (Lei et al., 2022). Fig. 10 illustrates segmentation examples for four target datasets, Further details are as follows:
Chest X-ray (Candemir al., 2013; Jaeger et al., 2013)
13

---

Self-Disentanglement and Re-Compasition for Cross-Domain Few-Shot Segmentation
is dataset for Tuberculosis screening, consisting of 566 high-resolution images (4020 4892 pixels). These images are drawn from 58 Tuberculosis cases and 80 normal cases. To handle the large image dimensions, we resize them to 1024 x 1024 pixels.
robustness and diserimination by aggregating information from multiple layers. In contrast, our approach decouples the encoder's output into independent semantic representa- tions, which improves transferability in gross-domain set- tings. As shown in Table 13 our method outperforms ex- isting fusion-based approaches on the CD-FSS task (for multimodal methods, we apply this approach solely to im- prove the vision encoder).
C. Comparison with Domain Transfer Methods
We compare our method with traditional disentanglement representation methods and multi-layer fusion approaches to validate its effectiveness. For a fair comparison, all methods are implemented on the same baseline and evaluated under the 1-shot setting on the CD-FSS benchmark.
FSS
Deepglobe
ISIC
Chest
Average
baseline
77.80
33.18
36.99
51.54
49,88
FPN
78.73
37.51
37.64
69.59
55.87
MEP3P
79.96
42.92
40.43
73.87
59.30
MMFuser
80.29
38.65
42.01
75.33
59.07
Disentanglement representation methods Disentangle- ment representation methods has long been a popular solu- tion for addressing domain shifts. InfoGAN (Chen et al., 2016) maximizes mutual information to learn disentangled representations in an unsupervised manner. Disentangled- VAE (Li et al., 2021b) excavate category-distilingg infor- mation from visual and semantic features for generalized zero-shot learning. DFR (Cheng al., 2023) separates discriminative features from class-irrelevant components. However, these methods require additional complex VAE- Discriminator networks, resulting in significant computa- tional overhead. Moreover, under the few-shot setting, these methods struggle to leverage limited data to learn a suitable latent space for disentanglement. In contrast, our approach does not introduce any extra branch networks. Effectively addresses few-shot scenarios while maintaining low com- putational overhead. As shown in Table 12 our method outperforms existing disentanglement-baan based approaches on the CD-FSS task.
SDRC (Ours)
80.31
43.15
46.57
82.86
63.22
Table 13: Compare our method to previous fusion-based methods under 1-shot setting.
Slot attention-based methods Our approach differs from slot attention-based methods (Locatello et al., 2020; Seitzer et al., 2022) in two fundamental aspects: 1) Slot attention pri- marily disentangles distinct objects through object-centric representation optimization, exhibiting coarser granularity, whereas our method focuses on disentangling different pat- terns within the same object at a finer granularity level. 2) While slot attention employs additional iterative atten- tion modules (external networks) for disentanglement, we leverage the inherent property of ViT layers that naturally attend to distinet spatial regions, augmented with orthogo- nality constraints to reinforce semantic separation, without requiring additional networks. To validate the effective- ness of our method, we conduct comparative evaluations against two representative slot attention-based disentangle- ment approaches, Slot-Attention (Locatello al., 2020) and DINOSAUR (Seitzer et al., 2022), as shown in Table 14.
FSS
Deepglobe
ISIC
Chest
Average
baseline
77.80
33.18
36.99
51.54
49,88
InfoGAN
78.73
35.62
38.19
65.45
$4,50
FSS1000
DeepGlobe
ISIC
ChestX
Mean
Disentangled-VV VAE
78.67
36.02
37.79
66.38
54.72
Baseline
77.80
33.18
36.99
51.54
49.88
DFR
79.18
39.21
40.62
72.85
57.97
Slot-Attention
79.05
37.83
41.22
69.59
56.92
SDRC (Ours)
80.31
43.15
46.57
82.86
63.22
DINOSAUR
79.62
38.57
40.89
72.33
57.85
SDRC (Ours)
80.31
43.15
46.57
82.86
63.22
Table 12: Compare our method to previous disentanglement- based methods under 1-shot setting.
Table 14: Comparison with slot attention-based methods under 1-shot setting.
Feature fusion methods FPN (Lin et al., 2017) proposes an in-network feature pyramid architecture that enhances semantic richness across all scales by combining high- and low-resolution features. MEP3P (Zhu et al., 2024) enhanced the original visual features input into MLLMs with image depth features and pseudo-3D positions. MMFuser (Cao et al., 2024) integrated features from multiple layers, enrich- ing the visual inputs for MLLMs by capturing multi-level representations from the vision encoder. These methods enhance the encoder's output representation by utilizing multi-layer information, They aims to increase the feature's
D. Comparison with Methods under Special Setting
Existing CD-FSS methods adopt a batch size of I during testing to prevent unfair advantages from incorporating in- formation from other samples. However, IFA (Nie et al., 2024) uses batch size of 96 during testing, which causes it to calculate foreground and background prototypes by aggregating all samples in a batch. For a fair comparison, we conduct comparisons with IFA using a batch size of 96.
14

---

Self- Disentangllment and Re-Compasition for Cross-Domain Few-Shot Segmentation
FSS-1000
Deepglohe
ISIC
Chost X-ray
1 shot
Foldo
Fold1
Fold2
Fold3
Mean
Mothod
I-shot
5-shot
I-shet
5-shot
I-shor
5-shot
I-shor
S-shot
82.4
50.6
58.8
66.3
69.8
Baseline
61.5
68.2
66.7
52.5
62.2
Ours
83.1
85.7
51.1
59.4
69.7
72.5
84.1
87.2
Ours
63.1
70.3
67.8
55.4
64.2
Table 15: Comparison with IFA under its specific testing setting, which uses batch size of 96.
Table 18: Performance on Pascal under FSS setting.
benefits for domain generalization.
E. Effectiveness in Swin Transformers
Baseline
Ours
Stage
Stage 2
Stage 3
Stage 4
Shape
H/4 WV4
H/X W/
H/32x W/32X/
FSS
72.6
75.3
Layer Num
2
18
2
Table 19; Performance under domain generalization setting,
Table 16: Configuration of Swin-B BB transformer architecture.
As shown in Table 17, we further validated the effectiveness of our method on Swin Transformer (Liu et al., 2021). Since the layers in Swin Transformer do not reside in the same feature space, two additional steps are required: (1) For features from stage 2 to stage 4, we upsample them spatially to H/4 x W/4; (2) For features from stage to stage 3, we add three mapping linear layers (C, 8C), (2C, 8C), and (4C, SC) map them to the same feature space that of stage 4. These mapping layers are trained alongside the model during the source domain training phase. The performance results demonstrate that our method is well- suited to Swin Transformer, and that Swin Transformer shows a significant improvement performance compared to VIT.
G. Connection to More fields
As deep learning advances and finds broader applica- tion across AI, ensuring model terpretability (Liu et al., 2025a; 2024b), causal reasoning (Liu et al., 2023b; 2024a; 2023a), and robustness (Liu et al., 2023c; 2022: Deng al., (023)-has become increasingly important. We recognize this and view it as promising direction for future work.
in-
H. More Visualization Results about Seff-Disentanglemeet and Re-Composition
Based the feature space consistency, we find natural decomposition in ViT's output, which inspires us to propose the concept of self-disentangleemnnt and re-composition of
FSS1000
Deepglobe
ISIC
ChestX
Mean
VIT-B
77.80
33.18
36.99
51.54
49.88
ViT features for the CD-FSS task, which disentangles fea- tures without the need for additional branch networks. In the main text, we presented some visualization examples of self-disentanglement and re-composition. Here, we provide additional examples to further validate our insights.
Swin-B
79.85
37.24
39.90
66.73
55.93
Swin-B Ours
81.02
46.63
49.19
83.85
65.17
Table 17: Performance under 1-shot setting with Swin Trans- former architecture.
Disentangle Feature by decomposing ViT structure: To validate our approach, we visualize the features of each layer of ViT-B, as shown in Figure 11. The outputs of all 12 layers of ViT-B show that features extracted differ- ent layers exhibit distinct semantic tendencies, focusing on different regions of the segmented object (e.g. the outline, head, wings, body, and tail of a bird). The result demon- strates that ViT inherently has the potential for semantic disentanglement and supports the feasibility of our insight: decompose the entangled semantic patterns through a struc- tural decomposition of the ViT output.
F. Applications in Other Settings 1) Benefit for few-shot segmentation task
We measure the FSS performance of our methods on Pascal, which consists of 20 classes and is set to a 4-fold configu- ration in the FSS setup. This means training is conducted on 5 classes, while testing is performed on 15 classes that were not seen during the training phase. The experimental results show that our method can also effectively improve the performance of general FSS tasks.
2) Benefit for domain generalization task
Cross-Pattern Comparison for re-composition: Due to the dynamic nature of ViT, the patterns captured by the different layer may be semantically similar (as demon- strated through experiments in the main text). Therefore, we adopt the
Under the domain generalization setting, our method trained on Pascal and tested on FSS1000 (with removed support sets and finetuning stage) demonstrates that feature disentangle- ment enhances model generalizability, yielding concomitant
Comparison (CPC) module,
15

---

Dell-Diseettngllmenn and Re-Compasition for Cross-Domain Few-Shot Segmentation
Image
Mask
Layer
12
Figure 11: Visualization of features extracted from different layers of ViT demonstrates the feasibility of disentangling the entangled patterns by decomposing the ViT structure.
query image mask
some examples of the comparison maps for re-composition
re-composition
Figure 12: Visualization of the heatmaps showing some examples of cross comparison maps and the results after re-
composition.
where the disentangled patterns are cross-compared to fa- cilitate effective re-composition. Here, we present addi- tional cross- comparsomaarisnn (some samples) visualizations and re-composition result visualizations. as shown in Figure 12. Through comparon,,,,,,,, the model focuses different
regions of the segmented object. These comparison maps are then re-composed, allowing the model accurately identify the complete segmentation region.
16


Reviews:
[{"review_id": "5mOgAcLqXY", "reviewer": "Reviewer_PmpS", "summary": "The paper addresses the feature entanglement problem in Cross-Domain Few-Shot Segmentation (CD-FSS) by leveraging the structural decomposition of Vision Transformers (ViTs). The authors identify that cross-layer comparisons in ViTs entangle meaningful and irrelevant patterns, leading to reduced transferability. To resolve this, they propose three modules: (1) Orthogonal Space Decoupling (OSD) to disentangle features via orthogonal constraints, (2) Cross-Pattern Comparison (CPC) to enable adaptive cross-layer feature matching, and (3) Adaptive Fusion Weight (AFW) for target-domain fine-tuning. Experiments on four CD-FSS datasets demonstrate that the proposed approach outperforms state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. The theoretical analysis of the effectiveness of ViT disentanglement\n\nIn cross-domain few-shot segmentation tasks, models need to transfer knowledge from the **source domain $\\mathcal{S}$** with abundant annotations to the **target domain $\\mathcal{T}$** with limited data. Let $\\mathcal{H}$ represent the hypothesis space of the segmentation model. The upper bound of the generalization error for target domain risk $\\epsilon_{\\mathcal{T}}(h)$ is defined as:\n$$\n\\epsilon_{\\mathcal{T}}(h) \\leq \\epsilon_{\\mathcal{S}}(h) + d_{\\mathcal{H}}(\\mathcal{S}, \\mathcal{T}) + \\lambda,\n$$\nwhere $h$ denotes features extracted by the encoder, $\\epsilon_{\\mathcal{S}}(h)$ is the source domain risk, $d_{\\mathcal{H}}(\\mathcal{S}, \\mathcal{T})$ represents the $\\mathcal{H}$-divergence (domain gap) between the source and target domains, and $\\lambda$ is the irreducible ideal joint risk. \n\nOur approach reduces $\\epsilon_{\\mathcal{T}}(h)$ through two mechanisms:\n\n1）**Adaptive Fusion Weights (AFW):** adaptively assigns higher weights to semantically appropriate matches, leading to better alignment (as confirmed by the experiments on the source domain in Answer 3), thereby optimizing source domain output and reducing $\\epsilon_{\\mathcal{S}}(h)$.\n\n2）**Domain-Invariant Component Isolation:** Minimizing $d_{\\mathcal{H}}(\\mathcal{S}, \\mathcal{T})$ by isolating domain-invariant patterns (e.g., object parts) via:\n$$\nd_{\\mathcal{H}}(\\mathcal{S}, \\mathcal{T}) \\approx \\sum_{i=1}^L \\sum_{j=1}^L w_{ij}d_{\\mathcal{H}}^{(ij)}(\\mathcal{S}, \\mathcal{T}) \\leq \\sum_{i=1}^L \\sum_{j=1}^L d_{\\mathcal{H}}^{(ij)}(\\mathcal{S}, \\mathcal{T})\n$$\nwhere $d_{\\mathcal{H}}^{(ij)}$ denotes the inter-layer domain discrepancy. By leveraging the self-disentangle property and the orthogonal constraints from the OSD module, inappropriate matches are learned to have small $w_{ij}$, reducing the inter-layer mutual information $\\mathbb{I}[\\mathbf{h}_i \\mathbf{h}_j^T]$, thereby tightening the boundary.\n\n## 2. Effectness in Swin Transformers\n\n|  Swin-B   |    Stage1     |     Stage2     |      Stage3      |      Stage4      |\n| :-------: | :-----------: | :------------: | :--------------: | :--------------: |\n|   shape   | H/4 * W/4 * C | H/8 * W/8 * 2C | H/16 * W/16 * 4C | H/32 * W/32 * 8C |\n| layer num |       2       |       2        |        18        |        2         |\n\nThank you for your suggestion. We further validated the effectiveness of the method on Swin Transformer. Since the layers in Swin Transformer do not reside in the same feature space, two additional steps are required: 1) For features from stage 2 to stage 4, we upsample them spatially to H/4*W/4; 2) For features from stage 1 to stage 3, we add three mapping linear layers (c,8c), (2c, 8c), and (4c, 8c) to map them to the same feature space as that of stage 4. The mapping layers are trained alongside the model during the source domain training phase. The performance results are as follows: our method is well-suited to Swin Transformer, and Swin Transformer shows a significant improvement in performance compared to ViT. Thank you again for your suggestion!\n\n|    1-shot     |  FSS1000  | Deepglobe |   ISIC    |  ChestX   |   Mean    |\n| :-----------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|     ViT-B     |   77.80   |   33.18   |   36.99   |   51.54   |   49.88   |\n|    Swin-B     |   79.85   |   37.24   |   39.90   |   66.73   |   55.93   |\n| Swin-B + Ours | **81.02** | **46.63** | **49.19** | **83.85** | **65.17** |\n\n## 3. Our methods can benefit general few-shot segmentation tasks and domain generalization tasks\nWe measure the FSS performance of our methods on Pascal. Pascal consists of 20 classes and is set to a 4-fold configuration in the FSS setup. This means training is conducted on 5 classes, while testing is performed on 15 classes that were not seen during the training phase. The experimental results show that our method can also effectively improve the performance of general FSS tasks.\n|1shot|Fold0|Fold1|Fold2|Fold3|Mean|\n| ------- | :------: | :------: | :------: | :------: | :------: |\n| Baseline|61.5|68.2|66.7|52.5|62.2|\n| Ours |**63.1**|**70.3**|**67.8**|**55.4**|**64.2**|\n\nUnder the domain generalization setting, our method, trained on Pascal and tested on FSS1000, demonstrates an improvement in domain generalization.\n|    | baseline |ours|\n| --- | :-----: | --- |\n|FSS|72.6|75.3|\n\n## 4. Discussion on computational efficiency\nWe compared our method with PATNet, HSNet, and SSP. Our method demonstrated higher computational efficiency compared to the other methods. This is because we do not require additional networks but instead leverage ViT’s inherent structure for feature separation.\n\n|              |PATNet|HSNet|SSP|Ours|\n| ------------ | :----: | :---: | :---: | :-------: |\n| FLOPs (G) |22.63|20.12|18.97|**18.86**|\n| Training Time (h)|6.32|5.61|5.12|**5.07**|\n\n## 5. Method section\nThank you for your suggestion; we will work on improving it in the final version.", "claims_and_evidence": "The claims are well-supported by empirical evidence:\n1. Feature entanglement: Validated via Centered Kernel Alignment (CKA) analysis, showing lower domain similarity for cross-layer vs. layer-wise matches (Fig. 4, Table 1).\n2. Module effectiveness: Ablation studies confirm the contributions of OSD, CPC, and AFW (Tables 3–7).\n3. Superior performance: Results on diverse datasets (Table 2) and visualizations (Figs. 6–9) substantiate improvements.", "methods_and_evaluation_criteria": "Methods: The ViT decomposition and proposed modules (OSD, CPC, AFW) are conceptually sound, leveraging ViT’s residual structure for disentanglement. The use of orthogonal constraints and adaptive weighting aligns with the goal of reducing feature entanglement.\n\nEvaluation: Standard CD-FSS benchmarks and metrics (mIoU) are appropriate. The inclusion of both natural and medical imaging datasets strengthens validity.", "theoretical_claims": "The paper does not present formal theoretical proofs but provides a mathematical interpretation of ViT decomposition (Eqs. 1–7) and entanglement analysis. The hypothesis about cross-layer comparisons is empirically validated through CKA and ablation studies. The derivations are logically consistent and align with the experimental results.", "experimental_designs_or_analyses": "The experiments are well-designed and cover multiple aspects:\n1. Comparison with SOTA on four CD-FSS datasets, demonstrating superior generalization ability.\n2. Ablation studies to verify the contribution of each module (OSD, CPC, AFW).\n3. Comparison of different distance metrics (Euclidean, Cosine, EMD, Dot Product).\n4. Feature visualization to support the claim that ViT decomposition enables disentanglement.\n\nOverall, the experimental methodology is robust and comprehensive.", "relation_to_broader_scientific_literature": "The paper is well-connected to related work in CD-FSS and feature disentanglement.\n1. Compared to existing CD-FSS approaches (e.g., PATNet, APSeg, DRA, ABCDFSS), this work introduces a novel ViT decomposition-based perspective.\n2. Compared to disentanglement methods (e.g., InfoGAN, Disentangled-VAE, DFR), the proposed method does not require additional networks but instead leverages ViT’s inherent structure for feature separation.\n\nHowever, additional discussion on ViT feature disentanglement in other contexts (e.g., domain generalization) could further strengthen the paper.", "essential_references_not_discussed": "The paper includes most relevant references, but it would be beneficial to cite works that analyze ViT feature disentanglement and its impact on domain generalization.", "other_strengths_and_weaknesses": "Strengths:\n1. Novel perspective: The paper introduces a unique way of handling feature entanglement in ViTs for CD-FSS.\n2. Methodological soundness: The proposed modules (OSD, CPC, AFW) are well-justified and experimentally validated.\n3. Strong experimental results: The method consistently outperforms SOTA across multiple datasets and settings.\n4. Computational efficiency: Unlike decoder-based architectures like APSeg, the proposed method only requires an encoder, making it more efficient.\n\nWeaknesses:\n1. Theoretical justification of ViT decomposition: While the empirical results support the method, a more formal theoretical analysis of why ViT decomposition is optimal for disentanglement could be provided.\n2. The author's method, limited to ViT, may be outdated. Its applicability to more recent architectures like Swin Transformer is questionable. This is something I care about deeply. If the author could answer this question, I would be very willing to significantly improve my score.\n3. Limited discussion on broader applicability: The method is focused on CD-FSS, but could it benefit general few-shot segmentation tasks?\n4. Limited discussion on computational efficiency (e.g., inference speed) are not deeply analyzed.", "other_comments_or_suggestions": "The method section is dense; a flowchart or pseudocode could improve readability.", "questions_for_authors": "No questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BBR7wLU6qa", "reviewer": "Reviewer_oMi8", "summary": "This paper addresses feature entanglement in Cross-Domain Few-Shot Segmentation (CD-FSS), discovering that ViT features assign equal weights to both meaningful and meaningless pattern matches when comparing images. To solve this, the authors propose a self-disentanglement and re-composition framework with three modules: OSD to reduce feature correlation, CPC for pattern re-composition, and AFW for target-domain adaptation. Their approach outperforms state-of-the-art methods by 1.92% and 1.88% in 1-shot and 5-shot settings respectively.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Compare with more disentangle-based methods and Discussion on slot attention\n\nOur approach differs from slot attention-based methods [1, 2] in two fundamental aspects: 1) Slot attention primarily disentangles distinct objects through object-centric representation optimization, exhibiting coarser granularity, whereas our method focuses on disentangling different patterns within the same object at a finer granularity level. 2) While slot attention employs additional iterative attention modules (external networks) for disentanglement, we leverage the inherent property of ViT layers that naturally attend to distinct spatial regions, augmented with orthogonality constraints to reinforce semantic separation, without requiring additional networks. To validate our method's effectiveness, we conduct comparative evaluations with two representative slot attention-based disentanglement approaches [1, 2]：\n\n|  1-shot  |  FSS1000  | Deepglobe |   ISIC    |  ChestX   | Mean      |\n| :------: | :-------: | :-------: | :-------: | :-------: | --------- |\n| Baseline |   77.80   |   33.18   |   36.99   |   51.54   | 49.88     |\n|   [1]    |   79.05   |   37.83   |   41.22   |   69.59   | 56.92     |\n|   [2]    |   79.62   |   38.57   |   40.89   |   72.33   | 57.85     |\n|   Ours   | **80.31** | **43.15** | **46.57** | **82.86** | **63.22** |\n\nWe promise to add the discussion on Slot Attention in the final version for both the related work and experiments.\n\n[1] Locatello, Francesco, et al. \"Object-centric learning with slot attention.\" Advances in neural information processing systems 33 (2020): 11525-11538.\n\n[2] Seitzer, Maximilian, et al. \"Bridging the gap to real-world object-centric learning.\" arXiv preprint arXiv:2209.14860 (2022).\n\n## 2. Misunderstanding of our cross-comparison design\n\nWe would like to point out that we did not deny the effectiveness of cross-comparison of layers. Instead, **we surely acknowledge the importance of cross-comparisons, and our method is majorly built upon cross-comparisons**. Specifically, \n\n(1) We hold that different layers tend to focus on different patterns that are complementary to each other. Our analysis is based on this characteristic, utilizing it to achieve self-disentanglement, and the complementary of patterns is achieved by the recomposition. The comparison of feature-similarity distributions measured by CKA is intended to demonstrate this dynamic nature, **indicating that different layers may still have meaningful connections, making it possible to complement different layers by the cross-comparisons, instead of denying it**.\n\n(2) It is because of the meaningful connections across different layers that **we design AFW to dynamically assign weights for these cross-comparisons for recomposition**, instead of simply forcing the position-wise comparison that ignores the cross comparison. In our experiments (Fig.9 and Tab.5), we also validated the effectiveness of the cross comparison.\n\n(3) Also, as validated in Fig.9, our **OSD does not suppress non-diagonal comparisons**; its role is to encourage layers to focus on more complementary patterns instead of those repeated ones that have been focused on by other layers. With the AFW module, our model still encourages the cross-comparisons across layers, while the complementary of patterns is enhanced by the OSD module.\n\n## 3. Computational efficiency\n\nWe compared our method with PATNet, HSNet, and SSP. Our method demonstrated higher computational efficiency compared to the other methods. This is because we do not require additional networks but instead leverage ViT’s inherent structure for feature separation.\n\n|                   | PATNet | HSNet |  SSP  |   Ours    |\n| ----------------- | :----: | :---: | :---: | :-------: |\n| FLOPs (G)         | 22.63  | 20.12 | 18.97 | **18.86** |\n| Training Time (h) |  6.32  | 5.61  | 5.12  | **5.07**  |", "claims_and_evidence": "The claims in this submission are generally well-supported by evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A - This is an experimental paper with no theoretical claims or proofs that require verification.", "experimental_designs_or_analyses": "All experiments use common benchmarks in the CD-FSS field. The experimental design looks rational, no significant issues.", "relation_to_broader_scientific_literature": "**CD-FSS**: Extends the benchmark established by PATNet (Lei et al., 2022), addressing the feature entanglement problem in existing methods (e.g., PFENet, HSNet) to improve cross-domain transferability.\n\n**ViT Structural Decomposition**: Builds on ViT interpretability studies (Gandelsman et al., 2023), proposing the first application of ViT’s residual stream decomposition for feature disentanglement—unlike classical approaches (e.g., InfoGAN), it requires no auxiliary networks.", "essential_references_not_discussed": "In disentangled representation methods, there is a lack of the Slot attention-based method. [1, 2, 3]\n[1] Locatello, Francesco, et al. \"Object-centric learning with slot attention.\" Advances in neural information processing systems 33 (2020): 11525-11538.\n[2] Seitzer, Maximilian, et al. \"Bridging the gap to real-world object-centric learning.\" arXiv preprint arXiv:2209.14860 (2022).\n[3] Chen, Chaoqi, Luyao Tang, and Hui Huang. \"Reconstruct and Match: Out-of-Distribution Robustness via Topological Homogeneity.\" Advances in Neural Information Processing Systems 37 (2024): 125588-125607.", "other_strengths_and_weaknesses": "Pros: \n1. The paper tackles disentangled representations, which is an important research area. It's the first to use disentangled features for CD-FSS tasks.\n2. Well-organized and easy to follow, with clear figures.\n3. They propose a novel pipeline, explain their motivation clearly, and back it up with experiments that validate their approach.\n4. Experiments are thorough and show excellent performance.\n\nCons:\n1. The paper assumes that comparing different layers of ViT is mostly meaningless. But in reality, ViT layers work together by dynamic self-attention: shallow layers focus on details (like edges), while deeper layers handle the big picture (like the whole bird). In Fig. 4, The CKA metric used in the paper only checks if features distribution similarity, but it misses how these layers complement each other. Also, in L197-L198, the The authors also acknowledge that dynamic self-attention mechanisms may introduce certain meaningful cross-layer associations. \n2. In Tab. 1, the CKA value for Top-12 Avg. (shifted matches) (0.8126) is significantly higher than that of Layer-wise Avg. (diagonal matches, 0.6107), suggesting that some cross-layer comparisons might be more effective. However, the authors did not exploit this observation to optimize their method; instead, they directly suppressed non-diagonal comparisons via the OSD module.\n3. The author did not compare with existing disentangled representations methods [1, 2].\n\n[1] Locatello, Francesco, et al. \"Object-centric learning with slot attention.\" Advances in neural information processing systems 33 (2020): 11525-11538.\n\n[2] Seitzer, Maximilian, et al. \"Bridging the gap to real-world object-centric learning.\" arXiv preprint arXiv:2209.14860 (2022).", "other_comments_or_suggestions": "In the related work section, should discuss some studies based on Slot Attention (SA) representations, since SA's motivation is also about disentangled and then reconstructing.", "questions_for_authors": "The paper mentions a low parameter count, but how does it perform in terms of FPS and overall time efficiency?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lRTIHYQzdl", "reviewer": "Reviewer_HFkz", "summary": "This paper addresses the challenge of feature entanglement in Cross-Domain Few-Shot Segmentation by leveraging the inherent structure of Vision Transformers. The authors identify that current methods often suffer from entangled semantic patterns, which hinder the transferability of features across domains. To tackle this, the paper proposes a framework that disentangles semantic patterns by analyzing ViT natural decomposition and re-composing these disentangled features to improve generalization and adaptation. The approach introduces mechanisms to reduce feature correlation, learns meaningful comparisons across layers, and dynamically adjusts feature weights during fine-tuning for better cross-domain segmentation. The method achieves state-of-the-art results across multiple benchmarks. Extensive experiments demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Orthogonal loss weight\n\nWe validated the impact of the weight of the orthogonal loss on performance. The results indicate that the optimal choice of the weight is in a wide interval, which means the tuning of this hyper-parameter is not difficult. Additionally, we used the same orthogonal loss weight in the Swin Transformer architecture as we did in the ViT architecture. Its performance indicates that our method design is not sensitive to this weight.\n\n| Orth Loss Weight | 0.01  | 0.05  |  0.1  |  0.2  |  0.5  |\n| :--------------: | :---: | :---: | :---: | :---: | :---: |\n| 1-shot Avg mIoU  | 62.59 | 63.01 | 63.22 | 63.18 | 62.87 |\n\n\n\n## 2. Apply to Swin Transformer\n\n|  Swin-B   |    Stage1     |     Stage2     |      Stage3      |      Stage4      |\n| :-------: | :-----------: | :------------: | :--------------: | :--------------: |\n|   shape   | H/4 * W/4 * C | H/8 * W/8 * 2C | H/16 * W/16 * 4C | H/32 * W/32 * 8C |\n| layer num |       2       |       2        |        18        |        2         |\n\nThank you for your suggestion. We further validated the effectiveness of the method on Swin Transformer. Since the layers in Swin Transformer do not reside in the same feature space, two additional steps are required: 1) For features from stage 2 to stage 4, we upsample them spatially to H/4*W/4; 2) For features from stage 1 to stage 3, we add three mapping linear layers (c,8c), (2c, 8c), and (4c, 8c) to map them to the same feature space as that of stage 4. The mapping layers are trained alongside the model during the source domain training phase. The performance results are as follows: our method is well-suited to Swin Transformer, and Swin Transformer shows a significant improvement in performance compared to ViT. Thank you again for your suggestion!\n\n|    1-shot     |  FSS1000  | Deepglobe |   ISIC    |  ChestX   | Mean      |\n| :-----------: | :-------: | :-------: | :-------: | :-------: | --------- |\n|     ViT-B     |   77.80   |   33.18   |   36.99   |   51.54   | 49.88     |\n|    Swin-B     |   79.85   |   37.24   |   39.90   |   66.73   | 55.93     |\n| Swin-B + Ours | **81.02** | **46.63** | **49.19** | **83.85** | **65.17** |", "claims_and_evidence": "1. Mutual information analysis shows reduced correlations between disentangled features after applying OSD.\n2. Quantitative results on four benchmarks demonstrate significant improvements, supported by qualitative visualizations.\n3. Ablation studies show consistent performance gains when CPC and AFW are added.", "methods_and_evaluation_criteria": "1. The proposed methods are well-motivated.\n2. The experimental setup is comprehensive, with comparisons to strong baselines and state-of-the-art methods.", "theoretical_claims": "The theoretical analysis of feature entanglement is convincing.", "experimental_designs_or_analyses": "1. The experimental design is robust, with extensive ablation studies to isolate the contributions of each module.\n2. The use of CKA similarity and mutual information to validate disentanglement is effective and aligns with the claims.", "relation_to_broader_scientific_literature": "The proposed approach adds a novel perspective by leveraging ViT’s inherent structure for disentanglement and re-composition.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of leveraging ViT’s structural decomposition for disentanglement is novel and insightful.\n2. The proposed method achieves state-of-the-art performance with significant improvements across benchmarks.\n\nWeakness:\n1. The orthogonal loss weight is manually tuned, which may limit adaptability.", "other_comments_or_suggestions": "None", "questions_for_authors": "Can the proposed framework be applied to other ViT architectures (e.g., Swin Transformer)? If so, what modifications are required?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hiXrVPnSTW", "reviewer": "Reviewer_4QAQ", "summary": "The paper finds that previous approaches have an entanglement problem, which tends to bind source domain patterns together, making each one difficult to transfer. They analyzed and explained the feature entanglement problem from a new perspective of the natural decomposition of ViT. On this basis, self-disentanglement and re-composition of CD-FSS are proposed.\n\n### update after rebuttal\nI would like to thank the authors for taking the time to address my questions and for providing the additional numerical experiments. I am satisfied with the authors' rebuttal. So I will keep my positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Design for the cross-domain part\n\nOur method design also highlights the cross-domain part of CDFSS in the following aspects. \n\n(1) Intuitively, for the cross-domain transferability, the overall semantics, like a whole bat in Fig.1, is much harder to transfer than the separated parts of it, like wings, claws. Based on this intuition, our method disentangles the overall semantics into small and transferable parts, and recomposes them on the target domain through efficient tuning of AFW. \n\n(2) Quantitatively, we validated in Fig.4 and Tab.1 that it is the mismatch between layers that harms the transferability between source and target domains by the domain similarity via CKA metrics. Therefore, our design addresses the mismatch problem and encourages the model to focus on more meaningful matches (typically with higher CKA value), and is verified in Fig.9 to show that our model can indeed highlight these matches, improving the cross-domain transferability.\n\n## 2. Portability of our methods\n\nOur method has excellent adaptability to other approaches. To verify this, we combined our method with APM and PATNet. Additionally, in response to reviewer HFkz and PmpS, we applied our method to Swin Transformer, demonstrating its suitability for different transformer architectures.\n\n| 1-shot（ViT-B） |  FSS1000  | Deepglobe |   ISIC    |  ChestX   | Mean      |\n| :-------------: | :-------: | :-------: | :-------: | :-------: | --------- |\n|       APM       |   79.84   |   39.78   |   50.37   |   79.29   | 62.32     |\n|   APM + Ours    | **80.73** | **43.67** | **52.82** | **83.01** | **65.06** |\n|     PATNet      |   72.03   |   22.37   |   44.25   |   76.43   | 53.77     |\n|  PATNet + Ours  | **80.50** | **43.18** | **46.62** | **82.49** | **63.20** |\n\n## 3. Failure cases\n\nHere https://anonymous.4open.science/r/Self-Disentangle-Rebuttal-for-ICML25-706E/failure_cases.PNG, we analyze some failure cases. We found that in DeepGlobe, areas with large contiguous regions tend to experience incomplete segmentation, particularly at the edges, where the segmentation granularity may not be fine enough. The main reasons are twofold: 1) DeepGlobe consists of remote sensing images, which are typically high-resolution, yet for computational efficiency, we standardize the images to 400x400; 2) DeepGlobe demands high local feature recognition capability from the model. ViT-based methods, due to their attention mechanism, are strong in global modeling but relatively weaker in recognizing local features compared to models like ResNet, which have inherent local priors. Therefore, we believe that future research could focus on enhancing ViT's ability to recognize local features in cross-domain scenarios.\n\n## 4. Background prototype\n\n#### (1) Role of the background prototype\n\nThe final result of segmentation is based on the probability map derived from both the support-set foreground prototype and the support-set background prototype compared with the query feature, rather than relying solely on the background prototype. Thus, the background prototype is complementary to the foreground prototype.\n\n#### (2) Support-set image and query-set image in the same background class\n\nAlthough the background of support and query image are not in the same class, most current prototype-based methods（PANet, SSP) utilize a single background prototype to model the background patterns, and have been verified to have good performance in both current works and our paper.\n\nIndeed, it is better to consider different background classes for different images. Therefore, here we further utilize clustering to obtain multiple background prototypes. By comparing the single-background prototype versus multi-background prototypes, we observed a slight performance improvement. However, the gains were not substantial enough to justify the computational overhead of clustering. Although this is not the main focus of our paper, we still appreciate your suggestion.\n\n|       1-shot        | FSS1000 | Deepglobe | ISIC  | ChestX | Mean  |\n| :-----------------: | :-----: | :-------: | :---: | :----: | ----- |\n| single BG prototype |  80.31  |   43.15   | 46.57 | 82.86  | 63.22 |\n| multi BG prototype  |  80.96  |   43.53   | 46.78 | 83.09  | 63.59 |\n\n## 5. Utilization under 5-shot setting\n\nFor the 5-shot setting, the methodology is consistent with the 1-shot approach, with the following differences: 1) During fine-tuning, each task has 5 support images available for learning; 2) Five support foreground prototypes are calculated for the same class and then aggregated by averaging.\n\n## 6. Minor writing error\n\nThank you very much for your careful correction. Equation 11 should indeed be $W_{out}$ instead of $W_{in}$​. We will take extra care to correct this in the final version to prevent any minor writing errors.\n\n## 7. FSS performance\nIn our response to reviewer PmpS's third point, we validated the effectiveness of our method in the FSS setting. Thank you for your suggestion.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "In fact, the authors present no new theories.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "They analyzed and explained the feature entanglement problem from a new perspective of the natural decomposition of ViT. On this basis, self-disentanglement and re-composition of CD-FSS are proposed.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:  \nThe motivation is clear, and the approach is sensible, straightforward, and highly applicable. The paper is easy to understand, the results are easy to reproduce, and a large number of ablation experiments have been done to prove the effectiveness of the results. Paper is well organized. The proposed approach for self-disentanglement and re-composition of CD-FSS from the perspective of natural decomposition of ViT is novel.\n\n\nWeaknesses：  \n1. It seems that the network decoupled the CD-FSS task into two tasks: Cross-Domain and FSS, and the model focused more on the FSS part, while there was not much special design for Cross-Domain.  \n2. I am curious about the portability of the proposed method, the authors did not test the feasibility of the proposed method on other existing methods.  \n3. Failure cases are not provided, which makes a complete discussion of the approach lacking.  \n4. The article uses the background prototype in CPC, I am doubtful about the role of background prototype. Because unlike the foreground, the background of the support image and the background of the query image are not constrained to the same class.\n5. The $W_{in}$ in Equation 11 should be $W_{out}$.  \n6. It doesn't describe how to use the network under 5-shot setting.", "other_comments_or_suggestions": "I suggest the authors test the performance of the model in the FSS setting as well. See if the model can achieve good results in different Settings.", "questions_for_authors": "See Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Dear authors, 
Thank you for the submitting draft. This draft received overall positive review, with three weak accept and 1 Accept. 
Reviewer oMi8 has indicated to raise the score and 4QAQ has stated to be "satisfied with the authors' rebuttal.". 
Taking these comments inconsideration, we are assigning Accept. 


regards

AC
Decision: Accept (poster)