A Closer Look at the CLS Token for Cross-Domain Few-Shot Learning
Authors: Yixiong Zou
Shuai Yi
Yuhua Li
Ruixuan Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Cross-Domain Few-Shot Learning
CLS Token
Vision Transformer
Primary area: machine_vision
TL;DR: We find the CLS token naturally absorbs domain information, and propose to decouple domain information from the CLS token and adapt it for cross-domain few-shot learning.

Abstract: Vision Transformer (ViT) has shown great power in learning from large-scale datasets. However, collecting sufficient data for expert knowledge is always difficult. To handle this problem, Cross-Domain Few-Shot Learning (CDFSL) has been proposed to transfer the source-domain knowledge learned from sufficient data to target domains where only scarce data is available. In this paper, we find an intriguing phenomenon neglected by previous works for the CDFSL task based on ViT: leaving the CLS token to random initialization, instead of loading source-domain trained parameters, could consistently improve target-domain performance. We then delve into this phenomenon for an interpretation. We find **the CLS token naturally absorbs domain information** due to the inherent structure of the ViT, which is represented as the low-frequency component in the Fourier frequency space of images. Based on this phenomenon and interpretation, we further propose a method for the CDFSL task to decouple the domain information in the CLS token during the source-domain training, and adapt the CLS token on the target domain for efficient few-shot learning. Extensive experiments on four benchmarks validate our rationale and state-of-the-art performance. Our codes are available at https://github.com/Zoilsen/CLS_Token_CDFSL.

A Closer Look at the CLS Token for Cross-Domain Few-Shot Learning
Yixiong Zou'
Shuai Yi Yuhua Li!
Ruixuan Li
School of Computer Science and Technology. "Scoool of Artificial Intelligence and Automation Huazhong University of Science and Technology fyixiongz, yishuai, idcliyuhua, rxli] Chust. edut.eeduuccc
Abstract
Vision Transformer (ViT) has shown great power in learning from large-scale datasets. However, collecting sufficient data for expert knowledge is always dif- ficult. To handle this problem, Cross-Domaie Few-Shot Learning (CDFSL) has been proposed to transfer the source-domain knowledge learned from sufficient data target domains where only scarce data is available. In this paper, we find an intriguing phenomenon neglected by previous works for the CDFSL task based VIT: leaving the CL.S token random initialization, instead of loading source- domain trained parameters, could consistently improve target-domain performance. We then delve into this phenomenon for an interpretation. We find the CLS token naturally absorbs domain information due to the inherent structure of the ViT, the low-frequency component in the Fourier frequency space of images. Based on this phenomenon and interpretation, we further propose a method for the CDFSL task decouple the domain information in the CLS token during the source-domain training, and adapt the CLS token on the target domain for efficient few-shot learning. Extensive experiments on four benchmarks validate our rationale and state-of-the-ar performance. Our codes are available at https://gthub.com//zaissncCCC.. Soooe_TTTT_______ Token_CDFSI.
which is represented
Introduction
Vision Transformer (ViT) has boosted the development of artificial intelligence due to its great capa- bility in learning from large-scale datasets 10 18 26 39 which is now the foundation model for many deep learning tasks. However, the generalization of ViT on downstream tasks has not been especially under the cross- -omainn data-scarce scenarios. To handle this problem, Cross- Dommai Few-Shot Learning (CDFSL) [14 21
fully explored yet (27 32
28
33 47 has been proposed
transfer the general knowledge from the tata-sufficient source domain (such as ImageNet [8) to
learn the expert knowledge in the data-scarce target domain (such as medical datasets (38] Due to domain gap and searce training data, CDFSL still remains challenging task.
To handle this task, we focus on the basic meanferrleariingee bbssd method, i.c., training the ViT model on the source domain by the all-class classification loss, and loading the model parameters on the target domain for classification. We find an intriguing phenomenon that is ignored by previous works: on the target domain dataset, not loading the CLS token parameters (i.e., leaving the CLS token to random initialization) would consistently lead to higher performance in most cases, although such operation may harm the nource-domain performance, as shown Fig.
In this paper, we delve into this phenomenon for an interpretation. We first study what information is encoded by the CLS token during the source-domain training. We find the CLS token naturally
"Corresponing author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

5-eay I-thot avcumcy
54.0
Fentee
92.0
ViI
33.0
S
90.0
32.5
GTS Taken
Tmage Tokken
Prolotype
Prototype
Finetune
Finetune
lllsstationn
Tampe-Densinn Perfermance
Figure I: (a) Vision Transformer (ViT) takes image tokens and learnable CLS token input. Few-Shot Learning (CDFSL) task, after the training on the source-domain dataset, evaluate the model on both the source-domain classes (b) and target-domain classes (c) by the 5-way I-shot classification. We find an intriguing phenomenon neglected by previous works: although not loading the CLS token parameters (i.e., leaving them to random initialization) on the source-domain classes harms the performance (b), not loading these parameters consistently improves the target-domain performance (c). In this paper, we delve into this phenomenon for an interpretation, and propose simple but effective method based on them for the CDFSL. task.
For the
absorbs domain information, represented as low-frequency components in the Fourier frequency space of images, due to the ViT's inherent structure which places the CLS token at the input layer and shares it with all imput image patches. Based this phenomenon and interpretation, we further propose a method for the CDFSL task. During the source-domain training. this method decouples the domain information from the CLS token make it domain agnostic, handling the domain gap problem. During the target-domain learning, this method finetunes the CLS token to efficiently absorb domain information, handling the few-shot learning problem. Experiments four benchmark datasets validate our rationale, and show that we can outperform current state-of-the-art works.
In summary, contributions can be listed follows.
We find phenomenon that is neglected by previous works: not loading the source-domain trained CLS token could consistently improve the performance for the target-domain generalization.
We delve into this phenomenon for an interpretation: the CLS token naturally absorbs domain information due to the inherent structure of vision transformers.
Based on this interpretation, further propose method to decouple the domain information in the CLS token, and make use the CLS token for efficient downstream few-shot learning.
Extensive experiments on four CDFSL benchmark datasets validate the rationale of our interpreta- tion and method, showing
outperform current state-of-the-art works.
Delve into the CLS Token for Cross-Domain Few-Shot Learning
2.1 Preliminaries
Cross-domain few-shot learning (CDFSL) requires the model to be firstly trained source-domain dataset (e.g., minilmageNee (34)), then applied to learn from a few training samples from each target-domain dataset, and finally evaluated on the test set of the target dataset. Specifically, denoting the source dataset as D = { . ] the baseline method to train the model f(()) on DS by minimizing the cross-entropy loss against the source-doman label space S
1 LCE N
CE(f(E)).)))
(I)
During the learning on each target dataset, the model will be applied to learn from the sampled K.M
support {=] v.y i=ljj=1 which called the K-way M-shot task (i.e., K classes in each support set with M samples in each class). Finally, the model will be evaluated on the query set {x }. Typically, the classification is based the distance between class prototypes and the test samples as
1 [(CT))
2 arg min di
M

---

Table 1: Not Loading the CLS token improves the cross-domain performance in most cases.
Method
CropDisease
EuroSAT
ISIC2018
ChestX
Ave.
(a.1) Baseline
81.22
72.08
32.75
22.84
52.22
(a.2) Baseline + Fix CLS as Initialization
82.50 -12
72.58 4ess
32.69 am
22.94 was
52.68 40.4
(a.3) Baseline + not loading CL.S
82.54 +1
73.02 -0s1
32.76
22.84
52.81 -059
1b Baseline+ + finctuning FC
83,14
72.81
33.82
23.09
53.22
(h.2) Baseline finetuming FC not loading CLS
83.32 w0
73.80 -0:s
33.37
23.19
53.42
(c.1) Baseline finetuning FC CLS
83,15
73.26
33.84
23.07
53.33
(c.2) Baseline + finetuning FC & CLS + not loading CLS
83.37 4000
73.63 -0:37
34.26 40
23.13 w0.36
53.60 -037
where f(x E Rd is the feature, and d(::)) denotes the Euclidean distance. Repeating the above sampling-evaluation process multiple times on target datasets, the performance will be obtained.
In this paper, we set the model f(.) to the Vision Transformer (ViT) [0]] with DINO 42 (other settings can be found in the appendix). ViT takes both the image tokens and learnable CLS token as inputs, written as
f(z) =
(3)
where 7 E Rd is the CLS token, T(x) E Rddxd combination of two kinds of tokens, as shown in Fig. I..
xd is the image tokens, and h(.-)) processes the
2.2 Wider verification of the phenomenon
To delve into this phenomenon, we first study this phenomenon under different training conditions for a detailed verification. Following [13, we first take the ViT model pretrained on the ImageNet dataset as an initialization. Then, the model is trained on the base classes of the minilmageNet dataset as the source-domain training. Finally, we learn and evaluate the model on four benchmark datasets (CropDiseases (25), EuroSAT [16). ISIC2018 (5) and ChestX (38) by the 5-way 1-shot accuracy. The target-domain learning methods include prototype- based method (Tab. [b],, which does not need to adapt the model parameters; finetuning- based methods (Tab. [b.)), which only adapts parts of
model parameters due to the scarcity of training data. Results are reported in Tab. [] where both the accuracy and the change of accuracy w.r.t. the first row of each block are included.
From this table, the model or not. This means the CLS token trained on the source domain may contain poisonous information for target-domain generalization. Therefore, we also try to finetune the CLS token (Tab. [ on the target domain, but this phenomenon still exists, which means such source-domain information strong enough that it cannot be simply handled by the target-domain finetuning.
this phenomenon exists in most cases, matter whether finetune
To ablate such information from the cource-domalle training, we directly fix the CLS token as random initialization for both the source and target domains (Tab. a.2). We can see that by abandoning the learning of the CLS token, the performance is also improved from the baseline method, but is slightly lower than training but not loading it (Tab. [ a..).. This means such information in the CLS token could be beneficial for the source-domain training. Therefore, in the following subsections, we will keep delving into such information to study what it is and how it is learned.
2.3 What information does the CLS token encode?
Intuitively, since not loading the CLS token improves performances only under cross-domain see- narios, it is natural to think of the CLS token's 's poisonous information as the domain information. Therefore, below we will validate and delve into this intuition both quantitatively and qualitatively.
2.3.1 Quantitative study: CLS token contains domain information
Firstly, we quantitatively measure the domain distance between source and target datasets by the CKA similarity following [7]. Specifically, given a backbone network, we extract features from images in different domains, and then calculate the CKA similarity by aligning the channel dimension. The larger the CKA similarity is, the smaller the domain distance will be, and it means the model contains less domain information. The results plotted in Fig. Zh. We can see:
(1) Not loading the CLS token can significantly increase the CKA similarity, indicating the CLS token contains domain information while other structures tend to capture domain-irrelevan information.
3

---

0.8
0.7
0.5
0.4
0.3
0.1
0.0
Simplesseases
FumNAA
ISIC2018
CheseX
Lood CLS
Losd
Fux CLS as Init
Ours
Domnin similittty messured the CKA similarity
(b) Visucization of the CLS telicn by retiival
Figure 2: (a) Not loading the CLS token significantly improves the domain similarity, indicating the CLS token contains domain information. (b) The similarity map between the CLS token and image tokens can roughly represent the background of the object (top two rows), which can hardly be transferred to target domains (bottom row). However, in some images (e.g., first row, second column), the highlighted regions are not necessarily the background but the dim regions (bottom-right region), which inspires us to consider whether the CLS token actually captures the uw-ffreuency components in the Fourier frequency space of images.
0.70
0.4905
0.00
0
0.55
0
Ang. Sin
0.45
0.4758
100000%
50,00%
12,30% 2.60%
0.8%%
Maistenot Proguancy (law)
(8) Visualization of the CLS images with high-frequency components removed
(b) Quantitative verification
Figure 3: (a) We only maintain the low-frequency component of images, and the frequency threshold decreases from left to right. The similarity map becomes brighter with the decreasing of the frequency, indicating the CLS token shows higher similarity to low-frequency components. (b) We quantitatively measure the average value of the similarity map and ratio of activated regions for different low- frequency images. With the decrease of the threshold, the similarity consistently increases.
(2) Fixing the CLS token random initialization in the source-domain training can further increase the CKA similarity, because the domain information inherited from the pertaining is now limited.
Therefore, we validate that the CLS token indeed contains domain information, and it plays an important role in learning such information.
2.3.2 Qualitative study: CLS token captures low Fourier-frequency components
To study the information encoded by the CLS token, we then visualize the CLS token by image retrieval. Specifically, we calculate the cosine similarity between the CLS token and the image token in the input layer, and report the similarity map in Fig. 2p. We can see the similarity map could roughly represent the contour of the object for the source dataset, by highlighting the background regions with higher similarity. Since identifying the background also means the model could find the foreground in the image, the CLS token can indeed facilitate the source-domain recognition. However, on the target datasets, such capability downgraded to activating just random regions (EuroSAT, ChestX), which limits the effectiveness of the CLSS token and is consistent with the performance change Tab.
Since the background region in ImageNet is always in bokeh due to the photography tools, we observe these regions being blurry. This inspires to consider whether the CLS token actually tends to find the low-frequency regions (with the Fourier Transform). To validate this hypothesis, we try to remove the high-fi gyggufrey components and maintain only the low-frequency components of input images, and compare the generated similarity map in Fig. 3h. Intuitively. by removing the frequency components from high to low (from left to right Fig. 3p), the input image becomes blurrier. The corresponding similarity map, as the brighter color indicates higher similarity, gradually gets brighter from left right. This indicates the information encoded by the CL.S token more likely to be contained by these blurry regions. Quantitatively, measure the average similarity value of

---

Slow (iong-rouge) change puiches
Shared by all patchees
Ine-freuunyy compusienss
BID
CropDisease
EuroSAT
ISIC2018
ChestX
Ave.
For imaage
82.54
73.02
32.76 mm
CLS rokin
X. Fasd (chariertagge cllengs of patches
76.43 sm
67.52 s:
30.09 am
21.98 6.5
49.01 .32
81.12
71.11
31.80
22.56
51.65
X High frouececy componicnss
79.21 136
72.20 am2
30.83 134
22.53 am
51.19 -.30
78.28 410
70.87 -.1
31.27 um
22.47 4.
50.72 .U51
10
80.16
70.40
31.06
22.59
51.05
For comsin
Shared by all images
Domollo informattinn
Table (Left) Interpretation of why the CLS token captures domain information. (Right) Only not
loading the CLS token in the first block can consistently improve performance.
all images in Fig. 3p as the blue curve to verify such intuition. We can see such values consistently increase with the decrease of maintained frequency.
Moreover, then utilize the similarity map generate the segmentation mask of the input image in the third row, where we set the masking threshold as the 50%-th similarity value in the original similarity map. Intuitively, we can observe that (1) with the decrease of the frequency, the masked regions get fewer, and (2) the regions (the guitarist, foreground) that are originally masked in the all- frequency (left-most) image are not masked in the low-frequency (right-most) image. This indicates the CLS token actually does not detect the fore- or background, because if the foreground (the guitarist) gets into low-frequency regions, the CLS token can no longer detect it, which verifies that is the low-frequency components that the CLS token detects. Quantitatively, we measure the ratio of regions that surpass the threshold (Act. Ratio) Fig. 3 as the red curve to verify such intuition. We can also see such ratios consistently increase as the frequency gets lower in all image
As the low-frequency information has been verified be relevant to the domain information [12), the above experiments also validate that the CLS token encodes the domain information.
In all, by qualitative and quantitative experiments, verify the CLS token encodes domain informa- tion, which contributes to the source-domain training but is harmful to target-domain generalization.
2.4 Why does the CLS token contain domain information?
Finally, we interpret why the CLS token encodes the domain information in two folds (Tab. 2
(1) The CLS token is shared by all image-patch tokens. As shown in Tab. 2 (left), the influence of such characteristics can be divided into two folds: (a) For each image, the CLS token adapts much slower than image tokens, so it cannot reflect the patch-wise change in each image. This means it tends to capture the trend of a relatively large number of patches, which is reflected in low-frequency components in the frequency space and is verified to be relevant to domain information.
(b) For each dataset, the CLS token captures the shared information for all patches. What information is shared by all patches in each dataset: the domain information, coinciding with the influence for each image.
(2) The CLS token captures lowelevel information that is more vulnerable to domain information. To verify this hypothesis, we move the CLS token to different ViT blocks and train the model on the source dataset, and report the performance change by loading the CLS token or not in Tab. P (right). We
only the first block (i.e., the input layer which is the default choice in ViT) shows consistent performance improvements by not loading the CLS token, which verifies our hypothesis.
2.5 Conclusion and Discussion
Based on the above experiments, we make the interpretation follows. The CLS token is originally designed to be placed at the input layer, which is treated as shared token for different input tokens. These two characteristics enable the CLS token to naturally absorb the domain information, since (1) the low-level information the input layer is vulnerable to domain changes, and (2) the CLS token is shared by all image tokens therefore it captures the now-frequency information in each image that is relevant to domain changes. Such information is beneficial for fource-domald learning as it partly highlights the background, but can hardly be transferred to target domains. As result, not loading the CLS token harms the source-domain performance but improves the target-domain performance.
Further experiments are conducted in the appendix show not all tokens tend to have such a characteristic.
5

---

Feature
Feature
D Demain Token
Phendoo alavain
ViT Backbone
ViI Backbone
CLS Token
CLS Token
Imuge Token
Imuge Token
Rondom Intuulttatiin
CLS Token
Source Dumain
CLSS Token
Turget Dowaii
Figure 4: Based on the phenomenon and interpretation, we propose to decouple the domain informa- tion from the CLS token to make it domain-agnosti during the source-domain training, and utilize the CLS token's is characteristic in absorbing domain information for efficient target-domain adaptation. Specifically, during the source-domain training, we generate pseudo domains on the source dataset by clustering, and apply a domain token for each pseudo domain. We fix the CLS token as the random initialization, and add the domain token to the fixed CLS token, that domain tokens will substitute the CLS token in absorbing domain information, which decouples the domain information from the CLS token. During the target-domain adaptation, we abandon domain tokens and finetune the CLS token to absorb target-domain information for efficient few-shot learning.
Morcover, we draw inspirations for the model design to facilitate the target-domain generalization and few-shot learning: (1) The model should the one hand maintain the learning of CLS token due its benefits in source-domain learning, and on the other hand alleviate its negative effect domain bias. Simply not loading the CLS token the target domain cannot fully take advantage of the CLS token's characteristics. (2) The CLS token is beneficial for downstream few-shot learning, as efficiently encodes domain information with just few parameters. Therefore, can focus the finetuning of
the CLS token for the downstream fast adaptation (akin to the prompt tuning [15 31 37 40))
Method: Decoupling Domain Information from the CLS Token
Based on the above interpretation, we aim to make use of the CLS token's 's characteristic of absorbing domain information for the CDFSL task. Therefore, we propose to decouple the domain information naturally absorbed by the CLS token during the source-domate training to make the CLS token domain-agnostic, and finetune the CLS token for efficient target-domain few-shot learning (Fig.
For the source-domain stage, our decoupling method includes three modifications to Sec. 2.1
(1) Not loading the pre-trained CLS token parameters, since the domain information absorbed in the CLS token does not help the CDFSL task.
(2) Random initialize the CLS token, so that it does not contain any domain information, and then fix it, as the learning of the CLS token will naturally absorb domain information.
(3) Explicitly adding source- doomain-sppciiii tokens to the CLS token, which would substitute the CLS token to absorb the domain information and will not be applied on the target domain.
In summary, we aim to decouple the original CLS token into a -pann--domai--ppecfficc part (we call domain tokens) and source-domains nource-domain-agnoste part (the fixed CLS token), where the domain tokens will be abandoned on target domains, and the CLS token fixed on source domains will be finetuned on target domains for efficient few-shot learning. Therefore, to absorb domain information more effectively. However, although the original CLS token could naturally absorb domain information, its efficacy may not be optimal, since the original CLS token is shared by all images in the source dataset, but the source dataset could demonstrate sub-domains it.
need to encourage domain tokens
To handle this problem, we create sub- do-maoomins by clustering the class prototypes (i.e., fully collececodeeted layer parameters) generated by baseline model. For clarity, we call these created domains as pseudo domains. Then, we assign each pseudo domain with an individual domain token, which makes such domain token only absorb information specific to such pseudo domain, by means of sharing different domain tokens among different pseudo domain images, instead of sharing domain token

---

among all source-domait images. This improves the efficacy of absorbing domain information, thereby helping source-domain learning.
Specifically, following section 2.1 we denote the ViT backbone network as f(x) = h(TC, (())) where 7 Rº is the CLS token and T(x) E R xd is image tokens. Then, we manually specify represent n pseudo domains on the source dataset. Given an input image x, besides its source-domait label y. we also have access to its pseudo-domal label 2, since these pseudo domains are created by ourselves. Therefore, we can obtain the domain token for this sample as T E R Subsequently, we replace the original input CLS token T by the combination of the domain token and the CLS token as
set of learnable domain tokens T.D RT
f(z) = h(T + +T))())).
(4)
Ideally, since now T is fixed as the random initialization, T. would substitute T to learn the information that is originally captured by TC. Since we explicitly specify different T for each pseudo domain, the domain information will be better absorbed by the domain tokens.
To explicitly encourage the decoupling of domain information, we also apply another loss to make the domain token orthogonal the CLS token, which represented as
n
TO
TP
Lorth nt
(5)
|
TPI|
Finally, the model is trained by the cross-entropy loss in Eq. D and Lorth with a
À as
== LCE + Xortth
(6)
For the target-domain stage, we abandon the domain tokens as they capture the source-domain information, and directly utilize the CLS token Eq. For the prototype-baseh classification, follow Eq. for target-domain recognition. For the finetuning- based classification, note that there is no domain gap between the training set and the test set the target domain. Since the CLS token naturally absorbs domain information, now we set the CLS token to learnable parameters to encourage it to absorb target-domain information. The finetuning is based on the cross-entropy loss on the support set, and we only train the classifier and the CLS token for efficient adaptation.
Experiments
4.1 Dataset and Implementation Details
Following current works [28 (3), we utilize the minilmageNet dataset [34] as our source domain with around 60k images and 100 annotated classes. We train models on the training split of the source dataset, then finetune and evaluate the generalization performance on four target-domain datasets, CropDisease (25), EuroSAT [16), ISIC (5), and ChestX (38), which are cross-domain datasets from the domain of agriculture, remote sensing and medical data (with significant domain gaps).
In implementation, domain token. We set to 100 to keep two losses on the same scale. We follow [[ 42) to take DINO on ImageNet as the pretraining of our backbone network, and scale the learning rate of the domain token to 1% of the backbone network. Experiments are conducted on NVIDIA A5000 GPUs.
set the domain number to 64, i.e., each source-domain class has specific
4.2 Comparison with State-of-the-Art Works
We report our comparison with state-of-the-art works utilizing the VIT-S backbone for both 5-shot and I-shot settings in Tab. 3 and For a fair comparison, we compare works with and without finetuning (FT) respectively. The asterisk (+) denotes a transductive setting. As can be seen, our results achieve the top average performance in all settings and consistently outperform current works, MEM-FS 35), AtyleAdv
13 PMF [29 and FLoR [46], in almost all datasets.
4.3
Ablation Study
The ablation study reported in Tab. 5 for the prototype-based classification which studies the source-domain training. We can see both the CLS decoupling and the orthogonal loss contribute to
7

---

Table 3: Comparison with state-of-th satte-of-the-arr art works by the 5-way 5-shot classification.
Methods
backbone
FT
Mark
Crop.
Euro.
ISIC.
Ches.
Ave.
LDP-net 44
ResNet10
CVPR-23
89,40
82.01
48.06
26.67
61.29
GNN+AFA
ResNet10
ECCV-22
88.06
85,58
46,01
25.02
61.67
SDT 22
ResNet10
X
NN-24
90.27
82.02
48.66
27.20
62.04
FLOR H6
ResNet10
X
CVPR-24
91.25
80.87
51.44
26.70
62.32
MEM-FS 35
VIT-S
TIP-23
93.74
86.49
47.38
26.67
63.57
StyleAdv
VIT-S
X
CVPR-22 23
94.85
88.57
47.73
26.97
64,53
MICM 43
VIT-S
X
MM-24
94.61
90.08
46.85
27.11
64.66
SDT [22
VIT-S
X
NN-24
95.00
89.60
47.64
26.72
64,75
Fl.oR 46
VIT-S
X
CVPR-24
95.28
90.41
49.52
26.71
65.48
CD-CLS
VIT-S
Ours
95.68
91.04
50.46
27.23
66.10
FLoR 46
ResNet10
CVPR-24
92.33
83.06
56.74
26.77
64.73
PMF 29
VIT-S
CVPR-22
92.96
85.98
50.12
27.27
64.08
StyleAdy
VIT-S
CVPR-23
95.99
90.12
51.23
26.97
66.08
FLoR 46
ViT-S
CVPR-24
96.47
90.75
53.06
27.02
66.83
CD-CLS
VIT-S
Ours
96.27
91.53
$4.69
27.66
67.54
LDP-net"
ResNet10
CVPR-23
91.89
84.05
48 44
26.88
62.82
RDC 19
ResNet10
CVPR-22
93.30
84.29
49.91
25.07
63.14
FL.oR 46
ResNer10
CVPR-24
93.60
83.76
57.54
26.89
65.45
MEM-FS+RDA'
35
VIT-S
TIP-23
95.04
88.77
$1.02
27.98
65.70
CD-CLS"
VIT-S
Ours
96.62
91.68
28.25
68.05
Table 4: Comparison with state-of-the-ar works by the 5-way 1-shot classification.
Method
buckbone
FT
Mark
Crop.
Euro.
ISIC.
Ches.
Ave.
GNN+AFA 17
ResNet10
BCCV-22
67.61
63.12
33.21
22.92
46.97
I.P--net 44
ResNet10
CVPR-23
69.64
65.11
33.97
23.01
47.18
Fl.oR [46
ResNet10
CVPR-24
73.64
62.90
38.11
23.11
49.69
SDT 22
ResNet10
NN-24
73.92
65.87
36.45
23.22
49.97
MEM-FS 35
ViT-S
TIP-23
81.11
68.11
32.97
22.76
51.24
StyleAdy
13
VIT-S
CVPR-23
81.22
72.15
33.05
22.92
$2.34
SDT 22
VIT-S
NN-24
81.03
72.71
33.40
22.79
52.48
FLoR 46
ViT-S
CVPR-24
81.81
72.39
34.20
22.78
$2.80
CD-CLS
VIT-S
Ours
83.51
74.08
34.21
22.93
53.68
FLoR H6
ResNet10
CVPR-24
84.04
69.13
38.81
23.12
53.78
PMF (29
ViT-S
CVPR-22
80.79
70.74
21.73
50.91
FLoR 46
ViT-S
CVPR-24
83.55
73.09
35.49
23.26
53.85
StyleAdv
ViT-S
CVPR-23
84.11
74.93
33.99
22.92
53.99
CD-CLS
ViT-S
Ours
84.54
74.97
35.56
23.39
54.62
I.PP-nt 44
ResNet10
V
CVPR-23
81.24
73.25
33.44
22.21
52.54
RDC 19
ResNet10
CVPR-22
85.79
70.51
36.28
22.32
53.73
FLoR" 46
ResNet10
CVPR-24
86.30
71.38
41.67
23.12
55.62
MEM-FS+RDA'
ViT-S
TIP-23
83.74
75.91
37.07
23.85
55.14
CD-CLS"
VIT-S
Ours
87.39
78.41
37.20
23.88
56.72
the performance, and the CLS decoupling contributes the most. Moreover, conduct experiments on the second block to compare different design choices from our final method.
4.3.1 Verification of fixing CLS token as random initialization
Since our method fixes the CLS token as the random initialization during the source-domain training, we first try to load the DINO pretrained CLS token (Tab. The performance drops from 66. 10 to 64.46. This is because the CLS token's capture of domain information is not bound to the label supervision. Although DINO is a self-supervised lcarning method, it still absorbs strong domain information from ImageNet, therefore we can hardly decouple the domain information from it.
Then, we try
the CLS token learnable parameters (Tab. 5p), and we can see the performance still drops from 66.10 10 to 64.83, which is slightly higher than (Tab. 5l).. This is because although we explicitly set the domain token for decoupling, it is still located in the input layer and changes slowly with different input tokens. Therefore, it still captures the domain information as long as we set it to learnable parameters, although the captured domain information is not as strong that from DINO.
characteristic the CLS token does not change:

---

Table Ablation study of source-domain training by the 5-way 5-shot accuracy.
Method
CropDisease
EuroSAT
ISIC2018
ChestX
Ave.
Baseline
94.62+002
88.62+02
46.08+0.3
26.25+o.19
63.89+014
Decoupling
95.55+0m
90,48
49.58+0as
27.03=0.13
65.66=015
Orth
95.68
91.04
50.46su11
27.23
66.10s017
(a) Load CLS
94.93+021
88. +
47. 17.15
26.77
64. 04.46=a014
(b) Learn CLS
95.36+022
89. +927
47.12 +0.2
26.94+020
64.83=0.1
(c) w/o Domain Token
94.84+002
89 0.33
46.63+002
26.99
64.50=0.16
(d) Not learn Domain Token
95.40+021
90.67
48. 48.45+t024
27.09
65.37
(c) Random Domain Choice
94.761021
89.87
46.08
26.8220019
64.38
(f) Decouple by Prompt
95.07.10.
89.87
46.681021
16.78
64.502014
Table 6: Ablation study of target-domain finetuning by the 5-way 1-shot accuracy.
Method
CropDisease
EuroSAT
ISIC2018
ChestX
Ave.
Train CLS
84.54
74.97
35.562017
23.392a1
54.622013
Fix CLS
84, 40
74.61 10
34.82ao2s
23. 23.13aun
54.242mm
Decouple CLS
80.82 #021
69.93
33.37
22.91
51.77#0.2
4.3.2 Contribution of domain tokens
Subsequently, we study the contribution of domain tokens. We first directly remove the domain token in Vab.5pp (i.e., fixing the CLS token to random initialization, Tab. a..2) We can see the performance higher than the baseline model, but is much lower than final method, which is consistent with
Tab. and verifies the domain token is crucial.
Then, we try to fix the domain token random initialization (Tab. 51). The performance is higher than Tab DT. indicating the separated domain information can indeed help to absorb domain information. Moreover, such absorption is enhanced by setting the domain token to learnable parameters ours.
Finally, we try to randomize the domain choice in Tab, 5t. i.e., randomly choosing domain tokens for each sample. We can see the performance is even lower than Tab. Te, indicating the domain token can indeed represent each pseudo domain and important for CLS decoupling.
4.3.3 Verification of (arget-domain finetuning
To study the contribution to target-domain finetuning, we report Tab. 6 for kietuning-based classifica- tions under the most challenging 1-shot scenario. Compared with the default choice of finetuning (Fix CLSS), method (Train CLS) can indeed improve the finetuning performance, verifying the effectiveness of the CLS token in fast learning the domain information.
Moreover, we try apply the CLS decoupling method to the finetuning, and the performance greatly drops. This is because there is no domain gap between the target-domain finetuning and evaluation, and the domain information is beneficial now. Therefore, the decoupling of domain information harms the performance, which on the contrary verifies the effectiveness CLS decoupling.
4.4 Verification of Domain Tokens and Hyper-parameters
To study what the domain token encodes, we follow Sec. 2.3 to plot the similarity map between the domain token and image tokens in the corresponding class in Fig. [ We can see the domain token can better capture the low-frequency regions (majorly represented as background) in each image compared with the baseline's 's CLS token (denoted as BL-CLS). Moreover, quantitatively validated the domain distance in Fig. 2.. where we can see the domain similarity significantly increases by applying the domain token. These results indicate the domain can better absorb the domain information, so that can better help to decouple such information from the CLS token.
Finally, we study the hyper- parameters in Fig. Fb.c. We can see only the CLS token in the first block (input layer) could be decoupled, consistent with Tab. 2 The best choice of cluster number number of source-domain classes, which means each class can be viewed as a pseudo domain. Since the source dataset (minilmageNet) is general classification dataset, the difference between each class is larger (c.g., than fine-grained datasets where domain information is clear). Therefore, for

---

66.0
65.0
64.0
63.5
8
9
10 11 12
(b) Block ID vs. S-shot performance
65.5
65.0
64.5
63.5
Image
BL-CLS
Domsin Tokon
Imsage
BL-CLS
Domsia Tokeu
(8) Visualization domain tokens
(c).Num cluster vs. 5-shot performance
Figure 5: (a) Domain tokens can better represent the background regions than the CLS token in the baseline method (BL-CLS), validating the effectiveness in absorbing domain information. (b) Applying the decoupling only on the first block (input layer) can improve the performance, consistent with Tab. 2 (c) The best pseudo-domain choice is to view each source-domain class domain.
minilmageNet, is reasonable to view each class as a domain. Further experiments in the appendix verify that domain tokens absorb domain information instead of class-speciff information.
Related Work
Cross-Domain Few-Shot Learning (CDFSL) has been studied by several works [14] 28 33 47 which focuses training model on the source domain that can generalize well target domain with limited examples. Current methods can be categorized into two types: meta-learning based approaches [12 [4 17 36), which aim at learning -annosic knowledge in order learn new 44 46 ttckling the problem based on reusing the model trained on the base classes data in a standard supervised learning way. However, these works are mostly restricted to the CNN architecture. Recently some works [3 29 35 39
tasks efficiently. and transfer learning based approaches 4 14 20,
focus on the transformer structure to solve the CDFSL tasks but these efforts have not fully dug out the potential of the VIT structure and the importance of the CLS token on CDFSL.
The CLS token. Adding extra special tokens the input sequence in the transformer architecture is popularized in BERT [9] However, most methods extend the input tokens with new tokens in order to use their output value as the output of the model. like ViT adding token the CLS token [O]. Recently, some works [2] 23 24) focus on using the CLS tokens in ViT to prune the network structure reduce computational efficiency, and some researches 6 focus on adding additional tokens the ViT to store and retrieve information. Different from them. our study is the first, as we know, to delve into the influence the CL.S token on cross-domain transferability, alleviating the domain gap by decoupling the CLS tokens into domain-specific and domain tokens.
Conclusion
In this paper, we find a phenomenon that not loading the pretrained CLS token parameters can improve the CDFSL performance. We delve into this phenomenon for interpretation, and find the CLS token naturally absorbs domain information due to ViT's structure. Based on the interpretation, further propose method to decouple the domain information in the CL.S token for the CDFSL task. Extensive experiments on four CDFSL benchmarks validate our rationale and effectiveness.
Acknowledgments
This work is supported by the National Natural Science Foundation of China under grants 62206 102, 62376103, 62436003 and 6230222222 18:; the Science and Technology Support Program of Hubei Province under grant 2022BAA046: Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund.
10

---

References
[1] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[2] Shuning Chang, Pichao Wang, Ming Lin, Fan Wang, David Junhao Zhang, Rong Jin, and Mike Zheng Shou. Making vision transformers efficient from a token sparsification view. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6195-6205, 2023.
[3] Xiangyu Chen, Qinghao Hu, Kaidong Li, Cuncong Zhong, and Guanghui Wang. Accumulated trivial atlention matters in vision transformers on small datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3984 1944-399, January 2023.
[4] Xinyang Chen, Sinan Wang. Mingsheng Long, and Jianmin Wang. Transferability vs. discriminabil- ity: Batch spectral penalization for adversarial domain adaptation. In Proceedings of the International Conference on Machine Learning, pages 1081-1090. PMLR, 2019.
[5] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpem. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), 2019.
[6] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024.
[7] MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, and Eugene Belilovsky. Reliability of cka as similarity measure in deep learning, 2022.
[8] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 248-255. Ieee, 2009.
[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4171- 4186, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[11] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE//VV International Conference on Computer Vision (ICCV), pages 6824-6835. October 2021.
[12] Yuqian Fu, Yu Xie, Yanwei Fu, Jingjing Chen, and Yu-Gang Jiang. Wave-san: Wavelet based style augmentation network for cross-domaiie sew-shot learning. 2022.
[13] Yuqian Fu, Yu Xie, Yanwei Fu, and Yu-Gang Jiang. Styleadv: Meta style adversarial training for cross- domain fow-shot learning, 2023.
[14] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In Proceedings of the IEEE/CV European Conference Computer Vision, pages 124-141. Springer, 2020.
[15] Zixian Guo. Bowen Dong, Zhilong Ji, Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo. Texts as images in prompt tuning for multi-label image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2808-28111 2817, 2023.
[16] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth, Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, 2019.
[17] Yanxu Hu and Andy J. Ma. Adversarial feature augmentation for cross-domain few-shot classification, 2022.
[18] Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets. arXiv preprint 4rX2v 2021.
11

---

[19] Pan Li, Shaogang Gong, Chengjie Wang, and Yanwei Fu. Ranking distance calibration for cross-domain few-shot learning, 2022.
[20] Hanwen Liang, Qiong Zhang, Peng Dai, and Juwei Lu. Boosting the generalization capability in cross- domain few-shot learning via noise-enhanced supervised autoencoder, 2021.
[21] Bingyu Liu, Zhen Zhao, Zhenpeng Li, Jianan Jiang, Yuhong Guo, and Jieping Yc. Feature transformation ensemble model with batch spectral regularization for cross-domain few-shot classification. arXiv preprint 3xii:20005084633 2020.
[22] Yicong Liu, Yixiong Zou, Ruixuan Li, and Yuhua Li. Spectral decomposition and transformation for cross-domain few-shot learning. Neural Networks, 179:106536, 2024.
[23] Sifan Long. Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10334-10343, 2023.
[24] Chenyang Lu, Daan de Geus, and Gijs Dubbelman. Content-aware token sharing for efficient semantic segmentation with vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23644,,340, 2023.
[25] Sharada P. Mohanty, David P. Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in Plant Science, 7(September), September 2016. Publisher Copyright: 2016 Mohanty, Hughes and Salathé.
[26] Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shah- baz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. In M. Ranzato, A. Beygelz- imer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 23296 23308. Curran Associates, Inc., 2021.
[27] Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Gustavo A. Vargas Hakim, David Osowiechi, Ismail Ben Ayed, and Christian Desrosiers. Tfs-vit: Token-level feature stylization for domain generalization. Pattern Recognition, 149:1 I 10213, 2024.
[28] Jaehoon Oh, Sungnyun Kim, Namgyu Ho, Jin-Hwa Kim, Hwanjun Song, and Se- Young Yun. Understand- ing cross-domain few-shot learning based domain similarity and few-shot difficulty, 2022.
[29] Jan Stühmer Shell Xu. Da Li. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference, 2022.
[30] Jake Snell, Kevin Swersky. and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the International Conference Neural Information Processing Systems, pages 1080-4090, 2017.
[31] Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, Han Zhang, Yuan Hao, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19840- 19851, 2023.
[32] Maryam Sultana, Muzammal Nascer, Muhammad Haris Khan, Salman Khan, and Fahad Shahbaz Khan. Self-distilled vision transformer for domain generalization. In Proceedings the Asian Conference on Computer Vision (ACCV), pages 3068-3085, December 2022.
[33] Hung- Yu Tseng, Hsin- HisinYYing Lee, Jia-Bin Huang, and Ming-Hsuar Yang. Cross-domain few-shot classifica- tion via learned feature-wiss wise transformation. Proceedings of the International Conference on Learning Representations, 2020.
[34] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the International Conference on Neural Information Processing Systems, pages 3637-3645, 2016.
[35] Recce Walsh, Islam Osman, and Mohamed S. Shehata. Masked embedding modeling with rapid domain adjustment for few-shot image classification. JEEE Transactions on Image Processing, 32:4907-4920, 2023.
[36] Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task augmenta- tion, 2021.
[37] Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position-guus guided text prompt for vision- language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23242-23251, 2023.
12

---

[38] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). JEEE, July 2017.
[39] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision (ECCV), 2022.
[40] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- lasual- prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6757-6767, 2023.
[41] Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao, Xianglong Liu, and Ziwei Liu. Delving deep into the generalization of vision transformers under distribution shifts. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7267-7276, 2022.
[42] Hao Zhang, Feng Li, Shilong Liu, Lci Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
[43] Zhenyu Zhang, Guangyao Chen, Yixiong Zou, Zhimeng Huang. Yuhua Li, and Ruixuan Li. Miem: Rethinking unsupervised pretraining for enhanced few-shot learning. Proceedings of the 32nd ACM In- ternational Conference on Multimedia, MM '24, page 7686-7695, New York, NY, USA, 2024. Association for Computing Machinery.
[44] Fei Zhou, Peng Wang, Lei Zhang, Wei Wei, and Yanning Zhang. Revisiting prototypical network for cross domain few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 20061-20070, June 2023.
[45] Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021.
[46] Yixiong Zou, Yicong Liu, Yiman Hu, Yuhua Li, and Ruixuan Li. Flatten long-range loss landscapes for cross-domain few-shot learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23575-2353 June 2024.
[47] Yixiong Zou, Shanghang Zhang, Jianpeng Yu, Yonghong Tian, and José MF Moura. Revisiting mid-level patterns cross-domain few-shot recognition. Proceedings of the ACM International Conference on Multimedia, pages 741-749, 2021.
13

---

A Appendix supplemental material
A.1 Detailed Dataset Description
minilmagenet
CropDiscases
BuroSAT
ISIC2018
ChestX
source domain
target domain
Figure 6: Samples source domain minilmageNet dataset(left) and target domain datasets (right), from left to right correspond to five distinct datasets: minilmageNet, CropDiseases, EuroSAT, ISIC2018, and ChestX.
a subset of the ImageNet[8] dataset that contains 100 categories, each consisting of 600 natural images. Following the current work 28 33 we split the minilmageNet into 64 classes as the source domain training dataset. addition, as shown in Figure 6 we utilize the datasets from four other different domains, like agriculture, remote sensing, and medical data, as target domains. We'll sequentially introduce each them below.
minilmageNet 34
25 consists of 38 distinct classes and total of 43,456 images, which are natural images, but are very specialized (specific to the agriculture industry), including various infected crops. healthy plants, and their corresponding disease category labels.
CropDiseases
EuroSAT [16]
contains total of 27, ,,000 satellite images of the Earth categorized into 10 distinct classes. The images in the EuroSAT are less similar to images minilmageNet since they lack perspective distortion, but still color images of natural scenes.
ISIC2018 B which is even less similar to the minilmageNer as they could not even represent natural scenes, encompasses 10,015 medical images for skin lesion classification across 7 different classes.
medical dataset for chest classification, consists of 25,847 images distributed across 7 distinct classes. The dataset is the most dissimilar to the minilmageNet in three criteria. Apart from the factors mentioned above, it loses color channels that appear in the ChestX.
ChestX 38
A.2 More Experiments
A.2.1 More Visualization of the CLS token by retrieval
To delve into the information encoded by the CLS token, we calculate the cosine similarity between the CLS token and the image token in the input layer of fixed ViT trained the source dataset and report the similarity map Figure 7 regions have higher similarity map, roughly representing the contour of the object for the source domain dataset. means that the CLS token can distinguish the background and the foreground of the image which could indeed facilitate source-domain recognition. However, as shown in Figure from above to bottom, with images that are less similar to the source domain, the ability to recognize the background for the CLS token gradually fades into mediocrity.
clear that on the source dataset, the background
14

---

minimageNett
source domain
CropDiscases
EuroSAT
ISIC2018
ChestX
target domain
Figure 7: Samples from top to bottom correspond the image token on the minilmageNet and four target domain datasets, like CropDiseases, ETroSAT, ISIC2018, and ChestX. It can be seen that the CLS token can distinguish the background and the foreground of the image which could indeed facilitate the source-domain recognition the source dataset while meeting more difficulties with larger domain gap.
similarity map between the CLS token and
A.2.2 More Visualization of the domain tokens by retrieval
Image
BL-CLS
domain token
Image
BL-CLS
domain token
Figure 8: Applying the domain token significantly improves the domain similarity compared to the CLS token of the baseline method (BL-CLS), validating the effectiveness of our approach in absorbing domain information
To delve into what the domain token encodes, re-utilize the similarity map plotted in Figure g as our measurement to compare our approach with baseline's CLS token (denoted as BL-CCL),. seems that when utilizing the domain token, the domain similarity significantly increases, proving
15

---

that the domain token can better absorb the domain information and efficiently help to decouple such information from the CLS token.
A.2.3 Applying Our Method Other Baselines
Table 7: Ablation study of our method with iBOT iBOT-pretrained VIT and DINO-Pretrained ViT-Base by
the 5-way 1-shot accuracy.
Method
CropDiseases
EuroSAT
ISIC2018
ChestX
Ave.
iBOT
81.17
72.71
31.44
22.56
51.97
iBOT + Ours
81.31
72.80
31,87
22.57
52.14
DINO- VIT Base
82.97
72.06
34.19
22,60
52.95
DINO-VIT-BAR Ours
83.11
73.77
34.75
22.98
53.65
Table 8:
Implementing method with meta-learning baseline.
Method
Crop.
Furo.
ISIC.
Ches.
Ave.
ProtoNet
93.59
86.92
46.15
25.68
63.09
ProtoNet + Ours
95.03
89.42
48.67
27.15
65.07
We also implement our approach on distinct backbones, like ViT pretrained by iBOT 45), and ViT-Base [42] pretrained by DINO. The results can be seen in Tab Specifically, iBOT represents the iBOT-pretrained ViT baseline, and DINO-ViT-Base corresponds the ViT-Base pretrained by DINO baseline. It is clarified from the average performance of four target domains that our approach shows improvements among both backbones in the 5-way 1-shot setting.
To verify our model also suits the meta-learning-basse -based baselines, we conduct experiments based on that our model also improves this kind of baseline method.
the ProtoNet (30 in Tab 8 We
A.2.4 Further Verification of the CLS Token
Randons (L1)
021
CLS
0.16
(0.1)
Ranles
(0.2)
Visaaiataionn d'undon CLS
(b).rt. Rations zanddasa tikes
Figure 9: To verify the CLS token that tends to capture low-frequency components, we (a) visualize similarity maps of random token and (b) use random tokens calculate the similarity map of low cow-frequency images, and find random tokens do not show the same results as the CLS token.
To verify is the CLS token that tends capture low- www-freuueccy components, we visualize similarity maps of random token in Fig. Ph.. We use the same color bar as in Fig.2b. We can see the similarity is much lower, a coarse contour of objects can still observed both the source and target domains, indicating a good transferability of detecting object contours, because no domain information is in the random token. However, the contour detected by the random token much worse than that by the CLS tokens as shown in Fig. and Fig.5a, indicating although the random CLS token can initially detect the object contour, the learning of the CLS token strengthens this characteristic to detect low- www-frequency images.
Then, we use random tokens to calculate the similarity map of low low- wweffreuy images, and find random tokens do not show the same results as the CLS token. Specifically, we measure the activation ratio of the CLS token and the random token. We take the top 10% or 20% value as examples in Fig. Pp. We can see the random tokens show tendency to decrease the activation ratio, while the CLS token shows a tendency to increase the ratio, indicating is the CL.S token that tends to be similar to the feature of low--fregg fow-frequency images.
16

---

Table 9: Verification of re-initializing the CLS token.
Method
Crop.
Euro.
ISIC.
Ches.
Ave.
Bascline
95.62
88.62
46.08
26.25
63.89
Ours
95.55
90.48
49.58
27.03
65.66
Ours Re-Init
95.20
89.60
50.23
26.79
65.31
A.2.5 Further Verification of Domain Tokens
To verify that model does not overfit the fixed random CLS token (so that view the fixed value as a kind new domain token), we re-initialize the CLS token during the source-domain training. The results are reported in Tab. and we can see the improvements also exist.
However, re-initializing the CLS token can be viewed as adding noise to the domain token, therefore harming the absorbed domain information, which then affects the domain irrelevaan information learned by other structures VÝT. As result, the Re-Init performance is slightly lower.
Indeed, domain tokens are encouraged to be orthogonal to the fixed CLS token, which would drive the model to view the fixed token as domain- Aomainnagnostic token. But note that the random token is already agnostic enough to every domain even without training, therefore our training would not essentially drive the model be more agnostic that CLS token, i.c. our model not bound the specific value of the CLS token.
Table 10: Training with datasets of 5 constructed domains.
Method
Crop.
Euro.
ISIC.
Ches.
Ave.
Baseline
93.85
89.72
49.74
26.07
64.77
320 classes as domain token
95.03
90.49
49.10
26.81
65.35
64 classes as domain token
95.16
90.61
47.86
26.77
65.10
5 domains domain token
95.52
90.62
50.77
27.00
65.98
To further ablate "domain-specifer and "class-specifie", then manually construct domains based on minilmageNet. Specifically, we take the amplitude (by Fourier transformation) from target domains the style information, and use the phase (by Fourier transformation) from the original source-domain images as the content information, thereby constructing new domains with the original 64 source nource-domain classes. Then, we train our model on a new dataset containing the constructed datasets and the original source dataset, and ablate different choice of domain tokens in Tab. I0
source
As can be seen, by introducing larger domain gaps, viewing each class as a domain is not the best choice. Instead, setting domain token for each domain could achieve the best performance, which validates the rationale of the domain token in absorbing the domain information.
A.3 Broader Impact
We propose CD-FSL method based on decoupling the CLS token into domain- sppcificc and domain- agnostic tokens in the ViT and making use of it for efficient downstream few-shot learning to alleviate the domain gap and generalize well to the target domain. This work also be adopted in other fields, like domain generalization, domain adaption, and few-shot class-incremental learning, where the challenge of enhancing the transferability of model exists universally. The evaluations of approach are mainly across four different target domains, which may not represent all possible real-world scenarios. The approach be evaluated on various target domains to be validated in more realistic setting.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not have theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We have reported the detailed settings and ppper-paramttes... We will release our codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
httpe/uuiides/oodeSuuiisssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpol/iiigggddde/ooooooo weweeeeeeeooeSbummmioooe ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have reported the detailed settings and our codes.
We will release
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We have reported the confidence interval in our ablation studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the computer resources the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///arrrr.ccccooo/cc//aaaaaa
Answer: [Yes]
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impact in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not have the risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have respected the license of assets used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not have assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "nEmT8BAJQJ", "reviewer": "Reviewer_epY7", "summary": "This paper presents a novel approach to Cross-Domain Few-Shot Learning (CDFSL) by investigating the role of the CLS token in Vision Transformers (ViT) for knowledge transfer under great domain gaps. The authors identify an intriguing phenomenon where not loading the CLS token parameters improves target-domain performance. They delve into this by proposing a method to decouple domain information from the CLS token during source-domain training and adapt it for efficient few-shot learning in the target domain. The paper is well-structured, presenting a clear problem statement, methodology, and extensive experiments across four benchmarks.", "questions": "The author explains the reason why the CLS token absorbs domain information is because it lies in the input layer and does not change according to input images. This insight is reasonable and interesting. As I know, the register token (Vision transformers need registers, ICLR 2024) also serves in the similar role as the CLS token. Therefore, in my understanding, if this explanation holds, the register token would show a similar phenomenon as the CLS token. Could the author conduct experiments to verify this?", "limitations": "Could this approach be applied to other tasks such as domain generalization or domain adaptation? How much computational cost is added if the regular CLS token is replaced by the proposed domain tokens?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a significant problem in CDFSL, providing a new perspective on the role of the CLS token in ViTs. The methodology is innovative, with a clear rationale behind decoupling domain information from the CLS token. The experiments are comprehensive, covering multiple datasets and ablation studies that validate the effectiveness of the proposed approach. The paper is well-written, with a clear presentation of the problem, methodology, and results.", "weaknesses": "The generalizability of this approach to other tasks beyond the ones tested is not fully discussed. In my understanding, other tasks involving domain gaps and finetuning would benefit from this method. Could the authors elaborate on how this approach could benefit related tasks, such as domain adaptation or domain generalization? This paper would benefit from a discussion on the computational efficiency of the proposed method compared to existing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n\n## 1. How could this method benefit other tasks\n\n​\tOur method could also benefit other cross-domain tasks. To verify this, we conduct experiments on the domain generalization task on 4 datasets (Sketch, Cartoon, Art Painting, Photo).  Each dataset shares seven object categories (dog, elephant, giraffe, guitar, house, horse, and person) with 9,991 images.\n\n​\tDue to time limitations, we implement the code based on our original setting, i.e., viewing miniImageNet as the source domain, and taking the 5-way 5-shot classification. The classifier on target domains is obtained by the linear probing method.\n\n| Method   | Sketch               | Cartoon              | Art Painting         | Photo                | Avg.      |\n| -------- | -------------------- | -------------------- | -------------------- | -------------------- | --------- |\n| Baseline | 55.68 $\\pm$ 0.31     | 70.09 $\\pm$ 0.27     | 79.83 $\\pm$ 0.19     | 97.29 $\\pm$ 0.08     | 75.72     |\n| Ours     | **60.88 $\\pm$ 0.31** | **71.33 $\\pm$ 0.27** | **83.23 $\\pm$ 0.18** | **97.72 $\\pm$ 0.07** | **78.29** |\n\n​\tAs can be seen, our method also benefits the domain generalization task, which further verifies the effectiveness of our method.\n\n\n\nDataset information: Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. 2017 IEEE International Conference on Computer Vision (ICCV), pages 5543–5551, 2017. 6\n\n\n\n## 2. Computational cost\n\n​\tThe only computational cost imported by us is the domain token. As there are 64 classes in miniImageNet,  64*384=24.6k parameters are added in our experiments, which is not a heavy burden since ViT-Small contains around 22M parameters (i.e., only 0.11% parameters are added).\n\n\n\n## 3. Registers\n\n​\tIndeed, the training and location registers are similar to the CLS token, therefore they could show similar behavior as the CLS token. To validate this hypothesis, we add registers[6] to the backbone network and report the performance as well as the domain similarity.\n\n| Method                                | 5-way 5-shot Accuracy | Domain Similarity |\n| ------------------------------------- | --------------------- | ----------------- |\n| Baseline                              | 63.89                 | 0.076             |\n| Train w/ Register + Test w/ Register  | 63.17                 | 0.047             |\n| Train w/ Register + Test wo/ Register | 64.73                 | 0.101             |\n\n​\tAs can be seen, such a phenomenon also exists in the registers, which verifies our analysis and interpretation.\n\n​\tNotably, our paper differs from registers[6] in:\n\n​\t (1) [6] finds ViT needs registers because it helps to reduce the artifacts in the feature map, but we find the CLS token, although its location is similar to registers, naturally absorbs domain information, and further interprets the reason behind it, which is not discovered in [6]; \n\n​\t(2) We further propose a method for the CDFSL task to take advantage of the CLS token's characteristics, which is not included in [6].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OqRGwrRXcq", "reviewer": "Reviewer_PfB7", "summary": "Based on the observation that pretrained ViT models perform better on cross-domain few-shot tasks when the cls-token is re-initialized, the authors hypothesize that this is due to the absorption of domain-specific information and consequently propose a modified training and inference scheme to combat this.", "questions": "**Main concerns, questions & potential improvements:**\n\n**[Q1]** Statements in l27. as well as equation 1 define a fully supervised setup, which is arguably very common; \n  However, the authors then use a ViT pretrained using DINO – which is a self-supervised method and does create quite a different  representation space, which can have a significant impact on downstream performance (e.g. see recent in-domain FSL works like [A]);\nEven though is then then followed with further supervised training (which in itself could be seen as fine-tuning), this raises some questions!\n\n- Have the authors investigated their approach with a fully-supervised pretraining? If so, what is the result there? Do your observations still hold?    \n   $\\textrightarrow$ (In addition to consistency with the motivation/pre-lim, this could add important additional insights!)\n-  I’d also be curious how well the DINO pretrained method as well as a supervised ImageNet pretrained one would perform across Table 1, i.e. without the miniImageNet training.\n\n_([A] Hiller et al., Rethinking Generalization in Few-Shot Classification; NeurIPS 2022)_\n\n\n**[Q2]** Training a backbone that is already trained on ImageNet further on miniImageNet seems very odd, given that miniImageNet is essentially a smaller subset; \n\n -  What is the reasoning behind this?  \n  $\\textrightarrow$ If supervised information shall be included, why not simply use a fully-supervised backbone (see before) or fine-tune using entire ImageNet?   \n  $\\textrightarrow$ I would understand this scheme if meta-finetuning or another paradigm was used, but it is not as far as I can see from the paper?   \n\n**[Q3]** The authors evaluate ‘prototype methods’ in Table 1 (& l.75), but do not state how their prototype is formed. Note that various method exist, including using the refined cls-token, averaging the patch tokens, etc.  \n$\\textrightarrow$ Some specification/description here would help the reader.\n\n**[Q4]** I’d appreciate some more insights on the experiments presented in Fig 2(a) and Tab 1 around fixing the cls token to random init. \n-  How is the performance of this on the actual source training task, i.e. does the model still train well? (what’s the gap if there is any); \n-  Why do the authors think that this setup does actually perform that well, would you imagine the domain information is simply not learnt at all or rather absorbed into the other tokens? \n-  Does the model `fit’ to work with the specific random init? And what happens if you then change/re-initialize this cls-token at the downstream task – does it fall back to the ‘not load cls’ performance?   \n  $\\textrightarrow$  All these insights would provide the reader with a much better basis for interpretation of your findings!\n\n**[Q5]** What is the similarity in terms of retrieval (i.e. similarity map) like shown in Fig 2(b) when using a randomly initialized cls token on the downstream datasets? Better/worse/same than using the one(s) trained without your method?\n\n**[Q6]** Fig 3 and the analyses are not necessarily convincing in my opinion. The blurry regions might cover a much larger distribution which could be significantly easier to match to then a very high-frequency one, couldn't it?\n\n**[Q7]** The best choice turns out to be one cls token for each class, which is obviously valid – somewhat questions the `domain’ information though, as it seems to mainly be a class identifier then.    \nWhile the efficacy on the cross-domain tasks are still valid, the interpretation and therefore generalization to other source datasets could significantly differ. E.g. would I need 1000 tokens if I was to train on ImageNet? (implications would be important to state for potential follow-up work)\n\n---\n**Additional comments:**  \nPotential misunderstanding of the word `doubt’: \n-  l92. seems confusing, as you effectively state that the reader should doubt (i.e. question/not believe) that the cls token contains domain information – but this is pretty much the main motivation of the paper? \n-  Same in caption of Figure 2 (b), and some other places", "limitations": "Although the authors state in the checklist that limitations are addressed in the appendix, there is only one sentence that recognizes the limitation in terms of the used datasets; I’d like to see the authors properly discuss some potential limiting factors/considerations in terms of their actual algorithmic and architectural choice (e.g. pretraining influence; still a good choice even if domain gap is small(er)?; and others, if they are aware of any)\n\n---- \n---- \n## Post rebuttal update:\nMost of my concerns have been addressed by the authors -- see rebuttal and comments;  \n\n*I have increased my rating accordingly to weak accept to reflect the new insights & clarifications.*", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Originality & Significance:**  \n- Motivation of the method based on a clear observation and backed up by intuition as well as corresponding analysis\n- Very interesting observation that is then leveraged to design a novel approach that improves results across various datasets\n\n**Quality:**   \n- Good range of experiments presented that mostly support the underlying intuition and argument of the paper\n- Experiments conducted with a good selection of alternate variants to gauge performance improvements of the main contribution across multiple datasets\n\n**Clarity:**  \n- Contributions and underlying motivation clearly stated in intro \n- The paper is mostly well written and easy to read and follow", "weaknesses": "_TLDR; I do like the underlying idea, but have a range of questions and concerns that I’d like the authors to clarify & address._\n\n- Inconsistency in terms of stated pretraining on the source domain – raising some questions in terms of generalizability of results – please see question section below.\n- Unclear reasoning behind training setup – see below.\n- Unstated assumptions around prototypical method that make it hard to follow some of the technical setup and conclusions. See question section below.\n- Many of the presented results would benefit from background info/analysis, see below.\n- Some concerns around result interpretation and baseline comparison, see below.\n- Some minor inconsistencies in wording.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n\n## Q1. DINO pretraining\n\n​\tThe training paradigm of DINO pretraining follows current cross-domain few-shot learning (CDFSL) works [13,42]. \n\n​\t1. Why this setting?\n\n​\t\tCurrent works [A] have shown that unsupervised pretraining could show better generalization than supervised pretraining. Therefore, an unsupervised pretraining on ImageNet could help the model generalization, especially under large domain gaps. To verify it, we compare unsupervised pretraining and supervised pretraining below, and we can see the unsupervised ones show better target-domain performance.\n\n​\t2. What if we tune on miniImageNet the model with full supervision on ImageNet?\n\n​\t\tFor the ImageNet fully-supervised setting, both the training on ImageNet and miniImageNet use the supervised classification loss. However, since all images and labels in miniImageNet are covered by ImageNet, the tuning on miniImageNet will be difficult. Therefore, unsupervised pretraining will be more suitable.\n\n​\tTo verify our method also fits other pretraining paradigms, we conduct experiments on other pretraining methods as follows.\n\n​\t*Please see Tab.6 in the PDF for results.*\n\n​\tHere, the CLIP pretraining can be viewed as a fully supervised pretraining. Since the pretraining of CLIP is not fully overlapped by miniImageNet, it is more suitable for validating the full supervision setting. As can be seen, our method and analysis still hold in this setting, including the ImageNet supervised pretraining and DINO without miniImageNet training.\n\n\n\n## Q2. First training on ImageNet then training on miniImageNet\n\n​\tPlease refer to Q1 for why the model is first trained on ImageNet and then tuned on miniImageNet. We would like to point out that some works [14] also have shown that our baseline method shows advantages in cross-domain transferring, which is the reason why we chose this simple method as our baseline. To verify our model also suits the meta-learning-based baselines, we conduct experiments based on the ProtoNet [4].\n\n​\t*Please see Tab.4 in the PDF for results.*\n\n​\tWe can see that our model also improves this kind of baseline method.\n\n\n\n## Q3. Prototypes\n\n​\tWe would like to point out that the prototype is defined in L61-62, Eq.2. Briefly speaking, the ViT features extracted from samples in each class are averaged as the prototype for each class.\n\n\n\n## Q4. Fig.2a and Tab.1 around fixing the CLS token to random initialization\n\n​\t*Please see Q2 in the global response for the explanation of a.1, a.2, and a.3.*\n\n​\tIf we re-initialize the CLS token during the source-domain training, the improvements also exist.\n\n​\t*Please see Tab.3 in the PDF for results.*\n\n​\tHowever, re-initializing the CLS token can be viewed as adding noise to the domain token, therefore harming the absorbed domain information, which then affects the domain-irrelevant information learned by other structures in ViT. As a result, the Re-Init performance is slightly lower. \n\n​\tIndeed, domain tokens are encouraged to be orthogonal to the fixed CLS token, which would drive the model to view the fixed token as a domain-agnostic token. But note that **the random token is already agnostic enough to every domain even without training (as shown in Q5)**, therefore our training would not essentially drive the model to be more agnostic to that CLS token, i.e., our model is not bound to the specific value of the CLS token.\n\n\n\n## Q5. Similarity map in Fig.2b with randomly initialized CLS token\n\n​\t*Please see Fig.1 in the PDF for results.*\n\n​\tWe use the same color bar as in Fig.2b. We can see the similarity is much lower, but a coarse contour of objects can still be observed in both the source and target domains, indicating a **good transferability** of detecting object contours, because no domain information is in the random token. However, the contour detected by the random token is much **worse** than that by the CLS tokens as shown in Fig.2a and Fig.5a, indicating although the random CLS token can initially detect the object contour, the learning of the CLS token strengthens this characteristic to detect low-frequency images.\n\n\n\n## Q6. Fig.3 Analysis of low-frequency images\n\n​\tTo verify it is the CLS token that tends to capture low-frequency components in images, we use random tokens to calculate the similarity map of low-frequency images, and find random tokens do not show the same results as the CLS token.\n\n​\tSpecifically, we measure the activation ratio of the CLS token and the random token.\n\n​\t*Please see Tab.5 in the PDF for results.*\n\n​\tWe take the top 10% or 20% value as examples. We can see the random tokens show a tendency to decrease the activation ratio, while the CLS token shows a tendency to increase to ratio, indicating it is the CLS token that tends to be similar to the feature of low-frequency images.\n\n\n\n## Q7. One CLS token for each class?\n\n​\tPlease see Q1 in the global response.\n\n## Additional comments\n\n​\tHere \"doubt\" is to express \"think\", which is a misuse of words. We promise to carefully revise the words in the paper. \n\n\n\n## Limitations\n\n​\tThe limitation of the CLS token analysis is that it gets less effective when the domain gap is smaller. The datasets used in the paper show large domain gaps with the source domain, which makes it very challenging for knowledge transfer [14]. Therefore, the domain information absorbed in the source domain is harmful to the target domain. However, when the domain gap is smaller, the information captured by the CLS token could be partly transferred to the target domain, and our analysis and method may only get smaller improvements in this situation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cyb8ejwxck", "reviewer": "Reviewer_aYYr", "summary": "This paper explores an intriguing phenomenon in Cross-Domain Few-Shot Learning (CDFSL) using Vision Transformers (ViT). The authors observe that randomly initializing the CLS token, instead of using source-domain pre-trained parameters, consistently improves target-domain performance. They attribute this to the CLS token naturally absorbing domain-specific information due to ViT's inherent structure, which manifests as low-frequency components in the Fourier space of images. To address this, the authors propose a novel method that decouples domain information in the CLS token during source-domain training and adapts the CLS token for efficient few-shot learning in the target domain. This approach aims to enhance the transferability and generalization of ViT in CDFSL tasks. The effectiveness of the proposed method is validated through extensive experiments on four benchmarks, demonstrating state-of-the-art performance.", "questions": "1.\tPoor paper writing. Some descriptions in the article are not analyzed or cited. Please see the weakness raised above for details.\n2.\tThe comparison method is not sufficient.\n3.\tThe research on related work is incomplete.\n4.\tPlease answer the above questions.", "limitations": "In the paper authors mention that “We discuss the limitations of the work in the appendix”. However, there’s no limitation discussion in the appendix. This work does not contain any negative social impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper describes and analyzes the impact of CLS on CDFSL performance in detail. This is an aspect that is easily overlooked. The authors analyze the impact of CLS and propose solutions.\n2.\tThe method achieved SOTA performance.", "weaknesses": "1.\tIn line 39-40 “During the target-domain learning, this method finetunes the CLS token to efficiently absorb domain information, handling the few-shot learning problem.”, the few-shot learning (FSL) problem cannot be solved through “efficiently absorb domain information”. Normally we solve FSL by addressing the overfitting due to the limited labeled target data.\n2.\tHow did the authors get the results of Tab. 1? Because the performance increase is not over 1%, it’s better to show how the results are obtained. It's better to follow what the existing methods do, take the average of 600 results.\n3.\tProviding theoretical support for this paper will make the paper more credible and sufficient. Such as in line 84-88 “we directly fix the CLS token as random initialization for both the source and target domains (Tab. 1 a.2). We can see that by abandoning the learning of the CLS token, the performance is also improved from the baseline method, but is slightly lower than training but not loading it (Tab. 1 a.3). This means such information in the CLS token could be beneficial for the source-domain training.”, why a.3 is better than a.2 in Tab.1? It will be better if authors provide the corresponding theoretical analysis.\n4.\tIn line 91-92 “Intuitively, since not loading the CLS token improves performances only under cross-domain scenarios, it is natural to doubt the CLS token’s poisonous information as the domain information.”, how authors obtain this conclusion (only under cross-domain scenarios)? Authors should compare the FSL performance changes between cross-domain and in-domain scenarios.\n5.\tIn line 98-99 “The larger the CKA similarity is, the smaller the domain distance will be, and it means the model contains less domain information.”, please explain why “it means the model contains less domain information”?\n6.\tIn line 100-101 “Not loading the CLS token can significantly increase the CKA similarity, indicating the CLS token contains domain information while other structures tend to capture domain-irrelevant information.”, how authors obtain the conclusion that “other structures tend to capture domain-irrelevant information”?\n7.\tIn Fig.3 (b), the similarity is between source and target domain data?\n8.\tThe CLS token is fixed as the random initialization in the source domain stage and fine-tuned in the target domain stage. It means that this CLS token does not work in the source domain stage, then why do you introduce this token in this stage? Why not only introduce and train the domain tokens in the source domain stage, and introduce the CLS token in the target domain stage? Seems introducing CLS token in the source domain seems unnecessary.\n9.\tIn line 191-193 “Interestingly, we find treating each class as a pseudo domain could achieve the best performance”, it means that every pseudo domain indicates a class. Therefore, how authors explain the domain tokens learned the domain-specific but not class-specific information?\n10.\tThe comparison method is not sufficient. Please compare with more existing SOTAs.\n11.\tThere is no explanation about Tab.4 (f). What it means?\n12.\tThe research on related work is incomplete. In recent years, there are many CDFSL papers, such as \"Deep Learning for Cross-Domain Few-Shot Visual Recognition: A Survey\", \"Free-lunch for cross-domain few-shot learning: Style-aware episodic training with robust contrastive learning\" \", \"Meta-fdmixup: Cross-domain few-shot learning guided by labeled target data\", and \"Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learning\", etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n\n## W1. Handling few-shot learning by absorbing domain information\n\n​\tWe would like to point out that for the cross-domain few-shot learning (CDFSL) problem, one of the most important challenges is the domain gap between the source and the target domain. Therefore, an important task for the target-domain few-shot finetuning is to effectively adapt to the target domain. This is also a challenge due to the scarce training data, as domain information cannot be fully represented by training samples, i.e., the model could overfit to the target-domain training data instead of learning the target domain information, which is well handled by our analysis and method.\n\n\n\n## W2. The confidence interval of Tab.1\n\n​\tWe also follow current works (e.g., [42]) to evaluate each model on thousands of episodes, as shown in Tab.4 and Tab.5 where the confidence intervals are included. Due to space limitation, the confidence intervals are abbreviated in Tab.1, and here we supplement them as follows.\n\n​\t*Please see Tab.8 in the PDF for results.*\n\n\n\n## W3. Why a.3 is better than a.2 in Tab.1\n\n​\t*Please see Q2 in the global response.*\n\n\n\n## W4. Comparison with in-domain performance to show domain information\n\n​\tWe would like to point out that the in-domain performance is already included in Fig.1b. Specifically, loading the CLS token leads to the 5-way 5-shot accuracy of 92.8%, while not loading it gives a lower accuracy of 90.6%, which is different from that of target domains. That is the reason why we say \"We think the CLS token contains the domain information\".\n\n\n\n## W5. Why \"the larger the domain similarity, the less the domain information\"\n\n​\tFollowing current works (e.g., [26]), the domain similarity is measured by comparing the distance between two batches of images, where each batch is sampled from a single domain. We follow [7] to take CKA as the similarity function. The domain information will make the model overfit to the given domain. Suppose a model is completely overfitted to one domain, given other domains' images, the extracted features could be just random noises, therefore the domain similarity will be downgraded to 0. Therefore, following current works (e.g., [26]), we hold that the larger domain similarity indicates the less domain information. \n\n\n\n## W6. Why do other structures tend to capture domain-irrelevant information?\n\n​\tBased on our experiments in Fig.2a, we can see that by randomizing the CLS token, the domain similarity increases significantly, which means other structures cannot easily extract domain-specific features without the help of the CLS token. This result indicates other structures tend to be more domain-agnostic than the CLS token. \n\n\n\n## W7. Similarity in Fig.3b\n\n​\tThe similarity is calculated as the cosine similarity between the CLS token and the input patch tokens of the first block, which is the same as Fig.2b (L108).\n\n\n\n## W8. Why introduce CLS token during the source-domain stage\n\n​\tThe CLS token fixed as random initialization can be viewed as **a placeholder for the downstream few-shot finetuning**. \n\n​\tDuring the source-domain stage, the domain tokens are added to the fixed CLS token to be fed into ViT. Therefore, the domain token is encouraged to learn domain information, while other structures and parameters are encouraged to learn domain-irrelevant information.  Since the CLS token is fixed as random initialization, it would also be domain-irrelevant, which is therefore suitable for the downstream target-domain finetuning.\n\n​\tDuring the target-domain stage, the domain token is then abandoned, and the remaining structures and parameters (i.e., the random CLS token and other parameters) are ideally domain-irrelevant, as validated in Fig.2a. Then, we un-fix the CLS token and finetune it to learn the target-domain information. Therefore, the domain-irrelevant CLS token will be specific to the target domain, and we also validated the effectiveness of this finetuning in Tab.5.\n\n\n\n## W9. \"Domain-specific\" vs. \"class-specific\"\n\n​\t*Please see Q1 in the global response.*\n\n\n\n## W10. Comparison with more SoTAs\n\n​\tWe list more comparisons with state-of-the-art works as follows, where we can see our method achieves the best performance.\n\n​\t*Please see Tab.1 and 2 in the PDF for results.*\n\n\n\n## W11. Tab.4f\n\n​\tTab. 4f means to add the domain token as appended tokens to the ViT's first block as prompts, instead of adding domain tokens to the CLS token. As can be seen, the performance is much lower, verifying our design of the model is better.\n\n\n\n## W12. Related work\n\n​\t*Please see Q3 in the global response.*\n\n\n\n## Other questions\n\nQ1: Please refer to the above responses.\n\nQ2: Please refer to W10.\n\nQ3: Please refer to W12.\n\nQ4. Please refer to the above responses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zu6coWyWyF", "reviewer": "Reviewer_dboq", "summary": "In this paper, the authors found a new phenomenon that the CLS token used in Vision Transformer (ViT) absorbs the domain information in Cross-Domain Few-Shot Learning (CDFSL). On the basis of the findings, they proposed a novel CDFSL method that updates only the CLS token during the target training. A comprehensive analysis provided verification of the findings, and comparative experiments confirmed the validity of the proposed method.", "questions": "+ The findings of this paper are very similar to the paper “Vision Transformer Needs Register” [6]. Is it possible to apply this method directly to CDFSL for comparison?", "limitations": "The limitation is well stated in the manuscript, and there does not seem to be anything additional mentioned.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The paper is well organized and it is easy to follow the contents.\n+ The findings that the CLS token absorbs the domain information in CDFSL sound new and worthwhile, and they will facilitate many future studies.\n+ The findings on the CLS token are supported by a comprehensive analysis, so I feel this paper is highly reliable.\n+ This paper has enough technical novelties that it proposed a novel CDFSL method based on the findings and confirmed the validity of the method.", "weaknesses": "+ There seemed to be a lack of validation using models other than DINO.\n  + Although the results of the validation using iBOT are reported in the Appendix, the improvement of the proposed method is not significant. I think this result alone does not sufficiently support that the findings can be broadly applicable to the CLS token in Vision Transformer in general.\n  + I consider that they should conduct an analysis on CLIP-pretrained Vision Transformer, if possible. Since CLIP is one of the most representative pre-trained computer vision models today, I believe it is crucial to verify whether the findings of this study are applicable.\n+ The performance difference when compared to existing methods does not appear to be very large. In some benchmarks, the difference is less than 1%. \n  + In particular, the ablation study results also appeared to show little improvement in ChestX. Is there any possible explanation for this result?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work!\n\n## W1. Verification of other pre-trained model\n\n​\tDue to time limitations, we did not fully tune the model in our appendix submission. Here we report the fully-tuned performance on the iBot and ViT-B model.\n\n| iBot | Crop.     | Euro.     | ISIC      | Ches.     | Avg.      |\n| ---- | --------- | --------- | --------- | --------- | --------- |\n| BL   | 81.17     | 72.71     | 31.44     | 22.56     | 51.97     |\n| Ours | **82.47** | **73.83** | **32.87** | **22.88** | **53.01** |\n\n| ViT-B | Crop.     | Euro.     | ISIC      | Ches.     | Avg.      |\n| ----- | --------- | --------- | --------- | --------- | --------- |\n| BL    | 82.97     | 72.06     | 34.19     | 22.60     | 52.95     |\n| Ours  | **83.43** | **74.42** | **35.78** | **22.98** | **54.15** |\n\n​\tAs can be seen, the improvements are clearer than in the appendix, which further verifies the generalizability of our method.\n\n​\tWe also train our model with the CLIP pretraining.\n\n| CLIP | Crop.     | Euro.     | ISIC      | Ches.     | Avg.      |\n| ---- | --------- | --------- | --------- | --------- | --------- |\n| BL   | 77.33     | 63.30     | 29.84     | 21.11     | 47.89     |\n| Ours | **78.55** | **64.88** | **30.95** | **22.25** | **49.16** |\n\n\n\n## W2. Why ChestX performance is lower\n\n​\tWe would like to point out that all current works show a low performance on the ChestX dataset, where we have already achieved the top performance. This dataset is difficult in two aspects: (1) the domain gap between it and the source domain is large among all target datasets, as validated in Fig.2a that the domain similarity is low; (2) the semantic shift is much larger, as ChestX is a fine-grained classification task with expert knowledge [14]. That is, even an untrained human can hardly distinguish different chest diseases in the X-ray image, which means much less prior knowledge can be transferred from the source datasets [14]. As a result, in current works [10, 49], the performance improvement on ChestX is always less than 1%.\n\n​\tHowever, given the difficulty in ChestX, we have still achieved the best performance in our work, which still demonstrates the effectiveness of our analysis and method.\n\n\n\n## Q1. Registers\n\n​\tNotably, our paper differs from registers [6] in:\n\n​\t (1) [6] finds ViT needs registers because it helps to reduce the artifacts in the feature map, but we find the CLS token, although its location is similar to registers, naturally absorbs domain information, and further interprets the reason behind it, which is not discovered in [6]; \n\n​\t(2) We further propose a method for the CDFSL task to take advantage of the CLS token's characteristics, which is not included in [6].\n\n​\tIndeed, the training and location registers are similar to the CLS token, therefore they could show similar behavior as the CLS token. To validate this hypothesis, we add registers[6] to the backbone network and report the performance as well as the domain similarity.\n\n| Method                                | 5-way 5-shot Accuracy | Domain Similarity |\n| ------------------------------------- | --------------------- | ----------------- |\n| Baseline                              | 63.89                 | 0.076             |\n| Train w/ Register + Test w/ Register  | 63.17                 | 0.047             |\n| Train w/ Register + Test wo/ Register | 64.73                 | 0.101             |\n| Ours                                  | **66.10**             | **0.655**         |\n\n​\tAs can be seen, such a phenomenon also exists in the registers, which verifies our analysis and interpretation. However, the improved performance is much lower than ours, which verifies our contribution.\n​", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable input.

## Q1. One CLS token for each class?

​	Since the source dataset (miniImageNet) is a general classification dataset, the difference between each class is larger (e.g., than fine-grained datasets where domain information is clear). Therefore, for miniImageNet, it is reasonable to view each class as a domain. 

​	To further ablate "domain-specific" and "class-specific", we then manually construct some new source domains based on miniImageNet. Specifically, we take the amplitude (by Fourier transformation) from target domains as the style information, and use the phase  (by Fourier transformation) from the original source-domain images as the content information, thereby constructing 4 new domains with the original 64 source-domain classes. Then, we train our model on a new dataset containing the 4 constructed datasets and the original source dataset, and ablate different choice of domain tokens.

​	*Please see Tab.7 in the PDF for results.*

​	As can be seen, by introducing larger domain gaps, viewing each class as a domain is not the best choice. Instead, setting a domain token for each domain could achieve the best performance, which validates the rationale of the domain token in absorbing the domain information. 

​	For the ImageNet training, since we can obtain the hierarchical structure of all classes (e.g., superclasses like animals, plants, ships, etc.) by class names, we do not need to assign 1k domain tokens for each class. Instead, we only need to assign each superclass with a domain token, which is affordable.



## Q2. Tab.1 a.1, a.2, and a.3

For the 5-shot task, the source-domain accuracy of the baseline method (a.1) is 97.94, for the "fix as random initialization" (a.2) is 97.50, and for the "not loading CLS" (a.3) is 96.64.

​	Tab.1 a.2 means to fix the CLS token as random initialization during both the training and testing. 

​	For the target-domain performance, since the fixed CLS token does not contain or learn any information, the capability of the CLS token is abandoned. Therefore, other structures in ViT need to take the place of the CLS token to learn the information that should originally be captured by the CLS token. However, **since other structures in ViT are not as capable of learning domain information as the CLS token, the domain information is not effectively captured**. Since the domain information is harmful to the target domain, a.2 is better than a.1. However, since the domain information is beneficial for the source domain, **a.3 is better than a.2 in the source-domain training**. Given that a.3 also randomizes the CLS token during the target-domain stage like a.2, a.3's final performance is therefore better than a.2.

​	For the source-domain performance, a.2 forces other structures in ViT to absorb source-domain information, which is not as capable of learning source-domain information as the CLS token, therefore its source-domain accuracy is lower than a.1. For a.3, as it also absorbs domain information by the CLS token, other structures in ViT tend to absorb the domain-irrelevant information. Therefore, by randomizing the CLS token in a.3, **the remaining structures have less source-domain information**, thereby showing the lowest source-domain performance.



## Q3. Related work

We provide an extended related work of CDFSL as follows.



​	Cross-Domain Few-Shot Learning (CDFSL) [14, 20, 26, 30, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54] focuses on training a model on the source domain that can generalize well to target domain with limited examples. Current methods can be grouped into two types: meta-learning-based approaches [12, 14, 17, 33, 44, 45, 46] and transfer-learning-based ones [4, 14, 19, 40, 42, 47, 48, 49, 50, 51, 52, 53, 54].  Meta-learning-based approaches aim at learning task-agnostic knowledge to learn new tasks efficiently [14], differing in their way of learning the parameter of the initial model on the base class data. MAML [45] aims at learning an initial parameter that can quickly adapt to new tasks, while FWT uses a feature-wise transformation to learn representations with improved ability to generalization. An alternative way to tackle the problem is transfer-learning-based approaches, tackling the problem based on reusing the model trained on the base class data in a standard supervised learning way [46]. Among these approaches, LRP [47] aims to use the explanation results to guide the learning process. STARTUP [48], and Meta-FDMixup [49] mainly aim at defining relaxed settings for CD-FSL. Wave-SAN [51] tackles CD-FSL by spanning distributions of source styles. SET-RCL [51] simulates the style distributions of unknown target domains. IM-DCL [52] sets the entire feature as positive and negative sets to learn the query set without accessing the source domain. However, these works are mostly restricted to the CNN architecture. Recently some works have focused on the transformer structure to solve the CDFSL tasks but these efforts have not fully dug out the potential of the VIT structure and the importance of the CLS token on CDFSL. 



Extended References:

[44] Cross-domain few-shot classification via learned feature-wise transformation

[45] Rapid learning or feature reuse towards understanding the effectiveness of maml.

[46] Deep learning for cross-domain few-shot visual recognition: A survey.

[47] Explanation-guided training for cross-domain few-shot classification.

[48] Self-training for few-shot transfer across extreme task differences.

[49] Meta-fdmixup: Cross-domain few-shot learning guided by labeled target data.

[50] Free-lunch for cross-domain few-shot learning: Style-aware episodic training with robust contrastive learning.

[51] Wave-san: Wavelet-based style augmentation network for cross-domain few-shot learning.

[52] Enhancing information maximization with distance-aware contrastive learning for source-free cross-domain few-shot learning.

Meta-review: This paper examines the hypothesis that the CLS token of the Vision Transformer absorbs domain information and proposes a new cross-domain few-shot learning (CDFSL) method utilizing the CLS token based on this insight. No prior research has thoroughly investigated and discussed the role of CLS in domain adaptation, making these findings a significant contribution to the field. Most reviewers acknowledged the strong motivation and focus of the study, the detailed analysis, and the clarity of the writing. The effectiveness of the proposed CDFSL method across multiple datasets was also highly appreciated. The authors' rebuttal effectively addressed the reviewers' concerns, resulting in an improved rating from two of the reviewers. Two reviewers noted that the writing of the paper needs improvement, and their comments are expected to be reflected in the final manuscript.
Decision: Accept (poster)