Attention Temperature Matters in ViT-Based Cross-Domain Few-Shot Learning
Authors: Yixiong Zou
Ran Ma
Yuhua Li
Ruixuan Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Cross-domain few-shot learning
Vision transformer
attention temperature
Primary area: machine_vision
TL;DR: We find a phenomenon for ViT-based CDFSL: multiplying a small temperature (even 0) to ViT's attention map can consistently improve performance. We delve into this phenomenon for an interpretation and propose an effective method for CDFSL.

Abstract: Cross-domain few-shot learning (CDFSL) is proposed to transfer knowledge from large-scale source-domain datasets to downstream target-domain datasets with only a few training samples. However, Vision Transformer (ViT), as a strong backbone network to achieve many top performances, is still under-explored in the CDFSL task in its transferability against large domain gaps. In this paper, we find an interesting phenomenon of ViT in the CDFSL task: by simply multiplying a temperature (even as small as 0) to the attention in ViT blocks, the target-domain performance consistently increases, even though the attention map is downgraded to a uniform map. In this paper, we delve into this phenomenon for an interpretation. Through experiments, we interpret this phenomenon as a remedy for the ineffective target-domain attention caused by the query-key attention mechanism under large domain gaps. Based on it, we further propose a simple but effective method for the CDFSL task to boost ViT's transferability by resisting the learning of query-key parameters and encouraging that of non-query-key ones. Experiments on four CDFSL datasets validate the rationale of our interpretation and method, showing we can consistently outperform state-of-the-art methods. Our codes are available at https://github.com/Zoilsen/Attn_Temp_CDFSL.

Attention Temperature Matters in ViT-Based Cross-Domain Few-Shot Learning
Ran Ma Yuhua Li School of Computer Science and Technology, Huazhong University of Science and Technology (yixiongz, ranma, idcliyuhua, rxli)Chust. edu.
Yixiong Zou
Ruixuan Li"
Abstract
Cross-domain few-shot learning (CDFSL) is proposed to transfer knowledge from large-scale source-domain datasets to downstream target-domain datasets with only few training samples. However, Vision Transformer (ViT), strong backbone network to achieve many top performances, is still under-explored in the CDFSL task in its transferability against large domain gaps. In this paper, we find interesting phenomenon of VIT in the CDFSL task: by simply multiplying a temperature (even as small as 0) to the attention in ViT blocks, the target-domain performance consistently increases, to a uniform map. this paper, we delve into this phenomenon for an interpretation. Through experiments, interpret this phenomenon as remedy for the ineffective target-domain attention caused by the query-key attention mechanism under large domain gaps. Based on
though the attention map is downgraded
further propose simple but effective method for the CDFSL task to boost VIT's transferability by resisting the learning query-key parameters and encouraging that of non-query-key ones. Experiments four CDFSL datasets validate the rationale of our interpretation and method, showing we can consistently outperform state-of-the-art methods. Our codes available at http:s//gthabbccom/ooiiss/ttttt Temp_ CDFSL.
Introduction
Deep networks have shown great power in learning from large-scale datasets [18] 11
However,
collecting sufficient training data for every domain is always challenging, which gives rise to the Cross- Domain Few-Shot Learning (CDFSL) task. CDFSL requires model be firstly trained on a large- scale pretraining dataset (source domain, general dataset, e.g., ImageNet (6]). and then transferred to downstream datasets (target domain, kppor--kowwedge dataset, e.g., medical dataset [5] 44 where only few training samples are available. Typically, large domain gaps exist between the source and target dataset, making the transferring and downstream learning difficult 2
[5].
Vision Transformer (ViT) 8 as a prevailing kind of deep network, has achieved top performances in many computer vision tasks 47 37] However, only few works [14
10 53 studied ViT's
transferability against large domain gaps for the CDFSL task. In this paper, we find an intriguing phenomenon for the ViT-based CDFSL task: multiplying temperature parameter to the attention map of ViT blocks, the downstream target-domain few-shot performance consistently increases when 1.0 even close
albeit the attention map is downgraded uniform map (Fig. [
In this paper, we delve into this phenomenon for an interpretation. find the query-key attention mechanism in the ViT network demonstrates high discriminability but low transferability, which makes the target-domain attention ineffective, and the temperature adjustment is interpreted as a remedy for the ineffective target-domain attention. Moreover, find the non-query-Ken key structures
"ooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

916
9422
454
Makil
63.4
Cmpp
51.2
E2ro
Chos
93.0
$3.0
940
15.4
44.6
254
VIT illostration
5-way S-shot Accuracy vs. Allention Temperature
Figure I: (a) Vision Transformer (ViT) is composed of multiple blocks of the MLP and attention networks. (b) We train a ViT model on minilmageNee (source dataset), and conduct prototype-based classification target datasets. By multiplying a temperature (even small 0) the attention in ViT blocks (e.g., the last block) during the target-dataset classification, we find the accuracy consistently increases on most target datasets (Crop. Euro. ISIC. and Ches., even with a uniform attention map) while that of the source dataset drops. In this paper, we delve into this phenomenon for an interpretation, and propose a simple but effective method for the CDFSL task based on
in ViT show complementary characteristics against the query-key parts, i.e., higher transferability but lower discriminability. Based on these interpretations and findings, we propose a method for the CDFSL task to encourage the learning of non-query-key parameters and resist that of query-key ones, so to improve the transferability of ViT against large domain gaps.
In summary, our contribution can be listed as
To the best of our knowledge, we are the first to unveil the importance of the attention temperature in VaT-based CDFSL methods.
Through experiments, we find the query-key attention mechanism shows limited transferability against large domain gaps, which causes ineffective target-domain attention and needs to be remedied by the temperature adjustment.
Based on it, propose method for the CDFSL task to boost VIT's transferability by resisting the learning of query-key parameters and encouraging that of the non-query-key ones.
Extensive experiments validate the rationale of our interpretation and method, and show that we can consistently outperform oftt--of-theaar works.
2 Delve into Attention in ViT-based Cross-Domain Few-Shot Learning
2.1 Preliminaries
Cross-Domain Few-Shot Learning (CDFSL) requires the model to first learn from a general dataset (i.e., source dataset) containing suffieient training samples, and then transfer to downstream target datasets where only scarce training data is available. We denote the source dataset as DS {x- "/ N where x and y represent the ith training sample and its label, respectively. Similarly, the target dataset termed as D {>] y/ comparison, current works
" - During the learning and testing on DT, for the fair 12 adopt a k-way n-shot paradigm to sample from D construct small datasets (i.c., episodes) consisting of & classes and n training samples in each class. Based on episodes, the model learns from these k samples (a.k.a. support set, {æaaj evaluated on testing samples from these l classes (a.k.a. query set, (2:)).
k.n V= and is
Vision Transformer (ViT) has recently been popular in vision tasks, which divides images into tokens and captures the image patterns through relf-attentions As shown in Fig. [ ViT is composed of stacked blocks containing an attention network and an MLP network, which can be represented as
f(z M (A(M( (IE(()))...)))
(I)
where (-) denotes the MLP network, A(-) denotes the self-attention network, and E(-) denotes the
embedding layer. In this paper, we focus on the ViT-based CDFSL method to study its downstream
generalization. Specifically, we follow (10) to initialize ViT on ImageNet [6] by DINO ( (see appendix for other settings), and train ViT on S by the cross-entropy loss with a fully-connected (FC) layer as
L = Lote (os(((((())))
(2)

---

CLS Token Aliantin Valae vs Block
minilmageNet
CropDiseases
wwggela Tagen
EuroSAT
Image Attention Densiry vs. Block
ISIC2018
ChestX
Chost
Tinget
(a) Attention Map Visualization
(b) Quantitative Measure Attention
Figure 2: (a) Visualization of VIT attention. Although ViT behaves well on the source domain (minilmageNet), on target domains ViT tends to (1) wrongly focus on the CLS token while omitting the input images, and (2) dispersively focus on large range of noisy image regions. (b) We quantitatively plot (top) the attention value the CLS token and (bottom) the density of attention maps on image tokens for all images in each domain. The black curve of minilmageNet is always below other curves of target domains, verifying the ineffectiveness of target-domain attention. Therefore, we interpret the temperature adjustment remedy for the ineffective target-domain attention.
where ob()) denotes the FC-based classifier and f(-) denotes ViT. Finally, we utilize the ProtoNet 33 with the distance function d(,; -) for the target domain few-shot learning as
04 = arg min d( 1 Σ (1rs)) ((zg))
"
In this section, follow 12 to take minilmageNeet 440 the source dataset, and take CropDis-
eases (30, EuroSAT 13, ISIC2018 [5] and ChestX 44
as target datasets.
2.2
Interpretation: Attention Temperature Remedies Target-Domain Attentions
2.2.1 Intuitive Observation of Ineffective Target-Domain Attentions
To study why the temperature-baset attention adjustment improves target datasets but harms the source dataset, we first visualize the attention map on each domain in Fig. 2h. As the backbone network utilizes the CLS token feature from the last VIT block the final feature, we take the attention map w.r.t. the CLS token in the last ViT block for visualization, where a large attention value indicates the color of red and small value refers blue. We plot the CLS token's attention value in the left-top of the map. We can see the source-domain-trainee ViT shows a good capability of discovering meaningful objects in the source dataset (minilmageNet)), but it always shows wrong attention on target datasets, which is represented in two aspects:
(L) It tends to excessively focus the CLS token and ignores all image tokens, the left-top patches are red but the image heatmap is blue for the first two columns. (2) For its focus on images, it tends to focus on large range of noisy regions instead of meaningful objects (the third column).
Ideally, if there no domain gap between the source and target domains, the target-domain attention should perform like the source-domain attention to focus on meaningful regions in the image. Therefore, these phenomena indicate that the attention network performs poorly on target domains.
2.2.2 Quantitative Verification of Target-Domain Attentions' Ineffectiveness
To verify this observation, we quantitatively measure (I) the attention value on the CLS token and (2) the sparsity of the attention on image tokens, on all images from different datasets and blocks. The attention value on the CLS token is measured as
((A)
Σ r(As.,.))))
(4)
b Y& Th
C.J.k
where xbxxaxXxxx the attention map. is the batch size, ny is the number of heads, ny the number of tokens and r(-) denotes the L no-oommalization Similarly, The sparsity of the image
3

---

Table 1: Ablation of the attention network from ViT's last block.
Method
mageNee
CropDiseases
EuroSAT
ISIC2018
ChestX
Average
Input Tokens
90.17
79.63
73.12
32.81
22.41
$1.99
Input Tokens SA
92.59
79.10
73.17
32.54
22.47
51.82
Input Tokens Identity SA
87,45
77.97
69.89
32.15
22.52
50.63
Input Tokens Cosine SA
88.80
79.98
74.35
32.65
22.57
$2.39
Input Tokens Average SA
89.53
80.73
74.59
32.04
22.64
$2.50
Table 2: Domain similarity w.r.t. ablated attention modules.
Method
minilmageNet
CropDiseases
EuroSAT
ISIC2018
ChestX
Average
Input Tokens
1.0
0.4569
0.4381
0.3608
0.3900
0.4115
Input Tokens SA
1.0
0,1853
0.1829
0.1344
0.1998
0.1756
Input Tokens Identity SA
1.0
0,5857
0,5873
0.5376
0.4836
0,5486
Input Tokens Cosine SA
1.0
0,2692
0.2252
0.1616
0.2295
0.2214
Input Tokens Average SA
1.0
0,2235
0,2226
0.1580
0.2002
0.2011
attention is measured by the averaged Ly-normalized L1 norm [54] of the attention map as
1
(orm(A) = b T& Th 1.J.k
Σ E
Results are plotted in Fig. b. As the smaller the L norm is, the sparser (i.e., less dense) the attention density the Y-axis to align with the L1 norm value. We can
would be, in Fig. 2p (bottom), see that curves of the source dataset (minilmageNNtt) are always located under those of target datasets in Fig. 2p. indicating the model averagely pays more attention to the CLS token or noisy image regions on target datasets, quantitatively validating the ineffectiveness of target-domain attention.
Therefore, we interpret the temperature adjustment as a remedy for ineffective target- domain attention: we apply a small temperature to the attention map, the attention map will be smoothed. For example, given the smallest temperature O, the attention map will be downgraded to a uniform map. Such smoothing operation would remedy the wrong attention on all tokens (i.e., CLSS token and image tokens), because uniform attention is at least better than wrong attention (e.g., uniform attention allows the model to collect information from image tokens, but the wrong attention restricts the model to only collect information from the CLS token). These also interpret why the temperature remedy only improves on target datasets: the source dataset is effective in finding meaningful regions, indicating good attention map that does not need to be remedied.
2.3 Why do attention networks get ineffective on target domains?
Then, we take the attention network in the last ViT block for ablation, and study the performance (5-way 1-shot accuracy, Tab. [ and domain similarity (Tab. 2 induced by these modules. The domain similarity is measured by the CKA similarity [17 23), where we extract features from images of different domains, and measure the CKA similarity between the source and each target dataset by aligning the channel dimension. Large domain similarities indicate less domain information contained. Since trade-off exists between discriminability and transferability [19 27 46), we utilize the source- source-domain (miniimageNet) accuracy in Tab. to measure the discriminability and use the average domain similarity in Tab. to measure the transferability, and the average target-domain accuracy in Tab. iis the result caused by the trade-off.
In Tab. 2 Input Tokens denote features input into the attention network (i.e., the output of the 11th VIT block), and SA (self-attention) denotes the output of the attention network (i.c., the input of the MLP network in the 12th VIT block), which is the default ViT self-attention module. By appending SA to Input Tokens, in Tab. I the source-domain accuracy increases. Accordingly, in Tab. the CKA similarity significantly drops, indicating SA is more the side of discriminability than transferability, which therefore makes target- accuracies decrease.
Then, we study the token relation modeled by the self-attention mechanism. We replace the default query-key relation with three simpler relations: the identity relation (tokens are not merged), the cosine relation (tokens are merged by the cosine similarity with softmax), and the average relation (tokens are merged uniformly, equivalent to the temperature set to 0). Firstly, we can see the identity SA shows low source and target accuracy but high domain similarity, indicating the merge of tokens

---

CLS Tokeo Value vs. Block
miniImageNet
CropDiseases
-a
EurOAAT
Image Activation Density va. Block
ISIC2018
ChestX
Activation Map Visualization
(b) Quantitative Measure of AcAivvation
Figure 3: (a) We visualize the activation maps of Input Tokens Tab. and find that although the activation maps are not perfect, they show (1) no exceeding activations on the CLS token, and (2) rough object contours in input images. (b) We further find CLS token activation values as well as the map density on image tokens are con-distinguishable between source and target domains, indicating a better transferability of representations against large domain gaps. These indicate the Input Token Tab. tends be more transferable than the query-key attention.
is crucial for discriminability. Therefore, target-domain attention majorly focuses on the CLS token (Fig. 2 the merge of image tokens would be abandoned, leading to downgraded performance.
By substituting the Identity SA with the Cosine and the Average SA, the performance improves on all datasets, which is lower on the source dataset but much higher on target datasets than the default SA. Also, the domain similarity higher than the default SA but lower than the Identity SA, indicating:
(1) The modeling of token relations contains domain information and discriminability itself, therefore it contributes the source domain and reduces the domain similarity. The increase in target-domain accuracies indicates the gain of discriminability is larger than the loss of transfcrability.
(2) The default query-key relation contains the most domain information and discriminability. which increases the source-domain performance the most and decreases the domain similarity the most. Therefore, the query-key relation even harms the target-domain performance.
As result, we can conclude that ineffective target-domain attention majorly caused by the self- attention mechanism in the attention network. Given the trade-off between discriminability and transferability, the modeling of token relations is not entirely harmful target-domain generalization. However, the query-key attention mechanism tends to be much less transferable, and some related works (36) have partly shown the tendency overfitting in this mechanism, therefore may harm the generalization target domains and lead to ineffective target-domain attention.
2.4 Handle the Ineffective Target-Domain Attention
Besides the query-key attention (SA) in Tab. D and 2 we can also find the rows of Input Tokens and Average SA demonstrate higher domain similarity and good target-domait performance, albeit the limited source-domain performance. This indicates these features. compared with the features the query-key attention, tend to be on the transferability side in the tradc-off between discriminability and transferability, since the query-key attention contains more learnable parameters.
To verify this hypothesis, we also visualize the activation map the Input Tokens of Tab. in Fig. 3h. We observe (1) no exceeding activation the CLS token, and (2) the activation maps can indeed reflect rough object contours in images from different domains, although not perfect.
Moreover, we measure the image token density and the CLS token value these features. Sim- ilar to norm (A) and ((A). these two criteria are written as norm = Σ, (r(())n4) and ((M)
E; r(M) where M E Rbxaaxe is the MLP outputs and r(M) M. /Il Σ M.,.Rll xx is the Inznormalized average activation of each token. The results are plotted in Fig. 3b. and we can see the source and target datasets are non-distinguishable on these two criteria, indicating better transfcrability and adcquate discriminability.
5

---

CTS Taken
Attention
MT.
Alteation
MatMul
Softmx
MLP
Image Tobra
T
Attcation
Attcution
:
Figure 4: During the source-domain training, we propose to randomly abandon the attention network by multiplying temperature of the attention in each block respectively, which resists the learning of the query-key attention parameters and enhances the non-query-Ker parts.
Therefore, we conclude that the query-key features tend be discriminative but less transferable, while the non-query-key features tend to be transferable but less discriminative. This inspires us to encourage the learning of the non-query-key parameters in VIT (since their ability to be discriminative and overfitting is limited by parameters), and resist the learning of the query-key parts (as they are discriminative enough after the pretraining, so the source-domain training may be needed).
2.5 Conclusion and Discussion
The self-attention mechanism is the design of ViT, where the query-key attention mechanism is capable of fitting large-scale training data. However, such characteristic also limits the transferability against large domain gaps, leaving the target-domain attention ineffective. Therefore, remedy of the attention map by the temperature adjustment is needed, as a uniform attention map is at least better than wrong attention map. Compared with the query-key attention, the non-query-ker components in VIT tend to be more transferable but less discriminative than the query-key components, which inspires improve the generalization of VIT by encouraging the learning of non-query-key parts and resisting the learning of query-key ones during the source-domain training.
3 Method
Based on the above analysis. we further propose a simple but effective method to boost ViT's transferability, which can be divided into source-domain stage and target-domain stage.
3.1 Source-Domain Attention Abandonment
Since the default query-key attention is verified to be vulnerable domain gaps, aim to enhance the non-query-ke part of ViT the source-domain training. We propose to stochastically abandon query-key attention of all ViT blocks during the source-domain training, by multiplying temperature of 0 (Fig. 4) so that the attention will be randomly downgraded to a uniform map. This is written as
A(x) = softmar(A'() 7 (p)).
where A'(x:) xbxmmxxxxxx denotes the un-normalized attention map in Eq. (p( scalar sampled from the binary distribution, and (typically set to 0.8) is the probability of being 0.
(0, 1) is
This operation will resist the learning of the query-key attention parameters (i.e., Q and K in Fig. 4. as query-key attention will be applied in the forward pass with a probability of 1 p. This operation will also encourage the learning of the non-query-key attention parameters, the average attention is applied with a probability of p. Moreover, will also resist the exceeding attention on the CLS token, as the CLS token's attention value in the uniform map would be 1/n, which is very small. Note that it is different from the Dropout 34 operation (1) Dropout is carried out element-wisely in the attention map, but directly shift the whole attention map uniform map; (2) Dropout is conducted after the softmax function, but ours is before it.
3.2 Target-Domaie Attention Adjustment
During the finetuning and evaluation of target dltassets, the default query-key attention still applied in the network, but we follow section 2 to set pre-defined apper-paramtter 7T for ViT blocks to

---

alleviate the influence of ineffective attention maps. This operation is written
A(x) softmar(AA(z)) ("")
Finally, we follow current works 12 to conduct the prototype-bard passeedd classification on the target- domain episodes Eq. [ finetune model the support of these episodes for the classifier-bass classification.
4
Experiments
Table 3: Comparison with state-of-the-art works by the 5-way 1-shot classification.
Method
hackhone
FT
Mark
Crop.
Furo.
ISIC.
Ches.
GNN+AFA 15
ResNer10
BCCV-22
67.61
63.12
33.21
22.92
46.97
LDP-net 50
ResNet10
CVPR-22 23
69.64
65.11
33,97
23.01
47.
FLoR 58
ResNet10
CVPR 24
73.64
62.90
38.11
23.11
49,69
SDT 29
ResNet10
NN-24
73.92
65.87
36.45
23.22
49.97
MEM-FS 42
VIT-S
TIP-23
81.11
68.11
32.97
22.76
51.24
StyleAdv
10
VIT-S
CVPR-23
81.22
72.15
33.05
22.92
52.34
SDT 29
VIT-S
NN-24
81.03
72.71
33.40
22.79
52.48
FLoR 53
VIT-S
CVPR-24
81.81
72.39
34,20
22.78
52.80
AttnTemp
VIT-S
Ours
84.02
74.35
34.92
23.19
54.12
FLoR [53
ResNer10
CVPR-24
84.04
69.13
38.81
23.12
53.78
PMF 14
VIT-S
CVPR-22
80.79
70.74
30.36
21.73
50.91
FLoR [53
ViT-S
CVPR-24
83.55
73.09
35.49
23.26
53.85
StyleAdv 10
VIT-S
CVPR-23
84.11
74.93
33.99
22.92
53.99
Attn'Temp
VIT-S
Ours
84.78
75.09
38.05
23.63
55.39
LDP-net"
ResNet10
CVPR-23
81.24
73.25
33.44
22.21
52.54
RDC 22
ResNet10
CVPR-22
85.79
70.51
36.28
22.32
53.73
FLoR 53
ResNet10
CVPR-24
86.30
71.38
41.67
23.12
55.62
MEM-FS+RDA"
42
ViT-S
TIP-23
83.74
75.91
37.07
23.85
55.14
AttnTemp
VIT-S
Ours
87.58
77.40
40.13
23.96
57.23
4.1 Datasets
Following current works 2 12
we utilize the minilmageNet dataset 40 as the source dataset, and
utilize 4 cross-domain datasets the target datasets, including CropDiseases (30), EuroSAT [13],
ISIC2018 [5] and ChestX [44] for few-shot training and evaluation, using the k-way n--hot classifica-
tion stated in section 2.1 minilmagenet [40] is subset of the large-scale ImageNet [6] dataset with 100 categories and 60,000 images, where 64 categories are utilized for training. CropDiseases [30] dataset for agricultural disease recognition, consisting of 38 categories and 43,456 images. EuroSAT [13 is satellite image dataset for land classification, comprising 10 classes and 27,000 images. ISIC2018 [5] is for skin lesion recognition, comprising categories with 10,015 images. ChestX 44 contains chest X-ray images, with 7 categories and 25,847 images.
4.2
Implementation Details
We follow StyleAdv [10 to take ViT-S as the backbone network and take the DINO pretraining on ImageNet as the initialization (other pretraining can be found in the appendix). During the source- domain training, we apply all ViT blocks to the Attention Abandonment. We use the Adam [I6] optimizer with learning rate of 0.001 for the classifier and 10-6 6 for the backbone network. During the target-domain few-shot evaluation, we set the temperature for the first two blocks as 0.3, and set the attention of the CLS token to () for blocks whose ID is greater than 4.
4.3 Comparison with State-of-the-Art Works
Tab. and report our results compared with state-of-the-art works for both 1-shot and 5-shot settings. We separately compare works with and without finetuning (FT) for fairness. The asterisk
(*) denotes a transductive setting. PMF [4], StyleAdv (10), MEM-FS 42] and FLoR [53] are introduced for comparison. For all results, our work achieves the top average performance in all
7

---

Table 4: Comparison with state-of-the-art works by the 5-way 5-shot classification.
Methods
backbone
FT
Mark
Crop.
Euro.
ISIC.
Ches.
Ave.
LDP-net 150
ResNer10
CVPR-23
89.40
82.01
48.06
26.67
61.29
GNN+AFA
ResNet10
BCCV-22
88.06
85.58
46.01
25.02
61.67
SDT 29
ResNet10
NN-24
90.27
82.02
48.66
27.20
62.04
FLoR [53
ResNet10
CVPR-24
91.25
80.87
51.44
26.70
62.32
MEM-FS
VIT-S
TIP-23
93.74
86.49
47.38
26.67
63.57
StyleAdv
VIT-S
CVPR-23
94.85
88.57
47.73
26.97
MICM
ViT-S
MM-24
94.61
90.08
46.85
27.11
64.66
SDT 29
ViT-S
NN-24
95.00
89.60
47.64
26.72
64.75
FLoR [3
ViT-S
CVPR-24
95.28
90.41
49.52
26.71
65.48
Attn'Temp
ViT-S
Ours
95.53
90.13
53.09
27.72
66.62
FLoR 53
ResNer10
CVPR-24
92.33
83.06
56.74
26.77
64.73
PMF 14
VIT-S
CVPR-22
92.96
85.98
50.12
27.27
64.08
StyleAdv 10
VIT-S
CVPR-23
95.99
90.12
51.23
26.97
66.08
FLoR 58
VIT-S
CVPR-22 24
96.47
90.75
53.06
27.02
66.83
AttnTemm
VIT-S
Ours
96.66
90.82
54.91
28.03
67.61
LDP-net 50
ResNet10
CVPR-23
91.89
84.05
48.44
26.88
62.82
RDC' P2
ResNed0
CVPR-22
93.30
84.29
49.91
25.07
63.14
FLoR" 53
ResNer10
CVPR-24
93.60
83.76
57.54
26.89
65.45
MEM-FS+RDA
42
ViT-S
TIP-23
95.04
88.77
51.02
27.98
65.70
Attn'Tmm
VIT-S
Ours
96.74
91.34
55.22
28.41
67.93
Table 5: Ablation study by the 5-way 5-shot accuracy.
Adjustment
Abandonment
CropDisease
EuroSAT
ISIC2018
ChestX
Ave.
94.24
88 62.622222
45.72.:0.3
25.66
63.53
24.48
88.73222s
49.12.20.
25.81 10:1
64.5322mm
95.40+a.3
=9.08=029
52.01
27.49+
66.00=a.19
95.53
90.13
53.09 18
27.72.1019
66.62.0.19
Learnable Temp.
94.26+aos
88.91 #0,09
46.45
26.44:aa.s
64.02+007
Dropout 34
14m7
89.62 62.622022
46.03
26.341uu1
64,11 1
Masking Diagonal Atten.
20
88, 38.952009
46.98=0.1
26.74
64.38
Fix Attention
94.67 1uzz
89, 29,66
46.22
26.31 im20
64.08
Totally abandon attention
95.07
88.90
49.23
27.44
65.1620.1
settings and achieves the highest performance in almost all datasets. This demonstrates that our method effectively reduces the domaín gap and enhances model transferability.
4.4 Ablation Study
We first ablate the Attention Adjustment and Abandonment. In Tab, 5 we can see both modules contribute to the performance, especially for the Attention Abandonment module which yields significant 2.47% average improvement. By adding the Adjustment module, the performance also increases but is smaller than that applied to the baseline model. This is because by applying Attention Abandonment, the attention is improved so that the remedy by temperature is not great need.
Then, we ablate our designs and compare them with other attention-temperature-- bbasdd works.
(1) Learnable or stochastic temperature? 152 applied a small network to dynamically learn tem- peratures, which shows only trivial improvements, showing the importance of stochastic temperature.
(2) Global or element-wise temperature? Dropout (34] ciement-wisely abandons the attention after softmax and LSA [20] masks the diagonal attention map, which shows only marginal improvements on CDFSL, indicating the importance of a global temperature for each attention map.
(3) Train attention not? Since abandoning the attention network can reduce its training and alle- viate overfitting, we directly fix the attention network for verification and see only slight performance improvements, the training of non-query-key parameters limited by ineffective attention.
(4) Maintain attention or not? Due to the ineffectiveness attention, we try to directly abandon all attention by setting temperature of 0 for all blocks. We can the performance is improved by

---

CLS Teken Alieniion Value vs. Dlock
minilmageNet
CropDiscases
EuroSAT
Image Alicatian Dasiiy Black
700
ISIC2018
ChestX
Caggloone Lenn
LectAL
228:2011
Timpet
(x) Our Atlention Map Visualization
(b) Quantitative Mensure of Our Attention
Figure 5: (a) Visualization of attention maps of our model. We can see images activated only on the CLS token in Fig. 2 are now correctly activated, and the model can focus on the meaningful and concentrated regions, verifying the improved attention networks. (b) By evaluating the image- loken-attention density and the CLS-token-attention value of our model, we can see these criteria are cin-distinuuishabee between the source and target domains (compared with Fig. (2),, indicating attention networks' transferability against domain gaps is improved.
Table 6: Verification of improved self-attention w.r.t domain similarity and target-domain accuracy.
Metric.
3
5
7
9
10
11
12
BL CKA
0.9805
0.9500
0.9667
0.9654
0.9455
0.9146
0.8940
0.8406
0.7446
0.6337
0.2063
0.1756
Ours CKA
0.9857
0.9590
0.9659
0.9704
0.9547
0.9347
0.9148
0.8763
0.7903
0.6655
0.2955
0.1886
BL Ace.
34.67
39.88
42.19
44.73
47.20
48.93
50.05
50.98
52.60
53.02
52.03
51.82
Ours Acc.
34.91
40.47
43.01
45.19
47.28
48.93
50.40
51.47
53.26
54,34
53.98
53.70
more than 1%, indicating relying the non-query-key outputs indeed help the cross-domain transferring. However, the accuracy is still lower than Attention Abandonment, showing that the query-key attention network still contains useful information for classification.
4.5 Verification of Improved Attention
4.5.1
Qualitative Study
We visualize attention maps of our model on both the source and target domain in Fig. 5h. In contrast to Fig. where attention primarily focuses on the CLS token the target domain, our model can correctly activate meaningful and discriminative tokens. Furthermore, compared to the dispersed attention observed in Fig. 2p. our model focuses on more concentrated regions within the image, indicating that model effectively transfers the attention network from the source target domains.
4.5.2 Quantitative Study
We compare the image token attention density and the CLS token attention value mentioned in Sec. 2.2 As depicted in Fig. these criteria are mostly consistent between the source and target domain, in contrast to the disparities in Fig 2p, suggesting improved transferability of attentions.
Moreover, in Tab. 6 we report the domain similarity and target-domain accuracy of the features output by the attention network from each ViT block. Following Tab. 2 we measure the domain similarity by the CKA similarity. From Tab. 6 we can see our model improves the domain similarity of each self-attention's 's output, indicating improved transferability of attention networks. Consistently, the target-domain accuracy is also improved by our method.
Finally, following Tab. [ we compare the domain similarity and performance between the default query-key attention and other attention choices in Tab, [6b.. We can see the Input feature of the attention network's 's CKA is decreased, as we encourage the learning of non-query-key parameters in ViT. However, such a decrease in CKA much smaller than the decrease brought by the query-key attention. In contrast, the CKA of the attention network's output consistently increases, indicating better transferability against domains. This can also be verified Tab. 66 that the performance difference between different attention choices is narrowed.

---

(a) Domain Similarity
(b) 1-thor
Act. Abun Probusiiity
(d) Ablate Block from Abandoument
0445
Conmagy
06.0
0:35
53
65.1
65.0
64A
64.5
Aug..
Harciie
Bitte
Block Abas
Renove Black from Abas.
Figure 6: (a) Domain similarity of self-attention outputs consistently increases. (b) The difference in accuracy between different attention choices is narrowed. (c) A high probability of abandoning query- key attention can help the transferability. (d) All blocks need to be put into Attention Abandonment.
4.5.3 Sensitivity Study of Hyper-parane paraam-psr
We plot the average target-domain accuracy vs. abandonment probability p in Fig. c When p it is downgraded to the baseline method. When = 1, it means all attentions are abandoned. A high probability of abandoning can help the cross-domain transferring. but we cannot simply abandon all attention. indicating useful information in the attention network. Moreover, we ablate ViT blocks added to the Abandonment method in Fig. (d.. By gradually adding or removing abandoned blocks, the accuracy of the target domain increases or decreases accordingly. This indicates each block positively contributes to the model, highlighting the necessity of all blocks for Attention Abandonment.
5 Related Work
Cross- Domain Few-Shot Learning (CDFSL) aims to acquire knowledge from the target domain with limited training samples [2]. The domain gaps between source and target domains make it challenging. Current works can be categorized into two groups: meta-learning based 2 38 43
15]
which simulates the data structure for the target-domain learning, and transferring-
22 32
28 25), involving training a model with strong generalization. However, they are mainly limited to the CNN structure, and while recent works [10 42 14, 45] have begun utilize the transformer architecture for CDFSL tasks, they have not fully leveraged the potential of the ViT architecture.
Domain Generalization (DG) aims to generalize models from seen to unseen domains [4 24
aligning with the objective of CDFSL. Recently, transformer-bant based approaches have been studied 448 31 35]. [36] discovered that self-attention is not indispensable. [46] discovered that self-attention
not adept at distinguishing the transferability and discriminability of features across different domains. Different from them, we explore the influence of temperature attention transferability against large domain gaps, without introducing any additional modules.
Attention Temperature can adjust the smoothness of the Softmax output distribution in the attention network. [39] proposes using a constant temperature scale the dot product alleviate small extreme gradient. Recently, some methods have emerged to dynamically adjust the temperature while training deep learning models (26 20 20 proposes apply a learnable temperature attention scores to address overly smooth distributions. (3] reduces attention noise by suppressing accumulated trivial attention weights. In contrast, our study the first, to our knowledge, delve into the impact of attention temperature on cross-domain transfcrability.
Conclusion
In this paper, we find a phenomenon for the temperature-base attention adjustment in the ViT-based CDFSL task and delve into it for an interpretation. We interpret remedy for the ineffective target-domain attention caused by the default query-key attention mechanism. Based on it, we further propose a method for CDFSL. Experiments validate our rationale and effectiveness.
Acknowledgments
This work is supported by the National Natural Science Foundation of China under grants 62206 102, 62436003, 62376103 and 623022228000 18;; the Science and Technology Support Program of Hubei Province under grant 2022BAA046; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund.
10

---

References
[1] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[2] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In Proceedings of the International Conference on Learning Representations, 2018.
[3] Xiangyu Chen, Qinghao Hu, Kaidong Li, Cuncong Zhong, and Guanghui Wang. Accumulated trivial attention matters in vision transformers on small datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages
January 2023.
[4] Yinbo Chen, Zhuang I. Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-haseline: Exploring simple meta-learning few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9062-9071, October 2021.
[5] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 01:: A challenge hosted by the international skin imaging collahoration (isic). arXiv preprint arXiv: 03368, 2019.
[6] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 248 255. Icce, 2009.
[7] Jacob Devlin, Ming-Wes Meig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4171- 4186, 2019.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[9] Haoqi Fan, Bo Xiong. Karttikeya Mangalam. Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 6824 6835, October 2021.
[10] Yuqian Fu, Yu Xie, Yanwei Fu, and Yu-Gang Jiang. Styleadv: Meta style adversarial training for cross- domain few-shot learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 24575-24584, June 2023.
[11] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT Press, 2016
[12] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In Proceedings of the IFFF/CVF European Conference on Computer Vision, pages 124-141. Springer, 2020.
[13] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selecied Topics in Applied Earth Observations and Remote Sensing, 12(7):2217 2226222662222222 2019.
[14] Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference. In Proceddings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 9068-9077 June 2022.
[15] Yanxu Hu and Andy J. Ma. Adversarial feature augmentation cross-domain sho-shot classification. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022, pages 20-37, Cham, 2022. Springer Nature Switzerland.
[16] Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6998 2014.
[17] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, In International Conference on Machine Learning, pages 3519-3529, PMLR, 2019.

---

[18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[19] Jogendra Nath Kundu, Akshay R Kulkarni, Suvaansh Bhambri, Deepesh Mehta, Shreyas Anand Kulkarni, Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for source-freee domain adaptation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 11710-11728. PMLR. 17-23 Jul 2022.
[20] Seung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets. arXiv preprint arXiv:21 134492, 2021.
[21] Bo Li, Yifci Shen, Jingkang Yang. Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwci Liu. Sparse mixture-of-experts are domain generalizable learners. In The Eleventh International Conference on Learning Representations, 2023.
[22] Pan Li. Shaogang Gong, Chengjie Wang, and Yanwei Fu. Ranking distance calibration for cross- doss-domain few-shot learning- In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 9099-9108, June 2022.
[23] Wei- Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9526-9535, October 2021.
[24] Yiying Li, Yongxin Yang. Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3915 -9244, PMLR, 09-15 15 Jun 2019.
[25] Hanwen Liang. Qiong Zhang. Peng Dai, and Juwei Lu. Boosting the generalization capability in cross- domain few- learning via noise-enhanced supervised autoencoder. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9424-9434, October 2021.
[26] Junyang Lin, Xu Sun, Xuancheng Ren, Muyu Li, and Qi Su. Learning when to concentrate or divert attention: adaptive attention temperature for neural machine translation. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun ichi Tsujii, editors. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2985 2990, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[27] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Man-Michacl Frahm, editors, Computer Vision ECCV 2020. pages 438 455, Cham, 2020. Springer International Publishing.
[28] Bingyu Liu, Zhen Zhao, Zhenpeng Li. Jianan Jiang, Yuhong Guo, and Jieping Ye. Feature transformation ensemble model with batch spectral regularization for cross-domain few-shot classification. arXiv preprint 2oxiv:2005.08463. 2020.
[29] Yicong Liu, Yixiong Zou, Ruixuan Li, and Yuhua Li. Spectral decomposition and transformation for cross-domain sho-shot learning. Neural Networks, 179:106536, 2024.
[30] Sharada P. Mohanty, David P. Hughes, and Marcel Salathé. Using deep learning for amage-based plant disease detection. Frontiers in Plant Science, 7(September), September 2016. Publisher Copyright: 2016 Mohanty, Hughes and Salathé.
[31] Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Gustavo A. Vargas Hakim, David Osowiechi, Ismail Ben Ayed, and Christian Desrosiers. Tis-vit: Token-level feature stylization for domain generalization. Pattern Recognition, 149: 10213, 2024.
[32] Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differ- ences. In International Conference on Learning Representations, 2021.
[33] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the International Conference on Neural Information Processing Systems, pages 4080-4090, 2017.
[34] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929 1958, 2014.
12

---

[35] Maryam Sultana, Muzammal Naseer, Muhammad Haris Khan, Salman Khan, and Fahad Shahbaz Khan. Self-distilled vision transformer for domain generalization. In Proceedings the Asian Conference on Computer Vision (ACCV), pages 3068-3085, December 2022.
[36] Tay, Dara Bahri, Donald Metzler, Da- Da-Cheng Juan, Zhe Zhao, and Che Zheng, Synthesizer: Rethinking self-attention for transformer models. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10183-10192. PMLR, 18-24 Jul 2021.
[37] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[38] Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming- Husuan Yang. Cross-domain few-shot classifica- tion via learned wise transformation. In Proceedings of the International Conference on Learning Representations, 2020.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[40] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the International Conference on Neural Information P Processing Systems, pages 3637-3645, 2016.
[41] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing unseen domains via adversarial data augmentation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa- Desa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[42] Recce Walsh, Islam Osman, and Mohamed S. Shehata. Masked embedding modeling with rapid domain adjustment for few-shot image classification. IEEE Transactions on Image Processing, 32:4907-4920, 2023.
[43] Hanqing Wang and Zhi- Thi-Hong Deng. Cross-domain few-shot classification via adversarial task augmenta- tion. In Zhi-Hus Zhou, editor, Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pages 1075-1081, International Joint Conferences on Artificial Intelligence Organi- Main Track.
zation, 8 2021.
[44] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEFF, July 2017.
[45] Kan Wu, Jinnian Zhang, Houwen Peng, Mengehen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision (ECCV), 2022.
[46] Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 520-530, January 2023.
[47] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Ha-a Jiang, Francis E.H. Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 558-567, Octoher 2021.
[48] Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao, Xianglong Liu, and Ziwei Liu. Delving deep into the generalization of vision transformers under distribution shifts. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 7267-7276. 2022.
[49] Zhenyu Zhang, Guangyao Chen, Yixiong Zou, Zhimeng Huang. Yuhua Li, and Ruixuan Li. Miem: Rethinking unsupervised pretraining for enhanced few-shot learning. In Proceedings of the 32nd ACM In- ternational Conference on Multimedia, MM '24, page 7686-7695 New York, NY, USA, 2024. Association for Computing Machinery.
13

---

[50] Fei Zhou, Peng Wang, Lei Zhang, Wei Wei, and Yanning Zhang. Revisiting prototypical network for cross domain few-shot learning. Proceedingy of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 20061-20070, June 2023.
[51] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie. Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. International Conference on Learning Representations, 2022.
[52] Xiang Zhou, Yuan Zeng, and Yi Gong. Learning scale temperature in masked self-altention for image inpainting. arXiv preprint arXiv:2302.061 0002006600, 2023.
[53] Yixiong Zou, Yicong Liu, Yiman Hu, Yuhua Li, and Ruixuan Li. Flatten long-range loss landscapes for cross-domain few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 23575-235555 June 2024.
[54] Yixiong Zou, Shanghang Zhang, Yuhua and Ruixuan Margin-based few-shot class- incrementel learning with class-level overfitting mitigation. In Proceedings of the International Conference on Neural Information Processing Systems. 2022.
14

---

A Appendix supplemental material
A.1 Dataset Description
CropDisease
Partias
EuroSAT
ISIC2018
ChestX
source domain
target domain
Figure 7: Samples of source domain minilmagenet dataset (left) and target domain datasets (right), from top to bottom correspond to CropDiseases, EuroSAT, ISIC2018, and ChestX. We can see large domain gaps between source and target domains.
minilmageNet 40 is subset derived from the larger ImageNet 6 dataset. It consists of 100 categories, each containing 600 natural images, Following the current works 12 we employ the training set of minilmageNet the source domain dataset, consisting of 64 classes and 38 38,400 images. Additionally, as depicted in Fig. [7] we utilize datasets from four distinct domains as target domains following [12), including plant discase images, surface satellite imagery, skin disease images, and chest X-ray images. We will introduce them sequentially below.
CropDiseases 30 is dataset for agricultural disease recognition, encompassing 38 distinct classes and total of 43,456 images. The dataset comprises images of various crops, including infected and healthy plants, and corresponding disease category labels.
EuroSAT ISS is comprehensive dataset comprising satellite imagery of the Earth. It encompasses a total of 27,000 images distributed across 10 distinct classes. The dataset offers a diverse range of geographical and topographical features.
ISIC2018 [5]
medical imaging dataset for skin lesion classification. The dataset consists of 10,015 images categorized into 7 distinct classes.
ChestX 44 is an X-ray medical imaging dataset for chest classification. The dataset consists of 25,847 images across 7 different classes.
A.2 More experiments
A.2.1
Applying Our Method to VIT Variants
Table 7: Our method with iBOT-pretrained ViT-S.
Method
Shot
ChestX
ISIC2018
EuroSAT
CropDiseases
Average
iBOT
22.67
31.61
72.85
81.35
52.12
OurT+Ours
23,01
34.69
73.04
82.39
53.28
iBOT
26.31
44.54
89.65
94.79
63.82
iBOT+Ours
5
27.63
51.06
89.21
95.20
65.78
We also apply our method to ViT-Small pretrained by iBOT(5]] and ViT-Base pretrained by DINO iBOT is self-supervised pre-training framework that learns semantic representations of images
15

---

Table 8: Our method with DINO-pretrained ViT-Base.
Method
Shot
ChestX
ISIC2018
EuroSAT
CropDiseases
Average
DINO-B
22.78
34.08
71.89
82.77
52.88
DINO-B+Ours
22.81
35,98
72.98
83.14
53.73
DINO-B
5
26.52
48.77
89.67
95,45
65.10
DINO-B+Ours
27.09
52.71
90.06
95.70
66.39
through Masked Image Modeling (MIM) and an online Tokenizer, enabling effective pre-training of vision Transformers without needing labeled data. The results are shown in Tab. 7 iBOT represents the
VIT baseline, while iBOT+Ours denotes our method applied to iBOT. As we see, method also shows considerable improvement the ViT pre-trained with iBOT, with 1.16 point increase in 1-shot and a 1.96 point increase 5-shot. The results our method applied to DINO-pretrained ViT-Base are shown in Tab. 8 DINO-B represents the DINO-pretrained ViT-Base baseline in our CDFSL task, and DINO-B+ ViT-Base. This also shows improvement in ViT-Base with 0.85 point increase in 1-shot and 1.29 point increase in 5-shot.
denotes our method applied to DINO-pretrained
A.2.2
The Effectiveness of Our Attention
Block
Block 2
Block
Block
670
662
65.4
64.6
Durs
Baseline
Oues
Baseline
OuTs - Btseline
Ours
Bascline
Block 5
Block 6
Block
Block 8
65.5
66.5
66.5
66.5
05.5
63.5
05.5
65.5
61.5
615
61.5
68.5
63.5
63.5
63.5
63.5
62.3
625
02.5
62.5
2.0
1.6
1.2
89
0.7
0.5
013
2.0
1.6
122
09
0.7
0.5
03
2.0
1.6
1.2
0.9
0.7
0.5
03
0.1
2.0
1.5
12
09
02
05
033
0.1
Block
Block
Block
Block
05
033
09
Ours
Baseline
Oues
Baseline
Olts
Baseline
Ours
Baseline
Figure 8: Average target domain accuracy temperature. The red point represents the temperature 1.0. If the attention is good enough, the attention adjustment will trivial. Our method shows less reliance the attention adjustments compared with the baseline method and achieves the best performance when the temperature is 1.0 in most blocks. This indicates the attention produced by our method is improved.
In Fig. we illustrate the average target domain accuracy of the baseline method and our model with attention adjustment. The red point means the temperature is 1.0 (i.e., no temperature is applied). The accuracy of the baseline method increases with the temperature decrease from 2.0 0.0 in most blocks, which means temperature adjustment is important in the target domain. If the attention already good enough, the impact attention adjustment trivial. As we can see, our method exhibits little variation in performance with temperature adjustment compared to the baseline method, and the highest accuracy achieved when the temperature 1.0 most blocks. This verifies the effectiveness of our attention.
16

---

A.3 Broader Impact
Our research introduces an improved VIT model based on attention temperature adjustment, aimed at addressing the ineffective target-domain attention caused by the query-key attention mechanism in CDFSL tasks. By suppressing the learning of query-key parameters and encouraging that of non- query-key parameters, our method significantly enhances the model's cross-domain transferabilityy This work is not only applicable to CDFSL but can also be extended to other domains, such as domain generalization, domaín adaption, and few-shot class-incremental learning, where enhancing the transferability of self-attention a prevalent challenge. While our method has been evaluated across four distinct target domains, providing a good initial assessment of our method's 's cross-domain applicability, the diversity of these domains may not encompass all possible real-world scenarios. Future work will aim to extend our evaluations to include wider range of target domains to understand their performance diverse real-world settings better.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not have theoretical results or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have reported the detailed settings and hyper-parameters. We will release our codes.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We have reported the detailed settings and ppper-paramttes... We will release our codes.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
httpe/uuiides/oodeSuuiisssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpol/iiigggddde/ooooooo weweeeeeeeooeSbummmioooe ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have reported the detailed settings and our codes.
We will release
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We have reported the confidence interval in our ablation studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the computer resources the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///arrrr.ccccooo/cc//aaaaaa
Answer: [Yes]
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impact in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not have the risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have respected the license of assets used in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not have assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not include human subjects information.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "NIVa3vijHj", "reviewer": "Reviewer_D8qR", "summary": "This paper studies the effectiveness of Vision Transformer (ViT) for Cross-domain few-shot learning (CDFSL). In particular, the authors found that by simply multiplying a temperature to the attention in ViT blocks, the target-domain performance consistently increases, even though the attention map is downgraded  to a uniform map. The authors investigated phenomenon through several experiments and propose a simple and efficient solution for boosting ViT’s transferability by resisting the learning of query-key parameters and encouraging that of non-query-key ones. Extensive experiments demonstrate the effectiveness of the proposed method for CDFSL.", "questions": "1. What does it mean by \"non-query-key structures\"?\n\n2. In Source-Domain Attention Abandonment, would the training strategy degrade the model performance on source data?\n\n3. Further explanations are needed for Equation (6) to clarify how it can improve learning better attention for CDFSL.\n\n4. The authors primarily consider a 5-way 5-shot scenario. How about increasing the number of samples in each class? For instance, in a 5-way 20-shot problem, would the proposed strategy still be effective?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper studies an important problem in few-shot learning called CDFSL and also investigates the effectiveness of ViT for this problem which is of high practical importance.\n\n2. The paper conducted a detailed analysis of attention temperature on addressing target-domain shift by visualizing the attention maps and quantitative analysis. \n\n3. The proposed solution is simple and effective for addressing the domain gap issue in CDFSL with ViT.", "weaknesses": "1. The paper mostly focuses on empirical analysis while lacking theoretical insights on why query-key features tend to be discriminative but less transferable.\n\n2. The proposed method needs to retrain the ViT in Source-Domain Attention Abandonment which can be costly. \n\n3. In Target-Domain Attention Adjustment, a pre-defined hyper-parameter is needed, which can be difficult to tune in CDFSL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n\n## W1. Theoretical insights. \n\nPlease refer to the global rebuttal Q1; we have conducted a theoretical analysis.\n\n## W2. Retrain ViT on the source domain\n\n​\tWe would like to point out that following current works [10,49], the size of the source-domain dataset (miniImageNet) is not large since this dataset is only a subset of ImageNet and contains only 64 classes with 600 samples in each class. The training is conducted on a single RTX3090 GPU for around 5 hours, which is affordable.\n\n​\tMoreover, we also decrease the number of samples in each class to verify how the source dataset size influences the performance.\n\n| Sample Number         | -     | 100   | 200   | 300   | 400   | 500   | 600   |\n| --------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| 5-way 5-shot accuracy | 63.53 | 65.29 | 65.69 | 65.86 | 65.95 | 66.00 | 66.00 |\n\n​\tWe can see the performance is not sensitive to the source dataset size, as we can achieve comparable performance even if the dataset size is halved to 300 samples in each class. In all, the retraining of the ViT on the source dataset is affordable.\n\n## W3. Pre-defined hyper-parameter of Attention Adjustment\n\n​\tThe hyper-parameter in the target-domain attention adjustment is the temperature, which is simply set to a fixed value (0.3) for all datasets. As shown in Fig.1b, the average performance plateaus when the temperature gets smaller. We also report the performance of our model w.r.t. the temperature as follows.\n\n|      Temperature      |  1.0  |  0.9  |  0.8  |  0.7  |  0.6  |  0.5  |  0.4  |  0.3  |  0.2  |  0.1  |\n| ------------------- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| 5-way 1-shot accuracy | 53.51 | 53.52 | 53.65 | 53.70 | 53.98 | 54.05 | 54.09 | 54.12 | 54.11 | 54.10 |\n\n​\tAs can be seen, the performance plateaus when the temperature gets smaller than 0.5. We also evaluated the temperature sensitivity in the appendix Fig.8. Therefore our model is not sensitive to the specific choice of the target-domain temperature, i.e., this hyper-parameter is not difficult to tune.\n\n## Q1. Non-query-key structures\n\n​\tThe non-query-key structure refers to the ViT backbone network without the query-key parameters, i.e., replace the query-key attention with the identity attention, the cosine attention, or the average attention, as shown in Tab.2. We promise to refine this demonstration.\n\n## Q2. Degrade the source-domain performance\n\n​\tThe source-domain performance is decreased from 97.94 to 97.56 by applying our method, which is only a marginal decrease and is affordable.\n\n## Q3. Explanation of Eq. 6\n\n​\tEq. 6 means to randomly multiply a temperature of 0 or 1 to the query-key attention in each block, and the probability of 0 is *p*. If the temperature is 0, then the attention is downgraded into average attention in Tab.2. If the temperature is 1, the original query-key attention is maintained.\t\n\n​\tIn section 2 we conclude that the query-key attention mechanism makes the model discriminative but less transferable, while the non-query-key ViT structures (ViT network without the query-key parameters) tend to make the model transferable but less discriminative. Therefore, we randomly abandon the query-key attention to encourage the non-query-key attention (i.e., the average attention) in the source-domain training. As the source-domain training will encourage the trained parameters to be discriminative but less transferable, this operation helps the learning of the non-query-key ViT structures to be more discriminative, and resist the learning of the query-key parameters to avoid them from being less transferable.\n\n​\tIn all, the query-key part is more discriminative but less transferable, therefore we resist its training to avoid it from being less transferable, while the non-query-key part is more transferable but less discriminative, therefore we encourage its training to improve its discriminability. As a result, the overall attention on the target domain is improved, due to the overcoming of shortcomings of each part.\n\n## Q4. Increase the number of shots\n\n​\tWe report our performance with a larger number of shots below.\n\n| 10-shot  | CropDiseases | EuroSAT | ISIC  | ChesX | Avg.  |\n| -------- | ------------ | ------- | ----- | ----- | ----- |\n| Baseline | 96.33        | 90.59   | 51.35 | 28.42 | 66.67 |\n| Ours     | 96.85        | 91.42   | 58.52 | 30.19 | 69.18 |\n\n| 20-shot  | CropDiseases | EuroSAT | ISIC  | ChesX | Avg.  |\n| -------- | ------------ | ------- | ----- | ----- | ----- |\n| Baseline | 97.15        | 91.76   | 55.56 | 30.73 | 68.80 |\n| Ours     | 97.59        | 92.63   | 62.38 | 32.85 | 71.36 |\n\n​\tWe can see our proposed strategy is still effective in these situations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "miLVXdaapi", "reviewer": "Reviewer_G6Ww", "summary": "This paper investigates the effectiveness of the attention mechanism in Vision Transformer (ViT) for solving cross-domain few-shot learning tasks. It finds that the traditional query-key attention operation is more on the side of discriminability than transferability in their trade-off balance, thus leading to downgraded performance for the target domain when there are large domain gaps. Based on a series of related analyses, this paper proposes an attention abandonment operation for source domain training and an attention adjustment operation for target domain finetuning.", "questions": "1.\tIn the section 4.2 implementation details, the authors propose to “set the attention of the CLS token to 0 for blocks whose ID is greater than 4” during the target-domain evaluation phase. Is this operation performed before or after the softmax function? Is the purpose of conducting this operation to make the image tokens unaffected by the CLS token? Could the authors explain the mechanism and impact of this operation?\n2.\tIn the top plot of Figure 5(b), I have noticed that the CLS token values in the 2nd-10th blocks are near zero on all the datasets. This seems strange because the entire line does not fluctuate within this interval. Does this mean that the CLS token and image patches tokens are orthogonal to each other?", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well organized, easy to follow and free of typos. \n2.\tFrom an observed phenomenon between target domain classification accuracy and attention temperature, this paper conducts comprehensive quantitative analyses of the effectiveness of different attention strategies in terms of the trade off between discriminability and transferability. The results fully support the authors’ claim that the non-query-key components in ViT tend to be more transferable but less discriminative than query-key parts.\n3.\tAll the operations (Source-Domain Attention Abandonment and Target-Domain Attention Adjustment) newly designed in this paper are supported by the above analyses and thus are technical sound.\n4.\tThis paper may provide some insights for developing novel attention operations for ViT based cross-domain few-shot learning methods.\n5.\tExtensive experiments on four datasets demonstrate the effectiveness of the proposed method and also showcase its superiority over other state-of-the-art approaches.", "weaknesses": "I personally like this paper, there are just some small weaknesses:\n\n1.\tIn Figure 2(a), 3(a) and 5(a), the CLS token attention values located at the left-top corner are too small. Moreover, the authors have not provided the vector diagram, so I also cannot see these values clearly by zooming in. Could the authors enlarge the CLS boxes in these figures? In addition, it maybe better to provide a color bar for these heat maps to clearly show the range of their values.\n\n2.\tSome references are not correctly formatted, the conference or journal names as well as page numbers are missing. For example, [4], [10], [13], [14], [15], [23], [29], [30], [47], [48], [49], etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work!\n\n## W1. Fig. 2a and Fig. 5a\n\nWe have added the color bar to the attention map and enlarged the class toke for a clear observation. Please refer to the global rebuttal PDF Fig.1 and Fig.2.\n\n## W2. Formatted references\n\n​\tWe have checked the references and completed the conference or journal names as well as page numbers.\n\n[4] Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9062–9071, October 2021.\n\n[10] Yuqian Fu, Yu Xie, Yanwei Fu, and Yu-Gang Jiang. Styleadv: Meta style adversarial training for cross domain few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24575–24584, June 2023.\n\n[13] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.\n\n[14] Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9068–9077, June 2022.\n\n[15] Yanxu Hu and Andy J. Ma. Adversarial feature augmentation for cross-domain few-shot classification.\n\nIn Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 20–37, Cham, 2022. Springer Nature Switzerland.\n\n[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.\n\n[24] Hanwen Liang, Qiong Zhang, Peng Dai, and Juwei Lu. Boosting the generalization capability in cross domain few-shot learning via noise-enhanced supervised autoencoder. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9424–9434, October 2021.\n\n[30] Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. In International Conference on Learning Representations, 2021.\n\n[47] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.\n\n[48] Xiang Zhou, Yuan Zeng, and Yi Gong. Learning to scale temperature in masked self-attention for image inpainting. arXiv preprint arXiv:2302.06130, 2023.\n\n[49] Yixiong Zou, Yicong Liu, Yiman Hu, Yuhua Li, and Ruixuan Li. Flatten long-range loss landscapes for cross-domain few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23575–23584, June 2024.\n\n## Q1. Set CLS token to 0\n\n​\tThis operation is conducted after the softmax operation. In Fig.5b, we validate that the attention value of the CLS token tends to be larger in target datasets than in the source dataset in the first few blocks, therefore we manually resist the influence of the CLS token in the target-domain attention map. \n\n## Q2. Near 0 attention on the CLS token\n\n​\tIntuitively, this is because the model focuses more on the image tokens, therefore the attention on the CLS token significantly decreases. Theoretically, we can observe that Fig.5b and Fig.3b are similar, with both the CLS token attention (or value) near 0. This is because according to our theoretical analysis (please refer to Q1 in the global response), our model decreases the eigenvalue of the query-key parameters (i.e., $W^k$ and $W^q$), therefore preventing the query-key attention from amplifying the perturbation on the representation $X$ [49]. As a result, the influence of the query-key attention brought to the attention's input feature $X$ decreases, therefore the output attention will be more similar to the activation map of $X$ as shown in Fig.3b. Since Fig.3b shows near 0 activation on the CLS token, the attention map of our model also shows the similar behavior.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3tf3pzG42", "reviewer": "Reviewer_ujLR", "summary": "This paper investigates the application of Vision Transformer (ViT) in Cross-Domain Few-Shot Learning (CDFSL). It fully analyzes the effectiveness of attention to CDFSL performance through experiments and identifies a method to enhance ViT's transferability across domains by adjusting attention mechanisms through temperature scaling. Despite reducing attention maps to uniform distributions, this adjustment effectively improves ViT's performance on target-domain datasets, mitigating issues with query-key mechanisms under large domain gaps. The proposed approach focuses on limiting query-key parameter learning while promoting non-query-key parameters, resulting in consistent outperformance across multiple CDFSL datasets compared to current state-of-the-art methods.", "questions": "Please answer and address the above-mentioned problems.", "limitations": "In the paper authors mention that “We discuss the limitations of the work in the appendix”. However, I couldn’t find anywhere about limitation discussion in the appendix. This work does not contain any negative social impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe analysis of how attention affects the CDFSL results is comprehensive.\n2.\tThe proposed method obtains the SOTA performance.", "weaknesses": "1.\tCan uniform attention be understood as an operation like randomly initializing attention?\n2.\tIs there any deep theoretical explanation for the conclusion that “Compared with the query-key attention, the non-query-key components in ViT tend to be more transferable but less discriminative than the query-key components”.\n3.\t“which inspires us to improve the generalization of ViT by encouraging the learning of non-query-key parts and resisting the learning of query-key ones.” in line 174-175. CDFSL requests both transferable and discriminative. Why resist the learning of query-key ones? If resist the learning of query-key ones, can authors still guarantee excellent performance in the source domain? Or does this paper not focus on the performance in the source domain even in the training phase, but only on transferability?\n4.\tThe method is not novel enough.\n5.\tThere is one way that the query-key attention be randomly downgraded to a uniform map after the source domain training and adjust it in the target domain inference. What is the difference between the above way and the proposed method? What are the advantages of the proposed method?\n6.\tThere are too few comparison methods, and it is also necessary to compare with some methods that use CNN as the backbone to show the superiority of the proposed method in ViT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n\n## W1. Random attention initialization\n\n​\tSince random attention initialization tends to produce a uniform attention map, we can view our attention abandonment method as producing a randomly initialized attention. However, our difference with the re-initialization is that we do not abandon the trained parameters and retrain them. Instead, we just \"skip\" the query-key attention to resist the learning of their parameters, but the trained parameters will not be abandoned. To the best of our knowledge, we are the first to propose this design.\n\n## W2. Theoretical explanation\n\nPlease refer to the global rebuttal Q1; we have conducted a theoretical analysis.\n\n## W3. Source-domain performance\n\n​\tIn section2, we conclude that the query-key attention tends to be discriminative but less transferable, while the non-query-key parts tend to be transferable but less discriminative. Since the backbone network is pretrained on the ImageNet dataset, the query-key parts are already discriminative enough. Therefore, it is not necessary to further train them on the source dataset (miniImageNet, a subset of ImageNet) to make it more discriminative. Instead, it is more important to prevent query-key parts from being too discriminative (i.e., less transferable), therefore we resist the learning of them.\n\n​\tOn the contrary, for the non-query-key parts, as they are transferable but less discriminative, we need to encourage learning of them, utilizing abandoning the query-key attention as the proposed method. Indeed, for the CDFSL task, the target-domain performance is more important, but we do not sacrifice the source-domain performance. To verify this, we measure the 5-way 5-shot accuracy on the source dataset of both the baseline method and ours. **The accuracy is 97.94 for the baseline method and 97.56 for ours, which is only a marginal decrease**.\n\n## W4. Novelty\n\n​\tWe would like to point out that our novelties are in\n\n​\t(1) We are the first to unveil the importance of the attention temperature in ViT-based CDFSL methods and interpret it as a remedy for the poorly transferred query-key attention.\n\n​\t(2) We are the first to encourage the learning of non-query-key parameters by abandoning the query-key attention through a random binary temperature, as recognized by Reviewer G6Ww and D8qR.\n\n​\tCould you please provide papers with similar contributions, so that we can address your concerns more effectively?\n\n## W5. Difference of source and target domain operation, and the advantage of our method.\n\n​\tDuring the source-domain training, we randomly multiply a temperature of 0 or 1 to the attention (before softmax) in each forward pass, so that the query-key attention map randomly switches between a uniform map and the original attention map. If the attention map is downgraded to a uniform map, the query-key attention will be abandoned, so that the query-key parameters will not be trained, thereby resisting the learning of these parameters and encouraging the learning of others.\n\n​\tDuring the target-domain stage, we set the temperature for each attention map to a fixed value (0.3), because our model is not sensitive to the temperature choice, as validated in Appendix Fig.8.\n\n​\tAs validated in experiments, our method effectively improves target-domain performance with comparable source-domain performance, which is simple but effective, as recognized by Reviewer G6Ww and D8qR.\n\n## W6. More comparisons with CNN-based methods.\n\nWe list more comparisons with state-of-the-art works as follows, where we can see our method achieves the best performance. Please refer to the global rebuttal PDF Tab.1 and Tab.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OeqnwT7Yts", "reviewer": "Reviewer_XXJ8", "summary": "This paper deals with ViT-based cross-domain few-shot learning. It first proposes an observation regarding ViT-based model. In cross-domain learning, the attention module in ViT seems to be hurting the model performance in the target domain. Based on this observed phenomenon, this paper proposes some fixes by including an additional temperature to reduce the effect of the attention map (e.g. essentially making the attention map to be uniform)", "questions": "1) Please explain how can you be sure that the phenomenon in the paper is not just overfitting, instead it is something more interesting about ViT.\n\n2) Please explain why not using all datasets used in [10,49] in the experiment.\n\n3) What exactly is the role of \"few-shot\" in this paper? It seems most of the paper is discussing the domain shift issue.", "limitations": "None", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper proposes an interesting phenomenon about ViT-based model in the cross-domain application", "weaknesses": "1) The whole paper is based on the phenomenon (that the attention module in ViT hurts cross-domain performance) analyzed in Sec 2. However, I do not think you can draw such conclusions convincingly from the analysis in Sec 2.  \n\nSec 2 uses the simplest possible to train the model parameters, i.e. the model (including a backbone and a classification head) is learned on the source data, then the backbone is directly used in the target domain using a prototype-based classification. This is the simplest baseline method of model training. There are a lot more sophisticated methods of model training (e.g. MAML, ProtoNet, etc) that are designed for few-shot learning. It is not clear whether the phenomenon observed in this paper is due to this simplistic model learning method (i.e. maybe this phenomenon will not appear if you use a more advanced learning method)\n\n2) This paper deals with few-shot cross-domain learning. But it is not clear whether the observed phenomenon is due to \"few-shot\" or due to \"cross-domain\". The analysis in Sec 2 only considers the \"cross-domain\" aspect, but did not provide anything about the \"few-shot\" aspect. It is not entirely clear whether this phenomenon is due to the domain shift between the source vs target domain, or due to the semantic difference between the classes observed on source and target domains. I would like to see a more rigorous analysis that seperate the two confounding factors\n\n3) It is possible that the phenomenon is merely an overfitting issue, i.e. when you have the learnable attention module (query, key matrices) in ViT, the model just overfits to the source domain easily. If that ia the case, the observed phenomenon may not be that interesting (i.e. it is just overfitting). The proposed solution simply reduces the overfitting by regularization\n\n4) This paper proposes an observation about some phenomenon, but it does not provide a convincing explanation on why this phenomenon happens or matters. This paper simply says that the attention module (learned on source data) does not generalize to target domain. But I think this is too generic -- it is well-known that if you train a model on the source domain, the model usually does not generalize to a target domain (this is the main motivation for research in domain adaptation).It seems the conclusion of this paper is just a specific manifestation of this well-known fact. \n\n4) In the previous papers used for comparison (e.g. [10,49]), those papers reported results on more datasets. What is the reason for omitting those additional datasets in the experiment? This makes me wonder whether the results are cherry-picked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your valuable comments. In the following, we respond to the concerns.\n\n## W1. Implementing with other baseline methods.\n\n​\tWe implemented our method with ProtoNet. Our method continues to be effective with this learning method and improves the performance on the target domain.\n\n| Method          | Crop.     | Euro.     | ISIC.     | Ches.     | Ave.      |\n| --------------- | --------- | --------- | --------- | --------- | --------- |\n| ProtoNet        | 93.59     | 86.92     | 46.15     | 25.68     | 63.09     |\n| ProtoNet + Ours | **95.07** | **89.46** | **48.64** | **27.14** | **65.08** |\n\n## W2. \"cross-domain\" vs. \"few-shot\", and \"domain shift\" vs. \"semantic difference\"\n\n#### \"domain shift\" vs. \"semantic difference\":\n\n​\tWe would like to point out that the miniImageNet performance is evaluated on the novel classes of this dataset, while the model is trained on the base classes of miniImageNet. Since there is no overlap between the base and novel classes, a semantic difference exists between these two sets of classes. As the model shows correct attention on these source-domain novel classes, such phenomenon is majorly due to the domain shift instead of semantic difference.\n\n​\tTo verify this, we construct datasets with source-domain semantic and target-domain style, by swapping the amplitude of the source-domain image with the amplitude from target-domain images and maintaining the source-domain phase (following WaveSAN). Then, we measure the performance of the baseline method and ours.\n\n| 5w5s | src semantic + src Style | src + Crop. style | src + Euro. style | src + ISIC style | src + Ches. style | Avg. target style |\n| ---- | ------------------------ | ----------------- | ----------------- | ---------------- | ----------------- | ----------------- |\n| BL   | 97.94                    | 79.10             | 64.24             | 71.70            | 60.27             | 68.83             |\n| Ours | 97.56                    | 81.40             | 70.92             | 76.11            | 62.90             | 72.83             |\n\n​\tFrom this table, we can see\n\n​\t(1) By swapping the style from the source domain to the target domains, the performance consistently decreases, verifying the rationale of the constructed pseudo-target domains.\n\n​\t(2) With the source-domain (src) style, by applying our method, the performance slightly decreases. In the meanwhile, with the target-domain (Crop., Euro., ISIC, Ches.) styles, our method significantly improves the performance.\n\n​\tSince the semantics are preserved in the constructed pseudo-target domains, we can conclude that the domain shift is the cause of the phenomenon in our paper.\n\n#### \"Cross-domain\" vs. \"few-shot\":\n\n​\tSince ineffective attention is observed by directly extracting features from target-domain images, no specific \"few-shot\" adaptation method is integrated. Therefore, this phenomenon is majorly due to the \"cross-domain\" aspect.\n\n​\tMoreover, by adapting our method, we can also contribute to the \"few-shot\" aspect. In the paper, we conclude that the query-key parameters tend to overfit the source domain, which makes the model discriminative but less transferable. For the \"cross-domain\" part, we resist the learning of these parameters during the source-domain stage to avoid overfitting. On the contrary, for the \"few-shot\" part, **we encourage the learning of them during the target-domain finetuning to better fit the target datasets**. To further verify the contribution to the \"few-shot\" part, we compare the finetuning of the query-key parameters and non-query-key ones.\n\n| 5-way 5-shot | Crop. | Euro. | ISIC. | Ches. | Avg.  |\n| ------------ | ----- | ----- | ----- | ----- | ----- |\n| FT non-QK    | 95.91 | 90.27 | 54.05 | 27.66 | 66.97 |\n| FT QK        | 96.01 | 90.36 | 54.30 | 28.31 | 67.25 |\n\n​\tWe can see the advantage of finetuning (FT) query-key (QK) parameters, albeit query-key parameters are much fewer than the non-query-key ones. In all, our contribution includes both the \"cross-domain\" and \"few-shot\" parts.\n\n## W3. Overfitting\n\n​\tWe would like to point out that although overfitting is a common phenomenon in machine learning, our contribution is in\n\n1. We are the first to unveil that the query-key parameters are more likely to be overfitting than other parameters, especially under large domain gaps.\n2. We are the first to find that temperature adjustment and abandonment can effectively handle the overfitting problem in ViT.\n\n​\tTo the best of our knowledge, we do not find papers with similar contributions. Could you please provide any specific papers for better comparison?\n\n## W4. Why this phenomenon happens and matters\n\nPlease refer to the global rebuttal Q1; we have conducted a theoretical analysis.\n\n## W5. Comparison of more datasets\n\n​\tWe would like to point out that many works (e.g., MEM-FS [40], TIP'23) only conduct experiments on the four datasets we experimented with in the paper. Here, we also report our performance on all 8 datasets as in [10, 49]. Please refer to the global rebuttal Q2. As can be seen, our model can still achieve a state-of-the-art performance average on all 8 datasets.\n\n## Other questions\n\nQ1: Please refer to W3.\n\nQ2: Please refer to W5.\n\nQ3: Please refer to W2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable input.

## Q1. Theoretical insights

#### 1. Our method reduces the sharpness of the model's loss landscape.

​	Theoretically, we analyze our findings from the sharpness of the loss landscapes (Foret 2021). That is, each value of model weights is viewed as a point in the weight space, and corresponds to a loss value. Each point and its loss value constructs a loss landscape, where the source-domain-trained model is viewed as a minimum in the landscape. The sharper the minimum is, the more vulnerable against domain gaps model will be [49]. Specifically, the sharpness is measured as
$$
R_\rho(\theta)=max_{||\epsilon||_2 \leq \rho} L(\theta + \epsilon) - L(\theta)
$$
where $\theta$ refers to the model weights, $\epsilon$ refers to the perturbation with the radius $\rho$. With this criterion, the generalization can be bounded as follows (Foret 2021).

**Theorem 1**. For and $\rho>0$ and any distribution $\mathscr{D}$, with probability $1-\delta$ over the choice of the training set $S \sim \mathscr{D}$,
$$
L_\mathscr{D}(\theta) \leq max_{||\epsilon||_2 \leq \rho} L_S(\theta + \epsilon) + \sqrt{\frac{k \space log(1+\frac{||\theta||_2^2}{\rho^2}(1+\sqrt{\frac{log(n)}{k}})^2) + 4 \space log \frac{n}{\delta} + \tilde O(1)}{n-1}}
$$
​	Following this criterion, we measure the sharpness given perturbations on different model weights.

| Perturbed Weights     | All    | Query Key |
| --------------------- | ------ | --------- |
| Sharpness of Baseline | 7.1483 | 7.0679    |
| Sharpness of Ours     | 5.9915 | 6.4024    |

​	We can see our method indeed reduces the sharpness of the loss landscapes, indicating better robustness against domain gaps [49]. Notably, the sharpness of the query-key parameters is also decreased, indicating we can effectively resist their overfitting to the source domain by the proposed method.

#### 2. Why the query-key mechanism increases the sharpness

​	The query-key attention is calculated as $QK^T = W^k X X^T (W^q)^T$.  Compared with other calculations in ViT, only the query-key attention involves the "square term" of $X$. Therefore, any perturbation added to representations [49] is likely to be amplified by the multiplication of $W^k$ and $W^q$. According to current studies (Chen, 2019), a well-trained model tends to increase the eigenvalue of its weights if overfitting happens. Since the eigenvalue here squarely influences $W^k X X^T (W^q)^T$ due to both $W^k$ and $W^q$​ in it, the query-key attention is more likely to amplify the perturbation added to representations and parameters. As the perturbation is finally forwarded to the classification loss, the query-key attention therefore increases the sharpness of the model, thereby making the model more vulnerable to domain shifts.

​	To solve this problem, our method resists the learning of the query-key attention, preventing $W^q$ and $W^k$ from learning large eigenvalues, therefore reducing the sharpness and benefiting the transferring. To verify the decreased eigenvalues, we measure the product of eigenvalues as follows.

|                                                | DINO Pretraining | Baseline Training | Ours  |
| ---------------------------------------------- | ---------------- | ----------------- | ----- |
| Average eigen value product of $W^q$ and $W^k$ | 14.54            | 14.56             | 14.52 |

​	We can see the baseline training increases the eigenvalue, while our method decreases it, verifying our theoretical analysis.

References

  Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization, 2021. 1, 2, 4, 5, 7, 8

  Chen, Xinyang, et al. "Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation." *International conference on machine learning*. PMLR, 2019.



## Q2. Comparison of more datasets

​	We would like to point out that many works (e.g., MEM-FS [40], TIP'23) only conduct experiments on the four datasets we experimented with in the paper. Here, we also report our performance on all 8 datasets as in [10, 49].

| 5shot       | CUB       | Cars      | Plac.     | Plan.        | Crop.     | Euro.        | ISIC      | Ches.     | Avg.      |
| ----------- | --------- | --------- | --------- | ------------ | --------- | ------------ | --------- | --------- | --------- |
| StyAdv [10] | 95.82     | 61.73     | 88.33     | **75.55**    | 94.85     | 88.57        | 47.73     | 26.97     | 72.45     |
| FLoR [49]   | 96.18     | 61.75     | 89.23     | 72.80        | 95.28     | **90.41**    | 49.52     | 26.71     | 72.74     |
| Ours        | **96.28** | **64.26** | **89.25** | *73.24* | **95.53** | *90.13* | **53.09** | **27.72** | **73.69** |

| 1shot       | CUB       | Cars      | Plac.     | Plan.        | Crop.     | Euro.     | ISIC      | Ches.     | Avg.      |
| ----------- | --------- | --------- | --------- | ------------ | --------- | --------- | --------- | --------- | --------- |
| StyAdv [10] | 84.01     | 40.48     | 72.64     | **55.52**    | 81.22     | 72.15     | 33.05     | 22.92     | 57.75     |
| FLoR [49]   | 84.60     | 40.71     | 73.85     | 51.93        | 81.81     | 72.39     | 34.20     | 22.78     | 57.79     |
| Ours        | **85.48** | **43.45** | **74.49** | *52.58* | **84.02** | **74.35** | **34.92** | **23.19** | **59.06** |

​	As can be seen, our model can still achieve a state-of-the-art performance average on all 8 datasets.

Finally, we appreciate the inspiring comments again and will thoroughly revise the paper accordingly. We hope our explanations have answered your questions.

Meta-review: This paper explores the effectiveness of the attention mechanism in Vision Transformer (ViT) for cross-domain few-shot learning. Specifically, the authors discovered that by simply multiplying a small temperature to the attention in ViT blocks, the performance in the target domain consistently improves, even though the attention map is downgraded to a uniform map. Based on a series of related analyses, the paper proposes an attention abandonment operation for source domain training and an attention adjustment operation for target domain fine-tuning. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method for cross-domain few-shot learning.

The paper received four reviews, two positive (D8qR and G6Ww) and two borderline (ujLR and XXJ8). The reviewers acknowledged that the paper finds an interesting phenomenon in ViT-based models, proposes a simple and effective solution for cross-domain few-shot learning, conducts comprehensive quantitative analyses, provides some insights for developing novel attention operations in ViT-based architectures, and achieves state-of-the-art performance. However, the paper is primarily empirical and lacks theoretical insights. After the rebuttal, one of the two borderline reviewers was convinced and upgraded their score from borderline reject to borderline accept.

Based on these considerations, the AC recommends acceptance and strongly suggests that the authors fully consider the reviewers' comments in the final version, including the additional analyses and experiments added during the rebuttal period, particularly the theoretical analysis and comparison of results on more datasets.
Decision: Accept (poster)