Focus Your Attention when Few-Shot Classification
Authors: Haoqing Wang
Shibo Jie
Zhi-Hong Deng
Venue: NeurIPS 2023 poster (2023)
Keywords: few-shot image classification
fine-tuning
vision transformers
TL;DR: We propose an attention enhancement method to adapt pre-trained vision transformers to few-shot image classification tasks.

Abstract: Since many pre-trained vision transformers emerge and provide strong representation for various downstream tasks, we aim to adapt them to few-shot image classification tasks in this work. The input images typically contain multiple entities. The model may not focus on the class-related entities for the current few-shot task, even with fine-tuning on support samples, and the noise information from the class-independent ones harms performance. To this end, we first propose a method that uses the attention and gradient information to automatically locate the positions of key entities, denoted as position prompts, in the support images. Then we employ the cross-entropy loss between their many-hot presentation and the attention logits to optimize the model to focus its attention on the key entities during fine-tuning. This ability then can generalize to the query samples. Our method is applicable to different vision transformers (e.g., columnar or pyramidal ones), and also to different pre-training ways (e.g., single-modal or vision-language pre-training). Extensive experiments show that our method can improve the performance of full or parameter-efficient fine-tuning methods on few-shot tasks. Code is available at https://github.com/Haoqing-Wang/FORT.

Focus Your Attention when Few-Shot Classification
Haoqing Wang School of Intelligence Science and Technology. Peking University (wanghaoqing, parsley, zhdeng) @ @pku.edu.cn
Shibo Jie Zhi-Hong Deng'
Abstract
Since many pre-trained vision transformers emerge and provide strong represen- tation for various downstream tasks, we aim to adapt them to few-shot image classification tasks in this work. The input images typically contain multiple enti- ties. The model may not focus on the class-related entities for the current few-shot task, even with fine-tuning on support samples, and the noise information from the class-independent entities harms performance. To this end, we first propose method that uses the attention and gradient information to automatically locate the positions of key entities in the support images, denoted position prompts. Then we employ the cross-entropy loss between their many-hot presentation and the attention logits to optimize the model to focus its attention on the key entities during fine-tuning. This ability then can generalize to the query samples. Our method is applicable to different vision transformers (e.g., columnar pyramidal ones), and also to different pre-training ways (e.g., single-modal or vision- languaee pre-training). Extensive experiments show that our method can improve the perfor- mance of full or parameter-efficient fine-tuning methods on few-shot tasks. Code is available httpps thub. thw.. haw.uuu..co//Hooigggaaaag
Introduction
Based on the accumulated experience, humans can learn from few labeled samples for solving novel
tasks, and this is a key aspect of human intelligence. Few-shot learning [12 31 aims to imitate this
ability and has attracted widely attention from the machine learning community [71]. Traditionally, base dataset is collected for learning prior knowledge before solving novel few-shot tasks. The meta-learning methods learn the task-shared inductive bias, e.g., parameter initialization [3] 41].
metric function [52
32 54
optimizer 2 47). The transfer learning methods [56 48 learn
generalizing representation to novel tasks, and achieve comparable or better performance.
Some works 56 [66] find that the well-learned embedding model contributes the most to few-shot classification performance, and even more important than the sophisticated meta- weta-nea algorithms. However, the commonly-used base datasets are far from large enough for learning generalizing embedding, e.g., minilmageNet [60], which severely limits the performance of few-shot classification. In fact, the few-shot learning ability of humans benefits from the supervised unsupervised learning from massive signals since birth, which helps us accumulating enough general knowledge. On the
other hand, self-supervised learning [66] 19 8 18 5 can learn well-gennt gell-leneraliiing representation from nearly-free massive unlaheled data and is beneficial for broad downstream tasks. Using these pre-trained models for few-shot tasks conforms the learning paradigm of humans and has achieved remarkable success 5 74 15] in Natural Language Processing (NLP). In this work, we aim to adapt the pre-trained models few-shot image classification tasks, without using any base datasets.
play an increasingly important role computer vision, so we use them the backbone for few-shot tasks. Note that it is non-trivial problem to fine- tune the preitedined large vision transformers on few support samples due to their data- hungry
Recently, vision transformers [
36 70
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Fine-tuning (uppoor we)
Testing (qury
Ingur
Rock Rooee
Head Prodiction
Ingut
Backbone
Head
Prodiction
vision transformer
ce(p.y) (clasification losa)
vision transformer
P
Attenttonn logits === = key positions
clasifaation
R
(etentton enfencement lows)
Prompes
Attn
Urig Attn
ATA
Our Atta
Figure I: Focusing on key entities via position prompts. The original pre-trained model may attend to multiple entities in single image, and the information from iaas-inndeeedden entities actually noise in the current task. The extremely few support samples make the standard fine-tuning with only classification loss unable to remove the noise information. We propose position prompts (red patches) to prompt the model where are the key patches of the input and focusing on them. This ability gained during fine-tuning can generalize from support samples to query ones. Here the white patches in attention visualization have the top highest attention scores and cover about 95% attention.
property. Inspired by the prompt methods 772 35] from NLP, some works 78 77 10 synthesize the classification weights by encoding the learnable text prompts that describe the objective classes, and then compare them with image features. Despite impressive performance on few-shot tasks, these works strongly rely on the semantic aligmment between image encoder and text encoder, and thus are specific vision-language pre-trained models 45 In this work, we design new form of prompts for vision data, named position prompts, to adapt the pre-trained vision transformers to few-shot image classification tasks. They can prompt explicit valuable information, and also be usable to different pre-trained vision transformers, without limited structures or pre-training ways.
Figure illustrates our framework and motivation, where we visualize the patches with top highest attention scores and covering about 95% attention (details at Section 4.2) of [CLS] token ViT- B/16 M.I. The input images usually contain several different entities, e.g., bird stands the branch (Figure []] 'Input'.. ) The pre- pre-trained model may attend to multiple entities at the same time (Figure Orig Attn') and the noise information from glass-independers entities harms performance. The standard fine-tuning methods tune full or partial [4 34 parameters of the model using the classification loss on the support samples, and can achieve better performance than the backbone- frozen methods. But even with fine-tuning, the extremely few support samples prevent the model from giving sufficient attention the class-related key entities (Figure 1 'FT Attn'). To this end, besides the classification loss, we use the position information of the key entities in each support sample guidance for fine-tuning, and the model is expected focus its attention on the key entities after fine-tuning, not only for the support samples, but also for the query ones, Concretely, we first automatically locate the position of key patches as position prompts (Figure I 'Prompts') in each support sample based on the deep explanation method, where the key patches refer to the most class-related ones and correspond to key entities. Inspired by Rollout (T. we integrate the attention information from multiple layers linearly for explanation, and further introduce gradient information as an auxiliary to ensure class-specific explanation. Unlike existing prompts (55,, ours are only used to guide the fine-tuning process and not available during testing, since we need label information to obtain them. Then, we use the position prompts as the prediction target attention, and optimize the cross-entropy loss between their many-hot presentation and the attention logits during fine-tuning. This loss enables the model to identify the key patches and focus on them (Figure [ 'Our Attn'), and this ability obtained during fine-tuning can generalize to query samples. We also theoretically prove that our method can increase the information from the key patches the output feature and reduce that from other patches (noise information).
Overall, our contributions can be summarized as follows:
We propose a new form of prompts for vision data, position prompts, to effectively adapt the pre-trained vision transformers to few-shot image classification tasks.
Our method, denoted as FORT (FOcus youR aTtention), is applicable to the attention modules in different vision transformers, e.g., Swin [36] or VIT, and not limited specific pre-training ways, e.g., single multi-modal ones.

---

We conduct extensive experiments and verify that our method can effectively improve the performance of full or parameter-cffiient [24 34 fine-tuning methods on few-shot tasks.
2
Related work
Few-shot classification.
The traditional paradigm of few-shot classification is the meta-learning
framework [22 67]. which learns the task-shared inductive bias [3] 52 32 63 on the training tasks
constructed from base dataset. When the base dataset is unlabeled or in the different domain novel
tasks, we also need to solve new challenges, unsupervised [23 22 64 or cross-domain [58] 62 [4]
few-shot classification. Some works [56] 46 find the well-generalizing representation is actually the key factor of few-shot ability, while the base dataset is typically not large enough to obtain it. In fact, adapting self- supervieee pre-trained models 6 76 65] to few-shot tasks can address all these concerns simultaneously: the training data is unlabeled and thus can be sufficiently large, and the novel tasks can come from different target domains. The pre-trained large models have far more generalizing representation and thus can obtain significantly better performance than the traditional setting, as shown in Appendix [B] Besides, it is more friendly privacy data, and we only need the pre-trained models instead of pre-training data. It also conforms to the learning paradigm of humans whose few-shot ability comes from learning on massive signals since birth, and has better realistic value. An exploratory work [9] uses the ensemble of frozen pre-trained models and a simple classifier, but without further adaptation to novel tasks. CLIP (55 introduces the vision- -aaguun pre-trained models, and some works 778 77 [0] use the text prompts to solve few-shot image classification based on them. These methods are limited to specific pre-trained models, and typically require the test classes to be seen in the training corpus [40]. Humans actually can recognize unseen objects based on only reference images, without knowing their class names. Our method is usable for both single and multi-modal pre-trained models. Similar to our work, some existing models (69) 79] also propose to highlight the key local features of each input for few-shot classification. However, they all design parametric modules and meta-learn them using the base dataset. It is not suitable for new setting, since the parametric modules can not been learned effectively using only few support samples. Our method instead introduces no new parametric modules and does not need meta-training.
Prompt engineering. Designing prompts 335 for adapting the pre-trained models to few-shot tasks originate from GPT-3 [5). which uses language prompts for in-context learning. Since the manual prompts require domain expertise and are likely to be sub-optimal even with significant effort, many focus on automating prompt generation to fine-tune pre-trained language models. The language prompts can also be used for few-shot image classification [78] 77 7 based on special vision-languse pre-trained models (45). VPT 26 introduces learnable prompts to the input space of each transformer layer, and is suitable for both single and multi-modal pre-trained models. But VPT performs unsatisfactorily on few-shot tasks, even worse than simple plug- -nd--aaddlll solvers 4 (2) with frozen backbone (Table D. Its prompts do not prompt any explicit information and still follow the form of language prompts. In this work, we propose position prompts for vision data explicitly prompt the key positions of input images to improve fine-tuning performance.
works 15
72 29
Deep explanation method. In order to build trust in intelligent systems, deep explanation methods [16] calculate the local relevancy of the input with model predictions, and can be used to locate the key parts. Many classic works have been designed for CNNs. such as gradient based methods 17]. For Transformers, these methods can not take advantage of the core attention information. We can use the raw attention scores to explain model predictions, but this ignores the attention components of other layers. Rollout M thus rolls out the attention scores of multiple layers to capture the propagation of information from input, but it is not class-specific. Inspired by Rollout, we also integrate the attention information from multiple layers, and further introduce gradient information to ensure class-specific explanation.
[51] [5) [5] 53 and relevance propagation based methods 3 38 25
Model
3.1 Problem formulation
In this work, we aim to adapt the pre-trained vision transformers to few-shot image classification tasks. These backbones can be columnar pyramidal architectures, e.g., ViT [I]] Swin [36], and
3

---

are pre-trained on massive data. In a few-shot classification task, we are typically given a support set T. as reference and query set Ty for test. The support set T. {( ) contains C² different classes with K labeled samples each class, which is denoted as "C-way K-shot' K is typically very small, e.g.. or 5, which makes it challenging task, especially for the data-hungry vision transformers. The query set T {x=] ((4],,))), 3 contains the samples belong the same classes with the support set, and need classify them correctly. Adapting a pre-trained model fg to few-shot task T = (Te, T can be formulated as
s = A(fo.
where the task solver A is designed to obtain the solution model sb of task T. For example, the task solver A can be linear probing [9] that combines the frozen model fo with linear classifier, or the that only tune partial existing or newly-added parameters for easy deployment, or using language prompts to generate classifier weights when fo is a vision-language pre-trained model (78) 77. The performance of solution t evaluated on the query set T with accuracy and on. In this work, we aim to design better task solver A.
parameter-efficient fine-tuning methods 34 24
3.2 Multi-head self-attention
The Multi-head Self-Attention (MSA) modules are the core components of various vision transformers. The input image is split into multiple patches, which are then projected to patch tokens. The learnable [CLS] token is appended to these tokens for final classification if needed. Let the input of a MSA module be Zin E R. containing N tokens with dimension d. Here Zin may contain all input tokens for global attention [[I], or partial ones for window attention [36]. MSA uses attention heads to jointly attend to the information from different representation sub-spaces, Within /-th attention head, we first project Zin to Query Qs, Key K,, and Value VA as
Q2 Zin W E Nxd.
K2 = Zin W E Nxd.
VA = Zz. W E Nxd.
(2)
where W q w and w are projection matrices. The self-attention matrix A₁ is calculated as
A softmax (0tttmmax(,,KK].... /v E N
The matrix A reflects the attention of different tokens to each other, and the large score A indicates the i-th token strongly attends to the j-th token. The output feature of the MSA module
MSA (Z..) Concat([A, V (=) W. + Zin.
where Concat(-) denotes the concatenation operation, W, is projection matrix. Obviously, the MSA module learns the increment of each token and introduces the information from other tokens, whose quantity is controlled by the attention matrices {Ap {h=
3.3 Analysis and motivation
We first examine various designs of the task solver A: 1) simple machine learning classifiers, e.g.. Nearest Neighbor classifier, Ridge
MotaOptNet
Regression and Support Vector Machines; 2) plug-and-play inductive
VPT
LORA
meta-solvvrs (e.g.. ProtoNet [52), R2D2 4 and MetaOptNet (32) and linear probing: 3) full or parameter-efficien fine-tuning, e.g.,
SSF
VPT 26 LoRA
24 and SSF
<4 The results provided
T where we use the ViT-B/16 pre-trained on ImageNet- 1 [49
with DINO 66] as fe and conduct on CUB 611, Cars [30], Places
(75 and Plantae (59] datasets. For each fine-tuning method, we use the best linear meta-solver, MetaOptNet, to obtain the initialization of the classifier weights. Otherwise, these fine-tuning methods perform even worse than MetaOptNet on few-shot tasks (Figure 2 As shown in Table [ the fine-tuning methods can achieve better performance than the backbone-frozen methods most cases. But they sometimes cannot significantly improve performance beyond the classifier initialization (i.e., MetaOptNet) or even perform worse, fact, the main challenges of few-shot tasks are: 1) the classes in the novel tasks could be unseen
Figure 2: Average 20-way ac- curacy CUB, Cars, Places and Plantae datasets. Here all fine-tuning methods the randomly-imitialzzed classifier. especially on 1-shot tasks. In

---

during pre-training phase; 2) there are only few (1 ~ 5) labeled support samples for each class in novel tasks. The imput images typically contain multiple entities. Although the pre-trained vision transformers may be able to separate different entities [6 [6). they do not pay enough attention to the key entities corresponding to the classes in the current few-shot task. Especially in the face of unseen classes, they may attend to multiple entities simultaneously [21]. When the support samples are sufficient, the model can attend to the frequently-occurinn this problem, as shown in Appendix B But the extremely few labeled samples prevent the fine-tuning methods from achieving this purpose, and the noise information from the class-independent entities misleads the fine-tuming process and harms the performance. For qualitative evidence, visualize the patches with top highest attention scores and covering about 95% attention of [CLS] token in Figure 4 more details and quantitative evidence at Section 4.2 As we can see, both the original model and fine-tuned one can not focus most attention on the key entities. To this end, we propose the position prompts to explicit prompt the model where are the key entities and guide the model to focus on them during fine-tuning. This ability then can generalize to the query samples.
key entities in each class more to alleviate
3.4 Position prompts
We need to locate the positions of key entities in each support image as prompts for fine-tuning, i... the most class-related key patches for the current few-shot task. Manual labeling may provide accurate positions, but requires domain expertise and labor costs. We aim to design an automatic method, which should I) be class-specife for locating class-related patches, and 2) make full use of the attention information, the core content of Transformers. Deep explanation methods [16] calculate the local relevancy of the input with model predictions, and can be used for highlighting the key patches for classification. For Transformers, Rollout []] finds that using the attention of top layer for explanation ignores most of the attention components, thus miscaleulating the importance of the patches, so we integrate the attention information from multiple layers like Rollout. Furthermore, only using the attention information cannot achieve class-specifie explanation for some pre-trained models (Figure 3 so we further introduce the gradient information as an auxiliary.
For columnar architecture MII with L layers, let the input feature map of (-th layer be Z',, E Nxd and the attention matrices be {A Mi=1- With the well-initialized classification head above, for support sample-label pair (xx, y²), we first compute the gradient of the prediction score for class y" Pyr, with respect to feature map Z!n as V /8Z'. E R- = L...., L. The gradient computed for single sample contains many noises, which could be amplified layer by layer along the back-propagation path. To this end, we only use the gradient to the input feature of the top layer, i.e. LL, and reserve its first principle component for denoise. The obtained gradient term is
(5) where svd(-) denotes Singular Value Decomposition (SVD) operation with V, USVT, and V, is the first column vector of matrix V, corresponding to the largest singular value of V.. The gradient term G can provide class-specific information. As shown in Equation the residual connection plays a important role in propagating information from inputs to model predictions and can preserve the position information during forward propagation. To this end, we use an identity matrix I to augment the attention matrix. Overall, the final attention graph of (-th layer is
G V VIL-VIE WIERN U,S. V svd(V
A' norm (I + +
+ G") E NXN
(6)
where norm(.) means to re-normalize the input so that the row sum is equal to 1, À can control the proportion of attention and gradient information, and we average the attention matrix over all heads, i.e., A' = 1, Σi= A/.- We assume the attentions are combined linearly and consider the path along the pairwise attention graph M.. The patch importance scores are calculated as
s mean( A A
A
E R
where (.) denotes matrix multiplication, mcan(-) denotes to average the row vectors of matrix. We use the positions of the top P patches with the highest importance scores as prompts.
For pyramidal architecture 36 70 since the downsampling operation destroys the patch position correspondence between layers, so we directly use the attention scores average on patches as the importance scores and introduce gradient information to achieve class-specific if necessary.
5

---

Table 1: Few-shot classification accuracy (%) 20-way 1-shot 5-shot tasks. We use the ViT-B/16 pre-trained on ImageNet- 49 with DINO for backbone initialization. 'NN', 'RR' and 'FT" denotes Nearest Neighbor classifier, Ridge Regression and full fine-tuning respectively.
Cars
Places
Plantae
Model
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
NN
52.8
73.0
20.9
36.2
49.5
64.3
35.0
54.3
RR
56.6
84.0
24.0
56.2
49.9
68.7
36.8
62.0
SVM
52.8
78.7
20.9
37.3
49.5
70.8
35.0
57.7
ProtoNet
52
52.8
79.8
20.9
39.3
49.5
71.2
35.0
57.7
R2D2
56.7
84.3
23.8
56.6
49.7
68.9
36.8
MetaOptNet
57.0
85.1
24.1
50.0
71.1
Linear Probing
41.9
78.2
18.3
47.2
41.0
65.8
27.2
56.6
VPT 26
52.9
81.1
23.3
54.5
48.0
69.6
33,9
60.2
FT
58.0
88.1
24.1
66.9
50.3
72.1
37.0
66.2
LoRA
24
57.9
88.2
23.3
64.3
49.9
71.3
37.1
65.7
SSF K4
57.8
88.4
23.8
62.3
50.2
73.4
37.2
66.0
FT + FORT
(1.5(
89.2 (1.1)
25.5 (1.4)
68.0
51.1 (0.8)
72.9 (0.8)
38.7 (1.7)
67.2 (1.0)
LoRA FORT
62.5 (4.6)
89.5 (1.3)
26.8 (3.5)
65.7 (1.4)
50.8 (0.9)
72.4 (1.1)
38.5 (1.4)
66.9 (1.2)
SSF FORT
62.3 (4.5)
89.6 (1.2)
26.5 (2.7)
64.2 (1.9)
(1.1)
74.4 (1.0)
39.0 (1.8)
67.5 (1.5)
3.5
Attention enhancement
The key patches contain the most class-related information of the input images, and the model should focus its attention them obtain discriminative representation in the current task, only for the support samples, but also for the query ones. Since the position prompts (or key patches) of the query samples are unknown without label information, unlike the existing prompts [35. our position prompts can not be used in the input or middle phase of forward inference. To this end, we use the position prompts the prediction target for attention, and optimize the cross-entropy loss between their many-hot presentation and attention logits during fine-tuning, As 3 given the input tokens (e.g.. [CLS] token and patch tokens) of the R RNxd, we obtain the Query Q {qn J Z!, W, and Key N, where W, Concat(( (ancatt[WW1kk H 1 and Wt Concat([Whlke k(A=1). The fine-tuning objective is
shown in Equation and
top layer / In J.
K [kn] ! Wa for calculating the attention logits {q.k)) /Vdx h. i.j = I,...,N,
N
exp (qnk?" /T) Σ exp (T,)
min
Σ (ce(fo(x)),))) a R), R N. [X0 JET
ΣΣ LEO
(8)
0
where 0 denotes tunable full or partial parameters of the model, ce(:,-) denotes cross-entropy classification loss, a is loss coefficient, is the index set of the key patches in x", is a temperature coefficient. We calculate the objective term R only at the top layer operate the features with more refine semantic. Specially, for the pyramidal architectures, R is applied to the top layer of each stage. This loss enables the model automatically identify the key patches and focus them during fine-tuning, and this ability then can generalize to the query samples. The position prompts are fixed during fine-tuning since updating them only obtains trivial improvement but introduces obvious cost.
Theoretical analysis. The key patches contain the most class-related information of the input and we want to increase their information in the output feature. We first prove that the objective term R
can increase the information from the key patches the input tokens Z (z., ) Concretely, we consider the mutual information T(2, 2n), where the random variable z presents the input token
with {z,) ) as samples, the random variable za presents the key patch with {2, E 2.} as its
samples. The InfoNCE lower bound estimate [42 44 of I [ (2,22) thus is
In f(zn, Zt)
InfoNNC((E,21) zn) N
S ((2,20)
J
NV Σm f(zn, Zm)
n=1
EE
where the critic f(..)) is used to model the density ratio and can be any positive real score function. When we use a big-bilinear function as critic, i.e.,
f(z,, 2m) exp (z., W, W 2 ///TT) exp (q kg
Vn,m m E (I....,N)
(10)
For index set Q. it is vector of length [S], with the values of the positions in Q be I and other ones be 0.

---

Table 2: Few-shot classification accuracy (%) 20-way 1-shot 5-shot tasks. We use the Swin-T/7 pre-trained on ImageNet-Il k with iBOT for backbone initialization.
CUB
Cars
Places
Plantae
Model
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
MetaOpiNet
49.1
81.1
21.0
51.1
48.9
70.4
36.0
63.1
FT
52.7
87.1
20.9
59.7
48.1
69.3
35.9
65.9
LORA
24
53.6
87.4
20.4
57.9
48.9
69.5
37.1
65.8
SSF 34
54.9
87.6
21.4
57.2
48.7
70.8
35.9
64.7
+ FORT
59.1 (6.4)
88.3 (1.2)
22.3
61.6 (1.9)
49.3 (1.2)
70.4 (1.1)
(1.3)
66.9 (1.0)
LoRA FORT
59.9 (6.3)
88.5
21.9
59.4 (1.5)
49.7 (0.8)
70.4 (0.9)
38.7 (1.6)
66.9 (1.1)
FORT
60.1 (5.2)
88.9 (1.3)
23.4 (2.0)
58.8 (1.6)
49.8 (1.I
72.0 (1.2)
(1.7
66.0 66.0 (1.3)
Table 3: Few-shot classification accuracy (%) 20-way 1 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on WTT 45] with CLIP for backbone initialization.
CUB
Cars
Places
Plantae
Model
1-shot
5-shot
1-shot
5-shot
I-shot
5-shot
1-shot
5-shot
MetaOptNer 32
68.0
88.7
67.6
90.7
51.6
73.8
46.2
73.0
zero-shot 45
84.1
84,1
88.0
88.0
76.6
76.6
61.2
61.2
CoOp 78
84.4
90.4
91.3
94.6
77.3
81.1
63,8
76.2
Tip-Adupter-F
73
86.9
92.0
92.2
95.3
79.8
82.0
68.3
79.3
PLOT+
87.4
92.0
92.2
95.5
79.9
82.7
67.7
78.8
LoRA 24
86.3
92.6
92.3
95.8
79.8
84.1
67.4
80.0
LoRA FORT
(1.5)
93.8 (3.2)
93.6 (1.3)
97.0 (1.2)
80.6 (0.8)
84.9 (0.8)
68.5 (1.1)
81.0 (1.0)
the objective term R satisfies
R (=noooncc(222 N
(11)
Therefore, increasing the objective term R can increase the mutual information between the input tokens and key patches. On the other hand, the objective term R can also be derived as
N
N
R= RA, R"
Σ
Σ
exp (4.
Σ
ΣEar AEO
1nnk (12)
N. [
Increasing RU reduces the similarity between input token pairs and obtains a more uniform dis- tribution of input tokens, which thus increases the entropy H2)). Increasing RA aligns the input tokens with the key patches and makes them more similar, which thus decreases the conditional entropy H (=]2n). Since [2, zn) (z)) Hz|aa)). the objective term R increase the mutual information [zz, =q). In fact, if p(z,, km) x exp (z,, W 4 W /T) ) n. m E [1,...,NN N}. R" and R.A have constant bias with the Monte-Carlo estimate of H(z) and - (=(2 respectively, see derivation in appendix. Finally, the objective term increases the information from the key patches in the output feature (c.g., [CLS] token or average patch tokens) from two aspects: 1) it increases the attention scores of the output feature to the key patches, which directly increases their contribution and also reduces the noise information from other patches, as shown in Equation 4 2) it increases the
information from the key patches in the input tokens, which further transports the output feature.
4
Experiments
4.1
Few-shot classification
Settings.
The experiments are conducted the few-shot benchmark from [58] (i.c., CUB [6], Cars
30
Places 77 and Plantae [59] datasets), and two other fine- nhe-gained datasets: Aircraft [39 and Pets
45
We use the pre-trained VYT-B/16 released by DINO (6) and CLIP 455 and Swin-T/7 released
by iBOT [76] for examining different architectures and pre-training ways. Here we only consider
contrastive pre-trained models (6 since their representations have better linear separability than
the generative pre-trained models [18 68 and are more suitable for few-shot classification. We evaluate both the bockbone-frozed methods, e.g., machine learning classifiers (Nearest Neighbor classifier, Ridge Regression and Support Vector Machine), meta-solvers (ProtoNet (52), R2D2 (4
7

---

Atfention
Attention Prompts
Attention-Grad Prompts
DING
Figure 3: The attention map of [CLS] token in the pre-trained ViT-B/16 from DINO and CLIP, and the position prompts (red patches) obtained using the attention w/ or w/o gradient information.
and MetaOptNet 32) and linear probing, and full or parameter-efffiient fine-tuning, e.g., VPT 226..
LoRA 24 and SSF 34 For the vision-language pre-trained model from CLIP, we also evaluate the -pesially-designed models: CoOp (78). Tip-Adapter 773 and PLOT+ 7. and use 'a photo of [CLASS]' as the prompts for classifier initialization in the fine- finentuing methods. The language prompts unusable for DINO and iBOT, number of the key patches P is set to 14 for ViT-B/16 and 7 for Swin-T77. We verify the effectiveness of our model, FORT, the fine-tuning methods. All models evaluated in the 20-way 1-shot / 5-shot settings, instead of overly-easy 5-way settings, using 2,000 randomly sampled tasks with 15 query samples per class, and we report the average accuracy. All experiments can be conducted on single Tesla V100 with 32GB memory. More fine-tuning details are provided in Appendix B where we also provide the results on 5-way settings for comparison with traditional setting.
use the classifier initialization based on MetaOptNet. The
Table 4: Few-shot classification accuracy (%) on 20-way 1-shot 5-shot tasks. We use the ViT- B/16 pre-trained on ImageNet-1K with DINO for backbone initialization.
Table 5: Attention-weighted summation of im- portance scores (x 10 3) on 20-way 1-shot / 5- shot tasks. We use the ViT-B/16 pre-trained on ImageNet-IK: with DINO for initialization.
Aircraft
Pets
1-shot
5-shot
Model
Model
I-shot
5-shot
I-shot
5-shot
support
query
support
query
MetaOptNet
32
25.5
55.5
71.9
89.6
Orig Atin
0.72
0.72
0.72
0.72
FT
27.1
64.6
71.6
89.0
FT
0.74
0.73
0.74
0.75
LORA 24
26.0
62.7
72.7
89.8
LORA 24
0.71
0.71
0.75
0.76
SSF 34
26.5
61.6
72.7
90.0
SSF 34
0.74
0.73
0.79
0.79
FT + FORT
28.7
66.0
72.6
89.9
FT FORT
0.80
0.77
0.90
0.91
LoRA + FORT
31.3
63.9
74.8
90.9
LoRA + FORT
1.00
0.97
0.96
0.97
SSF FORT
63.1
74.4
91.2
SSF FORT
1.06
1.03
1.04
1.04
Results. The results are provided in Table 1 2 3 and where the 95% confidence interval is +0.3
(1-shot) or +0.2 (5-shot) and the number in parentheses is the increment of accuracy beyond the base fine-tuning methods. As shown in Table 2 and with classifier initialization, the fine-tuning methods typically can achieve better performance than the model- methods, since the backbone further adapted to task distribution. However, fine-tuning sometimes struggles to achieve non-trivial improvement beyond the classifier initialization (i.e., MetaOptNet) or even performs worse, especially on the I-shot tasks. Our FORT can help the fine-tuning methods to obtain significant improvement on both ViT and Swin. Further, FORT typically achieves more improvement on 1-shot tasks than that on 5-shot tasks. This is intuitive, since with more support samples the model can attend to the frequently
entities more in each class, which are typically the key entities. FORT is more suitable for solving the hardest 1-shot tasks. As shown in Table on the vision-language pre-traine trained model, fine-tuning with simple classifier head (i.c., LoRA) achieves comparable better performance methods: CoOp, Tip-Adapter-F and PLOT++. In fact, Tip-Adapter-F
than the specifically-designee the ensemble of linear classifier head and the similarity- based label propagation module. PLOT++ introduces multiple learnable language prompts and visual prompts for obtaining multiple classifier weights and image embeddings, and then calculates their -traspo-t distance for classification, which has huge computational burden. Our FORT can further improve the performance beyond the fine-tuning methods. Finally, FORT is compatible with the parameter-efficien fine-tuning even better than with full fine-tuning, which is convenient for industrial deployment.
4.2 Ablation studies
8

---

Support
Query
FT
LoRA
SSP
Insut
Orig Alt
Am
Esprt
Cri Arta
Arm
Figure 4: Visualization of the patches (white parts) with top highest attention scores and covering about 95% attention of [CLS] token in ViT-B/16. We use DINO pre-trained model for initialization.
Position prompts. The different pre-training ways could lead different attention behavior in the vision transformers. Figure 'tttention' shows the attention map of [CLS] token in the top layer of pre-trained ViT-B/16 from both the single-modal (DINO) and multi-modal (CLIP) methods. DINO can attend to the foreground objects with the highest attention scores, while CLIP typically attends to more parts besides them. This is intuitive, since the supervision signals for the image encoder in CLIP are the sentence descriptions, which typically involve both the foreground objects and scene, e.g. 'a bird flies over the sea'. Importantly, the key entities typically can not get the highest attention scores in CLIP pre-trate -r-inraindd model, which makes it difficult to accurately locate them only based on the attention information. Figure 3 AAttention Prompts' shows the position prompts obtained using only the attention scores, i.... À = 0 in Equation For DINO, the obtained position prompts can indicate the key entities, while for CLIP, the obtained position prompts could point to other inaseddnde parts besides the key entities. To this end, we introduce the gradient information as an auxiliary. Figure 3 AAttentin++Grad Prompts' shows the position prompts obtained using both the attention and gradient information, where À = 1 for DINO and À = 50 for CLIP. We use MetaOptNet to calculate the classifier weight for DINO, and use language prompts for CLIP. The gradient information helps to accurately locate the key entities in CLIP, and also improve that in DINO.
Attention enhancement. We conduct qualitative and quantitative experiments to verify that our method can enhance the attention of the model to the key patches. Qualitatively, Figure 4 visualizes the patches with top highest attention scores of [CLS] token in ViT-B/16. Concretely, when the attention scores of all patches are ai,...,au v in descending order, we choose the patches (1...., n). which satisfy Sn- < 0.95 < S,, with S, = Ei= ag and SN = 1. We use the pre-trained model from DINO for backbone initialization and fine-tune on random 1-shot tasks. We provide the results of original attention ('Orig Attn') and the attention after fine-tuning w/ ("Our Attn') or w/o ('FT Attn') our method. The original model can not focus its attention on the key patches, nor can the full or parameter-efficttt the positions of key patches and help the model to focus most attention (95%) on them, and this ability can generalize from the support set to query samples. This phenomenon is not obvious on full fine-tuning, and the reason may be it uses smaller learning rate and less epochs than LoRA and SSF to avoid over-fitting. The full attention maps are provided in appendix and we can observe the same phenomenon. Further, we examine the ditentiio-weighhee summation of importance scores, ie.. Σm S; a,, where s is defined in Equation 7 and a is its rosition-aligned attention scores. The larger the value, the more focused the key patches the attention. Table 5 provides the results averaged on CUB, Cars, Places and Plantae datasets with 2000 tasks each setting. The fine-tuning methods obtain the similar scores with original attention, while increase. Consistently, the increment on full fine-tuning is smaller than that on LoRA and SSF For standard fine-tuning, the increment on 5-shot tasks is larger than that on I-shot tasks. In fact, with more support samples, the model can attend to the frequently-occurring key entities in each class.
fine-tuning achieve this property. Our method can prompt the model
FORT achieves non-trivial
Importance calculation. In Section 3.4 we combine both gradient and attention information for calculating the importance score of each patch. Here we ablate the different information item and compare their performance. We follow the experimental settings in Section 4.1 and use the per-trained models from DINO or CLIP for initialization. The results are provided in Table 6 where 'FORT (attn)' denotes using only attention information (i.e., setting À in Equation 6). 'FORT (grad)' denotes using only gradient term G, "FORT (attn+grad)" denotes using both attention and gradient information, that is our final method. Here we pick the best hypee-parameters for each method, even though they may differ from each other. For DINO pre-trained models, the attention information

---

Table 6: Few-shot classification accuracy (%) on 20-way 1-shot 5-shot tasks. We use the ViT-B/16
pre-trained by DINO or CLIP for backbone initialization.
DINO
CLIP
Model
CUB
Cars
CUB
Cars
1 1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
LoRA
57.9
88.2
23.3
64.3
86.3
92.6
92.3
95.8
FORT (attn)
62.1
89.0
26.3
65.0
86.6
92.6
92.5
95.8
FORT (grad)
60.0
88.8
25.2
64.7
87.8
93.8
93.6
97.0
FORT (attn+ggrad)
62.5
89.5
26.8
65.7
87.8
93.8
93.6
97.0
2010
tunt Surs
Satel Lot4 Sotel
Late:
lahel
-9- 3000 S-nnt
PT Sohal
-V- 52 Sohe.
6,cccurayy
Acccuma
20
(a)
(b)
(c)
Figure 5: Ablation studies hyper-parameter X, and We examine the absolute increment of accuracy beyond the base fine-tuning methods averaged on CUB, Cars, Places and Plantae datasets.
can effectively locate the key patches and contributes more to the performance than the gradient information. For CLIP pre-trained models, the attention information can not indicate the positions of the key patches and thus the gradient information plays a major role. Combining both attention and gradient information works for different pre-trained models and can obtain the best performance.
We conduct ablation studies on the hyper-parameters from Equation 6 and 8 X, 7 and a, and examine the absolute increment of accuracy beyond the base fine-tuning methods averaged on CUB, Cars, Places and Plantae datasets. For each its corresponding best values. For X, we use LoRA the base fine-tuning method and evaluate different pre-trained models, i.e., ViT-B B/16 from DINO and CLIP. For and a, we use the ViT-B/16 from DINO and evaluate different base fine-tuning methods, i.e., full fine-tuning, LoRA and SSF Figure 5 (a) shows that using DINO pre-trained model is suitable for small À since its attention is enough for locating key patches, while using CLIP pre-trained model needs large utilize gradient information. Figure (b) shows that our method is generally robust to the temperature coefficient
we set the other ones
T, and the performance on 1-shot tasks is relatively more sensitive than that on 5-shot tasks. Figure and needs to be carefully
5 (c) shows that the loss coefficient is an important dyper-paramedd chosen, and the performance on 5-shot tasks is more sensitive than that on 1-shot tasks. fact, the coefficient is too large, the attention enhancement loss could relatively inhibit the optimization of the classification loss (c.g., skew gradient direction), which is harmful to fitting the input-label distribution. The more training samples, the more important it is to fit input-label distribution, the 5-shot tasks prefer smaller coefficient a than I-shot tasks.
5 Conclusion and limitations
In this work, we aim to adapt the pre-trained vision transformers to few-shot image classification tasks. It is nontrivial, since even with well classifier initialization, the standard fine-tuning methods sometimes can not obtain significant improvement beyond the initialization, especially on 1-shot tasks. To this end, we propose a new prompt for vision data, position prompts, to explicitly prompt the model where are the key entities and focus on them during fine-tuning. Our method can effectively improve the performance of full or parameter-efficient fine-tuning on few-shot tasks. The limitations of our methods are: 1) our method is suitable for multi-head attention module, and unusable for other vision backbones without attention, e-g., ResNet (00 or MIP-Mixer [57]: 2) for some professional fields, e.g., medical imaging, may still need experts to manually label the position prompts.
10

---

References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
[2] Marcin Andrychowicz. Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
[3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-
Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by
layer-wise relevance propagation. PloS one, 10(7):e01 (000=e111110000 30140, 2015.
[4] Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprini /rXiv::805 00000....50 08136,,1000 2018.
[5] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660 2021.
[7] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. PLOT:: Prompt learning with optimal transport for vision-language models. In The Eleventh International Conference on Learning Representations, 2023.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[9] Arkabandhu Chowdhury, Mingchao Jiang, Swarat Chaudhuri, and Chris Jermaine. Few-shot image classification: Just use library of pre-trained feature extractors and simple classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9445-9454,
2021.
[10] Kun Ding. Ying Wang. Pengzhang Liu, Qiang Yu, Haojian Zhang. Shiming Xiang, and Chun- hong Pan. Prompt tuning with soft context sharing for vision-language models. arXiv preprint arXiv:2208. 13474, 2022.
[1I] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.123 11929, 2020.
[12] Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE
Iransactions on pattern analysis and machine intelligence, 28(4):594-611, 2006.
[13] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Mooning-Volume Learning- Volume 70, pages 1126-1135. JMLR. org, 2017.
[14) Yuqian Fu, Yu Xie, Yanwei Fu, and Yu-Gang Jiang. Meta style adversarial training for cross- domain few-shot learning. arXiv preprint arXiv:23003 302.09309, 2023.
[15] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, 2021.
[16] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(55:1-42. 2018.
[17] Shir Gur, Ameen Ali, and Lior Wolf. Visualization of supervised and self-supervised neural networks via attribution guided factorization. In Proceedings of the AAAI conference on artificial intelligence, 2021.
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
11

---

[21]
Markus Hiller, Rongkai Ma, Mehrtash Harandi, and Tom Drummond. Rethinking generalization in few-shot classification. In Advances in Neural Information Processing Systems. 2022.
[22]
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence,
44(9):5149-5163 2021.
[23] Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning, In 7th
International Conference on Learning Representations, 2019.
[24] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, -Zuu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022.
[25] Brian Kenji Iwana, Ryohei Kuroki, and Seüchi Uchida. Explaining convolutional neural networks using softmax gradient layer-wise relevance propagation. In 2019 IEEE/CVF In- ternational Conference on Computer Vision Workshop (ICCVW), pages 4185-4185 IEEE,
2019.
[26] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII. pages 709-727.
Springer, 2022.
[27] Shibo Jie, Haoqing Wang, and Zhi- Zhi-Hong Deng. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17217-17226, 2023.
[28] Siavash Khodadadeh, Ladislau Bölöni, and Mubarak Shah. Unsupervised meta-learning for few- shot image classification. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' AlAché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information P Processing Systems 32: Annual Conference on Neural Information Processing Systems, pages
10132-10142, 2019.
[29] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2023.
[30] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[31] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266): 133--1333 2015. [32] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10657-10665, 2019.
[33] SuBeen Lee, WonJun Moon, and Jae-Pil Heo. Task discrepancy maximization for fine- grained few-shot classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5331-5340, 2022.
[34] Dongze Lian, Zhou Daquan, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features:
A new baseline for efficient model tuning. In Advances in Neural Information Processing Systems, 2022.
[35] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35. 2023.
[36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings
of the IEEE/CV international conference on computer vision, pages 10012-10022, 2021. [37] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[38] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions.
Advances in neural information processing systems, 30, 2017.
[39] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- 2013.
grained visual classification of aircraft. arXiv preprint
[40] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023.
[41] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint urXiv:I803 02999, 2018
[42] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation lcarning with contrastive predictive coding, arXiv preprint arXiv:1807.0374. 037448, 2018
12

---

[43]
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3500 -3088. IEEE, 2012.
[44] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, PMLR, 2021.
pages
[46] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of MAML. In 8th International Conference on Learning Representations, 2020.
[47] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In Interna- tional conference on learning representations, 2017.
[48] Aniket Roy, Anshul Shah, Ketul Shah, Prithviraj Dhar, Anoop Cherian, and Rama Chellappa. Felmi: Few shot learning with hard mixup. In Advances in Neural Information Processing Systems, 2022.
[49]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 (15(3):21--255) 2015.
[50] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[51] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pages 31 3145-3153. PMLR, 2017.
[52] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087, 2017.
[53] Suraj Srinivas and François Fleuret. Full- mull-gradient representation for neural network visualization.
Advances in neural information processing systems, 32, 2019.
[54] Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv. Hicrarchical attention prototypical networks for few-shot text classification. In Proceedings of the 2019 conference empirical
methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 476-485, 2019.
[55] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning. pages 3319-3322 -328.. PMLR, 2017.
[56] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision-ECCV
2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XIV 16, pages 266-282. Springer, 2020.
[57] Ilya O Tolstikhin, Neil Houlsby. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261-24272, 2021.
[58] Hung- Yu Tseng, Hsin- Ying Lee, Jia- Bin Huang, and Ming- Hsuan Yang. Cross-domain sew- classification via learned feature-wist transformation. In 8th International Conference on Learning Representations, ICIR 2020, 2020.
[59] Grant Van Horn. Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 8769-8778, 2018
[60] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638,
2016.
[61] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200- 200--001 dataset. California Institute of Technology, 2011.
[62] Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task augmentation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 2021.
13

---

[63]
Haoqing Wang and Zhi- Hong Deng. Fhot learning with Issvm base learner and transductive modules. arXiv preprint 2xiv:2009.005886 2020.
[64]
Haoqing Wang and Zhi-Hong Deng. Contrastive prototypical network with wasserstein confi- dence penalty. In European Conference on Computer Vision, pages 665-682. Springer, 2022. Haoqing Wang, Xun Guo, Zhi- Shi- Deng, and Yan Lu. Rethinking minimal sufficient repre- sentation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16041-16050, 2022.
[65]
[66] Haoqing Wang, Huiyu Mai, Zhi-hong Deng, Chao Yang, Luxia Zhang, and Huai-yu Wang. Distributed representations of discases based on co-occurrence relationship. Expert Systems with Applications, 183:1 15418, 2021.
[67] Haoqing Wang, Huiyu Mai, Yuhang Gong, and Zhi-Hong Deng. Towards well-generalizingg meta-learning via adversarial task augmentation. Artificial Intelligence, page 103875, 2023.
[68] Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[69] Peng Wang. Lingqiao Liu, Chunhua Shen, Zi Huang, Anton Van Den Hengel, and Heng Tao Shen. Multi-attention network for one shot learning. In proceedings of the IEEE conference on computer vision and pattern recognition, pages 2721-2729, 2017.
[70] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 568-578, 2021. 2021.
[71] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1334, 2020. [72] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre- pre-trained language models better few-shot learners. In The Tenth International Conference on Learning Representations, 2022.
[73] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training free adaption of clip for few-shot classification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 493-510. Springer, 2022.
[74] Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
[75] Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places; A
10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, (466::452-1400) 2017.
[76] Jinghao Zhou, Chen Wei. Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Repre- sentations. 2022.
[77] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-languar models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[78] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9)22337-2333 22344888,, 2022. [79] Yaohui Zhu, Chenlong Liu, and Shuqiang Jiang. Multi-attention meta learning for few-shot fine mme-grained image recognition. In IJCAI, pages 1090-1096, 2020.
14

---

Table 7: Candidate values of hyper-parameters for fine-tuning.
hyper-parameter
value
learning rate #epoch
le-5, 3e-5, 5e-5, 8e-5, le-4, 3e-4, 5e-4, 8e-4, le-3, 3e-3, 5e-3, 8e-3, le-2 10, 15, 20
temperature
1,5, 10, 15, 20 5e-4, 8e-4, 1e-3, 3e-3, 5e-3, 8e-3, 1e-2, 3e-2, Se-2, 8e-2, 1e-1, 8e-1,
coefficient a
A Derivation of Monte-Carlo estimate
In Section 3.5 fl...., N). R and R have constant bias with the Monte-Carlo estimate of (z) and - - (2120) respectively. where
claim that if p(zn km) exp (z,, W, W/ (q, /r), n,
N
N
RT
In
exp (qnkg
RA
Ank /T
(13)
N
N
|
m=1
1=
tE
We define
p(z|8)= exp (2 W W /T)
p(2lan) exp (N W. W 2
(14)
2(
2(zn)
N where the random variable å follows the same distribution as and also has (z., m= as its some samples, and Z(.) is the partition function for normalization. Here we assume that the energy state remains the same for different 2 or 2p, i.e,, Z(.) keeps a constant Z. The detailed derivations are
H(z) = Ep(z) In p(2(2) Ep(=) In E (2))(((2))
In E (p)))(((213)
In plzn Zm
ln Σ cxp (qn (qnk2T/7) + +
N ==1
m=1
~ R + In NZ
(15)
and
- -H(2(zn) Ep (22(222))
In Inp(z, 1z))
N [
n= tEf
knk? / In Z
[2
R In
(16)
From these results, the above claim is proved. Besides, we have z() = H(z) H=222n) s RA In N= R
R
In N, i.e., obtain the Equation 11 again.
B More experiments
The resolution of input images is 224 x 224, and we do not use data aug- mentation following the standard setting. In fact, we find the base data augmentation (e-g., random resized cropping, flip or random erasing) can not obtain improvement the chosen datasets of our
Fine-tuning details.
15

---

Support
FT
LoRA
SSP
Imput
Atts
Esprt
buttoar)
Figure 6: Visualization of the patches (white ones) with top highest attention scores and covering about 95% attention of [CLS] token VIT-B/16. We use DINO pre-trained model for initialization.
LoRA
SSF
Input
Orig Altn
FT Attn (rutiion)
Figure 7: Visualization of the patches (white ones) with top highest attention scores and covering about 95% attention of [CLS] token in ViT-B/16. We use DINO pre-trained model for initialization.
experiments. We do not consider the pre-trained backbones from masked image modeling models for initialization, e.g., BEiT 76 or MAE [8], since these generative pre-training models can not obtain highly linearly separable features and perform poorly on few-shot tasks. We set the rank to for LoRA (24] and use 10 learnable prompts at each layer for VPT (26). We mainly use AdamW SGD optimizer with weight decay of le-3. We set the batch size to 20, same as the number of classes, for both 1-shot and 5-shot tasks. We set À 1 for DINO pre-trained model and X 50 for CLIP pre-trained model for simplicity. The other hyper-parameters, including learning rate, number fine-tuning epochs, temperature and coefficient a, changeable for different fine-tuning methods, with without classifier initialization and different datasets. To this end, we only provide their candidate values in Table 7 and choose them according to the average accuracy on 50 randomly sampled few-shot tasks. We only use 50 tasks for quick hyper-parameter selection. Considering the confidence intervals on random 50 tasks are too large to be accurate indication of performance, e.g., 1.7,, we fix the random seed so as to compare the performance of different tasks. We first determine the learning rate and the number of epochs for the base fine-tuning methods, and then determine the temperature т and the coefficient a for our method.
37 optimizer for fine-tuning, except for linear probing and VPT where
on the same 50 few-shot
Attention maps with sufficient training samples. We follow the experimental settings of Figure 4 except fine-tuning the 20-way 50-shot task from CUB and 20-way 200-shot task from Pets without attention enhancement. The attention maps after fine-tuning are shown in Figure 6 and Figure 7 as "FT Attn (sufficient)". As we can see, fine-tuning with sufficient samples indeed focuses on the key entities more than with few samples ("FT Attn") or original attention ("Orig Attn"), ) but is less focused than explicit attention enhancement ("Our Attn'). Note that this does not mean that few-shot fine-tuning with attention enhancement can obtain higher accuracy than sufficient-shi fine-tuning. In fact, sufficient training samples allow the models fit the input-label distribution better, which is more important for classification than attention enhancement. But when we can not obtain sufficient samples, attention enhancement can reduce hass-independent noise information in the final embedding as shown experimentally and theoretically, and thus is useful for classification.
Performance on 5-way tasks. In order to facilitate comparison with traditional few-shot learning setting and show the advantages of our new setting, we provide the results on 5-way 1-shot / 5-shot tasks in Table The performance are far more better than traditional few-shot learning setting. On minilmageNet, we can achieve 94.2% accuracy for 1-shot tasks and 98.5% 5% accuracy for 5-shot tasks, while the SOTA traditional model, FeLMi [48), only achieve 67.47% and 86.08% accuracy for 1-shot
16

---

Table 8: Few-shot classification accuracy (%) on 5-way 1-shot / 5-shot tasks. We the VIT-B/16
pre-trained on ImageNet-Ie Ik with DINO for backbone initialization.
minilmageNet
CUB
Cars
Pets
Model
1-shot
5-shot
1 I-shot
5-shot
1-shot
5-shot
1-shot
5-shot
LoRA
93.1
97.8
80.1
94.7
43.3
74.9
88.7
95.9
LoRA + FORT
94.2
98.5
84.4
96.2
45.2
76.1
89.9
96.8
SSF
92.9
97.5
78.1
94.1
43.5
74.7
89.4
96.5
SSF FORT
94.2
98.1
81.8
96.4
46.2
76.6
91.2
97.1
Support
Query
FT
LoRA
SSP
lisut
Orig Alln
FT Amm
Oue Ani
Esput
Origg Ana
FT Ama
Our Arta
Figure 8: Visualization of the attention map of [CLS] token the top layer of ViT-B/16, We use the ViT-B/16 pre-trained on ImageNet-IK with DINO for backbone initialization.
and 5-shot tasks respectively. Besides, although TDM [33 can achieve comparable performance on CUB with our method, it far lags behind our method on Pets, i.c. 55.8% 91.2% on 1-shot tasks and 72.5% vs 97.1% on 5-shot tasks. This means that TDM can not obtain stable good performance while our method can achieve satisfactory performance on different datasets and are more general. In fact, the traditional setting which uses base dataset is not inherently a true few-shot setting. want to obtain a few-shot model for CUB, we have to collect many labeled bird images for meta-training, which still needs massive labeling labor and seriously hinders the practical application. Importantly, for different domains (e.g., CUB, Cars), we have to label separate base datasets for them. Our new setting uses open-source large models pre-trained on massive unlabeled data and only uses the few support samples from different target domains for fine-tumin tne-tuning. which is a true few-shot setting and can handle different domains simultancously. Using pre- pre-trained large models is more effective and practically valuable than traditional setting. Finally, our method can further improve the fine-tuning performance of base fine-tuning methods, e.g., LoRA and SSF, even 5-way tasks.
Attention map of different fine-tuning methods. In Figure 4 we choose the patches with top highest attention scores and covering about 95% attention for visualization. Here provide the full attention map of original attention ('Orig Attn') and that after fine-tuning w/ ('Our Attn') or w/o ('FT Attn') our method in Figure 8 The settings the same, i.e.,
the ViT-B/16 pre- trained on ImageNet-1K KK [49] with DINO for backbone initialization and fine-tune on random 1-shot tasks. Neither the original nor fine-tuned models
focus their attention on the key entities. Our
method can obviously make the model to focus on the key entities and reduces the attention to the class-independent parts. This ability can generalize from the support to query samples.
Fine-tuning without classifier initialization. In Figure 2 we compare MetaOptNet 332 with full
or parameter-efficien fine-tuning methods, i.c., VPT 26 LoRA 24 and SSF [34] average accuracy on CUB [61]. Cars [30], Places (75) and Plantae [59] datasets. Here we provide detailed results in Table S where use the ViT-B/I6 pre-trained on ImageNet-IK
49 with DINO for
backbone initialization and the 95% confidence interval is +0.3 (1-shot) or 0.2 (5-shot). As we can see, all fine- tne-tuning methods perform worse than the backbone-frozen MetaOptNet on few-shot tasks. The extremely few labeled samples can not simultaneously lear good classifier and adapt the backbone to the data distribution. fact, the randomly initialized classifier back-propagates noisy gradients, which destroys the knowledge learned during pre-training and also fails to adapt new data distributions. Further, this phenomenon also shows it is very diffieult to learn new parametric modules from scratch on few-shot tasks, c.g., classifier head, and need to construct the well-
17

---

Table 9: Few-shot classification accuracy (%) 20-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on ImageNet- IK with DINO for backbone initialization. 'FT" denotes full fine-tuning. Here all fine-tuning methods use the eandomly-nittiaizeed initializzd classifier.
CUB
Cars
Places
Plantae
Model
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
MetaOptNet
57.0
85.1
24.1
57.9
50.0
71.1
36.9
63.7
VPT 126
38.3
73.3
17.5
43.1
35,9
64.6
25.1
54.1
FT
25.0
66.7
15.3
42.2
23.6
55.0
22.0
51.4
LoRA 224
55.3
83.6
22.5
54.7
48.4
68.2
34.2
62.1
SSF 34
54.8
83.4
22.6
53.9
47.9
69.4
33.7
61.2
initialization for them. Considering MetaOptNet is a linear solver with excellent performance, we use to calculate the classifier weights as initialization, which significantly improves the performance of various fine-tuning methods. Since the prompts in VPT are learned from scratch and we can not obtain their initialization, VPT still underperforms MetaOptNet on few-shot tasks even with classifier initialization. We recommend using linear solver (e.g., ProtoNet [52), R2D2 [ or MetaOptNet (32)) to obtain a good initialization of the classifier as a necessary preemptive operation when adapting pre pre-trained models to few-shot classification tasks in the future.
Patch attention. In the main text, we only examine the attention of the [CLS] token at the last layer. Since method calculates the attention enhancement loss on both the [CLS] and patch tokens, here we visualize the attention map of those tokens at the penultimate (11-th) and last (12-th) layer of ViT-B/16. We use the ViT-B/16 pre-trained on ImageNet- 1K with DINO for backbone initialization, and fine-tune the model using LoRA or SSF and evaluate the effect our method on them. We choose the [CLS] token and {14, 28, 42, 56, 70, 84, 98, 112. 126, 140, 154, 168. 182, 196) --th patches for visualization and the results are shown in Figure 9 After fine-tuning with LoRA or SSF, both the [CI.S] and patch tokens still attend the other parts besides the key entities, introducing class- independent noise information, and some patch tokens even attend to those parts more than the key entities. Our method enhance both the [CLS] and patch attention to make them focusing the key entities. Interestingly, although we do not calculate the attention enhancement loss at the penultimate layer, the attention of its [CLS] and patch tokens are still significantly enhanced and focus on the key entities. As the theoretical analysis in Section 3.5 our method can increase the mutual information between the input tokens of the last layer (i.e., the output tokens of the penultimate layer) and the key patches. In order to achieve this objective, the [CLS] and patch tokens of the penultimate layer should also pay more attention the key patches to introduce more information from them.
18

---

Support
LeRA
Layer-1l
LeRA
Layer-12
Query
LeRA
Layer-El
-Our
LeKAA
-Our
[1:8]]
1hach-24
Paich-42
Paich-te
Paich-00
Paich-44
LoRA
Support
-Our
Layer-12
Query
SSY
Layer-11
Layer-12
-Our
Pasto14 Past-25
Pack-42
Prat-112
Paut-12
Pact-154
Pach-133
Pach 1SS
(b) SSF
Figure 9: Visualization of the attention map of the [CLS] token and different patches at the penultimate (11-th and last (12-th) layer of ViT-B/16. Here we use the ViT-B/16 pre-trained on ImageNet- IK with DINO for backbone initialization and evaluate the effect of our method on LoRA and SSF.
19


Reviews:
[{"review_id": "nXXqpMoMmY", "reviewer": "Reviewer_8eth", "summary": "This paper proposes to force the pre-trained model to focus on class-related entities for few-shot image classification. To achieve this, it proposes position prompts that use attention and gradient information to automatically locate the positions of key entities. An attention enhancement loss is used to be trained with the cross-entropy loss. Experiments are conducted on CUB, Cars, Places, Plantae, Aircraft, and Pets datasets.", "questions": "Please see the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "\n+ The idea to focus attention for few-shot learning is quite interesting.\n+ This paper is well-organized, with intuitive figures which illustrate the motivation and approach clearly. \n+ The experiments are extensive. \n", "weaknesses": "+ The statement in Ln168 “When the support samples are sufficient, the model can attend to the frequently-occurring entities in each class more to alleviate this problem, since they are typically the key ones” can’t be well-supported by Fig. 4. Instead, attention maps with **sufficient training samples** should be included to prove that the “attention focus” is especially useful for few-shot learning. \n+ More illustration of the hyper-parameters $\\alpha$ should be included as it controls the core component of the proposed approach, the attention enhancement loss. For example, why does the performance decrease with higher $\\alpha$; why is the performance of 5-shot tasks more sensitive to the value? If as stated in Ln168, more samples would be less sensitive to the attention selection?\n+ The illustration of the position prompts is unclear and quite difficult to understand, which makes it difficult to evaluate its effectiveness.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please check the reference rules in the global response first to help following reading.\n\n**1.attention maps with sufficient training samples**\n\nWe follow the setup of Fig.4-main, except fine-tuning on the 20-way 50-shot task from CUB and 20-way 200-shot task from Pets without attention enhancement. The attention maps after fine-tuning are shown in Fig.1-rebuttal and Fig.2-rebuttal as “FT Attn (sufficient)”. As we can see, fine-tuning with sufficient samples indeed focuses on the key entities more than with few samples (“FT Attn”) or original attention (“Orig Attn”), but is less focused than explicit attention enhancement (“Our Attn”).  \nNote that this does not mean that few-shot fine-tuning with attention enhancement can obtain higher accuracy than sufficient-shot fine-tuning. In fact, sufficient training samples allow the models to fit the input-label distribution better, which is more important for classification than attention enhancement. But when we can not obtain sufficient samples, attention enhancement can reduce class-independent noise information in the final embedding as shown experimentally and theoretically, and thus is useful for classification.\n\n**2.more explanation about hyper-parameter alpha**\n\nAs an auxiliary of classification loss, the attention enhancement loss aims to guide the model to learn to classify the images by focusing on their key entities, so as to ignore class-independent information. Both the attention enhancement loss and the classification loss help to get better classification boundaries, but the latter one fits the input-label distribution and thus is more direct and effective. The attention enhancement loss aims to reduce noise when data is scarce and indirectly improves the generalization performance of the classifier. Therefore,  \n(1) If the coefficient alpha is too large, the attention enhancement loss could relatively inhibit the optimization of the classification loss (e.g., skew gradient direction), which is harmful to fitting input-label distribution.  \n(2) The more training samples, the more important it is to fit input-label distribution, so 5-shot tasks prefer smaller alpha than 1-shot tasks.\n\n**3.about position prompts**\n\nFor each input image, there are some patches corresponding to the key class entity, denoted as key patches. Their positions are used as “position prompts” to guide fine-tuning, i.e., the red patches in Fig.3-main.  \nAs described in Section_3.4-main, we use both attention and gradient information to locate position prompts for generality. In fact, it is difficult to locate them using only attention information sometimes, as shown in Fig.3-main. “Attention+Grad Prompts” is generally better than “Attention Prompts”.  \nAs described in Section_3.5-main, we use the position prompts as the prediction target for attention module, and optimize the cross-entropy loss between their many-hot presentation and the attention logits during fine-tuning, i.e., Eq.8-main. Unlike the existing prompts in NLP[a], our position prompts are not used in the input or middle phase of forward inference, since those of query samples are unknown without label information.  \nFinally, their effectiveness is quantitatively verified in Tab.1,2,3,4-main and qualitatively verified in Fig.4-main. The attention enhancement using position prompts indeed helps the model to focus most attention on key entities.\n\n[a] pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing."}, {"review_id": "cx5BzIuMD5", "reviewer": "Reviewer_TpQg", "summary": "- The paper adapts pre-trained vision transformers for few-shot classification. \n- Full/parameter-efficient fine-tuning using only a few examples may harm performance due to spurious correlations. \n- The proposed method uses an additional auxiliary loss to guide the attention of the top layer to focus on the class-related patches. \n- The target for this loss (“position prompts“) are obtained as a combination of deep explanation method (e.g. Rollout) and gradient information (of class logit wrt top layer input features). \n- Theoretical analysis shows how the proposed auxiliary loss increases mutual information between the input tokens and key class-related patches. \n- Experiments on CUB, Cars, Places, Plantae datasets show improvements over full/parameter-efficient fine-tuning methods and meta-learning based methods. ", "questions": "- DINO attention in Fig3 looks much better than Orig attention in Fig4. Is there a difference in how they are obtained? \n\n- How do the position prompts look when only using gradient term in Eq6 ?\n\n- Above points from the Weaknesses section", "limitations": "- The authors have discussed limitations:\n    - The method is not applicable for non-attention based network architectures \n\n    - Obtaining the position prompts may need manual labeling from experts for domains like medical, satellite, etc\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is written well and is easy to follow.\n    \n- The proposed Attention+Grad target gives better prompts than Attention alone (Fig3). (It would be good to see that it gives better prompts than Grad alone as well, or whether Grad is enough to get good position prompts.)\n    \n- Qualitative results of the resulting attention (Fig4) demonstrates improvements visually.\n    \n- Ablation wrt key hyper-params help understand their effect in different settings. (Please see Weaknesses for missing ablations)\n    \n- Theoretical analysis provides additional justification of the proposed approach.", "weaknesses": "- The most commonly studied settings in related work are 5-way 5-shot and 5-way 1-shot. It would be valuable to show results in these settings in order to compare with more recent methods than the ones in the paper. [a, b, c]\n    \n    - The used 20-way setting is more challenging, though it restricts comparisons with recent work\n        \n- Recent methods [a, b, c] also show results on the larger miniImageNet and tieredImageNet few-shot benchmarks\n    \n- Current SOTA pre-training methods are based on MAE. How does this method work when fine-tuning MAE pre-trained ViT?\n    \n- Cross-domain results to demonstrate that the method generalizes to unseen domains (as in [59] whose experimental setup is used)\n    \n- Ablation experiment using only the gradient term in Eq 6 to show the relative contribution of A\n\n- Missing comparisons and mention of closely related work [d, e]\n\n[a]: Roy, Aniket, et al. \"FeLMi: few shot learning with hard mixup.\" Advances in Neural Information Processing Systems 35 (2022)  \n[b]: Yiren Jian, et al. \"Label hallucination for few-shot classification.\" In Proceedings of the AAAI Conference on Artificial Intelligence (2022)  \n[c]: Afrasiyabi, Arman, Jean-François Lalonde, and Christian Gagné. \"Associative alignment for few-shot image classification.\" Computer Vision–ECCV 2020  \n[d]: Hou, Ruibing, et al. \"Cross attention network for few-shot classification.\" Advances in neural information processing systems 32 (2019)  \n[e]: Jiang, Zihang, et al. \"Few-shot classification via adaptive attention.\" arXiv preprint arXiv:2008.02465 (2020)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please check the reference rules in the global response first to help following reading.\n\n**1.difference between the attention visualization in Fig.3-main and Fig.4-main**\n\nFig.3-main visualizes the attention scores where the different colors represent different values, while Fig.4-main chooses the patches with top highest attention scores and covering about 95% attention for visualization where the chosen patches have the same color (i.e., white) no matter the attention scores. This can more clearly show where the models pay most attention to. Besides, the attention score visualization corresponding to Fig.4-main is provided in Fig.1-appendix.\n\n**2.position prompts using only gradient information**\n\nWe follow the setup of Fig.3-main to visualize the position prompts using only gradient information. The results are shown in Fig.3-rebuttal as “Grad Prompts” and the other ones are the same as Fig.3-main. As we can see, using only gradient information also can effectively locate the position prompts, but there could be some bias due to gradient noise.\n\n**3.results on 5-way 1-shot/5-shot tasks and comparison with [a,b,c]**\n\nIn this work, we abandon the traditional setting that learns inductive bias from a small base dataset, and instead propose a new paradigm that adapts the models pre-trained on massive data to target few-shot tasks. The new paradigm has many advantages: 1) the pre-trained large models have far more generalizing representation and thus can obtain significantly better performance; 2) it can handle supervised, cross-domain and unsupervised few-shot learning simultaneously: the pre-training data can be unlabeled and thus sufficiently large, and the novel tasks can come from different target domains; 3) it is more friendly to privacy data, and we only need the pre-trained models instead of pre-training data; 4) it conforms to the learning paradigm of humans whose few-shot ability comes from learning on massive signals since birth, and has better realistic value. Besides, we use the 20-way setting since it is more challenging and practically useful, and the 5-way setting is too simple.  \nWe provide the results on 5-way tasks in Tab.1-rebuttal. The results are far more better than those in [a,b,c], e.g., for 1-shot tasks from miniImageNet, the best accuracy of [a,b,c] is 68.28% and we can achieve 94.2%; for 1-shot tasks from CUB, [a] achieves 51.66% and we achieve 84.4%. This means using pre-trained large models is more effective and practically valuable than traditional paradigm [a,b,c]. Our FORT can also improve the performance of fine-tuning methods on 5-way tasks. Besides, [a,b,c] aim to pseudo-label the base dataset as auxiliary data, and this is not suitable for our setting. First, we may not obtain the pre-training data due to privacy. Second, even the pre-training data can be obtained, it is still too cost to pseudo-label the massive data, not like the small base dataset.\n\n**4.fine-tuning MAE pre-trained models**\n\nAlthough the MAE pre-trained models can obtain SOTA fine-tuning performance in dense prediction tasks, e.g., detection and segmentation, their can not obtain highly linearly discriminant representation like DINO and are not suitable for few-shot image classification. For example, fine-tuning ViT-B/16 from MAE with LoRA can only obtain 12.4% and 51.2% accuracy on 20-way 1-shot and 5-shot tasks from CUB respectively, far lag behind DINO pre-trained models whose results are 57.9% and 88.2% respectively.\n\n**5.cross-domain results**\n\nOur new setting is essentially a cross-domain setting: the massive pre-training data can come from different domains with target few-shot tasks, and even be unknown if there are privacy issues. We can adapt the pre-trained models to the few-shot tasks from any target domains.\n\n**6.ablation experiment using only the gradient term**\n\nWe follow the setup of Section_4.1-main and ablate the attention and gradient information. The results are shown in Tab.2-rebuttal where “+FORT (grad)” denotes using only gradient term, “+FORT (attn score)” denotes directly using attention scores, “+FORT” denotes using both attention and gradient information. Here we pick the best hyper-parameters for each model even though they may differ from each other. For DINO pre-trained models, the internal attention can locate the key patches and thus effectively assist the gradient information. For CLIP pre-trained models, the internal attention can not indicate key patches and the gradient information plays a major role.\n\n**7.comparison with related work [d,e]**\n\n[d] aims to use the correlation between the local features of support and query samples to calculate similarity precisely and proposes a cross-attention module. [e] aims to highlight the related features in query samples based on support samples and proposes the meta weight generator and spatial attention generator. These models follow the traditional meta-learning paradigm and need a base dataset to meta-learn the parametric module, thus can not applied in our new setting. Our method introduces no new parametric modules and does not require meta-training, thus is more flexible.\n\n[a]FeLMi: few shot learning with hard mixup. NeurIPS 2022.  \n[b]Label hallucination for few-shot classification. AAAI 2022.  \n[c]Associative alignment for few-shot image classification. ECCV 2020.  \n[d]Cross attention network for few-shot classification. NeurIPS 2019.  \n[e]Few-shot classification via adaptive attention. arXiv:2008.02465."}, {"review_id": "yChseIGTLN", "reviewer": "Reviewer_h2aC", "summary": "This paper addresses few-shot learning by resorting to pre-trained large vision models. A new prompt strategy, termed position prompt, is proposed during fine-tuning to encourage the model to focus on class-relevant patches. This is realized by an attention-based token selection module and an optimization objective that enhances the affinity values between image tokens and the selected key patches. The proposed method is applied to various Vision Transformer structures with different fine-tuning strategies to verify the effectiveness of the proposed method. Experiments on a set of few-shot learning tasks show the proposed method can effectively boost few-shot learning performance.", "questions": "In Eq. 5, G is a vector, this vector is repeated before applying to Eq. 6? What is the motivation to project gradients to input to its first principle component?", "limitations": "Limitation analysis is provided.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper makes an interesting attempt on how to prompt large pre-trained vision models during fine-tuning and is found beneficial for few-shot learning. \n2. It is good to see the proposed prompt strategy is applied to different combinations of Vision Transformer architectures and fine-tuning strategy to verify the generalizability of the proposed method.", "weaknesses": "1. The claim regarding the importance of locating class-specific patches appears too strong. Firstly, the datasets used in this paper primarily consist of single objects with context, so the statement that \"the input images typically contain multiple entities\" may not hold true. Secondly, contextual information is not always detrimental to classification and often provides valuable cues for recognizing the objects of interest.\n\n2. While the method employs a sophisticated strategy to locate key patches using both MHA and gradient information, this does not guarantee the accuracy of key patch identification. As there are no ground-truth annotations for key patches, limited visualizations alone are insufficient to justify their correctness. If the identified patches do not align with expectations, it contradicts the original motivation.\n\n3. The authors have overlooked existing attempts to locate discriminative image parts in few-shot learning that aim to enhance performance, such as the works referenced [1] and [2].\n\n[1] \"Multi-attention Meta Learning for Few-shot Fine-grained Image Recognition.\" IJCAI 2020.\n[2] \"Multi-attention network for one-shot learning.\" CVPR 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please check the reference rules in the global response first to help following reading.\n\n**1.vector *G* in Eq.6-main**\n\nYes, the vector *G^T* with shape of (1, *N*) is repeated row-wise to add to the attention matrix with shape of (*N*, *N*).\n\n**2.motivation to reserve the first principle component of gradient**\n\nThe gradient computed for single sample contains many noises and we reserve its first principle component for denoise. The similar idea can also be seen in some deep explanation methods, i.e., Eigen-CAM[a]. In fact, for the gradient *G* with shape of (*N*, *d*) where *N* is the patch number, we find keep the maximum value of each patch gradient can achieve similar effect, i.e., max(*G*, dim=1) --> *G* with shape of (*N*, 1).\n\n**3.importance of locating class-specific patches**\n\nFirstly, although the used datasets are typically object-centric, the images still contain multiple entities in most cases, e.g., birds and tree branches, cars and buildings, etc. The class-independent objects introduce noise information and harm the performance, especially for scarce training samples. Secondly, most contextual information can not reflect the essential characteristics of target classes, and thus misleads classification. Whether a bird lands on a tree branch or a roof should not affect its class. Besides, if some contexts are useful for recognizing the objects, we also treat their corresponding patches as the key ones.\n\n**4.correctness of locating the key patches**\n\nWe agree that the qualitative visualization of position prompts does have limited justification. Since there are no ground-truth annotations, it is impossible for us to provide quantitative verification. However, we conduct extensive visualization of position prompts beyond the few ones in the main text, and find our method can indeed effectively locate key patches. In fact, the supplementary materials also include some samples in Figures_ViT-B_DINO.zip, with the position prompts of 140 images, and please check them for further validation. Besides, it has been extensively validated that deep explanation methods can locate the most classification-related parts of input.\n\n**5.discussion about related works [b,c]**\n\n[b] proposes the channel and spatial attention modules to enhance the local features of each input. [c] aim to use class text tag to aid classification and propose a attention network which can generate the attention map based on text tag embedding to weight local features. Both methods need to meta-train the introduced parametric modules using a base dataset, and [c] even needs to conduct a new image-tag datasets. Therefore, they are not suitable for our setting where we only have a pre-trained backbone and the support set of the target task. Our method instead introduces no new parametric modules and does not require meta-training, thus is more flexible.\n\n[a]Eigen-CAM: Class Activation Map using Principal Components. IJCNN 2020.  \n[b]Multi-attention Meta Learning for Few-shot Fine-grained Image Recognition. IJCAI 2020.  \n[c]Multi-attention network for one-shot learning. CVPR 2017.  "}, {"review_id": "JL0bhsOCv0", "reviewer": "Reviewer_SFDV", "summary": "This paper introduce a method called FORT, aiming to adapt pre-trained vision transformers to few-shot image classification task. This method contains two steps. 1) Utilizing attention and the gradient information to locate important entities, which is denoted as position prompts. 2) A new loss is defined to enforce the attention close to the position prompts. Extensive experiments shows benefits of FORT over different datasets and different pre-trained model. \n\nContributions can be summarized as follow:\nThe position prompt and the new loss can make fine-tunning on pre-trained transformer based model more efficient when dealing with few-shot samples for classification task.\n", "questions": "1) for section 3.4, are the position prompts fixed during fine-tunning? Or it also updating along the fine-tuning?\n2) For each task, are the fine-tunning process only involved with a few-examples for each task? Or it is like traditional few-shot learning need a bunch of data like base-dataset.\n3) Can the method be compared with traditional trained few-shot learning? (train a vit from scratch using the data from seen classes of its own domain). If yes, would be nice to see some numbers. If not, please further explain a little more.\n4) When utilizing the pre-trained transformer, how does 5-way few shot classification works? Would be nice to see some numbers for comparison. I am wondering if it is too easy or too little data to be fine-tunned.\n5) For the compared baselines, are the MetaOptNet and SVM the same for fixed backbone? I am wondering why they differ so much in terms of the performance.\n\nIf the questions are addressed, I am willing to raise the score. Thank you in advance!", "limitations": "In general, I think the contribution might not be enough since the pre-trained model are proofed very powerful already. If the questions are addressed, I am willing to raise the score. Thank you in advance!", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The writing and presentation is logical.\n2) The intuition to make the attention concentrate to the class related entities are reasonable.\n3) Derivation of analysis makes sense.\n4) Extensive experiments shows some benefit of the proposed method.", "weaknesses": "1) The idea of denoising the attention more interpretable seems not very novel, especially for papers of interpretability and visualizations. Would be nice to discuss more in the related work about these methods.\n2) From my understanding, this method is only propose a regularizer in equation 8 combined with other standard fine-tunning procedure. The contribution of this is not significant. The improvement 1percent accuracy of experiments doesn't seem to be very siginificant. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please check the reference rules in the global response first to help following reading.\n\n**1.fixing or updating the position prompts along fine-tuning**\n\nThe position prompts are obtained before fine-tuning and fixed as supervision signals during fine-tuning process. We find that updating them along fine-tuning can not obtain non-trivial improvement but introduces obvious cost.\n\n**2.only support set of each novel task or using a base-dataset**\n\nIn this work, we aim to effectively adapt the pre-trained large models to target few-shot tasks. Therefore, for each task, we only have its support set that contains few labeled samples and a pre-trained model. We directly fine-tune the pre-trained large model in the few support samples, which is actually very challenging, especially for the data-hungry vision transformers.\n\n**3.comparison with traditional few-shot learning**\n\nIn this work, we abandon the traditional setting that learns inductive bias from a small base dataset, and instead propose a new paradigm that directly adapts the models pre-trained on massive data to target few-shot tasks. This new paradigm has many advantages: 1) the pre-trained large models have far more generalizing representation and thus can obtain significantly better performance; 2) it can handle supervised, cross-domain and unsupervised few-shot learning simultaneously: the pre-training data can be unlabeled and thus sufficiently large, and the novel tasks can come from different target domains; 3) it is more friendly to privacy data, and we only need the pre-trained models instead of pre-training data; 4) it has better realistic value and conforms to the learning paradigm of humans whose few-shot ability comes from learning with massive signals since birth. The reason why we use the 20-way setting is also to make the setting more realistic and practical useful. The 5-way setting is too simple to be a valuable one.   \nMost traditional few-shot learning methods typically design some parametric meta modules and need a base dataset for meta-training, and thus are not applicable in our setting. The results of some applicable ones, e.g., MetaOptNet and R2D2, are compared in Tab.1,2,3,4-main.\n\n**4.results on 5-way 1-shot/5-shot tasks**\n\nWe provide the results on 5-way tasks in Tab.1-rebuttal. The performance are far more better than traditional few-shot learning methods. For the 1-shot tasks from miniImageNet, we can achieve 94.2% accuracy while the SOTA traditional method, FeLMi[a], only achieve 68.28%. For the 1-shot tasks from CUB, we can achieve 84.4% accuracy while the SOTA traditional method, Wave-SAN[b], only achieves 50.3%. This means using pre-trained large models is more effective and practically valuable than traditional paradigm. Moreover, our method can further improve the fine-tuning performance of base fine-tuning methods, e.g., LoRA and SSF.\n\n**5.difference between MetaOptNet and “SVM”**\n\nFor “SVM”, we use the method from scikit-learn library, i.e.,\n\n    from sklearn.svm import SVC  \n    classifier = SVC(C=1., kernel='rbf')\n\nwhose implementation is based on libsvm library. Differently, MetaOptNet use multi-class kernel-based vector machines from [c]. Both MetaOptNet and “SVM” use the same fixed backbone, but the different multi-class support vector machine methods lead to different performance.\n\n**6.novelty of denoising**\n\nDenoising for better explanation is indeed a commonly-used method, e.g., Eigen-CAM[d], and we do not treat it as a contribution and thus do not discuss about denoising in the related works. Our main innovation is to effectively combine the attention and gradient information to obtain a method that can localize the key patches accurately and generally. The attention information is not enough for some pre-trained models, like CLIP, to locate the key patches, so we introduce the denoised gradient to the attention map, as shown in Eq.6-main.\n\n**7.contribution and improvement**\n\n(1)Our contribution is that we propose a new form of prompts for vision data, position prompts, to effectively adapt the pre-trained vision transformers to few-shot image classification tasks. We provide their definition, obtaining method (Section 3.4) and usage (Section 3.5). The reason why the usage is a simple regularization loss besides the classification loss is the limitation of the new setting. Concretely, for a target few-shot task, we only have few support samples without base dataset, so it is impossible to learn a new parametric module from scratch. This greatly limits the designing space of the model, and we have to resort to the simple non-parametric method. Our regularization method introduces no parametric modules and is simple, but can effectively enhance the attention of model to the key patches. (2) We conduct extensive experiments for different backbone, pre-training ways, fine-tuning methods and datasets. Our method can obtain significant improvement (+2.0~6.0%) in some cases, and also obtains small improvement in other cases. We believe this is normal in deep learning, as it is impossible for a method to be significantly effective in all cases.\n\nFinally, although the pre-trained models are proofed very powerful already, it is still meaningful to effectively adapt them to few-shot tasks to obtain better performance. In fact, it is a non-trivial problem, since fine-tuning the large data-hungry vision transformers on few samples is prone to over-fitting. As shown in Tab.1-main, the fine-tuning methods sometimes cannot significantly improve performance beyond the initial classifier (i.e., MetaOptNet) or even perform worse, especially on 1-shot tasks.\n\n[a]FeLMi: few shot learning with hard mixup. NeurIPS 2022.  \n[b]Wave-SAN: Wavelet based Style Augmentation Network for Cross-Domain Few-Shot Learning. arXiv:2203.07656.  \n[c]On the algorithmic implementation of multiclass kernel-based vector machines. JMLR 2001.  \n[d]Eigen-CAM: Class Activation Map using Principal Components. IJCNN 2020.  "}, {"review_id": "MrnR0R44Cs", "reviewer": "Reviewer_nhXH", "summary": "This paper proposes to directly adapt the large-scale pretrained model to the downstream classification task via fine-tuning on few-shot examples, therefore yielding a novel few-shot learning paradigm. Different from common few-shot classification methods, their paradigm is featured for 1) utilizing large-scale pretraining and 2) NO popular base training. Although large-scale pretraining usually promises substantial benefit and may bring unfair comparison against prior few-shot learning methods, the reviewer agrees with the author that this paradigm is more realistic and is of great value. Moreover, extensive experiments show that the proposed method achieves consistent improvement over various baselines (under the same paradigm), therefore validating the effectiveness of the proposed attention enhancement for few-shot adaptation. ", "questions": "- What is many-hot presentation？\n- In Table 1 and 2, “FT” (full finetuning) is slightly higher than all the three parameter-efficient tuning methods (VPT, LoRA, SSF). This is contrary to the intuition that PET methods is superior under the few-shot learning scenario. Can you give some explanations on this phenomenon?\n", "limitations": "The authors have adequately addressed the limitations. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed \"large-scale pretraining --> few-shot learning\" (no base training) is meaningful has good realistic value.\n2) Making the attention focus onto the key entities sounds reasonable for few-shot downstream task adaptation.\n3) Experiments validate the method with consistent improvement over a battery of baselines.  ", "weaknesses": "- In Eqn. 6, the rationale of adding an identity matrix to the attention matrix is unclear. Is it because the attention layer contains a residual operation？\n- The attention graph calculation for the columnar and pyramidal architecture differs a lot from each other.  How does the latter (and simpler) manner perform on columnar architecture? Ablation studies to investigate the detailed design of the attention graph are expected.\n- The results analysis (L269 to L287) needs better clarification. Currently, multiple observations are squeezed into a single paragraph and it is not always clear which results correspond to each observation. Some analysis in the main text is not explicitly consistent with the results in the tables. For example, Line 272 states that the fine-tuning methods can achieve better performance than the model-frozen methods. This observation does not stand in Table 2, where two model-frozen methods (LoRA and SSF) are higher than FT. \n- The “orig attn” visualization in Fig 1 is somehow misleading (which barely attends to the useful foreground), while the visualization in Fig. 4 is much more reasonable (the pre-trained cls token, though attends to many unrelated regions, partially covers some useful foreground). Please check whether the visualization in Fig. 1 is correct. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please check the reference rules in the global response first to help following reading.\n\n**1.many-hot presentation**\n\nFor an input image, we obtain its position prompts, i.e., the position index set *O* of the key patches. For a vector *z* of length *N*, where *N* is the patch number, we set its values of the positions in the index set *O* to 1 and other positions to 0, then *z* is the many-hot presentation of the position prompts. Similar to one-hot label, its cross-entropy loss is actually the negative log probability at key patch positions and the expanded formula is provided in Eq.8-main. Since the attention scores are computed from Softmax, so each score can be treated as the prediction probability to the corresponding position.\n\n**2.FT vs PET in few-shot learning**\n\nFor all fine-tuning methods, we initialize the classification head using closed-form linear classifier, e.g., MetaOptNet[a], before the fine-tuning process. As shown in Fig.2-main, without classification head initialization, the performance of the fine-tuning methods even lags significantly behind closed-form MetaOptNet, and PET fine-tuning indeed obtains better performance than full fine-tuning on few-shot tasks. The detailed accuracy are provided in Tab.2-appendix. The classification head initialization can significantly improve the performance of fine-tuning methods on few-shot tasks and also make full fine-tuning comparable with PET fine-tuning. We recommend this operation as the default for fine-tuning on few-shot tasks, which also is a contribution of this work.\n\n**3.Identity matrix in Eq.6-main**\n\nYes, the rationale of adding an identity matrix to the attention matrix is the attention\nlayer contains a residual connection. The residual connection plays a important role in propagating information from inputs to model predictions and can preserve the position information during forward propagation. Concretely, the input *X* and out *Y* of the attention layer approximately satisfy *Y = (W + I) X* with *W* is the attention matrix.\n\n**4.results of locating method from pyramidal architecture in columnar one**\n\nFor a pyramidal architecture, since the downsampling operation destroys the patch position correspondence between layers, so we directly use the attention scores average on patches as the importance scores. The results of this simple method in columnar architecture are provided in Tab.2-rebuttal as “+ FORT (attn score)”, the method used in our paper is shown as “+ FORT”. As we can see, this simple method can not achieve as effective improvement as the used method, especially for CLIP pre-trained model whose attention is not enough to locate the key patches and we need gradient information as auxiliary.\n\n**5.analysis between Line-269 to Line-287**\n\nWe guess the unclear reference to methods causes confusion. In this paper, the model-frozen methods refer to the simple machine learning classifiers (e.g., Nearest Neighbor classifier, Ridge Regression and Support Vector Machines), plug-and-play inductive meta-solvers (e.g., ProtoNet[b], R2D2[c] and MetaOptNet[a]) and linear probing. LoRA[d], SSF[e] and FT are all fine-tuning methods and adjust the parameters to adapt new tasks. Our FORT is used for these fine-tuning methods and improve their performance. Therefore, the fine-tuning methods indeed can achieve better performance than the model-frozen methods, e.g., MetaOptNet vs (SSF, LoRA or FT) in Tab.1,2,3,4-main. FORT can further help the fine-tuning methods to obtain significant improvement on both ViT and Swin, e.g., LoRA vs LoRA+FORT in Tab.1,2,4-main. We will make these more clear in the revision.\n\n**6.“orig attn” visualization in Fig.1-main**\n\nThe “orig attn” visualization in Fig.1-main is correct. The original attention attends to both foreground and background objects and thus the visualization of 95% attention may cover most positions.\n\n[a]Meta-learning with differentiable convex optimization. CVPR 2019.  \n[b]Prototypical networks for few-shot learning. NIPS 2017.  \n[c]Meta-learning with differentiable closed-form solvers. ICLR 2019.  \n[d]Lora: Low-rank adaptation of large language models. ICLR 2022.  \n[e]Scaling & shifting your features: A new baseline for efficient model tuning. NeurIPS 2022."}]

Author rebuttal:
We really appreciate all the reviewers for taking your precious time to make the valuable comments. Overall, **our work has the following strengths**:
1. **the proposed new setting is more realistic and has great value. [nhXH]**
2. **reasonable and interesting idea. [8eth], [SFDV], [nhXH]**
3. **sensible and useful theoretical analysis. [TpQg], [SFDV]**
4. **well-written and easy to follow. [8eth], [TpQg], [SFDV]**
5. **extensive experimental results. [8eth], [h2aC], [SFDV], [nhXH]**

We will carefully respond to the concerns of each reviewer. The reference contents come from the main text, the appendix or the rebuttal PDF. Therefore, for the convenience of reference, we mark "**xxx-main**", "**xxx-appendix**" and "**xxx-rebuttal**" to be the corresponding content xxx in the main text, the appendix and the rebuttal PDF respectively. For example, "Tab.1-main" refers to the Table 1 in the main text, and "Fig.1-rebuttal" refers to the Figure 1 in the rebuttal PDF. The valuable suggestions from reviewers will be added to the revision, such as more ablation results, more discussions about related works and more analysis on observations.

Meta-review: This paper obtained 4 borderline accepts and 1 borderline reject. All reviewers appreciated the interesting and intuitive idea, its practical value, and the extensive experiments, but there were concerns about unclear justification, missing related work, and limited analyses/comparisons. The authors addressed most of them in the rebuttal and subsequent discussion, providing additional experiments/analyses that show consistent improvement by the proposed technique. After carefully checking the reviews, the rebuttal, and the discussion, AC finds that the raised concerns are resolved and this submission has a good contribution to the few-shot learning area, recommending the acceptance.
Decision: Accept (poster)