SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoqing Wang Shibo Jie Zhi-Hong Deng*
{wanghaoqing, parsley, zhdeng}@pku.edu.cn
Since many pre-trained vision transformers emerge and provide strong representation for various downstream tasks, we aim to adapt them to few-shot image classification tasks in this work. The input images typically contain multiple entities. The model may not focus on the class-related entities for the current few-shot task, even with fine-tuning on support samples, and the noise information from the class-independent entities harms performance. To this end, we first propose a method that uses the attention and gradient information to automatically locate the positions of key entities in the support images, denoted as position prompts. Then we employ the cross-entropy loss between their many-hot presentation and the attention logits to optimize the model to focus its attention on the key entities during fine-tuning. This ability then can generalize to the query samples. Our method is applicable to different vision transformers (e.g., columnar or pyramidal ones), and also to different pre-training ways (e.g., single-modal or vision-language pre-training). Extensive experiments show that our method can improve the performance of full or parameter-efficient fine-tuning methods on few-shot tasks. Code is available at https://github.com/Haoqing-Wang/FORT.
Based on the accumulated experience, humans can learn from few labeled samples for solving novel tasks, and this is a key aspect of human intelligence. Few-shot learning [12, 31] aims to imitate this ability and has attracted widely attention from the machine learning community [71]. Traditionally, a base dataset is collected for learning prior knowledge before solving novel few-shot tasks. The meta-learning methods learn the task-shared inductive bias, e.g., parameter initialization [13, 41], metric function [52, 32, 54] or optimizer [2, 47]. The transfer learning methods [56, 48] learn generalizing representation to novel tasks, and achieve comparable or better performance.
Some works [56, 46] find that the well-learned embedding model contributes the most to few-shot classification performance, and even more important than the sophisticated meta-learning algorithms. However, the commonly-used base datasets are far from large enough for learning generalizing embedding, e.g., miniImageNet [60], which severely limits the performance of few-shot classification. In fact, the few-shot learning ability of humans benefits from the supervised or unsupervised learning from massive signals since birth, which helps us accumulating enough general knowledge. On the other hand, self-supervised learning [66, 19, 8, 18, 5] can learn well-generalizing representation from nearly-free massive unlabeled data and is beneficial for broad downstream tasks. Using these pre-trained models for few-shot tasks conforms to the learning paradigm of humans and has achieved remarkable success [5, 74, 15] in Natural Language Processing (NLP). In this work, we aim to adapt the pre-trained models to few-shot image classification tasks, without using any base datasets.
Recently, vision transformers [11, 36, 70] play an increasingly important role in computer vision, so we use them as the backbone for few-shot tasks. Note that it is a non-trivial problem to fine-tune the pre-trained large vision transformers on few support samples due to their data-hungry
Figure 1: Focusing on key entities via position prompts. The original pre-trained model may attend to multiple entities in a single image, and the information from class-independent entities is actually noise in the current task. The extremely few support samples make the standard fine-tuning with only classification loss unable to remove the noise information. We propose position prompts (red patches) to prompt the model where are the key patches of the input and focusing on them. This ability gained during fine-tuning can generalize from support samples to query ones. Here the white patches in attention visualization have the top highest attention scores and cover about  $95\%$  attention.
property. Inspired by the prompt methods [72, 35] from NLP, some works [78, 77, 10] synthesize the classification weights by encoding the learnable text prompts that describe the objective classes, and then compare them with image features. Despite impressive performance on few-shot tasks, these works strongly rely on the semantic alignment between image encoder and text encoder, and thus are specific to vision-language pre-trained models [45]. In this work, we design a new form of prompts for vision data, named position prompts, to adapt the pre-trained vision transformers to few-shot image classification tasks. They can prompt explicit valuable information, and also be usable to different pre-trained vision transformers, without limited structures or pre-training ways.
Figure 1 illustrates our framework and motivation, where we visualize the patches with top highest attention scores and covering about  $95\%$  attention (details at Section 4.2) of [CLS] token in a ViT-B/16 [11]. The input images usually contain several different entities, e.g., a bird stands on the branch (Figure 1, 'Input'). The pre-trained model may attend to multiple entities at the same time (Figure 1, 'Orig Attn') and the noise information from class-independent entities harms performance. The standard fine-tuning methods tune full or partial [24, 34] parameters of the model using the classification loss on the support samples, and can achieve better performance than the backbone-frozen methods. But even with fine-tuning, the extremely few support samples prevent the model from giving sufficient attention to the class-related key entities (Figure 1, 'FT Attn'). To this end, besides the classification loss, we use the position information of the key entities in each support sample as guidance for fine-tuning, and the model is expected to focus its attention on the key entities after fine-tuning, not only for the support samples, but also for the query ones. Concretely, we first automatically locate the position of key patches as position prompts (Figure 1, 'Prompts') in each support sample based on the deep explanation method, where the key patches refer to the most class-related ones and correspond to key entities. Inspired by Rollout [1], we integrate the attention information from multiple layers linearly for explanation, and further introduce gradient information as an auxiliary to ensure class-specific explanation. Unlike existing prompts [35], ours are only used to guide the fine-tuning process and not available during testing, since we need label information to obtain them. Then, we use the position prompts as the prediction target of attention, and optimize the cross-entropy loss between their many-hot presentation and the attention logits during fine-tuning. This loss enables the model to identify the key patches and focus on them (Figure 1, 'Our Attn'), and this ability obtained during fine-tuning can generalize to query samples. We also theoretically prove that our method can increase the information from the key patches in the output feature and reduce that from other patches (noise information).
Overall, our contributions can be summarized as follows:
- We propose a new form of prompts for vision data, position prompts, to effectively adapt the pre-trained vision transformers to few-shot image classification tasks.
- Our method, denoted as FORT (FOcus youR aTtention), is applicable to the attention modules in different vision transformers, e.g., Swin [36] or ViT, and not limited to specific pre-training ways, e.g., single or multi-modal ones.
- We conduct extensive experiments and verify that our method can effectively improve the performance of full or parameter-efficient [24, 34] fine-tuning methods on few-shot tasks.
Few-shot classification. The traditional paradigm of few-shot classification is the meta-learning framework [22, 67], which learns the task-shared inductive bias [13, 52, 32, 63] on the training tasks constructed from a base dataset. When the base dataset is unlabeled or in the different domain to novel tasks, we also need to solve new challenges, unsupervised [23, 28, 64] or cross-domain [58, 62, 14] few-shot classification. Some works [56, 46] find the well-generalizing representation is actually the key factor of few-shot ability, while the base dataset is typically not large enough to obtain it. In fact, adapting self-supervised pre-trained models [6, 76, 65] to few-shot tasks can address all these concerns simultaneously: the training data is unlabeled and thus can be sufficiently large, and the novel tasks can come from different target domains. The pre-trained large models have far more generalizing representation and thus can obtain significantly better performance than the traditional setting, as shown in Appendix B. Besides, it is more friendly to privacy data, and we only need the pre-trained models instead of pre-training data. It also conforms to the learning paradigm of humans whose few-shot ability comes from learning on massive signals since birth, and has better realistic value. An exploratory work [9] uses the ensemble of frozen pre-trained models and a simple classifier, but without further adaptation to novel tasks. CLIP [45] introduces the vision-language pre-trained models, and some works [78, 77, 10] use the text prompts to solve few-shot image classification based on them. These methods are limited to specific pre-trained models, and typically require the test classes to be seen in the training corpus [40]. Humans actually can recognize unseen objects based on only reference images, without knowing their class names. Our method is usable for both single and multi-modal pre-trained models. Similar to our work, some existing models [69, 79] also propose to highlight the key local features of each input for few-shot classification. However, they all design parametric modules and meta-learn them using the base dataset. It is not suitable for our new setting, since the parametric modules can not been learned effectively using only few support samples. Our method instead introduces no new parametric modules and does not need meta-training.
Prompt engineering. Designing prompts [35] for adapting the pre-trained models to few-shot tasks can originate from GPT-3 [5], which uses language prompts for in-context learning. Since the manual prompts require domain expertise and are likely to be sub-optimal even with significant effort, many works [15, 72, 29] focus on automating prompt generation to fine-tune pre-trained language models. The language prompts can also be used for few-shot image classification [78, 77, 7] based on special vision-language pre-trained models [45]. VPT [26] introduces learnable prompts to the input space of each transformer layer, and is suitable for both single and multi-modal pre-trained models. But VPT performs unsatisfactorily on few-shot tasks, even worse than simple plug-and-play solvers [4, 32] with frozen backbone (Table 1). Its prompts do not prompt any explicit information and still follow the form of language prompts. In this work, we propose position prompts for vision data to explicitly prompt the key positions of input images to improve fine-tuning performance.
Deep explanation method. In order to build trust in intelligent systems, deep explanation methods [16] calculate the local relevancy of the input with model predictions, and can be used to locate the key parts. Many classic works have been designed for CNNs, such as gradient based methods [51, 50, 55, 53] and relevance propagation based methods [3, 38, 25, 17]. For Transformers, these methods can not take advantage of the core attention information. We can use the raw attention scores to explain model predictions, but this ignores the attention components of other layers. Rollout [1] thus rolls out the attention scores of multiple layers to capture the propagation of information from input, but it is not class-specific. Inspired by Rollout, we also integrate the attention information from multiple layers, and further introduce gradient information to ensure class-specific explanation.
In this work, we aim to adapt the pre-trained vision transformers to few-shot image classification tasks. These backbones can be columnar or pyramidal architectures, e.g., ViT [11] or Swin [36], and
are pre-trained on massive data. In a few-shot classification task, we are typically given a support set  $\mathcal{T}_s$  as reference and a query set  $\mathcal{T}_q$  for test. The support set  $\mathcal{T}_s = \{(x_i^s,y_i^s)\}_{i = 1}^{C\times K}$  contains  $C$  different classes with  $K$  labeled samples in each class, which is denoted as 'C-way  $K$ -shot'.  $K$  is typically very small, e.g., 1 or 5, which makes it a challenging task, especially for the data-hungry vision transformers. The query set  $\mathcal{T}_q = \{(x_j^q,y_j^q)\}_{j = 1}^M$  contains the samples belong to the same classes with the support set, and we need to classify them correctly. Adapting a pre-trained model  $f_{\theta}$  to a few-shot task  $\mathcal{T} = \{\mathcal{T}_s,\mathcal{T}_q\}$  can be formulated as
$$
\psi = \mathcal {A} \left(f _ {\theta}, \mathcal {T} _ {s}\right) \tag {1}
$$
where the task solver  $\mathcal{A}$  is designed to obtain the solution model  $\psi$  of task  $\mathcal{T}$ . For example, the task solver  $\mathcal{A}$  can be linear probing [9] that combines the frozen model  $f_{\theta}$  with a linear classifier, or the parameter-efficient fine-tuning methods [34, 24, 27] that only tune partial existing or newly-added parameters for easy deployment, or using language prompts to generate classifier weights when  $f_{\theta}$  is a vision-language pre-trained model [78, 77, 7]. The performance of solution  $\psi$  is evaluated on the query set  $\mathcal{T}_q$  with accuracy and so on. In this work, we aim to design better task solver  $\mathcal{A}$ .
3.2 Multi-head self-attention
The Multi-head Self-Attention (MSA) modules are the core components of various vision transformers. The input image is split into multiple patches, which are then projected to patch tokens. The learnable [CLS] token is appended to these tokens for final classification if needed. Let the input of a MSA module be  $\mathbf{Z}_{in} \in \mathbb{R}^{N \times d}$ , containing  $N$  tokens with dimension  $d$ . Here  $\mathbf{Z}_{in}$  may contain all input tokens for global attention [11], or partial ones for window attention [36]. MSA uses  $H$  attention heads to jointly attend to the information from different representation sub-spaces. Within  $h$ -th attention head, we first project  $\mathbf{Z}_{in}$  to Query  $\mathbf{Q}_h$ , Key  $\mathbf{K}_h$  and Value  $\mathbf{V}_h$  as
$$
\mathbf {Q} _ {h} = \mathbf {Z} _ {i n} \mathbf {W} _ {q} ^ {h} \in \mathbb {R} ^ {N \times d _ {k}} \quad \mathbf {K} _ {h} = \mathbf {Z} _ {i n} \mathbf {W} _ {k} ^ {h} \in \mathbb {R} ^ {N \times d _ {k}} \quad \mathbf {V} _ {h} = \mathbf {Z} _ {i n} \mathbf {W} _ {v} ^ {h} \in \mathbb {R} ^ {N \times d _ {v}} \tag {2}
$$
where  $\mathbf{W}_q^h$ ,  $\mathbf{W}_k^h$  and  $\mathbf{W}_v^h$  are projection matrices. The self-attention matrix  $\mathbf{A}_h$  is calculated as
$$
\mathbf {A} _ {h} = \operatorname {s o f t m a x} \left(\mathbf {Q} _ {h} \mathbf {K} _ {h} ^ {T} / \sqrt {d _ {k}}\right) \in \mathbb {R} ^ {N \times N} \tag {3}
$$
The matrix  $\mathbf{A}_h$  reflects the attention of different tokens to each other, and the large score  $\mathbf{A}_h^{ij}$  indicates the  $i$ -th token strongly attends to the  $j$ -th token. The output feature of the MSA module is
$$
\operatorname {M S A} \left(\mathbf {Z} _ {i n}\right) = \operatorname {C o n c a t} \left(\left[ \mathbf {A} _ {h} \mathbf {V} _ {h} \right] _ {h = 1} ^ {H}\right) \mathbf {W} _ {o} + \mathbf {Z} _ {i n} \tag {4}
$$
where  $\mathrm{Concat}(\cdot)$  denotes the concatenation operation,  $\mathbf{W}_o$  is a projection matrix. Obviously, the MSA module learns the increment of each token and introduces the information from other tokens, whose quantity is controlled by the attention matrices  $\{\mathbf{A}_h\}_{h=1}^H$ .
3.3 Analysis and motivation
We first examine various designs of the task solver  $\mathcal{A}$ : 1) simple machine learning classifiers, e.g., Nearest Neighbor classifier, Ridge Regression and Support Vector Machines; 2) plug-and-play inductive meta-solvers (e.g., ProtoNet [52], R2D2 [4] and MetaOptNet [32]) and linear probing; 3) full or parameter-efficient fine-tuning, e.g., VPT [26], LoRA [24] and SSF [34]. The results are provided in Table 1, where we use the ViT-B/16 pre-trained on ImageNet-1K [49] with DINO [6] as  $f_{\theta}$  and conduct on CUB [61], Cars [30], Places [75] and Plantae [59] datasets. For each fine-tuning method, we use the best linear meta-solver, MetaOptNet, to obtain the initialization of the classifier weights. Otherwise, these fine-tuning methods perform even worse than MetaOptNet on few-shot tasks (Figure 2). As shown in Table 1, the fine-tuning methods can achieve better performance than the backbone-frozen methods in most cases. But they sometimes cannot significantly improve performance beyond
the classifier initialization (i.e., MetaOptNet) or even perform worse, especially on 1-shot tasks. In fact, the main challenges of few-shot tasks are: 1) the classes in the novel tasks could be unseen
Figure 2: Average 20-way accuracy on CUB, Cars, Places and Plantae datasets. Here all fine-tuning methods use the randomly-initialized classifier.
during pre-training phase; 2) there are only few  $(1\sim 5)$  labeled support samples for each class in novel tasks. The input images typically contain multiple entities. Although the pre-trained vision transformers may be able to separate different entities [6, 76], they do not pay enough attention to the key entities corresponding to the classes in the current few-shot task. Especially in the face of unseen classes, they may attend to multiple entities simultaneously [21]. When the support samples are sufficient, the model can attend to the frequently-occurring key entities in each class more to alleviate this problem, as shown in Appendix B. But the extremely few labeled samples prevent the fine-tuning methods from achieving this purpose, and the noise information from the class-independent entities misleads the fine-tuning process and harms the performance. For qualitative evidence, we visualize the patches with top highest attention scores and covering about  $95\%$  attention of [CLS] token in Figure 4, more details and quantitative evidence at Section 4.2. As we can see, both the original model and fine-tuned one can not focus most attention on the key entities. To this end, we propose the position prompts to explicit prompt the model where are the key entities and guide the model to focus on them during fine-tuning. This ability then can generalize to the query samples.
3.4 Position prompts
We need to locate the positions of key entities in each support image as prompts for fine-tuning, i.e., the most class-related key patches for the current few-shot task. Manual labeling may provide accurate positions, but requires domain expertise and labor costs. We aim to design an automatic method, which should 1) be class-specific for locating class-related patches, and 2) make full use of the attention information, the core content of Transformers. Deep explanation methods [16] calculate the local relevancy of the input with model predictions, and can be used for highlighting the key patches for classification. For Transformers, Rollout [1] finds that using the attention of top layer for explanation ignores most of the attention components, thus miscalculating the importance of the patches, so we integrate the attention information from multiple layers like Rollout. Furthermore, only using the attention information cannot achieve class-specific explanation for some pre-trained models (Figure 3), so we further introduce the gradient information as an auxiliary.
For a columnar architecture [11] with  $L$  layers, let the input feature map of  $l$ -th layer be  $\mathbf{Z}_{in}^{l} \in \mathbb{R}^{N \times d}$  and the attention matrices be  $\{\mathbf{A}_h^l\}_{h=1}^H$ . With the well-initialized classification head above, for a support sample-label pair  $(x^s, y^s)$ , we first compute the gradient of the prediction score for class  $y^s$ ,  $p_{y^s}$ , with respect to feature map  $\mathbf{Z}_{in}^{l}$  as  $\nabla_l = \partial p_{y^s} / \partial \mathbf{Z}_{in}^{l} \in \mathbb{R}^{N \times d}, l = 1, \dots, L$ . The gradient computed for single sample contains many noises, which could be amplified layer by layer along the back-propagation path. To this end, we only use the gradient to the input feature of the top layer, i.e.,  $\nabla_L$ , and reserve its first principle component for denoise. The obtained gradient term is
$$
\mathbf {G} = \nabla_ {L} \cdot \mathbf {V} _ {1} \in \mathbb {R} ^ {N \times 1}, \quad \mathbf {U}, \mathbf {S}, \mathbf {V} = \operatorname {s v d} (\nabla_ {L}) \tag {5}
$$
where  $\operatorname{svd}(\cdot)$  denotes Singular Value Decomposition (SVD) operation with  $\nabla_{L} = \mathbf{USV}^{T}$ , and  $\mathbf{V}_{1}$  is the first column vector of matrix  $\mathbf{V}$ , corresponding to the largest singular value of  $\nabla_{L}$ . The gradient term  $\mathbf{G}$  can provide class-specific information. As shown in Equation 4, the residual connection plays a important role in propagating information from inputs to model predictions and can preserve the position information during forward propagation. To this end, we use an identity matrix  $\mathbf{I}$  to augment the attention matrix. Overall, the final attention graph of  $l$ -th layer is
$$
\widehat {\mathbf {A}} ^ {l} = \operatorname {n o r m} \left(\mathbf {I} + \mathbf {A} ^ {l} + \lambda \cdot \mathbf {G} ^ {T}\right) \in \mathbb {R} ^ {N \times N} \tag {6}
$$
where  $\mathrm{norm}(\cdot)$  means to re-normalize the input so that the row sum is equal to 1,  $\lambda$  can control the proportion of attention and gradient information, and we average the attention matrix over all heads, i.e.,  $\mathbf{A}^l = \frac{1}{H}\sum_{h=1}^{H}\mathbf{A}_h^l$ . We assume the attentions are combined linearly and consider the path along the pairwise attention graph [1]. The patch importance scores are calculated as
$$
\mathbf {s} = \operatorname {m e a n} \left(\widehat {\mathbf {A}} ^ {1} \cdot \widehat {\mathbf {A}} ^ {2} \cdot \dots \cdot \widehat {\mathbf {A}} ^ {L}\right) \in \mathbb {R} ^ {N} \tag {7}
$$
where  $(\cdot)$  denotes matrix multiplication,  $\mathrm{mean}(\cdot)$  denotes to average the row vectors of a matrix. We use the positions of the top  $P$  patches with the highest importance scores as prompts.
For a pyramidal architecture [36, 70], since the downsampling operation destroys the patch position correspondence between layers, so we directly use the attention scores average on patches as the importance scores and introduce gradient information to achieve class-specific if necessary.
Table 1: Few-shot classification accuracy (%) on 20-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on ImageNet-1K [49] with DINO for backbone initialization. 'NN', 'RR' and 'FT' denotes Nearest Neighbor classifier, Ridge Regression and full fine-tuning respectively.
Model	CUB	Cars	Places	Plantae
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
NN	52.8	73.0	20.9	36.2	49.5	64.3	35.0	54.3
RR	56.6	84.0	24.0	56.2	49.9	68.7	36.8	62.0
SVM	52.8	78.7	20.9	37.3	49.5	70.8	35.0	57.7
ProtoNet [52]	52.8	79.8	20.9	39.3	49.5	71.2	35.0	57.7
R2D2 [4]	56.7	84.3	23.8	56.6	49.7	68.9	36.8	62.3
MetaOptNet [32]	57.0	85.1	24.1	57.9	50.0	71.1	36.9	63.7
Linear Probing	41.9	78.2	18.3	47.2	41.0	65.8	27.2	56.6
VPT [26]	52.9	81.1	23.3	54.5	48.0	69.6	33.9	60.2
FT	58.0	88.1	24.1	66.9	50.3	72.1	37.0	66.2
LoRA [24]	57.9	88.2	23.3	64.3	49.9	71.3	37.1	65.7
SSF [34]	57.8	88.4	23.8	62.3	50.2	73.4	37.2	66.0
FT + FORT	59.5 (1.5)	89.2 (1.1)	25.5 (1.4)	68.0 (1.1)	51.1 (0.8)	72.9 (0.8)	38.7 (1.7)	67.2 (1.0)
LoRA + FORT	62.5 (4.6)	89.5 (1.3)	26.8 (3.5)	65.7 (1.4)	50.8 (0.9)	72.4 (1.1)	38.5 (1.4)	66.9 (1.2)
SSF + FORT	62.3 (4.5)	89.6 (1.2)	26.5 (2.7)	64.2 (1.9)	51.3 (1.1)	74.4 (1.0)	39.0 (1.8)	67.5 (1.5)
3.5 Attention enhancement