SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Visualization of the patches (white ones) with top highest attention scores and covering about  $95\%$  attention of [CLS] token in ViT-B/16. We use DINO pre-trained model for initialization.
Figure 7: Visualization of the patches (white ones) with top highest attention scores and covering about  $95\%$  attention of [CLS] token in ViT-B/16. We use DINO pre-trained model for initialization.
experiments. We do not consider the pre-trained backbones from masked image modeling models for initialization, e.g., BEiT [76] or MAE [18], since these generative pre-training models can not obtain highly linearly separable features and perform poorly on few-shot tasks. We set the rank to 4 for LoRA [24] and use 10 learnable prompts at each layer for VPT [26]. We mainly use AdamW [37] optimizer for fine-tuning, except for linear probing and VPT where we use SGD optimizer with weight decay of 1e-3. We set the batch size to 20, same as the number of classes, for both 1-shot and 5-shot tasks. We set  $\lambda = 1$  for DINO pre-trained model and  $\lambda = 50$  for CLIP pre-trained model for simplicity. The other hyper-parameters, including learning rate, number of fine-tuning epochs, temperature  $\tau$  and coefficient  $\alpha$ , are changeable for different fine-tuning methods, with or without classifier initialization and different datasets. To this end, we only provide their candidate values in Table 7, and choose them according to the average accuracy on 50 randomly sampled few-shot tasks. We only use 50 tasks for quick hyper-parameter selection. Considering the confidence intervals on random 50 tasks are too large to be an accurate indication of performance, e.g.,  $\pm 1.7$ , we fix the random seed so as to compare the performance of different hyper-parameters on the same 50 few-shot tasks. We first determine the learning rate and the number of epochs for the base fine-tuning methods, and then determine the temperature  $\tau$  and the coefficient  $\alpha$  for our method.
Attention maps with sufficient training samples. We follow the experimental settings of Figure 4, except fine-tuning on the 20-way 50-shot task from CUB and 20-way 200-shot task from Pets without attention enhancement. The attention maps after fine-tuning are shown in Figure 6 and Figure 7 as "FT Attn (sufficient)". As we can see, fine-tuning with sufficient samples indeed focuses on the key entities more than with few samples ("FT Attn") or original attention ("Orig Attn"), but is less focused than explicit attention enhancement ("Our Attn"). Note that this does not mean that few-shot fine-tuning with attention enhancement can obtain higher accuracy than sufficient-shot fine-tuning. In fact, sufficient training samples allow the models to fit the input-label distribution better, which is more important for classification than attention enhancement. But when we can not obtain sufficient samples, attention enhancement can reduce class-independent noise information in the final embedding as shown experimentally and theoretically, and thus is useful for classification.
Performance on 5-way tasks. In order to facilitate comparison with traditional few-shot learning setting and show the advantages of our new setting, we provide the results on 5-way 1-shot / 5-shot tasks in Table 8. The performance are far more better than traditional few-shot learning setting. On miniImageNet, we can achieve  $94.2\%$  accuracy for 1-shot tasks and  $98.5\%$  accuracy for 5-shot tasks, while the SOTA traditional model, FeLMi [48], only achieve  $67.47\%$  and  $86.08\%$  accuracy for 1-shot
Table 8: Few-shot classification accuracy (%) on 5-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on ImageNet-1k with DINO for backbone initialization.
Model	miniImageNet	CUB	Cars	Pets
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
LoRA	93.1	97.8	80.1	94.7	43.3	74.9	88.7	95.9
LoRA + FORT	94.2	98.5	84.4	96.2	45.2	76.1	89.9	96.8
SSF	92.9	97.5	78.1	94.1	43.5	74.7	89.4	96.5
SSF + FORT	94.2	98.1	81.8	96.4	46.2	76.6	91.2	97.1
Figure 8: Visualization of the attention map of [CLS] token at the top layer of ViT-B/16. We use the ViT-B/16 pre-trained on ImageNet-1K with DINO for backbone initialization.
and 5-shot tasks respectively. Besides, although TDM [33] can achieve comparable performance on CUB with our method, it far lags behind our method on Pets, i.e.,  $55.8\%$  vs  $91.2\%$  on 1-shot tasks and  $72.5\%$  vs  $97.1\%$  on 5-shot tasks. This means that TDM can not obtain stable good performance while our method can achieve satisfactory performance on different datasets and are more general. In fact, the traditional setting which uses a base dataset is not inherently a true few-shot setting. If we want to obtain a few-shot model for CUB, we have to collect many labeled bird images for meta-training, which still needs massive labeling labor and seriously hinders the practical application. Importantly, for different domains (e.g., CUB, Cars), we have to label separate base datasets for them. Our new setting uses open-source large models pre-trained on massive unlabeled data and only uses the few support samples from different target domains for fine-tuning, which is a true few-shot setting and can handle different domains simultaneously. Using pre-trained large models is more effective and practically valuable than traditional setting. Finally, our method can further improve the fine-tuning performance of base fine-tuning methods, e.g., LoRA and SSF, even on 5-way tasks.
Attention map of different fine-tuning methods. In Figure 4, we choose the patches with top highest attention scores and covering about  $95\%$  attention for visualization. Here we provide the full attention map of original attention ('Orig Attn') and that after fine-tuning w/ ('Our Attn') or w/o ('FT Attn') our method in Figure 8. The settings are the same, i.e., we use the ViT-B/16 pretrained on ImageNet-1K [49] with DINO for backbone initialization and fine-tune on random 1-shot tasks. Neither the original nor fine-tuned models can focus their attention on the key entities. Our method can obviously make the model to focus on the key entities and reduces the attention to the class-independent parts. This ability can generalize from the support set to query samples.
Fine-tuning without classifier initialization. In Figure 2, we compare MetaOptNet [32] with full or parameter-efficient fine-tuning methods, i.e., VPT [26], LoRA [24] and SSF [34], via average accuracy on CUB [61], Cars [30], Places [75] and Plantae [59] datasets. Here we provide detailed results in Table 9, where we use the ViT-B/16 pre-trained on ImageNet-1K [49] with DINO for backbone initialization and the  $95\%$  confidence interval is  $\pm 0.3$  (1-shot) or  $\pm 0.2$  (5-shot). As we can see, all fine-tuning methods perform worse than the backbone-frozen MetaOptNet on few-shot tasks. The extremely few labeled samples can not simultaneously learn a good classifier and adapt the backbone to the data distribution. In fact, the randomly initialized classifier back-propagates noisy gradients, which destroys the knowledge learned during pre-training and also fails to adapt new data distributions. Further, this phenomenon also shows it is very difficult to learn new parametric modules from scratch on few-shot tasks, e.g., classifier head, and we need to construct the well
Table 9: Few-shot classification accuracy (%) on 20-way 1-shot / 5-shot tasks. We use the ViT-B/16 pre-trained on ImageNet-1K with DINO for backbone initialization. 'FT' denotes full fine-tuning. Here all fine-tuning methods use the randomly-initialized classifier.
Model	CUB	Cars	Places	Plantae
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
MetaOptNet	57.0	85.1	24.1	57.9	50.0	71.1	36.9	63.7
VPT [26]	38.3	73.3	17.5	43.1	35.9	64.6	25.1	54.1
FT	25.0	66.7	15.3	42.2	23.6	55.0	22.0	51.4
LoRA [24]	55.3	83.6	22.5	54.7	48.4	68.2	34.2	62.1
SSF [34]	54.8	83.4	22.6	53.9	47.9	69.4	33.7	61.2
initialization for them. Considering MetaOptNet is a linear solver with excellent performance, we use it to calculate the classifier weights as initialization, which significantly improves the performance of various fine-tuning methods. Since the prompts in VPT are learned from scratch and we can not obtain their initialization, VPT still underperforms MetaOptNet on few-shot tasks even with classifier initialization. We recommend using a linear solver (e.g., ProtoNet [52], R2D2 [4] or MetaOptNet [32]) to obtain a good initialization of the classifier as a necessary preemptive operation when adapting pre-trained models to few-shot classification tasks in the future.
Patch attention. In the main text, we only examine the attention of the [CLS] token at the last layer. Since our method calculates the attention enhancement loss on both the [CLS] and patch tokens, here we visualize the attention map of those tokens at the penultimate (11-th) and last (12-th) layer of ViT-B/16. We use the ViT-B/16 pre-trained on ImageNet-1K with DINO for backbone initialization, and fine-tune the model using LoRA or SSF and evaluate the effect of our method on them. We choose the [CLS] token and {14, 28, 42, 56, 70, 84, 98, 112, 126, 140, 154, 168, 182, 196}-th patches for visualization and the results are shown in Figure 9. After fine-tuning with LoRA or SSF, both the [CLS] and patch tokens still attend to the other parts besides the key entities, introducing class-independent noise information, and some patch tokens even attend to those parts more than the key entities. Our method can enhance both the [CLS] and patch attention to make them focusing on the key entities. Interestingly, although we do not calculate the attention enhancement loss at the penultimate layer, the attention of its [CLS] and patch tokens are still significantly enhanced and focus on the key entities. As the theoretical analysis in Section 3.5, our method can increase the mutual information between the input tokens of the last layer (i.e., the output tokens of the penultimate layer) and the key patches. In order to achieve this objective, the [CLS] and patch tokens of the penultimate layer should also pay more attention to the key patches to introduce more information from them.
(a) LoRA
(b) SSF
Figure 9: Visualization of the attention map of the [CLS] token and different patches at the penultimate (11-th) and last (12-th) layer of ViT-B/16. Here we use the ViT-B/16 pre-trained on ImageNet-1K with DINO for backbone initialization and evaluate the effect of our method on LoRA and SSF.