SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In addition, we evaluate the performance of our method using ID data. Models were trained on various datasets using 1, 2, 4, 8, and 16 shots, respectively, and subsequently evaluate performance on the respective datasets. Figure 5 illustrates a comparative analysis of the average performance between our method and existing methods on 11 datasets. As depicted, our method significantly outperforms other strategies and yields increasingly substantial performance enhancements as the shot number increases. Previous research indicates that WiSE-FT effectively elevates the fine-tuning performance of models in zero-shot learning. However, in few-shot learning, relative to fully fine-tuning, employing WiSE-FT with a fusion parameter of 0.5 results in a performance notably inferior to comprehensive fine-tuning. Our analysis posits that, in few-shot learning, the paucity of fine-tuning samples results in minimal alterations in model fine-tuning. Consequently, following fusion, the model parameters more closely resemble those of CLIP, thereby influencing the model performance.
4.4.1 The Generality of FD-Align
To validate the generalizability of models fine-tuned via FD-Align across various methods, we directly incorporated the FD-Align fine-tuned visual encoder into existing approaches and evaluate ID performance. As shown in Table 3, the visual encoder fine-tuned with FD-Align markedly improves the performance of existing methods, achieving a peak improvement of  $1.79\%$ . The visual encoder fine-tuned with FD-Align, can be seamlessly transitioned into existing methods, bolstering performance without incurring additional costs.
Methods	1shot	2shot	4shot	8shot	16shot
CLIP	60.33
Fully Fine-tuning CLIP	63.48	64.87	68.10	71.14	73.43
FD-Align (80 templates)	63.90	65.64	68.10	71.30	74.03
FD-Align (Tip templates)	61.14	62.39	63.37	60.34	66.30
FD-Align + SPC	63.92	65.68	68.63	71.66	74.38
Table 4: Ablation results. All methods use the same learning rate and calculate the average performance across 11 datasets. The setup of 80 templates means using all the features of 80 templates as spurious prototype. The setup of Tip templates means using the prompts templates proposed by Tip-Adapter, and SPC means spurious prototype correction.
4.4.2 Why Remove Outliers and Cluster?
In this section, we analyze the importance of eliminating outliers from spurious features and the subsequent clustering of residual points via empirical analysis. As illustrated in Table 4, employing the Spurious Prototype Clustering (SPC) method yields enhanced performance for the spurious prototype relative to directly utilizing all 80 templates. This underscores the efficacy of removing outliers and redundant values from spurious features. Furthermore, we experimented with features derived from manually designed prompt templates as spurious prototypes. While Tip-adapter handpicked 7 templates, our findings indicate that leveraging these 7 templates directly as spurious prototypes precipitates a marked performance degradation. A closer examination revealed that the template "itap of a {class}" from Tip-adapter was identified and excluded as an outlier by SPC during its initial phase. This underscores the ability of SPC to autonomously filter outliers of spurious prototypes, thereby circumventing potential pitfalls inherent in manual selections.
(a) All prompt templates
(b) Isolation Forest
Figure 6: (a) represents all the features of prompt templates. (b) represents the features processed by Isolation Forest. (c) represents the final results clustered by  $k$ -means.
(c)  $k$  -means
To more intuitively illustrate the impact of outlier removal and clustering, we visualize the spurious features. Figure 6a displays the features of all the prompt templates within the feature space, revealing the conspicuous presence of outliers and redundant values. Figure 6b shows the visualization after outlier elimination: red points represent preserved values, while black points denote deleted outliers, including "itap of a {class}" It is evident that numerous redundant values persist. As depicted in
Figure 6c, after the clustering process, the residual spurious features are uniformly distributed across various positions, thereby providing a rational representation of the spurious prototype.
(a) CLIP
Figure 7: Feature visualization for the same class under different domains.
(b) Fine-tuned CLIP
(c) FD-Align CLIP
4.4.3 OOD Visualization
To more vividly demonstrate the influence of our method on model feature extraction, we conducted a visualization of the image features on PCAS [53], utilized for investigating OOD tasks, which contains category and domain annotations for images. Figure 7 illustrates the visualization of features for one of our categories. It is observable that CLIP adeptly distinguishes features across different domains. The ability of fully fine-tuned CLIP to differentiate information across various domains is comparatively diminished. After fine-tuning with FD-Align, the model reacquires the ability to discern information across different domains, substantiating FD-Align's ability to preserve the model's OOD performance. Additionally, as depicted in Figure 8, we visualize the features of different categories in various domains. On the sketch data, the model, after fully fine-tuning, exhibits a diminished capacity to differentiate between categories, whereas CLIP and FD-Align distinguish between various categories accurately.
(a) CLIP
Figure 8: Feature visualization for different class under different domains.
4.4.4 Training Stability
Figure 9 depicts the evolution of model accuracy and loss on the validation set throughout the fully fine-tuning and FD-Align processes. Notably, during the initial phases of fine-tuning, fully fine-tuning prematurely encounters overfitting, culminating in a decline in accuracy and a surge in classification loss. Conversely, FD-Align adeptly maintains model robustness and stability throughout the fine-tuning process, effectively circumventing overfitting.
4.4.5 Limitation
As shown in Tables 2, our method is able to preserve the generalization of the model while fine-tuning the model. However, in certain specific cases, such as Traffic Sign, where the images primarily consist of target objects and lack category-independent information, the preserved pre-trained model's category-independent ability adversely impacts the performance on such data. Additionally, our method does not perform well on special scenarios, such as ChestX and EuroSAT, for which one possible reason is that the prompts we use do not contain the category-independent information of these data. Therefore, additional prompts may need to be designed for this type of data.
Figure 9: Validation accuracy and loss variation during the training process.
In this paper, we introduce a feature discrimination alignment fine-tuning (FD-Align) method pretrained models in few-shot learning. Leveraging the remarkable text-visual alignment capabilities of CLIP, we employ text features of category-agnostic descriptions as spurious feature prototypes. Furthermore, we constrain the probability distribution of image features extracted by the model on spurious features, both before and after fine-tuning, to ensure the robustness of the model subsequent to fine-tuning. Experimental results substantiate the efficacy of our approach in enhancing fine-tuning performance while ensuring robustness across distributions.
Huimin Ma and Bochao Zou are supported by the National Natural Science Foundation of China (No.U20B2062, No.62227801, No.62206015) and National Key Research and Development Program of China (2022ZD0117900). Weiran Huang is supported by 2023 CCF-Baidu Open Fund and Microsoft Research Asia.
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[2] Marcos V Conde and Kerem Turgutlu. Clip-art: Contrastive pre-training for fine-grained art classification. In CVPR, 2021.
[3] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021.
[4] Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. In AAAI, 2022.
[5] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In CVPR, 2022.
[6] Kwanyong Park, Sanghyun Woo, Seoung Wug Oh, In So Kweon, and Joon-Young Lee. Per-clip video object segmentation. In CVPR, 2022.
[7] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR, 2022.
[8] Soumya Suvra Ghosal, Yifei Ming, and Yixuan Li. Are vision transformers robust to spurious correlations? In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
[10] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NeurIPS, 2017.
[11] Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. Adaptive cross-modal few-shot learning. In NeurIPS, 2019.
A Detailed Results of Different ID Datasets
Figure 10 delineates the effectiveness of our approach across different datasets. Impressively, FD-Align surpasses both WiSE-FT and the fully fine-tuning on a vast majority of these datasets. As the shot count rises, the advantage of FD-Align becomes even more salient. It is pertinent to note that when data is scarce, WiSE-FT tends to underperform compared to fully fine-tune.
Figure 10: Performance comparison of different methods on 11 datasets.
B Time Cost of FD-Align
Table 5 shows the time cost of fully fine-tuning and FD-Align on 1-shot ImageNet. Evidently, FD-Align introduces only a negligible increase in time overhead. It is noteworthy that during the inference phase, both FD-Align and fully fine-tuning share identical processes, resulting in consistent inference cost. A significant advantage of FD-Align is its ability to boost performance across varied methodologies with just fine-tuning once.
METHOD	Time
Fully Fine-tuning	7min 9s
FD-Align	8min 53s
Table 5: Time cost of Fully Fine-tuning and FD-Align.
C More Results on OOD Datasets
We also evaluate the performance of fully fine-tuned models on the OOD dataset. All models are trained on miniImageNet and performance is tested on other datasets. As shown in Table 6, our method performs better on Meta-Dataset, but worse on BSCDFSL. After analysis, we believe that the reason is that the model is fine-tuned on miniImageNet, which mainly contains natural image information, so the model better retains the domain information of natural images. However,
BSCDFSL mainly contains satellite images, medical images, and other nonnatural images, and our model fine-tuning does not retain the domain information of these datasets. Furthermore, our model affects the classification ability in the classification process. Therefore, on such datasets, our method may be slightly degraded compared to the direct fine-tuning of CLIP.
Dataset	5way-1shot	5way-5shot
FT	WiSE-FT	FD-Align	FT	WiSE-FT	FD-Align
Meta-dataset	mini_test	94.15±0.19	93.55±0.17	95.04±0.18	98.13±0.12	98.44±0.06	98.52±0.07
cub-test	80.78±0.69	81.16±0.71	82.38±0.69	92.95±0.29	93.41±0.32	93.87±0.24
Textures	64.77±0.12	63.55±0.19	66.05±0.12	82.57±0.46	83.31±0.31	83.60±0.34
Traffic Signs	62.92±0.36	60.84±0.29	57.32±0.26	77.95±0.31	78.11±0.24	73.39±0.29
Aircraft	63.44±0.69	62.64±0.62	63.45±0.65	77.66±0.52	77.66±0.59	78.21±0.58
Omniglot	80.48±0.35	83.56±0.28	83.81±0.25	93.65±0.15	95.26±0.09	94.81±0.19
VGG Flower	93.81±0.11	94.16±0.23	93.50±0.24	98.81±0.11	99.06±0.09	98.95±0.09
MSCOCO	68.74±0.35	67.28±0.32	66.05±0.12	81.00±0.31	81.08±0.35	81.37±0.24
Quick Draw	63.07±0.44	62.54±0.59	64.49±0.58	82.11±0.41	82.78±0.37	82.78±0.28
Fungi	53.90±0.14	53.10±0.27	53.83±0.3	72.28±0.13	73.28±0.10	73.69±0.14
BSCDFSL	Plant Disease	73.99±0.35	75.66±0.33	75.13±0.33	90.27±0.38	91.78±0.31	91.84±0.19
ISIC	29.66±0.44	29.40±0.34	28.84±0.44	40.27±0.39	39.54±0.40	38.91±0.44
EuroSAT	64.49±0.34	63.99±0.39	60.39±0.43	81.14±0.25	80.96±0.19	77.25±0.16
ChestX	21.76±0.24	22.27±0.28	22.31±0.17	23.88±0.17	25.08±0.14	24.95±0.15
DomainNet	Real	92.48±0.20	89.96±0.26	92.45±0.28	97.22±0.05	97.16±0.02	97.36±0.04
Sketch	79.01±0.56	73.84±0.56	79.27±0.38	90.55±0.28	89.87±0.16	91.2±0.19
Infograph	65.48±0.27	61.93±0.47	65.61±0.17	81.52±0.48	80.87±0.30	82.02±0.37
Painting	79.34±0.31	74.92±0.33	79.06±0.32	90.98±0.16	90.26±0.22	91.37±0.21
Clipart	85.84±0.45	81.55±0.26	85.86±0.21	94.46±0.09	94.06±0.13	94.83±0.11
Table 6: Results on OODataset. FT stands for fully fine-tuning; WiSE-FT stands for performance after model fusion using Zero-Shot and fully fine-tuning; FD-Align stands for performance using our method.