FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning
Authors: Kun Song
Huimin Ma
Bochao Zou
Huishuai Zhang
Weiran Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: few-shot learning
CLIP
fine-tuning

Abstract: Due to the limited availability of data, existing few-shot learning methods trained from scratch fail to achieve satisfactory performance. In contrast, large-scale pre-trained models such as CLIP demonstrate remarkable few-shot and zero-shot capabilities. To enhance the performance of pre-trained models for downstream tasks, fine-tuning the model on downstream data is frequently necessary. However, fine-tuning the pre-trained model leads to a decrease in its generalizability in the presence of distribution shift, while the limited number of samples in few-shot learning makes the model highly susceptible to overfitting. Consequently, existing methods for fine-tuning few-shot learning primarily focus on fine-tuning the model's classification head or introducing additional structure. In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align). Our method aims to bolster the model's generalizability by preserving the consistency of spurious features across the fine-tuning process. Extensive experimental results validate the efficacy of our approach for both ID and OOD tasks. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements. Our code can be found in https://github.com/skingorz/FD-Align.

FD-Align: Feature Diserimination Alignment for
Fine-tuning Pre-Trained Models in Few-Shot Learning
Kun Song'
Huimin Ma
Bochao Zou'
Huishuai Zhang?
Weiran Huang
SCCE, University of Science and Technology Beijing
"Qnn Yuan Research Institute, SEIEE, Shanghai Jiao Tong University
3 Microsoft Research Asia
congkun@xs.ustb.eet (mhmpub, zoubochao) @ustbeddu.en, huzhang microsoft.com com, weiran huann wotaannkoullooccom
Abstract
Due the limited availability of data, existing few-shot learning methods trained from scratch fail to achieve satisfactory performance. In contrast, large-scale pre-trained models such as CLIP demonstrate remarkable few-shot and zero-shot capabilities. To enhance the performance of pre-trained models for downstream tasks, fine-tuning the model on downstream data frequently necessary. However, fine-tuning the pre-trained model leads to decrease in its generalizability in the presence of distribution shift, while the limited number of samples in few- shot learning makes the model highly susceptible to overfitting. Consequently, existing methods for fine-tuning few-shot learning primarily focus on fine-tuning the model's classification head introducing additional structure. In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align). Our method aims to bolster the model's generalizability by preserving the consistency of spurious features across the fine-tuning process. Extensive experimental results validate the efficacy of our approach for both ID and OOD tasks. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements. Our code could be found in https:// github. wwt.coooossii/eoc http///kinoozz/D---. Align.
Introduction
The Contrastive Language-Iman Pre-training model (CLIP) [1] represents a groundbreaking develop- ment in multi-modal deep learning. By utilizing contrastive learning, CLIP aligns visual and textual representations within unified embedding space, and exhibits superior performance
variety of downstream tasks, including image classification [2, 3], detection [4], and segmentation [5, 6]. which typically done by fully fine-tuning CLIP using downstream data. However, many real-world scenarios, there is often an insufficient amount of labeled data available. Thus, fully fine-tuning will lead to overfitting and significantly diminishing the model performance.
To mitigate this few-shot challenge, we consider using proxy dataset correlated to the downstream target dataset for fine-tuning CLIP, aiming obtain model that can efficiently generalize the few-shot target task. Directly fully fine-tuning CLIP on the proxy dataset is not feasible, as the fine- tuned model may overfit to the proxy data or have worse -uitt---isti (OOD) generalization [7], limiting its performance on the target task. As an example illustrated in Figure 1, the fully fine-tuned CLIP tends to focus more on local regions and less on the foreground compared to the original CLIP.
Cooresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original Image
(b) CLIP
(c) Fully Fine-Tuned CLIP
(d) FD-Align
Figure 1: (a) An image of a dog. (b) CLIP attention map, which pays more attention to the background in comparison with that to the dog. (c) The CLIP attention map after fully fine-tuning, which focuses more on locations with non-salient features. (d) The attention map after FD-Align tune-tuning, which tends to prioritize the dog's 's causal information while also paying attention to a minor portion of spurious information.
Such localized attention will weaken the model's robustness to spurious correlations [8], resulting in poor OOD generalization of the fully tunettuned CLIP.
In this paper, our objective to preserve the robustness of CLIP to spurious correlations during fine-tuning, i.e., its ability to distinguish between spurious and causal features. In particular, causal features represent the features related to the classes, while spurious features could be the features related to the context of the class. We want the fine-tuned CLIP can learn the causal features of the new classes while keeping the recognition ability for the spurious features. To achieve this, we propose method of Feature Discrimination Alignment (FD-Align). Specifically, we introduce spurious feature classifier, ensuring that the classification probability distribution of spurious features remains consistent throughout the fine-tuning, Taking advantage of the potent alignment capabilities of CLIP's text and visual features, we utilize text features category aggossiic descriptions (i.e., contexts) as spurious feature prototypes. The image features and spurious feature prototypes subjected to similarity measurement to ascertain the current image's probability distribution over spurious features. By constraining the probability distribution of the image features extracted by the model hefore and after fine-tuning, we ensure consistency in the spurious features extracted by the model. Concurrently, while learning classification capabilities on the proxy dataset, the robustness of the model to spurious associations after fine-tuning is also ensured.
Our method maintains the model's 's robustness to spurious correlations while fine-tuning model on proxy dataset. As illustrated in Figure 1d, on the one hand, compared to CLIP, the fine-tuned model with FD-Align better focuses the dog. On the other hand, compared to the localized attention of fully fine-tum finettuning CLIP, FD-Align pays slight attention to some spurious information. This balance between attending causal information and spurious information ensures the model's robustness to spurious correlations, thereby ensuring the model's OOD generalization. Extensive experiments validate the robust OOD performance of our approach, alongside the improvement of in-distribution (ID) performance. Furthermore, as shown in Figure 2, the model fine-tuned by FD-Align directly enhances the accuracy of existing methodologies without introducing additional inference cost.
Our paper makes the following contributions:
(1) We propose to use textual features to obtain spurious features of images;
ID-45gn
F3-Alig
(2) We propose a feature discrimination alignment fine- tuning architecture that ensures the OOD performance of the fine-tuned model by aligning spurious features ex- tracted model before and after fine-tuning:
APE-T
(a) ImageNct-S
(b) ImagoNet- V2
(3) Sufficient experiments demonstrate that our approach can significantly improve the performance of ID and OOD for few-shot learning, and it can improve the performance of existing methods without introducing additional training and inference costs.
Figure 2: Performance improvement for different datasets.
2 Related Work
Few-Shot Learning. The primary objective of few-shot learning is to train models of superior performance with small number samples. Prior methods primarily train models on base data and assess their performance on novel class data devoid of any shared categories. Approaches such as

---

MAML [9] adopt meta-learning to train base learner on base data, followed by fine-tumi fine-tuning it on limited amount of novel data derive a model suited for novel data. ProtoNet [10] introduces the use of metric learning for training. Recently, a line of work start to introduce the modality of text into the few-shot image classification, such as AM3 [11], TRAMI. [12] and FILM [13]. All these models are trained from scratch. With the emergence of pre-trained bi-modal models such as CLIP, the acquisition of more precise image features is presently achievable by leveraging these pre-trained models. Consequentyy, the majority of current research focuses how use features extracted by CLIP to enhance the capability of few-shot learning. For instance, CoOp [14] and CoCoOp [15] model prompt's 's context words with learnable vectors, keeping all pre-trained parameters fixed. Tip- Adapter [3] and APE [16] do not require any backpropagation to train the adapter but create the weights through key-value cache model constructed from the few-shot training set. VPT [17] introduces additional learnable parameters into the input space. However, all of these methods processed with the backbone frozen, while our paper aims to further explore the possibility of fine-tuning the backbone itself. Although these methods show excellent performance, they do not exploit the potential of pre-trained models.
Fine- Tine-Tuning of Pre-trained Model. The most direct approach is fine-tuning the pre-trained model directly. However, fully fine-tuning decreases the OOD performance of the pre-trained model [7]. WiSE-FT [18] enhances performance by ensembling the weights of the zero-shot and fine-tuned models. Kumar et al. [7] first perform linear probing and then perform fully fine-tuning to ensure the OOD performance of the model. Xuhong et al. [19] introduce an additional regularizer to constrain the l2 distance between zero-shot and fine-tuned CLIP. On the other hand, Mukhoti et al. [20] prevent the degradation of the foundational model's ' capacity by constraining image features extracted by zero-shot and fine-tuned CLIP in the feature space. Nevertheless, scant existing methods delve into the strategies for fine-tuning pre-train pre-trained models using limited data.
Spurious Correlations. Spurious correlations denote deceptive heuristics that hold for the majority of instances during training, but do not invariably apply [21]. For example, when the training causal
instance involves a cow grass, it is tempting to misconstrue the presence of grass determinant for categorizing cows. This misinterpretation, regarding grass as cows, embodies a form of spurious correlation. Eliminating spurious features generally leads enhanced OOD performance; however, this can be accompanied by a decrease in ID performance [22]. Furthermore, even armed with a comprehensive understanding of the spurious feature, its extraction remains non-trivial task [23]. The impact of spurious correlations persists within multimodal pre-trained models [24]. However, when model parameters and dataset sizes large enough, Vision Transformer (ViT) exhibits increased robustness against spurious correlations [8]. Furthermore, the CLIP architecture enhances the robustness of the vision encoder to spurious correlations [25]. Consequently, improving robustness against spurious correlations can effectively maintain the OOD performance of the model.
3
The Proposed Method
3.1 Problem Definition
Suppose that we have a pre-trained CLIP [1] which contains a visual encoder fo and a text encoder go- In addition, have access few-shot proxy dataset D X x V, where each class has very limited samples and each sample comprises an image & and its corresponding label y. The objective is to fine-tune the pre-trained CLIP using this proxy dataset, aiming enhance its zero-shot performance for unseen target tasks related to the proxy dataset.
3.2 Fine- Tone-Tuning on Proxy Dataset
We freeze CLIP's text encoder gu during fine-tuning and make the visual encoder learnable. First, we initialize the visual encoder fi using the parameters of the visual encoder of pre-trained CLIP fo- The visual encoder fe is then used to extract the feature fi f((z) of the image z. With the help of the remarkable text-vision alignment capability of CLIP, we use the textual features of the class names of each class as class prototypes. Following CLIP, for any class y, we combine M prompt templates (PI,..., Pas) with the class name and obtain M prompts [Pr,y],.... [PM. y). We then use the text encoder go to extract the features of the above M prompts. Subsequently, we calculated the means of
3

---

Frozen model
Text Embeedinn
Image Feature
NNN
Class Prototype mM]l
Spurious Prototype
Spurious Feature Constraint
Visual Encoder
Class
aplalc ofS3
CLIP
Embardiing
Pampiss
Avg.
ere
Encoder
Seybone ii
Spurious Prokeeves
Espurtous
Avg
Class Protoyye
KSSSSS
SSNNN
Trciass
Visual
Encoder
Fine Tune Model
Figure 3: The class names and prompts are combined and inputted into the text encoder to obtain text embeddings. We calculate the mean separately in the prompt and class dimensions to derive the class prototype and prompt embedding. On the one hand, the image features are extracted using the fine-tuned visual encoder, and class distribution are calculated based on the class prototype to calculate the class loss. On the other hand, we use spurious prototype correction (SPC) module to correct the prompt embedding. By calculating the cosine similarity between the image features and the spurious prototype, we obtain the distribution over spurious features and calculate the spurious loss.
the M features obtain the prototype of the corresponding class, namely, the prototype of class y is
M
class
= Σ vl). M 1=1
1
y
We calculate the cosine similarity s(.,-) between the image feature and the class prototypes and produce a distribution over classes for the image. Finally, the class loss is calculated using cross- entropy loss by
exp(s(f((11)) class) Eveyexple exp(s((i (x;), (g
Class
log (DD18))D
DD
where y is the label set.
3.3
Spurious Feature Constraint
Fully fine-tuning CLIP on the proxy data affects the robustness of the model to unseen data. To preserve the model's performance on out-of-distribttion data during fine-tuning, we maintain the model's robustness to spurious correlations during fine-tuning. That is, keeping the spurious feature extracted by the model before and after fine-tuning unchanged. We first calculate the mean of the features of each prompt template P3 over all classes as prototypes of the prompt template Pj, that is,
sparious
Σ g(P,.))))))) DEY
DJ
We can calculate the similarity between the feature extracted by the fine-tuned model and the spurious prototypes and produce the distribution over spurious features follows.
P spurrooss (2; ft) SoftMax (Ji(z), spurioos)
(Je(z),
spurious
)]
Mas

---

Similarly, we can use the pre-trained visual encoder fo to extract the feature and produce the distribution over spurious features follows.
P Pspurious (z; fo) = SoftMax (fo(z), spurions
(fo(z)) (fo(2)).PPa) PPs
sparious
We ensure that the spurious features of the models before and after fine-tuning remain consistent by keeping the probability distributions of the models over spurious features consistent before and after fine-tur the-ttnnng, that is
Eppurious
Σ
KL (Pspurious spurious (Iz; fi) ||
P Squrious (304 fo))
(2)
DDi
JED
Finally, we optimize both (1) and (2) during fine-tuning to ensure classification ability and OOD robustness:
Listal Lelnes + B Lsparionss
where we a to 1 and / to 20 in this paper.
3.4 Spurious Prototype Correction
Prompt templates are typically designed manually or generated by large-language models such as GPT. These templates often include redundant illogical prompts. Consequently, the prototypes of spurious features calculated by these imprecise and redundant prompt templates lack accuracy. Therefore, filtering and processing these prototypes of spurious features is necessary.
Certain prompt templates may exhibit a lack practical significance or irrationality, making them unsuitable for incorporation spurious features, exemplified by "itap of [class]." This scenario can result in the inaccuracy of spurious prototypes. To address this problem, we employ the Isolation Forest algorithm [26] to eliminate meaningless prototypes associated with spurious features, i.e., 4 spuriions = ISOLATION ppurious n)). We will retain the n prototypes that exhibit the highest degree of rationality.
Moreover, there are cases where certain prompts exhibit excessive similarity. For example, prompts such as "a photo of the [class].", "a photo of (class).". and "a photo of my (class)." show notable similarities. such scenarios, a single piece of spurious information may correspond to multiple prompts. However, some spurious information aligns with only one prompt. Consequently. the relative weight of the probability of spurious information corresponding to a single prompt diminishes during the classification process. To address this issue, employ the k-means algorithm to merge duplicate spurious features that arise from similar prompts, i.c., spprrious := k-means(yePunn) opurious k). where & is the number of cluster centers.
4 Experiments
In this section, we will validate the OOD robustness of our approach. Furthermore, our method demonstrates promising results in ID data.
4.1 Setup
Throughout this section, if not otherwise specified, we use open source ViT-B/32 as the backbone of the CLIP, and the prompts we use are OpenAl ImageNet prompt templates we set to 60, & to 20 in the spurious prototype correction stage. We employ the Stochastic Gradient Descent (SGD) optimizer for model fine-tuning, conducting the process over 60 epochs. Since little prior work on CLIP has been done to fine-tune the backbone for few-shot learning, we mainly compare our method with the fully fine-tuning strategy and WiSE-FT [18].
for Enginnecrinnggoooomggttteiiaaa

---

Dataset
For the OOD setting, evaluate method on two different datasets. On the one hand, train the model on ImageNet [27] following the few-shot task in CLIP and test the performance on two OOD variants of ImageNet [27]: ImageNerV2 [28] and ImageNet-Sketeh [29] with the same 1000 classes, On the other hand, we follow the traditional few-shot learning strategy and fine-tune the model on the train split of minilmageNet and evaluate the model on Meta-dataset [30], BSCDFSL benchmark [31] and DomainNet [32], for total of 19 datasets. These datasets cover a wide range of image styles, such as natural images, satellite images, medical images, and sketch images.
For ID setting, we use the 11 image recognition datasets following CoOp [14], which contain ImageNet [27]. StanfordCars [33], UCF101 [34], Caltech101 [35], Flowers 102 [36], SUN397 [37], DTD [38], EurOSAT [39], FGVCAireraft [40], OxfordPets [41] and Food101 [42]. These datasets cover wide range of different visual recognition tasks such generie object classification, fine- grained classification, action, scene, texture, etc.
Baselines
Baselines FD-Align
Method
CLIP
FT
Tip
Tip-F
APE
APE-T
Tip
Tip-F
APE
APE-T
ImageNet
63,34
64.91
65.49
68.43
66.55
68.74
66.39
65.49
68.70
67.59
69.15
ImageNetS
42.31
42.24
42.48
42.54
43.28
43.23
43,50
43.84
43.67
44.23
44.04
ImageNetV2
55,92
57.63
57.58
59.58
58,31
59.58
57.73
59.10
60.17
59.36
60.83
Table 1: OOD results. We fine-tune the model on 16-shot ImageNet and evaluate it on the variants of ImageNet. FT means fully fine-tuning.
4.2 Results under OOD Setting
In order to evaluate the robustness of our method in OOD data, we fine-tune the model on ImageNet in 16-shot settings and test the performance on the variants of ImageNet. As shown in Table in contrast to fully fine-tuning CLIP, our approach yields a performance boost of up to 1.75% on ImageNetA. Additionally, we directly integrate our fine-tuned backbone into Tip-adapter [3] and APE [16], leading to a substantial improvement in their generalization performance. The aforementioned findings demonstrate that FD-Align improves model performance while ensuring robust generalization. Furthermore, the fine-tuned model can seamlessly integrate into existing methods without requiring additional fine-tuning.
Sway-1shot
Sway-5shot
12.
10:
Sayes 4-
Chertit
igussaa
Omniggo
abraw
Power
Aircraft
Fungi
Drspase
Teatures
miri
MBcaco
cobo test
edig
Cipart
Pointing
Starthh
Signa
Chertt
Burds
fsic
oDran
Flownt
Ornnggot
A pipltes
Teatures
Airroft
Funq
Clpart
Diaaase
cub test
MacoCc
Painting
Slatch
Figure 4: Performance improvement/decline of our method on different OOD datasets.
In addition, we also evaluate the robustness of our method in the N-way K-shot few-shot learning task. We fine-tune the model on the minilmageNet train split utilizing prototypical networks [10] and subsequently evaluate its performance on several different datasets. In our testing phase, we evaluate the precision of 2000 tasks for 5-way- 1shot and 5-way-5shot scenarios in five distinct sets generated according to different random seeds. We then calculate the mean accuracy along with a confidence interval 95% As illustrated in Table 2 and Figure 4, our approach yields notable performance enhancements across the majority of datasets. In particular, on Sketch dataset, our method achieves a performance improvement of 12.03% compared to CLIP. Nevertheless, on certain datasets, such as Traffic Sign, marginal decrement in performance is observed with our approach. This may

---

Sway-Ishot
5way-5shot
Datasets
CLIP
WiSE-FT
FD-Align
CLIP
WiSE-FT
FD-Align
Mini-test [43]
88.21+0.33
93.35a0.11 17
95.04+0.18
97.46+0.07
98.44=0.06
98.5220007
CUB-test [44]
75.21
86.1610.71
82.38+0.69
91.48+0.34
93.41=0.32
93.87=0.24
Textures [38]
61.26#0.24
63.55a0.19
66.0550.12
82.40+0.40
83.3140.31
83.60:0.34
Traffic Signs [45]
58.51
60.8440.29
57.32+0.26
76.67=0.18
78.11=0.24
73.39=0.29
Aircraft [40]
60.57=0.54
62.64=0.62
63.45+0065
76.35+0.59
77.66+0.59
78.21
Omniglot [46]
83.27=0.37
83.56+0.28
83.81=0.25
94.29#0.13
95.2620.09
94.81=0.19
VGG Flower [36]
90.88=0.31
94.1620.23
93.50=0.24
98, 10
99.0620.09
98.95=0.09
MSCOCO [47]
62.30=a.38
67.28=0.32
69.1620.28
78.93=0.38
81.08+0.35
81.3720.24
Quick Draw [48]
62.2240.61
62.5420.59
64.49=0.58
82,65
82.78=0.37
82.78+0.28
Fungi [49]
50 50.42+0.32
53. 13.10:0.27
53.83+0.30
71.59+0.18
73.28+0.10
73.69=0.14
Plant Disease [50]
70. 0...44
75.66#0.33
75.13+0.33
89. 09.500
91.78=0.31
91.8420.19
ISIC [31, 51]
28 66.0.35
29.40=0.34
28.84
39. 39.02+0.24
39,54+0.40
38. 08.91
EurOSAT [39]
60.20#0.48
63.99+0.39
60.39++.43
77.43+0.21
80.9620.19
77.25=0.16
ChestX [52]
22.65=0.27
22.27+0.28
22.31
25.58
25.08&0.14
24.95
Real [32]
84 84.84+0.32
89. 89.9640.26
92.45+0.28
16.39
97.1640.02
97.36220.4
Sketch [32]
67.24=0.60
73.84=0.56
79.27=0.38
87.66=0.27
89.87=0.16
91.20e00.1
Infograph [32]
55.72+0.21
61.93+0.47
65.61=0.17
78. 23.30
80.87+0.30
82.02+0.37
Painting [32]
18.05=0.18
74.92+0.33
79.06:0.32
87. 87.99#0.21
90.26+0.22
91.37=0021
Clipart [32]
15.14=0.12
81.55=0.26
85.8620.21
92.52+0.09
94.06#0.13
94.8320.11
Table 2: The performance of CLIP, WiSE-FT and FD-Align with SPC (FD-Align)
be attributed to the dataset images primarily encompassing the object of identification, devoid of substantial contextual misinformation that might adversely impact direct fine-tuning. Moreover, our
method consistently outperforms WiSE-FT [18] in most datasets. In the few instances where we do not surpass its performance, our results remain comparable.
accurary
76
WISE
Methods
Ishot
8shot
16shnt
74
Tip
64.11
64.36
64.63
65.17
65.49
Align
Tip FD-Align
64.51
65.33
65.76
66.79
67.28
Tip-F
64.64
65.18
65.78
67.21
68.43
Tip FD-Align
64.86
65.61
66.11
67,58
68.70
APE
65.36
65.69
66.00
66.55
66.55
FD-Align
66.71
67.29
67.40
67.76
67.69
APE-T
65.89
66.18
66.82
67.99
68.74
2
APE-T + FD-Align
66.84
67.37
67.81
68.73
69.15
Shot Number
Figure 5: Performance comparison of different methods on 11 datasets.
Table 3: Results on ImageNet using our backbone with different methods.
4.3 Results under ID Setting
In addition, we evaluate the performance of our method using ID data. Models were trained on various datasets using 1, 2, 4. and 16 shots, respectively, and subsequently evaluate performance on the respective datasets. Figure 5 illustrates a comparative analysis of the average performance between our method and existing methods on datasets. As depicted, our method significantly outperforms other strategies and yields increasingly substantial performance enhancements as the shot number increases. Previous research indicates that WiSE-FT effectively elevates the fine-tuning performance of models in zero-shot learning. However, in few-shot learning, relative to fully fine- tuning, employing WiSE-FT with a fusion parameter of 0.5 results in a performance notably inferior to comprehensive fine-tuning, Our analysis posits that, in few-shot learning, the paucity of fine-tuning samples results in minimal alterations in model fine-tuning. Consequently, following fusion, the model parameters more closely resemble those of CLIP, thereby influencing the model performance.
7

---

4.4 Discussion
4.4.1 The Generality of FD-Align
To validate the generalizability of models fine-tuned via FD-Align across various methods, we directly incorporated the FD-Align fine-tuned visual encoder into existing approaches and evaluate ID performance. As shown in Table 3, the visual encoder fine-tuned with FD-Align markedly improves the performance of existing methods, achieving peak improvement of 1. 1.79%. The visual encoder fine-tuned with FD-Align, can be seamlessly transitioned into existing methods, boistering performance without incurring additional costs.
Methods CL.IP
Ishot
2shot
4shot
8shot
16shot
60.33
Fully Fine- sine-tuning CLIP
63.48
64.87
68.10
71.14
73.43
FD-Align (80 templates)
63,90
65.64
68.
71.30
74.03
FD-Align (Tip templates)
61.14
62.39
63.37
60.34
66.30
FD-Align SPC
63.92
65.68
68.63
71.66
74.38
Table 4: Ablation results. All methods use the same learning rate and calculate the average perfor- mance across 11 datasets. The setup of 80 templates means using all the features of 80 templates as spurious prototype. The setup of Tip templates means using the prompts templates proposed by Tip-Adapter, and SPC means spurious prototype correction.
4.4.2 Why Remove Outliers and Cluster?
In this section, we analyze the importance of eliminating outliers from spurious features and the subsequent clustering of residual points via empirical analysis. As illustrated in Table employing the Spurious Prototype Clustering (SPC) method yields enhanced performance for the spurious prototype relative directly utilizing all 80 templates. This underscores the efficacy of removing outliers and redundant values from spurious features. Furthermore, we experimented with features derived from manually designed prompt templates as spurious prototypes. While Tip-adapter handpicked 7 templates, our findings indicate that leveraging these 7 templates directly as spurious prototypes closer examination revealed that the template "itap
precipitates marked performance degradation. of a [class]" from Tip-adapter was identified and excluded as an outlier by SPC during its initial phase. This underscores the ability of SPC to autonomously filter outliers of spurious prototypes. thereby circumventing potential pitfalls inherent in manual selections.
Nonmal Embedding
Prompt Embedding
Cutlier Eribadding
Kmoans Embedding
(a) All prompt templates
(b) Isolation Forest
(c) k-means
Figure 6: (a) represents all the features of prompt templates. (b) represents the features processed by Isolation Forest. (c) represents the final results clustered by k-means.
To more intuitively illustrate the impact of outlier removal and clustering, we visualize the spurious features. Figure displays the features all the prompt templates within the feature space, revealing the conspicuous presence of outliers and redundant values. Figure 6b shows the visualization after outlier elimination: red points represent preserved values, while black points denote deleted outliers, including "itap of (class]". It is evident that numerous redundant values persist. As depicted in

---

Figure 6c, after the clustering process, the residual spurious features are uniformly distributed across various positions, thereby providing rational representation of the spurious prototype.
citice 1880
#TLpwninn CETION pr8te
Jaat
BTL20RI cattee
(a) CLIP
(b) Fine-tuned CLIP
(c) FD-Align CLIP
Figure 7: Feature visualization for the same class under different domains.
4.4.3 OOD Visualization
To more vividly demonstrate the influence of our method on model feature extraction, we conducted a visualization of the image features on PCAS [53], utilized for investigating OOD tasks, which contains category and domain annotations for images. Figure 7 illustrates the visualization of features for one of our categories. is observable that CLIP adeptly distinguishes features across different domains. The ability of fully fine-tuned CLIP to differentiate information across various domains comparatively diminished. After fine-tuning with FD-Align, the model reacquires the ability to discern information across different domains, substantiating FD-Align's 's ability to preserve the model's s OOD performance. Additionally, as depicted in Figure 8, we visualize the features of different categories in various domains. On the sketch data, the model, after fully fine-tuning, exhibits a diminished capacity to differentiate between categories, whereas CLIP and FD-Align distinguish between various categories accurately.
catooat hore
deeeh dog
outeen, tig arbsnn honn
canion, Fars anilth dog
dog
(a) CLIP
(b) Fine-tuned CLIP
(c) FD-Align CLIP
Figure 8: Feature visualization for different class under different domains.
4.4.4 Training Stability
Figure 9 depicts the evolution model accuracy and loss on the validation set throughout the fully fine-tuning and FD-Align processes. Notably, during the initial phases of fine-tuning. fully fine-tuning prematurely encounters overfitting, culminating in a decline in accuracy and surge in classification loss. Conversely, FD-Align adeptly maintains model robustness and stability throughout the fine-tuning process, effectively circumventing overfitting.
4.4.5 Limitation
As shown in Tables 2, our method is able to preserve the generalization of the model while fine-tuning the model. However, in certain specific cases, such as Traffic Sign, where the images primarily consist of target objects and lack enegoor--iddependent ability adversely impacts the performance on such data. Additionally, our method does not perform well on special scenarios, such as ChestX and EuroSAT, for which one possible reason that the prompts we use do not contain the category-independer information of these data. Therefore, additional prompts may need to be designed for this type of data.
information, the preserved pre-trained model's

---

0.96
Accuracy
Loss
0.25
Fully
0.24
FD-Align
0.23
0.95
0.21
0.94
0.20
Fully
0.18
0.93
FD-Align
0.17
100 200 300 400 500 600
700
0
100 200
300
400 Step
500 600 700
Figure 9: Validation accuracy and loss variation during the training process.
Conclusion
In this paper, introduce feature discrimination alignment fine-tuning (FD-Align) method pre- trained models in few-shot learning. Leveraging the remarkable text-visual alignment capabilities of CLIP, employ text features of category-agnosta descriptions spurious feature prototypes. Furthermore, we constrain the probability distribution of image features extracted by the model on spurious features, both before and after fine-tuning, ensure the robustness of the model subsequent to fine-tuning, Experimental results substantiate the efficacy of our approach in enhancing fine-tuning performance while ensuring robustness across distributions.
Acknowledgment
Huimin Ma and Bochao Zou are supported by the National Natural Science Foundation of China U2082202, No. 622272222 No.62206015) and National Key Research and Development Program of China (2022ZD0117900), Weiran Huang is supported by 2023 CCF-Baidu Open Fund and Microsoft Research Asia.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[2] Marcos V Conde and Kerem Turgutlu. Clip-art: Contrastive pre-training fine-grained art classification. In CVPR, 2021.
[3] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free free clip-adapter for better languaee modeling. arXiv preprint arXiv:21 2021.
[4] Sepideh Esmaeilpour, Bing Liu, Eric Rohertson, and Lei Shu. Zero-shot out-of-distribtiio detection based the pre-trained model clip. AAAI, 2022.
[5] Zhaoqing Wang, Yu Lu, Qiang Li, Xungiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In CVPR, 2022.
[6] Kwanyong Park, Sanghyun Woo, Seoung Wug Oh, So Kweon, Joon- Joon-Young Lee. Per-clip video object segmentation. In CVPR, 2022.
[7] Ananya Kumar, Aditi Raghunathan, Robbic Matthew Jones, Tengyu Ma, and Percy Liang, Fine-tuning distort pretrained features and underperform out-of-distribution. In /CLR, 2022.
[8] Soumya Suvra Ghosal, Yifei Ming, and Yixuan Li. Are vision transformers robust to spurious correlations In /CML 2022: Workshop Spurious Correlations, Invariance and Stability, 2022.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In JCML, 2017.
[10] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NeurIPS. 2017.
[11] Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. Adaptive cross-modal few-shot learning. In NewrIPS, 2019.
10

---

[12] Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng. Zhenguo Li, and Liwei Wang. Boosting few-shot learning with adaptive margin loss. In CVPR, 2020.
[13] Zihao Jiang, Yunkai Dang, Dong Pang, Huishuai Zhang, and Weiran Huang. Film: How can seo-shot image classification benefit from pre pre-trained language models? arXiv preprint arXiv:2307.041 2023.
[14] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. 1/CV, 2022.
[15] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- saiionlanguage models. In CVPR, 2022.
[16] Xiangyang 7hu, Renrui 7hang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint arXiv:2304.01 195, 2023.
[17] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[18] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Korblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In CVPR, 2022.
[19] LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In JCML. PMLR, 2018.
[20] Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308. 13320, 2023.
[21] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Perey Liang. Distributionally robust neural networks. ICLR, 2020.
[22] Ananya Kumar, Tengyu Ma. Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradcoff's under distribution shift. In UAI, 2022.
[23] Rafayel Darbinyan, Hrayr Harutyunyan, Aram H Markosyan, and Hrant Khachatrian. Identifying and disentangling spurious features in pretrained image representations. arXiv preprint arXiv:2306. /2673, 2023.
[24] Yang, Besmira Nushi, Hamid Palangi. and Baharan Mirzasoleiman. Mitigating spurious correlations in multi-modal models during fine fine-tuning. arXiv preprint arXiv:2304.0391 2023.
[25] Anil Palepu and Andrew Beam. Self-supervision images and text reduces reliance on visual shortcut features. In ICMI. 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
[26] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining. IEEE, 2008
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJV, 2015.
[28] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML. PMLR, 2019,
[29] Ross Wightman. Pytorch image models. https://glie wtt.thwww....//gtttaaaaaaaaaaaae 2019.
image-modelese
[30] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Mcta-datasct: A dataset of datasets for learning to learn from few examples. In ICLR, 2020.
[31] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.
The ham10000 dataset, large collection of molti-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1), 2018
[32] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019.
[33] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, 2013.
[34] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos the wild. arXiv preprint 122.0022,, 2012.
[35] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. IEEE, 2004.
[36] Maria-Elewe Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, IEEE, 2008.
11

---

[37] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large- scale recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 2010.
[38]
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[39] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep lcarning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7), 2019.
[40] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- sine-grained visual classification of aircraft. arXiv preprint arXiv: 130..5151, 2013.
[41] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR. TEFE, 2012.
[42] Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In ECCV. Springer, 2014.
[43] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurlPS, 2016.
[44] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uest birds-200-2011 dataset. 2011.
[45] Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. IJCNN. IEEE, 2013.
[46] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266), 2015.
[47] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV. Springer, 2014.
[48] Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The quick, draw!-ai experiment. Mount View, CA, accessed Feb, 17(2018), 2016.
[49] Brigit Schroeder and Yin Cui. Fgvex fungi classification challenge 2018. Available online: github. com/visipedhafgyvss_um ftwgi_comni_opooao_ee (accessed on 14 July 2021), 2018.
[50] Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant discase detection. Frontiers in plant science, 7, 2016.
[51] Noel Codella, Veronica Rotemherg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo,, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 201 8: A challenge hosted by the international skin imaging collahoration (isic). arXiv preprint 4r0iv:1902.033: 03368, 2019.
[52] Xiaosong Wang. Yifan Peng. Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax discases. In Proceedings of the IEEE conference computer vision and pattern recognition, 2017.
[53] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- Meta-learning for domain generalization. In AAAI 2018.
12

---

Appendix
A Detailed Results of Different ID Datasets
Figure 10 delineates the effectiveness of our approach across different datasets. Impressively, FD- Align surpasses both WiSE-FT and the fully fine-tuning on a vast majority of these datasets. As the shot count rises, the advantage of FD-Align becomes even more salient. is pertinent to note that when data is scarce, WiSE-FT tends to underperform compared to fully fine-tune.
disn
Jipe
Seruint
02.31
785
2977 1477100180
1EP0331
30.4
71.
CUP
0,100A
24-77 1111100000
3407 NOMMA
Figure 10: Performance comparison different methods on 11 datasets.
B Time Cost of FD-Align
Table 5 shows the time cost of fully fine-tuning and FD-Align 1-shot ImageNet. Evidently, FD-Align introduces only a negligible increase time overhead. is noteworthy that during the inference phase, both FD-Align and fully fine-tuning share identical processes, resulting in consistent inference cost. A significant advantage FD-Align is its ability to boost performance across varied methodologies with just fine-tuning once.
METHOD
Time
Fully Fine-tuning
7min 9s
FD-Align
8min 53s
Table 5: Time cost of Fully Fine-tuning and FD-Align.
C More Results on OOD Datasets
We also evaluate the performance of fully fine-tuned models on the OOD dataset. All models are trained on minilmageNet and performance tested on other datasets. As shown Table our method performs better on Meta-Dataset but worse on BSCDFSL. After analysis, we believe that the reason that the model is fine-tuned on minilmageNet, which mainly contains natural image information, the model better retains the domain information of natural images. However,
13

---

BSCDFSL mainly contains satellite images, medical images, and other nonnatural images, and our model fine-tuning does not retain the domain information of these datasets. Furthermore, our model affects the classification ability in the classification process. Therefore, on such datasets, our method may be slightly degraded compared the direct fine-tuning of CLIP.
5way-Ishot
5way-5shot
Datasct
FT
WiSE-FT
FD-Align
FT
WiSE-FT
FD-Align
mini_test
94.15=0.19
93.55=0.17 17
95.04+0.18
98.13+0.12
98.4440.06
98.52+0.07
cub-test
80.78+0.69
81.1640.71
82.38+0.69
92.95=0.29
93.41+0.32
93.87=0.24
Textures
64.77=0.12 12
63.55a0. 19
66.05#0.12
82.57=0.46
83.3140.31
83.60+0.34
Meta- Meta-dataset dataset
Traffic Signs
62.9220.36
60.84a0.29
57.3240.26
77.95+0.31
78.11+0.24
73.3940.29
Aircraft
63.44+0.69
62.6440.62
63.45:0.65
77.6640.52
77.66+0.59
78.21
Omniglot
80.4820.35
83,56=0.28
83.81=0.25
93.65=0.15
95.26+0.09
94.81=0.19
VGG Flower
93.81=0.11
94.1620.23
93.50=0.24
98.81=0.11
99.06+0.09
98.95=0.0r
MSCOCO
68.74=0.35
67.28=0.32
66.05=0.12
81 81.00=0.31
81.08=0.35
81.37=0.24
Quick Draw
63.07=0.44
62.54=0.59
64.49=0.58
82.11=0.41
82.78=0.37
82.78=0.28
Fungi
53.9020.14
10=0022
53.83+0.3
72.28=0.13
73.28+0.10
73.6920.14 14
BSCDFSL
Plant Disease
73.99=0.35
75.6620.33
75.13+0.33
90.27=0.38
91.78+0.31
91.84=0.19
ISIC
29.6620.44
29. 19,4004.34
28.8440.44 44
40.27=0.39
39. 39.54+0.40
38.91
EuroSAT
64.49=0.34
63.99=0.39
60.39+0.43
81.14+0.25
80.9640.19
77.25+0.16
ChestX
21.7620.24
22.27=0.28
22.31=0.17
23.88=0.17
25.08+0.14
24.95+0.15 15
Real
92.48+0.20
89. 89.9640.26
92.45+0.25
97.22+0.05
97.16+0.02
97.36+0004
DomainNet
Sketch
79.01
03.8
79.27+0.38
90.55=0.28
89.8740.16
91.2=0.19
Infograph
65.48+0.27
61.93+0.47
65.61+0.17
81.52+0.48
80.87+0.30
82.02+0.37
Painting
79.3420.31
74.92=0.33
79.0620.32
90.98=0.16
90.26+0.22
91.37=0.21
Clipart
85.8420.45
81.55=0.26
85.8620.21
94.4620.09
94.0620.13
94.83+0.11
Table Results on OODdataset. FT stands for fully fine-tuning: WiSE-FT stands for performance after model fusion using Zero-Shot and fully fine-tuning; FD-Align stands for performance using our method.
14


Reviews:
[{"review_id": "I1grQ5XD0G", "reviewer": "Reviewer_LEKB", "summary": "This paper presents a fine-tuning method for pre-trained models in few-shot learning via CLIP's text and visual feature alignment capability. Specifically, the authors use text information to assist in decoupling spurious information from causal information while keeping the spurious information unchanged during the training process. This way, the authors claim the proposed FD-Align model can maintain its generalizability while effectively mitigating the overfitting problem from previous fine-tuning approaches.", "questions": "According to the authors, keeping the original ability to extract spurious features can aid in solving the few-shot learning problem because it helps reduce overfitting. In situations where the image content is straightforward, identifying the correlation between the background and foreground can be a valuable tool for recognizing unfamiliar objects. For example, an airplane is commonly found in the sky or at an airport rather than in a mountainous environment. However, as image content becomes more complex, it is unclear how much background information is still helpful for this task. Additionally, causal/spurious feature disentanglement appears to be reliant on the quality of the prompt, which can introduce uncertainty into the performance since different individuals may provide different prompts.", "limitations": "The authors utilized the underperformed scores to explain the limitation of the proposed method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The concept of utilizing text to distinguish between causal and spurious feature extraction for enhancing the generalizability of few-shot model training is innovative. Additionally, the discovery that retaining the spurious feature extraction component unchanged pre and post fine-tuning can assist fellow researchers in addressing the few-shot learning issue is noteworthy. In addition to the originality of the concept, the results appear to be encouraging on various datasets when compared to other recent works. Finally, the authors also attached the code for easy reproducibility.", "weaknesses": "The presentation and illustrated figures could be improved for better clarity. Additionally, the experimental results do not show comprehensive superiority compared to WiSE-FT, indicating some drawbacks in the proposed approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The presentation and illustrated figures could be improved for better clarity.\n\nA1: Thanks for your feedback. In response to your suggestion, we have revisited and updated the visual representation of Figure 1. In the modified figure, the data flow and the loss term during the fine-tuning can be shown more clearly. For a more lucid depiction, please refer to **Figure 1 in the attached PDF of \"global\" response**. We believe the revised figures will offer better clarity and ease of comprehension. \n\nQ2: The experimental results do not show comprehensive superiority compared to WiSE-FT.\n\nA2: Thanks for bringing this to our attention. Upon careful examination, we determined that the minor superiority of our initial experimental results compared to WiSE-FT[2] was due to inadequate hyperparameter tuning. In fact, after re-tuning the hyperparameters of our method, a significant enhancement in performance can be observed. The updated results are shown in the table below.\n\n|      METHOD     |   1 shot  |   2 shot  |   4 shot  |   8 shot  |  16 shot  |\n|:---------------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|   CLIP(60.33)   |           |           |           |           |           |\n|   LP-CLIP [1]   |   22.17   |   31.90   |   41.20   |   49.52   |   56.13   |\n|   WiSE-FT [2]   |   63.36   |   64.27   |   66.16   |   68.13   |   69.55   |\n| Tip-Adapter [3] |   65.52   |   67.40   |   69.56   |   71.58   |   72.72   |\n|        FT       |   65.24   |   67.42   |   69.85   |   71.73   |   73.00   |\n| **FT+FD-Align** | **65.65** | **68.26** | **71.25** | **73.63** | **75.59** |\n\nQ3: It is unclear how much background information is still helpful as image content becomes more complex? \n\nA3: In few-shot learning, when the content in an image is more complex, it is more difficult for the model to learn the task-relevant causal information and overfit to task-irrelevant spurious information such as background. Our method can effectively avoid the model overfitting to task-irrelevant spurious information. The more complex the background information is, the more effective our method is.\n\nQ4: Causal/spurious feature disentanglement appears to be reliant on the quality of the prompt, which may lead to uncertainty in performance.\n\nA4: Thank you for your insightful question. Indeed, the performance will be influenced by the quality of the prompt. However, we would like to clarify that our method has conscientiously taken steps to mitigate this dependency. Firstly, our approach leverages an effective outlier removal algorithm (i.e., Isolation Forest) to eliminate impractical prompt instances, contributing to a more stable process and diminishing the influence of prompt quality. Secondly, we employ the k-means clustering to reduce redundant prompts, thereby ensuring a representative set of data. When sufficient prompts are available (e.g., large pre-trained language models like GPT can easily offer ample relevant prompts), the above techniques can reduce the impact of prompt quality on performance uncertainty to a minimal level. It's worth emphasizing that **an increase in the number of prompts does not adversely affect training time**, as context weight generation is executed solely prior to the training procedure. We hope this explanation addresses your concern and provides a clear understanding of our approach. We will add the above discussion into the revised paper.\n\n[1] Learning Transferable Visual Models From Natural Language Supervision\n\n[2] Robust fine-tuning of zero-shot models\n\n[3] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling"}, {"review_id": "yf3UV8UWBW", "reviewer": "Reviewer_EKx5", "summary": "This paper proposes a new few-shot learning method leveraging the pre-trained multi-model backbone CLIP. The method aims to eliminate the spurious information within the text embedding, and consequently regularize the image features to deliver better few-shot learning results. To do so, the authors propose to average pooling the text embeddings for the non-contextual information and enforce their KL distance with the image features. The evaluate the proposed method, the authors conduct experiments on multiple benchmarks in different settings. ", "questions": "See weakness.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-- The authors’ motivation sounds interesting. It’s a new angle to disentangle the text embeddings and accordingly regularize the image features. \n\n++ The results are looking good, the proposed method achieves good results on multiple benchmarks and settings (1, 2, 4, 8, 16 shots).", "weaknesses": "-- Although the motivation is clear and interesting and the authors claim to disentangle the causal information from the spurious information, I didn’t see any clear visualisations to support their claim. In Figure 1, the authors show better class centroids for their method, but it cannot directly show the effect of disentanglement. I suggest the authors show some similarity maps on both learned causal and spurious representations in future versions.\n\n-- For the same purpose, what if employing a small off-the-shell saliency model to first mask the spurious information?\n\n-- This paper can further benefit from improved organization. In Section 2.2, the authors simply list the related works without meaningful discussions. The readers, especially the ones that are not directly working on the same topic would be confused about details such as 1) Is it a common practice to jointly incorporate a frozen and a learnable visual encoder in few-shot learning with CLIP? Is any of the mentioned literature doing so? If not, what’s the gain of this design and what’s the learnable-parameter vs. accuracy trade-off? 2) Is there any changes in the prompt group compared with the literature? Authors claim to identify outliers with K-means. However, prompts like `a toy {c}` and `the plastic {c}` cannot describe most of the images and introduce outliers and it’s easy to simply just remove these manually designed prompts.\n\n-- Also, in section 2.1, the authors mostly introduce the datasets instead of introducing and discussing the few-shot learning literature. Looks like it can be moved to the experiment section instead of related work. \n\n-- In the proposed framework, the authors conduct Isolation forest as well as KMeans to eliminate outlier features, I wonder whether it slows down the training and inference.\n\n-- Minor: the scores for Zero-shot CLIP in tables 1 and 2 are not aligned well. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Lack of clear visualizations to show how causal information is disentangled from spurious information. Better class centroids do not indicate the effect of disentanglement. Similarity maps on both learned causal and spurious representations are encouraged to be shown.\n\nA1: Figure 1 of the paper visualizes the features of the same category under different domains. In order to better demonstrate the effectiveness of our motivation and the effect of disentanglement, we visualize the features of different categories under different domains. As shown in **Figure 2 in the attached PDF of \"global\" response**. We employ t-SNE to visualize distinct categories of image features within the contexts of both cartoon and sketch representations. This enables us to estimate the distribution of each category, delineated by an elliptic region. CLIP exhibits the capability to discern nuanced variations across diverse domains and categories. Subsequently, we fine-tuned the model on miniImageNet's training set. On sketch images with large differences in domain from miniImageNet, **the distinction of the feature extracted by fine-tunes model between the two categories deteriorates**. And the **distinction between domains becomes less obvious**. While our method can guarantee the differentiation between categories with larger differences and between domains.\n\nQ2: What if employing a small off-the-shell saliency model to first mask the spurious information?\n\nA2: Thanks for the question. Although the concept of utilizing a saliency model for masking spurious information seems align with image-related tasks, its applicability largely depends on the specific context. **Since the spurious information may be different across tasks, directly using such masking approach may get worse performance**. For instance, image background is usually considered as spurious information and should be masked, while it could also be useful information in some tasks. Paper [1] reports a worse out-of-distribution (OOD) performance when it employs a similar masking approach. In contrast, our method can be applied beyond \"object\" recognition. In particular, **employing a saliency model on datasets link eurosat is challenging due to the intricate task of satellite image \"terrain\" classification**. By utilizing non-contextual information from the text and constraining the model's ability to identify spurious information during fine-tuning, we ensure the preservation of model robustness, preventing the degradation of performance.\n\nQ3: Lack of meaningful discussions on related works.\n\nA3: Thanks for the suggestion. Due to space constraints, we have updated the content in global response.\n\nQ4: Is it a common practice to jointly incorporate a frozen and a learnable visual encoder in few-shot learning with CLIP? Is any of the mentioned literature doing so? If not, what’s the gain of this design and what’s the learnable-parameter vs. accuracy trade-off? \n\nA4: In scenarios with ample training samples, a larger number of parameters tends to correlate positively with heightened accuracy. Conversely, in few-shot learning, it's easy to overfit due to too many parameters and not enough data. Consequently, a prevailing trend among existing methodologies is to mitigate the number of learnable parameters. Within the few-shot learning paradigm, enhancing performance through an augmented parameters in the network while concurrently preventing overfitting proves to be challenging. It is worth noting that while existing methods have demonstrated better performance, few have attempted to exploit the potential for fine-tuning the backbone. Accordingly, this paper addresses this challenge by fine-tuning the backbone in few-shot learning  to make it less fitted and improve performance. The experimental results in the following table also demonstrate that applying our fine-tuned backbone directly to existing methods can effectively improve the performance.\n\n|METHOD|1shot|2shot|4shot|8shot|16shot|\n|-|-|-|-|-|-|\n|Tip-Adapter[3]|64.11|64.36|64.63|65.17|65.49|\n|Tip-Adapter+**FD-Align**|64.51|65.33|65.76|66.79|67.28|\n|Tip-Adapter-F[3]|64.64|65.18|65.78|67.21|68.43|\n|Tip-Adapter-F+**FD-Align**|64.86|65.61|66.11|67.58|68.70|\n|APE[4]|65.36|65.69|66.00|66.55|66.55|\n|APE+**FD-Align**|66.71|67.29|67.40|67.76|67.69|\n|APE-T[4]|65.89|66.18|66.82|67.99|68.74|\n|APE-T+**FD-Align**|66.84|67.37|67.81|68.73|69.15|\n\nQ5: Is there any changes in the prompt group compared with the literature? Authors claim to identify outliers with K-means. Why not delete prompts like a toy {c} manually\n\nA5: In order to ensure a fair comparison, we employ the identical prompts as presented in the original CLIP papar. \nIt should be corrected that we are using Isolation Forest, not k-means to remove outlies. K-means is employed to mitigate an overabundance of similar prompts, thereby preventing an undue emphasis on a particular context. Since we can't tell the validity of each prompt, there may be cases of mistaken or missed deletions. Consequently, we opt to employ Isolation Forest for the purpose of excluding irrational prompts, as opposed to manual deletion. As an illustration, in Tip-Adapter[3], the prompts after manual filtering by the author contain \"itap of a {c}.\" However, this prompt is not reasonable for our task and leads to performance degradation. \n\nQ6: Related work in Section2.1 and scores in tables 1and 2.\n\nA6: Thanks for your suggestion, we will move it to experiment in the revised version.\n\nQ7: Do Isolation Forest Kmeans slow down training?\n\nA7: No, Isolation Forest and k-means are only run during training initialization. So it doesn't affect training and inference speed.\n\n[1] Masked images are counterfactual samples for robust fine-tuning.\n[2] Conditional Prompt Learning for Vision-Language Models\n[3] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling\n[4] Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement\n[5] Visual Prompt Tuning"}, {"review_id": "aD4tYHgO49", "reviewer": "Reviewer_ePYg", "summary": "This paper studies the problem of fine-tuning a pre-trained CLIP to downstream classification tasks with few-shot samples. The authors propose to fine-tune the category-dependent feature while retain the category-independent feature in order to improve the robustness of the fine-tuned model. The proposed method is compared with several baselines on several benchmarks. ", "questions": "See Weaknesses.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The motivation is clear. The paper is clearly written.", "weaknesses": "- I'm not fully convinced by several key statements/arguments in the paper:\n    - L114-118, why retaining the spurious features can help model robustness? I think the less spurious feature it learns, the better it transfers to different domain? I also do not understand \"This allows the model to combine different spurious information based on the category information, thereby maintaining generalization when the object appears in a new context, style, or domain.\"\n    - L131-132, \"the object represents the causal information and 'a photo of' represents the spurious information\". I do not understand why \"a photo of\" can contain spurious information? As the authors explain, the spurious information includes domain, style, or background informaiton. However, \"a photo of\" contains neither of them. It does not discriminate between different domains or backgrounds or styles. Then the spurious feature extractor is not actually extracting any spurious informaiton. \n\n- Only fully fine-tuning and wise-ft are compared. What about other parameter-efficient fine-tuning methods such as LoRA [1] and VPT [2]? Also the gap between fully fine-tuning and FD-Align seems not really significant. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Why retaining spurious features can help model robustness?\n\nA1: Sorry for the confusion. In fact, our objective is not to preserve spurious features of an image, but to **preserve the ability of CLIP to distinguish spurious features**. Illustrated in Figure 1a, CLIP demonstrates the capability to differentiate task-irrelevant information (deemed as spurious information within the classification task). Regrettably, direct fine-tuning destroys this capability, resulting in a model's inability to differentiate images affected by distributional bias.\n\nIn few-shot tasks, during the fine-tuning process, the classifier encounters a combination of causal and spurious information within the features. Due to insufficient data, the fine-tuning procedure can readily lead to overfitting, causing the classifier to incorporate spurious features into its decision-making framework. This phenomenon leads to a decrease in the model's ability to recognize spurious information and thus errors in the testing phase.However, we impose constraints on the training process to preserve the model's ability to distinguish spurious features. At this point, the model learns mainly task-relevant information during the fine-tuning process. It refrains from overfitting spurious features onto the prevailing category attributes. Consequently, this safeguard guarantees that the model avoids overfitting to spurious information.\n\nQ2: How to understand \"This allows the model to combine different spurious information based on the category information, thereby maintaining generalization when the object appears in a new context, style, or domain.\"?\n\nA2: To facilitate comprehension, we employ the object recognition task as an illustrative example. Within the context of the target recognition task, the object represents our task-relevant information, while the scene embodies our task-irrelevant information, **coexisting simultaneously**. Owing to the limited volume of training samples inherent in few-shot learning. Moreover, direct fine-tuning often leads to pronounced overfitting within the scene information abstraction, thereby compromising the model's capacity to differentiate between spurious and causal information. Consequently, during instances where the current category emerges within an unseen scene, the model's ability to discern objects within said scene may falter, consequently yielding classification errors. Notably, our proposed methodology ensures the retention of the model's discriminative acumen between causal and spurious information. In scenarios where objects manifest in previously unseen scenes, our model adeptly discriminates between object-specific and scene-specific information, consequently enhancing its proficiency in recognizing scenes absent from the training dataset. This, in turn, augments the overall generalization.\n\n\nQ3: Why does \"a photo of\" contain spurious information?\n\nA3: \"a photo of\" can be regared as a **style information**. Compared to sketch and painting, photo has a distinctly different style. Therefore, it is also a kind of spurious information.\n\nQ4: Why not compared with parameter-efficient fine-tuning methods? \n\nA4: Our goal is to fine-tune a better performing, more generalizable backbone for fine-tuning, so that it can replace the existing method's backbone and improve performance. In the following table, we provide comparative performance results on imagenet for replacing our backbone with the existing method.\n\n|METHOD|1shot|2shot|4shot|8shot|16shot|\n|-|-|-|-|-|-|\n|Tip-Adapter[3]|64.11|64.36|64.63|65.17|65.49|\n|Tip-Adapter+**FD-Align**|64.51|65.33|65.76|66.79|67.28|\n|Tip-Adapter-F[3]|64.64|65.18|65.78|67.21|68.43|\n|Tip-Adapter-F+**FD-Align**|64.86|65.61|66.11|67.58|68.70|\n|APE[4]|65.36|65.69|66.00|66.55|66.55|\n|APE+**FD-Align**|66.71|67.29|67.40|67.76|67.69|\n|APE-T[4]|65.89|66.18|66.82|67.99|68.74|\n|APE-T+**FD-Align**|66.84|67.37|67.81|68.73|69.15|\n\nQ5: The gap between fully fine-tuning and FD-Align seems not really significant.\n\nA5: Thanks for bringing this to our attention. Upon careful examination, we determined that the minor gap of method compared to fully fine-tuning was due to inadequate hyperparameter tuning. In fact, after re-tuning the hyperparameters of our method, a significant enhancement in performance can be observed. The updated results are shown in the table below.\n\n|      METHOD     |   1 shot  |   2 shot  |   4 shot  |   8 shot  |  16 shot  |\n|:---------------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|   CLIP(60.33)   |           |           |           |           |           |\n|     LP-CLIP     |   22.17   |   31.90   |   41.20   |   49.52   |   56.13   |\n|     WiSE-FT     |   63.36   |   64.27   |   66.16   |   68.13   |   69.55   |\n|   Tip-Adapter   |   65.52   |   67.40   |   69.56   |   71.58   |   72.72   |\n|        FT       |   65.24   |   67.42   |   69.85   |   71.73   |   73.00   |\n| **FT+FD-Align** | **65.65** | **68.26** | **71.25** | **73.63** | **75.59** |"}, {"review_id": "k7cNzTqZBq", "reviewer": "Reviewer_vxTm", "summary": "This paper introduces a novel approach to tackle few-shot learning utilizing the powerful pretrained CLIP model. The primary objective is to construct multiple CLIP text prompts with different context and construct a term to regularize the learning process. This helps prevent overfitting to irrelevant correlations. Specifically, the method involves creating specific prompts for each class, combining category-independent factors such as \"toy,\" \"drawing,\" and \"plastic,\" with the object category name. A loss term is designed to ensure that the extraction of spurious information remains consistent both before and after fine-tuning the CLIP model. ", "questions": "1. What is Fine-tuning CLIP baseline in Table 2? Is it using the first loss term only?\n", "limitations": "Yes, the limitation has been properly addressed. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The notion of using the capabilities of the CLIP model to enumerate potential spurious factors, and subsequently constructing a regularization mechanism for few-shot learning, is a intriguing concept.\n\n2. Some experimental results presented in this paper, particularly the findings depicted in Figure 4 and Figure 6, seems to be promising and somehow validates the proposed method. ", "weaknesses": "1. The basic assumption of this paper, that if we fine-tuning CLIP model, the model tends to over-fit to the spurious information, needs a further validation. For example, it might be possible to use GradCAM or other visualization method to showcase such a phenomenon. \n2. The notation is somewhat confusing and a careful proof-reading is needed as there are many typos in this paper, for example:\n    (1) in line 134, The definition of C should be mentioned.\n    (2) the index i is used twice in the equation after line 142, this causes confusion\n    (3) W_spu was mistakenly written as W_sup in the same equation\n    (4) The prompts in Figure 3 give an good example of category-independent information mentioned in line 142. It is better to refer to Figure3 in line 142 as a concrete example.\n    (5) It is better to choose a different notation to denote object class, and context class, e.g., toy, plastic, ... in the equations \n3. What is Fine-tuning CLIP baseline in Table 2? Is it using the first loss term only? If yes, the proposed method only leads to marginal improvement over this simple baseline. If that is the case, then the benefit of the proposed strategy might not be significant.\n4. Also, the improvement by using IF and K-means is also marginal. In most cases, the improvement is less than 0.5%", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The validation of the basic assumption. \n\nA1: It is important to point out that our basic assumption is that **in few-shot learning, due to insufficient sample size to fine-tune the model, the model is prone to overfitting into the causal and spurious information it currently sees**. That is, the model may **incorrectly treat spurious information in the training sample as causal information of the current category**, thus losing the ability to distinguish between different spurious information. Our hypothesis can be demonstrated quite intuitively in Figure 1 in the paper. Figure 1 shows the t-SNE dimensionality reduction visualization of features extracted by different models for the same category over different domains. Compared to CLIP, the fine-tuned model is less capable of recognizing variance on data from different domains.\n\nQ2: The notation is somewhat confusing and careful proof-reading is needed as there are many typos in this paper.\n\nA2: Thanks for your suggestions, we will update them in later version.\n\nQ3: What is Fine-tuning CLIP baseline in Table 2? Is it using the first loss term only?\n\nA3: Yes, it is using the first term only. Previous submissions suffered from inadequate hyperparameter tuning. In response, we meticulously re-tuned our hyperparameters to bolster performance, consequently yielding superior results. To provide a comprehensive perspective, we present the latest results in the subsequent table. \n\n|      METHOD     |   1 shot  |   2 shot  |   4 shot  |   8 shot  |  16 shot  |\n|:---------------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|   CLIP(60.33)   |           |           |           |           |           |\n|     LP-CLIP     |   22.17   |   31.90   |   41.20   |   49.52   |   56.13   |\n|     WiSE-FT     |   63.36   |   64.27   |   66.16   |   68.13   |   69.55   |\n|   Tip-Adapter   |   65.52   |   67.40   |   69.56   |   71.58   |   72.72   |\n|        FT       |   65.24   |   67.42   |   69.85   |   71.73   |   73.00   |\n| **FT+FD-Align** | **65.65** | **68.26** | **71.25** | **73.63** | **75.59** |\n\n\nQ4: The improvement by using IF and K-means is marginal.\n\nA4: The utilization of IF aims to eliminate inappropriate prompts from the prompt group and mitigate the influence of redundant prompts. This strategy proves especially beneficial when the prompt group's quality is suboptimal. Given that we employ the prompts groups of CLIP for ImageNet, characterized by a paucity of inappropriate prompts, the performance enhancement is comparatively modest. Nevertheless, a performance gain is still attained. Furthermore, it is noteworthy that this procedure takes place during the training initialization phase, thereby exerting no impact on training velocity. Conversely, a reduction in the quantity of spurious factors contributes slightly to the acceleration of the training process."}, {"review_id": "DSuR3IQmOv", "reviewer": "Reviewer_vJi7", "summary": "This paper aims to enhance the performance of pre-trained CLIP for few-shot learning tasks while maintaining their generalizability and mitigating the risk of overfitting. To achieve this goal, the key contribution of this paper is the spurious information extractor that captures category-independent information from image features. By aligning the probability distributions of image features under the spurious weights, the fine-tuned model's ability to extract spurious features is preserved. Overall, this paper offers a robust CLIP fine-tuning method for improving the performance and generalization capabilities of pre-trained models in few-shot learning tasks.", "questions": "1) As mentioned in Weakness 1, the autors should give clearer explanations, or analyze the similarity between the pretraining dataset and the finetuning dataset.\n2) For weakness 2, it is suggested to provide some cross-domain comparisons on the same class, several examples on the classification confidence are also helpful.\n3) Typo errors such as \"\"wheile\" -> \"while\" in line 260.", "limitations": "potential limitations have been discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Cross-domain and OOD few-shot learning are important research topics in this field. In this respect, the goal of the proposed method is backed by strong motivations.\n2) Good performance. Fairly extensive experiments demonstrate that CLIP pretraining is promising for the few-shot and OOD few-shot learning tasks. Ablation studies indicate the innovations are helpful.", "weaknesses": "1) In my opinion, utilizing features from CLIP is knid of conflicting to few-shot settings. Sine CLIP is pretrained on multi-class large datasets, there could be some information leakage on similar classes, can it still be considered a few-shot learning task in this situation? As can be seen from Table 3, performance on some datasets that are unfamilar with pretrained CLIP (e.g., ChestX) are relatively poor.\n2) More discussions/results regarding the cross-domain and dataset settings should be added. For instance, how about the performance on the same class under different domains (e.g., painted, sketch and real dog)? What kinds of pre-training source images (domain or object) are more helpful?\n3) There is insufficient information about implementation details, such as the Isolation Forest and K-means algorithms, which makes it difficult to reproduce the work.\n4) Apart from the training stability, it is not specified how long the proposed method takes for the entire training (i.e., pretraining and fine-tuning) and how efficient it is compared to existing studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Is using features from CLIP kind of conflicting to few-shot settings?\n\nA1: The consideration of CLIP features within the setting of few-shot learning remains a valid approach. Despite CLIP's extensive pretraining on vast datasets, it is important to acknowledge the potential for significant **deviations between its acquired knowledge and the specific requirements of downstream tasks**. Notably, recent advancements have witnessed the emergence of numerous CLIP-based few-shot learning paradigms, exemplified by methodologies such as Tip-adapter[1], APE[2], VPT[3], and others. These endeavors underscore the ongoing relevance and applicability of CLIP-based techniques in the context of few-shot learning scenarios.\n\nQ2: Why is performance on ChestX so poor?\n\nA2: The setting in Table 3 involves fine-tuning CLIP on miniImageNet's training set and subsequently evaluating its performance on the ChestX dataset, which predominantly consists of medical images. The distinct dissimilarity in overall data distribution and categories between ChestX and miniImageNet contributes significantly to the observed disparity in performance. Given CLIP's classification performance is still low, the discernible performance difference is attributable to the lack of pertinent information within CLIP's training corpus. Notably, medical images are **difficult to classify without a specialized background** due to the domain-specific challenges they pose. This observed performance discrepancy aligns with analogous findings in related research[4].\n\nQ3: Some cross domain comparisions on the same class.\n\nA3: In response to your inquiry, we performed a comprehensive evaluation regarding cross-domain and dataset settings. Specifically, we divided the few shot training set on imagenet and fine-tuned the model. We then performed performance tests on ImageNet and its variants datasets. The results of our study on 16 shots are shown in the table below, and our approach significantly improves cross-domain performance compared to the direct fine-tuning alternative. And the direct application of our fine-tuned backbone to existing methods can significantly improve performance.\n\n|           Method           | imageNet | imageNetA | imageNetR | imageNetS | imageNetV2 |\n|:--------------------------:|:--------:|:---------:|:---------:|:---------:|:----------:|\n|            CLIP            |   63.34  |   31.57   |   68.45   |   42.31   |    55.92   |\n|             FT             |   64.91  |   30.05   |    68.7   |   42.24   |    57.63   |\n|       FT+**FD-Align**      |   66.39  |    31.8   |    69.7   |    43.5   |    57.73   |\n|         Tip-adapter        |   65.49  |     -     |     -     |   42.48   |    57.58   |\n|  Tip-adapter+**FD-Align**  |   65.49  |     -     |     -     |   43.84   |    59.10   |\n|        Tip-adapter-F       |   68.43  |     -     |     -     |   42.54   |    59.58   |\n| Tip-adapter-F+**FD-Align** |   68.70  |     -     |     -     |   43.67   |    60.17   |\n|             APE            |   66.55  |     -     |     -     |   43.28   |    58.31   |\n|      APE+**FD-Align**      |   67.69  |     -     |     -     |   44.23   |    59.36   |\n|            APE-T           |   68.74  |     -     |     -     |   43.23   |    59.58   |\n|     APE-T+**FD-Align**     |   69.15  |     -     |     -     |   44.04   |    60.83   |\n\nQ4: The implementation details about Isolation Forest and K-means algorithms.\n\nA4: We have explained the details of our experiment in section 4.1. Specifically, 60 data points were retained in Isolation Forest and subsequently clustered into 20 classes using K-means. In addition, we have provided the **code in the supplement**. Again, we will subsequently **open source the code** to ensure the reproducibility of the method.\n\nQ5: The time cost of our method\n\nA5: In order to address this issue, we have compared the time required by our method compared to the direct fine-tuning method. The following table shows the results of 1 shot fine-tuning on imageNet. The results show that our method does not introduce much extra time consumption. Our method can be trained once and then applied directly to existing methods.  Moreover, our approach does not introduce any additional time overhead compared to CLIP during the inference phase.\n\n| Method       | Time       |\n|--------------|------------|\n| Fine-tuneing | 7min 9s  |\n| Ours         | 8min 53s |\n\nQ6: Some typos of this paper.\n\nA6: Thanks for pointing out the problem, we will fix it in our latest paper.\n\n[1] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling\n\n[2] Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement\n\n[3] Visual Prompt Tuning\n\n[4] Channel Importance Matters in Few-Shot Image Classification"}, {"review_id": "0Q7G8luqKa", "reviewer": "Reviewer_TEuf", "summary": "This work aims to adapt large pre-trained vision-language model to few-shot tasks. To this end, the authors propose to decouple the category-related and category-independent information to alleviate overfitting when adapting large model to few samples. They claim to maintain the visual encoder's ability to extract category-independent information during fine-tuning. They conduct experiments on various datasets, including CoOp-datasets and OOD datasets to verify the effectiveness of proposed methods.", "questions": "Please answer the questions in weakness.", "limitations": "No.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "1.The paper is organized and written well and the paper looks well polished. The overall story is clear to me.\n\n2.The authors conduct experiments on extensive datasets for comprehensive evaluation.", "weaknesses": "1.The motivation is not very convincing. First, as shown in Figure 1 and line 36-42, I do not think the separability among domains could significantly affect the performance of downstream tasks, since we conduct classification within single domain. Second, as discussed in line 110-118, why we should contain the ability of extracting category-independent information. In fact, if the model can ignore the category-independent information and only extract the category-related information, it will obtain better performance and be very robust, even on different domain.\n\n2.The proposed method is kind of naive and can not be considered as a systematic approach.\n\n3.The experments are not very convicing, although they are conducted on various datasets. As shown in Table 1, some existing works are not compared, like CoOp, PLOT. I do not understand why this work use ViT-B as backbone, since many baselines use ResNet50. This work should conduct experiments with ResNet50 and compared with the reported results of baselines, instead of using a new backbone and reproduce the results baselines. Even with ViT-B, PLOT++ has achieve 70.6% average accuracy on 1-shot tasks, but this work only achieve 63.92%.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1 : Motivation is not very convincing.\n\nA1: Sorry for the confusion. Our primary objective focuses on **leveraging the pre-trained CLIP model for downstream few-shot tasks**. Specifically, when we have a dataset with limited samples, our aim is to enable CLIP to quickly adapt to it, and we also hope that the updated model can **generalize to other similar datasets**.\n\nThe most straightforward approach is to fine-tune CLIP using downstream data. However, there exist two issues. First, since the downstream data is limited in the few-shot setting, directly fine-tuning CLIP with **insufficient data may lead to overfitting**, thereby diminishing the model's performance. Second, this direct fine-tuning might cause the** loss of the ability to distinguish between causal information and spurious information** in the original CLIP, which can negatively impact the updated model's out-of-domain (OOD) generalization.\n\nTo address these two issues, we propose a controlled fine-tuning procedure for CLIP. Specifically, by imposing constraints that sustain its capability to differentiate between causal and spurious features, the updated model is steered away from both overfitting the few-shot data and fitting the spurious features. \n\nQ2: I do not think the separability among domains could significantly affect the performance of downstream tasks (Figure 1 and line 36-42).\n\nA2: Figure 1 of the paper visualizes the features of the same category under different domains. In order to better demonstrate the effectiveness of our motivation and the effect of disentanglement, we visualize the features of different categories under different domains. As shown in Figure 2 in the attached pdf of global response. We employ t-SNE dimensionality reduction to visualize distinct categories of image features within the contexts of both cartoon and sketch representations. This enables us to estimate the distribution of each category, delineated by an elliptic region. CLIP exhibits the capability to discern nuanced variations across diverse domains and categories. Subsequently, we fine-tuned the model on miniImageNet's training set. **On sketch images with large differences in domain from miniImageNet, the distinction of the feature extracted by fine-tunes model between the two categories deteriorates. And the distinction between domains becomes less obvious**. While our method can guarantee the differentiation between categories with larger differences and between domains.\n\nQ3: The proposed method is kind of naive and can not be considered as a systematic approach.\n\nA3: Thanks for the comments. While our approach may indeed seem simple on the surface, we view this **simplicity as one of its core strengths, rather than a limitation**. The ease of implementation and broad applicability that arise from this simplicity are valuable attributes in our opinion. We want to emphasize that **simplicity does not necessarily compromise innovation or the efficacy of a method**. In our particular case, this approach has demonstrated robust results, as substantiated by the experiments detailed in our submission.\n\nFor additional context, we can draw a comparison to one of our key baselines, WiSE-FT. Despite employing what might be considered a straightforward weighted fusion strategy for subsequent classification, WiSE-FT has achieved remarkable success, even being recognized as a finalist for the prestigious CVPR2022 Best Paper award.\n\nWe appreciate your insight and are open to further discussion if you have any specific suggestions or concerns that could enhance our work.\n\nQ4: Why not compared with some existing works.\n\nA4: Existing approaches mainly divide into two categories, one is to use additional adapters to adapt to the downstream task. The other is the learning of the prompt to improve the model performance. In contrast, our approach focuses on **improving the fine-tuning process of the CLIP backbone**. The fine-tuned backbone can be directly applied to previous methods to improve the performance of ID and OOD.\n\nWe acknowledge the omission of pertinent comparisons with CoOp, Plot++, and other pertinent works. Your feedback is invaluable, and we are committed to rectifying this oversight in the revised rendition of our related work section.\n\nFurthermore, the results in **Table 2  and Table 3 in global response** prove the efficacy of our fine-tuned backbone. Experimental results demonstrate that our improved backbone can be seamlessly integrated into existing methods with noticeable performance gains.\n\nQ5: Why use ViT-B/32 as backbone rather than ResNet.\nA5: We need to fine-tune the backbone relative to the existing methods, so we choose ViT-B/32 as the backbone with the smallest memeroy in the CLIP open source backbone. Nevertheless, our approach is applicable under different backbone. We will add the performance of our method on ResNet50 in the revised version. \n\nPrevious submissions suffered from inadequate hyperparameter tuning. In response, we meticulously re-tuned our hyperparameters to bolster performance, consequently yielding superior results. To provide a comprehensive perspective, we present the latest outcomes in the **Table 1 in global response**. Also, PLOT++ uses a stronger backbone ViT-B/16 relative to us.\n\n[1]  Robust fine-tuning of zero-shot models\n\n[2] Conditional Prompt Learning for Vision-Language Models\n\n[3] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models"}]

Author rebuttal:
## Additional discussion in related work.

In the context of few-shot learning, full fine-tuning of the pre-trained model often results in overfitting due to the limited sample size, which consequently diminish the model's generalization ability. Therefore, it is common practice in few-shot learning scenarios to fix the feature extractor while fine-tuning the classification head or training additional structures. For instance, CoOp [1] models prompt’s context words with learnable vectors, keeping the entire pre-trained parameters fixed. Tip-Adapter[2] and APE[3] do not require any back propagation for training the adapter but create the weights through a key-value cache model constructed from the few-shot training set. VPT[4] introduces extra learnable parameters into the input space. However, all these methods are processed with the backbone frozen, while our paper aims to further explore the possibility of fine-tuning the backbone itself.

## Figure
We have updated the method figure and the visualization of the features extracted by the different models in the pdf.

## Table 1

Previous submissions suffered from inadequate hyperparameter tuning. In response, we meticulously re-tuned our hyperparameters to bolster performance, consequently yielding superior results. To provide a comprehensive perspective, we present the latest outcomes in the subsequent table. 

|      METHOD     |   1 shot  |   2 shot  |   4 shot  |   8 shot  |  16 shot  |
|:---------------:|:---------:|:---------:|:---------:|:---------:|:---------:|
|   CLIP(60.33)   |           |           |           |           |           |
|     LP-CLIP     |   22.17   |   31.90   |   41.20   |   49.52   |   56.13   |
|     WiSE-FT     |   63.36   |   64.27   |   66.16   |   68.13   |   69.55   |
|   Tip-Adapter   |   65.52   |   67.40   |   69.56   |   71.58   |   72.72   |
|        FT       |   65.24   |   67.42   |   69.85   |   71.73   |   73.00   |
| **FT+FD-Align** | **65.65** | **68.26** | **71.25** | **73.63** | **75.59** |

## Table 2
Performance comparison of replacing our fine-tuned backbone onto an existing method on ImageNet.

|METHOD|1shot|2shot|4shot|8shot|16shot|
|-|-|-|-|-|-|
|Tip-Adapter[2]|64.11|64.36|64.63|65.17|65.49|
|Tip-Adapter+**FD-Align**|64.51|65.33|65.76|66.79|67.28|
|Tip-Adapter-F[2]|64.64|65.18|65.78|67.21|68.43|
|Tip-Adapter-F+**FD-Align**|64.86|65.61|66.11|67.58|68.70|
|APE[3]|65.36|65.69|66.00|66.55|66.55|
|APE+**FD-Align**|66.71|67.29|67.40|67.76|67.69|
|APE-T[3]|65.89|66.18|66.82|67.99|68.74|
|APE-T+**FD-Align**|66.84|67.37|67.81|68.73|69.15|

## Table 3
A comparison of the performance of replacing our fine-tuned backbone on the existing method in the OOD task. The backone is fine-tuned on 16 shots imageNet.

|           Method           | imageNet | imageNetA | imageNetR | imageNetS | imageNetV2 |
|:--------------------------:|:--------:|:---------:|:---------:|:---------:|:----------:|
|            CLIP            |   63.34  |   31.57   |   68.45   |   42.31   |    55.92   |
|             FT             |   64.91  |   30.05   |    68.7   |   42.24   |    57.63   |
|       FT+**FD-Align**      |   66.39  |    31.8   |    69.7   |    43.5   |    57.73   |
|         Tip-adapter        |   65.49  |     -     |     -     |   42.48   |    57.58   |
|  Tip-adapter+**FD-Align**  |   65.49  |     -     |     -     |   43.84   |    59.10   |
|        Tip-adapter-F       |   68.43  |     -     |     -     |   42.54   |    59.58   |
| Tip-adapter-F+**FD-Align** |   68.70  |     -     |     -     |   43.67   |    60.17   |
|             APE            |   66.55  |     -     |     -     |   43.28   |    58.31   |
|      APE+**FD-Align**      |   67.69  |     -     |     -     |   44.23   |    59.36   |
|            APE-T           |   68.74  |     -     |     -     |   43.23   |    59.58   |
|     APE-T+**FD-Align**     |   69.15  |     -     |     -     |   44.04   |    60.83   |

[1] Conditional Prompt Learning for Vision-Language Models

[2] Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

[3] Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement

[4] Visual Prompt Tuning

Meta-review: The paper received quite diverse ratings. After a thorough examination of the paper, reviews, authors' rebuttals, and the discussion, the AC has identified the following key points of contention:

Motivation and Its Adequacy: Reviewers EKx5, ePYg, and Teuf initially raised concerns about the clarity and sufficiency of the paper's motivation. However, the AC concur with the authors that the idea of "distinguishing between causal information and spurious information" is well-motivated and substantiated by Figures 1 and 2. Reviewers EKx5 and ePYg also acknowledge the motivation as clear and interesting. The authors' final response to Reviewer Teuf appears to have addressed the concerns adequately.

Novelty of the Approach: Reviewer vxTm questioned the novelty of the proposed method. The AC agrees with the authors that the concept of "utilizing text to distinguish between causal and spurious feature extraction for enhancing the generalizability of few-shot model training is innovative." However, it is essential for the authors to emphasize this concept in the Introduction to better convey the innovation to other researchers.

Insufficient Experimentation: The issue of insufficient experiments has been raised on several fronts:
a) Reviewer EKx5 inquired about the efficiency of the proposed method and its impact on inference speed. It appears that the authors' final response adequately addressed this concern.
b) Reviewer vxTm expressed concerns about the limited performance improvements demonstrated in the experiments. The authors' response seems to provide a solution to this issue. The AC strongly recommends that the authors revise the paper by including the updated results and offering a detailed explanation of how these results were achieved.

c) Reviewer Teuf suggested the inclusion of experiments with different backbones, such as ResNet50 and ViT-B/16, to provide a more comprehensive performance comparison. While this is a valuable suggestion to enhance the paper, it does not constitute a reason for rejection.

Overall, considering the paper's contributions in terms of novelty, methodology, and experiments, the AC believes it meets the criteria for acceptance at NeurIPS. However, the AC recommends that the authors make the suggested revisions to further strengthen the paper and address the concerns raised by the reviewers.
Decision: Accept (poster)