Title: FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: few-shot learning

CLIP

fine-tuning
Abstract: Due to the limited availability of data, existing few-shot learning methods trained from scratch fail to achieve satisfactory performance. In contrast, large-scale pre-trained models such as CLIP demonstrate remarkable few-shot and zero-shot capabilities. To enhance the performance of pre-trained models for downstream tasks, fine-tuning the model on downstream data is frequently necessary. However, fine-tuning the pre-trained model leads to a decrease in its generalizability in the presence of distribution shift, while the limited number of samples in few-shot learning makes the model highly susceptible to overfitting. Consequently, existing methods for fine-tuning few-shot learning primarily focus on fine-tuning the model's classification head or introducing additional structure. In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align). Our method aims to bolster the model's generalizability by preserving the consistency of spurious features across the fine-tuning process. Extensive experimental results validate the efficacy of our approach for both ID and OOD tasks. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements. Our code can be found in https://github.com/skingorz/FD-Align.

FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning

Abstract

Due to the limited availability of data, existing few-shot learning methods trained from scratch fail to achieve satisfactory performance. In contrast, large-scale pre-trained models such as CLIP demonstrate remarkable few-shot and zero-shot capabilities. To enhance the performance of pre-trained models for downstream tasks, fine-tuning the model on downstream data is frequently necessary. However, fine-tuning the pre-trained model leads to a decrease in its generalizability in the presence of distribution shift, while the limited number of samples in fewshot learning makes the model highly susceptible to overfitting. Consequently, existing methods for fine-tuning few-shot learning primarily focus on fine-tuning the model's classification head or introducing additional structure. In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align). Our method aims to bolster the model's generalizability by preserving the consistency of spurious features across the fine-tuning process. Extensive experimental results validate the efficacy of our approach for both ID and OOD tasks. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements. Our code could be found in https:// github.com/skingorz/FD-Align .

1 Introduction

The Contrastive Language-Image Pre-training model (CLIP) [1] represents a groundbreaking development in multi-modal deep learning. By utilizing contrastive learning, CLIP aligns visual and textual representations within a unified embedding space, and exhibits superior performance in a variety of downstream tasks, including image classification [2, 3], detection [4], and segmentation [5, 6], which is typically done by fully fine-tuning CLIP using downstream data. However, in many real-world scenarios, there is often an insufficient amount of labeled data available. Thus, fully fine-tuning will lead to overfitting and significantly diminishing the model performance.

To mitigate this few-shot challenge, we consider using a proxy dataset correlated to the downstream target dataset for fine-tuning CLIP, aiming to obtain a model that can efficiently generalize to the few-shot target task. Directly fully fine-tuning CLIP on the proxy dataset is not feasible, as the finetuned model may overfit to the proxy data or have worse out-of-distribution (OOD) generalization [7], limiting its performance on the target task. As an example illustrated in Figure 1, the fully fine-tuned CLIP tends to focus more on local regions and less on the foreground compared to the original CLIP.

Such localized attention will weaken the model's robustness to spurious correlations [8], resulting in poor OOD generalization of the fully fine-tuned CLIP.

In this paper, our objective is to preserve the robustness of CLIP to spurious correlations during fine-tuning, i.e., its ability to distinguish between spurious and causal features. In particular, causal features represent the features related to the classes, while spurious features could be the features related to the context of the class. We want the fine-tuned CLIP can learn the causal features of the new classes while keeping the recognition ability for the spurious features. To achieve this, we propose a method of F eature D iscrimination Align ment (FD-Align). Specifically, we introduce a spurious feature classifier, ensuring that the classification probability distribution of spurious features remains consistent throughout the fine-tuning. Taking advantage of the potent alignment capabilities of CLIP's text and visual features, we utilize text features of category-agnostic descriptions (i.e., contexts) as spurious feature prototypes. The image features and spurious feature prototypes are subjected to similarity measurement to ascertain the current image's probability distribution over spurious features. By constraining the probability distribution of the image features extracted by the model before and after fine-tuning, we ensure consistency in the spurious features extracted by the model. Concurrently, while learning classification capabilities on the proxy dataset, the robustness of the model to spurious associations after fine-tuning is also ensured.

Our method maintains the model's robustness to spurious correlations while fine-tuning model on proxy dataset. As illustrated in Figure 1d, on the one hand, compared to CLIP, the fine-tuned model with FD-Align better focuses on the dog. On the other hand, compared to the localized attention of fully fine-tuning CLIP, FD-Align pays slight attention to some spurious information. This balance between attending to causal information and spurious information ensures the model's robustness to spurious correlations, thereby ensuring the model's OOD generalization. Extensive experiments validate the robust OOD performance of our approach, alongside the improvement of in-distribution (ID) performance. Furthermore, as shown in Figure 2, the model fine-tuned by FD-Align directly enhances the accuracy of existing methodologies without introducing additional inference cost.

Our paper makes the following contributions:

We propose to use textual features to obtain spurious features of images;

We propose a feature discrimination alignment finetuning architecture that ensures the OOD performance of the fine-tuned model by aligning spurious features extracted model before and after fine-tuning;

Sufficient experiments demonstrate that our approach can significantly improve the performance of ID and OOD

FT

Tip

Tip-F

APE APE-T

40

45

Baseline

FD-Align

(a) ImageNet-S

FT

Tip

Tip-F

APE APE-T

55

60

Baseline

FD-Align

for few-shot learning, and it can improve the performance of existing methods without introducing additional training and inference costs.

2 Related Work

Few-Shot Learning. The primary objective of few-shot learning is to train models of superior performance with a small number of samples. Prior methods primarily train models on base data and assess their performance on novel class data devoid of any shared categories. Approaches such as

MAML[9] adopt meta-learning to train a base learner on base data, followed by fine-tuning it on a limited amount of novel data to derive a model suited for novel data. ProtoNet [10] introduces the use of metric learning for training. Recently, a line of work start to introduce the modality of text into the few-shot image classification, such as AM3 [11], TRAML [12] and FILM [13]. All these models are trained from scratch. With the emergence of pre-trained bi-modal models such as CLIP, the acquisition of more precise image features is presently achievable by leveraging these pre-trained models. Consequently, the majority of current research focuses on how to use features extracted by CLIP to enhance the capability of few-shot learning. For instance, CoOp [14] and CoCoOp [15] model prompt's context words with learnable vectors, keeping all pre-trained parameters fixed. TipAdapter [3] and APE [16] do not require any backpropagation to train the adapter but create the weights through a key-value cache model constructed from the few-shot training set. VPT [17] introduces additional learnable parameters into the input space. However, all of these methods are processed with the backbone frozen, while our paper aims to further explore the possibility of fine-tuning the backbone itself. Although these methods show excellent performance, they do not exploit the potential of pre-trained models.

Fine-Tuning of Pre-trained Model. The most direct approach is fine-tuning the pre-trained model directly. However, fully fine-tuning decreases the OOD performance of the pre-trained model [7]. WiSE-FT [18] enhances performance by ensembling the weights of the zero-shot and fine-tuned models. Kumar et al. [7] first perform linear probing and then perform fully fine-tuning to ensure the OOD performance of the model. Xuhong et al. [19] introduce an additional regularizer to constrain the l 2 distance between zero-shot and fine-tuned CLIP. On the other hand, Mukhoti et al. [20] prevent the degradation of the foundational model's capacity by constraining image features extracted by zero-shot and fine-tuned CLIP in the feature space. Nevertheless, scant existing methods delve into the strategies for fine-tuning pre-trained models using limited data.

Spurious Correlations. Spurious correlations denote deceptive heuristics that hold for the majority of instances during training, but do not invariably apply [21]. For example, when the training instance involves a cow on grass, it is tempting to misconstrue the presence of grass as a causal determinant for categorizing cows. This misinterpretation, regarding grass as cows, embodies a form of spurious correlation. Eliminating spurious features generally leads to enhanced OOD performance; however, this can be accompanied by a decrease in ID performance [22]. Furthermore, even armed with a comprehensive understanding of the spurious feature, its extraction remains a non-trivial task [23]. The impact of spurious correlations persists within multimodal pre-trained models [24]. However, when model parameters and dataset sizes are large enough, Vision Transformer (ViT) exhibits increased robustness against spurious correlations [8]. Furthermore, the CLIP architecture enhances the robustness of the vision encoder to spurious correlations [25]. Consequently, improving robustness against spurious correlations can effectively maintain the OOD performance of the model.

3 The Proposed Method

3.1 Problem Definition

Suppose that we have a pre-trained CLIP [1] which contains a visual encoder f 0 and a text encoder g 0 . In addition, we have access to a few-shot proxy dataset D ⊂ X ×Y , where each class has very limited samples and each sample comprises an image x and its corresponding label y . The objective is to fine-tune the pre-trained CLIP using this proxy dataset, aiming to enhance its zero-shot performance for unseen target tasks related to the proxy dataset.

3.2 Fine-Tuning on Proxy Dataset

We freeze CLIP's text encoder g 0 during fine-tuning and make the visual encoder learnable. First, we initialize the visual encoder f t using the parameters of the visual encoder of pre-trained CLIP f 0 . The visual encoder f t is then used to extract the feature f t ( x ) of the image x . With the help of the remarkable text-vision alignment capability of CLIP, we use the textual features of the class names of each class as class prototypes. Following CLIP, for any class y , we combine M prompt templates ( P 1 , . . . , P M ) with the class name and obtain M prompts [ P 1 , y ] , . . . , [ P M , y ] . We then use the text encoder g 0 to extract the features of the above M prompts. Subsequently, we calculated the means of

Visual

Encoder

……

Text Embedding

Class Prototype

CLIP

Text

Encoder

Visual

Encoder

Prompt Group

a photo of {c}

a drawing of {c}

a toy {c}

the plastic {c}

……

Class

𝐿𝐿

𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐

𝐿𝐿

Prompts

Embedding

……

Spurious

Prototype

SPC

𝑁𝑁

𝑀𝑀

𝑀𝑀

×

𝑁𝑁

𝑀𝑀

𝑵𝑵

𝐾𝐾

bird

fish

dog

Spurious Feature Constraint

Fine Tune Model

Image Feature

Text Embedding

Class Prototype

Spurious Prototype

the M features to obtain the prototype of the corresponding class, namely, the prototype of class y is

We calculate the cosine similarity s ( · , · ) between the image feature and the class prototypes and produce a distribution over classes for the image. Finally, the class loss is calculated using crossentropy loss by

where Y is the label set.

3.3 Spurious Feature Constraint

Fully fine-tuning CLIP on the proxy data affects the robustness of the model to unseen data. To preserve the model's performance on out-of-distribution data during fine-tuning, we maintain the model's robustness to spurious correlations during fine-tuning. That is, keeping the spurious feature extracted by the model before and after fine-tuning unchanged. We first calculate the mean of the features of each prompt template P j over all classes as prototypes of the prompt template P j , that is,

We can calculate the similarity between the feature extracted by the fine-tuned model and the spurious prototypes and produce the distribution over spurious features as follows.

Similarly, we can use the pre-trained visual encoder f 0 to extract the feature and produce the distribution over spurious features as follows.

We can ensure that the spurious features of the models before and after fine-tuning remain consistent by keeping the probability distributions of the models over spurious features consistent before and after fine-tuning, that is

Finally, we optimize both (1) and (2) during fine-tuning to ensure classification ability and OOD robustness:

where we set α to 1 and β to 20 in this paper.

3.4 Spurious Prototype Correction

Prompt templates are typically designed manually or generated by large-language models such as GPT. These templates often include redundant or illogical prompts. Consequently, the prototypes of spurious features calculated by these imprecise and redundant prompt templates lack accuracy. Therefore, filtering and processing these prototypes of spurious features is necessary.

Certain prompt templates may exhibit a lack of practical significance or irrationality, making them unsuitable for incorporation as spurious features, as exemplified by "itap of {class}." This scenario can result in the inaccuracy of spurious prototypes. To address this problem, we employ the Isolation Forest algorithm [26] to eliminate meaningless prototypes associated with spurious features, i.e., µ spurious := ISOLATIONFOREST ( µ spurious , n ) . We will retain the n prototypes that exhibit the highest degree of rationality.

Moreover, there are cases where certain prompts exhibit excessive similarity. For example, prompts such as 'a photo of the {class}.', 'a photo of a {class}.', and 'a photo of my {class}.' show notable similarities. In such scenarios, a single piece of spurious information may correspond to multiple prompts. However, some spurious information aligns with only one prompt. Consequently, the relative weight of the probability of spurious information corresponding to a single prompt diminishes during the classification process. To address this issue, we employ the k -means algorithm to merge duplicate spurious features that arise from similar prompts, i.e., ˜ µ spurious := k -means ( µ spurious , k ) , where k is the number of cluster centers.

4 Experiments

In this section, we will validate the OOD robustness of our approach. Furthermore, our method demonstrates promising results in ID data.

4.1 Setup

Throughout this section, if not otherwise specified, we use open source ViT-B/32 as the backbone of the CLIP, and the prompts we use are OpenAI ImageNet prompt templates 2 . we set n to 60, k to 20 in the spurious prototype correction stage. We employ the Stochastic Gradient Descent (SGD) optimizer for model fine-tuning, conducting the process over 60 epochs. Since little prior work on CLIP has been done to fine-tune the backbone for few-shot learning, we mainly compare our method with the fully fine-tuning strategy and WiSE-FT [18].

Dataset

For the OOD setting, we evaluate our method on two different datasets. On the one hand, we train the model on ImageNet [27] following the few-shot task in CLIP and test the performance on two OOD variants of ImageNet [27]: ImageNetV2 [28] and ImageNet-Sketch [29] with the same 1000 classes, On the other hand, we follow the traditional few-shot learning strategy and fine-tune the model on the train split of miniImageNet and evaluate the model on Meta-dataset [30], BSCDFSL benchmark [31] and DomainNet [32], for a total of 19 datasets. These datasets cover a wide range of image styles, such as natural images, satellite images, medical images, and sketch images.

For ID setting, we use the 11 image recognition datasets following CoOp [14], which contain ImageNet [27], StanfordCars [33], UCF101 [34], Caltech101 [35], Flowers102 [36], SUN397 [37], DTD [38], EuroSAT [39], FGVCAircraft [40], OxfordPets [41] and Food101 [42]. These datasets cover a wide range of different visual recognition tasks such as generic object classification, finegrained classification, action, scene, texture, etc.

4.2 Results under OOD Setting

In order to evaluate the robustness of our method in OOD data, we fine-tune the model on ImageNet in 16-shot settings and test the performance on the variants of ImageNet. As shown in Table 1, in contrast to fully fine-tuning CLIP, our approach yields a performance boost of up to 1.75% on ImageNetA. Additionally, we directly integrate our fine-tuned backbone into Tip-adapter [3] and APE [16], leading to a substantial improvement in their generalization performance. The aforementioned findings demonstrate that FD-Align improves model performance while ensuring robust generalization. Furthermore, the fine-tuned model can seamlessly integrate into existing methods without requiring additional fine-tuning.

Signs

ChestX

ISIC

EuroSAT

Omniglot

QDraw

Flower

Aircraft

Disease

Textures

mini-test

MSCOCO

cub-test

Real

Infograph

Painting

Sketch

0

2

4

6

8

10

12

5way-1shot

Signs

ChestX

EuroSAT

ISIC

QDraw

Flower

Omniglot

Real

mini-test

Textures

Aircraft

Disease

cub-test

MSCOCO

Painting

Sketch

Infograph

3

2

1

0

1

2

3

4

5way-5shot

In addition, we also evaluate the robustness of our method in the N -way K -shot few-shot learning task. We fine-tune the model on the miniImageNet train split utilizing prototypical networks [10] and subsequently evaluate its performance on several different datasets. In our testing phase, we evaluate the precision of 2000 tasks for 5-way-1shot and 5-way-5shot scenarios in five distinct sets generated according to different random seeds. We then calculate the mean accuracy along with a confidence interval 95%. As illustrated in Table 2 and Figure 4, our approach yields notable performance enhancements across the majority of datasets. In particular, on Sketch dataset, our method achieves a performance improvement of 12.03% compared to CLIP. Nevertheless, on certain datasets, such as Traffic Sign, a marginal decrement in performance is observed with our approach. This may

be attributed to the dataset images primarily encompassing the object of identification, devoid of substantial contextual misinformation that might adversely impact direct fine-tuning. Moreover, our method consistently outperforms WiSE-FT [18] in most datasets. In the few instances where we do not surpass its performance, our results remain comparable.

0

1

2

4

8

16

Shot Number

60

62

64

66

68

70

72

74

76

Average accuracy on 11 different datasets

WiSE-FT

Tip-Adapter

Fully Fine-Tuning

FD-Align

CLIP

4.3 Results under ID Setting

In addition, we evaluate the performance of our method using ID data. Models were trained on various datasets using 1, 2, 4, 8, and 16 shots, respectively, and subsequently evaluate performance on the respective datasets. Figure 5 illustrates a comparative analysis of the average performance between our method and existing methods on 11 datasets. As depicted, our method significantly outperforms other strategies and yields increasingly substantial performance enhancements as the shot number increases. Previous research indicates that WiSE-FT effectively elevates the fine-tuning performance of models in zero-shot learning. However, in few-shot learning, relative to fully finetuning, employing WiSE-FT with a fusion parameter of 0.5 results in a performance notably inferior to comprehensive fine-tuning. Our analysis posits that, in few-shot learning, the paucity of fine-tuning samples results in minimal alterations in model fine-tuning. Consequently, following fusion, the model parameters more closely resemble those of CLIP, thereby influencing the model performance.

4.4 Discussion

4.4.1 The Generality of FD-Align

To validate the generalizability of models fine-tuned via FD-Align across various methods, we directly incorporated the FD-Align fine-tuned visual encoder into existing approaches and evaluate ID performance. As shown in Table 3, the visual encoder fine-tuned with FD-Align markedly improves the performance of existing methods, achieving a peak improvement of 1.79%. The visual encoder fine-tuned with FD-Align, can be seamlessly transitioned into existing methods, bolstering performance without incurring additional costs.

4.4.2 Why Remove Outliers and Cluster?

In this section, we analyze the importance of eliminating outliers from spurious features and the subsequent clustering of residual points via empirical analysis. As illustrated in Table 4, employing the Spurious Prototype Clustering (SPC) method yields enhanced performance for the spurious prototype relative to directly utilizing all 80 templates. This underscores the efficacy of removing outliers and redundant values from spurious features. Furthermore, we experimented with features derived from manually designed prompt templates as spurious prototypes. While Tip-adapter handpicked 7 templates, our findings indicate that leveraging these 7 templates directly as spurious prototypes precipitates a marked performance degradation. A closer examination revealed that the template "itap of a {class}" from Tip-adapter was identified and excluded as an outlier by SPC during its initial phase. This underscores the ability of SPC to autonomously filter outliers of spurious prototypes, thereby circumventing potential pitfalls inherent in manual selections.

20

10

0

10

20

30

40

50

20

10

0

10

20

30

40

Prompt Embedding

(a) All prompt templates

20

10

0

10

20

30

40

50

20

10

0

10

20

30

40

Normal Embedding

(b) Isolation Forest

20

15

10

5

0

5

10

15

20

15

10

5

0

5

10

15

20

Kmeans Embedding

(c)

k

-means

To more intuitively illustrate the impact of outlier removal and clustering, we visualize the spurious features. Figure 6a displays the features of all the prompt templates within the feature space, revealing the conspicuous presence of outliers and redundant values. Figure 6b shows the visualization after outlier elimination: red points represent preserved values, while black points denote deleted outliers, including 'itap of a {class}'. It is evident that numerous redundant values persist. As depicted in

Figure 6c, after the clustering process, the residual spurious features are uniformly distributed across various positions, thereby providing a rational representation of the spurious prototype.

art_painting

cartoon

photo

sketch

(a) CLIP

art_painting

cartoon

photo

sketch

(b) Fine-tuned CLIP

art_painting

cartoon

photo

sketch

(c) FD-Align CLIP

4.4.3 OODVisualization

To more vividly demonstrate the influence of our method on model feature extraction, we conducted a visualization of the image features on PCAS [53], utilized for investigating OOD tasks, which contains category and domain annotations for images. Figure 7 illustrates the visualization of features for one of our categories. It is observable that CLIP adeptly distinguishes features across different domains. The ability of fully fine-tuned CLIP to differentiate information across various domains is comparatively diminished. After fine-tuning with FD-Align, the model reacquires the ability to discern information across different domains, substantiating FD-Align's ability to preserve the model's OOD performance. Additionally, as depicted in Figure 8, we visualize the features of different categories in various domains. On the sketch data, the model, after fully fine-tuning, exhibits a diminished capacity to differentiate between categories, whereas CLIP and FD-Align distinguish between various categories accurately.

cartoon_dog

cartoon_horse

sketch_dog

sketch_horse

(a) CLIP

cartoon_dog

cartoon_horse

sketch_dog

sketch_horse

(b) Fine-tuned CLIP

cartoon_dog

cartoon_horse

sketch_dog

sketch_horse

(c) FD-Align CLIP

4.4.4 Training Stability

Figure 9 depicts the evolution of model accuracy and loss on the validation set throughout the fully fine-tuning and FD-Align processes. Notably, during the initial phases of fine-tuning, fully fine-tuning prematurely encounters overfitting, culminating in a decline in accuracy and a surge in classification loss. Conversely, FD-Align adeptly maintains model robustness and stability throughout the fine-tuning process, effectively circumventing overfitting.

4.4.5 Limitation

As shown in Tables 2, our method is able to preserve the generalization of the model while fine-tuning the model. However, in certain specific cases, such as Traffic Sign, where the images primarily consist of target objects and lack category-independent information, the preserved pre-trained model's category-independent ability adversely impacts the performance on such data. Additionally, our method does not perform well on special scenarios, such as ChestX and EuroSAT, for which one possible reason is that the prompts we use do not contain the category-independent information of these data. Therefore, additional prompts may need to be designed for this type of data.

0

100 200 300 400 500 600 700

Step

0.93

0.94

0.95

0.96

Fully FT

FD-Align

0

100 200 300 400 500 600 700

Step

0.17

0.18

0.19

0.20

0.21

0.22

0.23

0.24

0.25

Loss

Fully FT

FD-Align

5 Conclusion

In this paper, we introduce a feature discrimination alignment fine-tuning ( FD-Align ) method pretrained models in few-shot learning. Leveraging the remarkable text-visual alignment capabilities of CLIP, we employ text features of category-agnostic descriptions as spurious feature prototypes. Furthermore, we constrain the probability distribution of image features extracted by the model on spurious features, both before and after fine-tuning, to ensure the robustness of the model subsequent to fine-tuning. Experimental results substantiate the efficacy of our approach in enhancing fine-tuning performance while ensuring robustness across distributions.

Acknowledgment

Huimin Ma and Bochao Zou are supported by the National Natural Science Foundation of China (No.U20B2062, No.62227801, No.62206015) and National Key Research and Development Program of China (2022ZD0117900). Weiran Huang is supported by 2023 CCF-Baidu Open Fund and Microsoft Research Asia.

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Marcos V Conde and Kerem Turgutlu. Clip-art: Contrastive pre-training for fine-grained art classification. In CVPR , 2021.

Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930 , 2021.

Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. In AAAI , 2022.

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In CVPR , 2022.

Kwanyong Park, Sanghyun Woo, Seoung Wug Oh, In So Kweon, and Joon-Young Lee. Per-clip video object segmentation. In CVPR , 2022.

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In ICLR , 2022.

Soumya Suvra Ghosal, Yifei Ming, and Yixuan Li. Are vision transformers robust to spurious correlations ? In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability , 2022.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NeurIPS , 2017.

Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. Adaptive cross-modal few-shot learning. In NeurIPS , 2019.

Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. Boosting few-shot learning with adaptive margin loss. In CVPR , 2020.

Zihao Jiang, Yunkai Dang, Dong Pang, Huishuai Zhang, and Weiran Huang. Film: How can few-shot image classification benefit from pre-trained language models? arXiv preprint arXiv:2307.04114 , 2023.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , 2022.

Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint arXiv:2304.01195 , 2023.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV , 2022.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR , 2022.

LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML . PMLR, 2018.

Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308.13320 , 2023.

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR , 2020.

Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In UAI , 2022.

Rafayel Darbinyan, Hrayr Harutyunyan, Aram H Markosyan, and Hrant Khachatrian. Identifying and disentangling spurious features in pretrained image representations. arXiv preprint arXiv:2306.12673 , 2023.

Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious correlations in multi-modal models during fine-tuning. arXiv preprint arXiv:2304.03916 , 2023.

Anil Palepu and Andrew Beam. Self-supervision on images and text reduces reliance on visual shortcut features. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability , 2022.

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining . IEEE, 2008.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV , 2015.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML . PMLR, 2019.

Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models , 2019.

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR , 2020.

Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data , 5(1), 2018.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV , 2019.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW , 2013.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop . IEEE, 2004.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing . IEEE, 2008.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Largescale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition . IEEE, 2010.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7), 2019.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR . IEEE, 2012.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In ECCV . Springer, 2014.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurIPS , 2016.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In IJCNN . IEEE, 2013.

Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science , 350(6266), 2015.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV . Springer, 2014.

Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The quick, draw!-ai experiment. Mount View, CA, accessed Feb , 17(2018), 2016.

Brigit Schroeder and Yin Cui. Fgvcx fungi classification challenge 2018. Available online: github. com/visipedia/fgvcx_fungi_comp (accessed on 14 July 2021) , 2018.

Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in plant science , 7, 2016.

Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 , 2019.

Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI , 2018.

Appendix

A Detailed Results of Different ID Datasets

Figure 10 delineates the effectiveness of our approach across different datasets. Impressively, FDAlign surpasses both WiSE-FT and the fully fine-tuning on a vast majority of these datasets. As the shot count rises, the advantage of FD-Align becomes even more salient. It is pertinent to note that when data is scarce, WiSE-FT tends to underperform compared to fully fine-tune.

0

1

2

4

8

16

SHOT NUMBER

60

62

64

66

68

70

72

74

76

accuracy

Average

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

20

22

24

26

28

30

32

accuracy

Aircraft

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

92

93

94

95

accuracy

Caltech101

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

45

50

55

60

65

70

accuracy

DTD

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

40

50

60

70

80

90

accuracy

EuroSAT

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

65

70

75

80

85

90

accuracy

Flowers102

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

78.0

78.5

79.0

79.5

80.0

80.5

81.0

accuracy

Food101

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

63.5

64.0

64.5

65.0

65.5

66.0

66.5

accuracy

ImageNet

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

82

84

86

88

90

accuracy

Oxfordpet

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

56

58

60

62

64

66

accuracy

StandfordCars

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

62

63

64

65

66

67

68

69

accuracy

SUN397

FT

WiSE-FT

FD-Align

ZS CLIP

0

1

2

4

8

16

SHOT NUMBER

60.0

62.5

65.0

67.5

70.0

72.5

75.0

77.5

80.0

accuracy

UCF101

FT

WiSE-FT

FD-Align

ZS CLIP

B Time Cost of FD-Align

Table 5 shows the time cost of fully fine-tuning and FD-Align on 1-shot ImageNet. Evidently, FD-Align introduces only a negligible increase in time overhead. It is noteworthy that during the inference phase, both FD-Align and fully fine-tuning share identical processes, resulting in consistent inference cost. A significant advantage of FD-Align is its ability to boost performance across varied methodologies with just fine-tuning once.

C More Results on OOD Datasets

We also evaluate the performance of fully fine-tuned models on the OOD dataset. All models are trained on miniImageNet and performance is tested on other datasets. As shown in Table 6, our method performs better on Meta-Dataset, but worse on BSCDFSL. After analysis, we believe that the reason is that the model is fine-tuned on miniImageNet, which mainly contains natural image information, so the model better retains the domain information of natural images. However,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to adapt large pre-trained vision-language model to few-shot tasks. To this end, the authors propose to decouple the category-related and category-independent information to alleviate overfitting when adapting large model to few samples. They claim to maintain the visual encoder's ability to extract category-independent information during fine-tuning. They conduct experiments on various datasets, including CoOp-datasets and OOD datasets to verify the effectiveness of proposed methods.\n\nSTRENGTHS:\n1.The paper is organized and written well and the paper looks well polished. The overall story is clear to me.\n\n2.The authors conduct experiments on extensive datasets for comprehensive evaluation.\n\nWEAKNESSES:\n1.The motivation is not very convincing. First, as shown in Figure 1 and line 36-42, I do not think the separability among domains could significantly affect the performance of downstream tasks, since we conduct classification within single domain. Second, as discussed in line 110-118, why we should contain the ability of extracting category-independent information. In fact, if the model can ignore the category-independent information and only extract the category-related information, it will obtain better performance and be very robust, even on different domain.\n\n2.The proposed method is kind of naive and can not be considered as a systematic approach.\n\n3.The experments are not very convicing, although they are conducted on various datasets. As shown in Table 1, some existing works are not compared, like CoOp, PLOT. I do not understand why this work use ViT-B as backbone, since many baselines use ResNet50. This work should conduct experiments with ResNet50 and compared with the reported results of baselines, instead of using a new backbone and reproduce the results baselines. Even with ViT-B, PLOT++ has achieve 70.6% average accuracy on 1-shot tasks, but this work only achieve 63.92%.\n\nQUESTIONS:\nPlease answer the questions in weakness.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to enhance the performance of pre-trained CLIP for few-shot learning tasks while maintaining their generalizability and mitigating the risk of overfitting. To achieve this goal, the key contribution of this paper is the spurious information extractor that captures category-independent information from image features. By aligning the probability distributions of image features under the spurious weights, the fine-tuned model's ability to extract spurious features is preserved. Overall, this paper offers a robust CLIP fine-tuning method for improving the performance and generalization capabilities of pre-trained models in few-shot learning tasks.\n\nSTRENGTHS:\n1) Cross-domain and OOD few-shot learning are important research topics in this field. In this respect, the goal of the proposed method is backed by strong motivations.\n2) Good performance. Fairly extensive experiments demonstrate that CLIP pretraining is promising for the few-shot and OOD few-shot learning tasks. Ablation studies indicate the innovations are helpful.\n\nWEAKNESSES:\n1) In my opinion, utilizing features from CLIP is knid of conflicting to few-shot settings. Sine CLIP is pretrained on multi-class large datasets, there could be some information leakage on similar classes, can it still be considered a few-shot learning task in this situation? As can be seen from Table 3, performance on some datasets that are unfamilar with pretrained CLIP (e.g., ChestX) are relatively poor.\n2) More discussions/results regarding the cross-domain and dataset settings should be added. For instance, how about the performance on the same class under different domains (e.g., painted, sketch and real dog)? What kinds of pre-training source images (domain or object) are more helpful?\n3) There is insufficient information about implementation details, such as the Isolation Forest and K-means algorithms, which makes it difficult to reproduce the work.\n4) Apart from the training stability, it is not specified how long the proposed method takes for the entire training (i.e., pretraining and fine-tuning) and how efficient it is compared to existing studies.\n\nQUESTIONS:\n1) As mentioned in Weakness 1, the autors should give clearer explanations, or analyze the similarity between the pretraining dataset and the finetuning dataset.\n2) For weakness 2, it is suggested to provide some cross-domain comparisons on the same class, several examples on the classification confidence are also helpful.\n3) Typo errors such as \"\"wheile\" -> \"while\" in line 260.\n\nLIMITATIONS:\npotential limitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to tackle few-shot learning utilizing the powerful pretrained CLIP model. The primary objective is to construct multiple CLIP text prompts with different context and construct a term to regularize the learning process. This helps prevent overfitting to irrelevant correlations. Specifically, the method involves creating specific prompts for each class, combining category-independent factors such as \"toy,\" \"drawing,\" and \"plastic,\" with the object category name. A loss term is designed to ensure that the extraction of spurious information remains consistent both before and after fine-tuning the CLIP model.\n\nSTRENGTHS:\n1. The notion of using the capabilities of the CLIP model to enumerate potential spurious factors, and subsequently constructing a regularization mechanism for few-shot learning, is a intriguing concept.\n\n2. Some experimental results presented in this paper, particularly the findings depicted in Figure 4 and Figure 6, seems to be promising and somehow validates the proposed method.\n\nWEAKNESSES:\n1. The basic assumption of this paper, that if we fine-tuning CLIP model, the model tends to over-fit to the spurious information, needs a further validation. For example, it might be possible to use GradCAM or other visualization method to showcase such a phenomenon. \n2. The notation is somewhat confusing and a careful proof-reading is needed as there are many typos in this paper, for example:\n    (1) in line 134, The definition of C should be mentioned.\n    (2) the index i is used twice in the equation after line 142, this causes confusion\n    (3) W_spu was mistakenly written as W_sup in the same equation\n    (4) The prompts in Figure 3 give an good example of category-independent information mentioned in line 142. It is better to refer to Figure3 in line 142 as a concrete example.\n    (5) It is better to choose a different notation to denote object class, and context class, e.g., toy, plastic, ... in the equations \n3. What is Fine-tuning CLIP baseline in Table 2? Is it using the first loss term only? If yes, the proposed method only leads to marginal improvement over this simple baseline. If that is the case, then the benefit of the proposed strategy might not be significant.\n4. Also, the improvement by using IF and K-means is also marginal. In most cases, the improvement is less than 0.5%\n\nQUESTIONS:\n1. What is Fine-tuning CLIP baseline in Table 2? Is it using the first loss term only?\n\nLIMITATIONS:\nYes, the limitation has been properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies the problem of fine-tuning a pre-trained CLIP to downstream classification tasks with few-shot samples. The authors propose to fine-tune the category-dependent feature while retain the category-independent feature in order to improve the robustness of the fine-tuned model. The proposed method is compared with several baselines on several benchmarks.

STRENGTHS:
The motivation is clear. The paper is clearly written.

WEAKNESSES:
- I'm not fully convinced by several key statements/arguments in the paper:
    - L114-118, why retaining the spurious features can help model robustness? I think the less spurious feature it learns, the better it transfers to different domain? I also do not understand "This allows the model to combine different spurious information based on the category information, thereby maintaining generalization when the object appears in a new context, style, or domain."
    - L131-132, "the object represents the causal information and 'a photo of' represents the spurious information". I do not understand why "a photo of" can contain spurious information? As the authors explain, the spurious information includes domain, style, or background informaiton. However, "a photo of" contains neither of them. It does not discriminate between different domains or backgrounds or styles. Then the spurious feature extractor is not actually extracting any spurious informaiton. 

- Only fully fine-tuning and wise-ft are compared. What about other parameter-efficient fine-tuning methods such as LoRA [1] and VPT [2]? Also the gap between fully fine-tuning and FD-Align seems not really significant.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new few-shot learning method leveraging the pre-trained multi-model backbone CLIP. The method aims to eliminate the spurious information within the text embedding, and consequently regularize the image features to deliver better few-shot learning results. To do so, the authors propose to average pooling the text embeddings for the non-contextual information and enforce their KL distance with the image features. The evaluate the proposed method, the authors conduct experiments on multiple benchmarks in different settings.\n\nSTRENGTHS:\n-- The authors’ motivation sounds interesting. It’s a new angle to disentangle the text embeddings and accordingly regularize the image features. \n\n++ The results are looking good, the proposed method achieves good results on multiple benchmarks and settings (1, 2, 4, 8, 16 shots).\n\nWEAKNESSES:\n-- Although the motivation is clear and interesting and the authors claim to disentangle the causal information from the spurious information, I didn’t see any clear visualisations to support their claim. In Figure 1, the authors show better class centroids for their method, but it cannot directly show the effect of disentanglement. I suggest the authors show some similarity maps on both learned causal and spurious representations in future versions.\n\n-- For the same purpose, what if employing a small off-the-shell saliency model to first mask the spurious information?\n\n-- This paper can further benefit from improved organization. In Section 2.2, the authors simply list the related works without meaningful discussions. The readers, especially the ones that are not directly working on the same topic would be confused about details such as 1) Is it a common practice to jointly incorporate a frozen and a learnable visual encoder in few-shot learning with CLIP? Is any of the mentioned literature doing so? If not, what’s the gain of this design and what’s the learnable-parameter vs. accuracy trade-off? 2) Is there any changes in the prompt group compared with the literature? Authors claim to identify outliers with K-means. However, prompts like `a toy {c}` and `the plastic {c}` cannot describe most of the images and introduce outliers and it’s easy to simply just remove these manually designed prompts.\n\n-- Also, in section 2.1, the authors mostly introduce the datasets instead of introducing and discussing the few-shot learning literature. Looks like it can be moved to the experiment section instead of related work. \n\n-- In the proposed framework, the authors conduct Isolation forest as well as KMeans to eliminate outlier features, I wonder whether it slows down the training and inference.\n\n-- Minor: the scores for Zero-shot CLIP in tables 1 and 2 are not aligned well.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a fine-tuning method for pre-trained models in few-shot learning via CLIP's text and visual feature alignment capability. Specifically, the authors use text information to assist in decoupling spurious information from causal information while keeping the spurious information unchanged during the training process. This way, the authors claim the proposed FD-Align model can maintain its generalizability while effectively mitigating the overfitting problem from previous fine-tuning approaches.\n\nSTRENGTHS:\nThe concept of utilizing text to distinguish between causal and spurious feature extraction for enhancing the generalizability of few-shot model training is innovative. Additionally, the discovery that retaining the spurious feature extraction component unchanged pre and post fine-tuning can assist fellow researchers in addressing the few-shot learning issue is noteworthy. In addition to the originality of the concept, the results appear to be encouraging on various datasets when compared to other recent works. Finally, the authors also attached the code for easy reproducibility.\n\nWEAKNESSES:\nThe presentation and illustrated figures could be improved for better clarity. Additionally, the experimental results do not show comprehensive superiority compared to WiSE-FT, indicating some drawbacks in the proposed approaches.\n\nQUESTIONS:\nAccording to the authors, keeping the original ability to extract spurious features can aid in solving the few-shot learning problem because it helps reduce overfitting. In situations where the image content is straightforward, identifying the correlation between the background and foreground can be a valuable tool for recognizing unfamiliar objects. For example, an airplane is commonly found in the sky or at an airport rather than in a mountainous environment. However, as image content becomes more complex, it is unclear how much background information is still helpful for this task. Additionally, causal/spurious feature disentanglement appears to be reliant on the quality of the prompt, which can introduce uncertainty into the performance since different individuals may provide different prompts.\n\nLIMITATIONS:\nThe authors utilized the underperformed scores to explain the limitation of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}