Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective
Authors: Yanan Zhang
Jiangmeng Li
Lixiang Liu
Wenwen Qiang
Venue: NeurIPS 2024 poster (2024)
Keywords: causal
adaptation
foundational models
Primary area: machine_vision
TL;DR: Guided by the theory of causation, we propose semantic decoupling and uncertainty modeling to conduct prompt tuning on CLIP for downstream tasks.

Abstract: Foundational Vision-Language models such as CLIP have exhibited impressive generalization in downstream tasks. However, CLIP suffers from a two-level misalignment issue, i.e., task misalignment and data misalignment, when adapting to specific tasks. Soft prompt tuning has mitigated the task misalignment, yet the data misalignment remains a challenge. To analyze the impacts of the data misalignment, we revisit the pre-training and adaptation processes of CLIP and develop a structural causal model. We discover that while we expect to capture task-relevant information for downstream tasks accurately, the task-irrelevant knowledge impacts the prediction results and hampers the modeling of the true relationships between the images and the predicted classes. As task-irrelevant knowledge is unobservable, we leverage the front-door adjustment and propose Causality-Guided Semantic Decoupling and Classification (CDC) to mitigate the interference of task-irrelevant knowledge. Specifically, we decouple semantics contained in the data of downstream tasks and perform classification based on each semantic. Furthermore, we employ the Dempster-Shafer evidence theory to evaluate the uncertainty of each prediction generated by diverse semantics. Experiments conducted in multiple different settings have consistently demonstrated the effectiveness of CDC.

Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective
Yanan Zhang 12, Janngmeng ; Lixiang Liu Wenwen Qiang*+
University of Chinese Academy of Sciences Institute of Software Chinese Academy of Sciences
1tt.y//man11109990ma11......... com,
aangaaaaaam...a.aannnnnnna.m {jiangmeng2019, lixiang, giangrenneem)@isaaaaacccoo
Abstract
Foundational Vision-Language models such CLIP have exhibited impressive generalization in downstream tasks. However, CLIP suffers from a two-level misalignment issue, i.e., task misalignment and data misalignment, when adapting to specific tasks. Soft prompt tuning has mitigated the task misalignment, yet the data misalignment remains challenge. To analyze the impacts of the data misalignment, we revisit the pre- pre-training and adaptation processes of CLIP and develop structural causal model. We discover that while expect to capture task- relevant information for downstream tasks accurately, the task-irrelevant knowledge impacts the prediction results and hampers the modeling of the true relationships between the images and the predicted classes. As task-irrelevant knowledge is unobservable, we leverage the front-door adjustment and propose Causality-Guice Guideed Semantic Decoupling and Classification (CDC) to mitigate the interference of task- irrelevant knowledge. Specifically, we decouple semantics contained in the data of downstream tasks and perform classification based on each semantic. Furthermore, we employ the Dempster-Shee -haffer evidence theory to evaluate the uncertainty of each prediction generated by diverse semantics. Experiments conducted in multiple different settings have consistently demonstrated the effectiveness of CDC.
Introduction
Benefiting from large-scale training data, foundational Vision-Language models such as CLIP []. ALIGN [2], and Florence 3 have demonstrated remarkable zero-shot generalization capabilities across a wide range of downstream tasks.
Despite the strong generalization of CLIP, there exists a two-level misalignment between CLIP and downstream tasks, which hinders its adaptation to these tasks. Specifically, the first misalignment caused by the discrepancy between the pre-training objectives of CLiP and the objectives of downstream tasks, i.e., the task misalignment. During pre-training, the texts associated with images often contain rich semantic information, e.g.. "a yellow Abyssinian lying on table". Accordingly, as shown in Figure [a), in the learned embedding space, the distance between the image and the entire textual description can be close, indicating a strong semantic relationship, but the distance between the image and the individual semantic elements such as "Abyssinian" or "table" is not sufficiently close. When performing a downstream task that requires classifying an image as an "Abyssinian", the model encounters challenge in capturing the specific semantics associated with "Abyssinian" and may fail to complete the task. To address this issue, soft prompt tuning 44 5 6 is proposed. By introducing learnable tokens into the model and tuning them with task-specific data, CLIP can identify task-related semantics and adapt to the corresponding task.
"Equal contributions.
Conresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

66
80
Accuracy (%)
Accuriicy (%)
New
70
Rase
53
yellow Abyssinian lying on table
0.37
Abyssinian
13
mall
0.25
Epoch (b)
Figure I: (a) A motivating example of task misalignment, illustrating the cosine similarities between an image and various descriptions in the embedding space of CLIP. (b) A motivating experiment on data misalignment, showing the accuracy trends for base and new classes across different training epochs on the DTD dataset.
However, not all downstream tasks have available annotated data for prompt tuning. As a result, inconsistency exists between the training and testing data, constituting the second misaligument, i.c., data misaligument. The inconsistency in data misalignment Label Inconsistency: The training and testing classes do not completely overlap. For instance, some classes present in the training data might not appear in the testing data and vice versa. We refer to the classes that appear in both training and testing as base classes and the classes that appear only in testing
arise due to main reasons: (1)
classes. (2) Distribution Inconsistency: Even if they share the same class names, the distributions of the classes in the training and testing data may differ, resulting in distribution inconsistency. In such cases, the testing classes are essentially also new classes. Researchers generally agree that prompt tuning potentially leads to the overfitting of the CLIP model to the base classes, deviating CLIP from its original behavior and thus hindering its generalization new classes (7] R. To explore whether the overfitting holds, we conduct prompt tuning based on MaPLe 66) on the DTD dataset [9. We record the performance of the base classes and the new classes under different training epochs. As shown in Figure [b), as the number of training epochs increases, the performance on base classes gradually improves; however, the performance on the new classes first increases and then decreases. The phenomenon is consistent with the definition of overfitting.
To explore why overfitting occurs and how it impacts the recognition of new classes, we revisit the pre-training and adaptation processes of CLIP and develop a Structural Causal Model (SCM) [10 analyze the causal relationships among variables abstracted from these two processes. From the perspective of data generation, generative factors refer to the elements that control the content of images and texts. We argue that the knowledge embedded in CLIP determines the content that can be obtained from CLIP. Therefore, the knowledge in CLIP is equivalent to the generative factors it contains. Via soft prompt tuning, we expect the learned prompts to capture the semantics determined by the task-relevant generative factors and eliminate the influence of task-irrelevant generative factors. However, due the data misalignment, the task-relevant generative factors estimated on the base classes potentially being task-irrelevant for the new classes. When estimating the causal relationship between images and the label space of new classes, task-irrelevant generative factors could interfere. In addition, task-irrelevant generative factors cannot be observed in practice, so cannot remove their effects by adjusting for them. To solve this problem, we introduce task-related semantics as intermediate variables from images to the label space of new classes and apply the front-door adjustment to estimate the true causal relationship between the two.
To implement the front-door adjustment, we propose Causality-Guider Semantic Decoupling and Classification (CDC), consisting of Visual-Language Dual Semantic Decoupling (VSD) and Decou- pled Semantic Trusted Classification (DSTC). Specifically, we incorporate multiple prompt templates within the model and encourage distinct templates to represent different semantics through VSD. In addition, decoupled semantics exhibit varying classification uncertainties. Therefore, we propose DSTC, which performs the classification task independently based on each decoupled semantic and estimates the corresponding uncertainties simultaneously. Subsequently, we fuse the results to obtain more accurate and reliable predictions. Through experiments conducted in various settings, our proposed CDC has effectively enhanced the performance of CLIP.

---

Our contributions include: (i) We identify the nwo-level misalignment that exists in the adaptation of CLIP to downstream tasks and illustrate that, due to the data misalignment, the prompt learned through soft prompt tuning becomes overfitted to the base classes; (ii) We develop an SCM to investigate the pre-training and adaptation processes of CLIP, revealing how the overfitting occurs and how it impacts the recognition of new classes. We discover that the task-irrelevant generative factors serve as the confounder when estimating the true causal relationship between the images and the label space of new classes; (iii) To mitigate the impact of task-irrelevant generative factors on downstream tasks, we propose CDC, which implements the front-door adjustment. Through experiments multiple datasets and various tasks, the effectiveness of CDC has been demonstrated.
Related Work
Adaptation in Vision Language models. To enhance the performance of vision-language models such as CLIP on downstream tasks, researchers propose to utilize few-shot adaptation. Generally, two main technical approaches have emerged: 1) adapter-basee methods 4 12 [3], which incorporate adapters into CLIP to fine-tune the features generated by CLIP, enabling better adaptation to specific tasks; 2) prompt-based methods, which primarily inject task information by appending learnable tokens to CLIP. While methods like CoOp [4) CoCoOp [5] and ProGrad [14] add tokens to the text input layer, VPT 15] and CAVPT [16] introduce tokens on the image branch as well. MaPLe [6] and UPT [17] incorporate tokens on both the image encoder and text encoder. In addition, MaPLe adds tokens into the intermediate layers of the network to enhance the model's adaptability to the task. Among all prompt- based upproaches, the most similar to method is ArGue (18), which aims to improve model performance by iterating over the attributes that each category possesses. However, ArGue relies the assistance of other large language models for attribute generation and requires additional preprocessing steps. In contrast, CDC achieves competitive performance with ArGue without relying on any external knowledge, solely through the end-to-end training.
Causal Inference. recent years, causal inference IO || has been extensively employed in computer vision and multi-modal learning. Generally, researchers explore the causal inference in two
promising directions: counterfactual [9 20, 21 [22] and intervention [23 24 26, 27]. 28 and
21 22
propose generate the counterfactual images to improve the robustness of the model. [27]
proposes intervening in the distribution of training data to eliminate the interference of the spurious correlation on predictions for domain generalization problems. To the best of our knowledge, are the first to apply causal inference to analyze CLIP's overfitting in downstream classification. We identify the confounders and eliminate their effects through the front-door adjustment.
3 Problem Formulation and Analysis
3.1 Problem Formulation
CLIP [ is an impressive vision-language model comprised of an image encoder and a text encoder, capable of embedding images and texts into a shared multi- latent space. Through contrastive learning, paired image-text embeddings are optimized for maximum cosine similarity, while unpaired ones are minimized. The intuition behind such design is based on a fundamental assumption: any they contain similar
paired images and texts are expected to form tight semantic correspondence, content. By pre-training on a large-scale dataset consisting of 400 million image-text pairs, CLIP demonstrates strong generalization in zero-shot and few-shot downstream tasks.
In zero-shot classification with C classes, an image x is transformed into an embedding v via the image encoder. For the text branch, CLIP [1] constructs prompts for each class using static template, e.g., "a photo of a [CLASS]", where [CLASS] denotes the class name of the e-th class, inputted into the text encoder to obtain embedding vectors w.
E 21, ..... C}. These prompts for each class. By calculating the cosine similarity between the image embedding e and the text embeddings we, we can obtain the probability of image belonging to the e-th class:
exp(sim((e, we)/T) ply = cla) IC eezpeap(simi we \/T)
(1)
where sim(;.) denotes the cosine similarity, and is temperature scalar.
3

---

In the few-shot setting, where a certain amount of labeled data is available, it is feasible to learn template instead of manually specifying one. Initially, researchers construct learnable template
t t= (PI, P2,.... .... Pd; l by appending the word embedding le of the c-th class to d learnable tokens as the input of the text encoder. recent years, intermediate layers of both text and image encoders also employ learnable tokens. These tokens are optimized with the cross-entropy loss to adapt to the specific task. The learned template performs the classification task the same way as the static one.
3.2 Problem Analysis: Causal Perspective
To understand how the knowledge contained in CLIP []] affects the downstream classification tasks, we propose an SCM shown in Figure 2 The directed acyclic graph
<<< V, E' > depicts the causal relationships among variables abstracted from the pre-training and adaptation processes. Each node V; E corresponds to a variable, Edge V1 - V E E signifies that V, is the cause of V-
Firstly, we define each variable in Figure 2 represents the data used in the pre-training stage. Generative factors, the fundamental semantic elements that control the gen- eration of images and texts, can determine the content of both images and texts. specific task, these generative factors can be divided into two subsets: the set of task- relevant factors G,- and the set of task-irrelevant factors G,, Taking the pet classification task as an example, task-relevant generative factors determine features such as coat colors, body shapes, and ear types, which are highly relevant to identifying pet breeds. Conversely, task-irrelevant generative factors influence semantics which do not provide helpful information for pet classification. For downstream task, X represents the image space, while Y corresponds to the true labels of these images. Y represents the predicted labels obtained through the learning. Additionally, S denotes the semantics within the images X closely relevant the current task.
Figure 2: SCMs. Solid and dashed cir- cles indicate the observable and unob- servable variables, respectively.
By training on diverse data, CLIP acquires relatively comprehensive visual and language knowledge. Therefore, it can be assumed that through the learning of D, we obtain comprehensive generative factors that constitute the entire visual space. Since the learning process of pre-training is not specific to any particular downstream task, these generative factors include not only task-related but also insk-irrelevant ones, i.c., D - G2 and D - G,,, for a specific task. By combining the generative factors, any image X can be generated, regardless of whether it has appeared in the pre-training dataset D. Images typically include not only content related to their category under a particular task but usually other content well, hence G, X and G; + X. However, only task-related generative factors can determine the true label Y of images, i.e., Y.
Given that the training data D for CLIP is invariant, in Figure aa). the path X +- G G. is blocked, resulting in X being associated with Y solely through G.. i.e., G. -Y. When performing downstream classification tasks, we aim to model the relationship between X and that arises from G,-. As shown in Figure 2 b). X + y represents the objective our modeling. G and G, unobservable and easily confused in practice. Therefore, we perform soft prompt tuning to help capture the information of G, and eliminate the influence of G, However, the estimated task-related factors
always accurate. Furthermore, due the data misalignment, the task- relevant generative factors estimated based on the training data of base classes are not necessarily the task-relevant factors for new classes, G; denotes the set of task-irrelevan generative factors that are incorrectly retained, and the set of factors that are task-relevant for the base classes but task iirelirrelevantt for the new classes. Since we incorrectly assume that G; task-relevant, G influences Y. i.c.. G, Y. When the learned prompt template overfitted the base classes, generative factors in & will further increase, hindering the estimation of the true causal relationship between X and V.
According to the definition of the backdoor path, when estimating the true causal relationship between X and Y, there exists a backdoor path X + G; + ý, where G; serves as a confounder. Given that G; is unobservable, it becomes impracticable to eliminate spurious relationships caused by G; via the back-door adjustment. To address this issue, we introduce an intermediate variable that represents task-relevant semantics and is derived from X, thus establishing the path X - S, as

---

Learnable Text Prompts
Lie PCF gldotze))
Images
e
Text Encader
Image Encoder
P2
Lggin
10
hand-crafted prompt
L
Augmentation
Figure 3: Framework of CDC. t denotes a single template, while p1, P2, Pa represent tokens in the template. Different colors indicate diverse templates. "fuse" refers the process of generating the final classification results from multiple template results as shown in Equation (8). The text encoder and the image encoder are frozen, and only the tokens in the prompt templates are learnable.
shown in Figure c). The classification for can be regarded as predicting based on the semantics that X possesses, thus + Y. S satisfies the front-door criterion for (X, Y). and the causal effect from X to y can be calculated using the front-door formula:
P(Ý y)do(x))
Σ P(s|z) Σ "uuld'ssP((e
first term
second term
To estimate the true causal relationship between X and Y, will discuss how to obtain the semantic s and how to estimate the probability of a sample æ belonging to class y.
Methodology
Guided by the front-door criterion, propose CDC, comprising two essential components: VSD and DSTC. Specifically, VSD focuses on decoupling the semantics represented by diverse templates. DSTC aims generate the classification scores via Equation (2) based the decoupled semantics. We will elaborate on the details these two components in Section 4.1 and Section 4.2 respectively.
4.1 Visual-Language Dual Semantic Decoupling
To decouple the semantics embedded in the images and texts, the concept of semantic set is introduced. semantic set serves as collection that encapsulates the semantics related to the same visual attribute. For example, multiple semantic sets be constructed for the task of pet classification, such as the set of coat colors, the set of body shapes, and the set of ear types. Each semantic set contains some possible values that represent the variations of the corresponding attribute. For instance, the semantic set "coat colors" includes semantics like red, white, purple, etc. The semantic sets enable us to understand and analyze the semantic information embedded in pet images.
To represent different semantic sets, we employ multiple templates in the CLIP model, i.e., tt {e,,2...,, (MM) in Figure 3 where M denotes the number of templates. Each template aims to characterize a unique semantic set. To achieve decoupling of the semantics represented by these templates, we propose VSD, which considers both text and image branches. On the image branch, we leverage diverse augmentation methods to generate the image inputs and employ the corresponding augmentation method for each individual template. The intuition behind this is to help different templates capture distinct feature invariance against different augmentations, c.g., performing random rotation augmentation enables the template capture features with rotation invariance. Since defining the precious semantic sets, such as coat colors and ear types, remains challenging without the help of external knowledge, group semanties that exhibit the same type of feature invariance into semantic set in practice.
On the text branch, we expect to maximize the diversity of embeddings corresponding to different templates, ideally achieving orthogonality. To achieve this objective, we aim to ensure that the embedding of the e-th class in the m-th template, denoted as w cannot be accurately classified by
5

---

embeddings from other templates, i.e., using any ut to classify w where m' * m, there is no bias towards the results. Specifically, the objective function can be formalized as:
M
1 dae M IC
Σ
ΣΣ
P(clw"" wn/) log P(clu" w
m/=1m fm
where P( w denotes the probability that w² is predicted as the ë-th class by the embedding of m/-th template. Specifically, P(clw" wm can be calculated as:
P(clu"" wn exp(sim((") w 3/T)
Ee exp( crep(sim(w") w²') )/7)
Intuitively, the embedding vector w from the m-th template can be regarded as a sample to be classified. The objective of Equation (3) is to maximize the entropy of the classification results, resulting in a uniform distribution. The classification results are obtained using the embedding of all other templates. Lae aims to enhance the semantic distinctions among different templates, thereby encouraging each template to maintain a unique semantic.
While boosting the diversity of the semantics in different templates, Lae may hinder the learning of the task relevant semantics. To alleviate this problem, we propose introducing a consisteney loss to assist each template in capturing information that is close to the hand-crafted template. The proposed consistency loss can be formulated as follows:
Lcon -
Σ log P(clw"", w²),
(5)
where w denotes the embeddings of the hand-crafted template 1 obtained via the original CLIP model. See Appendix A for the detailed templates.
Via VSD, we obtain distinct semantic sets that contain decoupled semantics. Different classes have varying values on the same semantic set, so we consider the values of s in Equation (2) to be the text embeddings of each class on all semantic sets, i.e, s E {w/" 1m=
4.2 Decoupled Semantic Trusted Classification
To implement Equation (2). we need to clarify how to estimate the ((((d) based on the decoupled semantics. Firstly, the probability P(s|z) that a sample has semantic s is obtained by calculating the cosine similarity between the representation v of and 8, i.e., P(s[æ) = N(sim(s, v)). where N denotes the normalization operation.
After determining the possible values of s, we clarify how to estimate the second term Equation (2). Let æ' represent the samples from the training set and s = w=" then: (i) if c # y, P(yl=', 0. By updating the template, we expect each class e carry unique semantics w.th Therefore, it cannot be assumed that the specific semantics of c attributable to another class; (ii) if e = y but x' does not belong to the e-th class, P(y|2', s) = O; (iii) if y and x' belong to the c-th class, we maximize the similarity between s and x' by prompt tuning so that the semantics of and x' are as consistent as possible. As x' contains both relevaelevnnt and iusk-irrelevant semantics, and its task-relevant semantic is decoupled to wh w ..., w.M via VSD, is the subset of x'. Therefore, the second term Equation (2) shrinks to P(yls).
Ideally, if 8 contains and only contains the category-specific semantic information that explicitly supports the classification of sample z' into c, P(y = cls) 1, and P(y # c|r', 0. However, because of the data misalignment, while applying the learned templates to the new classes, this ideal scenario does not happen. In reality, the qualities of the learned semantics vary for different templates and classes, allowing the value of P(y s) to alter, thereby resulting in significant differences in the uncertainty of classification results derived from different templates. To solve this issue, we propose to estimate the uncertainty 29 30 of each prediction from diverse templates via Dempster- Shafee theory 31 (2). Specifically, we consider e h(sim (" v)) as the evidence of classifying the feature as belonging to the e-th class under the m-th semantic set, where / represents a function that maps the cosine similarity to a non-negative value. We assume that there exists a Dirichlet
distribution a [a]" o22 mn) where at -.0" + 1. This formulation allows us quantify

---

the belief that the feature belongs to class well as the uncertainty associated with the prediction as:
C T
C
(6)
ATh
Am
where T Σ E/ + Once we have obtained the classification beliefs and uncertainties for c'=1
all semantic sets, we proceed perform evidence fusion to obtain the final classification results. Specifically, for the m-th and m'-th semantic set, after evidence fusion, obtain the belief of the sample æ belonging to class C, as well as the uncertainty of the classification as:
gam' --C (6 6th + b
um + b%t u")), umm
1- 1-C 2 um
(7)
The results from all prompt templates can be iteratively fused, shown in the following equation:
b1
m== =1 if 3lcma m <
B
6m
Bm- DmUm-1),,
1-0
(8)
u t"
m =1 if
U
M
In this formulation, B represents the belief after fusing the results from the first templates, while U denotes the corresponding uncertainty. The process of evidence fusion can be understood as the summation of results over all semantics in Equation (2). Consequently, the result in Equation (2 can be reformulated as:
BM
p(Ý cldo(x)) = ΣC _ BM
(9)
To calculate the probability that sample belongs to class e during the testing phase, we employ Equation (9). Accordingly, during the training stage, the cross-entropy loss which is used for learning the prompts is adjusted to the trusted cross-entropy loss to model the uncertainty information. For each training sample, the trusted eross-entropy be formalized as:
A
Li-ce
Σ ()a)))
(10)
where y denotes the index of the ground-truth class and v-(-) denotes the digamma function.
4.3 Overall Architecture
We demonstrate the architecture our proposed method in Figure 3. While training, the overall loss function can be formalized as:
LCDE Lice + BLde + Leonn
(11)
where 3 and denote the hyper-parameters to tune the influence of Laee and Leons respectively. Refer to Algorithm and Algorithm for the detailed training and testing pipeline.
Experiments
Following previous works 6.. conduct experiments to evaluate proposed method with three different settings. These settings encompass the base-to-new setting as well as two out-of-distribution (OOD) settings, i.c., the cross-dataset setting and the cross-domain setting. Refer to Appendix B for a detailed overview of the evaluation protocol.
5.1 Experimental Settings
Datasets. In the base-to-new setting, we conduct experiments based on 11 datasets: ImageNet 33
Caltech101 34. Oxford Pets 35), Stanford Cars (36), Flowers102
[37 Food101 [38) FGVC
Aircraft [39 SUN397 [40] DTD
9 EuroSAT
41, and UCF-101
42 In the following,
7

---

Algorithm The training pipeline of CDC
Input:
16-shot dataset X, the learning rate f, two hyper-parameter / and % the number of templates M, and the list of augmentation methods { ...,,.,...AA A..,,A)).
Initialize the parameters of CLIP with the parameters of the pre-trained model.
Randomly initialize the learnable tokens {0), 20,00 0a1 repeat
for i-th training iteration do Iteratively sample minibatch X' from X. for m-th template do Xmm (X')..
Generate the image features et of X with the image encoder of CLIP. Generate the text features u with the text encoder.
Generate the classification evidence ett with R and w": ex = h(sim(w e²))
Using Equation (IO) calculate Lt-ce.
Using Equation 3 and Equation (5) to calculate Laee and Ccon.
Using Equation 11 to calculate LCnc-
1 6th EVoLCDC.
end for end for until converge.
Algorithm 2 The testing pipeline of CDC
02, The testing dataset X, the number of templates M, and (0),02, BM.. Sample a test data x from X for m-th template do
Input:
Generate the image features n of æ.
Generate the text features wt with the text encoder.
Generate the classification evidence e with Tt and w":: e h(ssmmw"" , v²)).
end for
Using Equation
and Equation (9
to generate the final classification results.
abbreviate these datasets as ImageNet, Caltech, Pets, Cars, Flowers, Food, Aircraft, SUN, DTD, EuroSAT, and UCF. For the oetoo--generalizationn task, we adopt ImageNet as the source dataset. The remaining 10 are the target datasets in the cross-dataset setting, and ImageNetV2 43), ImageNet-
44, ImageNet-A
45 and ImageNet-R [46) are the target datasets in the cross dross-domain setting.
Experimental Details. We follow the experimental settings of the baseline method MaPLe. Specifi- cally, we utilize pre-trained CLIP with ViT-B/16 the visual encoder. The number of learnable tokens is fixed at 2, whereas the prompt depth varies, being 9 for the base-to-new setting and for the OOD setting. The learning rate is 0.035, and the batch size is 4. All models trained using an SGD optimizer on an NVIDIA 3090 GPU. Our proposed CDC introduces three additional hyperparameters: 3 and 7, which represent the weights for Lae and Leons respectively, and M, which denotes the number of prompts. Furthermore, different augmentation methods have a notable impact on the model's 's performance. We set 3 = ? 0.01, and M in the base-to-new setting, and 3 = 3, = 0.01, and A in the OOD setting. For a detailed analysis of the influence of varying param-parsmeterss and augmentation methods, please refer to Appendix C We report the accuracies of base classes and novel classes, along with their harmonic mean (HM), averaged over runs.
5.2 Base-to-New Generalization
As shown in Table we compare our method with three recent works, CoOp, CoCoOp. and MaPLe. For a fair comparison, do not include methods that utilize external knowledge, such ArGue (18), HPT 47]. and CoPrompt [8], although our method demonstrates competitive performance.
Compared to the baseline method MaPLe, our approach has achieved remarkable improvements in the accuracy of base classes by 1.06%, and the accuracy new classes by 2.24%, which leads to an improvement of 1.70% in HM. Specifically, across all 11 datasets, our method outperforms MaPLe
8

---

Table 1: The comparison with baseline methods on base-to-novel generalization setting.
CoOp
4
CoCoOp
[
MaPLe 6
CDC
Dataset
Base
New
HM
Base
New
HM
Base
New
HM
Base
New
HM
A
Avg
82.69 63.22 71.66 80.47 71.69 75.83 82.28 75.14 78.55
83.34 77.38 80.25
+1.70
ImageNet
76.47 67.88 71.92 75.98 70.43 73.10
76.66
70.54 73.47 77.50 71.73 74.51
+1.04
Caltech
98.00 89.91 93.73 97.96 93.81 95.84
97.74 94.36 96.02 98.20 94.37 96.25 +0.23
Pets
93.67 95.29 94.47 95.20 97.69 96.43
95.43 97.76 96.58 96.07 98.00 97.02 +0.44
Cars
78.12 60.40 68.13 70.49 73.59 72.01
72.94 74.00 73.47 73.80 73.97 73.88 +0.41
Flowers
97.60 59.67 74.06 94.87 71.75 81.71 95.92 72.46 82.56 96.93 75.07 84.61
+2.05
Food
88.33 82.26 85.19 90.70 91.29 90.99
90.71 92.05 91.38 90.87 92.33 91.59 +0.21
Aircraft
40.44 22.30 28.75 33.41 23.71 27.74
37.44
35.61 36.50 37.47 37.50 37.48 +0.98
SUN
80.60 65.89 72.51 79.74 76.86 78.27
80.82 78.70 79.75 82.37 80.03 81.18 +1.43
DTD
79.44 41.18 54.24 77.01 56.00 64.85
80.36
59.18 68.16 82.70 64.10 72.22 +4.06
SAT
92.19 54.74 68.90
87.49
60.04
71.21
94.07
73.23 82.35
95.10
82.33 88.26 +5.91
UCF
84.69 56.05 67.46 82.33 73.45 77.64
83.00 78.66 80.77 85.70 81.73 83.67 +2.90
Table 2: Comparison of CDC with recent approaches on cross-datases evaluation.
Source
Target
ImageNet Caltech
Pets
Cars
Flowers Food Aircraft SUN DTD SAT UCF Avg
CoOp
71.51
93.70
89.14 14 64.51
68.71
85.30
18.47
64.15 41.92 46.39 66.55 63.88
Co-CoOp
71.02
94.43
90.14 65.32
71.88
86.06
22.94
67.36 45.73 45.37 68.21 65.74
MaPLe
70.72
93.53
90.49 65.57
72.23
86.20
24.74
67.01
46.49 48.06 68.69 66.30
CDC
71.76
94,47
90.77 66.27
72.67
86.27
24.50
68.07 46.60 49.13
68. 60 66.73
11 datasets for base classes and in 10 datasets for novel classes. Notably, on ImageNet dataset, our approach achieves accuracy increase of 0.84% in base classes, 1.19% in new classes, and 1. 1.04% in HM. For the challenging datasets SAT, DTD, and UCF, our method delivers significant HM improvements of 5.91%. %, 4.06%, and 2.90%, respectively. In summary, significantly enhances the gencralization of the CLIP model for unseen classes, while ensuring stable improvements in the performance of base classes, demonstrating its superiority a wide range of datasets.
proposed method
5.3 Out-of-Distribution Generalization
Cross-dataset. We evaluate the cross-dataset generalizaiinnnnnnn CDC by learning prompts on ImageNet and then transferring them to the remaining 10 datasets. From Table 2 for the source dataset ImageNet, our proposed method achieves performance improvement by 1.04% compared MaPLe, outperforming all previous methods. Compared to MaPLe on the 10 target datasets, our method achieves better performance on 8 datasets. Considering all target datasets, our method achieves an average performance improvement of 0.43%.
Cross-domain. We transfer the prompts learning in ImageNet to its four invariants evaluate the coss-domain generalization of CDC. From Table compared to MaPLe, our proposed method improves the performance of ImageNetV2, ImageNet-S, and ImageNet-R by 0.80%, 1.18%, and 1. 1.12%, respectively, and brings slight performance decrease of 0.5% for ImageNet-A, A, thus leading to an average performance improvement of 0.65%. Generally, CDC enhances the generalization of the model when dealing with data from different domains.
5.4 Ablative Experiments
Table presents the experimental results obtained after adding some of the components to the baseline method MaPLe. The performance of MaPLe is shown in the first row.

---

Table 3: Comparison of CDC with recent approaches in the cross- doos--domain setting.
Source
Target
ImageNet
ImageNerV2
ImageNet-S
ImageNet-A
ImageNer-R
Avg.
CLIP
66.73
60.83
46.15
47.77
73.96
57.18
CoOp
71.51
64.20
47.99
49.71
75.21
59.28
Co-CoOp
71.02
64.07
48.75
50.63
76.18
59.91
MaPLe
70.72
64.07
49.15
50.90
76.98
60.28
CDC
71.76
64.87
50.33
50.40
78.10
60.93
The effectiveness of multiple templates and DSTC. In T all results except the baseline method are obtained by increasing the number of templates to four. In the second row, we train the learnable tokens within the templates using the conventional cross-entropy loss and average the predictions from different templates gener- ate the final classification results. Compared to the bascline, multiple templates lead to 0.68% improvement in the performance of base classes and 0.92% enhancement in the performance of new classes, resulting in an overall HM gain of 0.81% %. While modeling the uncertainty of each template, shown in the third line, the perfor- mance for new classes achieves an improvement
Table 4: The results the ablative experiments on pase-to-nove generalization setting.
VSD
M DSTC
Base
New
HM
Image Text
X
x
82.28 75.14 78.55
X
X
X
82.96
76.06 79.36
4
V
82.65
76.78 79.61
/
V
x
83.35
76.72 79.90
82.93 76.83 79.77
4
V
83.34 77.38 80.25
0.72%, and HM increases by 0.25%. Without explicitly decoupling the semantics within different templates, increasing the number of templates effectively boosts the performance. We argue that the semantics acquired by distinct templates, cach with varied initialization, is inherently independent (See analysis in Appendix C). Therefore, simply applying multiple templates guided by the front-door adjustment could enhance the performance.
The effectiveness of VSD. The fourth and fifth rows of Table present the experimental results obtained by decoupling semantics solely within the image and text branches, respectively. When compared to the third row, it becomes evident that decoupling semantics at either branch alone does not yield substantial improvement the accuracy of new classes. However, upon comparing the sixth row with the third row, it is observed that when both decoupling methods are employed concurrently, the model achieves a 0.69% increase in base class performance and 0.60% improvement in novel class performance, leading to overall HM enhancement of 0.64%. Despite the inherent decoupling of semantics among different templates, the utilization of VSD further promotes the learning of distinct semanties for each template, thereby enhancing the model's overall performance.
Conclusion and Future Discussion
Through motivating experiments, we identify the two-level misalignment that exists when CLIP is applied to downstream tasks. We further show that soft prompt tuning may worsen the second misalignment due to overfitting to the base classes, which impairs the generalization CLIP. To analyze this problem, we propose an SCM and discover that the task-irrelevant generative factors serve as the confounder while estimating the true causal relationship between images and label space of new classes. To address this issue, propose to implement front-door adjustment via CDC. Specifically, we introduce multiple templates to represent the decoupled semantics, then leverage VSD to further facilitate the decoupling of semantics, and finally fuse predictions from different templates via DSTC. The results under multiple experimental settings demonstrate that our proposed approach effectively improves the generalization of CLIP when adapted to downstream tasks.
Limitations and broader impacts. CDC incurs more time consumption, as applying different templates in the image branch requires encoding an image multiple times during both training and testing. The analysis of two-level misalignment CL.PP and the modeling of SCM inspiring for the community.
10

---

Acknowledgments
The authors gratefully acknowledge the valuable feedback provided by the anonymous reviewers. This work is supported by the Fundamental Research Program, China, Grant No.JCKY00213300000
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3748-8766 PMLR, 2021.
[2] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang. editors. Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904-4916. PMLR, 2021.
[3] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Shi, Lijuan Wang. Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. CoRR, abs/2111.11432 2021.
[4] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training- frrenngg-reeeeee adaption of CLIP for few-shot classification. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXV, volume 13695 of Lecture Notes in Computer Science, pages 493-510, Springer, 2022.
[5] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learn- ing for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 16795-16804, IEEE, 2022.
[6] Muhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Maple: Multi-moda prompt learning. IEEE/CVF Conference on Computer Vision and Putlern Recognition, CVPR 2023, Vancouver, BC, Canuda, June 17-24, 2023, pages 19113-19122, IEEE, 2023.
[7] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 15144-15154, IEEE, 2023.
[8] Shuvendu Roy and Ali Etemad. Consistency-guide prompt learning for vision-language models. 2024.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 3606-3613. IEEE Computer Society, 2014.
[10] Judea Pearl. Causal inference in statistics: An overview. Statistics surveys, pages 96 146, 2009.
[11] Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. Causal inference in statisties: primer.
John Wiley & Sons, 2016.
[12] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vasion-language models with feature adapters. Int. Comput. Vis., 132(2):581-595, 2024.

---

[13] Fang Peng, Xiaoshan Yang, Linhui Xiao, Yaowei Wang. and Changsheng Xu. Sgva-clip: cenaatic--uideed visual adapting of vision language models for few-shot image classification. IEEE Trans. Multim., 26:3469-3444 2024.
[14] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 15613-15624 IEFE, 2023.
[15] Menglin Jia, Luming Tang. Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, volume 13693 of Lecture Notes in Computer Science, pages 709-727. Springer, 2022.
[16] Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Clare- visual prompt tuning for vision- Saiiunla pre-trained model. CoRR, abs/2208.08344 2022.
[17] Sanjoy Chowdhury, Sayan Nag, and Dinesh Manocha. Apollo: Unified adapter and prompt learning for vision language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, P Proceedings of the 2023 Conference on Empirical Methods in Natural I Language Process- ing, EMNLP 2023, Singapore, December 6-10, 2023, pages 10173-10187. Association for Computational Linguisties, 2023.
[18] Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Argue: Attribute-giieed prompt tuning for vision-language models. CoRR, abs/2311. 11122311.144900 16494, 2023.
[19]
Chun-Hao Chang, Gcorge-Alexandry Adam, and Anna Goldenberg. Towards robust classifica- tion model by counterfactual and invariant data generation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 15212-15221. Computer Vision Foundation / IEEE, 2021.
[20] Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero-shot and open-set visual recognition. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 1, virtual, June 19-25, 2021, pages 15404-15414. Computer Vision Foundation / IERE, 2021.
[21] Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfac- tual samples synthesizing for robust visual question answering. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10797-10806. Computer Vision Foundation IEEE, 2020.
[22] Long Chen, Yuhang Zheng, Yulei Niu, Hanwang Zhang, and Jun Xiao. Counterfactual samples synthesizing and training for robust visual question answering. IEEE Trans. Pattern Anal. Mach. Intell., 45(1 (5511): 1):13218 - 2023.
[23] Yinjie Jiang, Zhengyu Chen, Kun Kuang, Luotian Yuan, Xinhai Ye, Zhihua Wang, Fei Wu, and Ying Wei. The role of deconfounding in meta-learnigg In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 10161-10176. PMLR, 2022.
[24] Wenwen Qiang, Jiangmeng Li, Changwen Zheng, Bing Su, and Hui Xiong. Interventional contrastive learning with meta semantic regularizer. In Kamalika Chaudhuri. Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18018-18030. PMLR, 2022.
[25] Jiangmeng Li, Yanan Zhang, Wenwen Qiang, Lingyu Si, Chengbo Jiao, Xiaohui Hu, Changwen Zheng, and Fuchun Sun. Disentangle and remerge: Interventional knowledge distillation for few-shot object detection from A conditional causal perspective. 2023.
[26] Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
12

---

[27] Xinyi Wang, Michael Saxon, Jiachen Li, Hongyang Zhang, Kun Zhang, and William Yang Wang. Causal balancing for domain generalization. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[28] Axel Sauer and Andreas Geiger. Counterfactual generative networks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[29] Zongho Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[30] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi view classifi- cation with dynamic evidential fusion. IEEE Trans. Pattern Anal. Mach. Intell., 45(2):2551-
2566, 2023.
[31] Glenn Shafer. A mathematical theory of evidence, volume 42, Princeton university press, 1976.
[32] Arthur P Dempster. Upper and lower probabilities induced by a multivalued mapping. In Classic works of the Dempster-Shafer theory of belief functions, pages 57-72. Springer, 2008.
[33] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255, IEEE Computer Society, 2009.
[34] Li Fei-Fei, Robert Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst., 106(1):59-70, 2007.
[35] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pages 3498-3505. IEEE Computer Society, 2012.
[36] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013, pages 554-561. IEEE Computer Society, 2013.
[37] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Pro- cessing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pages 722-729. IEEE Computer Society, 2008
[38] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 - mining discriminative components with random forests. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part VI, volume 8694 of Lecture Notes in Computer Science, pages 446-461. Springer, 2014.
[39] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. CoRR, abs/1306.5151, 2013.
[40] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In The Twenty-Third IEEE Con- ference Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pages 3485-3492. IEEE Computer Society, 2010.
[41] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 12(7):211-2226, 2019.
[42] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402. 2012.
[43] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June
13

---

2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5389-5400. PMLR, 2019.
[44] Haohan Wang, Songwei Ge, Zachary C. Lipton, and Eric Xing. Learning robust global representations by penalizing local predictive power. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 10506-10518, 2019.
[45] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 15262-15271. Computer Vision Foundation IEEE, 2021.
[46] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of
generalization. In 2021 JEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 8320- -3390 IEEE, 2021.
[47] Yubin Wang, Xinyang Jiang. De Cheng. Dongsheng Li, and Cairong Zhao. Learning hierar- chical prompt with structured linguistic knowledge for vision-language models. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Fighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 5749-5757, AAAI Press, 2024.
14

---

A Hand-Crafted Prompts
To prevent the text embeddings obtained via the learned prompts from deviating from the semantics of their respective categories, we introduce a consistency loss Leon to apply constraints on the learned prompts. For each dataset, borrow from CLIP hand-crafted prompt t to generate w.. The prompts employed in this process are detailed below.
"Pets": "a photo of a (), type of pet."
"Flowers": "a photo of a [}. type of flower."
"Aircraft": "a photo of a (1, type of aircraft."
"DTD": "a photo of a (). type of texture."
"SAT": "a centered satellite photo of ()."
"Cars": "a photo of (J."
"Food": "a photo of (J. type of food."
"SUN": "a photo of (]."
"Caltech": "a photo of f1."
"UCF": "a photo of a person doing ()."
"ImageNet": "a photo of ()."
B Evaluation Protocol
Base-to-New Setting. Following CoCoOp and MaPLe, split the classes into un-overlapping subsets, i.e., base classes and new classes, for each dataset. In training, base classes are leveraged for learning the prompts under the 16-shot setting, where each class contains 16 annotated samples. Subsequently, the prompts are transferred to new classes which are unseen during the training. In this setting, the evaluation metrics include accuracy on base classes, new classes, and their corresponding harmonic mean (HM), which can be calculated by:
HM
(12)
Base
New
where Base and New denote the accuracy of base classes and new classes, respectively.
onttr--Distributionn Setting. In the OOD setting, we initially train the model using the source dataset and then transfer the learned prompts the target dataset evaluate the robustness of the method. In our experiments, following COOP, CoCoOp and MaPI.e, the source dataset is ImageNet, employing all 1000 of its classes for prompt learning, where each class comprises 16 annotated samples. In the cross-dataset setting, 10 datasets, Caltech, Pets, Cars, Flowers, Food, Aircraft, SUN, DTD, EuroS and UCF, are target datasets. In the cross-domain setting, the target datasets include ImageNetV2. ImageNet-S, ImageNet-A, and ImageNet-R, all sharing the same classes as ImageNet but differing in distributions.
C Additional Experiments
C.1 Analysis of the
The number of templates M. Table presents the performance of the model in the base-to-nee setting and the corresponding computational complexity when the number of templates changes from to 8. The results in Table 5 are obtained solely using Li-ces without the help of VSD. As the number of templates increases, the performance of the model on the base classes and the new classes increases consistently, while at the same time, the computational overhead increases. Although better performance can be achieved when M is further increased to 8, we select M = 4 in order to balance the accuracy of classification with the computational overhead.
Weight (3 for Lae- Figure 4 gives the HM on both DTD and Aircraft datasets when varying the weights 3 of Lde From the figure, it is evident that using 3 5 and / 10 effectively improves
15

---

Table 5: Comparison of the performance under different template numbers.
M
Base
New
HM
Params
FPS
1
80.87
73.49
77.00
3.55M
600
2
82.80
76.59
79.57
7.10M
280.00
4
82.65
76.78
79.61
14.20M
185.71
8
83.40
77.29
80.23
28.40M
65.63
the performance of the model on these two datasets as compared to training without Lae. i.e., 3 = 0. When 3 further increased to 20, the performance model begins to decline. Considering the performance on each dataset, in the base-to-new setting, we set 5.
37.0
72.0
HM(%)
36.5
71.5
71.0
36.0
5
10
20
5
10
20
(a) DTD
(b) FGVC Aircraft
Figure 4: The impact of 3 on performance.
Weight 7 for Lcon. In Figure keeping = 5. From the results, we observe that setting y = 0.01 enables the performance both datasets to obtain an enhancement compared to y 0.0. When is further increased to 0.1, the performance on the Caltech10) 101 dataset shows a significant decrease. Considering all results together, set 7 0.01 in all base-to-new experiments.
provide the experimental results obtained by varying when
82.8
HM(%)
HM(%)
96.6
82.6
96.4
0.001
0.01
0.1
0
0.001
0.01
0.1
(a) UCF101
(b) Caltech101
Figure 5: The impact of 7 on performance.
Augmentation methods. Setting M 4, = 5, y 0.01, we vary the augmentation methods to explore the effect of different augmentations on model performance and demonstrate the results in T We set four of augmentations for 4 templates. Specifically, for Augmentation the augmentation methods include: (i) random crop and random flip; (ii) color jitter and random flip: (iii) random translation and random flip; (iv) random augmentation. For Augmentation 2, the four augmentation sets are: (i) random crop and random flip; (ii) random crop, random flip, and color jitter; (iii) random crop, random translation, and random flip; (iv) random crop and other random augmentations. Augmentation 2 adds the random crop to each augmentation set based on Augmentation From Table we notice that for most of the datasets such ImageNet, Caltech etc., there is no great difference in the performance obtained by Augmentation I and Augmentation 2.
16

---

Table The impact of augmentation methods on the performance.
Augmentation 1
Augmentation 2
Dataset
Base
New
HM
Base
New
HM
ImageNet
77.50
71.73
74.51
77.30
71.57
74.32
Caltech
98.63
93.67
96.09
98.20
94.37
96.25
Pets
95.97
97.67
96.81
96.07
98.00
97.02
Cars
72.33
71.63
71.98
73.80
73.97
73.88
Flowers
96.93
75.07
84.61
96.93
74.67
84.36
Food
90.93
92.23
91.58
90.87
92.33
91.59
Aircraft
36.83
36.23
36.53
37.47
37.50
37.48
SUN
82.37
80.03
81.18
82.00
79.77
80.87
DTD
82.70
64.10
72.22
82.13
61.73
70.49
SAT
95.10
82.33
88.26
92.87
75.83
83.49
UCF
85.70
81.73
83.67
85.40
81.23
83.26
However, for Cars and Aircraft, Augmentation 2 performs significantly better than Augmentation 1. Capturing discriminative information challenging for fine- mne-grained classification. We argue that the translation invariance and invariance to the object scale brought by random crop may be critical, so Augmentation performs significantly better than Augmentation! on these two datasets.
C.2 Template Study
provide the full results of our method under all templates, well the final performance after performing the evidence fusion which denoted by CDC. From Table CDC performs the best in 10 out of 11 datasets. Among them, several datasets gain essential improvements compared to the results under a single template, including ImageNet, Flowers, SUN, and DTD. search for better discriminative semantics
In Table
This suggests that the essence of approach is by increasing the number of templates but to eliminate the impacts of task-independent generative factors by combining multiple sets of decoupled semantics. Despite the moderate performance of any individual template, our method achieves a substantial performance enhancement by fusing them.
Table 7: The comparison with the results from different templates.
Dataset 70.10
S'
S
S3
S4
CDC
ImageNet
70.10
69.73
70.33
71.73
Caltech
94.10
94.73
94.57
94.17
94.37
Pets
97.60
97.60
97.37
97.87
98.00
Cars
72.93
73.57
70.87
72.63
73.97
Flowers
72.87
73.23
74.10
71.43
75.07
Food
91.60
91.40
91.73
91.73
92.33
Aircraft
34.77
35.33
35.70
35.60
37.50
SUN
78.53
77.87
77.73
78.17
80.03
DTD
59.60
57.83
57.87
60.43
64.10
SAT
73.73
75.67
74.57
73.73
82.33
UCF
77.83
76.00
77.60
78.13
81.73
C.3 Intrinsic Decoupling of Semantics from Different Templates
In Section 5.4, we demonstrate that increasing the number of templates can get a performance boost even without explicitly decoupling the semantics contained in different templates. We attribute this to the fact that templates based on different initializations do not leam exactly the same semantics. i.c., semantics contained different templates are intrinsically decoupled. To verify this suspicion, we collect the text embeddings obtained based on different templates without applying VSD, and compute the similarities between the embedding obtained from different templates. We plot the results of the DTD dataset in Figure 6 As can be seen from the figure, even for the same category,

---

0.57
0.53
0.42
0.46
0.41
0.64
0.58
0.42
0.40
0.45
0.68
0.63
0.50
0.51
0.64
0.43
0.50
0.44
0.59
0.67
0.46
0.46
0.50
0.64
0.70
0.50
0.50
0.42
0.60
0.40
0.40
0.55
0.54
0.66
0.47
0.47
0.39
0.44
0.39
0.49
0.50
0.42
0.64
0.53
0.57
0.58
0.52
0.69
0.58
0.40
0.45
0.39
0.52
0.59
0.48
0.50
0.40
0.49
0.61
0.54
0.56
0.48
0.56
0.62
(a)
(b)
(c)
Figure 6: The cosine similarities of the text embeddings under different templates based on the new classes of DTD. (a) The cosine similarities between the text embeddings generated by the O-th template and the 1-th template. (b) The cosine similarities between the text embeddings generated by the 1-th template and the 2-th template. (c) The cosine similarities between the text embeddings generated by the 2-th template and the 3-th template.
similarities between the text embedding obtained by different templates are still much less than (i.e., the value on the diagonal line in the figure). This indicates that the text embedding obtained by different templates has large discrepancy, which confirms our idea that the semantics learned by different templates are inherently decoupled. Therefore, better performance can be obtained through fusing results generated by these decoupled features.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contributions include: (i) we identify the problem of two-level misalignment in Lasion-Languaee Model adaptation: (ii) we propose an SCM to analyze the spurious correlations between images and the label space caused by the data misalignment; (iii) we propose CDC mitigate the spurious relationships while estimating the true causal relationships. These contributions are covered in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of our work in terms of computational efficiency in Section
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not include theorems and formulas in the paper. We analyze the mis- alignment problem from a causal perspective, and the main assumptions and corresponding analysis are provided in Section 3.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the details of our method Section 4 and present the necessary hyper-parameter in Section 5.1 to ensure the reproducibility of our method. Furthermore, we provide the code of our proposed method the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
20

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include the code our proposed method the supplementary material. The necessary environments and data preparation procedures are provided in the GitHub repository of our baseline method MaPLe.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
Cotps///SSSuuumiiiiiiice.a sttpbbmmsssoooooccccco for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https / wtppps.ccppublic.. corooliccggideeee htte://guuddee/eooSuuubmeiie wwtoobmsssiooooiiioiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the necessary hyper-parameters in Section 5.1 and analyze the hyper-parametee Appendix C.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We don't report the error bar in our experimental results because the previous works do not report the error bar and we follow them. All of our experimental results are averaged over runs of 3 different seeds.
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the compute workers in Section 5.1 and list the params and time exccution in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics hattps::/nnutri hhtce//hhhht///////op/pp//p//hhe
Answer: [Yes]
Justification: We conduct the research with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [Yes]
Justification: We provide the broader impacts of our work in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific sets), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets g.g code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All datasets, models, and code involved in our paper open source.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
23

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palasets.paperssttt paperssi thcode. com con/dataset package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "hFqtNMATGP", "reviewer": "Reviewer_Ptf4", "summary": "This paper investigates the task and data misalignment issues in pre-trained vision-language models such as CLIP. It discovers that the task-irrelevant information significantly affects the prediction of CLIP and soft prompt tuning cannot mitigate the data misalignment issue. The authors propose a novel Causality-Guided Semantic Decoupling and Classification method to mitigate the interference of task-irrelevant information. The experimental results show that the proposed method effectively mitigates the data misalignment and improves the generalization of CLIP.", "questions": "Is it feasible to adapt the proposed method to various tasks such as object detection, image captioning, or visual question answering.", "limitations": "The authors have discussed the limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-organized. The introduction of the method and the figures are clear and easy to understand. The description of the experiment setting is detailed, which makes the paper reproducible.\n2. The proposed methods to mitigate the task and data misalignment of CLIP are highly-motivated and intuitive.\n3. The authors design and conduct exhaustive experiments to demonstrate the effectiveness of the propose method. The proposed methods provide significant improvements on the generalization of CLIP.", "weaknesses": "1. In the experiments section, the method is currently adapted solely to the CLIP model. This limitation may not fully demonstrate the model's universality. The authors can adapt the method to various vision-language models with different architectures to showcase broader applicability.\n2. The experiments are exclusively conducted on image classification tasks. The authors can explore adapting vision-language models (VLMs) to a wider range of tasks, such as object detection, image captioning, or visual question answering, to further validate the model's versatility and performance across diverse applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Ptf4 for the valuable suggestions. The mentioned issues are addressed as follows:\n***\n**W1**: In the experiments section, the method is currently adapted solely to the CLIP model. This limitation may not fully demonstrate the model's universality. The authors can adapt the method to various vision-language models with different architectures to showcase broader applicability.\n\n**A1**: Thank you for your valuable suggestions. We believe that most current VLMs suffer from the misalignment problems we analyzed in our paper. The SCM we developed effectively models the overfitting problems that VLMs could encounter when adapting to downstream tasks. The proposed CDC, guided by the SCM, can effectively alleviate the interference of task-irrelevant information in VLMs on downstream tasks, thereby enhancing performance.\n\nTo validate the universality of our proposed method, we conduct further experiments based on two additional models:\n\n(1) **CLIP based on ViT-L/14**. ViT-B/16 is the most common setting in prompt tuning and is also employed in our paper. Compared with CLIP based on ViT-B/16, CLIP based on ViT-L/14 is a more powerful model, and experiments on ViT-L/14 can verify the effectiveness of CDC across VLMs of different capacities; \n\n(2) **ALIP based on ViT-B/32**. ALIP introduces a bi-path model that integrates raw text supervision and synthetic caption supervision. We conduct experiments on ALIP to demonstrate the generalization of our CDC to VLMs with different pre-training objectives. \n\nRefer to the **global rebuttal** for the experimental results and more analysis. The experimental results demonstrate that our proposed CDC can effectively improve the performance of various VLMs on downstream classification tasks.\n\n***\n**W2**: The experiments are exclusively conducted on image classification tasks. The authors can explore adapting vision-language models (VLMs) to a wider range of tasks, such as object detection, image captioning, or visual question answering, to further validate the model's versatility and performance across diverse applications.\n\n**A2**: We appreciate your valuable suggestions. To validate whether our proposed method is beneficial for applying VLMs to a wider range of tasks, we explore the application of our method in one-shot object detection. As shown in Table 3, our approach achieves the state-of-the-art performance. The experimental results confirm the effectiveness of our method.\n\nIn one-shot object detection, each foreground object category has only one labeled instance available for training. Due to the limited amount of training data, it is often inconsistent with the true data distribution of the dataset. Therefore, there exists a serious misalignment of the training data with the testing data, which is in line with our definition of data misalignment. Our proposed SCM framework is highly effective in addressing this challenge, which is achieved by transferring our SCM (Figure 2 in the original manuscript) to the scenarios of one-shot object detection. Specifically, $D$ in our SCM can be interpreted as the knowledge contained within the base model of one-shot object detection, which encompasses both knowledge related to one-shot object detection (denoted as $G_r$) and knowledge unrelated (denoted as $G_i$). The process of fine-tuning the base model on a single instance can be seen as an attempt to extract $G_r$ while eliminating $G_i$. However, due to the scarcity of samples required for fine-tuning, $G_i$ is often not accurately identified, thus interfering with the modeling of the true causal relationships between the testing instances and their corresponding categories. $\\hat{G_i}$ becomes a confounder.\n\nOur proposed CDC effectively mitigates such confounders. Concretely, we generate feature vectors for the foreground object categories using the text encoder of CLIP and align the visual features from the pre-trained base model with the corresponding text features in CLIP through a projector. Additionally, we enhance the feature alignment via prompt tuning and CDC within the CLIP text encoder. As shown in Table 3, our proposed CDC has achieved state-of-the-art performance among works in recent years. Compared to the baseline method DeFRCN [R2], our approach yields an average performance improvements of 1.61% across three different data splits, demonstrating its effectiveness in extending VLM to other tasks.\n\nIn future work, we will continue to explore the potential of CDC for enhancing the application of VLMs, e.g., CLIP, across a wider range of tasks.\n\n**Table 3** One-shot experimental results of AP50 on the VOC dataset [R3] (%).\n| Methods | Novel Set 1 | Novel Set 2 | Novel Set 3 |\n| --- | --- | --- | --- |\n|Pseudo-Labelling [R4]|54.50|32.80|48.40|\n| DeFRCN [R2] | 57.03|35.82 | 52.49  |\n| ICPE [R5] | 54.30 | 33.50 | 50.90 |\n| DiGeo [R6] | 37.90 | 26.60 | 30.40 |\n| CDC (Ours)| **59.62** | **37.67** | **52.89** |\n\n[R2] Qiao L, Zhao Y, Li Z, et al. Defrcn: Decoupled faster r-cnn for few-shot object detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 8681-8690.\n\n[R3] M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, A. Zisserman, The pascal visual object classes (VOC) challenge, Int. J. Comput. Vis. 88 (2) (2010) 303–338.\n\n[R4] P. Kaul, W. Xie, A. Zisserman, Label, verify, correct: A simple few shot object detection method, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 14237–14247.\n\n[R5] X. Lu, W. Diao, Y. Mao, J. Li, P. Wang, X. Sun, K. Fu, Breaking immutable: Information-coupled prototype elaboration for few-shot object detection, in: B. Williams, Y. Chen, J. Neville (Eds.), AAAI 2023, AAAI Press, 2023, pp. 1844–1852.\n\n[R6] J. Ma, Y. Niu, J. Xu, S. Huang, G. Han, S. Chang, Digeo: Discriminative geometry-aware learning for generalized few-shot object detection, in: CVPR 2023, IEEE, 2023, pp. 3208–3218.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HEljMeZXAK", "reviewer": "Reviewer_JJqY", "summary": "This paper addresses the two-level misalignment (task and data) issue in adapting CLIP to specific tasks. The authors develop a structural causal model to analyze CLIP's pre-training and adaptation processes, revealing how task-irrelevant knowledge interferes with predictions. To mitigate this, they propose Causality-Guided Semantic Decoupling and Classification (CDC), which implements front-door adjustment. CDC includes Visual-Language Dual Semantic Decoupling (VSD) to represent different semantics through multiple prompt templates, and Decoupled Semantic Trusted Classification (DSTC) to perform classification based on each decoupled semantic while estimating uncertainties. Experiments demonstrate CDC's effectiveness in enhancing CLIP's performance across various settings and tasks, addressing the challenge of data misalignment in vision-language model adaptation.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* CDC is well motivated from a causal perspective and has significant technical novelty.\n* Clear writing and well organized.\n* Experiment results show the effectiveness of CDC.", "weaknesses": "* Figure 1(a) appears to illustrate task misalignment. Consider enhancing the caption of Figure 1 with more detailed explanations to clarify this concept.\n* Regarding data misalignment, it would be beneficial to provide a more precise definition. Does it specifically refer to discrepancies in classes between training and testing processes? It's important to clarify that data misalignment encompasses both label misalignment and distribution misalignment. A brief explanation of each type would improve understanding.\n* In Figure 3, the term \"fuse\" is used. It would be helpful to clarify the meaning and context of this term within the figure.\n* How about accuracy if we directly use zero-shot test for CDC?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer JJqY for the valuable comments and constructive suggestions. The mentioned issues are addressed as follows:\n***\n**W1**: Figure 1(a) appears to illustrate task misalignment. Consider enhancing the caption of Figure 1 with more detailed explanations to clarify this concept.\n\n**A1**: Thanks for your valuable suggestions. The example we present in Figure 1(a) is indeed to illustrate the task misalignment problem. \n\nIn the final version, we will modify the caption of Figure 1 to: \"The motivating examples of the two-level misalignment. (a) Task Misalignment. Determined by the contrastive learning mechanism of CLIP, a given image is more similar to the entire textual description in the embedding space than to a single semantic element, which is inconsistent with the demands of the classification task. (b) Data Misalignment. The learned model intends to overfit on base classes. On the DTD dataset, as the number of training epochs increases, the accuracy of base classes rises, while the accuracy of new classes first rises and then drops.\"\n* * *\n**W2**: Regarding data misalignment, it would be beneficial to provide a more precise definition. Does it specifically refer to discrepancies in classes between training and testing processes? It's important to clarify that data misalignment encompasses both label misalignment and distribution misalignment. A brief explanation of each type would improve understanding.\n\n**A2**: Thank you for your suggestion. In our paper, data misalignment encompasses label inconsistency and distribution inconsistency. In the final version, we will further clarify the concept of data misalignment in the appendix to avoid confusion.\n\nData misalignment refers to the inconsistency between the distribution of training data and testing data. This inconsistency can arise due to two main reasons: \n\n(1) **Label Inconsistency**: The training and testing classes do not completely overlap. For instance, some classes present in the training data might not appear in the testing data and vice versa. We refer to the classes that appear in training as base classes and the classes that appear in testing as new classes. \n\n(2) **Distribution Inconsistency**: Even if they share the same class names, the distributions of the classes in the training and testing data may differ, resulting in distribution inconsistency. In such cases, the testing classes are essentially also new classes. \n\nLabel inconsistency and distribution inconsistency together constitute the problem of data misalignment.\n\nFurthermore, we have considered the effectiveness of CDC under both types of inconsistencies. The base-to-novel experimental setup mainly involves label inconsistency, while the cross-dataset and cross-domain setups address both label inconsistency and distribution inconsistency. Our proposed CDC has demonstrated effectiveness in all three experimental setups, highlighting its capability to address both types of inconsistencies.\n* * *\n**W3**: In Figure 3, the term \"fuse\" is used. It would be helpful to clarify the meaning and context of this term within the figure.\n\n**A3**: Thank you for your constructive suggestions. In Figure 3, \"fuse\" refers to iteratively combining the evidence from all templates according to Equation (7) to obtain the final classification results. We will add an explanation of the \"fuse\" operation to the caption of Figure 3: \"fuse\" represents the process of iteratively combining all evidence according to Equation (7) to obtain the final results.\n* * *\n**W4**: How about accuracy if we directly use zero-shot test for CDC?\n\n**A4**: Thanks for your valuable comment. The CDC method we proposed preserves the zero-shot ability of CLIP. In the cross-dataset transfer experiments, we use exactly zero-shot testing for the target datasets. To further verify the performance of our method under the zero-shot setting, in addition to the experiments in Table 2 of our manuscript, we have employed another architecture, i.e., ViT-L/14, to evaluate the zero-shot performance of CDC. The results of the additional experiments are presented in Table 4. \n\nAs shown in the table, compared to the baseline method MaPLe, our CDC method achieves an average performance gain of 0.43% under the zero-shot setting when using the ViT-B/16 structure, and an average zero-shot performance improvement of 1.05% when using the ViT-L/14 structure. These results further demonstrate the effectiveness of the CDC method in preserving the model's zero-shot ability.\n\n**Table 4**. Comparison of CDC on the zero-shot classification to evaluate the out-of-distribution generalization of CDC in the cross-dataset setting.\n| Method |Architecture| Caltech | Pets | Cars | Flowers | Food | Aircraft | SUN | DTD |SAT  |UCF  |Avg|\n| --- | --- |--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|CLIP| ViT-B/16| 93.30 | 89.10 | 65.6 | 70.7 | 85.90 | 24.70 | 62.60 | 44.00 | 48.40 | 67.70 | 65.20 |\n|CLIP+MaPLe| ViT-B/16| 93.53  | 90.49  | 65.57  |72.23  |86.20  |24.74  |67.01  |46.49  |48.06  |68.69  |66.30 |\n|CLIP+CDC| ViT-B/16| 94.47 |90.77 |66.27 |72.67 |86.27 |24.50 |68.07 |46.60 |49.13 |68.60 |66.73 |\n|CLIP|ViT-L/14|95.20|93.50|76.80|79.50|90.90|32.70|67.70|53.00|60.30|75.10|72.47|\n| CLIP+MaPLe| ViT-L/14 | 96.23  |92.57  |77.60  |76.90  |91.40  |30.07  |71.63  |54.23  |53.83  |75.90  |72.04  |\n|CLIP+CDC| ViT-L/14|96.70 |93.10 |77.33 |75.87 |91.53 |31.67 |72.87 |57.63 |58.00 |76.20|73.09|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zKZlg9Dccr", "reviewer": "Reviewer_7jjJ", "summary": "This paper investigates the two different misalignment issues between CLIP and downstream tasks, i.e., task misalignment and data misalignment.  The author designed several experiments that demonstrated that over-fitting occurs when tuning with the learnable prompt. They propose the Causality-Guided Semantic Decoupling and Classification(CDC) method to mitigate the impact of task-irrelevant generative factors on downstream tasks. The extended experiments demonstrate that the proposed CDC method is effective.", "questions": "see weaknesses", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper investigates the difficulty of adapting CLIP to downstream tasks via two-level misalignment, which is a bright idea to help the community understand the working mechanism. Then the author provides a comprehensive experiment that reveals how the overfitting occurs and impacts the recognition of new classes. The author uses the perspective of causal inference to alleviate the data misalignment and proposes CDC with front-door adjustment for implementation, which predicts with explicit evidence.", "weaknesses": "1. There is no obvious evidence to show that the CDC will improve the prediction in certain cases, which category previous wrong and correct with CDC. It would be better if several compared failure cases to demonstrate that the CDC can solve the question at which level and which case still needs more advanced methods.\n 2. It is interesting how the misalignment between CLIP and downstream pattern will change as the model’s capabilities increase, such as ViG-Large, and whether the more powerful model can solve the misalignment problem. Hence, it will be better if some comparisons of models can be added.\n3. As we all know, MLLMs already sweep through the multimodal community, it will be better if expand some discussion about the misalignment in this paradigm and current method whether easily general to that.\n4. How many parameters are tuned during downstream adapting? Compared to the pre-trained model, what is the ratio of tuned parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 7jjJ for the valuable feedback and constructive suggestions. The mentioned issues are addressed as follows:\n***\n**W1**: There is no obvious evidence to show that the CDC will improve the prediction in certain cases, which category previous wrong and correct with CDC. It would be better if several compared failure cases to demonstrate that the CDC can solve the question at which level and which case still needs more advanced methods.\n\n**A1**: Thank you for your suggestions. We provide several examples in the **attached PDF of the global rebuttal** to compare the prediction results of CDC with those of the baseline method MaPLe. We will include these analyses in the appendix to the final version of our paper, to enhance an intuitive understanding of our proposed CDC. \n\nIn the success cases in Figure 1, we observe that: \n\n(1) Based on different prompt templates, the model captures different semantic information of the samples and generates diverse prediction results. This validates the effectiveness of our semantic decoupling strategy. \n\n(2) By fusing the prediction results based on different templates (i.e., different semantic sets), CDC can obtain correct classification results. As shown in Figure 1 (e), the fused result is not a simple average of all predictions and may differ from all of them. This indicates that the fusion process comprehensively considers the credibility and consistency of each prediction result, enabling CDC to correctly classify samples that MaPLe fails to recognize.\n\nFor the failure cases in Figure 2, we discover that when the predictions based on each semantic set consistently lean towards the wrong category, CDC also struggles to correct the samples misclassified by MaPLe. Taking Figure 2 (a) as an example, intuitively, from different semantics such as the character's posture and the appearance of the yo-yo, the image is easily misidentified as playing the flute, making it difficult to classify. In the future, richer training data may enhance the model's ability to distinguish different objects and scenes, further improving CDC's performance.\n*** \n**W2**: It is interesting how the misalignment between CLIP and downstream pattern will change as the model’s capabilities increase, such as ViG-Large, and whether the more powerful model can solve the misalignment problem. Hence, it will be better if some comparisons of models can be added.\n\n**A2**: Refer to the global rebuttal for the details. \n***\n**W3**: As we all know, MLLMs already sweep through the multimodal community, it will be better if expand some discussion about the misalignment in this paradigm and current method whether easily general to that.\n\n**A3**: Thank you for your suggestions. The misalignment in MLLM is consistent with that presented in our paper. Theoretically, our approach can be applied to MLLM.\n\nMLLMs have demonstrated remarkable potential across a wide range of tasks. After aligning features from different modalities through pre-training, MLLMs often employ techniques such as instruction tuning to encourage the model to complete downstream tasks effectively.\n\nTake Instruct-BLIP as an example. During the instruction tuning phase, Instruct-BLIP learns from specific datasets for certain tasks to build an instruction-aware Q-Former, thus assisting the model in extracting informative features tailored to the given instruction. The learned Q-Former is expected to generalize to unseen datasets and unseen tasks. However, for different tasks, the informative features for similar instructions can be diverse. Therefore, the learned Q-Former risks overfitting the training data. This is similar to prompt tuning analyzed in our paper, where prompts overfit base classes. Therefore, we argue that the discrepancies between training and testing in Instruct-BLIP introduce \"data misalignment\" in a general sense.\n\nOur proposed CDC has the potential to address the data misalignment issue in Instruct-BLIP. To mitigate the interference of visual features that overfit training tasks, we can decouple the obtained features and estimate the importance of the semantics in the features. By performing a weighted fusion, we can assist the test tasks in extracting more task-relevant visual features, thereby improving overall task performance.\n\nIn conclusion, when aiming to enhance the zero-shot generalization performance of a model, the misalignment between training and testing data is a crucial issue that must be carefully considered. Our proposed SCM comprehensively models the misalignment problem in training and testing. CDC has the potential to be transferred to scenarios where misalignment exists, providing a valuable framework for addressing this common challenge in machine learning. In future work, we will investigate the specific implementation of CDC in MLLM to solve the data misalignment problem in it.\n***\n**W4**: How many parameters are tuned during downstream adapting? Compared to the pre-trained model, what is the ratio of tuned parameters?\n\n**A4**: Thank you for your valuable comments. In Table 5, we provide the number of learnable parameters of our method. Compared to MaPLe, our proposed CDC increases the number of learnable parameters from 3.55M to 14.20M, which brings an average performance improvement of 1.70% in the Base-to-New setting. \n\nIn the implementation, we can reduce the model overhead by having all templates share the V-L functions, i.e., CDC* in Table 5. CDC* adds 0.027M parameters compared to MaPLe, bringing an average performance improvement of 1.14%. The analysis of the parameters highlights the efficiency of CDC.\n\n**Table 5**. Comparison of prompting complexity among different methods. \n| Method | Params | Params % CLIP  | HM |\n| --- | --- | --- | --- |\n| CoOp | 2048 | 0.002 | 71.66 |\n| CoCoOp | 35360 | 0.03 | 75.83 |\n| Independent V-L | 31488 | 0.02 | 78.55 |\n| MaPLe | 3.55 M | 2.85  | 78.55 |\n| CDC(Ours) | 14.20 M | 11.40  | 80.25 |\n| CDC*(Ours) | 3.58 M | 2.87  | 79.69 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Response to **Weakness 1** of **Reviewer** **Ptf4** and **Weakness 2** of **Reviewer** **7jjJ**.
***
Thank you for your suggestions on exploring the impact of the misalignment issues we proposed across different models. We believe that most current VLMs can suffer from the misalignment problem when adapting to downstream tasks, regardless of the model's capacity and architecture.

Firstly, our proposed data misalignment arises from the distribution differences between training data and testing data during the adaptation process of VLMs, which is an inherent attribute of the task. During the process of tuning prompts based on training data, regardless of the model's architecture and pre-training method, the distribution of testing data remains unknown, causing the learned prompts to overfit the training data. Therefore, the data misalignment problem will persist.

Furthermore, to validate the efficiency of our proposed CDC in addressing the data misalignment problem regarding different VLMs and different model architectures, we conduct experiments based on two experimental settings: 

(1) **Base-to-New Generalization**. As shown in Table 1, we adopt an additional more powerful architecture, i.e., ViT-L/14, and another VLM, i.e., ALIP [R1], to validate the effectiveness of our CDC. Table 1 reports the harmonic mean (HM) of accuracy on base and new classes in the base-to-new experimental setup, based on CLIP and ALIP. As shown in the table, with ViT-L/14, CDC achieves a 5.78% average performance improvement over CLIP and a 2.66% improvement over MaPLe. With ViT-B/32, CDC achieves a 13.07% improvement over ALIP, and a 1.37% improvement over MaPLe.

(2) **Cross-Dataset Out-of-Distribution Generalization**. We also validate the efficiency of CDC in the cross-dataset experimental setting. Table 2 reports the zero-shot test results of MaPLe and CDC on 10 target datasets. As shown in the table, CDC achieves a 1.05% average performance improvement over MaPLe. 

The experimental results indicate that even with the use of a more powerful architecture and another VLM, CDC can still improve model performance by mitigating data misalignment issues during the prompt tuning process.


**Table 1**. The comparison with baseline methods on base-to-new generalization setting based on different VLMs and different architectures.
| Method |Architecture | Avg | ImageNet | Caltech | Pets | Cars | Flowers | Food | Aircraft | SUN | DTD |SAT  |UCF  |
| --- |--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |--- |
|CLIP |ViT-L/14  |78.75|76.51|96.69|96.63|79.23|82.75|94.35|40.62|75.54|65.84|76.70|80.30 |
|CLIP+MaPLe | ViT-L/14  |81.86|79.76|97.22|97.55|83.09|87.70|94.65|44.81|82.52|72.03|72.98|84.75 |
|CLIP+CDC | ViT-L/14 | 84.52|80.61|97.14|97.76|83.42|88.90|94.91|46.10|84.31|78.34|90.78|86.15|
|ALIP |ViT-B/32  |41.10 | 38.50 | 80.05 |41.76 |4.89 |60.36 |50.91 |4.05 |53.87 |28.60 |43.60 |43.94 |
|ALIP+MaPLe | ViT-B/32  |52.80 |45.84 |88.44 |60.64 |12.29 |74.70 |61.14 |7.10 |67.30 |36.27 |63.42 |56.43 |
|ALIP+CDC | ViT-B/32 | 54.17 |45.67 |90.06 |62.69 |12.44 |77.97 |62.31 |7.20 |67.91 |41.32 |65.99 |57.96 |


**Table 2**. Comparison of CDC on cross-dataset evaluation based on ViT-L/14.
| Method |Architecture| Caltech | Pets | Cars | Flowers | Food | Aircraft | SUN | DTD |SAT  |UCF  |Avg|
| --- | --- |--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|CLIP|ViT-L/14|95.20|93.50|76.80|79.50|90.90|32.70|67.70|53.00|60.30|75.10|72.47|
| CLIP+MaPLe| ViT-L/14 | 96.23  |92.57  |77.60  |76.90  |91.40  |30.07  |71.63  |54.23  |53.83  |75.90  |72.04  |
|CLIP+CDC| ViT-L/14|96.70 |93.10 |77.33 |75.87 |91.53 |31.67 |72.87 |57.63 |58.00 |76.20|73.09| 

[R1] Yang K, Deng J, An X, et al. Alip: Adaptive language-image pre-training with synthetic caption[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 2922-2931.

Meta-review: This paper effectively addresses the critical issues of task and data misalignment in adapting CLIP and similar models to downstream tasks. The proposed Causality-Guided Semantic Decoupling and Classification (CDC) method is well-founded in causal theory and demonstrates significant improvements in mitigating the interference of task-irrelevant information. Reviewers appreciated the thorough experimental validation, which consistently showed the effectiveness of CDC across various settings. However, some concerns were raised about the generalizability of the method beyond image classification tasks and across different vision-language models. The authors responded by providing additional experiments and clarifications, demonstrating CDC's broader applicability and robustness. They also addressed specific concerns regarding the explanation of concepts and the impact of the model’s increasing capabilities. Despite these minor concerns, the paper’s strengths, including its technical novelty, clear motivation, and strong empirical results, make it a valuable contribution to the field. Therefore, I recommend acceptance of this paper.
Decision: Accept (poster)