Divergence-enhanced Knowledge-guided Context Optimization for Visual-Language Prompt Tuning
Authors: Yilun Li
Miaomiao Cheng
Xu Han
Wei Song
Venue: ICLR 2025 Poster (2025)
Keywords: visual-language prompt tuning;few-shot learning;zero-shot learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Prompt tuning vision-language models like CLIP has shown great potential in learning transferable representations for various downstream tasks. The main issue is how to mitigate the over-fitting problem on downstream tasks with limited training samples. While knowledge-guided context optimization has been proposed by constructing consistency constraints to handle catastrophic forgetting in the pre-trained backbone, it also introduces a bias toward pre-training. 
This paper proposes a novel and simple Divergence-enhanced Knowledge-guided Prompt Tuning (DeKg) method to address this issue.
The key insight is that the bias toward pre-training can be alleviated by encouraging the independence between the learnable and the crafted prompt. Specifically, DeKg employs the Hilbert-Schmidt Independence Criterion (HSIC) to regularize the learnable prompts, thereby reducing their dependence on prior general knowledge, and enabling divergence induced by target knowledge. 
Comprehensive evaluations demonstrate that DeKg serves as a plug-and-play module can seamlessly integrate with existing knowledge-guided context optimization methods and achieves superior performance in three challenging benchmarks. We make our code available at https://github.com/cnunlp/DeKg.

Published as a conference paper at ICLR 2025
DIVERGENCE-ENHANCCE KNOWLEDGE-GUIDED CONTEXT OPTIMIZATION FOR VISUAL-LANGUAGE PROMPT TUNING
Yilun Li, Miaomiao Cheng", Xu Han, Wei Song
College of Information Engineering, Capital Normal University, Beijing, China {Yilun, miaomiao, hanxu, wsong) @enu. edu.cn
ABSTRACT
Prompt tuning vision-language models like CLIP has shown great potential learning transferable representations for various downstream tasks. The main is- sue is how to mitigate the over-fitting problem on downstream tasks with limited training samples. While knowledge-guides context optimization has been pro- posed by constructing consistency constraints to handle catastrophic forgetting in the pre-trained backbone, it also introduces bias toward pre- pre-training. This pa- per proposes a novel and simple Divergence-enhanee Tuning (DeKg) method to address this issue. The key insight is that the bias toward pre-training can be alleviated by encouraging the independence between the learnable and the crafted prompt. Specifically, DeKg employs the Hilbert- Schmidt Independence Criterion (HSIC) to regularize the learnable prompts, thereby reducing their dependence on prior general knowledge, and enabling di- vergence induced by target knowledge. Comprehensive evaluations demonstrate that DeKg serves as plug-and, and-play module that can seamlessly integrate with existing knowledge-guides context optimization methods and achieves superior performance in three challenging benchmarks. We make our code available https::/git com/ cwt..p//nnnng
Knowledge-guider Prompt
INTRODUCTION
Large-scale vision- laiguale models (VLMs) like CLIP (Radford et al. 2021) and ALIGN (Jia et al., (021) have demonstrated excellent capabilities in zero-shot recognition and generalization representation. Unfortunately, the large model sizes, high computational resource requirements, and massive trainable data restrict their deployment on real vision-language tasks. To address this problem, new paradigm of prompt tuning has been proposed and attracted increasing attention in recent years (Radford et al. 2021
Zhou et al. 20226).
Prompt tuning (Zhou al. 2022b) aims to optimize limited set of dynamic continuous prompt representations with the end-to-cns objective function, i.c., the cross-entropy loss, to transfer the pre-trained knowledge of VLMs to targeted tasks. These methods are less than optimal due to challenges in determining what should be preserved and what should be adapted for downstream tasks. For example, the base-to-new generalization task, as shown in Figure T CoOp (Zhou al., 2022b) can achieve a significant performance improvement over the manually prompted method CLIP (Radford et al.) 2021) on base (seen) classes, yet is inferior on new (unseen) classes in same dataset. This suggests that the prior general knowledge may be distorted by the limited task- specific labeled data, causing fine-tuned models to deviate from the pre-trained VLMs and leading to overfitting issues.
The overfitting issues can be attributed to the lack of regularization in the latent space to model the prior general knowledge for the unseen class distribution (Yao al., 2023). Since the frozen CLIP (Radford et al. (202I) coupled with crafted prompts exhibits robust abilities to unseen classes, indicating that the pre-trained backbone serves a valuable source of prior knowledge for each class, recent works (Yao et al.] 2023 Zhu et al. 2023a
Yao et al. 2024) all construct novel
Corresponding authors.

---

Published as a conference paper at ICLR 2025
constraint term by enforcing the consistency between the learnable and crafted prompts, called knowledge-guidde guided context optimization (KGCO). However, despite the benefits of regularization in preventing catastrophic forgetting, KGCO tends to be biased toward the pre- pre-trained model, pecially when the data distribution of the target task differs from that of the pre-trained data. For example, shown in Figure [ KgCoOp (Yao al. (023) improves CoOp new classes but de- grades on base classes, mainly due to the bias of the learnable prompts toward the representations of the pre-trained CLIP. Overall, an effective prompt tuning method should address the contradiction problem between catastrophic forgetting in fine-tuning and bias in pre- pre-training.
In this work, we propose a novel method, called Divergence-enbaneee
Knowledge-
KgCoOp
Delg
guided Prompt tuning (DeKg). We aim to maintain the advantage of knowledge-
250p
CLI
guided context optimization but alleviate the contradiction problem between catas-
trophic forgetting and bias towards general knowledge. Specifically, introduce novel constraint by employing the Hilbert- Schmidt Independence Criterion (HSIC) regularization (Gretton et al. 2005) to en- sure independence between learnable and crafted prompts. The proposed model en- courages the learnable prompts to maintain consistent yet independent relation with general knowledge, optimizing the balance between adapting general knowledge and fine-tuning for targeted tasks. As shown in Figure T DeKg overcomes the weakness of KgCoOp, performing best on both base classes and new classes.
(a) Base class performance.
(b) New class performance.
Performance comparison of DeKg with prompt tuning methods (CLIP/ CoOp, KgCoOp (base- line method), and DeKg (Ours)) under base-to-new generalization setting. We measure average accuracy on the base classes (a) and new classes (b) over 11 datasets. The red dotted line indicates the performance of CoOp for base classes and the zero-shot CLIP for new classes.
Figure 1:
Our contributions can be summarized as follows:
We tackle an inherent issue of knowledge-guing context optimization overly biasing general knowledge in pre-training, and propose novel HSIC-based regularization method DeKg for encouraging independence between the learnable and the crafted prompts.
DeKg integrates seamlessly with existing knowledge-guidee methods. Compared to the baselines, DeKg not only introduces divergence between the learnable and crafted prompts but also enhances differentiation between learnable prompts for distinct classes.
Extensive experiments demonstrate the superiority of the proposed method in three chal- lenging benchmarks: base-to-new generalization, cross-dataset generalization, and few- shot learning.
2 RELATED WORK
Vision-Language Models (VLMs) pre-trained on large-scale image-text association pairs through self-supervised methods have exhibited impressive performance in various visual tasks (Radford) et al. 2021 Jia et al. 2021). Despite the powerful generalization capacities, the enormous size of these models makes it challenging to fine-tune the entire models for downstream tasks, particularly when dealing with few-shot data. Such a trend raises the essential need to study different adaptation approaches, where prompting has been shown to be one of the simple and effective strategies.
Prompt Tuning for VLMs:: Prompting was initially proposed in the domain of Natural Language 1:1: Li Liang 2021), providing textual instructions to the task input for distilling task-relevant knowledge. For example, CLIP (Radford et al. 2021 utilizes a col- lection of crafted templates "a photo of [CLASS]" as textual inputs for category wise embeddings, and demonstrates exceptional zero-shot image recognition capabilities. However, building proper predefined prompt requires domain specific knowledge and enormous time. To circumvent this, a series of methods that automate learning embeddings the input tokens, known soft prompts,
Processing (NLP) (Lester et al. 2021;
2

---

Published as a conference paper at ICLR 2025
have emerged for fast adaptation to various downstream tasks. CoOp (houu al. 2022b) optimizes the prompt content by a continuous set of learnable vectors that are used as imput to the text encoder alongside the class name. However, the prompts learned by minimizing the classification error on a training set within the given base classes, resulting in weak generalization on new classes. Co- CoOp (Zhou et al. 2022a) further expands by constructing conditional prompts on specific image instances. However, such methods have a worse generalization than CLIP on the same task to the unseen classes. In addition to the textual prompt tuning, MaPLe (Khattak et al. 2023a) conducts the visual-textual prompt tuning by jointly conducting the prompt tuning on the visual and text encoders.
Knowledge-guided Prompt Tuning: To ensure that learnable prompts retain essential general tex- tual knowledge contained in frozen CLIP. ProGrad (Zhu et al., (202a)) and KgCoOp (Yao et al.) 2023 both constrain the consistency between the learnable prompt and the crafted prompt by em- ploying a novel constraint term. Specifically, ProGrad tries to optimize the learnable prompts with the aligned direction generated by the crafted prompts. KgCoOp adopts the Euclidean distance to minimize the discrepancy between textual embeddings generated by learned prompts and crafted prompts. PromptSRC (Khattak et al. 2023b) presents self-regulating approach to prompt learning, overcoming overfitting and improving generalization by leveraging mutual agreement, prompt self- ensembling, and textual diversity. Later, TCP (Yao et al.] (024) constructs an embedding module to inject the class-level textual knowledge into the learnable prompt tokens. While existing prompt learning techniques have boosted the generalization ability by applying consistency constraints on the textual input between learnable and crafted tokens, they exhibit limited capability capture specific knowledge. To mitigate this limitation, we propose novel textual prompting method that incorporates consistency and diversity enhance the generalization and discriminative capabilities of the learnable tokens.
3 METHODOLOOGY
Our method is built upon the framework of showledge-guiders context optimization (Yao et al. 2023), which enforces consistency constraint between the learnable and crafted prompts to dis- till knowledge from the frozen encoders, thus defying catastrophic forgetting. However, relying too much on pre-trained knowledge may hurt downstream knowledge and degrade performance. To mitigate this limitation, we propose a new method based on the Hilbert- Hilbert-Schmice Schmidt Independence Criterion (HSIC) regularization, to empower the capabilities of capturing task-specific information without forgetting task-agnostic general knowledge.
3.1 REVISITING KNOWLEDGE-GUIDER CONTEXT OPTIMIZATION
CLIP (Radford et al. 2021) is fundamental Vision-Language Model, offering a zero-shot transfer strategy by pre-training the visual backbone and textual encoder on 400M large-scale image-text pairs through contrastive learning. Benefiting its robust generalization capabilities, the frozen textual embeddings Welip = (w:lpp) N: of the crafted prompt "a photo of [class]" can be a valuable source of prior general knowledge where "[class]" replaced by one of the Ne class names. However, general knowledge is less able to accurately describe downstream tasks, mainly without considering the task-specific knowledge of each task.
i==1
To obtain discriminative target task knowledge, a sequence of learnable tokens t {t1, t2...., tt designed for generating task-specific textual embeddings of all classes, where M is the num- ber of tokens. The corresponding class token c; is concatenated with the learnable tokens, i.e., {t,, 12,,... , tar, ). for generating the encoded textual embedding w,. Then the learnable prompts (or contexts) W = N of all classes can be optimized by minimizing the contrastive loss between the given image's embedding and its class embedding Wy. which can be formulated
exp(sim(x, wy)/r)
Lee
Σ
N
(x.M)ED, IN, exp(sim(x, w:)/r)*
where D, denotes the seen dataset, N the number of training images, No is the number of classes, sim(.) represents the cosine similarity, and 7 refers to temperature parameter frozen in CLIP.
Following Yao et al. 2023). "general knowledge" in this work denotes the information contained in the pre-trained CL.IP model.
3

---

Published as a conference paper at ICLR 2025
Textual embeddings
Cress Pnetop
Vicual embedding
Crafted Frompt
%
photo of [class]
Consétency
dog
W2
Eug Incepandenss In Huli
map encoder
[class]
The -
Knowledge guided optemiration
The knowledge-guice context optimization framework of DeKg. Lee is the cross- ents-py loss, and Lkg is a consistency constraint. Lid is a regularization term that uses the Hilbert- Schmidt Independence Criterion (HSIC) to encourage the independence between learnable and crafted prompts.
Figure 2:
Despite delivering promising results, it can be observed that the learned prompt prone to overfit- ting to small training data and weakens the generalization capabilities to new classes (Zhou et al.] 2022a), mainly because the prompt is fixed once learned and only optimized for specific classes, i.c., catastrophic forgetting for pre-trained knowledge. To exploit the prior general knowledge contained in the frozen CLIP for learnable tokens optimization, a simple yet efficient consistency constraint is added during prompt tuning to prevent catastrophic forgetting (ao et al.] 2023). which can be expressed as
No No
Lkg
1
W
W
cli 2
The consistency constraint enforces that the learnable tokens have similar distributions as the crafted prompts, suggesting the potential bias toward pre- training.. The reason lies in that data distributions vary across different domains. Compared to the pre-trained VL.Ms, the training data of downstream tasks is extremely limited, resulting in the learnable prompts inevitably towards pre-trained knowl- edge distributions.
3.2 DIVERGENCE ENHANCED KNOWLEDGE DUUDDEEEGEGUDDED GUIDED CONTEXT OPTIMIZATION
Based the consistency constraint employed in the knowledge-guided context optimization (KGCO) methods (Yao et al. 2023 2024). the learnable tokens aim to preserve the task-agnostic general knowledge, enhancing the capabilities of new class prediction. However, the classifier generated by such textual tokens has poor task-specific discriminative ability to describe down- stream tasks. To alleviate the limitations of the KGCO methods, we propose a Divergence-enhanced Knowledge-
wwwledge-guided guided context optimization (DeKg) strategy by introducing an independence regular- ization into the KGCO methods, to adapt the pre-trained CLIP to the downstream tasks. As shown in Figure 2 DeKg designs the independence constraint to enhance the divergence between learn- able and crafted prompts, thereby strengthening task-specific knowledge to avoid bias towards pre- training and improving the discriminative ability of downstream tasks. Therefore, the independence constraint can act as a complement to the consistency constraint, enhancing the overall KGCO.
Given the textual embeddings embodied general knowledge Welip {w? clip1110 No Jim1 with N, classes, the independence constraint is proposed to enable learnable embeddings W = (w;),. N. to capture task-specific discriminative knowledge without interference from prior general knowledge. With the benefit of non-parametric, easy computability, rapid convergence, and small estimation bias with finite samples (Ma et al. 2020), the Seilbert-Schmitt Independence Criterion (HSIC) (Gretton et al., 2005) is adopted penalize the dependency between the learnable and crafted prompts. Specifically, it measures the degree dependency, with lower values indicating stronger independence and higher values suggesting greater correlation. The proposed constraint can be formulated as
Ltd HSIC(W, Wellp) (Ne 1) 1 (r)"">((KK""" (KHKel
(3)
4

---

Published as a conference paper at ICLR 2025
where K E RN.NN, K
RNN,N. with entries K, k( wy) and K.
k(w,"? olpp wwf""), k(.;-) matrix, which is used to remove the bias within each representation and focus on the inter-variable relationships: tr(-) represents the trace of the matrix.
is a kernel function; H IN. N, Iv. 1
INNNNNN is the centering
Define A HK H, then 3 can be rewritten as follows
Lid =
(N, - 1) (r-(k(
(4)
(Ne - (N)- EK,AA
6,
Notice that the elements Aig indicate the inter-class relationships among the crafted prompts Welp and are fixed, since they solely relate to the representations of the crafted prompts. Consequently, Led is influenced only by {Ki which describes the relationships within the of learnable prompts W. Therefore, we can identify two advantages of Lka-
First, the computation only related to W without introducing any extra parameters. In our imple- mentation, we use the inner product kernel function, i.e., Ki,, = w. w,,,, and promising performance achieved.
Second, leveraging HSIC regularization, Lkd fosters the independence between the learnable and crafted prompts by enhancing the divergence among the learnable prompts across various classes, which is influenced by the inter-class relationships inherent in the crafted prompts. So Led helps capture task-specific unique knowledge, to enhance model's discriminative capability.
Remind that the consistency constraint Lig aims to maximize the agreement between learnable and crafted prompts, enhance the model's ' generalization capability. Therefore, Led and Lkg focus on optimizing different aspects and complement each other.
As a result, we constrain the learnable prompts with both consistency and independence. The final objective function can be expressed
C = Lee AL.kg + j.Lkd:
(5)
where À and are trade-off hyperparameters encoding the belief degrees for consistency and inde- pendence constraints, respectively.
EXPERIMENTS
In this section, we conduct extensive experiments on three widely-used benchmarks to evaluate the ability of base-to-new generalization, cross-data generalization, and few-shot learning, and demon- strate the effectiveness of the proposed method by comparing with strong vision-language prompt tuning baselines.
4.1 EXPERIMENTAL SETUP
Datasets:
For downstream tasks, we follow previous work (Radford et al. 2021 Zhou et al.
2022a b), to conduct experiments on 11 representative image classification datasets, including Ima-
geNet (Deng al. 2009) and Caltech (Fei-Feei al., 2004) for generic object classification; Oxford-
Pets (Parkhi et al.] 2012). StanfordCars (Krause et al., 2013 Flowers (Nilsback & Zisserman, 2008),
Food101 (Bossard al. 2014). and FGVCAirerafi (Maji et al. 2013) for fine- ene-grained visual cate- gorization, AuroSAT (Helber et al. (211)) for satellite image classification, UCF101 (Soomro et al.)
2012 for action recognition, DTD Cimpoi al. 2014) for texture classification, and SUN397 (Xiao) al. 2010 for scene recognition.
Baselines: First, to demonstrate that DeKg can embody the advantage of preserving both the general knowledge frozen in CLIP and task-specific knowledge, we compare the results of CLIP (Radford) et al. 2021), CoOp (Zhou et al. 2022b). CoCoOp (Zhou et al. 2022a), and MaPLe (Khattak et al. 2023a), which exploits only general task-specific knowledge, i.c., only uses cross-entropy for prediction. Second, to show the significant advantage enhancing task-specific knowledge, we compared with two baselines, KgCoOp (Yao et al. 2023) and ProGrad (Zhu al. 2023a), which
5

---

Published as a conference paper at ICLR 2025
preserve the general knowledge by enforcing the consistency between the learnable tokens and crafted prompts. Besides, to highlight the importance of divergence guided by general knowledge and task-specific knowledge, compared with PromptSRC (Khattak et al. 2023b) and TCP (Yao) et al. 2024), which incorporate other strategies to consistency constraint, i.e., PromptSRC adds self-ensembling and textual diversity regularization, while TCP inserts class-specific knowledge into embeddings.
For the DeKg method which unifies the general knowledge preservation and divergence upon general-specific knowledge into one framework, four baselines, i.e., KgCoOp, ProGrad, TCP, and PromptSRC, can be expanded by adding the regularzztii to produce the divergence by target knowledge with general knowledge preservation. In our experiments, only KgCoOp and TCP are adopted and expanded to generate the final learnable tokens, denoted as DeKggy and DeKgTeP respectively. The main reason for this is that, on one hand, ProGrad aligns prompts with general knowledge of the gradient, while the others are directly aligned with the embeddings. On the other hand, PromptSRC includes visual prompts and textual prompts, while other baselines only include textual prompts.
Training Details: Our implementation is based on KgCoOp's Yao et al. 2023) and TCP's (Ya jet al. 2024) codes. To ensure a fair comparison, all experiments were conducted using the ViT- B/16 Dosovitskiy et al. (02I) the vision backbone and the context length set as 4. Additionally, we maintained consistency with the corresponding baselines in DeKg and DeKgreP for ran- dom prompt initialization, training epoch, training schedule, and data augmentation settings. In our experiments, we set the ratio of A/μ to 3/1 by grid search, which translates to A being 6 and being 2. All experiments were carried out using the HYGON DCU-ZI00L server.
4.2 PERFORMANCE COMPARISON AND ANALYSIS
4.2.1 BASE-TO-NEW GENERALIZATION
The base-to-new generalization setting aims to evaluate whether the models learned base tasks can generalize to new tasks without unseen classes, i.e., a category shift exists between base and new tasks. Following the baselines, on each dataset, we first construct base and new task by equally dividing the dataset into two groups, then perform prompt tuning on the base classes and test the learned model on both the base and new tasks. Table Table[] presents the performance of different methods across 11 datasets with 16-shot samples, where the best and second results are marked in bold and underlined, respectively. For convenience, we refer to the classification accuracy of base tasks and new tasks as base accuracy and new accuracy, respectively. The harmonic mean (H) of base accuracy and new accuracy is also computed to demonstrate the generalization trade-off.
Compared with zero-shot CLIP, the baselines optimized with only cross-entropy loss, i.e., CoOp, CoCoOp, and MaPLe, achieve improvement on base classes but show inferior performance on new classes except MaPLe. This suggests that they overall tend to overfit the task-specific data distribu- tions, losing the original generalization capability of the frozen CLIP model towards new tasks. Al- though KgCoOp alleviates the poor generalization problem in CoOp by preserving the prior general knowledge, it hardly outperforms CoOp in base accuracy in almost all benchmarks, i.e., KgCoOp has an average drop from 82. 22.64% 80. 70.73% compared with CoOp, while ProGrad has similar trend. This suggests that the learnable prompts may be skewed towards the general knowledge frozen in the CLIP, due the limited task-specific knowledge. In contrast, DeKg improves on both base and new classes over CLIP and CoOp. Specifically, DeKgrer obtains an average gain of 2.32% (i.e., 84.96% vs. 82.64%) over CoOp in base accuracy, and 2.16% (i.e., 74.22% vs. 76.38%) over CLIP in new accuracy, respectively. Additionally, DeKgxgcoop has a similar trend. This shows the benefits of DeKg by optimizing context explicit guidance by general and target knowledge, which aids base and new classes respectively.
PrompiSRC and TCP are two strong competitors because they both leverage task-specific knowledge and general knowledge together improve generalization. Fortunately, DeKgTeP demonstrates improved performance in recognizing both base and new classes. Specifically, DeKgicr outperforms PromptSRC 8 out of 11 datasets in terms of base accuracy and almost half of the datasets in new accuracy. Additionally, DeKgTcP shows improvement over TCP in almost all 11 datasets. The main reason is that PromptSRC and TCP guide the prompt with the token alignment strategy, limited by

---

Published as a conference paper at ICLR 2025
Table Comparison with existing methods in the base-to-new generalization setting with ViT-B/16 as the backbone. The context length M is 4 for prompt-based methods with the 16-shot samples from the base classes. H: Harmonic mean.
Datasets
CLIP
CoOp
CoCoUi
MaPLe
KgCoUp
Protami
PrompiSRC
TCP consistencc class specific
Dekgiganp
Dekgrre
Regulariizaion::
consistency
consistency and divensity
connistency and independence
Basc
69.34
82.64
80.47
$2.25
30.73
52.48
84.26
84.13
$2.59
84.96
Average
New
74.22
68.00
71.69
75.14
73.60
70.75
76.10
75.36
74.93
76.38
71.70
78.55
77.00
79.97
79.51
78.57
Base
72.45
76.46
75.9%
76.06
75,83
77.02
77.60
77.27
76.65
27.40
ImageNet
New
68.14
70:54
69.96
66.66
70.73
69.66
69.20
70.22
71.02
73.10
73.47
72.78
71.46
74.01
73.38
72.99
73.07
Base
96.84
98.11
97-96
97.74
97.72
98.02
98. 10
98.23
98.13
98.64
Caltech
H
95.40
95.76
95.84
96.02
96.03
95.91
96.02
96.42
95.59
96.89
Base
91.17
99.24
95.20
95.43
911.55
95.07
95.33
91.67
95.00
911,17
97.76
97.71
H
94.12
95.43
96.43
96.58
96.18
96.33
96.30
95.92
96.34
96.08
Base
63.37
76.20
70.49
71.14
71,76
77.68
78.27
MIL80
76.31
81,18
New
74.89
73.59
75.04
68.63
74.97
74.13
75.27
74.75
H
68.65
72:50
72.01
73.47
73.36
72.88
76.58
77.32
75.79
77.83
72.08
98.07
Flowers
New
77,80
69.55
71.75
72.46
74.73
71.87
76.50
75.57
74.04
75.18
H
74.83
81.23
81.71
82.56
$3.65
82.03
83.95
85.23
84.25
85,30
90.10
Food
New
91.22
$7.50
91.29
92.05
91.70
89.59
91.53
91.37
91.95
91.55
H
90.66
88.46
90.99
91.38
91.09
89.98
91.10
90.97
91.25
91.14
Aireraft
New
36.29
30.49
23.71
35.61
33.55
27.57
37,87
34.43
34,97
35.09
H
31.09
34,32
27.74
36.50
34.83
32,82
40.15
37.83
36.91
39,51
69.36
81.19
SUN397
New
75.35
68.34
76.86
78.70
766.53
74.17
78847
78.20
76.57
78.30
H
72.25
74.07
78.27
79.75
78.36
77.55
80.52
80.35
78.81
$0.35
Basc
53.24
30.17
77.01
80336
77.35
77.35
83.37
82.77
80.90
DTD
New
59.90
47.54
56.00
59.18
54,99
52.35
62.97
58.07
58.21
59.66
56.37
68.16
64.35
62.45
71.75
68.25
67.70
69,70
Base
56.48
91.34
87.49
94.07
83.64
90.11
92.30
91.63
$8.29
94,02
ET
New
64.05
$4.44
60.04
73.23
64.34
60.89
73.90
74.73
72,69
$1.69
79.73
Base
70.35
85.14
W2.33
X3000
$2.99
84.33
87.10
87.13
84.04
88.06
UCF101
New
77.50
64.47
73.45
78.66
76.67
74.94
78.80
80.77
78.04
81,77
80.77
handling domain shift in the test set. This demonstrates that DeKgrcr gains advantages by taking into account the textual embedding distribution with an independence constraint.
4.2.2 CROSS-DATASET GENERALIZATION
To further demonstrate that the proposed model can bridge the distribution gap between the pre- training dataset and the downstream evaluation set for zero-shot generalization, we compare DeKg with baselines under the cross-dataset generalization. In this experiment, we follow the baselines to regard ImageNet as the source dataset and the other 10 datasets as target datasets, i.c., there is distribution shift between the hase and new tasks.
From the comparison results in Table 2 we can see that our DeKgrcr obtains the highest average performance among all baselines (66.64% vs. 29% of TCP). By comparison, the performance on other datasets with distant and more fine-grained or specialized categories is much lower, such as Aircraft where the accuracy number well below 30%. Nonetheless, DeK gTCP exhibits much stronger transferability than TCP with an average gain of 1.60% (i.e., 25.05% vs. 23.45%) on Aircraft, as well as on most other fine- graiind or specialized datasets. Additionally, DeKgkg achieves inferior performance to PromptSRC and TCP, mainly due to the inability to explicitly model the downstream class distribution.
4.2.3 FEW-SHOT CLASSIFICATION
To verify the model's ability to develop robust representations with a severely limited amount of downstream data, we follow the previous work (Yao et al.) 2024) to train the model using K-shot labeled source images from each class and evaluate the testing domain with the same spaces as the training classes. A summary comparison of the 4-shot setting between the proposed DeKg and ex- isting baselines appears Table3] from which we can observe that: the proposed DeKgrep achieves the best average performance than all baselines. In addition, the baselines KgCoOp and TCP have

---

Published as a conference paper at ICLR 2025
Table Comparison in the cross-dataset generalization, The model is trained on the entire class of ImageNet (16 shots) and evaluated on the other 10 datasets.
CoOp
CoCoOp
MaPLe
ProCiad
PrompiSRC:
KgCoOp
DeKgngcopp
Dek &TCP
ImageNNt
66.70
71.51
71.02
70.72
72.24
71.27
70.66
71.40
71,34
72.33
Caltech101
93,30
93.70
94.43
93,53
91.52
93.60
93.92
93.97
93.87
94.73
Pets
89.10
89. 14
90.14
90.49
89.64
90.25
89.83
91.25
90.16
90.02
Cars
65.70
64.51
65.32
65.57
62.39
65.70
65.41
64.69
65.91
65.49
Flowers
70.70
68.71
71.88
72.20
67.87
70.25
70.01
71.21
70.60
72.39
Food101
85,90
85.30
86.06
86.20
85,40
86.15
86.36
86.69
86.37
86.59
Aireraft
18.47
24.74
20.16
23,90
22.51
23.45
23.37
25.05
SUN397
62.60
64.15
67.36
67.01
62.47
67.10
66.16
67.15
66.11
67.19
DTD
44,30
41.92
45.73
46.49
39.42
46.87
46.35
44.35
46.16
44.47
EuroSAT
48,30
46.39
45.37
48.06
43.46
45.50
46.04
$1.45
43.15
51.37
UCF101
67.60
66.55
68.21
68.69
64.29
68.75
68.50
68.73
68.17
68.78
Avg.
65.24
63.88
65.74
66.30
62.71
65.81
65.51
66.29
65,33
66.64
Table 3: Comparison of few-shot learning with s-shot samples.
Datasets
CLIP
CoOp
CoCoOp
MaPLe
FroGrad
PrompiSRC
KgCoOp
TCP
DeKgkgoocy
DeKg
ImageNet
66.70
69.37
70.53
70.67
70.21
70.80
70.19
70.48
70.24
70.19
Caltech101
93.30
94.44
94.98
94.30
94.93
94.77
94.65
95.00
94.97
95.21
Pets
89.10
91.30
93.01
92.05
93.21
93.23
93.20
91.90
93.10
92.15
72.73
71.75
71.98
72.24
Flowers
70.70
91.14
82.56
80.80
89.98
91.31
90.69
94.40
90.50
95.21
Food101
85,90
82.58
86.64
86.90
85.77
86.06
86.59
85,30
86.88
85.72
FGVC
24.90
33.18
30.87
29.03
32.93
32.80
32.47
36,20
32.88
37.02
SUN397
62.60
70.13
70.50
71.47
71.17
72.80
71.79
72.11
72.33
72.85
DTD
44.30
58.57
54,79
54,73
57.72
611.44
58.31
63.97
61.05
64.24
EurSAT
48.30
68.62
63.83
54.87
70.84
75.02
71.06
77.43
72.65
79.16
UCFI01
67.60
77.41
74.99
73.70
77.82
79.35
78.40
80.83
79.43
81.05
Avg.
65.37
73.59
71.98
70.66
74.21
75.33
74.48
76.72
75.12
77.06
shown respective improvements with independence constraint (i.e., DeKgksCoop and DeKgTcP) of the average gains of 0.64% (i.e., 75. 12% vs. 74.48%) and 0.44% (i.e., 77.06% vs. 76.12%) across 11 datasets. This demonstrates that optimizing the learnable prompts with independence and con- sistency constraints together is indeed beneficial.
Next, we will conduct more detailed investigations for DeKg. If there no special statement, all reported results represent average performance across over 11 datasets.
4.2.4 ABLATION STUDY AND ANALYSIS
To investigate the learning process of DeKg, we conduct ablative analysis including follows:
Effeet of Constraints Employed DeKg: DeKg contains two key constraints, including the consistency constraint Lkg and the independence constraint Lkd- We conduct constraint- analysis by adding one or two of them to the baseline method CoOp. Table 4 shows the results. We that the baseline CoOp provides high base class performance but suffers from poor generalization. By incorporating Ltg alone, the performance of new classes increases significantly by 5.62% (i.e.. 73.61% vs. 67.99 67.99%), but the base classes degrades from 82.63% to 90%73% This suggests that Lxn explicitly enforces the intra-class consistency between the learnable and crafted prompts capture generalizable features from frozen CLIP. In contrast, incorporating Led alone leads improvements in both base and new classes compared with CoOp, indicating its ability in balancing model adaptation and generalization. achieves the best performance in base classes but still lags behind KgCoOp in new classes. The main reason lies in that Lid aims to capture the task- specific knowledge by enhancing the divergence among learnable prompts across various classes with prior inter-class relationships inherent the crafted prompts. Finally, combining Led and Lkgs DeKg achieves improvements harmonic mean gains of 6.94% (i.e., 74.93% vs. 67 (7.99%) and 3.97% (i.e., 78.57% 74.60%). It reasonable because Lkg and Led are complementary. one for intra-class consistency and the other for inter-class divergence.
both base and new classes, leading to the average new class and
Comparison of Model Complexity: To better understand the benefits of the proposed DeKg, we examined the model complexity. As shown in Table 5) it can be observed that DeKg is efficient
8

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of model complexity with
Table 4: Effect of the constraints in our model.
KgCoOp.
Method
Base
New
Method
Total Parameters (M)
H
CoOp
82.63
67.99
74.60
KgCoOp
124.325
77.00
+Lkkg
80.73
73.61 77.00
DeKgkgCoOp
+0
78.57
+Lud
83.13
69.87
75.93
TCP
+0.329
79.51
+ +kg + Lkd (ours) 82.59
74.93
78.57
DeKgTeP
+0.329
80.44
method that performs better with the same model complexity of corresponding baselines. For exam- ple, DeKgggCoOp shows an average improvement of 1.57% (i.e., 78.57% vs. 77.00%) over KgCoOp without adding any parameters. The main reason is that DeKg simply adds efficient regulariza- tion for generating discriminative classifiers guided by better knowledge, i.e., optimizing the balance between adapting general knowledge and fine-tuning for targeted tasks.
Effect of Hyperparameter À and μr: To further investigate the impact consistency and inde- pendence constraints model performance, the proposed model DeKg, i.e., À controls the contribution of capturing general knowledge, and controls the divergence between task-specific knowledge and general knowledge. The effect of and j on DeKg with KgCoOp and TCP (i.e., DeKgkgCoOp and DeKgTcP) is shown in Figure 3a and Figure 3b respectively. It can be seen that the performance of new tasks becomes better A/μ increases, indicating that the consistency constraint effectively captures the essential knowledge for new classes. The results reach the best when A/1 / 3/1 for both DeKgkgCoop and DeKgTeP. After that, the performance decreases because a larger ratio forces the learnable prompts to rely strongly on general knowledge, failing to capture task-specific information. The trend for base precision is reversed. This result is reasonable because larger ratio of A/μ reduces the importance of task- specific knowledge, which essential for base tasks.
analyze the effect of hyperparameter À and j in
1234
02
86
S0
33-
7620
Accuracy
Accuracy
82-
2279
12.00
72
1-1
211
18
255
A/p
(a) De DeKgggO
(b) DeKgrer
Figure 3: Effect of hyperparameters À and μ DeKgkgCoop and DeKgTeP.
12
12
CoOp
KgCoOp
DeKg
Figure 4: Visualization the HSIC values between W and Welip in DTD dataset.
Visualization: As shown in Figure 4 the HSIC values obtained from the method KgCoOp very high. This indicates that the learnable tokens are highly correlated with

---

Published as a conference paper at ICLR 2025
the pre-trained general knowledge, which can lead poor performance on target tasks. In contrast, the HSIC values derived without wwvledge-guided context optimization method CoOp are very low. This suggests weak reliance general knowledge and tendency to overfit the target task, resulting in limited generalization ability for target tasks. 'hhe values obtained by DeKg are moderate compared to the baselines. This suggests that the proposed HSIC regularization term Led effectively maintains balance between dependence on general knowledge and task-specific knowledge.
CONCLUSION
Knowledge-guice context optimization is a representative visual-language prompt tuning frame- work. emphasizes the consistency between the learnable and crafted prompts to alleviate catas- trophic forgetting, which boosts the generalization ability but degrades the few-shot learning ability downstream tasks. In this paper, we propose a DeKg strategy by introducing an independence con- straint, which exploits Hilbert-Schmidt Independence Criterion regularization to enhance the diver- gence between learnable and crafted prompt for capturing tusk-specific unique knowledge, thereby enhancing the model's discriminative capability. Extensive evaluations on three challenging bench- marks demonstrate that DeKg is an effective and efficient prompt tuning method. Specifically, one of the main advantages of DeKg is its ability to seamlessly integrate with existing wewwwllde-guidded context optimization methods, such as KgCoOp and TCP, significantly enhancing their performance without requiring additional parameters. Another key advantage of DeKg its capability to outper- form strong baselines on both base classes and new classes in base-to-new settings, while existing methods often struggle keep balance between the generalization ability and few-shot learning ability. Additionally, it demonstrates superior performance in cross-datase generation and few-shot learning scenarios. the future, we plan to incorporate DeKg to more lisuaa-lagguage prompt tuning frameworks and applications.
ACKNOWLEDGEMENTS
This work is supported by grants from the National Natural Science Foundation of China (Grant Nos. 62306188, 62376166) and the R&DD Program the Beijing Municipal Education Commission IKM2202400228113) The Center for Experimental Teaching of Computer Science & Technology at Capital Normal University provides computational support.
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 101-mining discriminative compo- nents with random forests. In Proceedings of the 13th European Conference Computer Vision, pp. 446-661. Springer, 2014.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattemn Recognition, pp.
2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 245-255. IEEE, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations, 2021. URL https: //tp:////eeereeeeewee net. fo potorumid=Yiccbfaa
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the Conference on Computer Vision and Pattern Recognition workshop, pp. 778-178. IEEE, 2004.
10

---

Published as a conference paper at ICLR 2025
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmidt norms. In International Conference on Algorithmic Learning the- ory, pp. 63-77. Springer, 2005.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning. pp. 4904-4916. PMLR, 2021.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15190-1 11990-15200 15200. IEEE, 2023b.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graine -r--grrnned categorization. In Proceedings of the IEEE International Conference on Computer Vision work- shops, pp. 351-561. IEEE, 2013.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cexsing, pp. 3045-3059 ACI., 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of Annual Meeting of the Association for Computational Linguistics and Interna- tional Joint Conference on Natural Language Processing, pp. 4582-4597, 2021.
Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5085-5092, 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.553,, 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Sixth Indiam Conference on Computer Vision, Gruphics Image Processing, pp. 722-729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Pro- ceedings of the IEEE Conference on Computer Vision and Patter Recognition, pp. 3498-3505.
IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 9748-8763. PMLR, 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: /211./22222002 2012.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 3492-3492. IEEE, 2010.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 6757-6767. IEEE, 2023.

---

Published as a conference paper at ICLR 2025
Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: cextual-based alass-aware prompt tuning for visual-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14438-23448.... IEEE, 2024.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 168 16816-16825 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2348, 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023a.
Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, and Hanwang Zhang. Debiased fine-tuning for vision- language models by prompt regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3834 3842, 2023b.
12


Reviews:
[{"review_id": "2vdNLkiB2J", "reviewer": "Reviewer_u22y", "summary": "This paper introduces a simple yet effective knowledge-based prompt tuning method that leverages the Hilbert-Schmidt Independence Criterion (HSIC) to regularize learnable prompts. By reducing the reliance on prior general knowledge, this approach enables the prompts to better align with task-specific knowledge. The method is versatile and can be easily integrated into other frameworks. When applied to the TCP method, it demonstrates superior performance across most datasets.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and easy to follow. Figure 2 effectively illustrates the main idea by clarifying the roles of each loss function: the \\( L_{CE} \\) loss enforces alignment between text and vision embeddings, the \\( L_{kg} \\) loss encourages the learnable prompts to align closely with the CLIP textual embeddings, and the core \\( L_{HSIC} \\) loss ensures independence within the learnable prompt embeddings.\n\n2. The experiments are comprehensive, covering base-to-new generalization, cross-dataset generalization, and few-shot classification. The proposed DeKgTCP method achieves superior results across most datasets.", "weaknesses": "1. Why was the proposed method applied to KgCoOp and TCP rather than other state-of-the-art methods, such as PromptSRC, which performs even better than KgCoOp? Is it more challenging to integrate with PromptSRC, or are the results less effective? Providing additional clarification on this choice would enhance the paper.\n\n2. Figure 4 provides an insight into how the proposed method balances dependence and independence; however, the paper lacks further analysis on this. Expanding on this point would strengthen the reader’s understanding of the method's underlying mechanics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nbidEg04lc", "reviewer": "Reviewer_Km4y", "summary": "This paper tackles the inherent issue of knowledge-guided context optimization, which overly biases general knowledge in pre-training. It proposes a novel HISC-based regularization method, DeKg, for encouraging independence between the learnable and the crafted prompts. Extensive experiments demonstrate the superiority of the proposed method in three challenging benchmarks:", "questions": "Please see #Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+Using the Hilbert-Schmidt Independence Criterion (HSIC) is an interesting topic for encouraging independence between learnable and crafted prompts, which can boost performance in the seen classes.\n\n+Evaluation shows the effectiveness of the proposed method.\n\n+The proposed DeKg integrates seamlessly with existing knowledge-guided methods.", "weaknesses": "-As shown in Figure 1, the proposed DeKg obtains a higher performance than the performance of CoOp for base classes and the zero-shot CLIP for new classes. However, the Hilbert-Schmidt Independence Criterion (HSIC) contained in DeKg is a constraint between the learnable and crafted prompts without injecting additional information. Why can the proposed DeKg obtain a better performance?\n\n-L221: The proposed L_{kd} involves two terms: intra-class relations and inter-class relations. Moreover, the author claims that  penalizing\nL_{kd} encourages both intra-class and inter-class independence. Furthermore, the intra-class consistency is formulated between w_i and w_{i}^{clip}, which is the same as the L_{kg}. In other words, the proposed HSIC has contained the knowledge consistency L_{kg}. Therefore, the final objective of Eq.(5) should not contain L_{kg} because L_{kd} has been constrained by the intra-class consistency. However, the results in Table 4 are inconclusive with the above conclusion. Even more unfortunate, L_{kd} performs worse than L_{kg}. Why?\n\n-It is recommended to provide a code.\n\n-Since the proposed HSIC is model-independent, it is suggested that the module's generalization and plug-and-play be verified using more CoOp-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M0zq62OPaP", "reviewer": "Reviewer_z32d", "summary": "This paper proposes a novel method called Divergence-enhanced Knowledge-guided Prompt Tuning (DeKg), which employs Hilbert-Schmidt Independence Criterion (HSIC) regularization to maintain a degree of independence between the learnable prompts and pre-trained knowledge, addressing the bias problem caused by over-reliance on pre-trained knowledge. Built upon knowledge-guided context optimization, DeKg introduces an independence constraint, enabling learnable prompts to retain consistency with general knowledge while capturing task-specific features, thus achieving a better balance between base and novel classes.", "questions": "1. As mentioned in weakness1, more analysis of your motivation concerning why you choose Hilbert-Schmidt Independence Criterion would be insightful.\n2. The experiment of domain generalization seems to be missed in your paper, which is conducted by most prompt tuning method. Could you provide Dekg’s performance on this setting.\n3. In Tabel 4, it is obvious that the $L_{kg}$ works well in novel, while $L_{kd}$ performs well in base. Dose that mean these two losses are strongly coupled and your proposed $L_{kd}$ is not recommended to use independently? More analysis on above question would be insightful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses the inherent bias issue in knowledge-guided context optimization by introducing a novel Hilbert-Schmidt Independence Criterion (HSIC)-based regularization that encourages independence between learnable and crafted prompts. \n2. DeKg integrates with existing methods, enhancing class-specific prompt distinction without increasing model complexity.", "weaknesses": "1. The motivation of using HSIC as the constrain is not clearly elaborated. Further analysis of your motivation will be insightful. \n2. One of the proposed loss: $L_{kg}$ is already applied in existing methods, such as KgCoOp and PromptSRC, which weakens the novelty of overall method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sej0ycuDfg", "reviewer": "Reviewer_nZxi", "summary": "This paper proposes to adapt HSIC as an extra regularization term, which achieves a better trade-off between the performance on base and new classes. The experiments show the effectiveness of this regularization on varies experiment settings. The paper is well written and easy to understand.", "questions": "Please refer to the concern in the Weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1. I think this paper has a reasonable motivation to maximize the independence between learnable prompt and manual prompt. \n2. This paper has a very extensive experiment analysis on varies clip model adaptation task and show good results.", "weaknesses": "1. More analysis is needed to discuss why HISC is chosen as the metrics to measure the prompt independence. Other methods like information bottleneck can do that too. \n2. L_kd and L_kg seems to be a pair of totally contradictive losses. I wonder if this will cause the model to be difficult to converge. It would be better to provide more analysis on how the loss weight of these two losses affect the model convergence. \n3. More performance comparison and analysis on other state-of-the-art prompt tuning method like:\nYubin, et.al, Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)