Enhancing Target-unspecific Tasks through a Features Matrix
Authors: Fangming Cui
Yonggang Zhang
Xuan Wang
Xinmei Tian
Jun Yu
Venue: ICML 2025 poster (2025)
Keywords: Prompt learning
Primary area: general_machine_learning->supervised_learning
TL;DR: Enhancing Target-unspecific Tasks through a Features Matrix

Abstract: Recent developments in prompt learning of large Vision-Language Models (VLMs) have significantly improved performance in target-specific tasks. However, these prompting methods often struggle to tackle the target-unspecific or generalizable tasks effectively. 
It may be attributed to the fact that overfitting training causes the model to forget its general knowledge. The general knowledge has a strong promotion on target-unspecific tasks. To alleviate this issue, we propose a novel Features Matrix (FM) approach designed to enhance these models on target-unspecific tasks. Our method extracts and leverages general knowledge, shaping a Features Matrix (FM). Specifically, the FM captures the semantics of diverse inputs from a deep and fine perspective, preserving essential general knowledge, which mitigates the risk of overfitting. Representative evaluations demonstrate that: 1) the FM is compatible with existing frameworks as a generic and flexible module, and 2)  the FM significantly showcases its effectiveness in enhancing target-unspecific tasks (base-to-novel generalization, domain generalization, and cross-dataset generalization), achieving state-of-the-art performance.

Enhancing Target-unspecific Tasks through a Features Matrix
Fangming Cui
Yonggang Zhang?
Xuan Wang?
Xinmei Tian Jun Yu
Abstract
CLIP utilizes a large collection of 400 million text-image pairs. One of CLIP's most appealing features is its ability to perform zero-shot inference. During inference, CLIP utilizes hand-crafted text inputs, known as prompts, to gen- erate classification weights to predict image features, all without requiring any target-specific parameters training.
Recent developments in prompt learning of large Vision-Language Models (VLMs) have signifi- cantly improved performance in target-specific tasks. However, these prompting methods often struggle to tackle the target-unspeciin izable tasks effectively. It may be attributed to the fact that overfitting training causes the model to forget its general knowledge. The general knowl- edge has strong promotion on target-unspecific tasks. To alleviate this issue, propose a novel Features Matrix (FM) approach designed to en- hance these models on target-unspecific tasks. Our method extracts and leverages general knowl- edge, shaping a Features Matrix (FM). Specifi- cally, the FM captures the semanties of diverse inputs from deep and fine perspective. preserv- ing essential general knowledge, which mitigates the risk of overfitting. Representative evaluations demonstrate that: 1) the FM is compatible with existing frameworks as a generic and flexible mod- ule, and 2) the FM significantly showcases its ef- fectiveness enhancing target-unspecific tasks (base-to-novel generalization, domain generaliza- tion, and cross-dataset generalization), achieving state-of-the-ar performance.
or general-
In contrast cand-cratted prompts, a model-parameter tun- ing method has been proposed as prompt learning au- tomatically learn prompt embeddings (Zhou et al., 2022a). For instance, CoOp (Zhou et al., 2022b) represents the first method that specifically focuses on learning the text embed- dings of prompts with few-shot samples training while keep- ing the CLIP model frozen. Although CoOp (Zhou et al., 2022b) has shown significant performance improvements over hand-crafted prompts in target-specific base classes, it may yield inferior performance compared to the hand- crafted prompt CLIP in generalization tasks, e.g., the novel class of generalization from base-to novel (see Table 1). To overcome this challenge, some effective methods (Lu et al., 2022; Zhu et al., 2023; Zhou et al., 2022a;b; Yao et al., 2023a; Chen et al.; Li et al., 2024a) with tuning textual embeddings have been proposed. These methods aim to en- hance the performance of novel classes, surpassing the novel performance of the previous CoOp. Although the novel abil- ity of these methods surpasses that of CoOp (67.96%) it is unfortunate that the novel ability of these methods still falls short compared to the hand-crafted prompt method (CLIP: 74.22%), as shown in Table I (Novel). A possible reason (Zhou et al., 2022a) may be that these methods with tuning text embeddings tend to overfit the downstream data distributions. This overfitting can result in the model losing its inherent generalization capabilities (Yao et al., 2023a) obtained from hand-crafie prompts (Radford et al., 2021).
1. Introduction
Large vision-language models such as CLIP (Radford et al., 1) have attracted increasing attention for remarkable gen- eralization performance. Lisgua-Language Models (VLMs) trained to align textual and visual modalities by leverag- ing extensive datasets. For instance, prominent example of such models is CLIP, which has achieved remarkable success across wide range of downstream tasks (Greer et al., 2024; Sanghi et al., 2022; Etchegaray et al., 2024).
To alleviate this problem, we propose a novel Features Ma- trix (FM) with CLIP for target-unspecifi tasks. Our method incorporates multiple hand-crafted prompts extract gen- eral information as a pre-trained matrix from frozen CLIP to enhance generalization. The matrix of pre-trained fea- tures can delve into the semantics of different hand-crafted prompts finely and deeply, which reduces the risk of forget- ting the essential general knowledge of pre-trained CLIP. Importantly, our method is compatible with current prompt learning frameworks for textual or multi-modal prompt learning and serves as a flexible and generic module. As
Shanghai Jiao Tong University 'Honn Kong Baptist University Meituan Inc. Uniiversity of Science and Technology of China 3 HHrbii Institute of Technology (Shenzhen). Correspondence to: Jun Yu <yujun @ www.hht..ucc... edu.en.>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enhancing Target-unspecific Tasks through a Features Matrix
Domain Generalization
Hand-Crafted CLIP. During the pre-training phase of CLIP (Radford et al., 2021), the image and text encoders are jointly trained on large-scale text-image pairs using contrastive loss. This loss maximizes the cosine similar- ity between matching pairs and minimizes it between non- matching pairs, enabling the encoders to learn aligned visual and textual representations effectively. The final prediction score between the image x and text prompt p is computed using contrastive learning. The final prediction probability of alignment is computed by the matching score follows:
Base-to-novel Generalization
dross-dataset Generalization
exp fcos (E, &, (x))
p(1 = h xx)
K EK== exp {cos (E, Ex, /T}
M+DePT
Mi+ Ours
S+DePT
S+Ours
(1)
where I is the label x,, cos (+. stands for cosine similarity between two vectors, and r > 0 represents a temperature parameter. Here, the classifier consists of K textual features derived from prompts {Px 1 where the prompt P for the V'' -th class may have the form of "a photo of a".
Figure 1. Our method is orthogonal to representative architectures, such as MaPLe (M) and PromptSRC (S), surpassing the existing casy-to-use DePT (Zhang et al., 2024a) by a significant margin.
demonstrated in Figure I, our method, integrated with the representative MaPLe (Khattak al., 2023a) and Prompt- SRC (Khattak al., 2023b), exhibits competitive results in base-to-novel generalization, domain generalization, and cross-dataset generalization across 1l datasets. Meanwhile, our method surpasses the existing casy-to-use DePT (Zhang et al., 2024a) by a significant margin. This underscores the robustness of our method across various tasks.
Textual Prompting. In textual prompting, the class name is retained as prior knowledge, while the word embeddings (referred to context) of prompts parameters, as shown in CoOp (Zhou et al., 2022b) and Co- CoOp (Zhou et al., 2022a) of Figure 2. By modeling these context embeddings as trainable parameters, the model can optimize the prompts based on the specific requirements, enhancing the alignment between images and prompts. For class k, the tuning feature of the text encoder is denoted as fun in dataset with a total of K classes. The final pre- diction probability of cross-entropy loss for two-modalities alignment is computed as:
treated as learnable
Our contributions can be summarized as follows:
The proposed method incorporates multiple hand- crafted prompts with classes to extract general knowl- edge as a pre-trained Features Matrix (FM).
exp {cos (1tun Ex)x) /T]
p(1 = k Σx- exp {cos (Itun En (x)) /T}
(2)
We propose an easy-to-use design, which is compati- ble with representative textual or multi-modal prompt learning frameworks for adapting CLIP.
where cos (---) stands for cosine similarity, and T denotes a temperature parameter. In some learned text embedding methods, the method aligns with the CoOp methodology by setting the embeddings to be shared across different classes. During the inference phase, the prompts, along with the learned embeddings, can generate textual features that are used for classification purposes. By leveraging these learned embeddings, the model can produce more effective representations, leading to improved target-specific classification performance (Zhou et al., 2022b).
Various unspeciis tasks (base-to mosse-to-nove general- ization, cross-dataset generalization, and domain gen- eralization) across 11 datasets demonstrate that our method demonstrates its effectiveness.
2. Preliminaries
Considering pre-trained VLM, let (.).) its image encoder and (-) be its text encoder. The image encoder transforms input images into feature embeddings, capturing the visual information within the images. The text encoder generates representations for word embedding sequences, capturing the semantic information conveyed by the text prompts p. Generally, a hand-crafted prompt p may have the form of "a photo of a [Class]". In this paper, x represents an arbitrary image, and denotes the label.
Notations.
Multi-Modaa Prompting. Based on textual deep prompting, this design uses only a limited number of trainable parame- ters based on the image encoder, shown in MaPLe (Khat- tak et al., 2023a) and PromptSRC (Khattak et al., 2023b) of Figure 2. The visual prompts are introduced at every trans- former layer's s input space. In the visual branch, the input image x E xCx is divided into M patches. And, a tuning embedding of class ielass is appended with the input
2

---

Enhancing Target-unspecifie Tasks through Features Matrix
PromptSRC
Similarity
CoOp
Simiing--iilaitty
Tuning-Simin Similarity
Text Encoder
Dcep
Image
Txx Encoder
Image Encoder
Deep
Encoder
Class priors
T
Class priors
MaPLe
Tuning- Similarity
CoCoOp
Tuning-Simin Similarity
Image
Text Encoder
eep
Deep
Encoder
Text Encoder
Image Encoder
V
Class priors
T
Class priors
T
Figure 2. Illustration of representative textual prompting frameworks (CoOp and CoCoOp) and multi-modal prompting frameworks (MaPLe and PromptSRC). We propuse a flexible and generic design, which is compatible with these representative architectures. In the figure, "snowflake pattern" represents parameter freezing. "flame pattern" represents learnable pattern, "Deep" represents learnable tokens emhedded in several layers of the encoder, where "T"" represents learnable text embedding, "V" represents learnable visual embedding, "Class priors" represents which eategory it helongs to, "Tuning Similarity" represents the calculation of cosine similarity for fine-tuning architecture, and the light gray "Simillrrtt" represents the calculation of cosine similarity for frozen architecture. In the MaPLe architecture diagram, the "c" represents the matrix function that connects the encoders of two modalities several layers.
patches. The foundation of vision prompting is ViT (Vision Transformer) (Dosovitskiy et al., 2020) backbone, which shares the same image encoder as CLIP. Let P. denote the visual embeddings of prompts, and the image encoder pro- cesses the input tokens to generate tuning visual features, as follows (Jia et al., 2022):
abilities to mitigate performance degradation when dealing with target-unspecific classes
Table I. Our method is easy-to-use design, integrated into CoOp, obtaining a higher average performance (11 datasets) on novel (target-unspecific) classes.
x (Pu, idasss é, 82. im}
(3)
Method
Hand Features
Base
Novel
The tuning of visual embeddings are introduced in the image encoder for deep prompt tuning.
CLIP (ICML2021)
Single
69.34
74.22
CoCoOp (CVPR2022)
No
80.47
71.69
ProDA (CVPR2022)
No
81.56
72.30
3. Methodology
PLOT (ICLR2023)
No
81.24
72.98
ProGrad (ICCV2023)
No.
82.48
70.75
3.1. Current Challenge
CoOp (ICC2002)
No
82.38
67,96
+ DePT (CVPR2024)
No
83.66
71.82
As shown in Table 1, such textual prompting (Zhou et al., 2022a;b; Yao et al., 2023a; Lu et al., 2022) and multi-modal prompting (Chen et al.), have shown significant improve- ments in target-specific base classes. However, these meth- ods often suffer from overfitting, leading to poor general- ization in novel classes. To improve the novel performance of models, prompting regularization involves constraining the pre-traine tre-triiee and fine-tuning features in the text branch through computed loss. This constraint helps prevent forget- ting general knowledge and ensures that the model retains essential information for downstream tasks. The objective of this design is to activate and maintain the pre-trained gen- eral knowledge, leveraging its remarkable generalization
+ Kg (CVPR2023)
Single
80.73
72.70
+ Ours
Matrix
81.15 74.66
It is important to recognize that the fixed CLIP's fundamen- tal pre-trained characteristies demonstrate robust generaliza- tion capabilities. Recently, KgCoOp (Yao et al., 2023a) in- corporates the constraint with L1 loss fine-tuning feature with the pre-trained feature for the textual branch, avoiding the loss of CLIP's original generalization capability.
Motivation. However, the novel class performance of hand- brase KgCoOp (72.70%) still lower than pre- trained CLIP (74.22% (Radford et al., 2021) of Table
3

---

Enhancing Target-unspecific Tasks through Features Matrix
Scores Matrix
A easy- to-use design
Features Matrix
P1
P2
P3
P4
Classnamel
Designated
Pre-Trained
Classname2
Non- Designated
CL
Classname3
Tuning
Unexpected
Pre-Trained
Tuning
Tuning
Tuning
Classnamel Classname2 Classname3
CE
P1 "a photo of a
Text Encoder
8
Image
P2 "a picture of a"
Encoder
P3 "a drawing of a"
Deep
Deep
P4 "a photo of one"
T
T
T
T
V
V
Figure 3. Illustration our casy-to-use method. We propose a novel Features Matrix (FM) for enhancing target- ungeet-nnppecific tasks. Our method incorporates multiple hand-crafted prompts with classes extract general knowledge pre-trained features matrix. Various generalization tasks across 11 datasets demonstrate that our method outperforms existing prompt learning methods.
may be attributed to the fact that this single hand-crafted prompting regularization with CLIP cannot fully release and excavate the diverse semantics for utilizing essential general knowledge (Khattak al., 2023b).
through set of hand-crafted prompt templates (P1, P2, P3, P4, etc) with classes. In a dataset with a total of K classes, features belonging to the current label & of image samples are referred as designated features 4, while features from other classes are considered non-designated features tg.
3.2. Proposed Features Matrix (FM)
Our method computes the matching scores with the sine distance for each visual output feature pnun and des- ignated pre-trained text features of the features matrix as cos (lx, vfw)). Similarly, we compute the matching scores between each visual output feature and the non-designated pre-trained text features, denoted as cos (tg, tan). ). In corre- spondence with the features matrix, these matching scores can be viewed as scores matrix, depicted in Figure 3. Subsequently,
To tackle this challenge and problem, we propose a novel design called Features Matrix (FM) for enhancing target- unspecific tasks. Our method incorporates multiple hand- crafted prompts with classes of the same datasets to extract general information as pre-trained features matrix from CLIP to enhance generalization. In this way, we can explore and excavate the semantic information brought by each hand-crafted prompt more finely and deeply. By specifically aligning these multiple pre-trained hand-crafted unexpected features and tuning image features, our method aims to enhance the model's generalization and robustness. This involves focusing indistinguishable features that pose challenge to the model during training. As shown in Figure 3, we employ a set of hand-crafted prompt templates (P1, P2, P3, P4, etc) as inputs in the text encoder of pre- trained frozen CLIP. These prompt templates, such as photo of one", "a photo of ", "a picture of a"" "a drawing of a", etc. The number of prompt templates (Radford al., 2021) is set to 60. In dataset containing K classes, the input to the text encoder is the frozen embeddings of hand- crafted prompts, consisting of different classes. The output of the frozen text encoder can be viewed as features matrix, as depicted Figure? 3. Specifically, we let denote the sets of pre-trained text features extracted by a frozen text encoder
identify the textual feature with lower scores (low-5) among the designated features, which is re- ferred to as the unexpected designated features set an The low-3 is that there is a list scores sorted from low to high, with values assigned from the front the list. Simi- larly, we identify textual features with higher scores (top-3) among the non-designated features, which are referred to unexpected non non-designated feature sets Fk Accordingly, the contrastive loss LCL. of unexpected features similarity denoted as follows,
exp {cos (tx, odur))
- -log exp {cos (t,, otun)] } + exp {cos (ti, utun)
(4)
where the t4 E Fk, and & E FE denote the selected un- expected features for designated unexpected features and designaedd unexpected features. The objective is to explore and excavate the unexpected features of general

---

Enhancing Target-unspecific Tasks through a Features Matrix
information. We train the objective for aligning the un- expected text features and tuning visual features, and the contrastive loss LaL optimizes visual embeddings. For two- modal alignment, let (c)(+.)) denote the cross cross-entropy loss with prompts p for samples S, as follows,
epochs for the domain generalization setting and the cross- dataset evaluation setting in the first 3 transformer layers. We use SGD optimizer with a learning rate of 0.0025 on single GPU. Following the hand-crafted CLIP, we use the hand-crafted template sets from (Radford et al., 2021).
Datasets. In Table 10, the datasets cover multiple recogni- tion tasks including ImageNet (Deng et al., 2009) and Cal- tech101 (Fei-Fei et al., 2004) which consists of generic ob- jects, OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), and FGVCAircraft (Maji et al., 2013) for fine-grained classification, SUN397 (Xiao et al., 2010) for scene recognition, UCF101 (Soomro et al., 2012) for action recognition, DTD (Cimpoi et al., 2014) for texture classification, and EuroSAT (Helber et al., 2019) which consists of satellite images. We leverage datasets such as ImageNetA (Hendrycks et al., 2021b), ImageNet- R (Hendryeks et al., 2021a), ImageNet-Sketch (Wang et al., 2019), and ImageNetV2 (Recht et al., 2019) to assess the model's performance across different domain distributions.
CCE arg min &f {cos (t ( 11. P
(5)
Accordingly, the objective Lioaal with of our method can be formulated as follows,
y
Livial = LCR + LCa.-
(6)
Consequently, optimizing models with La. between the pre-trained textual unexpected features and image tuning features. The LCE represents two modalities of alignment of deep vision-language prompting. Importantly, our method is compatible with representative prompt learning frameworks as generic and flexible module for textual prompt learning, such as CoOp and CoCoOp, or multi-modal prompt learning, such MaPLe and PromptSRC.
4.2. Base-to-Novel Generalization Task
4. Main Generalization Tasks
In the base-to-novel generalization task, the datasets are divided into base and novel classes. The model is trained on the base classes in a 16-shot setting, and tested on both the base and novel classes across 11 different datasets. The number of classes for base and novel is the same, which means that all classes in a dataset are evenly divided into two groups of classes. The process of dividing all classes in the dataset is randomly selected. HM refers to harmonic mean. The HM evaluates the generalizable and non-generalizable ability of our methods.
We evaluate our method on generalization tasks. As evi- denced by systematic benchmarking in Figure 1, our frame- work, when synergistically integrated with MaPLe (Khat- tak et al., 2023a) and PromptSRC (Khattak al., 2023b), achieves state-of-the-art performance across three critical generalization dimensions (base-to-novel generalization, domain generalization, and cross-dataset generalization) spanning 11 heterogeneous benchmarks. Notably, our method surpasses the existing easy-to-use DePT (Zhang et al., 2024a) by a significant margin in generalization sce- narios through low-resource training.
Discussion. In Table 2, our method demonstrates signif- icant improvements all 11 datasets on HM. We found that our performance is higher than DePT (Zhang al., 2024a) (CVPR2024) when integrated with different meth- ods. Compared to DePT, our method significantly enhances the performance of base classes while also improving the accuracy on novel classes. In terms of average performance, our method based on PromptSRC (S) attains 85. 85.70% accu- racy on the base classes, and 77.35% accuracy on the novel classes. Our method, as an easy-to-use module, achieves performance improvements for various baseline frameworks, such as CoOp, CoCoOp, MaPLe (M), and PromptSRC (S).
4.1. Benchmark Setting
Compared Methods. In three generalization experiments, we compare our method with CoOp (UCV2022) (Zhou al., 2022b), CoCoOp (CVPR2022) (Zhou et al., 2022a), MaPLe (CVPR2023) (Khattak al., 2023a), Prompt- SRC (ICCV2023) (Khattak et al., 2023b), and DePT (CVPR2024) (Zhang et al., 2024a). The DePT is an easy-to- use method. We process frozen CL.IP using a linear probe method. Comparison methods are based on the VIT-B/16 architecture for fair comparison.
4.3. Domain Generalization Task
Implementation Details. We employ CLIP (Radford al., 2021) model based on the VIT-B/16 architecture. For the PromptSRC-based and MaPLe-based frameworks, we set the visual and textual embedding length to 4. We set the easy-to-use module y to 0.1 and matching scores (top and low) 3 to 5. Training for 30 epochs for a base-to-novel setting in the first transformer layers. Training for 20
We train our model using the ImageNet in 16 shots and we leverage ImageNetA, ImageNet-R, ImageNet-Sketch, and ImageNetV2 to assess the model's performance. The training ImageNet is a non-generalizable setting in Table (column-1) 1). This experiment aims to validate the potential of our method in domain shifts (Fang et al., 2023).
5

---

Enhancing Target-unspecific Tasks through a Features Matrix
Table 2. Base-to-novel generalization. In the base-to-nove generalization task, the datasets are divided into base and novel classes. The model is trained on the base classes in 16-shot setting, and lested on both the base and novel classes across 11 different datasets. Compared with easy-to-use DePT (CVPR2024) in detail, our method achieves consistent average performance improvement over different representative baselines (CoOp, CoCoOp, MaPLe, PromptSRC).
(a) Average over 11 datasets
(b) ImageNet
(c) Caltech101
(d) OxfordPets
Base
Novel|
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CoOp
82.69
63.22
71.66
CoOp
76.47
67.88
71.92
CoOp
98.00
89.81
93.73
CoOp +DePT 94.70
93.67
95.29
94.47
+DePT
83,66
71.82
77.29
+DePT
77.13
70.10
73.45
+DePT 98.33
94.33
96.29
97.63
96. 14
Ours
81.15
74.66
77.79
+ Ours
75.85
71.33
73.53
+ Ours
97.58
96.60
97.13
+ Ours
93.78
97.80
95.78
Co
80.47
71.69
75.83
Co
75.98
70.43
73.10
Co
97.96
93.81
95.84
Co
95.20
97.69
96.43
+DePT
83.80
72.89
77.97
+DePT
76.87
70.47
73.53
+DePT 98.37
93.87
96.06
+DePT 94.03
97.20
95.59
Ours
81.68
75.55
78.52
Ours
77.35
72.36
74.79
+ Ours
98.61
96.75
97.70
+ Ours
95.33
98.15
96.76
M
82.28
75.14
78.55
M
76.66
70.54
73.47
M
97.74
94.36
96.02
M
95.43
97.76
96.58
+DePT
84.85
74.82
79.52
+DePT
77.87
70.23
73.85
+DePT
98.53
95.03
96.75
+DePT 95.03
97.83
96.41
Ours
84.45
76.53
80.32
+ Ours
78.18
71.38
74.62
+ Ours
98.35
96.11
97.22
+ Ours
95.85
98.22
97.04
S
84.26
76. 10
79.97
S
77.60
70.73
74.01
S
98. 10 94.03
96.02
S
95.33
97.30
96.30
DePT
85.19
76.17
80.43
+ DePT
78.20
70.27
74.02
+ DePT
98.57
94.10
96.28
+ DePT
95.43
97.33
96.37
Ours
85.70
77.35
81.32
Ours
78.90
71.58
75.07
+ Ours
98.62
95.88
97.27
+ Ours
95.95
97.92
96.95
(e) EuroSAT
(f) UCF101
(g) StanfordCars
(h) Flowers
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CoOp
92.19
54.74
68.69
CoOp
84.69
56.05
67.46
CoOp
78. 12
60.40
68. 13
CoOp
97.60
59.67
74.06
DePPT
88.27
66.27
75.70
+ DePT
85.43
72.17
78.24
+ DePT
79.67
72.40
75.86
+ DePT
98.20
72.00
83.08
Ours
88.35
65.33
13
Ours
83.10
78.85
80.94
+ Ours
74.32
76.87
75.58
+ Ours
96.22
72.32
82.57
Co
87.49
60.04
71.21
Co
82.33
73.45
77.64
Co
70.49
73.59
72.01
Co
94.87
71.75
81.71
+DePT
90.27
66.87
76.82
+ DePT
85.70
72.80
78.73
+ DePT
79.87
73.33
76.46
+ DePT
98.33
72.57
83.51
+ Ours
88.15
70.11
78.12
+ Ours
82.95
77.21
80.00
+ Ours
72.88
76.10
74.46
+ Ours
95.61
74.93
84.03
M
94.07
73.23
82.35
M
83.00
78.66
80.77
M
72.94
74.00
73.47
M
95.92
72.46
82.56
+DePT
94.43
76.23
84.36
+ DePT
86.87
78.10
82.25
+ DePT
80.93
71.73
76.06
+ DePT
98.03
73.17
83.79
Ours
94.22
75.65
83.93
Ours
87.33
79.10
83.02
+ Ours
78.66
75.13
76.86
+ Ours
98.21
75.00
85.07
92.90
73.90
82.32
S
87.10
78.80
82.74
S
78.27
74.97
76.58
S
98.07
76.50
85.95
+ DePT
92.23
77.90
84.88
+ DePT
87.73
77.70
82.46
+ DcPT
80.80
75.00
77.79
+ DcPT
98.40
77.10
86.46
Ours
95.50
76.85
85.17
+ Ours
88.80
79.50
83,93
+ Ours
80.91
76.51
78.68
+ Ours
98.81 78.10
87.26
(i) Food101
(i) FGVCAircratt
(k) SUN397
()) DTD
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CoOp
88.33 82.26
19
CoOp
40.44
22.30
28.75
CoOp
80.60
65.89
72.51
CoOp
79.44
41.18
54.24
DePT
90.43
91.33
90.88
+ DePT
42.53
22.53
29.46
+ DePT
82.37
75.07
78.55
+ DePT
83.20
56.
67.04
+ Ours
89.98
92.85
91.41
+ Ours
37.32
34.61
35.92
+ Ours
79.12
78.38
78.77
+ Ours
77.10
56.38
65.15
Co
90.70
91.29
90.99
Co
33.41
23.71
27.74
Co
79.74
76.86
78.27
Co
77.01
56.00
64.85
DePT
90.30
91.30
90.80
+ DePT
43.07
31.30
36.25
+ DePT
82.20
76.73
79.37
+ DePT
82.77
55,40
66.37
Ours
90.61
91.93
91.28
Ours
37.87
34.91
36.33
+ Ours
80.32
79.00
79.68
+ Ours
78.90
59.61
67.93
M
90.71
92.05
191.38
M
37.44
35.61
36.50
M
80.82
78.70
79.75
M
80.36
59.18
68.16
+ DePT
90.33
91.53
90.93
+ DcPT
44.53
32.80
37.78
+ DcPT
82.90
76.40
79.52
+ DcPT
83.87
59.93
69.91
Ours
90.31
92.81
91.57
+ Ours
42.46
37.62
39.89
+ Ours
82.35
79.81
81.07
+ Ours
83.01
60.98
70.32
S
90.67
91.53
91.10
S
42.73
37.87
40.15
S
82.67
78.47
80.52
S
83.37
62.97
71.75
DePT
90.87
91.57
91.22
+ DePT 45.70
36.73
40.73
+ DePT 83.27
78.97
81.06
+ DePT
84.80
61.20
71.09
Ours
90.61
92.30
91.45
+ Ours
45.81
39.11
42.20
+ Ours
83.90
80.51
82.20
+ Ours
84.90
62.58
72.07
6

---

Enhancing Target-unspecifie Tasks through a Features Matrix
5.2. Embeddings Length
Table Domain generalization. We train our model using the Ima- geNet in 16 shots and test its performance on 4 different variants of the ImageNet. When compared to DePT, our method consistently outperforms it across all ImageNet variant datasets.
In Table 7, our findings indicate that the performance based PromptSRC reaches its peak when the length of embed- dings is set to on HM for average of 11 datasets. Our ablation studies are based on keeping all other settings un- changed. It indicates excessive fine-tuning of the model, causing it to lose CLIP generalization.
Source
Target
ImageNet
-V2
-S
-A
-R
Average
CoOp DePT
71.51
64.2
47.99
49.71
75.21
59.28
72.63
64.80
48.05
50.00
75.50
59.58
5.3. Analysis of y for LCL.
Ours
71.82
65.13
48.10
50.15
76.13
59.87
In Table 9, it is observed that as the number of 7y increases to 0.1, there is a peak in HM for an average of 11 datasets. We conduct analysis based on the PromptSRC model.
CoCoOp
71.02
64.07
48.75
50.63
76.18
59.91
DePT
72.77
65.10
49.10
51.00
76.85
60.51
+ Ours
72.10
65.25
50.13
52.11
77.18
61.16
MaPLe
70.72
64.07
49.15
50.9
76.98
60.27
+ DePT
73.27
65.33
49.05
51.25
77.50
60.78
5.4. Analysis of Top-3 and Low-3
+ Ours
71,56
65.45
50.33
52.32
78.10
61.55
In Table 8, it is observed that as the number of Top-3 and Low-3 increases to 5, there is a peak in HM. We conduct analysis based on the PromptSRC model for an average of 11 datasets. We keep other hyper-parameters unchanged.
PromptSRC
71.27
64.35
49.55
50,90
77.80
60.65
+ DePT
71.60
64.51
50.15
51.88
77.18
60.93
Ours
71.41
65.50
51.33
52.00
78.85
61.92
5.5. Applying to Other VIT Instances
Discussion. In Table 3, our method based on PromptSRC demonstrates the highest average improvement of 1.27% over the PromptSRC method. Additionally, when compared to DePT, our method consistently outperforms it across all ImageNet variant datasets.
In Table 6, we apply our method in various types of ViT instances. Our experimental performance is based on the average performance of 11 datasets. Our generic and flexible module based on PromptSRC has the lowest performance of ViT-B/32, with a value of 79.70%. And, our method, which leverages PromptSRC, achieves the best performance with ViT-L/14, with value of 84.20%.
4.4. Cross-Dataset Generalization Task
We train our model with 16 shots on the ImageNet dataset and test the model on 10 other unseen datasets. The training ImageNet is a non-generalizable setting in Table 4 (column- 1). This experiment aims to validate the potential of our method in a wide range of cross-dataset transfers.
5.6. Inference Stage Computational Cost
In Table 11, the compute cost analysis is performed using the SUN397 dataset over 10 epochs on a single GPU, Our method (row 3) may have slower inference speed as a result of multiple cosine similarity calculations. The num- ber of parameters consistent and stable in training. This implies that usually, our parameter quantity is lower than that of DePT.
Discussion. As shown in Table 4, our method based PromptSRC shows competitive performance in 10/ 10 over the generic and flexible method DePT. These findings sug- gest that our method excels in achieving better generaliza- tion across a diverse range of unseen datasets. However, our method for training ImageNet is lower than that of DePT.
introduced has remained relatively
6. Related Work
Vision-Language Models (VLMs). Recently, researchers have demonstrated strong generalization capability of Vision-Language Models (VLMs) (Alayrac et al., 2022; Shao et al., 2025), which involve training on large-scale datasets of image-text pairs. Using a large number of samples for training is indeed one of the most effective methods (Yang et al., 2021; Wang et al., 2023). Such as CLIP (Radford et al., 2021), which is a prominent and straightforward framework among existing VLMs. The strong generalization capability of CLIP has made it foun- dation for many methods in adapting pre-trained VLMs for downstream tasks (Sanghi et al., 2022; Maaz et al., 2022;
5. Further Studies
In this section, we discuss further studies, focusing on the impact of embedding length and depth, computational cost, applying to other ViT instances, analysis of 7 for La and analysis of Top-3 3 and Low-3 for pre-trained features matrix.
5.1. Learning Depth
In Table 5, we note that increasing the learning depth gener- ally increases the performance based on PromptSRC. As the number of layers increases to 11, the HM value decreases.

---

Enhancing Target-unspecifie Tasks through Features Matrix
Table Cross-dataset generalization. Our method achieves overall favorable performance.
Source
Target
ImageNet
Caltech101
OxfordPets
StanfordCars
Flowers 102
Food101
Aircraft
SUN397
EuroSAT
UCF101
Average
DTD
CoOp (Zhou al., 2022h)
71.51
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
DePT
72.63
93.30
90.00
65.53
70.50
85.97
21.90
66.07
43.17
44.97
68.80
65.02
+ Ours
71.82
94.10
90.33
65.82
70.01
86.10
20.71
65.11
43.98
46.55
68.00
65.07
CoCoOp (Zhou et al., 2022a)
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
+ DePT
72.77
94.10
90.63
66.23
72.17
86.27
22.90
67.30
45.50
44.17
69.53
65.88
Ours
72.10
94.88
90.57
65,80
72.15
87.00
22.78
68.12
45.98
46.15
69.10
66.25
MaPLe (Khattak et al., 2023a)
70.72
93.53
90.49
65.57
72.23
86.20
24.74
67.01
46.49
48.06
68.69
66.30
DePT
73.27
92.53
90.10
64.60
70.10
85.57
23.63
66.40
45.03
40.13
67.53
64.56
Ours
71.56
94.00
90.91
65.92
73.10
86.78
24.33
68.35
46.13
49.01
68.81
66.73
PromptSRC (Khattak al., 2023b)
71.27
93.60
90.25
65.70
70.25
86.15
23.90
67.10
46.87
45.50
68.75 75
65.81
DePT
71.60
93.80
90.13
66.00
70.93
86.27
24.30
67.23
46.60
45.83
69.10
66.02
Ours
71.41
94.95
92.14
66.50
73.25
87.53
25.81
68.10
49.01
49.20
69.74
67.62
Table 5. Analysis of learning depth based on PromptSRC for average datasets. HM refers harmonic mean.
Table 7. Analysis of embeddings length based on PromptSRC for an average of datasets. HM refers harmonic mean.
Learning Depth
3
5
7
11
Prompt Length
2
8
10
HM
77.11 78.03 79.12
80.01
81.32 80.21
HM
77.01 80.05 81.32 80.30
79.11
78.54
Table 6. Applying to other VIT instances.
Table 8. Identifying scores of Top-3 and Low-B 3 for average of 11 datasets. We conduct analysis based on the PromptSRC (Khat- tak al., 2023b) model. HM refers to harmonic mean.
Avg. (11 datasets)
B/32
B/16
L/14
Avg. (11 datasets)
HM (ViT)
79.70
81.32
84.20
4
5
6
HM (3) 78.21 79.81 81.32
80.51
Bangalath et al., 2022; Zhang et al., 2021; Wang et al., 2022; Xu et al., 2025). To enhance the generalization ability of VLMs, researchers have explored various approaches (Zhou al., 2025b;a). Some approaches involve enhancing the text encoder or the visual encoder (Vaswani et al., 2017). By improving the capabilities of the text encoder, the model can better capture the semantics and contextual information in the textual input (Zhang et al., 2024b). Similarly, enhancing the visual encoder allows the model to extract discriminative features from the visual input.
Table 9, Analysis o for LCL in pre-trained features matrix. We conduct analysis based on the PromptSRC (Khattak al.. 2023b) model an average of datasets. HM refers to harmonic mean.
Avg. (11 datasets)
0.05
0.5
0.9
HM (?)
80.13
81.32
81.00
79.88
Prompt Tuning in VLMs. Prompt tuning (Zhang & Tian, 2025; Pan et al., 2024; Cao et al., 2025; Zhou et al., 2024: Li et al., 2024a) is commonly employed technique in the field of Natural Language Processing (NLP) for training on down- stream tasks (Yu et al., 2025; 2024b;a). Leveraging text prompts, which are instructions given to the language model component of VLMs, is a prevalent practice to improve
task comprehension (Cui al., 2025b). Full fine-tuning and linear probes two commonly employed approaches for adapting VLMs to downstream tasks (Fang et al., 2025; Liu et al., 2025). The constraints of both methods have prompted research into innovative techniques influenced by prompt tuning within the realm of VLMs (Yin et al., 2025). CoOp (Zhou et al., 2022b), ProDA (Lu et al., 2022), and
8

---

Enhancing Target-unspecific Tasks through a Features Matrix
Table 10. Training and testing datasets. The dataset consists of 11 image classification datasets and four variant datasets of ImageNet.
Dataset
Classes
Train
Val
Test
ImageNet (Deng et al., 2009)
1,000
1.28 M
N/A
50,000
Caltech101 (Fei-Fei et al., 2004)
100
4,128
1,649
2,465
OxfordPets (Parkhi et al., 2012)
37
2,944
736
3,669
StanfordCars (Krause et al., 2013)
196
6,509
1,635
8,041
Flowers 102 (Nilsback & Zisserman, 2008)
102
4,093
1,633
2,463
Food101 (Bossard et al., 2014)
101
50,500
20,200
30,300
FGVCAineraft (Maji et al.. 2013)
100
3,334
3,333
3.333
SUN397 (Xiao et al., 2010)
397
15,880
3,970
19,850
DTD (Cimpoi et al., 2014)
47
2,820
1,128
1.692
EuroSAT (Helber et al., 2019)
10
13,500
5,400
8,100
UCFI01 (Soomro et al., 2012)
101
7,639
1,898
3,783
V22 (Recht et al., 2019)
1,000
N/A
N/A
10,000
Skkttch (Wang et al., 2019)
1,000
N/A
N/A 50,889
-A (Hendrycks et al., 2021b)
200
N/A
N/A
7,500
-R (Hendrycks et al., 2021a)
200
N/A
N/A 30,000
8. Conclusion
Table 11. The compute cost analysis is performed using the SUN397 (Xiao al., 2010) dataset over 10 epochs single GPU. 'N': the num of classes in the base task (Zhang et al., 2024a).
Prompt learning is promising method for adapting pre- trained Visual-Language Models (VLMs) for target-specific classification tasks. However, these optimization frame- works exhibit limited efficacy when applied target- unspecific generalizable scenarios. This performance degradation may stem from werfittingiinuucee catastrophic forgetting, where the model loses generalizable knowledge critical for adapting to unseen tasks during the fine-tuning process. In this paper, we propose a Features Matrix (FM) for vision-language models. Our method incorporates mul- tiple hand-crafted prompts extract general information as pre-trained features matrix from CL.IP to enhance gener- alization. The matrix of pre-trained features can delve into the semantics of different hand-crafted prompts profound level, which reduces the risk of forgetting the es- sential general knowledge of pre-trained CLIP. Our method focuses on the challenge of various target-unspecific tasks and scenarios across wide range of real datasets. Impor- tantly, our method is compatible with representative prompt learning frameworks for textual multi multi-modal prompts as generic and flexible module.
Method Train time Learnable para.
HM
CoOp
10.88min
8K
71.65
+ DePT
10.91min
(2+N/2)K
77.30
Ours
13.56min
OK
78.66
CoCoOp (Zhou et al., 2022a) fine-tune the CLIP model for few-shot image recognition by optimizing a continuous set of embeddings within the textual branch. The image- conditional prompt utilized in CoCoOp significantly con- tributes to improving generalization to unseen classes (Peng et al., 2025; 2023). By conditioning prompts on visual features, CoCoOp (Zhou et al., 2022a) ensures that the lan- guage model focuses on pertinent visual information when making predictions. Moreover, some approaches (Yao et al., 2023b;a; Cui et al., 2024; 2025a) constrain the learnable prompts to contain the essential general knowledge. In addi- tion, the approach (Zhang al., 2024a) introduces flexible approach to align its vision and language representations.
more
Acknowledgement
7. Future Works and Limitations
XM T was supported in part by the NSFC No. 622221 17. Y was supported in part by the NSFC No. 62125201 and U24B20174.
In the future, we plan to investigate the potential of our method in other tasks (Tian al., 2024; Yang et al., 2023: Li et al., 2019; Yu et al., 2023; Li et al., 2024c; Zhang et al., 2022; Wang et al., 2021; Li et al., 2025a;e; Jia al., 2025; Liu et al., 2023b;a; Li et al., 2024b) and downstream scenarios (Pan et al., 2025; Jin et al., 2025; Li et al., 2025b; Liu et al., 2024; Wu et al., 2024; Liu et al., 2023c;d). Due to multiple processes of our method, the training speed is slower. And, the ImageNet training in Table 4 lower.
Impact Statement
The societal implications of our work involve democratizing the availability of potent AI resources, as our plug-and-play method can achieve impressive performance even in the absence of extensive labeled data.

---

Enhancing Target-unspecifie Tasks through a Features Matrix
References
Etchegaray, D., Huang, Z., Harada, T., and Luo, Y. Find n' propagate: Open-vocabulary 3d object detection in urban environments. arXiv preprint 1Xiv:2403.133566 2024.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I,, Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23773 2022.
Fang, Z., Lu, J., and Zhang, G. An extremely simple algo- rithm for source domain reconstruction. IEEE Transac- tions on Cybernetics,
2023.
Bangalath, H., Maaz, M., Khattak, M. U., Khan, S. H., and Shahbaz Khan, Bridging the gap between object and image-level representations for open-vocabulary detec- tion. Advances in Neural Information Processing Systems, 0113771-33794 2022.
Fang, Z.., Yu, H., Xu, C.,, Li, Z.,, Ying, J., and Xie, S. Con- trastive zero-shot relational learning for knowledge graph completion. Knowledge-Bass Systems, pp. 1 113425, 2025.
Bossard, I.., Guillaumin, M., and Van Gool, I.. Food- 101- mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Pro- ceedings, Part VI 13. pp. 446-444646 461. Springer, 2014.
Fei-Fei, L.,, Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178-178. IEEE, 2004.
Cao. Q. Chen, Y., Lu, L., Sun, H., Zeng, Z., Yang, X., and Zhang, D. Generalized domain prompt learning for accessible scientific lsionnlanguuae models. Nexus, 2(2), 2025.
Greer, R., Antoniussen, B., Andersen, M. V., Møgelmose, A., and Trivedi, M. M. The why, when, and how to use active learning in large-data-driven 3d object detection for safe autonomous driving: An empirical exploration. arXiv preprint 1rXiv:2401.16663 2024.
Chen, G., Yao, W., Song, X., Li, X., Rao, Y., and Zhang, K. Plot: Prompt learning with optimal transport for vision- language models. In The Eleventh International Confer- ence on Learning Representations.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2228 2019.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M.. et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 8340-8349, 2021a.
Cui, F. Yang, X., Wu, C., Xiao, L., and Tian, X. Ad- vancing prompt learning through an external layer. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 807-816, 2024.
Cui, F., Fong, J., Zeng, R., Tian, X., and Yu, J. A similarity paradigm through textual regularization without forget- ting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 16100-16108, 2025a.
Hendrycks, D.,, Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021l..
Cui, F., Zhang, Y., Wang, X., Wang, X., and Xiao, Gener- alizable prompt learning of clip: A brief overview. arXiv preprint ar Xiv:2503..01233, 2025b.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A lcale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Iece, 2009.
Jia, Y., Wang, Z., Han, Z., Li, Zhang, M., Lao, Z., Han, Y., Gao, R., Gao, J., Zheng, Z., et al. Variable and intelligent catalyst design based on local chemical environments in sulfur redox reactions. Joule, 9(5), 2025.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Jin, K., Liu, Z., Wang, J., and Wang, H. Unmanned sur- face vehicle navigation under disturbances: World model enhanced reinforcement learning. JEFE/ASME Transac- tions on Mechatronics, 2025.
10

---

Enhancing Target-unspecifie Tasks through Features Matrix
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Liu, S., Chen, Z., Wu, M., Liu, C., and Chen, L. Wisr: Wireless domain generalization based style random- ization. IEEE Transactions on Mobile Computing, 23(5): 2023a.
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. Self-regulating prompts: Foun- dational model adaptation without forgetting. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision pp. 15190-15200, 2023b.
Liu, S., Chen, 7.,, Wu, M., Wang. H., Xing, B., and Chen, L. Generalizing wireless cross-muliple-factor gesture recognition to unseen domains. IEEE Transactions on Mobile Computing. 23(5):5083 5096, 2023b.
Liu, Z., Li, K., Hao, T., and Wang, H. Visual servoing of rigid-link flexible-joint manipulators in the presence of unknown camera parameters and boundary output. IEEE Transactions Systems, Man, and Cybernetics: Systems, 53(8):5096-5105 2023c.
Krause, J., Stark, M., Deng. J., and Fei-Fei, L.. 3d object rep- resentations for fine-gratd grained categorization. In Proceed- ings of the IEEE international conference on computer vision workshops. pp. 554-561, 2013.
Li, H., Huang, B., Chuai, M.. Zheng, Z., Chen, H., Piao, Z., Zhou, G., and Fan, H. J. Dual-plating aqueous zn-iodine batteries enabled via halogen-complexatin chemistry for lcrge-scale energy storage. Energy & Environmental Science, 18(7):3160-311 2025a.
Liu, Z., Zhai, Y., Li, J., Wang. G., Miao, Y., and Wang, H. Graph relational reinforcement learning for mobile robot navigation in large-scal sege-scale crowded environments. IEEE Transactions on Intelligent Transportation Systems, 24 (8)8776-878)) 2023d.
Li, H., Mai, Y., Gao, M., He, J., Liu, Z., and Wang. H. Large-scale lidar-based loop closing via combination of equivariance and invariance on (3). JEEF/ASME Trans- actions on Mechatronics, 2025b.
Liu, Z., Yu, H., and Luo, X. Federated graph anomaly detection via disentangled representation learning. In Proceedings of the ACM on Web Conference 2025, pp. 1216-1224, 2025.
Li, H., Zhang, M., Zheng, Z.,, Wu, X., Xiao, X., Piao, Z Li, C., Jia, Y., Yang. J., and Zhou, High-capacity and long-life aqueous zn-span batteries with tandem catalysis. Advanced Materials, 37(5)22499771, 2025c.
Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5206-5215, 2022.
Li, J., Zhang, Y., Liu, X., Liu, P., Yin, H., and Liu, D. A robust automatic ultrasound spectral envelope estimation. Information, 10(6):199, 2019.
Maaz, M., Rasheed, H., Khan, S., Khan, F. S., Anwer, R. M., and Yang, M.-H. Class-agnostic object detection with multi-modal transformer. In European Conference on Computer Vision, pp. 512-531. Springer, 2022.
Li, J., Li, H., Erfani, S. M., Feng, L.,, Bailey, J., and Liu, F. Visual-text cross alignment: Refining the similarity score in vision-language language models. In International Conference on Machine Learning, pp. 28018-28039, PMLR, 2024a.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515/ 2013.
Li, Q. Hu, Z., Wang, Y., Li, L., Fan, Y., King, I., Jia, G., Wang, S., Song, L., and Li. Y. Progress and opportunities of foundation models in bioinformatics. Briefings in Bioinformatics, bbbee55,,, 2024b.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Li, W., Guo, Y., Zheng, J., Lin, H., Ma, C., Fang, L., and Yang, X. Sparseformer: Detecting objects in hrw shots via sparse vision transformer. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 4851- 4860, 2024c.
Pan, B., Li, Q.. Tang, X., Huang, W., Fang. Z., Liu, F,, Wang, J., Yu, J., and Shi, Y. Niprompt: Noise-label prompt learning for vision-language models. arXiv preprint 2rXiv:2412.01256 2024.
Liu, J., Zhuo, D., Feng, Z., Zhu, S., Peng, C., Liu, Z.,, and Wang. H. Dvlo: Deep visual-lidar odometry with -oca-lto-globll feature fusion and bi-directional structure alignment. In European Conference on Computer Vision, pp. 475 493. Springer, 2024.
Pan, Y., Xu, Y., Liu, Z., and Wang. H. Planning from imagination: Episodic simulation and episodic memory for vision-and- lannluugge navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 6345-6333, 2025.

---

Enhancing Target-unspecific Tasks through Features Matrix
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Annual Conference of the IEEE Industrial Electronics Society, pp. 1-5. IEEE, 2021.
Wang, Y., Yan, Y., Shi, D., Zhu, W., Xia, J., Jeff, T., Jin, S., Gao, K., Li, X., and Yang, X. Nerf-ibvs: visual servo based on nerf for visual localization and navigation. Advances in Neural Information Processing Systems, 36: 8292 8304, 2023.
Peng, Z., Tian, Q.. Xu, J., Jin, Y., Lu, X., Tan, X., Xie, Y., and Ma, L.. Generalized category discovery in semantic segmentation. arXiv preprint arXiv:2311.
2023. Peng, Z.,, Ma, J., Sun, Z., Yi, R., Song, H., Tan, X., and Ma, L. Mos: Modeling object-scene associations in general- ized category discovery. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15118- 15128, 2025.
Wang. Z.,, Xu, Q., Yang. Z., He, Y., Cao, X., and Huang, Q. Openauc: Towards auc-oriented open-set recognition. Advances in Neural Information Processing Systems, 35: 25033-25045, 2022.
Wu, W. Wang. Q., Wang, G., Wang, J., Zhao, T., Liu, Y., Gao, D., Liu, Z., and Wang, H. Emie-map: Large-scale road surface reconstruction based on explicit mesh and implicit encoding. In European Conference on Computer Vision, pp. 370-386. Springer, 2024.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G. Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize imagenet? In Interna- tional conference on machine learning, pp. 5389-5400. PMLR, 2019.
Sanghi, A., Chu, H., Lambourne, J. G., Wang, Y., Cheng, C.-Y., Fumero, M., and Malekshan, K. R. Clip-forge: To- wards zero-shot text-to-shape generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18603-18613, 2022.
Xu, Y., Pan, Y., Liu, Z., and Wang, H. Flame: Learning to navigate with multimodal IIm in urban environments. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 9005-9013, 2025.
Yang, Z., Xu, Q., Bao, S., Cao, X., and Huang, Q. Learning with multiclass auc; Theory and algorithms. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 44 (11):7747-7763, 2021.
Shao, Z., Wang, M., Yu, Z., Pan, W., Yang, Y., Wei, T., Zhang, H., Mao, N., Chen, W., and Yu, J. Growing twig to accelerate large vision-language models. arXiv preprint arXiv:2503. 14075 2025.
Yang, Z., Xu, Q. Bao, S., Hc, Y., Cao, X., and Huang, Q. Optimizing two-way partial auc with an end-to-ent framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4528))10228-10246 2023. doi:
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.000 1212.0402 2012.
10.
Tian, F., Li, Y., Yan, Y., Guan, S., Ge, Y., and Yang, X. Postedit: Posterior sampling for efficient zero-shot image editing. arXiv preprint 10.04444 2024.
Yao. H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge gugiddggg context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6757-6767, June 2023a. Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge -uidedd context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6757-6767, 2023b.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predic- tive power. Advances Neural Information Processing Systems, 32, 2019.
Yin, H., Ren, Y., Yan, K., Ding, S., and Hao, Y. Rod-mllm: Towards more reliable object detection in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14358 14368, 2025.
Wang, Y., Liu, C., Li, S., Wang, T., Lin, W., and Yu, X. A tarteitargeetee tracking algorithm for fast-moving work- pieces based on event camera. In IECON 202 20th-47th
12

---

Enhancing Target-unspeciffe Tasks through Features Matrix
Yu, H., Liu, W., Zhu, N., Li, P., and Luo, X. In-gfd: An in- terpretable graph fraud detection model for spam reviews. IEEE Transactions Artificial Intelligence, 2024a.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional prompt learning for vision-language models. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 168 18816-16825, 2022a.
Yu, H., Wen, J., Sun, Y., Wei, X., and Lu, J. Ca-gnn: A competence-aware graph neural network for semi- supervised learning on streaming data. IEEE Transactions on Cybernetics, 2024b.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning prompt for vision-language models. International Jour- nal of Computer Vision, 130(0))2337-2344 2022b.
Zhou, Y., Xia, X., Lin, Z.,, Han, B., and Liu, Few-shot adversarial prompt learning on vision-language models. Advances in Neural Information Processing Systems, 37: 3122 3156, 2024.
Yu, H., Wen, J., and Zheng, Z. Camel: Cross-modality adaptive meta-learning for text-based person retrieval. IEEE Transactions on Information Forensics and Security, 2025.
Zhu, B., Niu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15659-15669, 2023.
Yu, 7., Zheng, I.., Zhao, Z., Wu, F., Fan, J., Ren, K., and Yu, Anetqa: large-scale benchmark for fine-grained com- positional reasoning over untrimmed videos. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23191-232000 2023.
Zhang, J., Wu, S., Gao, L., Shen, H. T., and Song, J. Dept: Decoupled prompt tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12924-12933, 2024a.
Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y., and Li, H. Tip-adapter: Training-free free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021.
Zhang, W., Lyu, S.. Xue, Yao, C., Zhu, Z., and Jia. Z. Predict the rover mobility soft terrain using articu- lated wheeled bevameter. IEEE Robotics and Automation Letters, 7(4): 10622-1269. 2022.
Zhang, W., Wu, L., Zhang, Z., Yu, T., Ma, C., Jin, X., Yang, X., and Zeng, W. Unleash the power of vision- language models by visual attention prompt and multi- modal interaction. IEEE Transactions on Multimedia, 2024b.
Zhang, Y. and Tian, X. Consistent prompt learning for vision-language models. Knowledge-Baud Based Systems, 310: 112974, 2025.
Zhou, C., Han, S., Yang, L., Zhou, Y., Cheng, X., Wang, Y., and Li, H. Rankadaptor: Hierarchical rank alloca- tion for efficient fine-tuning pruned llms via performance model. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 5781-5795, 2025a.
Zhou, C., Zhou, Y., Zhang. S., Wang, Y., and Liu, Z. Dy- namic operator optimization for efficient multi-tenant lora model serving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 19910-22918, 2025b.
13


Reviews:
[{"review_id": "kqSJXqvLXp", "reviewer": "Reviewer_xBkC", "summary": "This paper proposes a Features Matrix regularization method to improve model performance on target-unspecific tasks. FM preserves general knowledge and reduces overfitting by extracting and leveraging semantic information from diverse inputs. The approach, compatible with existing frameworks, enhances generalization and performs well across various datasets, including 11 datasets with limited labeled data. It also incorporates pre-trained CLIP features and multiple handcrafted prompts to prevent forgetting essential knowledge, demonstrating state-of-the-art performance in target-unspecific tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# **Thanks for review**:  \nWe thank the reviewer for the valuable time and consideration of our manuscript. The comments provided by the reviewers are very useful for our work, we express our heartfelt **gratitude**.\n\n## Response:  \n\n>**Q1**:  The detailed explanation of the illustrations can be improved. \n\n**A1**: Thank you very much to the reviewer for important comments on the illustration. Our initial focus in designing this illustration was on \"simplicity\", \"contrast\", and \"the designed artistic drawing\". However, we lack description of some components in the diagram. We **promise** to make serious revisions. Our explanation of the symbols in the figure is as follows:\n\nIn the figure, \"snowflake pattern\" represents parameter freezing, \"flame pattern\" represents learnable pattern, \"Deep\" represents learnable tokens embedded in several layers of the encoder, where \"T\" represents learnable text embedding, \"V\" represents learnable visual embedding, \"Class priors\" represents which category it belongs to, \"Tuning Similarity\" represents the calculation of cosine similarity for fine-tuning architecture, and the light gray \"Similarity\" represents the calculation of cosine similarity for frozen architecture. In the MaPLe architecture diagram, the \"e\" represents the matrix function that connects the encoders of two modalities in several layers. \n\n>**Q2**:  The more detailed discussion of the base classes and novel classes in the introduction.\n\n**A2**: We will add the revised records to the final version to express our **gratitude** to the valuable review. In terms of article writing, we need to improve our manuscript from various aspects. We **promise** to make serious revisions. Our explanation of the \"base and novel classes\" is as follows:\n\nIn the base-to-novel generalization task, the datasets are divided into base and novel classes. The model is trained on the base classes, and tested on both the base and novel classes. The number of classes for base and novel is the **same**, which means that all classes in a dataset are evenly divided into two groups of classes. The process of dividing all classes in the dataset is randomly selected.\n\n>**Q3**: Some ablation studies can be provided to verify the effectiveness of trainable text tokens and image tokens.\n\n**A3**: The experiments proposed by the reviewer are very meaningful and have greatly **helped** our work. We will include this part of the experiments in the main paper and express our gratitude to reviewer. We are now conducting ablation experiments on text embedding and visual embedding **separately** to explore more phenomena.\n\n1) We **add experiments** on the text and image tokens **length** for their effectiveness. Specifically, we set different learning tokens lengths for text tokens and visual tokens. When we conducted ablation experiments on visual embeddings, the value of text embeddings remained at best value (4). The reverse is also the same.\n\n* Base-to-novel generalization of 11 datasets (Textual Tokens Length) \n| Textual Tokens Length     |  1     |  2     |  4       | 6      |  8     | 10    | \n|:---------------             |:------:|:------:|:--------:|:------:|:------:|:------:|\n| HM (Ours based on MaPLe)    |  77.11 |  79.00 | **80.32**| 79.12   | 78.81  | 77.03 |\n| HM (Ours based on PromptSRC) |  78.02 |  79.34 | **81.32**| 81.00   | 80.86  | 78.94 |\n\n\n* Base-to-novel generalization of 11 datasets (Visual Tokens Length) \n| Visual Tokens Length |  1     |  2      |  4       | 6      |  8     | 10    | \n|:---------------       |:------:|:------:|:--------:|:------:|:------:|:------:|\n| HM (Ours based on MaPLe) |  77.01 |  78.11 |**80.32** | 79.33   | 78.11  | 77.50 |\n| HM (Ours based on PromptSRC) |  77.32 |  78.08 |**81.32** | 80.54   | 80.51  | 79.11 |\n\n2) We **add experiments** on the text and image tokens on different **learning depth**. When we conducted ablation experiments on visual embeddings depth, the value of text embeddings depth remained at best value (9). The reverse is also the same.\n\n* Base-to-novel generalization of 11 datasets (Textual Tokens Depth) \n| Textual Tokens Depth      |  1     |  3     |  5       | 7      |  9     | 11    | \n|:---------------             |:------:|:------:|:--------:|:------:|:------:|:------:|\n| HM (Ours based on MaPLe)    |  77.87 |  80.00 |  78.90    | 79.02   | **80.32**  | 77.51 |\n| HM (Ours based on PromptSRC) |  78.15 |  79.90 | 80.16| 80.80   | **81.32**  | 78.23 |\n\n\n* Base-to-novel generalization of 11 datasets (Visual Tokens Depth) \n| Visual Tokens Depth      |  1     |  3      |  5       | 7      |  9     | 11    | \n|:---------------             |:------:|:------:|:--------:|:------:|:------:|:------:|\n| HM (Ours based on MaPLe)     |  76.06 |  77.03 |78.65 | 78.16   | **80.32**  | 76.93 |\n| HM (Ours based on PromptSRC)  |  76.91 |  78.22 |78.33 | 81.00   | **81.32**  | 78.09 |\n\n3) If the reviewer has further suggestions on ablation experiments, we welcome discussions with reviewer.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No, this paper does not have the theoretical claims.", "experimental_designs_or_analyses": "Yes, I have checked the experimental designs and analyses.", "relation_to_broader_scientific_literature": "This paper proposes a better way to combine manually designed prompts and trainable tokens to improve the performance of target-unspecific tasks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. The illustrations are of great style, but the detailed explanation of the illustrations can be improved. The paper writing can be improved to make the paper more clear.\n2. The paper discusses the limitation of the proposed method, it is good.\n3. The experiments are comprehensive, but more ablation studies are needed to verify the effectiveness of the proposed method.", "other_comments_or_suggestions": "Some parts of the paper can be further improved to make the paper more clear, for example, the more detailed discussion of the base classes and novel classes in the introduction. The illustration of different types of prompt design (Fig 2) are not very clear, more details can be added in the figure caption.", "questions_for_authors": "1. What is the details of the split of base classes and novel classes in the experiments?\n2. Some ablation studies can be provided to verify the effectiveness of trainable text tokens and image tokens. Because one simple baseline may be to train a \"MOE\" to dynamically select the manually designed prompt.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YmdAy2m4H", "reviewer": "Reviewer_5hGa", "summary": "## Summary\n\nThis paper addresses out-of-distribution generalization problem in prompt fine-tuning \"CLIP\" kind of model.  The challenge is common: fine-tuning on a specific task boosts the performance in this task, but hurt the performance on other general tasks. \n\nThis work addresses this problem by adding a regularization during fine-tuning. This regularization, working in a constrastive process, encourages the model (during optimization) to output a close representation as the pretrained model.  This is an common, yet effective regularization. \n\nAs a regularization, it is supposed to help different \"fine-tuning\" methods. The experiment shows verifies this hypothesis on many tasks. \n\n\n-------\n\n## Strengthness\n- \"negative interference\" has long been a problem in model-based algorithm [1]. Fine-tuning in neural network is the common scenario that suffers from \"negative interference\". Thus it is interesting and important to research on negative interference in fine-tuning neural networks. \n\n- The regularization, which encourages the representation to be close to the pretraining one during fine-tuning, is an resonable and intuitive approach to reduce negative interference. \n\n- Many experimental results justifies the positive effect of the proposed method in this paper. \n\n----\n\n## Weakness \n\n- The novelty of the regularization is questionable. Many related works shares the same idea. For example, L2 weights decay towards pretrained weights rather than zero.  KL-divergence regularization in fine-tuning transformers via reinforcement learning manner, weights-averaging of pretrained and fine-tuned models, etc. \n\n\n\n\n[1] Atkeson, Christopher G., Andrew W. Moore, and Stefan Schaal. \"Locally weighted learning.\" Lazy learning (1997): 11-73.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# **Thanks for review**: \nThank you to the reviewer for taking time to take detailed research and valuable reference on our work. We would like to express our **gratitude**. We will also **carefully** revise according to the opinions of the reviewers.\n\nIn addition, thank you to the reviewer for providing such a comprehensive reference [1] in the comments, and we went to read this article. This article investigates many restrictive and regularization schemes, which we believe are very valuable. It is interesting and important to research on negative interference. The \"negative interference\" has long been a problem in model-based algorithm. Our future work will conduct an in-depth investigation of these regularization functions with prompting works. We will add this article to our citation and provide a descriptive introduction to the relevant work. \n\n[1] Atkeson, Christopher G., Andrew W. Moore, and Stefan Schaal. \"Locally weighted learning.\" Lazy learning (1997): 11-73\n\n\n## Response: \n\n>**Q1**:  L2 weights decay towards pretrained weights rather than zero. KL-divergence regularization in fine-tuning transformers via reinforcement learning manner, weights-averaging of pretrained and fine-tuned models, etc. Many related works shares the idea. \n\n\n**A1**: Thank you very much to the reviewer for specific technical analysis and insights. We will now introduce some **unique** aspects of our work, hoping that reviewer will have new perspectives on our work.\n\n* The **contribution** of our method: fully utilizing manual prompt words to mine information. \n\n\n* Our core module is **non-traditional** sense of regularization. In other words, the \"regularization\" mentioned in our paper is actually a **new** method.  \n\n* Compared to L2 and KL divergence, our method is flexible and **delves** into specific semantics. Our proposed scheme is closely integrated with manual prompt words. \n\nIn our future work, we will study and **draw on** the opinions provided by reviewers to design more works in earnest.", "claims_and_evidence": "check summary.", "methods_and_evaluation_criteria": "check summary.", "theoretical_claims": "no theory.", "experimental_designs_or_analyses": "yes.", "relation_to_broader_scientific_literature": "check summary.", "essential_references_not_discussed": "no.", "other_strengths_and_weaknesses": "check summary.", "other_comments_or_suggestions": "check summary.", "questions_for_authors": "check summary.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "isBW69Lc0C", "reviewer": "Reviewer_HJ59", "summary": "The paper proposes to mitigate the overfitting observed in parametric optimization methods when optimizing on the target domain. To address this issue, the authors introduce a feature matrix-based approach. This method leverages features extracted from multiple handcrafted prompts, combined with features from various classes, and employs a contrastive loss to optimize the process. The aim is to excavate general feature information, although some of it may not be directly relevant to the task. They validate their approach by integrating it with existing methods and evaluating it on multiple datasets. The experimental results, along with the average performance improvements across datasets, demonstrate that their method not only achieves better overall performance and also delivers significant improvements in certain cases.\n\n## update after rebuttal\nThanks for the efforts. I recommend the authors include the full version of Table 8 for every dataset in the paper (as promised). Moreover, the implementational details of the proposed method in conjunction with the other methods are not clear; hence, also include this in the paper from the rebuttal. In response, I have decided to retain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# **Thanks for review**: \nWe express gratitude to the reviewers: 1) for providing rich and **comprehensive** comments of our work, 2) for investing a significant amount of  time and effort in reviewing our work, and 3) for providing practical guidances. We will add acknowledgements in the final version and express our **gratitude**.\n\n## Response:\n\n>**Q1**: For the claim that common prompt learning methods suffer from overfitting, leading to poor generalization in novel classes (Lines 162–164), there are no references or empirical evidence. \n\n\n**A1**: Thank you very much for the reviewer's reminder. We apologize that our writing here is not very organized. In line (162-164), these methods refer to line (158-160) of paper. These methods are CoCoOp, CoOp, KgCoOp, ProDA, DPLQ, PLOT in line (158-160) of our paper.\n\n\n>**Q2**:  What does \"Low-β\" mean in Line 188? \n\n\n**A2**: 'Low-β' refers to a list of scores sorted from low to high, with β values assigned from the front. We **promise** to write it clearly in the final version.\n\n\n>**Q3**:  This term 'regularization' appears in the introduction and abstract but nowhere else in the paper. \n\n\n**A3**: Our method is **non-traditional** regularization . In other words, the \"regularization\" method mentioned in our paper is actually a new method. Based on this, this term 'regularization' does not appear elsewhere in our paper.\n\n\n>**Q4**:  The methodology does not explain convincingly why cosine similarity was chosen.\n\n\n**A4**: Our infrastructure (CLIP) and previous works (CoOp, CoCoOp, ProDA, KgCoOp, PLOT, MaPLe, PromptSRC, etc.) use cosine similarity in the alignment calculation of text and visual branches. Therefore, to ensure **fair** comparison, our method also uses cosine similarity here. Based on this, cosine similarity is also used to align the two modalities in the 'scores matrix' stage. Our idea is to **minimize** the introduction of other measurement methods, and to explore performance based on existing methods (CLIP, CoOp, MaPLe, etc) and hand-crafted templates.\n\n\n>**Q5**:  Demonstrate how many prompt templates are necessary for their method. How does the method perform with increases or decreases in features of templates? \n\n\n**A5**: We used 60 manual prompt templates, the contents of which are in our Appendix (A.6), the number 60 is fixed. We apologize for not including the number in the method details. We **promise** to write it clearly in the final version. Thank you for the careful review by the reviewer.\n\n>**Q6**:  I recommend providing the full version of Table 8 that will detail the utility of $\\beta$ for every dataset. Currently, with average performance it's unclear which similar features between source and target would be useful.\n\n**A6**: Thank you for the reviewer's comments. Due to the limited word count of Rebuttal, we now write the specific values of base and novel here to illustrate the relationship between source and target. And, we **promise** to refine the values of each dataset in the final version.\n\nBase-to-novel generalization of 11 datasets ($\\beta$) \n |        $\\beta$   |  3      |  4        |  5           | 6           | \n |:--------------- |:------:|:-------:|:---------:|:----------:|\n | Base           |  80.19 |  83.47 | **85.70**| 84.46  | \n | Novel          |  76.33 |  76.51 | **77.35**| 76.90  | \n | HM             |  78.21 |  79.81 | **81.32**| 80.51  | \n\n\n>**Q7**:  How are the features used in the model for the baseline methods to integrate the proposal? I recommend adding a subsection in the methodology to explain this process clearly. \n\n**A7**: Thank you for the reviewer's comments. In the final version, we promise to add pseudocode for the algorithm flow. In addition, we **promise** to add detailed implementation subsection for plug-and-play applying for CoOp CoOoOp、MaPLe、PromptSRC, and write it into a new subsection. Our brief introduction to the plug-and-play concept is as follows:\n\n* In Figure 3, Equation 5 refers to the part below the grey dashed box. After Equation 5, our plug-and-play framework begins to empower the results with Equation 5. Our plug-and-play architecture input consists of two parts: 1) text features generated by 60 manually prompted words, and 2) visual features generated by learnable visual embeddings. Therefore, learnable visual embeddings are necessary for the work we propose.\n\n* In Figure 2, we found that CoOp and CoCoOp do not have a visual embedding part, so we will first add learnable visual embeddings to the CoOp and CoCoOp architecture. Afterwards, based on this modified architecture, integrate the methods we proposed. For MaPLe and PromptSRC, our proposed method can be directly integrated.\n\n* It is worth noting that the input of the text encoder consists of two parts: 1) manual prompt word templates, and 2) learnable prompt words, which vectorize the text into a learnable matrix. These two parts will not affect each other during the input phase, as they are two separate input processes.", "claims_and_evidence": "My main critique is about detailing their method (proposal of the paper) and substantiating it with appropriate text and experiments. \n\n – For the claim that common prompt learning methods suffer from overfitting, leading to poor generalization in novel classes (Lines 162–164), there are no references or empirical evidence. Can the authors provide them to substantiate this?\n\n – What does \"Low β\" mean in Line 188? The authors should elaborate.\n\n – In the methodology, the claim in the abstract regarding the use of regularization is not clear. This term appears in the introduction and abstract but nowhere else in the paper. Can the authors clarify this?\n\n – The methodology does not explain convincingly why cosine similarity was chosen.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper includes minimal theory and theory investigation.", "experimental_designs_or_analyses": "– While the additional experiments exploring target k-shots and learning depth are interesting, they are insufficient. The authors should include more experiments that provide insights into their method. For instance, demonstrate how many P_{1,2,3} samples are necessary for their method. Similarly, how does the method perform with increases or decreases in t_{k}? What are the assumptions for the prompts, and are there any guidelines?\n\n– I recommend providing the full version of Table 8 that will detail the utility of \\beta for every dataset. Currently, with average performance it's unclear which similar features between source and target would be useful \n\n – The authors vaguely mention the need for a ‘set’ of prompts  (line 377) but fail to specify how many are needed. Can this be clarified and elaborated?", "relation_to_broader_scientific_literature": "The paper lacks significant contribution in terms of novelty.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "– Can the authors include the algorithm for their method? Additionally, for one of the baselines used, can they incorporate their algorithm into the paper’s algorithm (‘Ours’)? This would provide additional details and help describe the steps more clearly.\n\n – In the methodology, after Equation 5 is introduced, what happens next? How are the features used in the model for the baseline methods to integrate the proposal? I recommend adding a subsection in the methodology to explain this process clearly. Currently, it is unclear.", "other_comments_or_suggestions": "Figure 2 doesn't add value to the paper. I recommend shifting it to the appendix.", "questions_for_authors": "The authors do not provide enough details regarding methodology and experiments that would investigate their proposal even deeper. As a reader, I was intrigued to know more about features matrix and regularization, which are not detailed sufficiently.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VcxSkUX7hK", "reviewer": "Reviewer_gDtW", "summary": "Partial parameter optimization methods face challenges in handling target-unspecific tasks due to overfitting, which causes the model to lose its general knowledge essential for these tasks. To address this issue, this paper proposes a regularization technique using a Feature Matrix (FM). Through extensive evaluations across various tasks, the proposed method demonstrates significant improvements in enhancing performance on target-unspecific tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# **Thanks for review**:\n Thank you very much for the reviewer's appreciation of our work. Thank you to the reviewer for your valuable time for our paper, we express our **heartfelt** gratitude.\n\n## Response：\n\nOur work is based on pre-trained CLIP (ICML2021), and learnable and non learnable prompt words to explore and attempt under existing limited conditions and theories. \n\n* Our initial idea is to **minimize** the introduction of new theories, and to explore performance based on existing technology.\n\n* We focused more on exploring the performance of fine-tuning multimodal models on large-scale datasets for **realistic** application scenarios (11 datasets).\n\n* Our method is a plugin based approach, which appears to have practical value for **convenience** and low coupling. \n\nThank you to the reviewer for the insights on the theoretical aspects of our work. The theoretical analysis  will definitely increase the strictness of our work. According to the **guidance** of the reviewer, we will continue to improve this manuscript, including but not limited to theoretical analysis of  experiments.\n\nWe **promise** to  make revisions before submitting the final version.", "claims_and_evidence": "The paper asserts that partial parameter optimization struggles with novel classes due to overfitting, as evidenced by the results in Table 1. To address this issue, the proposed Feature Matrix (FM) enhances generalization by preserving general knowledge through multiple hand-crafted prompts. Experimental results demonstrate that FM significantly improves novel class accuracy compared to previous approaches.", "methods_and_evaluation_criteria": "The proposed methods are straightforward and easy to comprehend. The paper evaluates their effectiveness using a diverse set of benchmark datasets.", "theoretical_claims": "It appears that the paper does not present theoretical claims that require formal proof.", "experimental_designs_or_analyses": "The overall experimental setup appears to be well-designed, and the paper includes essential analyses, such as base and novel classification performance, the impact of additional parameters, prompt length, and cost comparisons with previous methods.", "relation_to_broader_scientific_literature": "This paper builds on previous research in vision-language models (VLMs), prompt tuning, and generalization techniques. It also introduces the Feature Matrix (FM) as a novel regularization method to enhance generalization in target-unspecific tasks.", "essential_references_not_discussed": "I am not a strict expert in this domain, so I cannot fully confirm whether the paper includes all the necessary references. However, it appears to cite the relevant works and uses them effectively for comparison to validate its proposed method.", "other_strengths_and_weaknesses": "- The paper introduces the Feature Matrix (FM) regularization technique to enhance the generalization of unseen classes in prompt learning. The proposed methods are well-structured, intuitive, and easy to understand.\n- To validate their approach, the paper evaluates performance across various benchmarks and includes essential analyses, such as hyperparameter tuning and cost comparisons.\n- However, a notable limitation is the lack of theoretical analysis, which could strengthen the explanation of why FM improves generalization.", "other_comments_or_suggestions": "No other suggestions.", "questions_for_authors": "Overall, the work is well-structured, easy to understand, and presents a plausible approach. The proposed methods effectively enhance performance on novel classes compared to previous approaches. One notable limitation is the lack of theoretical analysis, which could provide a stronger justification for why the proposed methods improve generalization. However, this does not seem to be a critical issue in evaluating the paper. Since I am not an expert in this domain, I believe it would be best to wait for comments from other reviewers for a more comprehensive assessment.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to improve generalization in target-unspecific tasks for CLIP-based models, leveraging Features Matrix (FM). The reviewers acknowledged the clear motivation, strong empirical results across a large number of datasets, plug-and-play compatibility, and thorough ablations. However, the more critical concern from the reviewers were about its unclear novelty over KgCoOp [Yao et al., 2023], which also regularizes tuning using pre-trained CLIP text features. While the proposed approach uses multiple prompts and contrastive loss on visual features, unlike KgCoOp which uses a single-prompt with L1 loss, the reviewers and the AC found the overall approach as conceptually very close and incremental. Another critical concern was its inference overhead from computing multiple cosine similarities, which may severely limit its practical applicability. Considering all these factors, I believe that this is a clear borderline paper, which could be accepted if there is enough room for presentation at the conference.
Decision: Accept (poster)