Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Authors: Liangchen Liu
Nannan Wang
Xi Yang
Xinbo Gao
Tongliang Liu
Venue: ICML 2025 poster (2025)
Keywords: transfer learning
prompt learning
vision-language model
parameter-efficient fine-tuning
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Prompt learning is a cutting-edge parameter-efficient fine-tuning technique for pre-trained vision-language models (VLMs). Instead of learning a single text prompt, recent works have revealed that learning diverse text prompts can effectively boost the performances on downstream tasks, as the diverse prompted text features can comprehensively depict the visual concepts from different perspectives. However, diverse prompt learning demands enormous computational resources. This efficiency issue still remains unexplored. To achieve efficient and diverse prompt learning, this paper proposes a novel \textbf{Surrogate Prompt Learning (SurPL)} framework. Instead of learning diverse text prompts, SurPL directly generates the desired prompted text features via a lightweight \textbf{Surrogate Feature Generator (SFG)}, thereby avoiding the complex gradient computation procedure of conventional diverse prompt learning. Concretely, based on a basic prompted text feature, SFG can directly and efficiently generate diverse prompted features according to different pre-defined conditional signals. Extensive experiments indicate the effectiveness of the surrogate prompted text features, and show compelling performances and efficiency of SurPL on various benchmarks.

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Liangchen Liu
Nannan Wang'
Xi Yang'
Xinbo Gao
Tongliang Liu
Abstract
(a) Single Prompt L Learning
Surrogate Prompt earning
Sinikech
Prompt learning is cutting-edge parameter- efficient fine-tuning technique for pre-trained vision-language models (VLMs). Instead of learn- ing a single text prompt, recent works have re- vealed that learning diverse text prompts ef- fectively boost the performances downstream tasks, the diverse prompted text features can comprehensively depict the visual concepts from different perspectives. However, diverse prompt learning demands enormous computational re- sources. This efficiency issue still remains un- explored. To achieve efficient and diverse prompt learning, this paper proposes novel Surro- gate Prompt Learning (SurPL) framework. In- stead of learning diverse text prompts, SurPI. di- rectly generates the desired prompted text fea- tures via a lightweight Surrogate Feature Gen- erator (SFG), thereby avoiding complex gra- dient computation procedure of conventional di- verse prompt learning. Concretely, based a basic prompted text feature, SFG can directly and efficiently generate diverse prompted fea- tures according to different pre-defined condi- tional signals. Extensive experiments indicate the effectiveness of the surrogate prompted text features, and show compelling performances and efficiency of SurPL various benchmarks. Code is available at https: htt///gtttu/b///////// com/ 11cllc1 (ec1lcc997/// /0///197///ss/ SurPI.
Secturres
surrogule text feabures
Fimal Eace
Lext Eacoder
diverse sisual features
pronpt tnkzend
Narmogah
class tokens
diverse conditional siguals
Starkkrity
Visual Encoder
Text Encoder
dherre
A
Flact
Text
prompt takens4
diaaese jconppt takres
class takens
Foeward Pass Gradient propagatinn
(b) Diverse Prompt Learning
Figure f. Illustration of different prompt learning frameworks. (a):
conventional single prompt learning. (b): diverse prompt learning, e.g., eetaace-dependen prompt learning and fine-graines prompt learning, etc. (c) our proposed Surrogate Prompt Learning.
standing. How to effectively and efficiently leverage these large-scale pre-trained models for downstream tasks has garered significant attention in research. In this context, prompt learning (Zhou et al., 2022c;b; Chen et al., 2023; Khattak et al., 2023a;b; Roy & Etemad, 2024; Yao et al., 2024; Lafon et al., 2024; Zhang et al., 2024a) comes to the fore as an emerging parameter-efficien fine-tuning (PEFT) technique, and has achieved compelling transferring perfor- mances for pre-trained VLMs..
Given downstream task, prompt learning targets to learn few extra prompt tokens at the text input position, so that the prompted text feature can obtain the task-specific knowl- edge, and provide more accurate descriptions of the visual concepts in the downstream tasks. While previous methods mainly focus on learning one single text prompt, recent ap- proaches start learn diverse text prompts and achieve sig- nificant improvements. These diverse prompted text features constitute more comprehensive descriptions of the visual concepts in downstream tasks from different perspectives. For example, instance-depende et al., 2022b) learns a unique visual-guided text prompt for each visual example; fine fine-grained prompt learning (Chen
1. Introduction
Recently, vision-language models (VLMs) (Radford al., 2021; Jia et al., 2021; Yao et al., 2021; Zhai al., 2022) have shown great capability in open-world visual under-
Xidian University, Xi'an 710071, Shaanxi, China. 'The Uni- versity Sydney, Darlington, NSW 2008, Australia, Correspon- dence to: Nannan Wang <nnwang @ xidian.cdu.cn..
prompt learning (Zhou
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
et al., 2023; Wang et al., 2023; Lafon al., 2024) learns several text prompts to align with different local-based vi- sual features, etc. We collectively denote these methods as diverse prompt learning.
performances of existing diverse prompt learning methods, while maintaining remarkable computation efficiency.
In a nutshell, we present our contributions as follows: 1) We propose a novel and efficient paradigm for diverse prompt learning, denoted as Surrogate Prompt Learning (SurPL... SurPL directly generates the diverse prompted text features instead of learning diverse text prompts from scratch, ad- dressing the efficiency issue in existing diverse prompt learn- ing: 2) To generate diverse text features, we propose Surro- gate Feature Generator (SFG). These features are controlled and generated by flexible and arbitrary conditional signals, allowing SurPL to adapt and integrate different types of diverse prompt learning frameworks; 3) Extensive exper- iments are conducted to indicate the effectiveness of our generated surrogate text features, and show remarkable per- formances and efficiency of SurPL on various benchmarks.
However, diverse prompt learning demands enormous computational resources. Although prompt learning is parameter-effin efficieet,, optimizing the prompt tokens at the in- put position still requires back mack-propagated gradient compu- tation through the entire text encoder, which causes consid- erable consumption on GPU memory and optimization du- ration. Given an M--clss visual classification task, learning a single text prompt needs to process the pack-propagatee gradient computation from M prompted text features to the input, simultaneously (shown in Fig. 1 (a)). Nevertheless, for instance- dependeent prompt learning, the optimization involves gradient computation w.r.t B x M text features, where B is the training batch size of the visual samples. For lieeggrained prompt learning, the gradient computation will increase to Z x M text features, where Z indicates the num- ber of applied local visual features. As illustrated in Fig.1 (b), these existing diverse prompt learning methods scale up the computation consumption from ((M) to O(B x M) or OZ x M), thereby leading severe efficiency issue.
2. Related Works
Single prompt learning: Recently, prompt learning has become a popular parameter-efficien fine-tuning technique that can rapidly adapt pre-trained VLMs (e.g., CLIP (Rad- ford et al., 2021), ALIGN (Jia al., 2021) and LiT (Zhai et al., 2022)) to various downstream tasks, without tuning any pre-trained parameters. Pioneer work CoOp (Zhou et al., 2022c) replaces hand-crafted prompt templates by a set of learnable prompt tokens at the text input position. After learning on downstream tasks, these prompt tokens capture task-specific knowledge and provide more accu- rate prompted text features align with task-related visual concepts, thus exhibiting better performances. Inspired by CoOp, subsequent approaches dedicate to further improve VLM's transfer ability by learning multi-modal prompts (Khattak et al., 2023a; Xu et al., 2023; Khattak et al., 2023b; Cho et al., 2023; Wang et al., 2023), generalizable prompts (Zhu et al., 2023; Yao et al., 2023; Liu et al., 2023; Yao et al., 2024; Zhang et al., 2024a), meta learning- based prompts (Li et al., 2023a; Zhao et al., 2024), neural architecture baaec prompts (Zhang et al., 2024b), etc.
To breakthrough the dilemma, this paper proposes Surro- gate Prompt Learning (SurPL). Instead of learning di- verse text prompts from scratch, SurPI. directly generates their prompted text features via a lightweight Surrogate Feature Generator (SFG). As shown in Fig. (c), SurPL only learns a single basic prompt. Given the basic prompted text feature, SFG can efficiently generate diverse text fea- tures according to different conditional signals. These sig- nals are inherently flexible and arbitrary. They can be pre- defined prior knowledge (e.g., semantic information of vi- sual instances), or can be initialized and learnable tokens that are supposed to convey specific knowledge (e.g., fine- grained information). SurPL bypasses the issue of enor- mous gradient computation inside the text encoder (i.e., keeping the computation consumption under (()),, but still provides diverse text features that can be exploited to comprehensively align with visual concepts.
Diverse prompt learning: Different from the above ap- proaches that only learn prompts to obtain a single prompted text feature, recent studies have manifested that leverag- ing diverse prompts can significantly improve the transfer performances, as their prompted text features can provide more comprehensive description of the visual concepts from different perspectives. CoCoOp (Zhou et al., 2022b) proposes instance deeendeet prompt learning, i.e., learning dynamic visual-conditional prompt for each visual example. PLOT (Chen et al., 2023) and ALIGN (Wang et al., 2023) learn multiple fine- -rained text prompts based on the opti- mal transport strategy. GalLoP (Lafon al., 2024) targets to learn both global and local text prompts that can align with the global and local representation features of the visual
We emphasize that the core idea of this paper is not designing new diverse prompt learning method, but proposing a novel and unified framework that can im- plement arbitrary diverse prompt learning approaches under the computation efficiency that is comparable to single prompt learning approaches. Qualitative and quan- titative experiments have verified that our generated sur- rogate features are sufficiently effective to replace those original prompted features. We further show that by sim- ply incorporating both instance-dependent and fine-grained prompt learning concepts into our surrogate prompt learn- ing framework, SurPL can even surpass the Maaeeoff-hhe-at
2

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
concepts. Although these diverse prompt learning methods have achieved remarkable performances, they still demand enormous computation resources.
replaces the hand-crafted prompts by several learnable prompt tokens = {st) E RLxd at the text input po- sition, where L and d indicates the prompt token length and token dimension. Therefore, the text input of m-th class can be formed as tm = [s, cm], where Cm is the m-th classname. Given a visual sample æ and the ground-truth label ymm of the downstream task, CoOp optimizes prompt token parameters s via cross-entropy loss Leg between the prediction probability and label (CLIP parameters are fixed):
3. Proposed Method
3.1. Overview
This paper proposes Surrogate Prompt Learning (SurPL) to achieve efficient and diverse prompt learning. Instead of directly learning diverse text prompts from scratch, SurPL. alternatively generates their prompted text features. As shown in Fig.2, SurPL introduces novel Surrogate Feature Generator (SFG). which can generate diverse text features based on the obtained basic text feature according to differ- ent conditional signals. These signals are inherently flexible and arbitrary, enabling SurPI. to incorporate any type of diverse prompt learning framework.
CCE(S) = - Um log p(m]
(2)
exp(cos(f f. wmm//T where p(m[z)] Σ/ exp(cos f,,,)/T)
Dense Visual-Text Prompt (DVLP). DVLP is a stronger baseline that has been widely-used in recent researches (Khattak et al., 2023a;b; Zhang et al., 2024a). DVLP ex- ploits multi-modal dense prompt tokens at each encoder layer. DVLP consists of text prompts s =
In this work, we focus on two primary types of diverse prompt learning methods: instance-dependent and fine- grained prompt learning. Similar to existing single prompt learning approaches, SurPL only optimizes few text prompt tokens to obtain basic prompted text feature. Using this basic text feature and various pre-defined conditional signals, SurPL efficiently generates instance-dependen and fine- mne-grained text features via SFG, which are supposed to be obtained by learning diverse text prompts. These generated text features, combined with the global-invariant text fea- ture, provide more comprehensive descriptions of the visual concepts from different perspectives, thereby significantly boosting the adaptation ability of VLMs.
and visual prompts u {u" ]KK Jk where K denotes the depth of prompts. The computation procedure of the k-th layer of DVLP can be expressed as:
{ [s cj -.c.c&lttii. | = T (*(ss, c [ant]] wxtt1 = V*& (lx u²))
The output of s* and ut are then substituted by the newly involved prompt tokens sh+1 and ++1, and reformed as the input for the next h + 1-th layer.
3.2. Preliminaries
Contrastive Language-Image Pre-training (CLIP). CLIP (Radford et al., 2021) has shown great potential on zero-shot image classification. CLIP consists of Ik=1 and text encoder T = ] Ik=1, where K denotes the depth of the encoder layers. Given an M-class task, the text inputs (tm IM are formed by the combination of the classnames (text labels) and the hand-crafted prompt, e.g., 'a photo of a [classname]'. The output text features can be obtained as w (t) = { [mm]] } m= where wm, denotes the feature corresponds to m-th class. Given the visual feature of an image x: f V(x), the prediction probability of æ belonging to m-th class can be expressed as:
3.3. Surrogate Feature Generator
visual-encoder V
{V*)
As shown in Fig.2, the proposed Surrogate Feature Gener- ator (SFG) is intrinsically a cross-attention module. The "query" is the basic prompted text feature while the "key" and "value" are the pre-defined conditional signals. Differ- ent from standard hidden layer dimension of the MLP (Multi-Layer Percep- tron), and keep SFG still parameter-efficient.
modules, we decrease the
M We denote w = [w m== E R Mx as the basic prompted text feature obtained from the text encoder, where M and d indicate the number of classes and the feature dimension. Generally, w can be obtained by any single prompt learning method. In this paper, we utilize DVLP as the baseline. The conditional signal E Rdd is flexible and arbitrary, which can be any pre-defined prior knowledge (c.g., semantic in- formation of visual instances), or initialized and learnable tokens that are supposed to convey specific knowledge (c.g., fine-grained information). The generated surrogate text fea-
exp(cos(f: wm m/T) M Σe= exp(cos(f.. w/)/
p(m m²)
(1)
where cos(:--) -) denotes the cosine similarity and r is a tem- perature parameter learned by CLIP.
Context Optimization (CoOp). CoOp (Zhou et al., 2022c)
3

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Basic visual
Global-Invariant
Surrogate F Feature Generator
feature
textueature
Surrogate text features
Lea/ Lissartunt
ID
Surrogate
conditional signals
Visura Encoder
ID lext
Text Encoder
prompt tokens
MLP/IN
feature
Lct
Surrogate F Feature
Generator
Pro
Pro
Proj
Basic
class tokens
LN
LN
prompted text feature
FG visual features
Surrogate FG text features
FG conditional signals
Basic prompted text feature
Conditional signals
Figure 2. Pipeline of our proposed Surrogatc Prompt Learning (SurPL). In this work, mainly generate two types of diverse prompted text features: instance-dependent features and fine-grained features. Notably, "MD" "FG" denote "instance-dependen" and "fine-grained" in the figure, respectively.
ture Mxd corresponding to can be derived as:
prompted text features via SFG. Concretely, for each visual instance Ø6, directly adopt the output visual feature fo V(ars) as the instance-dependen conditional signal o The surrogate instance-dependent (ID) text feature ID I xx can thus be expressed as: h OSFG(w, ob)) h encapsulates specific knowledge w.r.t œ6, thereby providing more accurate description to the visual instance. Finally, the corresponding instance-dependen optimization loss C is computed as:
wa
h (w, a) MLP ssotmaa
dd
(4)
where OSFG represents the parameters of SFG. For simplifi- cation, we omit the necessary feature projection (Proj) and layer norm (L.N operations in Eq.4, and provide detailed computation procedures in the appendix.
SFG uses conditional signals as guidance, the generated text feature obtains specific knowledge related to the given signal. Leveraging diverse condition signals can generate diverse text features that comprehensively align with visual concepts in downstream tasks from different perspectives. This operation avoids learning diverse prompts from scratch, hence addressing the efficiency problem in existing diverse prompt learning methods. In the following, we provide the concrete implementations of instance-dependent and fine-grained surrogate prompt learning.
B
CE
Σ ym log Pu (m)(maaaa)) (ml]zz)
B
(5)
exp(cos( 1 b//m)/T) j= exp(cos( f (b)))//))
where
ID (m|w)))
3.5. Fine-Grained Surrogate Prompt Learning
The output of visual example from the pre-trained VLM visual encoder is actually constituted by basic (global) visual feature f E Rd and N fine-grained (local) visual features f {f N I==0 E RNxd While traditional works typically utilize f as the representation of visual concepts, recent approaches start to fully exploit effective fine- fine-grained visual features embedded in f. and learn additional fine- grained (FG) text prompts to align with them. Suppose the number of newly involved prompts Z. Fine-grained prompt learning increases the computation cost of gradient pack-propagation in the text encoder by Z times compared to single text prompt learning methods.
3.4. Instance-dependent Surrogate Prompt Learning
Instance-dependeent (ID) prompt learning targets to embed visual information into text prompts, so that the prompted text features can dynamically adapt according to the input visual instances. Given of visual instances X { B (B denotes the batch size), instance-dependent prompt learning involves Bx computation cost of gradient back- propagation in the text encoder compared to single text prompt learning methods. This implementation drastically decreases the optimization efficiency, especially when using a sufficiently large batch size.
Since fine-grained information does not require prior knowl- edge, we simply pre-define the fine- fine- see-grained (FG) condi- tional signals learnable parameters: FG 2xd FG
To address this issue, we generate the instance-dependent

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
can automatically capture fine-grained cues during the op- timization procedure. The surrogate fine-grained (FG) text features h FG (h [
Lup= (0,0,,00r.... ,m,))) ) LEE + CE CE
PG
(9)
) ===1 E 2 xd can be expressed as:
During the inference phase, the final prediction probability of æ belonging to the m-th class is derived as:
h. = S aD, a ).
Following GalLoP (Lafon et al., 2024), we excavate ef- fective fine-grained visual features from f by measuring the similarity between f and h h Refer to recent works (Sun et al., 2022; Zhou al.. 2022a; Lafon et al., 2024), f E Nxd is obtained by forwarding them through the last visual encoder block, where the self-attention operation is removed and a linear projection layer is added at the end. Concretely, for the 2-th fine-gras grained text feature FG we se- lect the top 2 X 17 visual features with the highest similarities to h as the representative fine-grained visual information. Here 7) is a constant factor, which enables the selected fine- grained visual features different and multi-scaled for each FG The corresponding similarity between h and f of the m-th class is given by average similarities of the top X 1) regions:
Z
1 p(m ar) = (p (m|ar) + th (m]aa) +
Σ
/a)/ /3, P2
FG
Z
(10)
where p (0 |ar) indicates the prediction probability be- tween whdl and f.
Generalization ability of SurPL: Recent studies begin to explore the generalization ability of the learned prompts, and evaluate their performances on unseen tasks. To this end, we further propose a generalizable version of SurPI., denoted as SurPL-G. As many previous studies (Keskar et al., 2016; Dziugaite & Roy, 2017; Jiang et al., 2019) have manifested that flat loss landscape leads to better generaliza- tion ability, SurPL-G leverages Sharpness-awar Minimiza- tion (SAM) (Foret et al., 2021) to optimize the parameters Φ
{s, u. OSFG, a 0p
FG f) sim(h
znp=z(()) cos(h fn),
2 X 17
1,
if rank, (cos PG s,m fm)) S 27,,
CSAM = Late = C Fce c
(11)
where
Lop-sy(n))
0,
otherwise.
where Lo is computed according Eq.9 and p is the pertur- bation radius. This implementation encourages the neigh- boring parameters of Φ:: (Φ e) to have uniformly low loss values, thereby promoting a flatter loss landscape and en- hancing the generalization ability of SurPL-G. We provide the detailed algorithms of SurPL and SurPL-G in appendix.
(6)
Finally, we compute the fine-grained loss Ceg as:
2
CEE = 2 1=1
Σ
Um log p. (m|zz)
(m) [m)
m
exp(sim(htt. PG
2 m. f)/T)
where
2 (m|#))
4. Experiments
M j=1 exp(sim(h z.j f))/T)
PG
Dataset. To evaluate the effectiveness of SurPL, this paper exploits 15 public available visual classification datasets as downstream tasks, including ImageNet (Deng et al., 2009), Caltech101 (Fei-Fei et al., 2004), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers (Nilsback & Zisserman, 2008), Food101 (Bossard al., 2014), FGVCAireraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), UCF101 (Soomro et al., 2012), ImageNetV2 (Recht et al., 2019), LmageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021 These datasets constitute a com- prehensive benchmark.
3.6. Optimization and Inference
As shown in 2,g.2, besides the generated surrogate features ID and FG we also learn a global-invariant (GI) text fea- ture Gt based on w. which can provide the general and global text description. The global-invariant loss can be expressed as:
CC = LCE + Linvariant
(8)
= LCE + 11 - wh + - 8%)
where Leg indicates the cross-entropy loss calculated be- tween Gi and f according to Eq. w² and fd denote the text and visual features obtained from zero-shot text and visual encoders without prompts, Xı and X2 are the coefficients. During the optimization phase, the optimiz- ing parameters & include the dense text prompts s, dense visual prompts u. the Surrogate Feature Generator θSFG. fine-gratd grained conditional signals a PG and the linear projec- tion layer Pro to obtain f. The entire loss summed by:
Implementation details. We adopt Dense Visual- Language Prompt (DVLP) as the baseline model. We mainly evaluate our proposed method on 4 types of experiment settings: SurPL on few-shot learning, SurPL-G on base-to- novel generalization, cross-domain and cross-dataset gener- alization. Most of the implementation details are followed by PSRC (Khattak et al., 2023b). For SurPI. and SurPI-GG,
5

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table 1. Efficiency comparison between SurPI. and existing di- verse prompt learning methods, including CoCoOp(Zhou et al.,
Table 3. Ablation study of SurPI. few-shot setting and SurPI.-G on base-to-nove (B2N) setting. 'GI', 'ID' and "FG" denote lever- aging w htd and h tively. Averaged results on 11 datasets are reported here. Complete experiment results for individual datasets are given in appendix. HM" denotes the harmonic mean between base and novel results.
2022b). (ChT+ et al., 2023), ALagn et al., 2023) and GalLoP(Laaon et al., 2024). 'OOM" indicates that GPU Out
describe the visual concepts, respec-
of Memory on xx RTX3090 GPUs
Methods
Memory
Training
Testing
Acc
DVLP
GI
GI+ID
GI+FG
GI+ID+FG
EuroSAT
CoCoOp
7.79G
0min42s
1min17s
73.32
+LOT+
13.02G
0min53s
0min38s
92.00
Few-shot
82.92
83.70
83,96
84.99
85.12
ALIGN
14,00G
0min41s 1s
1min40s
90.77
GalLoP
4.69G
0min28s
0min24s
90.10
Base
79.61
81.84
84.77
85.52
86.37
SurPI.
5.60G
0min22s
0min22s
93.92
B2N
Novel
71.63
74.68
76.19
75.50
76.32
HM
75.41
78.09
80.25
80.20
81.03
CoCoOp
OOM
31,21
Aircraft
PLOT+
20.50G
8mini21s Is
0min21s
46.74
ALIGN
23.37G 31min23s
4min43s
49. 49.99
Gall.oP
18.77G
5min52s
0min23s
58.30
SurPL
7,37G
2min36s
0min 12s
60.51
time and few-shot learning accuracy. Compared these diverse prompt learning methods, SurPL significantly re- duces computational consumption in terms of GPU memory, training time, and testing time, while even simultaneously improving accuracy. These results indicate that SurPL has achieved our primary goal: improving the computation effi- ciency for diverse prompt learning.
ImageNet
CoCoOp
OOM
70.83
PI.OT+
OOM
72.60
ALIGN
OOM
72.45
GalLoP
OOM
75.10
SurPI.
23.80G
8min03s
6min21s
74.65
Table 2. Efficiency comparison between SurPL and existing single prompt leamning methods, including MaPLe( et al., 2023a),
Notably, for large-scale datasets such ImageNet, we ob- serve that existing diverse prompt learning approaches re- quire extremely enormous GPU memory. We also notice that the efficiency issue cannot be fully solved by parallel multiple GPUs, since all prompted text features should al- ways be repeatedly computed on each GPU, As we estimate, these methods require at least 80 100G memory for to- tal, and each GPU should afford 40 50G memory. In contrast, SurPL can conduct experiments on ImageNet with single 24G RTX 3090 GPU, which significantly improves the computation efficiency.
PSRC(Khattak et al., 2023b) and our baseline model DVLP.
Methods
Memory
Training
Testing
Acc
SAT
MaPLe
5.14G
0min15s
0min12s
92.33
PSRC
5,80G
0min17s
0min
92.43
DVLP
5.60G
0min21s
0min21s Is
92.58
SurPL
5.60G
0min22s
0min22s
93.92
Aireratt
MaPLe
6.90G
1min31s
Omin 10
48.40
PSRC
7.65G
1min43s
Omin OOs
50.83
DVLP
7.33G
2min29s
Omin 12s
52.88
SurPI.
7.37G
2min36s
Omin 22
60.51
ImageNet
MaPL.e
22.68G
5min53s
4min22s
72.33
PSRC
23.10G
6min14s
4min31s
73.17
Furthermore, we compare SurPL. with recently proposed single prompt learning methods. As shown in Tab.2, SurPL achieves significant improvements on accuracy, with only a slight increase in training and testing time. Notably, we find that the GPU memory consumption remains comparable to these single prompt learning methods, which highlights the computation efficiency of our proposed SurPL.
DVLP
23.30G
7min44s
5min48s
72.62
SurPL
23,80G
8min03s
6min21s
74.65
we utilize batch size B- = 4 for base-to-novel generalization, and batch size B- = 32 for other three settings. Similar to GalLoP (Lafon et al., 2024), the number of fine-grained text features and multi-scale constant factor are set as Z = 4 and 77 = 10. All experiments are conducted on single RTX 3090 GPU, All results are obtained with CLIP ViT-B/16 backbone and 16-shot visual samples unless specified. More detailed implementations are given in appendix.
4.2. Ablation Study
SurPL learns three kinds of text features: global-invariant (GI) text feature w the surrogate instance-dependent (ID) text feature ID and the surrogate fine-grained (FG) text feature PG These text features jointly provide a compre- hensive description the visual concept, thereby achieving compelling performance improvements over the baseline DVLP. We conduct ablation study verify the effective- ness of G 10 and H under both few-shot learning and noseell generalization settings.
4.1. Efficiency Analysis of SurPL
As shown in I. we compare the computation-effiicinns between SurPL and existing diverse prompt learning meth- ods, under three datasets that have different scales: Eu (10 classes), FGYCAireraft (100 classes) and ImageNet (1000 classes). We set the batch size B = 32 as default and report the results of GPU memory, training time (1 epoch for ImageNet and 10 epochs for the other two datasets), testing
As shown in Tab.3, each of the applied text features Gw h li and h H9) consistently improves the performance under both experiment settings. These results verify that learning
6

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table 4. Few-shot learning performance comparison between our proposed SurPI. and existing single prompt learning (CoOp(Zhou al.,
2022c). MaPLe(Khattak et al., 2023a), DAPT(Cho et al., 2023), PSRC(Khattak et al., 2023b) and LLaMP(Zheng et al., 2024)) and diverse prompt learning (CoCoOp(Zhou et al., 2022b), PLOT++(heen al., 2023), ALIGN(Wang et al., 2023) and GalLoP(Lafon et al., 2024)) approaches. The bold numbers and underlined numbers denote the best and second-best results, respectively.
Methods
ImageNet
Caltech
Pets
Cars
Flowers
Food101
Aireraft Sum397
DTD
EuroSAT
UCF101
AVG
CoOp
71.87
95.57
91.87
83.07
97.07
84.20
43,40
74.67
69.87
84.93
82.23
79.89
Single
MaPLe
72.33
96.00
92.83
83.57
97.00
85.33
48.40
75.53
71.33
92.33
85.03
81.79
DAPT
72.20
95.82
92.27
83.03
97.06
86.55
46.37
75.99
71.38
92.65
84.53
81.62
PSRC
73.17
96.07
93.67
83,83
97.60
87.50
50.83
77.23
72.73
92.43
86.47
82.87
LLaMP
73,49
97.08
94.21
86.07
98.06
87.62
56.07
77.02
74.17
91.31
86.84
83.81
Diverse
CoCoOp
70.83
95.16
93.34
71.57
87.84
87.25
31.21
72.15
63.04
73.32
78.14
74.90
PLOT+
72.60
96.04
93.59
84.55
97.56
87.11
46.74
76.03
71.43
92.00
85.34
82,09
ALIGN
72.45
96.00
94.17
86.75
96.57
86,90
49.99
76.57
71.40
90.77
85.69
82,48
GalLoP
75.10
96.70
94.10
89.20
98.80
86.50
58.30
77.20
75.50
90.10
86,90
84.
SurPL
74.65
96.92
94.22
89.00
98.84
87.63
60.51
77.67
74.75
93.92
88.18
85.12
to existing single and diverse prompt learning approaches, SurPL obtains the best average results over 11 datasets. By jointly exploiting different kinds of text features, we surpris- ingly observe that SurPL even surpasses the late-of-the-arr fine-grained-bare method GalLoP (Lafon et al., 2024) by 0. 0.72%, while significantly improving the computation effi- ciency. Beyond the average results, SurPL. also achieves the best performances on 7 out of 11 datasets and ranks within the top-2 on all datasets. Notably, for datasets such as FGVCAircraft, EuroSAT and UCF101, SurPL demonstrates substantial improvements over gains of 2.21%, 1.27% and 1.28%, respectively. The results in Tab.4 clearly show the superiority of SurPI., and verify that the surrogate text features can effectively substitute for original diverse prompted features.
Table 5. Base-to-novel generalization performance comparison be- tween our proposed SurPL-G and existing generalizable prompt learning approaches. For fair comparisons, we only list existing inductive methods in the table, without considering the transduc- tive approaches that leverage extra models (e.g., stronger VL L.LMs) or data improve the generalization ability. Averaged results on 11 datasets are reported here. Complete experiment results for individual datasets are given appendix.
Base
Novel
HM
CLIP (Radford et al., 2021)
68.21
73.36
70.69
CoOp (Zhou et al., 2022c)
82.68
64.15
72.25
results, with
CoCoOp (Zhou al., 2022b)
82.56
64.66
72.52
ProGrad (Zhu et 2023)
82.48
69.12
75.21
KgCoOp (Yao et al., 2023)
81.94
72.52
76.94
MaPLe (Khattak et al., 2023a)
82.28
75.14
78.55
PSRC (Khattak et al., 2023bb
84.24
75.68
79.73
ALIGN (Wang et al., 2023)
83.39
75.51
79.25
TCP (Yao et al., 2024)
84.13
75.36
79.51
Base-to-novel Generalization. Tab.5 compares the gen- eralization ability of SurPL-G with existing generalizable prompt learning methods under the base-to-novel gener- alization setting. We observe that SurPL-G achieves the highest average results over 11 datasets for both base and novel classes, thereby improving the generalization trf results (Harmonic Mean, HM) by 0.61% over the state-of- the-art approach DePT (Zhang et al., 2024a). Specifically, SurPL-G also achieves the best performances on out of 11 datasets. For datasets such as FGVCAireraft, EuroSAT and UCF101, SurPL-G demonstrates substantial improvements over state-of-the thefthhhartt art results, with gains of 1.46%, 2.60% and 1. 1.28%, respectively. The above detailed results are pro- vided in the appendix. The results in Tab.5 indicate that by simply incorporating the SAM optimization strategy with SurPL, SurPL-G exhibits compelling generalization ability.
DePT (Zhang et al., 2024a)
85.18
76.17
80.42
SurPL-G
86.37
76.32
81.03
diverse text features from different perspectives can signif- icantly benefit the VI.M's adaptation ability. Specifically, we observe that the surrogate instance-dependent feature ID and fine-grained feature h FG drastically boost VLM's generalization (base-to-novel results) and discriminative (few-shot results) ability, respectively. This observation manifests that different generated surrogate text features en- hance VLM's ability from different perspectives, which also indicates that 1) the generated surrogate text features are suf- ficiently effective to replace the original diverse prompted text features; 2) our proposed Surrogate Feature Genera- tor is flexible enough to simultancously generate different diverse prompted text features efficiently.
Cross-dataset Generalization. We further explore the gen- eralization ability of SurPL-G under the cross-datase set- ting. where prompts are learned on ImageNet (source do- main) and evaluated on 10 other datasets (target domain). As shown in Tab.6, SurPL-G achieves the best performances on both source and target domains compared to existing
4.3. State-of-the-art Comparison
Fhw-shot learning. As shown in Tab.4, SurPL achieves remarkable few-shot learning performances. Compared
7

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table Cross-dataset generalization performance comparison. Models learned ImageNet (source domain) and evaluated 10
other datasets (target domains). Averaged results of the 10 datasets are reported.
| CoOp
CoCoOp
ProGrad
KgCoOp
PLOT
MaPLe
PSRC
DePT
TCP
SurPL-G
Source
| 71.51
71.02
72.24
70.66
71.60
70.72
71.27
71.60
71.40
73.33
Target (average)
63.88
65.74
62.66
65.51
64.34
66.30
65.81
66.02
66.29
66.61
Table 7. Cross-domain generalization performance comparison. V2, Sketch, A and R indicate the ImageNetV2, ImageNet-Sketch, ImageNet-A A and ImageNet-R R datasets, respectively.
Feature similarity
0.9
0.8
0.7
Source
Target
0.6
ImtgeNet
V2
Sketch
A
AVG
0.5
CoOp
71.51
64.44
47.61
49.53
74.98
59.14
CoCoOp
71.02
64.07
48.75
50.63
76.18
59.91
CLSA1 CLS82 CLSB3 CLSW CLSHS CLSW6 C1S87 CLSHS CLSe9 CLSH1
ProGrad
72.24
64.27
48.10
49.72
75.84
59.48
KgCoOp
70.66
64.10
48.97
50.69
76.70
60.12
MaPLe
70.72
64.07
49.15
50.90
76.98
60.28
DAPT
72.20
64.93
48.30
48.74
75.75
59.43
PSRC
71.27
64.35
49.55
50.90
77.80
60.65
Figure 4. Comparison of normalized feature cosine similarity be- tween cos( f. e and cos 11 across different classes on the UCFI01 dataset.
TCP
71.20
64.60
49.50
51.20
76.73
60.51
SurPL-G
73.33
66.59
50.45
50.00
78.45
61.37
multi-scale description of the visual concept from coarse to fine. These evidences highlight the high quality of our surrogate text features, further validating the effectiveness of the proposed SurPL.
--1
2
Effectiveness of the surrogate instance-dependent text features ht" To explore the effectiveness of h we com- pute the normalized cosine similarity between h1th and the basic visual feature f. denoted as cos( f. h h² Specifi- cally, we randomly sample 10 classes from a dataset and report the average similarity across all visual samples within each class. We also perform the same operation between f and the global-invarian text feature w cos( w As shown in 4ig.4, the comparison results highlight that the sur- rogate instance-dependent text feature hD provides signifi- cantly better alignments with the visual concepts, indicating that ID effectively captures instance-level knowledge via the proposed Surrogate Feature Generator.
A photo of Bengal cut
3
4
Figure 3. Visualization results of the attention heat maps of differ- surrogate fine-grained text features hG Rig 1-
methods. This comparison indicates that SurPL-G exhibits remarkable generalization trade-off.
Cross-domain Generalization. We also investigate the generalization ability of SurPL-G under the cross-domain setting, where prompts are learned on ImageNet (source domain) and evaluated on 4 ImageNet-based domain-shifted datasets (target domain). The results illustrated in Tab.7 indicate that SurPL-G reaches significant improvements both source and target domains over the state-of-the-arts.
5. Conclusion
This paper proposes a novel Surrogate Prompt Learning (SurPL) framework to enable efficient and diverse prompt learning for vision-language models. Instead of learning diverse text prompts from scratch, SurPL directly gener- ates their prompted text features via a lightweight Surro- gate Feature Generator, achieving remarkable adaptation performances while maintaining efficiency comparable to single-prompt learning methods. By configuring appropri- ate conditional signals, SurPL. is adaptable and integrable to implement different diverse prompt learning ideas based on any single prompt learning approach. We hope this work
4.4. In-depth Analysis of Surrogate Features
Effectiveness of the surrogate fine-grained text features FG As shown in Fig.3, we visualize the attention heat map of different features {h FG Z J==1 on visual concept. The results manifest that 1) our surrogate fine-grat fine-grained features accurately capture the important information of the visual concept; 2) our surrogate fine- mn--graieed features provide
8

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
can bring some inspiration to the related fields.
L. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 248-255, 2009,
Limitations and future works. 1) The Surrogate Feature Generator (SFG) introduced in this paper is inherently cross-attention module, which is relatively outdated and inevitably increases the parameter sizes. We aim to design more performant and efficient variants in the future. 2) This paper mainly incorporates existing diverse prompt learning frameworks SurPL. We will further explore effective information that can boost rision-language alignment from new perspectives.
Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural net- works with many more parameters than training data. arXiv preprint arXiv: parXivv:77::
/703. / 1008, 2017.
Fei-Fei, L.,, Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004.
Acknowledgements
NNW was supported part by the National Natural Sci- ence Foundation of China under Grants U22A2096 and 62036007, in part by Scientific and Technological Inno- vation Teams in Shaanxi Province under grant 2025RS- CXTD-011, in part by the Shaanxi Province Core Tech- nology Research and Development Project under grant 2024QY2-GJHX in part by the Fundamental Research Funds for the Central Universities under GrantQTZZ22042.. TLL is partially supported by the following Australian Research Council projects: FT220100311 DP220102121, LP220100527, LP220200949, and IC190100031.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R.. Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision- lision-language models with feature adapters. International Journal of Computer Vision, 132(2)5811555. 2024.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2225 2019.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M.. et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEF/CVE International Conference Computer Vision, pp. 8340-8349, 2021a.
References
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food- 101- mining discriminative components with random forests. In European Conference Computer Vision, pp. 446- 461, 2014.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2021b. 2021b..
Chen, G., Yao, W., Song, X., Li, X., Rao, Y., and Zhang, K. PLOT: Prompt learning with optimal transport for vision-language models. In International Conference on Learning Representations, 2023.
Pattern Recognition, pp. 15262-15271,
Jia, C., Yang. Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMIR, 2021.
Cho, E., Kim, J., and Kim, H. J. Distribution- prompt tuning for vision-lanne languag models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 22004-22013 2023.
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 191..0::112.027... 2019.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-bator training for deep learning: Generalization gap and sharp minima. arXiv preprint 1609.046600044 2016.
Deng, J., Dong, W., Socher, R., Li, L.JJ., Li, K., and Fei-Fei,

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021.
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. S. Self-regulating prompts: Foun- dational model adaptation without forgetting. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15110011220) October 2023b.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In Interna- tional Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Roy, and Etemad, A. Consistency-guideed prompt learning for vision- lision-language models. International Conference on Learning Representations, 2024.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for ine-grained categorization. In Pro- ceedings of the IEEF/CVF International Conference on Computer Vision Workshop, pp- 554-5514 2013.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. 2012.
Lafon, M., Ramzi, E., Rambour, C.,, Audebert, N., and Thome, N. Gallop: Learning global and local prompts for vision-language models. European Conference on Computer Vision, 2024.
Sun, X., Hu, P., and Saenko, K. Dualcoop: Fast adapta- tion to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems, 35: 30569-30582, 2022.
Li, J., Gao, M., Wei, L., Tang, S., Zhang. W., Li, M., Tian, Q. Chua, T.-S., and Zhuang, Y. Gradient-regulate meta-prompt learning for generalizable vision-language models. Proceedings of the IEEE/CVE International Conference on Computer Vision, 2023a.
W., W.,
Wang, D., Li, M., Liu, X., Xu, M., Chen, B., and Zhang, H. Tuning multi-mode token-level prompt alignment across modalities. Advances in Neural Information Processing Systems, 36, 2023.
Li, M., Zhong, J., Li, C., Li, L., Lin, N., and Sugiyama, M. Vision-language model fine-tuning via simple parameter- efficient modification. In Conference on Empirical Meth- ods in Natural Language Processing, pp. 14394-14410, 2024.
Wang. H., Ge. S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predic- tive power. Advances in Neural Information Processing Systems, 32, 2019.
Xiao, J., Hays, Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3485- 3492, 2010.
Li, X., Lian, D., Lu, Z., Bai, J., Chen, Z., and Wang, X. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in Neural Information Pro- cessing Systems, 36, 2023b.
Liu, L., Wang, N., Zhou, D., Gao, X., Liu, D., Yang, X., and Liu, T. Gradient constrained sharpness-aware prompt learning for vision-lanm language models. arXiv preprint arXiv:2309.07866 2023.
Xu, C., Shen, H., Shi, F., Chen, B., Liao, Y., Chen, X., and Wang, L. Progressive visual prompt learning with contrastive feature re-formation. arXiv preprint arXiv:2304.08335 2023.
Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6757-6767. 2023.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.575/ 2013.
Nilsback, M.-F. and Zisserman, A. Automated flower clas- sification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729, 2008
Yao, H., Zhang, R., and Xu, C. Tep: Textual-based class- aware prompt tuning for visual-language model. In Pro- ceedings of the IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 23438-23448, 2024.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 2012.
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., and Xu, C. Filip: Fine-grained interactive language-iman -mage pre-training. arXiv preprint 1 2021.
10

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Yu, T., Lu, Z., Jin, X., Chen, Z., and Wang. X. Task residual for tuning vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10899-10909, 2023.
Zanella, M. and Ben Ayed, I. Low-rank few-shot adapta- tion of vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshop, pp. 1593-1603, 2024.
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123-18133, 2022.
Zhang, Wu, S., Gao, L., Shen, H. T., and Song, Dept: Decoupled prompt tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12924-12933, 2024a.
Zhang. R., Fang, R., Zhang, W., Gao, P. Li, K., Dai, Qiao, Y., and Li, H. Tip-adapter: Training-fre free clip-adapter for better vision-language modeling. European Conference on Computer Vision, 2022.
Zhang, Y., Zhou, K., and Liu, Z. Neural prompt search. IEEE Transactions Pattern Analysis and Machine In- telligence, 2024b.
Zhao, C., Wang, Y., Jiang, X., Shen, Y., Song, K., Li, D., and Miao, D. Learning domain invariant prompt for vision-language models. IEEE Transactions Image Processing, 2024.
Zheng, Z., Wei, J., Hu, X., Zhu, H., and Nevatia, R. Large language models good prompt learners for low-shot image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
pp.
Zhou, C., Loy, C. and Dai, B. Extract free dense labels from clip. In European Conference Computer Vision, pp. 696-712, 2022a.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Conditional prompt learning for lingua-angguaee models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816-16827 2022b.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-langn languge models. International Jour- nal of Computer Vision, 1330))23337-2488, 2022c
Zhu, B., Niu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023.

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
A. Detailed Implementations of Surrogate Feature Generator (SFG)
The parameters of Surrogate Feature Generator OSG include three input projection layers 0int, Oin2, 0n3. one output projection layer our and two fully connected layers 0 fel . fc2. Given the basic prompted text feature w and the conditional signal a, the input Query, Key and Value Matrix of the cross attention can be obtained by:
Q LN(w)) 0in1:
= (N m22
(12)
V (a) 0n3:
where LN is the layer norm operation. Then, the cross attention result Attn can be derived as:
QKI
Alln = w (so[lmax
-Oout-
(13)
vdz
sm11 in2 O in: Oott Rdxd where d denotes the feature dimension. Finally, the surrogate feature is obtained by passing the attention results through two-layer MIP:
h = Attn + ReLU (N((t)t) pma) f
(14)
where ffc E Rddddi and fe2 E R xd, To keep SFG parameter- we set dmid = d/2.
Notably, although our proposed SurPL significatly boosts the computation--effccenns the SFG module inevitably involves extra learnable parameters. Therefore, apart from the mainly concerned computation-efficieney problem, we also discuss the purameter-efficiengg of SurPL here. Tab.8 illustrates the parameter size comparison between SurPL and existing methods. We find the total parameter size of SurPL. remains acceptable. In fact, it is even smaller than some other prompt learning approaches e.g., MaPLe (Khattak et al., 2023a) and ALIGN (Wang et al., 2023).
Table 8. Parameter size comparison between SurPI. and existing methods (CoCoOp(Zhon et al., 2022b), PL.OT+(Chen et al., 2023), ALIGN(Wang et al., 2023), MaPte(Khattak al., 2023a), PSRC(Khattak et al., 2023b), Gall.P((Laaoo al., 2024)).
Method
CoCoOp
PLOT+
ALIGN
MaPLe
PSRC
GalLoP
DVLP (baseline)
SurPL
Params
0.035M
0.014M
3.582M
3.556M
0.046M
0.606M
0.061M
1.641M
B. Computation Complexity Analysis of SurPL
Here, we provide formal analysis of computational complexity of our proposed method SurPL.
Notations. loss C. classnames c = fcm |M Text encoder parameter: T = [TT
k== [[T] SFG parameter: fSFG, Text prompts parameter: = {** [ Encoder layer depth: K, the parameter size of SFG is much smaller than that of text encoder: 1
OSFG << T.
Derivation. Optimizing the text prompt s at the text encoder input position requires computing the gradient through the entire model. For each single surrogate output text feature, the back back-propagatton computation of SurPL can be written as:
Oc TsT 00 ( (IsK ct a))))
Oc
d0 sFs ([s" ct D:a))
OT ([s
K
OT ([s),cu.
(15)
UT "T*( ch.)
OTK- ((s
Øs
SFG
Text Encoder
This gradient computation can be conceptually divided into two parts, which correspond to the text encoder and SFG, respectively. We denote the computational complexity for each output text feature as Of and OSEG.
As shown in 1 (c), SurPL simultaneously involves M text features back-propagant w.r.t text encoder, which equals to the complexity in single prompt learning: (M) = M Or. For SFG, (B - Z)M text features are involved computation. The overall complexity of SurPL can thus be expressed as:
Sur PL = M Or + (B + Z)M OSFG
(16)
Since Asro << T, it follows that OSE << Or. Therefore, the second term is negligible, and we can approximate:
O OurPI M Or = O(M).
12

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table Performance comparison between SurPI. and SurPI.-G under the base-to-novel generalization experiment setting.
AVG Novel
ImageNet Novel
Caltech101 Novel
Base
HM
Base
HM
Base
HM
SurPL SurPL-G
86.35 86.37
71.77 76.32
78.39 81.03
78.78 78.74
68.25 70.49
73.14 74.39
98 88.84
94.76 95.16
96,76
98.77
96.93
OxfordPets Novel
StanfordCars Novel
Flowers Novel
Base
HM
Base
HM
Base
HM
SurPL SurPL-G
95, 35,96
96.92 97.41
96.44
85.05 83.57
68.95 72.77
76.16 77.80
98.77 98.90
70.19 72.88
82.06
96.37
96.89
83.92
Food101
FGVCAircraft Novel
SUN397 Novel
Base
Novel
HM
Base
HM
Base
HM
SurPL SurPL-G
90.27 90.92
89.97 91.81
90.12
49.52 49.20
12.96 36.93
20.54 42.19
83.19 83.43
76.55
79.73
91.36
78.96
81.13
DTD Novel
EuroSAT Novel
UCF101 Novel
Base
HM
Base
HM
Base
HM
SurPI. SurPL-G
85.26
55.84
67.48
95.19 94.63
77.28 81.33
85.31
89.06 89.44
77.84
83.07
86.07
62.04
72.11
87.48
79.74
84.31
C. SurPL-G: Generalizable version of SurPL by Sharpness-aware Minimization
Recent prompt learning works start to explore the generalization ability of the learned prompts. However, we observe that SurPL exhibits relatively poor generalization performances on unseen tasks. SurPL learns diverse text features to describe visual concepts from different perspectives. While these learned features, especially the surrogate fine-grained features, significantly improve the VLM's discrimination ability, they are also prone to overfitting when trained with Empirical Risk Minimization (ERM), i.e., cross-entropy loss.
To address the above issue, we further propose generalizable version of SurPL by exploiting sharpness-aware minimization (SAM) optimization strategy (Foret et al., 2021), denoted as SurPL-G. Given the theoretical background that flatter loss landscape yields better model generalization ability, SurPL-G further penalizes the loss sharpness by forcing the neighboring parameters of Φb: (Φ + E) to have the uniformly low loss values, and the corresponding SAM-based optimization loss can be interpreted by:
min
L LSAM where L CSAM
Ca+er ||ell.Sp
(17)
where the neighborhood region controlled by the perturbation radius p. Eq. can be explicitly solved by approximately finding the neighboring parameter (Φ 1 E) that has the maximum loss value using first-order Taylor expansion:
VLp
= argmax Lote R8 argmmax Lob + V مe
(18)
||e|l>Sp
Taking è back 17, we finally obtain the optimization loss for SurPL-G as shown in Eq.
We conduct the comparison experiment between SurPL (optimized via ERM) and SurPL-G (optimized via SAM) under the base-to-novel generalization setting. The results demonstrated in Tab.9 indicate that utilizing the SAM optimization strategy can significantly improve the performances on novel classes. This implementation directly endows SurPL with generalization capabilities without requiring any other modifications.
D. Algorithms of SurPL and SurPL-G
We provide the pseudo-code SurPL and SurPL-G in Algorithm.1, which demonstrates the comprehensive and detailed optimization and inference procedures.
E. Implementation Details of Experiments
Datasets. This paper adopts 15 public available visual classification datasets as downstream tasks, including ImageNet (Deng et al., 2009). Caltech101 (Fei-Fei et al., 2004), OxfordPets (Parkhi et al., 2012). StanfordCars (Krause et al., 2013), Flowers100 (Nilsback Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAireraft (Maji et al., 2013), EuroSAT
13

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Algorithm The optimization and inference procedures of SurPL. and SurPL-GG
Input: Pre-trained VLM with visual-encoder // and text encoder T, training visual sample: (Ztrain 8), testing visual example Tteat, class names с = {cm mm text prompt 8, visual prompts u, Surrogate Feature Generator AsrG. fine-grained conditional signals a linear projection layer Projj training epochs &, perturbation radius p.
Initialize the learnable parameters: Ø {s, u, OSFG; 2 Oproj
while Optimization phase do for i 1,2,, ...., & do
Obtain the basic prompted text feature for each class: Wm. = T[(ss, cml))
Obtain basic and fine-grained visual features: f:J. (( (attrann u4]; Proj)
Obtain the global-invariant text feature: w GT = Wm
Obtain the mestance-dependent conditional signal a =
Generate the surrogate instance-dependens text feature hD hib = Osro as)) Generate the surrogate fine-grained text features h RG; te Os ot Calculate Cgl L and L via Eq.8, Eq.5 and Eq.7, respectively if SurPL then
11
Optimize Φ {s, u, fsro, a PG Oproj} via Eq.9 if SurPL-G then Optimize o = {s, u, Osra, a PG p via Eq.11
12
13
14
15
end
Output: The optimized parameters of {s, u, OSFG, aG Oproj)
16
end
17
while Inference phase do
18
Repeat step 4 on Frest
19
Calculate ndl (ml (test)) pd) (test) and p= (m|[teest) ) via Eq.1, Eq.5 and Eq.7. respectively
20
Calculate the classification probabality ((m|aree) via Eq.10
Output: The classification probabality p(m (test).
end
(Helber et al., 2019), SUN397 (Xiao et al., 2010), DTD (Cimpoi al., 2014), UCF101 (Soomro al., 2012), ImageNetV2 (Recht al., 2019), ImageNet-Sketch (Wang al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021a). These datasets constitute a comprehensive benchmark, which includes classifications of generic objects, scenes, actions, satellites, textures and fine-grained categories. Detailed statistics of the datasets are given in Tab.
Experimental Settings. We evaluate the effectiveness the proposed prompt learning method under four experimental settings:
Few-shot learning: we optimize the parameters of SurPL on few-shot training examples of the downstream task, and evaluate the optimized SurPL. by classifying testing data that stem from the same classes
Base-to-novel generalization: we split each downstream task into disjoint base and novel classes. The parameters of SurPL are optimized with few-shot training samples of base classes, and evaluated on testing data from novel classes.
Cross-domain generalization: we optimize the parameters of SurPL on few-shot training examples of ImageNet, and evaluate the optimized SurPL on four Nomain-shifted datasets: ImageNetV2, ImageNet-Sketch, ImageNet-A and ImageNet-R. Notably, these datasets share the same classnames.
Cross-dataset generalization: we optimize the parameters SurPL on few-shot training examples of ImageNet, and evaluate the optimized SurPL on other datasets: Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAireraft, SUN397, DTD, EuroSAT, UCF101.
Implementations. This paper adopts CLIP-VIT-B/IE as the pre-trained VLM for research. DVLP is utilized as the baseline model. The depth and token-length of both text prompts and visual prompts u are set to K 12 and by default. All the prompt tokens are initialized by sampling from a zero-mean Gaussian distribution, except for the first-layer text
14

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table 10, Detailed statistics of the evaluated datasets.
Dataset
Task
Classes
Training Size
Testing Size
ImageNet
Object recognition
1000
1.28M
50000
10
Object recognition
100
4128
2465
OxfordPets
Fine mea-igra pets recognition
37
2944
3669
StanfordCars
Finc-grainee car recognition
196
6509
8041
Flowers 102
Fine-graine ane-grained flowers recognition
102
4093
2463
Food101
Fine-grained food recognition
101
50500
30300
FGVCAircraft
Fine-grained aircraft recognition
100
3334
3333
SUN3397
Scene recognition
397
15880
19850
DTD
Texture recognition
47
2820
1692
EuroSAT
Satellite image recognition
13500
8100
UCFI01
Action recognition
101
7639
3783
ImageNet-V2
Robustness of collocation
1000
N/A
10000
ImageNet-Sketch
Robustness of sketch domain
1000
N/A
50889
ImageNet-A
Robustness adversarial attack
200
N/A
7500
ImageNet-R
Robustness of multi-domains
200
N/A
30000
Table 11. Hyperparameter settings for few-shot learning and base-to-novel generalization experiments under different datasets.
StanfordCars
FGVCAircraft
ImageNet
Caltech101
OxfordPets
Flowers102
Food101
SUN397
EuroSAT
UCF101
DTD
Few-shot
Epochs
20
50
50
200
200
10
150
20
50
150
50
Epochs
30
30
20
30
10
30
Base-to-novel
0.1
0.1
0.1
0.1
0.1
0.1
0.05
0.1
0.05
0.02
0.1
K
12
12
12
12
12
12
12
12
12
12
prompt, which is initialized as 'a photo of a'. Similar to GalLoP (Lafon et al., 2024), the number of fine-grained text features and mulli-scale constant factor are set as Z = and 17 = 10. Similar to PSRC (Khattak et al., 2023b), we set the coefficient A1 = 25 and 22= = 10. The perturbation radius SurPL-G set to p = 0.1 by default. Both SurPL and SurPL-G are trained with the SGD optimizer. We initialize the learning rate 0.0025 with the cosine annealing decay. warm-up strategy is applied during the first training epoch, where the learning rate is fixed 0.00001. All the experiments are conducted single RTX3090 GPU, and results reported under 16-shot training unless specified. Each result is averaged over three random seeds for a fair comparison.
For the few-shot learning experiment, we set the batch size to B = 32 and the number of training epochs to 50 by default. However, we further find that large-scale datasets (e.g.. ImageNet and SUN397) benefit from fewer epochs, while datasets containing specific knowledge (e.g., FGVCAircraft, StanfordCars and EuroSAT) perform better with more training epochs. For the pase-to-novel generalization experiment, we set the batch size to for very individual datasets. For cross-dataset and cross-domain generalization experiments, and set the perturbation radius
and slightly adjust the perturbation radius p
batch size of B 32 0.2. A summary of hyperparameter adjustments is provided in 1.
F. Adaptability of SurPL on Different Single Prompt Learning Approaches
As mentioned before, SurPL is adaptable and integrable to implement different diverse prompt learning ideas based on any single prompt learning approach. To this end, we further explore the adaptability of SurPL by implemented our Surrogate Prompt Learning (SurPL) framework on different single prompt learning approaches.
Specifically, apply CoOp (Zhou al., 2022c), MaPLe (Khattak et al., 2023a) and PSRC (Khattak al., 2023b) as baseline models, and incorporates our surrogate instance-dependerr and fine-grained text features into each approach. We strictly follow their implementation details. The comparison results are presented in Tab. 12. By introducing the idea of SurPL into these approaches, we observe significant performance improvements for CoOp, MaPLe and PSRC, achieving average gains of 3.10%, 2.36% and 1.24%, respectively. This comparison further demonstrates the effectiveness and adaptability of our proposed SurPI..
15

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table 12. Performance comparison between w/o and w/ SurPI. different single prompt learning approaches.
ImageNet
Caltech101
Pets
Cars
Flowers
Food101
Aircraft
Sun397
DTD
EuroSAT
UCFI01
AVG
CoOp +SurPL
71.87
95.57
91.87
83.07
97.07
84.20
43,40
74.67
69.87
84,93
82.23
79.89
73.87
96.56
93.63
87.03
98.24
86.29
52.41
77.10
73.88
87.96
85.90
82.99
A
+2.00
+0.99
+1.76
+3.96
+1.17
+2.09
49.01
+2.43
+4.01
+3.03
+3.67
+3.10
MaPLe
72.33
96.00
92.83
83.57
97.00
85.33
48.40
75.53
71.33
92.33
85.03
81.79
+SurPL
73.98
96.67
93.71
87.79
98.37
86.38
55.76
77.45
74.33
93.49
87.76
84.15
+1.65
+0.67
+0.88
+4.22
+1.37
+1.05
+7.36
+1.92
+3.00
+1.16
+2.73
+2.36
PSRC
73.17
96.07
93.67
83,83
97.60
87.50
50.83
77.23
72.73
92.43
86.47
82.87
SurPL
74.95
96,96
94.25
86.99
98.27
87.42
53.04
78.22
75.02
92.37
87.66
84.10
+1.78
+0.89
+0.58
+3.16
+0.67
-0.08
+2.21
+0.99
+2.29
-0.06
+1.19
+1.24
Table 13.
Performance comparison with other PEFT methods.
Method
ImageNett
Caltech101
Pets
Cars
Flowers
Food101
Aireraft
Sun397
DTD
EuroSAT
UCF101
AVG
CLIP-Adapter
71.60
94.57
92.03
80.90
97.00
86.83
42.67
75.30
71.17
81.87
84.53
79.86
Tip-Adapter
73.10
95.79
92.70
83.09
96.18
87.24
45.59
74.99
72.05
87.46
84.50
81.15
TaskRes
73.00
95.80
92.40
83.50
97.50
86.90
44.90
76.10
71.50
82.70
84.00
80.75
CLIPFit
71.53
96.13
93.50
82.43
96.37
87.37
45.47
75.67
71.57
90.13
83.83
81.27
CLIP-LoRA
73.60
96.40
92.40
86.30
98.00
84.20
54.70
76.10
72.00
92.10
86.70
82.95
SurPL
74.65
96.92
94.22
89.00
98.84
87.63
60.51
77.67
74.75
93,92
88.18
85.12
G. Comparison with Other PEFT Methods
Apart from prompt learning, we also compare SurPI. with other parameter-efficien fine-tuning (PEFT) methods including adapter (CLIP-adapter (Gao et al., 2024), Tip-Adapter (Zhang et al., 2022) and TaskRes (Yu et al., 2023)), BitFit (CLIPFit (Li et al., 2024)) and LoRA (CLIP-LoRA (Zanella & Ben Ayed, 2024)) based approaches in Tab.13. 13. The results indicate that SurPI. also outperforms them by significant margin.
Adapter-based methods utilize lightweight networks to adapt the and visual features output by the VLM's encoders. While proposed Surrogate Feature Generator (SFG) and adapter both adapt the output text features, clarify the different purposes and motivations between them. SFG aims to generate novel text features based on the output text feature, which enables the description of visual concepts from different perspectives. Adapter mainly targets to simply self-enhance the discriminative ability of the output text feature.
H. Additional Experiment Results
Performance comparison under different number of few-shot learning (Tab..1)
Complete ablation study results of each individual datasets under few-shot learning setting (Tab.
Complete ablation study results of each individual datasets under base-to-novel generalization setting 16)..16).
Complete experiment results of each individual datasets under base-to-novel generalization setting (Tab.
Complete experiment results of each individual datasets under cross-dataset generalization setting (Tab,.8),
Additional visualization results of the heat maps (Fig.5).
16

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table Performance comparison under different number few-shot learning.
2-shot
4-shot
8-shot
16-shot
CoOp (Zhou al., 2022c)
70.65
74.02
76.98
79.89
CoCoOp (Zhou al., 2022b)
67.65
71.21
72.96
74.90
MaPLe (Khattak et al., 2023a)
72.58
72.58
78.89
81.79
PLOT (Chen et al., 2023)
74.00
76.90
79.60
82.10
PSRC (Khattak et al., 2023b)
75.29
78.35
80.69
82.87
ALIGN (Wang al., 2023)
73.56
76.85
79.58
82.48
GalLoP (Lafon al., 2024)
76.40
79.10
82.20
84.50
SurPL
75.39
79.24
82.52
85.12
Table 15. Complete ablation study results of each individual datasets under few-shot learning setting.
ImageNet Caltech101
Pets
Cars
Flowers Food101
Aircraft Sun397
DTD
EuroSAT UCF101
AVG
DVLP
72.62
96.24
93.14
85.75
97.84
87.30
52.88
76.15
71.96
92.58
85.64
82.92
GI
73.27
96.16
93.90
86.67
98.10
87.34
55.25
76.88
73.37
93.01
86.70
83.70
GI+ID
73.46
96.47
93.25 86.88
98.34
87.43
55.46
77.07
74.23
93.67
87.32
83.96
GI+FG
74.47
96.77
94.22
89.46
98.78
87.29
60.88
77.29
74.64
93.85
87.20
84.99
GI+ID+FG
74.65
96.92
94.22
89.00
98.84
87.63
60.51
77.67
74.75
93.92
88.18
85.12
Table 16. Complete ablation study results each individual datasets under nose-lo-novel generalization setting.
AVG Base Novel HM
ImageNet
Caltech101
OxfordPets
StanfordCars
Flowers
Base Novel HM
Base Novel HM
Base Novel HM
Novel HM
Base Novel
DVLP
882.00 74.25 78.06 75.98 71.44 73.64 98.15 15 95.09 96 96.12 97.93 97.02 72.81 75.05 73.91 97.66 74.14 84.29
GI
81.84 74.68 78.09 75.57 71.15 73.29 97.78 94.80 96.27 95.80 97.50 96.64 69.48 76.05 72.62 97.37 77.18 86.11
G[+ID
84.77
76.19 80.25 76.94 71.67 74.21 98.28 95.01 96.62 95.89 97.44 96.66 77.87 75.05 76.43
98.67 74.99 85.22
GI+FG
85.52
75.50 80.20 78.28 69.09 73.40 98.92 92 94.65 96.74 96.21 97.03 96.62 81.62 72.20 76.62 98.86
86
72.89 83.91
GI+ID+FG
86.37 76.32 81.03 78.74 70.49 74.39 98.77 95.16 96.93 96.37 97.41 96.89 83.57 72.77 77.80 98.9 72.88 83.92
Food101 Base Novel HM
FGVCAireraft
SUN397
DTD
EuroSAT
UCF101 Base Novel HM
Base Novel HM
Base Novel HM Base Novel HM
Base Novel HM
DVLP
990.87 92 91.48 42.64 36.49 39.33 79.81 81 79.49 79.65 83.29 29 59.50 69.41 81,94 56.94 67. 85,99 78.55 82.10
90.61 91.97 91. 41.08 29.63 34.43 79.00 79.76 79.38 82.91 61.64 70.71 84.94 61.26 71.18 85.68 80.51 83.01
GI+ID
90.87 92.06 47.12 37.29 41.63
81.90 79.85 80.86
84.45 61.55 71.20 71.83 80.65
88.52
81.32 84.77
GI+FG
90.42 91.09 90.75 45.06 36.43 40.29
82.82 77.76 80.21 86.46 62.60 72.62 93.60 77.14 84.58
88.52 79.65 83.85
GI+ID+FO
90.92 91.81 91.36 49.20
36.93
12.
83.43 78.96
81.13 86.07 62.04 72.11 94.63 81.33 87.48
89.44
79.74 84.31
17

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
z=1
z=2
A photo of water lily
z=3
Z-4
z-
z-2
A photo of Audi TT
z=3
z-4
z-1
2F2
A photo of a Abyssinian
ZF3
z=4
Figure 5. Additional visualization results of the attention heat maps.
18

---

Surrogate Prompt Learning: Towards Efficient and Diverse Prompt Learning for Vision-Language Models
Table 17. Complete experiment results of each individual datasets under base-to-nove generalization setting.
AVG
ImageNet Novel HM
Caltech101
OxfordPets
StanfordCars
Flowers102
Base Novel HM
Base
Base Novel HM
Base Novel HM
Base
Novel
HM
Base Novel HM
CLIP
68.21 73.36 70.69
72.42 68.13 70.21 97.29 94.10 95.67 89.47 96.81 93.00 63.87 74.97 68.98 69.23 76.74 72.79
CoOp
82.68
64.15 72.25
76.34 65.17 70.31 18.15 93.23 95.63 94.19 96.01 95.09 77.58 65.02 70.75
97.79 63.95 77.33
CoCoOp
82.56
64.66 72.52
76.09
69.73 72.77 97.92 91.78 94.75 93.48 95.77 94.61 77.65 63.00 69.56
97.44 60.97 75.01
ProGrad
82.48 69.12 75.21
76.72
67.80 71.98 98.30 93.96 96.08 94.59 96.98 95.77 77.11 69.89 73.32
96.52 69.86 81,05
KgCoOp
81,94 72.52 76.94
75.89 69.55 72.58 97.72 94.32 95.99 95.16 96.61 95.88 74.16 74.64 74.40
95.85 73.19 83.00
MaPLe
82.28
75.14 78.55
76.66
70.54 73.47 97.74 94.36 96.02 95.43 97.76 96.58 72.94 74.00 73.47
95.92 72.46 82.56
PSRC
84.24 75.68 79.73
77.62
70.55 73.92 98.06 93.85 95.91 95.20 97.31 96.24 78.42 75.26 76.81
98.01
77.07 86.29
ALIGN
83.39
75.51 79.25
76.89
72.15 74.44 98.37 94.70 96.50 95.67 97.93 96.79 77.24 76.38 76.81
97.70 73.30 83.76
TCP
84.13
75.36 79.51
77.27
69.87 73.38 98.23 94.67 96.42 94.67 97.20 95.92 80.80 74.13 77.32
97.73
75.57 85.23
DePT
85.18
76.17 80.42
78.20
70.27 74.02 98.57 94.10 96.28 95.43 97.33 96.37 80.80 75.00 77.79
98.40
77.10 86.46
SurPI-G
86.37 76.32 81.03
78.74 70.49 74.39 98.77 95.16 96.93 96.37 97.41 96.89 83.57 72.77 77.80 98.90 72.88 83.92
Food101 HM
FGVCAircraft
SUN397
DTD
EuroSAT Novel
UCF101
Base
Novel
Base Novel HM
Base
Novel HM
Base
Novel
HM
Base
HM
Base
Novel HM
CLIP
89.42 90.68 90.05 27.55 33.29 30.15 69.40 75.56 72.35 53.36 51.69 52.51 50.19 69.90 58.43 68.10 75.12 71.44
CoOp
88.68 85.31 86.96 39.94 24.62 30.46 80.58 63.90 71.28 79.82 45.17 57.69 91.25 47.26 62.27 85.20 56.05 67.62
CoCoOp
88.11 11 83.92 85.96 41.80 25.08 31.35 79.38 67.99 73.24 79.94 42.55 55.54 92.14 51.33 65.93 84.25 59.17 69.52
ProGrad
90.11 89.56 89.83 40.30 25.81 31.47 81.11 71.31 75.89 76.85 51.89 61.95 91.09 56.21 69.52 84.59
67.03 74.79
KgCoOp
90.53 91.01 90.77 38.72 29.63 33.57 80.71 76.28 78.43 80.44 56.69 66.51 88.15 60.42 71.70 84.00 75.37 79.45
MaPLe
90.71 92.05 91.38 37.44
35.61 36.50 80.82 78.70 79.75 80.36 59.18 68.16 94.07 73.23 82.35
83.00 78.66 80.77
PSRC
90.66 91.56 91.11 42.86
36.59 39.48 82.54 78.81 80.63 83.53 59.62 69.58 92.85 72.95 81.71 86.94
78.92 82.74
ALIGN
90.77
92.07 91.42
37.56
36.97 37.26 82.47 79.68 81.05 82.13 54.17 65.28 94.03 74.9 83.38
84.43
78.33 81.27
TCP
90.57
91.37 90.97
41.97
34.43
37.83 82.63 78.20 80.35 82.77 58.07 68.25 91.63 74.73 82.32
87.13
80.77 83.83
DePT
90.87
91.57 91.22
45.70
36.73
40.73
83.27
78.97 81.06 84.80 61.20 71.09 93.23
77.90
84.88
87.73
77.70 82.41
SurPI-G
90.92 91.81 91.36
49.20
36.93
42.19
83.43
78.96
81.13 86.07 62.04
72.11 94.63
81.33
87.48
89.44
79.74 84.31
Table 18. Complete experiment results of each individual datasets under cross-dataset generalization setting.
ImageNet Caltech101
Pets
Cars
Flowers Food101
Aircraft Sun397
DTD
EuroSAT
UCF101
AVG
CLIP
66.70
93.30
89.10
65.70
70.70
85.90
24.90
62.60
44.30
48.30
67.60
65.24
CoOp 71.51
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
CoCoOp
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
ProGrad
72.24
91.52
89.64
62.39
67.87
85.40
20.16
62.47
39.42
43.46
64.29
62.66
KgCoOp
70.66
93.92
89.83 65.41
70.01
86.36
22.51
66.16
46.35
46.04
68.50
65.51
PLOT
71.60
92.07
90.10 65.70
69.23
86.23
25.00
61.67
38.60
47.83
67.00
64.34
MaPLe
70.72
93.53
90.49 65.57
72.20
86.20
24.74
67.01
46.49
48.06
68.69
66.30
PSRC
71.27
93.60
90.25 65.70
70.25
86.15
23,90
67.10
46.87
45.50
68.75
65.81
DePT
71.60
93.80
90.13 66.00
70.93
86.27
24.30
67.23
46.60
45.83
69.10
66.02
TCP
71.40
93.97
91.25
64.69
71.21
86.69
23.45
67.15
44.35
51.45
68.73
66.29
SurPL-G
73.33
93.73
90.16
64.67
70.53
85,52
24.80
67.45
48.54
52.85
67.86
66.61
19


Reviews:
[{"review_id": "RrCOrlOles", "reviewer": "Reviewer_c1YQ", "summary": "This paper proposes Surrogate Prompt Learning (SurPL), a new approach to enhance the efficiency and diversity of prompt learning for VLMs. Instead of learning multiple diverse prompts, SurPL directly generates surrogate prompted text features through a lightweight Surrogate Feature Generator (SFG), reducing computational overhead while maintaining diversity.\n\nThe key contributions of this paper include: 1) A novel method that efficiently generates diverse text features instead of learning multiple prompts. 2) A cross-attention-based module that produces instance-dependent and fine-grained surrogate text features. 3) Extensive tests on 15 vision classification datasets show that SurPL achieves comparable or superior performance to existing diverse prompt learning methods, while significantly improving computational efficiency.4) SurPL-G, an extension of SurPL with Sharpness-aware Minimization (SAM), further enhances generalization across base-to-novel, cross-dataset, and cross-domain scenarios.\n\nOverall, this paper introduces an efficient and effective approach to prompt learning, addressing the computational cost issue while maintaining strong adaptation ability in VLMs.\n\n## update after rebuttal\nI appreciate the author's efforts during the rebuttal period and am willing to maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The O(M) computational complexity claim of SurPL lacks a formal derivation.\n\nA1: Thanks for the valuable comments. We provide a theoretical analysis of computational complexity here. \n\nNotations. \n\nLoss $L$, classnames ${c}=(c_m) _ {m=1}^M$, text encoder parameter: ${T}=({T}^{k}) _ {k=1}^K$, SFG parameter: $\\theta_{SFG}$, text prompts parameter: $s=(s^k) _{k=1}^K$, encoder layer depth: $K$, the parameter size of SFG is much smaller than that of text encoder: $\\theta _ {SFG}<<T$.\n\nDerivation.\n\nOptimizing the text prompt $s^1$ at the text encoder input position requires computing the gradient through the entire model. For each single surrogate output text feature, the back-propagation computation of SurPL can be written as:\n$$\n\\frac{\\partial L}{\\partial s^1}=\\underbrace{\\frac{\\partial L}{\\partial \\theta _ {SFG}(T^K([s^{K},c _ m^k]);\\alpha_j)}\\cdot \\frac{\\partial \\theta _ {SFG}(T^K([s^{K},c _ m^k]);\\alpha _ j)}{\\partial T^K([s^K,c _ m^K])}} _ {\\text{SFG}}\\cdot \\underbrace{\\frac{\\partial T^K([s^K,c _ m^K])}{\\partial T^{K-1}([{s}^{K-1},c _ m^{K-1}])}\\cdots \\frac{\\partial {T}^{1}([s^1,c _ m^1])}{\\partial s^1}} _ {\\text{Text Encoder}}.\n$$\nThis gradient computation can be conceptually divided into two parts, which correspond to the text encoder and SFG. We denote the computational complexity for each output text feature as $O_T$ and $O_{SFG}$, respectively. As shown in Fig.1 (c), SurPL simultaneously involves $M$ text features back-propagation w.r.t text encoder, which equals to the complexity in single prompt learning: $O(M) = M \\cdot O_T$. For the SFG stage, $(B+Z)M$ text features are involved in computation. The overall complexity of SurPL can thus be expressed as:\n$$\nO_{SurPL} = M \\cdot O _ T + (B+Z)M \\cdot O _ {SFG}\n$$\nSince $\\theta_{SFG}<<T$, it follows that $O_{SFG}<<O_T$. Therefore, the second term is negligible, and we can approximate: $O _ {SurPL} \\approx M \\cdot O_T = O(M)$.\n\n>Q2: Comparison with non-prompt-based PEFT methods.\n\nA2: Thanks for the suggestion. We further compare SurPL with Adapters (CLIP-Adapter(Gao et al., 2024), Tip-Adapter(Zhang et al., 2022), TaskRes(Yu et al., 2023)), BitFit (CLIPFit$^\\text{a}$) and LoRA (CLIP-LoRA$^\\text{b}$) based methods. The averaged performances of 11 datasets are shown below, which demonstrate the superiority of SurPL compared to other PEFT techniques. We will add more detailed results and analysis in the next version of the manuscript.\n\n| CLIP-Adapter | Tip-Adapter | TaskRes | CLIPFit | CLIP-LoRA |  DVLP | SurPL |\n|:------------:|:-----------:|:-------:|:-------:|:---------:|:-----:|:-----:|\n|     79.86    |    81.15    |  80.75  |  81.27  |   82.95   | 82.92 | 85.12 |\n\na. Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification. EMNLP 2024.\n\nb. Low-Rank Few-Shot Adaptation of Vision-Language Models. CVPRW 2024.\n\n>Q3: Experiments focus only on classification. Testing on VQA, object detection, or video understanding would provide a broader evaluation.\n\nA3: Thanks for your valuable suggestion. First, to our best knowledge, almost all studies of VLM-based PEFT techniques (e.g. prompts, adapters, LoRA, BitFit) mainly evaluate on classification tasks, and we simply follow these established protocols for fair comparison.\n\nSecond, although some researches have applied PEFT techniques on other visual tasks, they tend to leverage existing methods rather than explore new approaches. Diverse prompt learning has not been well-explored on other tasks, making it challenging to identify a suitable baseline for SurPL on such tasks. Additionally, exploring the effectiveness of diverse prompt learning on VQA and detection from scratch and then implementing SurPL require significant time, making it difficult to include such results within the rebuttal period. \n\nHowever, we sincerely appreciate this insightful suggestion and will actively explore this direction in the near future.\n\n>Q4: Hyperparameter sensitivity analysis is missing.\n\nA4: Thanks for the comments. Due to the character limit, we provide comprehensive hyperparameter analysis at the Q&A 2 of Reviewer jTad. The results indicate that the hyperparameters applied in this work are reasonable and relatively stable. Please refer to that response for details. \n\n>Q5: Why was cross-attention chosen as SFG over other feature generation techniques (e.g., Transformer-based mechanisms)?\n\nA5: Thanks for the comments. SFG aims to take basic prompted text feature and conditional signal as input, and generate the surrogate prompted feature according to the given signal. Cross-attention module is the most intuitive choice for this requirement. While Transformer-based mechanisms can also achieve it, stacking multiple attention blocks would significantly decrease the efficiency.\n\n>Q6: Minor suggestions of writing.\n\nA6: Thanks for the valuable suggestions. Due to the character limit, we are not able to provide detailed modifications in the rebuttal. We will revise them in the next version of the manuscript.", "claims_and_evidence": "The paper presents strong empirical results, but some claims need further support:\n\n1. The claim that SurPL reduces complexity to O(M) is based on experiments only, without a formal proof. A theoretical complexity analysis would make this more convincing.\n\n2. The surrogate feature generator (SFG) is effective, but there is no comparison with alternative feature generation methods. \n\n3. SurPL is only tested on classification tasks, making its generalization to other vision-language tasks unclear.", "methods_and_evaluation_criteria": "The proposed SurPL framework is well-motivated, and the SFG is a reasonable design. However, some aspects could be improved:\n\n1)  The method is only tested on classification tasks, limiting its applicability. Evaluating SurPL on VQA, object detection, or video understanding would better assess its generalization.\n\n2) The paper compares SurPL with diverse prompt learning methods but lacks comparisons with non-prompt-based fine-tuning approaches (e.g., Adapters, LoRA). Adding such baselines would clarify whether prompt learning is the best approach for efficiency.\n\n3) The claim of O(M) complexity is reasonable, but no detailed breakdown is provided. A formal complexity analysis would strengthen this argument.", "theoretical_claims": "The paper does not provide formal theoretical proofs for its main claims, particularly the O(M) computational complexity of SurPL. The argument is mainly supported by empirical results, which are convincing but not mathematically rigorous.", "experimental_designs_or_analyses": "The experimental setup is well-structured, but some aspects could be improved:\n\n1) The experiments focus only on classification, limiting the assessment of SurPL’s generalization. Testing on VQA, object detection, or video understanding would provide a broader evaluation.\n\n\n2) While the paper compares against diverse prompt learning methods, it lacks comparisons with non-prompt-based methods (e.g., Adapters, LoRA). Including these baselines would clarify whether prompt learning is the best approach.\n\n\n3) Key parameters (e.g., Z, η) are fixed without sensitivity analysis. Investigating their impact would strengthen the robustness of conclusions.\nOverall, the experiments are well-executed, but broader evaluations and deeper analyses would improve their reliability.", "relation_to_broader_scientific_literature": "The paper situates SurPL well within the literature on prompt learning for vision-language models, referencing key works on single and diverse prompt learning. However, there are some missing discussions:\n\n1. The paper does not compare SurPL with Adapters, LoRA, or other efficient fine-tuning methods, which are widely used alternatives to prompt learning.\n\n2. While the paper focuses on reducing computation, it does not connect with broader studies on efficient deep learning, such as low-rank adaptations, pruning, or quantization techniques.\n\n3. Most cited works focus on classification tasks, but SurPL's relevance to VQA, object detection, or multimodal reasoning is not discussed.", "essential_references_not_discussed": "The paper covers prompt learning literature well, but some essential references are missing （see previous section for details）:\n\n1) Comparison with efficient fine-tuning methods\n\n2) Computational efficiency research\n\n3) Prompt learning in broader tasks", "other_strengths_and_weaknesses": "Strengths\n\n1. The paper proposes an innovative method (SurPL) that improves prompt learning efficiency while maintaining diversity. The Surrogate Feature Generator (SFG) effectively reduces computational overhead, avoiding the high cost of traditional diverse prompt learning.\n\n2. The experimental results are strong, tested on 15 datasets, demonstrating that SurPL significantly improves computational efficiency while achieving comparable or superior performance to existing state-of-the-art methods.\n\n3. The paper is well-structured, with a clear motivation, well-designed experiments, and strong baseline comparisons. The methodology and experimental sections are logically presented and easy to follow.\n\nWeaknesses\n\n1.The O(M) computational complexity claim lacks a formal derivation and is currently supported only by empirical results. A complexity analysis would enhance the theoretical rigor of the argument.\n\n2. The method is tested only on classification tasks, making it unclear whether SurPL generalizes to VQA, object detection, and multimodal reasoning. Expanding the evaluation to broader tasks would verify its generalizability.\n\n3. The paper does not compare SurPL with non-prompt-based fine-tuning methods such as Adapters, LoRA, and BitFit, which are widely used for efficient VLM adaptation. Including such comparisons would clarify whether SurPL is the most effective approach for efficient model adaptation.", "other_comments_or_suggestions": "The writing is generally clear, but a few areas could be improved for better readability and precision. Below are some minor suggestions:\n\n- The introduction could better emphasize the novelty of SurPL compared to previous diverse prompt learning methods. While the paper discusses existing works, a more explicit contrast with traditional prompt learning and fine-tuning methods would help highlight its contributions.\n\n- Some notations and explanations in the methodology section could be clarified. For example, the role of Z (fine-grained feature count) and η (multi-scale factor) is not well explained, and a brief intuitive description would improve clarity.\n\n- The figures are informative, but adding a computational flow diagram for SurPL compared to traditional prompt learning would make the efficiency argument clearer.\n\n- There are some minor typos and grammar inconsistencies, particularly in the experimental section. Careful proofreading would improve the overall presentation.\n\n- The appendix could include more details on hyperparameter settings, optimizer choices, and training dynamics to enhance reproducibility.", "questions_for_authors": "1. The paper claims that SurPL reduces complexity to O(M) compared to O(BM) or O(ZM), but there is no formal derivation. Could you provide a step-by-step complexity breakdown?\n\n2. SurPL is compared against diverse prompt learning methods, but not with non-prompt-based adaptation techniques like Adapters, LoRA, or BitFit. Why were these omitted?\n\n3. The paper focuses only on classification tasks. Have you tested SurPL on VQA, object detection, or multimodal reasoning?\n\n4. Some hyperparameters, like Z (fine-grained feature count) and η (multi-scale factor), are fixed in the experiments. How sensitive is SurPL’s performance to these choices?\n\n5. SFG generates surrogate features efficiently, but why was cross-attention chosen over other feature generation techniques (e.g., Transformer-based mechanisms)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9KzeXjtHGx", "reviewer": "Reviewer_Zno2", "summary": "Prompt learning is an efficient fine-tuning technique that learns text prompts. Learning multiple text prompts instead of just one can improve performance while increasing computational cost. This paper proposes learning diverse text prompts without initializing additional parameters by generating specific text prompts with a lightweight model to avoid overwhelming the gradient process. The performance on several classification benchmarks is better than that of compared methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: How does the FG loss work with images that contain multiple classes? Would other instances in the figure disturb the classification since we only need to predict the most significant object? For instance, the author could provide heatmap visualization on such images.\n\nA1: Thanks for the insightful comments. Other instances will not disturb the classification process. In both diverse prompt learning and surrogate prompt learning, each input text description is comprised by the text prompts and the classname. Classnames are pre-defined and remain fixed during optimization, thereby providing consistent class-level semantic guidance. To this end, every FG prompted text feature embeds information that strongly associated with its corresponding class, which ensures the classification procedure not being disturbed by other instances in the image.\n\nTo further validate this problem, we provide additional visualization results. We select images that contain at least two distinct object categories, and visualize the attention heat maps of different surrogate fine-grained text features on such images. The corresponding heat maps are shown in “[multiclass-heatmap.png](https://postimg.cc/sBK9jYJB)”, which demonstrate the effectiveness of FG prompted text features in our proposed method.\n\n>Q2: The description in Figure 2 is not clear enough, such as how the ID conditional signals interact inside the Surrogate Feature Generator. Moreover, the structure on the right does not match Eq(4), where $w \\in M\\times d$, $\\alpha \\in Z\\times d$; then how does the output $h^{FG}$ become $Z\\times M\\times d$? If the conditional signals interact individually, you should improve the figure to show it, or it could be confusing to the reader.\n\nA2: Thanks for the valuable suggestion. We are sorry for the unclear description of Eq(4). The conditional signals interact individually with each basic prompted text feature $w_m$. We rewrite Eq(4) as:\n\n$$\nh_m = \\theta _ {SFG}(w_m,\\alpha).\n$$\n\nThis equation clearly states that for the basic prompted text feature corresponds to $m$-th class $w_m$, we will generate $Z$ fine-grained surrogate prompted text features $h^{FG} _ {m} \\in \\mathbb{R} ^ {Z \\times d}$. To this end, we have $h^{FG} \\in \\mathbb{R} ^ {Z\\times M\\times d}$ for total $M$ classes. \n\nWe will correct the corresponding description in the next version of the manuscript. We also add the input and output notations of SFG in Fig.2 for clear description, which is shown in “[Revised-Fig2.png](https://postimg.cc/JD1zLdt8)”.\n\n>Q3: Since the fine-grained module improves performance according to the ablation study, has the author tried to adopt this idea to existing prompt learning techniques? I wonder why there is no such usage in the current community.\n\nA3: Thanks for your comments. First, the idea of fine-grained prompt learning has been utilized in several VLM-based prompt learning methods, which we have already cited in the manuscript (e.g., Chen et al., 2023, Lafon et al., 2024). However, these methods suffer from the huge computation complexity problem, hindering their practical application in real-world scenarios.\n\nSecond, we claim that SurPL is adaptable and integrable to implement different diverse prompt learning ideas based on any single prompt learning approach. To this end, we have tried to implement SurPL on different existing single prompt learning methods, including CoOp, MaPLe and PSRC. Specifically, we consider these methods as baseline and additionally introduce both surrogate instance-dependent and fine-grained text features on each method. The comparison results are actually reported in Appendix Sec.E Table.12, and demonstrate that utilizing both instance-dependent and fine-grained can significantly improve the performance of existing single prompt learning methods.\n\nTo explicitly explore the effectiveness of the fine-grained idea, here we further conduct the following experiment. We consider CoOp, MaPLe and PSRC as baseline approaches, and exploit our method to only generate surrogate fine-grained features based on each approach. We provide the comparison of average performances on 11 datasets below. \n\n|       | Method | Method+FG | $\\Delta$ |\n|-------|:------:|:---------:|:--------:|\n|  CoOp |  79.89 |   83.15   |   3.26   |\n| MaPLe |  81.79 |   83.80   |   2.01   |\n|  PSRC |  82.87 |   83.33   |   0.46   |\n\n>Q4: Why is the ID signal derived from the visual feature while the FG signal is randomly initialized?\n\nA4: Thanks for the question. For existing ID prompt learning methods (such as CoCoOp), text prompts are not randomly initialized for optimization, but require visual information as prior knowledge. So, the ID signals are derived from the visual feature, thus providing visual guidance. For existing FG prompt learning methods (such as PLOT and GalLoP), text prompts are randomly initialized and supposed to capture fine-grained information during optimization. Therefore, we also keep FG signals randomly initialized.", "claims_and_evidence": "The claims in the submission are well-supported by clear evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no theoretical claims for the proposed methods, such as how the fine-grained loss promotes global minimum results through spatial feature consistency. The provided theoretical explanation for the optimization stages appears to be a simple regularization loss that mitigates overfitting, offering no new insights.", "experimental_designs_or_analyses": "The experimental designs and analyses are suitable and sufficient for this task.", "relation_to_broader_scientific_literature": "This paper proposes a more efficient approach to diverse text prompt learning for image classification transfer tasks.", "essential_references_not_discussed": "Nothing to supplement.", "other_strengths_and_weaknesses": "Strengths: \n1. The method is simple and easy to follow. \n2. The experiments are sufficient and show better speed compared to other diverse text prompt methods. \n\nWeaknesses: \n1. How does the FG loss work with images that contain multiple classes? Would other instances in the figure disturb the classification since we only need to predict the most significant object? For instance, the author could provide heatmap visualization on such images. \n2. The description in Figure 2 is not clear enough, such as how the ID conditional signals interact inside the Surrogate Feature Generator. Moreover, the structure on the right does not match Eq(4), where ω∈M×d, α∈Z×d; then how does the output h_FG become Z×M×d? If the conditional signals interact individually, you should improve the figure to show it, or it could be confusing to the reader. \n3. Since the fine-grained module improves performance according to the ablation study, has the author tried to adopt this idea to existing prompt learning techniques? I wonder why there is no such usage in the current community.", "other_comments_or_suggestions": "None", "questions_for_authors": "I am interested in the initialization of conditional signals, why is the ID signal derived from the visual feature while the FG signal is randomly initialized?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D9yReA0e3F", "reviewer": "Reviewer_jTad", "summary": "This paper introduces Surrogate Prompt Learning to address efficiency issues in diverse prompt learning for vision-language models (VLMs). SurPL leverages a lightweight Surrogate Feature Generator (SFG) to directly generate diverse prompted text features from a single basic prompt, avoiding the computational overhead of conventional approaches. Experiments across 15 datasets show SurPL's effectiveness.\n\n## update after rebuttal\nThe author has adequately addressed my concerns, thanks!", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The paper doesn't provide much theoretical justification for why surrogate features effectively replace the original prompted features, relying instead on empirical results.\n\nA1: Thanks for the valuable comments. We provide a theoretical analysis to demonstrate the effectiveness of the surrogate features.\n\nNotation.\n\nVisual instance $x$, basic text prompt $s$, explicit diverse text prompt $s_E$, conditional signal $\\alpha \\in \\mathbb{R}^d$, basic prompted text feature $w=T([s,c])\\in \\mathbb{R}^d$, surrogate prompted text feature: $h=\\theta _ {SFG}(w,\\alpha) \\in \\mathbb{R}^d$, the original explicit prompted text feature $w_E=T([s_E,c])\\in \\mathbb{R}^d$.\n\nProof.\n\nTo validate the effectiveness of replacing $w_E$ with $h$ in diverse prompt learning, we analyze their feature similarity through $||h-w_E||=||\\theta _ {SFG}(w,\\alpha)-w_E||$. Here, we mainly focus on providing the proof of surrogate instance-dependent prompt learning, which imposes stricter constraints (in surrogate fine-grained prompt learning, $\\alpha$ is also learnable parameters, which directly simplified the condition to $h=\\theta _ {SFG,\\alpha} (w)$).\n\n$w$, $\\alpha$, and $w_E$ are bounded-dimensional representations within the vector space $\\mathbb{R}^d$ (outputs of continuous neural network mappings), and their corresponding inputs $s$, $x$ and $s_E$ are related via $s_E=s+\\alpha=s+V(x)$ in instance dependent prompt learning (CoCoOp). Hence, there exists a continuous function $g$, such that: $w_E = g(w, \\alpha)$. According to the universal approximation theorem, for any continuous function $g(\\cdot)$ and any $\\epsilon>0$, there exists a neural network with sufficient capacity (here, we utilize the cross-attention module $\\theta_{SFG}$, which has been proved as a proper universal approximator$^\\text{a}$) such that for all $(w,\\alpha)$,\n$$\n|| g(w,\\alpha)-\\theta_{SFG}(w,\\alpha)||<\\epsilon.\n$$\nBy taking the approximation error $\\epsilon \\to 0$, we obtain $||h-w_E||\\to 0$ at the optimal parameters $\\theta_{SFG}^{\\star}$, which confirms that $h$ effectively approximates and replaces $w_E$. This analysis validates the theoretical existence of $\\theta_{SFG}^{\\star}$, while extensive empirical studies in the manuscript further elaborate on its implementation and optimization, and substantiate its practical effectiveness.\n\na. Are Transformers universal approximators of sequence-to-sequence functions? ICLR 2020.\n\n>Q2: Analysis of performance sensitivity to hyperparameter choices, particularly for different loss components, the number of fine-grained text features and multi-scale constant factor.\n\nA2: Thanks for the valuable suggestion. Here we carefully analyze the effect of different hyperparameters. Due to the character limit, we only report the averaged results on 11 datasets.\n\nDifferent loss components ($\\lambda_1$ and $\\lambda_2$): The results are quite stable under different loss coefficient hyperparameter settings. Since $\\lambda_1=25$ and $\\lambda_2=10$ achieve the best averaged performance, we choose them as the default setting. \n\n| $\\lambda_1=15$ | $\\lambda_1=20$ | $\\lambda_1=25$ | $\\lambda_1=30$ | $\\lambda_1=35$ |\n|:------------:|:------------:|:------------:|:------------:|:------------:|\n|     85.02    |     85.04    |     85.12    |     85.09    |     85.06    |\n\n| $\\lambda_2=5$ | $\\lambda_2=10$ | $\\lambda_2=15$ | $\\lambda_2=20$ |\n|:-----------:|:------------:|:------------:|:------------:|\n|    84.93    |     85.12    |     85.12    |     85.09    |\n\nThe number of fine-grained text features Z: Comparing the results between $Z=0$ (No FG prompts applied) and other settings validates involving FG information can significantly boost the performance. We observe consistent performance gains when increasing $Z$ from 1 to 4, which indicates applying sufficient FG prompt is necessary to capture comprehensive FG information. However, further increasing Z leads to performance degradation, which we attribute to the inclusion of ineffective information (e.g., background noise). Therefore, we select $Z=4$ as the default setting.\n\n| $Z=0$ | $Z=1$ | $Z=2$ | $Z=3$ | $Z=4$ | $Z=5$ | $Z=6$ |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| 83.96 | 84.91 | 85.02 | 85.08 | 85.12 | 85.04 | 84.95 |\n\nMulti-scale constant factor $\\eta$: In this paper, we utilize multi-scale strategy to obtain FG visual information under different scales. The results show that applying a relatively small $\\eta$ (5 or 10) can achieve better performance, since applying over-large scale may involve ineffective information (e.g., background noise), thus leading to a negative effect.\n\n| $\\eta=5$ | $\\eta=10$ | $\\eta=15$ | $\\eta=20$ |\n|:--------:|:---------:|:---------:|:---------:|\n|   85.08  |   85.12   |   84.99   |   84.86   |\n\n>Q3: Limited analysis of whether the surrogate features might be less effective for certain types of tasks or datasets.\n\nA3: Thanks for the comments. Due to the character limits, please refer to Q&A 3 of Reviewer 4GML for details analysis.", "claims_and_evidence": "The paper provides empirical validation through experiments on 15 different datasets across multiple settings. The efficiency claims are backed by direct comparisons of GPU memory usage, training time, and testing time. The effectiveness of the Surrogate Feature Generator is demonstrated through ablation studies, and the visualizations in Figures 3 and 4 provide qualitative evidence of the surrogate features' ability to capture relevant visual information.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are appropriate for the problem. The authors address the efficiency bottleneck in diverse prompt learning through a surrogate generation approach, which directly targets the computational overhead issue. The comparison metrics are relevant for assessing both performance and efficiency. The authors also appropriately evaluate their method against both single prompt learning approaches and diverse prompt learning approaches to demonstrate comprehensive improvements.", "theoretical_claims": "The paper does not contain formal mathematical proofs for theoretical claims. It presents algorithmic formulations and describes the approach using mathematical notation, but doesn't make rigorous theoretical claims requiring proof.", "experimental_designs_or_analyses": "Some potential improvements to the experimental analysis could include:\n\n1. The paper doesn't provide much theoretical justification for why surrogate features effectively replace the original prompted features, relying instead on empirical results.\n\n2. Analysis of performance sensitivity to hyperparameter choices, particularly for different loss components, the number of fine-grained text\nfeatures and multi-scale constant factor.\n\n3. While the paper shows improved efficiency and competitive performance, there's limited analysis of whether the surrogate features might be less effective for certain types of tasks or datasets.", "relation_to_broader_scientific_literature": "Related to prompt engineering in VLM.", "essential_references_not_discussed": "Nil.", "other_strengths_and_weaknesses": "Nil.", "other_comments_or_suggestions": "Nil.", "questions_for_authors": "Nil.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Cvx7T7bKo", "reviewer": "Reviewer_4GML", "summary": "This paper presents a novel Surrogate Prompt Learning (SurPL) framework for vision-language models (VLMs). SurPL aims to achieve efficient and diverse prompt learning by replacing explicit diverse prompt learning with a Surrogate Feature Generator (SFG) that generates diverse text features without requiring complex gradient computations. This approach significantly reduces the computational burden while maintaining competitive performance. The authors provide extensive experimental validation across multiple benchmarks, showing that SurPL improves both efficiency and performance compared to existing prompt learning methods. The paper also introduces SurPL-G, an extension designed to improve generalization through Sharpness-aware Minimization (SAM).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: Lack deeper theoretical justification on why surrogate features retain meaningful semantic information.\n\nA1: Thanks for the comments. We provide a theoretical analysis based on the universal approximation theorem. Due to character limits, please refer to the Q&A 1 of Reviewer jTad for detailed analysis.\n\n>Q2: No qualitative comparison between surrogate and explicitly learned prompts.\n\nA2: Thanks for the insightful comments. We provide direct visualization map comparisons between surrogate and explicit FG prompts (due to the computation resource limit, we can’t afford to reproduce explicit ID prompt learning). As shown in [explicit-versus-surrogate.png](https://postimg.cc/sQGnBFP5), heatmaps generated from explicit and surrogate prompts focus on the same regions of the image, particularly at large FG scales ($Z=3,4$). For small FG scales ($Z=1,2$), while the heatmaps show minor differences, they still consistently focus on the target object. These comparison results indicate the effectiveness of SFG.\n\n>Q3: No discussion about the failure cases of SurPL.\n\nA3: We first provide the direct comparison between explicit FG prompt learning and surrogate FG prompt learning on 9 datasets (except ImageNet and SUN397 due to the computational resource limit). The averaged results indicate surrogate features don’t exhibit significant failures compared to explicit features.\n\n| Explicit | Surrogate |\n|:--------:|:---------:|\n|   86.79  |   86.70   |\n\nWe further compare the generalization ability between explicit and surrogate prompts on 10 datasets (excluding ImageNet). The results indicate that surrogate prompts initially exhibit weaker generalization ability. We attribute it to the lightweight architecture of SFG. Generating features from SFG may overfit more easily. However, through our proposed SurPL-G, we effectively address this limitation and achieve remarkable performance.\n\n|               |  Base | Novel |   HM  |\n|:-------------:|:-----:|:-----:|:-----:|\n|    Explicit   | 86.76 | 73.15 | 79.38 |\n|   Surrogate   | 87.11 | 72.13 | 78.91 |\n| Surrogate+SAM | 87.13 | 76.90 | 81.70 |\n\nFinally, we compare SurPL with the state-of-the-art work GalLop. While GalLoP achieves marginally superior results on certain datasets (e.g., ImageNet), we attribute this advantage to its multiple global prompt learning and dropout strategy. This strategy induces diversity through randomization, enhancing the performance. We will further explore how to incorporate such strategy into our SurPL.\n\n>Q4: Hyperparameter sensitivity analysis is missing.\n\nA4: We provide hyperparameter analysis at the Q&A 2 of Reviewer jTad. The results indicate that the hyperparameters applied are reasonable and relatively stable.\n\n>Q5: Analyze how reducing SFG complexity affects performance.\n\nA5: We appreciate this insightful suggestion. Due to the time limit, we only explore this trade-off by reducing the dimension of $d_{mid}$ in $\\theta _ {fc1}$ and $\\theta _ {fc2}$ in SFG. The results indicate that accuracy remains relatively stable despite parameter reduction. This inspires us to further reduce the parameter of projection layers $\\theta _ {in1}$, $\\theta _ {in2}$, $\\theta _ {in3}$ and $\\theta _ {out}$.\n\n| $d_{mid}$ | Param(M) |  Acc  |\n|:-------:|:--------:|:-----:|\n|   256   |   1.31   | 85.12 |\n|   128   |   1.18   | 85.00 |\n|    64   |   1.11   | 84.99 |\n|    32   |   1.08   | 85.00 |\n\n>Q6: How does SurPL handle domain shifts compared to explicit diverse prompt learning?\n\nA6: Thanks for the question. Domain shift tightly associates with model’s generalization ability. Due to the computation resource limit, we can’t reproduce explicit prompt learning on cross-domain experiment (based on ImageNet). We alternatively conduct the comparison of explicit and surrogate prompt learning on base-to-novel setting. Please refer to Q&A 3 for detailed analysis.\n\n>Q7: How sensitive is SurPL to the choice of conditional signals?\n\nA7: Thanks for the question. The conditional signals are pre-defined according to the existing diverse prompt learning methods. Please refer to Q&A 4 of Reviewer Zno2 for details.\n\n>Q8: Can SFG be trained separately from the main model? Consider knowledge distillation techniques to refine surrogate feature generation.\n\nA8: Thanks for the valuable suggestion. At this stage, SFG cannot be trained separately since it is intrinsically a fine-tuning approach and builds upon $w$. We will keep on research to explore the idea of applying pre-trained models and knowledge distillation to replace the cross-attention module for SFG.\n\n>Q9: Does SurPL introduce additional latency during inference?\n\nA9: No significant additions compared with baseline DVLP. Please refer to Table.2 for details.\n\n>Q10: Effect of $\\rho$.\n\nA10: We explore the effect of $\\rho$ on cross-dataset setting. The results indicate a seen-unseen trade-off.\n\n|     |  Seen | Unseen |\n|:---:|:-----:|:------:|\n| 0.1 |  74.2 |  65.13 |\n| 0.2 | 73.33 |  66.61 |\n| 0.3 | 72.74 |  66.74 |", "claims_and_evidence": "Efficiency Improvement\n\nClaim: SurPL reduces computational costs by avoiding gradient-based optimization for diverse prompts.\nEvidence: Tables 1 and 2 compare SurPL with diverse and single-prompt learning baselines, showing reduced GPU memory usage and faster training/testing times.\nPerformance Enhancement\n\nClaim: SurPL achieves state-of-the-art accuracy while being computationally efficient.\nEvidence: Tables 4 and 5 demonstrate that SurPL outperforms existing diverse and single-prompt methods in few-shot learning and generalization.\nFlexibility in Prompt Learning\n\nClaim: SurPL generalizes to multiple diverse prompt learning strategies.\nEvidence: The authors show how SurPL can generate instance-dependent and fine-grained prompt representations.\nGeneralization with SAM (SurPL-G)\n\nClaim: SurPL-G enhances generalization using SAM.\nEvidence: Table 5 shows that SurPL-G achieves superior performance on base-to-novel generalization.", "methods_and_evaluation_criteria": "Methods and Evaluation Criteria\nThe paper employs rigorous evaluation methodologies:\n\nBenchmarks: 15 widely used datasets (e.g., ImageNet, EuroSAT, UCF101).\nComparison with state-of-the-art methods: Covers both single and diverse prompt learning approaches.\nAblation studies: Justifies the effectiveness of individual components (e.g., instance-dependent and fine-grained surrogate prompts).\nEfficiency analysis: Demonstrates reduced GPU memory and computational overhead.\nPotential Issues:\n\nThe paper does not explore the impact of different types of conditional signals on surrogate text feature generation.\nWhile the SAM optimization improves generalization, the effect of different perturbation radii is not analyzed in detail.", "theoretical_claims": "The paper does not introduce significant new theoretical contributions but relies on well-established concepts like contrastive learning, attention mechanisms, and SAM.\nThe derivations of loss functions (Equations 1-11) appear correct but lack deeper theoretical justification on why surrogate features retain meaningful semantic information.", "experimental_designs_or_analyses": "Strengths:\n\nThe experimental setup is comprehensive and includes efficiency, accuracy, and generalization evaluations.\nThe comparisons are fair, using the same baseline (Dense Visual-Language Prompt, DVLP).\nAblation studies effectively demonstrate the impact of different components.\nWeaknesses:\n\nThere is no qualitative analysis of how surrogate text features differ from learned diverse prompts.\nThe paper does not discuss potential failure cases (e.g., when SurPL might underperform compared to explicit prompt learning).\nHyperparameter sensitivity analysis is missing.", "relation_to_broader_scientific_literature": "The paper builds on prior work in:\n\nPrompt Learning: CoOp (Zhou et al., 2022), CoCoOp (Zhou et al., 2022b), PSRC (Khattak et al., 2023b).\n\nVision-Language Models: CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021).\n\nGeneralization Techniques: Sharpness-aware Minimization (SAM) (Foret et al., 2021).\n\nThe work is well-motivated and aligns with the ongoing trend of efficient fine-tuning methods for large pre-trained models.", "essential_references_not_discussed": "The paper could benefit from a discussion on meta-learning approaches for efficient adaptation, such as:\n\n- Ha, Hyeonmin, et al. \"Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service.\" Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.", "other_strengths_and_weaknesses": "Strengths\n\nAddresses an important limitation (high computation cost of diverse prompt learning).\n\nProvides a strong empirical validation across various benchmarks.\n\nThe proposed method is flexible and compatible with existing VLMs.\n\nWeaknesses\n\nThe theoretical foundations of surrogate feature generation are not well explored.\n\nNo qualitative analysis (e.g., feature visualization) to show how surrogate features compare to explicitly learned prompts.\n\nThe limitations section is brief and does not discuss potential failure modes.", "other_comments_or_suggestions": "Feature Interpretability: Provide visualization of the generated surrogate features.\n\nRobustness Analysis: Evaluate the sensitivity of SurPL to different choices of conditional signals.\n\nAblation on Surrogate Feature Generator: Analyze how reducing SFG complexity affects performance.", "questions_for_authors": "How does SurPL handle domain shifts compared to explicit diverse prompt learning?\n\nCan the surrogate feature generator be trained separately from the main model?\n\nDoes SurPL introduce additional latency during inference?\n\nHow sensitive is SurPL to the choice of conditional signals?\n\nHave you considered using knowledge distillation techniques to refine surrogate feature generation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposed a new prompt learning method called Surrogate Prompt Learning (SurPL) for adapting CLIP in downstream datasets. The main idea of SurPL is to learn a feature generator to generate diverse text features from a single prompt. The advantage of this framework over existing methods is that it does not require complex gradient computations while maintaining competitive performance. The paper received four reviews with 4x weak accept ratings and the reviews are mostly positive. Specifically, the reviewers appreciated the extensiveness of the results and the strong performance. On the negative side, the reviewers mentioned that the method lacks a theory that justifies why the complexity is low compared to existing methods. Currently only empirical results are provided. Considering that the paper has solid results and the method has novelty, the AC recommends that the paper be accepted.
Decision: Accept (poster)