Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation
Authors: Zixian Guo
Yuxiang Wei
Ming Liu
Zhilong Ji
Jinfeng Bai
Yiwen Guo
Wangmeng Zuo
Venue: EMNLP 2023 Findings (2023)
Keywords: Prompt Tuning
Black-box Model
Vision-language Model
TL;DR: We propose a black-box tuning method for few-shot adaptation of vision-language models.

Abstract: Parameter-efficient fine-tuning (PEFT) methods have provided an effective way for adapting large vision-language models to specific tasks or scenarios. Typically, they learn a very small scale of parameters for pre-trained models in a white-box formulation, which assumes model architectures to be known and parameters to be accessible. However, large models are often not open-source due to considerations of preventing abuse or commercial factors, hence posing a barrier to the deployment of white-box PEFT methods. To alleviate the dependence on model accessibility, we introduce collaborative black-box tuning (CBBT) for both textual prompt optimization and output feature adaptation for black-box models. Specifically, considering that the backpropagation gradients are blocked, we approximate the gradients of textual prompts by analyzing the predictions with perturbed prompts. Secondly, a lightweight adapter is deployed over the output feature of the inaccessible model, further facilitating the model adaptation process. Empowered with these designs, our CBBT is extensively evaluated on eleven downstream benchmarks and achieves remarkable improvements compared to existing black-box VL adaptation methods. Our code will be made publicly available.

Black-Box Tuning of Vision-Languaae Models with Effective Gradient Approximation
Zixian Guol
Yuxiang Wei
Ming Liu'
Zhilong Ji
Jinfeng Bai²
Yiwen Guo'
Wangmeng Zuo Zuo (.gg))
Harbin Institute Technology
"Tomorrow Advancing Life
Pazhou Lab, Guangzhou
"Independent Researcher zhi zhilangji eenggiihhtmmal...mmm com
zixian_guo@ff sttafooxaal..comm
yuxiang wwrig..wei.eo www. wei.ccs@gmail.cmm com
csml ssmliu@uutlook.co com
jfbai. tegmail. com quoyiwen89@gaiil.com
wrzzuehit it.iii..e edu.cn cn
Abstract
et al., 2022) tasks. Most existing methods conduct parameter-efficien fine-tuning (PEFT (Houlsby et al., 2019)), which updates a tiny fraction of the model parameters or introduces a small number of extra parameters for tuning, in order to trans- fer pre-trained knowledge in computation- and data-efficient manner.
Parameter-efficien fine-tuning (PEFT) meth- ods have provided effective way for adapting large vision- Aasion-anguuag models to specific tasks scenarios. Typically, they learn very small scale of parameters for pre-trained models in white-bon box formulation, which assumes model architectures be known and parameters to be accessible. However, large models are of- ten not open-source due to considerations of preventing abuse commercial factors, hence posing barrier to the deployment of white-box PEFT methods. To alleviate the dependence model accessibility, we introduce collabora- tive black-box tuning (CBBT) for both textual prompt optimization and output feature adap- tation for black-box models. Specifically, con- sidering that the backpropagation gradients are blocked, approximate the gradients of tex- tual prompts by analyzing the predictions with perturbed prompts. Secondly, lightweight adapter is deployed over the output feature of the inaccessible model, further facilitating the model adaptation process. Empowered with these designs, our CBBT is extensively eval- uated on eleven downstream benchmarks and achieves remarkable improvements compared to existing black-box VL adaptation methods. Our code will be made publicly available.
Although impressive improvements have been achieved, standard PEFT methods need to pass signals forward and backward through the entire pre-trained model to update the parameters, which relies on the availability of the architecture, param- eters, and even the inference source code of the model. Nevertheless, the trend of building machine learning models as a service leads to many propri- etary services that only provide an API interface for model inference, e,g., ChatGPT, Bard, and GPT- 4, where the parameters and inference code of the models are not open-source due commercial safety considerations. Under such black-box cir- cumstances, existing PEFT methods can hardly be adopted. Thus, worthwhile to develop methods that can tune pre-trained VL. models in black-box setting. Moreover, in the era of large foundation models, running super large pre-trained models on local devices can be very costly the scale of the pre-trained model has constantly increased. Al- though existing PEFT methods restrict learnable parameters to fairly small scale, is still burden to accommodate models with billions of parameters in limited computing resources for most users.
1 Introduction
Large-scale vision-language (VL) models (Radford et al., 2021; Jia et al., 2021: Li et al., 2021; Yao et al., 2021; Alayrac et al., 2022; Yuan et al., 2021) have demonstrated remarkable performance in wide range of applications. Various model fine- tuning methods have been proposed to exploit the potential of pre-trained VI. models for downstream vision (Zhou et al., 20226; Lu et al., 2022b; Wang et al., 2022; Sun et al., 2022c; Zhang et al., 2022; Wortsman et al., 2022: Li et al., 2023) and nat- ural language processing (Lu et al., 2022a; Yan
To tackle these problem of tuning black-box VL models, there exist a few very recent efforts. For instance, BlackVIP (Oh et al., 2023) pioneered black-box prompting for VI models by learning asymmetric autoencoder-sstlle coordinator with zeroth-order optimization modify visual prompts in the pixel space. However, modifying prompts in the large pixel space causes inefficiency and the method requires up to 9k parameters in the coordi- nator to achieve the goal. Besides, the performance
Work done when Zixian Guo was an intern at TAL..

---

of their visual prompts is subject to the diverse semantic features of well-trained generative self- supervised learning model. Even so, the method demonstrates limited performance improvements after prompting, showing that prompt tuning in the black-box setting is very challenging.
We adapt the visual features jointly with the tex- tual prompt when output features are available. The comprehensive comparison shows that our method achieves state-of-the-an performance compared to other black-box tuning approaches.
In this paper, we propose collaborative black- box tuning method dubbed CBBT for tuning pre- trained VL models and adapting them to down- stream tasks. Unlike in BlackVIP (Oh et al., 2023), we learn the prompt for the textual input instead of images, and we adapt the visual features using an adapter. The basic idca is illustrated in Fig. 1. A query- approximation method (Wier- stra et al.. 2014) is used estimate the gradients and optimize the textual prompt with the black- box pre-trained VL model, from which true gradi- ents are not accessible. Specifically, we query the model with randomly perturbed prompts and then summarize the change in model prediction loss to estimate the gradient of learnable parameters (i.e.. the prompts). We equip single-step gradient op- timization with information from history updates via a momentum strategy, which leads faster convergence and better results.
2 Related Work
Black-box Prompt Tuning for Large Language Models. BBT (Sun et al., 2022b) adopts derivative- free optimization using covariance matrix adapta- tion evolution strategy (CMA-ES) (Hansen et al., 2003) to optimize the prompt in a low-dimensional intrinsic subspace. With this method, the adapta- tion of large language models works well on natu- ral language tasks, surpassing even the white-box prompting performance. BBTV2 (Sun al., 2022a) further enhances the capacity of BBT by using deep prompt tuning. BDPL (Diao et al., 2022) tunes a set of discrete prompts for language models by model- ing the choice of words in the prompt as a policy of reinforcement learning, and variance-reduced policy gradient estimator (Williams, 1992; Dong et al., 2020; Zhou et al.. 2021) is used to optimize the discrete prompt based on loss value.
Under the circumstance where the output fea- tures are available for the pre-trained VL models, we further adapt the visual features by introducing a lightweight adapter module. As demonstrated in Fig. 1, the visual adapter can be learned effortlessly by supervised learning, without having knowledge of the pre-trained VL backbone.
Black-box Adaptation for VL Models. To the best of our knowledge, BIackVIP (Oh et al., 2023) is the first work to tackle black-box tuming problem of pre-trained VL models. It designs an asymmet- ric autoencoder-style coordinator to generate input- dependent image-shaped visual prompts and opti- mize the coordinator by zeroth-order optimization using simultaneous perturbation stochastic approx- imation (SPSA) (Spall, 1992, 1998, 1997). How- ever, the improvement brought by this method (af- ter visual prompting) is relatively limited compared to the baseline, ie., the pre-trained CLIP (Radford et al.. 2021). LFA (Ouali et al., 2023) liberalizes the regimes of black-box models by assuming pre- computed features from pre-trained backbones are accessible. They optimize projection layer for a better alignment between pre-computed image features and class prototypes by a multi-stage pro- cedure. They first solve the orthogonal procrustes problem (Schönemann, 1966) by singular value decompasition (SVD) and further refine the projec- tion matrix using adaptive reranking loss. Albeit superior adaptation performance is obtained, we ad- vocate that the complex-phased optimization can be substituted by end-to-end supervised learning with a lightweight adapter, which effortlessly provides comparable results given labeled image features.
With the joint optimization of the textual prompt and the visual adapter, our CBBT achieves signifi- cant model adaptation performance. To evaluate its effectiveness, we conduct extensive experiments on eleven downstream benchmarks, showing superior performance compared to existing black-box VL adaptation methods.
The main contributions of this work can be sum- marized as follows:
We advocate textual prompting for adapting pre- trained black-box VL models to downstream tasks. Satisfactory prompt tuning results are ob- tained with an effective gradient approximation algorithm.
We expedite the tuning process by utilizing his- tory updates as beneficial information for each optimization step, which brings about acceler- ated convergence and better results.

---

Perturbation Based Gradient Estimation
Prompt
Losses
+
And
hroppt
Prmpet
Input Interface
Text
Output Interface
Textual Class
X+
y
Embeddings
Encoder
q perturbations
y1 8 22...
Image
Feature
Encoder
Adapter
(Black Box)
Learnable Back Propagation
Gradient Estimation
Cosine Similarity
Figure I: Overview of our proposed method. We collaboratively optimize the textual prompt and the image feature
adapter for the adaptation of black-box pre-trained VL models. The prompt is optimized by estimated gradients since backpropagation cannot be applied to the black-box model. The visual adapter module learned by direct supervised learning given output features from the pre- pre-trained model.
3 Method
class of few-shot image samples:
3.1 PEFT in the Black-box Framework
Φ- arg min C(y, x, Φ)
Here we introduce the general form of prompt tun- ing and adapter method and the dilemma when upplied to black-box VL models.
(3)
arg min log P(ù 3/ ()
Prompt tuning for VL models. Given a pre- trained VL model, e.g., .,, CLIP (Radford et al., 2021), existing soft prompt tuning approaches (Zhou et al., 2022b,a; Sun et al., 2022c) for classification tasks typically prepend learnable embeddings to the class names of the target dataset:
When given a while-box model, it is straightfor- ward to calculate the gradient of with respect to the prompt, and optimization of the prompt can be performed via gradient descent:
Φ Φ: - VoCy,,
(4)
Unfortunately, in the black-box setting, the gradi- ents are unable to be backpropagated through the pre-trained black-box Enc and EneT via the chain rule, and the term VoC(y, x,, Φ) camnot be directly obtained. Thus, current gradient-basee prompt tun- ing methods are not feasible in this situation.
(c::) = [v]
ci)
(1)
where E .I,..., C. denotes the index of classes, C; denotes word embedding of the i-th class name Gi. For j E (1,..., M). 2 is a learnable word embedding whose dimension the same as the dimension normal word embeddings in the vo- cabulary. The prediction of an input image æ is obtained by computing similarities between the im- age feature f and prompted textual class features (t))
Adapter learning for VL models. Adapter learn- ing methods (Gao al., 2021; Zhang et al., 2022) for VI. models usually manipulate the output fea- tures of pre-trained models for adaptation target tasks. For instance, an adapter module can be intro- duced to transfer the visual features to new domains with f *f()). and then the prediction is obtained by:
(t, f\/T)
PΦù = =1x;;
(2)
Ej= E expl f//T
exp( f//T) ex- ft/)
PΦý i|æ; Φ)
(5)
where the features of images are encoded by pre- trained image encoder f - Enc((x), and textual class embeddings are generated by text encoder t; EnoT ( (b.)). (.--) calculates the cosine sim- ilarity and is temperature parameter.
Learning such an adapter module by minimiz- ing L(y, f. +/) does not require back-propagation through the entire pre-trained VL model, which provides convenience for adaptation without know- ing the details of the backbone model. But access
The objective of prompt module & is maximiz- ing the classification probability of the ground-truth

---

the output features of the pre-trained model is required to construct and optimize the adapter mod- ule (Zhang al., 2022; Ouali et al., 2023).
in Section 3.2. Then, we explain how expedite the tuning process by leveraging information from previous updates to achieve better optimization in Section 3.3. Finally, we introduce the adapter module and joint training schedule in Section 3.3.
Further Analyses of the Black-box PEFT. Given a black-box pre-trained model, the unavailabilityy of gradients set a barrier to prompt tuning. There- fore, we intuitively have the idea of optimizing the prompt by estimating gradients. Input gradient ap- proximation has been explored in the application of black-box model attacks (Ilyas et al., 20,a) and black-box model reprogramming (Tsai al., 2020). We employ perturbation-base gradient approximation method to estimate the gradient of learnable parameters the prompt. The estimated gradient serves as an effective guide for the tuning of the prompt.
3.2 Perturbation Based Gradient Approximation
Suppose the prompt module & has parameter 0 with dimension D. Let f(0) be the loss function defined in Eq. (3). To approximate the gradient of the loss function with respect to 6, one possible av- enue is to add small increment each dimension of 0 and sum up the slope of all dimensions:
D 1(0 + 3e,) 1(0)
g
(6)
Although the gradient approximation technique provides barely satisfactory optimizing guidance, it is still suboptimal compared to the real gradi- ents. Merely conducting single-step gradient de- scent based the results of the estimated gradient leads to inefficient training. Inspired by the pre- vious design of optimizers, we try to expedite the optimization based on the estimated gradient with a momentum. The basic idea that information from previous updates is useful for the current step, and accumulated gradients possibly provide more promising exploration directions. we empirically find that equipping the momentum strategy for gra- dient approximation brings expedited convergence and remarkable adaptation performance gain.
B
where 6 one-hot vector and its i-th element is equal to 1. Such an approximation may work well for low-dimensional parameters but is not suit- able for problems where D might be large. For example, the dimension of each word embedding of pre-trained CLIP is 512, i.e., O E R MG12, Thus M x 512 independent API calls for the black-box model must be applied obtain the complete esti- mated gradient of parameter which causes inef- ficiency.
To alleviate the cost of the above gradient esti- mation method, we adopt stochastic perturbation- based gradient estimation technique formulated as:
Although we have no access to the internal vari- ables of typical black-box models, under the cir- cumstance where output features of the pre-trained VL backbone are available, post-proue postpescessing adapter modules can be directly learned by labeled samples for PEFT.
9. b. f(0 + Be,) f()) E
(7)
3
9: the slope of the loss function along the direc- tion of the perturbation. e, is a vector randomly drawn from unit sphere with an L2-norm of 1. & is small value controlling the scale of perturba- tions. b is scaling factor balancing the bias and variance trade-off of the estimator.
Motivated by the above analyses, we propose to adapt black-box VL models with a collaborative PEFT consisting of optimization from two perspec- tives. Firstly, we tune a textual prompt under the guidance of the estimated gradient. Perturbation- based gradient approximation and effective opti- mization strategy used to facilitate the training. Secondly, learn a lightweight adapter transfer pre-trained visual features. Joint optimization of the prompt and adapter brings superior adaptation performance. The overview the proposed model is illustrated Fig. 1.
To mitigate noise in the estimated gradients, we sample random perturbation e; for times, and the gradient slope of directions (Wierstra et al., 2014; Ilyss et al., 2018a; Tu et al., 2019):
approximated by averaging the
g
9
(8)
In the following, we begin by presenting the perturbation- based gradient approximation method
The upper bound of the estimation g w.r.t the true gradient (f( is analyzed in Tu et al. (2019)'s

---

expedited convergence and superior adaptation per- formance.
1.4
ADAM
LAMB
1.2
SGD
SGD-M
Normalized Loss
1.0
ADANN
3.4
Visual Adapter Module
0.8
The pre-trained VL models can be effectively adapted to downstream tasks through the black-box prompt tuning method mentioned above. Mean- while, under the assumption that having access the output features of the black-box model (Ouali et al., 2023), a lightweight adapter module can be directly learned from labeled few-shot samples.
0.6
0.4
0.2
50
100
150
200
250
300
Number of iterations
Figure 2: Trend of loss during training on EurOSAT. We adopt ADAM optimizer for expedited convergence and superior adaptation performance.
Adapter modules (Houlsby et al., 2019; Gao et al., 2021; Zhang et al., 2022) have been proven to be effective in the adaptation of VL models. During the training process of the adapter, the gradients do not need to be back pack-propagated through the entire pre-trained model, making possible to equip the adapter module with black box models of which only the output features are available.
paper as:
82
b²
E glg V 4(
D
Dq
(b -D2 D2
2g + 62 32
[[)
The text features have been adapted in our method by tuning the learnable prompt. Thus, we introduce an adapter module only for the visual fea- tures to achieve collaborative adaptation. Specifi- cally, we add an adapter module to the output of the visual encoder of the pre-trained VL model. Access to computed image features and labels allows the adapter to be learned at ease through direct super- vised learning. During training, the visual adapter module and text prompts are optimized in turn to achieve joint adaptation.
Setting smaller 3 can reduce the last error term in Eq. (9) but may cause increase in noise due numerical precision. Increasing the number of samples q reduces the first error term but consumes more queries for the model API.
3.3 Effective Optimization Based on Estimated Gradient
To expedite the optimization based on the estimated gradient, we facilitate the tuning process by lever- aging the momentum strategy. SSeccificaly,, estimate the first-order moments of the parameters' gradient by m, 31 - (1 -1-33 (1) 9i- The first-order moments accelerate the optimization and reduce the noise in the gradient of cach step. And we obtain the adaptive estimation of the second- order moment by v; = B2 00 + 82) T7 which is used to adjust the learning rate of each dimension adaptively.
In our experiment, attempt two simple but effective adapter designs, CLIP-Adapter (Gao al., 2021) and Tip-Adapter (Zhang et al., 2022). Both of which can be well suited for the manipulation of image features for better adaptation.
4 Experiments
4.1 Implementation Details
Datasets. We perform the few-shot adaptation on black-box pre-trained CLIP (Radford al., 2021) for image classification tasks following the general protocol in existing methods (Zhou et al., 2022b; Ouali et al., 2023; Oh et al., 2023). In par- ticular, we adopt 11 commonly used datasets to evaluate our method, including ImageNet (Deng et al., 2009), Caltech101 (Li ct al., 2004), Oxford- Pets (Parkhi al., 2012), StanfordCars (Krause et al., 2013). Flowers 102 (Nilsback and Zisser- man, 2008), 101 (Bossard et al., 2014), FGV- CAircraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), UCF101 (Soomro et al., 2012), DTD (Cim-
In our experiments, we use optimizers that inte- grate the momentum as a practical implementation. To analyze the optimization results of different op- timizers, we illustrate the trend of normalized loss value [C(0") L(0) / /[("") - L(fo) in Fig. 2. Adam (Kingma and Ba, 2014) shows fast and steady convergence and satisfied final results. We have also tried more advanced techniques, e.g.. LAMB (You et al., 2019), but no significant im- provement performance observed. Empirical results show that optimizing the prompt with Adam optimizer based on the estimated gradient provides

---

Table 1: Few-shot adaptation performance on 11 image classification tasks. Black-box methods are indicated with
gray shadows.
Model
Method
Pets
Flowers
FGVCA
DTD
EamSAT
Cars
Foad101
SUN397
Caltech
UCF
ImageNNtt
Avg.
CoOp eto)
89.9
85.4
25.7
61.5
80.2
60.8
79.6
66.9
91.9
72.0
62.9
70.6
Adupter
67.4
77.0
92.5
77.8
60.4
Adapter
89.7
95.2
37.5
67.6
84.4
74.7
78.9
70.6
93.1
78.3
63.9
75.8
ZSCLIP
85.8
66.1
17.3
42.3
37.6
55.6
77.3
58.5
86.3
61.5
58.2
58.8
1.FA Ours (wlo Adapter)
86.8
94.6
35.9
66.4
84.1
73.6
76.3
71.3
92.7
77.0
63.7
74.8
60.9
91.6
69.6
69.3
Ours (CIIP. Addpter)
88.0
94.9
35.6
67.4
85.1
73.8
77.3
68.5
93.1
78.6
63.8
75.1
Ours (Tip-adapter)
89.9
95.2
37.3
68.4
85,3
74,5
78.5
71.0
92.6
79,4
64.6
76.1
CoOp ctx)
93.5
91.6
33.1
66.1
85.3
71.4
87.3
72.0
95.7
79.8
71.0
77.0
CoCoOp (1 cts)
92.8
86.7
31.4
61.7
73.8
68.9
87.1
71.6
94.8
77.4
70.5
74.2
VIT/B16
CoCoOp cts)
CLIP-Aduper
97.2
43.3
72.7
89.0
85.8
74.3
96.3
84.2
70.0
80.4
Tip-Adapter
91.3
97.5
46.8
73.7
88.3
83.9
87.5
76.1
95.8
84.3
71.8
81.7
ZSCLIP
89.2
71.3
24.7
44.4
47.6
65.3
86.1
62.5
92.9
66.8
66.7
65.2
BlackVIP
70.6
45.2
73.1
86.6
64.7
93.7
67.1
68.2
LFA
92.4
96.8
71.9
87.3
87.1
76.7
96.2
84.0
72.6
81.2
Ours (w/o adapter)
93.7
88.6
30.7
64.0
81.0
68.9
87.2
71.1
95.8
78.8
70.6
75.5
Ours (CLIP- (CLIP-Adapter)
92.2
91.2
45.3
73.3
88.8
81.2
86.1
74.8
95.8
84.6
71.9
81.0
Ours (Tip-Adapter)
93.8
97.8
46.6
74.1
88.3
83.5
87.3
75.9
95.9
84.9
72.4
81,9
poi et al., 2014), and EuroSAT (Helber et al., 2019).
78.0
For each dataset, labeled few-shot samples from each class used training data.
77.5
00.5
EunSAT
76.5
60.0
Learnable Prompts. The learnable prompts are
76.0
shared across all classes in the target dataset. By de- fault, the length of the prompt is set to be M = 1, which reduces the number of parameters in the learnable prompt. small parameter optimization space helps maintain the quality of the estimated gradients with limited resource for exploration, re- sulting in effective tuning results. The effect of different prompt sizes is analyzed in Sec. 4.4. To initialize the prompt with different length, we use "a", "a photo", "a photo of a", and "a photo of photo of a" for M 2, 1, 8, respectively.
75.5
50.0
75.0
24.5
58.5
EuroSAt
74.0
128
256
S12
Number of queqquussss used for gradient approximation
Figure 3: Ablation results of "Ours (w/o Adapter)" with different
4.2 Few-shot Adaptation Performance on Image Classification
We conduct extensive experiments 11 datasets evaluate our proposed method. Table reports the 16-shot adaptation performance competing meth- ods. For a comprehensive comparison, we include both white-box PEFT methods (i.e., CoOp (Zhou et al., 2022b), CoCoOp (Zhou et al., 2022a), CLIP- Adapter (Gao et al., 2021) 1), and Tip-Adapter (Zhang et al., 2022)) and black-box methods (i.e., Black- VIP (Oh al., 2023) and LFA (Ouali et al., 2023)) "ZSCLIP" denotes the outcomes obtained using manually designed hard prompts.
Aapter Module. Following CLIP-Adapter (Gao et al., 2021), our adaptor module adopts two-layer MLP that follows the pre-trained visual encoder. The input and output dimensions are the same as the dimension of the CLIP image feature, and the number of hidden units is a quarter. Following Tip-Adapter (Zhang et al., 2022), we use the aver- aged feature of random augmented training images from 10 epochs as the initialization of the cache to construct the projection layer.
Training Details. We employ the official CLIP model to evaluate our proposed method. For com- prehensive comparison, we conduct experiments with different visual backbones, i.e.. ResNet50 and ViT/B16. The query number q is set q = 256 by default, and its effect discussed in Sec. 4.4. The hyperparameters and Eq. (7) are set as and 1/D, respectively. D is the dimension of the parameter in the prompt.
From Table 1, our black-box prompt tuning method (ViT/B 16 backbone) surpasses previous work Oh et al. (2023) with an average accuracy mar- gin of 7.3% across 11 datasets, demonstrating the effectiveness of our black-box textual prompting for the adaptation of the VL model. Furthermore. when the context length of the prompt fixed M 1, our black- -ox prompt tuning method per- forms comparably to the white-box prompt method,

---

OxfordPeets
DTD
EuroSAT
30.4
85.0
64
00.2
62-
90.0
CoOp
60.0
adapteri
60
adapteri
Ours (we adapter)
09.0
89.6
75.0
56
89.4
89.2
54
89.0
16
16
1.2
16
Context longth of the promot
Context length of the prompt
Context length the prompt
Figure 4: Ablation results of "Ours (w/o Adapter)" with different context length of the prompt.
OxfordPots
DTD
EuroSAT
90.0
80
80
50
75
56-
54-
88.5
52-
50-
80
88.0
Colp 13 ctxl
48
Cctip cte)
CoOp ctdl
Ours (wit edepteri
Ours óeto adapter))
55
Ours twi wdapter)
46
Number labeled training exampies
Number labaled training examples
Namber labeled training axamples
Figure Ablation results of "Ours (w/o Adapter)" with different quantity of few-shot training data.
Table Comparison of different black-box optimizers.
our method with two other different optimization al- gorithms based on our implementation, In partieu- lar, CMA- ASES algorithm (Hansen et al., 2003) is con- sidered as state-of-the -o--t--rtttttttt in evolutionary computa- tion and is previously used to optimize the prompt for large language models (Sun et al., 2022b,a). SPSA-GC was proposed by BlackVIP (Oh et al.. 2023) to learn a visual prompt for adaptation of pre-trained CLIP.
Method
API calls/ iter
EuroSAT Acc.
CMA-ES
10
61.9
SPSA-GC
10
60.1
Ours (w/o adapter)
10
62.6
i.e., CoOp (1 ctx), with a slight difference of less than 2%.
By assuming pre-computed features are avail- able, LFA (Ouali et al., 2023) optimizes a projec- tion layer in multi-stage procedure as introduced in Section 2. We advocate that end-to-end learning of adapter methods (Gao et al., 2021; Zhang et al., 2022) provides much more brief avenue mean- while gives satisfactory performance. As shown in Table 1, optimizing the adapter module from CLIP-Adapter and Tip-Adapter can achieve com- parable performance with LFA. Thus, we integrate our black-box prompt tuning method with these more flexible adapter modules. From Table 1, the collaborative adaptation of black-box prompting and adapter module brings remarkable performance and achieves new state-of-the-ee result.
For a fair comparison, we unify the number API calls per iteration for all competitors to 10. This is achieved by: setting the population size of CMA-ES as 10; setting the number of repeated two-side estimations of SPSA-GC as 5; setting the number of samplings of our perturbation- gra- dient approximation as g 10. The experiments are conducted on CLIP ResNet50 model, and the prompt length was set to 1. All optimizers are trained for 750 iterations until convergence, and the results are listed in Table 2. From the Table, our method outperforms the SPSA-GC algorithm, which is also based gradient estimation. Al- though CMA-ES exhibits faster convergence, no- ticeable fluctuations are observed even in the later stages of training. Our perturbation- based gradi- ent approximation method more suitable for the adaption of the VL model.
4.3 Comparison with Black-Box Optimizers
Existing black-box prompt tuning methods have explored various effective optimization techniques when the gradient is unavailable. Here we compare

---

4.4 Ablation Study
fault configuration and vary the number of sam- ples used for prompt tuning, Both black box and white box models undergo the same number of iter- ations. As shown in Fig. 5, increasing the number of samples clearly leads to better adaptation results. Moreover, we observe that in extremely data-scarce scenarios with only 1-shot sample per class, tuning the prompt based on the estimated gradient outper- forms white-box tuning on all three datasets. One possible explanation is that optimizing with true gradients can lead overfitting when the amount of data is too small. In contrast, gradient approxima- tion provides a more robust optimization direction. As the amount of data increases, the advantages of direct white-box learning become more obvious. Table 3: Ablation study on the training schedule.
Ablation studies are performed to evaluate the ef- fect of various factors, including the number of queries, the prompt length, the number of few-shot samples, and the collaborative training schedule. The experiments are mainly on the CLIP ResNet50 model.
Effect of the number of queries q. The number of samplings g controls the times of querying the black-box model in each iteration. It has a signif- icant impact on the number of API calls required for learning the prompt. Fig. illustrates the adap- tation performance with different y values. Gener- ally, larger values of q yield more reliable gradients but also require more time and API calls for the black-box model. To trade-off the performance and computational cost, we use g = 256 for the results presented in Section 4.2.
Datasets
P-A
A-P
ALT
EuroSAT
84.0
82.7
85.1
DTD
66.4
65.7
67.4
Effect of prompt length. We further investigate the effect of Prompt length M. For comparison, all the experiments are conducted under 16- 16-shot training data, with the same number of sampling (q = 256) and iterations. The results are illustrated in Fig. 4. One can see that the trend of performance on different tasks varies as the context length of the prompt changes. For white-box prompt tuning, longer prompts usually can lead to better adapta- tion to downstream datasets, e.g., DTD and Eu- roSAT. However, blindly lengthening the context (e.g. M = 16) will not result in continuously ris- ing performance. Increasing the length of context brings little improvement for OxfordPets. We at- tribute these results to the varying degrees of data diversity among different tasks.
Caltech101
91.9
92.7
93.1
OxfordPets
87.9
87.5
88.0
Effect of the collaborative training schedule, our experiment, the prompt and the adapter module are optimized jointly to maximize their collabora- tive performance. During training, we alternately update the prompt and the adapter module at differ- ent epochs. To assess the effectiveness of this joint optimization schedule, conducted experiments using three different ways of training: (i) tuning the prompt until convergence and then optimizing the adapter module (P-A); (ii) tuning the adapter module until convergence and then optimizing the prompt (A-P); (iii) our collaborative training sched- ule (ALT). We train "Ours (CLIP-Adaples the above three schedules, and the results are shown in Table 3. As shown in the table, recurrently up- dating the prompt and the adapter alternately (ALT) achieves superior collaborative adaptation perfor- mance, demonstrating its effectiveness.
under
But in the case of black-box models, the experi- mental phenomenon changes due to the influence of gradient approximation. Lengthening the context of the prompt brings trivial benefits and may even result in noticeable performance degradation. The expanded parameter space of long context leads to practical difficulties in gradient estimation thus the optimization may lead to a suboptimal result. Increasing the number of sampling q may improve the reliability of estimated gradients, but scaling up q in proportion to the size of the prompt leads to severe inefficiency. Thus, we use the prompt length of as trade-off.
5 Conclusion
In this paper, we present CBBT, a black-box adapta- tion approach for VL models. We effectively tune a soft prompt for the text encoder by gradient approx- imation and jointly learn lightweight adapter mod- ule to transfer the visual features of the pre-trained backbone. Equipped with the textual prompt and the visual adapter, our method achieves a collabo- rative adaptation for both modalities. Experiments on various datasets show that our CBBT performs favorably against the state-of-the oh- art methods.
Effect of the number of few-shot samples. The number of few-shot samples determines the amount of training data used to adapt the pre-trained VL model. To demonstrate its effect, we keep the de-

---

Limitations
IEEE conference on computer vision and pattern recognition, pages 3606-3613.
We optimize the prompt in the original high- dimensional prompt embedding space, which leads to unsatisfactory optimization results for the prompt with long context, shown in Section 4.4. The high-ding nigh-dimensional parameter in the prompt also makes the gradient approximation more difficult. We have tried to optimize the prompt in a smaller subspace following the approach in BBT (Sun et al., 2022b). But the adaptation performance decreased a lot even though we only released a small pro- portion of the original dimensions. The intrinsic dimensionality property (Aghajanyan et al., 2020; Qin al., 2021) for vision-language pre-trained models needs further investigation.
Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, Xiao Zhou, and Tong Zhang. 2022. Black- box prompt learning for pre-trained language models. arXiv preprint N.Ii:2201.08555
7he Dong, Andriy Mnih, and George Tucker. 2020. Dis- arm: An antithetic gradient estimator for binary latent variables. Advances in neural information processing systems, 18:18637-18647
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li. and Yu Qiao. 2021. Clip-adapter: Better vision- language models with feature adapters. arXiv preprint arXiv:21
Besides, we optimize a continuous prompt with the need for the token embedding layer of pre- trained models. Learning a discrete prompt for the adaptation of VL models is worthy of explo- ration, considering that the discrete text prompt provides an explicit explanation, and discrete text inputs more suitable for the invocation of the latest pre-trained model APIs with natural language inputs and/or outputs.
Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. 2003. Reducing the time complex- ity of the derandomized evolution strategy with variance matrix adaptation (ema-es). Evolutionary computation, 11(1):1-18.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226,
Acknowledgements
This work was supported in part by National Key R&D Program of China under Grant No. 2020AAA0104500, and by the National Natural Science Foundation of China (NSFC) under Grant No. U19A2073.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficien transfer learning for nip. ternational Conference on Machine Learning, pages 2790-2799, PMLR.
References
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint 1355
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. 2018a. black-box adversarial attacks with limited queries and information. In International conference machine learning, pages 2137-2146. PMLR
Jean- San-BBatiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. arXiv preprini arXiv:2204.1 14198.
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. 2018b. Prior convictions: Black-box adversarial attacks with bandits and priors arXiv preprint arXiv:1807.074: 1807.07978.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101 min--ingg discriminative components with random forests. In Computer 2014: 13th European Conference. Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. 2021. Scaling up visual and vision- laniuug representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. De- scribing textures in the wild. In Proceedings of the
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69%0

---

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei- Fei. 2013. 3d object representations for fine-graine ine-grained categorization, In Proceedings of the IEFE inter- national conference on computer vision workshops, pages 554-561.
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, al. 2021. Exploring low- dimensional intrinsic task subspace via prompt tun- ing. arXiv preprint arXiv:21
Fei-Fei Li, Rob Fergus, and Pietro Perona. 2004. Learn- ing generative visual models from few training ex- amples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on com- puter vision and pattern recognition workshop, pages 178-178. IEEE
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Yangguang Li. Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Jun- jic Yan. 2021. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint 1rXiv:2110.05200.
Peter H. Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika. 31(1):1-10.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human ac- tions classes from videos in the wild. arXiv preprint arXiv:1212.0402.
Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. 2023. Clip surgery for better explainability with en- hancement in open-vocabulary tasks. arXiv preprint arXiv:2304.06653.
James C Spall. 1992. Multivariate stochastic approx- imation using a simultaneous perturbation gradient approximation. JEEF transactions on automatic con- trol, 37(3):333-33333 3411.
Yujie Lu, Wanrong Zhu, Xin Eric Wang, Miguel Eck- stein, and William Yang Wang. 2022a. Imagination- augmented natural language understanding. arXiv preprint arXiv:2204.08533
James C Spall. 1997. A one-measuremmnt form of si- multaneous perturbation stochastic approximation. Automatica, 33(1):009-11
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. 2022b. Prompt distribution learning. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 5206-5215.
James C Spall. 1998. An overview of the simultaneous perturbation method for efficient optimization. Johns Hopkins apl teclnical digest, 19(4):422-45 492.
Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuan-Jing Huang, and Xipeng Qiu. 2022a. Bbtv2: Towards gradient-free future with large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 3916-3930.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151.
Natalie Maus, Patrick Chao, Eric Wong, and Jacob Gard- ner. 2023. Adversarial prompting for black box foun- dation models. arXiv preprint arXiv:2302.044 042337.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022b. Black-box tuning for -aaggagg-model-s------e-
In International Con- ference on Machine Learning, pages 20841-2088 PMLR.
Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Com- puter Vision, Graphics & Image Processing, pages 722-729. IEEE
Ximeng Sun, Ping Hu, and Kate Saenko. 2022c. Dualcoop: Fast adaptation multi-label recog- nition with limited annotations. arXiv preprint 2rXiv:2206.09541.
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTack Lim, Geunyoung Jung. Jiyoung Jung, Hosik Choi, and Kyungwoo Song. 2023. Blackvip: Black-box visual prompting for robust transfer learning, arXiv preprint arXiv:2303 14773.
Yun-Yur Tsai, Pin-Yu Chen, and Tsung-Yi Yi Ho. 2020. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International Conference Machine Learning, pages 9614 0614-9624. PMLR.
Yassine Ouali, Adrian Bulat, Brais Martinez, and Geor- gios Tzimiropoulos. 2023. Black box few-shot adap- tation for languaee models. arXiv preprint arXiv:2304.0175. 01752..
Chun-Chen Tu, Paishun Ting, Pin- Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsich, and Shin- Ming Cheng. 2019. Autozoom: Autoencoder basedd zeroth order optimization method for attacking black- box neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33. pages 742-749.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEFE.

---

Feng Wang, Manling Li, Xudong Lin, Hairong Lv. Alexander G Schwing, and Heng Ji. 2022. Learn- ing to decompose visual features with latent textual prompts. arXiv preprint arXiv:2210.04287.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16816-16825.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Gold- blum, Jonas Geiping, and Tom Goldstein. 2023. Hard prompts made easy: Gradient-bass discrete opti- mization for prompt tuning and discovery. arXiv preprint arXiv:3302.0360
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9))3337-2348.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. 2014. Natural evolution strategies. The Journal of Machine Learning Research, 15(19999-990.
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. 2021. Efficient neural net- work training forward and backward propagation sparsification. Advances in Neural Information Pro- cessing Systems, 14:11211 16211-15229
Ronald Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Reinforcement learning, pages 5-32.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li. Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi. Ali Farhadi, Hongscok Namkoong, et al. 2022. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971.
Jianxiong Xiao, James Hays. Krista Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: I Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IBEE.
An Yan, Jiacheng Li, Wanrong Zhu, Yujie Lu, William Yang Wang, and Julian McAuley. 2022. Clip also understands text: Prompting clip for phrase un- derstanding. arXiv preprint urXiv:2210.05836.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang. Zhenguo
Li, Xin Jiang, and Chunjing Xu. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.0778.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel. Kurt Keutzer, and Cho-Jui Hsich. 2019. Large batch optimization for deep learn- ing: Training bert in 76 minutes. arXiv preprint arXiv: wwXi::/94.00096.
Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang. Boxin Li, Chunyuan Li, al. 2021. Florence: new foundation model for computer vision. arXiv preprint 2rX
Renrui Zhang, Zhang Wei. Rongyao Fang, Peng Gao. Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Tip-adapter: Training-free adaption of clip few-shot classification. arXiv preprint arXiv:2207.09519.

---

A Generalization Ability of Black-Box Prompt
scaled proportionately according to the size of the prompt, and the results are reported in Table 6.
From the table, with sufficient training time avail- able, proportionately scaling the samplings for tun- ing of the longer prompts achieves stable conver- genee and clear improvements (especially on Eu- roSAT). Nonetheless, optimized prompts con- sistently outperform hand-crafted hard prompts of any length.
To evaluate the generalization ability of our method, we conducted experiments on the extensively eval- uated domain shift benchmarks and base-to-new setting (training on samples from base classes, test- ing on samples from new classes) commonly used in studies for adaptation of CLIP.
Generalization to other domains. Following CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a), we evaluate the transferability of the prompt learned from ImageNet to the three spe- cially designed datasets. The results are shown in Table 4. Given the high variance inherent in these trials, the results are averaged over three random re-runs to ensure reliable comparisons.
C Computational Time Budget
The added computation burden of our method com- pared to white-box prompting methods lies within the multiple samplings required by the gradient approximation. We provide the training duration linked to the tuning methods presented in Table on the EuroSAT dataset in Table 7. All training pro- cedures are conducted on a single 3090 GPU. We record the minutes used for complete training and divide the time by the number of trained epochs to ascertain the time per epoch. While the sampling process inevitably elongates the training period, the overall consumed time acceptable.
Our prompt learned by black-box optimization performs better than CoOp with clear margin. Moreover, compared to CoCoOp, which relies input-conditioned prompts generated by meta- network, our vanilla prompt demonstrates superior performance on two of the three benchmarks.
Generalization from base to new classes. Fol- lowing CoCoOp (Zhou et al., 2022a), we split the classes of the target dataset into sets. In the
D Analysis of the Error in Gradient Estimation
base-to-new setting, the methods are trained us- ing data from base classes and tested separately on base and new classes to evaluate the generalization ability to unseen classes in training. The results are shown in Table 5.
The upper bound of the error of gradient approx- imation is ||V (0) according to Eq. (9). It is a theoretical value obtained through multiple bounding steps in the proof. The actual estima- tion error of the gradient during training is much lower than the theoretical upper bound since the ex- periments are conducted on reasonably annotated datasets with pre-trained CLIP and properly initial- ized prompts. As the training proceeds, the value of the true gradient becomes small, making the er- ror of the estimated gradient, bounded by the true gradient, become small simultaneously. Thus, the results of "Ours (w/o adapter)" are closely compa- rable to "CoOp (1 ctx)" in Table 1.
While CoOp improves pre-trained CLIP on base classes, fails grievously on novel classes. Co- CoOp optimizes for each instance to gain more generalization over an entire task. Our method achieves comparable results to CoCoOp by tuning a single prompt with the black-box optimizer. Opti- mizing the prompt by estimated gradient avoids the trend of overfitting to training samples, thus mak- ing up the superior of our method generalization ability white-box prompt tuning.
More Results with Longer Prompt
E Applying to Larger Black-Box Models
In Fig. 4 of our paper, we optimize prompts with different lengths under fixed training time budget by setting the same number of samplings q 256 for gradient approximation. Such setting ensures training efficiency but may lead to suboptimal re- sults for longer prompts, resulting in performance drop of longer prompts. To demonstrate this, we have conducted experiments in which the value is
It is promising to apply our method to larger black- box models. In fact, there exist closed-sourced model APIs, c.g., GPT-3, that provide the fea- ture extraction function. It is possible to adapt pre-trained models of this kind by transferring the extracted features. Additionally, inspired by re- cent discrete prompt tuning approaches in Maus et al. (2023); Wen et al. (2023), it is practically

---

Table Comparison manual and learned prompt domain generalization. The prompts are learned on 16-shot data from ImageNet.
Source ImageNet
Target
Methods
ImageNet-Sketch
ImageNet-A
ImageNet-R
Zero-Shot CLIP
66.7
46.2
47.8
74.0
CoOp (4 ctx)
71.5
48.0
49.7
75.2
CoCoOp (4 ctx)
71.0
48.8
50.6
76.2
Ours (w/o adapter) 70.7
48.7
50.7
76.6
Table 5: Comparison of manual and learned prompt in the base-to-new generalization setting. The prompts are learned from 16 images per base class.
OxfordPets
EuroSAT
DTD
Methods
Base
New
H
Base
New
H
Base
New
H
Zero-Shot CLIP
91.2
97.3
94.1
56.5
64.1
60
53.2
59.9
56.4
CoOp (4 ctx)
93.7
95.3
94.5
92.2
54.7
68.9
79.4
41.2
54.2
CoCoOp (1 ctx)
94.6
95.6
95.1
84.2
55.3
66.8
75.1
53.6
62.6
CoCoOp (4 ctx)
95.2
97.0
96.1
86.0
59.9
70.6
73.2
55.4
63.1
Ours (w/o adapter)
95.8
95.8
95.8
90.8
71.1
79.8
77.9
51.1
61.7
Table 6: More results with longer prompt and varying samplings "ctx" denotes the length of the prompt.
Methods
ctx
q
OxfordPets
DTD
EuroSAT
Zero-Shot CLIP
80.7
38.2
31.1
Ours (w/o Adapter)
1
256
89.2
60.9
77.3
Ours (CLIP-Adapter)
1
256
88.0
67.4
85.1
Ours (TIP-Adapter)
1
256
89.9
68.4
85.3
Ours (w/o Adapter)
2
256
89.0
60.2
77.5
Ours (w/o Adapter)
2
512
90.0
62.4
77.7
Ours (CLIP-Adapter)
2
512
87.5
67.2
85.4
Ours (TIP-Adapter)
2
512
89.6
68.8
85.0
Zero-Shot CLIP
4
83.6
40.0
24.2
Ours (w/o Adapter)
4
256
89.4
58.8
70.0
Ours (wlo Adapter)
4
1024
89.5
62.2
79.6
Ours (CLIP-Adapter)
4
1024
88.6
66.9
85.3
Ours (TIP-Adapter)
4
1024
89.8
69.3
85.2
Zero-Shot CLIP
8
84.2
39.3
31.0
Ours (w/o Adapter)
8
256
89.4
55.6
75.3
Ours (w/o Adapter)
8
2048
89.7
61.6
81.7
Ours (CLIP-Adapter
8 2048
88.4
66.8
85.5
Ours (TIP-Adapter)
8 2048
90.0
68.4
85.4
feasible to discretize the learned prompts by pro- jecting the continuous embedding discrete token space to support broader range of black-box mod- els that only allows discrete input, e.g., ChatGPT, Bard. Our research will persist in exploring more practical adaptation techniques for vision- languae models.
Table 7: Comparison of training time budget.
Methods
min / epoch
min / train
CoOp
0.017
3.3
CoCoOp
0.120
1.2
Ours (w/o Adapter)
0.095
14.2
Ours (CLIP- (CLIP-Adapter)
0.051
7.7
Ours (Tip-Adapter)
0.054
8.1


Reviews:
[{"review_id": "wPtAHjjcnJ", "reviewer": "Reviewer_QPix", "summary": "", "questions": "Please see the [reasons to reject] section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank **Reviewer QPix** for the valuable suggestions. Below we respond to the questions point by point.\n\n**Q1. Comparisons with more baselines**\n\nWe have conducted the comparisons with CoCoOp ([https://arxiv.org/abs/2203.05557](https://arxiv.org/abs/2203.05557)), and the results are reported in the below table. Specifically, we employ the official implementation of CoCoOp and train it with default training configuration, based on CLIP-ViT/B16, under the few-shot learning setting. As shown in the table, our method performs favourably against CoCoOp, demonstrating its effectiveness.\n\n|Method|Pets|Flowers|FGVCA|DTD|EuroSAT|Cars|Food101|SUN397|Caltech|UCF|ImageNet|Avg.|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|CoOp (1 ctx)|93.5|91.6|33.1|66.1|85.3|71.4|87.3|72.0|95.7|79.8|71.0|77.0|\n|CoCoOp (1 ctx)|92.8|86.7|31.4|61.7|73.8|68.9|87.1|71.6|94.8|77.4|70.5|74.2|\n|CoCoOp (4 ctx)|92.9|85.8|31.4|61.9|72.5|68.3|87.3|72.2|94.9|77.1|71.0|74.1|\n|CLIP-Adapter|92.1|97.2|43.3|72.7|89.0|79.0|85.8|74.3|96.3|84.2|70.0|80.4|\n|Tip-Adapter|93.3|97.5|46.8|73.7|88.3|83.9|87.5|76.1|95.8|84.3|71.8|81.7|\n|ZSCLIP|89.2|71.3|24.7|44.4|47.6|65.3|86.1|62.5|92.9|66.8|66.7|65.2|\n|BlackVIP|89.7|70.6|25.0|45.2|73.1|65.6|86.6|64.7|93.7|69.1|67.1|68.2|\n|LFA|92.4|96.8|46.0|71.9|87.3|82.2|87.1|**76.7**|**96.2**|84.0|**72.6**|81.2|\n|Ours (w/o adapter)|93.7|88.6|30.7|64.0|81.0|68.9|87.2|71.1|95.8|78.8|70.6|75.5|\n|Ours (CLIP-Adapter)|92.2|97.2|45.3|73.3|**88.8**|81.2|86.1|74.8|95.8|84.6|71.9|81.0|\n|Ours (Tip-Adapter)|**93.8**|**97.8**|**46.6**|**74.1**|88.3|**83.5**|**87.3**|75.9|95.9|**84.9**|72.4|**81.9**|\n\n**Q2. Experiments under natural data shift setting**\n\nThanks for the suggestion!  Following CoCoOp, we have conducted experiments on extensively evaluated domain shift benchmarks, and the results are shown in the table below. Given the high variance inherent in these trials, the results are averaged over three random re-runs to ensure reliable comparisons. From the table, our method also shows superior robustness to domain shifts.\n\n|Methods|Source|Target|Target|Target|\n|:----|:----|:----|:----|:----|\n|    |ImageNet|ImageNet-Sketch|ImageNet-A|ImageNet-R|\n|Zero-Shot CLIP|66.7|46.2|47.8|74.0|\n|CoOp (4 ctx)|71.5|48.0|49.7|75.2|\n|CoCoOp (4 ctx)|71.0|**48.8**|50.6|76.2|\n|Ours (w/o adapter)|70.7|48.7|**50.7**|**76.6**|\n\nBesides, to verify the generalizability of our method, we have also deployed our methods to the prevalent base-to-new setting (training on samples from base classes, testing on samples from new classes) commonly used in studies for adaptation of CLIP. From the table below, our black-box prompt tuning yields comparable results to while-box CoOp on base classes while exhibiting significantly improved generalization performance to new classes (especially on EuroSAT and DTD datasets). Such an advantage of our approach can be attributed to less overfitting due to the utilization of the approximated gradient.  It is worth noting that CoCoOp sometimes achieves slightly superior performance than ours on new classes; however, it comes at a considerable cost of degradation of performance on the base classes, which also elucidates the shortcomings of CoCoOp in the vanilla few-shot learning experiment.\n\n|Methods|OxfordPets|OxfordPets|OxfordPets|EuroSAT|EuroSAT|EuroSAT|DTD|DTD|DTD|\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|    |Base|New|H|Base|New|H|Base|New|H|\n|Zero-Shot CLIP|91.2|97.3|94.1|56.5|64.1|60.0|53.2|59.9|56.4|\n|CoOp (4 ctx)|93.7|95.3|94.5|92.2|54.7|68.9|79.4|41.2|54.2|\n|CoCoOp (1 ctx)|94.6|95.6|95.1|84.2|55.3|66.8|75.1|53.6|62.6|\n|CoCoOp (4 ctx)|95.2|97.0|96.1|86.0|59.9|70.6|73.2|55.4|63.1|\n|Ours (w/o adapter)|95.8|95.8|95.8|90.8|71.1|79.8|77.9|51.1|61.7|\n\n**Q3. Comparison with the hard prompt baseline**\n\nWe apologize for the unclear explanation in our paper. In fact, the ZSCLIP (which is the abbreviation of \"Zero-Shot CLIP\") method in Table 1 represents the method using the manually crafted hard prompt, e.g., \"a photo of a\". More carefully crafted prompt templates lead to marginal average score improvements on 11 datasets. Note that the learned prompts in our method are consistently better than hand-crafted hard prompts of any length.\n\n**Q4. Performance drop with respect to the prompt length**\n\nIn Fig. 4 of our paper, we optimize prompts with different lengths under a fixed training time budget by setting the same number of samplings $q$ as 256 for gradient approximation. Such a setting ensures training efficiency but may lead to suboptimal results for longer prompts, resulting in a performance drop of longer prompts. To demonstrate this, we have conducted experiments in which the value $q$ is scaled proportionately according to the size of the prompt, and the results are reported in the table below. From the table, with sufficient training time available, longer prompts achieve stable convergence and clear improvements (especially on EuroSAT). Nonetheless, our optimized prompts consistently outperform hand-crafted hard prompts of any length.\n\n|Methods|OxfordPets|DTD|EuroSAT|\n|:----:|:----:|:----:|:----:|\n|Zero-Shot CLIP (1 ctx, \"a\")|80.7|38.2|31.1|\n|Ours w/o Adapter (1 ctx, $q=256$)|89.2|60.9|77.3|\n|Ours w/o Adapter (2 ctx, $q=256$)|89.0|60.2|77.5|\n|Ours w/o Adapter (2 ctx, $q=512$)|90.0|62.4|77.7|\n|ZSCLIP (4 ctx, \"a photo of a\")|83.6|40.0|24.2|\n|Ours w/o Adapter (4 ctx, $q=256$)|89.4|58.8|70.0|\n|Ours w/o Adapter (4 ctx, $q=1024$)|89.5|62.2|79.6|\n|ZSCLIP (8 ctx, \"a very … photo of a\")|84.2|39.3|31.0|\n|Ours w/o Adapter (8 ctx, $q=256$)|89.4|55.6|75.3|\n|Ours w/o Adapter (8 ctx, $q=2048$)|89.7|61.6|81.7|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a black-box prompt tuning method for pretrain VLM, such as CLIP, showing comparable performance with white-box prompt tuning across 11 different datasets. This could be beneficial when the parameters of the VLM is not available to the users.", "reasons_to_reject": "1. I feel that paper has insufficiant baseline. For example, CoCoOp (https://arxiv.org/abs/2203.05557) is a widely used baseline for prompt tuning research in CLIP. Moreover, it would be nice to include the natural data shift setting as in most other prompt tuning papers for CLIP. \n\n2. It would be nice to include the hard prompt baseline in Table 1 to see the increase in performance of each method. \n\n3. I think the performance drop seen with respect to the prompt length (Figure 4) is a major limitation of this approach. For example, this phenomenon might make it so that using just a general hard prompt of length 4 ('a photo of a') would outperform the CBBT with length 4 or even CBBT with length 1.", "questions_to_authors": "Please see the [reasons to reject] section.", "paper_topic_and_main_contributions": "This research paper introduces Collaborative Black-Box Tuning (CBBT), a new method for adapting large vision-language models to specific tasks. CBBT addresses the common issue of inaccessible model parameters by optimizing textual prompts and adapting output features for black-box models. The approach utilizes perturbed prompts to approximate gradients and incorporates a lightweight adapter for output feature adaptation. CBBT, tested across eleven benchmarks, shows significant improvement over existing black-box vision-language adaptation methods.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3mMdjgP3XA", "reviewer": "Reviewer_ya24", "summary": "", "questions": "please refer to the reasons to reject.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank **Reviewer ya24** for the valuable suggestions. Below we respond to the questions point by point.\n\n**Q1. Experiments on situations with distribution shifts**\n\nThanks for the suggestion! Following CoCoOp, we have conducted experiments on extensively evaluated domain shift benchmarks, and the results are shown in the table below. Given the high variance inherent in these trials, the results are averaged over three random re-runs to ensure reliable comparisons. From the table, our method shows superior robustness to domain shifts.\n\n|Methods|Source|Target|Target|Target|\n|:----:|:----:|:----:|:----:|:----:|\n|    |ImageNet|ImageNet-Sketch|ImageNet-A|ImageNet-R|\n|Zero-Shot CLIP|66.7|46.2|47.8|74.0|\n|CoOp (4 ctx)|71.5|48.0|49.7|75.2|\n|CoCoOp (4 ctx)|71.0|**48.8**|50.6|76.2|\n|Ours (w/o adapter)|70.7|48.7|**50.7**|**76.6**|\n\nBesides, to evaluate the generalizability of our method, we have also deployed our methods to the prevalent base-to-new setting (training on samples from base classes, testing on samples from new classes) commonly used in studies for adaptation of CLIP. From the table below, our black-box prompt tuning yields comparable results to while-box CoOp on base classes while exhibiting significantly improved generalization performance to new classes (especially on EuroSAT and DTD datasets). Such an advantage of our approach can be attributed to less overfitting due to the utilization of the approximated gradient.  It is worth noting that CoCoOp sometimes achieves slightly superior performance than ours on new classes; however, it comes at a considerable cost of degradation of performance on the base classes.\n\n|Methods|OxfordPets|OxfordPets|OxfordPets|EuroSAT|EuroSAT|EuroSAT|DTD|DTD|DTD|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|    |Base|New|H|Base|New|H|Base|New|H|\n|Zero-Shot CLIP|91.2|97.3|94.1|56.5|64.1|60.0|53.2|59.9|56.4|\n|CoOp (4 ctx)|93.7|95.3|94.5|92.2|54.7|68.9|79.4|41.2|54.2|\n|CoCoOp (1 ctx)|94.6|95.6|95.1|84.2|55.3|66.8|75.1|53.6|62.6|\n|CoCoOp (4 ctx)|95.2|97.0|96.1|86.0|59.9|70.6|73.2|55.4|63.1|\n|Ours (w/o adapter)|95.8|95.8|95.8|90.8|71.1|79.8|77.9|51.1|61.7|\n\n**Q2. Details of the computational budget**\n\nThe added computation burden of our method compared to white-box prompting methods lies within the multiple samplings required by the gradient approximation. We report the training durations linked to the tuning methods presented in Table 1 on the EuroSAT dataset in the table below. All the trainings are conducted on a single 3090 GPU. We record the minutes used for complete training and divide the time by the number of trained epochs to ascertain the time per epoch. While the sampling process inevitably elongates the training period, the overall consumed time is acceptable.\n\n|Methods|min / epoch|min / train|\n|:----:|:----:|:----:|\n|CoOp|0.017|3.3|\n|CoCoOp|0.120|1.2|\n|Ours w/o Adapter|0.095|14.2|\n|Ours CLIP-Adapter|0.051|7.7|\n|Ours Tip-Adapter|0.054|8.1|\n\n**Q3. The results of longer prompts**\n\nWe have explored the implementation of more interpretable and semantically structured initializations for longer prompts, for instance, \"a very realistic and detailed photo of a\" for context length 8 (denoted as **\"8 ctx structured\"** in the table below). The prompts initialized with repeated \"a photo of a\" is denoted as **\"8 ctx\"**. From the table, more semantically structured initialization (\"8 ctx structured, $q=256$\" compared to \"8 ctx, $q=256$\") leads to improvements on the EuroSAT dataset but does not work on OxfordPets and DTD. \n\nActually, the reason for the performance drop of longer prompts is the insufficient number of samplings $q$ for gradient approximation. To demonstrate this, we have conducted experiments in which the value $q$ is scaled proportionately according to the size of the prompt, and the results are reported in the table below. From the table, with sufficient training time available, longer prompts achieve stable convergence and clear improvements (especially on EuroSAT). Considering that the enhancement resulting from a longer prompt and larger $q$ is not significant, we limit the scale of prompt and $q$ to ensure elevated training efficiency.\n\n|Methods|OxfordPets|DTD|EuroSAT|\n|:----:|:----:|:----:|:----:|\n|Ours w/o adapter (1 ctx, $q=256$)|89.2|60.9|77.3|\n|Ours w/o adapter (4 ctx, $q=256$)|89.4|58.8|70.0|\n|Ours w/o adapter (4 ctx, $q=1024$)|89.5|62.2|79.6|\n|Ours w/o adapter (8 ctx, $q=256$)|89.7|56.4|71.0|\n|Ours w/o adapter (8 ctx structured, $q=256$)|89.4|55.6|75.3|\n|Ours w/o adapter (8 ctx structured, $q=2048$)|89.7|61.6|81.7|\n\n\n**Q4. The performance is lower without the adapter**\n\nPrompt and adapter are two non-conflicting ways of adaptation. As illustrated in papers [r1, r2], the adapter modules facilitate a straightforward adaptation of pre-trained models to the target dataset. For a fair comparison with LFA [r2], we have also employed an adapter to transfer output features. The collaborative optimization of prompt tuning and adapter learning contribute to the overall improvements, surpassing the individual results of each. From Table 1 of main paper, though our method shows performance drop without the adapter, our results still outperform existing state-of-the-art black box prompt tuning method (i.e., BlackVIP) by a large margin.\n\n[r1] Learning multiple visual domains with residual adapters   \n[r2] Black Box Few-Shot Adaptation for Vision-Language models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This research paper introduces an efficient methodology for black-box tuning of pre-trained vision language models. \n2. The proposed method demonstrated strong performance across 11 different datasets specifically used for fine-grained image classification. \n3. The authors provide a viable solution to issues in previous research related to the black-box prompt tuning of pre-trained vision language models. They analyze their results from various perspectives, providing valuable insights into the black-box tuning process by comparing the white-box setting. ", "reasons_to_reject": "1. To provide the generalizability of the prompt optimization process, it might be beneficial to include situations with distribution shifts(e.g., https://arxiv.org/abs/1903.12261, https://arxiv.org/abs/2006.16241 …) This inclusion would be in line with other white-box setting research papers.\n\t\n2.  Could you provide details about the computational budget required for your method, such as the training time?\n\n3. The proposed method might yield better results with longer prompts if a more accurately structured prompt is used, rather than repeating \"a photo of a\". Have you considered this?\n\t\n4. Could you identify the reasons why the classification performances are generally significantly lower without the use of an adapter? ", "questions_to_authors": "please refer to the reasons to reject.", "paper_topic_and_main_contributions": "This paper proposes Collaborative Black-Box Tuning (CBBT) for the optimization of textual prompts and the adaptation of output features in the context of a vision language model. The model here is considered as a black-box, meaning that users only have access to itsAPI  but not to the inner workings of the model (i.e., they cannot directly adjust the gradient flow or model parameters).\n\nTheir method is evaluated based on its performance and its convergence - that is, how quickly and effectively it finds optimal solutions.\n\t\nThe paper's results demonstrate that this black-box tuning approach can yield results that are comparable to, or even surpass, those achieved in a white-box setting (where full access to the model's internals is allowed) under certain conditions. This indicates the potential effectiveness of their proposed methodology for real-world applications where full access to the model's parameters and gradients might not be possible.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AxbTeRZKR8", "reviewer": "Reviewer_ztat", "summary": "", "questions": "- A. A few questions about the upper bound Eq.(8)\n    - A1. In Eq.(9), if we set $b = D$, the upper bound is at least $4\\nabla||f(\\theta)||_2^2$. I wonder how such a loose upper bound guarantees the convergence of the optimization process.\n    - A2. Could the authors provide the difference between the estimated and the real gradient during training?\n\n- B. Why more learnable prompts lead to worse performance? This is not consistent with white-box methods such as CoOp, CoCoOp[1]. Does the new gradient estimation method have an impact on this? For example, it has a loose upper bound and cannot guarantee the convergence of the optimization.\n    - B1. I am not convinced by the \"expanded parameter space\" explanation as the adapter has more parameters than the prompt.\n\n- C. In Table 1, does LFA adopt adapters? If it does not use an adapter, it seems much better than the proposed method.\n\n- D. Finally, I think the setting of such black-box tuning is weird. We can inject learnable prompts (continuous vector rather than text) into the model. We can also get the feature of the model. This is what we cannot do for ChatGPT, Bard, and GPT-4 (lines 057-059 in the paper). What is the meaning of such a task?\n\n\n[1] https://github.com/KaiyangZhou/CoOp", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank **Reviewer ztat** for the valuable suggestions. Below we respond to the questions point by point.\n\n**Q-A1. The convergence of the optimization process**\n\nAlthough the upper bound of the gradient approximation error is $4 \\Vert \\nabla f(\\boldsymbol \\theta) \\Vert_2^2$ according to Eq. (9), the actual estimation error of the gradient during training is much lower than the theoretical upper bound, which makes \"Ours (w/o adapter)\" closely comparable in performance to \"CoOp (1 ctx)\" in Table 1.\n\nKindly refer to **Q-A2** for more details.\n\n**Q-A2. The difference between the estimated and the real gradient during training**\n\nDue to the inability to present figures in the rebuttal, we report the L-2 norm of the error relative to the norm of the real gradient in the table below by averaging every 150 iterations during training on the EuroSAT dataset. It shows that the error in practice is much lower than the theoretical upper bound in Eq. (9), i.e., 4, in our paper.\n\n|Training iteration|$150$|$300$|$450$|$600$|$750$|\n|:----:|:----:|:----:|:----:|:----:|:----:|\n|$\\Vert \\overline{\\boldsymbol g} -  \\nabla f(\\boldsymbol \\theta) \\Vert_2^2 / \\Vert \\nabla f(\\boldsymbol \\theta) \\Vert_2^2$|1.13|0.81|0.77|0.77|0.76|\n\n\n**Q-B. Performance drop with respect to the prompt length**\n\nIn Fig. 4 of our paper, we optimize prompts with different lengths under a fixed training time budget by setting the same number of samplings $q$ as 256 for gradient approximation. Such a setting ensures training efficiency but may lead to suboptimal results for longer prompts, resulting in a performance drop of longer prompts. Larger $q$ brings better estimation for gradients according to Eq. (9). To thoroughly investigate the performance of longer prompts, we have conducted experiments in which the value $q$ is scaled proportionately according to the size of the prompt, and the results are reported in the table below. From the table, with sufficient training time available, longer prompts achieve stable convergence and clear improvements (especially on EuroSAT), similar to the results in white-box CoOp and CoCoOp.\n\n|Methods|OxfordPets|DTD|EuroSAT|\n|:----:|:----:|:----:|:----:|\n|Ours w/o Adapter (1 ctx, $q=256$)|89.2|60.9|77.3|\n|Ours w/o Adapter (2 ctx, $q=256$)|89.0|60.2|77.5|\n|Ours w/o Adapter (2 ctx, $q=512$)|90.0|62.4|77.7|\n|Ours w/o Adapter (4 ctx, $q=256$)|89.4|58.8|70.0|\n|Ours w/o Adapter (4 ctx, $q=1024$)|89.5|62.2|79.6|\n|Ours w/o Adapter (8 ctx, $q=256$)|89.4|55.6|75.3|\n|Ours w/o Adapter (8 ctx, $q=2048$)|89.7|61.6|81.7|\n\n**Q-C. The details of LFA**\n\nYes, LFA also adopts a linear layer as an adapter, which is analogous to ours. From Table 1 in the main paper, under fair comparison, our method performs favourably against LFA.\n\n**Q-D. The setting of black-box tuning**\n\nOur black-box prompt tuning method follows settings in BBT[r1], BBTv2[r2], BSL[r3], where black-box optimization has been observed to yield better prompts than white-box tuning. Meanwhile, we link prompting with the available output features of the model by introducing adapters for a comprehensive comparison with the existing black-box adaptation method LFA[r4]. Actually, there exist closed-sourced model APIs, e.g., GPT-3, that provide the feature extraction function. Inspired by recent discrete prompt tuning approaches in [r5, r6], it's practically feasible to discretize the learned prompts by projecting the continuous embedding to discrete token space to support a broader range of black-box models, e.g., ChatGPT, Bard. Our research will persist in exploring more practical adaptation techniques for vision-language models.\n\n[r1] Black-Box Tuning for Language-Model-as-a-Service   \n[r2] BBTv2: Towards a Gradient-Free Future with Large Language Models   \n[r3] Black-box Prompt Tuning with Subspace Learning   \n[r4] Black Box Few-Shot Adaptation for Vision-Language models   \n[r5] Black Box Adversarial Prompting for Foundation Models   \n[r6] Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery  \n\n**Response to Other Concerns**\n\n**Q1. Why adopt the new gradient estimation method**\n\nPrompt tuning has been observed to enhance the adaptation to downstream tasks for white-box vision-language models. Existing prompt tuning methods typically employ gradient backpropagation for optimization. However, in the circumstances of black-box model, the unavailability of gradients set a barrier to prompt tuning. Therefore, we intuitively devise the idea of estimating the gradients for learnable prompts. We employ a perturbation-based gradient approximation algorithm to effectively steer the optimization of prompts, resulting in remarkable performance achievements.\n\n**Q2. Insufficient experimental analysis of the method.**\n\nWe have provided more experimental results to support our method. \n\n* Firstly, we have presented the value of the error term in gradient estimation during the training process (kindly refer to **Q-A**). The result shows that the error in practice is much lower than the theoretical upper bound in Eq. (9), which ensures competitive performance of our method compared to white-box prompting methods.\n* Secondly, we have provided the results of an investigation on different prompt lengths  (kindly refer to **Q-B**), which shows stable performance gain when elongating the prompt with properly set samplings $q$. \n* Thirdly, we have deployed our method to extensively evaluated domain shift benchmarks and base-to-new settings (training on samples from base classes, testing on samples from new classes) in line with other studies (kindly refer to **Q1 of Review ya24**). The results demonstrate preferable generalizability of our method.\n* Furthermore, we have provided the training time budget of our method (kindly refer to **Q2 of Review ya24**).  \nThese results offer a more in-depth analysis of our method.\n\n**Q3. Why this method achieves good performance with Tip-Adapter + Prompt**\n\nPrompt and adapter are two non-conflicting ways of adaptation, i.e., one for textual prompt optimization and the other for output feature adaptation. As illustrated in papers [r1, r2], the adapter modules facilitate a straightforward adaptation of pre-trained models to the target dataset. For a fair comparison with LFA [r2], we have also employed an adapter in our method, and the collaborative optimization of prompt and adapter contribute to the overall improvements, surpassing the individual results of each. Besides, we have also explored the effects of different adaptors (i.e., CLIP-adapter and Tip-Adapter). From Table 1, Tip-Adapter utilizes the training few-shot samples in an effective cached design, and contributes more to our overall performance.\n\n[r1] CLIP-Adapter: Better Vision-Language Models with Feature Adapters   \n[r2] Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper improves the gradient estimation method in black-box PEFT of a vision-language model.\n\n- They combine prompt tuning and adapter tuning in black-box PEFT and achieve pretty good performance.", "reasons_to_reject": "- Lack of in-depth theoretical analysis about why adopting the new gradient estimation method.\n- Lack of thorough experimental analysis of the method.\n- No explanations about why this method achieves good performance with Tip-Adapter + Prompt.", "questions_to_authors": "- A. A few questions about the upper bound Eq.(8)\n    - A1. In Eq.(9), if we set $b = D$, the upper bound is at least $4\\nabla||f(\\theta)||_2^2$. I wonder how such a loose upper bound guarantees the convergence of the optimization process.\n    - A2. Could the authors provide the difference between the estimated and the real gradient during training?\n\n- B. Why more learnable prompts lead to worse performance? This is not consistent with white-box methods such as CoOp, CoCoOp[1]. Does the new gradient estimation method have an impact on this? For example, it has a loose upper bound and cannot guarantee the convergence of the optimization.\n    - B1. I am not convinced by the \"expanded parameter space\" explanation as the adapter has more parameters than the prompt.\n\n- C. In Table 1, does LFA adopt adapters? If it does not use an adapter, it seems much better than the proposed method.\n\n- D. Finally, I think the setting of such black-box tuning is weird. We can inject learnable prompts (continuous vector rather than text) into the model. We can also get the feature of the model. This is what we cannot do for ChatGPT, Bard, and GPT-4 (lines 057-059 in the paper). What is the meaning of such a task?\n\n\n[1] https://github.com/KaiyangZhou/CoOp", "paper_topic_and_main_contributions": "This paper improves the gradient estimation method in black-box PEFT of a vision-language model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a black-box prompt tuning method for CLIP finetuning. The authors have added additional results and have done a nice job during the rebuttal. After rebuttal, two reviewers are happy about the paper, while one reviewer still shows concerns about accepting the paper.

One one hand, reviewers agree that the problem studied in this paper is interesting, and results are comprehensive. During rebuttal, additional results are added. There are a large attention recently towards black-box settings in language and vision-language domains and the proposed paper outperforms previous performance on black-box CLIP image classification setting, thus, reviewers recommend to accept the paper. 

On the other hand, some reviewers think that the newly added experiments in the rebuttal derive different conclusions from the experiments in the main paper, i.e., whether a longer prompt works. Besides, a more thorough analysis of the upper bound is needed. Therefore, rejection is recommended. Also, some reviewer commented that the black-box tuning setting is kind of weird, where learnable prompts can be injected, and the feature of the model can be obtained.
Decision: Accept-Findings