Black-Box Prompt Learning for Pre-trained Language Models
Authors: Shizhe Diao
Zhichao Huang
Ruijia Xu
Xuechun Li
LIN Yong
Xiao Zhou
Tong Zhang
Venue: Accepted by TMLR ()

Abstract: The increasing scale of general-purpose Pre-trained Language Models (\textbf{PLMs}) necessitates the study of more efficient adaptation across different downstream tasks. In this paper, we establish a Black-box Discrete Prompt Learning (\textbf{BDPL}) to resonate with pragmatic interactions between the cloud infrastructure and edge devices. Particularly, instead of fine-tuning the model in the cloud, we adapt PLMs by prompt learning, which efficiently optimizes only a few parameters of the discrete prompts. Moreover, we consider the scenario that we do not have access to the parameters and gradients of the pre-trained models, except for its outputs given inputs. This black-box setting secures the cloud infrastructure from potential attack and misuse to cause a single-point failure, which is preferable to the white-box counterpart by current infrastructures. Under this black-box constraint, we apply a variance-reduced policy gradient algorithm to estimate the gradients of parameters in the categorical distribution of each discrete prompt. In light of our method, the user devices can efficiently tune their tasks by querying the PLMs bounded by a range of API calls. Our experiments on RoBERTa and GPT-3 demonstrate that the proposed algorithm achieves significant improvement on eight benchmarks in a cloud-device collaboration manner.  Finally, we conduct in-depth case studies to comprehensively analyze our method in terms of various data sizes, prompt lengths, training budgets, optimization objectives, prompt transferability, and explanations of the learned prompts.

Xuechun Li Published in Transactions on Machine Learning Research (02/2023)
Black-Box Prompt Learning for Pre-trained Language Models
Shizhe Diao
Aikaona@Coonectt.ooo
The Hong Kong University of Science and Technology
Zhichao Huang
shuanghar@torrnecc.uu...k
The Hong Kong University of Science and Technology
Ruijia Xu
ymuogScconncc..ust.k
The Hong Kong University of Science and Technology
vul021 @ucsd. edu
University of California, San Diego
Yong Lin
dsindf@lcommecc.usthhh
The Hong Kong University of Science and Technology
Xiao Zhou
ezhoubi@commeet.us..kk
The Hong Kong University of Science and Technology
Tong Zhang"
tongzhang@rst.hh
The Hong Kong University of Science and Technology
Reviewed OpenReview: https:
httpp:://openrevvaau opwneevee........... net/ formf id= IvsGP7aRum
Abstract
The increasing scale of general-purpose Pre-trained Language Models (PLMs) necessitates the study of more efficient adaptation across different downstream tasks. In this paper, we establish Black-box Discrete Prompt Learning (BDPL) to resonate with pragmatic interactions between the cloud infrastructure and edge devices. Particularly, instead of fine-tuning the model in the cloud, adapt PLMs by prompt learning, which efficiently consider the scenario
optimizes only few parameters of the discrete prompts. Moreover, that we do not have access to the parameters and gradients of the pre-trained models, except for its outputs given inputs. This black-box setting secures the cloud infrastructure from potential attack and misuse to cause a single-point failure, which is preferable to the white-box counterpart. by current infrastructures. Under this black-box constraint, we apply a varianco-roduced policy gradient algorithm estimate the gradients of parameters the categorical distribution of each discrete prompt. In light of our method, the user devices efficiently tune their tasks by querying the PLMs bounded by range of AP1 calls, Our experiments on RoBERTa and GPT-3 demonstrate that the proposed algorithm achieves significant improvement on eight benchmarks in cloud-device collaboration manner. Finally, we conduct in-depth case studies to comprehensively analyze our method in terms of various data sizes, prompt lengths, training budgets, optimization objectives, prompt transferability, and explanations of the learned prompts
"Joint with Google research
The code available http::///gtthhhhc.oo//haaaaaaaaa http://aaaccc---orrrrra-aaaaaaaa

---

Published in Transactions on Machine Learning Research (02/2023)
Table 1: Comparison of different tuning methods. Frozen: be updated. Black-Box: there are no access to the parameters and gradients from the pre-trained model. Discrete: the prompts are discrete tokens (compared with soft prompts). Interpretable: the prompts are readable and interpretable. Learnable: the prompis are parametric and learnable with explicit or estimated gradients (compared with manual prompts). N/A: not applicable since the corresponding descriptions are for prompt learning.
the pre-trained model is frozen and will not
Methods
Frozen
Black-Box
Discrete N/A
Interpretable
Learnable
Vanilla
N/A
N/A
GPT-3's FineTTuning
N/A
N/A
N/A
PeatureProbe (Peters et al.
2019
ManualPrompt
LnContextLearring Brown
Brown ot
2020
Prompt PromptTuning (Lester al.
P-Tuning (Luu et. al. 2021
AutoPrompt. (Shin el.
2020
/
BBT Sun et al.
2022
BDPL (ours)
/
1
Introduction
Large Pre-trained Language Models (PLMs) have demonstrated impressive versatility across a wide spectrum
of downstream tasks, via either fine-tuning (FT) (Devlin et al.
2019 Liu et al.] 2019 Lewis et al. 2020
Zhang et al. 2020 Yang et al. 2020 Diao et al. 2020; Pan& et al. 2022 or prompt-based learning (PL) (Gao
ct al. 2021 Liu et al.) 2021b; Schick & Schütze 2021 Li & Liang] 2021 Liu et al., 2023). Traditionally,
these two tuning paradigms are conducted at a while-box seiting, where the parameters and gradients are accessible since the model is usually open-sourced and can be duplicated in user devices. Despite the fact that white-box methods have made remarkable progress, however, the increasing scale of PLMs renders this setting implausible. Nowadays, huge PLMs opt to serve users as commercial APIs deployed in the cloud, such as Op GPT-:P Tn particular, the service providers hide their model parameters and expose the query and prediction interface, which is termed the black-box setting in this paper.
Although solving NLP problems with APIs in a black-box setting is considerably challenging, it is indeed aligned with the new norm of the current interplay between the cloud infrastructure and edge devices. Specifically, from the position of cloud providers, it is reasonable parameters since commercial, cthical, legal, security, and other concerns might be raised (Bommasani et al., 2021 First,, under the white-box setting, the weaknesses and biases rooted in the underlying PLMs are at higher risk of being misused for harmful purposes. Second, the centralizing nature of PLMs exposes them to potential attacks to cause a single-point failure (Krishna et al. 2020). To this end, the black-box setting is more convincing to secure the cloud infrastructure from being condemned. As for the interests of user devices, the black-box paradigm grants them a more economical option. Otherwise, if we have access to the model's gradients, it requires transmitting gradients from cloud to device, causing high transmission costs.
restrict the access of pre-trained model
With this basic setting in mind, we further elaborate on a more pragmatic scenario, ie., the discrete prompt learning under the black-box constraint. Particularly, we opt for the prompt. learning mechanism instead of the fine-tuning counterpart, partially due to the fact that the prompt learning is more cost-effective by tuning fewer parameters and can climinate the gap between pro-training and downstream transfer. Morcover, black-box fine-tuning H requires users to upload their private labels and save the line-tuned model on the server, which strongly relies on the cloud provider as the single-point trust for the security of private data and model. On the contrary, our black-box prompt learning allows users to store the private label and tune the prompt locally, preventing potential data Icakage and protecting the users' commercial interests. For instance, our setting, each user device can query the output from the cloud and then update its prompts separately on its own data.
It is noteworthy that we optimize discrete prompts, which are more interpretable to power users from different backgrounds to develop their own applications with PLMs. On the contrary, continuous prompt learning
htttps://openat.com compsppp
https://beta..pemt http://ww......ccc///////eeeaaaa
2

---

Published in Transactions on Machine Learning Research (02/2023)
methods, c.g., BBT (black box tuning (Snn et al. 2022) ) are difficult to interpret their learned prompts. Moreover, these methods fail to be directly applied to prediction APIs because APIs only accept discrete inputs (e.g,, GPTT3). However, the discrete nature of our BDPL allows commercial prediction APIs to directly take the Icarned prompt tokens without pain. Overall, our established Black-Box Discrete Prompt Learning (BDPL) is closely in accordance with the recent. progress of huge PLMs, whose comparisons with the existing settings can be found in table ] As shown in the table, BDPL can be specified under the constraints that the PLMs are frozen and both their parameters and gradients are invisible and share the virtue of optimizing discrete, interpretable, and learnable prompt tokens simultaneously.
To embrace this paradigm shift of tuning PLMs, we design a policy gradient inspired framework that can be optimized without relying on the parameters and gradients of the pro-trained models. Specifically, we characterize the prompt learning procedure tokens are sampled according to categorical distribution. Because the PLM's parameters are invisible and gradients cannot be prok-propagatted the categorical distribution needs to be optimized by some gradient-fres algorithms. We resort to the policy gradient algorithm to estimate the gradients without back-propagation. Moreover, to eliminate the high variance issue of policy gradient, we adopted a variance-reducel policy gradient estimator.
discrete token selection problem, where the proper prompt
Experimental results on two kinds of datasets, i.c., datasets without domain-shift and datasots with domain- shift, demonstrate the effectiveness of the proposed black-box discrete prompt learning, which significantly improves the performance over a generic pre-trained model and outperforms all baseline models on eleven datasets. The results confirm that incorporating black-box prompt learning for pre-trained models is effective and efficient solution to the PLM adaptation. We also present further analyses by investigating the elfects of different training data sizes, prompt lengths, training budgels, and objectives. Our analyses demonstrated the robustness, scalability, and transferability of the proposed method.
The contributions of our work are as follows:
We propose a new setting called black-box prompt learning, where we only have access the output of prediction APIs without the need to access the PLM's parameters gradients. The black-box prompt is optimized without the requirements of tuning pre-trained models, saving the fine-tuning costs.
We propose new black-box discrete prompt learning (BDPL) method to solve this new problem, and demonstrate its effectiveness in dealing with domain shifts various tasks.
We conduct comprehensive analyses on eleven benchmark datasets under cloud-device collaboration settings, demonstrating its effectiveness for commercial APTs. BDPL has a much wider range of applications than previous methods, such transfer learning, model personalization, and decentralized training.
2 Approach
In our setting, the input is a sentence $ $152... .. St...... Sm with si indicating the 1-th token and the output corresponds to its category y. Our goal is to learn discrete prompt tokens T = 1. V[iiVVial V[jil ... Vinn. which are prepended to the input sentence to create the user query [T,S]. Note that V represents the vocabulary list consisting of total of N lokens, and 4; Vii is the i-th token in T and j--th token in V.
The overall architecture is shown in Figure D During the black-box training, we freeze the prediction model G with stop- goop-gradient strategy, and only optimize the discrete prompts T. Here, we assume independent categorical distribution for each prompt index je ~ Cat(p:), where the random variable j. is sampled with the probability distribution p: = [pr,,.... Nii,v] over the token indexes, where p; E C and C = {p: ||p|| = 1,0 < p < 1}. Since p; is independent of each other, the joint probability of the whole discrete prompt is P(T) II P(t.) 1P
Because the prediction model's parameters are invisible and gradients cannot be back pooc-garppggted to the prompts, it is no longer possible to directly update the prompts by back-propagating through V (CTT(, y)), where y the label. Inspired by the policy gradient algorithm in discrete optimization, resort to estimating the gradients without proppptoonn to accomplish black-box training.
3

---

Published in Transactions on Machine Learning Research (02/2023)
prediction
PLMs
Inference API
E(prediction.y)
2. Finetune on server
2. Transmit prediction
1. Download PLMs
1. Upload input & labal
1. Query by input
3. Query by input
sample by p-
update using our VR-PGE
2. Tune on device
Tune prompt on device
3. Query on device
Stop-gradient
Discrete prompt
From device to cloud
Input sentence
From cloud to device
(a) Vanilla finetuning prompt tuning
(b) GPT-3's Finetuning
(c) Our black-box discrete prompt optimization
Input label
9 PLMs
p Categorical distribution
Loss function
Figure 1: Schematic illustrations of the comparisons across various tuning paradigms and the cloud-device interplay at the training phase of our algorithm. Left: (a) Vanilla finetuning and prompt tuning can be conducted on user devices in a white-box manner since the PLMs are feasible to be duplicated at user devices. After tuning, users can still access services of PLMs on the device. (b) Increasing scale hinders the democratizing of PL.Ss. In GPT-3's finetuning, users have to upload the input and associated labels to the server. After finetuning, the model is saved on the server. (c) In our black-box discrete prompt learning setting, users send queries the server and then rely on PLMs' predictions to update their discrete prompts the devices using gradient-free optimization. Right: In framework, the user query is created by concatenating the discrete prompt and the input sentence, where the prompt tokens are sampled from their categorical distributions respectively. After calculating the loss between the PLMs predictions and input labels, we apply a variance-reduced policy gradient algorithm to estimate the gradient of the categorical distribution and update accordingly.
For abbreviation, we denote the L(G(T,,), v)) L(T) since S, y can be deemed constants here. By the virtue of the policy gradient estimator (PGE), can optimize the loss function via forward propagation with:
Ey [C(T]] / ((T))(T))T
(1)
and estimate the gradient of pe by:
v ET [C(T)] C(T)V,, dr
/ C(T) P(T)
P(T) V P(P dT
(2)
P(T)C(T)V log P(T) dT
-EP(T) [C(T) log [1,
EEr( [C(T)V p. log P(t))]
The j-th component of Vp, log P(t:) could be solved explicitly by:
V Pay log P(()) V Vpt.s log
33)
4

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm The black-box discrete optimization procedures.
Require: Input batch S, Label batch Y, Parameter of categorical distribution P1.-.. Pn: Prediction model G, Loss function L.
1: for k S I do
Sample j1 ~ Cat(pi), IJn ~ Cat(pn)
T(k) (k) (1) V15{11 Vin
end for
Lavg 1 Ek=1 ,
6: for < n do
7: 9p. T-T Ek=
k= V log P( (k) (CITTTW) ,,,,) - Lavg)
8: p: - o-rojc (p; 11- Ip.
9: end for
10:
return Pl PI...PP Pn
When j = jos it obvious that pis log P(t.) Pr.si
When j * jts equation 3
is calculated by:
N Σ
Pi. log P((:) =V Peis log(1
pi,k)
k=1,
(4)
kk Pik
PiJ.
However, consistent with previous policy gradient applications Sutton et al. 1999 Rezende et al. 2014 Jang
ct al. 2017 Zhou et al.) 2021). we observed that conventional PGE suffers from high variance, which makes challenging converge practice. Therefore, we adopted pariance-reduced policy gradient estimator (VR-PGE) as described in Williams 1992;;
Dong al. (2020) Zhou al. (2021).
The estimated gradient is
caleulated by:
9p:
((((
Σ
ccrooo) Vn log P(ts)
(5)
where ((K), k 1, 1,... I are sampled independently from P(T).
Thus, the prompt token distribution p, can be updated by a projected stochastic gradient descent algorithm:
p: + proje (p. -7 gp.), ,
(6)
where 17 is the learning rate of prompt learning, I is the sample size, and proje is the projection calculation (dctails are presented in the Appendix).
Here we introduce the detailed training procedure for updating the prompts using our proposed VR-PGE, whose mini-batch version is displayed in Algorithm]] [ Assuming the input data is divided into B batches, and within each batch, we will perform I iterations sampling to reduce the variance estimation. Specifically, at the k-th iteration within each batch, we first sample the sequence of prompl tokens (A) = Vo5 (k) ) (.."%%)) (k) according to the joint distribution P(T"). When (*k)) is created, we will prepend it to the input sentence S and feed the quory [A] into the black-bor pre-trained language model G, which will return back the prodiction. Then the estimated gradients 9p. for each p; could be obtained by executing Equation ( after sampling all prompt sequences for the training batch. Finally, the categorical distributions updated by Equation (
In light of the model prediction and ground-truth label Y, we then calculate the loss L(g(T(A)) "T) Y).
Vocabulary Construction A natural question how to construct the vocabulary and what is the size N. Inspired by the observation in Diao al. (022), which revealed the importance of domain-specific and

---

Published in Transactions on Machine Learning Research (02/2023)
task-specific words and ngrams in representation learning, we introduce such important ngrams as prompt candidates. Therefore, we adopt pointwise mutual information (PMI) to construct the vocabulary of candidate prompt tokens in an unsupervised way. For each sentence in the training set, we calculate the PMI by
p
PMI(G, = log p(T)p(T)
(7)
where 7 and are two adjacent words in the sentence, and p(z) is the probability of an n-gram x. If the PMI score between these two adjacent words is high, they have a high probability of co-oceurrence and are more likely to form n-gram, suggesting they are good collocation pairs. If the PMI score is lower than a threshold a, delimiter is inserted between I and T. As a result, the sentence will be segmented by several delimiters. Finally, we obtain a list of ngrams V by extracting those consecutive words after segmentation and with frequency of at least f. As for the size N, we observe that large N will cause an unstable optimization process and even divergence. Therefore, we keep N between 50 and 200.
3
Experimental Settings
In this section, we first introduce the datasets and evaluation metrics 3.2 Lastly, we describe the implementation details ($3.3).
3.1 followed by the baseline models
3.1
Datasets and Evaluation Metrics
In order to examine the model's ability in generic classification tasks as well as domain-specific classification
tasks, we include seven datasets from the GLUE benchmark (Wang et al., 2019): MNLI (Williams et al. 2018).
QQP
Tyer et al. 2017), SST-2 (Socher et al. 2013), MRPC (Dolan & Brockett
2005).
COLA (Warttdd
ct al. 2019), QNLI (Wang et al. 2019). RTE (Dagan et al. 2005 Haim et al. 2006;
Giampiccolo ct al.
2007:
Bentivogli et al. 2200)), and four domain-specific datasels: CITATIONINTENT (Jurgens el. al.; 2018).
ScIERC (Luan et al. 2018), RCT @ernoucourt & Lee 2017). HYPERPARTISAN (Kiesel et al. 2019 from
specific domains including computer science, biomedical science and news following Gururangan et al.
2020
Diao et al. 2021]) The statistics of these datasets are shown in Table 2 Considering the data sparsity issue and large query costs]] in cloud-device collaboration, we conduct our experiments on popular and more realistic setting- few-shot learning, where huge models have shown their powerful ability (Brown et al.) 2020). We follow Perez et al. (2021) to simulate a true k-shot learning sctting. We randomly sample l data from the original training set for each class to construct. the training set and another different. data to construct the validation set. The original validation set will be used as the test set. Because the size of the QQP and RCT validation sets is too large, we randomly sample 1K data to save costs. We adopt Matthews Correlation Coefficient for CoLA, Fi-score for QQP, MRPC, CITATIONINTENT, ScIERC, HYPERPARTISAN RCT, and accuracy for SST-2, RTE, QNLI, SST-2, IMDB, CR, MR, and MPQA following Wang et al. 2019). Diao et al. (2021) MNLI results are an average of MNLI-match and MNLI-mismatch accuracy.
3.2 Baselines
For GPT-3-based models, because previous white-box tuning methods and black-box continuous prompt tuning methods (e.g., BBT) cannot be applied to GPT-3, we compare our model with the following baselines.
GPT-3's Fininng a GPT-3 inference AP1 that is fine-tuned entirely on a labeled dataset (black-box)) MrnualPrompt: a GPT-3 inference API with manually composed prompts to conduct the zero-shot evaluation. The human-writn written prompts are shown in Appendix A..5 (black-box).
InContext (Brown et al., 2020):: a GPT-3 inference API with a set of examples containing training sentences and labels as the input prefix, which is then prepended the input texts (black-box).
For RoBERTa-base based models, we adopt the RoBERTaalarge as the backbone and the following tuning methods.
Vanilla FineTuniigg (Liu et al. 2019): a RoBERTaalarge model that is fine-tuned entirely on a labeled dataset (white-box).
*Fr example, only one training epoch 10, 000 sentences with 300, 000 tokens will cause 6 USD costs for GPTni-Davinci. Not. mention of hundreds of rounds of training.
htttps:/bbea..ot con/doca/ cowm//acca/giidd////ieeeeeeeeeee
6

---

Published in Transactions on Machine Learning Research (02/2023)
The statistics of seven datasets in the generic domain and four datasets in the specific domain. CI, SE, HP denote CITATIONINTENT, ScIERC, HyPERPARTISAN, respectively. [[] number of classes for classification tasks. Note that sample the few-shot training split and development split from the original training split for few-shot setting as described in Section 3.1
Table 2:
Dataset
L
|rain
Dev
Test
Type
Metrics
Domain
Generic Tasks
MNLI
3
393K
9.8K
9.8K
NL1
aCc. F1 aCc. F1 Matthows corr. acc.
fiction, reports Quora
QQP
2
364K
40K
391K
paraphrase
SST-2
2
6.7K
872
1.8K
sentiment
movie reviews
MRPC
2
3.7K
408
1.7K
paraphrase
news books, articles
CoLA
2
8.6K
1K
1K
acceptability
QNLI
2
105K
5.5K
5.5K
NLI
Wikipedia news, Wikipedia
RTE
2
2.5K
277
3K
NLI
acc.
Domain-Specippe Tasks
CI
6
1.6K
114
139
citation intent
F1
computer science
SE
7
3.2K
455
974
relation classification
F1
computer science
RCT
5
180K
30K
30K
abstract sentence roles
F1
biomedical
HP
2
516
64
65
review helpfulness
F1
reviews
PromptTuning (Lester al. 2021 frozen RoBERTa-large model with continuous prompt embeddings
prepended to the input, and learned by gradients (white-box).
P-Tuning v2 (Liu et al. 2021a : frozen RoBERTaalarge model with continuous prompt embeddings
prepended to each layer, and learned by gradients (white-box).
2020 a frozen RoBERTa-large model with discrete prompts optimized basod AutoPrompt (Shin et al.
on gradient-guided search (white-box).
FeaturcProbe (Poters al. 2019): a frozen RoBERTa-large model outputs the features given inputs and newly added classification layer is trained with the gradients (white-box).
ManualPrompt: frozen RoBERTa-large model with manually composed prompts to conduct the zero-shot evaluation. The human-written prompts are shown in Appendix A.5 (black-box).
InContextLearning (Brown et al. 2020): a frozen larERTa-aarge model with a set of examples containing
training sentences and labels as the input prefix, which is then prepended to the input texts (black-box).
BBT (Sun et al. 2022 a frozen RoBERTa-large model with continuous prompts that are optimized by covariance matrix adaptation evolution strategy (black-box).
RLPrompt (Deng et. al. 20022): frozen RoBERTa-large model with discrete prompts that generated by policy network and optimized by reward function (black-box).
3.3 Implementation
For GPT-3 experiments, we conduct experiments with four variants: GPT-3-Ada, GPT-3-Babbage, GPT-3- Curie, and GPT-3-Davinci. The batch size of training and evaluation is set to 4 to fulfill the query length limit (i.e., 2048). We call the APIs directly from OpenAl's service For RoBERTa- large experiments, we initialize it with pre-trained weights by Huggingface's Transformers library The batch size training and evaluation set to 16 and 32, respectively. The number of APT calls is limited 8000 across all datasets.
For BDPL, we optimize the prompts by AdamW [Loshchilov & Hutter 2019) for 30 epochs with a learning rate of x 10-4 The prompt length is 50, and the size the candidate prompt list N is 100. Other hyper-parameters are detailed the Appendix A.3
hatpe:://openai.comm/s///

---

Published in Transactions on Machine Learning Research (02/2023)
The overall performance of black-box prompt and the comparison on eleven datasets with GPT-3. We report average scores across three random seeds, with standard deviations subscripts. AvG. denotes the average score across all tasks. SCoST denoles the money cost in US dollars for calling GPT-3's API during training and inference. FT: GPT-3's FineTuning, MP: ManualPrompt. ICL: InContextLearning
Table 3:
DATASET MNLI QQP SST-2 MRPC CoLA QNLI RTE CI
SE
RCT
HP
AvG.
SCOST
GPT-3 Ada
FT
38.5o.
44.51. 71.612
45.74a
0.0o.0
49.821
52.71. 27.7a.2
3.5a.
57.04.2
24.41.4 37.8
5.6
MP
26.5o.
31.21.s s
63.11a
35.62.1
0.06.0
45.61.
47.32. 26.92.
1.2a.6
15.8
15.2
28.0
0.5
ICL
36.3e.r 40.31s
64.6ass
40.5ts
1.31.4
48.81r
48.71a 28.2a.a
2.50.4
22.72a
20.02.3
32.2
5.7
BDPL
37.1o. 45.1o.6
68.8a.
43.22.
2.00.4
51.2a.4
52.71. 28.31.
3.8a.s
45.72a
22.41. 7 36.4
3.2
GPT-3 Babbage
FT
40.7o.s
46.2
8
66.4
0.3o.1
50.9o
52.31.
5.2o.
4.11.
61.1sa
33.31.
40.7
8.5
MP
28.9o.s
34.11a
8
62.4s.2
0.2o.1
48.814
51.20.6 31.42s
1.7a.s
21.723
27.21.s 35.6
0.6
ICL
35.7o. 45.21.
86.21.
65.4
2.60.0
48.3a. 51.5o.4 13.11s
2.5a.9
36.71.
32.21.4
38.1
7.1
BDPL
41.0o. 50.41.
86.41
67.71.
2.8o.1
52.1oa
53.11.
40.22.
3.2o.s
45.2a.a
30.4a.a
43.0
4.0
GPT-3 Curie
FT
42.2> 53.3
88.9s
7
3.41.a
49.0ts
54.5
28.41.a
5.1os
50.6r.
43.3ts 45.0
42.3
MP
34.51.
44.3a
84.21.
73.31.2
2.0o.6
47.2a.
44.0r.
19.2
2.80.a
31.01.
37.11s 38.1
2.5
ICL
38.02.1
4
87.0..
81.0r.
2.8..0
46.81.
46.2.. 15.2I.
4.81.s
50.12s
39.02.s
41.6
28.5
BDPL
42.5L. 52.01s
88.02
82.61.
4.01.s
50.1o.s
55.8L.a 25.51.a
3.41.
49.62.
39.41.
44.8
16.2
GPT-3 Davinci
FT
60.2>s 67.82.
92.924
84.61.a
55.31.
54.22. 57.0.. 35.4.:
10.32a
51.62.
60.1 57.2
423.2
MP
40.2.a 39.2
86.727
69.7a.
55.224
28.01.
55.3rs a
25.6
4.91.
26.8ta
52.6
44.0
25.0
ICL
52.72. 55.6a.4
87.2as
82.4
56.72a
17.91.
56.62.a 30.1aa
9.21.
44.4aa
55.42.
49.8
206.1
BDPL
54.624 57.82.1 89.3a.
83.41.4
58.41.
56.21.
57.2o.s
34.62.a
6.62.1
48.8a.
58.52.4
55.0
161.5
4
Experimental Results
The overall results eleven datasets are reported in Tables and
We first verify our proposed method's effectiveness on purely black-box setting with GPT-3 APls. From Table BDPL shows great superiority across eleven datasets. Compared with MamualPrompt and InCon- LextLearning, BDPT, demonstrates significant improvements, which are 8.35% and 4.35% on average of Ada, Babbage, Curie, and Davinci models. BDPL also achieves comparable performance with GPT-3's fine-tuning, which requires large money costs and uploading user's data. In Babbage, BDPL even outperforms GPT-3's fine-tuning. As the experiments are conducted on the few-shot setting, where a small mumber of data are available finc-tune the models' parameters, for large models like GPT-3, overfitting could be a serious problem that deteriorates the performance so that fine-tuning is inferior BDPL. Although careful adjustment of the fine-tuning algorithm may mitigate overfitting and improve accuracy, it needs lot of manual effort and money costs, which is implausible for cloud-device collaboration. Moreover, it is observed that MamualPrompt and InContextLcarnig with less capable versions of GPT-3 (c.g., Ada and Babbage) fail in some challenging datasets (e.g,, CoLA and SE) but BDPL performs well on them. With the increase of the model capacity (from Ada to Davinci), we observed MamualPrompt and InContextLearning could also solve them, which consistent with recent observations of large model's emergent abilitics (Wei al. (022)). From this perspective, our method offers another option in addition to increasing model size, which an efficient solution for less capable models to perform challenging tasks.
8

---

Published in Transactions on Machine Learning Research (02/2023)
Table 4:
The overall performance of black box prompt and the comparison on eleven datasets with RoBERTa- large. We report average scores across three random sceds, with standard deviations as subscripts. AvG.
denotes the average score across all tasks. FT: Vanilla FineTuning. ICT: InContextLearning
DATASET
MNLI QQP SST-2 MRPC COLA QNLI RTE
CI
SE
RCT
HP
AvG.
White- Bor Methods
FT PromptTuning
50.8.
60.8t.
86.52.
78.41s
20.4..9
53.2.s
55.62.s
37.4.r
23.11.a
45.2a2
55.52s
51.5
36.5a.
50.21.s
70.72.
52.7a.4
8.0
53.5t.
56.3..c
34.4as
28.6a.s
36.7a1
47.4s.
43.2
P-Tuning v2
44.2..r
57.424
80.41a
62.42.
8.92
51.5.a
53.1.
31.4a2
24.62s
35.4a.
55.44.
45.9
AutoPrompt
40.1
45.7%.
71.5a.1
63.8a.
5.42a
50.2
52.11e
27.92a
21.5aa
29.6as
40.6s.s
40.8
FeatureProbe
46.51s 56.31.1
79.51.
68.91r
15.6.s
50.5o.
54.12s
22.32a
20.8a.c
31.2ar
60.
46.0
Black Box Methods
ManualPrompp
35.91 49.8o.
7
70.41.
0.6o.
49.2
48.2o.c
12.32
9.6r.
11.71s
35.71.6
36.4
ICT
3
50.1o.
8
72.123
1.10.4
50.8o.s
49.32.
14.6
9.2r.
25.8
38.524
39.2
BBT
40.62.
55.2s.1
85.3a.o
66.4a.7
5.52.7
55.4a.
52.62.2
17.4a.
16.40.
31.71.
47.24.s
43.1
RLPrompt
42.8aa 53.72.a 88.41.
68.92.
5.01.
52.6.4
51.8t.s
19.2s.
18.8ts
30.12r
44.92.4
43.3
BDPL
42.51s 56.41. 87.62.1
78.1a.r
4.61.2
53.11.
53.5o.s 24.01. 21.52.0
36.6s.2 45.6s.4
45.8
In addition to the auto-regressive model, we also conduct additional experiments with an encode-only model, RoBERTa-lagge. Because the weights of RoBERTa-large are released and gradients can be leveraged, several white-box baseline models are introduced for comparison. First, our model outperforms all black-box methods, demonstrating the effectiveness of our proposed black-box discrete prompt optimization. Second, BDPL achieves comparable performance compared with white-box prompt-based methods including both discrete and continuous prompts. It is observed that BDPL even outperforms some white-bon box methods (e.g., PromptTuning and AutoPrompt). We attribute this phenomenon to the overfitting of white-box methods in terms of the given few-shot examples while BDPL does not suffer severe overfitting due to its exploration mechanism. We perform an ablation study in Section D to reveal the effect of data size and accuracy. Given that white-box prompt tuning methods cannot be applied in black- box settings when the gradients are unavailable, previous black-box methods such as InContextLearnigg and BBT achieve 2.82% and 6.66% improvement average over ManualPrompp BDPL outperforms the previous black-box methods BBT and RLPrompt by an average of 2.7% and 2.5%, respectively. Compared with BBT, our method, BDPL, not only outperforms it but is also more practical considering its discrete nature. BBT is optimizing continuous prompts and cannot be directly fed into current prediction APIs. We also notice that there is still a large gap between FineTuning and all other methods. FineTuning updates the full model with gradients and huge parameters, serving as an upper bound for all methods. Across eleven tasks, it is observed that the BDPL on domain-specifi datasets is as effective as on generic datasets. While it is known that domain shift introduced difficulyy for models to deal with, BDPL offers an effective solution to domain-specific datasets.
5 Analysis
We analyze several aspects of BDPL, including the effects of different training data sizes, prompt lengths, training budgets, and learning objectives. In addition, we examine the transferability of our learned prompts under the transfer learning setting and the explanation of prompts. We choose GPT-3 Babbage as the backbone model in the following discussion. The details are illustrated in this section.
5.1 Ablation Study
Effects of Training Data Size First, we analyze the effects brought by four different training data sizes: 4-shot, 8-shot, 16-shot, and 32-shot, Experiments are conducted on MRPC and RCT datasets. As shown in the left part of Figure 2 (a) and (b), with the increase in training data size, the performance of FT, InContextLearinn and BDPL is improved on both MRPC and RCT, which is consistent with the assumption that more data brings sufficient training. Compared with baseline models, model achieved consistent.
9

---

Published in Transactions on Machine Learning Research (02/2023)
(a) MRPC
(b) RCT
(c) MRPC
(d) RCT
Poera
10
90 DEPAL
E00L
6P22
42
04
00
numbor ohlncinin dar2
runborr ofAP1 c2ll
nfrber coll
Figure 2: The effects of training data size, prompt length, and the number of API calls on MRPC and RCT. FT, MP, and ICL denote GPT-3's Tuning, ManualPrompt, and InContextLearning respectively.
(a) Loss Type
(b) Transfer Learning
06
P-- Ch
Perforranee (60
Type CE 1leg
Metttd
Manuehhoop
200%
CEUL
B620
131
Figure 3: (a) Ablations of loss function. CE and Hinge represent cross-entrops loss and hinge loss, respectively. (b) Transfer learning performance. SST-2 is the source task while IMDB and CR are two tasks in the target domain.
improvement over ManualPrompt and InContextTearning verifying its effectiveness and scalability under different data sizes.
Effects of Prompt Length It is known that prompt-based methods sensitive to many aspects of prompts, including contexts (Jiang et al. 2020), orders (Tu et al., 2022) and lengths (Lester et al. 2021 and inappropriately designed prompts lead to bad performance. Here we study the effects of different prompt lengths on MRPC and RCT. Considering the maximum number of tokens is 2048 for the input of GPT-3 API, too many prompt tokens (e.g., more than 100) cause additional costs and even failure of queries. Therefore, we conduct experiments with length 10, 25, 50, and 75. The results are shown in the Figure 2 (c) and (d). With the increase prompt length, the performance increases at first and decreases when the prompt length reaches 75 in most cases. We conclude that the approximate best prompt length is 50, since shorter prompt length limits the representation capacity while longer prompt length might involve noises contained in the training data and is hard to optimize.
Effects of Training Budgets Training budgets are essential factors for efficient-tuning methods. An efficient method is expected to achieve good performance with few training budgets. We measure the budgets by the number of prediction API calls and report the performance of our models with different numbers of API calls in Figure 2 (c) and (d). It is observed that with the increase of API calls, BDPL under different settings obtains performance gains because of sufficient training. All settings could converge within 12, 000 API calls. We also find that the 50-prompt performs well at first but the gap between 75-prompt and 50-prompt narrows down when the training budget grows, and finally, 75-prompt achieves competitive performance. Tt told us that if we do not have sufficient. training budget, it would be better to apply fewer prompt tokens, which are easier to optimize.
Effects of Different Objectives In the previous experiments, the prompts are optimized with the cross- entropy loss, and here explore the effectiveness of our model with different objectives. With the same
10

---

Published in Transactions on Machine Learning Research (02/2023)
Task
Prompt Input
Prediction
Label
Our frtends won't this analysis. alone the next we propuse. <dgp>
Unacesptable
CoLA
as time game second family group company take way only ege Our friends) Way this analysis, alone next one we propose. </gp
Acceptable
Acceptable
<pp> one of the dead was a child, suid a doctor at Civil Hospital Karachi, 
 spp A doctor was killed by his parents </p>>
Entailment
RTE
got because people during go N or work both support come also sap> one of the dead was child, suid a doctor at Civil Hospital Karachi. </g>> <g> doctor Killed his percets <ig>>
Net Entailment
Fntailment
<op- This appeared to solve the problem, and the results presented later for the average degree of generalisation do not show an ovsr-generaliisnioon compared with those given in Li and Abe ( 1998) ) s/gs
Background
CompareOr Contrast
may man show most state well around later for the average degree of generalisation do not show an over generalisation compared with those given in Li and Abc 1998). </g>
uppeared solve the problem, and results presented
CompareOr
Contrust
not clear whether these patients would henefli from antifungal treatment
<is>
Results
RCT
go such time part event city use season play news people is nod clear whether these patients benefit from antifingga treatment <<7g>
Background
Background
Figure Four correctly predicted examples by BDPL. We display the prompts and salience map of the token 
. The prompts are in green and the input tokens are in red. The salient tokens are highlighted in blue background, where the darker color denotes the more dominant weights for the prediction.
conduct further experiments with hinge loss on four datasets: MRPC, QNLI, CI, and RCT. We find that our model with both objectives can achieve comparable results. As shown in Figure 3 (a), the model with hinge loss outperforms that with cross-entropy loss on MRPC, CI, and RCT, but underperforms it on QNLI. On average, our approach with hinge loss works as well as that with cross-entropy loss. It is flexible enough to work with different objectives, and we hope to extrapolate to any kind of human desiged objectives.
setting as our main experiment,
Effects of Transfer Learning A critical advantage of discrete prompts is the possibility of transferring prompts learned from one lask to another because discrete tokens share the same text space instead of specific latent space for continuous prompts. To verify the transferability of black box optimized prompt tokens, we conduct experiments on three sentiment analysis datasets (f.e., SST-2 (Socher et al. 2013), IMDB (Maas et al. 2011), and CR (Hu & Liu, 2004)) with GPT-3-Babbage model in the 16-shot setting, First, use SST-2 as the source task following Vu et al. (2022) and optimize the discrete prompt tokens with our proposed BDPL. Then we obtain those selected prompt tokens and simply prepend them to the beginning of the imput of the target task, IMDB and CR. Our setting assumes no training data in the target domain, so we directly test the performance the largel. lask. Following Wang et al. (2021a), for CR, we randomly sample 2,000 instances as the test set. The results are shown in Figure (b). Consistent with the previous observation in Section 4 BDPL outperforms ManualPrompt in the source task by a large margin. Moreover, learned prompts are helpful in two target tasks, demonstrating that our black-box method is robust under transfer learning settings. The experimental results display the expansion potential of prompt transfer, which is a promising practical application of BDPL, especially when there are N edge devices sharing a similar task, but they have no training data. We can update the black-box prompts in a general domain and then transfer them to the target domain.
5.2 Prompt Explanation
To intuitively understand the prompts, we visualize the salience maps using the Language Interpretability Tool (LIT) (Tenney et al. 2020). We choose CoLA, RTE, CI, and RCT datasets because the sentences contained in these datasets are easy for human interpretation. The comparisons between models with discrete prompts and without them are shown in Figure 4 By adding discrete prompt tokens, the model is able to find coherence-related clues. For example, CoLA aims to distinguish the acceptability of sentences. The grammar-relate related word 'won't' and phrase 'let alone" dominate its prediction, which consistent with the decision process of human beings. Similar observations are found other datascts. Due to space limitations, more visualized examples are not shown here. Based on considerable empirical evidence, we conclude that BDPL can capture helpful information to guide the model.
11

---

Published in Transactions on Machine Learning Research (02/2023)
We notice that most of the optimized prompts are readable but in-comprehensible, useful to model improvement, but semantically confusing to humans. [yyas et al. (2019h) find that neural networks rely on some non- robust features to achieve the highest possible accuracy. These non-robust features are usually semantically meaningless to humans, but we can still train a well-performed model only using these featurcs. We argue that while the optimized prompts that our method finds are meaningless to humans, they are useful for the models to make a more accurate prediction. In contrast, forcing the prompts to have semantic meaning may remove the useful information and leads to degraded performance. This observation is consistent with previous discrete prompt learning studies (Shin et al. 2020
6 Related Work
In this section, we present the review on prompt learning for pre-trained language models and black-box optimization.
6.1 Prompts for Pre-trained Models
Large pre-trained language models are of great importance and standard paradigm is pre-training language model on large unlabeled corpus and then fine-tuning the pre-trained model on different supervised tasks. This approach shows great improvements on lots of downstream tasks but it needs lots of computational resources to change all the parameters and has to save a copy for each task. Therefore, prompt-based learning, which does not require tuning the large model, is proposed to solve the problem. Based on the format of prompls, the prompt-based learning can be categorized into two kinds: discrete prompt (Wallace Gao et al. 2021; David et al.) 2022) and continuous Hanbardzumyan et al. 2021 Liu et al. 2021b Han et al., Li & Liang, 2021). The discrete prompt is usually a sequence of tokens or natural language phrases
et al. 2019 Shin al.] 2020; Jiang et al. 2020
prompt [Zhong et al., 2021 Qin & Eisner 2021
2021
while the continuous prompt is designed as a sequence of vectors. However, all of these studies are limited to a white-box setting, which requires accessing all the parameters of a pre-trained model so that the gradients could be back- pooe-propaggted to optimize the prompts. Recently, Sun et al. (2022) proposed black-box tuning real applications, because most.
methods but they are optimizing contimuous prompls, which is impractical of the commercial APIs do not accept continuous vectors as input. Our method, black-box prompt learning, provides a truly black-box solution with discrete optimization, which optimizes a set of discrete prompts without accessing the pre-trained model. There are some concurrent works (Deng et al.
2022 Hou et al.
2022 exploring black-box tuning methods for large language models. PromptBoosting Hou et al. 2022
ensembles a large number of weak learners by the AdaBoost algorithm to pair pre-generated prompts with different elements of the LM's output distribution. They learn the verbalizer instead of discrete prompts of the language model, which is complementary to BDPL. RLPrompt (Deng et al. 2022) generates discrete prompts by a policy network and optimizes by a reward function. In contrast, we apply a varianco-reduced policy gradient estimator to optimize few independent categorical probabilities to select the appropriate prompt tokens. In addition, we are the first to show that the black-box prompt learning methods can generalize to real-world large language models like GPT-3.
6.2
Black- Box Optimization
One of the applications of black-box optimization the score-based black-box adversarial attack πyas et al. 2018 2019a Huang & Zhang, 2020: Andriushchenko et al. 2020 Cheng al. 2019), where the models are also invisible to the attacker. These studies use zeroth-order optimization methods such natural evolution strategy (NES) (Wierstra et al. 2014) to optimize the input and increase the loss to fool the model. Instead of detcriorating the models' performance in the adversarial attack, our direction is to find the inputs that improve the accuracy. Policy gradient (Sutton et al. 1999), which also belongs to the black-box optimization, is widely used in reinforcement learning to find the best policy. In contrast to NES that can only be used to search in the continuous scarch, policy gradient allows the choice of discrete policy and can be used to find the optimal discrete prompts. BDPL uses black-box optimization methods to find the optimal prompts, which is a novel application direction of these methods.
12

---

Published in Transactions on Machine Learning Research (02/2023)
Another line of research to adapt the black-box model is knowledge distillation (KD) Hinton et al. 2015 which learns a student model with the outputs of large models. KD can be used to learn black-box models (Nguyen et al., 2022 Wang 2021,, perform domain adaptation (Liang et al. 2022), and adversarially attack the model Zhang et al. (2022). Despite the wide applications of black-box KD, learning the models with KD still requires a large number of queries and data for training the student network. On the contrary, our proposed approach, BDPL, much more lightweight, and only needs to train a few prompts with a small amount of data. Moreover, under the scenario of cloud-device collaboration, BDPL only needs negligible computation the edge devices while KD has to train the local student network with large computational resources. Therefore, BDPL is more practical for the scenario of cloud-device collaboration.
7
Conclusion
This paper proposes novel setting for text categorization namely black-box prompt learning, where a large pre-trained model is invisible so that the gradients cannot be back- propagaaee to update the prompts. Compared with the standard pre-training then fine-tuning paradigm, our approach only requires updating very few parameters. Compared with previous prompt-based methods, our approach does not require the visibility of pre-trained models, and thus it provides more flexibility in practical applications. We propose black-box prompt learning method, BDPL, which employs variance-reduced policy gradient estimator to approximate the gradients, and then update the prompts. Experimental results demonstrate that approach out performs all black-box mot and is comparable with white-box methods, illustrating the effectiveness of black-box optimization. Experiments on the transfer learning settings further show the potential of our approach in realistic sccnarios, where the pro-trained model is deployed on the cloud, and the prompt learning can be implemented on each device.
In the future, we would like to explore the effectiveness of our proposed methods on more commercial classifiers, such as Google Cloud APIs, Microsoft Azure APIs and so on. The black-box prompt learning for large multi-modal models (Wang et al. 2021b Singh et al. 2022 Wang et al.) 2021cc Zhou et al. 2022 Wang
et al. 2022 Diao et al. 2023 is another important scenario to explore in future work.
Acknowledgments
We thank the anonymous reviewers for their valuable suggestions. This work was supported by the General Research Fund (GRF) of Hong Kong (No. 16310222 and No. 16201320). Shizhe Diao, Ruijia Xu, and Yong Lin were supported by the Hong Kong Ph.D. Fellowship Scheme (HKPFS).
References
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-effiiient black-box adversarial attack via random search. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23 28-28, 2020, Proceedings, Part XXIII, pp. 484-501. Springer, 2020.
Eyal Ben-David, Nadav Oved, and Roi Reichart. PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains. Transactions of the Association for Computational Linguistics, 10:414-433, 04 2022. ISSN 2307-387X. doi: 10.1162/1acl a 00468. URL httts:://oi.orgg::: http//...111//.a.. t1tettacc_aaaaeee 00468
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. ArXiv preprint, abs/2108.00758 2021. URL https::/aarav..rg/aab////000
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nco- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,
13

---

Published in Transactions on Machine Learning Research (02/2023)
Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (ods.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- vul Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https : .ttml
Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial attacks with transfer-based prior. In Hanna. M. Wallach, Hugo Tarochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Newral Information Processing Systems
32: Annual Conference on Newral Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pp. 10932 10942, 2019. URL htttp::/pproceedingggg wtw.eeeddinnnnnuuiisss..mmmmm www.ips..cc/ppppppeeee cc/pperr Abstract .tml
2019 ht0p://aaaa22220005f144466688880
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsich, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. R.Prompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3369 3391, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL 222
https://aclanthollo. www.gg....g//2// 2022.
Franck Dernoncourt and Ji Young Lee. PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 308-313, Taipei, Taiwan, 2017. Asian Federation of Natural Language Processing. URL https: httta:://aclaanhooooggo hwwphhhooooooooggIIii. 2172222
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North Americon Chapler of the Associalion for Compulational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1 /t1//////11111/// 10099-1-444 1423. URL htapss://cclamtooooo.oro////////
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4729-4740, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/ wtg//000.0.000000000
2020. wewddings-emnll.44e URL
wiidiinnnnnennnnn emmlp..mmpp... 425
Shizhe Diao, Ruijia Xu, Hongjin Su, Yilei Jiang, Yan Song, and Tong Zhang. Taming pre-trained language models with n-gram representations for low-resource domain adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3336 3349, Online, 2021. Association for Computational Linguistics. doi: 200653/v1/2221..// acl- 259. URL https://aclanthollgy org/202/ 2021
acl-long. 259
Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, and Jiawei Wang. Write and paint: Generative vision- language models are unified modal learners. In International Conference on Learning Representations, 2023.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https
// //aclanthology.. 50002
Zhe Dong, Andriy Mnih, and George Tucker. Disarm: An antithetic gradient estimator for binary latent variables. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (ods.), Advances in Neural Information Processing Systems 39: Annual Conference on Newral Information Processing Systems 2020, NewrIPS 2020, December 6- 12, 2020, virtual, 2020. URL https: :
// Awst:tttt.bbttttttt.e.aaaaaac... html
14

---

Published in Transactions on Machine Learning Research (02/2023)
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pro-trained language models better few-shot learners.
In Proceedings of the 59th Anaual Meeting of the Association for Compulational Linguisties and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3816 3830, Online, 2021. Association for Computational Linguistics. doi: /1.18633/112021.acchhl2 295. URL
attps://aclantholoogy org. wa001..allloomo.. acl-long. 295
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. Tn Proceedings of the ACI-PASCAL, Workshop on Textual Entaimment and Paraphrasing, pp. 1 9, Prague, 2007. Association for Computational Linguistics. URL https://
aclanthology wwwp//oggy.rg//0-110000000
Suchin Gururangan, Ana Marasović, Swabha Swayamdipla, Kyle Lo, Iz Bellagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Online, 2020. Association org/
for Computational Linguistics. doi: 2..18633vv1/2000 wttp///0.222..mmaii......m mai-mai. URL
2020. acl.main.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Sapektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, 2006.
Karen Hambardzumyann, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial RePro- gramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp- 4921 4933, Online, 2021. Association for Computational Linguistics. doi: 10. URL httppp://acllaabboooog.oo/////// acl-11a..clooooo 1811
acl- long.ol
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. PTR: Prompt Tuning with Rules for Text Classification. ArXiv preprint, abs/2105 11259, 2021. URL attp:://aarxv.orr//bb/100.... 11259
Geoffrey Hinton, Oriol Vinyals, Jell Dean, et al. Distilling the knowledge in neural network. ArXie preprinl, abs/15003.025 2015. URL https://arxiv org/ hts//bb//100 1100.. 150310253111111111
Bairu Hou, Joe O'Connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. Promptboosting: Black- box text classification with ten forward passes. ArXie preprint, abs/2222.00957, 2022. URL https Slaxiv.orgg/bb//2211.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 118-177, 2004.
Zhichao Huang and Tong Zhang. Black-box adversarial allack with transferable model-based embedding. In 8th International Conference on Learning Representations, TCLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proccedings of Machine Learning Research, pp. 2142-2151. PMLR, 2018. URL http: 7/proceedings.nt www.iinnnmmmr.....mmmmm...mme wta.igg..mr......eee..e wwt.:pereeees//eeeaeeeeeee
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. In 7th International Conference on Learning Represeniations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview. net, 2019a. URL httpge///operrerissineefffooooot BKM1WAR5K7
Andrew Tlyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alch Emily B. Fox, and Roman Garnett (eds.), Advances in Neural
15

---

Published in Transactions on Machine Learning Research (02/2023)
Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 125- 136, 2019b. URL https:/7
htt.://cccbbbbtt11aabttttttteee Abstract.html
Shankar 1yer, Nikhil Dandekar, Kornél Csernai, et al. First quora dataset release: Question pairs. data, quora. com, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview
2017. URL http://paaareeeeeeee////rrrr////
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig, How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020. doi: 10.1162/tacl a 00324.
URL
wttp///0...aa11....... tacl-1.28
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. Mcasuring the evolution of scientific field through cilation frames. Transaclions of the Associalion for Compulational Linguislics, 6: 2018. doi: 10.1162/tack
00028. URL htttps::/aclaattoto//t//// www.ggg...rgrggg// org/Q18-1028
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. SemEval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pp. Association for Computational Linguistics. doi: 10. 10.188653v1 /1/00//11/11215555 2eg/
Minncapolis, Minnesota, USA, 2019.
URL.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit Tyyer. Thieves on sesame street! model extraction of bert-based apis. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview net,, 2020. URL httpps://openrevier http://wwwwwneefrrree www.:nteefooumdddyyyeetma ttee/
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v /t////112222222 eennl- main. 243. URL httpss://aclaatoooooggggo wwwp:t/twwooooooooo222o0... org/2021. wwwee-enaaaaeeeee mma...ain..... 243
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising requence-to-sequene pre-training for natural language gencration, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguislics, pp. 7871-78808 Online, 2020. Association for Computational Linguistics. doi: 10.11853/vl// 1t/p//1/22222//// acl 703. URL https:
wwg//o////00// foce-main..e 703.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, 2021. Association for Computational Linguistics. doi: 10. 10111855vv/2200 acl-long.3333 URL https:
//aclanthology. wwwpoology.ooor///// org//021..c1- acl- loo.ooon..... 353
Jian Liang, Dapeng Hu, Jiashi Feng, and Ran Hc. Dine: Domain adaptation from single and multiple black-box predictors. In Proceedings of the FEEE/CVI Conference on Compuler Vision and Pattern Recognition, pp. 1003-8013 2022.
Penglei Liu, Weinhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuming can be comparable to fine-tuning universally across scales and tasks. ArXie preprint, abs/2110.07602, 2021a. CRL fittps://arxiv.oot org/abs/2110.0760
16

---

Published in Transactions on Machine Learning Research (02/2023)
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT Understands, Too. Xiv preprinl, abs/2103. 10385, 2021b. URT htttps://arxtti httpa////axxvo.o/a////////// www./or//////..//// 10385
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv preprint, abs/1907. 1692, 2019. URL
11692
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Repressenaatinns, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview .neti 2019. URL stttp:://opentevieevnne///// fdg.f
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order scnsitivity. In Proceedings of the 60th Anneal Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 8086 8098, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2002
URL
http://toooooooooo///..a...aaa-. acl-long........... 556
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3219 3232, Brussels, Belgium, 2018. Association for Computational Lingnistics. doi: 10.18653/v /1//3///1/11-///// 11///D1-11000 1360.
URL
Andrew L.. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, 2011. Association for Computational Linguistics. URL httpss////claathhooogoooor/////
Dang Nguyen, Sunil Gupta, Kien Do, and Svetha Venkatesh. Black- box few-shot knowledge distillation. In European Conference on Computer Vision, pp. 196-211. Springer, 2022.
Rui Pan, Shizhe Diao, Jianlin Chen, and Tong Zhang. Extremebert: A toolkit for accelerating pretraining of 2022. URL
customized bert. ArXio preprint,
17201
Ethan Perez, Douwe Kicla, and Kyunghyun Cho. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:11054 11070, 2021.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse lasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLPP-001)) pp. 74,,, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v 11/111-1/WWW222 URL https:://cclanthoooog..ooo
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chopter of the Association for Computational Linguisties: Human Language Technologies, pp. 5203 5212, Online, 2021. Association for Computational Linguistics. doi: 1..18633/v12022 maaclll main.410. URL 410
org/ naacl-main
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pp. 1278 1286. JMLR.org, 2014. URL httpp://proceedings. mlr wwt.ss/////reeeeeeel.l. html
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-2352. Online, 2021. Association for Computational Linguistics. doi: 10.18653/vv 11.8633/v1/20 v1/2021 naacl-main. 185. URL h 2021 maacc-mmiin.. 185
17

---

Published in Transactions on Machine Learning Research (02/2023)
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222 4235, Online, 2020. Association for Computational Linguistics. doi: https://acllattool.g. wrgp://0.0/0000000 http//200.eemmmmmma...... 346
http///112220.emm//// main.346. URL
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638 15650, 2022.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empérical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, 2013. Association for Computational Linguistics. URL httpps://aclaatthololl
org/D13-1170
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipong Qiu. Black-box tuning for language- moddel-as---serviee ArXio preprini, abs/2201. 003514,,, 2022. URL httpps////axxio.ogaaaa////////// 03514
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
lan Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 107-118, Online, 2020. Association for Computational Linguistics. doi: 10.18633/v1 http://////220..mmpp/eeemeooo 15
URL
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT: Belter frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5039-5059, Dublin, Ircland, 2022. Association for Computational Linguistics. doi:
URL
-ong acl-long. 346
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 2153-21001 2162, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19- 1221. URL
http://acllattoooooorrg/////
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi- task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nee net, 2019.
URL https::/openrevier
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple serqueecc--t-sequueeee learning framework. ArXiv preprint, bbs/2222.03052, 2022. URL https://arxiv org//bs///000000
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as Few-Shot Learner. ArXie preprint, 0tte/2114..16999 2021a. URL https: //ar:iv.///ivv//// wwwpsiiiiorr.//ooooe wttpg/bbs22004... 14690
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-languugee proptaiiingggtrrriiiningg with mixture- of-modality-experts. ArXio preprinl, abs/211 2/11//211002258,,, 2021b. URL https://arxxv..or
02358
Zi Wang. Zero-shot knowledge distillation from a decision-based black-box model. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, TCML. 2021,
18

---

Published in Transactions on Machine Learning Research (02/2023)
18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10675-10685. PMLR, 2021. URL htttp://proceedings... mlr. wwt.ee.ww..rreeseeeeeeeee https///11/wwwwaggge html
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arv preprint, 2021c.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641, 2019. doi: 10.1162/tacl a 00290.
URL
10400
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarlen Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Talsunori Hashimolo, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. URL https://openreeieeu hwt.://e....oooommoooyyyssssseia
Survey Cortification.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural Evolution Strategies. The Journal of Machine Learning Research, 15(1):949 980, 2014.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding Uhrough inference. In Proceedings of the 2018 Conference of the North American Chapler of the Association for Computational Línguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112 1122, New Orleans, Louisiana, 2018. Association for Computational Linguisties. doi: 10.18653/ v1/N18- 1101. URL httpe:///cccanttooooooor///////
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, ((3):229-266) 1992.
Ze Yang, Wei Wu, Can Xu, Xinnian Liang, Jiaqi Bai, Liran Wang, Wei Wang, and Zhoujun Li. StyleDGPT: Stylized response generation with pre-trained language models. In Findings of the Associalion for Compu- tational Linguistics: EMNLP 2020, pp. 1548 1559, Online, 2020. Association for Computational Linguistics. doi: 10.18653/vi v1/////////220////// wwtp//2...iiiinne//eee findings-ennnl....... URL hattps:/acclathhll.g/ org/2020. findings- - emmlp
140
Jie Zhang, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Lei Zhang, and Chao Wu. Towards efficient data free black- adversarial attack. In Proceedings of the IEEE/ /CFF Conference on Computer Vision and Pattern Recognition, pp. 15115-15125, 2022.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 270-278 Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1 a020.acl-demos.30. URL https://aclanthooe
org/2020. acl-demos. 30
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguislics: Human Language Technologies, pp. 5017-5033, Online, 2021. Association for Computational Linguistics. doi: 10.18653/vl. (1/653/v1/200 v122021 maacl- maiee39............ URL https://accaathhllgg/ org/2021. maacclmaine 398
Wangchunshu Zhou, Yan Zeng, Shizhe Diao, and Xinsong Zhang. VLUE: A multi-task multi-dimennionn benchmark for evaluating vision-language pre-training. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27395 27411. PMLR, 17-23 Jul 2022. URL httpp:: h/tp::/::///rooedinggss mlr, pptss//162//hooo2///e html
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang, Efficient neural network training via forward and backward propagation sparsification. Advonces in Neural Information Processing Systems, 24:15216-15229, 2021.
19

---

Published in Transactions on Machine Learning Research (02/2023)
A
Implementation Details
A.1 Computing Infrastructure
For experiments on GPT-3, we directly call its APIs without any GPU for computation. For experiments on RoBERTa, they are conducted with NVIDIA 2080Ti GPUs with 11GB memory.
A.2 Evaluation Measures
For tasks from the GLUE Benchmark, we adopt Matthews correlation cocfficient for CoLA, F1 for MRPC, and accuracy for RTE and QNLI following their original metric choices. We adopt macro-F1 for CITATIONINTENT, SCIERC, RCT, and HYPERPARIISAN as evaluation metrics.
A.3 Bounds of Hyper-parameters
Table 5: Bounds of hyper-paramett parameters.
Hyper-parameter GPT-3
RoBERTa
number of epochs train batch size eval and test. batch size
30
30
4
32
4
16
prompt length learning rate
{10, 12, 25, 50, 75]
[le-5, le-3)
dropout
0.1
learning rate optimizer loss type
AdamW
(hinge loss, cross-entropy loss}
A.4 Configuration of Best Model
The configuration of the best model for each dataset is shown in Table
Table Configuration of the best model for each datasot. The rest hyper-parameters are the default value
Table 5
DATASET
MNLI
QQP
SST-2
MRPC
COLA
QNLI
RTE
C1
SE
RCT
HP
prompt length
10
25
50
50
50
50
50
50
50
50
50
learning rate
2e-4
le-4
2e-4
le-4
3e-4
2e-4
1e-4
le-4
1e-4
le-4
le-4
A.5 Manual Templates
The manual templates shown in Table 7
B Projection Calculation
The projection from z to C can be calculated by:
Algorithm Projection from z to
Require: a vector 2. 1: Solve V1 from 1 [min(1, max(0, z - 1)1))]
2: p f- min(1, max p 1]))).
output p
20

---

Published in Transactions on Machine Learning Research (02/2023)
Table Prompts and label descriptions of Mamualrrompt method. Most of them are from Gao et al. 20021)
Dataset
Template
MNLI
sentence; @ntailmmnt?[MMAS]) sentencez. (yes/no)
QQP
senlencer ? [MASK] sentence2. (yes/no)
SST-2
sentences. It was [MASK]. (great/terribbe)
MRPC
sentence; ? [MASK] sentenceg. (yes/no)
CoLA
senlencer. correct? [MASK]. (yes/no)
QNLT
sentence, entailment? [MASK]; sentenceg. (yes/no)
RTE
sentence, entailment? [MASK] sentenceg. (yes/no)
CI
sentence1. What is the intent? [MASK]. (background//com
SE
sentence1- What is the relation? [MASK] (canparrroojiinttioo///llaaaasrr http://///eeaeeeeeoooo/////////a
hyponym/part/usecc)
RCT
sentence1. It is [MASK]. haaolgouuu//oomtttsssin//e/e//br
HP
sentence1. It is [MASK] (yes/no)
IMDB
sentences. It was [MASK] (great/terriibe)
CR
sentence1. It was [MASK] (great/terrible)
Proof. The projection from to set C can be formulated in the following optimization problem:
min PER 2 p x
s.t. p and P: 1.
Then we solve the problem with the Lagrangian multiplier method.
1
o),
マ 1p 2|| v(1 p p-1)
(8)
1 1p (z - e1 1l + v(1
(9)
2
with 0 p: S 1. Minimize the problem with respect to p, we have
p = 1=-1121 + 11)1>>=-1>0
(10)
Then we have
g(v) =00)=((p))) v)
1 - - -r1]
-11-- - [2 (v+ + 1) n
+v(1 T = -1)
2 v
- 1112-v1]-10
[|
(v + 1)
+ + = -
2 b²a
d'(v)
[1l - -1+ + 1 + 1)1 - 2]
+
1) -1)==1)--10
-1 min(1, ma,(0,a000 v1))
It is easy to verify that g((v) is a monotone decreasing function with respect to e and we can use a bisection
method solve the equation g' g((v) = 0 with solution vi. Finally we have
p" =1 = + (z of 11))12=-11100
(11)
= min (1, max z 1)).
(12)
21

---

Published in Transactions on Machine Learning Research (02/2023)
Table 8:
The effects of different prompt token positions. We report average scores across three random soeds, with standard deviations as subscripts. AvG. denotes the average score across all tasks. FT: GPT-3's FineTu MP: ManualPrompt. ICL: InContex
DATASET
MNLI QQP SST-2 MRPC CoLA QNLI RTE
CT
SE
RCT
HP
AvG.
GPT-3 Babbage
FT
40.7a.s 46.2..4
87.4.s
66.41.7
0.3
50.9a.
52.31.a
5.20.4
4.11.
61.1sa
33.31a
40.7
MP
28.9o.s
34.11a
83.51a
62.4a.a
0.21.
48.81
51.2o.a
31.4a.
1.7a.
21.72.a
27.21.
35.6
ICL
35.7a.
45.2la
86.2
65.4
2.60.0
48.3a.
51.5a.
13.11s
2.5a.
36.71.
32.21
38.1
BDPL
41.0o.a
50.4..s
86.41.1
67.71.2
2.80.1
52.1a.
53.11.
40.22.
3.2a.s
45.22.2
30.42a
43.0
BDPI-infix
1..
35.6
80.223
60.31.4
0.50.
50.80.s
51.2..
15.22s
2.0
10.50.
20.2
32.0
BDPL-suffi
39.11. 47.52.1 85.63.1 64.22.6
1.10.4
51.81.
52.91.
23.71.
2.9..7
43.12.
28.81
40.1
C Effects of Prompt Positions
In our main experiments, we follow existing prompt-based learning studies (Li & Liang 2021 to prepend some prompt tokens to the original sequence. We also investigate the effects of prompt positions. First, introduce two new baselines based GPT-3-babbage: suffix-tuning (placing prompt tokens after the original sequence) and infix-tuning (placing prompt tokens in the middle of the sequence). The results are shown in g From the results, we observed that prefix-tuning putperossm suffix-tuning and infix-tuning by a large margin. We attribute this performance drop to the position embedding of the learned prompts. Compared with infix-tuning and sulfix-tuning, prefix-luning keeps the prompt tokens at the same position, which consistent. during the learning process. However, the tunix-tuning and suffix-tun tunnx--tunnnggggn require the prompts have adapting ability to dynamic positions. In addition, putting the prompt tokens in the middle of the sequence may break the semantic meaning of the original sequence, leading to even worse results than ManualPromph, These observations are consistent with the prefix-tuning (Li & Liang 2021 so we decide to adopt grefix-tuning in our main method.
D Case Studies
More case studies for prompt explanation are provided in Figure
E Performance with More Data
It is not very straighttorward that black-box methods can outperform the white box methods (i.e., Prompt- Tuning and AutoPrompt). However, some similar observations were reported in previous and concurrent black-box studies. For example, BBT outperforms Prompt and AutoPrompt, and RLPrompt out- performs AutoPrompt. According our experiments, we attribute this phenomenon to the overfitting of white-box methods in terms of the given few-shot examples. First, in our experiments, found that PromptTuming and AutoPrompt have lower training losses. It suggests that white-box methods tend to overfit the small training data (our experiments are 16-shot). Furthermore, we gradually increase the number of training data and verify whether the experimental results are consistent with our conjecture. Specifically, we increase the training data from 16-shot to 32-shot, 64-shot, and 128-shot. The results are shown in Figure B It observed that as the amount of training data increases, the white-box methods will outperform black-box methods, which demonstrates that whito-box methods are better than black- box methods given sufficient data. Based these experiments, believe black-box methods are good at few-shot settings due to their exploration mechanism, which can mitigate overfitting.
22

---

Published in Transactions on Machine Learning Research (02/2023)
Tisk
Prompt Input
Prediction
Label
fiicads this analysis,
propeso <i80
Unaceeptable
CoLAA
any
be you last could and John his never got like mun would book on will did who it gave can time saw about girl sent in <e> Our friends won buy analysis. alone one propose..
any said about
Acceptable
Acceptabee
<g> one of the dead was a child, said a Boctog at Civil Hospital Karachi < 
 A doctor was killed by his parents. <dA>>
Entailment
RTE
found would million had from than died being years made because including three said will government announced company South political United more other people under been for many could new will died from against million being because would
Eutailmeut
Nor Entailment
finst including will
Civil Hospital Karachi. <is> <g> doctor was killed by his parents.. </g>
This appeared solve the problem, and results presented for the average degree generalisation do show an cecrrgencaaiisatoon compared with those given in Li and Abe 1998 5/5
Background
such word can between used use example loarning information analysis proposed described parsing stracture models lexical berween miles proposed similar syntactic
CI
translation semantic
system leamnigg grummar translation text language word truining work information learning used described structure leurning lexical unalysis between structure results information syntactic parsing <g> This appeared solve the problem, and the results presented average dogree generalisation not an over generalisation petpiiaatiiooomrrreeeeeeee compured
CompareOr Contrast
CompareOr Contmst
given and
Abe 1998) ) </8>
<g>> It not clear Whether these patients would benefit from antifungal freamment </6>
Results
group treatment stady placebo during clinical significantly after weeks significant peimary bascline during placebo clinical randomized groups was therapy risk symptoms significantly group patients was changes versus
RCT
Background
Backgnound
therapy more days all also blood participants We who that months was and ego not clear whether these
patients henefir from antifungal treatment
<ip>
Figure 5: Four correctly predicted examples by BDPL. We display the prompts and salience map of the token 
. The prompts are in green and the input tokens are in red. The salient tokens are highlighted in blue background, where the darker color denotes the more dominant weights for the prediction.
(a) MRPC
(b) RCT
45.0
80
42.5
40.0
Fl-score
37.5
65
35.0
method
32.5
method
PromptTuning
Promptiuuiin
AutoPromat
AutoPrompt
30.0
EDPL
BDPL
27.5
20
60
80
100
120
100
120
number training data
number of training data
Figure 6: The effects of training data size on MRPC and RCT with RoBERTa-large model. PromptTuning and AutoPrompt are two white-box methods. BBT and BDPL are two black-bor methods.
23


Reviews:
[{"review_id": "nIQSfFxbi0", "reviewer": "Reviewer_u59W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As explained, the black-box prompt learning setting could be more realistic when we can only access large language models’ outputs without other information such as gradients. However, it doesn’t make sense how and why black-box prompt learning could outperform white-box prompt learning in terms of accuracy and cost because it is a more restrictive setting. Moreover, I wonder whether compared baselines are not strong enough. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a method called Black-box Discrete Prompt Learning (BDPL) for adapting Pre-trained Language Models (PLMs) for different downstream tasks in a cloud-device collaboration setting. BDPL optimizes PLMs through prompt learning, which adjusts a few discrete parameters rather than fine-tuning the entire model. BDPL was tested on RoBERTa and GPT-3 and showed significant improvement on multiple benchmarks, with case studies also conducted to analyze the method under various conditions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could you provide a plausible (and theoretical) explanation of how black-box prompt learning could perform better than white-box one?\n\nCan you compare other prompt learning methods with reinforcement learning [1, 2]?\n[1] Deng et al., RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning\n[2] Hou et al., PromptBoosting: Black-Box Text Classification with Ten Forward Passes\n\nWhat’s the difference between ManualPrompt and InContextLearning?\n\nThe prompt length N looks large (50~200). Isn’t it expensive? The length of the prompts in Figure 4 is short. why?\n\nPage 7: first InContextLearning -> GPT-3 not RoBERTa-large\n\nQuestion: how do you initialize prompt tokens? From random tokens? ", "broader_impact_concerns": "The black-box setting of BDPL is preferable for protecting the cloud infrastructure from potential attacks and misuse, which could have a positive impact on security and reliability. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "82F9A3gakR", "reviewer": "Reviewer_k32w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper introduces the black-box prompt learning problem, which to the best of the reviewer's knowledge, is a new and interesting problem. \n- The paper takes practical constraints into consideration, such as the cloud LM APIs only taking discrete tokens and no easy access to model weights/gradients into consideration when designing and evaluating their approach.\n- A comprehensive evaluation of different tuning methods (10 methods).\n- The paper is clearly written and easy to follow.\n\nWeaknesses:\n- Lack of discussion with alternative methods for black-box adaptation. \n- Evaluation datasets appear to be cherry-picked.\n- Lack of explanation on why some baseline results are noticeably worse than their original reported results. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks into the problem of adapting pretrained models to downstream tasks through prompt learning and without the access to the model weights and gradient information, the so called black-box setting where pre-trained LMs are exposed through only inference APIs. In particular, the paper looks into the black-box prompt learning scenario, where the end devices can  query the outputs of the pre-trained model and use that to optimize prompts locally. Finally, the paper focuses on discrete prompts learning because existing cloud LM prediction APIs only accept discrete prompts as inputs. Under this setup, the paper formulates this black-box prompt learning as a discrete token selection problem and optimizes a categorical distribution via a policy gradient algorithm (gradient-free). Furthermore, the paper also introduces a variance-reduced policy gradient estimator to reduce the high variance issue in policy gradient optimization. Evaluation on RoBERTa and GPT-3 APIs show that the proposed method can effectively and efficiently adapt pretrained models through black-box manner. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper overall looks into a very interesting problem and makes a good attempt of resolving the problem that may lead to an impactful direction. There are a few comments the work:\n\n1. While the proposed black-box prompt learning is indeed an interesting problem, the paper does not discuss too much about alternative methods for adapting pre-trained LMs without accessing model parameters. For example, it might be worthwhile discussing how the proposed method relates to knowledge distillation which also learns a separate model by querying the outputs of the teacher model (in this case, the pre-trained PLM). In particular, KD also does not require the client to have access to model weights/gradients. The resulting student model can be used for adaptation. Do we really need to formulate a new problem when existing techniques might already help resolve the problem? \n\t\n2. It is unclear why the paper only uses a subset of the GLUE datasets for evaluation, e.g., MNLI, QQP, SST-2, and STSB in GLUE  are not included. This raises questions on whether the reported datasets are cherry-picked. To be more convincing, it would be better to report the remaining GLUE task results.\n\t\n3. The paper compares its approach with white-box fine-tuning methods, and it seems that the black-box prompt learning methods are getting similar accuracy as the white-box fine-tuning method in several cases. For example, BDPL is getting <1 point difference and sometimes better performance on MRPC, QNLI, RTE on both GPT-3 and RoBERTa. There are two issues related to this evaluation results: i) the fine-tuning results for RoBERTa on these tasks are noticeably worse by a large margin than their originally reported scores (e.g., please refer to Table 5 in RoBERTa paper and GLUE leaderboard). (ii) MRPC, CoLA, RTE have large variances that are hard to interpret their results without confidence intervals. To be more convincing, the authors might want to explain why the RoBERTa fine-tuning results on those GLUE tasks are much worse than the original paper and include the other GLUE tasks together with standard deviation to the results. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qm6qbxEGgL", "reviewer": "Reviewer_v5DN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This is a very interesting paper that introduces a novel discrete prompt-learning method, this method enables a more efficient and secure way of applying large-scale language models.  \n\n- The empirical study is very solid. The proposed methods are compared with various alternatives under different tasks and datasets. The performance boost is significant. \n\n\nWeakness:\n\n- In general, I just have some small concerns about some design decisions without justification. For example, why the learned prompt tokens are placed before the original sequences? Alternatives could be to place them after the original sequence, or insert some learned tokens between the original sequences. I am not suggesting the proposed method is inappropriate, I am just curious if such alternatives have been considered, if so, why they did not work. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a black-box prompt learning method, where some generated discrete prompt tokens are prepended to the original sequence to improve the quality of the generated sequences. The learning procedure leverages a no back-propagation based approach. The model is evaluated on various tasks and datasets and shows an advanced performance boost.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide some justification for the algorithmic design decisions of the proposed methods. \n\n- Release the code for review if possible. ", "broader_impact_concerns": "Not applicable. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies a new setting named black-box prompt learning and proposes a method BDPL for the setting, which achieves good performance on several public datasets.

Reviewers are all positive about the paper and their concerns are well addressed through discussions and revisions. I recommend acceptance. 
Decision: Accept (poster)