Decoder Tuning: Efficient Language Understanding as Decoding
Year: 2023

Abstract:
With the evergrowing sizes of pre-trained models (PTMs), it has been an emerging practice to only provide the inference APIs for users, namely model-as-a-service (MaaS) setting. To adapt PTMs with model parameters frozen, most current approaches focus on the input side, seeking for powerful prompts to stimulate models for correct answers. However, we argue that input-side adaptation could be arduous due to the lack of gradient signals and they usually require thousands of API queries, resulting in high computation and time costs. In light of this, we present Decoder Tuning (DecT), which in contrast optimizes task-specific decoder networks on the output side. Specifically, DecT first extracts prompt-stimulated output scores for initial predictions. On top of that, we train an additional decoder network on the output representations to incorporate posterior data knowledge. By gradientbased optimization, DecT can be trained within several seconds and requires only one PTM query per sample. Empirically, we conduct extensive natural language understanding experiments and show that DecT significantly outperforms state-of-the-art algorithms with a 200× speed-up. Our codes are available at https://github.com/thunlp/DecT.

heading: 1 Introduction
text: Recent advances in pre-trained models (PTMs) demonstrate the power of the “pre-trainingfine-tuning” paradigm, which empowers broad downstream NLP tasks with a single backbone model (Devlin et al., 2019; Raffel et al., 2020; Radford et al., 2019). Given the million even billionscale models, model-as-a-service (MaaS) has become an emerging practice in deploying massive PTMs, where users can only get access to model inference APIs (Brown et al., 2020; Sun et al., 2022b). Under such a scenario, PTMs’ parameters
∗Corresponding Author.
are frozen, and users cannot fine-tune the model on downstream tasks for adaptation. To find an alternative way, researchers have studied MaaS PTM adaptation methods extensively.
Most existing approaches in this line are based on prompts, which modify inputs with specific patterns. By wrapping inputs into cloze-style questions or prepending inputs with a few demonstrative examples, PTMs could produce the right outputs directly and show strong “in-context” learning abilities (Petroni et al., 2019; Brown et al., 2020) without any parameter update. Besides heuristic prompt design, some recent works try to optimize the input prompts without gradients. Among them, Black-box Tuning (BBT) (Sun et al., 2022b) and BBTv2 (Sun et al., 2022a) apply evolutionary algorithm (Hansen and Ostermeier, 2001) on continuous prompt tokens, while RLPrompt (Deng et al., 2022) adopts reinforcement learning to find discrete prompt tokens. Nevertheless, gradientfree optimization is rather difficult and these input-
15072
side methods need to query the PTMs thousands of times for optimization, which leads to huge inference costs in terms of time and computation resources. Moreover, their final performance is not satisfying as well.
Given the flaws of input-side adaptation, we turn to output-side adaptation, which builds tunable decoder networks on model outputs. Comparatively, output-side adaptation enjoys two major advantages: (1) We can directly tune decoder networks on top of model outputs with back-propagation rather than arduous alternatives. (2) We can reduce thousands of model queries to only once per sample. However, designing decoder networks is not straightforward. Past studies have shown that merely tuning an MLP or LSTM (Hochreiter and Schmidhuber, 1997) over output features cannot provide satisfying results (Sun et al., 2022a,b), leaving this path underexplored.
In this work, we aim to solve the performance issue for output-side adaptation, and we argue that there are two critical reasons behind it: (1) Simply utilizing PTMs as feature extractors ignores the infilling ability of PTMs, which is a strong prior for adaptation. (2) MLP and LSTM are not proper networks especially when training data is not sufficient.
Based on these findings, we present Decoder Tuning (DecT), an enhanced output-side adaptation method. Specifically, DecT has two crucial design choices to address the above issues. First, DecT queries the PTM with prompts and adopts model output scores as the initial predictions, which takes advantage of internal model knowledge. Second, on top of the output representations, we select a Prototypical Network (ProtoNet) (Snell et al., 2017) as the decoder network and train it to fit the training data, which is more suitable for few-shot learning. In this way, DecT modifies the initial model scores with subsequent training data, thus achieving better performance.
Through few-shot learning experiments on ten language understanding datasets, we highlight three advantages of DecT (see Figure 1). (1) DecT achieves over 3% absolute accuracy improvement on average, greatly outperforming previous works. (2) DecT is highly efficient. Compared with major prompt engineering baselines, DecT dramatically reduces the average adaptation time from over 9,800 seconds (BBTv2) to 3 seconds. (3) DecT only requires one PTM query for each example,
while other input-side optimization methods need about 104 calls. This advantage is vital when PTM calls are not for free. In addition, we conduct extensive ablation studies and validate the impact of each component of DecT.

heading: 2 Preliminaries
text: Given a set of training data Dtrain = {(xi, yi)}Ni=1 and PTM M, we need to predict the label y ∈ {1, . . . ,K} for sample x, where K is the number of classes. We assume that each class has the same amount of n training samples.
In the MaaS setting, M is a black-box inference API with fixed parameters. Therefore, we can only query the model with input x and get corresponding outputs. To better utilize the PTMs, it has been a common practice to wrap input samples into prompts. Specifically, we enclose each input x into a template T with a [MASK] token (here we assume using a masked language model). Then, we query M with T (x) and get the final layer hidden states h at the [MASK] position and scores s = SM(T (x)) ∈ RK over label words V . Take sentiment analysis as an example, we can use
T (x) = x In summary, it was [MASK].
as the template with V = {bad, great} as label words for negative and positive sentiment respectively. The output scores on these label words further correspond to the classes.

heading: 3 Methodology
text: In this section, we elaborate on our proposed Decoder Tuning (DecT) method for the classification task. We start with reviewing current input-side adaptation methods, then give an overview of DecT and finally detail it step-by-step.

heading: 3.1 Input-side Adaptation
text: Previous MaaS adaptation methods seek for optimal prompts that stimulate PTMs to output correct answers1. Without loss of generality, we formulate these methods with a transformation function f(·) which pre-processes the input x. f(·) can be specialized by adding demonstrations (Brown et al., 2020), discrete prompt tokens (Deng et al., 2022) or soft ones (Sun et al., 2022a,b). Denote the final score as q(x) and probability as
1BBTv2 (Sun et al., 2022a) further optimizes prompt tokens in the intermediate layers, but we omit this here.
P (y|x) = Softmax(q(x)), these methods define q(x) = SM(f(x)) and optimize f(·) for correct predictions. Although optimizing f(·) without model gradients is possible, we argue that it is highly burdensome. Forwarding through a large “black box” model M, it is rather challenging to find corresponding inputs for specific outputs without the guidance of gradient signals. As a result, users may get suboptimal performance with expensive query costs. We empirically validate it in experiments.

heading: 3.2 Overview of DecT
text: For more effective and efficient PTM adaptation, we turn to output-side adaptation rather than inputside. Overall, output-side adaptation can be viewed as a post-processing of model outputs which uses another function g(·) to process the model outputs, and get the final scores q(x) = g(SM(T (x))). Different from input-side ones, output-side adaptation is easy-to-optimize with gradient descent, and for each sample, we only need to query the PTM once.
For DecT, as shown in Figure 2, we model the post-processing as decoding, which refers to a post-modification to the initial model predictions. Specifically, we first query the PTM with promptenclosed inputs to get model outputs, including the scores for each class and hidden states. Intuitively, output scores contain prior knowledge inside the PTM, so we retain them as part of the final scores.Then, we tune an additional decoder function on the hidden states to fit the training data and make final predictions. Next, we describe how we query the model and then specify the implemen-
tation of the score function.

heading: 3.3 Querying with Prompts
text: To get model outputs, we simply follow the procedure in Section 2 and query the model with manual template-wrapped inputs. We then process the scores by calibration.
Calibration. As stated in Zhao et al. (2021), PTMs tend to assign higher probabilities on those frequent label words, leading to biased output scores. To eliminate the prediction bias, we further calibrate the output scores with empty input xc =“” following (Zhao et al., 2021). Querying the model with xc, we can obtain the calibaration scores sc and normalize them by sc/mean(sc). Then we calibrate s by
ŝ = diag(sc/mean(sc))−1s. (1)
After that, the calibrated scores ŝ are balanced over classes.

heading: 3.4 Tuning the Outputs
text: After getting the hidden states and calibrated scores, we perform DecT outside the PTM to modify the output scores fitting the training data. Denote the final score on class k as q(x, k), we calculate it by the following function:
q(x, k) = Dec(h, k) + λŝk, (2)
where Dec(·) is a trainable decoder function, λ is a hyperparameter controlling the weight of PTM scores and ŝk is the k-th logit in ŝ. By tuning Dec(·), the final predictions incorporate training
data on top of PTM outputs, which combine both knowledge effectively.
The design choice of Dec(·) is fairly flexible. In practice, we select Prototypical Networks (ProtoNet) (Snell et al., 2017) due to their simplicity and remarkable performance in few-shot learning and prompt-based tuning (Cui et al., 2022). For this, we project the hidden states with a linear layer parameterized by W and get sample representation
v = Wh. (3)
On prototypes, classical approaches model them as points in the embedding space, which overlook the different class characteristics. Inspired by Ding et al. (2022a), we model prototypes as hyperspheres with an additional radius parameter. Concretely, the prototype for class k contains two parameters, center position vector zk and radius scalar rk. We randomly initialize zk and initialize rk as the average distance between zk and instances in class k:
rk = 1
Nk
yi=k∑
i
‖vi − zk‖2. (4)
As for the score function, we calculate the Euclidean distances between instances and prototypes.
Dec(h, k) = −‖Wh− zk‖2 + rk. (5) According to Eq. 2, the final logit is
q(x, k) = −‖Wh− zk‖2 + rk + λŝk. (6) From a geometric view, the score function calculates the distance from instance x to the “surface” of the prototype, where rk+λŝk is the whole radius acting like the bias term. With the scores, we can calculate the predicted probability by the Softmax function:
P (y = k|x) = exp(q(x, k))∑K k′=1 exp(q(x, k ′)) , (7)
and we can optimize W and rk by the crossentropy loss
L = − 1 N
N∑
i=1
logP (yi|xi). (8)

heading: 4 Experiments
text: In this section, we first introduce the experimental settings (Section 4.1), then discuss the results for few-shot experiments (Section 4.2), efficiency comparison (Section 4.3), and experiment results for more training data (Section 4.4).

heading: 4.1 Experimental Settings
text: Datasets. We conduct experiments on four typical natural language understanding tasks. For sentiment analysis, we select SST2 (Socher et al., 2013), Yelp P. (Zhang et al., 2015) and IMDB (Maas et al., 2011). For text classification, we use AG’s News, Yahoo (Zhang et al., 2015) and DBPedia (Lehmann et al., 2015). For natural language inference (NLI), we adopt RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). For entity typing, we experiment on FewNERD (Ding et al., 2021b). We report dataset statistics in Appendix A.1.
Splits. We randomly sample n = 1, 4, 16 data instances for each class from the training set for few-shot learning, and sample same amount data for validation. For datasets in GLUE (Wang et al., 2019) (SST2, RTE, MNLI) and SNLI, we use the original validation sets as test sets following Zhang et al. (2021). For other datasets, we evaluate on their original test sets.
Baselines. We compare with representative MaaS PTM adaptation methods. Prompt refers to directly performing zero-shot classification with template-wrapped examples. In-context learning (ICL) (Brown et al., 2020) further concatenates some exemplars before the test samples. BBT (Sun et al., 2022b) optimizes soft prompt tokens with an evolutionary algorithm, and BBTv2 (Sun et al., 2022a) further inserts deep prompts to intermediate layers for better performance. RLPrompt (Deng et al., 2022) is another recent algorithm that optimizes discrete prompts with reinforcement learning. PromptBoosting (Hou et al., 2022) is a concurrent work that applies boosting algorithm for prompt ensembling. We report the details of baselines in Appendix A.2.
Environments. For all experiments, we use NVIDIA A100 and RTX 2080 Ti GPUs. We implement DecT with PyTorch (Paszke et al., 2019), HuggingFace Tansformers (Wolf et al., 2020), and OpenPrompt (Ding et al., 2022b).
Implementation Details. For all methods, we use the same RoBERTaLARGE (Liu et al., 2019) as the backbone model. For DecT, we set the representation dimension to 128 and optimize the parameters for 30 epochs with Adam optimizer (Kingma and Ba, 2015). The learning rate is 0.01. On the
selection of λ, we directly set λ = 1/n for most datasets based on the intuition that λ should decrease as the amount of training data increases. On MNLI and FewNERD, we tune λ on the validation set and select λ = 1 and λ = 1/16 respectively. We give the templates and label words in Appendix A.3.

heading: 4.2 Main Results
text: Table 1 presents the main few-shot learning results. From the results, we have these observations:
Overall, DecT outperforms the state-of-theart baseline methods by a large margin (more than 3% on average), especially under extreme data scarcity, showing its superior performance. Across different tasks, DecT and baselines obtain similar results on some easy sentiment analysis and topic classification tasks, but we highlight that DecT is much more favorable on difficult datasets, such as Yahoo and FewNERD. While other baseline methods struggle to optimize well, DecT surpasses them significantly (about 10% on Yahoo and 20% on FewNERD under 16-shot setting compared with BBTv2 and ICL).
On stability, DecT also has consistently low variance and some baselines (ICL, RLPrompt and PromptBoosting) are unstable. Given the difficulty of few-shot PTM adaptation, it is of great significance that the adaptation method is robust to random seeds.
On baselines, optimization-free methods, i.e.
zero-shot prompt and ICL are strong baselines. However, as shown in the table, ICL gives the best results in the 1-shot setting, and it can hardly improve with more training data due to the input length restriction. To compare, merely optimizing the input prompts (BBT and RLPrompt) can hardly outperform them, showing the limitation of input-side prompt optimization. In contrast, two other baselines, BBTv2 and PromptBoosting, are more powerful because they either inserts additional learnable prompt tokens inside the PTM or ensembles the outputs of different prompts. With the superior results of DecT, we argue that outputside optimization is a promising way for MaaS PTM adaptation.

heading: 4.3 Efficiency Comparison
text: Despite the superior performance, another major advantage of DecT is its high efficiency. In Fig-
ure 1, we plot average accuracy versus training time for each method under different shots. We also provide detailed statistics of training time, query numbers, and parameter numbers for 16-shot experiments in Table 2.
From Figure 1 and Table 2, we clearly see that DecT can be optimized quickly and only requires one model query per training sample, which is about 200×faster and queries 10×fewer than all prompt optimization methods. For BBT, BBTv2, and RLPrompt, users have to query the model near 104 times and spend several hours for sufficient optimization even in the few-shot scenario. When the inference API is not for free such as OpenAI API 2, using these methods would be expensive, and this further burdens their usage in the scenarios of rich data and large models.
In terms of tunable parameters, DecT demands 130K additional parameters for the linear projection layer, which is less than 0.04% of RoBERTaLARGE (355M) that largely saves storage space. Although some other methods (BBT, BBTv2 and PromptBoosting) require fewer parameters, DecT is much easier to optimize.

heading: 4.4 Beyond Few-shot
text: As shown in Section 4.3, the simple architecture and high efficiency enable DecT to scale on more training data, while baseline methods struggle to finish training within acceptable time limits. To explore the scalability of DecT beyond the few-shot setting, we conduct experiments with increased training data (n = 64 and 256). For reference, we compare DecT with fine-tuning, the strongest baseline which update full model parameters.
The detailed results are presented in Figure 1 and Table 3 and we have the following conclusions. (1) DecT continually improves its performance on more training data at a low cost. The average accuracy gains 6% from 16-shot to 256-shot while the average training time is less than 100 seconds.
2https://openai.com/api/
(2) Compared with fine-tuning, DecT is even on par with it in the 64-shot scenario and gradually falls behind in the 256-shot setting, which is reasonable as we only tune a small portion of parameters outside the model. Through further task-level observation, we find DecT still performs well on sentiment analysis and topic classification, but cannot catch up with fine-tuning on NLI and entity typing, which are identified as harder tasks as they require complex reasoning or fine-grained semantic understanding. (3) In experiments, we find fine-tuning is more sensitive to random seeds in the few-shot setting due to the huge amount of trainable parameters and relatively few loss signals, which is evidenced by the high variance in the 64-shot setting. In such scenario, DecT has lower variances due to most parameters are frozen. Therefore, the stability advantage of DecT has been verified again.
To conclude, we take the first step to applying MaaS methods beyond few-shot learning. The results show that DecT is competitive against fine-tuning on regular classification tasks, but is limited on difficult tasks. How to adapt PTMs on challenging tasks without parameter updates still needs further exploration.

heading: 5 Analysis
text: In addition to main experiments, we further provide more analytical experiments for understanding DecT. We conduct ablation study on several components in Section 5.1. Then we evaluate the scaling effect (Section 5.2), the impact of hyperparameter λ (Section 5.3) and templates (Section 5.4) respectively. We further conduct transferability experiments in Appendix B.

heading: 5.1 Ablation Study
text: To validate each component of our proposed DecT, especially the effect of model scores s, radius parameter r, and ProtoNet, we conduct extensive ablation studies. We present results in Table 4 and Figure 4.
Ablating model scores. Apparently, model scores contribute largely to the few-shot performance of DecT, especially when the training data is extremely scarce (1-shot), which illustrates that model scores contain beneficial prior model knowledge for language understanding. Also, incorporating training data reduces the variance. When there are more training data, model scores bring less enhancement, which is reasonable as the relative weights of model and ProtoNet scores change.
Ablating radius. Meanwhile, the radius is also helpful under low-shot scenarios, which characterizes the difference across classes. But as the number of training data increases, ProtoNet dominates model predictions and the impact of r is weakened as well.
Ablating decoder. As stated previously, the design choice of the decoder function is flexible. We replace ProtoNet with a two-layer MLP and evaluate the performance. In Figure 3 we can see that ProtoNet significantly outperforms MLP in the 1-shot setting, which matches the advantages of ProtoNet in the few-shot setting. In 4- shot and 16-shot experiments, ProtoNet still gets higher scores, but with smaller margins. On stabil-
ity, ProtoNet achieves consistently lower standard deviation scores, which serve as another advantage. Overall, we find ProtoNet is a vital component in DecT, and simply replacing it with MLP would worsen the performance.

heading: 5.2 Model Scaling
text: In this section, we explore how DecT applies to PTMs with different architecture and scales. We select T5 (Raffel et al., 2020), an encoderdecoder PTM, at different scales, from T5Small, T5Base, T5Large to T53B. Table 5 presents the full results of T5 models. First of all, DecT has been successfully deployed on T5, a generative language model, which verifies its transferability across PTMs. Furthermore, we can observe an apparent trend of the scaling effect where larger models consistently perform better. We also evaluate the DecT on CPM-Bee3, which is a bilingual generative pre-trained language model with 10B parameters. Table 6 presents the results of CPM-Bee in different settings. The results show that DecT strongly enhances the adaptation of large PLM on downstream tasks. Moreover, CPM-Bee achieves great performance on NLI tasks, which flags that DecT could deal with more difficult tasks with powerful backbone models.

heading: 5.3 Impact of λ
text: As a hyperparameter, λ controls the relative importance of model scores and prototype scores. Here we examine its impact on AG’s News and SST2. In Figure 4, we can observe that: (1) λ largely affects DecT in the 1-shot settings. As λ increases, DecT gradually performs better and stabler, which illustrates the importance of model knowledge in this case. (2) With the shot number increases, the impact of varying λ is weakened, and the best practices become smaller. These observations validate our selection strategy in Section 4.1, which effectively balances model and data knowledge.

heading: 5.4 Impact of Templates
text: Although DecT is an output-side adaptation method, the choice of templates also affects the final performance. To assess the influence of templates, we conduct experiments on AG’s News and SST2 and show results in Table 7. Overall, DecT does not rely much on templates. While different templates may induce fluctuant zero-shot perfor-
3https://live.openbmb.org/models/bee
mance, DecT largely moderates the gaps between them. Additionally, we try two templates searched from RLPrompt (Deng et al., 2022) and they both achieve satisfying results. On SST2, the template from RLPrompt is even better than manually designed ones. Therefore, we highlight that DecT is complementary with input-side adaptation algorithms, and they can work together for better performance.

heading: 6 Related Work
text: Our work explores how to efficiently adapt large PTMs. In this section, we review three lines of research for prompt-based tuning (data efficiency), parameter-efficient tuning (parameter efficiency), and MaaS adaptation methods respectively.

heading: 6.1 Prompt-based Tuning
text: Prompt-based tuning aims to bridge the gap between pre-training and downstream tasks for data-
efficient model adaptation. The major practice for prompt-based tuning (Liu et al., 2021) is wrapping text pieces into human-designed templates. By this means, prompt-based tuning converts downstream tasks to pre-training tasks (e.g. masked language modeling) and greatly enhances the zero/few-shot learning ability of PTMs. Prompt-based tuning is first applied in knowledge probing (Trinh and Le, 2018; Petroni et al., 2019), and it has been adopted broadly in NLP (Schick and Schütze, 2021; Hu et al., 2022b; Ding et al., 2021a; Han et al., 2021; Cui et al., 2022). Despite manually designed prompts, other works also investigate automatic and learnable prompts (Shin et al., 2020; Gao et al., 2021; Hambardzumyan et al., 2021; Schick et al.,
2020; Lester et al., 2021) to alleviate the prompt engineering efforts. However, the optimization of prompts is a non-trivial problem (Ding et al., 2022c; Lester et al., 2021) which sometimes leads to more computation costs and suboptimal performance. Thus in our work, we adopt manual prompts to stimulate model knowledge and help data-efficient model adaptation.

heading: 6.2 Parameter-efficient Tuning
text: Another line of work explores tuning a small fraction of model parameters to reduce computation and storage budgets, namely parameterefficient tuning (PET) (Ding et al., 2022c). Typical PET methods include inserting tunable modules (Houlsby et al., 2019; Li and Liang, 2021; Hu et al., 2022a), adding soft prompt tokens (Lester et al., 2021) and specifying certain parameters (Zaken et al., 2022). Although PET methods achieve remarkable performance with few parameter updates, they still require model gradients, which are unavailable in the MaaS setting.

heading: 6.3 MaaS Adaptation
text: With inference-only APIs, there are also works that adapt models without tuning any model parameters. Brown et al. (2020) present in-context learning, which concatenates test inputs with several exemplars. Further improvements focus on reliving the instability issues caused by model biases (Zhao et al., 2021; Han et al., 2022) and examplar orders (Lu et al., 2022). PromptBoosting (Hou et al., 2022) employs boosting algorithm for prompt ensembling, giving strong performance. Other approaches try to optimize prompts with either blackbox optimization methods (Sun et al., 2022a,b) or reinforcement learning (Deng et al., 2022). However, due to the lack of gradient signals, they need thousands of model queries, resulting in high costs when the model is large and API calls are not for free. Different from the abovementioned methods, we adapt models at the output side, which need not optimize the distant input prompts. We demand only one API call for each training sample and achieve better results across tasks.

heading: 7 Conclusion
text: In this paper, we present DecT, which performs both data and parameter-efficient adaptation with off-shelf PTMs. By fusing prior model knowledge and posterior data knowledge, DecT achieves su-
perior performance on ten language understanding tasks. Meanwhile, DecT exceeds existing baselines by three orders of magnitude in terms of training time and the number of queries, highlighting its advantages in real-world deployment. In future works, we are eager to explore how to combine input and output-side adaptation methods for better PTM adaptation, and how to extend this line of research to more challenging scenarios.
Limitation
DecT explores how to adapt black-box PTMs on downstream tasks. As we show in Section 4.4, our method is not comparable to fine-tuning on hard tasks with increased data points. Moreover, we only focus on classification tasks in this work and do not testify DecT on free-form generation tasks. In the future, we will work toward more general MaaS adaptation strategies across tasks.
Ethical Statement
As large language models are getting more and more popular in NLP research and application, DecT provides a cost-efficient way to adapt these large models. However, we need also to be cautious about the improper adaptation of large language models, such as generating toxic and biased speeches.

heading: Acknowledgements
text: This work is supported by the National Key R&D Program of China (No.2022ZD0116312), Institute Guo Qiang at Tsinghua University and sponsored by Tsinghua-Toyota Joint Research Fund.
Ganqu Cui made the original research proposal and wrote the paper. Ganqu Cui and Wentao Li conducted experiments. Ning Ding revised the paper and participated in the discussion. Longtao Huang, Zhiyuan Liu and Maosong Sun advised the project.

heading: A Experiment Details
text: A.1 Dataset Statistics
We provide dataset statistics in Table 8. We obtain AG’s News, Yahoo, and Yelp P. from https:// github.com/zhangxiangxiao/Crepe under the BSD-3-Clause license. We get FewNERD from https://ningding97.github. io/fewnerd/ with CC BY-SA 4.0 license. Other datasets are downloaded using Huggingface Datasets (Lhoest et al., 2021).
A.2 Baselines
In-context Learning (ICL). To guarantee meaningful results, we randomly permute the demonstrations and prepend them before input prompts. Due to the input length limit, we truncate the demonstrations which exceed input length.
BBT and BBTv2. We reproduce BBT and BBTv2 using the official codes4. For datasets adopted in their work, we follow the original implementations including templates, label words, and hyperparameters. For other datasets, we reproduce with our templates, label words, and default hyperparameters. We take existing 16-shot experiment results from the paper and run other experiments with 5 random seeds for a fair comparison.
RLPrompt. We also use the official codes5 for reproduction and take some experiment results from their original paper. It is worth noting that RLPrompt adopts the test set of SST2 and we use the
4https://github.com/txsun1997/ Black-Box-Tuning
5https://github.com/mingkaid/rl-prompt
validation set, so we report the reproduced results on the SST2 validation set.
PromptBoosting. We follow the official codes6 for reproduction. Since the number of additional parameters is related to number of classes, we compute the average numbers across datasets.
Fine-tuning. We adopt prompt-based fine-tuning which uses the same templates and label words with DecT. We tune the whole model for 5 epochs with AdamW optimizer (Loshchilov and Hutter, 2019) using a 3× 10−5 learning rate.
A.3 Templates and Label Words We report the used prompt templates and label words in Table 9. Most of them are taken from OpenPrompt (Ding et al., 2022b).

heading: B Transferability
text: We conduct transferability experiments on sentiment analysis and present the results in Figure 5. We see that DecT is highly transferable across datasets. On SST2 and IMDB, DecT trained on other datasets even surpasses the original performance. More surprisingly, we find DecT trained on Yelp, a restaurant review dataset, performs best on SST2 and IMDB, which are two movie review datasets. This result further shows the great domain generalization ability of DecT.
6https://github.com/UCSB-NLP-Chang/ PromptBoosting
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the last section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? In experiments we use datasets

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.1
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.1
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future works, we are eager to explore how to combine input and output-side adaptation methods for better PTM adaptation, and how to extend this line of research to more challenging scenarios. Limitation
DecT explores how to adapt black-box PTMs on downstream tasks. As we show in Section 4. 4, our method is not comparable to fine-tuning on hard tasks with increased data points. Moreover, we only focus on classification tasks in this work and do not testify DecT on free-form generation tasks. In the future, we will work toward more general MaaS adaptation strategies across tasks