SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning
Authors: Wei Zhu
Ming Tan
Venue: EMNLP 2023 Main (2023)
Keywords: Prompt tuning
neural architecture search
parameter efficient tuning

Abstract: Prompt tuning prepends a soft prompt to the input embeddings or hidden states and only optimizes the prompt to adapt pretrained models (PTMs) to downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the potential of prompt tuning. In this work, we propose a novel framework, \underline{S}elective \underline{P}rompt \underline{T}uning (SPT), that learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We further propose a novel bi-level optimization framework, SPT-DARTS, that can better optimize the learnable gates and improve the final prompt tuning performances of the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The results demonstrate that our SPT framework can perform better than the previous state-of-the-art PETuning baselines with comparable or fewer tunable parameters. \footnote{Codes will be publicly available upon acceptance. }

SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning
Wei Zhu'; Ming Tan?
East China Normal University
2 Southern University of Science and Technology
Abstract
Prompt tuning prepends a soft prompt to input embeddings hidden states and only op- timizes the prompt to adapt pretrained models (PTMs) downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the poten- tial of prompt tuning. In this work, propose a novel framework, Selective Prompt Tuning (SPT), that learns to select the proper prompt layers by inserting prompt controlled by learnable probabilistic gate at each intermediate layer. We further propose novel bi-level opti- mization framework, SPT-DARTS, that can bet- ter optimize the learnable gates and improve the final prompt tuning performances the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The re- sults demonstrate that our SPT framework can perform better than previous state-of-the-art PETuning baselines with comparable fewer tunable parameters.
SPT
Average Performance
LPT LotA
MERE
Pluliig V2
Parameters
Figure I: Overall comparison between our SPT method and baselines under the few-shot scenario with 100 train- ing samples for each task. All methods are evaluated ten text classification tasks using RoBERTa-large The radius of every circle indicates training speed (tokens millisecond).
number of internal or additional parameters (Ding et al., 2022; Zhang et al., 2023b; Zhu et al., 2023a). Prompt tuning (Lesterr et al., 2021) is a simple and popular PETuning method that prepends a se- quence of soft prompt tokens to the input sequence and only tunes the prompts to adapt the PTM back- bones to downstream tasks. Despite its advantages in parameter efficiency and convenience in PTM deployment, prompt tuning suffers from lower per- formance and convergence rate than other PETun- ing methods like Adapters (Houlsby et al., 2019; Preiffer et al., 2021: Mahabadi et al., 2021: Zhang et al., 2023b), and BitFit (Ben-Zaken et al., 2021). Recently, there has been a branch of literature inves- tigating the advanced techniques for improving the performances of prompt tuning. P-tuning v2 (Liu et al., 2021) improves the performance of prompt tuning by inserting soft prompts into every hidden layer of PTMs. However, it is difficult optimize and needs more training steps attain competitive performance. BBT (Sun et al., 2022) optimizes the inserted prompts with derivative- free optimization.
1 Introduction
Increasingly large pre-trained models (PTMs) (Han et al., 2021; Devlin et al., 2019; Peters et al., 2018; Liu et al., 2019;; Radford and Narasimhan, 2018; Raffel et al., 2019; Zhu, 202 Guo et al.,
2021: Zuo et al., 2022; Sun et al., 2020) have achieved the state-of-the-an (SOTA) performances on most NLP tasks. Full-model fine-tuning is one of the most widely used method for utilizing PTMs. However, fine-tuning (Devlin et al., 2019; Zhu et al., 2023b, 2021a,b; Zhu, 2021a; Gao et al., 2023; Zhang et al., 2023a) needs to tune all pa- rameters of PTMs for each task, resulting in large GPU memory and storage costs, especially for su- persized PTMs (Brown et al., 2020; Wang et al., 2021a). Parameter-efficient tuning (PETuning) is new fine-tuning paradigm that can reduce the adap- tation costs of PTMs by only tuning very small
Coorresponding author: michachwzhu91 @egail.com

---

IDPG (Wu et al., 2022) employs a prompt gener- ator with parameterized hyper-complex multipli- cation (Le et al., 2021) to generate instance-aware soft prompts. LPT (Liu et al., 2022a) inserts an instance-aware late prompt generated by the neu- ral prompt into an intermediate layer of the PTM instead of the input layer or all layers. Liu et al. (2022a) can achieve competitive performance un- der both full-data and few-shot scenarios. Note that the above methods adopt heuristic strategies to determine where insert prompts on the PTMs.
PETuning baselines by a clear margin. Figure depicts the overall comparison between our SPT method and baselines.
To summarize, our contributions are:
We propose the SPT framework, which au- tomatically learns to insert instance-aware prompts the proper intermediate layers of PTMs.
We propose SPT-DARTS, which contains two novel techniques improve the optimization process the prompt hyper-network.
In this paper, we first conduct pilot experiment to show that simple modifications to the prompt inserting strategies in Liu et al. (2022a,b) can result in better performances than these baselines with comparable tunable parameters. The pilot exper- iments demonstrate that there dire need for the optimal strategy of setting prompt layers into PTMs. Predictably, such an optimal strategy may vary across tasks or PTM backbones and difficult to construct manually. Therefore, we propose the Selective Prompt Tuning (SPT) framework (Figure 2). which automatically searches for the optimal strategy of inserting prompts into the PTMs.
We verify our SPT framework in the full-data and few-shot scenarios across ten benchmark text classification tasks and three different PTM backbones.
Related work
2.1 Prompt-based tuning
A major research line of PETuning the prompt- based tuning that inserts some additional soft prompts into the embeddings or hidden states on specific layers of PTMs. Prompt tuning (Lester et al., 2021) and P-tuning (Liu et al., 2021) in- sert soft prompt on the word embedding layer only and can achieve competitive results when ap- plied large-scale PTMs. Prefix tuning (Li and Liang, 2021) and P-tuning v2 (Liu al., 2021) in- sert prompts to every hidden layer of PTM. (Zuo et al., 2022) propose to utilize prompt tuning in the continual learning of rumor detection. BBT (Sun et al., 2022) optimizes the inserted prompt with derivative-free optimization. Recently, there have been few works investigating instance-aware prompting. IDPG (Wu et al., 2022) uses prompt generator to generate soft prompt for every in- stance. Context tuning (Tang et al., 2022) uses the pretrained BERT model (Devlin et al., 2019) as the prompt generator and focuses on NLG tasks. IPL (Jin et al., 2022) first calculates relevance scores between prompt tokens and inputs, then uses the scores to re-weight the original prompt tokens. However, IPL needs to tune all parameters of PTM. LPT inserts an instance-aware late prompt into the middle intermediate layer of the PTM instead of the embedding layer or all the Transformer layers and achieves competitive performances despite its simple design.
Our SPT framework considers a simple search space of whether to insert the generated instance- aware prompts into an intermediate layer of the PТM. As depicted in Figure 2, we initialize prompt hyper-network where each intermediate layer of PTMs inserts prompt controlled by learnable probabilistic gate Oim- We follow the bi- level optimization strategy of Liu al. (2019a) to optimize the learnable probabilistic gates. After optimization, we keep the top prompt layers that receive the highest probabilitics to meet the tunable parameter budgets. To better optimize the learn- able gates and obtain better prompt layer settings, we propose SPT-DARTS, which consists of two novel techniques to improve the optimization pro- cess of Liu et al. (2019a). Our SPT framework can automatically determine the suitable prompt- inserting strategy that achieves high-quality trade- off between parameter efficiency and model perfor- mance.
Extensive experiments are conducted on six benchmark datasets from the GLUE benchmark and four widely studied text classification bench- marks. The results show that SPT performs compa- rable to or outperforms the previous SOTA PETun- ing methods. Especially in the few-shot scenario with 100 training samples, SPT outperforms the

---

(M)LM head
Layer M
Prompt generator
Layer M-1
Up Proj
Layer M-2
Activation
a m
am
Pooling
+
Layer m
New prompt
Propagated prompt
Down Proj
Layer 2
Tunable
Frozen
Layer
Embeckling
Figure The overview of our SPT framework. Left: a prompt generator with bottleneck architecture, Right: forward pass of SPT. At each transformer layer of PLM, one has to decide whether to the prompt propagated from the previous layer or inject newly generated prompt.
Our work contributes the literature by propos- ing SPT, which selectively inserts prompts on cer- tain intermediate layers of PTMs and effectively boosts prompt tuning performance while maintain- ing high parameter efficiency.
Recently, there are work conducting automatic configurations of PETuning modules, such Hu et al. (2022); Zhang et al. (2023b). Compared we focus on the prompt layer selection, which is not included in the search space of Hu et al. (2022). Thus our work can be seen as meaningful complement to the existing literature.
2.2 Other PETuning method
One important research line of PETuning is the adapter-based tuning (Ding et al., 2022) that in- serts certain adapter modules between around the self-attention feed-forward modules of the Transformer layer and only tunes these adapters in downstream training for model adaptation. Adapter (Houlsby et al., 2019) inserts adapter modules with bottleneck architecture between every con- secutive Transformer (Vaswani et al., 2017) sublay- ers. DropterDrop (Rücklé et al., 2020) improves efficiency by removing adapters from lower lay- Compacter (Mahabadi et al., 2021) used low- rank optimization and parameterized hypercom- plex multiplication (Le et al., 2021) compress adapters. Zhang et al. (2023b) propose to optimize the adapter architectures order to obtain better fine-tuning performances. Adapter-based tuning methods have comparable results with model tun- ing when training data is sufficient but work less well in the few-shot scenario (Wang et al., 2021b). There also some other popular PETuning meth- ods, such BitFit (Ben-Zaken et al., 2021) which only tunes the bias terms, LoRA (Hu et al., 2021) which optimizes low-rank decomposition matrices the weights within self-attention layers.
Problem Formulation
For PTM full fine-tuning, the input samples are usually reformulated as [CLS] (Si) (SEP] if the in- puts are single sentences, and as [CLS] (S1) [SEP] (S2) [SEP] if the inputs are sentence pairs. After the PТM backbone encodes the inputs, the final hidden states of the [CLS] token will be used to predict classification labels with linear classifica- tion head.
In the settings of prompt tuning, the downstream tasks are reformulated masked language model tasks to close the gap between pre-training and fine- tuning. Specifically, we insert randomly initialized soft prompt p on the word embeddings, and also modify the original inputs using different manually designed templates with [MASK] token for task adaptations. For example, in single-sentence tasks, the input will be transformed into template like
concat(p, B(CCL) (Si) > It was [MASK]. [SEP))
where B(x) means to map the tokens in the input sequence x into embedding vectors. Then, we map the original labels y to some words (label words) in the vocabulary V of M. Then the final hidden

---

states of [MASK] token will be fed into the pre- trained masked language modeling (MLM) head to predict label words. During downstream task tuning, the PTM backbone and the MLM head will be frozen, and only the soft prompt will be tuned. This way, the downstream tasks are formulated as a masked language modeling task to close the gap between pre-training and downstream task tuning.
(m < d) via linear layer down. Then it obtains the prompt p with length through average pooling operation Pooling(). The pooled prompt will go through an activation function and be up- projected to dimension d via another linear layer MLP up
p = MLP ( (pooooling down (h))))
(1)
In the setting of our proposed SPT framework (depicted in Figure 2), we investigate the prob- lem of whether to insert newly generated instance- aware prompts at the word embeddings or certain intermediate layers of PTM. For convenience, we will refer to the word embedding layer as the 0- th layer of the PTM. We refer to the layers at which new prompts are inserted the prompt lay- ers (PLs). At certain prompt layer a prompt generator PG; to generate a prompt P; from a given input's ' hidden states at layer i.
Following Mahabadi al. (2021); Wu et al. (2022), employ the parameterized hyper-complex mul- tiplication (PHM) layer (Le et al., 2021) with pa- rameter n to reduce the parameters MLPdown
and MLP up- PHM substitutes the weight matrix of a linear layer to sum of Kronecker products, thus having parameter complexity of (mmd/n) n)). reducing the parameters of the projection layers by at most _.
will use
4.3 Prompt hyper-network
We aim to search for the optimal setting of prompt layers under the limited tunable parame- ter budgets. Assume the parameter budget allows K prompt layers. Since not all prompt layers con- tribute equally to task performance, only a fraction of layers should be selected as prompt layers to avoid redundancy of the tunable parameters.
4 SPT: Selective Prompt Tuning
In this section, we will elaborate our Selec- tive Prompt Tuning (SPT) framework, which is depicted in Figure 2. We first discuss motiva- tions. Then we will elaborate on our method for determine the prompt layers.
Thus, we initialize prompt hyper-network where the embedding layer and all the intermediate layers have prompt generation layer controlled by a learnable probabilistic gale. Introducing a vero-initialized learnable parameter a; E R. the learnable gate at layer is given by
4.1 Motivation
We have conducted a pilot experiment on the RTE (Dagan et al., 2005) and Subj (Pang and Lee, 2004) datasets in which we manually design a series of strategies to set the prompt layers of RoBERTa-large (Liu et al., 2019b). The details are put in Appendix A due to limited length. The experimental results demonstrate that: (a) simple manually designed strategies of inserting prompts could perform comparably with recent strong baseline prompt tuning method Liu al. (2022a), with comparable numbers of tunable parameters. (b) setting too many prompt layers will instead hurt the tuning performances. The above observations raise vital research question which we will ad- dress:
= Sigmoid(as).
(2)
where Sigmoid() is the Sigmoid activation func- tion. 0g C (0, 1) can be seen as the probability of activating the prompt generator at layer i. At each layer of the hyper-network, the prompt P; consists of the prompt p; (prev) propagated from the previ- ous layer, and the prompt P (new) generated from the prompt generator PG; at layer i. Formally, the prompt p; at layer is given by
R.G.1: how do we find the optimal strategy of prompt injection, given the task at hand?
(prer) T P P; = (1 - T 04) * =P,
(new)
(3)
where 7 E {0.5, 1.0} is a hyper-parameter de- termining whether to discard the previous layer's 's prompt p (pree) when new prompt is generated at layer Note that = 1.0 is similar to Liu et al. (2021) where instance-indepen indeeedeend new prompts are inserted at each intermediate layer.
4.2 prompt generators
A prompt generator is simple feed-forward layer with bottleneck architecture (Wu et al., 2022). It first dooneprojects the hidden states h of PTM from dimension to dimension m

---

Through optimization, the probabilistic gate az's value will move toward 0 1, acting as importance scores for the prompt layers. The top K layers that receive the highest probabilistic gate values will be set as prompt layers to meet the parameter budget, and the model with such group of prompt layers will be referred to as the learned SPT model.
thus not optimizing fully reveal the difference in their contributions to the prompt hyper-network. We now introduce a re-parameterization step to aj before the calculation of equation 3:
0: = aa * # aj ED(a())
(5)
Σ,a 2.
Our hyper-network, which is the backbone model with soft prompts each layer and the prompts are controlled by the learnable gates asş- The parameters o; are learnt jointly with the model parameters. So they are not hapar-parameters and do not require hyper-paramett tuning to determine their values.
where GD() detaches the parameter from the com- putational graph, and the parameter will never have gradients. The above equation does not change the value of a, since has a value of 1. And equation becomes
(prev) + * *
(new) (6)
pj (1 (1-17 d;) P
4.4 Optimization of prompt hyper-network
Now the gradient of a is given by:
Following DARTS (Liu et al., 2019a), we con- sider all the parameters Og of the learnable proba- bilistic gates architectural parameters, denoted as a, and optimize them via bi-level optimization. De- note the hyper-networks' prompt generator param- eters as w. The bi-level optimization optimize a conditioned on the optimized parameters of prompt generators w". At each epoch, the train set is split into two splits Da and Dw- The inner and outer levels of optimization are conducted on these two separate splits. which is analogous to validating architectures trained on Dw using different split Da to avoid over-fitting. Thus the optimization objective is:
dc
OC dā.
Σ
Da;
dāg da,
Oc da:
OC Σ, GD(a:) da
=0
Σ ak 0ük
Dās Oos
a;)²
Das
Oo:
0L
Σ aj dL
*
(7)
da;
0ā;
Σ, ag Ōā:
We can see that our re- pa-armmtterization technique introduces an extra term Σσ
Cg. OL
the gra-
Σ, aj Rāg
dient. This way, we explicitly introduce the interac- tions among the gating parameters from different layers during gradient computations.
min L(Das w", a):
Architectural consistency learning Note that the final optimized model we want is sparse, with most layers' prompt generators being pruned. To close the gap between the hyper-network and the final model, we assign a Bernoulli distributed ran- dom mask m; E {0, }} with mean value E (0, 1) to each learnable probabilistic gate a; Thus, equa- tion 6 becomes
s.f. w² arg min L(Dw,
(4)
where C() is the objective function on a given downstream task. The above bi-level optimiza- tion problem is approximated with an alternating optimization strategy. The gradients of the prompt generators are calculated with batches of samples from Dw, and the gradients of a are calculated on Da-
p = (1-m (prev) + +m++ (new)
(8) Now we ask the same input to go through the forward pass twice, once with the architectural masks applied (Equation 8) and once with the chitectural masks turned off (Equation 6), resulting in different hidden representations (1) and (2) for the input sample. We now introduce a consistency
Although DARTS is widely applied, it known to produce unstable gradients and sub-optimal per- formances (Dong and Yang, 2020). We propose two novel techniques to improve the optimization of architectural parameters a. We will refer to our modifications to DARTS as SPT-DARTS.
ce-parameterization of probabilistic gates Note that probabilistic gates a; is calculated by equation 3. Their optimization does not explic- itly consider the trade-offs across different layers,
regularization objective in addition to the task's objective function:
C. = MSE( .N!)), (()))))))
(9)

---

where MSE is the mean squared error loss function. Note that this regularization term will be added to both the inner and outer objectives in Equation 4. 5.3 Baselines
random seeds under the full-data scenario and 10 random seeds under the few-shot scenario. More implementation details are provided Appendix C.
Our consistency regularization objective is in- spired by the recent works in consistency learn- ing (Liang et al., 2021). Here, we apply the idea of consistency learning to enhance the op- timization process of the learnable probabilistic gates. Intuitively, this regularization term encour- ages the hyper-network to output consistent hidden states when different sets of prompt generators are pruned. It ensures that each prompt generator is well-trained and bridges the gap between the hyper- network and the final discretized SPT model. As a result, the optimization of a; can better reflect the contributions of each prompt generator, and thus the final learned model will obtain better perfor- mance.
We compare our SPT framework with the current SOTA baseline methods.
Fine-tune The traditional fine-tuning method that trains all parameters in the PTM backbone.
Adapter-based tuning Adapter (Houlsby et al., 2019); (2) AdapterDrop (Rücklé et al., 2020).
compare with (1)
Prompt-based tuning For crompt-based tuning methods, we compare with (I) Prompt Tuning (Lester et al., 2021), (2) P-tuning v2 (Liu et al., 2022b), (3) IDPG (Wu et al., 2022), and (4) LPT (Liu et al., 2022a). The prompt length for all these methods are set to 10.
5 Experiments
Other PETuning methods We also compare: (1) BitFit (Ben- et al., 2021); (2) LoRA (Hu et al., 2021); (3) S by Hu et al. (2022).
5.1 Evaluation datasets
We evaluate our method on five single-sentence and five sentence-pair classification tasks, includ- ing six tasks from GLUE benchmark (Wang et al., 2018) and four other popular tasks, including MPQA (Wiebe et al., 2005), MR (Pang and Lee, 2005), Subj (Pang and Lee, 2004), and TREC (Voorhees and Tice, 2000) tasks. All details about the dataset statistics and evaluation metries can be found in Appendix B.
We implement Adapter, AdapterDrop. BitFit, and LoRA using OpenDelta" library. Other base- lines are implemented using their open-sourced codes. For fair comparison, we do not use supple- mentary training like Wu al. (2022) to enhance performance.
5.4 Results the few-shot scenario
We first evaluate our SPT framework in the few- shot scenario. Following Wu et al. (2022); Liu et al. (2022a), we consider three settings where the training set size 100, 200, and 500, respectively. Under a given random seed, we randomly sample the training samples from the original training set. For every baseline and our SPT method, we will run the experiments over 10 different random seeds and report the mean and deviation on each task. The development and test sets are the same the full-data scenarios.
5.2 Experiment Settings
All experiments are conducted on NVIDIA GTX A40 GPUs. We use Pytorch (Paszke al., 2019) and HuggingFace's Transformers (Wolf et al., 2020) libraries to implement our SPT method. We evaluate our method in both full-data and few-shot scenarios on three PTM backbones, RoBERTa- large (Liu et al., 2019b), DaBgRTa-large (He et al., 2020), and GPTe-large (Radford et al., 2019). Un- less otherwise specified, the number of prompt lay- ers K is set 4, the prompt length / is 10, and we set T 0.5 and the coefficient de of the consis- tency regularization term Eq to 1.0. That is, our method will keep the previous layer's prompt when inserting new prompts. Moreover, we report the average performances and standard deviations on the test set of the learned SPT model across 5
The results for the few-shot scenario of 100 sam- ples are presented in Table 1. The results for the few-shot scenario of 200 and 500 samples are in Table 12 of Appendix E. Our SPT method out- performs all the baseline methods in the few-shot settings. Especially when the training set has only 100 samples, the SPT method outperforms model tuning by 3.3 points on average and Adapter by 3.7 points. Our method also outperforms all the
'We set the coefficient X of this term to 1.0. Without further hyper-parameter tuning, this regularization can already help improve the model performances.
hhtnss//jiiiii.coommttmlpppppppp

---

Methed
Tunable
SST-2
MPQA
MR
Subi
TREC
MNLI
MRRC
QNLI
QQP
RTE
Avg
Params
(oce)
(aec)
(acc)
(acc)
(ace)
(acc)
lace (l)
(acc)
face and (1)
(acc)
Model luning
355M
(12)
(2.5)
(1.)
(3.3
7..0
(6.9
(2.1
74.7
Adapier
L.M
(1.3)
(5.00
32.9015)
88.7 (0:8)
(1.7
(1.5
76.8
(2.5
61.5
(2.1
74.3
AdigeerDrop
81KK
26.8.02)
(2.3)
(1.)
(2.
(8.2
764
(3.
65.7
51408))
73.5
Bifiit
273K
(0.9
62.0:2.11
(1.1
(L.3
(1.0 (1.7
78.4
69.3 (6.5)
69.7
55.8
75.5
LoRA
788K
88.5
(1.3
(8.6 (0.5
(0.9
(2.3
0..8
69.9 (5.7)
(0.3
(2.0
75.8
293k
(2.3) (1.2
(1.3 8.84
(1.
51.8
(1.3
(9.7 (1.6
(6.9
76.2
Prompe Tuning
21K
75.5 (3.8) (2.2)
(1.1
(2.6
(3.7
(1.3
6288 74.2
(2.6
20.8
P-tuning v2
985K
87.8 (0.6)
(1.6
(1.6
87.7 (1.4)
(3.1
41.3
(1.6
66.2 (3.3)
(3.0)
(21))
72.3
IDPG
296K
(1.7)
77.5 (5.8)
(1.7
(1.5
(2.6
(1.8
(1.6
68.6 (3.1)
(1.6
(2.8
73.5
LPT
263K
89.7 (0.8)
82.8
(1.5)
89.2 (1.7) (1.8)
(2.5
(2.0
(1.6
(1.9)
$7.1 (3.5)
76.5
method
SPT (ouns)
152K
90.8 (1.0)
84.5 (1.6)
$4.3(0.5) (0.5)
90.8 (0.9)
(1.8
(1.7
(1.5(1.5)
(2.3)
(1.3
58.9
78.0
SPT (r 1.0)
152k
90.1 (1.1)
(1.3)
(1.)
(1.2 (2.2)
53.1 (2.1)
(7.9.(19)
72.1 (2.5)
714(1.)
(2.1
76.8
Table I: Results in the few-shot scenario of 100 training samples. We report mean and standard deviation of performance across 10 random seeds. Bold and Underline indicate the best and the second best results. All the results are obtained using RoBERTa-large.
prompt-based baseline methods with comparable or less tunable parameters. The results demonstrate that our method can achieve good generalization performance when the training data is very scarce.
Prompt layers determined by
1804
M8-
Subj
TREC
MNLI
From Table 1 and 12, we see that although outperforming the prompt tuning- based baseline methods. SPT with
ONU
00P
generally performs less well than SPT with = 0.5. This result intuitive. The prompt propagated from the previous layers carries different semantic or syntactic information (Clark et al., 2019), which can help the current and future layers to better encode the input sample.
RTE
Lapor
Layer
2
Layor
Layer
Layer
Layer
Layer Caper Layer
Lapur Loyer
Loyer
17 Lover 10 Caper 20 Caper Layer Layera
22 Layer Lapir
Figure 3: The heat map representing the chosen prompt layers by our SPT method on each task, using RoBERTa- large as backbone. For each cell, dark green represents prompt layer, while bright yellow means not.
5.5 Results in the full-data scenario
The overall comparison of our SPT framework and the baselines in the full-data scenario is re- ported Table 11 in Appendix D. From the experi- mental results, we can see that our SPT method can outperform the PETuning baselines with compara- ble or fewer tunable parameters. We can also ob- serve from Table 11 that: (a) Generally, the prompt- based methods are weaker than the baseee methods under the full-data settings, especially on sentence-pair tasks, which is consistent with the re- sults from Sun al. (2022). However, our method overcomes this shortcoming by properly setting the prompt generators at certain intermediate lay- ers. (b) Our method SPT with the learned prompt layer setting is comparable with or outperforms the strong baselines, like AdapterDrop, BitFit, and LoRA, with even fewer tunable parameters.
rows represent different tasks, and the columns cor- respond to the layer indices. For each cell, dark green represents a prompt layer, while bright yel- low means not. We can observe the following pat- terns: (a) all the tasks decide to insert prompts after the embedding layer (layer 0) and the first four transformer layers, which is a similar obser- vation to (Liu al., 2022a). (b) Layers 10 to 19 of RoBERTa-large are frequently chosen as the prompt layers. Similarly, Liu et al. (2022a) ob- serve that the middle intermediate layers are the most performing prompt layers. (c) SPT discards the last four layers. We hypothesize that if we set prompt layers among these layers, the newly gener- ated prompt will not be propagated long enough to formulate useful task rask-related information.
The effects of the number of prompt layers In the main experiments (Tables and 11), mainly the number of prompt layers K 4. To inves- tigate how K affects SPT's performance, we now run the SPT method with K E (1,2,8, 16). We adjust n so that different settings of & have compa- rable tunable parameters. The results of the learned
5.6 Analysis and ablation studies
Visualization and discussions of the learned SPT models We visualize the learned SPT models on the ten tasks with RoBERTa-large backbone via heat map, as depicted in Figure 3. In Figure 3, the

---

Settings
Tunable
Params
SST-2
(ace)
MNLI
(acc)
RTE
(ace)
(a0c)
Subj
Source
SST-2
MNLJ
Target
RTF
Subj
SPTCK 1)
149K
94.9 (0.2)
(0.6)
76.6(1.1)
94.1 (0.3)
SK 2)
150K
94.9 (0.1)
88.8 (0.4)
(1.5
94.6 (0.2)
SST-2
95.2 (0.1)
88.9 (0.2) 79.1 (1.4)
95.0 (0.3)
(P 4)
152K
96.2 (0.1)
89.0 (0.3)
(1.2
95.3 (0.1)
MNLI
95.1 (0.1)
89.0
78.9 (0.8)
94.9 (0.2)
95.3 (0.2)
88.9 (0.2)
78.9 (1.2)
95.1 (0.2)
RTE
95.0 (0.2)
88.7 (0.3)
(1.1
94.7 (0.3)
SP 16)
166K
95.0
(0.0
95.2 (1.0
(011)
Subj
94.9 (0.2)
88.8 (0.1)
78.6 (0.6)
95.3 (0.1)
Table 2: Results of SPT on four tasks with different
Table 3: Evaluation of the prompt layer settings trans- ferred from source datasets to target datasets under the full-data scenario. The target datasets are in the column names, and the source datasets are in the row names.
numbers of prompt layers K. The pretrained model is RoBERTa-large. Bold indicates the best result among PETuning methods.
Tunable
SST-2
MNLI
RTE
Suhj
SPT models can be found in Table 2. The results show that: (a) our main setting K = performs comparable to or better than other settings of larger K, showing that we can not achieve performance bumps simply by adding more prompt layers. (b) Note that by learning the placement of single prompt layer, our SPT (K 1) model performs comparable to or better than the strong baseline Liu et al. (2022a). The results demonstrate that our method indeed has the ability to discover the proper prompt layers.
Method
Params
(acc)
(acc)
(ace)
(acc)
DeBRRTn-lurge backhone
Model tuning
406M
(0.7
89.9 (0.2) 83.5 (0.7)
(6.4
Procupt tuning
26K
94.0 (0.3) 86.1 (0.4)
(1.5)
92.9 (0.4)
LPT
263K
(0.9)
89.1 (0.3)
(1.3
94.7 (0.3)
SPT
152K
19.3 (6.4
81.8
98.5 (0:4)
GPT2-lrgg backbone
Model tuning
(0.6)
89.1 (0.3) (1.2(
(0.3)
Prompr tuning
31K
93.8 (0.2)
86.2 (0.3) 59.7 (2.0)
(0.4)
LPT
329K
94.7 (0.3)
89.1 (0.4) (1.6)
94.1 (0.3)
SPT
186K
(5.3
(0.5
77.1 70.5
Table 4: Results on 4 GLUE tasks using DeBERTa- large and GPT2-large models the backbone under the full-data scenario. Bold indicates the best result among PETuning methods.
The effects of prompt length In the main exper- iments (Table 1), the prompt length for our SPT method and LPT, IDPG is set to 10 following LPT (Liu et al., 2022a). The same prompt length on the three methods ensures that the comparisons of the three methods are fair. Now, we change to 5 or 20, and the performances on 4 tasks are reported in Ta- ble 14 of Appendix G. From the results, can that: (a) our method can consistently outperform the baseline method under different prompt lengths. (b) Our method less affected by the prompt length ppeer-prraeter, which is important to real-world application since increasing the prompt length in- creases computations quadratically.
(He et al., 2020) and GPT2-large (Radford et al., 2019) backbones. The results are reported in Table 4. The results show that our method works well on the two backbones and successfully outperform the baselines.
Training efficiency of the SPT framework Compared with LPT (Liu et al., 2022a), our op- timization framework consumes 4-5 times training time and 1.6 times GPU memory due to bi-level op- timization and multiple forward passes. However, considering that the training is done off-line, it is affordable compared to manually designing differ- ent prompt layer settings and running numerous evaluations.
Transferability of the learned PG settings We now evaluate the transferability the learned SPT models. In table 3, we select four datasets, SST-2, MNLI, RTE, and Subj, and treat them as the source or target dataset. We search the prompt layer set- ting on the source dataset and train with the learned prompt layer on the target task. We can see from Ta- ble 3 that the transferred prompt layer settings can perform close to the directly learned settings and already achieve better performances than most of the baseline models. The transferability guarantees the re-usability of our SPT framework.
Inference efficiency We run inference on the RTE test set, with three different tasks: prompt tun- ing, LPT, and the learned SPT model, with batch size 32 and maximum length 128. The memory footprint and speed are recorded in Table 13 (Ap- pendix F). We can see that during inference, all three methods consume almost equal GPU mem- ory sizes, and SPT is 3.1 1% slower than LPT. The results show that our SPT method achieves superior performances while still being efficient.
Working with other pre-trained encoders To demosntrate that our method's 's superiority does not rely on a specific pre-trained backbone, we run our SPT method and baselines on the DeBERTa-large
5.7 Results on large language models
To demonstrate that our method can generalize to larger language models, we now conduct the

---

Tunable
SST-2
MNLI
RTE
Subj
Method
Params
(ace)
(acc)
(acc)
(acc)
RoBERTa-lang + LPT
263K
89.7 (0.8)
52.5 (2.0)
57.1 (3.5)
89.7 (1.7) (1.7)
RoBERTa-large + SPT
152K
90.8 (1.0)
54,9 (1.7)
(2.3)
90.8 (0.9)
LlaMA-13b LPT
1310K
90.6 (0.6)
52.9 (2.3)
(3.9)
90.8 (1.5)
1JaMA-13h SPT
672K
91.3 (0.8)
$5.7 (1.9)
(2.1)
91.2 (0.7)
Table 5: Results on 4 GLUE tasks using the popular LLM, LlaMA-13b. The fine-tuning is done in the few-data scenario (100 samples).
following three groups of experiments with open- sourced language models.
COPA
ShaRE-13
MultiArith
Method
(ace)
(fl)
(acc)
ChatGPT
0.732
0.331
0,953
Classification tasks Continuing the experiments
LlaMA-2 13B LoRA
0.718
0.532
0.888
in Table 1 and 4, first experiment with the
LlaMA-13b LPT
0.823
0.536
0.879
LlaMA-13b (Touvron al., 2023) (13 billion pa- rameters) on the SST-2, MNLI, RTE and Subj tasks. The results are presented in Table 5. We can see that by selecting proper prompt layers, our SPT method successfully help the LlaMA-13b back- bone to achieve clear performance gains over the LPT baseline. In addition, we can see that LLM presents strong performances under the few-data settings. However, the LLMs requires much higher computational complexity and memory costs, Other English tasks We now also conduct ex- periments on other English tasks of different types: (a) COPA, a task focused on commonsense reason- ing. (b) ShaRE-13, nested named entity recog- nition (NER) task within the biomedical domain. (c) MultiArith, task centered around arithmetic reasoning, To be consistent with Table 1, 4 and we sample 100 samples from the training sets of these tasks as our training set. The hyper-parameter settings are the same with Table The results of these experiments are presented in Table 6. From the above table, the following observations can be made: (1) ChatGPT demonstrates strong perfor- mance in many NLP tasks without fine-tuning, In contrast, when employing our SPT method with 100 training samples, fine-tuned LlaMA-2 13B ex- hibits either comparable or superior performance compared ChaIGPT. (2) On all the above tasks, when having comparable tunable parameters, SPT can outperform LoRA or LPT under the few-data setting.
LiaMA-135 SPT
0.836
0.553
0,907
Table 6: Results fine-tuned LlaMA-13b on 3 tasks. The fine-tuning is done in the few-data scenario (100 samples).
P-DARTS (Chen al., 2021), FairNAS (Chu al., 2021) or Lo regularization method (Louizos et al., 2017). The results is presented in Table 15 (Ap- pendix H.1). We can see that our SPT method can obtain better learned SPT models than the other methods.
SPT-DARTS on the NAS benchmark Note that the architectural consistency learning regu- larization of our SPT-DARTS method is gener- ally applicable to neural architecture search. We now evaluate SPT-DARTS on the widely used NAS benchmark, #AS-benchmark-201 (Dong and Yang, 2020). We follow the same search setting as DARTS on NAS-benchmark-200 The results are in Table 16 of Appendix H.2. The results show that our SPT-DARTS outperform ENAS and DARTS by clear margin NAS-benchmark-201.
Conclusion
In this work, propose the SPT framework automatically determine the optimal settings for prompt layers under the given PTM backbone and downstream task. We initialize a prompt hyper- network in which each layer has a prompt generator controlled by learnable probabilistic gate. To bet- ter optimize the prompt Syper-network, we propose novel SPT-DARTS method containing two novel modifications to the original DARTS' bi-level opti- mization process. Experiment results in full-data and few-shot scenarios demonstrate that SPT achieve comparable or better performance than state-of-the-as PETuning methods while maintain- ing parameter and inference efficiency.
5.8 Validating SPT-DARTS method
In order to validate the effectiveness of our SPT- DARTS method, now conduct two experiments. Ablation on the hyper-network optimization method The first experiment is to substitute SPT- DARTS to DARTS (Liu et al., 2019a) or its variants

---

Limitations
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
We showed that our proposed method can greatly improve the performance of prompt tuning on di- verse NLU tasks and three different pre-trained models (i.e., RoBERTa-large, DeBERTa-large, and GPT2-large). However, the more large-scale pre- trained models with tens of billions more param- eters were not studied due to limited computation resources. In addition, more comprehensive range tasks, like text generation, should be investigated. Our framework can be easily transferred to other backbone architectures and different types of tasks. We are eager to validate our framework to a broader range of scenarios in future work.
Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zong- han Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu. Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv, 0bs/2203.06904
Xuanyi Dong and Yi Yang. 2020. Nas-bench-200 Ex- tending the scope of reproducible neural architecture search. ArXiv, abs/2001 .00326.
Ethics Statement
Our proposed method aims to improve prompt tuning in terms of performance under a budget of parameter efficiency. The datasets we experiment with are widely used in previous work and, to our knowledge, do not have any attached privacy or ethical issues.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3816-3830, Online. Association for Computa- tional Linguisties.
References
Xiangxiang Gao, Wei Zhu, Jiasheng Gao, and Congrui Yin. 2023. F-pabee: Flexible- patience- based early exiting for single-label and multi-label text classifica- tion tasks. ArXiv, abs/2305. 1996..
Flexible-patience-
Elad Ben-Zaken, Shauli Ravfogel, and Yoav Gold- berg. 2021. Bitfit: Simple parameter-effiiient fine-tuning for transformer-based masked language- models. ArXiv, abs/2 106. 10199.
Zhao Guo, Yuan Ni, Keqiang Wang, Wei Zhu, and Guo Tong Xie. 2021. Global attention decoder for chinese spelling error correction. In Findings.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1999.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song. Jie Tang, Ji rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre- fre-trained models: Past, present and future. ArXiv, abs/2106.07139.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. 2021. Pro- gressive darts; Bridging the optimization gap for nas the wild. ArXiv, abs/1912. 10952.
Pengeheng Hc, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding- enhanced bert with disentangled attention. ArXiv, abs/2006.03654.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. 2021. Fairas:: Rethinking evaluation fairness of weight sharing neural architecture search. 2021 IEEF/CVF International Conference Computer Vision (ICCV), pages 12219-12228.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, Parameter- transfer learning for nip. In In- ternational Conference on Machine Learning, pages 2790-2799, PMLR.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert's attention. In Black- boXNLP@ACL.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges Workshop.
Edward J Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint a1X.v:2100.09685.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang. Yasheng Wang. Zhiyuan Liu, and Maosong Sun. 2022. Sparse structure search for parameter-effiiien tuning. ArXiv, abs/2206.0735 07382.
Christos Louizos, Max Welling, and Diederik Kingma. 2017. Learning sparse neural networks through 10 regularization. ArXiv, abs/ 7 12.01312.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypereomplex adapter layers. In NeurlPS.
Feihu Jin, Jinliang Lu, Jiajun Zhang, and Chengqing Zong. 2022. Instance-aware prompt learning for language understanding and generation. ArXiv, 005/2201.07126.
Bo Pang and Lillian Lec. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Annual Meeting of the Association for Computational Linguistics.
Tuan Le, Marco Bertolini, Frank No'e, and Djork-Arn Clevert. 2021. Parameterized hypercomplex graph neural networks for graph classification, In Interna- tional Conference on Artificial Neural Networks.
Bo Pang and Lillian Lee. 2005. Sceing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. Annual Meeting of the Association for Computational Linguistics.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiet prompt tuning, arXiv preprint arXiv:2104.0080 104.08691.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach DeVito, Martin Raison, Alykhan Tejami, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Jun- jie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary. In Neural Information Processing Systems.
Xiang Lisa Li and Percy Liang. 2021. -tunin: Optimizing continuous prompts for generation. Pro- ceedings of the 59th Annual Meeting of the Associa- tio Computational Linguistics and the Ith Inter- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), abs/2101 110.00090.... .00190.
Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang. Qi Meng, Tao Qin, Wei Chen, M. Zhang, and Tie-Yan Liu. 2021. R-drop: Regularized dropout for neural networks. ArXiv, abs/2 14448.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zeulemoyer. 2018. Deep contextualized word repre- sentations. In North American Chapter of the Associ- ation for Compulational Linguistics.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019a. Darts: Differentiable architecture search. ArXiv, 006.085
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Tryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Lin- guistics.
Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu. 2022a. Late prompt tuning: A late prompt could be better than many prompts. ArXiv, abs/2210.11229
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ArXiv, abs/2110.07602.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. In ICML.
Xiao Liu, Kaixuan Ji. Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022b.
Alec Radford and Karthik Narasimhan. 2018. Im- proving language understanding by generative pre- training. In OpenAI.
P-tuning: Prompt tuning can be comparable to fine- tuming across scales and tasks. In Annual Meeting of the Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Suiskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT Understands, Too. arXiv e-prints, page arXiv:2103.10385
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, 0bs//110.11e 10683.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: robustly optimized bert pretraining ap- proach. ArXiv, abs//1907. 16692.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Tryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. In Conference on Empir- ical Methods in Natural Language Processing.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.

---

Haixia Sun, Jin Xiao, Wei Zhu, Yilong He, Sheng Zhang. Xiaowei Xu, Li Hou, Jiao Li. Yuan Ni, and Guo Tong Xie. 2020. Medical knowledge graph to enhance fraud, waste, and abuse detection on claim data: Model development and performance evalua- tion. JMIR Medical Informatics, 8.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-thh-ee natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38 88-45, Online, Association for Computational Linguistics.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuan- jing Huang, and Xipeng Qiu. 2022. Black-box ArXiv,
tuning for aodgg---ddll------sseee- 0bs/2201.03514.
Tianyi Tang, Junyi Li, and Wayne Xin Zhao. 2022. Context-tuning: Learning contextualized prompts for natural language generation. In International Confer- ence on Computational Linguistics.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yux- iao Dong, V. G. Vimod Vydiswaran, and Hao Ma. 2022. Idpg: An instance-dependee ation method. In North American Chapter of the Association for Computational Linguistics.
prompt gener-
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Wein- berger, and Yoav Artzi. 2020. Revisiting few-sample bert fine-tuning. ArXiv, abs/2006.05987.
Xinpeng Zhang. Ming Tan, Jingfan Zhang, and Wei Zhu. 2023a. Nag-ner: unified non-autorerressive generation framework for various ner tasks. In An- nual Meeting of the Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, ness/706.03762..
Yuming Zhang, Peng Wang, Ming Tan, and Wei-Guo Zhu. 2023b. Learned adapters are better than man- ually designed adapters. In Annual Meeting of the Association for Computational Linguistics.
Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC- 8 question answering track. In Proceedings of the Second International Conference on Language Re- sources and Evaluation (LREC*00), Athens, Greece. European Language Resources Association (ELRA).
Wei Zhu, 2021a. LeeBERT: Learned early exit for BERT with cross-level optimization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2968-2980, Online. Association for Computational Linguistics.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis plat- form for natural language understanding. In Black- boxNLP @ EMNLP.
Wei Zhu. 202 Myp-bert: Multi-vocah pre-training for chinese bert. In Annual Meeting of the Associa- tion for Computational Linguistics.
Shuohuan Wang. Yu Sun, Yang Xiang, Zhihua Wu, Siyu Ding, Weibao Gong, Shi Feng, Junyuan Shang, Yan- bin Zhao, Chao Pang, Jiaxiang Liu, Xuyi Chen, Yuxi- ang Lu, Weixin Liu, Xi Wang, Yangfan Bai, Qiuliang Chen, Li Zhao, Shiyong Li, Peng Sun, Dianhai Yu, Yanjun Ma, Hao Tian, Hua Wu, Tian Wu, Wei Zeng, Ge Li, Wen Gao, and Haifeng Wang. 2021a. Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and genera- tion. ArXiv, abbs/2112.1277
Wei Zhu, Yilong He, Ling Chai, Yunxiao Fan, Yuan Ni, Guotong Xie, and Xiaoling Wang. 2021a. paht_nnj @ MEDIQA 2021: Multi-grained query focused summarization. In Proceedings of the 20th Workshop on Biomedical Language Process- ing, pages 96-102, Online. Association for Computa- tional Linguistics.
Wei Zhu, Peifeng Wang. Yuan Ni, Guo Tong Xie, and Xiaoling Wang. 2023a. Badge: Speeding up bert inference after deployment via block-wise bypasses and divergence- based early exiting, In Annual Meet- ing of the Association for Computational Linguistics.
Yaqing Wang. Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021b. List: Lite prompted self self-training makes parameter-efficient few-shot learners. In NAACI- İII.T
Wei Zhu, Peng Wang, Xiaoling Wang. Yuan Ni, and Guotong Xie. 2023b. Acf: Aligned contrastive fine- tuning for language and vision tasks. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions language. Language Resources and Evaluation, 39:165-210.

---

Wei Zhu, Xiaoling Wang, Yuan Ni, and Guotong Xie. 2021b. BERT: Improving BERT early exit- ing by gradient aligned mutual learning. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3033-3044, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Tunable parameters
RTE
Subj
Model
(acc)
(acc)
Mo
263k
76.4 (1.3)
93.9 (0.2)
Mi
262k
76.7 (0.9)
93.8 (0.2)
M2,1
295k
75.1
92.5 (0.2)
M2
295k
75.6 (0.9)
92.9 (0.4)
M2,3
295k
(1.1)
93.6 (0.1)
Yuhui Zuo, Wei Zhu, and Guoyong GUET Cai. 2022. Continually detection, rapidly react: Unseen rumors detection based on continual prompt-tuning In Proceedings of the 29th International Conference on Computational Linguisties, pages 3029-304 -041, Gyeongju, Republic of Korea. International Com- mittee on Computational Linguisties.
M2,
295k
(0.7)
94.2 (0.2)
Table 7: The experimental results for the pilot experi- ments, with the RoBERTa-large backbone. evalua- tion metric for the RTE and Subj tasks is accuracy (acc).
Model Mi different from LPT (Liu et al.. 2022a), we set the 13-th and 19-th layers as the prompt layers. The bottleneck dimension r of the prompt generators is set to 64 to main- tain comparable number of tunable parame- ters with model Moo
A Pilot experiments for manually designed prompt inserting strategies
Prompt tuning (Lester al., 2021) only adds unified prompt the word embeddings and does not perform well on downstream tasks. P-tuning v2 (Liu et al., 2021) improves upon prompt tuning by inserting prompts at each intermediate layer of PTM but suffers from difficulties in convergence. Liu et al. (2022a) argue that prompt tuning per- forms poorly mainly due to the long propagation path of task-related information, which causes the loss of task-related information during propagation in the frozen model and thus affects test perfor- mances. They further propose a late prompt tuning framework that brings significant improvements by generating an independent prompt for each sam- ple and inserting these prompts at the 13-th layer of the RoBERTa-large model (Liu et al., 2019b). Since RoBERTa-large has 24 Transformer blocks, the prompts that carry task-related information still have go through many transformer layers. Will another prompt generation layer after the 13-th layer help to preserve the task-related information and improve the test performances? Will adding more prompt generation layers be beneficial? Will adding too many prompt layers hurt the model per- formance?
Model M2,x: starting from layer 1, we set a prompt layer for every k layers (k: 1,2, 3, 4). The bottleneck dimension of the prompt generators are set 6, 12, 18, and 24, respectively.
Note that the above models, if a prompt layer has prompt propagated from the previous layers, we will average the newly generated prompt with the old prompt and insert the resulting prompt into the prompt layer. We can see that the above models simple variants of LPT (Liu et al., 2022a). The experimental settings for this pilot experiment are consistent with Section 5.2. The experiment results are reported in Table 7.
From Table 7, the following observations can be made: (a) we can see that some of the above simple manually designed models can perform compara- bly with or slightly outperform Liu et al. (2022a), with comparable numbers of tunable parameters. (b) Although adding more prompt layers can im- prove the performances (as for M222), setting too many prompt layers (like model M2,1) will hurt the model performances. The above observations suggest that we could obtain better performances of prompt tuning by appropriately setting prompt layers at certain intermediate layers.
Now conduct pilot experiments the RTE (Dagan al., 2005) and Subj (Pang and Lee, 2004) datasets using the RoBERTa-large model to shed light on the above research questions. We consider the following model settings with RoBERTa-large as the backbone model:
B Datasets and evaluation metrics
Model Mo: following LPT (Liu al., 2022a),
Dataset splits For SST-2, MNLI, MRPC, QNLI, QQP and RTE datasets from the GLUE bench-
we only insert soft prompt the 13-th layer of PTM. The bottleneck dimension r of the prompt generator 128.
Phtps:/wwww.qun..ccm/qquaaratete

---

Category
Datasets
Itrainl
Idevl
Itestl
[3]
Type
Labels
SST-2
66349
1000
872
2
sentiment
positive, negative
MPQA
7606
1000
2000
2
opinion polarity
positive, negative
Single-sentence
MR
7662
1000
2000
2
sentiment
positive, negative
Subj
7000
1000
2000
2
subjectivity
subjective. objective
Trec
4952
500
question classification
abbr, entity, description, human, loc., num.
MNLI
391702
1000
19647
3
NLI
entuilment, neutral, contradiction
MRPC
3668
204
204
2
paraphrase
equivalent, not equivalent entailment, not entailment equivalent, equivalent entailment, not entailment
Sentence-pair
QNLI
103743
1000
5463
2
NLI
QQP
362846
1000
40430
2
puruphrase
RTE
138
NLI
Table 8: The statistics of datasets evaluated in this work. For MNLI task, the number of samples in development and test sets summed by matched and mismatched samples. [}]] is the number of classes for dataset.
mark (Wang et al., 2018), the original test sets are not publicly available. Thus follow Zhang et al. (2020) and Mahabadi al. (2021) construct train/dev/test splits as follows: (a) for datasets with fewer than 10k samples (RTE, MRPC), we divide the original validation set in half, using one half for validation and the other for testing. (b) for larger datasets, we split Ik samples from the training set the development set and treat the original devel- opment set as the test set.
new prompts. We also consider T 1 for compar- ison. We set the bottleneck dimension and the targeted number of prompt layers K 4. For the PHM layers, we use the PyTorch imple- mentation of Le al. (2021), and set for the main experiments (Tables 11 and 1). We also run models with K E 81 for comparison.
128
Settings for SPT-DARTS For optimizing the hyper-network and learning the optimal prompt layer settings, we follow DARTS (Liu et al., 2019a)' bi-level optimization method. We equally split the original training set Drrain into two splits, D and Da on epoch start. Du is used to optimize the parameters in the prompt generators, and Da is used to optimize the gating parameters. The mean value for the Bernoulli architectural mask is set to 0.6. The learned SPT model is retrained from scratch in the original Dirain set and evaluated on the dev and test sets. All the parameters are opti- mized with the AdamW (Loshchilov and Hutter, 2019) optimizer with linear learning rate decay, 6% warmup ratio, and learning rate 5e-4. Under the full-data scenario. the bi-level optimization and retraining procedures will run for T epochs with batch size B. We set T = 30 and B = 8 for the MPQA, subj. TREC, MRPC, and RTE tasks, and T 15 and B 64 for SST-2 and QQP tasks. T 5 and 128 for MNLI and QQP tasks. For the GPT2-large model, we use the gradient ac- cumulation technique to avoid out-of-memory and control the actual batch size. Under the few-data scenario, the bi-level optimization and retraining procedures will run for 1000 steps with batch size step for
For four other classification datasets, we select a certain number of samples from the training set as the development set. The number of samples for each label is determined according to its proportion in the original training set. The dataset statistics after splitting are shown in Table 8.
Metries for the tasks For MNLI, we report the average accuracy score the matched and mis- matched test sets. For MRPC and QQP, report acc-fl, the average of the accuracy and F1 scores. And we report accuracy for all other tasks.
Implementation Details
Pretrained backbones We evaluate our method in both full-data and few-shot scenarios on three PTM backbones, RoBERTa-large (Liu et al., 2019b), DeBERTa-large (He et al., 2020), and GPT2-large (Radford al., 2019).
SPT Model settings We use the HugginFace Transformers (Wolf et al., 2020) as the code base for implementing our method. The prompt length is 20. We follow Gao et al. (2021) and show the used manual templates and label words in Table 9 and Table 10, respectively. Note that since the vocabulary of the GPT2 model does not have the [MASK] token, we justly use it to represent the po- sitions that are needed to predict. Unless otherwise specified, we set r 0.5. That is, our method will keep the previous layer's prompt when inserting
B 8. For the bi-level optimization, the model parameters and one for the architectural parameters constitute a complete step. The bi-level optimization process will only run once. We report the average performances and standard deviations on the test set of the learned SPT model across 5

---

Task
Template
Label words
SST-2
(S1) It was
MASK
positive:
great, negative: terrible
MPQA
(S1) It was
MASK
positive: great, negative: terrible
MR
(S1) It was MASK
positive:
great, negative: terrible
Subj
(S1) It was
MASK]]
subjective: subjective, objective: objective
abbreviation: Expression, entity: Entity, description: Description
TREC
[MAS]]: (SI)
human: Human, location: Location, numeric: Number
MNLI
(Si) ? [MASK] (S2)
entailment: Yes, netural: Maybe, contradiction: No
MRPC
(Si) [MASK]] (82)
equivalent: Yes, not equivalent: No
QNLI
(S1) ? [MASK] (12)
entailment: Yes, not entailment: No
QQP
(S1) [MASK], (82)
equivalent: Yes, not equivalent: No
RTE
(S1) ? [MAKK] (S2)
entailment: Yes, not entailment: No
Table 9: The manual templates and label words used on RoBERTa and DeBERTa models.
Task
Template
Label words
Subj
(S1) was [MASK].
subjective: subjective, objective: objective
abbreviation: Expression, entity: Entity, description: Description
TREC
[MASK]: (S1)
human: Human, location:
Location, numerie: Number
MRPC
(S1) (S2) They are [MSKK]
equivalent: Yes, not equivalent: No
RTE
(Si) (52) They are MMSKK]
entailment: Yes, not entailment: No
Table 10: The manual templates and label words used on the GPT2-large model.
random seeds under the full-data scenario and 10 random seeds under the few-shot scenario.
E Experimental results under few-shot settings
Settings for the baselines We also add man- ual templates in Tables 9 and 10 to transfer the downstream tasks to (masked) language modeling tasks. For adapter-based tuning methods, we set the down-projection size to 16. We set the soft prompt length 20 for prompt tuning (Lester et al., 2021). P-tuming v2 (Liu et al., 2022b), IDPG (Wu et al., 2022) and LPT (Liu et al., 2022a). Besides, we set the down-projcction size m IDPG (Wu et al., 2022) and LPT (Liu et al., 2022a) to 128. The hyperparameter r and a in LoRA (Hu et al., 2021) are set to 8 and 16 on RoBERTa-large 4 and 32 on GPrg-large. The settings for the optimizer, learning rate, warm-up, and batch size the same our SPT method. And we also report the age performances and standard deviations on the test set across random seeds under the full-data scenario and 10 random seeds under the few-shot scenario.
In the main content, we present the results for the few-shot scenario of 100 samples. Here, we present the results for the few-shot scenario of 200 samples and 500 samples. Under a given random seed, we randomly sample the training samples from the original training set. We will run the experiments over 10 different random seeds and report the mean and deviation on each task. The pretrained backbone model is RoBERTa-large. The results are presented in Table 12.
F Efficiency evaluations for SPT
In this section, we measure the memory con- sumption and inference speed for three methods: prompt tuning, LPT and SPT. The pre-trained back- bone is RoBERTa-lagee, the batch size is set to 32 and the maximum sequence length is 128. We report the measures in Table 13.
D Experimental results under the full-data setting
In the main content, we present the results for the few-shot scenario of 100 samples. Here, we present the results for the full-data scenario. The results are presented in Table 11.
G Effects of the prompt length /
We now present the experimental results for changing the prompt length in Table 14.

---

Tunable
SST-2
MPQA
MR
Subj
TREC
MNLI
MRPC
QNLI
QQP
RTE
Method
Arg
Params
(60c)
(ace)
(acc)
(ac))
(a0c)
(ace)
(ac-f1))
(80)
(ace-())
(acc)
Model tuning
355M
(0.1)
(0.6005)
90.5 (0.3) 95.8 (0.4)
93.6 (0.3)
89.3 (0.1)
89.5 (..)
(0.3
98.7 (0.1)
(1.))
90.8
Adapter
16M
25.1 (0.2)
(0.5
89.3 (0.4)
(0.004
92.3
88.5 (0.1)
(..3
(0.3
(9.4
18.21
89.7
AdapteeDrup
S1IK
(0.3)
89.1
(0.5)
(4.4
92.2 (0.5
88.3 (0.2)
88.1
90.4 (0.3)
07.6
(1.4)
89.2
Bilfi
273K
(0.1
(0.1
(9.9...3)
94.3 (0.1)
(0.3)
$8.9 (0.11
(7.7
90.9 (0.21
(1.91.44)
71.3(11)
39.1
LoRA
778K
(0.1 (0.3 (0.3)
92.3 (..0)
(8.6
91.1
00.3
(2.1
89.7
Prompe Tuning
21K
(0.9
58.8 (0.8)
88.6 (0.5)
92.6 (0.6)
50. (6.6)
(0.4)
(..3
(0.1
(1.2
(0.5)
85.1
P-tuning v2
985K
(0.4)
89.9 (0.6)
89,4 (0.4)
93.2
92.4 (0.6)
88.5 (0.3)
(2.1
(0.3)
87.4
(2.3)
88.0
IDPG
296K
94.3
89.5
89.6
93.3
91.9 (0.4)
87.9 (0.5)
88.6 (0.4)
(1.3
(1.99)))
88.2
LPT
263K
94.8 (0.2)
89.1 (0.3)
89.5 (0.1)
93.9 (0.2) 92.1 (0:2)
88.6 (0.3)
88.3 (1.0)
89.7 (0.5)
88.5 (0.4)
76.4 (1.3)
89.1
Our SPT method
SPT
152K
95.2 (0.1)
90.5
90.2 (0.2)
(0.1) 93.5 (0.3)
89.0 (0.3)
89.1 (0.7)
90.8
79.2 (9.2.
90.2
SPT (r 1.0)
152K
94.9 (0.2)
89.9 (0.3)
89.8 (0.4)
94.7 (0.2) 92.7 (0.2)
88.6 (0.4)
88.8 (1.0)
90.2 (0.5)
88.6 (0.4)
(1.3)
89.7
Table 11: The Overall comparison in full-data scenario. We report mean and standard deviation of performance over 5 different random seeds for all the methods. Bold and Underline indicate the best and the second best results. PT-256 indicates prompt tuning with prompt length 256. All the results are obtained using RoBERTa-large as the pre-trained backbone.
Tunable Params
SST-2 (acc)
MNLI
RTE (acc)
Subj (acc)
Method
(acc)
Fho-shot scenario: 200 samples
Model tuning
355M
88.9 (0.9) 53.3 (2.1)
51.6 (2.4)
89.9 (1.1)
Prompt tuning
26K
88.2 (0.7)
49.6 (1.7)
58.1 (2.2)
86.7 (1.2)
LPT
329K
90.9 (0.9) 53.6 (1.5)
59.4 (1.7)
90.7 (0.9)
SPT
186K
92.3 (0.7) 56.1 (1.6)
61.6 (1.9)
91.6 (1.0)
Few scenario:
500 samples
Model tuning
355M
(0.7)
63.1 (1.8)
57.2 (1.6)
90.9 (0.7)
Prompt tuning
26K
89.1 (0.7)
54.3 (2.1)
63.5 (1.5)
(1.1)
LPT
263K
91.1 (0.9)
62.7 (1.6)
66.8 (1.6)
91.3 (0.8)
SPT
152K
92.6 (0.7) 65.3 (1.4) 68.1 (1.4)
92.1 (0.9)
Table 12: Results for the few-shot scenario of 200 samples and 500 samples. The pretrained backbone model is RoBeRTa-large. Bold indicates the best result among the PETuning methods.
Method
Speed (it/s)
Memory (GB)
H.2 Results on the BAS-Bench-201
Prompt tuning
9.14
3.92
NAS-Bench-201 (Dong and Yang, 2020) is the most widely applied and investigated NAS bench- mark analyzing various NAS methods. NAS- Bench-201 provides a DARTS-like search space, containing 4 internal nodes with 5 associated op- erations. The search space consists of 15,625 ar- chitectures, with the ground truth performance of CIFAR-10, CIFAR-100 and ImageNet16-122 of each architecture provided. On MA.S-BBnch-2000 the searching settings are kept the same as DARTS on (Dong and Yang, 2020). Table 16 reports the performances.
LPT
9.75
3.95
SPT
9.46
4.05
Table 13: Results for the efficiency measures. The memory and speed during inference on the RTE test set is presented. The backbone is RoBERTa-large, the batch size is set to 32 and maximum sequence length is 128.
H Appendix for analysis on the SPT-DARTS method
H.1 Ablation on the NAS methods
Table 1.5 report the performance of SPT when the SPT-DARTS method is replaced by the other popular NAS methods: ENAS (Pham al., 2018). DARTS (Liu et al., 2019a), P-DARTS (Chen et al.. 2021), FairNAS (Chu al., 2021), and Lo regular- ization (Louizos et al., 2017). The model backbone is RoBERTa-large

---

Prompt length
SST-2
MNLI
RTE
Subj
Method
(acc)
(acc)
(acc)
(ace)
LPT
89.7 (0.8)
52.5 (2.0)
(3.5)
89.7 (1.7)
LPT
5
89.3 (0.9)
(1.8
(2.9)
88.9 (1.6)
LPT
20
89.9 (0.8)
(2.1)
(3.6
SPT
10
90.8 (1.0)
(1.9
(8.9
90.8 (0.9)
SPT
5
90.7 (0.9)
(1.6)
(2.1)
90.8 (0.8)
SPT
20
90.8
54.8 (1.8)
(2.3)
90.9 (1.1)
Table 14: Results on 4 GLUE tasks with different prompt length using RoBERTa large as the backbone under the few-data scenario (100 samples). Bold indicates the best result among PETuning method.
Tunable
SST-2
MNLI
RTE
Subj
Method
Params
(acc)
(acc)
(acc)
SPT-DARTS
152K
95.2 (0.1)
89.0 (0.3)
79.2 (1.1)
95.3 (0.1)
La regularization
152K
94.1 (0.3)
87.9 (0.7)
78.1 (1.9)
93.7 (0.4)
ENAS
152K
94.3 (0.2)
88.2 (0.5)
77.6 (1.7)
94.3 (0.3)
DARTS
152K
94.2 (0.1)
88.1 (0.5)
78.3 (1.8)
93.8 (0.4)
P-DARTS
152K
94.6 (0.2)
88.4 (0.7)
78.3 (1.6)
94.5 (0.3)
FaiNNAS
152K
94.4 (0.3)
88.2 (0.6)
78.1 (1.8)
94.3 (0.2)
Table 15: Comparisons of different hyper-network optimization methods on our SPT framework.
costs (hours)
CIFAR-
CIFAR- 100
TmageNet16-115
valid
test
valid
test
valid
test
ENAS
3,9
37.51 +: 3.19
53.89
009900.5 0.58
13.37 = 2.35
13.96 +: 2.33
15.06 +: 1.95
14.84 2.10
DARTS
3.2
39.77
0.00
54.30 0.00
15.03 - 0.00
15.61 0.00
16.43
0.00
16.32 4 0.00
PC-DARTS
89,96
0.15
93.41 0.30
67.12 = 0.39
67.48
0.89
40.83
0.08
41.31 + 0.22
SPT-DARTS
5.2
91.16 + 0.45
93.89 0.41
70.45 = 0.52
70.61 + 0.50
44,92 + 0.48
45.23 + 0.58
Table 16: Performance comparison NAS-Bench-201 benchmark.


Reviews:
[{"review_id": "0X49l4Rg6F", "reviewer": "Reviewer_AMiu", "summary": "", "questions": "A: At training stage, will all the $a_i$ be updated? If yes, it seem that SPT actually need $l$ prompt generators, where $l$ is the number of layers of the language model. If that is the case, then SPT still need to train $l$ prompt generators since SPT-DARTS first optimize $\\omega$ on $\\mathcal{D}_{\\omega}$.\n\nB: How is the hyper-parameter choosed for baseline models?\n\nC: Do the author(s) think the method or idea proposed in this paper can directly facilitate or have any implications on the usage and adaption of cutting-edge large language models (e.g. ChatGPT)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough reviews provided by the reviewer. We will carefully consider the valuable questions and suggestions as we prepare the final version.\n\n\n### Question 1\n\nQUESTION: Given the emergent of cutting-edge large language models and their decent in-context performance, the technique proposed in this paper seems to be redundant and outdated.\n\nResponse: We hold a different perspective. OpenAI's recent announcement on August 22, 2023, reveals the availability of Fine-tuning for GPT-3.5 Turbo, followed by fine-tuning support for GPT-4 later this fall. (URL: https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates) OpenAI explains that fine-tuning offers distinct advantages compared to prompting or in-context learning: (a) Enhanced steerability. (b) Improved adherence to specific formats. (c) Customizable tone. (d) Reduced prompt length, leading to lower API fees, enhanced efficiency, and decreased GPU memory usage. \n\nIn the following, we will present experimental evidence demonstrating that even language models weaker than ChatGPT or GPT-4 can outperform these models when fine-tuned on a diverse set of tasks.\n\n\n#### (1a) Results of English tasks\n\nFirst, we experimented with English tasks. In the case of ChatGPT, we provided as many demonstrations as possible to fill its 4096-context length. During the fine-tuning process, we utilized the recently released open-source LLM, LlaMA-2 13B by Meta (available at: https://huggingface.co/meta-llama/Llama-2-13b-hf) as the backbone model. This model was fine-tuned using a few-data approach, utilizing only 200 samples.\n\nFor our SPT method, we generated soft prompts by utilizing the hidden states of the instructions (or prompts). These soft prompts were then concatenated after the instruction. We retained a total of 8 prompt layers, each with a length of 10 tokens. The bottleneck dimension was set at 512, and the rank 'n' was set to 128. With this configuration, the number of tunable parameters amounted to 11.5 million.\n\nIn the case of LoRA method, we set the rank to 4, and this rank was applied to all linear layers within the transformer block. This resulted in a total of 15.6 million tunable parameters.\n\nThe experiments were conducted on the following tasks: (a) RTE, which involves natural language inference. (b) COPA, a task focused on commonsense reasoning. (c) ShaRE-13, a nested named entity recognition (NER) task within the biomedical domain. (d) MultiArith, a task centered around arithmetic reasoning. The results of these experiments are presented in the following table.\n\n\n| Task      | ChatGPT      |  LlaMA-2 13B + LoRA      |   LlaMA-2 13B + SPT      |\n|-------------|--------|--------|--------|\n| RTE        |   0.848   |  0.857   |   0.869   |\n| COPA        |   0.732   |  0.887   |   0.906   |\n| ShaRE-13        |   0.331   |  0.743   |   0.764   |\n| MultiArith     |   0.953   |  0.956   |   0.959   |\n\nFrom the above table, the following observations can be made: \n- (1) ChatGPT demonstrates strong performance in tasks such as RTE and MultiArith. In contrast, when employing our SPT method with 200 training samples, fine-tuned LlaMA-2 13B exhibits either comparable or superior performance compared to ChatGPT.\n- (2) Across the ShaRE-13 and COPA tasks, LlaMA-2 13B showcases a clear performance advantage over ChatGPT.\n- (3) On all the above tasks, when having comparable tunable parameters, SPT can outperform LoRA under the few-data setting. \n\nThe outcomes of these experiments underscore the effectiveness of SPT in enhancing the performance of an open-source LLM, enabling it to either match or surpass ChatGPT's performance across diverse tasks.\n\n\n\n#### (1b) Results of Chinese tasks\n\nFor Chinese tasks, we choose the CMeEE-v2 (medical NER), CMeIE (medical triple extraction task), and IMCS-V2-MRG (medical report generation) tasks from PromptCBLUE (https://tianchi.aliyun.com/competition/entrance/532085/introduction). The backbone LLM is Chinese LlaMA-2 13B (https://huggingface.co/ziqingyang/chinese-llama-2-13b/tree/main) The other experimental settings are kept the same with above.\n\n| Task      | ChatGPT      |  Chinese LlaMA-2 13B + LoRA      |   Chinese LlaMA-2 13B + SPT      |\n|-------------|--------|--------|--------|\n| CMeEE-v2        |   0.469   |  0.673   |   0.687   |\n| CMeIE        |   0.305   |  0.602   |   0.618   |\n| IMCS-V2-MRG        |   0.325   |  0.474   |   0.486   |\n\nFrom the results, we observe the following: (a) Fine-tuned Chinese LlaMA-2 13B outperforms ChatGPT significantly in the context of the two Chinese medical information extraction tasks. (b) While ChatGPT demonstrates strong performance in open-domain summarization, its performance in IMCS-V2-MRG is notably inferior to that of fine-tuned Chinese LlaMA-2 13B in terms of Rouge-L score. This task necessitates the model to succinctly summarize patient-doctor dialogues, and Language Models (LLMs) exhibit superior summarization capabilities in this style following fine-tuning.\n\n\nIn the final version, we will also present the results of GPT-4 on these tasks. And we will also try out the OpenAI’s fine-tuning APIs to see how ChatGPT performs after fine-tuning with the same set of 200 training samples. \n\n\n\n### Question 2\n\nQUESTION: Will $a_i$ be updated? \n\nResponse:  Yes, the learnable gate parameters $a_i$ will be updated during training following our equation 4 and equation 7. \n- Through optimization, the probabilistic gate $a_i$’s value will move toward 0 or 1, acting as importance scores for the prompt layers. Then the prompt generators with the $K$ largest $a_i$ values will be kept, while other prompt generators will be discarded. And the gate parameters $a_i$ will be set to 1 for those layers with prompt generators, and 0 for other layers. \n- Then the learned SPT model with the selected K prompt generators will be trained on the training set again, following the standard model training procedure.\n- Thus, during inference, the learned SPT model will only has K prompt generators.\n\n\n### Question 3\n\nQUESTION: I suggest the author add a subsection in section 4 illustrating that SPT uses only the layers with larger $a_i$ for prompt inserting.\n\nResponse:  We will take the time to thoroughly revise Section 4 to ensure that it is more reader-friendly.\n\nIf K prompt generators are selected, then in the transformer layer without a prompt generator, we will utilize the soft prompt from the preceding layer, or in some cases, no soft prompt at all. When the current transformer layer contains a prompt generator, it combines the soft prompt from the preceding layer (or no previous soft prompt) with the newly generated soft prompt. Our Figure 3 (on page 7) demonstrates the K selected prompt layers on the RoBERTa-large backbone.\n\n\n### Question 4\n\nQUESTION: How is the hyper-parameter choosed for baseline models?\n\n\nResponse:  For the baseline comparisons, we implement the baseline methods using OpenDelta (https://github.com/thunlp/OpenDelta) or their official code repositories. With these methods, we only adjust the hyperparameters directly associated with tunable parameter quantities (such as rank, bottleneck dimension, and prompt length). This ensures that the tunable parameter quantities align with those presented in Table 1.\n\nAs for the other hyperparameters, particularly the training-related ones, we determine their values using the following approach: If OpenDelta is utilized, we adopt the parameter settings outlined in its corresponding paper, 'Delta tuning' (https://arxiv.org/abs/2203.06904). In cases where the original code of a method is employed, we adhere to the hyperparameters suggested by the original authors.\n\n\n\n### Question 5\n\nQUESTION: Do the author(s) think the method or idea proposed in this paper can directly facilitate or have any implications on the usage and adaption of cutting-edge large language models (e.g. ChatGPT)?\n\n\nResponse:  Despite the fact that ChatGPT's fine-tuning method is controlled by ChatGPT, we believe that our method can be applied to fine-tune the more recent LLMs. In the response above to question 1, we have presented some of our recent experimental results with the open-sourced LlaMA-2 models.\n\nIn the final version, we will present more comprehensive experimental results using various open-source LLMs such as ChatGLM-6B, Baichuan-13B, and Falcon, among others.\n\n\n\n\n### Question 6\n\nQUESTION: On the missing References.\n\nResponse:  We will correct this error in the final version. Additionally, we will thoroughly review all references to ensure the proper citations have been made. \n\n\n### Question 7\n\nQUESTION: On Typos Grammar Style And Presentation Improvements.\n\nResponse:  We will thoroughly proofread the entire manuscript and enhance the presentation of this work in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a novel prompt tuning approach, that is, automatically selecting several layers to insert instance-dependent soft prompts. This paper also proposes the corresponding optimization method.\n2. The results of the experiments verify the effectiveness of proposed method.\n3. The ablation study is comprehensive.", "reasons_to_reject": "1. The writing in section 4.3 and 4.4 is confusing. If my understanding is right, I suggest the author add a subsection in section 4 illustrating that SPT uses only the layers with larger $a_i$ for prompt inserting.\n2. Given the emergent of cutting-edge large language models and their decent in-context performance, the technique proposed in this paper seems to be redundant and outdated.", "questions_to_authors": "A: At training stage, will all the $a_i$ be updated? If yes, it seem that SPT actually need $l$ prompt generators, where $l$ is the number of layers of the language model. If that is the case, then SPT still need to train $l$ prompt generators since SPT-DARTS first optimize $\\omega$ on $\\mathcal{D}_{\\omega}$.\n\nB: How is the hyper-parameter choosed for baseline models?\n\nC: Do the author(s) think the method or idea proposed in this paper can directly facilitate or have any implications on the usage and adaption of cutting-edge large language models (e.g. ChatGPT)?", "paper_topic_and_main_contributions": "This paper studies the selection of prompt layer in prompt tuning methods. The authors first conduct a pilot exam to show that prompt inserting strategies affects the performance of prompt tuning. The authors further propose an optimization method to automatically search for the optimal layer to insert prompt. Results of experiments shows the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There seems a missing citation in line 185."}, "extra_text": {"Missing_References": "It seems that the citation to P-tuning (Liu et al., 2022b) in line 131 should be replaced to [1].\n\nReferences:\n\n[1] Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021). GPT understands, too. arXiv preprint arXiv:2103.10385."}}, {"review_id": "x3jPtLlcaf", "reviewer": "Reviewer_XNTB", "summary": "", "questions": "Please refer to the reasons to reject section. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for conducting a comprehensive review. The insightful questions and suggestions provided will be carefully considered as we work on the final version.\n\n\n### Question 1\n\nQUESTION: To ensure the illustration aligns with the method's description, I suggest revising the figure. \n\nResponse: We will adjust Figure 2 to more accurately depict our method, particularly emphasizing Equation 3.\n\n\n### Question 2\n\nQUESTION: Could we train just one prompt generator, with the differing input hidden states provided from layer to layer?\n\nResponse: This is a thought-provoking question. The configuration of the shared prompt generator, referred to as SPT-shared-PG, is definitely worth investigating. We now conduct experiments to compare SPT-shared-PG with our main model, SPT.\n\n#### (2a) Experiments with RoBERTa-large \n\nWe first conduct experiments on the RoBERTa-large backbone, same as Table 1. For SPT-shared-PG, the number of finally kept prompt layers are 4, and the prompt length is 10, and bottleneck dimension is 128, and rank n = 32. Under this setting, the tunable parameter number is 152K, the same with SPT on Table 1. In the few-shot scenario of 100 training samples, the avg score on the 10 classification task is 77.2, while our main model, SPT, achieve the avg score of 78.0. In the few-shot scenario of 200 training samples, the avg score on the 10 classification task is 77.8, while our main model, SPT, achieve the avg score of 78.5.\n\n\nWe initially conducted experiments using the RoBERTa-large backbone. For the SPT-shared-PG model, we retained a total of 4 prompt layers, with each prompt having a length of 10 tokens. The bottleneck dimension was set to 128, and the rank 'n' was chosen as 32. With these parameters, the number of tunable parameters was 152K, which aligns with the SPT configuration in our Table 1.\n\nIn the scenario where only 100 training samples were available, SPT-shared-PG attained an average score of 77.2 across the 10 classification tasks. In contrast, our primary model, SPT, achieved a slightly higher average score of 78.0. As the number of training samples increased to 200 in the few-shot scenario, the average score of SPT-shared-PG for the 10 classification tasks improved to 77.8. Once again, our primary model, SPT, exhibited superior performance with an average score of 78.5.\n\n\n\n#### (2b) Experiments with open-sourced LLMs\n\n\nWe now conduct experiments using the recently released open-source language model, LlaMA-2 13B, developed by Meta (available at: https://huggingface.co/meta-llama/Llama-2-13b-hf). We perform fine-tuning with a limited dataset comprising 200 samples. Our experiments cover the following tasks: a) Recognizing Textual Entailment (RTE) - a natural language inference task. b) Choice of Plausible Alternatives (COPA) - a task involving commonsense reasoning. c) ShaRE-13 - a nested Named Entity Recognition (NER) task within the biomedical domain. d) MultiArith - a task focused on arithmetic reasoning.\n\n\n For SPT, the soft prompts are generated using the hidden states of the instructions (or prompts), and they are concatenated after the instruction. Specifically, eight prompt layers are retained, each with a prompt length of ten. The bottleneck dimension is set at 512, and the rank, denoted as 'n', is 128. With these settings, the total number of tunable parameters is 11.5 million. For SPT-shared-PG, the experimental setup is similar, except for setting 'n' to 512 to ensure comparability in the number of tunable parameters. \n\n The results are reported below. SPT-shared-PG performs slightly better than LoRA under the few-data setting of 200 samples, but it underperforms compared to our main model, SPT.\n\n\n\n| Task         |  LlaMA-2 13B + LoRA      |   LlaMA-2 13B + SPT      | LlaMA-2 13B + SPT-shared-PG      |\n|-------------|--------|--------|--------|\n| RTE        |    0.857   |   0.869   |   0.857   |\n| COPA        |    0.887   |   0.906   |   0.901   |\n| ShaRE-13        |    0.743   |   0.764   |   0.756   |\n| MultiArith     |    0.956   |   0.959   |   0.957   |\n\n\n\nFrom the results presented above, it is evident that the configuration of SPT-shared-PG demonstrates commendable performance across various tasks. However, our primary model, SPT, outperforms it. This outcome is to be expected, as we employ distinct prompt generators to align with the hidden states of different layers. \n\n\n### Question 3\n\nQUESTION: Why the prompt length for the proposed method and other comparing methods are set differently?\n\nResponse:  We apologize for any confusion caused by the typo. In line 457, the prompt length for all the baseline prompt-tuning-based methods is actually 10, not 20.\n\nAs stated in line 559, we configured the prompt length in the main experiment (Table 1) to be 10 in order to ensure a fair comparison. Additionally, the outcomes of various prompt lengths are provided in Table 13 of Appendix I.\n\n\n### Question 4\n\nQUESTION: It would be beneficial to include a discussion or analysis showing the progression of the learnable gates throughout the training.\n\nResponse:  We thank the reviewer for this great question. In the final version, we intend to incorporate a visual representation illustrating the dynamic changes of the learnable gates throughout the course of training. \n\nBased on our observations, we typically notice fluctuations in the values of $a_i$ during the initial steps of training. After around 200 steps, we observe a smoother convergence of $a_i$ towards either 0 or 1.\n\n\n### Question 5\n\nQUESTION: If equations 3 and 8 are accurately bracketed, it implies that you’re highly weighting the previous prompt at the outset of the training.\n\n\nResponse:  Actually, in equation 3, we use the hyper-parameter $\\tau \\n {0.5,1.0}$ to determine whether the prompt layer will incorporate the soft prompt propagated from the previous layer. We now explain how will $\\tau$ the final learned SPT model (where a_i is discretized to 0 or 1)\n- If $\\tau=0.5$, \n\t- at the prompt layer, a_i=1, so the soft prompt will be the average of the previous layer’s soft prompt and the newly injected soft prompt. \n\t- At other layers, a_i=0, so that no new soft prompt will be included. And this layer will use the soft prompt propagated from the previous layer (or no soft prompt at all). \n- If $\\tau=1.0$,\n\t- At the prompt layer, $a_i=1$, so the soft prompt will be only the newly injected soft prompt. \n\t- At other layers, $a_i=0$, so that no new soft prompt will be included. And this layer will use the soft prompt propagated from the previous layer (or no soft prompt at all).\n\n\n### Question 6\n\nQUESTION: how the model performs when consistency regularization is not applied.\n\n\nResponse:  In Section 5.7 (lines 621 to 631), we conducted an ablation study on the hyper-network optimization method introduced in Section 4.4. This method involves the re-parameterization of probabilistic gates and architectural consistency learning. The results, presented in Table 14 of Appendix J, demonstrate that our approach achieves better performance compared to other hyper-network optimization methods.\n\nAs the reviewer pointed out, we haven’t conducted an ablation study solely on consistency regularization. In the following table, we present the results. 'SPT w/o consistency' refers to conducting the hyper-network optimization method without the consistency regularization objective. The other settings remain the same as the responses for Question 2.\n\n\n| Task         |  LlaMA-2 13B + LoRA      |   LlaMA-2 13B + SPT      | SPT w/o consistency      |\n|-------------|--------|--------|--------|\n| RTE        |    0.857   |   0.869   |   0.863   |\n| COPA        |    0.887   |   0.906   |   0.903  |\n| ShaRE-13        |    0.743   |   0.764   |   0.759   |\n| MultiArith     |    0.956   |   0.959   |   0.956   |\n\n\nWe observe that when utilizing the LlaMA-2 13B backbone along with a limited dataset of 200 samples for fine-tuning, the inclusion of the consistency regularization term results in notable performance improvements. However, it's noteworthy that even in the absence of consistency regularization, SPT consistently outperforms the LoRA method. Therefore, if computational resources permit, it would be advisable to incorporate the consistency regularization technique for enhanced training outcomes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The researchers propose a technique for identifying the optimal layer for prompt integration. Instead of utilizing an empirical selection process, this approach employs an algorithmic search for finding the most effective layer.\n\n2 .To add clarity and depth to their findings, the authors provide a comprehensive analysis from various perspectives.\n\n3. The proposed method outperforms the prior related research.", "reasons_to_reject": "1. Based on my understanding, the proposed method appears to adaptively update the prefix using a learnable probabilistic gate. However, the provided figure seems to depict a process of concatenating the prompt instead. To ensure the illustration aligns with the method's description, I suggest revising the figure. \n\n2. With regards to training, I'm curious about the requirements for the prompt generator. Is it necessary to have a separate prompt generator for each layer? Or, could we train just one prompt generator, with the differing input hidden states provided from layer to layer?\n\n3. Why the prompt length for the proposed method and other comparing methods are set differently? (searched as hyperparameter?)\n\n4. If equations 3 and 8 are accurately bracketed, it implies that you’re highly weighting the previous prompt at the outset of the training. It would be beneficial to include a discussion or analysis showing the progression of the learnable gates throughout the training. This would provide valuable insights into how the previous prompts are integrated over time.\n\n5. I'm interested in how the model performs when consistency regularization is not applied. If reasonable performance can be achieved without the use of this regularization technique, it could potentially allow for a trade-off between performance and computational cost. Specifically, this might eliminate the need for multiple forward passes, thereby reducing computational demand.\n", "questions_to_authors": "Please refer to the reasons to reject section. ", "paper_topic_and_main_contributions": "This study presents the method for selecting specific layers to input a prompt with a learnable probabilistic gate. Through the bi-level optimization, they were able to enhance the convergence of the learnable gates. Such improvement results in enhanced overall performance. Numerous experiments have been conducted to demonstrate the efficacy of this framework.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. I'm wondering the usage of brackets is correct in eq. 3, and eq.8 \n2. In table 12, 13 there is not bold in the table. please modify the tables or captions. "}, "extra_text": {}}, {"review_id": "2yoq60fkTy", "reviewer": "Reviewer_yfyq", "summary": "", "questions": "See my questions in \"reasons to accept\", and \"reasons to reject\".", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for conducting a comprehensive review. The insightful questions and suggestions provided will be carefully considered as we work on the final version.\n\n\n### Question 1\n\nQUESTION: Whether our method works for other use cases w/ LMs or even other models?\n\n\nWe thank the reviewers for the great question. We will use experiments to show that our method works well with the recent released LLMs, on a wide range of tasks. Notably, after fine-tuning, the open-sourced models will perform better than ChatAGPT or GPT-4 with ICL. \n\n\n#### (1a) Results of English tasks\n\nFirst, we experimented with English tasks. In the case of ChatGPT, we provided as many demonstrations as possible to fill its 4096-context length. During the fine-tuning process, we utilized the recently released open-source LLM, LlaMA-2 13B by Meta (available at: https://huggingface.co/meta-llama/Llama-2-13b-hf) as the backbone model. This model was fine-tuned using a few-data approach, utilizing only 200 samples.\n\nFor our SPT method, we generated soft prompts by utilizing the hidden states of the instructions (or prompts). These soft prompts were then concatenated after the instruction. We retained a total of 8 prompt layers, each with a length of 10 tokens. The bottleneck dimension was set at 512, and the rank 'n' was set to 128. With this configuration, the number of tunable parameters amounted to 11.5 million.\n\nIn the case of LoRA method, we set the rank to 4, and this rank was applied to all linear layers within the transformer block. This resulted in a total of 15.6 million tunable parameters.\n\nThe experiments were conducted on the following tasks: (a) RTE, which involves natural language inference. (b) COPA, a task focused on commonsense reasoning. (c) ShaRE-13, a nested named entity recognition (NER) task within the biomedical domain. (d) MultiArith, a task centered around arithmetic reasoning. The results of these experiments are presented in the following table.\n\n\n| Task      | ChatGPT      |  LlaMA-2 13B + LoRA      |   LlaMA-2 13B + SPT      |\n|-------------|--------|--------|--------|\n| RTE        |   0.848   |  0.857   |   0.869   |\n| COPA        |   0.732   |  0.887   |   0.906   |\n| ShaRE-13        |   0.331   |  0.743   |   0.764   |\n| MultiArith     |   0.953   |  0.956   |   0.959   |\n\nFrom the above table, the following observations can be made: \n- (1) ChatGPT demonstrates strong performance in tasks such as RTE and MultiArith. In contrast, when employing our SPT method with 200 training samples, fine-tuned LlaMA-2 13B exhibits either comparable or superior performance compared to ChatGPT.\n- (2) Across the ShaRE-13 and COPA tasks, LlaMA-2 13B showcases a clear performance advantage over ChatGPT.\n- (3) On all the above tasks, when having comparable tunable parameters, SPT can outperform LoRA under the few-data setting. \n\nThe outcomes of these experiments underscore the effectiveness of SPT in enhancing the performance of an open-source LLM, enabling it to either match or surpass ChatGPT's performance across diverse tasks.\n\n\n\n#### (1b) Results of Chinese tasks\n\nFor Chinese tasks, we choose the CMeEE-v2 (medical NER), CMeIE (medical triple extraction task), and IMCS-V2-MRG (medical report generation) tasks from PromptCBLUE (https://tianchi.aliyun.com/competition/entrance/532085/introduction). The backbone LLM is Chinese LlaMA-2 13B (https://huggingface.co/ziqingyang/chinese-llama-2-13b/tree/main) The other experimental settings are kept the same with above.\n\n| Task      | ChatGPT      |  Chinese LlaMA-2 13B + LoRA      |   Chinese LlaMA-2 13B + SPT      |\n|-------------|--------|--------|--------|\n| CMeEE-v2        |   0.469   |  0.673   |   0.687   |\n| CMeIE        |   0.305   |  0.602   |   0.618   |\n| IMCS-V2-MRG        |   0.325   |  0.474   |   0.486   |\n\nFrom the results, we observe the following: (a) Fine-tuned Chinese LlaMA-2 13B outperforms ChatGPT significantly in the context of the two Chinese medical information extraction tasks. (b) While ChatGPT demonstrates strong performance in open-domain summarization, its performance in IMCS-V2-MRG is notably inferior to that of fine-tuned Chinese LlaMA-2 13B in terms of Rouge-L score. This task necessitates the model to succinctly summarize patient-doctor dialogues, and Language Models (LLMs) exhibit superior summarization capabilities in this style following fine-tuning.\n\n\n\n### Question 2\n\nQUESTION: On the contributions of the work.\n\nWe believe that our work is a non-trivial addition to the current literature:\n- Our approach, called SPT, focuses on the automatic selection of the prompt layer in pre-trained transformer models. This is achieved by optimizing the learnable probabilistic gate $a_i$, in conjunction with the parameters in the prompt generator. To enhance the optimization of the learnable probabilistic gate $a_i$, we introduce the re-parameterization method and architectural consistency regularization techniques. \n- Through experiments on a series of classification tasks, commonsense QA, information extraction tasks and medical report summarization tasks, we demonstrate that our SPT method performs well with different pretrained models, including the recently released LlaMA-2 models. Our method consistently achieves superior fine-tuning performance compared to LoRA, especially when working with limited training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Their asked question is meaningful to the community, as \"how do we find the optimal strategy of prompt injection, given the task at hand?\". In PETuning (parameter-efficient tuning), it would be interesting to ask and answer this question from a theoretical view, even though this work is more from an empirical view. \n2. For their designs, their optimization objective looks novel and interesting, combing and adapting existing work's key designs to their approach.\n3. The experiments and discussions look convincing, and especially, their conclusion of \"Transferability of the learned PG settings\" is very interesting to me. I am really looking forward to hearing from you about explaining this phenomenon!", "reasons_to_reject": "- Honestly, I do think such small modification is very incremental to the community (though still good questions), as you just focus on the simplest classification tasks. Right now, with very large language models, simple ICL (in-context learning) performance is already very strong. Therefore, it would be very good to show your advantages on other use cases w/ LMs or even other models.", "questions_to_authors": "See my questions in \"reasons to accept\", and \"reasons to reject\".", "paper_topic_and_main_contributions": "Existing parameter-efficient tuning framework, especially soft-prompt tuning, has not select the optimal prompt layer settings for downstream adaptations.This work discusses how to adaptively select the ideal prompt layers for task adaptions. Specifically, they inject a probabilistic gate per layer, and propose a bi-level optimization framework for their approach. Under few-shot and full-data settings, their framework does show competitive performance on prompting-based classification tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for learning to select prompt tuning layers. The authors formulate this problem as a bi-level optimization problem, and proposes an approach for learning gating functions which modulate the layer selection procedure. Experiments show that the proposed approach is able to outperform various baselines across standard datasets. During the discussion period, the authors have performed extensive additional experiments to address many of the reviewers' concerns. This paper is a solid contribution to the parameter-efficient learning literature.

[As a side note, some reviewers raised concerns that this paper may be outdated or irrelevant given the the ICL + LLM paradigm with closed source models. I do not share this viewpoint, and have not given much weight to these concerns in the decision process.]
Decision: Accept-Main