Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning
Year: 2023

Abstract:
Fine-tuning large pre-trained language models on various downstream tasks with whole parameters is prohibitively expensive. Hence, Parameter-efficient fine-tuning has attracted attention that only optimizes a few task-specific parameters with the frozen pre-trained model. In this work, we focus on prefix tuning, which only optimizes continuous prefix vectors (i.e. pseudo tokens) inserted into Transformer layers. Based on the observation that the learned syntax and semantics representation varies a lot at different layers, we argue that the adaptive prefix will be further tailored to each layer than the fixed one, enabling the fine-tuning more effective and efficient. Thus, we propose Adaptive Prefix Tuning (APT) to adjust the prefix in terms of both fine-grained token level and coarse-grained layer level with a gate mechanism. Experiments on the SuperGLUE and NER datasets show the effectiveness of APT. In addition, taking the gate as a probing, we validate the efficiency and effectiveness of the variable prefix.

heading: 1 Introduction
text: Vanilla fine-tuning strategy usually adjusts all the parameters to adapt the pre-trained language model to downstream tasks. Parameter-efficient learning (He et al., 2022; Houlsby et al., 2019; Lester et al., 2021; Guo et al., 2021; Ben Zaken et al., 2022) is an emerging framework that freezes the pre-trained model and only tunes a few number of task-specific parameters for downstream tasks. For instance, Prefix tuning (Li and Liang, 2021; Liu et al., 2022) prepends length-equivalent pseudo prefix tokens, i.e. continuous task-specific vectors to each layer of the pre-trained model, achieving comparable even superior performance with only 0.1-3% parameters.
In previous works, the length of prefix tokens (or the number of trainable parameters) is usually the same at each layer. However, a potential observation lies in that the structure information and
representational capacity embedded in each layer are prone to be inconsistent (Jawahar et al., 2019). It is generally considered that the bottom layers of the language model tend to capture concrete and shallow phrase-level features, while the top layers concerns more with abstract semantic information (Tenney et al., 2019). Based on the perspective, we assume adaptive prefix can grab the emphasis more flexibly to adapt to various downstream tasks.
In light of above motivation, we investigate the adaptive prefix in this work. We propose Adaptive Prefix Tuning (APT) with an adaptive gate mechanism at both fine-grained token level and coarsegrained layer level. Specifically, as shown in Figure 1, for fine granularity, APT scores each individual prefix token via gated weight assignment. Then, the scaled weight is utilized to balance the inserted task-specific prefix tokens and original input tokens for current layer at coarse-grained level.
Extensive experiments against prefix tuning on the sentence and token classification tasks in full data and low resources setting validate the effectiveness of APT. In addition, the gate learned from APT could be served as a probing for the number of necessary parameters in different layers, guiding us to directly apply variable prefix to the original prefix tuning. The probing experiment further demonstrates the effectiveness of adaptive prefix.
1239

heading: 2 Related Works
text: Since fine-tuning the whole model is prohibitively expensive, parameter-efficient language model finetuning becomes a lightweight alternative that only optimizes a small number of parameters while keeping most pre-trained parameters frozen (He et al., 2022). Adapter tuning (Houlsby et al., 2019) inserts two tunable task-specific modules after multihead attention and feed-forward network, achieving comparable performance with only 2-4% of the parameters. Prompt tuning (Lester et al., 2021) and Prefix-Tuning (Li and Liang, 2021) only train soft prompts by adding prefix tokens to the input or hidden states. Recently, Liu et al. (2022) extend the prefix tuning to the natural language understanding tasks, which matches the performance of fine-tuning with only 0.1%-3% tuned parameters.
Furthermore, with an overlap of our motivations that each layer of the pre-trained language model focuses on different aspects of feature for various tasks (Jawahar et al., 2019; Clark et al., 2019b) and extra parameters are probably not necessary for certain tasks (Houlsby et al., 2019; Fan et al., 2020; Rücklé et al., 2021), Adaptable Adapters (Moosavi et al., 2022) selects beneficial adapter layers and learns task-specific activation function for downstream tasks to make adaptor dynamic for each task and layer. In addition to different frameworks (adapter versa prefix tuning), our key difference from their work lies in that we aim to dynamically filter required information at each layer in a soft way, while they choose whether to add trainable modules at the layer level in a hard manner.

heading: 3 Methodology

heading: 3.1 Prefix Tuning
text: As prefix tuning is an extension on Transformer (Vaswani et al., 2017), we first recap the structure of Transformer. Transformer is the block consisting of multi-head attention concatenated by multiple single self-attention functions and a fully connected feed-forward network. Formally speaking, the Transformer block is calculated as follows:
Attn(Q,K,V ) = softmax( QKT√
d V ) (1)
FFN(x) = ReLU(xW1 + b1)W2 + b2 (2)
Prefix tuning prepends pseudo prefix tokens of length l to each layer of the language model, which is implemented by concatenating inserted
keys and values matrix with original corresponding items in each multi-head attention. Specifically, let Pk,Pv ∈ Rl×d be the keys and values of the engaged prefix separately, where l denotes the length of prefix and d corresponds to the dimension, thus self-attention function can be reformatted as:
Attn(Q,K ′,V ′) = softmax( Q(K ′)T√
d V ′) (3)
where K ′ = [Pk;K],V ′ = [Pv;V ]
Here, [; ] donates concatenation function.

heading: 3.2 Adaptive Prefix Tuning
text: The length of prefix is usually a manually set hyperparameter for each task and fixed in distinct layers of the model. However, existing work demonstrates each layer of the language model pays attention to different aspects of the input feature. We assume the prefix in fixed length is insufficient to tailor different layers and tasks. To dynamically customize the prefix at each layer, APT performs a gate mechanism via fine-grained gated weight assignment and coarse-grained scaled weight specification.
Specifically, to capture the diversity of information utilization at different layers, we go deep into the token level at the fine-grained granularity. The token-level gate can inspire us on how many trainable parameters (i.e. pseudo tokens in prefix tuning) are required for this layer, which will be discussed in Section 4.4. Thus, APT yields the gated weights of l pseudo tokens at each layer. We use the hidden states to represent the information encoded in the layer and calculate the gated weights αi = [αi1, αi2, . . . , αil] for i-th layer as:
αi = sigmoid(hi−1Wi) (4)
Here, hi−1 is the d-dimensional hidden states from the previous layer, and Wi ∈ Rd×l corresponds to the parameters to be learned.
Besides, we also design a coarse-level gate to balance the information brought from task-specific prefix tokens and original input tokens by learning a layer-level weight. A learnable scaled weight λi is added to the representation of pseudo prefix tokens at the i-th layer.
With the above strategy, the keys-values pair Pi = [Pik,Piv] derived from pseudo prefix tokens in i-th layer is updated to P̂i as:
P̂i = λiαi ⊙ [Pik,Piv] (5)
⊙ is the element-wise multiplication. Accordingly, the calculation of the self-attention function in APT is similar to Eq.(3) without further elaboration.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: We conduct 5 NLU tasks on SuperGLUE (Wang et al., 2019) benchmark including BoolQ (Clark et al., 2019a), COPA (Roemmele et al., 2011), RTE (Wang et al., 2018), WiC (Pilehvar and Camacho-Collados, 2019) and WSC (Levesque et al., 2012) as well as 3 Named Entity Recognition (NER) tasks including CoNLL03 (Tjong Kim Sang and De Meulder, 2003), CoNLL04 (Carreras and Màrquez, 2004), and OntoNotes 5.0 (Weischedel et al., 2013). With BERT-base / large (Devlin et al., 2019) and RoBERTa-large (Liu et al., 2019) instantiated by HuggingFace Transformers (Wolf et al.,
2020), we compare APT with vanilla fine-tuning and P-Tuning v2 (Liu et al., 2022) which is an implementation of the prefix tuning, configured with hyper-parameters public in the released code1. We also verify our method with DeBERTa-xlarge (He et al., 2020) on NER tasks following P-Tuning v2.

heading: 4.2 Results
text: We report the main results in Table 1. For BERTbase, we can observe that APT achieves 1.5% and 0.7% improvements over P-Tuning v2 on SuperGLUE and NER tasks, respectively. For BERTlarge, APT outperforms P-Tuning v2 by 1.8% on SuperGLUE tasks and 1.4% on NER tasks. For RoBERTa-large, APT surpasses P-Tuning v2 by 1.5% on SuperGLUE tasks and 0.2% on NER tasks. On NER tasks with DeBERTa-xlarge, APT is supe-
1https://github.com/THUDM/P-tuning-v2
rior to P-Tuning v2 by an average of 0.8%. Compared with vanilla fine-tuning, APT is comparable or even better on part of tasks. In addition, we explore the experimental performance under low resource settings on SuperGLUE benchmark. As shown in Table 2, APT is a better few-shot learner than P-Tuning v2, which exceeds 4.2%, 3.4% in 16-shot setting, and 2.9%, 3.6% in 32-shot setting for BERT-base and BERT-large, respectively.

heading: 4.3 Ablation Study
text: We conduct an ablation study in order to explore the separate effect of token-level gated weight α, layer-level scaled weight λ and the hidden states h from the previous layer which is used to calculate token-level gated weight α in Eq.(4). As shown in Table 3, it can be found that removing any strategy hurts the performance to varying degrees, demonstrating that they are all advantageous. Specifically, the beneficial effect of λ for APT is slightly greater than α overall. Besides, it is effective and meaningful to introduce the context (i.e. the hidden states h from the previous layer) when obtaining the gated weight, especially for SuperGLUE tasks.

heading: 4.4 Discussion
text: What is prefix weight distribution learned by APT? The gate mechanism for prefix serves as the key strategy of the proposed APT, where the learned prefix weight distribution turns out to be a critical point. Figure 2 illustrates the gate weights of the pseudo prefix token for COPA and CoNLL04,
respectively. It can be found that CoNLL04 is concerned with bottom layers in the language model which are regarded as phrase-level features, while COPA pays more attention to the higher layers, indicating semantic information. The observation is consistent with the characteristics of corresponding tasks. NER is a token-level task while COPA is a causal reasoning task sensitive to the semantics of sentences, which reminds us that it is worth placing various prefix tokens on specific layers according to the task properties.
Does variable prefix work better than fixed one? To verify the effectiveness of adaptive prefix under the proposed architecture, we wonder if the learned ratio at each layer can be directly transferred to P-Tuning v2. Taking the gate as a probing indicator, we reset the prefix length of P-Tuning v2 from fixed to variable in different layers based on the ob-
servation of the learned ratio (e.g. the distribution shown in Figure 2). From the comparison between PT-2 and PT-2∗ in Table 4, we demonstrate that the variable prefix with less trainable parameters surprisingly outperforms the original implementation in fixed prefix. Nonetheless, it is also worth noting that there is still a gap between P-Tuning v2 with variable prefix and APT, where the latter continuously adjusts the weight of prefix during the training phase while the former only initializes with a one-time mask probing.
Whether the adaptive structure benefits the finetuning? Compared to P-Tuning v2, APT learns extra gated and scaled weights. To figure it out whether the improvement of APT is brought from more trainable parameters or the adaptive model structure, we adjust the hyper-parameter, i.e., enlarge the prefix length of P-Tuning v2 by 1.5 times to align the number of parameters with our APT. As shown in the comparison between PT-2+ and APT of Table 4, we observe that APT still outperforms enlarged P-Tuning v2 with 1.9%, 0.4% on average for SuperGLUE and NER tasks respectively, validating the superiority of the gate mechanism.

heading: 5 Conclusion
text: In this paper, we investigate prefix tuning and assume that adaptive prefix is probably more efficient and effective than fixed prefix. Firstly, we propose APT that leverages the token-level and the layerlevel gate mechanism which achieves an improvement of performance over original prefix tuning. Then, we illustrate the weight distribution learned by APT and take it as a probe, which validates the variable prefix can work better than the fixed one. The above experiments and analysis demonstrate that the adaptive prefix can be served as a promising strategy for parameter-efficient fine-tuning.
Limitations
The proposed approach in this paper also suffers from certain limitations, i.e. we adapt APT on the encoder model and lack design for the other architectures such as decoder-only and encoder-decoder. In addition, it is better to generalize the key idea to other parameter-efficient learning approaches. A unified solution for existing work may be worth exploring in the future.

heading: A Experimental Details
text: Datasets In the full data setting, all train-dev-test splits follow P-Tuning v2 (Liu et al., 2022). For low resources setting, to generate k-shot (k = 16, 32) datasets on SuperGLUE, the fixed set of random seed [11,21,42,87,100] is utilized to sample instances in training and development set, while the entire development set is treated as test set, where the average performance is reported in Table 2.
Experimental Setting We grid search the learning rate over [5e-3, 7e-3, 1e-2, 1e-4], training epoch over [20, 40, 60, 80, 100, 120], batch size over [8, 16, 32], and random seeds over [11, 21, 42, 87, 100]. For a fair comparison, the prefix length utilized by APT is consistent with P-Tuning v2. In low resources setting, the batch size we used is 2. In Eq.(4), we take the hidden states of the first input token as representation in previous layer.
Experimental Computation We use the pretrained model BERT-base with 110M parameters, BERT-large with 335M parameters, RoBERTalarge with 355M parameters and DeBERTa-xlarge with 750M parameters. We conduct experiments on NVIDIA V100 or A100 GPUs for each task.

heading: B Further Ablation Results
text: We demonstrate further ablation results on BERTlarge and RoBERTa-large as shown in Table 5. It can be found that the beneficial impact of the three strategies and the observation is consistent with BERT-base in Section 4.3 in general.

heading: C Prefix Length
text: The prefix length is an important hyper-parameter for prefix tuning and APT. Figure 3 illustrates the performance of APT and P-Tuning v2 with different prefix lengths over a range. It can be observed that APT is superior to P-Tuning v2 in most prefix length settings, indicating that APT has a relatively wider range of prefix length to achieve better performance.

heading: D Scientific Artifacts
text: We use datasets involving SuperGLUE (Wang et al., 2019) benchmark including BoolQ (Clark et al., 2019a), COPA (Roemmele et al., 2011), RTE (Wang et al., 2018), WiC (Pilehvar and Camacho-Collados, 2019) and WSC (Levesque et al., 2012) as well as 3 Named Entity Recognition (NER) tasks including CoNLL03 (Tjong Kim Sang and De Meulder, 2003), CoNLL04 (Carreras and Màrquez, 2004), and OntoNotes 5.0 (Weischedel et al., 2013). The pre-trained model we used are BERT-base / large (Devlin et al., 2019), RoBERTalarge (Liu et al., 2019) and DeBERTa-xlarge (He et al., 2020). We use HuggingFace Transformers (Wolf et al., 2020) and P-Tuning v2 (Liu et al., 2022) as the codebase implemented by PyTorch 2. They are all open-source and we only use for academic research which is consistent with their intended use.
2https://pytorch.org/
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section abstract and section 1 introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 4.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: section D Scientific Artifacts

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section D Scientific Artifacts
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use open-source datasets and do not change datasets for a fair comparison.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. It can be found in the cited paper.
C 3 Did you run computational experiments? section 4 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Table 1 and section appendix A Experimental Computation
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section appendix A Experimental Details

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 2 report the mean and std results.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We follow the existing work and keep consistent with them.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: A unified solution for existing work may be worth exploring in the future