Towards a Unified View of Parameter-Efficient Transfer Learning

1 INTRODUCTION . Transfer learning from pre-trained language models ( PLMs ) is now the prevalent paradigm in natural language processing , yielding strong performance on many tasks ( Peters et al. , 2018 ; Devlin et al. , 2019 ; Qiu et al. , 2020 ) . The most common way to adapt general-purpose PLMs to downstream tasks is to fine-tune all the model parameters ( full fine-tuning ) . However , this results in a separate copy of fine-tuned model parameters for each task , which is prohibitively expensive when serving models that perform a large number of tasks . This issue is particularly salient with the ever-increasing size of PLMs , which now range from hundreds of millions ( Radford et al. , 2019 ; Lewis et al. , 2020 ) to hundreds of billions ( Brown et al. , 2020 ) or even trillions of parameters ( Fedus et al. , 2021 ) . To mitigate this issue , a few lightweight alternatives have been proposed to update only a small number of extra parameters while keeping most pretrained parameters frozen . For example , adapter tuning ( Houlsby et al. , 2019 ) inserts small neural modules called adapters to each layer of the pretrained network and only the adapters are trained at fine-tuning time . Inspired by the success of prompting methods that control PLMs through textual prompts ( Brown et al. , 2020 ; Liu et al. , 2021a ) , prefix tuning ( Li & Liang , 2021 ) and prompt tuning ( Lester et al. , 2021 ) prepend an additional l tunable ∗Equal Contribution . Order determined by random dice rolling . 1Code is available at https : //github.com/jxhe/unify-parameter-efficient-tuning . prefix tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks . More recently , Hu et al . ( 2021 ) learn low-rank matrices to approximate parameter updates . We illustrate these methods in Figure 1 . These approaches have all been reported to demonstrate comparable performance to full fine-tuning on different sets of tasks , often through updating less than 1 % of the original model parameters . Besides parameter savings , parameter-efficient tuning makes it possible to quickly adapt to new tasks without catastrophic forgetting ( Pfeiffer et al. , 2021 ) and often exhibits superior robustness in out-of-distribution evaluation ( Li & Liang , 2021 ) . However , we contend that the important ingredients that contribute to the success of these parameterefficient tuning methods are poorly understood , and the connections between them are still unclear . In this paper , we aim to answer three questions : ( 1 ) How are these methods connected ? ( 2 ) Do these methods share design elements that are essential for their effectiveness , and what are they ? ( 3 ) Can the effective ingredients of each method be transferred to others to yield more effective variants ? In order to answer these questions , we first derive an alternative form of prefix tuning that reveals prefix tuning ’ s close connections with adapters ( §3.1 ) . Based on this we then devise a unified framework that frames the aforementioned methods as different ways to modify the hidden representations of frozen PLMs ( §3.2 ) . Our unified framework decomposes previous methods along a shared set of design dimensions , such as the function used to perform the modification , the position in which to impose this modification , and how to integrate the modification . This framework allows us to transfer design choices across approaches to propose new variants such as adapters with multiple heads ( §3.3 ) . In experiments , we first show that existing parameter-efficient tuning methods still lag behind full fine-tuning on higher-resource and challenging tasks ( §4.2 ) , as exemplified in Figure 2 . Then we utilize the unified framework to identify critical design choices and validate the proposed variants empirically ( §4.3-4.6 ) . Our experiments on four NLP benchmarks covering text summarization , machine translation ( MT ) , text classification , and general language understanding , demonstrate that the proposed variant uses less parameters than existing methods while being more effective , matching full fine-tuning results on all four tasks . 2 PRELIMINARIES . 2.1 RECAP OF THE TRANSFORMER ARCHITECTURE . The transformer model ( Vaswani et al. , 2017 ) is now the workhorse architecture behind most stateof-the-art PLMs . In this section we recap the equations of this model for completeness . Transformer models are composed of L stacked blocks , where each block ( Figure 1 ) contains two types of sub- layers : multi-head self-attention and a fully connected feed-forward network ( FFN ) .2 The conventional attention function maps queries Q ∈ Rn×dk and key-value pairs K ∈ Rm×dk , V ∈ Rm×dv : Attn ( Q , K , V ) = softmax ( QKT√ dk ) V , ( 1 ) where n andm are the number of queries and key-value pairs respectively . Multi-head attention performs the attention function in parallel over Nh heads , where each head is separately parameterized by W ( i ) q , W ( i ) k , W ( i ) v ∈ Rd×dh to project inputs to queries , keys , and values . Given a sequence of m vectors C ∈ Rm×d over which we would like to perform attention and a query vector x ∈ Rd , multi-head attention ( MHA ) computes the output on each head and concatenates them:3 MHA ( C , x ) = Concat ( head1 , · · · , headh ) Wo , headi = Attn ( xW ( i ) q , CW ( i ) k , CW ( i ) v ) , ( 2 ) where Wo ∈ Rd×d . d is the model dimension , and in MHA dh is typically set to d/Nh to save parameters , which indicates that each attention head is operating on a lower-dimensional space . The other important sublayer is the fully connected feed-forward network ( FFN ) which consists of two linear transformations with a ReLU activation function in between : FFN ( x ) = ReLU ( xW1 + b1 ) W2 + b2 , ( 3 ) where W1 ∈ Rd×dm , W2 ∈ Rdm×d . Transformers typically use a large dm , e.g . dm = 4d . Finally , a residual connection is used followed by layer normalization ( Ba et al. , 2016 ) . 2.2 OVERVIEW OF PREVIOUS PARAMETER-EFFICIENT TUNING METHODS . Below and in Figure 1 , we introduce several state-of-the-art parameter-efficient tuning methods . Unless otherwise specified , they only tune the added parameters while the PLM ’ s are frozen . Adapters ( Houlsby et al. , 2019 ) : The adapter approach inserts small modules ( adapters ) between transformer layers . The adapter layer generally uses a down-projection with Wdown ∈ Rd×r to project the input h to a lower-dimensional space specified by bottleneck dimension r , followed by a nonlinear activation function f ( · ) , and a up-projection with Wup ∈ Rr×d . These adapters are surrounded by a residual connection , leading to a final form : h← h + f ( hWdown ) Wup . ( 4 ) Houlsby et al . ( 2019 ) places two adapters sequentially within one layer of the transformer , one after the multi-head attention and one after the FFN sub-layer . Pfeiffer et al . ( 2021 ) have proposed a more efficient adapter variant that is inserted only after the FFN “ add & layer norm ” sub-layer . Prefix Tuning ( Li & Liang , 2021 ) : Inspired by the success of textual prompting methods ( Liu et al. , 2021a ) , prefix tuning prepends l tunable prefix vectors to the keys and values of the multihead attention at every layer . Specifically , two sets of prefix vectors Pk , Pv ∈ Rl×d are concatenated with the original key K and value V . Then multi-head attention is performed on the new prefixed keys and values . The computation of headi in Eq . 2 becomes : headi = Attn ( xW ( i ) q , concat ( P ( i ) k , CW ( i ) k ) , concat ( P ( i ) v , CW ( i ) v ) ) , ( 5 ) Pk and Pv are split into Nh head vectors respectively and P ( i ) k , P ( i ) v ∈ Rl×d/Nh denote the i-th head vector . Prompt-tuning ( Lester et al. , 2021 ) simplifies prefix-tuning by only prepending to the input word embeddings in the first layer ; similar work also includes P-tuning ( Liu et al. , 2021b ) . LoRA ( Hu et al. , 2021 ) : LoRA injects trainable low-rank matrices into transformer layers to approximate the weight updates . For a pre-trained weight matrix W ∈ Rd×k , LoRA represents its update with a low-rank decomposition W+∆W = W+WdownWup , where Wdown ∈ Rd×r , Wup ∈ Rr×k are tunable parameters . LoRA applies this update to the query and value projection matrices ( Wq , Wv ) in the multi-head attention sub-layer , as shown in Figure 1 . For a specific input x to the linear projection in multi-head attention , LoRA modifies the projection output h as : h← h + s · xWdownWup , ( 6 ) 2In an encoder-decoder architecture , the transformer decoder usually has another multi-head cross-attention module between the self-attention and FFN , which we omit here for simplicity . 3Below , we sometimes ignore the head index i to simplify notation when there is no confusion . where s ≥ 1 is a tunable scalar hyperparameter.4 Others : Other parameter-efficient tuning methods include BitFit ( Ben Zaken et al. , 2021 ) , which only fine-tunes bias vectors in the pre-trained model , and diff-pruning ( Guo et al. , 2021 ) , which learns a sparse parameter update vector . 3 BRIDGING THE GAP – A UNIFIED VIEW . We first derive an equivalent form of prefix tuning to establish its connection with adapters . We then propose a unified framework for parameter-efficient tuning that includes several state-of-the-art methods as instantiations . 3.1 A CLOSER LOOK AT PREFIX TUNING . Eq . 5 describes the mechanism of prefix tuning which changes the attention module through prepending l learnable vectors to the original attention keys and values . Here , we derive an equivalent form of Eq . 5 and provide an alternative view of prefix tuning:5 head = Attn ( xWq , concat ( Pk , CWk ) , concat ( Pv , CWv ) ) = softmax ( xWqconcat ( Pk , CWk ) > ) [ Pv CWv ] = ( 1− λ ( x ) ) softmax ( xWqW > k C > ) CWv + λ ( x ) softmax ( xWqP > k ) Pv = ( 1− λ ( x ) ) Attn ( xWq , CWk , CWv ) ︸ ︷︷ ︸ standard attention +λ ( x ) Attn ( xWq , Pk , Pv ) ︸ ︷︷ ︸ independent of C , ( 7 ) where λ ( x ) is a scalar that represents the sum of normalized attention weights on the prefixes : λ ( x ) = ∑ i exp ( xWqP > k ) i∑ i exp ( xWqP > k ) i + ∑ j exp ( xWqW > k C > ) j . ( 8 ) Note that the first term in Eq . 7 , Attn ( xWq , CWk , CWv ) , is the original attention without prefixes , whereas the second term is a position-wise modification independent of C. Eq . 7 gives an alternative view of prefix tuning that essentially applies a position-wise modification to the original head attention output h through linear interpolation : h← ( 1− λ ( x ) ) h + λ ( x ) ∆h , ∆h : = softmax ( xWqP > k ) Pv . ( 9 ) The Connection with Adapters : We define W1=WqP > k , W2=Pv , f=softmax , and rewrite Eq . 9 : h← ( 1− λ ( x ) ) h + λ ( x ) f ( xW1 ) W2 , ( 10 ) which reaches a very similar form to the adapter function in Eq . 4 , except that prefix tuning is performing weighted addition while the adapter one is unweighted.6 Figure 3b demonstrates the 4The public code of LoRA at https : //github.com/microsoft/LoRA uses different s in different datasets , and we have verified the value of s could have a significant effect on the results . 5Without loss of generalization , we ignore the softmax scaling factor √ d for ease of notation . 6h in adapters and prefix tuning are usually different , as described more below . However , here we mainly discuss the functional form as adapters can , in principle , be inserted at any position . computation graph of prefix tuning from this view , which allows for abstraction of prefix tuning as a plug-in module like adapters . Further , we note that W1 ∈ Rdh×l and W2 ∈ Rl×dh are lowrank matrices when l is small , and thus they function similarly to the Wdown and Wup matrices in adapters . This view also suggests that the number of prefix vectors , l , plays a similar role to the bottleneck dimension r in adapters : they both represent the rank limitation of computing the modification vector ∆h . Thus we also refer l as the bottleneck dimension . Intuitively , the rank limitation implies that ∆h is a linear combination of the same l ( or ≤ l ) basis vectors for any x . The Difference from Adapters : In addition to the gating variable λ , we emphasize three differences between prefix tuning and adapters . ( 1 ) As demonstrated in Figure 3 , prefix tuning uses x , the input of the PLM layer , to compute ∆h , while adapters use h , the output of the PLM layer . Thus , prefix tuning can be thought of as a “ parallel ” computation to the PLM layer , whereas the typical adapter is “ sequential ” computation . ( 2 ) Adapters are more flexible with respect to where they are inserted than prefix tuning : adapters typically modify attention or FFN outputs , while prefix tuning only modifies the attention output of each head . Empirically , this makes a large difference as we will show in §4.4 . ( 3 ) Eq . 10 applies to each attention head , while adapters are always single-headed , which makes prefix tuning more expressive : head attention is of dimension d/Nh – basically we have full rank updates to each attention head if l ≥ d/Nh , but we only get full-rank updates to the whole attention output with adapters if r ≥ d. Notably , prefix tuning is not adding more parameters than adapters when l = r.7 We empirically validate such multi-head influence in §4.4 .

Summary: This paper formulates different approaches for parameter-efficient transfer learning (such as adapters, prefix-tuning, LoRA) under a common framework. Approaches vary alongside different design dimensions: functional form, insertion form (sequential or parallel), which representation is directly modified by the approach and the composition function of the main representation with the adaptation one.   The authors describe how existing approaches fit this framework. Then, they show how changing these design choices leads to new approaches, including parallel adapters (PA), multi-head PA and scaled PA.  The authors then compare their newly introduced methods and existing ones empirically. They find that: 1. Parallel insertion is generally better 2. Under a low parameter budget (~0.1% of original) it is preferable to modify the attention of the transformer. Under a higher budget, changing the feed-forward network is better. 3. Their methods generally perform on par or better than existing approaches.  Finally, the authors combine their insights to develop “Mix-and-match” adapters and show that it does well overall.