Towards a Unified View of Parameter-Efficient Transfer Learning

1 INTRODUCTION . Transfer learning from pre-trained language models ( PLMs ) is now the prevalent paradigm in natural language processing , yielding strong performance on many tasks ( Peters et al. , 2018 ; Devlin et al. , 2019 ; Qiu et al. , 2020 ) . The most common way to adapt general-purpose PLMs to downstream tasks is to fine-tune all the model parameters ( full fine-tuning ) . However , this results in a separate copy of fine-tuned model parameters for each task , which is prohibitively expensive when serving models that perform a large number of tasks . This issue is particularly salient with the ever-increasing size of PLMs , which now range from hundreds of millions ( Radford et al. , 2019 ; Lewis et al. , 2020 ) to hundreds of billions ( Brown et al. , 2020 ) or even trillions of parameters ( Fedus et al. , 2021 ) . To mitigate this issue , a few lightweight alternatives have been proposed to update only a small number of extra parameters while keeping most pretrained parameters frozen . For example , adapter tuning ( Houlsby et al. , 2019 ) inserts small neural modules called adapters to each layer of the pretrained network and only the adapters are trained at fine-tuning time . Inspired by the success of prompting methods that control PLMs through textual prompts ( Brown et al. , 2020 ; Liu et al. , 2021a ) , prefix tuning ( Li & Liang , 2021 ) and prompt tuning ( Lester et al. , 2021 ) prepend an additional l tunable ∗Equal Contribution . Order determined by random dice rolling . 1Code is available at https : //github.com/jxhe/unify-parameter-efficient-tuning . prefix tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks . More recently , Hu et al . ( 2021 ) learn low-rank matrices to approximate parameter updates . We illustrate these methods in Figure 1 . These approaches have all been reported to demonstrate comparable performance to full fine-tuning on different sets of tasks , often through updating less than 1 % of the original model parameters . Besides parameter savings , parameter-efficient tuning makes it possible to quickly adapt to new tasks without catastrophic forgetting ( Pfeiffer et al. , 2021 ) and often exhibits superior robustness in out-of-distribution evaluation ( Li & Liang , 2021 ) . However , we contend that the important ingredients that contribute to the success of these parameterefficient tuning methods are poorly understood , and the connections between them are still unclear . In this paper , we aim to answer three questions : ( 1 ) How are these methods connected ? ( 2 ) Do these methods share design elements that are essential for their effectiveness , and what are they ? ( 3 ) Can the effective ingredients of each method be transferred to others to yield more effective variants ? In order to answer these questions , we first derive an alternative form of prefix tuning that reveals prefix tuning ’ s close connections with adapters ( §3.1 ) . Based on this we then devise a unified framework that frames the aforementioned methods as different ways to modify the hidden representations of frozen PLMs ( §3.2 ) . Our unified framework decomposes previous methods along a shared set of design dimensions , such as the function used to perform the modification , the position in which to impose this modification , and how to integrate the modification . This framework allows us to transfer design choices across approaches to propose new variants such as adapters with multiple heads ( §3.3 ) . In experiments , we first show that existing parameter-efficient tuning methods still lag behind full fine-tuning on higher-resource and challenging tasks ( §4.2 ) , as exemplified in Figure 2 . Then we utilize the unified framework to identify critical design choices and validate the proposed variants empirically ( §4.3-4.6 ) . Our experiments on four NLP benchmarks covering text summarization , machine translation ( MT ) , text classification , and general language understanding , demonstrate that the proposed variant uses less parameters than existing methods while being more effective , matching full fine-tuning results on all four tasks . 2 PRELIMINARIES . 2.1 RECAP OF THE TRANSFORMER ARCHITECTURE . The transformer model ( Vaswani et al. , 2017 ) is now the workhorse architecture behind most stateof-the-art PLMs . In this section we recap the equations of this model for completeness . Transformer models are composed of L stacked blocks , where each block ( Figure 1 ) contains two types of sub- layers : multi-head self-attention and a fully connected feed-forward network ( FFN ) .2 The conventional attention function maps queries Q ∈ Rn×dk and key-value pairs K ∈ Rm×dk , V ∈ Rm×dv : Attn ( Q , K , V ) = softmax ( QKT√ dk ) V , ( 1 ) where n andm are the number of queries and key-value pairs respectively . Multi-head attention performs the attention function in parallel over Nh heads , where each head is separately parameterized by W ( i ) q , W ( i ) k , W ( i ) v ∈ Rd×dh to project inputs to queries , keys , and values . Given a sequence of m vectors C ∈ Rm×d over which we would like to perform attention and a query vector x ∈ Rd , multi-head attention ( MHA ) computes the output on each head and concatenates them:3 MHA ( C , x ) = Concat ( head1 , · · · , headh ) Wo , headi = Attn ( xW ( i ) q , CW ( i ) k , CW ( i ) v ) , ( 2 ) where Wo ∈ Rd×d . d is the model dimension , and in MHA dh is typically set to d/Nh to save parameters , which indicates that each attention head is operating on a lower-dimensional space . The other important sublayer is the fully connected feed-forward network ( FFN ) which consists of two linear transformations with a ReLU activation function in between : FFN ( x ) = ReLU ( xW1 + b1 ) W2 + b2 , ( 3 ) where W1 ∈ Rd×dm , W2 ∈ Rdm×d . Transformers typically use a large dm , e.g . dm = 4d . Finally , a residual connection is used followed by layer normalization ( Ba et al. , 2016 ) . 2.2 OVERVIEW OF PREVIOUS PARAMETER-EFFICIENT TUNING METHODS . Below and in Figure 1 , we introduce several state-of-the-art parameter-efficient tuning methods . Unless otherwise specified , they only tune the added parameters while the PLM ’ s are frozen . Adapters ( Houlsby et al. , 2019 ) : The adapter approach inserts small modules ( adapters ) between transformer layers . The adapter layer generally uses a down-projection with Wdown ∈ Rd×r to project the input h to a lower-dimensional space specified by bottleneck dimension r , followed by a nonlinear activation function f ( · ) , and a up-projection with Wup ∈ Rr×d . These adapters are surrounded by a residual connection , leading to a final form : h← h + f ( hWdown ) Wup . ( 4 ) Houlsby et al . ( 2019 ) places two adapters sequentially within one layer of the transformer , one after the multi-head attention and one after the FFN sub-layer . Pfeiffer et al . ( 2021 ) have proposed a more efficient adapter variant that is inserted only after the FFN “ add & layer norm ” sub-layer . Prefix Tuning ( Li & Liang , 2021 ) : Inspired by the success of textual prompting methods ( Liu et al. , 2021a ) , prefix tuning prepends l tunable prefix vectors to the keys and values of the multihead attention at every layer . Specifically , two sets of prefix vectors Pk , Pv ∈ Rl×d are concatenated with the original key K and value V . Then multi-head attention is performed on the new prefixed keys and values . The computation of headi in Eq . 2 becomes : headi = Attn ( xW ( i ) q , concat ( P ( i ) k , CW ( i ) k ) , concat ( P ( i ) v , CW ( i ) v ) ) , ( 5 ) Pk and Pv are split into Nh head vectors respectively and P ( i ) k , P ( i ) v ∈ Rl×d/Nh denote the i-th head vector . Prompt-tuning ( Lester et al. , 2021 ) simplifies prefix-tuning by only prepending to the input word embeddings in the first layer ; similar work also includes P-tuning ( Liu et al. , 2021b ) . LoRA ( Hu et al. , 2021 ) : LoRA injects trainable low-rank matrices into transformer layers to approximate the weight updates . For a pre-trained weight matrix W ∈ Rd×k , LoRA represents its update with a low-rank decomposition W+∆W = W+WdownWup , where Wdown ∈ Rd×r , Wup ∈ Rr×k are tunable parameters . LoRA applies this update to the query and value projection matrices ( Wq , Wv ) in the multi-head attention sub-layer , as shown in Figure 1 . For a specific input x to the linear projection in multi-head attention , LoRA modifies the projection output h as : h← h + s · xWdownWup , ( 6 ) 2In an encoder-decoder architecture , the transformer decoder usually has another multi-head cross-attention module between the self-attention and FFN , which we omit here for simplicity . 3Below , we sometimes ignore the head index i to simplify notation when there is no confusion . where s ≥ 1 is a tunable scalar hyperparameter.4 Others : Other parameter-efficient tuning methods include BitFit ( Ben Zaken et al. , 2021 ) , which only fine-tunes bias vectors in the pre-trained model , and diff-pruning ( Guo et al. , 2021 ) , which learns a sparse parameter update vector . 3 BRIDGING THE GAP – A UNIFIED VIEW . We first derive an equivalent form of prefix tuning to establish its connection with adapters . We then propose a unified framework for parameter-efficient tuning that includes several state-of-the-art methods as instantiations . 3.1 A CLOSER LOOK AT PREFIX TUNING . Eq . 5 describes the mechanism of prefix tuning which changes the attention module through prepending l learnable vectors to the original attention keys and values . Here , we derive an equivalent form of Eq . 5 and provide an alternative view of prefix tuning:5 head = Attn ( xWq , concat ( Pk , CWk ) , concat ( Pv , CWv ) ) = softmax ( xWqconcat ( Pk , CWk ) > ) [ Pv CWv ] = ( 1− λ ( x ) ) softmax ( xWqW > k C > ) CWv + λ ( x ) softmax ( xWqP > k ) Pv = ( 1− λ ( x ) ) Attn ( xWq , CWk , CWv ) ︸ ︷︷ ︸ standard attention +λ ( x ) Attn ( xWq , Pk , Pv ) ︸ ︷︷ ︸ independent of C , ( 7 ) where λ ( x ) is a scalar that represents the sum of normalized attention weights on the prefixes : λ ( x ) = ∑ i exp ( xWqP > k ) i∑ i exp ( xWqP > k ) i + ∑ j exp ( xWqW > k C > ) j . ( 8 ) Note that the first term in Eq . 7 , Attn ( xWq , CWk , CWv ) , is the original attention without prefixes , whereas the second term is a position-wise modification independent of C. Eq . 7 gives an alternative view of prefix tuning that essentially applies a position-wise modification to the original head attention output h through linear interpolation : h← ( 1− λ ( x ) ) h + λ ( x ) ∆h , ∆h : = softmax ( xWqP > k ) Pv . ( 9 ) The Connection with Adapters : We define W1=WqP > k , W2=Pv , f=softmax , and rewrite Eq . 9 : h← ( 1− λ ( x ) ) h + λ ( x ) f ( xW1 ) W2 , ( 10 ) which reaches a very similar form to the adapter function in Eq . 4 , except that prefix tuning is performing weighted addition while the adapter one is unweighted.6 Figure 3b demonstrates the 4The public code of LoRA at https : //github.com/microsoft/LoRA uses different s in different datasets , and we have verified the value of s could have a significant effect on the results . 5Without loss of generalization , we ignore the softmax scaling factor √ d for ease of notation . 6h in adapters and prefix tuning are usually different , as described more below . However , here we mainly discuss the functional form as adapters can , in principle , be inserted at any position . computation graph of prefix tuning from this view , which allows for abstraction of prefix tuning as a plug-in module like adapters . Further , we note that W1 ∈ Rdh×l and W2 ∈ Rl×dh are lowrank matrices when l is small , and thus they function similarly to the Wdown and Wup matrices in adapters . This view also suggests that the number of prefix vectors , l , plays a similar role to the bottleneck dimension r in adapters : they both represent the rank limitation of computing the modification vector ∆h . Thus we also refer l as the bottleneck dimension . Intuitively , the rank limitation implies that ∆h is a linear combination of the same l ( or ≤ l ) basis vectors for any x . The Difference from Adapters : In addition to the gating variable λ , we emphasize three differences between prefix tuning and adapters . ( 1 ) As demonstrated in Figure 3 , prefix tuning uses x , the input of the PLM layer , to compute ∆h , while adapters use h , the output of the PLM layer . Thus , prefix tuning can be thought of as a “ parallel ” computation to the PLM layer , whereas the typical adapter is “ sequential ” computation . ( 2 ) Adapters are more flexible with respect to where they are inserted than prefix tuning : adapters typically modify attention or FFN outputs , while prefix tuning only modifies the attention output of each head . Empirically , this makes a large difference as we will show in §4.4 . ( 3 ) Eq . 10 applies to each attention head , while adapters are always single-headed , which makes prefix tuning more expressive : head attention is of dimension d/Nh – basically we have full rank updates to each attention head if l ≥ d/Nh , but we only get full-rank updates to the whole attention output with adapters if r ≥ d. Notably , prefix tuning is not adding more parameters than adapters when l = r.7 We empirically validate such multi-head influence in §4.4 .

Summary: This paper investigates recent parameter-efficient methods that have been shown to achieve good performance in a variety of NLP tasks. The authors analyse their connections and propose a unified framework which subsumes a number of existing approaches. The authors then compare their performance across four NLP tasks, with varying level of difficulty and amount of available resources. Here, compared to full fine-tuning, they find that parameter-efficient approaches perform well on simpler tasks (MNLI and SST2), while showing larger gaps on more challenging tasks (XSum and MT). A series of controlled experiments centred around the proposed framework shows that parallel adapters applied to the FFN module of a Transformer generally perform better. Applying them to the attention module leads, however, to better gains when the number of additional parameters is small. Finally, the authors combine these findings into a novel MAM adapter module that leads to further gains in every task.