Towards a Unified View of Parameter-Efficient Transfer Learning
Authors: Junxian He
Chunting Zhou
Xuezhe Ma
Taylor Berg-Kirkpatrick
Graham Neubig
Venue: ICLR 2022 Spotlight (2022)
Keywords: parameter-efficient transfer learning
unified view
natural language processing

Abstract: Fine-tuning large pretrained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pretrained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pretrained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.

Published as a conference paper at ICLR 2022
TOWARDS A UNIFIED VIEW OF GARAMETER-EFFICEER TRANSFER LEARNING
Junxian He
Chunting Zhou'
Camegii Mellon University junxianh& cs cmu. edu
Carnegie Mellon University chuntinz@cs cmu edd.
Xuezhe Ma
Taylor Berg-Kirkpattick UC San Diego tbergeeng, uesd. edu
Graham Neubig
University of Southern California xuezhema edu
Carnegie Mellon University gneubig@cs cmu. edu
ABSTRACT
Fine-tuning large pretrained language models on downstream tasks has become the de-facto learning paradigm NLP. However, conventional approaches fine- unne all the parameters of the pretrained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a va- riety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. this paper, we break down the design of state-of-the-art parameter-efficiiet transfer learning methods and present unified framework that establishes connections between them. Specifically, we re-frame them as modifi- cations to specific hidden states in pretrained models, and define set of design dimensions along which different methods vary, such as the function to compute the modification and the position apply the modification. Through comprehen- sive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, utilize the unified view identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as result we are able to instantiate new parameter-efficiee fine-tuning meth- ods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks
INTRODUCTION
Transfer learning from pre-trained language models (PLMs) now the prevalent paradigm natural language processing, yielding strong performance many tasks (Peters et al. 2018 Devlin et al.) 2019 Qiu et al., 2020). The most common way to adapt general-purpose PLMs to downstream tasks to fine-tune all the model parameters (full fine-tuning). However, this results in a separate copy of fine-tuned model parameters for each task, which is prohibitively expensive when serving models that perform a large number of tasks. This issue is particularly salient with the ever-inereasing size of PLMs, which now range from hundreds of millions (Radford et al. Lewis et al.) 2020) to hundreds of billions (Brown et al. 2020) even trillions of parameters (Fedus et al. (021))
To mitigate this issue, a few lightweight alternatives have been proposed to update only a small num- ber of extra parameters while keeping most pretrained parameters frozen. For example, adapter tun- ing (Houlsby et al. (2019) inserts small neural modules called adapters to each layer of the pretrained network and only the adapters are trained at fine-tuning time. Inspired by the success of prompting methods that control PLMs through textual prompts (Brown et al. 2020 Liu et al.) 2021a), prefix
tuning (Li & Liang 202I) and prompt tuning (Lester et al. 2021 prepend an additional tunable
"Equal Contribution. Order determined by random dice rolling.
Code available at http:///gihh.oooommmmiaiiyaaaaae

---

Published as a conference paper at ICLR 2022
Adapter
Fine-tuning 21.94
Ours 21.90
Adapter Sigeeee
21
Adupler 20.98
W
Add Layeer Norm
ROUGE-2
Prefix Tuning 20.46
LoRA 20.50
20
Adaps:
Preftx Tuning
Attention
19
18
LoRA
BitFit 17.32
Hiddann States
WE
10 15
Fine-tuned Parameters (%)
Figure 1: Illustration of the transformer architecture and several statc-of-the-ar art parameter-officiet tuning methods. We use blocks with dashed borderlines to represent the added modules by those methods.
Figure 2: Performance of different methods the XSum (Narayan al. 2018)) summarization task. The number of fine-tuned parameters is relative the tuned parameters in full fine-tuning.
prefix tokens to the input or hidden layers and only train these soft prompts when fine-tuning on downstream tasks. More recently, Hu et al. (2021) learn low-rank matrices to approximate param- eter updates. illustrate these methods Figure [ These approaches have been reported to demonstrate comparable performance to full fine-tuning different sets of tasks, often through up- dating less than 1% of the original model parameters. Besides parameter savings, parameter- efficien wning makes it possible quickly adapt new tasks without catastrophic forgetting (Preiffer al., 2021 and often exhibits superior robustness duiti--dist evaluation (Li & Liang 2021
However, contend that the important ingredients that contribute to the success of these parameter- efficient tuning methods are poorly understood, and the connections between them are still unclear. In this paper, we aim to answer three questions: (I) How these methods connected? (2) Do these methods share design elements that are essential for their effectiveness, and what are they? (3) Can the effective ingredients of each method be transferred others yield more effective variants?
In order to answer these questions, we first derive an alternative form of prefix tuning that reveals prefix tuning's close connections with adapters ($3. Based on this we then devise unified frame- work that frames the aforementioned methods as different ways to modify the hidden representations of frozen PLMs ($3.2). Our unified framework decomposes previous methods along shared set of design dimensions, such as the function used to perform the modification, the position in which to impose this modification, and how to integrate the modification. This framework allows us to transfer design choices across approaches to propose heads (33.)). In experiments, we first show that existing parameter-efficien tuning methods still lag behind full fine-tuning higher-resource and challenging tasks ($4.2). as exemplified in Fig- ure 2 Then we utilize the unified framework to identify critical design choices and validate the proposed variants empirically ($4.3 4.6). Our experiments four NLP benchmarks covering text summarization, machine translation (MT), text classification, and general language understanding, demonstrate that the proposed variant less parameters than existing methods while being more effective, matching full fine-tuning results on all four tasks.
variants such adapters with multiple
2 PRELIMINARIES
2.1 RECAP OF THE TRANSFORMER ARCHITECTURE
The transformer model (aswwan et al. 2017 is now the workhorse architecture behind most state- of-the-art PLMs. In this section we recap the equations of this model for completeness. Transformer models composed of I. stacked blocks, where each block (Figure contains types of sub-
2

---

Published as a conference paper at ICLR 2022
layers: multi-head self-attention and fully connected feed-forward network (FFN The conven- tional attention function maps queries Q E Rnxdx and key-value pairs K Rmnda
E mxd,
Attn Q, K, softmax QKT
V
VTk
where n and the number of queries and key-value pairs respectively. Multi-head attention per- forms the attention function in parallel over N₁, heads, where each head is separately parameterized
by W W w E Rdxds to project inputs to queries, keys, and values. Given sequence of m vectors C' RRxd over which we would like perform attention and query vector Rd, multi-head attention (MHA) computes the output on each head and concatenates them 5)
MHA(C, x) Concat(head,,, head,) ) Wo: head; = Attn(a CW CW."). (2)
where Wa E Rdxdd d is the model dimension, and in MHA d, is typically set to d/N, to save parameters, which indicates that each attention head is operating on lower-dimensional space. The other important sublayer is the fully connected feed-forward network (FFN) which consists of two linear transformations with a ReLU activation function in between:
FFN(x) RcLU(>W; + b1) W2 + b2.
(3)
where W1 E _dddd, W2 E Rd xd Transformers typically use a large dm e.g. dm 4d. Finally, residual connection is used followed by layer normalization (Ba et al., 2016).
2.2 OVERVIEW OF PREVIOUS CHAAMETER-EFFIIESS TUNING METHODS
Below and in Figure I we introduce several state-of-the-att parameter-efficient tuning methods.
Unless otherwise specified, they only tune the added parameters while the PLM's are frozen.
Adapters (Houlsby et al. 2019): The adapter approach inserts small modules (adapters) between transformer layers. The adapter layer generally uses down- poon-proecction with Waowa E R to project the input h to a lower-dimensional space specified by bottleneck dimension r, followed by nonlinear activation function f()), and up-projection with W up Rrxd These adapters surrounded by residual connection, leading to a final form:
h - h + fhh down) W.p
(4)
Houlsby et al. (019)) places two adapters sequentially within one layer of the transformer, one after the multi-head attention and one after the FFN sub-layer. Pfeiffer et al. 2021) have proposed a more efficient adapter variant that is inserted only after the FFN "add & layer norm" sub-layer.
Prefix Tuning (Li & Liang, 2021): Inspired by the success of textual prompting methods Liu et al. 2021a), prefix tuning prepends / tunable prefix vectors to the keys and values of the multi- head attention at every layer. Specifically, two sets of prefix vectors Pa. P. E Rixd are concatenated with the original key K and value V. Then multi-head attention performed on the new prefixed keys and values. The computation of head, in Eq. 2 becomes:
head; = Attn x ( concat p(() (" concat( (() CW.
(5)
P& and P. are split into N₂, head vectors respectively and P P! R/xd/N, denote the i-th head vector. Prompt-tuning (Lester et al.) (021) simplifies prefix-tuning by only prepending to the
input word embeddings in the first layer; similar work also includes P-tuning (Liu et al. 2021b).
LoRA (Hu et al.) 2021]: LoRA injects trainable low-rank matrices into transformer layers to approximate the weight updates. For a pre-trained weight matrix W Bddk LoRA represents its update with a low-rank decomposition W W+AW- = W W+W Waomn Wap- where Wdown E Raxr, W grxk tunable parameters. LoRA applies this update to the query and value projection matrices (W,, W,) in the multi-head attention sub-layer, as shown in Figure I For a specific input æ to the linear projection in multi-head attention, LoRA modifies the projection output h
h
s W down Wap+
II an encoder-decoder architecture, the transformer decoder usually has another multi-head cross-aties attennit module between the self-attention and FFN, which we omit here for simplicity. Below, we sometimes ignore the head index simplify notation when there is confusion.
3

---

Published as a conference paper at ICLR 2022
OO Q
1O000
MO O O
MOOO
10000
O
#0 $0000
OOO
(a) Adapter
(b) Prefix Tuning
(c) LoRA
(d) Parallel Adapter
(c) Scaled
Figure 3: Graphical illustration existing methods and proposed variants. "PLM module" represents certain sublayer of the PLM (c.g. attention or FFN) that is frozen. "Scaled PA" denotes scaled parallel adapter. We do not include multi-head parallel adapter here to save space.
where is a tunable scalar hyperparameter
Others: Other parameter-efficien tuning methods include BitFiit (Ben Zaken et al. 2021 only fine-tunes bias vectors in the pre-trained model, and diff- pfffiirunngg (Guo 2021
which
which
learns sparse parameter update vector.
3 BRIDGING THE GAP A UNIFIED VIEW
We first derive an equivalent form of prefix tuning to establish its connection with adapters. We then propose unified framework for parameter-efficiet tuning that includes several state-of-the-an methods as instantiations.
3.1 A CLOSER LOOK AT PREFIX TUNING
Eq. 5 describes the mechanism of prefix tuning which changes the attention module through prepending learnable vectors to the original attention keys and values. Here, we derive equiva- lent form Eq. 5 provide an alternative view of prefix tuning 5
head = Att q: concat( Pk: CW&),, concat P... CW.))
P.
softmax (a W W, concat Pk, C)
CW
(7)
(1 ((a))softmax(:WWWW W C. CWW, + ((z))softmax(( (rWW )
= (1 A(x)) Attn( CW&, CW.) +X(x) Aun(a: 9, Pk, P.).
standard attention
independent of C
where X(x:) is scalar that represents the sum of normalized attention weights the prefixes:
E, exp(xw,,P
X(x) Σ, exp(xW P. E, exp CT);
Note that the first term in Eq. Aut((a CW, CW.). is the original attention without prefixes, whereas the second term is position-wise modification independent of C. Eq. gives alterna- tive view of prefix tuning that essentially applies a position-wise modification to the original head attention output through linear interpolation:
h (1 X(x)) Akx)A,
Ah softmax WW, P..
The Connection with Adapters: We define W Wo=P,, f= f=softmax, and rewrite Eq 9 (1 - 1-)x(())) + (=)
(10)
which reaches a very similar form to the adapter function in Eq. performing weighted addition while the adapter one is unweighted
except that prefix tuning is
Figure 3b demonstrates the
uses different in different datasets, and we have verified the value of could have significant effect on the results.
'The public code LoRA at httpp::/githhb..com http://iwwwi.cocooooo/oo/o/////t
'Without loss of generalization, we ignore the softmax scaling factor /d ease notation.
'h in adapters and prefix tuning usually different, as deseribed more helow. However, here mainly discuss the functional form adapters can, in principle, be inserted any position.
4

---

Published as a conference paper at ICLR 2022
Table 1: Parameter-efficient tuning methods decomposed along the defined design dimensions. Here, for clarity, we directly write the adapter nonlinear function as ReLU which is commonly used. The bottom part of the table exemplifies new variants by transferring design choices of existing approaches.
Method
Ah functional form
insertion form
modified representation
composition function
Existing Methods parallel sequential
Prefix Tuning
softmax(ar W P.P..
&
head atin ffi/aatn
h (1 Ajh A&h h h+ Aht
Adapter
ReLU W down W
LoRA
W: Wap
parallel
atin key/val
h + h + Ah
Proposed Variants
Parallel adapter
ReLU (WW Wame W
parallel
fin/attn head attn ffn//tttn
h - h Ah - + Ah
Muti-heal parallel adapter Sealed parallel adapter
ReLU (h WWWaaa down W
paralicl
ReLU Lowa Wap
paraliel
h Ah
computation graph of prefix tuning from this view, which allows for abstraction of prefix tuning a plug-in module like adapters. Further, note that W1 E Rd xi and W2 E Rxxds are low- rank matrices when / is small, and thus they function similarly to the down and WTp matrices in adapters. This view also suggests that the number of prefix vectors, I, plays similar role to the bottleneck dimension r in adapters: they both represent the rank limitation of computing the modification vector Ah. Thus also refer I as the bottleneck dimension. Intuitively, the rank limitation implies that Ah is a linear combination of the same (or S 1) basis vectors for any æ.
The Difference from Adapters: In addition to the gating variable X, we emphasize three differ- ences between prefix tuning and adapters. (1) As demonstrated in Figure prefix tuning uses æ. the input of the PLM layer. to compute Ah, while adapters use h. the output of the PLM layer. Thus, prefix tuning can be thought of as "parallel" computation to the PLM layer, whereas the typical adapter is "sequential" computation. (2) Adapters are more flexible with respect to where they are inserted than prefix tuning: adapters typically modify attention or FFN outputs, while prefix tuning only modifies the attention output each head. Empirically, this makes large difference as we will show in $4.4. (3) Eq. 10 applies to each attention head, while adapters are always single-headed. which makes prefix tuning more expressive: head attention is of dimension d/N; basically we have full rank updates each attention head if / 2 d/N,, but we only get full-rank updates to the whole attention output with adapters if r 2 d. Notably, prefix tuning is not adding more parameters We empirically validate such multi-head influence in $4.4
than adapters when 1 r
3.2 THE UNIFIED FRAMEWORK
Inspired by the connections between prefix tuning and adapters, we propose a general framework that aims to unify several state-of-the-art parameter-efficient tuning methods. Specifically, we cast them as learning a modification vector Ah, which is applied to various hidden representations. Formally, we denote the hidden representation to be directly modified as h, and the direct input to the PLM sub-module that computes h as æ (e.g. and æ can be the attention output and input respectively). To characterize this modification process, we define set of design dimensions, and different methods can be instantiated by varying values along these dimensions. We detail the design dimensions below, and illustrate how adapters, prefix tuning, and LoRA fall along them in Table [
Functional Form is the specific function that computes Ah. We have detailed the functional form for adapters, prefix tuning, and LoRA in Eq. these methods are similar with proj.dows + nonlinear + proj_up architecture, while "nonlinear" degenerates the identity function LoRA.
4 6 and 10 respectively. The functional forms of all
K Insertion Form is how the added module is inserted into the network. As mentioned in the previous section and shown in Figure 3, traditionally adapters are inserted at a position in a sequential manner, where both the input and output are h. Prefix tuning and LoRA- although not originally described in this way- turn out to be equivalent to a parallel insertion where x is the input.
Modified Representation indicates which hidden representation is directly modified
We will detail 44. the number parameters added different methods. SStrictly speaking, all the hidden representations would be indirectly influenced by modifying the ones hefore them. Here we refer the position being directly modified by the added module.
5

---

Published as a conference paper at ICLR 2022
Composition Function is how the modified vector Ah is composed with the original hidden repre- sentation h to form the new hidden representation. For example, adapters perform simple additive composition, prefix tuning uses gated additive composition shown in Eq. T0] and LoRA scales Ah by a constant factor and adds it to the original hidden representation as in Eq.
We note that many other methods not present in Table I fit into this framework well. For example, prompt tuning modifies the head attention in the first layer in a way similar to prefix tuning, and various adapter variants (Pfeiffer al. 2021 Mahabadi et al. 2021) can be represented in a similar way as adapters. Critically, the unified framework allows us to study parameter-efficient tuning methods along these design dimensions, identify the critical design choices, and potentially transfer design elements across approaches, as in the following section.
3.3 TRANSFERRING DESIGN ELEMENTS
Here, and in Figure 3 we describe just few novel methods that can be derived through our uni- fied view above by transferring design elements across methods: (1) Parallel Adapter is the variant by transferring the parallel insertion of prefix tuning into adapters. Interestingly, while we moti- vate the parallel adapter due to its similarity to prefix tuning, concurrent work (Zhu et al. 202T) independently proposed this variant and studied it empirically; (2) Multi-head Parallel Adapter a further step to make adapters more similar to prefix tuning: we apply parallel adapters to modify head attention outputs prefix tuning. This way the variant improves the capacity for free by uti- lizing the multi-head projections as we diseuss in $3.1 (3) Scaled Parallel Adapter is the variant by transferring the composition and insertion form of LoRA into adapters, as shown in Figure 3e
Our discussion and formulation so far raise a few questions: Do methods varying the design elements above exhibit distinct properties? Which design dimensions are particularly important? Do the novel methods described above yield better performance? We answer these questions next.
4
EXPERIMENTS
4.1 GENERAL SETUP
Datasets: We study four downstream tasks: (1) XSum (Narayan et al.] 2018)) is an English sum- marization dataset where models predict summary given news article; (2) English to Romanian translation using the WMT 2016 en-ro dataset (Bojar et al. 2016); (3) MNLI (Williams et al.] 2018)) an English natural language inference dataset where models predict whether one sentence entails, contradicts, or is neutral to another. (4) SST2 (Socher et al. 2013) is an English sentiment classifi- cation benchmark where models predict whether sentence's sentiment is positive negative.
Setup: We use BARTTARGE (Lewis et al. 2020) and a multilingual version of it, mBARTTARGE
al. 2020a), as the underlying pretrained models for XSum and en-ro translation respectively, and we use RoBERTapace (Liu et al. 2019) for MNLI and SST2. We vary the bottleneck dimension within {1, 30, 200, 512, 1024] if needed We mainly study adapters, prefix tuning (prefix), and LoRA which greatly outperform bitfit and prompt tuning in our experiments. In the analysis sections ($4.3 we insert adapters either the attention or FFN layers for easier analysis, but include the results of inserting at both places in the final comparison (4..)) We re-implement these methods
based on their respective public code I0 We use the huggingface transformers library (Wolf et al.,
2020 for our implementation. Complete setup details can be found in Appendix A
Evaluation: We report ROUGE 1/2/L scores (R-1/2/L, Lin 20044) on the XSum test set, BLEU scores (Papineni et al. 2002) the en-ro test set, and accuracy the MNLI and SST2 dev set. For MNLI and SST2, we take the median of five random runs. We also report the number of tuned parameters relative to that in full fine-tuning (#params).
Number of Tunable Parameters: BART and mBART have an encoder decoder structure that has three types of attention: encoder self-attention, decoder self-attention, and decoder cross-attention. RoBERTa only has encoder self-attention. For each attention sub-layer, the number of parameters
II some settings we use other values to match the number added parameters of different methods. We verify that our re-implementation can reproduce adapter and prefix tuning XSum, and LoRA A on MNLI, by comparing with the results of running the original released code.

---

Published as a conference paper at ICLR 2022
21.91
Table 2: Accuracy the dev set of
20.98
22
MNLI and SST2. MAM Adapter proposed in $4.6
SSan ROUGEE
14-
Bittit numbers are 2021
02
from Ben Zaken et al.
LoRA
10-
300
Adapter
Method (# params)
MNLI SST2
28
PrefieTuning
Full-FT (100%)
87.6,4
94.6
18-
Birfit (0.1 %)
84.7 93.7
26
Fine-tuning
Prefix (0.5%)
86.34. 94.0=11
10
10
15
LoRA (0.5%)
87.2+4 94.2+
Fine-tuned Parameters (%)
Fine-tuned Parameters (%)
Adapter (0.5%)
87.2 94.2
Figure 4: Performance of previous state-of-the-art parameter- efficient tuning methods XSum (left) and en-ro (right).
MAM Adapter (0.5%) 87.4.., 94.2
Table 3:
Comparison of different insertion forms for adapters, Le. sequential adapter (SA) and parallel adapter (PA). We in- clude the results of prefix tuning as a reference point.
Table 4: Results on en-ro dataset.
Method
# params MT (BLEU)
PA (=tt
==200
3.6%
35.6
Method
params
XSum 1/2/L MT (BLEU)
Prefix, /=200
3.6%
35.6
MH PA (uttm). r=200
3.6%
35.8
Prefix, 1=200
3.6%
43.40/20.46/35.5
35.6
SA (atin). r=200
3.6%
42.01/19.30/34
35.3
Prefix, /=30
0.1%
35.2
SA (ffn), r=200
2.4%
43.21/19. 43.21/19.98/35.08
98/35.08
35.6
-gating. (=30
0.1%
34.9
PA (ffn), r=30
0.1%
33.0
PA (attn), r=200
3.6%
43.58/20.31 31//20031/35.44
35.6
PA (attn), r=30
0.1%
33.7
PA (ffi), r=200
2.4%
43.93/20.66/3566
36.4
MH PA (attn). r=
0.1%
35,3
used of each method is: (1) prefix tuning prepends / vectors to the keys and values and uses 2 x / x d parameters; (2) adapter has W down and W up thus uses 2 x d parameters; (3) LoRA employs a pair of W Wdowa and W up for query and value projections, hence uses 4 x x d parameters. For the adapter modification fin, it uses x r x parameters which is the same as adapter at attention. Therefore, for a specific value of or I, prefix tuning uses the same number of parameters as adapters, while LoRA uses more parameters. More details can be found Appendix
4.2 THE RESULTS OF EXISTING METHODS
We first overview the results of existing methods on the four tasks. As shown in Figure Table while existing methods can achieve competitive performance on MNLI and SST2 by tuning fewer than 1% parameters, a large gap is still present if add 5% parameters in XSum and en-ro. The gap remains significant even though we increase the relative parameter size to > 10%. Even larger gaps have been observed in Raffel et al. (2020) on high-resource MT tasks. This shows that many methods that claimed comparable results to full fine-tuning on the GLUE benchmark with an encoder-only model (Guo al. 2021] Ben Zaken et al.) [021] Mahabadii et al., (021), or relatively simple generation benchmarks such as E2E (Novikova et al. 2017) with an encoder-decoder model (Li & Liang 202T. may not generalize well to other standard benchmarks. The influencing factors could be complicated including the number of training samples, task complexity, or model architecture. We thus advocate for future research on this line to report results on more diverse benchmarks to exhibit a more complete picture of their performance profile. Below, our analysis will mainly focus on the XSum and en-ro datasets to better distinguish different design choices. We note that these two benchmarks are relatively high-resource performed with encoder-decoder model (BART), while we will discuss the results on MNLI and SST2 with an encoder-only model (RoBERTa) in $4.6
4.3 WHICH INSERTION FORM SEQUENTIAL OR PARALLEI.?
We first study the insertion form design dimension, comparing the proposed parallel adapter (PA) variant to the conventional sequential adapter (SA) over both the attention (att) and FFN modifica- tion. We also include prefix tuning reference point. As shown in Table 3 prefix tuning, which uses parallel insertion, outperforms attention sequential adapters. Further, the parallel adapter is able to beat sequential adapters in all cases with PA (ffn) outperforming SA (ffn) by 1.7 R-2 points on
More results with different be found in Appendix which exhibits similar observations.
7

---

Published as a conference paper at ICLR 2022
21.25
37.0
XSum ROUGED
21.00
MT BLEU
36.5
Frefxx (2tin)
20.76
36.0
(attr)
LoRA (sitn)
20.50
35.5
(ffe)
LoRA
20.25
35.0
2.5
5.0
7.5
10.0
12.5
2.5
5.0
7.5
10.0
12.5
Fine-tuned Parameters (%)
Fine-tuned Parameters (%)
Figure 5: Results XSum (left) and en-ro (right). PA represents parallel adapter. Blue and red markers apply modifications at attention and FFN sub-layers respectively (hest viewed in color).
XSum and 0.8 BLEU points en-ro respectively. Given the superior results of parallel adapters over sequential adapters, we focus on parallel adapter results in following sections.
4.4 WHICH MODIFIED REPRESENTATION ATTENTION OR FFN?
Sctup: We now study the effect of modifying different representations. We mainly compare at- tention and FFN modification. For easier analysis we categorize methods that modifies any hidden representations in the attention sub-layer (e.g. the head output, query, ett) as modifying the atten- tion module. We compare parallel adapters at attention and FFN and prefix tuning. We also transfer the FFN modification to LoRA to have LoRA (ffn) variant for complete comparison. Specifi- cally, we use LoRA to approximate the parameter updates for the FFN weights W1 E Rd and W₂ E Rd xd In this case WTp up in LoRA for W (similar for down of W2) would have dimensions of r x dm. where dm = 4d as described in 2 Thus typically use smaller r for LoRA (ffn) than other methods to match their overall parameter size in later experiments.
Results: As shown in Figure 5 any method with FFN modification outperforms all the methods with attention modification in all cases (the red markers are generally above all the blue ones, the only exception is ffn-PA with 2.4% params), often with fewer parameters. Second, the same method applied at FFN always improves over its attention counterpart. For example, LoR A (ffn) improves LoRA (attn) by R-2 points on XSum. We also highlight that prefix tuning does not keep improving when we further increase the capacity, which is also observed in Li & Liang (202I). These results suggest that FFN modification can utilize the added parameters more effectively than attention, no matter what the functional form or composition function is. We hypothesize that this is because the FFN learns task-specific textual patterns (Geva al.) (021), while attention learns pairwise positional interactions which do not require large capacity for adapting to new tasks.
Is the story different when we use 0.1% parameters? In 3.1 we reason that prefix tuning
more expressive than adapters (attn), which, however, is not reflected in Figure 5 We conjecture that this is because multi-head attention is only superior when the parameter budget is small. To validate this hypothesis, we compare prefix tuning to parallel adapters when they add 0.1% of the pretrained parameters. To ablate the impact of the composition function, we also report the results of removing the gating in prefix tuning h + Ah. We include the results of the multi-hean parallel adapter variant (MH PA) described in $3.3 As shown in Table 4 the multi-head methods prefix tuning and MH PA (attn) outperform all others by at least 1.6 BLEU points when using 0.1% of the parameters. Surprisingly. reducing / from 200 to 30 only causes 0.4 BLEU loss for prefix tuning while PA (attn) loses 1.9 points. The gating composition function in prefix tuning slightly helps the results by 0.3 points. We highlight that the MH parallel adapter improves the single-headed version by 1.6 points, which again verifies the effectiveness of the multi- head formulation.
Combining the results in Figure FigureBland and Table 4 we conclude that modifying head attention shows the best results when the parameter budget very small, while the FFN can better utilize modifications at larger capacities. This suggests that it may be effective to allocate larger parameter budget to FFN modification instead of treating attention and FFN equally as Houlshy al. (0019).
4.5 WHICH COMPOSITION FUNCTION?
We have presented three composition functions in $32 simple addition (adapter), gated addition (prefix tuning) and scaled addition (LoRA). As it is unnatural to incorporate the exact gated ad- dition into methods whose functional form does not softmax, we examine the other two by
8

---

Published as a conference paper at ICLR 2022
Table 6: Comparison of various parameter-efficieee tuning methods and the proposed variants. "f" results copied from Lewis et al. (2200 and Liu al. (2020b). We could not reproduce exactly the same full fine- tuning numbers with the same hyperparameters or even searching them. The reason may be the different libraries which the training code is based on - full fine-tuning is very sensitive to training hyperparameters. For the most performant methods we run with random seeds and report mean and standard deviation.
Method
# params
XSum (R-1/22 1/2/L)
MT (BLEU)
Full fine-tuning!
100%
45.14/22.27/37..22
37.7
Full fine-tuning (our run)
100%
44.81/21 944336883
37.3
Bitfit (Ben Zaken et al. 2021
0.1%
40.64/17.32/5
26.4
Prompt tuning (Lester al
2021
0.1%
38,91/15. 98/30.83
1.991/15.8/30.88
21.0
Prefix tuning (Li & Liang, 2021
200
3.6%
43.40/20.46/33
35.6
Pfeiffer adapter (Preitter et al. 2021
r=000
7.2%
44.03/20.89/
011220889935.89..1113 13
36.91.1
LoRA (ffn), r=102
7.2%
14.533/11.9/33.222
36.8
Parallel adapter (PA, ftin. 1=20
12.3%
37.2+
PA (attn, r=30) + PA (ffn, 512)
6.7%
37.2
Prefix tuning (alln, 3=3 + LoRA (fin, 102)
6.7%
44.84221711/66.7722m010
37.0+
MAM Adapter (our variant, /=30, r=512)
6.7%
45.06/21..9//6..7....o
37.5
ablating LoRA and comparing with the proposed scaled parallel adapter (Scaled PA), we con- strain modified representation to be FFN since is generally more effective as shown in $4.4
Treporrs the results XSum. We set 512 for adapters and 102 for LoRA so that their tuned parameter sizes the same. We select based on the R-2 score on the dev set. We observe that LoRA (s 4) performs better than parallel adapter. However, the advantage disappears if we remove the scaling by setting s
Table 5: Results on XSum when using different composition functions. The modified representa- tion is FFN. The bottleneck dimension 512 for (Scaled) PA and 102 for LoRA.
Method (# params)
XSum (R-1/2/LSum)
Through plugging the composition function of LoRA into parallel adapter, the resulted Scaled PA improves the vanilla parallel adapter by 0.56 ROUGE-2 points. We also experi- ment with a Iearned scalar which does not give bet- ter results. Therefore, we conclude that the scaling composition funnctio better than the vanilla additive one while being easily applicable.
LoRA (6.1%). LoRA (6. (6)1%), s=1 PA (6.1%)
14.59/21.31/36
44.17/20.83//
44.35/20.98.. 017220098/35.00
Scaled PA (6.1%)), s=4
44.85/21.54/36.58
Scaled PA (6.1%) trainable
44.56/21.31/36.29
4.6 AN EFFECTIVE INTEGRATION BY TRANSFERRING FAVORABLE DESIGN ELEMENTS
We first highlight three findings in previous sections: (1) Scaled parallel adapter is the best variant to modify FFN; (2) FFN can better utilize modification at larger capacities; and (3) modifying head attentions like prefix tuning can achieve strong performance with only 0.1% parameters. Inspired by them, we mix and match the favorable designs behind these findings: specifically, we use prefix tuning with small bottleneck dimension parameter budgets to modify FFN representation using the scaled parallel adapter (r = 512). Since prefix tuning can be viewed form of adapter in our unified framework, we name this variant as Mix-And-Mari adapter (MAM Adapter). In Table D we compare MAM adapter with various caaumetrr-fffcienn tuning methods. For completeness, we also present results of other combination versions in Table 6 using parallel adapters at both attention and FFN layers and combining prefix tuning (attn) with LoRA (ffn) both of these combined versions can improve over their respective prototypes. However, MAM Adapter achieves the best performance on both tasks and is able to match the results of our full fine-tuning by only updating 6.7% of the pre-trained parameters. In Table present the results of MAM Adapter on MNLI and SST2 as well, where MAM Adapter achieves comparable results to full fnn-ttunin by adding only 0.5% of pretrained parameters.
30) at the attention sub-layers and allocate more
DISCUSSION
We provide a unified framework for several performant parameter-tuning methods, which enables us to instantiate more effective model that matches the performance of full fine-tuning method through transferring techniques across approaches. We hope our work can provide insights and guidance for future research on parameter-efficient tuning.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Our work proposes a method for efficient fine-tuning of pre-trained models, in particular language models. Pre-trained language models have wide variety of positive applications, such as the appli- cations to summarization, translation, or language understanding described in our paper. At the same time, there are a number of ethical concerns with language models in general, including concerns regarding the generation of biased or discriminative text (Bordia & Bowman 2019), the leakage of private information from training data (Carlini et al. 2020), and environmental impact of training or tuning them (Strubell et al. 2019).
Our method attempts to train language models making minimal changes to their pre-existing param- eters. While it is an interesting research question whether parameter-efficient fine-tuning methods exacerbate, mitigate, or make little change to issues such as bias or information leakage, to our knowledge no previous work has examined this topic. It is an interesting avenue for future work.
With respect to environmental impact, the methods proposed this paper add a small number of extra parameters and components to existing models, and thus they have nominal negative impact on training and inference time for example, the final MAM Adapter needs 100% 150% training time of full fine-tuning in four benchmarks since parameter-efficient tuning typically needs more epochs to converge; the inference time is roughly the same as the model obtained by full fine-tuning. On the other hand, the methods proposed this paper may obviate the need for full fine-tuning, this may also significantly reduce the cost (in terms of memory/deployed servers) of serving models. Notably, the great majority of the experimentation done for this paper was performed data center powered entirely by renewable energy.
REPRODUCIBILITY STATEMENT
In addition to the setup description in 4.. we have detailed the complete experiments setup such as batch size, optimizer, learning rates in Appendix A Besides, we have publicized our source code. These resources should be sufficient to reproduce results the paper.
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their comments. This work was supported in part by the CMU-Portugal MAIA Project, a Baidu PhD Fellowship for Junxian He, and a CMU Presidential Fellowship for Chunting Zhou.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1x0iv 06450, 2016.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models. arXiv e-prints, pp. arXiv-2106, 2021.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 conference machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, 2016.
Shikha Bordia and Samuel Bowman, Identifying and redueing gender bias in word-level language models. In Proceedings of the 2019 NAACL: Student Research Workshop. 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voon Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:20/2.07800 2020.
10

---

Published as a conference paper at ICLR 2022
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, 2019.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03661, 2021.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In Proceedings of EMNLP, 2021.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-effiient transfer learning with diff prun- ing. In Proceedings of ACL, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of ICCV, 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In Proceedings of ICML, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen, LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2 (0fi::0000.008833 09685, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of ICLR, 2015.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficien prompt tuning. In Proceedings of EMNLP, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising scquence-to-sequencee pre- training for natural language generation, translation, and comprehension. In Proceedings of ACL, 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of ACL,, 2021.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization
Branches Out, 2004.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and prediet: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.1335
202
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. arXiv:2103. 2021b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv: I 1907. 1692, 2019.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zeutlemoyer. Multilingual denoising pre-training for neural machine translation. Trans- actions of the Association for Computational Linguistics, 2020a.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine transla- tion. Transactions of the Association for Computational Linguistics, 8.726-742, 2020b. doi:
10. 00343. URL //acl antho ogy org/ 20../2220..000 tacll....aclll. aattacll.44
Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Proceedings of NeurIPS, 2021.
11

---

Published as a conference paper at ICLR 2022
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of EMNLP, 2018.
Jekaterina Novikova, Ondłej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp.
Saarbrücken, Germany, August 2017. doi: 10. 10.18633/1/W11-55555
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL, 2002.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACI, 2018.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- Fusion: Non-destructive task composition for transfer learning, Proceedings of EACL. 2021.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, 2013.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of ACL, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, 2017.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. Proceedings of NAACI, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP: System Demonstrations, 2020.
Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li, Serial or parallel? plug- able adapter for multilingual machine translation. arXiv preprint arXiv:2104.000 08154444 2021.
12

---

Published as a conference paper at ICLR 2022
A EXPERIMENTS
A.1 SETUPS
Table 7: Datasct Statistics of the four tasks.
Dataset
#train
#dev
#test
XSum
204,045
113,332
113,334
WMT16 en-ro
610,320
1,999
1,999
MNLI
392,702
9815
9832
SST-2
67,349
872
1,821
We implement all the parameter-efficien tuning methods using the huggingface transformers li- brary WWlff et al. 2020). We use BARTIARGE (Lwis et al. (020) and mBARTIARGE (Liu et al., 2020b) (mBART-cc25) for the summarization and machine translation tasks respectively, and use ROBERTaBASE (Liu et al. 2019) for MNLI and SST2. BARTLARGE and mBART have the same encoder-decoder architectures. mBARTLARGE public checkpoints from the transformers library in experiments. For MT and classifications tasks, the max token lengths of training data are set be 150 and 512 respectively. For XSum, we set the max length of source articles to be 512 and the max length of the target summary to be 128. The detailed dataset statistics is present in Table 7 examples for validation to save time.
pre-trained on 25 languages. We use their
In our summarization experiments, we only use 1600
While we vary the bottleneck dimension within {1,30, 512, 1024) as mentioned in $ 4.1 we test bottleneck dimension 1024 only when the modified representation is FFN, because the training of prefix tuming does not fit into 48GB GPU memory when 1024. While other methods do not have memory issues, keep the bottleneck dimension of attention modification most 512 to have relatively fair comparison with prefix tuning. For LoRA we always tune its scaling hyperparameters on the dev set.
A.2 TRAINING AND EVALUATION
We present some training hyperparameters of parameter-efficient tuning methods in Table 8 For all the tasks, we train with the Adam optimizer (Kingma & Ba (015), and use polynomial learning rate scheduler that linearly decays the learning rate throughout training. We set the warm up steps of learning rate to be for both MT and summarization tasks, and for the classification tasks, learning rate is linearly warmed up from for the first 6% of the total training steps before decay. For full fine-tuning we set these training hyperparameters following Lewis et al. (2020) (XSum), Liu et al. (2020b) (en-ro), and (Liu et al. 2019) (MNLI and SST2). We also did hyperparameter search in the full fine-tuning case to try to reproduce their results. We set dropout rate to be 0.1 for all the tasks. We use ROUGE-2 and perplexity as the validation metrics for summarization and MT respectively.
For MT and text summarization, we use beam search for decoding and set the number of beams to be and 5 following previous work (Li Liang 202T Liu et al. 2020b). The min and max generation lengths for summarization and MT are set be (10, 60) and (1, 200) respectively.
A.3 OTHER EXPERIMENTAL DETAILS
Prefix Tuning: Following Li & Liang 2021), we reparameterize the prefix vectors by MLP
network which is composed of a small embedding matrix and large feedforward neural network. This is conducive for learning due to the shared parameters across all layers.
LoRA: LoRA and adapter employ different parameter initialization methods: LoRA uses a ran- dom Kaiming uniform (He et al., 2015) initialization for Wdown and zero for W up (LoRA init), while adapters use the same intialization as BERT (Devlin et al. 2019). We found it beneficial to the same initialization method LoRA in scaled PA.
13

---

Published as a conference paper at ICLR 2022
Table 8: Training hyperparameters of parameter-efffcient tuning methods the four tasks. and Is represents learning rate and label smoothing respectively.
Tasks
Ir
batch size
Is
max grad norm
weight decay
train steps
XSum
5e-5
64 sents
0.1
0.1
0.01
100K 50K 0.01
enro MT
5e-5
16384 tokens
0.1
1.0
MNLI/SST2
le-4
32 sents
1.0
0.1
10 epochs
B COMPUTATION OF TUNABLE PARAMETERS
Table 9: Number of attention or FFN sub- layers in each layer of the pre-trained mod- els.
Table 10: Number of parameters used at each sub-layer for dif- ferent methods.
NW
Arfa
BART/BBARTYARGE RoBERTABASE
Prefix Tuning
21d
Natta Nut
Adapter variants
2nd 2 2rd 4rd 2 (rd 4dr) 10rd
2rd
LoRA
We compute the number of tunable parameters based on where the tunable module is inserted into and how it is parameterized. The pretrained-models for summarization or MT have an encoder- decoder structure and each has L layers, whereas RoBERTapase for classification tasks only has encoder layers. To simplify the computation of tunable parameters, we compute the sum of parameter used in encoder layer and one decoder layer the parameter overhead of one single layer the pre-trained mncoder-decoder model. Each layer has Namn sub-layers and Ntt. sub- layers. For the encoder-decoder models, Nastn 3: the encoder self-attention, the decoder self- attention and the decoder cross-attention. For the classification tasks, RoBERTapase only has the encoder self-attention, thus Nattn = 1. We present the number of attention and ffn sub-layers for different pre-trained models in Table TO For modifications applied at the attention sub-layers, the number of tunable parameters computed by [0lattn N x Nuttn x L,, where Nw denotes the number of parameters (Wdown or Wup) used for one attention sub-layer. Similarly, the number of tunable parameters for the FFN sub-layers is computed by [O] Nit x Nn. x we have explained in $4.4 LoRA approximates the update of each weight matrix with a pair of Wdown and Wup. up, thus LoRA typically uses more parameters with the same r as other methods. Finally, the total number of tunable parameters for prefix tuning, adapter variants and LoRA is | | + 1 fin as applicable. Prompt tuning prepends 1 tunable vectors at the input layer and uses x d number of parameters. Using MBART/BART
L. In Table 10 we show the number of parameters for one sub-layer.
example, we present the number of parameters used by several representative methods throughout our paper in Table II where adapter variants include sequential adapter, parallel adapter, scaled adapter and multi-head adapter.
Table 11: Number of tunable parameters of various parameter-effiiient tuning methods with BARTMBBRT
models (L 12) an example.
Method
number of parameters
Prompt Tuning
x d
Prefix Tuning (attn)
2/d x 3 x 12
Adapter variants (attn)
2rd 3 x
Adapter variants (ffn)
2rd x 2 x 12
LoRA (attn)
4rd 3 x
LoRA (ffn)
10rd x x 12
MAM Adapter (our proposed model)
21d x + 2rd x 12
C FULL RESULTS ON DIFFERENT BOTTLENECK DIMENSIONS
14

---

Published as a conference paper at ICLR 2022
Table 12: Performance on the test sets of abstractive summarization (XSum) and WMT EN-RO translation.
Method
# params (%)
XSum (R-1/2/L)
MT BLEU
Modified Representation:
attention
Prefix Tuning, 200
3.6
43.40/20.46/35551
35.6
Prefix Tuning, = 512
9.2
43.29/20.40/355
35.1
LoRA, r 200
7.2
43.09/20.29/35..1
36.2
Sequential Adapter, 200
3.6
42.01/19. 02.01/19.30/34..4
35.3
Sequential Adapter, 512
9.2
41.05/11.87/33.77
34.7
Parallel Adapter, 200
3.6
43.58/20.31/35.34
35.6
Parallel Adapter, r = 512
9.2
43.99/20.83/35.77
36.2
Modified Representation: FFN
LoRA, r = 102
6.1
44.59/21.1136.25
36.5
Sequential Adapter,
200
2.4
33.21/19.98/35.08
35.6
Sequential Adapter.
512
6.1
43.72/20.75/35.66
36.3
Sequential Adapter, 1024
12.3
43.95/21
05.95/221.0/35.99
36.7
Parallel Adapter, r -200
2.4
43.93/20.66/35.63
36.4
Parallel Adapter, r =5112
6.1
14.35/20.98/35.99
37.1
Parallel Adapter, 1021
12.3
44.53/21. 24/36.23
37.3
15


Reviews:
[{"review_id": "XbEUCnnma0e", "reviewer": "Reviewer_EtWw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A new perspective of parameter-efficient methods within a unified framework.\n- A number of controlled studies which give insights into which components are meaningful and in which cases.\n- The released code base can help boost future research in this area.\n- Clear and well-written.\n\nWeaknesses:\n- No major weaknesses identified.\n\nQuestions:\n- It would be interesting to see the variance observed when applying parameter-efficient methods. Previous work found large variance in downstream performance across different seeds when performing full fine-tuning [1,2,3]. This is something you could add, for example, to Table 2.\n\nTypos:\n- In Figure 2, the performance of Adapter is 21.00, not 20.46\n- The \"v\" subscript is missing in Eq. 5\n\n[1] Dodge et al. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. 2021\n[2] Mosbach et al. On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines. ICLR 2021\n[3] Bugliarello et al. Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs. TACL 2021", "summary_of_the_paper": "This paper investigates recent parameter-efficient methods that have been shown to achieve good performance in a variety of NLP tasks. The authors analyse their connections and propose a unified framework which subsumes a number of existing approaches. The authors then compare their performance across four NLP tasks, with varying level of difficulty and amount of available resources. Here, compared to full fine-tuning, they find that parameter-efficient approaches perform well on simpler tasks (MNLI and SST2), while showing larger gaps on more challenging tasks (XSum and MT). A series of controlled experiments centred around the proposed framework shows that parallel adapters applied to the FFN module of a Transformer generally perform better. Applying them to the attention module leads, however, to better gains when the number of additional parameters is small. Finally, the authors combine these findings into a novel MAM adapter module that leads to further gains in every task.", "summary_of_the_review": "This paper provides a unified framework for parameter-efficient NLP, an important task when serving models at scale. The experiments provide new insights into the performance of existing methods, that, when combined, result in further performance improvements. I think this is a solid paper and recommend its acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PBPjBdrsEve", "reviewer": "Reviewer_JZ1E", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The analysis and conclusion on Adapters, Prefix Tuning, and LoRA are very penetrating. Specifically, the paper derives an equivalent form of preﬁx tuning to establish its connection with adapters in a unified view.\n\n\n2. The paper proposes a uniﬁed framework for parameter-efﬁcent tuning that includes several state-of-the-art methods as instantiations.\n\n3. The paper is well organized and well-motivated with theoretical analysis and proof. \n\n4. Experiment setting and analysis are comprehensive; the proposed method Scaled PA consistently shows its advantages against the other baselines.\n", "summary_of_the_paper": "This paper breaks down the design of state-of-the-art parameter efﬁcient transfer learning methods and presents a uniﬁed framework that establishes connections between them. The paper re-frames them as modifications to specific hidden states in pre-trained models and defines a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Experiments on machine translation, text summarization, language understanding, and text classification have been conducted, which indicates that the unified framework enables can instantiate new parameter-efficient fine-tuning methods that tune fewer parameters.", "summary_of_the_review": "Overall, this paper is well-written and well-motivated. I think this paper attracts lots of researchers and will inspire future works in parameter efficient training. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-ZIWcdyn-tP", "reviewer": "Reviewer_eByL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- The common framework for different approaches is useful overall. In particular, I liked the section clarifying the connection between Prefix Tuning and Adapters. Although there is nothing groundbreaking here, I believe these frameworks can be useful and help researchers.\n- The authors show how the framework can be used to derive new approaches. They also show that these approaches can be more effective than existing ones.\n- The experimental setup is well-described overall, with hyperparameters being provided\n- This is a hard analysis to perform, with many possible things to change. I feel like the experiment set chosen is convincing overall, with the caveat mentioned below.\n\nCons:\n\n- Significance of results:\n\nThere are no standard deviations for any results in the paper, which makes it hard to assess significance of many results, esp. since some of the performance gaps are small.\nFor instance, in table 6, authors have a 0.4 BLEU discrepancy in their replication of full fine-tuning performance but draw conclusions on MAM being best based on a 0.2 BLEU gap. The gap is bigger when comparing MAM to methods not introduced by the authors, but still.\nThe story makes sense but I am not 100% confident in the robustness of the results. Using two decimal numbers for tasks (e.g: XSum) also gives a false impression of precision. \nI understand fine-tuning on some of these tasks (MT / XSUM) can be resource-intensive but having standard deviations in even a subset of the experiments would be useful.\n\n- Interpretation of Figure 4 in Section 5.2 and effectiveness of MAM Adapters for encoder models\n\nThe authors highlight that while existing methods perform well on MNLI/SST, they are underwhelming on en-ro / XSum. They conclude that this means existing parameter-efficient transfer learning (PETL) approaches are not great for higher-resource / more challenging tasks.\n\tHowever, I would point out that this data can also support a different conclusion: existing PETL approaches work well for encoder-only models but are not great for encoder-decoder models. Including the T5 datapoint is also not very relevant since in that case Superglue is treated as a single task (w/ one adapter). Answering “For which tasks / architectures do PETL methods perform well” is an interesting question in itself and I feel like this section expedites this. It is likely that both the architecture and #datapoints in the task matter. \n\tThe author’s choice at the end of section 4.2 is to focus on XSum / en-ro MT. It would be great to highlight that not only are those higher-resource tasks, but they are also generative ones, and thus all the conclusions of this paper might not apply widely to encoder models. Indeed, the results of MAM adapters in table 2 are quite mixed. Right now the paper is claiming more generality than deserved.\n\tTwo possible fixes here: (1) Make claims less general and specific to enc-dec models, (2) Conduct more experiments on encoder-only models \n\n- Writing:\n  - The writing is subpar right now. Example: section 4.4: “its counterpart at attention” -> “its attention counterpart”,  “FFN can better utilize modification at larger capacities” -> modifications or “modification of the FFN is better at larger capacities”, etc. \n  - Minor: The citation style is often wrong for the sentence. Use \\citet more often.\n  - Minor: Replace “For classifications” with “for classification tasks” in the Appendix (twice)\n  - Typos: Section 5 name Discussions -> Discussion ;  Appendix A3 Learning -> Learning. \n\n- Minor:\n  - For Figure 2, I feel like the full fine-tuning number should come from the original baseline, not the replication.\n  - The protocol in 4.5 for choosing the scaling parameter for Scaled PA is a bit surprising. I would just suggest that it should be an hyperparameter, the current description makes it seem like it is based on hyperparam tuning for LoRA. \n  - It seems to me that 4.5 is about composition function *and* \\delta h functional form since LoRA changes both. \n\n\n~~~~~~~~\n\nRevised score upwards after response, see below\n", "summary_of_the_paper": "This paper formulates different approaches for parameter-efficient transfer learning (such as adapters, prefix-tuning, LoRA) under a common framework. Approaches vary alongside different design dimensions: functional form, insertion form (sequential or parallel), which representation is directly modified by the approach and the composition function of the main representation with the adaptation one. \n\nThe authors describe how existing approaches fit this framework. Then, they show how changing these design choices leads to new approaches, including parallel adapters (PA), multi-head PA and scaled PA. \nThe authors then compare their newly introduced methods and existing ones empirically. They find that:\n1. Parallel insertion is generally better\n2. Under a low parameter budget (~0.1% of original) it is preferable to modify the attention of the transformer. Under a higher budget, changing the feed-forward network is better.\n3. Their methods generally perform on par or better than existing approaches.\n\nFinally, the authors combine their insights to develop “Mix-and-match” adapters and show that it does well overall.\n", "summary_of_the_review": "Compelling framework to unify parameter-efficient transfer learning approaches, with interesting new methods emerging from it. The set of experiments is well-chosen overall, but the lack of standard deviations and the focus on generative tasks / enc-decoder models makes the results less robust and general. The writing would benefit from more work as well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper reviews and draws connections between several parameter-efficient fine-tuning methods.

All reviewers found the paper addresses an important research problem, and the theoretical justification and empirical analyses are convincing.
Decision: Accept (Spotlight)