Parameter-Efficient Transfer Learning with Diff Pruning
Authors: Demi Guo
Alexander M Rush
Yoon Kim
Venue:  (2021)
Keywords: transfer learning
parameter efficiency

Abstract: While task-specific finetuning of deep networks pretrained with self-supervision has led to significant empirical advances in NLP, their large size makes the standard finetuning approach difficult to apply to multi-task, memory-constrained settings, as storing the full model parameters for each task become prohibitively expensive. We propose $\textit{diff pruning}$ as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the $L_0$-norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5$\%$ of the pretrained model's parameters per task.

Under review as a conference paper at ICLR 2021
PARAMETER-EFFICIET TRANSFER LEARNING WITH DIFF PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
While task-specific finetuning of deep networks pretrained with self-suupeviiion has led to significant empirical advances in NLP, their large size makes the stan- dard finetuning approach difficult to apply to multi-task, memory-consttrinne set- tings, as storing the full model parameters for each task become prohibitively expensive. We propose diff pruning as simple approach to enable parameter- efficient transfer learning within the peetrain-finetune views finetuning learning a task-specific "diff" vector that is applied top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable ap- proximation to the Lo-norm penalty to encourage sparsity. Diff' pruning becomes parametee--ffficientt the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. We find that models fine- tuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model's parameters per task.
framework. This approach
INTRODUCTION
Task-specific finetuning of pretrained deep networks has become the dominant paradigm in contem- porary NLP, achieving state-of-the-att results across a suite of natural language understanding tasks (Devlin et al., 2219901Lu et al. (2019c; Yang et al. 2019 Lan et al. 2020). While straightforward and empirically effective, this approach difficult to scale to multi-task,
ceemory-constriing settings (e.g. for on-device applications), as it requires shipping and storing a full set of model parameters for each task. Inasmuch as these models are learning generalizable, task-agnostic language representa- tions through self-supervised pretraining, finetuning the entire model for each task is an especially inefficient use of model parameters.
A popular approach to parammter-efffcieency is to learn sparse models for each task where subset of the final model parameters are exactly zero (Gordon al. 2020 Sajjad et al. 2020 Zhao et al. 2020 Sanh et al. 2020). Such approaches often face a steep sparsity/performance tradcoff, and substantial portion of nonzero parameters (c.g. 10%-30%) )) are still typically required to match the performance of the dense counterparts. An alternative is to multi-task learning or feature-based
transfer for more parameter-efficien transfer learning with pretrained models (Liu et al. 2019b]
Clark al. 2019 Stickland & Murray| [2019 Reimers& & Gurevych) 2019 Feng et al. 20200. These methods learn only a small number of additional parameters (c.g. a linear layer) on top of a shared model. However, multi-task learning generally requires access all tasks during training to prevent catastrophic forgetting (French 1999), while feature-based transfer learning (e.g. based on task- agnostic sentence representations) is typically outperformed by full finetuning (Howard Ruder 2018).
Adapters (Rebuffi et al., 2018) have recently emerged as promising approach to parameter- efficient transfer learning within the pretrain-finetune paradigm (Houlsby et al. 2019 |Peiiferr [ et al.) 2020a blc). Adapter layers are smaller, task-specific modules that are inserted between layers of pretrained model, which remains fixed and is shared across tasks. These approaches do not require access to all tasks during training making them attractive in settings where one hopes to obtain and share performant models new tasks arrive in stream. Houlsby et al. (201)) find that adapter lay- ers trained on BERT can match the performance of fully finetuned BERT on the GLUE benchmark (Wang et al. 2019a) while only requiring 3.6% additional parameters (on average) per task.

---

Under review as a conference paper at ICLR 2021
In this work, consider similar setting adapters but propose new diff pruming approach with the goal of even more parameter-efficien transfer learning. Diff pruning views finetuning as learning task-specific difference vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. In order to learn this vector, we reparameterize the task-specific model parameters as uusk- pretzanned musk, where the pretrained parameter vector O pretrained is fixed and the task-specific diff vector ōusk is finetuned. The diff' vector is regularized with a differentiable approximation to the Lo-norm penalty (Louizos et al., 2018) encourage sparsity. This approach can become parameter-efficient as the number of tasks increases as it only requires storing the nonzero positions and weights of the diff vector for each task. The cost of storing the shared pretrained model remains constant and is amortized across multiple tasks, On the GLUE benchmark (Wang et al., 2019a), diff pruning can match the performance of the fully finetuned BERT baselines while finetuning only 0.5% of the pretrained parameters per task, making it a potential alternative to adapters for parameter-efficiine transfer learning.
2 BACKGROUND: TRANSFER LEARNING FOR NLP
The field of NLP has recently seen remarkable progress through transfer learning with pretrain- and-finetune paradigm, which initializes a subset of the model parameters for all tasks from a pre- trained model and then finetunes a task specific objective. Pretraining objectives include context prediction (Mikolov et al.] 2013), autoencoding (Dai & Le (013),, machine translation (McCann al. 2017), and more recently, variants of language modeling (Peters al. 2018 Radford al. Devlin et al., 2019) objectives.
2018
Here consider applying transfer learning to multiple tasks. We consider a setting with poten- (n) N
tially unknown set of tasks, where each 7 E T has an associated training set {of"), yr In=1 For all tasks, the goal is to produce (possibly tied) model parameters 0, to minimize the empirical risk,
Σ ( + AR(0,)
min 6,
where ((: is a parameterized function over the input (e.g- a neural network), L(:,)) is a loss function (e.g. cross-entropy), and R(-) is optional regularizer with hyperparameter X.
This multi-task setting can use the pretrain- then- wwwntthhettueeee finetune approach by simply learning indepen- dent parameters for each task; however the large size of pretrained models makes this approach exceedingly parameter inefficient. For example, widely-adopted models such as BERTBASE and BERTLARGE have 110M and 340M parameters respectively, while their contemporaries such as TS (Raffel et al., 2020), Megatron-L.M (Shoeybi et al.) (019). and Turing-NLG (Rajbhandari al. 2019) have parameter counts in the billions. Storing the fully finetuned models becomes difficult even for a moderate number of tasks R A classic approach to tackling this parameter- inefficiency (Caruana 1997p is to train a single shared model (along with a task-specific output layer) against multiple tasks through joint training. However, the usual formulation of multi-task learning requires the set of tasks T to be known in advance in order to prevent catastrophic forget- ting (French) 1999) | making it unsuitable for applications in which the set of tasks is unknown (e.g.
when tasks arrive in stream).
3
DIFF PRUNING
Diff pruning formulates task-specific finetuning as learning a diff vector 6, that is added to the pretrained model parameters 0 pretrnined We first reparameterize the task-specific model parameters,
0
preeraained + 6.,
'Therefore our setup different from the classic multitask setting which usually assumes that set of tasks known
"An intriguing line of work suggests that large-scale language models can be used without finetuning for a
variety of tasks if given the appropriate context (Radford et al. 2019 Brown al. 2020). While interesting.
these models generally underperform task-specific models and require billions of parameters, though recent
work suggests that they can be made substantially smaller (Schick & Schutze 2020
'However, work on continual learning mitigates these issues to an extent Shin et al. 2017 Lopez- Paz. &
Ranzato 2017
Lee et al. 2017 Kirkpatrick et al. 2017 Parisi et al. 2018.
2

---

Under review as a conference paper at ICLR 2021
which results in the following empirical risk minimization problem,
NV.
min 6, N
1
Σ C ( m=
pretrained + -6)), + XR(O pretrained + 67).
This trivial reparameterization equivalent to the original formulation. Its benefit comes in the multi-task setting where the cost of storing the pretrained parameters prerrnined amortized across tasks, and the only marginal cost for new tasks is the diff' vector. If we can regularize 6, to be sparse such that ||6, lle << I pretrained llu, then this approach can become more parameter-effiiemt as the number of tasks increases. We can specify this goal with an Ly Lo-morm penalty on the diff vector,
pretrainee + 5,) = =16-1lo = 1 + 0}.
3.1 DIFFERENTIABLE APPROXIMATION TO THE NORM
This regularizer is difficult directly optimize as it is non-differentiable In order to approximate this Lo objective, we follow the standard approach for gradient-based learning with Lo sparsity using a relaxed mask vector (Louizos et al. 2018). This approach involves relaxing binary vector into continuous space, and then multiplying it with a dense weight vector to determine how much of the weight vector is applied during training. After training, the mask is deterministic and large portion of the diff vector is true zero.
To apply this method we first decompose o, into binary mask vector multiplied with a dense vector,
6 2 W
2, E (0, 13d w, E Rdd
We can now instead optimize expectation with respect to Zr, whose distribution p(zr; cr) is initially Berouuli with parameters Cty,
N
min Ez, (ppe.;aa) Wr
I
C (f(oft)) Σ 1=1
; ppettriiee + 2, WT.) v(fi) + A115+110
N
This objective is still difficult in practice due z, 's being discrete (which requires the score func- tion gradient estimator), but the expectation provides some guidance for empirically effective relax- ations. We follow prior work (Louizos et al., 2018 Wang et al.) 2019b) and relax 2,- into continuous space [0, 1]d with stretched Hard-Concrete distribution (Jang et al. 2017 Maddison et al. 2017 which allows for the use of pathwise gradient estimators. Specifically, ly. 2, is now defined to be a deterministic and (sub)differentiaabl function of a sample from a uniform distribution.
u ~~ (O, 1), = (r-1)+1
8, (log u log(1 - u) + ar),
2, min (1, max
Here 1 and r > 1 are two constants used to stretch 8, into the interval (1, r) before is clamped to [0, 1]d with the min(1, max(0, .)) operation. In this case we have differentiable closed-form expression for the expected Lo-norm,
d Σ 1=1
ar, log , )
E |||
Σ E[z 031
1=1
Thus the final optimization problem given by,
min E.-U(O W-
1 (F(af"))
N
c
0 pectrained + 2, WT, (1)
σ ar, log 7
N
and we now utilize pathwise gradient estimators optimize the first term with respect to ar since the expectation no longer depends on it After training we obtain the final diff vector 8, by sampling once obtain 2, (which is not necessarily binary vector but has significant number of dimensions equal to exactly zero due to the clamping function), then setting 5, = 2, @ w, F
"To reduce notation clutter we subsume the parameters of the task-specific output layer, which not pre- trained, into pprttined Wc do not apply the Lo-norm penalty on these parameters during training.
We found sampling once work as well more complicated alternatives (e.g. hased on multiple samples).
3

---

Under review as a conference paper at ICLR 2021
3.2 Ly- Lu-BALL PROJECTION WITH MAGNITUDE PRUNING FOR SPARSITY CONTROL
Differentiable Lo regularization provides a strong way to achieve high sparsity rate. However, it would be ideal have more fine-grained control into the exact sparsity rate in the diff vector, espe- cially considering applications which require specific parameter budgets. As À is just the Lagrangian multiplier for the constraint ||] < 1j for some 1). this could be achieved in principle by search- ing over different values of X. However we found it more efficient and empirically effective to achieve an exact sparsity rate by simply projecting onto the Lo-ball after training.
Specifically we use magnitude pruning on the diff vector 8, and target a sparsity rate 1% by only keeping the top 5% x values in o F Note that unlike standard magnitude pruning, this is based on the magnitude of the diff vector values and not the model parameters. As is usual in magnitude pruning, we found it important to further finetune o, with the nonzero masks fixed to maintain good performance (Han et al.) 2016). Since this type of parameter-efficieng through projection onto the Lo-ball can be applied without adaptive diff pruning such an approach will serve as one of our baselines in the empirical study.
3.3 STRUCTURED DIFF PRUNING
Diff pruning, as presented above, is architecture-agnostic and does not exploit the underlying model structure-each dimension of z, is independent from one another. While this makes the approach potentialll more flexible, we might expect to achieve better sparsity/performance tradeoff through a structured formulation which encourages active parameters to group together and other areas to be fully sparse. Motivated by this intuition, we first partition the parameter indices into groups {g(1), ................................ (G)) where g(j) is subset of parameter indices governed by group g(j) [ We then introduce a scalar z (with the associated parameter ax)) ) for each group g()), and decompose the spec-fie parameter for index i E g(j) as 8', = Zr x z x Wr,i. The expected Lo-norm is then given by,
) Σ Σ 0 caaaat log 7 x a ar -log r
|11 = Σ Σ
[1
2 > 0)]
1=1 kE
J=1 (E)(j)
and we can train with gradient-based optimization as before.
EXPERIMENTS
4.1 MODEL AND DATASETS
For evaluation we use the GLUE benchmark (Wang et al.) 2019b). popular finetuning dataset. Following adapters (Houlsby et al. 2019), we test our approach on the following subset of the GLUE tasks: Multi-Genre Natural Language Inference (MNLI), where the goal is two predict whether the relationship between two sentences is entailment, contradiction, neutral (we test on both MNLI and MNLI which respectively tests on matched/miismatheed domains); Quora Question Pairs (QQP). a classification task to predict whether Natural Language Inference (QNLI). which must predict whether a sentence a correct answer the question; Stanford Sentiment Treebank (SST-2), sentence classification task predict the sentiment of movie reviews; Corpus of Linguistic Acceptability (CoLA), where the goal is predict whether a sentence is linguistically acceptable or not; Semantic Textual Similarity Benchmark (STS- B). which must predict similarity rating between two sentences; Microsoft Research Paraphrase Corpus (MRPC), where the goal is to predict whether two sentences are semantically equivalent; Recognizing Textual Entailment (RTE), which must predict whether second sentence is entailed by the first. For evaluation, the benchmark uses Matthew's correlation for CoLA, Spearman for STS-B, F1 score for MRPC/QQC, and accuracy for MNLI/OLLISS../RTE.
question semantically equivalent; Question
WWang et al. (2019b) show that it also is possible to inject such constraint softly into training objec- tive by regularizing the expected model size towards a certain rate. However, since the constraint is soft this approach also makes it difficult to target an exact sparsity rate.
"Coneretely, one obtain through usual finetuning, set 6, 0, Openrined and then apply magnitude
pruning followed by additional finetuning on 8,.
WWhile groups can be defined in various ways, we found that defining groups based on each matrix/bias vector of the pretrained model was simple and worked well enough.
4

---

Under review as a conference paper at ICLR 2021
Total
New parums per task
QNLI
SST-2
MNLI,.
MNL.um
CoLA
MRPC
STS B RTB
QQP
Avg.
params
Full finetuning
9.00x
100%
91.1
94.9
86.7
85.9
60.5
89.3
87.6
70.1
72.1
80.9
Adapters (8- (8-256)
1.32x
3.6%
90.7
94.0
84,9
85.1
59.5
89.5
86.9
71.5
71.8
80.4
Adapters (64)
1.19x
2.1%
91.4
94.2
85.3
84.6
56.9
89.6
87.3
68.6
71.8
79.8
Full finetuning
9.00x
100%:
93.4
94.1
86.7
86.0
59.6
88.9
86.6
71,2
71.7
80.6
Last layer
1.34x
3.8%
79.8
91.6
71.4
72.9
40.2
80.1
67.3
58.6
63.3
68.2
Non-adap. diff pruning
1
0.5%
89.7
93.6
84,9
84.8
51.2
81.5
78.2
61.5
68.6
75.5
Diff pruning
1.05x
0.5%
92.9
93.8
85.7
85.6
60.5
87.0
83.5
68.1
70.6
79.4
Diff pruning (struct.)
1.05x
0.5%
93.3
94.1
86.4
86.0
61.1
89.7
86.0
70.6
71.1
80.6
Table 1: GLUE benchmark test server results with BERT models. (Top) Results with adapter bottleneck layers (brackets indicate the size bottlenecks), taken from from Houlsby et al.] (2019). (Bottom) Results from this work. "QNLI results are not directly comparable across the two works as the GLUE benchmark has updated the test set since then. To make our results comparable the average column is calculated without QNLI.
For all experiments, we use the BERT LARGE model from Devlin et al. (2019), which has 24 layers, 1024 hidden size, 16 attention heads, and 340M parameters. We use the Huggingface Transformer library (Wolf et al. 2019) to conduct our experiments.
4.2 BASELINES
We compare both structured and non-structured variants of diff pruning against the following base- lines: Full finetuning, which fully finetunes BERTLARGE as usual; Last layer finetuning, which only finetunes the penultimate layer (along with the final output layer Adapters from Houlsby et al. (2019), which train task-specific bottleneck layers between between each layer of a pretrained model, where parameter-efficiency can be controlled by varying the size of the bottleneck layers; and Non-adaptive diff pruning. which performs diff pruning just based on magnitude pruning (i.e., we obtain 0, through usual finetuning, set 6, 0, pretrained. and then apply magnitude pruning followed by additional finetuning on 8,). For diff pruning we set our target sparsity rate to 0.5% and investigate the effect of different target sparsity rates in section 5.1
4.3 IMPLEMENTATION DETAILS AND HYPERPARAMETERS
Diff pruning introduces additional hyperparameters 1, r (for stretching the Hard-Concrete distribu- tion) and À (for weighting the approximate Lo-norm penalty). We found 1 = - -1.5, r 1.5, À 1.25 X 10-7 to work well across all tasks. We also initialize the weight vector w,, to 0, and a to a positive vector (we use 5) to encourage Z, to be close to 1 at the start of training. While we mainly experiment with BERT LARGE to compare against prior work with adapters (Houlsby et al. 2019), in preliminary experiments
found these hyperparameters to work for finetuning RoBERTa (Liu et al. 2019c) and XLNeet (Yang et al. 2019) models as well.
For all tasks E 64, 8, 10} and the number of epochs E (2,3, 4, 5} T0 However we found the default settings used for regular finetuning as suggested in the original BERT paper work well for most tasks. Finetuning with the fixed mask after projecting onto the Lo-ball with magnitude pruning is done with learning rate of 5 10 for 3 or epochs (3 epochs for QNLI, SST-2, MNLI-m, MNLI-mm, CoLA, QQP, epochs for MRPC, STS-B, -B, RTE). Grouping for the structured version of diff pruning based on the matrix/bias vectors (i.e. parameters that belong to the same matrix or bias vector are assumed to be in the same group), which results in 393 groups [I
a learning rate of 10-5 and perform a hyperparameter search over batch size
5 RESULTS AND ANALYSIS
Our main results the GLUE benchmark are shown in Table Structured diff pruning can match the performance of a fully finetuned BERTLARGE model while only requiring 0.5% additional pa-
Wu et al. (2220) observe that finetuning later layers generally performs better than finetuning earlier layers For the larger QNLI, SST-2, MNLI-m, MNLI-mm, CoLA, QQP datasets, we use batch size over epochs. For the smaller MRPC, STS-B, B, RTE datasets, we use batch size of over 3 epochs.
since it depends on how one concatenates the input vector before each affine layer. Our grouping is based on Huggingface's BERT implementation at /ttp://eeecccccccc2cc222c 9e2cc7 (available at htttts:://ggtttaaaa http://665661132222629922333 /2 9b 2ecceeecccccece 9e2cc77
"This definition of groups is implementation-speiife
commit (5561388622966666326 huggingface wwtpe///aaeetaatttneeeeeeee wtw.trrannsorssoeooooeoo sra/trans formers/ mode1 ing bert. py). In preliminary experiments we found this simple definition work well compared to alternative group dehnitions (c.g. hased on individual neurons).
com/
5

---

Under review as a conference paper at ICLR 2021
90%
GLUE Average Performance
Pruned Diff Groups
Non-structured
Structured
60%
%
MRPC
24
6.1
52
13.2
70%
STS-B
25
6.4
12.2
Non adaptive DIf Pruning
RTE
28
7.1
12.7
Pruning
Avg
25.7
6.5
50.0
12.7
Di Pruning (Strudured)
0000%
0.20%
0.50%
0.70%
1,00%
Vector Sparsity
Figure 1: (Left) Average performance on the GLUE validation set across different target sparsity rates for the different methods. (Right) Number of groups where all of the parameters in the group are fully zero for structured non-structured diff pruning 0.5% target sparsity. We group based each matrix/bias vector, resulting groups in total.
Dift voctor
QNII
SST-2
MNLI
MNLI
CaLA
MRPC
STS-B
RTE
QQP
Arg
target sparsity
0.10%
92.7
93.3
85.6
85.9
58.0
87.4
96.3
68.6
85.2
82.5
0.25%
0.50%
93.4
94.2
86.4
86.9
63.5
91.3
89.5
71.5
86.6
84.8
1.00%
93.3
94.2
86.4
87.0
66.3
914
89.9
71.1
86.6
85.1
100%
93.5
94.1
86.5
87.1
62.8
91.9
89.8
71.8
87.6
85.0
Table Structured diff pruning results on validation set with different target sparsity rates. Average performance includes all tasks.
rameters per task. Diff pruning without structured sparsity also performs well, though slightly worse than the structured approach. Non-adaptive diff pruning, which magnitude prunes the diff vector without learning the binary mask 2,, performs significantly worse, indicating the importance of learning the masking vector. Compared to adapters, diff pruning obtains similar performance while requiring fewer parameters per task, making it a potential alternative for parameter-efficient transfer learning 12
We now perform series of analysis experiments on the validation set.
5.1 VARYING THE TARGET SPARSITY
In Figure (left), we plot results the GLUE validation set averaged across all tasks at target sparsity rates of 0.1%. 0.25%, 0.5%, 1.0% for the different baselines. Structured diff pruning con- sistently outperforms non-structured and and non-adaptive variants across different sparsity rates. The advantage of adaptive methods becomes more pronounced at extreme sparsity rates. In Table report the breakdown of accuracy of structured diff pruning across different tasks and sparsity rates, where we observe that different tasks have different sensitivity to target sparsity rates. This suggests that can obtain even greater carameter-efficiene through targeting task-specifie sparsity rates the diff vector.
5.2 STRUCTURED vs. NON-STRUCTURE DIFF PRUNING
Structured diff pruning introduces an additional mask per group. which encourages pruning of entire groups. This is less restrictive than traditional group sparsity techniques that have been used with Lg-norm relaxations which force all parameters
group to share the same mask (Louizos et al.) 2018 Wang et al. 2019b). However we still expect entire groups to be pruned out more often in the structured case, which might bias the learning process towards either eliminating completely or clustering together nonzero diffs. In Figure I (right), we indeed find that structured diff pruning leads to finetuned models that are much more likely to leave entire groups unchanged from their pretrained values (zero diffs).
5.3 MASK-SPECIHIC SPARSITY
Different layers of pretrained models have argued to encode different information (Liu et al. 2019a Tenney et al. 2019). Given that each task will likely recruit different kinds of language phenomena embedded in the hidden layers, we hypothesize that diff pruning will modify different parts of the
However diff pruning incurs additional storage cost due storing the nonzero positions of the diff vector.

---

Under review as a conference paper at ICLR 2021
ONLI
SST-2
MNLI
CoLA
MRPC
STS-B
RTE
QQP
Figure 2: Percentage of modified parameters attributable to each layer for different tasks at 0.5% target sparsity. The layers are ordered from earlier to later (i.e. the embedding layer is shown at the top). The x-axis for each plot goes from 0% to 20%.
QNLI
SST-2
MNLI
MNLI
CoLA
MRPC
STS-B
RTE
QQP
Avg
Sparsity Performance
1.5%
0.6%
0.8%
0.8%
1.6%
2.4%
3.3%
0.7%
0.6%
1.4%
93.8
94.0
86.2
86.8
63.1
91.9
89.7
71.8
86.5
84,9
With 0.5% sparsity
93.4
94.2
86.4
86.9
63.5
91.3
89.5
71.5
86.6
84.8
Table 3: (Top) Sparsity and performance before magnitude pruning on the validation set with structured diff' pruning. (Bottom) Performance with 0.5% target sparsity.
pretrained model through task-specific finetuning. Figure shows the percentage of nonzero diff parameters attributable to the different layers for each task. We find that different tasks indeed modify different parts the network, although there are some qualitative similarities between some tasks, for example between QNLI & QQP (both must encode questions), and MRPC & STS-B (both must predict similarity between sentences). The embedding layer is very sparsely modified for all tasks. While some of the variations in the sparsity distributions is due to simple randomness, do observe some level of consistency over multiple runs the same task, as shown in Figure 3 of the appendix.
The ability to modify different parts of the pretrained model for each task could explain the improved parameter-=fficiency of our approach compared Houllyy al. (2019)'s 's adapter layers, which can only read/write to the pretrained model at certain points of the computational graph 13 This poten- tially suggests that adapter layers with more fine- ine-grained access into model internals (e.g. adapters for sey/value/query transformations) might result in even greater -eeuuntte--fffcieeey.... While left as future work, we also note that diff pruning can be applied conjunction with adapters, which might further improve results.
5.4 EFFECT OF Lo- LO-BALL PROJECTION VIA MAGNITUDE PRUNING
Applying magnitude pruning to project onto the Lo-ball was crucial in achieving exact sparsity targets. As shown in T we observed little loss performance through magnitude pruning. We re-iterate that it was crucial to finetune with the fixed mask in order to maintain good performance [4]
5.5 SQUAD EXTRACTIVE QUESTION ANSWERING
To demonstrate the effectiveness of our approach beyond classification, we additionally experiment on the extractive question answering task SQuAD, which asks model to select the answer span to question given a Wikipedia paragraph. To make direct comparisons with Houlsby et al. (2019), we run all experiments our full finetuning baseline|
SQuAD v1.1. For diff pruning, 18 Results are shown in Table 4 Diff pruning is able achieve comparable better performance with only 1% additional parameters. Notably, improve the F1 score of full finetuning baseline by a significant margin (e.g. 90.8% = 93.2%)
the same general tlyer-parametten as
that our method can
To simulate this restricted setting, we tried applying diff pruning only on the dense transformations just before the output of each layer (i.c. after self-attention layers), and observed much worse performance.
"#vven for the approach that does not apply magnitude pruning, we found it helpful fix the mask z, after an initial training phase and finetune just w..
15 hhtps:/huggingffcc..c http:://hggnnngaaa.e.ceetttteeee 2e/transformers/22.5.. wttp:///.....eeees...... l/examples.hmml

---

Under review as a conference paper at ICLR 2021
Sparsity
Full finetuning
100%
90.7%
Adapters
2%
90.4%
Full finetuning 100%
90.8%
Diff pruning
1 %
92.1%
Diff pruning (struct.)
1 1%
93.2%
Table 4: SQuAD validation results with BERTAARGE model.
while modifying many fewer parameters (e.g., 100% = 1%), which potentially implies that diff pruning can have a useful regularization effect.
DISCUSSION
6.1 MEMORY REQUIREMENTS
For training, our approach requires more memory than usual finetuning due to additionally opti- mizing a and w, This did not present significant challenge for pretrained models that experimented with in this study, since majority of GPU memory was utilized by the minibatch's activation layers. However, this could present issue model sizes get larger and larger. While training efficiency was not primary concern of this work, diff pruning takes approxiamtely 1.5x 2x more time per batch, which results in slower training.
After training, storing the task-specific diff' vector requires storing a compressed version with both the nonzero positions and weights, which incurs additional storage requirements.
6.2 INFORMATION EFFICCEETT TRANSFER LEARNING
Efficiently representing pretrained models adapted new tasks is becoming an increasingly impor- tant problem in contemporary NLP. This paper focuses rather narrow definition of efficiency- parameter-efficiencyy An interesting direction might be to target generalizations of parameter- efficiency, for example, information-efficienee, which aims minimize the number of bits required to represent the task-specific model when given the pretrained model for free. This view can suggest other avenues for achieving information-efficien transfer learning: for example, "what the min- imum number of (potentially synthetic) datapoints that we can finetune BERT on to obtain a good task- sask-specific model?" or "what is the shortest prefix string that we can condition GPT3 on for to become good task- model"?
7 RELATED WORK
Multi-fask learning Multi-task learning (Caruana 1997). broadly construed, aims to learn models and representations that can be utilized across diverse range of tasks, and offers natural approach training parameter-efficien deep models. Several works have shown that a single BERT model can obtain good performance across multiple tasks when jointly trained (Liu et al. 2019b] Clark al., 2019 (Stickland Murray 2019). Adapter layers, which are task-specific layers that read and write to layers of shared model (Rebuffi et al.) 20188) offer an alternative approach to multi-task learning that does not require access to all tasks during training, and have also been applied to obtain parameter-efficien BERT models (Houlsby al. 2019: Pfeiffer et al. 2020a
bl c related line of work targets extreme parameter saraamete--fffcienncy through task sentence representations that can be used without finetuning for downstream tasks (Le & Mikolov 2014 Kiros al. al. (2015] Wieting al. 2016 Hill et al. 2016 Arora et al. [0177] Conneau et al. [01]]017]] e et al. 2018 /Zhang et al. 2018 al.] 2020). [Reimers & Gurevych (2019) 99, building the earlier work of Conneau et al. (2017), show that BERT finetuned on natural language inference obtains sentence representations that perform well across multiple sentence-level tasks. These feature-based transfer learning methods are however generally outperformed by fully finetuned models (Howard
Subramanian
Ruder 2018).
Dataset distillation WWng al. 2018) tackles this question the context of vision models.
8

---

Under review as a conference paper at ICLR 2021
Model compression There has been much recent work compressing pretrained trained with self-supervision (see Ganesh et al. (020)) for a recent survey). A particularly promising line of work focuses obtaming smaller pretrained models (for subsequent finetuning) through weight pruning (Gordon et al. 2020; Sajjad et al.) 2020; Chen et al. 2020) and/or knowledge distillation
(Sanh et al. 2019 Sun al. 2019 Ture all] 2019 Jiao et al. 2019 Sun et al. (020). It would be
interesting to see whether our approach can be applied on top of these smaller pretrained models to for even greater parameler-efficiency,
Learning to prune Our work is closely related the line of work learning to prune pretrained models with differentiable relaxations of binary masks (Wang et al., (2119)] Zhao et al. [2020] Sanh etal. 2020 Radiya-Diis Dixit Wang 2020). While these works also enable parameter-efficient transfer learning. they generally apply the masks directly on the pretrained parameters instead of on the difference vector as in the present work.
Regularization towards pretrained models Finally, diff pruning also related works which regularize the learning process towards pretrained models for continual learning (Kirkpatrick et al.) 2017 Schwarz et al., 2018), domain adaptation (Wiese et al. 2017,] Miceli Barone et al. (0077). and stable finetuning (Lee et al., (200)) These works typically do not utilize sparse regularizers and target a different goal than parameter-effiiinnyy
8 CONCLUSION
We propose diff pruming as a simple approach for parameter-efficient transfer learning with pre- trained models. Experiments standard NLP benchmarks and models show that diff pruning can match the performance of fully finetuned baselines while requiring only a few additional parameters per task. We also propose a structured variant of diff pruning which provides further improvements. Future work will consider (i) applying this approach to other architectures (c.g. ConvNets for vi- sion applications), (ii) injecting parameter-effiiieng objectives directly into the pretraining process (to pretrain models that are better suited towards sparse transfer learning), and (iii) combining diff pruning with other techniques (e.g. adapters) to achieve even greater parameter-efficinncy.
REFERENCES
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A Simple but Tough-to- Bough-to-Beat Baseline for Sentence Embeddings. In Proceedings of ICLR, 2017.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. 2020.
Rich Caruana.
Multitask Learning. Machine Learning, 1997.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Con- stant, Mario Guajardo-Cespedess Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. Uni- versal sentence encoder for English. In Proceedings of EMNLP: Syslem Demonstrations, 2018.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang. and Michael Carbin. arXiv:2007.12223. 2020.
The Lottery Ticket Hypothesis for Pre-trained BERT Networks.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! Born-Again Multi-Task Networks for Natural Language Understanding. In Proceedings of ACL, 2019.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Pro- ceedings of EMNLP, 2017.

---

Under review as a conference paper at ICLR 2021
Andrew Dai and Quoe V. Le. Semi-Supervised Sequence Learning, In Proceedings of NIPS, 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL, 2019.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic BERT Sentence Emhedding. arXiv:2007.01852, 2020.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3, 1999.
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang. Deming Chen, Marianne Winslett, Hassan Sajjad, and Preslav Nakov. Compressing Large-Scale Transformer-Based Mod- els: A Case Study on BERT. arXiv:2002.11985, 2020.
Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews, of Weight Pruning on Transfer Learning. In Proceedings of Rep4NLP 2020 Workshop at ACL 2020, 2020.
Compressing BERT: Studying the Effects
Song Han, Huizi Mao, and William J. Dally. Deep Compression: Compressing Deep Neural Net- works with Pruning, Trained Quantization and Huffman Coding. Proceedings of ICLR, 2016.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of ACI,, 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, and Mona Attariyanand Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of ICML, 2019.
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classifica- tion. In Proceedings of ACL, 2018.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical Repurameterization with Gumbel-Softmant In Proceedings of ICLR, 2017.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TERyBERT: Distilling BERT for Natural Language Understanding. arXiv: 10351, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A.
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrassa-Barrwinsh Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 14:3521-3526, 2017.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urta- sun, and Sanja Fidler. Skip-Thought Vectors. In Proceedings of NeurIPS, 2015.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: Lite BERT for Self-supervised Learning of Language Representations. In Proceedings of ICLR, 2020.
Quoc V. Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. In Proceedings of ICMI, 2014.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective Regularization to Fine- tune Large-scale Pretrained Language Models. In Proceedings of ICLR, 2020.
Sang- Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang, Overcoming catastrophic forgetting by incremental moment matching. In Advances Neural Information Processing Systems. 2017.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of ACL, 2019a.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-Tase Deep Neural Networks for Natural Language Understanding. In Proceedings of ACI., 2019b.
10

---

Under review as a conference paper at ICLR 2021
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv: 2019c.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient Episodic Memory for Continual Learning. In Proceedings of NeurIPS, 2017.
Christos Louizos, Max Welling, Diederik P. and Kingma. Learning Sparse Neural Networks through Lo Regularization. In Proceedings of ICLR, 2018.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In Proceedings of ICLR, 2017.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Proceedings of NeurlPS. 2017.
Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regularization techniques for fine-tuning in neural machine translation. In Proceedings of EMNLP, 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Repre- sentations Vector Space. arXiv:1301.33: 2013.
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual Lifelong Learning with Neural Networks: Review. arXiv:1802.07566 2018.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In Proceedings of NAACL, 2018.
Jonas Pfciffer, Aishwarya Kamath, Andreas Ruckle, and Kyunghyun Cho amd Iryna Gurevych. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. arXiv:2005.00247, 2020a.
Jonas Pfeiffer, Andreas Ruckle, Clifton Poth, Aishwarya Kamath, Ivan Vulic. Sebastian Ruder, and Iryna Gurevych Kyunghyun Cho. AdapterHub: A Framework for Adapting Transformers. wwXiv:2007.07779 2020b.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-based Frame- work for Multi-task Cross-lingual Transfer. arXiv:2005.00033 00052, 2020c.
Alec Radford, Kantik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
Evani Radiya-Dixit and Xin Wang. How fine can fine-tuning be? Learning efficient language mod- cls. In Proceedings of AISTATS, 2020.
Colin Raffel, Noam Shazeer, Katherine Lee Adam Roberts, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. Journal of Machine Learning Research, 21, 2020.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 7eRO: Memory Optimiza- tions Toward Training Trillion Parameter Models. arXi::1910.0205.. 2019.
S. Rebuffi, A. Vedaldi, and H. Bilen. Efficient Parametrization of Multi-domain Deep Neural Net- works. In Proceedings of CVPR, 2018.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks. In Proceedings of EMNLP, 2019.
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor Man's BERT: Smaller and Faster Transformer Models, 034844,00403384 2020.

---

Under review as a conference paper at ICLR 2021
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter. In Proceedings of 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing. 2019.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement Pruning: Adaptive Sparsity by Fine-Tuning, arXiv:2005.07683, 2020.
Timo Schick and Hinrich Schutze. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. arXiv:2009.07//6 2020.
Jonathan Schwarz, Jelena Luketina. Wojeiech M. Czamecki, Agnieszka Straska-Barwwint Yee Whye Tch, Razvan Pascanu, and Raia Hadsell. Progress & Compress: A scalable frame- work for continual learning. In Proceedings of ICMI., 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual Learning with Deep Gener- ative Replay. In Proceedings of NeurIPS. 2017.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron- LM: Training Multi-Billion Parameter Language Models Using Model Par- allelism. 1XXiv:9900000533 2019.
Asa Cooper Stickland and lain Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of ICML, 2019.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning. In Proceed- ings of ICLR, 2018
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient Knowledge Distillation for BERT Model Compression. In Proceedings of EMNLP, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou Mobile- BERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of ACL,, July 2020.
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of ACI., 2019.
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Students Learn Better:
On the Importance of Pre- fre-training Compact Models. arXiv: 08962, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE:
A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR, 2019a.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexci A. Efros. 1:X1v:811111095: 2018
Dataset Distillation.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured Pruning of Large Language Models. arX1v:1910.04332. 2019b.
Georg Wiese, Dirk Weissenborn, and Mariana Neves. Neural domain adaptation for biomedical question answering. In Proceedings of CoNLL, August 2017.
John Wicting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards Universal Paraphrastic Sentence Embeddings. Proceedings of ICLR, 2016.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, abss/1910.0371, 2019.
John M. Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similarity Analysis of Contextual Word Representation Models. In Proceedings of ACL., 2020.
12

---

Under review as a conference paper at ICLR 2021
Figure Percentage of modified parameters attributable to each layer different runs SST-2 0.5% target sparsity. The layers are ordered from earlier to later (i.e. the embedding layer is shown at the top). The x-axis for each plot goes from 0% to 20%.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of NeurIPS, 2019.
Minghua Zhang, Yunfang Wu, Weikang Li, and Wei Li. Learning universal sentence representations with mean-max attention autoencoder. In Proceedings of EMNLP, 2018.
Yan Zhang, Ruidan He. Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An Unsupervised Sentence Embedding Method byMutual Information Maximization. In Proceedings of EMNLP, 2020.
Mengjie Zhao, Tao Lin, Martin Jaggi, and Hinrich Schutze. Masking an Efficient Alternative to Finetuning for Pretrained Language Models. arXiv:2004. 12406, 2020.
A APPENDIX
A.1 CONSISTENCY OF NONZERO PARAMETERS
Figure 3 shows the percentage of modified parameters attributable to each layer across 5 runs of SST- We find that there nonotrivial variation in sparsity across runs, but also degree of consistency. For example, the first layer is modified considerably more than other layers across all runs.
2.
13


Reviews:
[{"review_id": "y-n1aaUhRhD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present an interesting approach to learn task-specific models with only a few tunable parameters. They propose learning a diff vector with a sparsity constraint and then pruning the vector using magnitude pruning. They also impose a structured sparsity constraint by introducing a group penalty.\n\nThis work is interesting and important with the growing size of the pretrained models. It enables the model to learn a new task with relatively few parameters would be very beneficial. The experiments and analysis conducted by the authors is thorough.\n\nA few questions/thoughts which could improve the paper: \n* It would be good to list the epochs for training for each of the different approaches? Authors mention 2,3,4,5 epochs for training but would good to highlight the cost or savings of fine tuning with their approach.\n* In table 1, it would be also helpful to include some information about the finetuning steps/sec with the different approaches. This would help understand the tradeoff of memory vs compute for the proposed approach.\n* What’s the intuition for structured pruning performing better than non unstructured? Typically, enforcing some structure should have worse accuracy. If the structure seems to help, then the model should be able to learn structure without the group constraint as well.\n\nOverall, I recommend accepting the paper. As the size of pre-trained models is growing quite rapidly, research that investigates parameter sharing and adapting a pretrained model to a new task with few parameters is essential.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "li-u1Vjqkgi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work combines model pruning with transfer learning/multi-task learning in NLP. Instead of finetuning pretrained models on each individual task, the author propose to learn 'residual' parameters with sparse masks for each task independently, hence reducing per-task parameter requirements. The evaluation on GLUE shows some promising results with better efficiency compared to adapter networks. However, I am concerned with the motivation and applicability of the proposed method, such that I am left with the impression that the method may be hard to use in practice.\n\nPros:\n1. To the best of my knowledge, combining pruning on pretrained models to transfer learning in NLP is novel.\n2. Empirical results show the method can performance comparably with adapter networks but with fewer parameters.\n\nCons (see below for detailed questions):\n1. Although the method is proposed for 'multi-task' learning, it is not really evaluated on multi-task settings. The motivation and applicability are not very clear to me.\n2. Compared to adapter networks, the proposed method contains more hyper-parameters and is harder to tune in practice.\n\nQuestions:\n1. In section 2, the author introduced multi-task learning as the background. However, the proposed method is not really designed for multi-task learning (training multiple tasks simultaneously), but rather for transfer learning on multiple tasks independently. In practice, however, training some tasks together can improve performance (GLUE for example), but the proposed method is trained for each task independently and thus there is no positive transfer between tasks. So have you evaluated your method when trained multiple tasks simultaneously (where the shared parameters are jointly finetuned)? How does that work? This is an important extension that might be useful for the community.\n2. Please point it out if I miss this information, but how did you select your hyper-parameters for your approach? Besides, how sensitive they are compared to adapter networks?\n3. Prior work [1] has shown the resulting mask of pruning can be compared to evaluate task similarities. Did you have similar observations in your experiments?\n\nMissing reference:\nThe idea of adding 'residual' parameters for new tasks is not new in lifelong learning. It is good to mention [2] and [3]. [1] is also related of comparing pruning masks for pretrained models.\n\nTypo:\nPage 4, last line: QQC -> QQP\n\n[1] On negative interference in multilingual models: findings and a meta-learning treatment. Wang et al., EMNLP 2020.\n\n[2] Progressive neural networks. Rusu et al., arxiv 2016\n\n[3] BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. Wen et al., ICLR 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "keH6gGlHB9l", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes diff pruning, an alternative paradigm for parameter-efficient transfer learning of pre-trained models. Similar to adapters, diff pruning leaves the body of the pre-trained model unchanged. Rather than inserting additional task-specific parameters into the pre-trained model, diff pruning adds reparameterizes the parameters of the transferred model $\\theta_\\tau$ by adding a diff vector $\\delta_\\tau$ to them: $\\theta_\\tau = \\theta_{\\text{pretrained}} + \\delta_\\tau$. Parameter efficiency is achieved by regularizing $\\theta_\\tau$ to be sparse. The authors achieve this by using a relaxed mask vector to approximate the $L_0$ norm. They also propose a way to control for a specific sparsity rate via projection onto the $L_0$ ball after training and to enforce group sparsity that takes the model's structure into account. The approach is evaluated on the GLUE benchmark where it achieves competitive performance to full fine-tuning a BERT Large model and adapters while being more parameter-efficient than both of them.\n\nPros:\n1. The proposed method is intuitive and the different modelling choices are principled and well motivated.\n2. The method achieves strong results. It is competitive with full fine-tuning and more parameter-efficient than adapters, the prevalent approach for parameter-efficient transfer learning.\n3. The authors show how to effectively control the sparsity rate and incorporating structure via group sparsity brings further gains.\n4. The authors conduct extensive analyses, which pre-empted many of my questions, such as the variation across different sparsity masks, sparsity patterns across different tasks, etc. Overall, the analyses shed additional light on the characteristics and preferences of different tasks in transfer learning.\n\nCons:\n1. The approach is potentially more complicated than the baseline, so it is important that the authors open-source their code.\n2. The diff vector is distributed over the entire set of parameters of the model rather than focused in a few layers. This makes it potentially harder to combine the diff vectors from different tasks as can be done with adapters (see e.g. https://arxiv.org/abs/2005.00247) and to compose multiple diff vectors. \n\nQuestions:\n1. Does a visualization of diff vectors of different tasks (such as using t-SNE) reveal any interesting patterns?\n2. Are there any transfer settings that the addition of task-specific parameters can model but inserting layer-specific transformations via adapters cannot (or vice versa)? Adapters have been used to transfer across modalities such as languages (see e.g. Pfeiffer et al. (2020), https://arxiv.org/abs/2005.00052) and I am wondering whether the same would be possible by adding task-specific parameters.\n3. How long does your approach take to converge in comparison to the baselines? How much longer do you need to fine-tune with non-zero masks for magnitude pruning for sparsity control? What is the performance benefit of this further fine-tuning?", "extra_scores": {}, "extra_text": {}}, {"review_id": "h7S5ugPJwKK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies the problem of parameter-efficient transfer learning in the paradigm of pretraining/finetuning. The proposed method, diff pruning, can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5 of the pretrained model's parameters per task.\n\nI don't get the problem definition of this work. In other words, the authors need to better motivate and justify parameter efficiency. Given that all the parameters of a big model are used in downstream tasks, what are the benefits of only modifying a few parameters? Here are several possibilities.\n1. Does it speed up the finetuning process by modifying a few parameters? \n2. Does it speed up inference for downstream tasks?\n3. Another possibility is model size reduction, which leads to reduced storage costs. \"This approach can become parameter-efficient as the number of tasks increases as it only requires storing the nonzero positions and weights of the diff vector for each task. The cost of storing the shared pretrained model remains constant and is amortized across multiple tasks.\" Unfortunately, in real-world scenarios, inference of  large models like BERT_base/large is usually conducted on servers or in cloud, where storage is not a big issue while latency (speed) is critical. Inference in edge devices like phones, IoT sensors cares about storage, speed, and power consumption. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies a problem setup of parameter-efficient transfer learning for large-scale deep models. The approach consists of learning a diff vector with a sparsity constraint and then pruning the vector using magnitude pruning. A group penalty is also introduced to enhance structured sparsity. The main motivation is that for each new task, we only need to add a few parameters based on a pre-trained model without fine-tuning it.

The proposed approach possesses technical soundness and shows empirical efficacy for the studied problem setup. During the rebuttal and discussion phases, two of the reviewers raised two major concerns based on which they strongly disagreed with acceptance:
- The problem setup is not elaborated sufficiently and falls short of plausibility. An approach targeting at efficiency should either improve inference speed or reduce storage cost. Unfortunately, neither advantage has been well approached.
- The technical novelty is somewhat incremental, given the rich previous work on residual adapter, network re-parameterization, and network compression (pruning, sparsity etc.).

AC read the paper and agreed that, while the paper has some merit such as a better model for the particular problem setup, the reviewers' concerns are reasonable and need to be addressed in a more convincing way. For example, try to study a practical application in which the proposed approach is essential and useful for efficiency enhancement.
Decision: Reject