Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Authors: Rabeeh Karimi mahabadi
James Henderson
Sebastian Ruder
Venue: NeurIPS 2021 Poster (2021)
Keywords: parameter-compact fine-tuning methods
adapters
low-rank methods
hypercomplex multiplication layers
adapting large-scale language models

Abstract: Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods,  but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers.

Specifically, Compacter inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently as a sum of Kronecker products between shared ``slow'' weights and ``fast'' rank-one matrices defined per Compacter layer. By only training 0.047% of a pretrained model's parameters, Compacter performs on par with standard fine-tuning on GLUE and outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our code is publicly available at https://github.com/rabeehk/compacter. 

COMPACTER: Efficient Low-Ramn Hypercomplex Adapter Layers
Rabeeh Karimi Mahabadi EPFL University, Idiap Research Institute rabeeh. karimi 0iimi.iii.....mmm ch
James Henderson Idiap Research Institute j james henderson@idiap ch
Sebastian Ruder
DeepMind
ruder@google.com com
Abstract
Adapting large-scale pretrained language models to downstream tasks via fine-tuning the standard method for achieving state-of-the-an performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-infffiieee unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-effict efficiint fine-tuning methods, but these approaches either still require relatively large number of parameters or underperform standard fine-tuning. In this work, we propose COMPACTER, a method for fine-tuning large-scale language models with better trade-off between task performance and the number of trainable parameters than prior work. COMPACTER accomplishes this by building top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers.
Specifically, COMPACTER inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently sum of Kronecker products be- tween shared "slow" weights and "fast" rank-one matrices defined per COMPACTER layer. By only training 0.047% of a pretrained model's s parameters, COMPACTER performs par with standard fine-tuning fine-tuning on SuperGLUE and low fow-resource settings. Our code is publicly available at Llltttps://gtthbb com http://rbbeehhcooooee compptterr
GLUE and outperforms standard
Introduction
State-of-the-art pretrained language models (PLMs) in natural language processing (NIP) have used heavily over-parann wer-paramettrized representations consisting of hundreds of millions billions of parameters to achieve success on a wide range of NLP benchmahssss 22 3 4 These models are generally applied to downstream tasks via fine-tuning
With four parameters can elephant, and with five I can make him wiggle his trunk.
John Neumann
[] which requires updating all parameters and storing one copy of the fine-tuned model per task. This causes substantial storage and deployment costs and hinders the applicability of large-scale PLMs to werldworld applications. Additionally, fine-tuning of over--vramenn pverparraeterizee models on low-resource datasets has been shown to be subject instabilities and may lead to poor performance 7].
Inspired by John von Neumann's quotation, we ask, given that we have already learned general-purposs language representations via a PLM (i.e. we have fit our elephant), how many more parameters
35th Conference on Neural Information Processing Systems (NeurIPS 2021)

---

Compacier CompaciefpHM- PHM--da
Layer norm
Compacter
Addpter-Adepte
000
Adapter- Roykank
Adapter
T5
Adep Dhop
Feed forward projection
IADD
BIIF
Feed forward
Layer norm
0
GLUE Soore
Nontinearity
Adapter
Feed forward down projection
Muttihead attention
Transformer Layer
Adapter Layer
Figure 2: Left: Adapter integration in pretrained transformer model. Right: Adapter architecture. Follow- ing Houlsby et al. [I], we include adapters after the attention and feed- forward modules. During training. we only update layer normalizations and adapters (shown yellow), while the pretrained model is fixed.
PromptTuning
78
0.01
0.10
1,00
10.00
100.00
Percenlage the Trained Parameters Per Task (Relative T5)
Figure 1: The average score GLUE (y axis), percentage of trainable parameters per task (x axis, in log scale), and memory footprint (size of the circles) of different methods.
do we need to reach state-of-the-an art performance on standard NLP tasks. Specifically, we aim to develop practical, memory-efficien methods that train minimum set of parameters while achieving performance on par or better than full fine-tuning for state of-the-art NLP models.
Recent literature has introduced purameter-effiient fine-tuning methods. These approaches generally keep the pretrained model's parameters fixed and introduce a set of trainable parameters per task, trading off the number of trainable parameters with task performance. At one end of the spectrum, prompts, i.c. natural language descriptions of a task, together with demonstrations have been used to achieve reasonable performance without any parameter updates some benchmarks [8] but their performance generally lags behind fine-tuned models. They also require huge models to work well but choosing good prompts becomes harder with larger model sizes [9]. Soft prompt methods treat prompts as trainable continuous parameters, which are prepended to the inputs at the input layer or intermediate layers [10 1I [2] Such methods, however, often require large models to achieve good performance and are very sensitive to initialization and unstable during training.
The theoretically motivated low-rank methods train small number of parameters that lie in low-dimensional subspace using random projections 14. However, storing the random projection matrices causes substantial memory overhead and leads slow training times. At the other end of the spectrum, adapter methods [1 [15 that insert trainable transformations at different layers of the pretrained model require more parameters than the aforementioned approaches but are more memory--fficient and obtain performance comparable to full fine-tuning [[]
In this work, we propose COMPACTER, method for fine- the-tuning fange-scale language models with an excellent trade-off between the number of trainable parameters, task performance, and memory footprint, compared to existing methods (see Figure COMPACTER builds on ideas from adapters 0. low-rank methods [K], as well as recent hypercomplex multiplication layers [77. Similar to adapters, COMPACTER inserts task-specifie weight matrices into a pretrained model's 's weights Each COMPACTER weight matrix is computed as the sum of Kronecker products between shared "slow" weights and "fast" rank-one matrices defined per COMPACTER layer (see Figure 3). As a result, COMPACTER achieves a parameter complexity of (k++d) compared to (kd) for regular adapters, where the adapters are of size & d. In practice, COMPACTER trains 0.047% of PLM's parameters. On the standard GLUE [8] and SuperGLUE [19 benchmarks, COMPACTER outperforms other parameter-efficieet tne-ttunng methods and obtains performance par or better than full fine-tune fine-tuning. On low-resource settings, COMPACTER outperforms standard fine-tuning.
In summary, we make the following contributions: 1) We propose COMPACTER (Compact
Adapter) layers, a parameter-effiiient method to adapt large-scale language models. 2) We show that COMPACTER obtains strong empirical
peapiriaa performance GLUE and SuperGLUE. 3) We demonstrate that

---

COMPACTER outperforms fine-tuning in low-resouree settings. 4) We provide parameter complexity analysis of COMPACTER, showing that requires dramatically fewer parameters than adapters and fine-tuning. 5) We provide systematic evaluation of recent parameter-effcient fine-tuning methods terms of training time and memory consumption. We release our code to facilitate future work.
2
Background
We start by introducing the required background on the Kronecker product and adapter layers 1
2.1
Kronecker Product
The Kronecker product between matrix A E mRx and B g RPx9, denoted by A Be mpxx /4 is mathematically defined as:
411
a1 B
A& B=
(I)
am B
anf B
where azy shows the element in the 7th row and jth column of A.
2.2 Adapter Layers
Recent work has shown that fine-tuning all parameters of a language model can lead to a sub-optimal solution, particularly for low-resource datasets [6]. As an alternative, Rebuffi al. [15] and Houlsby et al. I propose to transfer a model to new tasks by inserting small task task-specific modules called adapter layers within the layers of pretrained model, depicted in Figure They then only train adapters and layer normalizations, while the remaining parameters of the pretrained model remain fixed. This approach allows pretrained language models efficiently adapt new tasks.
Each layer of a transformer model composed of two primary modules: a) an attention block, and b) a feed-forward block. Both modules are followed by a skip connection. As shown in Figure Houlsby ct al. [I] suggest to insert an adapter layer after each of these blocks before the skip connection.
Adapters are bottleneck architectures. By keeping the output dimension similar to their input, they cause no change to the structure parameters of the original model. The adapter layer A' for layer consists of down-prood wwwn-projection, D' E kxd GeLU non- non-linesn -ineariy [20), and up- projectionn U E Rddk where k the input dimension, and d is the bottleneck dimension for the adapter layer. Adapters are defined as:
A' (GEL((D (x)))++2
(2)
where x is the input hidden state.
3
Method
In this section, we present COMPACTER, a compact and efficient way to adapt large-scale PLMs.
Problem formulation We consider the general problem of fine-tuning large-scale language models, where we are given the training data D ((x" y/ {H with P samples. We assume we are also given a large- scale pretrained language model fo (.) parameterized by 0 that computes the output for input x Our goal is to fine-tune f.).) efficiently to enable the model to adapt to new tasks.
3.1 Compact and Efficient Adapter Layers
In this section, we introduce an efficient version of adapter layers, building on top of recent advances in parumeterized hypercomples multiplicution layers (PHM) [17] To the best of our knowledge, are the first to exploit PHM layers for efficient fine-tuning of large-scale transformer models. The PHM layer has a similar form as fully-connected layer, which converts an input E IR* to output E Rd.
y W +b.
3
3

---

Size of W
B
Parameter size of W
Independent one weights
Shared weight A
Independent rank one weights
Shared wright Ag
Shared parameters size
B
B
Parameter size of W
Size of W2
Figure 3: Illustration of generating weights of two different COMPACTER layers: W1 E BR (first row) and W2 E Rdxk (second row). We generate W, and W2 using W, = = Ai 8 B
Σi_1, @ ts (5) by computing the sum Kronecker products shared matrices A, and adapter-specific matrices Bi with i }n. and adapter index E {1,2). We generate each B by multiplying independent rank one weights. In this example r = 2, d 6, and 8
where W E RAxd The key difference is that in a PHM layer, W is learned as sum Kronecker products, Assume that and are both divisible by a user-defined hyperparameter E EZ Then, the matrix W in (3) is computed as the sum of n Kronecker products as follows:
W
i=1
where A, A and B; E x The PHM layer has a parameter complexity of O reducing
parameters by at most 1 17 (see $4).
3.2 Beyond Hypercomplex Adapters
Prior work indicates that some of the information captured in pretrained models can be ignored for transfer 22]. Similarly, redundancies have been observed in the information captured by adapters, with adapters in lower layers being less important [.. In addition, sharing adapters across layers leads comparatively small drop of performance for some tasks [23] Motivated by these insights, we propose the following two extensions to make hypercomplex adapters more efficient.
Sharing information across adapters Sharing all adapter parameters across layers is overall too restrictive and is not able to perform on par with fine-tuning or using regular adapters (23]: however, our decomposition of adapters into A; and B; matrices in Eq. (4) allows to be more flexible. Consequently, we divide our adaptation weights into shared parameters that capture general information useful for adapting to the target task and adapter-specific parameters that focus on capturing information relevant for adapting each individual layer. Specifically, we define A; as shared parameters that are common across all adapter layers while B; are adapter-specific parameters.
Low-rank parameterization Low-rank methods 13 44] have demonstrated that strong performance can be achieved by optimizing task in low-rank subspace. Similarly, we hypothesize that model can also be effectively adapted by learning transformations
low-rank subspace. To this end, we as a low-rank matrix, which is the product of two low-rank weights 8; E ER and 4 r 4, where r is the rank of the matrix Putting both extensions together, we propose the low-rank parameterized hypercomplex multiplication layer (LPHM)):
propose to parameterize B; E IR &
W Σ A X (,,0tt
(5)
In general, task, r can be set to a higher value Figure illustrates our method. Overall, the LPHM layer reduces
1 so that B, is rank-one matrix. Depending the complexity of the target
I We not factorize A; as they are small, shared between all layers, and factorization hurts performance. II factors per-parraetterized COMPACTER be used for overcomplete knowledge distillation [24]

---

complexity further to O + d) (see $4. The LPHM layer can also be seen as leveraging "slow" weights A; that are shared across adapters and capture general information and "fast" weights B; that learn adapter-specific information for adaptation of each individual layer [25
COMPACTER Based the above formulation, introduce COMPACTER layers, which replace the down-projection and up-projection layers in adapters as follows:
(x) -LPHM (GeLU(LPHM" D' (x)))
where the up-projection weights LPHM are computed as in (5) replacing the layer U in
Similarly, down-projection weights LPHM replace the layer D While the two adapters in each layer of a transformer have their own 8; and l rank-one weights, we share the A, across all layers and positions of the adapter layers.
Parameter Efficiency
In this section, we compare the number of parameters of COMPACTER with adapters.
Adapters parameters In the standard setting, two adapters are added per layer of transformer model []] Each adapter layer consists of 2kd parameters for the down and up-projection matrices (U', D`) respectively where is the size of the input dimension and is the adapter's s bottleneck dimension. The total number of parameters for adapters for a transformer model with I. layers of both an encoder and decoder is, therefore, (2k(d),),, which scales linearly with all three variables.
PHM-ADAPTER parameters In the conventional PHM layer 17 as depicted in Eq. 4). parameters of A; E =R" and B; E E define the degree of freedom for W as ni( kud 12 +n²) kut +n With the mild condition that kd > n². n² then kd dominates and the overall parameter size of the PHM layer in (4)
( kd This condition satisfied for typical values for adapters, PHM layers, and large-scale PLMs such as Tarlarge, with hidden size k == 1024, adapter hidden size de (24,32,48, 48,966, and n n= 2,4,8,122 12. Hence, the PHM layer offers parameter reduction of almost 1 compared to standard fully-connected n
n
layers, which are (kk) F
Similarly, employing PHM layers for modeling down and up- pp-projection matrices offers a parameter reduction of almost n. Each adapter with a PHM layer has total 2( kd + n²*) parameters. For a Transformer model with L layers, the total number of parameters of
is +n²). COMPACTER parameters COMPACTER shares the trained weight matrices {A,) in (5) consisting of n parameters across all layers. COMPACTER also has two rank-one weights for each adapter, $4,4 (5) consisting of & + parameters, resulting in total of 2n #) parameters for down and up-projection weights. Therefore, the total number of parameters COMPACTER is AL(k+d)) +n for transformer with L layers in the encoder and decoder.
In settings with a large number of layers, the dominant term is 4L(k d). Therefore, with a mild condition that 4L(k+d) >> n², COMPACTER has a complexity of (k-+d), which is far more efficient compared to adapters' (kd)) and FHIM-DDAPED ' O kd complexity respectively. In settings where is large. the number of parameters for shared weight matrices (A,) for all layers remain constant in COMPACTER with total of n parameters while this scales linearly with the number of layers L for PHM and adapter layers. As an example, in the T5DASE model with 222M parameters [3], COMPACTER only learns 0.047% of the parameters, and maintains comparable performance to full fine-tuning
5 Experiments
Datasets Following Raffel al. 3, we evaluate the performance of the methods on the GLUE 18 and SUPERGLUE 19 benchmarks. These benchmarks cover multiple tasks of paraphrase detection (MRPC, QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI, CB), linguistic acceptability (CoLA), inesttion-answering (MultiRC, ReCoRD, BoolQ), word sense disambiguation (WiC), and sentence completion (COPA) As the original test sets are not publicly
Even for smaller models where the n term dominates, observe substantial reduction of parameters compared adapters.
'Following Devlin et al. (2). Raffel et al. 3. as common practice, we do not experiment with WNLI 26 due to its adversarial nature with respect to the training
5

---

available, we follow Zhang et al. 227 and split off Ik samples from the training set that we use for validation, while we use the original validation data as the test set. For datasets with fewer than 10k samples (RTE, MRPC, STS-B, CoLA, COPA, WiC, CB, BoolQ, MultiRC), we divide the original validation set in half, using one half for validation and the other for testing.
Experimental details We use the state-of-the-art encoder- TS model [3] as the underlying model for all methods in our experiments. For computational efficiency, we report all results on TARSE models (12 encoder and decoder layers and 222M parameters). We use its HuggingFace PyTorch implementation 28). We fine-tune all methods for epochs on large datasets and 20 epochs for low-resource datasets of GLUE (MRPC, CoLA, STS-B. B, RTE, BoolQ, CB, COPA, WiC) to allow the models to converge
(27). For all adapter-basee based methods, we experiment with adapters of bottleneck size of {96, 48, 24). We save a checkpoint every epoch for all models and report the results for the iyper-parameters performing the best on the validation set for each task. For the PHM layers, we use the PyTorch implementation of Le et al. (29). We include low-level details in Appendix [A] For our methods, we experiment with n 12) and report the model performing the best. We include the results for all values of in Appendix B
Following Mahabadi et al. [30] we freeze the output layer of the pretrained model for all tasks across all methods 5 al. we update the layer normalization parameters for all methods where applicable
We show the results with fine-tuning the output layer in Appendix Following Houlsby
5.1 Baselines
We compare against several recently proposed purumeter-efficeent fine-tuning methods:
T5RASE We compare our method to the standard practice of fine-tuning T5, where we fine-tune all parameters of the model on each individual task.
ADAPTER We compare to a strong adapter baseline I. which adds adapters for each task after the feed-forward and attention modules in each transformer block of T5.
PFEIFFER- AAPPER Pfeiffer et al. [31 propose a more efficient adapter variant, which keeps only one of the adapters in each layer for better training efficiency. We experimented with keeping either adapter and found keeping the adapter after the self-attention module in each layer to perform the best.
ADAPTER-LOwRAR RANK We parameterize each adapter's 's weight as product of two rank-one weights.
PROMPT TUNING Prompt tuming [12 is the successor variant of Li and Liang [10), which prepends a randomly initialized continuous prompt to the input (PROMPT TUNING-R). We also compare to variant, which initializes prompts using token embeddings of the pretrained language model's vocabulary (PROMPT TUNING-T) T) 12]
INTRINSIC-SAID The Structure Aware Intrinsic Dimension [4] fine-tunes the model by reparame-
terizing the parameters in llwwer-dimensional subspace 0e (d' D): 0.- =0P. +X, where parameter f.0 are the pretrained model's parameters and P Pe R= RR. " is random linear projec- tion via the Fastfood transform 33.. They then consider the total number of weight matrices in the PLM, m, and attribute weight each of them, resulting in X E RMm in total by trading m parameters from the low dimensional space 0d' EIR.". Then, the total trainable parameters are 0d' ERR -m and A.
ADAPTERDROP We apply the method of Rücklé et al. 23 which drops the adapters from lower transformer layers for a better training efficiency to T5 with ADAPTER. Consequently, we drop adapters from the first five layers of both the encoder and the decoder in T5
BITFIT Cai al. 33 propose to freeze the weights and only train the biases. By not storing intermediate activations, this method enables substantial memory savings. Ravfogel el. [34] study similar method for PLMs that fine-tunes only the biases and the final output layer!
'This is much more efficient as the output layer includes 1 11.1% of the parameters TSBASE. Tasks are formulated in text-to-text format so the model can be applied to them without learning new output layer B
We note that this is in contrast to the original adapter setting, which used an encoder-only masked PLM ([]. 'For BITFFT, we only update the biases. For PROMPT TUNING, the entire model is frozen.
"Note that in the HuggingFace T5 implementation. the biases in layer normalizations, linear layers, the output layer and self-attention layers are removed. We re-introduce these biases for BITFIT.

---

Table 1: Performance of all models the GLUE tasks. For each method, report the total number of parameters across all tasks and the number of parameters that are trained for each task as a multiple and proportion of BASE model [3. For MNLI, we report accuracy on the matched validation set. For MRPC and QQP, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation coefficients. For CoLA, we report Matthews correlation. For all other tasks, we report accuracy. Bold fonts indicate the best results. For the results with t, due to insatiability during training, we restarted experiments with 6 random seeds and report the best. For INTRINSIC- d' is set to 20K.
Trained
#Total params
Method
params/ per task
CoLAA SST-2 MRPC
QQP
STS-B
MNLI QNLI RTE
Avg
Baselines
TSKSE
8.0x 1
100%
61.76
94.61
90.2093.06
91.63/8.88
89.68/89,99
97
86.78
93.01
71.94
86,50
ADAPTER
1.065
0.832%
64.02
93.81
85.29/89.77 73 90. 20.18/87.22 20
90.73/91 0.02
86.49
93.21
71.94
85.78
PREIFFER ADAPTER
1,032
0.427%
62.9
93,46
86. 06.76/90.85
90.14/87.15
91.13/91.34 1.34
86.26
93.30
76.26
86.32
ADAPTERDROP
1,038
0.494%
62.7
93.58
86.27/90.60 60
90:2/87.25
91.37/91.61
86.27
93.23
71.22
85.85
ADAPTER LOWRANK
1,004
0.073%
59.19
93.69
88.24/91
90.23/87.01
90.8/91.33
85.8
92.9
73.38
85.82
PROMPT TUNING-R
1,003
0.034%
0
87.61
68.14/81 LS
88, 93.93/85.55
90,25/90.59
46,831
92.33
54.68
71.49
PROMPT TUNING-T
1.003
0.034%
10.59
90.94
14/81/5
89.69/86.14
89.84/9.221
81.46
92.75
54.68
75.95
INTRINSIC-SAID
1.001
0.009%
58.69
94.15
88.24/91.78
78 90. 20.22887.13
90.06/90.4
85.23
93.39
70.50
85.45
BITFIT
1.010
0. 126%
58. 16
94,15
86, 86.76/90.53
90 90.06/86.99
90.88/91 1226
85.31
92.99
67.63
84.97
Proposed Methods
PII ADAPTER (n 12)
1.013
0.179%
57.35
94.50
91.67/93.86
90.25/87.05 90.45/90.84
85,97
92.92
75.54
86.40
COMPACTER 4m==4)
1.004
0.073%
63.75
93.00
89.22/92.31
90.23/87.03
90.31/90,74 85.61
92. 88
77.70
86.62
(1
1.002
0.047%
61.27
93.81
90.69/93.33
97.17/86.93 90.46/90.93
85.71
93.08
74.82
86.47
5.2 Our Methods
PHM-ADAPTER We learn the weights of adapters using PHM layers as in 4 To our knowledge, we the first who exploit the idea of PHM (17) for efficient fine-tuning of large-scale language models.
COMPACTER We learn adapter weights using LPHM layers as described in (5) We also explore a variant where we only keep the COMPACTER layer after the block [
layer in each transformer
5.3 Results on the GLUE Benchmark
Tshow the results on GLUE with T5BASE (see Appendix for results on T5SMALL). COMPACTER and COMPACTER+ outperform all previous parameter-efficiee methods and perform on par with full fine-tuning while only training 0.07% and 0.047% of parameters respectively. We now discuss the different methods in detail.
Seapter-based methods For ADAPTER, not fine-tuning the classifier hurts the performance substantially 785.78 versus 86.48; cf. Appendix(C). PFEIFFER- ADAPTER, which adds adapters only after the self-attention module outperforms the standard ADAPTER while being more parameter- efficient. ADAPTERDROP obtains lower performance than fine-tuning, demonstrating that adapting the lower layers of an encoder-decoder T5 model is important for its performance. Additionally, MHAPTER-LOWRAND is not expressive enough perform well on this benchmark.
Prompt tuning and BitFit For PROMPT TUNING, we observe high sensitivity to initialization and learning rate, as also confirmed in [10]. We experimented with multiple random seeds but performance lags behind tne-tuning substantially. in particular on low resourseee datasets. This can be explained by the low flexibility of such methods as all the information needs to be contained in the prefixes. As a result, the method only allows limited interaction with the rest of the model and good performance requires very large models [1]. In addition, increasing the sequence length leads to memory overhead (see $5.5) and the number of prompt tokens is limited by the number of tokens that can fit in the model's maximum input length, which makes such methods less flexible and unsuitable for dealing with large contexts. Similarly, BITFIT performs worse than fine-tuning, especially on low-resource datasets.
Intrinsic-SAID Interestingly, the average performance of INTRINSIC-SA SAID, which fine-tunes only 0.009% of a model's 's parameters is only 1.05 points below the fine-tuning baseline. However, this method has two practical drawbacks, a) storing the random projection matrices results in substamtial
We found this slightly outperform keeping the COMPACTER layer after the self-attentic layer instead.
7

---

Table 2: Performance of all methods on the SUPERGLUE tasks. For each method, we report the total number of parameters across all tasks and the percentage of parameters that are trained for each task as a multiple and proportion of BASE model A] For CB, we report accuracy and F1. For MultiRC, we report Fl over all answer-options (FI,)) and exact match of each question's is set of answers (EM) [9]. For ReCoRD, we report F1 and EM scores. For all other tasks, we report accuracy. For
PNTRINSIC-SAID. d' is set to 20K. Bold fonts indicate the best results in each block.
#Total Trained params
Method
params/
BoolQ CB
COPA MultiRC
ReCoRD
WIC
Avg
per task
Baselines
TSRAS:
6.0x
100%
81.10
85.71/78.21
52.0
68.71/47.0
74.26/73.33
70.22
70.06
ADAPTER
1.049
0.832%
82.39
85.71/73.52
52.0
72.75/53.41
74.55/73.58
67.08
70.55
PFEIFFER-ADAS ADAAPEER
1.024
0.427%
82.45
85.71/75.63
54.0
72.53/51.76
74.69/73.70
68.65
71.01
ADAPTERDROP
1.028
0.494%
82.26
85.71/75.63
42.0
72.92/53.30
74.68/73.70
68.34
69.84
ADAPTER-LOWRANE
1.003
0.073%
80.31
78.57/55.37
54.0
72.58/51.98
74.77/73.87
64.58
67.34
PROMPT TUNING-R
1.002
0.034%
61.71
67. 67.86/46.99
48.0
59.23/16.33
75.27/74.36
48.90
55.41
PROMPT TUNING-T
1.002
0.034%
61.71
67.86/46.89
52.0
57.66/19.44 75.37/74.41
48,90
56.03
INTRINSIC-SAID
1.001
0.009%
78.72
75.00/51.8 833 54.0
69.98/52.78
74.86/73.91
65.83
66.32
BITFIT
1.008
0.126%
79.57
78.57/54.40
56.0
70.73/48.57
74.64/73.64
69.59
67.30
Our Proposed Methods
PHM-ADAPTER (n= 0
1.013
0.240%
80.31
85.71/73.52
44.0
71.99/51.65
74.62/73.60
67.40
69.20
COMPACTER (n- 12) 1.003
0.073%
78.59
96.43/87.44
48.0
70.80/49.67
74.49/73.54 65,20
71.57
COMPACTER++ (n= 12) 1.002
0.048%
78.84
92.86/84.96
52.0
70.68/50.99 74.55/73.50 68.03
71.82
Despite this, INTRINSIC-SAID provides insights regarding the effectiveness of low-rank optimization of pretrained language models [14]. which motivates the development of parameter-efficient methods such as COMPACTER.
memory overhead; b) very slow to train (see $5.5).
COMPACTER For our proposed methods, we observe fine-tuning the output layer for both PHM-ADAPTER and COMPACTER++ does not provide much performance difference (see Appendix C. PHM-ADAPTER reduces the parameters of ADAPTER from 0.83% to 0.17%% (with n = 12), being 4.64x more carameter-efffcient COMPACTER reduces the number of parameters to the remarkable rate of 0.073% while obtaining comparable results to full fine-tuning By removing the COMPACTER layer after self-atte attention, COMPACTER* obtains similar performance, while reducing the parameters to 0.047%. Adaptation without updating the layer normalization can be promising direction to reduce the parameters further, for instance by building on recent advances in normalization- models (35 which we leave to future work.
5.4 Results on the SUPERGLUE Benchmark
Table the performance of the methods on SUPERGLUE 19
We include the results for all values of in Appendix [ We observe similar pattern as GLUE in Ta COMPACTER and COMPACTER++ ++ perform substantially better compared to other parameter-effiiient fine-tuning methods and even outperform full fine-tuning while only training 0.073% and 0.048% of the parameters.
5.5 Efficiency Evaluation
In this section, we compare the efficiency of our proposed methods with various recently proposed parameter-compact fine-tuning methods under the same computation budget. To this end, we train all methods for epoch on the MNLI dataset. For cach method, we select the largest batch size that fits a fixed budget of the GPU memory (24 GB). For all adapter-based methods, we fix the adapter size to 24. For PROMPT TUNING,
set the number of prefix tokens to 100. For AITRINSIC-SAID we set = 1400. Finally, we set n = 4. In Table 3 we report the percentage of trained parameters per task, training time per epoch, and memory usage of each method. Moreover, Figure [ shows the trade-off between quantitative performance, percentage of trained parameters, and memory footprint.
Our approaches have several attractive properties. Based on our analysis in Table COMPACTER and COMPACTER++ obtain the best combination of high GLUE score averaged across all tasks, plus a sub- stantially lower number of parameters (0.073% and 0.047% respectively). In addition to COMPACTER++

---

Table Percentage of trained parameters per task, average peak memory and training time for all methods. A% % is the relative difference with respect to full tining (T5 (A5B)s)) Lower is better.
Trained params/
Memory (MB)
Time/Epoch (min)
Method
%
A %
per task
BABEE
100%
167.99
42.13
ADAPTER
0.832%
124.02
-35,45%
31.81
-24.50%
PFEIFFER-ADAS
0.427%
118.4
-41.8%%
28.19
-33.09%
ADAPTERDROP
0,494%
119.41
-00.6%%
28.08
-33.35%
0.073%
123.8
-35.69%
32.71
-22.36%
PROMPT TUNING
0.034%
222.27
24.42%
44.54
5.72%
INTRINSIC-SAID
0.009%
285.40
41.14%
144.01
241.82%
BITFIT
0.126%
102.31
-64.20%
27.36
-35.06%
PHM-ADAPTER
0.179%
123.93
-35.55%
35.55
-15.62%
COMPACTER 0.073%
123.91
-35.57%
36.48
-13.41%
COMPACTER++ 0.047%
118.35
-41.94%
30.96
-26.51%
performing well, its memory requirement is the second best among all methods, reducing memory usage by -4194% compared to TSASSE COMPACTER and COMPACTER++ also speed up training substantially, by -13.41% % and 226551 16.51% relative to TASE On the other hand, BITFIT, by not storing intermediate activations, has the lowest memory requirement (-64.2% relative TASE) and is the fastest (-35.06% relative to TA at the cost of lower quantitative performance (1.53 points lower; see Table
Methods relying pruning adapters, i.c., PFEiFTER-ADAPTES and ADAPTERDROP reduce the memory overhead and improve training time. However, their number of parameters almost an order of magnitude more compared to COMPACTER++, with 9.1 and 10.5x more parameters respectively. Moreover, although, PFEIFFER-ADAPEER performs on par with full fine-tuning with slight degradation ( ADAPTERDROP obtains lower performance (-0.65 less average across all tasks.). We note that dropping adapters from transformer layers is general technique and could be applied to COMPACTER for improving efficiency even further, which we leave future work. Similarly, although ADAPTER-L Low R ANK reduces the memory overhead and improves the training time, it obtains lower performance (Table (-0.68 less average across all tasks.).
At the other end the spectrum, INTRINSIC-SAID and PROMPT TUNING methods have the lowest number of parameters. However, they both come with high memory overhead (41.14% and 24.42% relative to full fine-tuning (T5RAsE) respectively), are slowest to train, and their performance substantially lags behind full fine-tuning (see Table ID. For PROMPT TUNING, high memory costs are due to the fact that the computational complexity self-attention, which requires storing the full attention matrix for gradient computation, scales quadratically with the sequence length [36). For INTRINSIC-S SAD,, the high memory requirement is due to storing large random projection matrices, which limits the application of INTRINSIC-SAIT for fine-tuning large-scale PL.Ms. Moreover, computing projections via FastFood transform, although theoretically possible in O(Dogdd") [32], slow in practice even with CUDA implementation. For pretrained language models with large number of parameters, allocating random projections for the full parameter space is intractable. While using Fastfood transform partially ameliorates this issue reducing the memory usage from O(d") to O()), the memory issue with such methods remains unresolved.
Overall, given the size of large-scale transformer models with millions and billions of parameters, such as T5 [3,, efficient memory usage is of paramount importance for practical applications. COMPACTER and COMPACTER++ offer great trade-off in terms of performance, memory usage, and training time. With regard to our inspiration of von Neumann's 's quotation, we thus find that only comparatively small number of additional parameters necessary for the practical and efficient adaptation of PLMs.
5.6 Low-resource Fine-tuning
COMPACTER++ has substantially fewer parameters compared to T5 In this section, we investigate whether this could help COMPACTER++ to generalize better in resource-limited settings. subsample each dataset of GLUE for varying sizes in the range 1100,500,1000,,000,,000 Figure shows the

---

Average scores on GLUE
84
80
78
+ TSRASE
Compacter++
76
1000
2000
3000
4000
# Samples per task
Figure 4: Results GLUE for the various number of training samples per task
(100,500,1000 100,,004,0000,,004400000,0, We show mean and standard deviation across seeds.
results, COMPACTER++ substantially improves the results in the low-resource setting, indicating more effective fine-tuning in this regime.
Related Work
Adapters Adapters have recently emerged as a new paradigm for fine-tuning pretrained language models [1] In another line of work, Üstün al. (37 proposed a multilingual dependency parsing method based on adapters and contextual parameter generator networks (38) where they generate adapter parameters conditioned trained input language embeddings. This, however, leads to large number of additional parameters compared to the base model. Contemporaneously, Mahabadi et al. 30 use a single compact hypernetwork allowing to generate adapter weights efficiently conditioned on multiple tasks and layers of a transformer model. Pilault et al. [39] also proposed a task-conditioned transformer for multi-task learning which is less parameter-efficieen.. The aforementioned work is complementary to COMPACTER, and one could potentially combine COMPACTER with contextual parameter generation to generate adapter modules. Compared to Mahabadi et al. 30. COMPACTER+ reduces the parameters by 6.2x.
Hypercomplex representations Deep learning advances in the hypercomplex domain in nascent stage, and most work is fairly recent [40 41 42
Replacing matrix multiplications standard networks with Hamilton products that have fewer degrees of freedom offers up a 4x saving of param- eter size in a single multiplication operation 42 44). Very recently, Zhang et al. [17] extend such meth- ods way that they could reduce the parameters of fully connected layer under mild condition to 1/n, where n is a user-specified parameter. To the best of our knowledge, there is no previous work that attempts to leverage the hypercomplex space for efficient fine-tuning of large-scale language models.
43
4
Other parfmeter-efficient models et et al. B and Aghajanyan et al. 14 study training models in -iw--eenniionll randomly oriented subspace instead of their original parameter space. Another recent line of work has shown that pretrained models such as BERT are redundant their capacity, allowing for significant sparsification without much degradation in end metrics 45 46 47 Such methods, however, remain not well supported by current hardware and often perform worse compared dedicated efficient architectures (48..
7 Conclusion
We have proposed COMPACTER, a light-weight fine-tuning method for large-scale language models. COMPACTER generates weights by summing Kronecker products between shared "slow" weights and "fast" rank-one matrices, specific to each COMPACTER layer. Leveraging this formulation, COMPACTER reduces the number of parameters in adapters substantially from (kd) to Okk dd). Through extensive experiments, we demonstrate that despite learning 2127.66x fewer parameters than standard fine-tuning, COMPACTER obtains comparable or better performance in full-data setting and outperforms fine-tuning in data-limited scenarios.
10

---

Acknowledgements
We are grateful to Dani Yogatama for feedback on a draft this manuscript. The authors would like to thank Tuan Le for his assistance in reproducing the results of Zhang al. [17. We would like to also thank Armen Aghajanyan for his assistance to reproduce the results his work [4]. We thank Jue Wang for his comments on an earlier version of this paper. The authors are grateful to Brian Lester, Rami Al-Rfou, Noah Constant, and Mostafa Dehghani for their assistance. Rabeeh Karimi Mahabadi was supported by the Swiss National Science Foundation under the project Learning Representations of Abstraction for Opinion Summarization (LAOS), grant number "FNS-30216".
References
[1] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiient transfer learning for nlp. In ICML, 2019.
[2] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI,
[3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020.
[4] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 16692, 2019.
[5] Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In ACL, 2018.
[6] Matthew E Peters, Sebastian Ruder, and Noah Smith. To tune or not to tune? adapting pretrained representations diverse tasks. In RepL4NLP, 2019.
[7] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv preprint ar 20Xiv:2002.06305. 2020.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz I. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
[9] Ethan Perez, Douwe Kicla, and Kyunghyun Cho. True Few-SFrt with Language Models. arXiv preprint 2rXiv:2105.11441 2021. UtL.bttp://arrii.rrrg//
11447
[10] Xiang Lisa Li and Percy Liang. Prefix- Prefix-tuning: optimizzig prompts for generation. ACL, 2021.
[11] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. ACI., 2021.
[12) Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP, 2021.
[13] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In /CLR, 2018.
[14] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. ACL, 2021.
11

---

[15] Sylvestre- Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi-domain deep neural networks. In CVPR, 2018.
[16] Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficient transfer learning. In EMNLP Findings, 2020.
[17] Aston Zhang. Yi Tay. SHUAI Zhang, Alvin Chan, Anh Tuan Luu, Siu Hui, and Jie Fu. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. In ICLR, 2021.
[18] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer I Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[19] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: a stickier benchmark for general-purposs language understanding systems. In NeurlPS, 2019.
[20] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.084/1 0,..11666..811,,..1 2016.
[21] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi Revisiting Faw-le BERT Fine-tuning. In ICLR, 2021.
[22] Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. Rethinking Embedding Coupling in Pre-trained Language Models. In ICLR, 2021.
[23] Andreas Rückll Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nits Reimers, and
Iryna Gurevych. AdapterDrop: On the Efficiency of Adapters in Transformers. EMNLP, 2021.
[24] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterizztion In ICML, 2018.
[25] Yeming Wen, Dustin Tran, and Ensemble and Lifelong Learning. ICLR, 2020.
and Jimmy Ba. BatchEnsemble: An Alternative Approach to Efficient
[26] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR, 2012.
[27] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi Revisiting fample bert tne-tuning. In ICLR, 2021.
[28] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers:
natural language processing. In EMNLP: System Demonstrations, 2020.
[29] Tuan Le, Marco Bertolini, Frank Noé, and Djork-Amé Clevert. Parameterized hypercomplex graph neural networks for graph classification. ICANN, 2021.
[30] Rabeeh Karimi Mahabadi, Sehastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. In ACI. 2021.
[31] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Cho Kyunghyun, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. FACI. 2021.
[32] Quoe Le, Tamás Sarlós, and Alex Smola. Ristfood-approxinng time. In ICML, 2013.
kernel expansions in loglinear
[33] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. NeurIPS, 2020.
[34] Shauli Ravfogel, Elad Ben-Zaken, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for mansformer-basee masked languagemodels. arXiv preprint arXiv:2106.10199
12

---

[35] Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. high-performance large-scale image recognition without normalization. ICML, 2021.
[36] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 20Xix:2000604766., 2020.
[37] Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. Udapter: Language adaptation for truly universal dependency parsing. In EMNLP, 2020.
[38] Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. Contextual parameter generation gennerationn for universal neural machine translation. In EMNLP, 2018.
[39] Jonathan Pilault, Amine El hattami, and Christopher Pal. Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. ICLR, 2021.
[40] Chase Gaudet and Anthony S Maida. Deep quaternion networks. In IJCNN, 2018.
[41] Titouan Parcollet, Ying Zhang, Mohamed Morchid, Chiheb Trabelsi, Georges Linarès, Renato de Mori, and Yoshua Bengio. Quaternion convolutional neural networks for end-to-end automatic speech recognition. In Interspeech, 2018.
[42] Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi, Renato De Mori, and Yoshua Bengio. Quatermion recurrent neural networks. In ICLR, 2018.
[43] Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In E 2018.
[44] Yi Tay, Aston Zhang, Anh Tuan Luu, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. Lightweight and efficient neural natural language processing with quaternion networks. ACL, 2019.
[45] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang. and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. NeurlPS, 2020.
[46] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When BERT Plays the Lottery, All Tickets Are Winning. In EMNLP, 2020.
[47] Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. In DeepLo, 2019.
[48] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint wrXXiv2003.03033 2020.
[49] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. TACL, 2019.
[50] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013..
[51] William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In /WP, 2005.
[52] Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez- and Lucia Specia. SemEval-20177 task
1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval, 2017.
[53] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2018.
[54] Pranay Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
[55] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
13

---

[56] Roy Bar- Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second pascal recognising textual entailment challenge. Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006.
[57] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. 2007.
[58] Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
[59] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium Series, 2011.
[60] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung. 2019.
[61] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprchension over multiple sentences. In NAACL, 2018.
[62] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv: 1810. / 12885, 2018.
[63] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACI, 2019.
[64] Mohammad Taher Pilehvar and Jose Camacho-Collados. Wie: the word-in-context dataset for evaluating context-sensitive meaning representations. In NAACL, 2019.
[65] Karin Kipper Schuler. Verbnet: broad-coverage, comprehensive verb lexicon. PhD) Thesis,
2005.
[66]
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 1995.
[67] Thomas Wolf, Quentin Lhoest, Patrick von Platen, Yacine Jernite, Mariama Drame, Julien Plu, Julien Chaumond, Clement Delangue, Clara Ma, Abhishek Thakur, Suraj Patil, Joe Davison, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angie McMillan-Major, Simon Brandeis, Sylvain Gugger, François Lagunas, Lysandre Debut, Morgan Funtowicz, Anthony Moi, Sasha Rush, Philipp Schmidd, Pierric Cistac, Victor Muštar, Jeff Boudier, and Anna Tordjmann. 2020.
Datasets. GitHub. Note: http:/gghhh..ooo//ugggg////////e
14


Reviews:
[{"review_id": "oz96xvNQB7h", "reviewer": "Reviewer_Ndst", "summary": "The authors presented an efficient parameterization of task-specific adapters for parameter-efficient multi-task transfer learning with large pre-trained language models based on transformer architecture.  \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the method is well described and demonstrated with pre-trained T5-base on GLUE.  The idea is novel and its motivations from prior work are clearly stated.  That said, I do have a few criticisms that I hope the authors would address.  \n\n- **Model capacity**.  It is not clear whether Compacter is applicable to transformer LMs of different sizes, since only T5-base is demonstrated.  When hidden dimensions are smaller, are low-rank adapters still adequate?  \n- **Task difficulty**.  It is not clear how the applicability of Compacter vary with difficulties of tasks.  Out of the GLUE tasks studied, how does the effectiveness of Compacter correlate with, say, word-order sensitivity (arXiv:2012.15180)?  What about even harder tasks and longer sequences?  Any results on SQuAD?  \n- **Parameter efficiency**.  Though the authors conducted a comprehensive comparison on parameter efficiency, some other approaches might be even more efficient in parameters--for example, it has been shown that BERT can be fine-tuned for any GLUE task by setting a small fraction of pre-trained weights to zero, i.e. _sparsification as fine-tuning_ (arXiv:2004.14129), in which case only the row/column indices of the small fraction of weight entries are necessary to store in addition to the pre-trained weights.  \n- **Practical significance**.  The approach of adding task-specific adapters to pre-trained LMs, however parameter and/or compute efficient the adapters are, still leads to a computational cost no less than that of the pre-trained network.  Pre-trained SOTA transformer models are grossly overparameterized and, just like other large NNs, can be compressed (by sparsification, quantization and/or distillation) for efficient inference in practice.  Thus, the adapter approach might not be as desirable as simultaneous compression/fine-tuning approaches such as _sparsification as fine-tuning_.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "cuAiS1j6on8", "reviewer": "Reviewer_CErC", "summary": "This paper develops a new parameter-efficient adapter layer, which extends the parameterized hypercomplex multiplication layers by sharing part of parameters across layers and decomposing adapter-specific parameters with two low-rank weights. Experimental results on the GLUE benchmark prove that the proposed approach outperforms other parameter-efficient finetuning methods and obtains performance on par with full finetuning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The authors propose a novel parameter-efficient adapter layer to adapt the large-scale language model.  \n2. The proposed method requires dramatically fewer parameters than adapters and finetuning, while outperforming other parameter-efficient fine-tuning methods.\n\nWeaknesses:\n1. The novelty of this paper is limited, which is a straightforward combination of parameterized hypercomplex multiplication layers, sharing information across adapters, and low-rank parameterization. These methods are widely known.\n\nDetailed Comments:\n\nIn general, this paper is easy to follow, the idea is very straightforward, and the parameter efficiency is analyzed in detail. Experimental results on the GLUE benchmark verify the effectiveness of the proposed method in terms of the parameter-efficient and accuracy. But the proposed method is not novel enough to me. It is a straightforward combination of some previous techniques, including parameterized hypercomplex multiplication layers, sharing information across adapters, and low-rank parameterization. The low-rank parameterization method actually could further reduce parameters of the original adapter layer or the AdapterFusion method. Adding these experimental comparisons could make the experiment more convincing. In addition, the authors only verify the effectiveness of the proposed method on NLU tasks. Actually, the proposed method can directly apply to harder NLG tasks. I am curious whether this method achieves high performance on downstream NLG tasks with such few parameters.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the detailed comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Ckj-6zK9P-i", "reviewer": "Reviewer_zZfs", "summary": "This paper propose an parameter-efficient fine-tuning method called Compactor by inserting a low-rank matrix decomposition adapter. Experiments support the claim. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Pros\nCompactor basically combines the low-rank decomposition with adapter. The writing is clear and idea is very easy to follow. Experimental results seem reasonable.\n\n# Cons\n1. Limited novelty: the idea is not new to the community. As in the reference [24], the low-rank decomposition has already demonstrated effectiveness in batch ensemble learning. Extending it to multitask fine-tuning is natural and intuitive.\n\n2. Weak results: authors only show results on GLUE benchmarks, which is not convincing. What about the performance on SuperGLUE, low-resource tasks and text generation?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 "}, "extra_text": {}}, {"review_id": "756HCb2N80g", "reviewer": "Reviewer_s8Se", "summary": "This paper propose COMPACTER, a parameter-efficient fine-tuning method  for large-scale language models. On GLUE datasets, COMPACTER only requires training 0.047% parameters without loss of accuracy, and moreover it outperforms standard fine-tuning in low-resource settings.\n", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Pros\n  * Method is novel and interesting. COMPACTER builds on ideas from adapters, low-rank methods as well as recent hypercomplex multiplication layers.\n  * Experiments and analysis are solid. It compares against a large number of baselines, and did thorough analysis on memory and training time. \n  *  Results are strong — it can achieve only 0.05% without loss of accuracy on GLUE, outperforming other parameter-efficient fine-tuning methods.  \n   * The paper is well-written.\n  * Parameter-efficient NLP method is an important direction.\n* Cons\n  * It will be great to also validate on datasets other than GLUE, e.g. Question Answering. \n* Missing references\n  * Guo et al, Parameter-efficient transfer learning with Diff Pruning", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper address efficient finetuning of pertrained models and investigate applications in NLP domains. The idea is to insert a parameter compact layer called Compacter in the pretrained models. By further parametering the Compacter layers with low-rank representation, its achieves a better trade-off between model performance, trained parameters and memory footprint compared with existing methods.

The idea is quite simple and the techniques are well explored in different scenarios by existing works. Thus most reviewers consider the technical contribution to be incremental. However, it demonstrates very good results in NLP pretrained models. Some reviewers were not satisfied with the limited experiments provided in the original submission. In the rebuttal, the authors was able to provide more results in a more complex data and model. And the results are in consistent with the results provided in the paper. The reviewers are generally satisfied with the rebuttal. Considering the technical limitation and the support from the reviewers, I recommend a weak accept and the authors to incorporate the new results and other comments from the reviewers into the revision.
Decision: Accept (Poster)