Venue: NAACL 2022

# Adaptable Adapters

Nafise Sadat Moosavi\({}^{1}\); Quentin Delfosse\({}^{3}\), Kristian Kersting\({}^{2,3}\), Iryna Gurevych\({}^{2,4}\)

###### Abstract

State-of-the-art pretrained NLP models contain a hundred million to trillion parameters. Adapters provide a parameter-efficient alternative for the full finetuning in which we can only finetune lightweight neural network layers on top of pretrained weights. Adapter layers are initialized randomly. However, existing work uses the same adapter architecture--i.e., the same adapter layer on top of each layer of the pretrained model--for every dataset, regardless of the properties of the dataset or the amount of available training data. In this work, we introduce adaptable adapters that contain (1) learning different activation functions for different layers and different input data, and (2) a learnable switch to select and only use the beneficial adapter layers. We show that adaptable adaptters achieve on-par performances with the standard adapter architecture while using a considerably smaller number of adapter layers. In addition, we show that the selected adapter architecture by adaptable adapters transfers well across different data settings and similar tasks. We propose to use adaptable adapters for designing efficient and effective adapter architectures. The resulting adapters (a) contain about 50% of the learning parameters of the standard adapter and are therefore more efficient at training and inference, and require less storage space, and (b) achieve considerably higher performances in low-data settings.1

Footnote 1: The code is available at [https://github.com/UKPLab/adaptable-adapters](https://github.com/UKPLab/adaptable-adapters).

\({}^{1}\) Department of Computer Science, The University of Sheffield

\({}^{2}\) Hessian Center for AI (hessian.AI)

\({}^{3}\) AI & Machine Learning Lab, \({}^{4}\) Ubiquitous Knowledge Processing Lab (UKP Lab),

Department of Computer Science, Technical University of Darmstadt

[https://www.ukp.tu-darmstadt.de](https://www.ukp.tu-darmstadt.de)

+
Footnote †: footnotemark:

The work has been mostly carried out during the employment at the UKP Lab, TU Darmstadt.

## 1 Introduction

Recent improvements in NLP are heavily skewed towards using larger pretrained models Roberts et al. (2020) and given their considerably better performances, using them is becoming unavoidable Kaplan et al. (2020). Their improvements, however, come at the cost of significant computational resources at training and inference times. For instance, the number of parameters in recent pre-trained models can vary from 110M in BERT-base Devlin et al. (2019) to 11 billion in T0 Sanh et al. (2022) to trillion parameters in Switch Transformers Fedus et al. (2021). Using such models for each downstream application requires a vast amount of storage, training, and inference computation budget that is not accessible to every user.

Instead of fine-tuning these massive numbers of parameters for each downstream task, we can use adapter architectures Houlsby et al. (2019); Pfeiffer et al. (2020). Adapters are lightweight neural network layers that are added on top of each layer of the pretrained model. As opposed to the standard model fine-tuning, in which all layers are fine-tuned for the target task, adapter-based tuning freezes the transformer layers and only trains the newly added adapter layers. Since the majority of parameters--i.e., the layers of the large pretrained model--are shared between different downstream tasks, the use of adapters results in parameter-efficient transfer learning. In addition to their parameter-efficiency, He et al. (2021) show that training adapter-layers (a) outperforms fine-tuning the whole model on low-data and cross-lingual settings, and (b) is more robust to overfitting.

Existing work suggests that (a) different layers of the pretrained models may capture different aspects of the form, syntax, or meaning of the input text Tenney et al. (2019); Clark et al. (2019), and (b) they may not be all needed for performing a given task Houlsby et al. (2019); Fan et al. (2020); Ruckle et al. (2021). In addition, adapter layers are initialized randomly. Therefore, it is not necessary to use the same adapter architecture for different downstream tasks and given different amounts of annotated data. However, existing works use the same adapter architecture for all the different input data, i.e., (a) one adapter layer on top of all the pretrained layers while using all the layers may not be necessary, and (b) the same activation func

tion for all the layers and different tasks while the best activation function may vary for different tasks (Delfosse et al., 2021).

In this paper, we propose a systematic approach for designing more adequate and flexible adapter architectures by introducing the adaptable adapter (_AA_). Adaptable adapters (1) use a learnable activation function--called Rational activation (Molina et al., 2020)--instead of a constant activation in adapter layers allowing the adapter model to learn different activation functions at different adapter layers and for different tasks, and (2) consist of a learnable switch at each adapter layer to determine the beneficial adapter layers during training and to only use the selected layers during inference.

We evaluate adaptable adapters on the GLUE benchmark (Wang et al., 2018) that consists of various text classification tasks. We perform evaluations based on different data settings in which different amounts of annotated examples are available for training. Our results show that adaptable adapters achieve on-par performances with the full adapter architecture while using considerably fewer adapter layers at the inference.

We further propose to use adaptable adapters for designing efficient adapter architectures--i.e., to only add an adapter layer to the layers that are selected by the adaptable adapter. We show that while the selected adapter architecture by _AA_, called _AA-focused_, is considerably more efficient at both training and inference times and requires less storage, it achieves on-par performances with the full adapter architecture when trained on all available training data and considerably outperforms it on low-data settings. In addition, we show that the selected adapter architecture by _AA_ transfers well across similar tasks and different data settings. Therefore, we can train _AA_ using a limited amount of training data, and for one of the tasks, and then use the resulting _AA-focused_ architecture for different data settings and other similar tasks.

Overall, the contributions of this paper are as follows:

* We propose adaptable adapters that introduce flexibility in adapter architectures by (a) selecting the beneficial adapter layers to use, and (b) learning the suitable activation function for each layer and each task.
* We propose to use adaptable adapters to design efficient adapters that require less training time, inference time, and storage space.
* We show that using fewer adapter layers with a learnable activation function considerably improves the performance on low-data settings.

## 2 Related Work

### Rational Activation

Rational activation functions, empirically introduced as Pade Activation Units (Molina et al., 2020), are learnable activation functions that can approximate common activation functions as well as learn new ones. The rational activation function \(R(x)\) of order \(m,n\) is defined as follows:

\[R(x)=\frac{\sum_{j=0}^{m}a_{j}x^{j}}{1+|\Sigma_{k=1}^{n}b_{k}x_{k}|} \tag{1}\]

where \(a_{j}\) and \(b_{k}\) are learnable parameters. These rational functions use an absolute value in the denominator to avoid potential poles, which will make the training unstable. Such rational activation functions provide stable training, as empirically shown in image classification and reinforcement learning (Molina et al., 2020; Delfosse et al., 2021). \(R(x)\) can be initialized to initially approximate any of the known activation functions or with constant functions. Molina et al. (2020) show that rationals outperform other commonly used activation functions in common image classification tasks. Rational activation functions are also integrated in Generative Adversarial Networks (Boulle et al., 2020). Delfosse et al. (2021) show that some of the layers in very deep pretrained Residual Networks tend to approximate activation functions' behavior, and we can achieve on-par or better performances with the full network by replacing some of the complete layers with rational activation functions. Similar to this observation, as we show in SS 5, using rational activation functions instead of a constant activation (ReLU) in adapters allows them to achieve high accuracy using a fewer number of adapter layers.

### Reducing Model's Size for Efficiency

Improving the efficiency of large pretrained models has received particular attention for the inference time. The argument is that the effect of training cost is limited, i.e., the model can be trained once but it will be used many times. However, the inference time has a wide impact on the everyday use of NLP models.

Existing approaches for improving the inference-time efficiency belong to two different categories: 

(a) the distillation and pruning techniques that create a smaller model for inference but often require re-training or fine-tuning the smaller model [19, 20, 34, 56], and (b) on-demand network size reduction at the inference time.2 There are two different approaches in the second category, namely layer dropping and early exiting.

Footnote 2: There is another category that requires changes in the models’ architectures. However, it would require re-training the large model. E.g., Sukhbaatar et al. (2019) propose new attention mechanisms that can process larger context with no additional computational or memory costs.

Fan et al. (2020) use layer dropping during the training that randomly drops the model's layers to make the model robust to the inference time layer selection. They show that it is possible to select sub-networks of any depth from large models at inference with limited impact on the performance and without the need for additional finetuning. Layer dropping was previously investigated by Huang et al. (2016) who propose to drop layers during training for regularizing the model and reducing the training time of deep convolutional networks. Ruckle et al. (2021) use layer dropping for adapter architectures. They show that by randomly dropping adapter layers during training, they can prune the adapter model on-demand at the inference time.

Schwartz et al. (2020) propose to add an output layer to each transformer layer. At inference time, while the model calculates the layer-wise representation, from the bottom layer to the top layer, it also makes the prediction using the associated classification layer. They use the output labels' scores of the classification layers as confidence scores to decide whether to exit early if the classifier is confident or to proceed to process the input with the next layers. This hierarchical architecture offers an inference time-accuracy tradeoff by setting the confidence threshold. The early exiting approach is similar to layer dropping in which the dropped layers are always from the last top layers.

All these approaches select the number of layers to drop and the dropped layers heuristically at the inference time with the goal of improving the inference time. Instead, the adaptable adapter is a systematic approach for selecting the useful adapter layers for the given task during training. Besides layer selection, an adaptable adapter allows for learning the desired activation function for different inputs. As we show, we can use adaptable adapters to design efficient adapter architectures with a considerably smaller number of training parameters with on-par or considerably higher performances, especially with larger models and in low-data settings.

## 3 Proposed Architecture

### Learnable Activation

Empirical observations of performances have led experts in several fields to use different activation functions for different tasks. Functions from the ReLU family are usually used for neural network-based visual computing, Tanh has been used in PPO for reinforcement learning, while GeLU has progressively been adopted in transformers. With the growth of the models, and the complexity of the tasks they are applied to, choosing one fixed activation function to equip the complete architecture is suboptimal. By using rational (8 2.1), we let the adapter layer learn the suitable activation function at each different adapter layer, task, and dataset. In adaptable adapters, we replace the constant activation function of each adapter layer--i.e., ReLU in the default configuration used in AdapterHub [7]--with rational.

Figure 1 shows a standard adapter layer as well as an adapter layer in adaptable adapters.

### Learnable Layer Selection

Houlsby et al. (2019) examined various choices of adapter architectures. They report that using

Figure 1: (a) a standard adapter layer with linear feedforward layers and a fixed activation, (b) an adapter layer in adaptable adapters with linear feedforward layers and a rational activation. Learnable parameters are shown within pink boxes.

 

two feedforward linear layers--one down-project and one up-project layer--results in good performances while only introducing a few parameters. Assuming \(d\) is the dimensionality of the input--i.e., the embedding size of the transformer layer--the down-project layer maps the input dimension to \(n\) where \(n<d\), and the up-project layer maps the input dimension back to \(d\). \(n\) is called the hidden size of the adapter. Each adapter contains a skip-connection that lets an adapter layer approximate an identity function, i.e., to pass the input of a transformer layer unchanged to the next layer. The learnable switches in adaptable adapter explicitly model the selection between the feedforward adapter layer and the identity function. By examining the switch probabilities we can determine the adapter layers that are beneficial for the overall performance of the model.

As mentioned in SS 1, existing work shows that different layers of the pretrained models capture different aspects of the input data, and not all of them are necessary for performing various tasks. Therefore, for different input data, different layers may be of different importance. Adding a learnable switch at each adapter layer provides a more systematic approach to determining the beneficial layers for each input task during training. We use the Gumbel Softmax (\({\cal GS}\)) estimator as an end-to-end differentiable switch (hard attention) to make the network attend to an element of a set. Assuming \(\pi_{i}\) are the probabilities of selecting each element of the set, i.e., \(\forall_{i}\pi_{i}\geq 0,\sum_{i}\pi_{i}=1\), \({\cal GS}\) estimates the hard attention \(y_{i}\) as follows:

\[y_{i}=\frac{exp((log(\pi_{i})+g_{i})/\tau)}{\sum_{j}exp((log(\pi_{j})+g_{j})/ \tau)} \tag{2}\]

where \(g_{i}\) are i.i.d. samples from a Gumbel distribution, and \(\tau\) is a temperature parameter. Setting \(\tau\) to small values results in distributions that are similar to categorical ones.

### Adaptable Adapters

The adaptable adapter (_AA_) is the combination of the learnable layer selection and the learnable activation function. The learnable layer selection--i.e., a Gumbel Softmax estimator--selects between an adapter layer, with no skip connection, and an identity function with zero parameters that passes the input without any changes to the next layer. The adapter layers in adaptable adapters consist of two linear layers--i.e., down-project and up-project layers--, and the non-linearity function between these two linear layers consists of a rational activation function. The adaptable adapter allows to learn different adapter architectures for different input data by (a) learning to use a subset of adapter layers, and (b) learning a potentially different activation function at each layer. Figure 3 shows the structure of an adapter layer in adaptable adapters.

## 4 Experimental Setup

### Datasets

We use the English text classification datasets from the GLUE benchmark Wang et al. (2019) including MNLI Williams et al. (2018), QQP3, QNLI Rajpurkar et al. (2016), SST-2 Socher et al. (2013), CoLA Warstadt et al. (2019), STS-B Cer et al. (2017), MRPC Dolan and Brockett (2005), RTE Dagan et al. (2006), and WNLI Levesque et al. (2011). Table 1 shows the number of training examples and the evaluation metric for each dataset.

Figure 2: The adaptable adapter layer that consist of a Gumbel Softmax to choose between an adapter layer with a rational activation and an identity function.

 



### Transformer Model

As the base model, we use the BERT-large model Devlin et al. (2019). BERT-large contains 24 layers, an embedding size of 1024, and a total number of 340M parameters.4

Footnote 4: The results for BERT-base are reported in the appendix. BERT-base contains 12 layers, an embedding size of 768, and 110M parameters.

### Adapter Models

BaselineAs a baseline adapter, we use the adapter layers with the Pfeiffer configuration from AdapterHub Pfeiffer et al. (2020). The adapter layers with the pfeiffer configuration are similar to the one in Figure 1, in which learnable parameters include two feedforward layers. For BERT-base, each pfeiffer layer consists of 73.7k parameters5 resulting in a total number of 884.7K. For BERT-large, the number of parameters for each adapter layer is 131K, and the total number of parameters is 3.1M. We see that as the underlying model gets larger, the number of parameters in adapters also increases notably. Therefore, adapter architecture selection using _AA_ is a potential solution to control this exponential increase to some extent.

Adaptable Adapter (AA)For the rational activation, similar to Molina et al. (2020), we use order \(m=5\) and \(n=4\) for rational. Therefore, the rational activation function only consists of ten learnable parameters. The rational activation can be initialized to initially estimate an existing function. Based on our preliminary experiments, using \(f(x)=1\) for initializing \(R(x)\) results in better performances on the GLUE benchmark.

For the Gumble-Softmax switch, we set the temperature parameter \(\tau\) to 0.1, and we initialize \(\pi_{i}\) to 0.5 for both inputs--i.e., the same initial probability for the rational adapter and the identity function.

AA-focusedWe can use the selected architecture by _AA_ for designing a new adapter architecture, i.e., to only include an adapter layer--with a rational function--at layers in which the switch has selected the adapter layer over the identity function. We call this architecture _AA-focused_. Note that compared to _AA_, _AA-focused_ is more efficient both at training and inference time, as it includes a fewer number of layers and no switch functions. It also requires less storage space for saving the new adapter weights. Also, training _AA_ includes both the architecture selection and training of the adapter layers, which are initialized randomly, simultaneously. As a result, as we see in our evaluations, _AA-focused_ achieves higher performances as its training is only focused on training the adapter layers.

AdaperDrop Ruckle et al. (2021)During training, AdapterDrop randomly drops the first \(n\) layers in which \(n\) varies for different iterations. At inference, \(n\) can be set to any desired number of layers. In our experiments, we select \(n\) based on the number of dropped layers by _AA_, i.e., the number of layers that are not selected by the switch functions.

### Experiments

We evaluate the models in different settings: (a) using full training data, and (b) low-data settings. For all the experiments, we consider 25% of the training data as the development set and use the official development sets as the test data. We perform the low-data evaluations when 100, 300, and 500 annotated examples are available.6 The test data is the same for all the evaluations. We run all the low-data experiments for 20 epochs and five different random seeds7. We report the average and standard deviation over the five different runs. When training on full datasets, the experiments are computationally very expensive using BERT-large. Therefore, for this setting, we only report the results using the first random seed. All experiments are done on one A100 NVIDIA GPU. All implementations are based on AdapterHub Pfeiffer et al. (2020).

Footnote 6: Selected training examples for low-data experiments are the same for all models given the same random seed.

Footnote 7: 42, 92, 111, 245, and 651.

## 5 Evaluation

Table 2 presents the results of _Baseline_, _AdapterDrop_, _AA_, and _AA-focused_. _AA_ selects different layers for different tasks and different random seeds.8 We evaluate three configurations for _AA-focused_:

Footnote 8: For instance, the selected layers for _RTE_ are as follows for different runs of _Low-data-100_: {0, 2, 5, 11, 12, 13, 16, 17}, {3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 19, 21}, {2, 3, 4, 6, 9, 12, 14, 16, 17, 18, 20, 22, 23}, {0, 2, 6, 8, 9, 11, 13, 14, 17, 19, 23}, {1, 2, 5, 10, 11, 14, 16, 20, 21, 22, 23}





\begin{table}
\begin{tabular}{l l c c c c c c c c c}  & MNLI & QQP & QLNLI & SST-2 & CoLA & STS-B & MRPC & RTE & WNLI & Avg \\ \hline
**Baseline** & 33.89\({}_{402}\) & 36.50\({}_{38}\) & 58.78\({}_{1,81}\) & 56.13\({}_{6,13}\) & 5.20\({}_{84}\) & 40.00\({}_{6,64}\) & 74.80\({}_{0,0}\) & 49.39\({}_{2,86}\) & 55.21\({}_{1,01}\) & 44.87 \\
**AA** & 33.64\({}_{2,66}\) & 30.88\({}_{2,39}\) & 59.61\({}_{1,19}\) & 51.28\({}_{2,72}\) & -0.55\({}_{1,87}\) & 45.18\({}_{1,17}\) & 74.80\({}_{0,0}\) & 50.11\({}_{3,44}\) & 55.48\({}_{2,46}\) & 44.49 \\
**AdapterDrop\({}^{A[A]}\)** & 33.72\({}_{2,84}\) & 30.62\({}_{0,40}\) & 57.50\({}_{5,05}\) & 54.01\({}_{2,99}\) & 41.07\({}_{0,95}\) & 36.53\({}_{5,93}\) & 74.80\({}_{0,0}\) & 49.39\({}_{2,86}\) & **56.06\({}_{3,38}\)** & 44.08 \\
**AdapterDrop\({}^{[3]}\)** & 33.71\({}_{2,17}\) & 30.61\({}_{0,11}\) & 58.39\({}_{2,77}\) & 53.44\({}_{2,46}\) & 3.91\({}_{1,76}\) & 36.23\({}_{6,58}\) & 74.80\({}_{0,01}\) & 51.48\({}_{2,87}\) & 55.08\({}_{1,91}\) & 44.04 \\
**AA-focused\({}^{qmei}\)** & 35.28\({}_{2,06}\) & 44.37\({}_{1,61}\) & 63.75\({}_{1,39}\) & 52.94\({}_{4,64}\) & 5.68\({}_{0,91}\) & **62.79\({}_{3,74}\)** & 74.80\({}_{0,01}\) & 51.48\({}_{2,82}\) & 54.08\({}_{1,91}\) & 49.47 \\
**AA-focused\({}^{qmei}\)** & **36.36\({}_{2,61}\)** & 44.37\({}_{1,61}\) & 63.36\({}_{3,46}\) & 55.87\({}_{4,42}\) & 4.75\({}_{4,9}\) & 59.37\({}_{6,78}\) & **74.94\({}_{0,92}\)** & 51.13\({}_{4,55}\) & 51.83\({}_{4,12}\) & 49.11 \\
**AA-focused\({}^{qmei}\)** & 34.77\({}_{3,18}\) & **45.78\({}_{1,40}\)** & 63.13\({}_{4,30}\) & **61.85\({}_{1,09}\)** & **17.54\({}_{1,19}\)** & 59.89\({}_{7,70}\) & **74.70\({}_{0,07}\)** & **52.20\({}_{2,93}\)** & 51.83\({}_{5,52}\) & **51.28** \\ \hline
**Baseline** & 24 & 24 & 24 & 24 & 24 & 24 & 24 & 24 & 24 & 24 \\
**IAAl** & 13.2\({}_{1,7}\) & 15.03\({}_{0,13}\) & 13.62\({}_{1,46}\) & 14.64\({}_{0,10}\) & 15.8\({}_{2,1}\) & 16.4\({}_{2,7}\) & 13.0\({}_{1,8}\) & 11.2\({}_{1,8}\) & 12.3\({}_{5,7}\) & \\
**IAdapterDrop\({}^{[A]}\)** & 14 & 13 & 15 & 16 & 16 & 14 & 15 & 13 & 16 & \\
**IAdapterDrop\({}^{[3]}\)** & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & \\
** IAA-focused\({}^{qmei}\)** & 14 & 13 & 15 & 16 & 16 & 14 & 15 & 13 & 16 & \\
**IAa-focused\({}^{qmei}\)** & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & \\
**AA-focused\({}^{qmei}\)** & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & 13 & \\ \hline
**Low-data-300** & & & & & & & & & & & \\
**Baseline** & 36.55\({}_{4,70}\) & 61.50\({}_{8,66}\) & 69.62\({}_{2,24}\) & 79.86\({}_{1,45}\) & 30.40\({}_{40,48}\) & 78.24\({}_{2,81}\) & 76.55\({}_{1,31}\) & 51.62\({}_{3,21}\) & 45.92\({}_{3,31}\) & 58.91 \\
**AA AdapterDrop\({}^{[3]}\)** & 37.14\({}_{2,49}\) & 66.07\({}_{7,38}\) & 71.33\({}_{2,72}\) & 72.51\({}_{2,16}\) & 26.05\({}_{7,14}\) & **82.06\({}_{7,03}\)** & 51.83\({}_{2,84}\) & 47.04\({}_{7,06}\) & 58.68 \\
**AdapterDrop\({}^{[4]}\)** & 38.86\({}_{3,59}\) & 62.98\({}_{1,84}\) & 66.72\({}_{1,91}\) & 79.20\({}_{1,17}\) & 16.89\({}_{1,92}\) & 78.51\({}_{9,79}\) & 77.50\({}_{6,07}\) & 51.19\({}_{3,35}\) & 46.76\({}_{1,76}\) & 57.44 \\
**adapterDrop\({}^{[3]}\)** & 37.95\({}_{5,56}\) & 63.72\({}_{4,84}\) & 66.71\({}_{2,91}\) & 80.01\({}_{4,47}\) & 16.31\({}_{2,05}\) & 77.52\({}_{2,08}\) & 76.70\({}_{0,92}\) & 51.33\({}_{3,34}\) & 46.48\({}_{1,27}\) & 57.34 \\
**AA-focused\({}^{qmei}\)** & 44.62\({}_{1,11}\) & 66.83\({}_{1,06}\) & 73.72\({}_{1,09}\) & 85.87\({}_{2,94}\) & 34.51\({}_{5,13}\) & 81.16\({}_{2,04}\) & 76.72\({}_{1,06}\) & 54.58\({}_{4,72}\) & 46.20\({}_{1,92}\) & 62.69 \\
**AA-focused\({}^{qmei}\)** & **46.64\({}_{2,19}\)** & **49.25\({}_{1,35}\)** & **74.16\({}_{2,95}\)** & **75.70\({}_{2,72}\)** & 35.65\({}_{2,26}\) & 71.72\({}_{1,72}\) & 79.51\({}_{5,15}\) & **56.98\({}_{5,56}\)** & **52.39\({}_{7,26}\)** & **64.48**

the experiments of _RTE_ for _Low-data-100_--i.e., over the five different random seeds--. However, it is different for the rest of the tasks and different data settings.
* **AA-focused\({}^{uni}\)**: we design this adapter architecture of all tasks and data settings based on a single random seed, single task, and a single data regime, i.e.-- random seed 42, the _QQP_ task, and _low-data-100_. We choose _low-data-100_ because the architecture selection process--i.e., training _AA_--is very fast in this setting. We select the selected architecture by _QQP_ because _AA_ selects the smallest number of layers for _QQP_ when the random seed is 42. The selected layers are {2, 6, 10, 12, 14, 15, 16, 18, 19, 20, 21, 22, 23}, i.e., three layers from the first half of the original 24 layers, and ten layers from the second half. The results of _AA-focused\({}^{uni}\)_ compared to AA-focused\({}^{spec}\) indicate whether the selected architecture by _AA_ transfers between similar tasks and different data settings.
* **AA-focused\({}^{sim}\)**: we design a simplified adapter based on _AA_ in which we only use the number of selected layers, instead of the layer numbers, in a single random seed, single task, and a single data setting. We use the number of selected layers when the random seed is 42 for the _QQP_ task and the _low-data-100_ setting, i.e., 13. As investigated by Houlsby et al. (2019), the last adapter layers are in general more effective. As a result, we add adapter layers, with rational activation, to the last 13 transformer layers in _AA-focused\({}^{sim}\)_ experiments. The results of _AA-focused\({}^{sim}\)_ compared to AA-focused\({}^{uni}\) show whether only the number of selected layers by _AA_ matters or it is also important to specify at which layers to add the adapters.

The number of inference layers for _AdapterDrop\({}^{A}\)_ are equivalent to the number of layers in _AA-focused\({}^{spec}\)_ experiments for each task and data setting. The number of layers for _AdapterDrop\({}^{13}\)_ is 13, which is the same as _AA-focused\({}^{uni}\)_ and _AA-focused\({}^{sim}\)_. Note that the number of layers for _AA-focused_ experiments are the same both at training and inference while it is not the case for _AdapterDrop_.

The \(|\)_AA\(I\)_ rows in Table 2 show the average number of selected layers for each task over the five different random seeds. _AA-focused\({}^{*}\)_I rows report the number of added adapter layers in the corresponding _AA-focused\({}^{*}\)_ experiments. \(|\)_AdapterDrop\({}^{*}\)_1rows report the number of included adapter layers for the corresponding _AdapterDrop_ experiments at the inference time.

Footnote 9: It even outperforms _AA-focused\({}^{spec}\)_ showing that _AA-focused\({}^{spec}\)_ may have overfitted to the development sets. We have not performed hyperparameter selection for our experiments. Using better hyperparameters may improve the results of different settings.

We make the following observations from the results of Table 2:

* _AA_ achieves on-par performances with the _Baseline_, and on average it uses about 13-15 layers out of 24 layers. We can use this insight for designing efficient adapter architectures.
* All _AA-focused_ architectures considerably outperform _Baseline_ in all the the tasks in low-data settings while using considerably smaller number of parameters, and therefore, being considerably more efficient. For instance, while _AA-focused\({}^{uni}\)_ only uses 13 layers out of 24 layers--i.e., reducing the number of training parameters from 3M to 1.7M--, it outperforms the _Avg_ score by 4.24, 5.57, and 2.35 points in _Low-data-100_, _Low-data-300_, and _Low-data-500_, respectively.
* The high performances of _AA-focused\({}^{uni}\)_ show that the selected architecture by _AA_ for one task and one data setting transfers well to other data regimes and similar tasks.10 Therefore, it is not necessary to design the adapter architecture separately for a different amount of available data and similar tasks. Footnote 10: It even outperforms _AA-focused\({}^{prec}\)_ showing that _AA-focused\({}^{spec}\)_ may have overfitted to the development sets. We have not performed hyperparameter selection for our experiments. Using better hyperparameters may improve the results of different settings.
* _AA-focused\({}^{sim}\)_ and _AdapterDrop\({}^{13}\)_ both use the last 13 adapter layers during the inference while the results of _AA-focused\({}^{sim}\)_ are considerably higher for all data regimes. This indicates the importance of rational activation in adaptable adapters. We will further investigate the impact on rational activation in the next section.
* In average, _AdapterDrop\({}^{AA}\)_ contains more inference layers compared to _AdapterDrop\({}^{13}\)_. However, there is not a significant difference between their performances. They achieve on-par or lower results compared to _Baseline_.

 

Evaluating the impact of AA on selecting the number of beneficial layers.In the results of Table 2, we select the number of layers in _AA-focused\({}^{sim}\)_, i.e., 13 , based on the minimum number of selected layers by _AA_ on the _low-data-100_ setting and for random seed 42. _AA-focused\({}^{sim}\)_ is equivalent to an adapter architecture in which only the last 13 adapter layers are added to the model. To investigate whether the improvements of _AA-focused\({}^{sim}\)_ over the baseline are only due to using a fewer number of adapter layers, we report the results of an adapter architecture in which only the last \(n\) adapter layers are added to the model, e.g., for \(n=13\) the resulting architecture is the same as _AA-focused\({}^{sim}\)_. Table 3 shows the result of this experiment for \(n=13,12,11\). We observe that by decreasing the number of layers from 13 to 12, the overall performance drops notably from 63.61 to 57.37.

#### Evaluating the impact of rational activation.

The results of _AA-focused_ experiments vs. _Baseline_ in Table 2 mostly emphasize the impact of layer selection by the learnable switches in _AA_. In this section, we investigate the impact of learnable activation functions in more details in the evaluations of Table 4.

First, we replace all rationals in _AA_ with ReLU. The results are reported in the _Switch-Only_ row. By comparing the results of _AA_ and _Switch-only_ we observe that the use of rational activation considerably improves the performance of _AA_, i.e., using rational is a key component to achieving higher performances with fewer layers.

Second, we replace the activation functions in the standard adapter with rational. The results are reported in _Rational-only_ rows. The results of _Baseline_ compared to _Rational-only_ show that the impact of rational is prominent when the model contains fewer parameters and using rational with an overparameterized model is not very effective, i.e., both layer selection and learnable activation play an important role.

Third, we only add a standard adapter layer at the last 13 layers of BERT-large (_Baseline\({}^{13}\)_), which is the same number of adapter layers in _AA-focused\({}^{sim}\)_. The difference is the activation function that is used in these 13 adapter layers is ReLU in _Baseline\({}^{13}\)_ and rational in _AA-focused\({}^{sim}\)_. The considerably higher performances of _AA-focused\({}^{sim}\)_ show that higher performances of _AA-focused_ are due to both layer selection as well as a learnable activation function.

Learned rational activation functions.Figure 3 shows the learned activation functions across different layers of the same trained adapter and different tasks. We see that the learned activation differs

\begin{table}
\begin{tabular}{l l l l l l l l l l}  & **MNLI** & **QQP** & **QNLI** & **SST-2** & **CoLA** & **STS-B** & **MRPC** & **RTE** & **WNLI** & **Avg** \\ \hline \multicolumn{8}{c}{**Low-data-300**} \\ \hline
**Baseline** & 36.55,74.76 & 61.50,86.66 & 69.62,124 & 79.86,14.15 & 30.40,458 & 78.24,81 & 76.55,131 & 51.62,321 & 45.92,433 & 58.91 \\
**AA** & 37.14,29.49 & 66.07,038 & 71.33,182 & 72.52,16.59 & 26.05,74 & **82.08**,16 & 74.03,221 & 51.83,284 & 47.04,76 & 58.68 \\
**Switch-Only** & 35.05,281 & 43.81,62.62 & 65.59,261 & 61.86,266 & 97.1286 & 75.74,14.29 & 75.37,47 & 50.18,344 & 45.92,033 & 51.44 \\
**Rational-Only** & 37.72,288 & 64.75,251 & 69.69,104 & 79.86,14.15 & 23.20,38 & 78.58,94 & 75.84,017 & 52.27,311 & 46.48,388 & 58.70 \\
**Baseline\({}^{13}\)** & 37.98,500 & 33.74,74 & 68.76,155 & 85.16,631 & 21.11,69 & 77.96,23 & 75.25,071 & 54.44,206 & 45.35,372 & 57.80 \\
**AA-focused\({}^{sim}\)** & 45.97,028 & 68.36,136 & 79.928,68 & 86.83,190 & **37.43**,130 & 78.81,578 & **76.66**,130 & 55.96,281 & 48.44,53,53 & 63.61 \\ \hline
**|AA|** & 17.01,3 & 16.21,0 & 14.81,8 & 12.83,2 & 16.82,2 & 18.61,9 & 16.01,1 & 12.42,12 & 12.42,1 & \\
**|Switch-Only|** & 14.01,1 & 15.82,5 & 17.01,9 & 16.22,8 & 16.49,19 & 16.41,15 & 17.8,17 & 15.02,1 & 14.01,7 & \\ \end{tabular}
\end{table}
Table 4: Evaluating the impact of rational in adaptable adapters. Experiments are run for five different random seeds. _Switch-only_ shows the results when learnable switches are used with standard adapter layers, i.e., linear layers with the ReLU activation. _Rational-only_ shows the result when all the activation functions in the standard adapter are replaced with rational. _Baseline\({}^{13}\)_ contains a standard adapter layer on the last 13 transformer layer. _AA-focused\({}^{sim}\)_ contains adapter layers with rational activation on the last 13 layers.

\begin{table}
\begin{tabular}{l l l l l l l l l l l}
**Adap. layers** & **MNLI** & **QQP** & **QNLI** & **SST-2** & **CoLA** & **STS-B** & **MRPC** & **RTE** & **WNLI** & **Avg** \\ \hline
**13** & 45.97\({}_{2.08}\) & 68.36\({}_{1.36}\) & 73.98\({}_{2.86}\) & 86.83\({}_{1.9}\) & 37.34\({}_{3.31}\) & 78.81\({}_{3.58}\) & 76.66\({}_{1.3}\) & 55.96\({}_{2.81}\) & 48.44\({}_{5.53}\) & 63.61 \\
**12** & 36.84\({}_{5.51}\) & 62.43\({}_{5.65}\) & 65.77\({}_{3.43}\) & 84.63\({}_{3.64}\) & 13.23\({}_{2.63}\) & 77.08\({}_{2.36}\) & 75.27\({}_{0.39}\) & 54.30\({}_{3.75}\) & 46.76\({}_{5.45}\) & 57.37 \\
**11** & 36.16\({}_{5.12}\) & 62.59\({}_{5.8}\) & 67.93\({}_{1.42}\) & 79.95\({}_{14.16}\) & 16.32\({}_{11.65}\) & 73.22\({}_{6.75}\) & 76.42\({}_{1.19}\) & 56.53\({}_{2.02}\) & 46.24\({}_{1.12}\) & 57.26 \\ \end{tabular}
\end{table}
Table 3: Evaluating the impact of the number of adapter layers on the overall performance. The adapter layers are added to the top \(n\) layers of the model for \(n=13,12,11\). Adapter layers contain rational activation, i.e., \(n=13\) is equivalent to _AA-focused\({}^{sim}\)_. Results are reported for the _low-data-300_ setting.

 

for different layers of the same task as well as for different tasks.

## 6 Conclusion

In this paper, we propose adaptable adapters. They consist of a learnable switch to select a subset of beneficial adapter layers and a learnable activation function to learn the suitable activation at each adapter layer and for each input data. The results of adaptable adapters show that we can achieve on-par performances with the full adapter architecture by using a smaller subset of layers. We show that adaptable adapters are viable tools for designing efficient and effective adapter architectures that require less storage space, lower training and inference time with high performances.

## Acknowledgements

The authors would like to thank Jorge Cardona for his valuable contribution to the implementation of adaptable adapters. We thank Michael Bugert, Jius Ling Lee, and Soumya Sarkar for their constructive suggestions and feedback on this work. We would like to thank Jonas Pfeiffer and Clifton Poth for always being very helpful with all questions about adapters and AdapterHub. This research work has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) within their joint support of the National Research Center for Applied Cybersecurity ATHENE. It benefited from the HMWK cluster project "The Third Wave of AI".

## References

* Bai et al. (2021) Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2021. BinaryBERT: Pushing the limit of BERT quantization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics. Boulle et al. (2020) Nicolas Boulle, Yuji Nakatsukasa, and Alex Townsend. 2020. Rational neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS). Cited by: SS4.
* Cern Diab et al. (2017) SemEval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, pp. 1-14. External Links: Link Cited by: SS1, SS2.
* K. Clark, U. Khandelwal, O. Levy, and Christopher D. Manning (2019)What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Florence, Italy, pp. 276-286. External Links: Link Cited by: SS1, SS2.
* I. Dagan, O. Glickman, and B. Magnini (2006)The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177-190. Cited by: SS1, SS2.
* Quentin Delfosse, Patrick Schramowski, Alejandro Molina, and K. Kersting (2021)Recurrent rational networks. arXiv preprint arXiv:2102.09407.

Figure 3: Learned rational activation functions differ according to their place within the network and to the task they are trained for. Right: activation functions at different layers within adapters trained on the QNLI task. Left: activation functions trained at layer 2 of adapters trained on different tasks.

 

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. External Links: ISBN 978-3-546-3861-4 Cited by: SS1.
* W. B. Dolan and C. Brockett (2005)Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), Cited by: SS1.
* A. Fan, E. Grave, and A. Joulin (2020)Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, Cited by: SS1.
* W. Fedus, B. Zoph, and N. Shazeer (2021)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. Cited by: SS1.
* R. He, L. Liu, H. Ye, Q. Tan, B. Ding, L. Cheng, J. Low, L. Bing, and L. Si (2021)On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 2208-2222. External Links: Link, Document Cited by: SS1.
* N. Houlsby, A. Giurgiu, S. Iastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019)Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pp. 2790-2799. Note: PMLR External Links: Link Cited by: SS1.
* G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger (2016)Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Cited by: SS1.
* J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020)Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: SS1.
* H. J. Levesque, E. Davis, and L. Morgenstern (2011)The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, Vol. 46, pp. 47. Cited by: SS1.
* A. Molina, P. Schramowski, and K. Kersting (2020)Pade activation units: end-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations, Cited by: SS1.
* J. Pfeiffer, A. Ruckle, C. Roth, A. Kamath, I. Vulic, S. Ruder, K. Cho, and I. Guerevych (2020)Adapterbub: a framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, Online, pp. 46-54. External Links: Link, Document Cited by: SS1.
* P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016)SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 2383-2392. External Links: Link, Document Cited by: SS1.
* A. Roberts, C. Raffel, and N. Shazeer (2020)How much knowledge can you pack into the parameters of a language model?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 5418-5426. External Links: Link, Document Cited by: SS1.
* A. Ruckle, G. Geigle, M. Glockner, T. Beck, J. Pfeiffer, N. Reimers, and I. Gurevych (2021)AdaperDrop: on the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 7930-7946. External Links: Link, Document Cited by: SS1.
* NeurIPS 2019, Cited by: SS1.
* V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. Saiful Bari, C. Xu, U. Thakker, S. Sharma Sharma, E. Szczechla, T. Kim, G. Chhabplani, N. Jayak, D. Datta, J. Chang, M. Tian-Jian Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bowden, T. Wang, T. Neneralj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. Alan Fries, R. Teehan, T. Le Scao, Stella Biderman, L. Gao, T. Wolf, and A. M Rush (2022)Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, Cited by: SS1.
* R. Schwartz, G. Stanovsky, S. Swayamdipta, J. Dodge, and N. A. Smith (2020)The right tool for the job: matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 6640-6651. External Links: Link, Document Cited by: SS1.
* R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and 

Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Sukhbaatar et al. (2019) Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 331-335, Florence, Italy. Association for Computational Linguistics.
* Sun et al. (2020) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2158-2170, Online. Association for Computational Linguistics.
* Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. _arXiv preprint arXiv:1903.12136_.
* Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
* Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
* Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _International Conference on Learning Representations_.
* Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. _Transactions of the Association for Computational Linguistics_, 7:625-641.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
 



\begin{table}
\begin{tabular}{l l c c c c c c c c c}  & MNLI & QQP & QNLI & SST-2 & CoLA & STS-B & MRPC & RTE & WNLI & Avg \\ \hline
**Baseline** & **83.53\({}_{0.19}\)** & **88.12\({}_{0.14}\)** & **96.63\({}_{0.26}\)** & **91.74\({}_{0.36}\)** & **56.51\({}_{0.84}\)** & **88.48\({}_{0.14}\)** & 84.81\({}_{0.17}\) & 63.83\({}_{1.4}\) & **54.08\({}_{6.64}\)** & **77.97** \\
**AA** & 82.89\({}_{0.43}\) & 88.09\({}_{0.16}\) & 89.96\({}_{0.25}\) & 91.31\({}_{0.51}\) & 51.44\({}_{1.82}\) & 88.25\({}_{0.17}\) & **85.09\({}_{1.06}\)** & **64.25\({}_{1.72}\)** & 52.11\({}_{f.61}\) & 77.05 \\ \hline
**AA-Layers** & 9.8\({}_{0.3}\) & 11.2\({}_{0.7}\) & 10.6\({}_{1.0}\) & 9.8\({}_{1.1}\) & 8.6\({}_{2.1}\) & 11.4\({}_{0.4}\) & 9.0\({}_{0.6}\) & 9.4\({}_{0.7}\) & 8.0\({}_{1.4}\) & \\ \hline \multicolumn{11}{c}{**Low-data-100**} \\ \hline
**Baseline** & 35.66\({}_{3.38}\) & 29.70\({}_{0.86}\) & 60.51\({}_{4.5}\) & 51.54\({}_{1.24}\) & -1.27\({}_{3.56}\) & 41.52\({}_{5.93}\) & **74.86\({}_{0.12}\)** & **50.42\({}_{9.98}\)** & 54.93\({}_{5.84}\) & 44.21 \\
**AA** & **37.05\({}_{2.35}\)** & **30.59\({}_{0.68}\)** & **62.52\({}_{2.27}\)** & **52.73\({}_{2.55}\)** & **-0.08\({}_{0.16}\)** & **48.73\({}_{23.91}\)** & 74.83\({}_{0.07}\) & 50.18\({}_{3.21}\) & **55.21\({}_{6.13}\)** & **45.75** \\ \hline
**IAAI** & 6.4\({}_{1.8}\) & 8.6\({}_{2.1}\) & 8.8\({}_{1.7}\) & 8.6\({}_{1.6}\) & 74.2\({}_{4.2}\) & 10.8\({}_{0.7}\) & 9.4\({}_{1.4}\) & 9.4\({}_{1.4}\) & 8.2\({}_{0.9}\) & \\ \hline \multicolumn{11}{c}{**Low-data-300**} \\ \hline
**Baseline** & 37.88\({}_{4.09}\) & 49.24\({}_{10.32}\) & 68.17\({}_{2.9}\) & 75.53\({}_{4.9}\) & 3.40\({}_{5.59}\) & 69.39\({}_{15.05}\) & 75.99\({}_{1.2}\) & 54.22\({}_{2.96}\) & **47.61\({}_{4.91}\)** & 53.49 \\
**AA** & **40.27\({}_{4.78}\)** & **66.31\({}_{1.86}\)** & **74.03\({}_{2.03}\)** & **76.42\({}_{6.07}\)** & **3.56\({}_{5.49}\)** & **82.06\({}_{2.24}\)** & **76.12\({}_{0.89}\)** & **54.73\({}_{0.09}\)** & 47.04\({}_{5.46}\) & **57.84** \\ \hline
[AA|\) & 10.4\({}_{1.6}\) & 10.8\({}_{0.7}\) & 11.0\({}_{0.8}\) & 9.4\({}_{1.3}\) & 7.6\({}_{2.0}\) & 10.8\({}_{0.7}\) & 9.6\({}_{1.0}\) & 9.8\({}_{1.4}\) & 8.2\({}_{1.1}\) & \\ \hline \multicolumn{11}{c}{**Low-data-500**} \\ \hline
**Baseline** & 42.82\({}_{2.4}\) & 67.63\({}_{1.44}\) & 72.7\({}_{1.31}\) & 83.46\({}_{0.64}\) & **20.9\({}_{1.4}\)** & 81.97\({}_{0.89}\) & 76.51\({}_{0.95}\) & **57.11\({}_{2.93}\)** & **52.11\({}_{6.96}\)** & 61.69 \\
**AA** & **47.72\({}_{1.67}\)** & **69.27\({}_{0.89}\)** & **75.649\({}_{1.9}\)** & **84.52\({}_{1.18}\)** & 19.13\({}_{14.46}\) & **83.74\({}_{0.67}\)** & **78.03\({}_{2.33}\)** & 55.96\({}_{3.08}\) & 51.83\({}_{6.13}\) & **62.87** \\ \hline
[AA|\) & 9.8\({}_{1.1}\) & 10.4\({}_{1.3}\) & 10.0\({}_{0.8}\) & 9.2\({}_{0.7}\) & 9.4\({}_{1.8}\) & 10.6\({}_{1.4}\) & 9.8\({}_{1.6}\) & 9.6\({}_{1.0}\) & 8.0\({}_{1.5}\) & \\ \end{tabular}
\end{table}
Table 5: Comparing the results of (a) the baseline adapter model that includes an adapter layer on all BERT-base layers (_Baseline_), and (b) the adaptable adapter (_AA_). The reported results are averaged over five different random seeds. The subscript reports the corresponding standard deviation. _|AA|_ reports the average number of selected adapter layers by the adaptable adapter over different runs. The _full data_ results show the performance when the model is trained on all the available training data. The _Low-data-X_ settings report the results when only \(X\) examples are used for training the model. The test data is the same for all the experiments.