SlowGuess/NAACL_2022_OCR huggingface dataset:
ferent random seeds.  $|AA\text{-focused}^*|$  rows report the number of added adapter layers in the corresponding  $AA\text{-focused}^*$  experiments.  $|\text{AdapterDrop}^*|$  rows report the number of included adapter layers for the corresponding AdapterDrop experiments at the inference time.
We make the following observations from the results of Table 2:
-  $AA$  achieves on-par performances with the Baseline, and on average it uses about 13-15 layers out of 24 layers. We can use this insight for designing efficient adapter architectures.
- All AA-focused architectures considerably outperform Baseline in all the tasks in low-data settings while using considerably smaller number of parameters, and therefore, being considerably more efficient. For instance, while AA-focused $^{uni}$  only uses 13 layers out of 24 layers—i.e., reducing the number of training parameters from 3M to 1.7M—, it outperforms the Avg score by 4.24, 5.57, and 2.35 points in Low-data-100, Low-data-300, and Low-data-500, respectively.
- The high performances of  $AA$ -focused<sup>uni</sup> show that the selected architecture by  $AA$  for one task and one data setting transfers well to other data regimes and similar tasks.<sup>9</sup> Therefore, it is not necessary to design the adapter architecture separately for a different amount of available data and similar tasks.
-  $AA$ -focused $^{sim}$  and AdapterDrop $^{13}$  both use the last 13 adapter layers during the inference while the results of  $AA$ -focused $^{sim}$  are considerably higher for all data regimes. This indicates the importance of rational activation in adaptable adapters. We will further investigate the impact on rational activation in the next section.
- In average, AdapterDrop $^{AA}$  contains more inference layers compared to AdapterDrop $^{13}$ . However, there is not a significant difference between their performances. They achieve on-par or lower results compared to Baseline.
<sup>9</sup>It even outperforms AA-focused<sup>spec</sup> showing that AA-focused<sup>spec</sup> may have overfitted to the development sets. We have not performed hyperparameter selection for our experiments. Using better hyperparameters may improve the results of different settings.
Adap. layers	MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	WNLI	Avg
Low-data-300
13	45.972.08	68.361.36	73.982.68	86.831.9	37.433.1	78.813.58	76.661.3	55.962.81	48.445.53	63.61
12	36.845.51	62.435.65	65.773.43	84.633.64	13.2312.63	77.082.36	75.270.39	54.303.75	46.765.45	57.37
11	36.165.12	62.595.8	67.931.42	79.9514.16	16.3211.65	73.226.75	76.421.19	56.532.02	46.24.12	57.26
Table 3: Evaluating the impact of the number of adapter layers on the overall performance. The adapter layers are added to the top  $n$  layers of the model for  $n = {13},{12},{11}$  . Adapter layers contain rational activation,i.e.,  $n = {13}$  is equivalent to  ${AA}$  -focused  ${}^{\text{sim }}$  . Results are reported for the low-data-300 setting.
MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	WNLI	Avg
Low-data-300
Baseline	36.554.76	61.508.66	69.621.24	79.8614.15	30.405.48	78.242.81	76.551.31	51.623.21	45.924.33	58.91
AA	37.142.49	66.070.38	71.331.82	72.5216.59	26.058.74	82.081.6	74.032.21	51.832.84	47.044.76	58.68
Switch-Only	35.052.81	43.816.02	65.592.61	61.866.26	9.7712.86	75.413.29	75.370.7	50.183.44	45.923.03	51.44
Rational-Only	37.723.88	64.752.51	69.691.04	79.8614.15	23.208.33	78.581.94	75.841.07	52.273.11	46.483.88	58.70
\( Baseline^{13} \)	37.985.80	63.374.72	68.761.55	85.163.63	12.1112.69	77.962.23	75.250.71	54.442.06	45.353.72	57.80
\( AA-focusedsim \)	45.972.08	68.361.36	73.982.68	86.831.90	37.433.10	78.813.58	76.661.30	55.962.81	48.445.53	63.61
|AA|	17.01.3	16.21.0	14.81.8	12.83.2	16.82.2	18.61.9	16.01.1	12.41.2	12.42.1
|Switch-Only|	14.01.1	15.82.5	17.01.9	16.22.8	16.41.9	16.41.5	17.81.7	15.02.1	14.01.7
Table 4: Evaluating the impact of rational in adaptable adapters. Experiments are run for five different random seeds. Switch-only shows the results when learnable switches are used with standard adapter layers, i.e., linear layers with the ReLU activation. Rational-only shows the result when all the activation functions in the standard adapter are replaced with rational. Baseline<sup>13</sup> contains a standard adapter layer on the last 13 transformer layer. AA-focused<sup>sim</sup> contains adapter layers with rational activation on the last 13 layers.
Evaluating the impact of AA on selecting the number of beneficial layers. In the results of Table 2, we select the number of layers in  $AA$ -focused $^{sim}$ , i.e., 13, based on the minimum number of selected layers by AA on the low-data-100 setting and for random seed 42. AA-focused $^{sim}$  is equivalent to an adapter architecture in which only the last 13 adapter layers are added to the model. To investigate whether the improvements of AA-focused $^{sim}$  over the baseline are only due to using a fewer number of adapter layers, we report the results of an adapter architecture in which only the last  $n$  adapter layers are added to the model, e.g., for  $n = 13$  the resulting architecture is the same as AA-focused $^{sim}$ . Table 3 shows the result of this experiment for  $n = 13, 12, 11$ . We observe that by decreasing the number of layers from 13 to 12, the overall performance drops notably from 63.61 to 57.37.
Evaluating the impact of rational activation. The results of AA-focused experiments vs. Baseline in Table 2 mostly emphasize the impact of layer selection by the learnable switches in AA. In this section, we investigate the impact of learnable activation functions in more details in the evaluations of Table 4.
First, we replace all rationals in  $AA$  with ReLU. The results are reported in the Switch-Only row. By
comparing the results of  $AA$  and Switch-only we observe that the use of rational activation considerably improves the performance of  $AA$ , i.e., using rational is a key component to achieving higher performances with fewer layers.
Second, we replace the activation functions in the standard adapter with rational. The results are reported in Rational-only rows. The results of Baseline compared to Rational-only show that the impact of rational is prominent when the model contains fewer parameters and using rational with an overparameterized model is not very effective, i.e., both layer selection and learnable activation play an important role.
Third, we only add a standard adapter layer at the last 13 layers of BERT-large (Baseline $^{13}$ ), which is the same number of adapter layers in AA-focused $^{sim}$ . The difference is the activation function that is used in these 13 adapter layers is ReLU in Baseline $^{13}$  and rational in AA-focused $^{sim}$ . The considerably higher performances of AA-focused $^{sim}$  show that higher performances of AA-focused are due to both layer selection as well as a learnable activation function.
Learned rational activation functions. Figure 3 shows the learned activation functions across different layers of the same trained adapter and different tasks. We see that the learned activation differs
Rational activation functions
Figure 3: Learned rational activation functions differ according to their place within the network and to the task they are trained for. Right: activation functions at different layers within adapters trained on the QNLI task. Left: activation functions trained at layer 2 of adapters trained on different tasks.
for different layers of the same task as well as for different tasks.
In this paper, we propose adaptable adapters. They consist of a learnable switch to select a subset of beneficial adapter layers and a learnable activation function to learn the suitable activation at each adapter layer and for each input data. The results of adaptable adapters show that we can achieve on-par performances with the full adapter architecture by using a smaller subset of layers. We show that adaptable adapters are viable tools for designing efficient and effective adapter architectures that require less storage space, lower training and inference time with high performances.
The authors would like to thank Jorge Cardona for his valuable contribution to the implementation of adaptable adapters. We thank Michael Bugert, Ji-Ung Lee, and Soumya Sarkar for their constructive suggestions and feedback on this work. We would like to thank Jonas Pfeiffer and Clifton Poth for always being very helpful with all questions about adapters and AdapterHub. This research work has been funded by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) within their joint support of the National Research Center for Applied Cybersecurity ATHENE. It benefited from the HMWK cluster project "The Third Wave of AI".
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2021. BinaryBERT: Pushing the limit of BERT quantization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4334-4348, Online. Association for Computational Linguistics.
Nicolas Boullé, Yuji Nakatsukasa, and Alex Townsend. 2020. Rational neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS).
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising textual entailment, pages 177-190. Springer.
Quentin Delfosse, Patrick Schramowski, Alejandro Molina, and Kristian Kersting. 2021. Recurrent rational networks. arXiv preprint arXiv:2102.09407.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations.
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2208-2222, Online. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. 2016. Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, page 47.
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. 2020. Padé activation units: End-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations.
Jonas Pfeiffer, Andreas Rückle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Andreas Rückle, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930-7946, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640-6651, Online. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and
Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331-335, Florence, Italy. Association for Computational Linguistics.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158-2170, Online. Association for Computational Linguistics.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
A BERT-base Results
MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	WNLI	Avg
Baseline	83.530.19	88.120.14	90.630.26	91.740.36	56.510.84	88.480.14	84.810.07	63.831.4	54.086.64	77.97
AA	82.890.43	88.090.16	89.960.25	91.310.51	51.441.82	88.250.17	85.091.06	64.251.72	52.117.61	77.05
AA-Layers	9.80.3	11.20.7	10.61.0	9.81.1	8.62.1	11.40.4	9.00.6	9.40.7	8.01.4
Low-data-100
Baseline	35.663.38	29.700.86	60.514.5	51.542.14	-1.273.56	41.525.93	74.860.12	50.42.98	54.935.84	44.21
AA	37.052.35	30.590.68	62.524.27	52.732.55	-0.080.16	48.7323.91	74.830.07	50.183.21	55.216.13	45.75
|AA|	6.41.8	8.62.1	8.81.7	8.61.6	7.42.4	10.80.7	9.41.4	9.41.4	8.20.9
Low-data-300
Baseline	37.884.09	49.2410.32	68.172.9	75.533.49	3.408.59	69.3915.05	75.991.2	54.222.96	47.614.91	53.49
AA	40.274.78	66.311.86	74.032.03	76.426.07	3.565.49	82.062.24	76.120.89	54.733.09	47.045.46	57.84
|AA|	10.41.6	10.80.7	11.00.8	9.41.3	7.62.0	10.80.7	9.61.0	9.81.4	8.21.1
Low-data-500
Baseline	42.822.4	67.631.44	72.71.31	83.460.64	20.94.14	81.970.89	76.510.95	57.112.93	52.116.96	61.69
AA	47.721.67	69.270.89	75.6491.9	84.521.18	19.1314.46	83.740.67	78.032.33	55.963.08	51.836.13	62.87
|AA|	9.81.1	10.41.3	10.00.8	9.20.7	9.41.8	10.61.4	9.81.6	9.61.0	8.01.5
Table 5: Comparing the results of (a) the baseline adapter model that includes an adapter layer on all BERT-base layers (Baseline), and (b) the adaptable adapter (AA). The reported results are averaged over five different random seeds. The subscript reports the corresponding standard deviation. |AA| reports the average number of selected adapter layers by the adaptable adapter over different runs. The full data results show the performance when the model is trained on all the available training data. The Low-data-  $X$  settings report the results when only  $X$  examples are used for training the model. The test data is the same for all the experiments.