SlowGuess/NAACL_2022_OCR huggingface dataset:
We observed that many of the trained parameters in  $L_{\alpha}$  have values that are extremely close to zero after tuning on downstream tasks, which might cause redundancy of the parameters. To further encourage the sparsity of AdapterBias, we add  $L_{0}$ -norm regularization to  $L_{\alpha}$  during the training stage.
Method	Params	CoLA	SST-2	MRPC	QNLI	RTE	STS-B	MNLI-m	MNLI-mm	QQP	Avg
Share v	56.1K	50.1	90.8	87.1	87.6	65.0	84.9	77.5	77.9	65.1	76.2
Share Lα	30.7K	50.4	91.9	88.1	89.1	65.4	85.2	79.8	79.9	66.6	77.4
Share v+Lα	22.3K	46.8	90.9	87.3	87.8	64.8	85.7	77.7	78.0	64.9	76.0
AdapterBias	64.5K	51.6	93.1	87.5	89.4	66.1	84.6	80.9	80.5	67.9	78.0
Table 4: Analysis of more parameter-efficiency methods in AdapterBias. The settings are the same as in Table 1. The backbone model is BERT-base. Share  $\nu$ , Share  $L_{\alpha}$ , and Share  $\nu + L_{\alpha}$  means that we share vector, linear layer, and both of them, respectively.
Method	CoLA	SST-2	MRPC	QNLI	RTE	STS-B	MNLI-m	MNLI-mm	QQP	Avg
BB	Full-FT	52.1	93.5	88.9	90.5	66.4	85.8	84.6	83.4	71.2	79.6
BB	AdapterBias	51.6	93.1	87.5	89.4	66.1	84.6	80.9	80.5	67.9	78.0
BB	AdapterBias (L0)	53.7	92.5	87.5	90.3	68.3	85.7	81.7	81.5	69.8	79.0
BL	Full-FT	60.5	94.9	89.3	92.7	70.1	87.6	86.7	85.9	72.1	82.2
BL	AdapterBias	60.0	94.4	88.2	91.2	70.5	87.5	84.3	83.9	70.5	81.2
BL	AdapterBias (L0)	58.0	93.7	88.2	91.5	69.2	87.2	84.2	84.1	71.2	80.8
Table 5: Performance of our AdapterBias with  $L_{0}$ -norm regularization. Here we experiment with two models: BERT-base (BB), and BERT-large (BL). The settings are the same as in Table 1. The Full-FT represents fine-tuning the whole PLM without adding adapters.
In Table 5, we use BERT-base (BB) and BERT-large (BL) as the PLMs. We compare the performance of fine-tuning, the original AdapterBias, and the one trained with  $L_{0}$ -norm regularization. The experiment shows that adding  $L_{0}$ -norm regularization during the training step improves the performance on 7 out of 9 tasks in BERT-base models. However, the performance did not improve when applied to BERT-large models. As for the parameter efficiency of applying  $L_{0}$ -norm penalty, the linear layer  $(L_{\alpha})$  with  $L_{0}$ -norm penalty saves about  $17\%$  parameter on average compared to the original AdapterBias. The details of the reduced parameters of each task are shown in Appendix A.3.
4.7 What AdapterBias learns
AdapterBias has good interpretability due to its simplicity. Compared to the similar work BitFit (Ben Zaken et al., 2021), where the shifts are identical for all tokens, AdapterBias adds token-dependent shifts to the output representation. By observing these token-dependent shifts, we analyze what AdapterBias learns when adapting to downstream tasks.
4.7.1 Average representation shifting in transformer layers
In light of the works of Liu et al. (2019a); Tenney et al. (2019); Kovaleva et al. (2019), which show that different information is being encoded by different transformer layers of PLMs. We assume that AdapterBias provides different repre
Figure 5: We analyze the average absolute value of weights vector  $\alpha$ , the output of the linear layer  $(L_{\alpha})$ , in each layer for different tasks. The y-axis represents the index of transformer layers, ordered from earlier to later (i.e. the embedding layer is shown at the top). The x-axis represents the average absolute value of  $\alpha$ .
Figure 6: Word cloud of CoLA, a corpus of linguistic acceptability. We utilize BERT-base model as the PLM and words come from validation data. The weights of the words are the summation of their weights produced by the linear layer  $(L_{\alpha})$  in twelve transformer layers.
sensation shifts to the transformer layers through task-specific fine-tuning.
In AdapterBias, the linear layer  $(L_{\alpha})$  produces a weights vector  $\alpha$  for representation shifts, therefore, the average absolute value of vector  $\alpha$  can give us a look at the shifting amount in the transformer layers when adapting to downstream tasks. In Figure 5, the layers are ordered from lower to upper. From the experimental result, we find that the weight in each layer is considerably different in different tasks in general.
CoLA (Warstadt et al., 2019) is a syntactic task that consists of English acceptability judgments in the GLUE benchmark. As shown in Figure 5, its average shift at the ninth layer is the highest among all layers, which is quite different from the others. We speculate that the ninth layer has the ability to extract the syntactic information, leading AdapterBias to add the largest shift in this layer. Our experiment has a similar observation with the work of Jawahar et al. (2019). They observe on a syntactic task with BShift (Conneau et al., 2018) that the ninth layer of BERT embeds a rich hierarchy of syntactic information. (Jawahar et al., 2019)
Moreover, we observe similar distributions between specific tasks. For instance, RTE (Giampiccolo et al., 2007; Bentivogli et al., 2009) and MNLI (Williams et al., 2017), where both recognize textual entailment, have higher values in the upper layers than the lower ones.
Based on these findings, we find that Adapter-Bias assigns suitable representation shifts in different tasks. For tasks with similar objectives, AdapterBias tends to add similar representation shifts.
4.7.2 Which kind of word does  $L_{\alpha}$  focus on
Since  $\alpha_{i}$  represents the weight of the representation shift for  $i^{th}$  token in a transformer layer, we can observe the significance of  $i^{th}$  token from the summation of  $\alpha_{i}$  in all the transformer layers. Special tokens, including [CLS], [SEP], and [PAD], are not included for analysis. We use the validation sets of CoLA and SST-2, and word cloud is used for visualizations.
In Figure 6, we visualize all words in the validation data of CoLA. The result shows that Adapter-Bias focuses more on reflexive pronouns, such as yourself, himself, and myself. This is because there are many incorrect sentences with misused reflexive pronouns, such as "He washed yourself."
In Figure 7, we visualize all words in the validation.
Figure 7: Word cloud of SST-2, a corpus of movie reviews categorized in two sentimental classes (i.e. positive, negative). The visualization approach is the same as in Figure 6.
tion data of SST-2. The result shows that Adapter-Bias focuses more on adjectives, such as "bad", "awful", and "worst". SST-2 is a binary sentiment analysis dataset, which classifies movie reviews into positive and negative classes. AdapterBias learns that adjectives often constitute a crucial factor in sentiment analysis during tuning, and adds larger shifts to these adjective tokens.
In this study, we present AdapterBias. By adding token-dependent representation shifts to the PLM, AdapterBias shows competitive results even though it uses far fewer parameters than the existing methods. Through extensive experiments, not only does AdapterBias reach competitive results on the GLUE benchmark, but also obtain good performance on small-to-medium datasets. In addition, we demonstrate the robustness of AdapterBias to different PLMs. Finally, we provide analysis on what AdapterBias learns by comparing  $\alpha$ , the weights of representation shift for different tokens, finding AdapterBias has the ability to identify task-specific information. Our study is different from the previous architectures of adapters by proposing a simple adapter that can produce suitable representation shifts for different tokens.
Ankur Bapna, Naveen Arivazhagan, and Orhan First. 2019. Simple, scalable adaptation for neural machine translation. arXiv preprint arXiv:1909.08478.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv e-prints, pages arXiv-2106.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo
Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC.
Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.01070.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9.
Demi Guo, Alexander M Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics.
Ian T Jolliffe. 2002. Springer series in statistics. *Principal component analysis*, 29.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.08593.
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. 2019a. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Christos Louizos, Max Welling, and Diederik P Kingma. 2017. Learning sparse neural networks through  $l_{-}0$  regularization. arXiv preprint arXiv:1712.01312.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint arXiv:2106.04647.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247.
Jonas Pfeiffer, Andreas Rückle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020b. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Andreas Rückle, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Asa Cooper Stickland and Iain Murray. 2019. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In International Conference on Machine Learning, pages 5986-5995. PMLR.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtopicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Yu Zhang and Qiang Yang. 2017. A survey on multitask learning. arXiv preprint arXiv:1707.08114.
We train our model on Pytorch. The training details are shown in Table A. In addition, the bottleneck of Adapters (Houlsby et al., 2019) and is 32.
A.2  $L_{0}$ -norm regularization in AdapterBias
In Table B, we report the remaining parameters of utilizing  $L_{0}$ -norm regularization compared with the original AdapterBias. BERT-base (BB) and BERT-large (BL) are used as PLMs.
A.3 The direction of representation shifts in different tasks
Different from BitFit (Ben Zaken et al., 2021), where all the representation shifts are identical within one task, AdapterBias produces different weights for the shift based on each token. In this section, we compare the transformed tokens in AdapterBias and BitFit. We utilize PCA (Jolliffe, 2002) to reduce the dimension of the vectors. In Figure A, we input five sentences from the evaluation set of SST-2. We experiment on the last transformer layer since it has the most obvious shifts compared to the previous layers. '0' with lighter color indicates the representation before shifting, which is the output of the first layer normalization. '1' with darker color is the shifted representation, which is the output of the second layer normalization. The color red represents positive sentences, and blue are the negative ones.
The result shows that BitFit shifts all tokens towards the same direction regardless of the ground-truth label. On the other hand, AdapterBias discerns the label of the sentences and thus shifts the tokens of different sentences toward different directions.
Figure A: We utilize PCA (Jolliffe, 2002) to visualize the shifting difference between Bitfit (Ben Zaken et al., 2021) and AdapterBias on SST-2 validation set. '0' with light color means the embedding before shifting. '1' with dark color means the embedding after shifting. The color red represents positive sentences, and blue represents negative sentences.
CoLA	SST-2	MRPC	QNLI	RTE	STS-B	MNLI-m	MNLI-mm	QQP
Max_len	128	128	128	512	350	512	128	128	350
Batchsize	32	32	32	16	32	16	32	32	32
Learning rate	10-3	10-3	10-3	10-4	4 × 10-4	10-3	4 × 10-4	4 × 10-4	4 × 10-4
Epoch	20	10	10	10	20	20	10	10	10
Table A: Our training details of GLUE benchmark(Wang et al., 2018).
Method	CoLA	SST-2	MRPC	QNLI	RTE	STS-B	MNLI-m	MNLI-mm	QQP
BB	AdapterBias (L0)	26.2%	82.0%	83.1%	82.3%	81.0%	83.0%	83.2%	83.3%	83.4%
BL	AdapterBias (L0)	83.2%	83.0%	83.3%	83.7%	83.2%	83.2%	83.4%	83.7%	83.6%
Table B: Percentage of remaining parameters compared with the original parameters of the linear layer  $(L_{\alpha})$ . Here we experiment with two models: BERT-base (BB) and BERT-large (BL). The setting follows by Table 1.
Figure B: Word cloud of SST-2 in layer 0 to layer 6.
Figure C: Word cloud of SST-2 in layer 7 to layer 12.
Figure D: Word cloud of CoLA in layer 0 to layer 6.
Figure E: Word cloud of CoLA in layer 7 to layer 12.