SlowGuess/NAACL_2022_OCR huggingface dataset:
Chin-Lun  $\mathbf{F}\mathbf{u}^{1\star}$
Zih-Ching Chen\*
Yun-Ru Lee3
Hung-yi Lee
$^{1}$ Department of Electrical Engineering, National Taiwan University
$^{2}$ Graduate Institute of Communication Engineering, National Taiwan University  
 $^{3}$ Department of Computer Science and Information Engineering, National Taiwan University  
{b06505011, r09942176, b06902107, hungyilee}@ntu.edu.tw
Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed. AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias. The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pretrained models. We further find that AdapterBias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration.<sup>1</sup>
While large pre-trained language models (PLMs) reached state-of-the-art results on natural language processing (NLP) tasks, PLMs require updating all parameters and storing the fully fine-tuned model for each downstream task. These requirements have led to difficulties in real-world applications. Moreover, fine-tuning PLMs on low-resource datasets is subject to instabilities.
To tackle these shortcomings, Adapters (Houlsby et al., 2019), a more parameter-efficient alternative training strategy for the transformer architecture (Vaswani et al., 2017) have been proposed. Instead of full fine-tuning the whole model, Adapters introduce extra tunable weights and freeze the original parameters of PLM. Adapters demonstrated comparable performance with fully fine-tuning the
1. Token from positive sentence
2. Before shifting
Figure 1: Overview of the main concept of our work compared to BitFit (Ben Zaken et al., 2021). Left: BitFit tends to add the same representation shift to different tokens. Right: Our work applies different representation shifts to tokens considering their importance to the downstream task and their characteristics. The shifts of the input words that are more task-related is more significant than that of other tokens. For example, in SST-2 (Socher et al., 2013), which is a semantic task, the representation shifts of the semantic words, such as "kind" and "worse", are larger than that of other words.
Token from negative sentence After shifting
entire model. Although Adapters solve the problem of the PLM's massive parameters, researchers are curious about how many more parameters are required to reach state-of-the-art performance on standard NLP tasks. The results in Houlsby et al. (2019) have shown that the performance on GLUE benchmark (Wang et al., 2018) is almost the same when removing the Adapters in the lower layers, which indicates that not every adapter is useful. It raises the question of whether adapters can be even more parameter-efficient.
To develop practical and memory-efficient methods of utilizing PLMs, Diff pruning (Guo et al., 2020) enables parameter-efficient transfer learning that scales well with new tasks. The approach learns a task-specific "diff" vector that extends the original pre-trained parameters and encourages the sparsity of the vector through  $L_{0}$ -norm regularization. Another approach is BitFit (Ben Zaken et al.,
The source code is available at: https://github. com/Allen0307/AdapterBias
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2608 - 2621 July 10-15, 2022 ©2022 Association for Computational Linguistics
2021), which shows that with small-to-medium training data, fine-tuning only a subset of the bias terms of pre-trained BERT models (Devlin et al., 2018) is competitive with fine-tuning the entire model. The central concept of these approaches is to add task-specific shifts to each output representation of the PLM layers so as to adapt to different tasks. In the previous works, Ben Zaken et al. (2021); Guo et al. (2020) both add the same shifts to the output representation regardless of which token is more relevant to the task. However, considering some specific tokens might be more critical to a particular task, the representation can better adapt to the downstream task under a limited amount of parameters if these shifts are based on the input tokens.
Based on this concept, in this study, we add token-dependent biases to the shifts by proposing AdapterBias, which consists of a vector and a linear layer  $(L_{\alpha})$ . The vector represents the task-specific shift, and  $L_{\alpha}$  produces the weights for input tokens. Thus, with the vector and the weights, AdapterBias can add a token-dependent shift to the transformer layer. Since the concept of BitFit (Ben Zaken et al., 2021) is similar to AdapterBias by adding a shift to the representation, we demonstrate the difference between BitFit and AdapterBias in Figure 1. BitFit assigns identical shifts to all the tokens, while AdapterBias adds more significant shifts to the representations that are related to the task.
With fewer trainable parameters required, AdapterBias achieves comparable performance on the GLUE benchmark with Houlsby et al. (2019); Pfeiffer et al. (2020a); Guo et al. (2020); Ben Zaken et al. (2021); Hu et al. (2021). We further decrease the parameters of AdapterBias in different ways, including partial weight-sharing in AdapterBias and adding  $L_{0}$ -norm regularization. Finally, AdapterBias has better interpretability due to its simplicity. We use different tools, including word cloud and PCA (Jolliffe, 2002), to visualize what AdapterBias has learned, and we found that the proposed approach automatically learns to assign larger representation shifts to the task-related tokens.
For NLP tasks, adapters are introduced for the transformer architecture. A set of adapter parameters was added at each transformer layer, which is mostly bottleneck architectures Houlsby et al. (2019). By keeping the output dimension identical,
they cause no change to the structure or parameters of the original model.
Adapters quickly gained popularity in NLP with various applications. For multi-task learning (Caruana, 1997; Zhang and Yang, 2017; Liu et al., 2019b), a projected self-attention layer is proposed by Stickland and Murray (2019), while Bapna et al. (2019) proposed an additional layer norm suitable for machine translation.
Besides the applications of adapters, researchers are also dedicated to improving their performance. Based on the architecture introduced by Houlsby et al. (2019), AdapterFusion (Pfeiffer et al., 2020a) leveraged knowledge from multiple tasks with a new two-stage learning algorithm. Despite the recent popularity of these methods, they still train a relatively large number of training parameters.
Recently, studies start to focus on improving the parameter-efficiency of adaptation to a new task. Diff-pruning (Guo et al., 2020) achieves parameter efficiency by adding a sparse, task-specific difference-vector to the fixed original parameters. The vector is adaptively pruned during training with a differentiable approximation to the  $L_{0}$ -norm penalty to encourage sparsity. Rucklé et al. (2020) introduced AdapterDrop, which has been recently integrated into AdapterHub (Pfeiffer et al., 2020b). It removes adapters from lower transformer layers during training and inference, which can dynamically reduce the computational cost. Mahabadi et al. (2021) proposed Compacter, which improved the trade-off between performance and trainable parameters per task with low-rank optimization.
On the other hand, without modifying the architecture of the PLM, BitFit (Ben Zaken et al., 2021) shows that fine-tuning only the bias terms of a large PLM is also competitive with fine-tuning the entire model. Fine-tuning only the bias terms can be considered as adding a task-specific shift to the token representation. BitFit is most similar to our work. While in BitFit, the shifts added to all the representations are exactly the same for all input tokens, in our work, the shifts are token-dependent.
In this section, we present AdapterBias, an efficient way to adapt large-scale PLMs. In order to better adapt to different downstream tasks, the adapter module should be token-specific. AdapterBias produces a suitable weight for the bias based on the input token.
Figure 2: Model architectures comparison of Houlsby et al. (2019), BitFit (Ben Zaken et al., 2021), and the proposed method AdapterBias. The orange blocks indicate the trainable parts, while the gray blocks indicate the frozen parameters during the training stage. Left: Houlsby et al. (2019) add their Adapters after the feed-forward layers, and their Adapter consists of two linear layers and an active function. Middle: BitFit tunes all biases from the original transformer layers. Right: AdapterBias, consisting of a linear layer  $(L_{\alpha})$  and a vector  $(\nu)$ , is added after the second feed-forward layer only in each transformer layer.
Problem Formulation We consider the general problem of fine-tuning PLMs, where the training data  $D = (x_{i},y_{i})_{n = 1}^{N}$  is given. Assume that given a PLM with parameters  $\theta$  and AdapterBias with parameters  $\theta^{\prime}$ . During the training stage, we freeze  $\theta$  and tune  $\theta^{\prime}$  only.
3.1 AdapterBias
The architecture of AdapterBias is shown in the right part of Figure 2. AdapterBias consists of two modules: a vector  $(\nu)$  and a linear layer  $(L_{\alpha})$ .  $\nu$  is a task-specific shift added to the output of each transformer layer. The tokens which are more related to the task should be assigned larger representation shifts than other tokens. The linear layer  $(L_{\alpha})$  produces a token-dependent weight vector  $\alpha = [\alpha_{1},\alpha_{2}\dots \alpha_{m}]^{T}$ , where  $\alpha_{i}$  is the weight of the  $i^{th}$  token's representation shift. By applying the token-specific weight to the task-specific representation shift  $(\nu)$ , AdapterBias can focus on the tokens that are more important to the task and is able to adapt to different downstream tasks efficiently.
We define the output of AdapterBias as the bias  $(B)$ , which is the outer product of  $\nu$  and the learned weights vector  $\alpha$ . When the dimension of the token's representation is  $r$  with  $m$  input tokens, the function can be defined as follows:
$$
B = v \otimes \alpha^ {T} = \left( \begin{array}{l l l l} \alpha_ {1} v & \alpha_ {2} v & \dots & \alpha_ {m} v \end{array} \right) \quad (1)
$$
where  $v\in \mathbb{R}^r$ $\alpha \in \mathbb{R}^{m}$  , and  $B\in \mathbb{R}^{r\times m}$
To further elaborate on the details of AdapterBias, we give an example of how AdapterBias produces  $B$  and how  $B$  adds to the transformer layer. In Figure 3, we assume that there are three representation outputs  $(r_1, r_2, r_3)$  after the first layer normalization. The dimension of  $r_1$ ,  $r_2$  and  $r_3$  is the dimension of the  $2^{\mathrm{nd}}$  feedforward layer, while the input dimension of the linear layer  $(L_{\alpha})$  is the output dimension of the first feed-forward layer with the token representation  $(r_1, r_2, r_3)$  as its inputs. The linear layer  $(L_{\alpha})$  produces  $\alpha$ , where  $\alpha \in \mathbb{R}^3$ . The blocks in different colors represent the difference of the weights  $(\alpha_1, \alpha_2, \alpha_3)$ . Take BERT-base for example, after performing outer product with the weights vector  $\alpha$  and the vector  $(\nu)$ , the dimension of  $B$  becomes  $768 \times 3$ . For example,  $b_1$ , the first column of  $B$ , is the shift for the first token representation.
3.2 Further improvement on parameter-efficiency of AdapterBias
In this section, we experiment on two different methods to make AdapterBias more parameter efficient. One is partial weight-sharing of AdapterBias among transformer layers, another is enforcing the weights of the linear layer  $(L_{\alpha})$  to be sparse by utilizing  $L_{0}$ -norm penalty.
3.2.1 Cross-layer parameters sharing in AdapterBias
Redundancies have been observed in the information captured by adapters, with adapters in lower layers being less important (Houlsby et al., 2019). In addition, sharing parameters of the Adapter across layers leads to a comparatively small drop in performance in some tasks. In light of the above information, we further reduce the number of parameters required for each task by partially sharing the weights of the adapters across all transformer layers. The experimental results are discussed at Section 4.6.1.
3.2.2  $L_{0}$  regularization in AdapterBias
Sparsity has been utilized in various parameter-efficient methods. For applications in NLP tasks, Diff-pruning (Guo et al., 2020) learns a sparse vector added to the whole PLM with  $L_{0}$ -norm penalty. Inspired by their work, we further apply  $L_{0}$ -norm regularization to  $L_{\alpha}$  in the AdapterBias module, aiming to encourage the sparsity of  $L_{\alpha}$ . We choose to drop  $L_{\alpha}$  because it contributes most of the parameters in AdapterBias. Encouraging its sparsity can further increase the parameter efficiency. Note that we specifically apply  $L_{0}$  regularization in Section 4.6.2.
In AdapterBias, we add  $L_{0}$ -norm penalty to the linear layer  $(L_{\alpha})$ . The optimization problem can be expressed as,
$$
\min  _ {\theta^ {\prime}} L (D; \theta , \theta^ {\prime}) + \lambda \| \theta_ {L _ {\alpha}} ^ {\prime} \| _ {0}, \tag {2}
$$
where  $L(D; \cdot)$  represents the original loss with training data  $D$ .  $\lambda$  is the hyperparameter for  $L_0$ -norm penalty. Note that  $\theta'$  represents trainable parameters and  $\theta_{L_\alpha}'$  represents the parameters of  $L_\alpha$  in AdapterBias. Following the work of Diff-pruning, we utilize a relaxed mask vector (Louizos et al., 2017) with a stretched Hard-Concrete distribution (Jang et al., 2016; Maddison et al., 2016) to encourage  $L_0$  sparsity.
In this section, we evaluate the effectiveness of our proposed adapter module in NLP training tasks, and provide the analysis of what AdapterBias has learned in different tasks.
We base our experiments on HuggingFace PyTorch implementation (Wolf et al., 2019) of BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019c)
Figure 3: The detailed architecture of how AdapterBias produces the bias  $(B)$  and how  $B$  is added to the output of transformer layers.
models. The learning rate is set in the range  $[10^{-4}, 10^{-3}]$ , with AdamW (Loshchilov and Hutter, 2017) as the optimizer. GLUE benchmark (Wang et al., 2018) and SQuAD v1.0 (Rajpurkar et al., 2016) are the training data in our settings.
The training details are shown in Appendix A.3. Note that the second layer normalization in each transformer layer is also tuned during the training stage, corresponding to the orange component in the right part of Figure 2. We experiment with 3 random seeds and choose the seed with the best performance on the validation set to evaluate on the GLUE server. We report the test metrics provided on the submission website<sup>2</sup>.
4.2 Results on GLUE
In this section, we compare AdapterBias to other parameter-efficient methods, including Adapters (Houlsby et al., 2019), Diff-pruning (Guo et al., 2020), BitFit (Ben Zaken et al., 2021), and LoRA (Hu et al., 2021). In Table 1, we report the test scores on the GLUE benchmark and the required new parameters per task. Here we use BERTlarge as the PLM. AdapterBias reaches 81.2 average score in GLUE benchmark, with the smallest amount of parameters (0.17M) added per task. AdapterBias shows competitive performance as its
$^{2}$ https://gluebenchmark.com/
Method	Params	CoLA	SST-2	MRPC	QNLI	RTE	STS-B	MNLI-m	MNLI-mm	QQP	Avg
BERTLARGE	340M	60.5	94.9	89.3	92.7	70.1	87.6	86.7	85.9	72.1	82.2
Adapters (Houlsby et al., 2019)	7.14M	56.9	94.2	89.6	91.4	68.8	87.3	85.3	84.6	71.8	81.1
Diff-Pruning (Guo et al., 2020)	1.7M	61.1	94.1	89.7	93.3	70.6	86.0	86.4	86.0	71.1	82.0
BitFit (Ben Zaken et al., 2021)	0.27M	59.7	94.1	88.9	92.0	72.0	85.5	84.5	84.8	70.5	81.3
LoRA (Hu et al., 2021)	0.39M	60.6	94.0	87.9	92.2	70.3	85.6	84.2	84.0	70.0	81.0
AdapterBias	0.17M	60.0	94.4	88.2	91.2	70.5	87.5	84.3	83.9	70.5	81.2
Table 1: Performance of all methods on the GLUE testing sets scored by the GLUE evaluation server. For each method, we report the new adding parameters per task. For QQP, we report the F1 score. For STS-B (Cer et al., 2017), we report Spearman correlation coefficients. For CoLA (Warstadt et al., 2019), we report Matthews correlation. For all other tasks, we report accuracy. Bold fonts indicate the least trainable parameter per task. The first row  $(\mathrm{BERT}_{\mathrm{LARGE}})$  represents fine-tuning the whole BERT-large model without adding new parameters. The results of baselines including (Houlsby et al., 2019; Guo et al., 2020; Ben Zaken et al., 2021) are their reported performance and Pfeiffer et al. (2020a); Hu et al. (2021) performance is reproduced on our setting. Due to instability during training, we restart experiments with 3 random seeds and report the best.
Method	Params	CoLA	SST-2	MRPC	QNLI	RTE	STS-B	MNLI-m	MNLI-mm	QQP	Avg
BB	Full-FT	110M	52.1	93.5	88.9	90.5	66.4	85.8	84.6	83.4	71.2	79.6
BB	BitFit	0.10M	47.2	92.4	87.4	89.7	65.5	87.6	80.8	80.9	67.8	77.7
BB	AdapterBias	0.06M	51.6	93.1	87.5	89.4	66.1	84.6	80.9	80.5	67.9	78.0
BL	Full-FT	340M	60.5	94.9	89.3	92.7	70.1	87.6	86.7	85.9	72.1	82.2
BL	BitFit	0.27M	62.0	93.1	86.8	89.8	66.6	87.2	84.1	84.3	67.2	80.1
BL	AdapterBias	0.17M	60.0	94.4	88.2	91.2	70.5	87.5	84.3	83.9	70.5	81.2
RoB	Full-FT	125M	61.3	94.7	90.4	92.0	74.4	87.5	87.4	86.8	71.9	82.9
RoB	BitFit	0.10M	62.7	94.8	89.7	91.3	73.6	88.5	85.3	84.9	68.1	82.1
RoB	AdapterBias	0.06M	61.9	94.5	90.2	91.1	74.1	88.7	85.3	85.1	70.5	82.4
RoL	Full-FT	355M	63.3	96.7	92.3	95.4	84.5	92.2	90.8	90.2	74.3	86.6
RoL	BitFit	0.26M	64.7	95.8	91.5	94.2	80.9	90.6	89	88.9	72.0	85.3
RoL	AdapterBias	0.17M	63.9	96.4	90.4	94.7	83.6	91.3	89.8	89.4	72.3	85.8
Table 2: Performance of AdapterBias adding in different PLMs. Here we experiment with four models : BERT-base (BB), BERT-large (BL), RoBERTa-base (RoB), and RoBERTa-large (RoL). The settings are the same as in Table 1. The Full-FT corresponds to fine-tuning the whole PLM without adding adapters.
parameters are  $40 \times$  less than the works of Houlsby et al. (2019). Although Diff-pruning (Guo et al., 2020) achieves the best average score among all parameter-efficient methods, their work trains an additional vector whose parameter count is equivalent to the parameters of the whole PLM. Thus, Diff-pruning requires 340M trainable parameters of BERT-large during the training stage, while AdapterBias only trains 0.17M parameters. Furthermore, AdapterBias achieves comparable performance with BitFit and LoRA with fewer parameters needed per task. This shows that AdapterBias is a worthwhile targeted fine-tuning method.
4.3 Different base models
To analyze the generalization ability of this approach to different PLMs on different models of AdapterBias, as shown in Table 2, we apply AdapterBias in different transformer-based PLMs, including BERT-base (BB), BERT-large (BL), RoBERTa-base (RoB), and RoBERTa-large
(RoL), on the GLUE benchmark. All results are scored by the GLUE evaluation server. Compared with BitFit, In Table 2, not only can AdapterBias perform well on BERT but also achieve competitive performance on larger PLMs such as RoBERTa.
4.4 Size of training data
In the previous experimental results, we observe that AdapterBias tends to have higher performance on tasks with a smaller amount of data (i.e. CoLA, SST-2, and RTE). To further validate this observation, we follow the work of BitFit (Ben Zaken et al., 2021) by training AdapterBias on subsets of SQuAD v1.0 (Rajpurkar et al., 2016) of increasing size. The experiments are conducted with BERT-base. The results on the validation set of the SQuAD dataset are listed in Figure 4, which shows the tendency of AdapterBias outperforming full fine-tuning when the size of the training dataset is smaller. However, with more training data available, the trend is reversed. The results
Method	Params	CoLA	SST-2	MRPC	QNLI	RTE	STS-B	MNLI-m	MNLI-mm	QQP	Avg
w/o Lα	27.6K	45.6	91.5	87.4	88.3	65.6	81.0	77.9	78.4	65.7	75.7
AdapterBias	64.5K	51.6	93.1	87.5	89.4	66.1	84.6	80.9	80.5	67.9	78.0
Table 3: Evaluating the importance of the linear layer  $(L_{\alpha})$  in AdapterBias. The settings are the same as in Table 1. The backbone model is BERT-base. w/o  $L_{\alpha}$  means that there is only a vector  $(\nu)$  in AdapterBias.
Figure 4: Comparison of Finetune, BitFit (Ben Zaken et al., 2021), and AdapterBias with BERT-base on SQuAD validation set. The x-axis represents the total number of training examples while the y-axis represents the exact match score.
show that AdapterBias has the ability to outperform fine-tuning the whole PLM with small-to-medium data size, similarly to BitFit.
4.5 Investigation on the effectiveness of token dependent representation shift
Different from BitFit (Ben Zaken et al., 2021), where the bias terms in all transformer layers are tuned, we claim that the bias added to the representation should be token-dependent, and proposed AdapterBias based on this concept. We conduct ablation studies to verify this claim. In this experiment, the linear layer  $(L_{\alpha})$  in AdapterBias that produces the token-dependent weights vector  $(\alpha)$  is removed; that is, only the  $\nu$  is trained. All shifts added to the representation outputs are identical within the same transformer layer. The experiments are conducted with BERT-base model. We report the test scores on the GLUE benchmark in Table 3. The performance of AdapterBias without the linear layer  $(L_{\alpha})$  dramatically decreases. Without  $L_{\alpha}$ , it is hard for the vector  $(\nu)$  to adapt to different downstream tasks. This result demonstrates the importance of  $L_{\alpha}$ . In other words, assigning different shifts to different token representations improves
the performance of the method.
4.6 Improving the parameter efficiency of AdapterBias
We further apply two additional methods to AdapterBias to enhance its parameter efficiency. Experiments are conducted to examine whether AdapterBias can be more parameter-efficient by sharing its components across all layers. Moreover, we experiment on adding  $L_{0}$ -norm regularization during the training stage to encourage the sparsity of AdapterBias.
4.6.1 Sharing components in AdapterBias
In this experiment, we conduct an ablation study of partial weight-sharing in the AdapterBias module. In Table 4, we share components of Adapter-Bias among different transformer layers. Share  $\nu$  represents sharing  $\nu$  across all transformer layers, while Share  $L_{\alpha}$  means sharing the linear layer  $(L_{\alpha})$ . Share  $\nu + L_{\alpha}$  denotes sharing one Adapter-Bias across all transformer layers. As can be seen in Table 4, the performance of Share  $L_{\alpha}$  stands out among other partial weight-sharing methods, while Share  $\nu$  leads to a poor performance.
From the experiments above, we conclude that the linear layer  $(L_{\alpha})$  captures general task information by learning the weights of the bias for different tokens. Thus, sharing  $L_{\alpha}$  across all layers results in better performance compared to other components. The vector module  $(\nu)$  in AdapterBias aims to learn local information in each transformer layer. If  $\nu$  among different transformer layers are shared, the performance drops dramatically. This might be due to a failure of  $\nu$  to learn general information which can be adapted to each individual transformer layer.
4.6.2  $L_{0}$ -norm regularization in AdapterBias