SlowGuess/NAACL_2022_OCR huggingface dataset:
Pruning Adapterfusion with Lottery Ticket Hypothesis
Jiarun Wu $^{1}$ , Qingliang Chen $^{*,2}$ , Zeguan Xiao $^{1}$ , Yuliang Gu $^{1}$ , Mengsi Sun $^{1}$ $^{1}$ Department of Computer Science, Jinan University, Guangzhou 510632, China  
 $^{2}$ Yunqu-Jinan University Joint AI Research Center, Guangzhou 510632, China
Pre-trained language models have shown great success in multiple downstream tasks. However, they are computationally expensive to fine-tune. Thus, transfer learning with adapter modules has been introduced to alleviate this problem, helping to extract knowledge of the downstream tasks. Adapterfusion models are an example of the transformers-with-adapter-modules, which merge multiple adapters to incorporate knowledge from different tasks. However, merging multiple adapters will inevitably cause redundancies, increasing the training and inference time massively. Therefore, in this paper, we propose an approach to identify the influence of each adapter module and a novel way to prune adapters based on the prestigious Lottery Ticket Hypothesis. Experiments on GLUE datasets show that the pruned Adapterfusion model with our scheme can achieve state-of-the-art results, reducing sizes significantly while keeping performance intact.
Transfer learning with transformer-based pretrained language model has become a go-to method for solving multiple NLP tasks (Vaswani et al., 2017). The language models are pre-trained on large amounts of unlabeled text data with methods such as masked language modeling (e.g. BERT (Devlin et al., 2019a), Roberta (Liu et al., 2019), and XLNet (Yang et al., 2019)). Despite the state-of-the-art performance for most natural language understanding tasks, they are notoriously deep requiring millions or even billions of parameters to gain great results (Kaplan et al., 2020). For different tasks, models needs to be fine-tuned entirely, which is computationally expensive and requires large storage.
Therefore, adapter modules (Houlsby et al., 2019) are introduced to tackle this issue. It's an
Figure 1: Pruning Adapterfusion. Adapters from different tasks are combined into Adapterfusion after pruning with proposed method separately.
alternative way of transfer learning that achieves comparable performance to full fine-tuning on most NLP tasks, without the need of fine-tuning the whole model for a downstream task. Adapter is a small residual neural network inserted in each layer of the transformer. During training, only the parameters in adapters are fine-tuned, while the rest of the parameters are frozen. This approach can reduce the number of parameters needed to be trained at the training phase and enable parameter sharing among tasks. Moreover, recent studies have revealed that the adapter is capable of extracting knowledge from the target task (Rücklé et al., 2020b; Pfeiffer et al., 2020), so research attempts have also been made to fuse multiple adapters across multiple tasks to incorporate different aspects of knowledge (e.g. Adapterfusion (Pfeiffer et al., 2021), K-adapter Wang et al. (2021)). How
* Corresponding author: tpchen@jnu.edu.cn
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1632 - 1646 July 10-15, 2022 ©2022 Association for Computational Linguistics
ever, the fusing of adapters in these models can inevitably cause a lot of redundancies. So, Rücklé et al. (2020a) have recently proposed AdapterDrop which aims to drop the redundant adapters. They tried to remove adapters from lower transformer layers during training and inferences, resulting in faster training and inference speed with some performance cost. However, the utilization of each adapter has not been fully analysed yet and how to introduce new pruning strategies remains to be explored.
To address these deficits, in this paper, we propose an approach to model the utilization of different adapters in the transformer layer, and a novel way to prune adapters in the model while keeping the loss of the performance to be negligible. The contributions are summarized as follows:
- We propose a new indicator LIA (Layer Influence Of Adapter) to quantify the utilization of adapters at each layer and identify the most influential adapters in the model.
- We introduce a novel way for pruning adapter modules, inspired by the prestigious Lottery Ticket Hypothesis (Frankle and Carbin, 2019), which states that dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that can have test accuracy comparable to the original network in a similar number of iterations when trained in isolation.
- We have evaluated the proposed approach on the GLUE datasets. For the performance and LIAs of the pruned adapters in the latest state-of-the-art Adapterfusion model, we can remove more than half of the adapters and reduce computation of the Adapterfusion model by nearly  $40\%$  with little performance loss.
2 Adapterfusion pruning
Adapterfusion model (Pfeiffer et al., 2021) is to merge multiple adapters from different tasks. And inference time of the model increases drastically after the fusion. However, not all the adapter modules in the model are utilized in the downstream task. We prune adapter modules from model to improve inference speed. Our proposed pruning strategy to remove redundant adapters from Adapterfusion model will be two-stage. First, we prune single task adapters before the fusion using Lottery Ticket Hypothesis (Frankle and Carbin, 2019). We then fuse
the less utilized adapters after training the model. The whole framework is presented in Figure 1. To identify the roles of adapters, we firstly define a new indicator LIA (Layer Influence Of Adapter) for the adapter module to measure its utilization in each layer.
2.1 Pruning single task adapter with Lottery Ticket Hypothesis
Since not all adapters in the model are created equal, removing some of the adapters does not compromise the performance too much. We will prune the single task adapter before the Adapterfusion.
Inspired by Lottery Ticket Hypothesis (Frankle and Carbin, 2019), we prune the adapter iteratively to find the sub-network (winning ticket) that can reach the same accuracy when trained in isolation. After every pruning, we reinitialize the weights of the adapter to the initial values when the first iteration starts.
We explore to find the winning ticket in adapters by training and pruning them iteratively. Since the importance of adapters is different in each layer, we are performing the pruning globally. We train the transformer model with adapters as  $f(x; \theta_0; \alpha)$  with initial parameter in adapters  $\theta = \theta_0 \sim D_\theta$  and transformer parameter  $\alpha = \alpha \sim D_\alpha$ . Then the winning ticket can be found by the following steps:
1. Randomly initialize adapter parameters in the model  $f(x; \theta_0; \alpha)$ .
2. Train the adapters for  $j$  iterations, arriving at parameters  $\theta_{j}$
3. Prune  $p\%$  of the adapters in  $\theta_{j}$ .
4. Reset the remaining parameters in adapters to  $\theta_0$ , and go back step 2 to train the model  $f(x; \theta_0; \alpha)$  if it is not a winning ticket yet.
We prune adapters based on their sum of weights. Here we do not use LIA yet because it only represents the influence of adapter at each layer and it can not distinguish the influence between layers.
Let  $\theta_{t,l}$  be the weights of the adapter at layer  $l$  at iteration  $t$  and  $a_{i,j}$  denote the parameters in  $\theta_{t,l}$ . The importance of an adapter of size  $N$  with input size of  $H$  is  $\sum_{i,j}^{N,H}|a_{i,j}|$ . Adapters are then sorted by the sum of weights in the descending order as well, and the  $p\%$  smallest adapters in list  $R$  are removed from the model. And the remaining adapters step back to their initial weights for the
re-training. The whole procedure is elaborated in Algorithm 1. See Appendix B for more details.
2.2 Layer influence of Adapter
To identify the usage of adapter module, we propose a new indicator LIA(Layer influence of Adapter).
Figure 2: Layer influence of Adapter.
In a adapter module, let  $\vec{a}$  denote the output of adapter up projection module,  $\vec{b}$  be the residual connection of the adapter, and  $\vec{c}$  be the output of the adapter. See Figure2 for more detail. Their connection can be modeled as equation 1:
$$
\vec {a} + \vec {b} = \vec {c} \tag {1}
$$
Let  $\vec{d}$  denote the projection of  $\vec{a}$  on  $\vec{c}$ ,  $\vec{e}$  be the projection of  $\vec{b}$ . Therefore  $\vec{c}$  is the combination of  $\vec{d}$  and  $\vec{e}$ .
Since the activation of adapters varies between different inputs and different layers, it's difficult to
see the influence of adapter in each layer based on the activation. So we use the projection vector  $\vec{d}$  to represent  $\vec{a}$  influence and normalize it by the length of  $\vec{c}$ . We name this quantity as Layer Influence Of Adapter (LIA), which is defined as:
$$
L I A _ {\vec {a}} = \frac {| \vec {a} | \times \cos \theta}{| \vec {c} |} = \frac {\vec {a} \cdot \vec {c}}{| \vec {c} | ^ {2}} \tag {2}
$$
With LIA, we can model the importance of adapters and streamline the model accordingly. We compare the LIAs of adapter before and after the pruning.
In Adapterfusion model, adapters are fused with a adapterfusion layer which is a self-attention layer. It calculate the attention of adapters and use the weighted sum of adapters as the output. Their connection can be modeled as equation 3:
$$
\vec {c} = \vec {b} + \sum_ {i = 1} ^ {n} W _ {i} * \vec {a} _ {i} \tag {3}
$$
We use  $W_{i} * \vec{a}_{i}$  to calculate the LIA of each adapter. We compare LIA between different adapters to show whether the adapters are fully utilized, and identify which adapter is more useful for certain layer.
3 Experimental studies
In this section, we examine the influence of the single task adapter under different downstream tasks, and present the results of our proposed pruning scheme evaluated on the prestigious GLUE datasets (Wang et al., 2018).
We use the public BERT-Based uncased model which has 12 layers and a total of 110M parameters as our base model. And we apply the similar approach in (Devlin et al., 2019b) to perform a text classification task. In each input sequence, the first token is a classification token. Its embedding is then fed into a linear layer to make a prediction. In the training of Adapterfusion, a new linear layer is initialized for classification and Adapterfusion model is inserted in each transformer layer.
In the experiment of pruning single task adapters, we set the adapter size to 128 because engineering practices (Bengio et al., 2005) suggest that overparameterized networks are easier to train. We use Adam optimizer to train the single task adapter model and perform hyperparameter search using TPE algorithm (Bergstra et al., 2011). We run 30
trials on learning rate settings in  $\{1\times 10^{-4},5\times$ $10^{-4},1\times 10^{-3}\}$  , and number of epochs in  $\{3,4,5\}$  We select the best settings for pruning experiments. We prune  $20\%$  of the adapters from the model at each pruning iteration, and use an early-stopping strategy with patience of 3 to speed up training, and we use the minimum validation loss for earlystopping criterion.
In the experiment of pruning Adapterfusion model, we set the learning rate to  $5 \times 10^{-5}$  and use AdamW optimizer as suggested by Pfeiffer et al. (2021). We run each task with 4 epochs and set the batch size to 32, and each model for each task with five different random seeds.
We have evaluated the single adapter with pruning, and Adapterfusion on GLUE datasets (Wang et al., 2018), which contain eight sentence or sentence-pair language understanding tasks. And we treat MNLI  $\mathrm{mm}$ , MNLI  $\mathrm{m}$  equally. We have reported the test results of the single task adapter through the GLUE submission website.
3.2 LIAs in single task adapter
In order to analyse the influence of single task adapter at each layer, we run a test on the standard adapters and analyse the utilization of each of them.
In the evaluation step, we store the residual output and the output of each adapter to calculate the LIAs at each step. We then average LIAs of each adapter across the datasets. And we run the test from tasks with small datasets to large datasets, whose results are shown in Figure 3.
We have found that as the size of dataset gets bigger, LIAs of adapters also become larger, implying that the size of target task dataset affects the influence of the adapters. After training on larger datasets, adapters learn more and extract more knowledge, and thus become more essential for the whole model. This could explain why in Adapterfusion (Pfeiffer et al., 2021), using adapters from large datasets can help improve the performance of the task with small datasets.
For adapters in large dataset task  $(\mathsf{QQP}^3$  QNLI(Rajpurkar et al., 2016), MNLI(Williams et al., 2018)), most of them have large LIAs, suggesting they are already concise and there are no
many redundant parameters in them.
3.3 Pruning single task adapter
We insert adapters of size 128 into each layer of transformers in the BERT-Based model. For different text classification task, we put a task-specific classifier at the end of the model. Only the parameters in the adapters and task-specific classifier are fine-tuned, and the rest of the parameters in the model are untouched.
We iteratively prune the parameters in the adapter by  $20\%$  per iteration. We perform 11 iterations for layer pruning since there will be less than one adapter left after the 11-th iteration.
Results on GLUE test sets are presented in Table 1. We select the best result in all iterations of pruning. The best model is chosen by metrics of the corresponding task. And we evaluate the result on GLUE testing sever. We can see there is a 0.2 percentage performance gap between the full fine-tuning model and the adapter model, and there is a small performance gap between the adapter model and the pruned adapter model. Therefore, we preserve most of the essential adapters while pruning the redundant ones.
We evaluate the adapter model on GLUE development datasets after each iteration of pruning and obtain the average score of three runs, see Figure 5. We discover that there is no major performance loss before the number of adapters drops below 9 (40%). By contrast, AdapterDrop model (Rücklé et al., 2020a) removed the first 5 layers of adapters and preserve most of the performance with about 60% of the adapters left. We thus can prune adapters 20% further than their work. The speed comparison between the two models is shown in Table 2. We discover that our model is faster than AdapterDrop in most tasks.
We further analyse how the adapters are distributed when there is only nine adapters left, see Figure 6. Interestingly, we can see that most of the adapters close to the output layer are pruned. These layers are removed but no harm is done to the performance, and we think that maybe the last few layers are just a redundant extension of classification layer.
We also find that there are more adapters after feed-forward layers than self-attention layers, implying that adapters after feed-forward layers are more valuable. Similar phenomena can be found in the experiments of ALBERT (Lan et al., 2020),
<sup>1</sup>We omit WNLI because it is not evaluated in BERT (Devlin et al., 2019a). <sup>2</sup>https://gluebenchmark.com <sup>3</sup>https://quoradata.quora.com/ First-Quora-Dataset-Release-Question-Pair
Figure 3: LIAs of each layer in single task adapter for different tasks. att denotes the adapter after self-attention layer, out is the adapter after output layer. Darker colors represent higher values of LIA. The target tasks are arranged in the order of the size of datasets from small to large.
CoLA	SST-2	MRPC	STSB	QQP	MNLI	QNLI	RTE	AVG
Full fine-tuning	52.10	93.50	88.90	85.80	71.20	84.60	90.50	66.40	79.60
Adapter128	51.70	93.10	88.50	85.60	71.50	83.40	90.50	67.30	79.42
Prune layer	49.50	92.60	88.00	83.50	71.50	84.10	90.80	70.60	79.30
Figure 4: LIAs of adapters before and after pruning with Lottery Ticket Hypothesis
Table 1: Test results on GLUE test sets using GLUE server. CoLA is evaluated using Matthew's Correlation. STS-B is evaluated using Spearman's correlation coefficient. MRPC and QQP are evaluated using F1 score. The rest of the tasks are evaluated by accuracy.
Origin	AD	LTH	Speed up
AD	LTH
RTE	208.3	203.9	203.5	2.11%	2.30%
MRPC	227.6	222.7	218.3	2.15%	4.09%
STSB	221.6	216.9	217.8	2.12%	1.71%
COLA	63.5	62.1	61.7	2.20%	2.83%
SST2	127.5	124.7	123.9	2.20%	2.82%
QNLI	41.4.	40.5	40.2	2.17%	2.90%
QQP	85.8	83.9	83.7	2.21%	2.45%
MNLI	42.8	41.9	41.5	2.10%	3.04%
Table 2: Floating points  $(10^{9})$  operation origin adapter (Origin), AdapterDrop (AD) and Pruned adapters (LTH) in each tasks and percentage change in speed after using AdapterDrop or Pruned adapters.
where most of the performance drop appears to come from sharing the feed-forward layer parameters, while sharing the attention layer parameters results in no performance loss.
Since different tasks require different number of adapters, in the following experiments, we use the best adapter model of all iterations of pruning in each task.
After using layer pruning with Lottery Ticket Hypothesis, the average influence of each adapter increases as shown in Figure 4. In most tasks, the LIAs increase after the pruning, which means the
redundant part of the adapters are removed. And we can see a significant LIA boost, mostly in tasks of small datasets, especially in MRPC. However, in larger dataset task like MNLI, pruning the adapters causes a small decrease in LIAs, implying that most of the adapters for large datasets are playing an important role for the task. In summary, we have greatly increased the utilization of each adapter after pruning, because we deleted most of the less essential layers, and the model is thus streamlined to perform better.
Figure 5: Performance of pruning schemes on GLUE validation sets at every pruning iteration. Horizontal line represents the performance of adapters before pruning starts. X-axis denotes the number of adapters remains, and Y-axis denotes the score in corresponding task metrics.
Figure 6: Left: Percentage of adapter remaining in each adapter layer when there is only 9 adapters left. Center: Adapter distribution inserted after Feed-forward layer (Extracted from the Left image). Right: Adapter distribution inserted after attention layer (Extracted from the Left image)
3.4 Merging pruned adapters to Adapterfusion
In this experiment, we fuse eight single task adapters to construct the Adapterfusion model. An extra self-attention layer is inserted in each layer of the model to fuse the results of multiple adapters. Only the parameters in these newly inserted self-attention layers are fine-tuned, and the rest of the parameters (including adapters) remains unchanged.
We run our test on the same eight tasks of GLUE datasets as in the previous experiment. We compare the fusion of full-size adapters and the pruned adapters with 5 runs each. We use the best adapter model of all iterations of pruning for the fusion. Then, we calculate the mean and variance of each task, whose results are in Table 3. It shows that there is not much difference between the pruned Adapterfusion model and the full-size Adapterfusion model. And there is even a mild improvement of performance using the pruned one. Therefore, it justifies that the proposed pruning of adapters is very effective and we can obtain a small and dense version of the general Adapterfusion model for GLUE datasets.
The original Adapterfusion model has 192 adapters (8tasks  $\times$  12layer  $\times$  2adapter/layer), while after pruning the redundant adapters from the
AF	AF w. LTH
CoLA	55.16 ±1.1	55.96 ±2.3
SST-2	91.67±0.7	91.87±0.4
MRPC	91.46 ±1.6	92.16 ±1.5
STSB	89.83±0.33	89.27±0.64
QQP	86.74±0.39	86.88±0.22
MNLI	83.13±0.42	83.16±0.18
QNLI	90.84±0.21	90.73±0.33
RTE	75.90±3.43	73.00±6.68
Table 3: Development score of Adapterfusion and Adapterfusion pruned with Lottery Ticket Hypothesis (LTH). CoLA is evaluated using Matthew's Correlation. STS-B is evaluated using Spearman's correlation coefficient. MRPC and QQP are evaluated using F1 score. The rest of the tasks are evaluated by accuracy.
model, it only has 89, with more than half of the adapters removed. For layers left without adapters, there isn't a self-attention layer insertion for fusion, resulting in the reduction of depth as well.
We also have measured the total number of floating point operations (FLOPs) for each task in the evaluation process. We average the FLOPs across different tasks, as shown in Table 4 and Table 5. And it reveals that after pruning, we reduce the computation by about  $40\%$ , which is a very significant improvement for the inference speed.
To better analyse the training and inference speed, we use the basic fine-tuning model as the base model and calculate the relative speed of dif
Num of adapter	FLOPs (109)	Steps/sec
AF	192	554.8	9.86
AF-LTH	89	332.8	13.86
Table 4: FLOPs of the standard Adapterfusion and pruned Adapterfusion
AF	AF-LTH	Saved (%)
RTE	1157.9	703.1	39.23%
MRPC	934.9	565.6	39.50%
STSB	805.9	487.4	39.52%
COLA	221.3	133.6	39.62%
SST2	513.2	310.1	39.57%
QNLI	248.4	150.3	39.49%
QQP	356.1	215.3	39.53%
MNLI	432.2	261.4	39.51%
ferent models including single task adapter model, Adapterfusion model and pruned Adapterfusion model. We use one NVIDIA RTX 3090 with a batch size of 16 to test the models and results are shown in Table 6. We can see that fusing 8 adapters slow down the model both in training and inference. However, using pruning the model can reduce the effect of slowing down to a certain level.
Table 5: Floating points (10 $^9$ ) operation of standard Adapterfusion and pruned Adapterfusion in each tasks
Adapter	AF	AF-LTH
Train	0.93	0.52	0.79
Eval	0.63	0.12	0.18
Table 6: The average relative training speed and inference speed of Adapter, Adapterfusion and pruned Adapterfusion
Moreover, we analyse the LIAs of each adapter for different target tasks. LIAs of the original Adapterfusion and the pruned Adapterfusion are shown in 3D heat map in Figure  $7^{4}$ , where X-axis represents eight different target tasks for the model, y-axis is the source of each fused adapters, z-axis denotes the adapters in different layers ranging from 1 to 24, the odd number layers in z-axis represent the adapter modules inserted after attention layers, and the even number layers in z-axis are the ones inserted after the output layer.
Figure 7(a) shows the LIAs of the original Adapterfusion model. We discover that there are a number of adapters not utilized in the original Adapterfusion model and most of the essential adapters are at the back of the cube which are the adapters trained on large datasets. As the layers
get deeper, more adapters in the model are utilized. Figure 7(b) shows the LIAs of the pruned Adapter-fusion model. Compared to the original Adapter-fusion model, the pruned Adapterfusion model has much fewer adapters. Furthermore, most of the adapter modules have larger LIA values, which implies that most of the adapters become more important for the task.
Then, we average the LIAs of adapters in Adapterfusion across 12 layers and 5 test runs, as shown in Figure 8. We find that for the original Adapterfusion model, most of the adapters' LIAs are zeros, which means that most of them are not used in the model. And most of the tasks use the adapters trained in QQP and MNLI, both of which are tasks with large datasets. Moreover, with a larger dataset, the model will utilize more of the adapters trained from the same task.
By comparing the original Adapterfusion and the pruned Adapterfusion model, we have seen that more of the adapters are utilized after the pruning. And there are fewer zeros of LIAs in the pruned Adapterfusion model, suggesting that remaining adapters after pruning have become much more influential on average and the model are using more adapters from different tasks.