SlowGuess/NAACL_2022_OCR huggingface dataset:
Pre-trained language model Language models pre-trained on large corpora are widely used in multiple NLP tasks to improve performance. However, these models are often very large. Recently, Transformer-based (Vaswani et al., 2017) models have become the most popular pre-trained language models. There are plenty of model variants, such as BERT (Devlin et al., 2019a), GPT-3 (Brown et al., 2020), XLNET (Yang et al., 2019), and Roberta (Liu et al., 2019), etc. Transformer models are huge models, ranging from 110M parameters in BERT-Base to trillions (Fedus et al., 2021; Lepikhin et al., 2020) in the largest, best-performing models. Due to the resource constraints in GPU/TPU memory and computational power, it is difficult to run a large model. So Lan et al. (2020) propose an approach to reduce the amount of training parameters by sharing weights among all transformer layers. The model named ALBERT can lower the usage of memory and speed up the training process of BERT. By contrast, ALBERT reduces the amount of parameters needed to be trained, while adapters introduce new parameters and deepen the model.
4In Appendix A, we present an in-depth visualization of the Adapterfusion with LIAs
(a) Adapterfusion
(b) Adapterfusion with LTH
(a) Adapterfusion
Figure 8: LIAs of Adapterfusion w/o pruning
Figure 7: LIAs of Adapterfusion w/o pruning.
(b) Adapterfusion with LTH
Adapters and fine-tuning Since fine-tuning approaches have proven to have better performance than feature-based approaches (Peters et al., 2019), researchers often prefer fine-tuning approaches to feature-based ones. Most of the state-of-the-art results of NLP tasks are achieved by fine-tuning a complex pre-trained model. Fine-tuning does not require a task-specific design beforehand, which is more general across tasks than feature-based ones. However, every time a model is fine-tuned on a new task, a new set of parameters are created and trained, leading to pretty low degree of parameter sharing among tasks.
Adapter model is a lightweight fine-tuning approach introduced by Houlsby et al. (2019). They insert a small set of newly initialized neural networks named adapters in each layer of the transformers. At training step, only parameters of adapters will be updated and the parameters in the pre-trained language model will be unchanged.
Therefore it reduces the number of parameters to be trained in the training phase and enables efficient parameter sharing between tasks by combining many task-specific or language-specific adapters.
Merging and pruning adapters Adapters have achieved great results in multi-task (Pfeiffer et al., 2021), cross-lingual transfer learning (Pfeiffer et al., 2020) and infusing knowledge (Wang et al., 2021). Adapters are capable of extracting knowledge from different tasks that can be applied to fuse knowledge they learned from different tasks (Rücklé et al., 2020b; Pfeiffer et al., 2020; Wang et al., 2021). There are plenty of ways to merge multiple adapters, including stacking (Pfeiffer et al., 2020), fusing and concatenating adapters (Pfeiffer et al., 2021). However, adapters are still far from being concise, and merging multiple adapters from different tasks will introduce redundancies and slow down the inference speed of the model. Therefore, Rucklé et al. (2020a) have firstly intro
duced a way to remove adapters from lower transformer layers. By removing the first few layers of the adapters, it effectively speeds up the training and inference of the adapter models.
Neural networks are easily overparameterized and carry plenty of redundancies. To tackle this problem, distillation (Ba and Caruana, 2014; Hinton et al., 2015) and pruning (LeCun et al., 1990; Han et al., 2015) are introduced to streamline the model while perserving good performance. And there are several research directions in this field. For pruning before training, MobileNets (Howard et al., 2017) is designed for image-recognition networks. For pruning after training, LeCun et al. (1990) use the second derivatives to truncate the neural networks. For pruning during training, Bellec et al. (2018) reinitialize weights near zeros with random number after training the model. Moreover, we can prune models based on activations (Hu et al., 2016), filters (Li et al., 2017; Molchanov et al., 2017) or channels (He et al., 2017).
The most influential theory recently for pruning comes from Frankle and Carbin (2019), in which they prove that a dense neural network contains subnetworks (winning ticket) that can have the same performance as the original network when trained isolated. Their experiments reveal that not only the structure of the pruned networks matters but also the initial weights of these networks can affect the performance of the model. They also find that a subnetwork extracted from pruning learns faster than the original model and even reaches higher test accuracy. Our pruning approach is inspired by this theory and can prune the original Adapterfusion model to a much more concise one.
In this paper, we propose a new approach to model the utilization of the adapters at each layer by defining a new indicator LIA (Layer Influence Of Adapter) with which we can identify the most influential adapters. Moreover, we introduce a novel way of pruning adapter modules inspired by the prestigious Lottery Ticket Hypothesis. The proposed pruning strategy has been extensively evaluated on the GLUE datasets, whose results show that we can prune adapters up to  $40\%$  of its original size while keeping the performance intact. We further examine the performance and LIAs of the pruned adapters in the latest state-of-the-art Adapterfusion model, and we can remove more than half of the
adapters and reduce computation of the Adapterfusion model by nearly  $40\%$  with little performance loss.
This work can be further extended in many ways. For instance, iterative pruning is time-consuming. Running 10 iterations of pruning also means training the model for almost 10 times. We can try to find the redundant adapters before and after the training by introducing new elaborate measurements.
This research is supported by Qinghai Provincial Science and Technology Research Program (grant No.2021-QY-206), National Natural Science Foundation of China (grant No. 62071201), and the Natural Science Foundation of Guangdong (grant No. 2022A1515010119).
Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2654-2662.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert A. Legenstein. 2018. Deep rewiring: Training very sparse deep networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Yoshua Bengio, Nicolas Le Roux, Pascal Vincent, Olivier Delalleau, and Patrice Marcotte. 2005. Convex neural networks. In Advances in Neural Information Processing Systems 18 [Neural Information Processing Systems, NIPS 2005, December 5-8, 2005, Vancouver, British Columbia, Canada], pages 123-130.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pages 2546-2554.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1135-1143.
Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 1398-1406. IEEE Computer Society.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. 2016. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yann LeCun, John Denker, and Sara Solla. 1990. Optimal brain damage. In Advances in Neural Information Processing Systems, volume 2, pages 598-605. Morgan-Kaufmann.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan First, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017. Pruning filters for efficient convnets. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP-2019), pages 7-14, Florence, Italy. Association for Computational Linguistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Andreas Rückle, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020a. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918.
Andreas Rückle, Jonas Pfeiffer, and Iryna Gurevych. 2020b. MultiCQA: Zero-shot transfer of self-supervised text matching models on a massive scale. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2471-2486, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405-1418, Online. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
A Detail Results: AdapterFusion model LIA
We calculate the LIA of each adapters in the Adapterfusion model. In order to have a better insight of the model, we gradually remove adapters with small LIA from the 3D heatmap, see Figure 9. In standard adapters modules, we discover that there are a number of adapters not utilized in the original Aadapterfusion model and most of the essential adapters are at the back of the cube which are the adapters trained on large datasets. In pruned Adapterfusion model, most of the adapters are essential and most of the adapters become more important for the task.
Figure 10 and Figure 11 shows the LIAs of each adapters in Adapterfusion model in each tasks which is cross-section of the cube. We find that in tasks with small datasets, the model uses adapters from different tasks, while in tasks with large datasets, the model mainly uses the adapter trained from the same task (e.g. QNLI, QQP and MNLI). However, after pruning the Adaterfusion model, it uses adapters trained on different tasks even when the target task is the one with a large dataset.
B Detail of Pruning Adapters
The single task adapter model contains 24 adapter modules. There are 12 layer of transformers block in Bert-base. Each layer with 2 adapter modules. We prune  $20\%$  of the adapters for each iteration of pruning.
For a better understanding of the pruning algorithm 1. We summarized the proposed pruning strategy and demonstrate the inner structure of adapter in Figure 12.
(a) Standard Adapterfusion with LIA
(b) Pruned Adapterfusion with LIA
(c) Standard Adapterfusion with LIA  $>0.1\%$
(d) Pruned Adapterfusion with LIA  $>0.1\%$
(e) Standard Adapterfusion with with LIA  $>1\%$
(f) Pruned Adapterfusion with LIA  $>1\%$
Figure 9: LIAs of Adapterfusion w/o pruning
(b) MRPC
(a) RTE
(d) CoLA
(c) STSB
(e) SST-2
(g)QQP
(f)QNLI
(h) MNLI
Figure 10: LIAs of Standard Adapterfusion on different target tasks
(f)QNLI
(h) MNLI
Figure 11: LIAs of Adapterfusion with pruning on different target tasks
Figure 12: The Process of Pruning adapters using Lottery Ticket Hypothesis