SlowGuess/NAACL_2022_OCR huggingface dataset:
As mentioned in Sec. 3 we opt for a full-rank decomposition rather than a low-rank one. Our hypothesis is that a low-rank decomposition would lead to the loss of useful information, hindering the effective adaptation to different tasks given our low parameter budget. We have empirically verified this hypothesis by reducing the rank of our decomposition during the training procedure initialization. Concretely, even slight dimensionality reduction (e.g. from 1024 to 768) leads to a significant decrease in performance (e.g. from  $61\%$  to  $35\%$  in the CoLA dataset/task), with performance becoming gradually worse when further reducing dimensionality.
For the first set of experiments, our results in table 1 show that our MHA variant yields competitive results on the GLUE dataset, being comparable or better than the adapter-based alternative of (Houlsby et al., 2019), albeit requiring significantly less total and trainable parameters. In fact, given that the large version of BERT consists of 330M parameters, our approach requires four times less total parameters for all of the GLUE tasks and, for each task, we train at most  $1\%$  of the initial model's parameters. Compared to adapters, this leads to an almost four times reduction in the number of training parameters - without relying on extensive parameter search, on which adapters are heavily reliant (Stickland and Murray, 2019; Houlsby et al., 2019). Our method does not introduce extra hyperparameters since the decomposition is always full rank, thus it is also simpler to implement and optimize.
Our second set of experiments is related to recent findings that the fully connected modules of the transformer architecture tend to be more important for a model's performance (Kitaev et al., 2020; Tay et al., 2020a). A number of works have confirmed this claim under different settings (Houlsby et al., 2019; Tay et al., 2020b; Lu et al., 2021), where either the network is trained by remaining frozen except for the FFN module or the second fully connected layer of the FFN is chosen for some weight modulation method. We also selected the second fully connected layer of the FFN module and applied our collective factorization scheme to it. However, when comparing the two variants of our method, the MHA module variant seems to consistently be the better method when it comes to both the Glue test set (table 1) and the dev set (table 2).
We introduced a collective weight factorization method for adapting a pre-trained transformer to
multiple NLP tasks in an efficient manner. The proposed method is more efficient in terms of total parameters needed and percentage of model parameters trained and stored for each new task. Furthermore, we investigated the effect of our method when applied to different components of the Transformer. The results indicate that the multi-head attention mechanism can effectively adapt to different tasks and achieve competitive results when it is the only trained component, leading to the conclusion that more research needs to be done on the contribution of the different parts of the Transformer to its performance, and the correlation of the performance to the data used for training.
A limitation of out work is that the proposed factorization scheme operates on a single type of module, i.e. requiring same number of weight matrices and matrix dimensions. We plan to extend our method in order to be able to incorporate different types of modules (such as the FFN and MHA) in a joint factorization scheme. Taking into account recent advances in the extension of pretrained transformers to different modalities (Lu et al., 2021), one can also investigate the extension of the proposed method to adapt pretrained Transformer networks to new modalities, incorporating diverse tasks while maintaining a minimum overhead in terms of additional trainable and total parameters.
The research work was supported by the Hellenic Foundation for Research and Innovation (H.F.R.I.) under the "First Call for H.F.R.I. Research Projects to support Faculty members and Researchers and the procurement of high-cost research equipment grant" (Project Number: HFRI-FM17-2351). This work was also supported by a grant from The Cyprus Institute on Cyclone under project ID p055.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33.
Adrian Bulat, Jean Kossaifi, Georgios Tzimiropoulos, and Maja Pantic. 2020. Incremental multi-domain learning with network latent tensor factorization. Proceedings of the AAAI Conference on Artificial Intelligence.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. 2000. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing.
David Ha, Andrew M. Dai, and Quoc V. Le. 2017. Hypernetworks. In 5th International Conference on Learning Representations.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning.
Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan V. Oseledets. 2019. Tensorized embedding layers for efficient model compression. CoRR, abs/1901.10787.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In 8th International Conference on Learning Representations.
Tamara G Kolda and Brett W Bader. 2009. Tensor decompositions and applications. SIAM review.
Jean Kossaifi, Zachary C Lipton, Arinbjörn Kolbeinsson, Aran Khanna, Tommaso Furlanello, and Anima Anandkumar. 2020. Tensor regression networks. Journal of Machine Learning Research.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. 2019. Tensorly: Tensor learning in python. Journal of Machine Learning Research.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Conference of the Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. 2021. Pretrained transformers as universal computation engines. CoRR, abs/2103.05247.
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry P. Vetrov. 2015. Tensorizing neural networks. In Advances in Neural Information Processing Systems.
Ivan V Oseledets. 2011. Tensor-train decomposition. SIAM Journal on Scientific Computing.
Yannis Panagakis, Jean Kossaifi, Grigorios G Chrysos, James Oldfield, Mihalis A Nicolaou, Anima Anandkumar, and Stefanos Zafeiriou. 2021. Tensor methods in computer vision and deep learning. Proceedings of the IEEE, 109(5):863-890.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics.
Jonas Pfeiffer, Andreas Rückle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterhub: A
framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020b. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Amir Rosenfeld and John K. Tsotsos. 2020. Incremental learning through deep adaptation. IEEE.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing.
Asa Cooper Stickland and Iain Murray. 2019. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2020a. Synthesizer: Rethinking self-attention in transformer models. CoRR, abs/2005.00743.
Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan. 2020b. Hypergrid: Efficient multitask transformers with grid-wise decomposable hyper projections. CoRR, abs/2007.05891.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2017. Compressing recurrent neural network with tensor train. In 2017 International Joint Conference on Neural Networks.
L. R. Tucker. 1963. Implications of factor analysis of three-way matrices for measurement of change. In C. W. Harris, editor, *Problems in measuring change*, pages 122-137. University of Wisconsin Press, Madison WI.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.