DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models

1 INTRODUCTION . Most recent NLP applications have been following the pre-train then fine-tune paradigm , where we start from a gigantic pre-trained model and fine-tune it towards downstream tasks . Conventional fine-tuning works by updating all of the parameters of the pre-trained model . As the size of pretrained models grows , updating all parameters becomes less feasible in most practical scenarios , due to the expensive memory and computational requirements . For example , BERTBASE ( Devlin et al. , 2019 ) has 110M trainable parameters , while GPT-2 ( Radford et al. , 2019 ) has up to 1.5B and the largest version of GPT-3 ( Radford et al. , 2019 ) has an astonishing 175B trainable parameters . As such , conventional fine-tuning of the larger models could require hundreds of GPU hours . Another downside of this paradigm is that it requires storing as many parameters , as in the large-scale pretrained models , for each downstream task . That poses impediments to the deployment in real-world resource-constrained environments . One solution to address the extensive resource requirement of conventional fine-tuning is model pruning ( LeCun et al. , 1990 ; Han et al. , 2015a ; Ren et al. , 2018 ; He et al. , 2017 ; Liu et al. , 2017 ) , where unnecessary weights are eliminated to shrink the model size . For example , Chen et al . ( 2021b ) leverages ` 1 regularization to remove insignificant attention heads and gains 35 % ∼ 45 % training time with comparable performance . Guo et al . ( 2020 ) learns sparse task-specific “ diff ” vectors for various downstream fine-tuning tasks , leading to great memory savings . All these studies indicate the rise of sparsity naturally during fine-tuning a general-purpose pre-trained model , 1All codes are provided in the supplement . to some specialized downstream functionality . One potential interpretation , of why sparsity arises , is that different subsets of the parameters may be responsible for different downstream tasks and data domains ( Sanh et al. , 2020 ) . However , identifying appropriate sparse pruning masks requires burdensome training of models with full weights , which can still be unaffordable for many practitioners . For example , fine-tuning a large pre-trained language model like GPT-3 for just one step consumes at least 1.2TB of VRAM and requires 96 pieces of NVIDIA Tesla ( Hu et al. , 2021 ) . One parallel alternative is designing parameter-efficient fine-tuning algorithms , which aims to only optimize a small portion of weights while fixing most of the pre-trained weights during the downstream task training step . Pioneering works along this line , which utilize adapters ( Houlsby et al. , 2019 ) or low-rank decomposition ( Hu et al. , 2021 ) , can significantly reduce the number of trainable parameters while preserving good fine-tuning performance . Introducing only a small number of task-specific parameters for each new downstream task can substantially improve the memory and deployment efficiency of models as it allows us to reuse the remaining unchanged/shared parameters . However , there are two major hurdles of current parameter-efficient fine-tuning : ( i ) it does not yield any inference efficiency gains since the full pre-trained weights are still required to calculate outputs ; and ( ii ) current methods assume the weight update to be either sparse ( Guo et al. , 2020 ) or low-rank ( Hu et al. , 2021 ) , yet those assumptions might be oversimplified and overly restricted to allow for effective updates . For example , the low-rank assumptions on weight matrices might be overly strong since some weights can not be fitted in the low-rank space ( Yu et al. , 2017 ) . Recently Chen et al . ( 2021a ) also find that using both sparse and low-rank components performs better than either of them . These observations have inspired us to explore better parameter-efficiency methods . To tackle both resource- and parameter-efficiency issues of large model fine-tuning , we explicitly draw on the prior of sparsity for both weight updates and the final weights , and establish a dually sparsity-embedding efficient tuning ( DSEE ) framework . From a pre-trained model , DSEE first adopts a sparsity-aware low-rank weight update to achieve parameter efficiency of the fine-tuning process ; and then enforces a sparse weight structure by masking to achieve resource efficiency of the fine-tuned model at inference time . Our contributions can be summarized as follows : • We propose the dually sparsity-embedding efficient tuning ( DSEE ) , which unifies sparsityaware weight update and sparse pretrained weight in fine-tuning gigantic pre-trained models . It is the first attempt towards jointly optimizing both parameter efficiency of the finetuning process , and the resource efficiency of the fine-tuned model . • We exploit both unstructured and structured sparse patterns in the DSEE framework . For weight updates , we find the injected sparsity prior to greatly enhance existing parameterefficient update schemes , and to further trim down the needed amount of trainable parameters . For the final weights , we learn well-performed sparse masks from the pre-trained weights , leading to substantial computation and memory reductions at inference . • Extensive experiments demonstrate the effectiveness of our proposal across various representative pre-trained languages models , such as BERT , GPT-2 , and DeBERTa ; and on diverse evaluation benchmarks , such as E2E , DART , WebNLG , and GLUE . Specifically , on ( E2E , Dart , WebNLG ) , our methods can achieve a BLUE score of ( 69.75 , 55.40 , 46.66 ) with less than 1 % of total trainable parameters . On BERT , our method can save about 35 % FLOPs , compared to traditional downstream fine-tuning . 2 RELATED WORK . Pre-trained language models . Transformer ( Vaswani et al. , 2017 ) is a sequence-to-sequence model that heavily uses the concept of self-attention . Later on , numerous transformer-based models have been proposed and show overwhelming performance on natural language processing ( NLP ) and on computer vision ( CV ) tasks . Devlin et al . ( 2019 ) designed BERT that pre-train deep bidirectional representations from unlabeled text and reached powerful performance . Liu et al . ( 2019 ) found that BERT were terribly undertrained and proposed RoBERTa , an enhanced training recipe for BERT which can greatly boost the performance . He et al . ( 2020 ) proposed decoding-enhanced BERT with disentangled attention ( DeBERTa ) that incorporates the disentangled attention mechanism and an improved mask encoder to enhance BERT and RoBERTa . More variants like XL-Net , Albert , and Electra have also been proposed in recent years ( Yang et al. , 2019 ; Lan et al. , 2019 ; Clark et al. , 2019 ) . The series of GPT models ( Radford et al. , 2019 ; Brown et al. , 2020 ) are later developed based on transformers decoder blocks rather than encoder blocks like BERT , which again have shown superior performance on different tasks . These large models pretrained on a large amount of unlabelled texts would need to be further fine-tuned on downstream tasks for better performance . One of the accompanying disadvantages of these pre-training models with tremendous parameter counts ( e.g. , 175B in GPT-3 ) is the unaffordable computational cost for further fine-tuning . Pruning and Low-rank decomposition . Pruning is a widely-used model compression technique . It can reduce the number of parameters inside models , which possibly brings training and inference efficiency . Along with weight pruning method ( Han et al. , 2015b ) being one of the most effective methods ( Gordon et al. , 2020 ) , various criterion have been proposed to select insignificant weights for pruning , such as Taylor approximation ( Molchanov et al. , 2019 ) , Hessian score approximation ( Hassibi & Stork , 1993 ) , and other saliency scores such as SNIP ( Lee et al. , 2018 ) , GraSP ( Wang et al. , 2019 ) and SynFlow ( Tanaka et al. , 2020 ) . Several pruning methods have been commonly adapted to compress language models ( McCarley et al. , 2019 ; Gordon et al. , 2020 ; Sanh et al. , 2020 ; Wang et al. , 2020 ; Chen et al. , 2021b ) . Specifically , McCarley et al . ( 2019 ) proposed to prune attention heads that had less contribution to the model . Wang et al . ( 2020 ) pruned BERT models by involving low-rank factorization and ` 0 regularization . Sanh et al . ( 2020 ) invented an improved version of magnitude pruning ( i.e. , pruning based on the weight change ) that can better suit the transfer learning . Chen et al . ( 2021b ) performed structured pruning on BERT via ` 1 sparse regularization , which reduced a large portion of parameters and decreased the training cost . Low-rank approximation ( Ye , 2005 ) is also vastly studied . One classical scenario is robust principle component analysis ( Candès et al. , 2011 ) , which decomposes a matrix into a low-rank plus a sparse component . Existing literature shows that in deep learning , the learned over-parameterized models often naturally bear approximate low-rank weight structures ( Oymak et al. , 2019 ; Yu et al. , 2017 ) . Some ( Jaderberg et al. , 2014 ; Povey et al. , 2018 ; Sainath et al. , 2013 ; Zhang et al. , 2014 ; Zhao et al. , 2016 ) have explicitly imposed the low-rank constraint during training . Wang et al . ( 2020 ) ; Hu et al . ( 2021 ) utilized low-rank decomposition to shrink the model size and trim down the trainable parameters during fine-tuning . However , to our best knowledge , integrating sparsity and low-rank structures has never been studied before for efficient fine-tuning of pre-trained language models . Parameter-efficient adaptation . Parameter-efficient adaptation aims at reducing the number of trainable parameters when fine-tuning the models across different downstream domains . Unlike pruning , it generates updates that can be represented by fewer parameters instead of building sparse models . Various approaches are invented to achieve the goal . Rebuffi et al . ( 2017 ) ; Houlsby et al . ( 2019 ) inserted and only trained adapters between existing layers , whose parameters are much less compared to the pretrained models . Guo et al . ( 2020 ) leveraged ` 0 regularization to limit the number of non-zero elements in the update vectors . Lester et al . ( 2021 ) ; Li & Liang ( 2021 ) introduced efficient prompt tuning which optimizes only a small continuous task-specific vector . Hu et al . ( 2021 ) proposed a low-rank decomposition-based method that can also significantly reduce the number of trainable parameters . However , fine-tuned models yielded by these methods work have the same amount of weights as the pre-trained starting point ; hence they contribute no resource efficiency of the final model . 3 METHODOLOGY . In this section , we begin by describing our notations and definitions of sparsity generation and parameter-efficient fine-tuning in Section 3.1 . Then , we introduce the ( dually ) sparsity-embedded efficient fine-tuning algorithms in Sections 3.2 and 3.3 . 3.1 PRELIMINARIES . Sparsity generation and resource-efficient fine-tuning . We adopt both unstructured and structured pruning methods to produce sparsity . They can lead to resource-efficiency including memory and computation savings . Let W ∈ Rm×n denote a weight matrix . The goal of pruning is to find a binary mask S ∈ { 0 , 1 } ‖W‖0 , where ‖W‖0 is the number of parameters in W . The mask S is applied to W and results in a sparse weight W S. For unstructured pruning , only memory cost is saved ; but for structured pruning , it helps save computational cost since the sparse weights can be smaller in size by wiping out all-zero columns or rows . However , the performance of networks after structured pruning is often shown to be inferior compared with the unstructured pruning counterpart . Parameter-efficient fine-tuning . To leverage the knowledge in pre-trained weights W , downstream models learn task-specific weight update ∆W via fine-tuning and generate predictions with weights W + ∆W . The output of models is therefore calculated as ( W + ∆W ) x where x is the input . Since ∆W has the same size of W , learning the update matrices usually requires massive resources as the size of the pre-trained model increases . Parameter-efficient fine-tuning try to solve this problem by using as few trainable parameters as possible to represent ∆W , while maintaining competitive downstream fine-tuning performance . Previous literature reaches the goal via either sparsifying weight update matrices ∆W ( Guo et al. , 2020 ) or leveraging low-rank decomposed matrices to compute ∆W ( Hu et al. , 2021 ) , while in our work we combine both of them . Algorithm 1 : Sparsity-Embedded LowRank Decomposition Input : Pretrained weightsW , number of non-zero elements N Output : Sparse matrices S2 1 Initialize S2 to be an empty set . 2 for each self-attention projection weights wi inW do / * Decomposition * / 3 Perform matrix decomposition : wi ≈ UV + S ′ by solving the optimization problem in Eqn.1 . / * Identify important elements to form S2 * / 4 Perform thresholding on S ′ : Keep N elements in S ′ with top magnitudes , and set the rest 0 . 5 Append S ′ into S2 . 6 end Algorithm 2 : DSEE Input : Pretrained weightsW , number of non-zero elements N , desired sparsity s , loss function L Output : Sparse mask S1 , matrices U , V , S2 1 DecomposeW into U , V and S2 2 ( Re- ) Initialization : U = 0 , V ∼ N ( 0 , 0.02 ) , Ω = indexes of non-zero elements in S2 , S = 0 / * I : train before pruning * / 3 Train U , V , S with respect to L under the constraint of PΩC ( S ) = 0 . / * II : pruning the model * / 4 Prune ( 1-s % ) parameters inW globally by sorting the magnitude ofW + UV + S , deriving the sparsity mask S1 / * III : tuning after pruning * / 5 Tune U , V , S2 for E epochs for recovering the performance .

Summary: This paper proposes methods to improve the efficiency of large pre-trained models (millions or billions of parameters) by combining multiple notions of _sparsity_ and/or _dimensionality reduction_ on the model weights. This is achieved via (1) a low-rank update matrix with fine-tunable parameters, and (2) pruning the original pre-trained weight matrix. These two sets of parameters are combined during inference. The authors validate their proposed method on several model architectures and datasets.