DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models

1 INTRODUCTION . Most recent NLP applications have been following the pre-train then fine-tune paradigm , where we start from a gigantic pre-trained model and fine-tune it towards downstream tasks . Conventional fine-tuning works by updating all of the parameters of the pre-trained model . As the size of pretrained models grows , updating all parameters becomes less feasible in most practical scenarios , due to the expensive memory and computational requirements . For example , BERTBASE ( Devlin et al. , 2019 ) has 110M trainable parameters , while GPT-2 ( Radford et al. , 2019 ) has up to 1.5B and the largest version of GPT-3 ( Radford et al. , 2019 ) has an astonishing 175B trainable parameters . As such , conventional fine-tuning of the larger models could require hundreds of GPU hours . Another downside of this paradigm is that it requires storing as many parameters , as in the large-scale pretrained models , for each downstream task . That poses impediments to the deployment in real-world resource-constrained environments . One solution to address the extensive resource requirement of conventional fine-tuning is model pruning ( LeCun et al. , 1990 ; Han et al. , 2015a ; Ren et al. , 2018 ; He et al. , 2017 ; Liu et al. , 2017 ) , where unnecessary weights are eliminated to shrink the model size . For example , Chen et al . ( 2021b ) leverages ` 1 regularization to remove insignificant attention heads and gains 35 % ∼ 45 % training time with comparable performance . Guo et al . ( 2020 ) learns sparse task-specific “ diff ” vectors for various downstream fine-tuning tasks , leading to great memory savings . All these studies indicate the rise of sparsity naturally during fine-tuning a general-purpose pre-trained model , 1All codes are provided in the supplement . to some specialized downstream functionality . One potential interpretation , of why sparsity arises , is that different subsets of the parameters may be responsible for different downstream tasks and data domains ( Sanh et al. , 2020 ) . However , identifying appropriate sparse pruning masks requires burdensome training of models with full weights , which can still be unaffordable for many practitioners . For example , fine-tuning a large pre-trained language model like GPT-3 for just one step consumes at least 1.2TB of VRAM and requires 96 pieces of NVIDIA Tesla ( Hu et al. , 2021 ) . One parallel alternative is designing parameter-efficient fine-tuning algorithms , which aims to only optimize a small portion of weights while fixing most of the pre-trained weights during the downstream task training step . Pioneering works along this line , which utilize adapters ( Houlsby et al. , 2019 ) or low-rank decomposition ( Hu et al. , 2021 ) , can significantly reduce the number of trainable parameters while preserving good fine-tuning performance . Introducing only a small number of task-specific parameters for each new downstream task can substantially improve the memory and deployment efficiency of models as it allows us to reuse the remaining unchanged/shared parameters . However , there are two major hurdles of current parameter-efficient fine-tuning : ( i ) it does not yield any inference efficiency gains since the full pre-trained weights are still required to calculate outputs ; and ( ii ) current methods assume the weight update to be either sparse ( Guo et al. , 2020 ) or low-rank ( Hu et al. , 2021 ) , yet those assumptions might be oversimplified and overly restricted to allow for effective updates . For example , the low-rank assumptions on weight matrices might be overly strong since some weights can not be fitted in the low-rank space ( Yu et al. , 2017 ) . Recently Chen et al . ( 2021a ) also find that using both sparse and low-rank components performs better than either of them . These observations have inspired us to explore better parameter-efficiency methods . To tackle both resource- and parameter-efficiency issues of large model fine-tuning , we explicitly draw on the prior of sparsity for both weight updates and the final weights , and establish a dually sparsity-embedding efficient tuning ( DSEE ) framework . From a pre-trained model , DSEE first adopts a sparsity-aware low-rank weight update to achieve parameter efficiency of the fine-tuning process ; and then enforces a sparse weight structure by masking to achieve resource efficiency of the fine-tuned model at inference time . Our contributions can be summarized as follows : • We propose the dually sparsity-embedding efficient tuning ( DSEE ) , which unifies sparsityaware weight update and sparse pretrained weight in fine-tuning gigantic pre-trained models . It is the first attempt towards jointly optimizing both parameter efficiency of the finetuning process , and the resource efficiency of the fine-tuned model . • We exploit both unstructured and structured sparse patterns in the DSEE framework . For weight updates , we find the injected sparsity prior to greatly enhance existing parameterefficient update schemes , and to further trim down the needed amount of trainable parameters . For the final weights , we learn well-performed sparse masks from the pre-trained weights , leading to substantial computation and memory reductions at inference . • Extensive experiments demonstrate the effectiveness of our proposal across various representative pre-trained languages models , such as BERT , GPT-2 , and DeBERTa ; and on diverse evaluation benchmarks , such as E2E , DART , WebNLG , and GLUE . Specifically , on ( E2E , Dart , WebNLG ) , our methods can achieve a BLUE score of ( 69.75 , 55.40 , 46.66 ) with less than 1 % of total trainable parameters . On BERT , our method can save about 35 % FLOPs , compared to traditional downstream fine-tuning . 2 RELATED WORK . Pre-trained language models . Transformer ( Vaswani et al. , 2017 ) is a sequence-to-sequence model that heavily uses the concept of self-attention . Later on , numerous transformer-based models have been proposed and show overwhelming performance on natural language processing ( NLP ) and on computer vision ( CV ) tasks . Devlin et al . ( 2019 ) designed BERT that pre-train deep bidirectional representations from unlabeled text and reached powerful performance . Liu et al . ( 2019 ) found that BERT were terribly undertrained and proposed RoBERTa , an enhanced training recipe for BERT which can greatly boost the performance . He et al . ( 2020 ) proposed decoding-enhanced BERT with disentangled attention ( DeBERTa ) that incorporates the disentangled attention mechanism and an improved mask encoder to enhance BERT and RoBERTa . More variants like XL-Net , Albert , and Electra have also been proposed in recent years ( Yang et al. , 2019 ; Lan et al. , 2019 ; Clark et al. , 2019 ) . The series of GPT models ( Radford et al. , 2019 ; Brown et al. , 2020 ) are later developed based on transformers decoder blocks rather than encoder blocks like BERT , which again have shown superior performance on different tasks . These large models pretrained on a large amount of unlabelled texts would need to be further fine-tuned on downstream tasks for better performance . One of the accompanying disadvantages of these pre-training models with tremendous parameter counts ( e.g. , 175B in GPT-3 ) is the unaffordable computational cost for further fine-tuning . Pruning and Low-rank decomposition . Pruning is a widely-used model compression technique . It can reduce the number of parameters inside models , which possibly brings training and inference efficiency . Along with weight pruning method ( Han et al. , 2015b ) being one of the most effective methods ( Gordon et al. , 2020 ) , various criterion have been proposed to select insignificant weights for pruning , such as Taylor approximation ( Molchanov et al. , 2019 ) , Hessian score approximation ( Hassibi & Stork , 1993 ) , and other saliency scores such as SNIP ( Lee et al. , 2018 ) , GraSP ( Wang et al. , 2019 ) and SynFlow ( Tanaka et al. , 2020 ) . Several pruning methods have been commonly adapted to compress language models ( McCarley et al. , 2019 ; Gordon et al. , 2020 ; Sanh et al. , 2020 ; Wang et al. , 2020 ; Chen et al. , 2021b ) . Specifically , McCarley et al . ( 2019 ) proposed to prune attention heads that had less contribution to the model . Wang et al . ( 2020 ) pruned BERT models by involving low-rank factorization and ` 0 regularization . Sanh et al . ( 2020 ) invented an improved version of magnitude pruning ( i.e. , pruning based on the weight change ) that can better suit the transfer learning . Chen et al . ( 2021b ) performed structured pruning on BERT via ` 1 sparse regularization , which reduced a large portion of parameters and decreased the training cost . Low-rank approximation ( Ye , 2005 ) is also vastly studied . One classical scenario is robust principle component analysis ( Candès et al. , 2011 ) , which decomposes a matrix into a low-rank plus a sparse component . Existing literature shows that in deep learning , the learned over-parameterized models often naturally bear approximate low-rank weight structures ( Oymak et al. , 2019 ; Yu et al. , 2017 ) . Some ( Jaderberg et al. , 2014 ; Povey et al. , 2018 ; Sainath et al. , 2013 ; Zhang et al. , 2014 ; Zhao et al. , 2016 ) have explicitly imposed the low-rank constraint during training . Wang et al . ( 2020 ) ; Hu et al . ( 2021 ) utilized low-rank decomposition to shrink the model size and trim down the trainable parameters during fine-tuning . However , to our best knowledge , integrating sparsity and low-rank structures has never been studied before for efficient fine-tuning of pre-trained language models . Parameter-efficient adaptation . Parameter-efficient adaptation aims at reducing the number of trainable parameters when fine-tuning the models across different downstream domains . Unlike pruning , it generates updates that can be represented by fewer parameters instead of building sparse models . Various approaches are invented to achieve the goal . Rebuffi et al . ( 2017 ) ; Houlsby et al . ( 2019 ) inserted and only trained adapters between existing layers , whose parameters are much less compared to the pretrained models . Guo et al . ( 2020 ) leveraged ` 0 regularization to limit the number of non-zero elements in the update vectors . Lester et al . ( 2021 ) ; Li & Liang ( 2021 ) introduced efficient prompt tuning which optimizes only a small continuous task-specific vector . Hu et al . ( 2021 ) proposed a low-rank decomposition-based method that can also significantly reduce the number of trainable parameters . However , fine-tuned models yielded by these methods work have the same amount of weights as the pre-trained starting point ; hence they contribute no resource efficiency of the final model . 3 METHODOLOGY . In this section , we begin by describing our notations and definitions of sparsity generation and parameter-efficient fine-tuning in Section 3.1 . Then , we introduce the ( dually ) sparsity-embedded efficient fine-tuning algorithms in Sections 3.2 and 3.3 . 3.1 PRELIMINARIES . Sparsity generation and resource-efficient fine-tuning . We adopt both unstructured and structured pruning methods to produce sparsity . They can lead to resource-efficiency including memory and computation savings . Let W ∈ Rm×n denote a weight matrix . The goal of pruning is to find a binary mask S ∈ { 0 , 1 } ‖W‖0 , where ‖W‖0 is the number of parameters in W . The mask S is applied to W and results in a sparse weight W S. For unstructured pruning , only memory cost is saved ; but for structured pruning , it helps save computational cost since the sparse weights can be smaller in size by wiping out all-zero columns or rows . However , the performance of networks after structured pruning is often shown to be inferior compared with the unstructured pruning counterpart . Parameter-efficient fine-tuning . To leverage the knowledge in pre-trained weights W , downstream models learn task-specific weight update ∆W via fine-tuning and generate predictions with weights W + ∆W . The output of models is therefore calculated as ( W + ∆W ) x where x is the input . Since ∆W has the same size of W , learning the update matrices usually requires massive resources as the size of the pre-trained model increases . Parameter-efficient fine-tuning try to solve this problem by using as few trainable parameters as possible to represent ∆W , while maintaining competitive downstream fine-tuning performance . Previous literature reaches the goal via either sparsifying weight update matrices ∆W ( Guo et al. , 2020 ) or leveraging low-rank decomposed matrices to compute ∆W ( Hu et al. , 2021 ) , while in our work we combine both of them . Algorithm 1 : Sparsity-Embedded LowRank Decomposition Input : Pretrained weightsW , number of non-zero elements N Output : Sparse matrices S2 1 Initialize S2 to be an empty set . 2 for each self-attention projection weights wi inW do / * Decomposition * / 3 Perform matrix decomposition : wi ≈ UV + S ′ by solving the optimization problem in Eqn.1 . / * Identify important elements to form S2 * / 4 Perform thresholding on S ′ : Keep N elements in S ′ with top magnitudes , and set the rest 0 . 5 Append S ′ into S2 . 6 end Algorithm 2 : DSEE Input : Pretrained weightsW , number of non-zero elements N , desired sparsity s , loss function L Output : Sparse mask S1 , matrices U , V , S2 1 DecomposeW into U , V and S2 2 ( Re- ) Initialization : U = 0 , V ∼ N ( 0 , 0.02 ) , Ω = indexes of non-zero elements in S2 , S = 0 / * I : train before pruning * / 3 Train U , V , S with respect to L under the constraint of PΩC ( S ) = 0 . / * II : pruning the model * / 4 Prune ( 1-s % ) parameters inW globally by sorting the magnitude ofW + UV + S , deriving the sparsity mask S1 / * III : tuning after pruning * / 5 Tune U , V , S2 for E epochs for recovering the performance .

Summary: This paper proposes to enforce weight sparsity constraints into a prior parameter-efficient tuning method (LoRA) to achieve both parameter-efficient tuning and resource-efficient inference. Specifically,  this paper (1) adds another sparse, learnable weight matrix to the original low-rank decomposition in LoRA; (2) incorporates unstructured and structured weight pruning techniques to save computation at inference time. Experiments are conducted on GLUE and several data2text generation benchmarks to demonstrate the effectiveness of the proposed method.