Year: 2021

Abstract:
Gigantic pre-trained models have become central to natural language processing (NLP), serving as the starting point for fine-tuning towards a range of downstream tasks. However, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g., 175B parameters for GPT-3), even the fine-tuning process can be time-consuming and computationally expensive; (b) the fine-tuned model has the same size as its starting point by default, which is neither sensible due to its more specialized functionality, nor practical since many finetuned models will be deployed in resource-constrained environments. To address these pain points, we propose a framework for resourceand parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning by enforcing sparsity-aware weight updates on top of the pretrained weights; and (ii) resource-efficient inference by encouraging a sparse weight structure towards the final fine-tuned model. We leverage sparsity in these two directions by exploiting both unstructured and structural sparse patterns in pre-trained language models via magnitude-based pruning and `1 sparse regularization. Extensive experiments and in-depth investigations, with diverse network backbones (i.e., BERT, GPT-2, and DeBERTa) on dozens of datasets, consistently demonstrate highly impressive parameter-/training-/inference-efficiency, while maintaining competitive downstream transfer performance. For instance, our DSEE-BERT obtains about 35% inference FLOPs savings with < 1% trainable parameters and comparable performance to conventional fine-tuning. 1

heading: 1 INTRODUCTION
text: Most recent NLP applications have been following the pre-train then fine-tune paradigm, where we start from a gigantic pre-trained model and fine-tune it towards downstream tasks. Conventional fine-tuning works by updating all of the parameters of the pre-trained model. As the size of pretrained models grows, updating all parameters becomes less feasible in most practical scenarios, due to the expensive memory and computational requirements. For example, BERTBASE (Devlin et al., 2019) has 110M trainable parameters, while GPT-2 (Radford et al., 2019) has up to 1.5B and the largest version of GPT-3 (Radford et al., 2019) has an astonishing 175B trainable parameters. As such, conventional fine-tuning of the larger models could require hundreds of GPU hours. Another downside of this paradigm is that it requires storing as many parameters, as in the large-scale pretrained models, for each downstream task. That poses impediments to the deployment in real-world resource-constrained environments.
One solution to address the extensive resource requirement of conventional fine-tuning is model pruning (LeCun et al., 1990; Han et al., 2015a; Ren et al., 2018; He et al., 2017; Liu et al., 2017), where unnecessary weights are eliminated to shrink the model size. For example, Chen et al. (2021b) leverages `1 regularization to remove insignificant attention heads and gains 35% ∼ 45% training time with comparable performance. Guo et al. (2020) learns sparse task-specific “diff” vectors for various downstream fine-tuning tasks, leading to great memory savings. All these studies indicate the rise of sparsity naturally during fine-tuning a general-purpose pre-trained model,
1All codes are provided in the supplement.
to some specialized downstream functionality. One potential interpretation, of why sparsity arises, is that different subsets of the parameters may be responsible for different downstream tasks and data domains (Sanh et al., 2020). However, identifying appropriate sparse pruning masks requires burdensome training of models with full weights, which can still be unaffordable for many practitioners. For example, fine-tuning a large pre-trained language model like GPT-3 for just one step consumes at least 1.2TB of VRAM and requires 96 pieces of NVIDIA Tesla (Hu et al., 2021).
One parallel alternative is designing parameter-efficient fine-tuning algorithms, which aims to only optimize a small portion of weights while fixing most of the pre-trained weights during the downstream task training step. Pioneering works along this line, which utilize adapters (Houlsby et al., 2019) or low-rank decomposition (Hu et al., 2021), can significantly reduce the number of trainable parameters while preserving good fine-tuning performance. Introducing only a small number of task-specific parameters for each new downstream task can substantially improve the memory and deployment efficiency of models as it allows us to reuse the remaining unchanged/shared parameters. However, there are two major hurdles of current parameter-efficient fine-tuning: (i) it does not yield any inference efficiency gains since the full pre-trained weights are still required to calculate outputs; and (ii) current methods assume the weight update to be either sparse (Guo et al., 2020) or low-rank (Hu et al., 2021), yet those assumptions might be oversimplified and overly restricted to allow for effective updates. For example, the low-rank assumptions on weight matrices might be overly strong since some weights cannot be fitted in the low-rank space (Yu et al., 2017). Recently Chen et al. (2021a) also find that using both sparse and low-rank components performs better than either of them. These observations have inspired us to explore better parameter-efficiency methods.
To tackle both resource- and parameter-efficiency issues of large model fine-tuning, we explicitly draw on the prior of sparsity for both weight updates and the final weights, and establish a dually sparsity-embedding efficient tuning (DSEE) framework. From a pre-trained model, DSEE first adopts a sparsity-aware low-rank weight update to achieve parameter efficiency of the fine-tuning process; and then enforces a sparse weight structure by masking to achieve resource efficiency of the fine-tuned model at inference time. Our contributions can be summarized as follows:
• We propose the dually sparsity-embedding efficient tuning (DSEE), which unifies sparsityaware weight update and sparse pretrained weight in fine-tuning gigantic pre-trained models. It is the first attempt towards jointly optimizing both parameter efficiency of the finetuning process, and the resource efficiency of the fine-tuned model.
• We exploit both unstructured and structured sparse patterns in the DSEE framework. For weight updates, we find the injected sparsity prior to greatly enhance existing parameterefficient update schemes, and to further trim down the needed amount of trainable parameters. For the final weights, we learn well-performed sparse masks from the pre-trained weights, leading to substantial computation and memory reductions at inference.
• Extensive experiments demonstrate the effectiveness of our proposal across various representative pre-trained languages models, such as BERT, GPT-2, and DeBERTa; and on diverse evaluation benchmarks, such as E2E, DART, WebNLG, and GLUE. Specifically, on (E2E, Dart, WebNLG), our methods can achieve a BLUE score of (69.75, 55.40, 46.66) with less than 1% of total trainable parameters. On BERT, our method can save about 35% FLOPs, compared to traditional downstream fine-tuning.

heading: 2 RELATED WORK
text: Pre-trained language models. Transformer (Vaswani et al., 2017) is a sequence-to-sequence model that heavily uses the concept of self-attention. Later on, numerous transformer-based models have been proposed and show overwhelming performance on natural language processing (NLP) and on computer vision (CV) tasks. Devlin et al. (2019) designed BERT that pre-train deep bidirectional representations from unlabeled text and reached powerful performance. Liu et al. (2019) found that BERT were terribly undertrained and proposed RoBERTa, an enhanced training recipe for BERT which can greatly boost the performance. He et al. (2020) proposed decoding-enhanced BERT with disentangled attention (DeBERTa) that incorporates the disentangled attention mechanism and an improved mask encoder to enhance BERT and RoBERTa. More variants like XL-Net, Albert, and Electra have also been proposed in recent years (Yang et al., 2019; Lan et al., 2019; Clark et al., 2019). The series of GPT models (Radford et al., 2019; Brown et al., 2020) are later developed based on transformers decoder blocks rather than encoder blocks like BERT, which again have
shown superior performance on different tasks. These large models pretrained on a large amount of unlabelled texts would need to be further fine-tuned on downstream tasks for better performance. One of the accompanying disadvantages of these pre-training models with tremendous parameter counts (e.g., 175B in GPT-3) is the unaffordable computational cost for further fine-tuning.
Pruning and Low-rank decomposition. Pruning is a widely-used model compression technique. It can reduce the number of parameters inside models, which possibly brings training and inference efficiency. Along with weight pruning method (Han et al., 2015b) being one of the most effective methods (Gordon et al., 2020), various criterion have been proposed to select insignificant weights for pruning, such as Taylor approximation (Molchanov et al., 2019), Hessian score approximation (Hassibi & Stork, 1993), and other saliency scores such as SNIP (Lee et al., 2018), GraSP (Wang et al., 2019) and SynFlow (Tanaka et al., 2020). Several pruning methods have been commonly adapted to compress language models (McCarley et al., 2019; Gordon et al., 2020; Sanh et al., 2020; Wang et al., 2020; Chen et al., 2021b). Specifically, McCarley et al. (2019) proposed to prune attention heads that had less contribution to the model. Wang et al. (2020) pruned BERT models by involving low-rank factorization and `0 regularization. Sanh et al. (2020) invented an improved version of magnitude pruning (i.e., pruning based on the weight change) that can better suit the transfer learning. Chen et al. (2021b) performed structured pruning on BERT via `1 sparse regularization, which reduced a large portion of parameters and decreased the training cost.
Low-rank approximation (Ye, 2005) is also vastly studied. One classical scenario is robust principle component analysis (Candès et al., 2011), which decomposes a matrix into a low-rank plus a sparse component. Existing literature shows that in deep learning, the learned over-parameterized models often naturally bear approximate low-rank weight structures (Oymak et al., 2019; Yu et al., 2017). Some (Jaderberg et al., 2014; Povey et al., 2018; Sainath et al., 2013; Zhang et al., 2014; Zhao et al., 2016) have explicitly imposed the low-rank constraint during training. Wang et al. (2020); Hu et al. (2021) utilized low-rank decomposition to shrink the model size and trim down the trainable parameters during fine-tuning. However, to our best knowledge, integrating sparsity and low-rank structures has never been studied before for efficient fine-tuning of pre-trained language models.
Parameter-efficient adaptation. Parameter-efficient adaptation aims at reducing the number of trainable parameters when fine-tuning the models across different downstream domains. Unlike pruning, it generates updates that can be represented by fewer parameters instead of building sparse models. Various approaches are invented to achieve the goal. Rebuffi et al. (2017); Houlsby et al. (2019) inserted and only trained adapters between existing layers, whose parameters are much less compared to the pretrained models. Guo et al. (2020) leveraged `0 regularization to limit the number of non-zero elements in the update vectors. Lester et al. (2021); Li & Liang (2021) introduced efficient prompt tuning which optimizes only a small continuous task-specific vector. Hu et al. (2021) proposed a low-rank decomposition-based method that can also significantly reduce the number of trainable parameters. However, fine-tuned models yielded by these methods work have the same amount of weights as the pre-trained starting point; hence they contribute no resource efficiency of the final model.

heading: 3 METHODOLOGY
text: In this section, we begin by describing our notations and definitions of sparsity generation and parameter-efficient fine-tuning in Section 3.1. Then, we introduce the (dually) sparsity-embedded efficient fine-tuning algorithms in Sections 3.2 and 3.3.

heading: 3.1 PRELIMINARIES
text: Sparsity generation and resource-efficient fine-tuning. We adopt both unstructured and structured pruning methods to produce sparsity. They can lead to resource-efficiency including memory and computation savings.
Let W ∈ Rm×n denote a weight matrix. The goal of pruning is to find a binary mask S ∈ {0, 1}‖W‖0 , where ‖W‖0 is the number of parameters in W . The mask S is applied to W and results in a sparse weight W S. For unstructured pruning, only memory cost is saved; but for structured pruning, it helps save computational cost since the sparse weights can be smaller in size by wiping out all-zero columns or rows. However, the performance of networks after structured pruning is often shown to be inferior compared with the unstructured pruning counterpart.
Parameter-efficient fine-tuning. To leverage the knowledge in pre-trained weights W , downstream models learn task-specific weight update ∆W via fine-tuning and generate predictions with weights W + ∆W . The output of models is therefore calculated as (W + ∆W)x where x is the input. Since ∆W has the same size of W , learning the update matrices usually requires massive resources as the size of the pre-trained model increases. Parameter-efficient fine-tuning try to solve this problem by using as few trainable parameters as possible to represent ∆W , while maintaining competitive downstream fine-tuning performance. Previous literature reaches the goal via either sparsifying weight update matrices ∆W (Guo et al., 2020) or leveraging low-rank decomposed matrices to compute ∆W (Hu et al., 2021), while in our work we combine both of them.
Algorithm 1: Sparsity-Embedded LowRank Decomposition Input: Pretrained weightsW , number of non-zero elements N Output: Sparse matrices S2 1 Initialize S2 to be an empty set. 2 for each self-attention projection weights wi inW do /* Decomposition */ 3 Perform matrix decomposition: wi ≈ UV + S ′ by solving the optimization problem in Eqn.1. /* Identify important elements to form S2 */ 4 Perform thresholding on S ′: Keep N
elements in S ′ with top magnitudes, and set the rest 0.
5 Append S ′ into S2. 6 end
Algorithm 2: DSEE Input: Pretrained weightsW , number of
non-zero elements N , desired sparsity s, loss function L
Output: Sparse mask S1, matrices U ,V,S2 1 DecomposeW into U ,V and S2 2 (Re-)Initialization: U = 0,V ∼ N (0, 0.02),Ω = indexes of non-zero elements in S2,S = 0 /* I: train before pruning */ 3 Train U ,V,S with respect to L under the constraint of PΩC (S) = 0. /* II: pruning the model */ 4 Prune (1-s%) parameters inW globally by sorting the magnitude ofW + UV + S, deriving the sparsity mask S1 /* III: tuning after pruning */ 5 Tune U ,V,S2 for E epochs for recovering the performance.

heading: 3.2 SPARSITY-EMBEDDED PARAMETER-EFFICIENT FINE-TUNING
text: A recent study (Hu et al., 2021) enforces low-rank constraint to weight update tensors ∆W , and obtains a satisfactory trade-off between parameter-efficiency and model quality. However, as revealed experimentally by (Yu et al., 2017), a part of the important information in the trained weights will also scatter outside the low-rank subspace, creating sparse “residuals”. Inspired by these observations, we investigate a new sparsity-aware low-rank subspace of ∆W , and introduce the first component of our proposal in Figure 1, i.e., sparsity-embedded parameter-efficient fine-tuning.
Specifically, we identify a sparse subspace Ω, that we can project our update matrices ∆W to. The update matrix is then decomposed into two components: a low-rank part which is represented by the multiplication of two low-rank matrices U ∈ Rm×r and V ∈ Rr×n, and a sparse residual
S2 = PΩ(∆W), where PΩ(∆W) =
{ si,j , (i, j) ∈ Ω
0, (i, j) ∈ ΩC , i = 1, 2, . . . ,m, j = 1, 2, . . . , n.
As illustrated in Figure 1, the update matrix ∆W can be approximated by UV + S2, in which the U , V , and the non-zero element of S2 are learnable parameters while Ω is fixed once determined. Compared to the full fine-tuning schemes has m × n individual trainable parameters, our method only has (m+n)×r+card(Ω). If r is smaller than m×n−card(Ω)m+n / 0.5 min{m,n}, which is a loose bound, our method is capable of substantially reducing trainable parameters for downstream finetuning. In practice, the value of r is very small compared tom and n so the savings are considerable.
The next question is how to find appropriate Ω. Motivated by (Candès et al., 2011), we formulate our sparsity-aware decomposition as the following optimization problem:
min U,V,S2
1 2 ‖W − UV − S2‖2F
s.t. rank(U) ≤ r, rank(V) ≤ r, card(S2) ≤ c, (1)
where rank(·) indicates the rank and card(·) indicates the cardinality of a matrix. Here we derive Ω from pre-trained weights W , which is different from previous low-rank techniques for model compression since we perform prior-training decomposition and can not access ∆W before finetuning. In this way, we actually assume that ∆W shares a similar crucial subspace Ω with W (Sun et al., 2018). We use the GreBsmo algorithm (Zhou & Tao, 2013) to solve this optimization problem. This method adopts a similar idea as random projection and can solve the optimization problem rapidly. The details of the algorithm are in Appendix.
Algorithm 1 summarizes the detailed procedure of our proposal. We first decompose the pretrained weightsW into three matrices: U , V and S2. U and V are matrices of low-rank and S2 is a sparse matrix. After the decomposition, we re-initialize U and V to be of shape Rm×r and Rr×n. The initial values for U are set to 0, and the initial values for V will follow a normal distribution as Hu et al. (2021) did. We do not use the decomposed value inside S2, but only the index of non-zero elements. We collect the indexes of non-zero elements into Ω, and reset the values of S2 back to 0. During the training, we only update the elements of S2 whose indexes are in Ω, as well as U and V .

heading: 3.3 DUALLY SPARSITY-EMBEDDED EFFICIENT TUNING (DSEE)
text: Besides parameter-efficiency, resource-efficiency is another major goal of our proposed framework. Sparsity-embedded low-rank weight updates alone do not necessarily lead to computational cost reductions since it works together with the dense pre-trained weights and the massive computation during forwarding processes is inevitable. To achieve both resource- and parameter-efficiency, we also promote a sparse weight structure to the final fine-tuned model, as demonstrated by the sparse mask S1 in Figure 1. Our DSEE explores both unstructured/structured sparsity patterns as follows. B Pruning with unstructured sparse masks. The unstructured sparse mask is the most widely used mask since it usually produces almost undamaged performance compared to its dense counterpart (Han et al., 2015a). It applies fine-grained manipulation on each individual model weight, while the generated irregular sparse masks bring limited hardware speedup (Wen et al., 2016). Specifically, based on the weight matrixW , a sparse mask S1 can be calculated with various approaches, either heuristic or optimization-based. We adopt the one-shot magnitude pruning (Han et al., 2015a) in DSEE, due to its simplicity and competitive performance.
In our context, we create the sparse pattern S1 from W + UV: First, it sorts the magnitude (i.e., absolute value) of individual weights and removes parameters with bottom 1 − k% magnitude. Second, we further tune the value of the low-rank update matrices for a few epochs, which is important to keep the performance as stated in Han et al. (2015b). Third, the sparse mask S1 that we derive will be applied on the pretrained weights only and do not affect the output from the update matrices, as W S1 + UV . For an input sample x, the output is calculated by (W S1 + UV + S2)x = W S1x + (UV + S2)x, where the first term W S1x brings significant resource-efficiency, and the second term brings parameter-efficiency.
B Pruning with structured sparse masks. The second method exploits structured sparse masks, whose regular patterns are more hardware friendly yet have worse performance than unstructured masks. Our DSEE considers `1 sparse regularization (Liu et al., 2017; Chen et al., 2021b) to craft high-quality structurally pruned subnetworks. More precisely, motivated by Chen et al. (2021b), we introduce learnable coefficients ξ before each attention head module, and append a `1 sparse penalty on ξ to the original loss. Detailed formulations are depicted as below:
We add trainable coefficients c before attention heads. The parameters c are optimized together with the decomposed matrices, i.e., U ,V and S2. An extra term λ‖c‖1 will be added to the training loss for sparse regularization. The value of λ is set to 1e-4. After training, we prune the attention heads that have the lowest contribution to the model (i.e., lowest c). We use a layer-wise pruning scheme that prunes the same proportion of heads in each attention layer. After pruning, several epochs of tuning are run to recover the performance of the pruned model. The size of update matrices will change after structured pruning, so we also need to change the dimension of U and V . Specifically, we change the size of V from Rr×n to Rr×[n×s] where s is the pruning ratio of the corresponding self-attention layer and [x] is the biggest integer not greater than x. The size of S2 is shrunk accordingly.

heading: 4 EXPERIMENT RESULTS
text: Datasets and models. We use three classical pre-trained language models in our experiments: BERTBASE (Devlin et al., 2019), GPT-2 (Radford et al., 2019), and DeBERTa-large (He et al., 2020), which have 12/24/24 layers with hidden size of 768/1024/1024 and 110/354/380M trainable parameters, respectively. For BERT and DeBERTa, we evaluate our method on the GLUE benchmarks (Wang et al., 2018). For GPT-2, we use E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017), and DART (Nan et al., 2021) for evaluations.
Training and evaluation details. For BERT and DeBERTa, we follow the default settings in Wolf et al. (2019); Devlin et al. (2019). We use the AdamW (Loshchilov & Hutter, 2017) optimizer for downstream training. The batch size for BERT and DeBERTa is 32, 8 per GPU. We train three epochs to search the sparse mask S1, and continue to tune the model for three epochs to converge. The initial learning rates are reported in A1, and we linearly decay them. For GPT-2, we follow the same hyperparameters as in Hu et al. (2021). We train the model for five epochs to search for the mask, and further tune the pruned model for two epochs.
Evaluation Metrics. For tasks on BERT and DeBERTa, we use the accuracy, Matthew’s Correlation, and Pearson’s r in the evaluation by default, which is also the conventional setting in the NLP community. On GPT-2, we use BLEU (Papineni et al., 2002), METEOR (Denkowski & Lavie, 2014), TER (Snover et al., 2006) and NIST (Doddington, 2002) as our evaluation metrics. To evaluate the efficiency of models, we use the number of trainable parameters to measure the parameter efficiency, use Sparsity in Pretrained Weights to measure the resource efficiency, and FLOPs to evaluate the computational efficiency. We add a star sign (∗) to indicate the structured sparsity (i.e., structurally prune the pretrained weights).
Baselines. On BERT and DeBERTa, we conduct comparisons with the following baseline methods: ¶ Fine-tune: directly fine-tunes the full model; · EarlyBERT (Chen et al., 2021b); ¸ BERT Tickets (Chen et al., 2020); ¹ OMP: prunes the fine-tuned weights by magnitude, and fine-tune afterwards; and º LoRA (Hu et al., 2021). We also report the Huggingface’s fine-tuning results.
On GPT-2, we conduct comprehensive comparisons with multiple baseline methods: ¶ Adapters (Houlsby et al., 2019): insert adapters after linear layers; · FT-Top2: fine-tune the top 2 layers only; ¸: Prefix: prefix tuning introduced by Li & Liang (2021); and ¹ LoRA: low-rank decomposition, which assumes ∆W = UV; most results are directly cited from Hu et al. (2021).

heading: 4.1 EFFICIENT TUNING WITH DSEE
text: Parameter-efficiency with sparse masks. To verify that using simple low-rank adaptation (i.e., LoRA) has limitations, we compare its performance with the performance of our sparsity-embedded efficient fine-tuning. Table 1 proves that on MNLI, SST-2, CoLA, and STS-B, the simple decomposition form shows unsatisfactory results compared to sparsity-embedded decomposition. Adding a small proportion of parameters (only 3072 trainable parameters) can bring a performance boost to the model. Specifically, under
rank 8, the metrics increase (0.69/0.13/0.008/0.003) on SST-2/MNLI/CoLA/STS-B, respectively. Moreover, using only approximately half of the number of trainable parameters, our method can achieve comparable parameters compared to the state-of-the-art method LoRA.
Table 2 shows the performance on GPT-2. For all three tasks, our method can achieve comparable results with only about half of the trainable parameters with LoRA with rank four and make substantial performance gain compared to LoRA with rank two. On WebNLG, our method even achieves a higher
BLEU score, 55.56 versus 55.29, with half of the trainable parameters. Such a phenomenon indicates that adding a sparse matrix to the low-rank decomposition can make substantial improvement.
Resource- and parameter-efficiency with sparse masks. We report the performance of DSEE, including both the number of trainable parameters and the sparsity in pretrained weights. We use an unstructured sparsity of 50%, and structured sparsity of 25% and 33%, which is equal to pruning 1/4 and 1/3 heads from each attention layer. For BERTBASE and DeBERTa-large, we set the r (the low-rank dimension) to be 16, and N (the number of non-zero elements in S2 ) to be 64. We also prune each of the intermediate layers using a structured sparsity of 40% as in (Chen et al., 2021b). For GPT-2, we set r to be 2 in DSEE and 4 in LoRA. The choice of N remains 64.
Table 3 summarizes the performance on BERTBASE. On BERTBASE, our DSEE method can: ¶ achieve parameter-efficiency at high level and retain the performance on various downstream tasks. On BERTBASE, with only about 600K trainable parameters (110/0.5929 ≈ 200× smaller than the full model), DSEE can achieve comparable performance on all GLUE tasks. · DSEE can achieve resource-efficiency in the final models. Using either unstructured or structured sparse masks can reach our goal of reducing the number of parameters in the pretrained weights without sacrificing much performance. At the sparsity level of 50%, unstructured DSEE can achieve an accuracy of 90.46% on QNLI, surpassing the baseline of fine-tuning, whose result is 90.15%. At the sparsity level of 25%, the structured DSEE can have performance gains ranging from 0.46% to 2.22%, except for QQP and QNLI. At the sparsity level of 33%, the performance gains range from 0.11% to 2.04%, except for QQP and QNLI. ¸ Compared to other baselines, e.g., EarlyBERT and BERT Tickets, our method benefits from parameter efficiency, which updates the weights more efficiently. These observations validate the effectiveness of our DSEE method.
We also calculate the inference FLOPs of BERTBASE on STS-B dataset. The original BERTBASE on STS-B takes FLOPs of 3.7835 × 1014, while LoRA takes FLOPs of 3.8096 × 1014, which is 0.69% higher. Conversely, the structured version of DSEE takes a FLOPs of 2.4921 × 1014 at the sparsity of 25%, and 2.3867× 1014 at the sparsity of 33%, which is 34.61% and 37.38% lower than LoRA. This indicates that a large proportion of computational cost can be saved at inference phase.
Table 4 summarizes the performance on GPT-2, which shares a similar trend. Unstructured DSEE can achieve 2000× reduction in trainable parameters and 2× reduction in the final fine-tuned model size with almost no loss in downstream task performance compared to conventional finetuning. When compared to LoRA, the unstructured DSEE can retain the same level of the number of trainable parameters and downstream task performance, while showing a 2× reduction in the final finetuned model size. The structured DSEE on GPT-2 seems to be less competitive than on BERT, but it can still hold performance on E2E and WebNLG after pruning 25% of heads in attention modules.
Finally, we validate our method on DeBERTa. The results are displayed in Table 5. We use four datasets, CoLA, MNLI, MRPC, and RTE. They have greatly varied sizes, which are representatives of the GLUE benchmark. DeBERTa is a larger model, so applying low-rank decomposition with our
hyperparameters cannot match the performance of fine-tuning. Compared to LoRA, our method reaches higher performance on downstream tasks, albeit it needs slightly more training epochs. However, such a slight extra cost for searching the sparse mask S1 can be amortized by the efficiency gained in the future inference, since the burden on resources such as storing the pretrained weights is relieved.
4.2 UNDERSTANDING DSEE
The position of embedded sparsity masks plays a crucial role in our proposed DSEE. For instance, applying sparse masks to the pre-trained weights or weight updates produces resourceand parameter-efficiency. Precisely, we compare four different methods on BERTBASE:
one-shot magnitude weight pruning (W S1), two DSEE’s variants (W S1 + UV) and W + UV + S2), DSEE (W S1 + UV + S2). The results are collected in Table 6. We can see that: ¶ NO embedded sparsity in the pretrained weights yields the overall best performance. This is intuitive since the valuable knowledge learned from massive pretraining data is intact and not removed. · Embedding sparsity into the pretrained weights harms only little to no performance. A similar conclusion is drawn by Chen et al. (2020) which validated that a pruned model can have a similar performance as the dense model. ¸ Using sparsity-embed efficient fine-tuning with the sparse pre-trained weights can also preserve performance, as well as achieve parameter efficiency.

heading: 4.3 ABLATION AND VISUALIZATION
text: Different methods to generate sparse matrix S2 In this section, we compare the performance of BERTBASE using different methods to generate the sparse matrix S2 and the corresponding space Ω. Except for the aforementioned matrix decomposition method, we try to (1) randomly sample indexes into Ω; and (2) directly select the indexes of elements of highest magnitude ofW into Ω. In Figure 2 we can see that using the matrix decomposition method has the highest metric overall. It is also noteworthy that sparse matrices generated by other methods will sometimes harm the performance.
Another factor is the number of non-zero elements inside S2. More non-zero elements in S2 reduce the parameter efficiency, while less non-zero elements increase the efficiency but may not be much beneficial to the accuracy or other metrics. Figure 2 also shows the relationship between number
of non-zero elements in S2 and the performance on fine-tuned model on SST-2. From the graph, we can see that using an N of 64 seems to have the most stable results compared to other choices. Another important piece of information we can derive from the figure is that a higher number of non-zero elements in the S2 does not guarantee better performance.
Ablation of # ranks. The rank of lowrank decomposition r is crucial to the transfer learning performance. A small r will result in lower representation ability, and a large r will bring more parameters and reduce the parameter efficiency. To find the best value, we conduct experiments with different ranks r on four datasets, SST-2, MNLI, CoLA, and STSB. The results are displayed in Figure 3. We add quadratic trend lines in the graph along with the discrete points to smooth
the results. We can draw two conclusions from the graphs: ¶ Overall, the final performance is positively correlated with the number of trainable parameters; however, on some tasks (MNLI, CoLA) higher number of parameters (namely, the r for U and V) will lead to lower performance. · With a sparse matrix embedded, the performance of the trained models can be improved within a range of trainable parameters. On SST-2 and CoLA, the applicable range seems to be 104.5 ∼ 106. On STS-B, our method can consistently outperform the performance of using low-rank decomposition only. For MNLI, the two methods behave similarly while our method is slightly better.
Ablation of sparsity. Although creating sparsity in pretrained weights W does not change the number of trainable parameters of models, the different levels of sparsity control the number of non-zero elements and thereby influence the representation ability. Therefore, we conduct experiments on BERT with different sparsity, ranging from 10% to 60%. The results are in Figure A5. From the figures, we can draw conclusions: ¶ DSEE out-performs magnitude pruning at low sparsity (< 50%) with respect to the performance on down-
stream tasks. · DSEE surpasses vanilla magnitude pruning with respect to the number of parameters. For each self-attention layer, using vanilla magnitude pruning needs to train all the weights, while DSEE only needs to update the three low-rank matrices.
Visualizations. Figure 4 shows the distribution of weight change. From the graph, we can see that most weights are located around 0. Such a phenomenon indicates that a natural sparsity exists within the update matrices, motivating us to explore sparse structures along with matrix decomposition.

heading: 5 CONCLUSION
text: This paper draws on the prior of sparsity and establishes the DSEE framework. It is the first attempt towards jointly optimizing both parameter efficiency of the fine-tuning process, and the resource efficiency of the fine-tuned model. On state-of-the-art large-scale language models (e.g., BERT, GPT, and DeBERTa) and across several of datasets, DSEE consistently demonstrates highly impressive parameter, training, and inference efficiency, in addition to preserving a competitive downstream transfer performance. Our future work targets extending DSEE to the finetuning of large-scale computer vision and/or multi-modal pre-trained models.

heading: A1 MORE IMPLEMENTATION DETAILS
text: We report the setting of other hyperparameters of our experiments, such as learning rate, in Table A7. The device we used for experiments are various, including NVIDIA GeForce GTX 1080 Ti, GeForce RTX 2080 Ti, Titan RTX, and A6000.
Architecture Method Parameters Dataset MNLI QNLI QQP SST-2 CoLA MRPC RTE STS-B
BERTBASE Fine-tune Learning Rate 5e-5 BERTBASE DSEE (before pruning) Learning Rate 5e-5 5e-5 5e-5 2e-4 1e-3 8e-4 1e-3 8e-4 BERTBASE DSEE (after pruning) Learning Rate 5e-5 5e-5 5e-5 5e-5 1e-3 5e-4 5e-4 5e-4
DeBERTa-large LoRA & DSEE (before pruning) Learning Rate 1e-5 - - - 1e-3 8e-4 8e-5 - DeBERTa-large DSEE (after pruning) Learning Rate 1e-5 - - - 5e-5 8e-4 6e-5 -
Table A7: Hyper-parameters we used on different datasets.
GreBsmo Algorithm GreBsmo (Zhou & Tao, 2013) is an algorithm for solving the Robust PCAlike methods. The optimization of U , V , and S follows the following iterative rules:
 Uk = Q,QR ( (X − Sk−1)V Tk−1 ) = QR Vk = Q T (X − Sk−1)
Sk = Sλ (X − UkVk) , (2)
where X is the original dense matrix, QR(·) means the QR decomposition, Sλ(·) indicates the soft-threshold operation, and the subscripts k indicates the optimization step.

heading: A2 MORE EXPERIMENT RESULTS
text: Ablation of Sparsity To study the relationship between sparsity of unstructured pruning and the behavior of our DSEE, we conduct an ablation study on various datasets in GLUE benchmarks. The results are shown in Figure A5
0.880
0.885
0.890
5.5 6.0 6.5 7.0 7.5 8.0 log10(#Trainable Parameters)
Pe ar
so n'
s r
Methods Magnitude Pruning DSEE0.880
0.885
0.890
20 40 60 Sparsity
Pe ar
so n'
s r
Methods Magnitude Pruning DSEE
ST SB
89.5
90.0
90.5
91.0
91.5
5.5 6.0 6.5 7.0 7.5 8.0 log10(#Trainable Parameters)
A cc
ur ac
y
Methods Magnitude Pruning DSEE
89.5
90.0
90.5
91.0
91.5
20 40 60 Sparsity
A cc
ur ac
y
Methods Magnitude Pruning DSEE
SS T2
83
84
85
86
5.5 6.0 6.5 7.0 7.5 8.0 log10(#Trainable Parameters)
A cc
ur ac
y
Methods Magnitude Pruning DSEE
83
84
85
86
20 40 60 Sparsity
A cc
ur ac
y
Methods Magnitude Pruning DSEE
M R PC
67 68 69 70 71 72 73
5.5 6.0 6.5 7.0 7.5 8.0 log10(#Trainable Parameters)
A cc
ur ac
y
Methods Magnitude Pruning DSEE
67
68
69
70
71
72
73
20 40 60 Sparsity
A cc
ur ac
y
Methods Magnitude Pruning DSEE
R T E
80
81
82
5.5 6.0 6.5 7.0 7.5 8.0 log10(#Trainable Parameters)
A cc
ur ac
y
Methods Magnitude Pruning DSEE
80
81
82
20 40 60 Sparsity
A cc
ur ac
y
Methods Magnitude Pruning DSEE
M N L I
0.52
0.56
0.60
5.5 6.0 6.5 7.0 7.5 8.0 log10(#Trainable Parameters)
C or
re la
tio n
Methods Magnitude Pruning DSEE0.52
0.56
0.60
20 40 60 Sparsity
C or
re la
tio n
Methods Magnitude Pruning DSEE
C oL A
Figure A5: DSEE performance compared to vanilla magnitude pruning at different sparsity. Magnitude Pruning: vanilla magnitude pruning which tunes W directly.
Results with more runs. We conduct experiments with three runs. The mean and standard deviation of accuracy of LoRA and DSEE (without pruning) are shown in Table A8. We have also included the p-value of t-tests to show the significance of the performance gaps. On half of the
A14
datasets, the p-value is around 0.1; and on three datasets the p-value is between 0.2 and 0.4. The reason why p-value is large on some datasets is probably because the number of experiments still has room to increase.
Table A8: Performance comparison of different methods on BERTBASE on GLUE benchmarks. The first row of each method indicates the mean accuracy, and the second row indicates the standard deviation.
Methods # Trainable Sparsity in DatasetParameters Pretrained Weights CoLA STS-B MNLI QQP QNLI MRPC RTE SST-2
LoRA 589.8K 0% 57.38 88.81 80.88 86.45 88.49 86.03 71.00 92.171.43 0.23 0.41 0.14 0.10 1.49 1.10 0.27
DSEE 592.9K 0% 58.73 89.16 81.03 86.53 88.58 85.78 71.12 92.550.95 0.03 0.62 0.06 0.17 1.07 0.62 0.12
P-value of matched pair t-tests 0.086 0.054 0.383 0.121 0.276 0.572 0.277 0.031
DSEE 592.9K 50% 55.60 88.39 81.16 87.23 88.88 84.88 70.09 91.000.81 0.21 0.23 0.03 0.30 0.93 1.37 0.13
Inference time on BERTBASE We have recorded the inference time of BERTBASE on various GLUE benchmarks, QQP, STS-B, CoLA and RTE. The number of samples in their evaluation set range from 277 to 40430, so they can be considered representative. The results are shown in Table A9. From the table we can see that, the inference time are greatly saved after using the structured version of DSEE.
Table A9: Inference time of different methods on BERTBASE on GLUE benchmarks. The sparsity with star signs indicates that it is a structured sparsity.
Dataset # eval samples MethodsLoRA DSEE (25%∗) DSEE (33%∗)
QQP 40,430 50.773 36.391 34.946 STS-B 1,500 2.033 1.506 1.337 CoLA 1,043 1.316 0.931 0.877 RTE 277 0.464 0.420 0.376
Convergence Speed We have compared the convergence speed (i.e., how the test accuracy changes) of LoRA and DSEE. From Figure A6 we can see that the convergence speeds of the two methods are not significantly different. This means introducing a sparse component into the decomposition form does not affect the convergence dynamics.
0.75
0.80
0.85
0.90
0 50 100 Evaluation Step
Te st
A cc
ur ac
y
Method
DSEE
LoRA
Figure A6: Convergence speed of two methods (LoRA and DSEE). The x-axis represents evaluation steps and y-axis represents the test accuracy.
A15

Review summaries:
This paper proposes methods to improve the efficiency of large pre-trained models (millions or billions of parameters) by combining multiple notions of _sparsity_ and/or _dimensionality reduction_ on the model weights. This is achieved via (1) a low-rank update matrix with fine-tunable parameters, and (2) pruning the original pre-trained weight matrix. These two sets of parameters are combined during inference. The authors validate their proposed method on several model architectures and datasets.

This paper proposes a principled way for achieving both parameter efficient finetuning and resource efficient inference. They combines the two steps of (i) computing a low-rank decomposition of parameter updates with a sparsity matrix (ii) further prune the trained (ie., finetuned) model. The first step aimed to achieve the parameter efficient finetuning but w/ the sparse residual components based on sparsity from the pretrained model to preserve the performance. The second step can reduce the memory of parameter storage. On three benchmarks with three different model architectures such as encoder-only, encoder-decoder they a large number of experiments with ablation studies. Studies show several a number of promising enhancement over other baselines.  

This paper proposes to enforce weight sparsity constraints into a prior parameter-efficient tuning method (LoRA) to achieve both parameter-efficient tuning and resource-efficient inference. Specifically,  this paper (1) adds another sparse, learnable weight matrix to the original low-rank decomposition in LoRA; (2) incorporates unstructured and structured weight pruning techniques to save computation at inference time. Experiments are conducted on GLUE and several data2text generation benchmarks to demonstrate the effectiveness of the proposed method.