DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Year: 2023

Abstract:
Gigantic pre-trained models have become central to natural language processing (NLP), serving as the starting point for fine-tuning towards a range of downstream tasks. However, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g., 175B parameters for GPT-3), even the fine-tuning process can be time-consuming and computationally expensive; (b) the finetuned model has the same size as its starting point by default, which is neither sensible due to its more specialized functionality, nor practical since many fine-tuned models will be deployed in resource-constrained environments. To address these pain points, we propose a framework for resourceand parameterefficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning by enforcing sparsity-aware low-rank updates on top of the pre-trained weights; and (ii) resourceefficient inference by encouraging a sparse weight structure towards the final fine-tuned model. We leverage sparsity in these two directions by exploiting both unstructured and structured sparse patterns in pre-trained language models via a unified approach. Extensive experiments and in-depth investigations, with diverse network backbones (i.e., BERT, RoBERTa, and GPT-2) on dozens of datasets, consistently demonstrate impressive parameter/inference-efficiency, while maintaining competitive downstream performance. For instance, DSEE saves about 25% inference FLOPs while achieving comparable performance, with 0.5% trainable parameters on BERT. Codes are available at https://github.com/VITA-Group/ DSEE.

heading: 1 Introduction
text: Most recent NLP applications have been following the pre-train then fine-tune paradigm, starting from
a gigantic pre-trained model and fine-tuning it towards downstream tasks. Conventional fine-tuning works through updating all of the parameters in the pre-trained model. However, as the size of pre-trained models grows, updating all parameters becomes less feasible in most practical scenarios, due to the expensive memory and computational requirements. For example, BERTBASE (Devlin et al., 2019) has 110M trainable parameters, while GPT-2 (Radford et al., 2019) has up to 1.5B and the largest version of GPT-3 (Radford et al., 2019) has an astonishing 175B trainable parameters. As such, conventional fine-tuning of the larger models could require hundreds of GPU hours. Another downside of this paradigm is that it requires storing as many parameters as in the large-scale pretrained models for each downstream task, which poses impediments to the deployment in real-world resource-constrained environments.
One solution to address the extensive resource requirement of conventional fine-tuning is model pruning (LeCun et al., 1990; Han et al., 2015; Ren et al., 2018; He et al., 2017; Liu et al., 2017), where unnecessary weights are eliminated to shrink the model size. For example, Chen et al. (2021b) leverages ℓ1 regularization to remove insignificant attention heads and gains 35 ∼ 45% training time with comparable performance; Chen et al. (2021a); Dao et al. (2022) leverage sparse matrices with fixed structures to reduce pretrained models’ sizes. All these studies indicate the rise of sparsity naturally during fine-tuning a general-purpose pre-trained model, to some specialized downstream functionality. One potential interpretation, of why sparsity arises, is that different subsets of the parameters may be responsible for different downstream tasks and data domains (Sanh et al., 2020). However, identifying appropriate sparse masks can be burdensome: fine-tuning a large pre-trained language model like GPT-3 for just one step consumes at least 1.2TB of VRAM and requires 96 pieces of
8208
NVIDIA Tesla (Hu et al., 2021), and these methods either require access to pre-trained weights or introduce additional learnable coefficients (such as importance scores of attention heads).
A parallel alternative is to design parameterefficient fine-tuning algorithms, which aim at optimizing a small portion of weights while fixing most of them when fine-tuning on downstream tasks. Pioneering works along this line, which utilize adapters (Houlsby et al., 2019), learnable embeddings (Li and Liang, 2021; Liu et al., 2021), low-rank decomposition (Hu et al., 2021) or their combination (He et al., 2021), can significantly reduce the number of trainable parameters while preserving good fine-tuning performance. Although these methods can substantially improve the storage and deployment efficiency of models, there are two major hurdles: (i) they does not yield any inference efficiency gains since the full pretrained weights are still required to calculate outputs; and (ii) current methods assume the updates on pretrained weights to be either sparse (Guo et al., 2020) or low-rank (Hu et al., 2021), yet those assumptions might be oversimplified (Yu et al., 2017) and overly restricted to allow for effective updates. These observations have inspired us to explore better parameter-efficiency methods.
To improve both resource- and parameterefficiency during model fine-tuning, we explicitly draw on the prior of sparsity for both weight updates and the final weights, and establish a dually sparsity-embedding efficient tuning (DSEE) framework. Starting from a pre-trained model, DSEE first adopts a sparsity-aware low-rank weight update to achieve parameter-efficiency of the finetuning process; and then enforces a sparse weight structure directly from weight updates by masking to achieve resource-efficiency of the fine-tuned model at inference time. Our contributions can be summarized as follows:
• We propose the dually sparsity-embedding efficient tuning, which unifies sparsity-aware parameter-efficient weight update and sparse pretrained weight in fine-tuning gigantic pre-trained models. It is the first attempt toward jointly optimizing both parameter-efficiency of the finetuning process and the resource-efficiency of the fine-tuned model.
• Both unstructured and structured sparse priors are investigated in our proposed DSEE algorithm. For weight updates, the injected sparsity
prior enhances existing parameter-efficient update schemes (e.g., low-rank decomposition). As for the final weights, we draw superior sparse masks, either unstructured or structured, directly from the weight updates, which requires neither additional parameters nor access to the pretrained weights and saves the sparsification cost.
• Extensive experiments demonstrate the effectiveness of our proposal across various representative pre-trained language models (BERT, GPT-2, and RoBERTa) and on diverse evaluation benchmarks (E2E, DART, WebNLG, and GLUE). On GPT-2, our methods can achieve a BLUE score of {69.5,54.9,47.5} with 0.1% of trainable parameters on {E2E, WebNLG, DART} with 20% parameter removed in pre-trained weights. On BERT, DSEE can fine-tune only 0.5% parameters and save about 25% inference FLOPs, while losing less than 2% performance.

heading: 2 Related Work
text: Pruning and Sparsification Pruning is a classical model compression technique that can reduce the number of parameters, which can bring training and inference efficiency. Researchers have proposed several pruning methods for pre-trained language models: McCarley et al. (2019); Chen et al. (2021b) pruned attention heads that had less contribution during finetuning; Sanh et al. (2020) proposed a pruning criterion targeting the weight change after training, which suits the transfer learning better; Wang et al. (2020) incorporated lowrank factorization and ℓ0 regularization for pruning. Recently, there is a series of sparsification works that utilize sparse masks with specific structures, called Butterflies, and achieve high efficiency in pretraining models (Chen et al., 2021a) or finetuning on downstream tasks (Dao et al., 2022). However, these methods do not allow for parameterefficient updates.
Low-rank decomposition Low-rank approximation (Ye, 2005) has broad applications in the machine learning community and is vastly studied. One classical scenario is the robust principal component analysis (Candès et al., 2011), which decomposes a matrix into a low-rank plus a sparse component. Existing literature shows that in deep learning, the learned over-parameterized models often naturally bear approximate low-rank weight structures (Oymak et al., 2019; Yu et al., 2017).
Some (Jaderberg et al., 2014; Povey et al., 2018; Sainath et al., 2013; Zhang et al., 2014; Zhao et al., 2016) have explicitly imposed the low-rank constraint during training. Wang et al. (2020); Hu et al. (2021) utilized low-rank decomposition to shrink the model size and trim down the trainable parameters during fine-tuning. However, to our best knowledge, integrating sparsity and low-rank structures has never been studied before for efficient fine-tuning of pre-trained language models.
Parameter-efficient adaptation. Parameterefficient fine-tuning aims to reduce the number of trainable parameters when fine-tuning the models across different downstream domains. Unlike pruning, it aims at adapting models with fewer parameters instead of building sparse models. Various approaches are proposed to achieve this goal: Rebuffi et al. (2017); Houlsby et al. (2019) inserted and only trained adapters between existing layers, whose parameters are much less compared to the pretrained models. Guo et al. (2020) leveraged ℓ0 regularization to limit the number of non-zero elements in the update vectors. Lester et al. (2021); Li and Liang (2021); Liu et al. (2021) introduced efficient prompt tuning which optimizes only a small continuous task-specific vector. Zaken et al. (2021) fine-tunes only the bias terms inside models. Hu et al. (2021) proposed a low-rank decomposition-based method, and He et al. (2021) combined low-rank and adapter-based methods for efficient finetuning. However, fine-tuned models yielded by these methods have the same amount of weights as the pre-trained models; hence they contribute no resource efficiency.

heading: 3 Methodology
text: In this section, we describe our notations and definitions of sparsity generation and parameter-efficient fine-tuning in Section 3.1, and then introduce the dually sparsity-embedded efficient fine-tuning algorithms in Sections 3.2 and 3.3.

heading: 3.1 Preliminaries
text: Sparsification and resource-efficient fine-tuning. We adopt both unstructured and structured pruning methods to produce sparsity. They can lead to resource-efficiency including memory and computation savings.
Given W ∈ Rm×n a weight matrix, pruning aims at finding a binary mask M ∈ {0, 1}m×n which is applied to W and generating a sparse
weight W ⊙ M. The weights at the positions where M have “0” value are considered as pruned. Pruning methods can be classified into two classes by the structure of M: For unstructured pruning where M does not have sparse structures such as rows and columns, the memory cost is saved due to fewer number of nonzero parameters; for structured pruning, it also helps save computational cost since the sparse weights can be smaller in size. One of the most widely used unstructured pruning methods is the weight magnitude (Han et al., 2015), i.e., remove the weights with the smallest absolute values. One common structured pruning method in the NLP field is the head pruning (McCarley et al., 2019), which tries to remove unimportant attention heads from the model. Parameter-efficient fine-tuning. To leverage the knowledge in pre-trained weights W , downstream models learn task-specific weight update ∆W via fine-tuning and generate predictions with weights W + ∆W , where the output of models is calculated as (W + ∆W)x with x as the input. Since ∆W has the same size as W , learning the update matrices usually requires massive resources as the size of the pre-trained model increases. Parameterefficient fine-tuning tries to solve this problem by using as few trainable parameters as possible to represent ∆W , while maintaining competitive downstream fine-tuning performance. Previous literature reaches the goal via either sparsifying weight update matrices ∆W (Guo et al., 2020) or leveraging low-rank decomposed matrices to compute ∆W (Hu et al., 2021), while in our work we combine both of them.

heading: 3.2 Sparsity-Embedded Parameter-Efficient Fine-tuning
text: A recent study (Hu et al., 2021) enforces low-rank constraint to weight update tensors ∆W , and obtains a satisfactory trade-off between parameterefficiency and model quality. However, as re-
Algorithm 1: Sparsity-Embedded LowRank Decomposition Input: Pretrained weights W , number of
non-zero elements N , number of weights to decompose n.
Output: Indices sets Ωi, i = 1, 2, . . . , n. 1 Initialize each Ωi to be an empty set. 2 for each weight matrix Wi in W do /* Decomposition */ 3 Perform matrix decomposition
Wi ≈ AB + S ′ by solving the optimization problem 1.
/* Extract important elements from S ′ into Ωi. */
4 Perform thresholding on S ′: Keep N elements in S ′ with top magnitudes, and append their locations into Ωi. 5 end
vealed experimentally by (Yu et al., 2017), a part of the important information in the trained weights scatters outside the low-rank subspace, creating sparse “residuals". Inspired by this observation, we investigate a new sparsity-aware low-rank subspace of ∆W , and introduce the first component of our proposal in Figure 1, i.e., sparsity-embedded parameter-efficient fine-tuning.
Specifically, the weight updates ∆W are consisted of two components as illustrated in Figure 1,: (1) a low-rank component ∆Wl built by the multiplication of two matrices U ∈ Rm×r and V ∈ Rr×n; and (2) a sparse residual ∆Ws = PΩ(S) where S ∈ Rm×n is a learnable matrix, PΩ(S) ={ si,j , (i, j) ∈ Ω 0, (i, j) ∈ ΩC , i = 1, 2, . . . ,m, j = 1, 2, . . . , n, wi,j is the parameter of S at location (i, j), and Ω is a indices set containing the positions of non-zero elements in S. The update matrix ∆W is expressed as ∆Wl + ∆Ws, with U , V and S as the learnable parameters while Ω is fixed once determined. Compared to the full finetuning which has m × n trainable parameters for a matrix with size m × n, our method only has (m + n) × r + card(Ω) trainable parameters. If r is smaller than m×n−card(Ω)m+n ⪅ 0.5min{m,n}, our method is capable of reducing trainable parameters for downstream fine-tuning. In practice, the value of r is very small compared to m and n so the savings are significant.
One question for the above pipeline is how to
find a high-quality indices set Ω. Inspired by the observation that the low-rank component ∆Wl is highly correlated with the low-rank structure of W (Hu et al., 2021), we hypothesize that the indices set Ω should be highly correlated as well. More concretely, we hypothesize that the sparse residuals that are not in the low-dimensional subspace of W may also lay outside ∆Wl, which motivates the design of sparse update ∆Ws. We formulate the problem of discovering the sparse residuals of W as a Robust Principal Component Analysis (Candès et al., 2011). Formally, we aim at solving the following optimization problem:
min U,V,S
1 2 ∥W − UV − S∥2F
s.t. rank(U) ≤ r, rank(V ) ≤ r, card(S) ≤ c.
(1)
where rank(·) and card(·) indicate the rank and the number of non-zero elements of a matrix, respectively. S ′ represents the sparse residuals that cannot be fit in the low-rank component AB, and we acquire the locations of elements with non-zero magnitude into Ω. To solve Problem 1 efficiently, we adopt an SVD-free algorithm called GreBsmo (Zhou and Tao, 2013) (refer to Section A.2). Algorithm 1 summarizes the detailed procedure of constructing sparse indices sets Ω. Empirically, we set the size of Ω (i.e., c) to be 16 since it yields high test performance (refer to Section 4.2) while imposing little overhead on parameters. The initial values of V and S are set as 0 so these matrices do not affect outputs at the beginning of training.

heading: 3.3 Dually Sparsity-Embedded Efficient Tuning (DSEE)
text: Adapting pre-trained models with ∆Wl and ∆Ws can bring significant parameter-efficiency, but does not directly bring any resource-efficiency such as memory or computational cost. Motivated by such, we propose a unified framework called DSEE pursuing both parameter- and resource-efficiency simultaneously. We leverage the sparsity in pretrained models’ weights to enhance the resource efficiency, as demonstrated in Figure 1. More specifically, we derive sparse masks M directly from the parameter-efficient updates ∆W , and apply the sparse masks by pruning the pre-trained weights W to seek resource-efficiency. It requires no additional parameter for sparsifying the model and no access to the underlying pretrained weights, which is favorable due to the lower sparsification cost.
As shown in Algorithm 2, DSEE handles unstructured and structured pruning at the same time: for unstructured pruning, we sort the magnitude of ∆W , generate a sparse mask M by assigning “1” to the position where ∆W have largest magnitude and “0” to the rest; for structured pruning, we sum the magnitude of ∆W of each head and remove those with least scores. We also shrink ∆W accordingly by removing the corresponding weight columns in V and ∆Ws to match the shape while keeping U intact. A comparison of different pruning criteria is shown in Section 4.2.1, which demonstrates that ∆W is a superior choice due to the high downstream task performance and no access to the pretrained weights W .
Given a parameter budget, the number of parameters per module decreases if we choose to adapt more modules, which imposes a trade-off. We study different choices of modules to adapt in Section 4.2.2, and we find the optimal modules to adapt are Wq and Wv, where Wq and Wv stand for the projection weights for query and value in attention heads. Since some modules are not adapted during fine-tuning (i.e., ∆W = 0), we prune them separately according to the magnitude of the corresponding pre-trained weights. After applying the mask M to the pretrained weights W , we conventionally tune ∆Wl(= UV) and ∆Ws(= PΩ(S)) for several epochs to recover the performance (Han et al., 2015).

heading: 4 Experiment Results
text: Datasets and models. We use three classical pre-trained language models in our experiments: BERTBASE (Devlin et al., 2019), RoBERTaLARGE (Liu et al., 2019) and GPT2 (Radford et al., 2019), which have 12/24/24 layers with hidden size of 768/1024/1024 and 110/380/354M trainable parameters, respectively. For BERT and RoBERTa, we evaluate on the GLUE benchmarks (Wang et al., 2018), and for GPT-2 we use E2E (Novikova et al., 2017), WebNLG (Gardent et al., 2017) and DART (Nan et al., 2021).
Training and evaluation details. For BERT and RoBERTa, we follow the default settings in Wolf et al. (2019); Devlin et al. (2019). We use the AdamW (Loshchilov and Hutter, 2017) optimizer for downstream fine-tuning, and a batch size of 32 for BERT and RoBERTa, and a batch size of 2 for GPT-2. The rest hyper-parameters for training are reported in Table 11.
Algorithm 2: DSEE Input: Pretrained weights W , number of
non-zero elements N , desired sparsity s, loss function L.
Output: Sparse mask M, matrices U ,V,S . Derive Ω from pretrained weights W . Initialization: U = 0,V ∼ N (0, 0.02), and S = 0. /* I: train before pruning */ Train U ,V,S with respect to L under the constraint of PΩC (S) = 0. /* II: pruning the model */ if using unstructured pruning then
Prune (1− s%) weights in W by sorting the magnitude of ∆W .
else Prune (1− s%) heads by sorting the aggregated magnitude of ∆W of heads. Shrink V and S accordingly to match the shape. end if /* III: tuning after pruning */ Fine-tune U ,V,S to recover the performance.
Evaluation Metrics. For the GLUE benchmark, we report the accuracy, Matthew’s correlation, and Pearson’s correlation in the evaluation. On GPT2, we use BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014), TER (Snover et al., 2006) and NIST (Doddington, 2002) as the evaluation metrics. To evaluate the efficiency of models, we report the number of trainable parameters to measure the parameter-efficiency, the number of total parameters (the number of non-zero parameters in the model) to measure the resourceefficiency, and FLOPs for the computational efficiency.
Baselines. On BERT and RoBERTa, we conduct comprehensive experiments with the following baseline methods: ❶ Fine-tune: directly finetuning the full model; ❷ EarlyBERT (Chen et al., 2021b): learn importance scores for heads and perform pruning based on them afterwards; ❸ BERT Tickets (Chen et al., 2020): IMP-based unstructured pruning; ❹ P-Tuning v2 (Liu et al., 2021); ❺ Bitfit (Zaken et al., 2021): fine-tuning bias terms only; and ❻ LoRA: low-rank decomposition, which learns ∆Wl only (Hu et al., 2021). On GPT2, we conduct comparisons with multiple baseline methods: ❶ Adapters (Houlsby et al., 2019): insert
·
adapters after linear layers; ❷ FT-Top2: fine-tune the top 2 layers only; ❸: Prefix: prefix tuning introduced by Li and Liang (2021); and ❹ LoRA.

heading: 4.1 Efficient Tuning with DSEE
text: Parameter-efficiency with sparse residuals. To verify that using a simple low-rank component ∆Wl has limitations, we compare its performance with our sparsity-embedded efficient fine-tuning. Table 1 shows that on four benchmarks (i.e., SST-2, RTE, CoLA, and MRPC), adding a sparse residual in weight updates can bring performance gain: at the level of approximately 600K trainable parameters, adding sparse residuals with only 384 nonzero elements (12 × 2 × 16 = 384) can increase the validation performance on all benchmarks except CoLA by 0.23% ∼ 1.09%; at the level of approximately 300K trainable parameters, adding sparse residuals can bring performance gain ranged from 0.34% to 1.08% on all four benchmarks.
We further verify that adding sparse residuals ∆Ws could benefit NLG tasks with GPT-2. Table 2 shows that under different levels of parameters, adding sparse residuals ∆Ws yields higher performance for most metrics on three tasks. At the level of 0.39M parameters, adding sparse residuals can improve all metrics on WebNLG and DART, and sightly boost the NIST score on E2E. At the level of 0.20M parameters, ∆Ws helps increase all metrics across three tasks. We also show the standard deviation in Table 10.
Resource- and parameter-efficiency with unstructured sparse masks. We verify that DSEE is capable of enhancing both parameter- and resource-efficiency, while preserving performance on downstream tasks, on various architectures. Table 3 summarizes the experiment results on BERTBASE, and we observe that introducing unstructured sparsity patterns inside pretrained weights not only brings resource-efficiency (manifested by the fewer number of total parameters) but also potentially improves the performance on downstream tasks. Specifically, at 80% and 70% of total parameters, DSEE can remain compara-
ble performance on downstream tasks, and even present a performance boost on QQP, RTE, and SST-2 compared to LoRA. At the level of 50% parameters, performance on smaller datasets such as CoLA and RTE drops by a wider margin; but on larger datasets such as QQP, DSEE can maintain comparable performance (< 1.5% gap) after sparsification.
On GPT-2, we observe a similar trend as shown in Table 4. DSEE can achieve superior performance with unstructured sparse patterns with 80% total parameters compared to finetuning the entire model, and remain highly competitive with other baselines with fewer parameters in the model. Using only 50% of parameters in pre-trained weights, DSEE can achieve comparable performance with the full fine-tuning on E2E and DART.
Finally, we validate if DSEE can work on the larger model RoBERTaLARGE. We conduct experiments on four datasets (CoLA, SST-2, QNLI, and RTE), and present the results in Table 5. Compared to full-finetuning, LoRA, and Adapter, our method reaches comparable performance on these four downstream tasks and saves resources at the same time. The performance gap is maximal 1% but 30% parameters in the models are removed.
Resource- and parameter-efficiency with structured sparse masks. DSEE can directly perform structured pruning on weights without additional parameters such as importance scores of heads. In Table 6 we show the performance of structured pruned BERTBASE on several tasks in the GLUE benchmark, where we study the testing accuracy after removing 3, 6 and 9 attention heads on SST-2, MNLI, QNLI and QQP, as well as the inference FLOPs ratios of the model. Firstly, removing 3 heads from the model reaches comparable performance against full fine-tuning (improved on SST-2, MNLI, and QNLI) and LoRA (improved on SST-2 and QQP), while taking advantage of reduced inference FLOPs. Secondly, removing 6 heads from the model will lead to lower performance since half of the parameters in the projection matrices are eliminated. However, the performance of DSEE is still higher than EarlyBERT. Lastly, DSEE with 9 heads removed from the model leads to comparable performance with EarlyBERT, but the number of trainable parameters is substantially smaller (0.6M versus 66M).

heading: 4.2 Ablation and Visualization
text: We study several choices of parameters and provide visualization in this section.

heading: 4.2.1 Different criteria for sparse masks
text: We find the magnitude of weight updates (i.e., |∆W|) is an effective solution for preserving performance with both unstructured and structured pruning. We conduct experiments on the adapted weights (i.e., Wq and Wv), and compare against two baselines: ❶ Random: perform random pruning on the adapted modules; ❷ |W + ∆W|: perform pruning based on the magnitude of final adapted weights. Table 7 shows the results on RTE and SST-2 with BERTBASE. We can see from the table that: ❶ performing unstructured pruning without accessing the pretrained weights can achieve comparable performance on RTE and SST-2, only slightly weaker than pruning with final adapted weights; ❷ performing structured pruning according to ∆W yields the highest performance on both datasets after training. These observations verify the effectiveness of our proposal.

heading: 4.2.2 Different choices of modules to adapt
text: We study the choices of modules to adapt for DSEE on RTE. We choose possible modules to adapt within Wq, Wk, Wv, and Wo, representing the projection matrix for query, key, value, and output, respectively. We hold the number of trainable
parameters at the same level and set the sparsity level at 30%. Table 9 summarizes the performance with different adapted weights, which demonstrates that adapting Wq and Wv yields the highest performance. Each module will be given fewer parameters when adapting more modules and the model may not be sufficiently fine-tuned when adapting fewer modules and leading to inferior performance.
Different methods for identifying Ω. We compare our proposal against various methods to identify Ω from pretrained weights W: ❶ Magnitude, which selects the position of elements with highest magnitude into Ω; ❷ Random, which randomly samples positions into Ω. The results are shown in Figure 2. We can observe that our proposal can identify high-quality Ω for finetuning on downstream tasks, shown by the consistently higher performance with different sizes of the indices set Ω.
Different sizes of Ω. We search over 8 ∼ 256 to find the optimal size of Ω. Ω with a smaller size brings fewer performance gains, and Ω with a larger size may harm the efficiency. Figure 2 shows the relationship between the size of Ω and the performance on SST-2. We find the optimal choice for this task is 16 where the model achieves the highest performance. Consequently, we by default set the size of Ω to 16 for simplicity.

heading: 5 Conclusion
text: This paper draws on the prior of sparsity and establishes the DSEE framework. It is the first attempt toward jointly optimizing both parameterefficiency of the fine-tuning process, and the resource-efficiency of the fine-tuned model. On
state-of-the-art large-scale language models (e.g., BERT, GPT, and RoBERTa) and across several datasets, DSEE consistently demonstrates highly impressive parameter and inference efficiency, in addition to preserving a competitive downstream transfer performance on various tasks. Our future work targets extending DSEE to the finetuning of large-scale computer vision and/or multi-modal pre-trained models.
Limitation The unstructured sparse patterns we introduce are not as hardware-friendly as the structured patterns, suggesting the speedup of using unstructured patterns maybe limited due to the implementation. The number of parameters of models we are studying are only at the level of 100 ∼ 300M, and the datasets are focus on GLUE, E2E, WebNLG, and DART. We will generalize to wider choices of datasets in future works.

heading: 6 Ethical and Broader Impacts
text: DSEE aims at reducing the number of trainable parameters when fine-tuning the models, which can help save the cost of saving new weights. This can be helpful to companies who are fine-tuning large-
scale language models on various downstream tasks, suggesting our work has potentially positive broader impact. On the other hand, our work does not have obvious ethical impacts, as we focusing on model tuning.

heading: A More Implementation Details
text: A.1 Hyper-parameters We report the learning rates, the batch sizes, and the max sequence length for DSEE in Table 11. The device we used for experiments are various, including NVIDIA GeForce GTX 1080 Ti, GeForce RTX 2080 Ti, Titan RTX, and A6000. We follow (Hu et al., 2021) to set the evaluation protocols on E2E, WebNLG, and DART.
A.2 Decomposition Method GreBsmo (Zhou and Tao, 2013) is an algorithm for solving the Robust PCA-like methods. The optimization of U , V and S follows the following iterative rules:
   Uk = Q,QR ( (X − Sk−1)V Tk−1 ) = QR Vk = Q T (X − Sk−1)
Sk = Sλ (X − UkVk) ,
(2) where X is the original dense matrix, QR(·) means the QR decomposition, Sλ(·) indicates the softthreshold function (i.e., Sλ(x) = x1|x|≥λ) , and the subscripts k indicates the optimization step.
A.3 Statistics and Usage of Datasets We report the statistics of datasets in Table 8. For GLUE tasks we report the sizes of the train, the dev and the test set, and for non-GLUE tasks we report the sizes of the train, validation (dev), and test set. We follow the conventional use of these datasets (Hu et al., 2021) and do not modify the conventional splits.

heading: B More Experiments Results
text: B.1 Ablation Studies Table 9 summarizes the performance with different adapted weights, which demonstrates that adapting Wq and Wv leads to the highest performance.
Performance of different Ω. We conduct additional ablation study experiments (three runs for each experiment) on the sizes of Ω on three datasets in GLUE (i.e., STSB, QNLI and MRPC). The results shown in Table 12 below verify that our method are generalizable to other datasets. On STSB and QNLI, using a size of 16 can achieve the best performance, while on MRPC it can achieve a comparable test accuracy.
Compare with recent methods. We have conducted a set of experiment to compare our methods
GLUE
Weights Test Acc. Weights Test Acc.
Wq 68.59 (0.21) Wk 67.87 (0.21) Wv 68.23 (1.82) Wo 68.23 (1.05) Wq,Wk 68.95 (1.11) Wq,Wv 71.48 (2.16) Wk,Wv 70.04 (0.75) Wq,Wk,Wv 69.31 (2.56)
with MAM Adapter (He et al., 2021). We train a RoBERTa-large with their method on SST-2, QNLI, RTE, by following the same hyperparameters used in the original work. The results are shown in Table 13. We observe that our method, even with sparse models, achieves same-level performance with LoRA and MAM Adapter.
Other Pruning Methods. We apply the iterative magnitude pruning method on RTE. Specifically, we train the model for 10 epochs, prune 10% of the remaining weights, and fine-tune for 10 epochs before the next pruning. Table 14 shows that directly applying iterative magnitude pruning does not bring performance improvements over the oneshot pruning baseline.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Our future work targets extending DSEE to the finetuning of large-scale computer vision and/or multi-modal pre-trained models. Limitation The unstructured sparse patterns we introduce are not as hardware-friendly as the structured patterns, suggesting the speedup of using unstructured patterns maybe limited due to the implementation. The number of parameters of models we are studying are only at the level of 100 ∼ 300M, and the datasets are focus on GLUE, E2E, WebNLG, and DART. We will generalize to wider choices of datasets in future works