DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
Authors: Xuxi Chen
Tianlong Chen
Yu Cheng
Weizhu Chen
Zhangyang Wang
Ahmed Hassan Awadallah
Venue: ICLR 2022 Submitted (2022)

Abstract: Gigantic pre-trained models have become central to natural language processing (NLP), serving as the starting point for fine-tuning towards a range of downstream tasks. However, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g., $175$B parameters for GPT-3), even the fine-tuning process can be time-consuming and computationally expensive; (b) the fine-tuned model has the same size as its starting point by default, which is neither sensible due to its more specialized functionality, nor practical since many fine-tuned models will be deployed in resource-constrained environments. To address these pain points, we propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed $\textbf{D}$ually $\textbf{S}$parsity-$\textbf{E}$mbedded $\textbf{E}$fficient Tuning (DSEE), aims to achieve two key objectives:  (i) $\textit{parameter efficient fine-tuning}$ -  by enforcing  sparsity-aware weight updates on top of the pre-trained weights; and (ii) $\textit{resource-efficient inference}$ - by encouraging a sparse weight structure towards the final fine-tuned model. We leverage sparsity in these two directions by exploiting both unstructured and structural sparse patterns in pre-trained language models via magnitude-based pruning and $\ell_1$ sparse regularization. Extensive experiments and in-depth investigations, with diverse network backbones (i.e., BERT, GPT-2, and DeBERTa) on dozens of datasets, consistently demonstrate highly impressive parameter-/training-/inference-efficiency, while maintaining competitive downstream transfer performance. For instance, our DSEE-BERT obtains about $35\%$ inference FLOPs savings with $<0.1\%$ trainable parameters and comparable performance to conventional fine-tuning.

Under review as a conference paper at ICLR 2022
DSEE: DUALLY SPARSITY-EMBEDEED EFFICIENT TUNING OF PRE-TRAINED LANGUAGE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Gigantic pre-trained models have become central to natural language processing (NLP), serving the starting point for fine-tuning towards range of downstream tasks. However, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g.. 175B parameters for GPT-3), even the fine-tuning pro- cess can be time-consuming and computationally expensive; (b) the fine-tuned model has the same size as its starting point by default, which is neither sen- sible due to its more specialized functionality, nor practical since many fine- tuned models will deployed in resource-constrainee environments. To address these pain points, we propose framework for resource- and parameter-efficien fine-tuning by leveraging the sparsity prior both weight updates and the final model weights. Our proposed framework, dubbed Dually Bearsity-Embedeee Efficient Tuning (DSEE). aims to achieve two key objectives: (i) parameter ef- ficient fine-tuning- by enforcing sparsity-aware weight updates top of the pre- trained weights; and (ii) resource-efficient inference by encouraging a sparse weight structure towards the final fine-tuned model. We leverage sparsity in these two directions by exploiting both unstructured and structural sparse patterns in pre-trained language models via magnitude-based pruning and l sparse regu- larization. Extensive experiments and in-depth investigations, with diverse net- work backbones (i.e., BERT, GPT-2. and DeBERTa) on dozens of datasets, con- sistently demonstrate highly impressive while maintaining competitive downstream transfer performance. For instance, our DSEE-BERY obtains about 35% inference FLOPs savings with < 1% train- able parameters and comparable performance to conventional fine-tuning,
INTRODUCTION
Most recent NLP applications have been following the pre-train then fine-tune paradigm, where start from a gigantic pre-trained model and fine-tune it towards downstream tasks. Conventional fine-tuning works by updating all of the parameters of the pre-trained model. As the size of pre- trained models grows, updating all parameters becomes less feasible in most practical scenarios, due to the expensive memory and computational requirements. For example, BERT BASE. (Devlin let al. 2019) has 1 10M trainable parameters, while GPT-2 (Radford er al.) 2019) has up to 1.5B and the largest version of GPT-3 (Radford al. 2019) has astonishing 175B trainable parameters. As such, conventional fine-tuning of the larger models could require hundreds of GPU hours. Another downside of this paradigm is that requires storing as many parameters, as in the large-scale pre- trained models, for each downstream task. That poses impediments to the deployment in real-world essuurceconstrraiee environments.
One solution to address the extensive resource requirement of conventional fine-tuning is model pruning (LeCun et al., 1990, [an et al. 2015a Ren et al., 2018 He et al. 2017 Liu et al. 2017). where unnecessary weights are eliminated to shrink the model size. For example, Chen et al. (002b) leverages & regularization to remove insignificant attention heads and gains 35% ~ 45% training time with comparable performance. Guo et al. (020)) learns sparse task-specific "diff" vectors for various downstream fine-tuninn fine-tuning tasks, leading to great memory savings. All these stud- ies indicate the rise of sparsity naturally during fine-tuning general-purpont pre-trained model,
All codes are provided in the supplement.

---

Under review as a conference paper at ICLR 2022
to some specialized downstream functionality. One potential interpretation, of why sparsity arises, is that different subsets of the parameters may be responsible for different downstream tasks and data domains (Sanh et al.) 2020). However, identifying appropriate sparse pruning masks requires burdensome training of models with full weights, which can still be unaffordable for many practi- tioners. For example, fine-tuning large pre-trained language model like GPT-3 for just one step consumes at least 1. 1.2TB of VRAM and requires 96 pieces of NVIDIA Tesla (Hu et al. 2021).
One parallel alternative is designing parameter-effiient fine-tuning algorithms, which aims only optimize a small portion of weights while fixing most of the pre-trained weights during the down- stream task training step. Pioneering works along this line, which utilize adapters (Houlsby et al. 2019) or low-rank decomposition (Hu et al.) 2021), can significantly reduce the number of trainable parameters while preserving good fine-tuning performance. Introducing only a small number of task-specific parameters for each new downstream task can substantially improve the memory and deployment efficiency of models as it allows us to reuse the remaining unchanged/shared parame- ters. However, there are two major hurdles of current parameter--fffiiiet fine-tuning: (i) it does not yield any inference efficiency gains since the full pre-trained weights are still required to calculate outputs; and (ti) current methods assume the weight update to be either sparse (Guo et al., 2020) or low-rank (Hu et al. 202I), yet those assumptions might be oversimplified and overly restricted to allow for effective updates. For example, the low-rank assumptions on weight matrices might be
overly strong since some weights cannot be fitted in the low-rank space (Yu et al. 2017). Recently Chen et al. 2021a also find that using both sparse and low-rank components performs better than either of them. These observations have inspired us to explore better parameter-efficieng methods.
To tackle both resource- and wrrameter-efficinng issues of large model fine-tuning, we explicitly draw the prior of sparsity for both weight updates and the final weights, and establish a du- ally sparsity-emeedding efficient tuming (DSEE) framework. From a pre-trained model, DSEE first adopts paasity-aaarre low-rank weight update to achieve parameter efficiency of the fine-tuning process; and then enforces a sparse weight structure by masking to achieve resource efficiency of the fine-tuned model at inference time. Our contributions can be summarized as follows:
We propose the dually parsity-embedding efficient tuning (DSEE), which unifies sparsity- aware weight update and sparse pretrained weight in fine-tuning gigantic pre-trained mod- els. It is the first attempt towards jointly optimizing both parameter efficiency of the fine- tuning process, and the resource efficiency of the fine-tuned model.
We exploit both unstructured and structured sparse patterns the DSEE framework. For weight updates, we find the injected sparsity prior to greatly enhance existing parameter- efficient update schemes, and to further trim down the needed amount of trainable param- eters. For the final weights, we learn well-performed sparse masks from the pre-trained weights, leading to substantial computation and memory reductions at inference.
Extensive experiments demonstrate the effectiveness of our proposal across various rep- resentative pre-trained languages models, such as BERT, GPT-2, and DeBERTa; and on diverse evaluation benchmarks, such as E2E. DART, WebNLG, and GLUE Specifically, on (E2R, Dart, WebNL.G) .G), our methods can achieve BLUE score of (69.75, 55.40, 46.66) with less than 1% of total trainable parameters. On BERT, our method can save about 35% FLOPs, compared to traditional downstream fine-tuning.
RELATED WORK
Pre-trained language models. Transformer (Vaswani et al. 2017) is a sequence-to-sequeeec model that heavily uses the concept of self-attention. Later on, numerous transformer-based models have been proposed and show overwhelming performance on natural language processing (NLP) and on computer vision (CV) tasks. Devlin et al. (2019) designed BERT that pre-train deep bidi- rectional representations from unlabeled text and reached powerful performance. [iu et al. (2019) found that BERT were terribly undertrained and proposed RoBERTa, an enhanced training recipe for BERT which can greatly boost the performance. He et al. (202)) proposed decoding-ennie -ecodiin-ennhanedd BERT with disentangled attention (DeBERTa) that incorporates the disentangled attention mecha- nism and an improved mask encoder to enhance BERT and RoBERTa. More variants like Xet, Albert, and Electra have also been proposed in recent years (Yang et al. 2019 Lan et al. 2019 Clark (et al., 2019). The series of GPT models (Radford et al.) 2019: Brown et al. 2020) are later devel- oped based on transformers decoder blocks rather than encoder blocks like BERT, which again have
2

---

Under review as a conference paper at ICLR 2022
shown superior performance different tasks. These large models pretrained large amount of unlabelled texts would need to be further fine-tuned on downstream tasks for better performance. One of the accompanying disadvantages of these pre-training models with tremendous parameter counts (e.g. 175B in GPT-3) is the unaffordable computational cost for further fine-tuning.
Pruning and Low-rank decomposition. Pruning is a widely-used model compression technique. It can reduce the number of parameters inside models, which possibly brings training and infer- ence efficiency. Along with weight pruning method (Han et al., 2015b) being one of the most effective methods (Gordon et al., 2020), various criterion have been proposed to select insignifi- cant weights for pruning, such as Taylor approximation (Molchanov et al. 2019). Hessian score approximation (Hassibi & Stork 1993), and other saliency scores such as SNIP (Lee et al. 2018), GraSP (Wang et al.. 2019) and SynFlow (Tanaka et al., 2020). Several pruning methods have been commonly adapted to compress language models (McCarley al., 2019; Gordon et al. 2020 [annh et al. 2020; Wang et al. 2020 [Chen et al. (0211b) Specifically, McCarley et al. (2019) proposed to prune attention heads that had less contribution to the model. [Wang et al. (2020) pruned BERT models by involving low-rank factorization and &o regularization. [Sanh et al. (2220) invented an improved version of magnitude pruning (i.e., pruning based on the weight change) that can better suit the transfer learning. Chen et al. (2021b) performed structured pruning BERT via l sparse regularization, which reduced a large portion of parameters and decreased the training cost.
Low-rank approximation Yee| 2005) is also vastly studied. One classical scenario is robust principle component analysis (Candès et al.) 2011). which decomposes a matrix into a low-rank plus a sparse component. Existing literature shows that in deep learning, the learned over- veer-parameterieed models often naturally bear approximate low-rank weight structures (Oymak et al. 2019;; Yu et al.) 2017)
Some (Jaderberg et al. 2014 Povey et al., 2018 Sainath al.) 2013: Zhang et al. [014]] Zhao
et al. al.
2016) have explicitly imposed the low-rank constraint during training. Wang et al. (2020); Hu
shrink the model size and trim down the trainable parameters during fine-tuning. However, to our best knowledge, integrating sparsity and low-rank
(202T) utilized low-rank decomposition
structures has never been studied before for efficient fine-tuning of pre-trained language models.
Parameter-efficient adaptation. Parameter-efffcient adaptation aims at reducing the number of trainable parameters when fine-tuning the models across different downstream domains. Unlike pruning. it generates updates that can be represented by fewer parameters instead of building sparse models. Various approaches are invented to achieve the goal. Rebuffi al. (2017); HHuulby et al. (019)) inserted and only trained adapters between existing layers, whose parameters are much less compared to the pretrained models. [uo et al. (2020) leveraged to regularization to limit the number of non-zero elements in the update vectors. Lester et al. (2021); Li & Liang (2021) introduced effi- cient prompt tuning which optimizes only small continuous task-specific vector, Hu et al. (202T) proposed a low-rank decomposition based method that can also significantly reduce the number of trainable parameters. However, fine-tuned models yielded by these methods work have the same amount of weights as the pre-trained starting point; hence they contribute no resource efficiency of the final model.
3 METHODOLOGY
In this section, we begin by describing our notations and definitions of sparsity generation and parameter-efficient fine-tuning in Section 3.1 Then, we introduce the (dually) sparsity-
efficient fine-tuning algorithms in Sections 3.2 and 3.3
3.1 PRELIMINARIES
Sparsity generation and resource-efficient fine-tuning. We adopt both unstructured and struc- tured pruning methods to produce sparsity. They can lead to resource--fffiienny including memory and computation savings.
Let W Rmxn denote weight matrix. The goal of pruning is to find binary mask S {0, 13llWll,, where ||Wllo is the number of parameters in W. The mask S is applied to W and results in a sparse weight W o S. For unstructured pruning, only memory cost is saved; but for structured pruning, it helps save computational cost since the sparse weights can be smaller in size by wiping out all-zero columns or rows. However, the performance of networks after structured pruning is often shown to be inferior compared with the unstructured pruning counterpart.
3

---

Under review as a conference paper at ICLR 2022
Sparse-Embedded Pre-trained Weights
Input Embeddings
Point-wise Product
Pre-re rained
ghis
Point-wise Addition
Trainable Weights
Pruned Weights
Sparse Matrix
W E I m (Fixed)
S E {0, 1) MX
Sparse-Embeddel Decomposition
(Trainable)
Output
V R T
4 E ReXT
S₂ E RR
Figure 1: The overview of proposals. The sparse masks have unstructured structured patterns, which leads to training and inference efficiency. During the fine-tuning, we only train decomposed matrices '4, and non-zero clements in S2.
Parameter-efficient fine-tuning. To leverage the knowledge in pre- pre-trained weights W, down- stream models learn task-specific weight update AW via fine-tuning and generate predictions with weights W + AW. The output of models is therefore calculated as (W AW)r where is the input. Since AW has the same size W, learning the update matrices usually requires massive resources as the size of the pre-trained model increases. Parameter-efficient fine-tuning try solve this problem by using as few trainable parameters as possible to represent AW, while maintaining competitive downstream fine-tuning performance. Previous literature reaches the goal via either sparsifying weight update matrices AW (Guo et al.) (020) or leveraging low-rank decomposed ma- trices compute AW (Hu et al. 2021 while our work we combine both of them.
Algorithm 1: Searsity-Embedee Low-
Algorithm 2: DSEE
Rank Decomposition
Input: Pretrained weights W, number of
Input: Pretrained weights W. number of
non-zero elements N, desired sparsity
non-zero elements N
8, loss function
Output: Sparse matrices S₂
Output: Sparse mask S matrices U, V,
Initialize S2 to be an empty set.
Decompose W into V1, and Sq
for each self-attention projection weights W; in W do
(Re-)nnitializatio::
= 0, V (0, 0.02),
/* Decomposi i
indexes of non-zero elements in S
Perform matrix decomposition: W; UV + S' by solving the optimization problem in E
/* I: train before pruni ing
//
Train U1 V with respect to L under the constraint of (S) 0.
Identify important
II: pruning the model
elements to form S Perform thresholding on S': Keep N elements in S' with top magnitudes, and set the rest 0. Append S' into Sz. end
Prune (1-5% parameters in W globally by sorting the magnitude of W +WV + deriving the sparsity mask S
/* 111: tuning after pruning Tune V, for E epochs for recovering the performance.
3.2 SPARSITY-EMBEDDED PARAMETER EFFERCEET FINE-TUNING
recent study (Hu et al. 202 enforces low-rank constraint to weight update tensors AW, and obtains satisfactory trade-off between parameter-efficienng and model quality. However, vealed experimentally by Yu et al. (017). a part of the important information in the trained weights will also scatter outside the low-rank subspace, creating sparse "residuals". Inspired by these ob- servations, we investigate a new sparsity-aware low-rank subspace AW, and introduce the first parameter-efficeett fine-tuning.
component of proposal in Figure i.e.,

---

Under review as a conference paper at ICLR 2022
Specifically, we identify sparse subspace Q2 that can project our update matrices AW to. The update matrix is then decomposed into two components: a low-rank part which is represented by the multiplication of two low-rank matrices U E R and V E R"x", and a sparse residual
Si.j+ (i,))
S2 = Pn(AW), where Pa(AW)
0, (i.j) so i= 1,2. m. j = 1.2.....
As illustrated in Figure I the update matrix AW can be approximated by UV+ S2, in which the U. V, and the non-zero element of S2 are learnable parameters while S2 is fixed once determined. Compared to the full fine-tuning schemes has m x n individual trainable parameters, our method only has (m+n)) card If is smaller than mxn-card(S) / 0.5 mi,, nt), which is a loose m++1
bound, our method is capable of substantially reducing trainable parameters for downstream fine- tuning. In practice, the value of is very small compared to m and so the savings considerable.
The next question is how to find appropriate Q. Motivated by (Candès et al. 201T), we formulate our sparsity- aware decomposition as the following optimization problem:
min I 10-- U.V.S² 2
S.L. (2) sr, rank(V) sr, card(S2) S c,
where rank(-) indicates the rank and card(-) indicates the cardinality of a matrix. Here we derive Ω from pre-trained weights W, which is different from previous low-rank techniques for model compression since we perform prior-training decomposition and can not access AW before fine- tuning. In this way; we actually assume that AW shares similar crucial subspace Ω with W (Sun et al. (018)). We use the GreBsmo algorithm (Zhou & Tao (2033) to solve this optimization problem. This method adopts similar idea as random projection and can solve the optimization problem rapidly. The details of the algorithm are in Appendix.
Algorithm I summarizes the detailed procedure of our proposal. We first decompose the pretrained weights W into three matrices: U, V and Sg. 24 and V are matrices of low-rank and Sy is a sparse matrix. After the decomposition, we re-initialize 2 and to be of shape RR and R The initial values for VI are set to O, and the initial values for V will follow normal distribution as Hu jet al. (202T) did. We do not use the decomposed value inside Sz, but only the index of non-zero elements. We collect the indexes of non-zero elements into Ω, and reset the values of S2 back to O. During the training, we only update the elements of S2 whose indexes are in Q as well as lI and V.
3.3 DUALLY SPARSITY-EMBEDDEE EFFICIENT TUNING (DSEE)
Besides pfrfmeter-efffcienncy. resource-efficiency is another major goal of our proposed framework. Sparsity-embedded low-rank weight updates alone do not necessarily lead to computational cost reductions since it works together with the dense pre-trained weights and the massive computation during forwarding processes is inevitable. To achieve both resource- and parameter--fficienng also promote a sparse weight structure to the final fine-tuned model, as demonstrated by the sparse mask S in Figure 1 Our DSEE explores both unstructuured/srruutuuedd sparsity patterns as follows.
> Pruning with unstructured sparse masks. The unstructured sparse mask is the most widely used mask since it usually produces almost undamaged performance compared to its dense counter- part (Han et al. 201 It applies fine-graines manipulation on each individual model weight, while the generated irregular sparse masks bring limited hardware speedup (Wen et al. 2016). Specifically, based on the weight matrix W, sparse mask S₁ can be calculated with various approaches, either heuristic or optimization oppimization-based. We adopt the one-shot magnitude pruning (Han al.) 2015a) in DSEF, due to its simplicity and competitive performance.
In our context, we create the sparse pattern S from ++ First, it sorts the magnitude (i.e.,
absolute value) of individual weights and removes parameters with bottom 1 - 1% magnitude. Second, we further tune the value of the low-rank update matrices for a few epochs, which is stated in Han et al. (2015b). Third, the sparse mask Si
important to keep the performance that we derive will be applied on the pretrained weights only and do not affect the output from the update matrices, (W O S + uv + Sz)æ  = W O Siæ + (UV + S2)a, where the first term W @ Sj brings sig- nificant resource-efficiency. and the second term brings parameter-effiiiengg
W o S + UV. For an input sample x, the output is calculated by
5

---

Under review as a conference paper at ICLR 2022
Pruning with structured sparse masks. The second method exploits structured sparse masks, whose regular patterns are more hardware friendly yet have worse performance than unstructured masks. Our DSEE considers & sparse regularization (Liu et al.] 2017] Chen et al. 2 to craft high-quality structurally pruned subnetworks. More precisely, motivated by Chen et al. (2021b), we introduce learnable coefficients & before each attention head module, and append & sparse penalty on & to the original loss. Detailed formulations are depicted as below:
We add trainable coefficients e before attention heads. The parameters c are optimized together with the decomposed matrices, i.e., 24, V and S2. An extra term A||[|| will be added to the training loss for sparse regularization. The value of À is set to le-4. After training, we prune the attention heads that have the lowest contribution to the model (i.e., lowest c). We use layer-wise pruning scheme that prunes the same proportion of heads in each attention layer. After pruning, several epochs of tuning are run to recover the performance of the pruned model. The size of update matrices will change after structured pruning, so we also need to change the dimension of U and V. Specifically, we change the size of from I=** R [[xx)] where is the pruning ratio of the corresponding self aelf-attention layer and [z) is the biggest integer not greater than x. The size of Sg is shrunk accordingly.
4 EXPERIMENT RESULTS
Datasets and models. We use three classical pre-trained language models in experiments: BERT (Devlin et al. 2019). GPT-2 (Radford et al. 2019), and DeBERTa-large (He et al. 2020), which have 12/24/2 24 layers with hidden size of 768/ 1054/ 1024 and 110/354/380N train- able parameters, respectively. For BERT and DeBERTa, we evaluate our method on the GLUE benchmarks (Wang al.) 2018). For GPT-2, we use E2E (Novikova et al., 2017). WebNLG (Gar- dent et al. 2017). and DART (Nan et al. 2021) for evaluations.
Training and evaluation details. For BERT and DeBERTa, we follow the default settings in Wolf al. (2019); Devlin et al. (2019). We use the AdamW (Loshchilov & Hutter 2017) optimizer for downstream training. The batch size for BERT and DeBERTa is 32, 8 per GPU. We train three epochs search the sparse mask Sr, and continue tune the model for three epochs to converge. The initial learning rates are reported in AT and we linearly decay them. For GPT-2, we follow the same hyperparameters as in Hu al. (2021) We train the model for five epochs search for the mask, and further tune the pruned model for two epochs.
Evaluation Metries. For tasks on BERT and DeBERTa, we use the accuracy, Matthew's Corre- lation, and Pearson's 's in the evaluation by default, which is also the conventional setting in the NLP community. On GPT-2, we use BLEU (Papineni et al. (2002), METEOR (Denkowski & Lavie 2014). TER (Snover et al. 2006) and NIST (Doddington) (002) as evaluation metrics. To eval- uate the efficiency of models, we use the number of trainable parameters to measure the parameter efficiency, use Sparsity in Pretrained Weights to measure the resource efficiency, and FLOPss to eval- uate the computational efficiency. We add star sign (x) to indicate the structured sparsity (i.e., structurally prune the pretrained weights).
Baselines. On BERT and DeBERTa, conduct comparisons with the following baseline meth- ods: Fine-tune: directly fine-tunes the full model; @ EarlyBERT (Chen et al. 2021b): 0 BERT Tickets CCen et al. 2020); O OMP: prunes the fine-tuned weights by magnitude, and fine-tune afterwards; and LoRA (Hu al. 2021). We also report the Huggingface's fine-tuning results.
On GPT-2, conduct comprehensive comparisons with multiple baseline methods:
Adapters (Houlsby et al. 2019): insert adapters after linear layers; @ FT-Top2: fine-tune the top 2 layers only; Ø: Prefix: prefix tuning introduced by Li & Liang (202 and 2 LoRA: low-rank
decomposition, which assumes AW- = LVV; most results are directly cited from Hu et al. 2021
4.1 EFFICIENT TUNING WITH DSEE
Parameter-eflcienncy with sparse masks. To verify that using simple low-rank adap- tation (i.e., LoRA) has limitations, we compare its performance with the performance of our sparsity-embeddee
efficient fine-tuning. Table [ proves that on MNLI, SST-2, CoLA, and STS-B, the simple decomposition form shows unsatisfactory results compared to sparsity- decomposition. Adding a small proportion of parameters (only 3072 trainable parameters) bring performance boost to the model. Specifically, under

---

Under review as a conference paper at ICLR 2022
rank 8, the metrics increase (0.69/0. (0.69/.11//0000//// 13/00000000000003 on SST-2/NLI/COLASTSS respec- tively. Moreover, using only approximately half of the number of trainable parameters, our method can achieve comparable parameters compared to the sfate-of-the-art method LoRA. Table I: Performance comparison with BERTRASE on SST-2, MNLJ, CoI.A, and STS-B.
Table _ shows the performance on GPT-2. For all three tasks, our method achieve comparable results with only about half of the trainable parameters with LoRA with rank four and make substantial performance gain compared to LoRA with rank two. On WebNLG, our method even achieves a higher BLEU score, 55.56 versus 55.29, with half of the trainable parameters. Such a phenomenon indi-
# Traimblee Parameters
Dataset
SST-2
MNLI
CoLA
STS-B
Fine tune
109.9M
92.32
82.12
0.570
0.890
AW 2/V
589.8K
92.09
80.79
0.575
0.891
AW = WV
294.9K
92.09
80.49
0,580
0.892
AW 2VV &
298.0K
92.78
80.64
0.588
0.895
cates that adding sparse matrix to the low-rank decomposition can make substantial improvement.
Table 2:
comparison of different decomposition on GPT-2. The formulas of decomposition are reported.
Results of fine-tuning and the results of Trainable Parameters = 0.39M are cited from Hu et al. (2021
Datasct
Trainable Parameters
E2B
WeNNLG
DART
Metric
BLBU
MET
NIST
BLEU
MЕT
TER
BLEU
MET
TER
Fine-tune
354,92M
68.2
46.2
8.62
47,60
0.39
0.50
46.0
0.39
0.46
uv
0.39M
70.4
46.9
8.84
55,29
0,4143
48.23
0.392
AW W
0.20M
69.2
45.9
8.74
55,23
0.4134
0.3957
46.49
0.387
0.477
AW W + S.
0.20M
69.8
46.5
8.79
55.56
0.4132
0.3916
47.08
0.390
0.472
Resource- and parameter-efficennyy with sparse masks. We report the performance of DSEE, including both the number of trainable parameters and the sparsity in pretrained weights. We use an unstructured sparsity of 50%, and structured sparsity of 25% and 33%, which is equal to pruning 1/4 and 1/3 heads from each attention layer. For BASE and DeBERTa-large, we set the r (the low-rank dimension) to be 16, and N (the number of non-zero elements in S2 ) to be 64. We also prune each of the intermediate layers using a structured sparsity of 40% as in (Chen et al. 20216 For GPT-2, we set to be 2 in DSEE and 4 in LoRA. The choice of N remains 64.
Table 3 summarizes the performance on BERTBASE- On BERTBASE, our DSEE method can: achieve parameter--fficiency high level and retain the performance on various downstream tasks. On BERTBASE. with only about 600K trainable parameters (110/0 (110/0.5929 x 200x smaller than the full model), DSEE can achieve comparable performance on all GLUE tasks. @ DSEE can achieve resource-efficiency in the final models. Using either unstructured or structured sparse masks can reach our goal of reducing the number of parameters in the pretrained weights without sacrificing much performance. At the sparsity level of 50%, unstructured DSEE can achieve an accuracy of 90.46% on QNLI, surpassing the baseline of fine-tuning. whose result is 90.15%. At the sparsity level of 25%. the structured DSEE can have performance gains ranging from 0.46% to 2.22%. except for QQP and QNLI. At the sparsity level of 33%, the performance gains range from 0. % to 2.04%, except for QQP and QNLI. @ Compared to other baselines, e.g., EarlyBERT and BERT Tickets, our method benefits from parameter efficiency, which updates the weights more efficiently. These observations validate the effectiveness of our DSEE method.
We also calculate the inference FLOPs of BERT BASE on STS-B dataset. The original BERT on STS-B takes FLOPs of 3.7835 x 10 while LoRA takes FLOPs of 3.8096 x 10 which is 0.69% higher. Conversely, the structured version of DSEE takes a FLOPs of 2.4921 x 10 at the sparsity of 25%, and 2.3867 10 at the sparsity of 33%, which is 34.61% 1% and 37 37.38% lower than LoRA. This indicates that a large proportion of computational cost can be saved at inference phase.
Table 3: Performance comparison different methods on BERTBASE on GLUE benchmarks. The star sign (+) in the sparsity column indicates that it represents structured sparsity.
Methods
Trainable Parameters
Sparsity in Pretrained Weights
Datasel
CoLA
STS-B
MNLI
QQP
QNLI
MRPC
RTE
SST-2
Fine-tune
110M
0% 33%
$7.02
88.97
82.12
91.01
90.15
85.29
70.40
92.32
EaryyBRR
110M
41.00
79.97
89.44
89.86
80.39
61.01
90.94
BERT Tickets
110M
(905, tol, 265 JMSS, 703, 505, 10w.8005..00000000 009)
$4.5
88.4
82.4
90.2
89.1
85.2
66.2
92.1
OMP
~ -55M
50%
56.17
88.66
81.97
90.72
89.80
82.84
70.04
92.18
LoRA
589.8K
0%
58.58
89.10
80.79
86.43
88.16
86.27
71.48
92.09
DSEE
592.9K
50%
56.74
88.77
81.41
87.21
90.46
85.05
70.04
90.83
DSEE
592.9K
25% 33%
59.01
89,90
84.34
90.99
90.94
86.76
71.84
92,78
DSEE
592.9K
57.79
89.93
84.16
90.78
91.09
87.25
71.48
92.43

---

Under review as a conference paper at ICLR 2022
Table 4 summarizes the performance on GPT-2, which shares similar trend. Unstructured DSEE can achieve 2000 reduction in trainable parameters and 2x reduction in the final fine-tuned model size with almost no loss in downstream task performance compared to conventional finetuning. When compared to LoRA, the unstructured DSEE can retain the same level of the number of train- able parameters and downstream task performance, while showing 2x reduction in the final fine- tuned model size. The structured DSEE on GPT-2 seems to be less competitive than on BERT, but it can still hold performance on E2E and WebNLG after pruning 25% of heads in attention modules.
Table 4: Performance comparison of different methods on GPT-2 on E2E, WebNLG and DART. LORA':
reproduced results. The star sign (*) in the spansity column indicates that represents structured sparsity.
Methods
Parameters
*Trainable
Sparsity Pretriined Weights
E2E
WehNLG
DART
Fine-tura
354.22M
0%
68.2
0.462
8.62
47.6
0.39
0.50
46.0
0.39
0.46
Adapiers
11.48M
0%
45.4
FT-Top2
25.
0%
68.1
0.460
8.59
33.5
0.26
0.75
38.1
0.34
0.56
Profix
0.35M
0%
69.7
0.461
8.81
54.4
0.41
0.41
45.7
0.38
0.46
LoRA
0.39M
0%
70.4
0.468
8.85
55.3
0.41
0.39
47.5
0.39
0,45
LaRA"
0.3
0%
70.05
0.467
8.84
55.29
0.4145
0.3938
48.23
0.392
0.469
DSRE
0.20M
3056
69:33
0.465
8.73
55.78
0.4165
0.3921
47.21
0.390
0.471
DSEE
0.20M
5056
09:25
0.469
8.78
55.40
0.4124
0.3935
46.66
0.389
0.471
DSEE
0.20M
25%5"
69.48
0.464
8.75
54,64
0.4105
0.4030
26.94
0.247
0.727
Finally, we validate our method on De-
Table 5: Performance comparison of different methods on DeBERTa-large on CoLA, MNLI, MRPC and RTE.
BERTa. The results are displayed in Ta- ble 5 We use four datasets, CoLA, MNLI, MRPC, and RTE. They have greatly varied sizes, which are representatives of the GLUE benchmark. DeBERTa is larger model, so applying low-rank decomposition with our hyperparameters cannot match the performance of fine-tuning. Compared to LoRA, reaches higher performance on downstream tasks, albeit it needs slightly more training epochs. However, such slight extra cost for searching the sparse mask S₁ can be amortized by the effi- ciency gained in the future inference, since the burden on resources such as storing the pretrained weights is relieved.
Methods
T'rainable
Sparsity Protmined Weights
CalAA
MNI.1
Dataset
MRPC
RTF
LoRA DSFF DSEE DSBE
7B6.4K
0%
75.00
30%
67.35
90:58
90.20
77.08
789 SK
50%
63.82
90.03
$9.71
74.73
789.5K
26%
03.62
39.93
89.96
75.24
method
4.2 UNDERSTANDING DSEE
The position of embedded sparsity masks plays crucial role in proposed DSEE. For in- stance, applying sparse masks to the pre-trained weights or weight updates produces resource- and parameter-efficiency. Precisely, we com- pare four different methods on BASE: one-shot magnitude weight pruning (W @ S1). two DSEE's variants (W 3
Table 6: Performance comparison on BERTBAS BAsE with different masks embedded.
Mushate
Triiealee Parseeeess
Sporsty Promined Weight
MNLI
Daseett
CaLA
STS-8
Pire-ue
116M
92.32
82.12
52.02
$8.97
WS &
55M
99%
91.38
81.92
0.567
0.677
9929K
91.97
$9.56
0.500
0.355
WES MV
9929K
50%
90.53
$1.41
0.567
0.558
81
+
((V)
and
W + uVV + S2), DSEE (W & S + 2V S2). The results are collected in Table 6 We can see that: 1 NO embedded sparsity in the pretrained weights yields the overall best performance. This is intuitive since the valuable knowledge learned from massive pretraining data intact and not removed. @ Embedding sparsity into the pretrained weights harms only little to no performance. A similar conclusion is drawn by Chen et al. (2020) which validated that a pruned model can have similar performance the dense model. 4 Using sparsity embed efficient fine-tuning with the sparse pre-trained weights can also preserve performance, as well as achieve parameter efficiency.
4.3 ABLATION AND VISUALIZATION
Different methods to generate sparse matrix S2 In this section, we compare the performance of BERT BASE using different methods to generate the sparse matrix S2 and the corresponding space Q. Except for the aforementioned matrix decomposition method, we try to (1) randomly sample indexes into Q2; and (2) directly select the indexes of elements of highest magnitude of W into S2. In Figure 2 see that using the matrix decomposition method has the highest metric overall. It is also noteworthy that sparse matrices generated by other methods will sometimes harm the performance.
Another factor is the number of non-zero elements inside Sz. More non-zero elements in S₂ reduce the parameter efficiency, while less non-zero elements increase the efficiency but may not be much beneficial to the accuracy or other metrics. Figure also shows the relationship between number
8

---

Under review as a conference paper at ICLR 2022
of non-zero elements in S2 and the performance fine-tuned model on SST-2. From the graph, we can see that using an of 64 seems to have the most stable results compared to other choices. can derive from the figure is that higher number of
Another important piece of information non-zero elements in the S₂ does not guarantee better performance.
Ablation of # ranks. The rank of low- rank decomposition r is crucial to the transfer learning performance. A small r will result in lower representation abil- ity, and large r will bring more parame- ters and reduce the parameter efficiency. To find the best value, we conduct ex- periments with different ranks r four datasets, SST-2, MNLI, CoLA, and STS- B. The results are displayed in Figure [ We add quadratic trend lines in the graph along with the discrete points to smooth the results. We can draw two conclusions from the graphs: ! Overall, the final performance is pos- itively correlated with the number of trainable parameters; however, on some tasks (MNLI, CoLA) higher number of parameters (namely, the r for U and V) will lead to lower performance. @ With sparse matrix embedded, the performance of the trained models can be improved within range of trainable parameters. On SST-2 and CoLA, the applicable range seems to be 104.5 ~ 10%. On STS-B, our method can consistently outperform the performance of using low-rank decomposition only. For MNLI, the two methods behave similarly while our method is slightly better.
92.5
91.0
Trammbie Parandeer))
logglaak)
Figure 2: Left: Different method for generating S2 in BERT Right: Different number of non-zero elements in S2. Empty: no non-zero element in Sg. Decompose: matrix decomposition method aforementioned. Magnitude: picking elements with highest magnitude. Random: random matrix.
MNLL NeM
82,00
92.30
$1.0
12.00
scuo
cuss
nuss
X
0325
0.50
91.00
MyurrTrieaa Panmsted
AggefTrraatt Parncun)
Performance comparison of two decomposition methods under different ranks. We add quadratic trend lines for better visualization quality.
Figure 3:
Ablation of sparsity. Although creating sparsity in pre- trained weights W does not change the number of trainable parameters of models, the different levels of sparsity control the number of non-zero elements and thereby influence the representation ability. Therefore, we conduct experiments on BERT with different sparsity, ranging from 10% 60%. The results are in Figure A5 From the figures, we can draw con- clusions: 1 DSEE out-performs magnitude pruning at low sparsity (< 50%) with respect to the performance down- stream tasks. @ DSEE surpasses vanilla magnitude pruning with respect to the number of param- eters. For each self-attention layer, using vanilla magnitude pruning needs train all the weights, while DSEE only needs to update the three low-rank matrices.
Love
Layoaa
Weghe
Figure 4: Weight change distributions.
Visualizations. Figure [ shows the distribution of weight change. From the graph, we see that most weights are located around O. Such phenomenon indicates that natural sparsity exists within the update matrices, motivating us explore sparse structures along with matrix decomposition.
CONCLUSION
This paper draws on the prior of sparsity and establishes the DSEE framework. It is the first attempt towards jointly optimizing both parameter efficiency of the fine-tuning process, and the resource ef- ficiency of the fine-tuned model. On state-of-the-art large-scale language models (e.g.. BERT, GPT, and DeBERTa) and across several of datasets, DSEE consistently demonstrates highly impressive parameter, training, and inference efficiency, in addition to preserving competitive downstream transfer performance. Our future work targets extending DSEE to the finetuning of large-scale com- puter vision and/or multi-modal pre-trained models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. arXiv preprint arXiv:2005.
2020.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-77, 2011.
Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Uni- fying sparse and low-rank attention approximation, 2021a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint MrXiv:2007.12223. 2020.
Xiaohan Chen, Yu Cheng, Shuohang Wang. Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021b.
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019.
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pp- 376-380, 2014.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-44 1186, 2019.
George Doddington. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pp. 138- 145, 2002.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Berer-Beltrachhni. The webnig challenge: Generating text from rdf data. In Proceedings of the 10th International Conference Natural Language Generation, pp. 124-133, 2017.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effects of weight pruning transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 143-155, 2020.
Demi Guo. Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff prun- ing. arXiv preprint arXiv:2012.07463, 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:1510.0014 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 2015b.
Babak Hassibi and David G Stork. Second order derivatives for network pruming: Optimal brain surgeon. Morgan Kaufmann, 1993.
Pengcheng He, Xiaodong I Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0366 03654, 2020.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp- 1389-1397, 2017.
10

---

Under review as a conference paper at ICLR 2022
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 100.0066...... 2021.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for sulf-super learning of language representations. arXiv preprint arXiv:1909.11990 2019.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural information processing systems, pp. 598-59888 0886605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2018.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0869/ 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019.
approach. arXiv preprint
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 171105 2017.
JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of a bert-based question answering model. arXiv preprint 1eXiv:1900.06380 2019.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264-11272 2019.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Dart: Open-domain structured data record to text generation, 2021.
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. The e2e dataset: New challenges for end-to- end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, 2017.
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05333 35322, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.

---

Under review as a conference paper at ICLR 2022
Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and San- jeev Khudanpur. Semi-orthogonal low-rank matrix factorization for deep neural networks. In Interspeech, pp. 3743-37477 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 506-516,220 2017.
Ao Ren, Tianyun Zhang, Shaokai Ye, Jiayu Li. Wenyao Xu, Xuchai Qian, Xue Lin, and Yanzhi Wang. Admm-nn: An -erithhmmaaddwrr co-design framework of dnns using alternating direc- tion method of multipliers, 2018.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high sigh-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655- 6659. IEEE, 2013.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine- tuning. In NeurlPS, 2020.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pp. 223-231, 2006.
Mengying Sun, Inci M Baytas, Liang Zhan, Zhangyang Wang, and Jiayu Zhou. Subspace network: deep multi-task censored regression for modeling neurodegenerative diseases. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp- 2259-2268, 2018
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Pro- cessing Systems, 33, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2018.
Chaoqi Wang, Guodong Zhang, and Roger Grosse.
Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In
Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6151-6162, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems (NeurIPS), pp- 2074-2082, 2016.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint 19X1v:1910.03771, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
12

---

Under review as a conference paper at ICLR 2022
Jieping Ye. Generalized low rank approximations of matrices. Machine Learning, 61(1-3):167-191, 2005.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7370-7379, 2017.
Yu Zhang, Ekapol Chuangsuwanich, and James Glass. Extracting deep neural network bottleneck features using low-rank matrix factorization. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 185-189. IEEE, 2014.
Yong Zhao, Jinyu Li, and Yifan Gong. Low-rank plus diagonal adaptation for deep neural networks. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5005-5009. IEEE, 2016.
Tianyi Zhou and Dacheng Tao. Greedy bilateral sketch, completion & smoothing. In Artificial Intelligence and Statistics, pp. 650-658. PMLR, 2013.
13

---

Under review as a conference paper at ICLR 2022
All sections of newly added results and discussions highlighted.
A1 MORE IMPLEMENTATION DETAILS
We report the setting of other hyperparameters experiments. such as learning rate, in Table The device we used for experiments are various, including NVIDIA GeForce GTX 1080 Ti, GeForce RTX 2080 Titan RTX, and A6000.
Architecture
Method
Parametens
Dalaset
MNLI
QNLI
QOP
SST-2
CoLA
MRPC
RTE
STS-B
BERTHASE
Fine-tune
Learning Rate
5e-5
BERTHASE
DSEE (before proning)
Learnnng Rate
5e-5
5o-5
50-5
20-4
1o-3
8o-4
1o-3
8c-4
BASE
(after pruning)
Leaminn
5e-5
56-5
5e-5
1e-3
5e-4
DeBr-Rh--lare
LoRA & DSEE (before pruning)
Leaming Rate
le-5
le-3
8e-4
Re-5
pruming)
Learnnng
Table A7: Haper-parameters used different datasets.
GreBsmo Algorithm GreBsmo (Zhou & Tao 2013) an algorithm for solving the Robust PCA- like methods. The optimization of U, V, and S follows the following iterative rules:
Uj Q, QRQRR ((X - -k-1) -QR
V2 ="= (X Sk-1)
S Sx (X U&Va)
where X is the original dense matrix, QR(-) means the QR decomposition, Sx(-) indicates the soft-threshold operation, and the subscripts indicates the optimization step.
A2 MORE EXPERIMENT RESULTS
Ablation of Sparsity To study the relationship between sparsity of unstructured pruning and the behavior of our DSEE, conduct ablation study on various datasets in GLUE benchmarks. The results are shown in Figure A3
CatA
0.29
Hebick
0.32
000
63
Spart
0.
0.90
S15-8
MIRPC
0.500
View 141
Vollnch
Vigeisi honle
Momole
37
Spurily
bigefiiee Pocoeater)
Spardyy
1087
Coa
Figure A5: DSEE performance compared to vanilla magnitude pruning at different sparsity. Magnitude Pruning: vanilla magnitude pruning which tunes W directly.
Results with runs. We conduct experiments with three The mean and standard deviation of accuracy of LoRA and DSEE (without pruning) are shown in Table A8 We have also included the p-value of t-tests show the significance of the performance gaps. On half of the
A14

---

Under review as a conference paper at ICLR 2022
datasets, the p-value is around 0.1; and three datasets the p-value is between 0.2 and 0.4. The reason why p-value is large on some datasets is probably because the number of experiments still has room to increase.
Table A8: Performance comparison of different methods on BARE on GLUE benchmarks. The first row
of each method indicates the accuracy, and the second row indicates the standard deviation.
Trainable Parameters
Sparsity
Dataset
Methods
Pretrained Weights
CoL.A
STS-B
MNLI
QQP
QNLI
MRPC
RTF
SST-2
57.38
88.81
80.88
86.45
88.49
86.03
71.00
92.17
LoRA
589.8K
0%
1.43
0.23
0.41
0.14
0.10
1.49
1.10
0.27
58.73
89.16
81.03
86.53
88.58
85.78
71.12
92.55
DSEE
592.9K
0%
0.95
0.03
0.62
0.06
0.17
1.07
0.62
0.12
P-value of matched pair t-lests
0.086
0.054
0.383
0.121
0.276
0.572
0.277
0.031
55.60
88.39
81.16
87.23
88.88
84.88
70.09
91.00
DSEE
592.9K
50%
0.81
0.21
0.23
0.30
Inference time on BERT BASE We have recorded the inference time of BERT BASE on various GLUE benchmarks, QQP, STS-B, CoLA and RTE. The number of samples their evaluation set range from 277 to 40430, so they can be considered representative. The results are shown in Ta- ble From the table we can see that, the inference time are greatly saved after using the structured version of DSEE.
Inference time different methods on BERTBASE on GLUE benchmarks. The sparsity with star Table A9:
signs indicates that it is structured sparsity.
Methods DSEE (25%*)
Dataset eval samples
LoRA
DSEE (33%*
QQP
40,430
50.773
36.391
34.946
STS-B
1,500
2.033
1.506
1.337
CoLA 1,043
1.316
0.931
0.877
RTE
277
0.464
0.420
0.376
Convergence Speed We have compared the convergence speed (i.e., how the test accuracy changes) of LoRA and DSEE. From Figure A6 can that the convergence speeds of the two methods are not significantly different. This means introducing a sparse component into the decomposition form does not affect the convergence dynamics.
Test Accuracy
0.80
Method
DSEE
0.75
LoRA
100
Evaluation Step
Figure A6: Convergence speed of two methods (LoRA and DSEE). The x-axis represents evaluation steps and y-axis represents the test accuracy.
A15


Reviews:
[{"review_id": "cgg7cEMhbFZ", "reviewer": "Reviewer_KvuY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths and weaknesses are detailed below:\n\n*Strengths:*\n\n1. Combining weight-pruning and parameter-efficient tuning is a valuable attempt to further improve resource efficiency especially at inference time\n\n2. Experiments demonstrate the effectiveness of the proposed method. The analysis sections regarding DSEE are also appreciated.\n\n*Weaknesses:*\n\n1. (This is a minor sanity check question) On GLUE tasks, are the results from one run or several random runs? This detail seems not to be mentioned in the paper – it is common practice to reports results from several random runs since the GLUE results are known to be sensitive to initialization \n\n2. The necessity of adding a sparse matrix (i.e. S2 in the paper) into LoRA is not well supported. The justification experimental results in Table 1 and Table 2 are not very convincing to me because (1) the improvement from “+S2” is very small in most cases (and sometimes worse), for example, I do not think 0.008/0.003 are meaningful differences to claim in the paper; without significance test, most of the small differences may not be trustable; (2) In Table 1 and Table 2, many results from UV of 2x sizes (i.e. 589.8K in Table 1 and 0.39M from Table 2) are better than UV + S2. I know that this is not a fair comparison because of different models sizes, however, I suggest the authors double the size of UV + S2 for a fair comparison instead of downsizing UV – on the scale of 589.8K or 0.39M parameters, the number of parameters is not really a practical concern anymore. For example, your method can tune 1M less parameters than others, but such a 1M parameter saving may not lead to any meaningful differences in most practical settings.  \nTherefore, the sparse S2 adds complexity to the method and requires additional algorithm 1 in the pipeline, yet it does not yield very effective improvement as argued above, thus I do not think this is a proper design.\n\n3. Without considering the questionable contributions of S2 because of point 2, I think the remaining technical contributions of this paper are limited because it is a combination of the existing parameter-efficient tuning method (LoRA) and existing weight pruning methods. Such a combination is empirically valuable but technically trivial.\n\n4. The improvement of DSEE over LoRA seems from sparsity in pretrained weights, it is better to explain this phenomenon in the paper why weight pruning leads to a significant performance boost.\n\n5. There is a sentence above Section 4.2 “DeBERTa is a larger model so applying low-rank decomposition with our hyperparameters cannot reach the performance of fine-tuning.” This statement is misleading and contradicts the claims by other papers [1] [2], which conclude that a larger model typically requires a lower-dimensional parameter update space.\n\n6. The are some minor presentation flaws:  \n(1) In the first paragraph of the introduction section, citations to related papers should be included such as when mentioning BERT and GPT2.   \n(2) Above section 4.2, the authors say that Table 5 uses SST-2 …. which are not present in Table 5\n\n\n[1] Lester et al. The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021  \n[2] Aghajanyan et al. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021\n", "summary_of_the_paper": "This paper proposes to enforce weight sparsity constraints into a prior parameter-efficient tuning method (LoRA) to achieve both parameter-efficient tuning and resource-efficient inference. Specifically,  this paper (1) adds another sparse, learnable weight matrix to the original low-rank decomposition in LoRA; (2) incorporates unstructured and structured weight pruning techniques to save computation at inference time. Experiments are conducted on GLUE and several data2text generation benchmarks to demonstrate the effectiveness of the proposed method.\n", "summary_of_the_review": "Some core method design is not well-justified and the technical contributions are limited. There are some valuable empirical contributions though. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VhieWNZcoI4", "reviewer": "Reviewer_uLTd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The topic of this paper is timely. Models are growing to immense sizes (at faster and faster rates), and the community is starting to come to a reckoning with their computational (and storage) budget. Reducing the impact both from a memory footprint, ability to fine-tune, and perform efficient inference can have immense impact.\n- As an empirical result, some of the findings in this paper points towards the proposed method being worthwhile to try in practice.\n\nWeaknesses:\n- In general, the motivation and exact contributions of this work are hard to follow. In fact, I found it hard to understand what exactly the proposed method (and inference mechanism) is without first reading LoRA, the paper on which this work is based. For example, how outputs are even calculated are not explicitly spelled out (via equations) until Section 3.3. Personally, I also find the notation of $\\mathcal{W} + \\Delta\\mathcal{W}$ to be slightly confusing when also talking in the context of \"sparse updates\" etc: it makes me think that $\\Delta\\mathcal{W}$ is a repeated gradient update of $\\mathcal{W}$.\n\n- It seems like a core deviation of this work from an ensemble of prior techniques is the addition of the sparse residuals. As such, it seems appropriate that this should be highlighted more/expanded on in the main introduction for this work. On this note, I'm still not that convinced by the motivation, particularly on identifying the subspace $\\Omega$ from the pre-trained $\\mathcal{W}$. Can you provide more explanation for why you assume $\\Delta\\mathcal{W}$ shares this subspace? Or why $\\Omega$ isn't computed from the _sparse_ version of $\\mathcal{W}$? Compared to random (Fig. 2), the performance gap seems fairly small, and is inconsistent across scales.\n\n- It's hard to judge the practical benefit of the proposed approaches simply by looking at the number of trainable parameters and the number of flops. Actually exploiting these reductions to the full extent is hard (as the authors noted as well). It's also not really clear where the frontier is on total parameters vs. trainable parameters vs. accuracy is. Is it possible to give some real efficiency (i.e., wall clock) times? This would help shed some more light on the actual benefits vs. remaining challenges (how to actually implement this real-time). \n\n- The benefit of $\\mathcal{S}_2$ seems dubious. The differences are quite small, which makes me wonder if it is worth the extra complexity. ", "summary_of_the_paper": "This paper proposes methods to improve the efficiency of large pre-trained models (millions or billions of parameters) by combining multiple notions of _sparsity_ and/or _dimensionality reduction_ on the model weights. This is achieved via (1) a low-rank update matrix with fine-tunable parameters, and (2) pruning the original pre-trained weight matrix. These two sets of parameters are combined during inference. The authors validate their proposed method on several model architectures and datasets.", "summary_of_the_review": "Overall, I think that the topic the paper tackles is important and timely. The paper proposes a compelling combination of existing techniques to increase various aspects of efficiency. That said, unfortunately, in its current version the paper is not so clear to follow (in motivation and implementation). In terms of technical novelty, it seems that the added components over LoRA might need a bit more work for their performance gains to be more convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LEP5CqQTRDm", "reviewer": "Reviewer_gnKh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n* Developing more parameter-efficient and resource-efficient NLP models is important\n* Experiments show promising results\n\nWeakness:\n* Language and presentation of the paper can be improved. For example, after reading the introduction, it is still quite confused for me to understand the method and motivation because of confusing word usages — what exactly is the relationship and difference between “sparsity-aware weight update” and “sparse final weight”? \n* Experimental results can be better validated. Compared to LoRA, they claim the advantage of DSEE is that the “burden on resources such as storing the pertained weights are relieved”. To further validate this claim, it will be insightful to also compare with LoRA applied on sparse pertained model (i.e. first apply pruning on pertained model, then apply LoRA). There is some discussion on FLOPs in e.g. section 4.1, but it will be more clear to have a complete table on training/inference time and memory.\n* Technical novelty is somewhat limited, or at least can be better justified.\n\n", "summary_of_the_paper": "* This paper proposes a Dually Sparsity-Embedded Efficient Tuning (DSEE) which enforces sporty-aware weight updates on top of the pertained weights, and encourages a sparse weight structure towards the final fine-tuned model. On a diverse set of backbones and datasets, DSEE achieves 35% inference FLOPs savings with <0.1% trainable parameters without loss of accuracy.\n", "summary_of_the_review": "This paper addresses an important problem and shows some promising results to improve the efficiency of NLP models. However, the presentation of the paper can be improved and there should be more justification of the motivation and novelty of the method. In particular, it seems that the main difference between this method and LoRA is the sparsity of pertained models, which is achieved mostly with existing techniques, yet it seems that LoRA (and/or similar method) can also be applied on top of sparse pre-trained models.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F16ks68RSjq", "reviewer": "Reviewer_q6NF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting direction of achieving two different goals simultaneously. Below is my comments on this. \n\nStrength:\n1) Clearly defines the problem, goal and most part of the method\n2) Although not fully novel to scratch, the present an interesting idea of achieving both parameter and resource efficiency. They uniquely studied the problem of this and presented a way of making it work.   \n3) Multiple solid experimental results with a diverse set of models, benchmarks, and metrics showing the generalization of their approach. \n\nWeakness:\n1) My major concern is the evaluation. It is difficult to combine the experiments to support the claims. The gains in performance are marginal in compare to LORA, where LORA is a simpler method. With the S1 matric in Table 6, it even became worse in comparison to LORA or the finetuned model. The main claim is to excel for the large models but as found, with large models like DeBERTA,  the gains in performance is even lower. The FLOPs results are extremely insufficient to stand as an evidence (only on SST-B are reported). \n2) It's not described why S1 is only applied to pretrained weights W \n3) Unclear when to use unstructured or structured sparsity. \n4) Definitely just having a small reduction in memory usage does not mean to a significant resource utilization as it may lead to performance drop or may take an increased amount to training time. Analogously here, searching the sparsity also needs addition overhead (e.g., more epoch to tune UV after pruning etc.) the corresponding gain in performance is incremental specially when compare to the baseline LORA approach. \n\nTypos and tips:\n1) 1/3 heads from each attention head -> 1/3 heads from each attention leyer.\n2) reset the values of S2 back to 0. During the update, we only update the elements of S2 whose indexes are in Ω -> this should be in correspondence to Algo-2. \n3) More details how UV is computed without seeing pretrained weights W should be discussed (not just referring previous works) \n4) FLOs on BERT -> FLOPs on BERT\n\n\n", "summary_of_the_paper": "This paper proposes a principled way for achieving both parameter efficient finetuning and resource efficient inference. They combines the two steps of (i) computing a low-rank decomposition of parameter updates with a sparsity matrix (ii) further prune the trained (ie., finetuned) model. The first step aimed to achieve the parameter efficient finetuning but w/ the sparse residual components based on sparsity from the pretrained model to preserve the performance. The second step can reduce the memory of parameter storage. On three benchmarks with three different model architectures such as encoder-only, encoder-decoder they a large number of experiments with ablation studies. Studies show several a number of promising enhancement over other baselines.  ", "summary_of_the_review": "Results with initial success but rejected as further evaluation is needed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BtkHYM6OZ2Z", "reviewer": "Reviewer_UaPn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The paper is presented in an easy to read manner.\n\n2. I believe it's an important research topic to work on which can have a broad impact.\n\nWeakness:\n\n1. The organization can be improved. I feel the main motivation is actually in section 3 (page 4) \n\n\"However, as revealed experimentally by (Yu et al., 2017), a part of the important information in the trained weights will\nalso scatter outside......\"\n\nbut authors didn't put it in the introduction. This makes me feel the method is not well motivated. And consequently I think the proposed method is a bit ad-hoc. There are many details of thinking not well explained so maybe it can be an interesting method but currently it reads to me that it's fairly incremental. I think to make me increase the score, the introduction should introduce more rationale of previous methods and why the proposed method can improve.\n\n2. I believe the authors overlook an important work in the line of efficient fine-tuning (https://arxiv.org/abs/2101.00190). I didn't run the code myself but one of my collaborators told me this method in practice is fairly competitive without any effort of parameter tuning. I think the comparison to the work should be added.\n\n3. I think listing #trainable is not a fair comparison. To best of my knowledge, storing the sparse coordinate information requires additional coding of the row/col of the scalar which is omitted by most sparse-pruning papers. But when it comes to comparison of low-rank methods, it will create an unfair situation as low-rank methods doesn't require additional memory usage so maybe other researchers doesn't put emphasis on this but I insisted the memory footprint information should be represented in the result tables.\n\n4. Also, sparse method is in general slow without hardware accelerator. I think authors also know this as they wrote in the paper:\n\nIt applies fine-grained manipulation on each individual model weight, while the generated irregular sparse masks bring limited hardware speedup (Wen et al., 2016). \n\nBut somehow I didn't see any actions to cope with this and they just listed FLOP reduced which is again unfair to other structured methods such as structure pruning and low-rank. I think it's ok to require extra hardware accelerator. But to really benefit the research community, it's important also to present the weakness of method and I believe it's important to show the real wall-clock inference time instead of just the simulated #FLOP reduced. It's also interesting to know what's the wall-clock time saving in efficient fine-tuning part. As the low-rank method in general is easy to train and the proposed method seems to have many steps. So it seems to me that it's possible low-rank methods will have a bit more parameters but converges faster so the real time saving is better than the proposed method.\n\nIn fact, I think the empirical experiments doesn't really show a significant improvement over previous methods. For example, in Table 4 LoRA with  0.39M outperforms DSEE 0.17M and honestly 0.39M really doesn't increase too many parameters compared to DSEE. So I think the justified the efficacy of each step in DSEE and frankly present strength/weakness is rather important.\n\n5. This leads to my concern that each functionality of the method is not well discussed. In particular, the ablation study of solving eq 1 looks weird to me. As an apparent way to solve eq 1 is via iterative methods. Fix S2, UV can be optimally decided by SVD and fixing UV S2 can also be determined easily. So to me I think iteratively solving UV and S2 is the best candidate to compare but it's not appearing in the paper. And we don't even know what's inside the GreBsmo algorithm as authors didn't explain it at all. That confused me a lot. \n\nAnd if we look into the ablation study in Figure 2, there are some cases where other heuristics actually outperformed the DSEE. It's important to study why such cases happen but author didn't touch this part. I feel this then make the experiments not able to justify the proposed method.\n \n\n", "summary_of_the_paper": "This paper tries to solve both efficient fine-tuning and model size compression simultaneously. It adopts the previous framework to model W and \\delta W together and use different existing approaches to learn 2 parts in order. ", "summary_of_the_review": "Overall, I think the paper present an easily understandable method to an important and interesting question. But it reads to me many details are hidden or omitted so the efficacy of the proposed claim cannot be justified properly. I didn't list all of my concerns but at least the above mentioned five points should be addressed in order for me to increase the ratings.\n\n\nPost rebuttal:\n\nAfter author's reply, I still cannot confirm there is significant technical contribution. In addition, despite authors posted some additional experiment as requested, the numbers cannot match the original paper so I am not convinced the empirical result as well. In sum, there is no enough argument that the proposed method should work or any explanation why it should work. On the contrary, the experiments suggested that initilization plays important roles which might imply the claimed contribution might not be the real meat. Empirical evaluation also don't support the claim so I decide to remain my original ratings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper integrates several dimensionality reduction and sparsity methods for improving the efficiency of large pre-trained models. Overall, the paper is interesting and discusses an important topic. However, it seems that it is not ready to be published at the current stage. I would encourage the authors to take reviewers comments into account and further improve the paper 

The pros and cons of the papers are summarized in the following: 

Pros: 
+ Improving the efficiency of large pre-trained models is an essential research issue. 
+ The idea is interesting although the technical novelty is a bit limited. 

Cons: 
- The key concern is that the technical and practical benefit of the proposed approach is not clear based on the results demonstrated in the experiments. 
- The writing of the paper can be further improved in general to make the motivation more clear.
Decision: Reject