Sparse Low-rank Adaptation of Pre-trained Language Models
Authors: Ning Ding
Xingtai Lv
Qiaosen Wang
Yulin Chen
Bowen Zhou
Zhiyuan Liu
Maosong Sun
Venue: EMNLP 2023 Main (2023)
Keywords: Parameter-efficient
Sparse Adaptation

Abstract: Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. 
The popular method of low-rank adaptation (LoRA) offers a notable approach, hypothesizing that the adaptation process is intrinsically low-dimensional. Although LoRA has demonstrated commendable performance, it is implemented with a fixed and unalterable intrinsic rank that might not always be the ideal choice. Recognizing the need for more flexible adaptation, we extend the methodology of LoRA to an innovative approach we call sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process.  We achieve this through the incorporation of a gate unit optimized with proximal gradient method in the training stage, controlling the cardinality of rank under the sparsity of the gate. In the subsequent inference stage, we eliminate the parameter blocks corresponding to the zeroed-out ranks, to reduce each SoRA module back to a concise yet rank-optimal LoRA. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters via updating in a sparse way. We further introduce a sparsifying scheduler for SoRA, aiming to examine the impact of the number of non-zero parameters on the model's memorization and generalization. Our experimental results demonstrate that SoRA can outperform other baselines even with 70\% retained parameters and 70\% training time.

Sparse Low-rank Adaptation of Pre-trained Language Models
Ning Ding" Xingtai Lv Qiaosen Wang", Yulin Chen
Bowen Zhou', Zhiyuan Liu Maosong Sun
'Department of Electronic Engineering, Tsinghua University "Department of Computer Science and Technology, Tsinghua University
3 BNRIST, IAI, Tsinghua University, 'Department of Statistics, The University of Chicago @mail. tsinghua, edu. en, 1vxt20@mail seriiissssssngggn tsinghua..dee cn
Abstract
et al., 2022: Ding et al., 2023; Hu et al., 2023) man- ner is increasingly gaining traction within the re- search community. The methods this paradigm typically keep most of the parameters of the un- derlying model unchanged, either insert additional trainable parameters into the model (Houlsby et al., 2019; Li and Liang, 2021), or specify small num- ber of parameters (Zaken et al., 2021; Liu et al., 2021; Su al., 2023) be trainable or reparame- terize the adaptation process into more efficient form (Hu et al., 2021; Qin et al., 2021). They have been validated to be effective across various mod- els and tasks, often yielding comparable or even better results than full-parameter mie-tuning.
Fine-tuning pre-trained large language mod- els in parameter-efficiet manner is widely studied for its effectiveness and efficiency. The popular method of low-rank adaptation (LoRA) offers notable approach, hypothe- sizing that the adaptation process is intrinsi- cally low-dimensional. Although LoRA has demonstrated commendable performance, it is implemented with fixed and unalterable in- trinsic rank that might not always be the ideal choice. Recognizing the need for more flexi- ble adaptation, we extend the methodology of LoRA innovative approach call sparse low-rank adaptation (SoRA) that enables dy- namic adjustments to the intrinsic rank during the adaptation process. We achieve this through the incorporation of gate unit optimized with proximal gradient method in the training stage, controlling the cardinality of rank under the sparsity the gate. the subsequent inference stage, we eliminate the parameter blocks cor- responding to the zeroed-out ranks, to reduce each SoRA module back to a concise yet rank- optimal LoRA. Our approach strengthens the representation power of LoRA by initializing it with higher rank, while efficiently taming a temporarily increased number of parameters via updating in a sparse way. We further intro- duce sparsifying scheduler for SoRA, aiming to examine the impact of the number of non- zero parameters on the model's memorization and generalization. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
The development potential of parameter-efficien fine-tuning became evident after extensive vali- dation of its performance. These methods offer the opportunity adapt the base model fit any data, allowing for enhancements and customiza- tion of language models tailored to specific tasks and personalized user characteristics. Due to the lightweight nature of the optimized parameters, they can be seamlessly plugged into the model, al- lowing targeted enhancements to be made. Among these methods, low-rank adaptation (LoRA (Hu et al., 2021)) is considered one of the most efficient methods at present. It assumes that the change of the model's parameters after adaptation is "intrin- sically low-dimensional" and performs adaptation by optimizing the matrix obtained from low-rank decomposition. LoRA avoids forward propaga- tion latency caused by inserting additional neural modules while demonstrating stable performance. Although effective, the setup of the intrinsic rank (normally hyperparameter) is still unclear. In- tuitively, a larger rank brings larger optimization space and creates the capacity to handle more chal- lenging tasks. However, in practice, the optimal intrinsic rank would vary according to multiple factors such as the backbone model and the task.
1 Introduction
Adapting large-scale pre-trained language mod- els (Devlin et al., 2019; Brown et al., 2020; He et al.. 2020; Bommasani et al., 2021; Han al., 2021; Touvron et al., 2023) a parameter-efficicnt (He
cqual contributions
corresponding authors
Given the enormous computational cost of

---

searching hyperparameters on large-scale models (such GPT-3 (Brown et al., 2020) with 175 bil- lion parameters and LLaMA (Touvron et al., 2023) with 700 million to 65 billion parameters), develop- ing a method based on adaptive ranks is natural approach. Some existing work has attempled to explore this direction (Valipour et al., 2022; Zhang et al., 2023), but they are largely heuristic intro- duce additional costs. In this paper, we propose SoRA, simple, effective, and automated method for adaptive parameter-efficient fine-tuning. We introduce& a gating module with a proximal gradient decent update under LI regularization to control the sparsity of the updated matrices. After train- ing, the zero entry of the gating vector records the columns of the down-projection matrix and the rows of the up-projection matrix, which be simply dropped and stored in more parameter- efficient manner. Compared other adaptive ap- proaches, the proximal gradient method has clear mathematical meaning and does not have to involve other computations and heuristics. For example, AdaLoRA (Zhang et al., 2023) introduces an addi- tional regularizer to ensure that the lower and upper projection matrices strictly adhere to the definition of singular value decomposition (SVD), with each matrix being orthogonal. However, this regular- ization term incurs substantial computational over- head due to the gradient calculations. In contrast, we eliminate this requirement and instead selec- tively filter low-rank components by controlling the intermediate diagonal matrix. We detailedly compare SoRA and related methods in Section 3.
2 A Closer Look to Adaptive Rank
Related Work. Before introducing our approach, we first briefly recap parameter-efficien tuning and our backbone low-rank adaptation (LoRA). Parameter-efficient tuning is a set of methods that only optimize a small portion of parameters and keep the main model untouched for adaptation. Some parameter-efficient methods would insert ad- ditional neural modules or parameters to the back- bone model, such as Adapter (Houlsby al., 2019). Prefix and Prompt Tuning (Li and Liang, 2021; Lester et al., 2021). And another line of such meth- ods attempts to specify particular parameters to be trainable or prunable (Guo et al., 2021; Zhao et al., 2020; Zaken et al., 2021). Researchers derive a series of variants of parameter-efficien methods to improve the effectiveness efficiency (Karimi Ma- habadi et al., 2021: Hu et al., 2022: Sung et al., 2022: He et al., 2022). Recently, the applications of parameter-efficient fine-tuning are expanded to modal and instruction-tuning scenarios (Gao et al., 2023; Dettmers et al., 2023). In this paper, we focus more on LoRA (Hu et al., 2021), which uses low-rank matrices to approximate the change of weights.
In LoRA, pre-trained weights (denoted as Wo E RPx)) are frozen, and the trainable LoRA modules are low-rank decomposition matrices Wa E grxy and W, E Rxx of the change of each weight ma- W.W E Rex4, In this way, the output of the current layer h could be represented as
trix
- Wox + W. Wax.
(1)
where r < min (p, is a hyper-parameter of "intrinsic dimension" that controls the size of low-rank matrices and the number of trainable parameters. In this section, we primarily focus on the last term, denoting
The mechanism of SoRA also allows us to con- trol the sparsity temporarily and investigate the relationship between the number of non-zero train- able parameters and memorization and generaliza- tion capabilities. We propose a sparsifying sched- uler and find that the process of model adapta- tion exhibits a strong "compression capability"" and even a tiny portion of parameters (lower than LoRA rank being I) could retain considerable per- formance. Extensive experiments are conducted to demonstrate the effectiveness of our method. Par- ticularly, our model could consistently outperform parameter-efficient baselines with fewer parame- ters and 30% shorter training time on wide range of downstream tasks. The code of this work will be publicly available at https: /tt//gttit github...... com/ Ts #singhuaC31/ I/SORA.
W.X..
Adaptive Rank on LoRA. Despite great step forward in tractability and efficiency, LoRA is still restricted by its inflexibility in selecting the opti- mal rank r. Unlike continuous hyperparameters such as learning rate and weight decay that can be tuned adaptively online during the training process, LoRA rank r takes discrete values the change of which will directly alter the model structures. The optimal choice of rank can vary across different backbone models and downstream tasks. A conser- vative choice of huge rank r can waste training time and computation resources, while progressively set- ting r tiny may degrade model performance and

---

lss (40
()
(o
2
W
h
g
h
W
X
Up- projection
Down- projection
Gating
Updatee g
Zero values
Drop after training
1/s V VgLo( (A)
Non-zero values
Proximal Gradient Descent
Input& & Output
T(-)
An illustration of sparse low-rank adaptation (SoRA). At the training stage, the gate g will control the sparsity of Wa and W.. At the inference stage, zero vectors in Wa and W., indexed by the zero entries of g, would be eliminated.
Figure I:
lead from- from-scratch re-training. These limitations highlight the importance of upgrading LoRA with an sllaptiverrnnk-sllccte plug-in.
Then, letting d (p. q). we can reformulate the LoRA forward propagation as
U. V 1=xx),
(4)
Several remedies have been proposed in re- cent years to enable the flexible tuming of LoRA rank. For example, rather than setting fixed rank. Valipour et al. (Valipour al., 2022) introduce Dy- LoRA which pre-defined discrete distribution Pn(-) is cast over a range of rank choices. This approach is related to but different from nested dropout (Rippel et al., 2014), and can be regarded optimizing a mixture model with LoRA modules of different ranks.
where denotes clement-wise dot product (Hadamard product). Note that rank( || lo which the to norm of g. Therefore, tuning the LoRA rank suffices to control the sparsity the vector g. Zhang et al. precede along this SVD-based track with their methodology named AdaLoRA (Zhang et al., 2023). In AdaLoRA. the elements in vector g calibrated such that the number of nonzero entries is smaller than pre-defined budget b. To be specific, they pre- serve only the entries with top-b importance score which their newly proposed metric of "sensitiv- ity" heuristically constructed from weight-gradient product. The nonnegativity of g entries is reason- ably dropped since a negative gi can be simply reduced to the positive case by flipping the sign of either u, V,. Besides, they transform the constrained optimization problem into uncon- strained version by replacing the orthogonality con- ditions UT U Ip and V V -Ig with regular- ization term
Nevertheless, tuning LoRA rank straightfor- wardly and deterministically appears to be a more attractive approach. To devise such an approach, we first gain a crucial hint from the connection be- tween matrix's 's rank and its singular value decom- position (SVD). Let us denote the tunable incre- mental weight matrix in LoRA by A W. Wa- We can then formulate its SVD
U pxp
VT pxq 9xq)
(2)
pxg
in which U and V are orthogonal respectively, and E (rectangular) diagonal matrix with diagonal elements being the singular values of A: a (A)
R(U, = I 111 - + V V
11
(ar Z 02 2 ... 2 min(p.g) 2 0). For notation convenience, we reshape the diagonal of Σ into column vector
(5)
In spite of the effectiveness demonstrated through experiments, there are still two problems in AdaLoRA that demand rethinking of the method- ology and wait for further improvements. First, the
g = (01,02, 02, ... 0miu(p.g))

---

sparsity selection criterion in A is based on their newly proposed importance score relied on the moving average of weight-gradient prod- uct. Despite its effectiveness empirical study, this criterion is largely heuristic, lacking theoreti- cal motivation. Second, both the moving average operation of importance scores and the gradients of orthogonality regularization (5) add up to addi- tional computation cost. Compared to AdaLoRA
methods as in LoRA, while each gate g updated in different searsity-promonee way:
B++1 <- T T/e VgC(()a)
(10)
in which Lo(-) is the original loss function of the language model, A denotes the complete tunable parameter (including the gates), 7/ > 0 stands for the step-size at the I-th iteration, and À > 0 works as the regularization strength hyperparameter that promotes sparsity. Besides, --)- in the above expression stands for the element-wiss -iise broadcast of the following soft soft-thresholding function:
with the aforementioned limitations, our approach, SoRA, serves as an amelioration with highly simpli- fied updating rules and backed up by the theory of sparsity regularization and proximal gradient methods. Detailed methodology of SoRA will be elaborated in the next section.
-E-5, E, E
Te(x) == <<<os t x x
(11)
3 Our Approach
with ç 1/2 À being the threshold. In practice, the true gradient VgLo in (10) is approximated by its mini-batch stochastic counterpart.
The key idea of our approach, sparse low-rank adap- tation (SoRA), is to dynamically adjust the intrinsic rank the training process with a sparse gating unit trained by proximal gradient method. SoRA adopts the previously introduced framework of low- rank decomposition because of its widely validated effectiveness and parameter efficiency.
Post-pruning. When training is completed, we further prune the SoRA weights to drop the zeroed- out ranks and reduce the module back to the LoRA form. To be specific, for the k-th SoRA module, let
( = {i -
[1 rmax]
(k) g -
(12)
3.1 Sparse Low-rank Adaptation
be the index of zero entry in the k-th gating vector g(()), We drop the (A) rows of down-projection d to obtain W(AA) ttee (Ak)-th columns of up- projection w((k) to obtain W((),, as well as the ((h) entry of gate (*) to obtain (i(k) In this way, during inference time the k-th SoRA module will proceed as usual LoRA module of rank Tmax (*)) with down-projection matrix w-(k) and up- projection matrix ((()) diag(g(()
Module Structure. At the start of building SoRA module, we pre-define a maximum accept- able rank Tmax according to practical or research concerns. Then, each SoRA module will inherit two matrices Wa E Tmax Xq and W., E Rpxrma from LoRA for down projection and up projection. The maximum rank max is set to be relatively large, but we will show in the subsequent paragraph how to tame it efficiently in sparse sense. In fact, this is realized by injecting a gating unit g E Rims be- tween the projection matrices, which imitates the formulation of SVD. The forward propagation of the SoRA module proceeds as follows:
((())
3.2 Interpretation and Comparison
Theoretical interpretation. The update rule (10) in fact an application of the proximal gradient method for & loss (Chambolle et al., 1998; Beck and Teboulle, 2009). This follows immediately (6) once we reformulate (10) equivalently as
down projection
W ax;
gating
h g h:
(7)
gr+1 f arg min Ttt A/lg|/1
z
up projection
W.h
(8)
g
1g (g - (oo(g))
(13)
or, more compactly,
The above equation (13) is exactly the proximal gradient update of the e regularized loss function
- W. (g @ (Wxx)).
K
Optimization. We optimize down-projection and up-projection matrices with stochastic gradient
((A) = :0:==o(A1 Lo(A) +
l1g (
(14)

---

where (8)) denotes the gate of the k-th SoRA mod- ule. This sparsity-rromoting strategy dates back to LASSO estimator (Tibshirani, 1996) and com- pressed sensing (Candes et al., 2006), and is also adopted by many works within the realm of deep learning (Wen et al., 2016; Scardapane et al., 2017).
atively small value to &. Consequently, the SoRA model is initially "dense" and is trained until con- vergence. Once this stage achieved, we introduce a scheduler to incrementally increase the value of ç, thereby enhancing the model's 's sparsity. During this transition from dense to sparse model, it becomes possible to evaluate the model's memo- rization and generalization abilities by examining performance the training and testing data respec- tively. The procedure is reported in Algorithm 1. The process can be regarded as exploring the "compression loss" in the scenario of model adaptation. Here, "compression loss" refers to the reduction in model performance due to the increased sparsity, providing measure of how well the model can retain its predictive power under constraints. Investigating this "compres- sion loss" is meaningful to understanding the behavior of model adaptation and can facilitate developing efficient, compact models that maintain Bigh-performance levels.
Comparision with AdaLoRA. Inspired alike by SVD decomposition, our approach SoRA differs from the preceding work AdaLoRA (Zhang et al., 2023) in the following sense. First, we do not apply the orthogonality regularization (5) used in AdaLoRA. The reason that for rank selection purposes, sparsifying the gate g will be sufficient. Sticking to the original requirements of SVD can result in additional computation expenditure. Sec- ond, the moving averaged importance score in AdaLoRA works as an approximation to the change in loss when the corresponding entry is zeroed out, which is regarded as heuristic measurement of parameter "sensitivity".. However, model's 's's tempo- ral sensitivity to a certain parameter cannot imply that the parameter should be retained, since there is no rigorous theory for doing so. By contrast, our rank selection based on soft-thresholding oper- ation (10) proceeds in a much cleaner form and is soundly justified by the theory of proximal gradient iteration. As is explained earlier this section, the updating rule of SoRA module exactly follows the trade-off
Algorithm 1: Scheduling Algorithm of g
Input :M, Eo- Emax 6g, D
Output: (Ma, M1....
É < So:
M' < 0;
M = TrainUnnilConeergene(MM D.E.)
M' M'dadd(M):
first principle of cetrpolationcommpeete by minimizing a regularized loss objective (14).
E Ex;
while & S Emux do
Beyond the formal simplicity and theoretical clearness is SoRA's superior experimental perfor- mance achieved with fewer parameters in less wall- clock time, which will be presented in Section 4.
for epoch 1 to 5 do M Upddtee(MM D,
end M''add(MM):
& E + 6g;
3.3 Scheduling & to Explore Memorization and Generalization
end
We dub the threshold & as sparsity indicator. As the name implies, this parameter could directly de- termine the sparsity of SoRA in the training process. It can be set as a constant to heuristically control the sparsity according to the budget of parame- ters and expected performance. When dynamically changing & in the adaptation process, SoRA serves an effective tool to assess the memorization and generalization under model M and dataset D. In other words, we can visually observe how many additional parameters are required to achieve a par- ticular point of performance given the model M and data D. We elaborate the fundamental idca as follows. The process starts by assigning a rel-
Experiments
Extensive experiments are carried out to assess the effectiveness of our approach comprehensively. Generally speaking, section: (1) the performance and corresponding analysis as normal parameter-efficient method; and (2) the investigation of memorization and gen- eralization in virtue of the sparsity nature of SoRA.
explore two aspects in this
4.1 Experimental Settings.
Baselines. Our baselines comprise full parameereet fine-tuning and other rell-recognized parameter- efficient methods, including Adapter (Houlsby

---

Method
#Params
CoLA
SST-2
MRPC
QQP
STS-B
MNLI
QNLI
RTE
Avg.
Fine-Tune
184M
69.21
95.64
89.22
92.05/89.31
91.59
89.98/89.95
93.78
82.49
87.82
Adapter
1,41M
69.00
95.16
89.90
91 41/48/8888
92.21
90.11/90.11
93.79
82.44
87.85
Bitfit
0.IM
68.70
94.38
87.16
87 27.86/84.0
89.71
87.45/87.45
91.90
76.12
85.18
LoRA (r=8)
1.33M
69.73
95.57
89.71
91.95/89.26
91.86
90.47/90.46
93.76
85.32
88.38
AdaLoRA
1,27M
70.86
95.95
90.22
92, 41
91.39
90.27/90.30
94.28
87.36
88.83
SoRA
0.91M
71.48
95.64
91,98
92.39/89.87
92.22
90.35/90.38
94.28
87.77
89.36
Table I: Test results of SoRA and other baselines the GLUE benchmark. We denote the best result in bold and underline the second best result. The standard deviations of results from different methods are similar and we show them in Table 8 in Appendix A.4.
et al., 2019), BitFit (Zaken al., 2021), LoRA (Hu et al., 2021) and A (Zhang et al., 2023). We omit the variants of the Adapter since we find that the performance between them is very close. We also do not include Prompt Tuning since we find that it takes considerably longer time for con- vergence and cannot yield non-trivial performance on backbone models.
SoRA achieved an accuracy of 91. 91.98% surpassing AdaLoRA by 1.76%. On average, SoRA surpassed LoRA and AdaLoRA on the GLUE benchmark by 0.98% and 0.52%, respectively, using 31.5% and 28.3% fewer parameters. To take a closer look at the effectiveness of adaptive rank, we conduct experiment to compare LoRA and SoRA with different ranks in Table 2. The results affirm that SoRA's superiority is consistent across different budgets of parameters, that is, SoRA could out- perform the LoRA baseline in all settings while utilizing over 30% fewer parameters.
Datasets For evaluation, we adaopt the GLUE benchmark (Wang et al.). including CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), QQP (Wang et al.), STS-B (Wang al.), MNLI (Williams al., 2017), QNLI (Rajpurkar al., 2016) and RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo al., 2007; Bentivogli et al., 2009). We mainly use DeBERTaV3-bass (He et al., 2021) as the backbone model. Additionally, we also use RoBERTa-large (Liu et al., 2019) for analysis. Other experimental details are described in Appendix A.
4.3 Sparsifying Scheduler
We apply the sparsifying scheduler introduced Section 3.3 by enlarging the sparse indicator g (starting from le-4) of SoRA progressively in the adaptation process. As illustrated in Figure 2, we plot the memorization and generalization curve of RoBERTa-large (Liu et al., 2019) on MRPC, RTE, STS-B, CoLA, QNLI, and SST-2, where the mem- orization is gauged by the performance on the train- ing set and the generalization measured by the performance on the validation set. Intriguingly, we observe a robust "compression performance" across almost all the datasets. Among these, SST-2 emerges the most "compressible" task, where the model sustains over 99% performance when restricted to 47,104 non-zero parameters. Remark- ably, a mere 4,096 parameters can still conserve above 90% memorization and generalization capa- bilities. As the sparsifying process proceeds, the model encounters an "inflection point" on differ- ent data, after which the performance significantly plummets. This consistent phenomenon suggests that there exist some critical parameters that un- derpin the performance and worth further in- vestigation. Insight gleaned from the graph also indicates varying degrees of adaptation difficulty for the model across different datasets. For exam- ple, certain datasets, like CoLA, prompt an earlier
4.2 Results
We first conduct an evaluation on GLUE bench- mark, widely recognized benchmark for natural language understanding. The experimental perfor- mance of SoRA, well other baseline method- ologies, is recorded in Table 1. We reproduce these methods in our infrastructure and present the av- erage results drawn from 5 random seeds. Our findings indicate that both AdaLoRA and S consistently outperform the initial LoRA baseline. This underlines the validity of adaptive rank po- tent solution for enhanced model adaptation. Most notably, SoRA outshines all other basclines, partic- ularly LoRA and AdaLoRA, despite utilizing fewer parameters. This lends credence the argument that our proximal gradient method may constitute a more efficacious and essential approach to achiev- ing adaptive rank. For instance, on the MRPC,

---

Method
4Params
CoLA
SST-2
MRPC
QQP
STS-B
MNLI
QNLI
RTE
Avg.
LoRA
0.17M
68.60
94,95
88.24
91 91.20/88.37
91.41
90.09/90.25
93.35
81.29
87.23
SoRA.-1
20.49%
70.24.us4
95.14,0.9
89.22,0.8
91.52/88.3..a
91.41.0.00
90.08/90.41
93.43. 0.2
83.02
87.85
LoRA,=2
0.33M
68.93
95.04
88.43
91.59/88.8
91.53
90,35/90.30
93.63
84.17
87.79
2
0.25M 24
70.22
95.64.0.00
89.71
01.88/89.12.0 0.27
91.63
90.37/90.51 ia
93.78 0.15
88.18 1,01
88.39,
LoRA,-1
0.66M
69.27
95.55
89
91.40/88.44 41
91.69
90.36/9049
93.83
82.01
87.74
0.47M. Tw
71.05.174
90.20
92.06/89.444..r
91.76,cmn
90.38/90.43.om
93.92.
86.04...3
88.71
LoRA,
1.33M
69.73
95.57
89.71
91.95/89.22
91.86
90.47/90.46
93.76
85.32
88.38
SoRAres
0.91M.s1 105
71.48,1.7
95.644m17
91.98
92.39/89.87
92.22
90.35/99.38., to
94.28....5
87.77
89.36,
Lo
2.65M
92.22/89.63
90.55/90.31
So
71.93.2.6
95.61.018
92.00
92.37/89.84...
92.05
30.34/9044.oom
94.11
87,41.005
89,33.
Table 2: Test results and number of parameters of SoRA initialized with different Tmax on the GLUE benchmark, compared with LoRA of the same rank. The standard deviations of results from different methods are similar and we show them in Table 9 in Appendix A.4.
COLA
SST-2
MRPC
100
100
Inflection Point
90
90
Inflection Point
-80
Performances %6
Infiaction Point
80
660
70
40
Memorization
60
Memorization
50
Memarization
20
Generalization
Generalization
Gonsralization
5
4
3
80
40
16
8
6
Param(10*)
Param(10*)
Param(105)
STSB
QNLI
RTE
100
100
100
Inflection Point
80
Inflection Point
Inflection Point
Performancee (%6)
60
800
40
40
40
Memorization
20
Memortzation
20
Memorization
20
Ceneralization
Generatization
Genoratization
5
14
10
6
Param(10*)
Param(10*)
Peram(10*)
Figure 2: The memorization and generalization curve on six datasets. The "Param" axis indicates the number of non-zero parameters. The sparsity indicator increases every 5 epochs.
and more pronounced decline in performance com- pared to others. Another finding is that the trend of memorization and generalization is consistent in the sparsifying procedure, which is aligned with intuition. Our observations also indicate tendency for the parameters of intermediate and deep layers to maintain their density, while those of the shallow layers show higher propensity towards sparsity.
converges with SoRA on four datasets in Figure 3. Quite obviously, the trained parameter matrices on QQP are exceedingly dense and others do not exhibit such density, which echos the existence of different levels of difficulties. This phenomenon also suggests that leveraging the performance and the parameter budget does not have an invariable constant law, but needs specific considerations in different situations.
4.4 Rank Analysis
4.5 Applying SoRA to Different Weights
An intuitive statement is that single model suf- fers from varying extents of difficulty when being adapted different downstream datasets. Concur- rently, it evident that not all parameters within the model carry equal importance- some are more crit- ical to performance than others. In this section, we visualize the final ranks after the training process
In experiments in Table 1, utilize LoRA, AdaLoRA, and SoRA on all weight matrices enhance performance. It should be noted that the performance may fluctuate when parameter- efficient fine-tuning applied to various positions within the model, as evidenced by previous

---

QQP
MNLI
Query
6
Key
Key
Value
8
9
Value
6
9
Attn Output
8
9
Alth Output
5
8
FFW1
7
10
11
QNLI
MRPC
Query
Query
Key
0
2
Key
1
2
2
Value
0
0
2
Value
3
3
Ntn Output-
Attn
2
5
W2
1
FRW2
2
3
2
5
5
Figure 3: The final ranks after training with SoRA on four datasets (l.e., QQP, MNLI, QNLI, and MRPC). The X-axis is index DeBERTaV3-hase layers, and Y-axis indicates different layers SoRA applies to.
research (Zaken al., 2021; Hu et al., 2022; Zhang et al., 2023). We carry out such ablation experiments with SoRA on three datasets to investigate the impact. Although SoRA is not budget-oriented method, we adjust À to approxi- mately equate the retained non-zero parameters. As reported in Table in most cases, the applica- tion of SoRA all weight matrices resulted in considerable improvement in performance com- pared to the application of merely one or several types of weights, which suggest that uniformly applying SoRA to all weight matrices can serve a beneficial strategy. And merely applying SoRA to Wq,k will experience considerable performance drop, which is aligned with LoRA.
and batch size. shown in Table 4, SoRA takes about 30% less training time than AdaLoRA. certain instances, such as the CoLA, QNLI, and RTE datasets, SoRA exhibits a significant edge in efficiency over its counterpart. Conversely, while SoRA consistently outpaces AdaLoRA on other datasets, the margin is not wide. This discrep- ancy could be attributable the different rank dis- tributions of AdaLoRA and SoRA under varying tasks. Such distributions exert influence on the calculation of regularization in AdaLoRA.
Datasets
AdaLoRA (s)
SoRA (8)
CoLA
160.2
57. 57.2.64.99%
SST-2
491.0
433.0.11.81%
MRPC
27.3
24.8.9.
#Params
CoLA
MRPC
STS-B
STS-B
48.2
38.4. 38.4.20.33%
Wax
0.80M
65.56_224
86.43
72.30
QNLI
1001.0
676.3 32.44%
Naxv
0.69M
69.07-217
18.24
90.82+a.70
RTE 79.8
45.1 433.48%
WQKKA
71.99=1.0
90.20
91.71
0.77M
7148-1.17
91.98+1.16
92.22+0.24
Avg.
301.3
212.5 29.49%
Table 3: Test results that applying SoRA to different weights. Q K, V. and A.O represent query, key, value and attention output layers respectively. #Params means the number of parameters that would remain after training.
Table 4: The average training time per epoch on six datasets. For each task, the experiments with AdaLoRA and SoRA have the same batch size 32.
5 Conclusion
4.6 Efficiency Analysis
Our work presents Sparse Low-Rank Adapta- tion (SoRA), an innovative method for parameter- efficient fine-tuning large pre-trained language models. Upon the hypothesis that the adaptation process could be intrinsically sparse, we offer a dynamic alternative rank by introducing an opti- mizable gate with a proximal gradient method to
We elaborate that SoRA is a theoretically clear and computation-efficcent method in Section 3.2. To evaluate this, measure the efficiency of SoRA and AdaLora in this section. We compute the clock time of average epoch of AdaLoRA and SoRA on six datasets with identical compute infrastructure

---

regulate sparsity, thereby expanding the optimiza- tion space while enhancing parameter efficiency, The method is simple and theoretically supported with promising performance across various tasks. Utilizing SoRA as a tool, we propose a sparsify- ing scheduler to analyze the correlation between parameters and memorization and generalization.
Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Limitations
Despite the encouraging results demonstrated by SoRA, there are certain limitations in our current study that are worth acknowledging. This paper only evaluates the effectiveness of SoRA on tra- ditional natural language processing tasks. How- ever, recent studies demonstrate that parameter- efficient methods could be applied to cross-modal or instruction-tuning scenarios, In those cases, how the sparsity of SoRA is displayed is still unknown and worth investigating. Our sparsifying scheduler could provide insights the adaptation process of language models, but it still challenging rigor- ously explain the procedure and more efficiently to assess the difficulty of an adaptation process.
Emmanuel J Candes, Justin K Romberg, and Terence Tao. 2006. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 1908):1207
Antonin Chambolle, Ronald A De Vore, Nam- Yong Lee, and Bradley J Lucier. 1998. Nonlinear wavelet im- age processing: variational problems, compression, and noise removal through wavelet shrinkage. IEFE Transactions Image Processing, 7(3):319-335
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges Workshop. pages 177-190. Springer.
Acknowledgements
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 143144 14314.
This work supported by the National Key R&D
Program of China (No. 20227D01 19101), Na- tional Natural Science Foundation of China (No. 62236004), the Young Elite Scientists Sponsorship Program by CAST, and Institute Guo Qiang T's- inghua University.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
References
Amir Beck and Mare Teboulle. 2009. A fast itera- tive toinkage-thresooddig algorithm for linear in- verse problems. STAM journal on imaging sciences,
Ning Ding. Yujia Qin, Guang Yang. Fuchao Wei. Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. 2023. Parameter-efficiet fine-tuning of large-scale pre- trained language models. Nature Machine Intelli- gence, 5(3):220-235.
11)::133200.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccoo. 2009. The fifth pascal recognizing textual entailment challenge. In Proceedings of Text Analysis Conference.
Bill Dolan and Chris Brockett 2005. Automati- cally constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255
Dan Friedman, Ben Dodge, and Danqi Chen. 2021. Single-dataset experts for multi datti-ddtaset question an- swering. ArXiv preprint, hts/2109.11882
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. 2023. Llama-adapter v2: Parameter-efficien visual instruction model. arXiv preprint arXiv:2304.15011 15010.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Mors,rt-Voss,,

---

Danilo Giampiceolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
Opendelta: A plug-and -nd--aadyyyyy library for parameter- efficient adaptation of pre-trained models. arXiv preprint urXiv:2307.03000
Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. 2022. Sparse structure search for parameter-efficient tuning. arXiv preprini urXiv:2206.07382.
Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-efficien transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Lan- guage Processing ( (Volume I: Long Papers), pages 4884-4896, Online. Association for Computational Linguistics.
Rabech Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low- low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 1022-0005.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. ArXiv preprint, abs/2 104.0861..
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo. Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop on Recognising Textual Entailment, volume 7.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu. Lewis Tunstall, Joe Davison, Mario Šaško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xu Han, Zhengyan Zhang. Ning Ding. Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang. Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past. present and future. Al Open.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards unified view parameter-efficien transfer learning. In International Conference on Learning Representa- tions.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. 3ebertav3; Improving deberta using electra-style pre- training with ing. arXiv preprint arXiv:2111.095.
Xiang Lisa Li and Percy Liang, 2021. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
embedding shar-
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0665..
Peiyu Liu, Ze- Ze-Feng Gao, Wayne Xin Zhao, Zhong-X Lu, and Ji-Rong Wen. 2021. Enabling lightweight fine-tuning for pre-trained language model compres- sion based on matrix product operators. arXiv preprint 2xXx:2106.022000
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beuch. California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv preprint, abs/1900.11692.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. ArXiv preprint, 10s/2106.09685.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Shengding Hu, Ning Ding, Weilin Zhao, Xingtai Lv, Zhen Zhang, Zhiyuan Liu, and Maosong Sun. 2023.

---

Yujia Qin, Xiaozhi Wang. Yusheng Su, Yankai Lin. Ning Ding, Zhiyuan Liu, Juan-Zi Li. Lei Hou, Peng Li, Maosong Sun, and Jie Zhou. 2021. Exploring low-dimensional intrinsic task subspace via prompt tuning. ArXiv, abs/21 10.07867.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016. Learning structured sparsity deep neural networks. Advances in neural informa- rion processing systems, 29.
Adina Williams, Nikita Nangia, and Samuel Bow- man. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:/606.05255.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et 2020. Transformers: State-of-the the-re- natural language processing. In Proceedings of the 2020 con- ference empirical methods in natural language processing: system demonstrations, pages 38 45.
Oren Rippel, Michael Gelbart, and Ryan Adams. 2014. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746-1754. PMLR.
Simone Scardapane, Danilo Comminiello, Amir Hus- sain, and Aurelio Uncini. 2017. Group sparse regular- ization deep neural networks. Neurocomputing. 2-1:81-89.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Gold- berg. 2021. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language- models. ArXiv preprint, abs/2106.10199.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Mamning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:230. 10512.
Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang. Ning Ding, Zhiyuan Liu, and Maosong Sun. 2023. Ar- bitrary few parameters good enough for adapt- ing large-scale pre-trained language models. arXiv preprint 2rX.:22306.02200.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hin- rich Schütze. 2020. Masking tive to finetuning for pretrained language models. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226-2241, Online, Association for Computa- tional Linguistics.
efficient alterna-
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and mem- ory efficient transfer learning. arXiv preprint
arXiv:2206.06522.
Robert Tibshirani. 1996. Regression shrinkage and se- lection via the lasso. Journal of the Royal Statistical Society: Series B (Methddoloicaal), 58( 58(1):267-228
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1 13971.
Mojtaba Valipour, Mehdi Rezagholizaadh, Ivan Kobyzev, and Ali Ghodsi. 2022. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel Bowman. Glues A multi-task benchmark and analysis platform for natural language understanding.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641

---

A Experimental Details
The Huggingface Transformers (Wolf al., 2020) and Py PyTorch (Paszke et al., 2019) are utilized for all the experiments. We use NVIDIA GeForce RTX 3090 (maximum GPU memory=24268MB) and the application of SoRA with a batch size of 8 occupies 61 L0MB GPU memory on average.
A.1 Datasets
The GLUE benchmark, consisting of CoLA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), QQP (Wang et al.. STS-B (Wang et al.). MNLI (Williams al., 2017), QNLI (Rajpurkar al., 2016) and RTE (Dagan et al., 2005; Haim al., 2006; Giampiccolo al., 2007; Bentivogli et al., 2009), is used for natural language under- standing. The details and the evaluation metric are reported in Table 5. We source each dataset from Huggingface Datasets (Lhoest et al., 1021) and utilize the full dataset for our experiments. For almost all experiments, we run 5 times using different random seeds and report the average results in order to ensure statistical significance.
A.3 Optimization of Hyperparameters
In this section, we delve into the optimization of hyperparameters. The results of different Ymmx are proved in Table and supplement the results of two other important hyperparameters, & and 7j in the Table 6 and Table 7. The performance of SoRA highly stable with respect to different choices of & and And for each fixed and 7), the variance of performance rather low. In general, we suggest setting & to le-4 level and around 1e-1~~e..
COLA
STS-B
Dataset
#Train
#Valid
#Test
Metric
1c-3
63.25
90.85
CoLA
8.5k
1,043
1,063
Mee
8e-4
64.98+1.25
91.12+0.38
SST-2
67k
872
1.8k
Ace
Se-4
66.94
91.61 10.24
MRPC
3.7k
408
1.7k
Acc
QQP
364k
40.4k
391k
Acc/F1
3e-4
68.61
92.22+0.24
STS-B
5.7k
1.5k
1.4k
Corr
le-4
70.18
92.01
92.0110.14 10.14
MNLI
393k
9.8k/9.8k 8k
9.8k/9.8k
Acc(m/mm)
8c-5
68. (8.788...44
92. 92.00
QNLI
105k
5,5k
5.5k
Acc
5c-5
71.48
92.02
RTF
2.5k
277
3k
Acc
3e-5
69.68 MM
92. 92.18 18.15
1e-5
69.65
92.08 10.15
Table 5: The size and evaluation metric of the datasets in GLUE benchmark. "Mec", "Acc". "F1" and "Corr" represent matthews correlation coefficient, accuracy, the score and pearson correlation coefficient respectively. And "Aco(m/mm)" represents the results corresponding to matched and mismatched datasets of MNLI while the metric accuracy.
Table 6: Test results of the optimization experiments on different
1/c
COLA
STS-B
10
69.0642.18
91.75+0.41
1
70.54
92.02.0.20
A.2 Implementation Details
0.1
71.48
92.22+00.4
0.01
68.7 68.78442.29
92.06
Regarding hyper-parameters, we set the learning rate to 8e-4. Based on the size and training conver- gence speed of the datasets, we set the number of epochs for CoLA, MRPC, and STS-B to 20, and the number of epochs for the remaining tasks to 10. As for RTE, we reference the settings of Friedman et al. 2021, which entail a learning rate of 1.2e-3 and an epoch count of 50. We set À to 0.1 in all our experiments, and select & with grid search in [1e-5, 5e-5, le-4). When dealing with MRPC, RTE, and STS-B datasets, common trick in cer- tain studies is that using the best model checkpoint on the MNLI dataset could boost the performance. In our experiments, we do not use this strategy and instead opt for standard initializations across all models.
0.001
69.8 1..5
91.83
0.0001
69. 69.70+1.70
92.13
Table 7: Test results of the optimization experiments on different 1/.-
A.4 Results with Standard Deviations
The test results in Table are shown in Table 8 and results in Table 2 are shown in Table 9.

---

Method
#Params
CoLA
SST-2
MRPC
QQP
STS-B
MNLI
QNLI
RTE
Avg.
69.21
95.64
89.22
92.05/89.31
91.59
89.98/89.95
93.78
82.49
Fine-Tune Tune
184M
87.82
(2.24)
(0.52)
(0.69)
(0.09)(0.00)
(0.47)
(0.06)/(0.33)
(0.02)
(1.48)
69.00
95.16
89.90
91.45/88.88
92.21
90.11/90.11
93.79
82.44
Adapter
1.41M
87.85
(0.91)
(0.46)
(2.10)
(0.18)((.40)
(0.33)
(0.57M(0.57)
(0.07)
(1.74)
68.70
94.38
87.16
87. 86/84,20
89.71
87 47/84/8744
91.90
76.12
Bitfit
0.IM
85.18
(1.85)
(0.28)
(0.58)
(0.52y(0.74)
(0.58)
(0.76)((076)
(0.14)
(1.54)
69.73
95.57
89.71
91.95/89.22
91.86
90.47/90.46
93.76
85.32
LoRA (r=8)
1.33M
88.38
(1.42)
(0.21)
(1.32)
(0.12M(0.18)
(0.29)
(0.23)/(0.12)
(0.36)
(0.86)
70.86
95.95
90.22
92.13/88.41
91.39
90.27/90.50
94.28
87.36
Adal.oRA
1.27M
88.83
(1.43)
(0.37)
(0.40)
(0.06)((.05)
(0.25)
(0.11)((.18)
(0.11)
(0.30)
71.48
95.64
91.98
92.39/89.87
92.22
90.35/90.38
94.28
87.77
SoRA
0.91M
89.36
(1.17)
(0.23)
(1.16)
(0.17)((.27))
(0.24)
(0.09)(0.12)
(0.06)
(1.56)
Table
Test results SoRA and other baselines on the GLUE benchmark. We denote the best result in bold and underline the second best result. The standard deviation is provided in parentheses.
Method
#Params
CoLAA
SST-2
MRPC
QQP
STS-B
MNLI
QNLI
RTE
Avg.
68.60
94.95
88.24
91.20/88.37
91.41
90.09/90.22
93.35
81.29
LoRA,-1
0.17M
87.23
(1.58)
(0.20)
(1.42)
(0.20/0.33)
(0.29)
(0. (0.16/0.14)
(0.34)
(2.83)
70.24
95.14
89.22
91.52/88.73
91.41
90.08/90.41
93.43
83.02
SoRA--1
0.12M
87.85
(1.10)
(0.25)
(2.12)
(0.14/0.22)
(0.29)
(0.18/0.12)
(0.33)
(2.47)
68.93
95.04
88.43
91.59/88.88 87
91.53
90.35/90.30
93.63
84,17
LoRA,-2
0,33M
87.79
(1.50)
(0.37)
(1.06)
(0.06/0.12)
(0.32)
(0. (0.13/0.17)
(0.37)
(1.20)
70.22
95.64
89.71
91.88/89.15 12
91.63
90.37/90.51
93.78
85,18
SoRA.
0.25M
88.39
(1.03)
(0.09)
(0.81)
(0.13/0.17)
(0.30)
(0.06/0.08)
(0.23)
(0.73)
69.27
95.55
89.22
91.40/88.44 .41
91.69
90.36/90.49
93.83
82.01
LoRA,
0.66M
87.74
(2.22)
(0.56)
(1.64)
(7.177).27)
(0.24)
(0. (0.12/0.3))
(0.35)
(1.76)
71.05
95.57
90.20
92.06/89.44
91.76
90,38/90.43
93.92
86.04
SoRA
0.47M
(0.74)
(0.05)
(1.60)
(0.17/0.14)
(0.17)
(0.04/0.25)
(0.09)
(1.96)
69.73
95.57
89.71
91.95/89.20 26
91.86
90.47/99.46
93.76
85,32
LoRA,=3
1.33M
88.38
(1.42)
(0.21)
(1.32)
(0.12/0.18)
(0.29)
(0.23/0.12)
(0.36)
(0.86)
71.48
95.64
91.98
92,39/89.87
92,22
90.35/90.38
94.28
87.77
SoRA
0.91M
89.36
(1.17)
(0.23)
(1.16)
(0.17/0.27)
(0.24)
(0.09/0.12)
(0.06)
(1.56)
69.87
95.53
89.91
92.22/89.66 66
91.79
90.55/90.31
93.46
87.05
LoRA,=16
2.65M
88.62
(0.86)
(0.15)
(1.69)
(0.05/0.04)
(0.16)
(0.10/0.03)
(0.12)
(3.11)
71.93
95.61
92.00
92,37/89.84
92.05
90.34/90.47
94.11
87.41
SoRA,-16
1.78M
89.33
(0.97)
(0.11)
(0.23)
(0.15/0.19)
(0.16)
(0.13/0.04)
(0.07)
(1.08)
Table results and number of parameters of SoRA initialized with different raus on the GLUE benchmark, compared with LoRA of the same rank. The standard deviation is provided in parentheses.


Reviews:
[{"review_id": "MtzqoQ25mX", "reviewer": "Reviewer_kCU2", "summary": "", "questions": "1. Compare with Table 1, why are the results of QQP and MNLI missing in Table 2?  \n2. This paper does not provide a detailed analysis of the computational complexity of SoRA, which may limit its use in resource-constrained environments.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful and constructive comments and suggestions.\n\n> The comparison experiment with the MPOP method is lacking (See Missing References [1]). MPOP method is a lightweight fine-tuning method based on matrix decomposition and low-rank approximation, and it has a strong relevance to the method proposed in the paper. However, there is no comparison with this baseline method in the paper.\n- We appreciate your advice. We have added the reference to MPOP in our revised paper. Yet we do not deem the relation between SoRA and MPOP a “strong relevance”. MPOP alleviates PLM computation cost by compressing the existing weights – a model compression approach; LoRA in contrast freezes those weights and tunes plugged-in modules on downstream tasks – a parameter-efficient fine-tuning (PEFT) methodology. SoRA, as an amelioration of LoRA, is consequently orthogonal to the realm of MPOP and other model compression methods. What is more, SoRA seeks for optimizable rank in the training process, as opposed to fixed-rank decomposition applied in either LoRA or MPOP. In spite of the lack of direct relevance, we acknowledge that MPOP makes use of the same mathematical truth – reduced-rank decomposition of matrices/tensors – as LoRA, and LoRA variants including our work SoRA. For this reason, we mention MPOP in our new version for a more comprehensive literature review. It will be updated in the revised version.\n\n> In Table 4, only the training time of the proposed method and AdaLoRA is compared, lacking the efficiency comparison with LoRA, Bitfit, and Adapter.\n- Thanks for your suggestion. We mainly compare SoRA to AdaLoRA, our primary baseline, to prove that our simpler method could achieve better performance and more efficiency when implementing adaptive rank. As mentioned in the last response (about complexity), the time complexity is approximately the same as LoRA with slightly more overheads. In fact, the gradient computation of SoRA can further benefit from the algorithmic optimization of deep learning frameworks since many of the matrices involved are either row-sparse or column-sparse due to the stop-gradient effect of gating. We provide the training time for all five methods in the table below. Detailed analysis of the training time for SoRA and LoRA can be found in the answer to the fifth question.\n\n| Datasets | AdaLoRA (second) | SoRA (second) | LoRA (second) | Adapter (second) | Bitfit (second) |\n| :------: | :--------------: | :-----------: | :-----------: | :--------------: | :-------------: |\n|   CoLA   |      160.2       |     57.2      |     51.0      |       53.3       |      38.4       |\n|  SST-2   |      491.0       |     433.0     |     379.0     |      413.4       |      297.7      |\n|   MRPC   |       27.3       |     24.8      |     23.2      |       23.6       |      16.5       |\n|  STS-B   |       48.2       |     38.4      |     33.3      |       34.7       |      25.9       |\n|   QNLI   |      1001.0      |     676.3     |     594.9     |      655.0       |      478.9      |\n|   RTE    |       79.8       |     45.1      |     17.9      |       15.3       |      11.3       |\n|   Avg.   |      301.3       |     212.5     |     183.2     |      199.2       |      144.8      |\n\n> In the experimental section of the paper, the standard deviation after multiple experiments is not provided. The improvement brought by SoRA compared with the baseline is quite limited, which may be due to random fluctuations. The author should clarify which effects are within the range of standard deviation fluctuations and which are improvements brought by the SoRA method.\n- Thank you for the reminder. We have supplemented the standard deviation in the table below, which will be included in the next version of the paper. The standard deviation corresponding to SoRA is not significantly different from other methods, indicating that the improvement in performance on the GLUE tasks is indeed attributed to the SoRA method.\n\n- **Table 1**\n\n|   Method   | #Params |        CoLA         |        SST-2        |        MRPC         |                QQP                |        STS-B        |               MNLI                |        QNLI         |         RTE         |     Avg.     |\n| :--------: | :-----: | :-----------------: | :-----------------: | :-----------------: | :-------------------------------: | :-----------------: | :-------------------------------: | :-----------------: | :-----------------: | :----------: |\n| Fine-Tune  |  184M   |    69.21 (2.24)     |    95.64 (0.52)     |    89.22 (0.69)     | 92.05 (0.09)/ 89.31 (0.07)|    91.59 (0.47)     |    89.98 (0.06)/ 89.95 (0.33)     |    93.78 (0.02)     |    82.49 (1.48)     |    87.82     |\n|  Adapter   |  1.41M  |    69.00 (0.91)     |    95.16 (0.46)     |    89.90 (2.10)     |    91.45 (0.18)/ 88.88 (0.40)     | 92.21 (0.33)|    90.11 (0.57)/ 90.11 (0.57)     | 93.79 (0.07)|    82.44 (1.74)     |    87.85     |\n|   Bitfit   |  0.1M   |    68.70 (1.85)     |    94.38 (0.28)     |    87.16 (0.58)     |    87.86 (0.52)/ 84.20 (0.74)     |    89.71 (0.58)     |    87.45 (0.76)/ 87.45 (0.76)     |    91.90 (0.14)     |    76.12 (1.54)     |    85.18     |\n| LoRA (r=8) |  1.33M  |    69.73 (1.42)     |    95.57 (0.21)     |    89.71 (1.32)     |    91.95 (0.12)/ 89.26 (0.18)     |    91.86 (0.29)     |  **90.47 (0.23)/ 90.46 (0.12 )**  |    93.76 (0.36)     |    85.32 (0.86)     |    88.38     |\n|  AdaLoRA   |  1.27M  | 70.86 (1.43)|  **95.95 (0.37)**   | 90.22 (0.40) |    92.13 (0.06)/ 88.41 (0.05)     |    91.39 (0.25)     |     90.27 (0.11)/ 90.3 (0.18)     |  **94.28 (0.11)**   | 87.36 (0.30) | 88.83|\n|    SoRA    |  0.91M  |  **71.48 (1.17)**   | 95.64 (0.23)|  **91.98 (1.16)**   |  **92.39 (0.17)/ 89.87 (0.27)**   |  **92.22  (0.24)**  | 90.35 (0.09)/ 90.38 (0.12)|  **94.28 (0.06)**   |  **87.77 (1.56)**   |  **89.36**   |\n\n- **Table 2**\n\n|   Method    | #Params |     CoLA     |    SST-2     |     MRPC     |            QQP             |     STS-B     |            MNLI            |     QNLI     |     RTE      | Avg.  |\n| :---------: | :-----: | :----------: | :----------: | :----------: | :------------------------: | :-----------: | :------------------------: | :----------: | :----------: | :---: |\n| LoRA (r=1)  |  0.17M  | 68.60 (1.58) | 94.95 (0.20) | 88.24 (1.42) | 91.20 (0.20)/ 88.37 (0.33) | 91.41 (0.29)  | 90.09 (0.16)/ 90.28 (0.14) | 93.35 (0.34) | 81.29 (2.83) | 87.23 |\n| SoRA (r=1)  |  0.12M  | 70.24 (1.10) | 95.14 (0.25) | 89.22 (2.12) | 91.52 (0.14)/ 88.73 (0.22) | 91.41 (0.29)  | 90.08 (0.18)/ 90.41 (0.12) | 93.43 (0.33) | 83.02 (2.47) | 87.85 |\n| LoRA (r=2)  |  0.33M  | 68.93 (1.50) | 95.04 (0.37) | 88.43 (1.06) | 91.59 (0.06)/ 88.87 (0.12) | 91.53 (0.32)  | 90.35 (0.13)/ 90.30 (0.17) | 93.63 (0.37) | 84.17 (1.20) | 87.79 |\n| SoRA (r=2)  |  0.25M  | 70.22 (1.03) | 95.64 (0.09) | 89.71 (0.81) | 91.88 (0.13)/ 89.12 (0.17) | 91.63 (0.30)  | 90.37 (0.06)/ 90.51 (0.08) | 93.78 (0.23) | 85.18 (0.73) | 88.39 |\n| LoRA (r=4)  |  0.66M  | 69.27 (2.22) | 95.55 (0.56) | 89.22 (1.64) | 91.40 (0.17)/ 88.41 (0.27) | 91.69 (0.24)  | 90.36 (0.12)/ 90.49 (0.32) | 93.83 (0.35) | 82.01 (1.76) | 87.74 |\n| SoRA (r=4)  |  0.47M  | 71.05 (0.74) | 95.57 (0.05) | 90.20 (1.60) | 92.06 (0.17)/ 89.44 (0.14) | 91.76 (0.17)  | 90.38 (0.04)/ 90.43 (0.25) | 93.92 (0.09) | 86.04 (1.96) | 88.71 |\n| LoRA (r=8)  |  1.33M  | 69.73 (1.42) | 95.57 (0.21) | 89.71 (1.32) | 91.95 (0.12)/ 89.26 (0.18) | 91.86 (0.29)  | 90.47 (0.23)/ 90.46 (0.12) | 93.76 (0.36) | 85.32 (0.86) | 88.38 |\n| SoRA (r=8)  |  0.91M  | 71.48 (1.17) | 95.64 (0.23) | 91.98 (1.16) | 92.39 (0.17)/ 89.87 (0.27) | 92.22  (0.24) | 90.35 (0.09)/ 90.38 (0.12) | 94.28 (0.06) | 87.77 (1.56) | 89.36 |\n| LoRA (r=16) |  2.65M  | 69.87 (0.86) | 95.53 (0.15) | 89.91 (1.69) | 92.22 (0.05)/ 89.63 (0.04) | 91.79 (0.16)  | 90.55 (0.10)/ 90.31 (0.03) | 93.46 (0.12) | 87.05 (3.11) | 88.62 |\n| SoRA (r=16) |  1.78M  | 71.93 (0.97) | 95.61 (0.11) | 92.00 (0.23) | 92.37 (0.15)/ 89.84 (0.19) | 92.05 (0.16)  | 90.34 (0.13)/ 90.47 (0.04) | 94.11 (0.07) | 87.41 (1.08) | 89.33 |\n\n- **Table 3**\n\n|                          | #Params |     CoLA     |     MRPC     |     STS-B     |\n| :----------------------- | :-----: | :----------: | :----------: | :-----------: |\n| $\\mathbf{W}_{Q,K}$       |  0.80M  | 65.56 (2.24) | 86.43 (0.83) | 72.30 (2.61)  |\n| $\\mathbf{W}_{Q,K,V}$     |  0.69M  | 69.07 (2.17) | 88.24 (1.84) | 90.82 (0.70)  |\n| $\\mathbf{W}_{Q,K,V,A.O}$ |  0.87M  | 71.99 (1.30) | 90.20 (1.83) | 91.71 (0.34)  |\n| **All**                  |  0.77M  | 71.48 (1.17) | 91.98 (1.16) | 92.22  (0.24) |\n\n- The results of the standard deviation indicate that the improvement is not brought by fluctuations. Also, we think the improvement in GLUE tasks at this range could be considered non-trivial.\n\n> Compared with Table 1, why are the results of QQP and MNLI missing in Table 2?\n- QQP and MNLI tasks are very resource-intensive and we feel the results of the other six datasets could sufficiently demonstrate effectiveness at the time of preparing the manuscript for analysis. However, we have supplemented the results in the table below. Thanks for your suggestion!\n\n- **Table 2**\n|   Method    | #Params |     CoLA     |    SST-2     |     MRPC     |            QQP             |     STS-B     |            MNLI            |     QNLI     |     RTE      | Avg.  |\n| :---------: | :-----: | :----------: | :----------: | :----------: | :------------------------: | :-----------: | :------------------------: | :----------: | :----------: | :---: |\n| LoRA (r=1)  |  0.17M  | 68.60 (1.58) | 94.95 (0.20) | 88.24 (1.42) | 91.20 (0.20)/ 88.37 (0.33) | 91.41 (0.29)  | 90.09 (0.16)/ 90.28 (0.14) | 93.35 (0.34) | 81.29 (2.83) | 87.23 |\n| SoRA (r=1)  |  0.12M  | 70.24 (1.10) | 95.14 (0.25) | 89.22 (2.12) | 91.52 (0.14)/ 88.73 (0.22) | 91.41 (0.29)  | 90.08 (0.18)/ 90.41 (0.12) | 93.43 (0.33) | 83.02 (2.47) | 87.85 |\n| LoRA (r=2)  |  0.33M  | 68.93 (1.50) | 95.04 (0.37) | 88.43 (1.06) | 91.59 (0.06)/ 88.87 (0.12) | 91.53 (0.32)  | 90.35 (0.13)/ 90.30 (0.17) | 93.63 (0.37) | 84.17 (1.20) | 87.79 |\n| SoRA (r=2)  |  0.25M  | 70.22 (1.03) | 95.64 (0.09) | 89.71 (0.81) | 91.88 (0.13)/ 89.12 (0.17) | 91.63 (0.30)  | 90.37 (0.06)/ 90.51 (0.08) | 93.78 (0.23) | 85.18 (0.73) | 88.39 |\n| LoRA (r=4)  |  0.66M  | 69.27 (2.22) | 95.55 (0.56) | 89.22 (1.64) | 91.40 (0.17)/ 88.41 (0.27) | 91.69 (0.24)  | 90.36 (0.12)/ 90.49 (0.32) | 93.83 (0.35) | 82.01 (1.76) | 87.74 |\n| SoRA (r=4)  |  0.47M  | 71.05 (0.74) | 95.57 (0.05) | 90.20 (1.60) | 92.06 (0.17)/ 89.44 (0.14) | 91.76 (0.17)  | 90.38 (0.04)/ 90.43 (0.25) | 93.92 (0.09) | 86.04 (1.96) | 88.71 |\n| LoRA (r=8)  |  1.33M  | 69.73 (1.42) | 95.57 (0.21) | 89.71 (1.32) | 91.95 (0.12)/ 89.26 (0.18) | 91.86 (0.29)  | 90.47 (0.23)/ 90.46 (0.12) | 93.76 (0.36) | 85.32 (0.86) | 88.38 |\n| SoRA (r=8)  |  0.91M  | 71.48 (1.17) | 95.64 (0.23) | 91.98 (1.16) | 92.39 (0.17)/ 89.87 (0.27) | 92.22  (0.24) | 90.35 (0.09)/ 90.38 (0.12) | 94.28 (0.06) | 87.77 (1.56) | 89.36 |\n| LoRA (r=16) |  2.65M  | 69.87 (0.86) | 95.53 (0.15) | 89.91 (1.69) | 92.22 (0.05)/ 89.63 (0.04) | 91.79 (0.16)  | 90.55 (0.10)/ 90.31 (0.03) | 93.46 (0.12) | 87.05 (3.11) | 88.62 |\n| SoRA (r=16) |  1.78M  | 71.93 (0.97) | 95.61 (0.11) | 92.00 (0.23) | 92.37 (0.15)/ 89.84 (0.19) | 92.05 (0.16)  | 90.34 (0.13)/ 90.47 (0.04) | 94.11 (0.07) | 87.41 (1.08) | 89.33 |\n\n> This paper does not provide a detailed analysis of the computational complexity of SoRA, which may limit its use in resource-constrained environments.\n- Thanks for the suggestion. Compared to LoRA, SoRA introduces additional computation only in the gate control part, and only during training, not inference. In forward propagation, the gating operation is simply a vector-by-vector element-wise multiplication, which takes merely $\\mathcal{O}(r_{\\text{max}})$ time. As for backward propagation, the change in gradient computation complexity is difficult to explicitly formulate on PLM. But our experiments show that SoRA merely requires 10% more wall-clock training time than LoRA. In fact, the gradient computation of SoRA can further benefit from the algorithmic optimization of deep learning frameworks since many of the matrices involved are either row-sparse or column-sparse due to the stop-gradient effect of gating. Similarly, the increase in training-phase storage will be in the same order $\\mathcal{O}(r_{\\text{max}})$ since the optimizable gate is the only weight additional to LoRA. Our experimental results can also help demonstrate that SoRA is approximately as resource-economical as LoRA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed SoRA can dynamically adjust the intrinsic rank during the adaptation process, thereby improving parameter efficiency. This idea is quite novel.\n2. SoRA introduces an optimizable gate unit that controls the sparsity of the adaptation using the proximal gradient method, thereby expanding the optimization space, this technique makes sense.\n3. The authors demonstrate the effectiveness of SoRA on various natural language processing tasks, proving its superiority in terms of parameter efficiency and performance.\n4. This paper is well written.", "reasons_to_reject": "1. The comparison experiment with the MPOP method is lacking (See Missing References [1]). MPOP method is a lightweight fine-tuning method based on matrix decomposition and low-rank approximation, and it has a strong relevance to the method proposed in the paper. However, there is no comparison with this baseline method in the paper.\n2. In Table 4, only the training time of the proposed method and AdaLoRA is compared, lacking the efficiency comparison with LoRA, Bitfit, and Adapter.\n3. In the experimental section of the paper, the standard deviation after multiple experiments is not provided. The improvement brought by SoRA compared with the baseline is quite limited, which may be due to random fluctuations. The author should clarify which effects are within the range of standard deviation fluctuations and which are improvements brought by the SoRA method.", "questions_to_authors": "1. Compare with Table 1, why are the results of QQP and MNLI missing in Table 2?  \n2. This paper does not provide a detailed analysis of the computational complexity of SoRA, which may limit its use in resource-constrained environments.", "paper_topic_and_main_contributions": "This paper introduces a new approach called sparse low-rank adaptation (SoRA) that allows for dynamic adjustments to the intrinsic rank during the adaptation process, proposing a gate unit optimized with a proximal gradient method to control the sparsity of the adaptation, and demonstrating the effectiveness of SoRA on various natural language processing tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1]. [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://aclanthology.org/2021.acl-long.418) (Liu et al., ACL-IJCNLP 2021)"}, "extra_text": {}}, {"review_id": "k82mVD6nvd", "reviewer": "Reviewer_rN3k", "summary": "", "questions": "- Question A: Table 3 shows the `#Params` of \\mathbf{W}_{Q,K} is 0.8M. However, the `#Params` of All, including the weights of $Q, K, V, A.O, f_1, f_2$ is 0.77M. Does `#Params` means the total parameters of the weight matrices after applying SoRA? If it is, why \\mathbf{W}_{Q,K,V}'s 0.69M is smaller than \\mathbf{W}_{Q,K}'s 0.80M? (It seems that the latex symbols do not render correctly, so I remove all latex $ here for better reading)\n- Question B: Will the source code be public if accepted?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful comments and suggestions.\n> Question A: Table 3 shows the `#Params` of \\mathbf{{W}_{Q,K}} is 0.8M. However, the #Params of All, including the weights of \n is 0.77M. Does `#Params` means the total parameters of the weight matrices after applying SoRA? If it is, why \\mathbf{W}{Q,K,V}'s 0.69M is smaller than \\mathbf{W}_{Q,K}'s 0.80M? (It seems that the latex symbols do not render correctly, so I remove all latex $ here for better reading)\n- We sincerely apologize for not providing a detailed explanation of the specific meaning of `#Params` in the table caption. Throughout the paper, all instances of `#Params` related to SoRA refer to the number of non-zero parameters (parameters that will remain after training) in the Low-rank Adaptation after the Post-pruning, rather than the number of the initial parameter of the Low-rank Adaptation, corresponding to $W_d\\in R^{r_{\\text{max}} \\times  q}$ and $W_u∈R^{p × r_{max}}$. \nThis is because zero parameters will be removed after training in SoRA. \n- In the corresponding experiments of Table 3, to eliminate the influence of parameter quantity on results, it is necessary to ensure consistency in the number of non-zero parameters after the Post-pruning, when we apply SoRA to different weights. To achieve this, we introduce different values of $r_{max}$ in different experiments. For instance, when we apply SoRA to $W_{Q,K}$, $r_{max}$ is set to its maximum, while for applying SoRA to all weights, $r_{max}$ is minimized. In the experiments, we try our best to make the count of non-zero parameters (`#Params`) as consistent as possible, but ensuring uniformity in the `#Params` is extremely difficult, given that $r_{max}$ and the `#Params` do not exhibit a strict functional correspondence.\n- In a word, the `#Params` in Table 3 means the number of parameters that would remain after training, and it will be clearly stated in the next version of the paper. It should be kept as consistent as possible and has no relation to the weights that underwent the application of SoRA.\n\n> Will the source code be public if accepted?\n- Of course! We will include the GitHub repository link for the source code in the paper if accepted. We have proved in Table 2 that SoRA could be a consistently more effective and efficient method compared to the original LoRA. We would be pleased if you find our code useful and decide to utilize it.\n\n> The parameter for controlling model sparsity $\\xi$ is worth exploring, as mentioned in the Limitations. \n- Yes, we feel it will be interesting to conduct relevant analysis (especially theoretical) about model sparsity in parameter-efficient fine-tuning. We will keep working on it. Thanks for you comments again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method SoRA enlights a way to apply the lower-rank adaptation method in practice. The rank analysis illustrated in Section 4.4 and Figure 3 shows that ranks vary across tasks and layers. Hence, compared to the identical rank in LoRA, SoRA has more potential to be extended to more applications.\n2. Parameter-efficient fine-tuning of large language models has become critical as ChatGPT attracted wide attention recently. Incorporating large language models into applications extensively with various downstream tasks requires efficient fine-tuning as this work does.", "reasons_to_reject": "1. The parameter $\\xi$ for controlling model sparsity is worth exploring, as mentioned in the Limitations. More examples are required to understand the potential weakness of the proposed method. But overall, I do not have a strong reason to reject this paper.\n", "questions_to_authors": "- Question A: Table 3 shows the `#Params` of \\mathbf{W}_{Q,K} is 0.8M. However, the `#Params` of All, including the weights of $Q, K, V, A.O, f_1, f_2$ is 0.77M. Does `#Params` means the total parameters of the weight matrices after applying SoRA? If it is, why \\mathbf{W}_{Q,K,V}'s 0.69M is smaller than \\mathbf{W}_{Q,K}'s 0.80M? (It seems that the latex symbols do not render correctly, so I remove all latex $ here for better reading)\n- Question B: Will the source code be public if accepted?\n", "paper_topic_and_main_contributions": "The authors proposed a parameter-efficient method for large language model fine-tuning, SoRA (Sparse Low-Rank Adaptation), which aims to resolve the limitation of the existing method, Low-Rank Adaptation (LoRA (Hu et al., 2021)). This work's contribution includes introducing a flexible way to dynamically adjust the low intrinsic rank with theoretical support. Instead of viewing the rank of the lower-rank matrices $r$ as a hyperparameter, the authors add a gating unit $\\textbf{g}$ between two trainable low-rank matrices (i.e., $B$ and $A$ in the original LoRA paper) with a fixed $r_{max}$. The introduced gating unit is updated by proximal gradient descent with a soft-thresholding function that sets values within $(-\\xi,\\xi]$ to zeros which can be dropped in the post-pruning phase, where $\\xi$ is a changeable threshold for incrementally increasing model sparsity.\nThe results show that the proposed method SoRA can improve LoRA in 7 out of 8 tasks on the GLUE benchmark, retaining ~70% of parameters (Table 1) with the rank $r$ set to 8.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XOQ8waic4f", "reviewer": "Reviewer_mFGF", "summary": "", "questions": "1. How is $r_{max}$ defined? If it's set during initialization, shouldn't hyperparameter optimization be conducted?\n2. Does $\\eta_t$ being larger than 0 or $\\eta_t > 0$ represent the step size at the t-th iteration?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your valuable and constructive feedback.\n> The study falls short in terms of ablation analysis and optimization of hyperparameters.\n- Thanks for your comment. Our method, SoRA, is kind of an integrated strategy for parameter-efficient fine-tuning rather than a modularized method, and it is counter-intuitive to eliminate any parts of it. For example, if we remove the proximal gradient part, the method itself will not hold.\n- We agree that the effect of hyperparameters should be taken into consideration. In Table 2, we prove the results of different r_max. Now we have supplemented the results of two other important hyperparameters, $\\xi$ and $\\eta$ in the following tables.\n- From the following tables, we discover that the performance of SoRA is highly stable with respect to different choices of $\\xi$ and $\\eta$. And for each fixed $\\xi$ and $\\eta$, the variance of performance is rather low. In general, we suggest setting $\\xi$ to 1e-4 level and $\\eta$ around 1e-1 ~ 1e-3. The analysis would be updated in the revision.\n\n| $\\xi$ |     CoLA     |    STS-B     |\n| :---: | :----------: | :----------: |\n| 1e-3  | 63.25 (0.71) | 90.85 (0.53) |\n| 8e-4  | 64.98 (1.25) | 91.12 (0.38) |\n| 5e-4  | 66.94 (0.98) | 91.61 (0.24) |\n| 3e-4  | 68.61 (1.23) | 92.22 (0.24) |\n| 1e-4  | 70.18 (1.05) | 92.01 (0.14) |\n| 8e-5  | 68.78 (1.43) | 92.00 (0.15) |\n| 5e-5  | 71.48 (1.17) | 92.02 (0.18) |\n| 3e-5  | 69.68 (1.94) | 92.18 (0.15) |\n| 1e-5  | 69.65 (1.93) | 92.08 (0.15) |\n\n\n| $\\eta_t$ |     CoLA     |    STS-B     |\n| :------: | :----------: | :----------: |\n|    10    | 69.06 (2.18) | 91.75 (0.41) |\n|    1     | 70.54 (1.15) | 92.02 (0.20) |\n|   0.1    | 71.48 (1.17) | 92.22 (0.24) |\n|   0.01   | 68.78 (2.29) | 92.06 (0.17) |\n|  0.001   | 69.86 (1.54) | 91.83 (0.22) |\n|  0.0001  | 69.70 (1.70) | 92.13 (0.40) |\n\n> The problem definition could benefit from a clearer explanation; the extensive use of symbols can sometimes lead to confusion.\n- We apologize for the potential confusion. We find it hard to narrow down our passage since most of the background introductions and mathematical definitions are vital to readers who are new in this field. However, it is our pleasure to explain our idea more briefly here to you. To be concise, our goal is to develop a LoRA amelioration with optimizable ranks to achieve an automatic perfection of both high performance and low cost. For the symbolic issues, readers may focus on Eq 9 and Eq 10, which are all of our ideas. We will definitely try to reduce the symbol use and make the method clearer, thanks!\n\n> The experiment should compute the mean and variance.\n- Thank you for the reminder.  All results are the mean results already. We have supplemented the standard deviation and displayed it in the table below.We will also include it in the next version of the paper. The standard deviation corresponding to SoRA is not significantly different from other methods, indicating that the improvement in performance on the GLUE tasks is indeed attributed to the SoRA method.\n\n- **Table 1**\n\n|   Method   | #Params |        CoLA         |        SST-2        |        MRPC         |                QQP                |        STS-B        |               MNLI                |        QNLI         |         RTE         |     Avg.     |\n| :--------: | :-----: | :-----------------: | :-----------------: | :-----------------: | :-------------------------------: | :-----------------: | :-------------------------------: | :-----------------: | :-----------------: | :----------: |\n| Fine-Tune  |  184M   |    69.21 (2.24)     |    95.64 (0.52)     |    89.22 (0.69)     | 92.05 (0.09)/ 89.31 (0.07)|    91.59 (0.47)     |    89.98 (0.06)/ 89.95 (0.33)     |    93.78 (0.02)     |    82.49 (1.48)     |    87.82     |\n|  Adapter   |  1.41M  |    69.00 (0.91)     |    95.16 (0.46)     |    89.90 (2.10)     |    91.45 (0.18)/ 88.88 (0.40)     | 92.21 (0.33)|    90.11 (0.57)/ 90.11 (0.57)     | 93.79 (0.07)|    82.44 (1.74)     |    87.85     |\n|   Bitfit   |  0.1M   |    68.70 (1.85)     |    94.38 (0.28)     |    87.16 (0.58)     |    87.86 (0.52)/ 84.20 (0.74)     |    89.71 (0.58)     |    87.45 (0.76)/ 87.45 (0.76)     |    91.90 (0.14)     |    76.12 (1.54)     |    85.18     |\n| LoRA (r=8) |  1.33M  |    69.73 (1.42)     |    95.57 (0.21)     |    89.71 (1.32)     |    91.95 (0.12)/ 89.26 (0.18)     |    91.86 (0.29)     |  **90.47 (0.23)/ 90.46 (0.12 )**  |    93.76 (0.36)     |    85.32 (0.86)     |    88.38     |\n|  AdaLoRA   |  1.27M  | 70.86 (1.43)|  **95.95 (0.37)**   | 90.22 (0.40) |    92.13 (0.06)/ 88.41 (0.05)     |    91.39 (0.25)     |     90.27 (0.11)/ 90.3 (0.18)     |  **94.28 (0.11)**   | 87.36 (0.30) | 88.83|\n|    SoRA    |  0.91M  |  **71.48 (1.17)**   | 95.64 (0.23)|  **91.98 (1.16)**   |  **92.39 (0.17)/ 89.87 (0.27)**   |  **92.22  (0.24)**  | 90.35 (0.09)/ 90.38 (0.12)|  **94.28 (0.06)**   |  **87.77 (1.56)**   |  **89.36**   |\n\n\n- **Table 2**\n\n|   Method    | #Params |     CoLA     |    SST-2     |     MRPC     |            QQP             |     STS-B     |            MNLI            |     QNLI     |     RTE      | Avg.  |\n| :---------: | :-----: | :----------: | :----------: | :----------: | :------------------------: | :-----------: | :------------------------: | :----------: | :----------: | :---: |\n| LoRA (r=1)  |  0.17M  | 68.60 (1.58) | 94.95 (0.20) | 88.24 (1.42) | 91.20 (0.20)/ 88.37 (0.33) | 91.41 (0.29)  | 90.09 (0.16)/ 90.28 (0.14) | 93.35 (0.34) | 81.29 (2.83) | 87.23 |\n| SoRA (r=1)  |  0.12M  | 70.24 (1.10) | 95.14 (0.25) | 89.22 (2.12) | 91.52 (0.14)/ 88.73 (0.22) | 91.41 (0.29)  | 90.08 (0.18)/ 90.41 (0.12) | 93.43 (0.33) | 83.02 (2.47) | 87.85 |\n| LoRA (r=2)  |  0.33M  | 68.93 (1.50) | 95.04 (0.37) | 88.43 (1.06) | 91.59 (0.06)/ 88.87 (0.12) | 91.53 (0.32)  | 90.35 (0.13)/ 90.30 (0.17) | 93.63 (0.37) | 84.17 (1.20) | 87.79 |\n| SoRA (r=2)  |  0.25M  | 70.22 (1.03) | 95.64 (0.09) | 89.71 (0.81) | 91.88 (0.13)/ 89.12 (0.17) | 91.63 (0.30)  | 90.37 (0.06)/ 90.51 (0.08) | 93.78 (0.23) | 85.18 (0.73) | 88.39 |\n| LoRA (r=4)  |  0.66M  | 69.27 (2.22) | 95.55 (0.56) | 89.22 (1.64) | 91.40 (0.17)/ 88.41 (0.27) | 91.69 (0.24)  | 90.36 (0.12)/ 90.49 (0.32) | 93.83 (0.35) | 82.01 (1.76) | 87.74 |\n| SoRA (r=4)  |  0.47M  | 71.05 (0.74) | 95.57 (0.05) | 90.20 (1.60) | 92.06 (0.17)/ 89.44 (0.14) | 91.76 (0.17)  | 90.38 (0.04)/ 90.43 (0.25) | 93.92 (0.09) | 86.04 (1.96) | 88.71 |\n| LoRA (r=8)  |  1.33M  | 69.73 (1.42) | 95.57 (0.21) | 89.71 (1.32) | 91.95 (0.12)/ 89.26 (0.18) | 91.86 (0.29)  | 90.47 (0.23)/ 90.46 (0.12) | 93.76 (0.36) | 85.32 (0.86) | 88.38 |\n| SoRA (r=8)  |  0.91M  | 71.48 (1.17) | 95.64 (0.23) | 91.98 (1.16) | 92.39 (0.17)/ 89.87 (0.27) | 92.22  (0.24) | 90.35 (0.09)/ 90.38 (0.12) | 94.28 (0.06) | 87.77 (1.56) | 89.36 |\n| LoRA (r=16) |  2.65M  | 69.87 (0.86) | 95.53 (0.15) | 89.91 (1.69) | 92.22 (0.05)/ 89.63 (0.04) | 91.79 (0.16)  | 90.55 (0.10)/ 90.31 (0.03) | 93.46 (0.12) | 87.05 (3.11) | 88.62 |\n| SoRA (r=16) |  1.78M  | 71.93 (0.97) | 95.61 (0.11) | 92.00 (0.23) | 92.37 (0.15)/ 89.84 (0.19) | 92.05 (0.16)  | 90.34 (0.13)/ 90.47 (0.04) | 94.11 (0.07) | 87.41 (1.08) | 89.33 |\n\n- **Table 3**\n\n|                          | #Params |     CoLA     |     MRPC     |     STS-B     |\n| :----------------------- | :-----: | :----------: | :----------: | :-----------: |\n| $\\mathbf{W}_{Q,K}$       |  0.80M  | 65.56 (2.24) | 86.43 (0.83) | 72.30 (2.61)  |\n| $\\mathbf{W}_{Q,K,V}$     |  0.69M  | 69.07 (2.17) | 88.24 (1.84) | 90.82 (0.70)  |\n| $\\mathbf{W}_{Q,K,V,A.O}$ |  0.87M  | 71.99 (1.30) | 90.20 (1.83) | 91.71 (0.34)  |\n| **All**                  |  0.77M  | 71.48 (1.17) | 91.98 (1.16) | 92.22  (0.24) |\n\n> How is $r_{max}$ defined? If it's set during initialization, shouldn't hyperparameter optimization be conducted?\n- Yes. $r_{max}$ is a hyperparameter standing for the maximum rank of SoRA. To compare SoRA experiments initialized with different  r_max, please find them in Table 2, in which the “$r$” of SoRA results is exactly the hyperparameter $r_{max}$. From Table 2, two conclusions can be drawn:\n    - For a given rank $r$, SoRA initialized with $r_{max}=r$ can steadily end up with better performance and fewer parameters than LoRA fixed with the same rank $r$.\n    - SoRA is highly robust to a wide range of $r_{max}$ choices. Slightly increasing $r_{max}$ generally introduces better results.\nIn summary, we have demonstrated the tuning results of hyperparameter $r_{max}$ in Table 2, which consequently proves the robustness of SoRA to different  $r_{max}$ choices. Empirically, we suggest setting $r_{max}$ to be 1x $\\sim$ 2x of the commonly used LoRA rank.\n\n> Does $\\eta_t$ being larger than 0 or $\\eta_t > 0$  represent the step size at the $t$-th iteration?\n- You are right. $\\eta_t$ here is exactly the $t$-th step size of proximal gradient descent for the gate unit.\n\n> Your algorithm should also use \\toprule, \\midrule, and \\bottomrule\n- Thank you very much for your suggestion. We have modified the manuscript and will update it in the next version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(+) The author presents a fresh approach known as SORA, designed to enhance the representational strength of LoRA by initializing it at a higher rank.\n(+) The paper provides a lucid theoretical explanation, detailing the underlying principles of SORA.", "reasons_to_reject": "(-) The study falls short in terms of ablation analysis and optimization of hyperparameters.\n(-) The problem definition could benefit from a clearer explanation; the extensive use of symbols can sometimes lead to confusion.\n(-) The experiment should compute the mean and variance.", "questions_to_authors": "1. How is $r_{max}$ defined? If it's set during initialization, shouldn't hyperparameter optimization be conducted?\n2. Does $\\eta_t$ being larger than 0 or $\\eta_t > 0$ represent the step size at the t-th iteration?", "paper_topic_and_main_contributions": "The paper proposes a method called sparse low-rank adaptation (SoRA) for parameter-efficient fine-tuning of large pre-trained language models. SoRA introduces a sparse gating unit that dynamically adjusts the intrinsic rank of the low-rank decomposition matrices during the training process, using a proximal gradient method with sparsity regularization. It improves the representation power and parameter efficiency of the existing low-rank adaptation (LoRA) method, and outperforms other baselines on various natural language understanding tasks. It also enables the exploration of the relationship between the number of non-zero parameters and the memorization and generalization capabilities of the model, using a sparsifying scheduler.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Your algorithm should also use $\\toprule$ , $\\midrule$ and $\\bottomrule$"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is strong consensus that this work is both very exciting and very sound. The reviewers agree that the idea and its intuition is important and insightful and that the experimental results are excellent. As such, I recommend acceptance to the main conference.
Decision: Accept-Main