Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning 
Authors: Qingru Zhang
Minshuo Chen
Alexander Bukharin
Pengcheng He
Yu Cheng
Weizhu Chen
Tuo Zhao
Venue: ICLR 2023 poster (2023)
Keywords: Adaptive budget allocation
Parameter-efficient fine-tuning
Natural language processing

Abstract: Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP. However, common practice fine-tunes all of the parameters in a pre-trained model, which becomes prohibitive when a large number of downstream tasks are present. Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments. These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters. As a consequence, the fine-tuning performance is suboptimal. To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition. Such a novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings. Our code is publicly available at https://github.com/QingruZhang/AdaLoRA .

Published as a conference paper at ICLR 2023
ADAPTIVE BUDGET ALLOCATION FOR PARAMETER- EFFICIENT FINE-TUNING
Qingru Z Minshuo Chen', Alexander Bukharin', Pengcheng He Yu Cheng", Weizhu Chen and Tuo Zhao "Georgia Institute of Technology 'Princeton University Microsoft Azure Al {qingru. zhang, abukharin3, tour me0750@princeton. .ddu edu fpenhe, yu. cheng, weweeeenemmrroottt.ooo
edu
ABSTRACT
Fine-tuning large pre-trained language models on downstream tasks has become important paradigm in NLP. However, common practice fine-tunes all of the parameters in pre-trate pre-trained model, which becomes prohibitive when a large number of downstream tasks are present. Therefore, many fine-tuming methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments. These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters. As a consequence, the fine- tuning performance is suboptimal. To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates the form of singular value decomposition. Such novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings. Our code is publicly available at https: / //g//ttttt thub. com/ Qingružhang/ AdaLoRA.
INTRODUCTION
Pre-trained language models (PLMs) have manifested superior performance in various natural language processing tasks (Devlin et al., 2019; Liu et al., 2019; He et al., 2021b; Radford et al., 2019; Brown et al., 2020). The most common way adapt pre-trained models to down-stream tasks is to fine-tune all the parameters (full fine-tuning, Qiu et al. (2020); Raffel et al. (2020)). However, pre-trained models typically incurs large memory footprint. For example, BERT model (Devlin et al., 2019) consists up to 300 million parameters; T5 (Raffel et al., 2020) comprises up 11 billion parameters and GPT-3 (Brown al., 2020) contains up 175 billion parameters. When building a NLP system upon these pre-trained models, we usually handle multiple tasks that arrive simultaneously (Radford et al., 2019). Given large number of down-stream tasks, full fine-tuning requires that each task maintains a separated copy of large models. The resulting memory consumption is prohibitively expensive.
To address this issue, researchers have proposed two main lines research to reduce the fine-tuning parameters, while maintaining of research focuses on adding small neural modules to PLMs and fine-tune only these modules for each task the base model is kept frozen and shared across tasks. In this way, only small number of task-specific parameters are introduced and updated, greatly enhancing the practicality of large models. For example, adapter tuning (Houlsby et al., 2019; Rebuffi al., 2017; Pfeiffer al., 2020;
improving the performance of PLMs. Specifically, one line
Work was done during Qingru Zhang's intersship at Microsoft Azure AI.

---

Published as a conference paper at ICLR 2023
90.00
$9.99
88.15
88.6
Auc
89.75
MNLI Macched Ace
86
89.50
99.36
84
89.25
9:2
89.00
38.98
80
88.75
78
77.87
88.50
1%
16
W;
1,2.3
4,5.6
10,11.12
(a) Sclected weight matrix
(b) Selected layers
Figure I: Given the total trainable parameters as 0.28M, we apply LoRA only to selected weight matrices (left) selected layers (right) of DeBERTaV3 V--base and compare the fine-tuning performance on MNLI-m. Figure la: we only fine-tune selected type of weight matrix of every transformer layer. including query/key/value projection (W,. Wx, W.). output projection (W,) in self-attention, and two weight matrices (Wi, W22) in two-layer FFNs. In Figure 1b, we apply LoRA to every weight matrix of the selected layers.
He et al., 2022) inserts small neural modules called adapters between the layers of the base model. Prefix tuning (Li & Liang, 2021) and prompt tuning (Lester et al., 2021) attach additional trainable prefix tokens to the input or hidden layers of the base model. These methods have shown achieve comparable performance to full fin-tuning, while only updating less than 1 1% of the original model parameters, significantly releasing the memory consumption.
Another line of research proposes to model the incremental update of the pre-trained weights in a serameter-efficient way, without modifying the model architecture (Zaken et al., 2021; Guo et al., 2020; Hu et al., 2022). Given pre-trained weight matrix W( for example, diff pruning (Guo et al., 2020) models its incremental update A as sparse matrix. Diff pruning initializes A as the same dimension (7()) and then prunes A element-wist based on the magnitude of the entries. As such, diff pruning can increase the parameter efficiency substantially by adaptively retaining important updates and pruning unimportant ones. Nonetheless, diff' pruning has several limitations. First, relies on low-level implementation to speed up the computation of unstructured sparse which is not well supported by existing deep learning frameworks. Therefore, we have to store as a dense matrix during training. Second, it needs to update every entry of A with their gradients and then prune them. This results in similar computational cost as full fine-tuning (Guo al., 2020).
matrices,
To overcome these drawbacks, Hu et al. (2022) propose a method named LoRA, which parameterizes A as a low-rank matrix by the product of two much smaller matrices:
W (0)) + A (0 BA,
where W(0), E pddixda, A R rx2 and E R x with r < (d1,d2), During fine-tuning, only A and B are updated. The rank is chosen to be much smaller than the dimension of W (e.g.. 7 8 when di = d2 = 1024). With less than 0.5% additional trainable parameters, the training overhead be reduced up 70%, compared to full fine-tuning. However, LoRA achieves comparable or even better performance than full fine-tum finettuning (Hu et al., 2022). Mcanwhile, the product of two samll matrices is more friendly implement and deploy than unstructured sparse matrices in diff pruning.
LoRA still has limitations as it prespecifies the rank r of each incremental matrix identical. This ignores the fact that the importance of weight matrices varies significantly across modules and layers when fineituning pre-trained models. To illustrate this point, we present an concrete example in Figure 1. We compare the performance of LoRA when fine-tuning specific modules or layers with the same number of trainable parameters. Figure shows that fine-tuning feed-forward networks (FFN) achieves better performance than self-attention modules. In addition, Figure 1b demonstrates that weight matrices top layers are more important than those bottom layers.
Adding more trainable parameters to the critical weight matrices can lead to better model performance. In contrast, adding more parameters to those less important weight matrices yields very marginal gains or even hurt model performance. Given the parameter budget, i.c., the number of total trainable parameters, always prefer allocate more parameters to those important modules. Distributing the budget evenly to all weight matrices/layers, like LoRA and other methods (e.g.. adapter and prefix tuning), often gives suboptimal performance. To this end, a natural question
How can we allocate the parameter budget adaptively according to importance of modules to improve the performance of parameter-efficent fine-tuning?
Unless specified otherwise, we use (D) to denote any pre-trained weight matrix.
2

---

Published as a conference paper at ICLR 2023
To answer this question, we propose new method AdaLoRA (Adaptive Low-Rank Adaptation), which dynamically allocates the parameter budget among weight matrices during LoRA-alike fine- tuning. Specifically, AdaLoRA adjusts the rank of incremental matrices to control their budget. Critical incremental matrices are assigned with high rank such that they can capture more fine-grained and task-specific information. Less importance ones are pruned to have lower rank to prevent overfitting and save the computational budget. There are some methods to control the rank of matrices in the existing literature of matrix approximation (Cai al., 2010; Koltchinskii et al., 2011; Toh & Yun, 2010). Most of them directly compute singular value decomposition (SVD) of matrix and then truncate the smallest singular values. Such an operation can manipulate the rank explicitly and, more importantly, minimize the difference between the resulting matrix and the original matrix. However, for fine-tuning large models, it becomes prohibitively expensive to iteratively apply SVD for a large number of high-dimensional weight matrices. Therefore, instead of computing SVD exactly, we parameterize A as A = PAQ to mimic SVD. The diagonal matrix A contains singular values while the orthogonal matrices I' and Q represent left/right singular vectors of A. To regularize the orthogonality of P and Q, an additional penalty is added to training loss. Such a parameterization avoids the intensive computations of SVD. Besides, another advantage that only need to drop the unimportant singular values while the singular vectors are maintained. This preserves the possibility of future recovery and stabilizes the training. See detailed comparison to LoRA in Section 3.
Based on our SVD parameterization, A dynamically adjusts the rank of A = PVQ by importance scoring. Specifically, we divide the incremental matrix PAQ into triplets, where each triplet Gj contains the i-th singular value and the corresponding singular vectors. To quantify the importance of triplets, we propose a novel importance metric, which takes account of the contribution of every entry in G, to the model performance (Sanh et al., 2020; Liang et al., 2021; Zhang et al., 2022). Triplets with low importance scores are granted low priority and hence the singular values are zeroed out. Triplets with high importance are retained for fine-tuning. Moreover, we also propose global budget scheduler to facilitate the training. In particular, we start from initial parameter budget, which is slightly higher than the final budget, and then gradually reduce until matching the target. Such scheduler can improve the training stability and model performance. Please see Section 3 for a detailed description of our importance metric and budget scheduler.
We conduct extensive experiments on a wide range of tasks and models to demonstrate the effec- tiveness of AdaLoRA. Specifically, we evaluate the performance using DeBERTa V3-base (He et al., 2021a) on natural language understanding (GLUE, Wang et al. (2019)) and question answering (SQuADvI, Rajpurkar al. (2016) and SQuADv2, Rajpurkar et al. (2018)) datasets. We also apply our methods to BART-large (Lewis et al., 2019) and evaluate the performance on natural language generation (XSum, Narayan al. (2018) and CNll Hermann al. (2015)) tasks. We show AdaLoRA consistently outperforms the baseline, especially under low budget settings. For example, with less than 0.1% trainable parameters of full fine-tuning, AdaLoRA achieves 1.2% F1 improvement on the SQuAD2.0 dataset compared with tate-of-the-ane approaches.
BACKGROUND
Transformer-basel Models. A typical transformer model consists of L stacked blocks, where each block contains two submodules: a multi-head attention (MHA) and a fully connected FFN. Given the input sequence X E Baxd MHA performs the attention function in parallel h heads:
MHA (X) Concat(head )ead;).. head; Softmax (xW,, (XW,( /dn)) dn XW 112
where W, E [ddd is an output projection and W g. WR,, W., E Rdxda are query, key and value projections of head dp typically set to d/h. The other important module is a FFN which consists of two linear transformations with ReLU activation in between: FFN(X) = ReLU((W,, b,)W,, f² + b2. where W,, E dxd, and W,, E Rd xd Finally, residual connection is used followed by a layer normalization (Ba et al., 2016).
Low Rank Adaptation. LoRA (Hu et al., 2022) models the incremental update of the pre-trained weights by the product of two small matrices. For W7)),, x, the modified forward pass is:
h ((O) x + A = (()) x + BAx,
where W( A E xddixd, A E Rr and B E B with r < {d,, d2). A typically adopts a random Gaussion initialization while B is initialized with zero to have A = 0 at the beginning of
3

---

Published as a conference paper at ICLR 2023
row of A, Bug the i-th column of B, and 9; = {Ai,. B.:) } training. We further denote A₁x as the row
as the i-th doublet. Hu et al. (2022) only apply LoRA to query and value projections (i.e, Wg and W;) in the MHAs. He et al. (2022) extend it to weight matrices of FFNs (i.e. W f and W),). leading to the performance improvement. Meanwhile, they propose a unified view of various efficient tuning methods including adapter tuning, prefix tuning and LoRA.
3 ADALORA METHOD
Our method contains two important components: (i) SVD-based adaptation, which formulates the incremental matrices in the form singular value decomposition; (ii) Importance-aware rank allocation, which prunes redundant singular values based on newly-designed importance metric.
3.1 SVD-BASED ADAPTATION
As mentioned in Section 1, we propose to parameterize the incremental updates of the pre-trained weight matrices in the form of singular value decomposition:
W = ( (0) + A ( + PAQ.
where P E Rdixe xr and Q E Rrxda represent the left/right singular vectors of A and the diagonal matrix A E Rexr contains the singular values {A, (AiJisa isig with r & min(di, d2). We further denote G. = [Pus, Ai Q++} as the triplet containing the i-th singular value and vectors. In practice, since A is diagonal,
only need to save it vector in R" A is initialized with zero while P and Q adopt a random Gaussian initialization to ensure A = O at the beginning of training. To enforce the orthogonality of P and Q, i.e., P P = QQ I, we utilize the following regularizer":
R(P, Q) | P 111 + +QQ - I1
In our method, A is iteratively pruned to adjust the rank after each gradient decent step. As mentioned in Section 1, one directly compute SVD for every A manipulate singular values. The computational complexity, however, is O(min(d), d2 )d d2). It becomes extremely expensive to iteratively apply SVD for large number of high-dimensional incremental matrices. In contrast, our parameterization avoids intensive SVD computation, greatly releasing the computational overhead.
We remark that one can also apply structured pruning to LoRA to control the rank (i.e., prune B doublet-wis. wise in (1)). whereas it has the following disadvantages. First, when doublet is measured as unimportant, we have to prune all of its elements. It makes scarcely possible to reactivate the pruned doublets as their entries are all zeroed out and not trained. contrast, AdaLoRA only masks out the singular values based on (3) while the singular vectors are always maintained. It preserves the potential of future recovery for the triplets dropped by mistake. Second, A and B of LoRA are not orthogonal, meaning the doublets can be dependent with each other. Discarding the doublets can incur larger variation from the original matrix than truncating the smallest singular values. Therefore, the incremental matrices are often altered dramatically after each step of rank allocation, which causes training instability and even hurts generalization. To demonstrate this point, we present an ablation study in Section 4.4, which compares AdaLoRA with structured pruning for LoRA.
3.2 IMPORTANCE-AWARE RANK ALLOCATION
We apply the SVD-based adaptation (3) to every weight matrix including WG, Ws. W., W,, and Wr, of each transformer layer. In order to control the budget, we iteratively prune singular values in correspondence to their importance score during the training. For clear reference, we use k to index the incremental matrix, i.e., A = A& Qk for k 1..... ... where n is the number of adapted weight matrices. We denote the i-th triplet of A as Gz = {Pr. An,i, QR and its importance score as Sk.s. We further denote the parameter sets P {P E (A
Q [Q+]% = and training cost as C(P, E, Q). With the regularization (4), the training objective is given by L(P,E, Q) = C(P,E,Q) Q) + R(P, Qk), where > 0 is the regularization
coefficient, At the I-th step, we first take stochastic gradient step to update p(() , and Q!! for k l.....n. Specifically, for (t)
(t)
A
A (1)
1/ C(p(())
(5)
We present the experiments in Appendix I to verify the effectiveness of the regularization.
4

---

Published as a conference paper at ICLR 2023
where 7/ 0 is learning rate. Then, given importance score s the singular values pruned following
A ((+1)_ = - - ), with (( ={ A k.st Sk, is in the top-b()) of 5to),
(0)
(6)
otherwise,
where 8(1) = -1s ( hiskH, contains the importance score of all triplets. Here ((() the budget of remaining singular values at the t-th step, which we explain more Section 3.3. In this way, we leave more budget the incremental matrices of higher priority by pruning the singular values of less important ones. In the sequel, we introduce several options to design the importance score.
Magnitude of singular values is the most straightforward way to quantify the importance of every triplet, i.e., Sk,s | In this way, only the least significant singular values discarded. It minimizes the deviation from the original matrix and further stabilizes the training. Many existing methods use this criterion control the rank of matrix (Cai et al., 2010: Koltchinskii al., 201 Toh & Yun, 2010). However, we remark that such simple metric cannot properly quantify the contribution of parameters model performance.
Sensitivity- based importance is another option for importance scoring, which quantifies the sensi- tivity of parameters to the training loss (Molchanov et al., 2019; Sanh et al., 2020; Liang et al., 2021; Zhang et al., 2022). The prior work, however, leverages the sensitivity to quantify the importance of single entries and applies it for unstructured pruning that prunes weights element-wise. When it turns to our case, we have to design a new metric as the triplets are discarded group-wise. Every entry's 's sensitivity ought to be considered and properly combined to quantify the overall contribution of the triplet to model performance. Therefore, we propase a newly- importance metric account of both the singular value and vectors in triplet GR,e:
1 d2
Sk.i = s(Aaa) + di
1 Σ (PP.j)) +
Σ s(Qu,ss) d2 j=1
where we calculate the mean importance of Ph..: and Qk. such that Sk,, does not scale with the number of parameters in GR Here s(.) is specific importance function for single entries. We can adopt the sensitivity for s-)), which is defined as the magnitude of the gradient-weight product:
1(way) Way V or Cl,
where wiy is any trainable parameter. (8) essentially approximates the change in loss when parameter is zeroed out. If the removal of parameter has a large influence, then the model is sensitive to it and we should retain it (Molchanov et al., 2019; Liang et al., 2021; Zhang et al., 2022).
However, Zhang et al. (2022) point out that the sensitivity in (8) is not yet a reliable importance indi- cator. Such score is estimated the sampled mini batch. The stochastic sampling and complicated training dynamics incur high variability and large uncertainty for estimating the sensitivity with (8). Therefore, Zhang al. (2022) propose to resolve this issue by sensitivity smoothing and uncertainty quantification:
(w),) =3,7 (1-1) (w)r) + +1 -1-11:111 (=-))
(9)
U (1) (wy) =B2 (wwy + (1- B2) (war) 77 (ww)).
(10)
where 0 < 31, 82
(t) the smoothed sensitivity by exponential moving average and U the uncertainty term quantified by the local variation between I(t) and (()) Then they define the importance as the product between 7()) and (1() which can be another option for s(-):
(wxy = 7 (wwy (w))).
(11)
We present detailed ablation study in Section 4.4 to compare the performance of different importance metrics. We find the proposed metric (7) based on the sensitivity variant (11) generally performs best. We summarize the detailed algorithm in Algorithm presented in Appendix A.
5

---

Published as a conference paper at ICLR 2023
3.3 GLOBAL BUDGET SCHEDULER
As mentioned in Section I, adjusting the rank is naturally to control the parameter budget in the context of low-rank adaptation. Hence we define the budget b(t) as the total rank of all incremental matrices, i.e., the number of total singular values. Recall that the budget allocation is iteratively conducted during the fine-tuning. To facilitate the training, we propose a global budget scheduler. Specifically, we start from an initial budget ((() that slightly higher than the target budget ((47 (e.g.. 1.5 times of b(2"). We set the initial rank of each incremental matrix as r = ((0) /n. We warm up the training for & steps, and then follow cubic schedule to decrease the budget b( until it reaches B(4) Finally, we fix the resulting budget distribution and fine-tune the model for y steps. The exact equation for the budget schedule is presented in Appendix B. This allows AdaLoRA to explore the parameter space first and then focus on the most important weights later.
4 EXPERIMENTS
We implement AdaLoRA for fine-tuning DeBERTaV3-bai (He et al., 2021a) and BaRT-large (Lewis et al., 2019). We evaluate the effectiveness of the proposed algorithm on natural language understanding (GLUE, Wang et al. (2019)). question answering (SQuADvI, Rajpurkar et al. (2016) and SQuADv2, Rajpurkar et al. (2018)), and natural language generation (XSum, Narayan et al. (2018) and CNN/DailyMail Hermann et al. (2015)). All the gains have passed significant tests with p < 0.05.
Implementation Details. We use PyTorch (Paszke al., 2019) to implement all the algorithms. Our implementation is based on the publicly available Huggingface Transformers' (Wolf et al., 2019) code-base. All the experiments are conducted on NVIDIA V100 GPUs.
LoRA scales A by a/r where is constant in As result, the magnitude of output can be consistent given different r. It reduces the efforts of retuning learning rate when varying r. Typically is set as 16 or 32 and never tuned (Hu et al., 2022; Yang & Hu, 2020). Following LoRA, we add the same scaling for (3) and fix a as LoRA. Besides, in Algorithm 1, we prune singular values every Ar steps (e.g.. År 100) such that the pruned triplets can still get updated within these intervals and possibly reactivated in future iterations.
Baselines. We compare AdaLoRA with the following methods:
Full fine-tuning is the most common approach for adaptation. During fine-tuning, the model is initialized with pre-trained weights and biases, and all model parameters undergo gradient updates.
Birfit (Zaken et al., 2021) is an effective parameter-efficient fine-tuning method. The method only fine-tunes bias vectors in the pre-trained model.
Adapter tuning (Houlsby et al., 2019; Pfeiffer et al., 2020) inserts two-layer adapters between transformer blocks. We compare with two types of adapter. Houlsby adapter as proposed in Houlsby et al. (2019) is inserted between the self-attention module and the FFN module followed by subsequent residual connection. Recently, Pfeiffer et al. (2020) propose more efficient design with adapters only applied after FFN modules and LayerNorm modules (Ba et al., 2016), which we call Pfeiffer udapter. The number of trainable parameters is determined by the number of layers, the hidden dimension of adapters and the dimension of their inputs.
LoRA (Hu et al., 2022) is state-of-the oftee-of-hheart method for crrameter-effiieen fine-tuning. The method parameterizes incremental updates by two small matrices and only fine-tune them. The number of trainable parameter is controlled by the rank and the number of adapted weight matrices n. Hu et al. (2022) apply LoRA to query and value projections only. In empirical, we find that applying LoRA to all weight matrices, i.e., W,, WR. W.. W,, f, and W,, further improve its performance (Please see Appendix H). Hence, we compare with this generalized LoRA to maximize its performance. We use publicly available implementation to run all the baselines. Please refer Hu al. (2022) and reference therein for details.
'https:/ htwp:hcc.o///ogggggga///eeeeeeee hhtpps://gt wtt....com/iccoooooooooooooooooe

---

Published as a conference paper at ICLR 2023
Table I: Results with DeBERTaV3-buse GLUE development set. The best results on each dataset are shown bold. We report the average correlation for STS-B. Full FT, HAdapter and PAdapter represent full fine-tuning Houlsby adapter, and Pfeiffer adapter respectively. We report mean of 5 runs using different random seeds.
MNLI
SST-2
CoLA
QQP
QNLI
RTE
MRPC
STS-B
All
Method
# Params
m/mm
Acc
Mee
Acc/F1
Ace
Acc
Acc
Corr
Ave.
Full FT
184M
89.90/90.12
95.63
69.19
92.40/89.80
94.03
83.75
89.46
91.60
88.09
BitFit
0.1M
89.37/89.91
94.84
66.96
88.41784.95
92.24
78.70
87.75
91.35
86.02
HAdapter
1.22M
90.13/90.17
95.53
68.64
91.91/89.27
94.11
84.48
89.95
91.48
88.12
PAdapter
1.18M
90.33/90.39
95.61
68.77
92.04/89 40
94.29
85.20
89,46
91.54
88.24
Lo
1.33M
90.65/90.69
94.95
69.82
91. 91.99/89.3 ,38
93.87
85.20
89,95
91.60
88.34
AdaLoRA
1.27M
90.76/90.79
96.10
71.45
92.23/89.74
94.55
88.09
90.69
91.84
89.31
HAdapter
0.61M
10/92/90.
95.30
67.87
91. 91.65/88.99 .5
93.76
85.56
89,22
91.30
87.93
PAdapter
0.60M
10.15/90.28
95.53
69.48
91. 11.62/88.00 86
93.98
84.12
89.22
91.52
88.04
HAdapter
0.31M
90.10/90.02
95.41
67.65
91.54/88.81
93.52
83.39
89.25
91.31
87.60
PAdapter
0.30M
89.89/90.06
94.72
69.06
60.8888.62
93.87
84.48
89.71
91.38
87.90
LoRA,=2
0.33M
90.30/90.38
94.95
68.71
91.61/88.91
94.03
85.56
89.71
91.68
88.15
AdaLoRA
0.32M
90.66/90.70
95.80
70.04
91.78/89.16
94.49
87.36
90.44
91.63
88.86
4.1
NATURAL LANGUAGE UNDERSTANDING
Models and Datasets. We evaluate the fine-tuning performance of DeBERTaV3-bar (He et al., 2021a) using the proposed algorithm. We conduct experiments on the General Language Understand- ing Evaluation (GLUE, Wang et al. 2019) benchmark. The benchmark includes two single-sentence classification tasks, three similarity and paraphrase tasks and four natural language inference tasks. Dataset details are summarized in Appendix D.
Implementation Details. DeBERTaV3 base consists of 183 millions parameters. We compare AdaLoRA with the basclines under different budget levels, for example, given the total trainable parameters as 0.3/0.6/1.2 million. In order to match the parameter budget, we select the hidden dimensions of adapters from (8, 16, 32, 64), set the rank of LoRA {2, 4, 8), and choose the final budget B(T) of AdaLoRA from {144, 288, 576). Then set ((0) as 1.5 times of ((7) for AdaLoRA and select the regularization coefficient from 10.1, 0.3, 0.5). We set the exponential moving average parameters 3, and 32 as their default value 0.85. We select the learning rate from (5 x 10-5,8 8 x 10-5, 1 x 10-1, 2 x 10- More details are presented in Appendix E.
Main results. We compare A with the baseline methods under different budget settings. Table 1 shows experimental results on the GLUE development set. We see that AdaLoRA achieves better or on par performance compared with existing approaches on all datasets under all budget levels. For example, when the parameter budget is 0.3M, AdaLoRAA achieves 87.36% accuracy on RTE, which is 1.8% higher than the best-performing baseline. Besides, AdaLoRA with extreme low budget can often perform better than the baselines with higher budget. For example, AdaLoRA achieve 70.04% Mec. score on CoLA with 0.3M fine-tuning parameters, which is higher than all baseline methods with lager budget (e.g., 0.6M and 1.2M).
4.2 QUESTION ANSWERING
Models and Datasets. We evaluate performance of the proposed algorithm on two question answering (QA) datasets: SQuAD v1.1 (Rajpurkar et al., 2016) and SQuADv2.0 (Rajpurkar et al., 2018), where we use AdaLoRA to fine-tune DeBERTaV3 --bbas. These tasks are treated as a sequence labeling problem, where we predict the probability of each token being the start and end of the answer span. Dataset details can be found in Appendix F.
Implementation Details. We compare AdaLoRA with the baseline methods under different parameter budgets. That is we have the number of trainable parameters as 0.08% 00% 10.1%% /0.3%%% 00.65% of total pre-trained parameters. To match the budget requirements,
select the hidden dimensions of adapters from (4, 8, 16, 32, 64), set the rank r of LoRA as {1, 4, 8} and choose the final total rank b(T) of AdaLoRA from {72, 144, 288, 576}. We set the batch size as 16. We use AdamW (Loshchilov & Hutter, 2019) as the optimizer and set the learning rate 1 x 10 for AdaLoRA. Please refer to Appendix F for more details.
Main Results. Table summarizes experimental results when we fine-tune DeBERTaV3- under 4 different budget settings: 0.08%, 0.16%, 0.32% and 0.65% of total pre-trained parameters. From the
7

---

Published as a conference paper at ICLR 2023
Table 2: Results with DeBERTaV3-berd on SQuAD v1.1 and SQuADv2.0. Here # Params is the number of trainable parameters relative to that in full fine-tuning. We report EM/FI. The best results in each setting shown in bold.
SQuADv1.1
SQuADv2.0
FT
86.0 / 992.7
85.4 / 88.4
# Params
0.08%
0.16%
0.32%
0.65%
0.08%
0.16%
0.32%
0.65%
HAdapter
84.4/91.5
85.3/92.1
86. 1/92.
86.7/92.9
83.4/86.6
84.3/87.3
84.9/87.9
85.4/88.3
PAdapter
84.4/91.7
85.9/92.5
86.2/92.8
86.6/93.0
84.2/87.2
84.5/87.6
84.9/87.8
84.5/87.5
LoRA
86.4/92.8
86.6/92.9
86. 86.7/93.1
86. 3
84.7/87.5
83.6/86.7
84.5/87.4
85.0/88.0
AdaLoRA
87.2/93.4
87.5/93.6 87.5/93.7 87.6/93.7
85.6/88.7
85.7/88.8
85.5/88.6
86.0/88.9
result, we that Ada consistently outperforms existing approaches under all the budget levels in term of two evaluation metrics: exact match (EM) and F1. Notice that the performance of Houlsby adapter and Pfeiffer adapter are notably decreased when we reduce the parameter budget. In contrast, our method shows the consistent performance under different budget levels. For example, AdaLoRA achieves 88.7% F1 on SQuADv2.0 with the smallest budget 0.08%. It is close its performance under the high budget and it is also 1.2% higher than the best-performing baseline.
4.3 NATURAL LANGUAGE GENERATION
Table 3: Results with BART-large XSum and CNN/Daily Mail. Here # Params is the number of trainable parameters relative to that in full tine-tuning. We report R- R-1///L. The best results are shown bold.
# Params
Method
XSum
CNN/DailyMail
100%
Full FT
25.49/22.33/31
44.16/21.28 / 40.90
LoRA
43.95 20.72 35.68
45.03/ 42.15
2.20 %
AdaLoRA
44.72/21.46 36.46
45.00 21.89 42.16
LoRA
43.40 20.20/ 35.20
44.72/21.58 41.84
1.10% %
AdaLoRA
44.35 21.1/21.1/// 36.13
44.96 21.77 42.09
LoRA
43.18/ / 19.89 / 34.92
43.95 / 20.91 / 40.98
0.26% %
AdaLoRA
43.55 20.17 35.20
44.39 / 21.28 41.50
LoRA
42.81 19.68 34.73
43.68 / 20.63 40.71
0.13%
AdaLoRA
43.29 / 19.95 35.04
43.94 / 20.83 40.96
Models and Datasets. To provide a comparison with the state-of-the-art in natural language gener- ation (NLG) tasks, we apply AdaLoRA to fine-tune BART-large model (Lewis et al., 2019). We evaluate model performance on two datasets: XSum (Narayan et al., 2018) and CNN/DailyMail (Hermann al., 2015).
Implementation Details. Similarly as DeBERTav3-base we apply low-rank/SVD-biit adaptation to every weight matrix of both encoder and decoder layers. We report ROUGE 1/2/L scores (R-1/2/L, Lin (2004)). We set the training epochs as 15. For XSum, we set the beam length as 8 and batch size as 64. For CNN/DailyMail,, we set the beam length as 4 and batch size as 32. Please see Appendix G for the detailed configuration.
Main Results. Experimental results are summarized in Table 3, where we compare the fine-tuning performance under four budget levels: the number of trainable parameters is 0.13%, 0.26%, 1.10% and 2.20% of total pre-trained parameters. We see that AdaLoRA achieves better or on par performance compared with the baseline both datasets (XSum and CNN/DailyMail) under all the budget levels. For example, AdaLoRA achieves 21.13 R-2 score when budget level is 1.10%, compared with 19.89 for LoRA.
4.4 ANALYSIS
Different budget levels. Figure illustrates experimental results of fine-tuning DeBERTaV3 --abaee under different budget levels. We see that all the three datasets (MNLI-m. SQuADv2.0 and XSum), AdaLoRA achieves consistent performance improvement under all the budget levels compared with the baseline. The performance gain is more significant when increasing the budget for the XSum
8

---

Published as a conference paper at ICLR 2023
89.0
90.7
88.5-
21.5
90.6
w0.5
68.0-
ROUGE:
21.0
90.4
87.5-
20.5-
90.3
LoRA
67.0-
AdaloKA
90.2
008 0.10 0.32 BU6
1:30 1.5 2.80
00S
0.16
0.32 0065 1:30 2.70 4.05
0T3
0.20
1.1
2.2 455
7.9 125
Parms
Pananss 193)
8 Panuns (9)
(a) MNLI
(b) SQuADv2.0
(c) XSum
Figure 2: Fine-tuning performance under different budget levels. We compare AdaLoRA with the generalized LoRA that applies to every weight matrix.
task, suggesting a high budget can help NLG tasks. Note that on the MNLI and SQuADv2.0 datasets, the performance of AdaLoRA under low budget levels ( 1%) can match the results of high budget settings. For example, AdaLoRA achieves 88.78% FI on SQuADv2.0 when the budget is 0.16%. It close to the performance (8%.8% FI) of the highest budget (4. (4.55%) with more significant gain over the baseline.
Comparison to low-rank parameterization. As mentioned in Section 3.1, one can alternatively prune LoRA doublet-wise to conduct the rank allocation. In this case, the doublets are zeroed out entirely, raising the barrier to reactivate them. It can cause training instability and hurt the generalization when some crucial doublets are pruned by mistake. In Table 4, we compare AdaLoRA with pruning LoRA on three datasets (SST-2, RTE, and CoLA) illustrate this point. We apply the same importance score, budget scheduler and training setups as Section 4.1 for pruning LoRA. We that AdaLoRA outperforms pruning LoRA on all the datasets under all the budget levels.
Table 4: We present two ablation studies in this table: (i) Comparison between AdaLoRA and structured pruning on LoRA. (ii) Comparison of different importance metries for AdaLoRA.
SST-2
RTE
CoLA
Params
0.08%
0.16%
0.65%
0.08%
0.16%
0.65%
0.08%
0.16%
0.65%
Prune LoRA
94.84
94.50
94.95
86.28
86.15
87.00
66.71
69.29
69.57
AdaLoRA
95.52
95.80
96.10
87.36
87.73
88.09
70.21
70.04
71.45
s(.) I(-)
94.61
95.30
95.64
87.36
87.71
88.10
66.71
68.83
70.19
S. |>:
95.41
95.41
95.87
87.00
86.28
88.00
67.67
68.44
70.38
Variants of the importance score. Recall that in Adal.oRA, the importance score is defined by the sensitivity and uncertainty of every entry in the triplet (7). In Table 4, we examine two variants the importance score: (i) changing s(-) in (7) to sensitivity-only; (ii) directly defining S, as AA/). From the results, we can see that the proposed importance score generally performs best. The other two variants can degenerate the model performance up to 0.9%.
The resulting budget distribution. Figure 3 in Appendix C shows the resulting rank of each incremental matrix of DeBERTaV3-bass fine-tuned with AdaLoRA. We find that AdaLoRA always prefers to allocating more budget to FFNs and top layers. Such behavior aligns with our empirical conclusions presented Figure that weight matrices FFN moduels and top layers are more important for model performance. Hence, it validates that proposed importance metric AdaLoRA to focus on crucial modules. Meanwhile, the rank distribution generated by AdaLoRA is consistent across different budget levels, tasks and models. It means the number of remaining parameters is linearly scaled with b(22) and hence we can tune ((2") to control the remaining parameters.
guide
5 CONCLUSION
We propose a parameter-efficien fine-tuning method AdaLoRA that adaptively allocates the parameter budget according to importance scoring. In AdaLoRA, we parameterize the incremental updates of weight matrices the form of singular value decomposition. Then, we dynamically allocate the parameter budget among incremental matrices by manipulating the singular values based on a new importance measurement. Such an a pproach effectively improves the model performance and parameter efficiency. We conduct extensive experiments natural language processing, question answering and natural language generation tasks. Results show that AdaLoRA outperforms existing approaches.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 107.v:660.006450, 2016.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz L Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin ), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
Jian-Feng Cai, Emmanuel Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956-19955 1982, 2010.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Minneapolis, Minnesota, 2019.
Technologies, Volume / (Long and Short Pupers). pp. Association for Computational Linguistics. doi: 10. 1 8653/v1/N19- 1423.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficiee transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- KirkkaKikttttk,, and Graham Neubig, Towards unified view of parameter-effiient transfer learning. International Conference on Learning Representations, 2022. URL https: wpwpeeeveeeeeeeeeeeeeeeeeeeeeeee ew net/ forum ? d=(RDcddAxxok.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with cradient-disentangee embedding sharing. arXiv preprint arXiv:2111.09543; 2021a.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhangn bert with disentangled attention. International Conference on Learning Representutions, 2021b.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiee transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: tettts::/oppnntt /ttp//ppeeeevieeeeeeeeeeeeeeeeee net/ forum? id-nZeVKeeFYYs,
Vladimir Koltchinskii, Karim Lounici, and Alexandre B Tsybakov. Nuclear- norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics, 3965)2330--2399, 2011.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3050 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18853/v 1/2021 1/2021eemnlp 243. URL https: //acl aalahhho ogy- .org/ 2222.. emn 1 p- ma 243.
10

---

Published as a conference paper at ICLR 2023
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to gruuence-t-seqqenee -o-seuuencee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/ICNNPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 4582-4597. Association for Computational Linguistics, 2021. doi: 10. 00.18653/112022..2 awl.oon..cl-o1..35555 URL https: //tpp:::////..// org. 1tt.//10.165//v1/000 acl acl-long 353.
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. Super tickets in pre-trained language models: From model compression to improving generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6524-6538, Online, 2021. Association for Computational Linguistics. doi: 10.18853/v1// a/nn
Chin- Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pp. 74-81, 2004.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 117.11190. 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11264-11272. Computer Vision Foundation / IEEE, 2019. doi: 10. (09/CVPR.2019.00)
Shashi Narayan, Shay В Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint 10X1v:1808.08745 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, eigh-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'chhe Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp.
2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):
1872-1897, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140): 2020.
11

---

Published as a conference paper at ICLR 2023
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10. 0.188533/1/DD46 1264.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, 2018. Association for Computational Linguisties. doi: 10.18653/11/P18-2111
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning, 2020.
Kim-Chuan Toh and Sangwoon Yun. An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of optimization, 66615-640):15, 2010.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-att natural language processing. ArXiv preprint, abs/1910.03771, 2019.
Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint M:Xiv:2011.1452: 2020.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter--fficieet fine-tuning for transformer-base masked language language-modelss models. arXiv preprint arXiv:2106.10193 2021.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and
Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning, pp.
PMLR, 2022.
12

---

Published as a conference paper at ICLR 2023
A THE DETAILED ALGORITHM
Algorithm 1 AdaLoRA
1: Input: Dataset D: total iterations T; budget schedule (b()) hyperparameters 7, 81.
2: for .,..., T do
3: Sample mini-batch from D and compute the gradient VL(P,E,
4: Compute the sensitivity ((t) in (8) for every parameter in {P.E,0)
5: Update 7 as (9) and ( as (10) for every parameter in {P,E, Q);
6: Compute shit by (7), for k 1,...;;.. and i I,...,r;
7: Update p(t+1) p(() P. Q) and @ 2 7/ q. ((P.E Q):
8: Update A (++1) T(A 1/ C(P, Sk given the budget b(0)
9: end for
10: Output:
B GLOBAL BUDGET SCHEDULE
As mentioned in Section 3.3, we propose global budget scheduler gradually decrease the budget B(t) following cubic schedule. The detailed equation is given follows:
((
0st< t
b09)
((T) + (b(0) - b(7)) (1 (1-7-1 ((T)
144-4
3
4SI<T-1y
(12)
O.W.
C
THE BUDGET DISTRIBUTION
12
W
/22
2
10
0
10
W
f
The final rank
W
8
12
12
12
W
10
6
10
11
11
11
12
12
11
9
W
5
5
10
11
12
12
12
2
W
2
3
4
5
6
7
8
9
10
11
12
Layer
Figure The resulting rank of each incremental matrix when fine-tuning DeBERTaV3-bar on MNLI with AdaLoRA. Here the z-axis is the layer and the y-axis represents different types of adapted weight matrices.
Figure shows the resulting rank of each incremenal matrix when fine-tuning DeBERTaV3. on MNLI with AdaLoRA. We can see that AdaLoRA allocates more parameter budget to weight matrices of 8, 9, 10, and layers, especially for Wn, and W,. Such behavior aligns with our conclusions presented in Figure 1.
D GLUE DATASET STATISTICS
We present the dataset statisties of GLUE (Wang et al., 2019) in the following table.
13

---

Published as a conference paper at ICLR 2023
Table 5: Summary of the GLUE benchmark.
Corpus
Task
#Train
#Dev
#Test
#l.abel
Metrics
Single-Sentence Classification (GLUE)
CoLA
Acceptability
8.5k
lk
lk
2
Matthews corr
SST
Sentiment
67k
872
1.8k
2
Accuracy
Pairwise Text Classification (GLUE)
MNLI
NLI
393k
20k
20k
3
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Accuracy/F1
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy/Fi
QNLI
QA/LII
108k
5.7k
5.7k
2
Accuracy
Text Similarity (GLUE)
STS-B
Similarity
7k
1.5k
1.4k
1
Pearson/Spearman corr
E
NATURAL LANGUAGE UNDERSTANDING
E.
BUDGET CONFIGURATION
For each budget level, we tune the final budget ((7) for AdaLoRA, the rank r for LoRA, the hidden dimension for two adapters to match the budget requirements.
Table 6: Detailed budget setup for GLUE benchmark.
# Params
Houlsby Adapter (d)
Pfeiffer Adapter (d)
LoRA (r)
AdaLoRA (b(T)
1.2M
32
64
576
0.6M
16
32
4
288
0.3M
8
16
2
144
Alternatively, we can also set the final average rank F(T) = B(T) /n for AdaLoRA to control the budget, which set as 2, 4, and 8 given the final budget as 144, 288, and 576 respectively. Then we select the initial rank r from {4, 6, 12) for the final average rank (2, 4, 8) respectively.
E.2 TRAINING DETAILS
We tune the learning rate from {8 10-5, x 10-5, 10-5, x 10 3 x 10 10-1,8 10-1,1 1 x 10-3) and pick the best learning rate for every method. For each dataset, the batch size set as identical for every method.
Table 7: Hyper-parameter setup of AdaLoRA for GLUE benchmark.
Dataset
learning rate
batch size
# epochs
AT
t,
MNLI
5 10-4
32
0.1
8000
100
50000
RTE
1.2 10-3
0.3
1800
QNLI
1.2 10
32
5
0.1
2000
100
8000
MRPC
10-3
0.1
1800
QQP
5 x 10
32
5
0.1
8000
100
25000
SST-2
8 10-4
32
24
0.1
6000
100
22000
CoLA
5 10
32
25
0.5
800
10
3500
STS-B
2.2 x 10
32
25
0.1
800
10
2000
QUESTION ANSWERING
F.I BUDGET CONHIGRATION
Given the budget, we control the trainable parameters for each method the following table.
14

---

Published as a conference paper at ICLR 2023
Table 8: Detailed budget setup for question answering.
# Params
Houlsby Adapter
Pfeiffer Adapter
LoRA
AdaLoRA BTT//((1// /r
d
0.65%
32
64
576/8/ / 12 288 /4/6
0.32%
32
0.16%
16
144 144/2/4
0.08%
8
72/1/2
F.2 TRAINING DETAILS
We set the batch size as 16. We select the learning rate from (8 x 10-6, x 10-6, x 10-6, x
10-1,3 3 x 10-4, 10-4, 8 x 10-4, x 10=3} and pick the west-performing learning rate for every method. The configuration of AdaLoRA is listed in the following table.
Table 9: Hyper-parameter setup of AdaLoRA for question answering tasks.
Dataset
learning rate
batch size
# epochs
7
Ar
f,f
SQuADv1.1
x 10-3
16
10
0.1
5000
100
25000
SQuADv2.0
10-3
0.1
5000
100
50000
F.3 DATASET
The statistics question answering datasets are summarized in Table 10.
Table 10: Statistics of the SQuAD dataset.
# Train
# Validation
SQuAD v1.1
87,599 130,319
10,570 11,873
SQuAD v2.0
G NATURAL LANGUAGE GENERATION
G.1 BUDGET CONHIURRTIION
Given the budget, control the trainable parameters for each method the following table.
Table Detailed budget setup for summarization tasks.
# Params
Houlsby Adapter
Pfeiffer Adapter
LoRA r
AdaLoRA (T) fr
d
0.65%
576/8
0,32%
16
32
288 /
0.16%
16
144 14/4
0.08%
8
72/1/2
G.2 TRAINING DETAILS
We set the batch size as 16. We select the learning rate from {8 x 10-5, 10-5,3 10 10 10-4,3 3 x 10-4, 4 x 10 10-4, 8 x 10-4, x 10-33 and pick the best persfrrrfonnng learning rate for every method. The configuration of Adal A is listed in the following table.
15

---

Published as a conference paper at ICLR 2023
Table 12: Hyper- papper-arammttr setup of AdaLoRA for summarization tasks.
Dataset
learning rate
batch size
epochs
t;
Ar
fr
XSum
5 x 10 10=4
64
25
0.1
6000
100
50000
CNN/DailyMail
5 x 10
32
15
0.1
5000
100
85000
H ABLATION STUDY FOR LORA
As mentioned in Section 4, we find that the performance of LoRA can be further improved when applying every weight matrix, compared fine-tuning Wy and We only (Hu al., 2022). This observation aligns with the empirical results of He et al. (2022). In Table 13, we follow the same training configuration as Section 4.1 and present an ablation study illustrate this point.
Table 13: We compare the fine-tuning performance when apply LoRA to every weight matrix or Wg. W, only. The parameter budget fixed as 0.3M. We report accuracy for QQP and MRPC, accuracy(m) for MNLI, and average correlation for STS-B.
MNLI
QQP
CoLA
RTE
QNLI
SST-2
MRPC
STS-B
LoRA (W,, Ws)
89.80
90.48
67.04
83.75
93.69
94.84
90.20
91.05
LoRA (all)
90.30
91.61
68.71
85.56
94.31
94.95
90.44
91.68
ORTHOGONAL REGULARIZATION
109
10-1
10-1
113
10-1
10-2
/m
100
my
10-1
10-7
10
20000
40000
20000 Iscrations
40000
lierations
(a) P of W. at the first layer.
(b) Q of W. the first layer.
10a
10-
-1l
10-2
Maw yno yom
10-2
WM
hmml pw hum
1000
10
10-3
10
10
400000
lterations
Iserations
(c) W5a the first layer.
Q of Wia the first layer
Figure 4: We plot the 1P P T1I2 and IIQQ 7|22 when fine-tuning DeBERTaV3-ban SST-2.
To verify the effectiveness of (4), we plot P ZJF and J!QQT - / to show whether P and Q are regularized to be orthogonal. We fine-tune a DeBERTaVas V--baae model on SST-2 with AdaLoRA and follow the same training configuration along the training horizon. From Figure 4, we can see that two regularization terms can be optimized to very small value (e.g., 0.001) the beginning of training. Therefore, both P and can be
Section 4.1. We set y 0.1 and plot the two terms
16

---

Published as a conference paper at ICLR 2023
enforced be orthogonal quickly during the initial warm-up of AdaLoRA. It ensures that the triplets are not dependent with each other.
J THE ROLE OF TWO COMPONENTS
We remark that both two components of our method SVD adaptation and adaptive budget allocation, play vital roles for the performance gain. To demonstrate it, we compare AdaLoRA with the following variants: (i) SVD-LoRA: fine-tuning only with the proposed SVD-based adaptation in (3) and (4); (ii) LoRAzegu: LoRA with orthogonal regularization (4) on A and B; (iii) AdaLoRA, =
AdaLoRA without orthogonal regularization (4). Table 14 present the results when fine-tuning that fine-tuning only with SVD adaptation shows an improvement over LoRA but cannot match the performance of Adal Meanwhile, without SVD orthogonal regularization, the performance of AdaLoRA can degenerate. These results validate that both components contribute to the model performance.
DeBERTaVe-bar
SST-2 and MNLI. We
Table 14: We present ablation studies about SVD-based adaptation, orthogonal regularization, and budget allocation in this table. For MNLI, we report the average score of m/mm acc.
SST-2
MNLI
Params
0.08%
0.16%
0.32%
0.65%
0.08%
0.16%
0.32%
0.65%
LoRA
94.38
94.95
94.95
90.19
90.34
90.57
LoRAregu
94.61
94.72
94.61
90.30
90.40
90.66
SVD-LORA
95.33
95.18
95.07
95.53
90.28
90.25
90.52
90.62
AdaLoRA, =0
95,41
95.10
95.30
95.10
90.37
90.34
90.56
90.43
AdaLoRA
95.64
95,80
96.10
96.10
90.65
90.68
90.66
90.77
COMPARISON OF TRAINING COST K
We compare the training cost between AdaLoRA and LoRA in the following table. We use two methods to fine-tune DeBERTaV3-bass on a single NVIDIA V100 GPU. We do training only and set hyperparameters, e.g., batch size and training epochs, the same in Section
Table 15: Comparison of practical training cost between AdaLoRA and LoRA.
Dataset
Param
Method
GPU Mem
Time/epoch
LoRA
11.094 GB
105 min
0.08%
AdaLoRA
11.104 GB
116 min
LoRA
11.098 GB
105 min
MNLI
0.16% 1
AdaLoRA
11.110 GB
117 min
LoRA
11.128 GB
105 min
0.65%
AdaLoRA
11.188 GB
117 min
LoRA
13.138 GB
60 min
0.08 %
AdaLoRA
13.148 GB
71 min
LoRA
13.142 GB
61 min
SST-2
0.16%
AdaLoRA
13.164 GB
71 min
LoRA
13.170 GB
61 min
0.65% %
AdaLoRA
13.226 GB
71 min
Table 1.5 shows that MARVEL incurs 11% additional training time on MNLI and 16% on SQuADv2 under different budgets. The memory footprint of two methods are quite close. Such results demonstrate that MARVEL does not incur significant training overheads. The reason behind is that we only evaluate the importance score for small incremental matrices PAQ. Their total number of parameters is usually less than 1% of pre-trained weights. Therefore, it does not lead to significant computational cost to update the importance scores of these well-structured small matrices, compared to nowaardbackwaad pass of full model.
17


Reviews:
[{"review_id": "mP6O6iKHrPx", "reviewer": "Reviewer_z2BR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a space-efficient method for finetuning large LMs. The proposed method is a simple extension of the LoRA method: (1) instead of decomposing the update matrix into general A*B, the authors use SVD matrices, which allows reducing the matrix size rather partially than all or none. (2) They propose an importance ranking method for deciding which SVD values to prune in which matrix. The overall method obtains very strong results: it outperforms the LoRA method, as well as several baselines, and even the baseline vanilla fine-tuning.\n", "summary_of_the_review": "A simple yet effective extension of the LoRA method for space-efficient fine-tuning of LMs. The proposed method performs very well on several benchmarks. There are a few clarity issues and some methodological concerns, such as an important missing baseline. Overall I am supportive of accepting this paper.", "strength_and_weaknesses": "Strengths:\n- A simple and very space-efficient FT method. \n- Very strong empirical results. \n- The method is quite simple (though there seem to be a lot of important details)\n\nWeaknesses: \n- An important ablation seems missing: running LoRA with the proposed SVD decomposition rather than the AB one. This would help tease apart the two proposed contributions (the SVD part and the importance part). \n\n- The writing can be repetitive at times, but still a bit cryptic in other places. In particular, I had to read the SVD part several times to understand the differences between the method and the LoRA method. I still have a few questions, see below. \n- The authors only report validation results as far as I understand. Reporting such results is important for reproducibility, but does not replace reporting test results. Given the large number of small decisions made in developing the proposed method, it is particularly important to evaluate it on a truly held-out set.\n\nQuestions:\n- How much does the initial stage (Sec. 3.3) increase the cost of fine-tuning (e.g., in runtime)?\n\n- \"All the gains have passed significant tests with p < 0.05.\" -> Which statistical test was used?\n\n- \"such that the pruned triplets can still get updated within these intervals and possibly reactivated in futur iterations.\" -> maybe I am missing something: once the warmup stage is over, and the budget is fixed, do importance scores change? In my understanding, the importance of a given singular value cannot change since it is always pruned. Or am I missing something?\n\n- Table 1: Why does the bitfit baseline use far fewer parameters than all the others? \n\n- What are the NLG tasks? Summarization? What are the evaluation measures? These are very important details, NLG is not a task.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fXVV9XSNlY", "reviewer": "Reviewer_97di", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a parameter-efficient fine-tuning method for pre-trained language models. The method first approximates the gradient update of weight matrics by low-rank factorization and uses a sensitivity-based importance score to prune unimportant triplets. Experiments on NLU, QA, NLG tasks show the effectiveness of the proposed method compared to other methods with the same amount of fine-tuning parameters.", "summary_of_the_review": "The motivation for adjusting the rank of weight matrics is not very clear. Experimental results in terms of FLOPS (or training time) are needed to show the efficiency of the proposed method.\n\n------------------\nThe score is updated after the responses.", "strength_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow. The research problem is important as pre-training LMs are growing in scale, resulting in the difficulty in fine-tuning all model parameters. \n- The method is simple, straightforward, and effective. \n\nWeaknesses:\n- This paper mainly compares methods under the same amount of fine-tuning parameters. There is no comparison in terms of FLOPS/training time.\n- Besides, it's also unclear about the speedup gain v.s. the overhead of computing the sensitivity score w.r.t. the whole training process. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PiGyee_TmpJ", "reviewer": "Reviewer_byZf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes MARVEL, a method for parameter-efficient fine-tuning on big pre-trained language models. MARVEL is largely based on LoRA, a method that fine-tunes low-rank decomposition on parameter matrices. MARVEL uses SVD and dynamically allocate rank to each layer as resources, and it outperforms LoRA consistently with both DeBERTa and BART on natural language understanding and natural language generation respectively.", "summary_of_the_review": "This paper proposes MARVEL, a method for parameter-efficient fine-tuning on big pre-trained language models. The topic is of significance. The method is based on LoRA, so technical contribution is incremental. However, the paper includes very solid experiments showing that MARVEL outperforms LoRA and other baselines consistently. There is a minor soundness problem: some experiments could be unfair against LoRA because training time is not compared. This paper is overall solid so I would like to give a borderline accept recommendation.", "strength_and_weaknesses": "## Strengths\n\n1. Parameter-efficient fine-tuning is a topic of significance in the application of big pre-trained LMs to real-world problems.\n\n2. MARVEL uses SVD and dynamically allocate rank to each layer as resources. It is a novel technical contribution.\n\n3. The paper compares MARVEL with multiple baselines on very well-established NLP models (DeBERTa, BART) and benchmarks (GLUE, SQuAD, CNN/DM, XSum). In these experiments, MARVEL outperforms other methods, especially LoRA, consistently.\n\n4. This work includes an analysis section, where various design choices of MARVEL are compared. It helps readers understand how MARVEL works. The figure in Appendix B is also provides great insight on how different layers of big LM contributes to its downstream performance.\n\n## Weaknesses\n\n1. The method, MARVEL, is based on LoRA. Most important comparisons in this paper is comparing MARVEL against LoRA. The contribution of MARVEL is mostly incremental and empirical.\n\n2. MARVEL introduces many new hyperparameters compared with LoRA (see Section 4.1). How could you rule out the possibility that a **larger hyperparameter search space** is what leads to a better performance? Also, MARVEL is considerably more complicated than LoRA, so fine-tuning with MARVEL could be slower than fine-tuning with LoRA. However, the **training time** is not compared in this paper either. To address these two concerns, could you show the time required for each run (including full hyperparameter sweep) of both MARVEL and LoRA?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "N0IOWiXhAG", "reviewer": "Reviewer_f1zB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach for adaptively allocating fine-tuning parameters during transfer of pre-trained models to downstream tasks.  The authors propose an SVD inspired decomposition of the adapter matrices and develop various importance scores to assess which triplets in the SVD decomposition are removable. This allows adaptively tuning the ranks of the adapter matrices across layers. The authors demonstrate empirical gains from using their method.", "summary_of_the_review": "I think the ideas in the paper are solid. However, there are several missing empirical elements that I would need to be fully convinced that the method works as advertised (significantly improves downstream performance at same overall budget compared to simpler non-adaptive methods).\nConditioned on adequate responses being made to the weaknesses outlined above, I would be willing to raise my score.\n\n****** \n\nScore updated after responses. ", "strength_and_weaknesses": "**Strengths**\n1. The idea of adaptive computation budget allocation is interesting and highly relevant for using pre-trained models for downstream tasks in a memory efficient fashion\n2. The experiments, though limited in some respect, are quite extensive in others. \n\n**Weaknesses**\n1. Missing Baselines / Experiments\n     1.  MAM [1] seems to be the current state of the art but this was not compared against though it is referenced in the paper.\n     2.  What is the training time cost of  Marvel compared to the non-adaptive methods ? It seems that Marvel adds non-trivial overhead compared to the simpler non-adaptive methods.  If the training overhead is significant, it might hinder Marvel's applicability, just like the authors mention with Diff Pruning. \n2. Some Missing ablations \n    1. Impact of orthogonality constraint - would be good to see ablations with and without this constraint\n    2. Is the success of the gradient-weight product (as a sensitivity measure) possibly coming from the exponential smoothing ? Was exponential smoothing applied to the simple singular value magnitude measure (?)\n    3.  LoRA version as presented has no regularization applied to the learned AB matrices.  It is possible that the gain produced by the current method is coming from “effective regularization effect\"  of the global budget scheduler ?  Could LoRA be run with varying regularization (say L2) on the AB matrices  to ensure that such a regularization effect isn't indeed the cause of gains.\n\n\n**Nits**\n1. Figure 1\n    1. We are not giving the reference result for full fine-tuning so it’s hard to put this figure in context\n    2. Would be good to also include results for when budget is split equally across all layers / types of parameters\n\n[1] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards\na unified view of parameter-efficient transfer learning. In International Conference on Learning\nRepresentations, 2022. URL https://openreview.net/forum?id=0RDcd5Axok.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CzYjUF6FC6", "reviewer": "Reviewer_qo5b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "* This paper proposes a novel parameter-efficient fine-tuning method called MARVEL, which adaptively allocates the parameter budget among weight matrices according to their importance score. Specifically, it proposes to use a learned approximation of SVD decomposition, and prunes out singular values of unimportant updates. Learned approximation of SVD circumvent intensive exact SVD computations, and importance of singular values is defined by a novel metric correlating to the contribution to the model performance. It additionally proposes a global budget scheduler to facilitate training process.\n", "summary_of_the_review": "This paper presents a novel method that achieves better efficiency without much loss of accuracy, compared to a comprehensive set of baselines, on a variety of tasks. However, the novelty of the method is somewhat limited and there can be potential drawbacks such as longer training time. So I recommend weak acceptance.\n", "strength_and_weaknesses": "* Strength\n    * The evaluation is extensive. The paper compares with a comprehensive set of baselines and evaluates on a variety of tasks (language generation, GLUE, and question answering).\n    * The paper achieves great experimental results, e.g. achieving 1.2% F1 improvement on Squad2. With less than 0.1% trainable parameters.  \n    * This paper proposes a novel approach for parameter-efficient finetuning, and can have wide applications.  \n* Weakness\n    * It will be great to also provide numbers of training/inference memory/time. Even though it's faster than SVD, is the training time longer than other methods? What are some trade-offs?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "0ruzQ8oVRz", "reviewer": "Reviewer_9ZWX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes MARVEL, a parameter-efficient fine-tuning method. MARVEL parameterizes the incremental updates of the pre-trained weight matrices in the form of SVD for parameter-efficient fine-tuning. Besides, it allocates the parameter budget adaptively according to the importance of modules to improve the final performance. The authors conduct extensive experiments on Natural Language Understanding (NLU), Question Answering (QA) and Natural Language Generation (NLG) tasks. Results show that MARVEL outperforms existing approaches like BitFit, LoRA, etc.", "summary_of_the_review": "In general, the paper is well-written and techniquely-reasonable. Extensive experiments demonstrate the superiority of the proposed method.", "strength_and_weaknesses": "*Strengths*\n\n1. The paper is well-organized and easy to follow.\n2. The proposed algorithm is well-motivated. Specifically, previous work like LoRA treats all parameters equally, but actually some parameters contribute more to the final performance and thus should be updated in higher priority when the budget is limited. Therefore, the paper proposes to adaptively allocate the parameter budget among weight matrices according\nto their importance score, which improves the model performance.\n3. Experiments conducted on various tasks including NLU, QA, and NLG demonstrate that MARVEL can achieve better performance compared with baselines without adaptive parameter budget allocation.\n\n*Weaknesses*\n\n1. The experiments are only based on two backbones (DeBERTaV3-base, BART-large). It would be better to provide more results on other backbones, especially on large-scale models, e.g., T5-3B, to show the generalization of the proposed method to larger models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster