Enhancing Parameter Efficiency and Generalization in Large Models: A Regularized and Masked Low-Rank Adaptation Approach
Authors: Yuzhu Mao
Zihao Zhao
Siqi Ping
Yang Liu
Wenbo Ding
Venue: Accepted by TMLR ()

Abstract: Large pre-trained models, such as large language models (LLMs), present significant resource challenges for fine-tuning due to their extensive parameter sizes, especially for applications in mobile systems. To address this, Low-Rank Adaptation (LoRA) has been developed to reduce resource consumption while maintaining satisfactory fine-tuning results. Despite its effectiveness, the original LoRA method faces the challenge of suboptimal performance. This paper investigates the intrinsic dimension of the matrix updates approximated by the LoRA method and reveals the performance benefits of increasing this intrinsic dimension. By employing regularization and a gradient masking method that encourages higher intrinsic dimension, the proposed method, termed Regularized and Masked LoRA (RM-LoRA), achieves superior generalization performance with the same or lower trainable parameter budget compared to the original LoRA and its latest variants across various open-source vision and language datasets.

Published in Transactions on Machine Learning Research (01/2025)
Enhancing Parameter Efficiency and Generalization in Large
A Regularized and Masked Low-Rank Adaptation Approach
Models:
Yuzhu Mao
my::20@mais..siive Lsinghus. edu. zheo-zh21 @mails tsinghua edu.co cn edu.cn
Zihao Zhao
Siqi Ping
Tsinghua- Barnghua-Berkeley Shenzhen Institute
Tsinghua Shenzhen International Graduate School, Tsinghua University
Yang Liu
FiayO8@@airtssinghuee edw...
Institate for AI Industry Research (AIR), Tsinghua Universily Shanghai Artificial Intelligence Laboratory Wenbo Ding"
Binng.wennboss..iinnggueuu edu.cr
Tsinghua- Berkeley Shenzhen Institute
Tsinghua Shenzhen International Graduate School, Tsinghua University Shanghai Artificial Intelligence Laboratory
Reviewed on OpenReview: https: // openreview. net/ per/ffrmm? 2d=2gOhPLABff
Abstract
Large pre-trained models, such large language models (LLMs), present significant resource challenges for fine-tuning due to their extensive parameter sizes, especially for applications in mobile systems. To address this, Low-Rank Adaptation (LoRA) has been developed to reduce resource consumption while maintaining satisfactory fine-tuning results. Despite its effectiveness, the original LoRA method faces the challenge suboptimal performance. This paper investigates the intrinsic dimension of the matrix updates approximated by the LoRA method and reveals the performance benefits of incrcasing this intrinsic dimension. By employing regularization and a gradient masking method that encourages higher intrinsic dimension, the proposed method, termed Regularized and Masked LoRA (RM-LoRA),, achieves superior generalization performance with the same lower trainable parameter budget compared to the original LoRA and its latest variants across various open-source vision and language datasets.
1 Introduction
Large pre-trained models, such as large-scale language models (LLMs), have showcased remarkable performance across varicty of tasks in computer vision and natural language processing (Zhang et al.] 2022 Brown
et al. 2020 Touvron al. 2023 Ouyang al. 2022). Nonetheless, fine-tuning these models for specific downstream tasks often presents substantial resource challenges due to their extensive parameter sizes. In
this context, parrameter-efficcentt fine-tuning (PEFT) methods have been extensively explored to alleviate resource consumption while prescrving or enhancing fine-tuned model performance (Zaken et al. 2022
Hu
et al.
2021 Li & Liang 2021: Lester et al.
2021 Vu et al.) 2022 Guo el al. 2021 Zhang et al. 2023 Liu
et al.
2022a Houlsby et al. 2019 Liu et al. 2022b Sung et al. 2021 2022 Mao et al. 2022 Lee et al. 2020
Among these mothods,
Adaptation (LoRA), which involves freczing the pro-trained weights and
approximating updates in weight matrices using the multiplication of low-rank matrices, has emerged
* *Corresponding Author

---

Published in Transactions on Machine Learning Research (01/2025)
promising approach to balance computational efficiency and task performance during the fine-tuning process (Hu et al. 2
Despite its effectiveness, LoRA fine-tuning encounters challenges in determining the optimal size of LoRA matrices for a given model and task. On one hand, excessively small matrices, with limited number of trainable parameters, inevitably harm training convergence and generalization performance. On the other hand, large matrices introduce redundant trainable parameters, which could be reduced to enhance parameter efficiency. Moreover, some studies have indicated that large LoRA matrices may exacerbate overfitting, redundant parameters primarily contribute to training accuracy rather than test accuracy (Qiang et al. 2024 Karimi Mahabadi et al. 2021
Several approaches have been proposed to determine or adaplively adjust. the size of LoRA matrices, often referred to as the LoRA rank R, for improved efficiency and generalization (Valipour et al. 2023 Benedek & +Wolf 2024 Kopiezko et al. 2023 Ding et al. 2023 Zhang et al., (023a). However, none of these methods investigate the intrinsic dimension r of the approxinated matrix update AW -B given by the product of low-rank LoRA matrices A and B. This intrinsic dimension r, instead of the previously studied LoRA rank R, has been proven to play crucial role in LoRA fine-tuning. Specifically, it has been theoretically demonstrated that for fully connected neural networks, the LoRA approximation error given by an approximated update AW = BA with intrinsic dimension r is related to the r-th singular value of the discrepancy E = W Warget Wfrozen between the target weight matrix and the frozen pre-trained weight matrix (Zeng & Lee 2023). In this context, the previously studied LoRA rank R determined by the size of matrices A and B acts as just an upper bound for AW's intrinsic dimension
In other words, encouraging the intrinsic dimension of AW approximate the given LoRA rank R benefits the generalization of LoRA fine-tuning under given trainable parameter budget. Inspired by this theoretical conclusion, this paper first adopts regularization technique to encourage LoRA matrices to span higher intrinsic rank in their parameter space. Additionally, to maintain a reasonable budget of trainable parameters, a gradient masking method is introduced to randomly mask a subset of parameters in each epoch instead of updating all parameters in LoRA matrices. Experiments on multiple datasets have proven this method also helps promote the growth of the intrinsic rank r and thus yields lower approximation error and better generalization performance.
The contributions of this paper can be summarized follows:
1. This paper extends previous theoretical bounds for LoRA approximation error from simulated datasets to real-world datasets, providing further insights into the trade-off between LoRA rank R and generalization performance.
2. Based on the analysis of LoRA rank and generalization performance, this paper designs a strategy for fine-tuning LoRA matrices that encourages the growth of intrinsic rank r= rank(AW) within the LoRA parameter space defined by R. This strategy effectively alleviates the problem of overfitting the training data by encouraging the LoRA matrices to explore the parameter space.
3. The experimental results across multiple open-source datasets demonstrate that this Regularized and Masked version of LoRA (RM-LoRA) method manages to strike better efficency-generalizeen tradeoff compared to the original LoRA method and its state-of-the-art variations, with better generalization performance achieved with the same or lower trainable parameters budget.
2 Related Works
In attempts to address the computational challenges posed by updating the enormous amount of weights in large pre-trained models, LoRA achieves outstanding model generalization with significantly reduced budget of trainable parameters during fine-tuning (Hu et al. 2021 However, LoRA still faces the challenges of sub-optimal performance. Previous research addressing LoRA's 's main challenges is briefly discussed as
follows:
Underfitting. While LoRA has demonstrated remarkable parameter efficiency and generalization perfor- mance, it may lead to insufficient. fine-tuning of large-scale models with high embedding dimensions (Hayou)
2

---

Published in Transactions on Machine Learning Research (01/2025)
2024 In some cases, there is contradictory phenomenon where higher LoRA rank doesn't necessarily yield better training results than lower ToRA rank. Much research has been devoted to further enhancing both the performance and efficiency achieved by LoRA, including the adaptive choice of LoRA rank (Zhang
et al.
ct al. 2023a Ding et al., 2023 Valipour et al.) 2023 adjustment of learning rate (Hayou et al.) 2024).
random projection (Kopiezko et al. 2023), derivative-free optimization (Jin el al., 2024), ; and pre-trained weights optimization (Zi et al.
Nevertheless, none of these mothods consider the role of LoRA updates' intrinsic dimension in mitigating the performance gap under a given LoRA rank setting.
2023
Overfitting. Fine-tuning large pre- trained models with a large number of parameters overfitting (Karimi Mahabadi et al. (202]. In the AdaLoRA method, the LoRA matrices of less important pre-trained weight matrices are assigned a lower rank to prevent overfitting (Zhang et al. 2023a). However, according to the experiments by Qiang et al. (2024), LoRA and AdaLoRA still clearly overfit the training data as fine-tuning progresses, with decreases in training losses but increases in test losses. To alleviate the overfitting problem, Qiang al.] (2024) developed the BiLoRA method, which iteratively trains different subsets of trainable parameters using different subsets of training data. Furthermore, HHaoou et al. (2024) pointed out that the overfitting of LoRA and its variants is due to some directions of LoRA matrices not being sufficiently updated, and thus the change in model weights approximated by LoRA is restricted by the vector (sub)space generated by the LoRA matrices' columns at initialization.
easily lead to
The previous analysis of LoRA's existing limitations and solutions gives rise to the idea and method employed in this work. Specifically, for better performance under a given LoRA rank setting, this paper proposes a fine- tuming strategy that promotes the growth of the intrinsic dimension of LoRA updates through regularization and gradient masking, bridging the gap between practical performance and theoretical optimal performance. The superiority of the two proposed techniques aligns with some of the previous studies. For example, the proposed regularizer encourages parameter space exploration. It helps reduce the performance gap as indicated by the performance benefits achieved by SoRA, which also tries to maintain a larger optimization space by keeping each epoch's sparsified components unchanged for the next epoch of updating (Ding et al.) 2023). Additionally, the gradient masking method shown to further improve generalization performance due to more efficient updates in certain LoRA directions. Such improvements can also be observed in DyLoRA, and column in LoRA matricos in cach step (Valipour et al. 2023
which only updates
3 Preliminary
Transformer Models. A seraformer-based pre-trained model typically involves L stacked encoder /deoodr blocks, with a multi-head attention module followed by fully connected feed-forward network (FFN) in each block. Given an input sequence X E Rxd the output of the multi- head attention module can be written as:
MultiHead(X)) Concat(head head, W with head, Atlention Qi, K;, V.),
Q,K. Vdz
and Attention Q2 K,, Vi) softmax
V.,
(1)
where Q XW?, K; XW and V, XWY are matrices of queries, keys, and values of head, respectively, with projection matrices w e gdxd, W. E Rdxd, WY E dxd,, and W E g xd We refer readers to the original paper for a more comprchensive introduction to attention calculations general (Vaswani et al. 2017).
Given the output of the multi-head attention module, the FFN further projects the d-dimensional output X' for each position. A two-layer FFN with ReLU activation operates as follows:
FFN(X') max X'W,, + 61) b
(2)
where W E Rdxdm. and W f2 E Rd xd Moreover, a residual connection followed by layer normalization is applied to each layer to generate the output of each transformer block given the input sequence X in the following way:
Norm + FFN(LayyrNorm(XX MultiHead (X))))
(3)
3

---

Published in Transactions on Machine Learning Research (01/2025)
LoRA Fine-tuning. For pre-trained matrix Wo R4xdd LoRA,, proposed by Hu et al. (2021)
approximates its update AW by AW = BA, where A E IR and B E RR with rank R < min d1 d2) During model fine-tuning, the weight matrix Wo is frozen, with only the LoRA adapters A and B being trainable. The modified LoRA forward pass is:
h= Wox + AWX = Wox+ + BAx.
(4)
Typically, the low-rank matrice A initialized using a random Gaussian distribution, while is initialized to zero, ensuring AW- 0 at the start of fine-tuning. Current approaches fine-tuning large pre-trained models with LoRA apply pair of matrices to all weight matrices involved in each transformer block's multi-head atlention module and FFN (He et al.
2021 Zhang et al. 2023a
The Expressive Power of LoRA. Zeng & Lee (2023) investigates the LoRA approximation error under a mild non-singularity assumption. To begin with, a L-layer width-D fully connected ReLU neural network is denoted as FN (W (bi)f=2), where W, E RDDD are the weight. matrices and b, E RDD are the biases for each layer I E [Z]. In LoRA method, the primary objective is adapt a pre-trained frozen FNN fo to approximate larget FNN J, which are represented follows:
Target FNN == FNN L.D W
(5) (6)
Frozen FNN fo := FNNL,D(: (b.)).
where Wi E RDxD and by E RD are the weight matrix and bias vector for the (-th layer of the target model f, while W, E RxD and b, E RD are those for the l-th layer of the pre-trained frozen model fo-
With a LoRA rank setting R E [D], the frozen FNN fo is adapted into a new model f:
Adapted FNN f == FNNL,D(: (Wr + Af
(7)
where AW, E RxD the weight update approximated by LoRA with rank(AW,) S RM, and b, is the updated bias vector or I [L]. Given that a large pre-trained model tends to be overparameterized, reasonable to assume that L> L, which means the pre-trained model is much deeper than the larget model to be approximated. Therefore, Zeng & Lee (2023) further introduces ordered partition p == (P....., Pr] to partition the L layers in the adapted model f, such that U._11 P. = [L]. Each partition element P. E P consists of consecutive integers 1 E Pi, which indicate the index of layers in the adapted model that will be used to approximate the i-th layer in the target model.
is
With the above definition, the following theoretical result provides an upper bound on the approximation error for the adapted model.
Theorem 3.1 (Theorem from ZZng Lee 2023) If ΣiEr, Ra 2 rank( - IIEP, Wi) for all E [L], there crists LoRA adapters (AWi) with rank A Wi) S Rí and biases (bi) such that the adopted model f can exactly approzimate the target model f. Furthermore, define the approzimation error of the i-th layer as e = Eien, Liep, Ri+1 (W: -ILEP, Wi), and the magnitude of the weight parameters and the input as
(VIE II
B = max
Σ II f=lk=
W |B58l|| (1bl1111bbll)
VIEEI
[E[
==1
Then, there erists LoRA adapters (AWi)_ with rank( AW)) s Rs and biases (Bi) such that for any input x with ExxT = E, the approzimation error can be bounded as
E El|f(x) - fxx) SB Σ
max W/ + tx) kc(T.)
ex) I-t Cs.
(8)
4

---

Published in Transactions on Machine Learning Research (01/2025)
In the above bound, B and W capture the magnitude of the weight parameters in the target model and the input. The LoRA rank setting R2 for all layers 1 E [7.] in the adapted model contributes to this bound through the term e; for all i E [4]. The following section focuses on the interconnection among the constituting parts of the & term and explains how this thcoretical conclusion can be utilized to enhance
LoRA adaptation on real-world datasets.
4 RM-LoRA Method
Based on the analysis of LoRA's expressive power described Section we introduce Regularized and Masked LoRA (RM-LoRA), a robust method aimed enhancing the gencralization performance of LoRA within given parameter budget. This section presents the details of RM-LoRA, including its underlying principles and specific techniques.
4.1 Influence of the Intrinsic Dimension of LoRA Adapter AW
Note that the partition P; of the pre-trained model for the i-th layer in the target model is an intrinsic but unknown property during adaptation, and consequently, the number and index of layers E Ps also unknown. Nevertheless, with a pre-trained model fo to be adapted and the larget model f determined by a given downstream task, the partition can be considered deterministic, as can the discrepancy between the pre-trained model and the target model E, = W; IeEP Wi Consider the case where the LoRA rank setting for each layer I E P; is the same as R. The e; term in Equation can be rewritten as:
eLrank(AWicr)) = Eier, n+1 (E,) for each layer i E [L),
(9)
where rank(A A W (EP,) R represents the LoRA adapter for each layer I E P, satisfies the rank constraint. rank(AW) s R.
Clearly, increasing Amk(AW) helps relax the constraint on LoRA rank R to achieve a certain level of approximation error. For example, consider two LoRA rank settings R and Rs with Rı < R2. Tf rank(A Wi) < Rg < R2, then Giraak((AWeer) )SR, degenerates to Ei,rak )SR, for E [L], despite the larger size of LoRA matrices under the LoRA setting of R2. In this case. LoRA rank R₁ and R2 yield the same LoRA approximation error fx) - f(xx)l2 according to Equation
4.2 Regularization on LoRA Weights
Let pair LoRA low-rank matrices be denoted as W and WB respectively. To enforce the growth rank of AW WB4 the following regularizer is first used to encourage W and W to be orthogonal:
Reg(W A W || (WWA(WAY IlI + +|( TW Il
(10)
The orthogonality of W and WB helps increase the Pank(W-) and rank According to the lower bound for the rank of the matrix product, for matrices A E Rxxd and B RdxR the rank of their product matrix C = BA satisfies rank C) > max(rank(A) (A) rBnk(B)- BB))) R, 0). This lower bound ensures the growth of the intrinsic rank of the LoRA adapter AW = " as the rank(W") and rank(W") increase with the regularizer shown in Equation 10 Note that there exist her alternative regularizers that theoretically can also encourage the growth of rank AW), but are infeasible in reality due to considerations of differentiabilityy numerical stability, and computational costs
4.3 Gradient Masking for Partial Updates
The gradient masking algorithm in RM-LoRA is designed to perform partial updates in LoRA matrices. The algorithm takes as input the total number of steps T, the LoRA rank R, and the number of directions
The nuclear (trace) norm involves expensive computation for singular value decomposition, especially with large matrices. Regularization on the determinants suffers from numerical instability since the determinant caleulation is highly sensitive small changes in the matrix elements. Constraints eigenvalues or singular values the matrix are directly differentiable.
5

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm 1 Gradient Masking Algorithm
1: Input: Total steps T, LoRA rank R, number of updated directions A.
2: for t = 0 to T - 1 do
3: for each pair of LoRA weight matrices (W.. w in the model do
4: Sample a mini-batch data & and compute the gradients (Ve, W V&. W
Initialize gradient masks (M,', < with the same shape Vs W.1 Vg, W?)): Construct the set R2 by randomly selecting & distinct integers from 2,....... R). for each i in R₂ do for all 1, ....,
8: M
9: end for
10:
for each j in R, do
[ = for all = 1.2...., R.
11:
12: end for
13: Apply gradient mask Ve, W. + V., W. O MA V. W VV. W O M
14: Perform optimization step W. W. V. W - VV.. W.P
15: end for
16: end for
17: Output: Updated LoRA weight matrices (W4, WT) for each fine-tuned module.
update in each step. In each training step f, it samples mini-batch of data & and computes the gradients Vs, W.A. and Vg, W for each pair of LoRA weight matrices. The corresponding gradient masks are first initialized to zero, before set R2 of distinct directions is randomly selected. The RM-LoRA method then sets the relevant entries in the gradient masks to one according to the selected directions. These masks are applied to the gradients to restrict the update directions. Finally, the algorithm updates the weight matrices W, and W using the masked gradients, thus achicving the partial update of LoRA weight matrices. The complete process of gradient masking summarized in Algorithm []
By strategically updating a subset of directions, gradient masking helps promote sparsity in gradient flow, which aligns with the inherent low-rank structure of many real-world datasets. To further improve efficiency, the gradient masking process can be optimized by avoiding the computation of gradients for the masked positions altogether. By skipping these unnecessary calculations, the computational overhead involved in the backpropagation phase can be significantly reduced, especially in large-scale models with high-dimensional parameter spaces. Modern training frameworks, such as PyTorc,, provide native support for mechanisms like partial gradient computation, which can be leveragod achieve a more efficient implementation of the proposed gradient masking.
5 Experiments
With the regularization and gradient masking tochnique described in Section 4 the proposed RM-LoRA method is expected to alleviate the problem of overfitting and achieve enhanced generalization for fine-tuning pre-trained models. In this section, RM-LoRA is comprehensively evaluated against state-of-thhe teet---rttttetttt LoRA variants multiple datasets.
5.1 Experimental Setup
The details of the experiment for the evaluation of the proposed RM-LoRA method are outlined as follows:
Models and Datascts. This paper compares our proposed RM-LoRA with the original LoRA and its recent variants across both computer vision and natural language tasks. For the vision task, we fine-tune a Vision Transformer (VIT-B/16) model, which has 86M parameters. This model was pre-trained on ImageNet dataset as described in the original paper (Dosovitskiy et al.] 22200) Fino-tuning performed on the CIFAR-100 dataset and Food-101 dataset to evaluate our approach in the computer vision domain. For language tasks, fine-tune the DeBERTa-v3-haar model with 184M parameters (He et al. 2022), and GPT-2 Small model

---

Published in Transactions on Machine Learning Research (01/2025)
Test Loss V.S. Epoch
Test Accuracy v.s. Epoch
2.50
80
2.25
0.03
Test Loss
1.75
Accurucy
78
0.75
1.90
23
0.33
76
0.75
10
Epoch
Epoch
Train Accuracy v.s. Epoch
Generalization Error v.s. Epoch
18
94-
16
92
Generalization
12
83
Train
84
80-
10
20
30
40
50
60
20
30
50
60
Epoch
Epoch
GM LoRA (R 52)
R RcLRAA (R- 32)
GM LR -64)
R-LoRA (R (R--6
GM LRRA 50)
8 LoRA
3M-LoRA
(R 54)
RM-LORA (R
LoRA
Full finestuning
Figure Results with ViT model on CIFAR-100.
with 117M parameters. Evaluations are conducted on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2019 for language understanding and the Stanford Question Answering Dataset (SQuAD 1.1) (Rajpurkar
2016) for question answering.
Baselines. The following baselines implemented within the same HuggingFace's Transformers framework for fair comparison (Wolf et al. 2019)
Full fine-tuning (FT) uses the pre-trained model as the initialization point and updates all parameters in the model through gradient backpropagation, resulting in a very large trainable parameter budget that may be impractical in resource-limee limitedd environments.
LoRA Hu ct al. 2021 approximates the incremental updates in pre-trained model weights by using the product of two trainable matrices with rank R, which siguificantly reduces the number of trainable parameters needed to adapt pre- pre-trained models while achieving competitive generalization performance on downstream tasks.
AdaLoRA (Zhang el al. 2023a uses the product of three small matrices in the form of singular value decomposition to parameterize the updates pre-trained model weights, and then prunes the singular values of lower importance the diagonal matrix to achieve a pre-sct total parameter budget across all adapter weight matrices. Compared to LoRA with a fixed parameter budget, AdaLoRA provides a more flexible solution for task-specific fine-tuning, resulting in better afficiency-gnnerlliataaan trade-off.

---

Published in Transactions on Machine Learning Research (01/2025)
Test Loss v.s. Epoch
Test Accuracy v.s. Epech
76
1.6
74
Accuracy
Test Loss
1.4
1w215
22
0299
1.2
66
10
Epoch
Epoch
Train Accuracy V.s. Epoch
Generalization Eroor v.s. Epoch
$0.0
77.5
Generoiization Error
Train Accuricy
75.0
70.0
67.5
10
20
30
46
60
20
30
40
S0
60
Epoch
Epoch
GM-LoRA
(R
GM-LoRA RM-L.RA (R 80)
R-LoRA
LORA
RM-LoRA (R 64)
LoRA (R 64)
Figure 2: Results with ViT model on Food-101.
SoRA DDigg et al. 2023) parameterizes the updates in pre-trained model weights similarly to AdaLoRA, with an additional gate unit in between, and controls the sparsity of the gate by pruning components with absolute values lower than a pre-set threshold A. By retaining pruned components until the pruning step in the final epoch, SoRA maintains a larger parameter space for exploration during fine-tuning compared to AdaLoRA.
These aforemmentioned LoRA methods aim to strike a balance between parameter elficiency and generalization performance in adapting pre-trained models. Sharing this goal, our proposed RM-LoRA addresses the challenge by exploring and enhancing the role of the intrinsic dimensions of LoRA matrices. For a fair comparison of all the LoRA variants, including evaluated under the same parameter budget during inference. All experiments are conducted across three independent runs with different random seeds, as discussed in the following sections of this paper.
proposed RM-LoRA method, their performance is
5.2 Image Classification
Figure Iillustrates the results achieved by the ViT model on the CIFAR-100 dataset, serving as a preliminary mcasure of the performance of LoRA and the enhancement techniques for LoRA method proposed in this paper. To simulate the theoretical results based on the fully connected layer, only the last classification layer of the ViT model is fine-tuned by LoRA low-rank matrices. The four sub-figures of Figure display

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Results with DeBERTav3 model on GLUE benchmark
Method
* T / Parama
CoLAA
ST3-B
QNLI
MNLI
WNLI
ITE
MRPC
QQr
SST-2
Mee
Peatson / Spearzman
Acc
Acc
Aec
Acc
Acc / FL
Ase / Fl
Acc
Full FT
184M
184M
0.068
0.892 / 0.590
0.540
0.907
0.565
0.552
0.853 y 0.891
0.915
0.891
0.994
LoRAges
1.26M / 1.26M
0.680
0.012 0.912
0.939
0.901
0.715
0.586
0.892
0.922
0.912 / 0.853
0.934
AdaLoRA
1.22M/
1:26M
0.863
0.905
0.909
0.934
0.904
0.365
0.845
0.860
0.911
0.912
0.854
0.955
S
1.92M/
1.36M
0.655
0.905 / 0.907
0.926
0.889
0. 890
0.848
0.895
0.924
0.920
0.782
0.820
R--olRA R-d
1.26M/
1.26M
D. 0.685
0.814
0.914
0.841
0.903
0.718
0.883
0.897
0.925
0.917
0.889
0.943
RM-LaRA R-d
1.26M 1.26M
0.689
0.915 / 0.915
0,942
0.904
0.732
0.866
0. 0.892
0.922
0.913
0.851
0.943
LoRAg-S
1.99M/
1.92M
0 0.672
0.813 / 0.914
0.835
0.900
0.718
0.856
0 0.000
0.928
0,916 / 0.858
0.936
AdaboRAA
3.25M
1.92M
0 0.004
0.012/ 0.913
0.042
0.006
0.578
0.541
0.697
0.924
0,912
0,855
0.945
SoRA
3.26M
2.34M
0 0.005
0.907
0.910
0.080
0.890
0.680
0.541
0,807
0.907
0.919
0.761
0.526
R-LoRAgga_
192M
1,22M
0.094
0.914
0.914
0.943
0.004
0.782
0.568
007
0.008
0.017
0.890
0.940
RM-LoRAp-s
1.92M/
1.923
0.688
0.916
0.915
0.944
0.907
0.747
0.870
0.014
0.038
0.913
0.886
0.948
Table 2: Results with DeBERTaV3 model on SQuAD
Method
#
Rank
# / Params
Full FT
N/A
N/A
184M
184M
87.83 + 0.01
93.02 = 0.01
LoRAg-s R-8
8
8
1.33M
1.33M
87.47 + 0.01
93.33 + 0.01
SoRAAgmbe-4
16
16
2.66M
1.28M
62.73 = 0.02
90.49 0.01
RM-LaRAR-S
8
1.33M
1.33M
87.87 0.03
93.67 0.01
LoRAR = 4
4
0.67M
0.67M
87.43 + 0.02
93.35 + 0.02
AdaLoRA
8
4
1.33M
0.67M
86.48 + 0.02
92.83 : 0.01
SoRAsg=0e
8
1.33M
0.6IM
79.64 + 0.02
88.01 0.02
RM-LoRAR-4
0.67M
0.67M
87.57 0.03
93.51 0.02
the test loss, test accuracy, train accuracy, and generalization error (measured by train accuracy minus test. accuracy) for each method respectively.
As observed in Figure [ the regularization and gradient masking techniques proposed in this paper both effectively mitigate overfitting and achieve higher accuracy on the test dataset. Specifically, Regularized LoRA (R-LoRA) and Gradient Masking LoRA (GM-LoRA) represent the application of each technique individually, while RM-LoRA combines them together described in Section 4 Furthermore, Table 5 presents the orthogonal loss ([AW(AW)) describes its spatial distribution. The results in Table 5 demonstrate that the orthogonal penalty term for the LoRA matrices WA and W n in Eq. I0 effectively promotes the orthogonality of their product. Meanwhile, as the orthogonal loss of AW decreases, the accuracy of LoRA tnn-tuning on the test data increases. Therefore, by effectively promoting the reduction of AW's orthogonal loss, the RM-LoRA method proposed in this paper achieves the best generalization performance across all rank settings.
I of A AW after being fine-tuned by each method, which
Figure presents the results achieved by the ViT model on the Food- 101 dataset, which exhibit similar trends to those observed in Figure D and further demonstrate the generalizability of the proposed advancements.
5.3 Natural Language Understanding
The GLUE benchmark includes two single-sentence classification tasks (CoLA, SST-2), three similarity and paraphrase tasks (MRPC, STS-B, QQP), and four natural language inference tasks (QNLI, WNLI, MNLI, RTE). The proposed RM-LoRA method is compared against the baseline methods under multiple LoRA rank settings to demonstrate its superiority. Table I shows the performance achieved by different methods on GLUE lasks, as well as the number of trainable and inference parameters (# T / Params respectively). The standard deviations (STD) across three independent runs are limited within 0.01 but are omitted in the table due space constraints. The best result for each task highlighted with the proposed regularizer consistently achieves performance gains under the same or lower inference parameter budget compared to other methods in most cases. Furthermore, RM-LoRA with gradient masking
bold. R-LoRA
9

---

Published in Transactions on Machine Learning Research (01/2025)
Table 3: Results with GPT-2 model on SQuAD
Method
#
Rank
T
Params
EM
F1 Score
Full FT
N/A
N/A
74.41
LoRAR-10
1.60M
L.9M
61.41
0.02
72.76 0.03
AdaLoRA
32
16
3.30M
1.69M
61.70 0.01
72.30 1 0.03
SoRAA2=8e
32
3.39M
L62M
60.80 0.02
71.80 0.04
RM-LoRAR=I6
1.69M
73.50
LoRAR-s
0.85M
0.85M
60.34
0.02
71.92 0.02
AdaLoRA
16
1.69M
0.85M
60.20 + 0.02
71.40 1 0.02
SoRAA2=0e
16
1.69M
0.81M
59.80 0.02
70.90 0.03
RM-LoRAR=s
Table 4: Hyperparameters for GLUE tasks
Learning Rate
SoRA Parameters
Datascts
Batch Size
# Epochs
LoRA
Sparsity
CoLA
0.00001
0.001
32
3
0.001
68.96%
SST-2
0.00001
0.001
32
3
0.005
53.90%
MRPC
0.00001
0.001
32
5
0.001
79.27%
STS-B
80.62%
QQP
0.00001
0.001
32
3
0.005
53 53.90%
QNLI
0.00001
0.001
32
3
0.0001
58.06%
MNLI
0.00001
0.001
32
3
0.005
61.77%
WNLI
62.53%
RTE
0.00001
0.001
32
3
0.005
61.23%
outperforms R-LoRA in a majority of task settings. The specific fine-tuning hyperparameters adopted by each method on the GLUE benchmark are summarized in Table
Table 5: Correlation between the orthogonality of AW and generalization performance
Method
Rank AW
Orthogonality Loss of AW
Test Ace (mean std)
LoRA
29
585.835
76.97 + 0.02
GM-LoRAg-s3
29
277.919
77.72 0.02
RR-LoRAR-12
122.079
77.00
RAM-LoRA--8
30
95.240
78.05 0.02
LoRA 8-64
55
410.154
78.41 0.03
GM-LoRAR
52
175.010
79.60 0.05
R--.oRAR-G4
RM-LoRARand
57
58.010
80.07 + 0.04
LoRA
64
342.197
78.13 0.03
GM-LoRARRs50
155.010
79.08 0.02
R-LoRAR-50
79.63
RM-LoRA
R2
69
48. 48.882
80.20 + 0.02
10

---

Published in Transactions on Machine Learning Research (01/2025)
5.4 Question Answering
The performance of different methods using the DeBERTaVe V3 model and the GPT-2 model the SQuAD dataset is shown in Table 2 and Table 3 Similarly, the proposed RM-LoRA method outperforms other baselines under the same or lower inference parameter budget across varying LoRA rank settings, with EM denoting the average exact match score and F1 referring to the average F1 score. These results highlight that the RM-ToRA method consistently improves LoRA's fine-tuning performance across various benchmarks. The capability of achieving better or comparable performance with a reduced parameter budget is especially significant for practical deployment in mobile systems, where efficiency and resource utilization crucial factors.
6 Conclusion
In conclusion, the exploration of the intrinsic dimension in LoRA fine-tuning reveals critical insights into optimizing parameter efficiency and enhancing model generalization. The thcoretical foundation indicates that the intrinsic dimension of the approximated matrix updales is more pivotal in achieving effective LoRA fine-tuning than the previously emphasized LoRA rank. By employing a regularization technique and a gradient masking method to encourage parameter space exploration while controlling the trainablc parameters budget, this paper presents an advanced low-rank adaptation strategy that addresses the challenges of sub-optimal performance associated with LoRA. The better generalization performance achieved by the proposed R.M-LORA under the same or lower parameter budget compared to other methods represents progress in the field of parameter-effiiinnt fine-tuning for large pre-trained models.
7
Acknowledgement
We would like to acknowledge the support from the National Key R&D Program of China under Grant No. 2022ZD0160504 and the Shenzhen Key Laboratory of Ubiquitous Data Enabling under Grant No. 2DSYS20220622111400005 Their support has been invaluable the completion of this work.
References
Naday Benedek and Lior Wolf. PRILoRA: Pruned and rank-increasing low-rank adaptation. arXiv preprint 2rXiw:2401.11816, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 2020.
Ning Ding, Xingtai Tw, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In Conference on Empirical Methods in Natural Language Processing, 2023.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Annual Meeting of the Association for Computational Linguistics, pp. 4884-4896, 2021.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. arXie preprint arX:::2402. 12354, 2024.
Junxian Hc, Chunting Zhou, Xuezhe Ma, Taylor Borg-Kirkpatrick, and Graham Neubig. Towards a unificd view of carfmeter-cffiiint transfer learning. In International Conference on Leorning Representations, 2021.
11

---

Published in Transactions on Machine Learning Research (01/2025)
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using electra-style pre-training with gradient-disentanllel embedding sharing. In International Conference on Learning Representations, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Atlariyan, and Sylvain Gelly. Parameter-eficient transfer learning for NLP. Intervational Conference on Machine Learning, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Feihu Jin, Yin Liu, and Ying Tan. Derivative-free optimization for low-rank adaptation in large language models. arXiv preprint orXiv:2403.0117 2024.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercom- plex adapter layers. Advances in Neural Information Processing Systems, 34, 2021.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. VelA:: Vector-based random matrix adaptation. arXiv preprint (xXit:2310.1145.. 2023.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parametor-officient prompt tuning. Conference on Empirical Methods in Naturul Language Processing, pp. 3045 3059, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuinng Optimizing continuous prompts for generation. In International Joint Conference on Natural Language Processing, pp. 3582-4597, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A fine-tuning is better and cheaper than in-context learning. Advances
Raffel. Few-shot in Neural Information Processing Systems, 35, 2022a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang, P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. Annual Meeting of the Association for Computational Linguistic, pp. 61-68, 2022b.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen Tau Yih, and Madian Khabsa. UNIPELI: A unified framework for parameter-effiiient language model tuning. In Annual Meeting of the Association for Computational Linguistics, pp. 6253-6264, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35, 2022.
resilient low-rank adaptation of large pre-trained models. arXiv preprint 2r 130.13373, 2024. Rushi Qiang, Ruiyi Zhang, and Pengtao Xie. BiLoRA: A Bi-level optimization framework for overfitting-
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing, pp- 2016.
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34, 2021.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information. Processing Systems, 35, 2022.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint arXi::2002.188 2023.
Mojtaba. Valipour, Mehdi Rezagholizadeh, Tvan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-effeient tuning of pre-trained models using dynamic search-frec low-rank adaptation. In Conference of the European Chapler of the Associalion for Compulational Linguislies, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Milia Polosukhin. Attention is all you need. Advances in Newral Information Processing Systems, 30, 2017.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Annual Meeting of the Association for Computational Linguistics, pp- 5039 5059, 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXi::191000077 03771,, 2019.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficiet fine-tuning for transformer- based masked anguage-modekt In Annual Meeting of the Association for Computational Linguistics, pp. 9, 2022.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In International Conference Learning Representations, 2023.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-cffcicnt fine-tuning. In International Conference on Learning Representations, 2023a.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. ar preprint XrX::22003. 16199, 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXio preprint Xiv: 01068, 2022.
Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang, LoRAa- Fine-tuming high-rank parameters with the delta of low-rank matrices. aiv preprint arXiv:2309 024111 2023.
13


Reviews:
[{"review_id": "XzThjtzdJR", "reviewer": "Reviewer_2F9x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The problem is well-motivated.\n* The intuition behind the design choices is explained properly.\n* Experiments in the NLP domains are more comprehensive.\n* The method shows some improvement compared to the prior work. \n\nWeakness:\n* Baselines can be improved. The authors have named several works in the prior work sections but only selected two methods for their comparison baseline.\n* In some scenarios, the improvement is negligible.\n* The method is not very novel. Adding regularization to training parameters is a common practice, and prior works have already explored the benefits of different masking methods on top of the LoRA blocks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to improve the performance of the LoRA method (an efficient method for finetuning). First, the authors explain the importance of the intrinsic dimension of $\\Delta W = W_{finetuned} - W_{pretrained}$ in the final performance. Then, they claim prior works have two problems: limited LoRA dimension and LoRA finetuning can lead to over/underfitting. To mitigate these problems, they propose two changes to the LoRA blocks: 1) regularization in the weights and 2) random gradient masking.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The citation and text have been combined in several locations, such as Page 2 - line 2, Page 3 - line 1, Page 4- line 3 and 14. \n\n* The improvement in the method is insignificant (for example, less than 1 % in Table 1).\n   1) The information about random seeds is missing.\n   2) LoRA training can be sensitive to learning rate, and using this parameter can change the final observation.\n\n* Some tasks, such as CIFAR100, are very simple and cannot show the importance of finetuning methods in ViT. Authors can include more sophisticated datasets to show the limitations of LoRA and their method's superiority.\n\n* The authors mention various PEFT methods in the related works; however, they have only selected 3 as their baseline. Could you please justify why some of the newer LoRA versionmodifications and algorithms are irrelevant here? \n\n* The authors claim gradient masking can make the training more efficient, but dynamic masking during training is not very beneficial in terms of computation reduction since the gradients are randomly set to zero after calculation. Could you please share any information or evidence that shows the computational superiority of this method?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2TvbzH80BY", "reviewer": "Reviewer_LdFz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\nThis paper addresses the fundamental problem of improving LoRA efficiency with the proposed regularization method.\n\nIf the algorithm is implemented robustly and scale to industry infra, it could help to large scale model training.\n\nThe experiments offers comprehensive comparison of various LoRA baselines, which helps readers to understand its effectiveness.\n\n**Weakness**\n\nThe paper discussed theorems in the main section yet only to cite other's contribution. This should be better put to related works section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose improve LoRa by adding regularization and mask mechanisms. It did so by investigating the intrinsic dimension. It proposes that employing regularization and a gradient masking method could encourage higher intrinsic dimension. The proposed method achieves superior generalization performance with the same or lower trainable parameter budget.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be better to not discuss other work's theorem in the main section.", "broader_impact_concerns": "The reviewer does not see broader impact concerns with regard to this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vBfk1vLVBr", "reviewer": "Reviewer_u7Bu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The motivation of the regularizer is clearly explained.\n2. The paper provides numerical results to validate the proposed algorithms' motivation by Tab. 4.\n\nWeakness:\n1. The motivation of the gradient masking requires extra explanation. Sec. 3.4 is too short to explain the proposed gradient masking method. \n2. Lacks discussion and comparison with existing methods. The description in Sec. 4.1 on the baselines should be moved to Sec. 3 with a more detailed discussion on the connection and difference between the proposed method and these baselines.\n3. In sufficient numerical results. Since the paper is purely empirical, more substantial numerical evidence is required.\n    1. The STD is missing in the tables. The experiments should report the STD of multiple runs so readers can decide if the performance improvement is statistically significant, especially when the algorithm introduces extra randomness in sampling $\\hat{r}$.\n    2. The lines in Fig. 1 are hard to distinguish; please consider using more distinguishable color/line type coding.\n    3. The reported results for ViT do not match SOTA results, e.g., in [This GitHub Project](https://github.com/bwconrad/vit-finetune), the reported test acc. using LoRA goes to 92% for CIFAR-100, in [This Paper](https://arxiv.org/pdf/2406.16282), it is 92%. However, the paper only reports less than 81%. A similar issue appears in the results of AdaLoRA, where the reported numbers do not match the ones in the [original paper](https://arxiv.org/pdf/2303.10512).\n    4. The ViT and DeBERTa-V3 model size is not specified. Since these models have many different settings, the author should provide the specific choice used for the experiment, including the size and the pretrained source.\n    5. The numerical result is limited to only two models, ViT and DeBERTa-V3. The author should compare the performance of more models, e.g., OPT and Llama.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on improving PEFT, specifically the performance of LoRA. With the goal of increasing the intrinsic rank of the LoRA matrix $\\Delta W$, the paper proposes adding a regularizer on the LoRA matrices to make the rows/columns orthogonal to each other. Additionally, the paper proposed using masked gradient updates to randomly update $\\hat{r}$ rows/columns in the LoRA matrices. \nNumerical results are conducted on CIFAR-100 with ViT and GLUE&SQuAD with DeBERTa-V3. In the experiments, the paper compares the proposed method (R-LoRA, GM-LoRA, and RM-LoRA) with several baselines, including full FT, LoRA, AdaLoRA, and SoRA. Numerical results show that the proposed RM-LoRA outperforms existing algorithms. The results also show that there is a strong correlation between the orthogonality and the FT model performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the above weaknesses, specifically:\n1. Clarify the mismatch of the numbers in the experiments.\n2. The STD of multiple runs\n3. Clarify the models used\n4. Add comparison on more models\n\nMinor issues:\n1. Change the color or line type in Fig. 1 to improve readability.\n2. Add a discussion about the algorithms' connection", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend accepting the paper with minor revisions. The authors have largely addressed the reviewers' concerns in their responses and revised manuscript. The paper offers valuable insights and a potentially impactful method for improving LoRA.
Decision: Accept (poster)