RidgeLoRA: Matrix Ridge Enhanced Low-Rank Adaptation of Large Language Models
Authors: Junda Zhu
Jun Ai
Yujun Li
Yichun Yin
Yasheng Wang
Lifeng Shang
Qun Liu
Venue: NeurIPS 2025 spotlight ()
Keywords: Parameter-Efficient Fine-Tuning
Large Language Model
Low-Rank Adaptation
Primary area: deep_learning
TL;DR: RidgeLoRA adopts a novel series connection architecture, and enables different parameter initialization approaches for better performances.

Abstract: As one of the state-of-the-art parameter-efficient fine-tuning~(PEFT) methods, Low-Rank Adaptation (LoRA) enables model optimization with reduced computational cost through trainable low-rank matrix. However, the low-rank nature makes it prone to produce a decrease in the representation ability, leading to suboptimal performance. In order to break this limitation, we propose RidgeLoRA, a lightweight architecture like LoRA that incorporates novel architecture and matrix ridge enhanced full-rank approximation, to match the performance of full-rank training, while eliminating the need for high memory and a large number of parameters to restore the rank of matrices. We provide a rigorous mathematical derivation to prove that RidgeLoRA has a better upper bound on the representations than vanilla LoRA. Furthermore, extensive experiments across multiple domains demonstrate that RidgeLoRA achieves better performance than other LoRA variants, and can even match or surpass full-rank training.

RidgeLoRA: Matrix Ridge Enhanced Low-Rank
Adaptation of Large Language Models
Junda Zhu' Jun Ai'
Yujun Li-
Yichun Yin?
Yasheng Wang" Lifeng Shang? Qun Liu
Beeihang University HHuawei Noah's 's Ark Lab
junda_ zundda_kk@ootooo com 1iyujun1450gmailccmmm
Abstract
As one of the state-of-the-e parameter-efiin fine-tuning (PEFT) methods, Low- Rank Adaptation (LoRA) enables model optimization with reduced computational cost through trainable low-rank matrix. However, the low-rank nature makes it prone to produce a decrease in the representation ability, leading to suboptimal per- formance. In order to break this limitation, we propose RidgeL.RRA, lightweight architecture like LoRA that incorporates novel architecture and matrix ridge en- hanced full-rank approximation, to match the performance full-rank training, while eliminating the need for high memory and large number of parameters restore the rank of matrices. We provide a rigorous mathematical derivation to prove that RidgeLoRA has better upper bound on the representations than vanilla LoRA. Furthermore, extensive experiments across multiple domains demonstrate that RidgeLoRA achieves better performance than other LoRA variants, and can even match or surpass full-rank training.
Introduction
Large Language Models (LLMs) with large number of parameters [ have demonstrated ex- ceptional performance natural language generation tasks. These models acquire their primary knowledge during the pre-training phase, through training on massive high-quality datasets from both real-world corpora or model-generated synthetic data. To align with real-world scenarios (9). LLMs also require supervised fine-tuning (SFT). Traditionally, this fine-tuning process employs full-parameter (also full-rank) training (FFT) to achieve optimal performance. However, this approach demands substantial computational resources.
When adapting LLMs for downstream tasks, training is often constrained by limited computational resources, calling for efficient and lightweight solutions. Paftmeter-Effficient Fine- Tuning [1 PEFT) methods achieve comparable performance Low-rank adaptation (LoRA A, 12). as a representative PEFT method, is widely adopted due to the fact that downstream tasks largely rely on the generic capabilities developed in pre-training. To maintain model performance, LoRA's initial state should also align with the original model's 's output, which we refer to as "the
full-parameter fine-tuning with minimal cost.
restriction".
However, vanilla LoRA, while significantly reducing the number of trainable parameters, is often criticized for its lower performance ceiling (3315]. Its low-rank nature results in significantly lower representation capability compared to full parrmmeerr fine-tuning, making it prone to underfitting in downstream tasks [14). Existing LoRA variants (16 primarily focus on matrix decomposition or
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LoRA
RidgeLoRRA
Xou
W
AB
Xont X
AB\W
rask
Tull reaks
Baue
des
B
B
Pre-truined Matrix
X..r
Pre-trained Matrix vilas
Figure 1: Differences between LoRA (left) and RidgeLoRA (right): As is depicted, though LoRA reduces the number of parameters to be trained, the matrix rank severely shrinks. order to be comparable with full rank training. the proposed RidgeLoRA introduces matrix ridge (formalized as XΣ) to complement the rank of the trainable parameters.
numerical stability. proposing better parameter initialization methods or updating strategies. However, these methods lack theoretical investigation full-rank matrix approximation and the fundamen- tal challenges of representation ability in low-rank settings. Additionally, exploring architectural alternatives beyond the vanilla LoRA framework could potentially unlock opportunities for improvement.
Unlike existing works, RidgeLoRA incorporates full-rank module to achieve performance compa- rable to full-parameter training. We redesign the architecture by replacing the parallel connection in vanilla LoRA with series connection module. Inspired by the Matrix Ridge algorithm 222, enhance vanilla LoRA by incorporating a ridge term, thus introducing RidgeLoRA. The architectural differences between RidgeLoRA and vanilla LoRA are illustrated in Figure Based on observations from previous works [8] and our experiments, appropriate initialization methods can significantly improve low-rank training performance. As RodgeLoRAA oRA adopts a series connection architecture, enables different parameter initialization approaches even under "the ruusform-calibratinn re- striction". RidgeLoRA achieves comparable performance full-parameter training minimal cost without introducing additional computation and parameters. Our main contributions are summarized three-fold:
We propose RidgeLoRA, novel LoRA variant that replaces parallel connection with series connection, enabling better parameter initialization and enhanced representation capability.
We introduce diagonal ridge term alongside the low-rank matrices, inspired by the matrix ridge algorithm [22]. which improves approximation flexibility while maintaining computa- tional efficiency, better represent the high-rank updates of LLM training.
We provide theoretical analysis of RidgeLoRA's representation capability and demonstrate its superior performance through extensive experiments across multiple datasets.
Related Works
2.1 Matrix Low-Rank Decomposition and LLMs
Matrix decomposition aims at optimizing the following objective:
min JW -w
where W E gedenxdou is the target matrix, W, represents its low-rank approximation with rank < min(din, dout) and || denotes the Frobenius norm. Given a restricted rank r, Singular value decomposition (SVD, has a bounded (also the minimum) error, which justifies wide application, whose details can be found in Appendix A.3 In order to accelerate the inference

---

LoRA Method
Number Parameterr
Weight Initialization Complexity
Formalization
Computation Complexity
FFT
dless
OL1) Otrdin)
W"
Old dear,
Lotd
12
F. (din in deat)
Xin(W 2.4
thatl dens
thas)
① Cinns thas) minffh Guull O manellon thane) minn C... Gaall
NuatM 9.485
DNCC,
KrSA
18
FX (fh + thas 1]
NW, 9.47.
Lidg
2din (din + 1)
Olybhanl
Xin(AE PAD2WW
(ldf,)
Table 1: Comparisons between FFT and other LoRA methods, here we focus on analyzing the number of parameters from the perspective of a single matrix. The comparison here also considers the complexity of weight initialization, where some methods may conduct SVD or matrix multiplications. W and W denote different ways to initialize W in their works. RidgeLoRA is showcased with simple initialization method, small number of parameters, and will not increase the computation and memory requirements during training.
rea
speed, SVD-LLM [24] utilizes SVD compression method together with data whitening [25 to minimize ||W X| I
compress LLMs. MoDeGPT (26) conducts detailed analysis of each matrix calculation operation inside an LLM and selects the corresponding decomposition method accordingly, namely SVD, Nyström approximation (27] and CR decomposition
[28] As inspiration [22] proposes an algorithm that approximates a positive semi-definite matrix using a combination of incomplete matrix decomposition and ridge term. This achieves tighter approximation than both incomplete Cholesky decomposition [29) and incomplete spectral decomposition, while ensuring that the condition number of the approximated matrix does not exceed that of the original matrix.
of this paper, Matrix Ridge
2.2 Parameter-Efficien Fine-Tuning
LoRA and its Variants LoRA and its variants 12
30 351 train extra low-rank weights on top of the model, which makes low-rank matrix decomposition naturally suitable for the improvement restriction" ", advanced LoRA variants like PiSSA (17], LoRA-GA (30], MiLoRA (31], LoRA-XS [32] and KaSA [18] conduct SVD the original matrices to endow the low-rank matrices with better initializations, thus achieving better performance. Fur- thermore, some works focus more the numerical stability of matrices, rsLoRA K4 and proposes a better setup of the scaling factor from statistical point of view. To achieve training stability, DoRA [16] decomposes the updates of matrices to magnitude factor and direction factor, hereby adding a scaling factor during training. VeRA 351 adds learnable scaling vectors which can be updated and tune frozen random matrices across layers, which changes bit of the architecture.
16 21
of it. Considering "the
Apart from parameter-based methods, LoRA+ 20 assigns the matrix initialized with all zeros with larger learning rate, LoRA-Pro [19] updates the matrices from the perspective of transformation invariance. These works all focus on how to deal with the gradients in the optimization stage.
Other PEFT Methods LoRA-based methods and variants can be attributed to one type of PEFT 36
method, which also includes (1) Selective training: BitFit [37]; (2) Soft prompt: Prefix
Tuning 38 and P-Tuning [39 40]: (3) Adapter-based method: Serial Adapter 41 and Parallel
The latter two types also insert extra trainable modules while keeping the pre-trained matrices frozen like LoRA does.
Adapter 42
3 RidgeLoRA: Matrix Ridge Enhanced LoRA
3.1
Architecture of RidgeLoRAA
RidgeLoRA switches the computation graph to the following formalization:
out X r. AB)W,
'The biggest difference between RidgeLoRA's series connection and serial adapter is that the training parameters of RidgeLoRA can be absorbed into the original weights after the training is completed, whereas the extra modules inserted by serial adapter cannot be absorbed due the presence of activation values inside of it.

---

where E is diagonal matrix (referred to as the Ridge) that requires gradient descent. It has dag learnable parameters on the matrix diagonal, with its matrix rank din À denotes the "Ridge Intensity" which can be updated. Specifically, RidgeLoRRA inserts additional trainable ridge beside the matrix and novelly converts the extra modules into series connections, as opposed the parallel connections used in LoRA variants. Moreover, the newly trained part can also be absorbed back to the original weight as described in the following formalization:
W = (AE AB)W,
(2)
where the trained matrix W the product of two full-rank matrices. This absorption recovers the original architecture while endowing the model with brand new optimized weights during inference, which provides the same advantage as LoRA and its variants. Moreover, in order to provide a clear comparison main-stream tuning methods, we further list the properties of them in Table 1
3.2 Theoretical Analysis of RidgeLoRA
According to the setups of RidgeLoRR,, AΣ initialized from a part of a diagonal matrix, thus enabling it to dominate the spectrum of XΣ AAB and achieve high rank of din. Here we conduct derivations on how closely the enhanced RidgeLoRA architecture can approximate the full-rank weight update AW. By introducing this ridge term, we effectively increase the model's rank expressiveness and improve the overall performance.
Theorem 3.1. Let K E Raxd be a rank-k matrix (k S d). D E Rdddd be a diagonal matrix, M dxd be arbitrary matrix. Given that M has Singular Value Decomposition (SVD) M- = UEV T Let A E Ra(n-1)xk 1)xk be a matrix whose rows are indexed by ordered pairs (p, q) where P.q E {1, 2, .... n and + q. Each row a(pq) E R xk of A has entries given by Up Vas for j E (1,2, ...,, k). Let E 1n)= be vector whose entries are Cpq = =j=n4t Upj8j Vaj. where
8j denotes the corresponding entries in Σ. We have that
min
K + D MI & I1( A(A A(ATA) AdllE
(3)
K.D
where (-)t denotes the pseudo inverse of matrix.
This demonstrates the advantages of adding Ridge. Specifically, the right side of the above expression is less than or equal to the LoRA case. For the detailed proof, please refer to Appendix A.1 Next, we will elaborate on this point.
In the case LoRA, is a zero matrix, and the minimization objective becomes
min |K M M
Let K be a rank-k matrix. According to the Eckart- Young Mirsky theorem 43 the optimal choice
K UVVV" V minimizes the Frobenius norm ||
M Substituting this optimal K, we obtain
KK M -/ 10 - V
where Σn denotes the best rank-k approximation of Σ, obtained by retaining the top-k largest singular values (ordered in descending magnitude) and zeroing out the rest.
In the proof of Theorem 3.1 after introducing the diagonal matrix D, if keeping K fixed K UE V the problem reduces to a least squares optimization with respect to the entries D. Clearly, D 0 is not the optimal solution. The bound we established demonstrates that our method is more effective, detailed analysis can be found in Appendix A. to illustrate a theoretical measure.
3.3 Detailed Designs of RidgeLoRA
On top of the theoretical analysis of how RodgeLoRA facilitates full-rank training, additional mecha- nisms are introduced to optimize performance. Specifically, incorporates a novel weight initializa- tion strategy and an auxiliary loss function for the efficient updates of parameters.

---

Algorithm Weight Initialization of RidgeLoRA
Input: Input dimension din, Target low rank r, Scaling factor
Output:
X, A, A,
I: Initialize À 6-11
> Initialize the intensity of ridge term.
2: Sample noise vector N E R'' using
N o [NNN R;)];
where (-) is the sigmoid function.
Split small portion initialize low-rank matrices.
3: Construct matrix Σ Rd X xd
as
diag(N)
E
d:n--r]
where diag(N) is an r x r diagonal matrix, and Ld..-r is an identity matrix.
4: Initialize low-rank matrix A E RRd xr with Gaussian Noise (00, 1/r)
5: Conduct SVD matrix A: UVEAVA
A.
6: To ensure the condition
b This ensures the "transform-calibrating restriction".
AE AB=1,
is initialized with
B= ~ (Iai. -E). a
Note that UAUA VIVA 4.
Weight Initialization In the field of deep learning, matrices are usually initialized with well- designed methods [4446 for a better starting point. However, vanilla LoRA is constrained by the cauussforn-calbrrttin restriction" [12, which allows only the initialization of matrix A with such methods, while B is initialized as an all-zero matrix. This limitation hinders the ability of LoRA to fully explore the parameter space. With the novel series connection, this restriction can be fulfilled with AΣ + ę AB I, eliminating the strong constraint that requires B = 0. This ensures good initializations can be adopted on all matrices, endowing them with the possibility to get efficiently updated. In practice, given AB's target low rank small portion (r r) of the identity matrix is initialized with a diagonal matrix diag (N). This leads to be both part (the Ridge and the low-rank matrices) to be non-zero, allowing for good initialization of AB while ensuring XΣ dominates the spectrum. Subsequently, RidgeLoRA initializes the matrix A (with a Gaussian distribution, by default), and then compute matrix B to ensure the sum of these modules equal to the identity matrix. The whole procedure is also demonstrated in Alg. for a clear presentation.
It is noteworthy that this process also differs from SVD-based methods [17 that initialize matrices with the eigenvalues and eigenvectors of the pre- pre-trrined matrix. They require massive computation to conduct SVD on the full-rank original matrices and split corresponding portion to initialize the low-rank matrices, while Ridgel LoRA only requires SVD on the low-rank matrix A, quantitative comparisons can also be found at Table
Ridge Squashing Loss The initialization of the low-rank modules ensures that their sum equals an identity matrix I, hereby adhering to the "transform-caliratti restriction". Furthermore, to encourage the matrix explore a broader space, RidgeLoRA adds loss term LRS, pushing the weight update towards higher rank:
a LB ΣΣ MesEes S ALL
LRS
where L is the total number of layers the model, B denotes the number of matrix blocks in each layer, and || |l. is the nuclear norm [47], which measures the rank of the weight updates. Notably, PRS is set to negative value, which encourages rank updates the non-diagonal portions, hereby increasing the effectiveness of training the matrices A and B. We further incorporate this loss into the original objective:
L Lmodel + BRSCRS.
(5)
5

---

Method
Trainable
BoolQ
PIQA
SIQA (Acc.)
HellaSwag (Acc.)
WinGGrande (Acc.)
ARC-e (Acc.)
ARC-c
OBQA
Avg.
Parameters (%)
(Acc.)
(Acc.)
(Ace.)
(Ace.)
(Ace.)
Llama- 2
FFT
100% (6.748)
71.16
83.84
80.81
92.45
83.66
82.87
72.35
84.40
81.52
LoRA
0.30% (20.02M)
66.92
81.66
77.43
90.68
75.53
84.01
66.55
74.40
77.15
DoRA
0.32% (21 (21.37M)
67.50
81.23
48
90.78
76.16
84.34
67.06
74.20
77.35
PISSA
0.30% (20.02M)
69.81
83.68
79.94
93.60
80.51
85.94
71.42
79.20
80.51
KaSA
0,30% (20.05M)
66.52
80.85
76.77
90.28
75.37
83.54
65 87
73.60
76.60
RidgeLoRA
0.29% (19.42M)
71.43
83.08
81.27
93.24
81.53
86.66
71.76
82.00
Iaama-.,
FFT
100% (8,038)
70.88
83.62
79.02
90.93
83.11
85.10
75..4
83.20
81.40
LoRA
(21.00M)
71.67
DoRA
0.28% (22.37M)
71.43
88.63
80.45
94,92
83.66
93.27
80.80
85.20
84, 84.80
PISSA
0.26% (21.00M)
73.56
89.39
82.65
95.16
87.61
93.73
83.36
88.00
86.68
KaSA
0.26% (21.03M)
71.79
88.57
79.99
94.63
83.19
92.89
80.89
84.00
84.49
RidgeLoRA
0.26% (21 23M)
73.90
89.34
83.67
95.79
86.98
93.52
83.87
87.00
86.76
Mistral-r0.3-77
FFT
100% (7.25B)
71.92
85.64
79.73
92.19
85,08
83,46
73.63
84.00
81,96
LORA
0.29% (21.00M)
74.47
90.26
82.50
96.18
87.45
92.72
82.00
88.89
86.81
DoRA
0.31% (22.37M)
90.32
81.93
96.43
88.56
92.72
82.25
89.60
87.10
PISSA
0.29% (21)
75.18
90.48
82.86
96.61
87.77
93.10
82.34
91.00
87.42
KaSA
(21.03M)
89.61
96.21
RidgeLoRA
0.29% (21.23M) 74.63
91.35
81.88
96.38
88.63
93.18
82.59
90.80
87.43
Table 2: Accuracy comparison of Llama-2-7B (MHA). Llama-3.1-88 (GQA) and
Mistral-v0.3-7B (GQA) with various tuning methods on eight commonsense reasoning datasets. The numbers of trainable parameters of different methods are also included for a clear comparison. The highest accuracy scores achieved by low-rank methods are marked as Bold.
where Lmodel is the task-related loss for the PEFT-enhanced base model, and fRS is hyper- parameter controlling the influence of the rank-decreasing penalty. Our experimental results and ablation study also indicate that selecting a negative value further improves performance.
Experiments
4.1 Experimental Setup
Datasets In order to showcase the validity RidgeLoRA and demonstrate its good performance. In comparisons with state-of-the-art low rank methods, we conduct comprehensive experiments across different tasks, which are widely utilized for evaluation in previous works: namely, (i) Commonsense Reasoning, (ii) Math & Code Problems and (iii) Multi-modal Understanding tasks. We adopted all of its training split for fine-tuning for fixed number of steps to ensure fair comparisons. Reported metrics are evaluated on the official test splits.
Baselines Across all our experiments, we mainly compare RidgeLoRA with low-rank based tuning methods, namely vanilla LoRA [12, DoRA [16] and SVD-based PiSSA [17] and KaSA [18]. To further demonstrate the comparable performance RidgeL.RRA on par with FFT, we also include FFT in main results. Details of the experiments be found at Appendix B.4
Detailed Setups Throughout our experiments, we adopt a cosine learning rate schedule and use AdamW [8] as the optimizer. Unless otherwise specified, all LoRA variants share the same maximum learning rate for a given task. Concretely, we use learning rate of x 10-5 for Math & Code tasks, and 10 5 for Commonsense tasks. The rank of the low-rank matrices is set to 64 for Math & Code tasks, and for Commonsense tasks. For mooti- understanding, we follow the configuration proposed in DoRA [16] for a fair comparison.
4.2 Main Experimental Results
Commonsense Reasoning As showcased in Table across eight commonsense reasoning datasets, the average scores achieved by our proposed RidgeLoRA outperforms most of LoRA variants. When choosing Llama-2-7B as the base model, RidgeLoRA outperforms state-of-the-art baseline, i.e., PiSSA, by an improvement of 0.86% in the average accuracy. When analyzing each single dataset, we observe that Rod surpasses most of the tuning methods including FFT by large margins. For example, RidgeLoRA outperforms LoRA by 6.0% accuracy improvement WinoGrande with

---

Llama-2-7B as the base model, and is generally better on every dataset than KaSA across different base models. From the results we can observe that only in few datasets that RidgeLoRA may not surpass baselines, with the performance drops usually do not exceed 1 1%.
As for low-rank methods' performance in calibrating FFT, our proposed RidgeLoRA is the closest to the performance of FFT when selecting Llama-2-7B as the base model, which demonstrates the advantages of full-rank training of RidgeLoRA, with performance drop of only 0.15%. We also observe that when selecting Llama-3. 1-B and Mistral-v0.3-7B as the base model, the performances of FFT are commonly exceeded by low-rank based methods, which are also observed by many previous works [7 18 131, that base model may find it harder to converge given limited data in some tasks comparing to low-rank methods.
Math & Code Problems We also evaluate RidgeLoRA together with its baselines on math and code prob- lem solving datasets, where mod- els are usually required to gener- long-form arithmetic reasoning trace (math) and complete executable program (code), to further test the performance. As is demonstrated Table RidgeLoRA surpasses al- most all of its baselines with differ- ent LLMs as base models. Taking the math benchmarks with Llama-2-7B as the base model as an example, except for FFT, RidgeLoRA outperforms all of its low-rank baselines, even outper- forms KaSA by 7.26% on GSMSK. Similar results hold for Llama-3.1- 8B and Mistral-O.3-7B models, where RidgeLoRA outperforms its baselines by 0.56% in average and even passes FFT by an improvement of 2.42% in the result accuracy. In few datasets, RidgeLoRA may get surpassed by SVD-based PiSSA or KaSA, but with an average margin of only 0.30%.
Method
J Trainable Parameters (%)
GSMSK (Ace.)
MATH
HumanEyul (Possoe 1)
MBPP (+)
[Aec.]
(Puss
Llame-2-77
FFT DoRA
100% (6.71B)
66.32
17.72
(35,4)
15.2 (36.8)
22.6(18.3)
2.34% (161.3M)
52.25
3.08
25.0 (20.1)
36.2 (31.0)
PISSA
2.52% (159.9M)
56.29
9.28
36.8 (29.4)
(1955
(29.))
Ridgel oRA
(147.0M)
56.44
26.2 (23.8)
37.3 (29.9)
Llama-3. -33
FPT
100% (8.031)
23.84
(85.55
64.0
LoRA
2.0%% (167 XM)
25.97
28.30
51.2 (47.0)
67.5 (56.6)
DoRA
2.06% (169. (169.2M)
76.05
27.82
51.2 (48.2)
67.7 (56.6)
POSSA
2.05% (167.5) 8MD
77.92
30.26
53.7 (50.0)
65.1 (56.1)
7M)
(50.6)
RidgeLoRA
1.96% (160.7M)
78.07
30.12
53.7 (51.8)
69.8 (59.8)
EEistral.v0.. 3.771
100% (7,25B)
(47.0)
(45.1)
LoRA
2.26% (167.8M)
71.56
21.56
(39.()
62.2 (51.1)
DoRA
2.28% (169. 2M)
72.83
22.08
45.1 (39.0)
60.8 (51.3)
PISSA
2.26% (167.8M)
72.53
22.96
47.6
62.7 (51.3)
KaSA
(168.7M)
74.33
(40.2)
(50.5)
RidgeLoRA
2.17% (160.7M)
73.88
24.02
48.2 (41.5)
63.2 (53.7)
Table 3: Performances of different models on math & code problem benchmarks. For the code benchmarks, (+) denotes the enhanced datasets with more difficult test cases, whose metrics are in the parentheses, best performances of low-rank methods are marked as Bold.
Similar results hold for the code benchmarks. On the datasets with original test cases, RidgeLoRA surpasses every low-rank method and outperforms the best of its bascline performances with 0.65% improvement. As for datasets with enhanced test cases, i.e., the ones with (+), RodgeLoRA outperforms the low-rank methods by 1.4% improvement, showeasing its good performance and generalizability on hard cases. Like Commonsense Reasoning, we also observe several low-rank methods even achieve better performance than FFT, the reason is that the base models already possess enough capabilities, thus possible to achieve good results by tuning fewer parameters on the training set with less data.
Multi-modal Understanding In or- der to expand the applicable domain of RidgeLoRA, we further evaluate RidgeLoRA's performance in align- ing a pre-trained a language model with multi-modal projector. The evaluation results can be found at Ta- ble As is showcased, is show- cased that language model tuned with RidgeLoRA achieves better perfor- mances than DoRA, with an improve- ment of at most 1.4% % comparing to
Trainable Parameters
GQA
SQA
VQA
POPE
Avg.
Method
FFT
100%
61.9
66.8
58.2
85.9
68.20
LORA
4.61%
62.9
68.4
58.2
86.4
68.98
DoRA
4.63%
62.9
69.9
57.0
87.2
69.25
RidgeT
4.26%
61.9
69.2
58.4
87.6
69.28
Table 4: Multi-modal understanding evaluation results of different tuning methods on 4 vision-language tasks follow- ing the setups of DoRA [16), best performances are marked Bold.

---

Method
BoolQ
SIQA
HellaSwag (Ace.)
WinoCrunde (Ace.)
ARC-
ARC-c (Aeci)
Avg/C)
(Acc.)
(Acc.)
(Ace.)
(Ace.)
(Acc.)
(Acc.)
Ridge Enhanced Parallel Connections
LoRA-
67.
82. 27
78 (5
91.77
76 70.40
84.977 (4.97
688 (6..04)
74. (4040))
77. (10083)
(7.23
32-43em asy
1..7 say
ye.03
76. 76.20:re.00 (4100)
78. (8.34
PISSA
70.64 x9
83.57
(6.42
80.66
86,24
1_,33,0 10:00000
80.60 40)
80.85;
KaSA-Ridge
68.4
83. 83.13
79.84
92.57
77.74
85,14
68.5 68.52
79.20
79.32
Weight Initialization
Diagonal
75. 1.
Guvrrionaedatn
71.43
X3.08
81.27
93.24
N1.51
86.66
71.76
02.00
81.37
Kaiming (A)
70.88
83,08
80.50
93.77
82,00
85.48
72,10
83.80
81.45
Kaiming (24)
70.02
82.48
79.48
93.67
82.40
85.65
71.59
81
80.86
Xavier (N))
78.22
Xavier (2G)
67.99
$2.21
78.45
91.86
77.74
85:44
68.43
75.20
73.42
Table 5: Experimental results of Commonsense Dataset from two ablation studies, namely: (1) Parallel Connection; (2) Weight Initialization. .N denotes the normal distribution while 4 is the uniform distribution of corresponding method. AVg.(C) denotes the average accuracy score of Commonsense datasets. The subscripts represent the performance change of adding the Ridge enhancement compared to itself. Best scores of initialization methods are marked Bold.
DoRA on VQA". Consistent with the training of language-only models, the ratio of trainable parameters remains lower than low-rank baselines, demonstrating the lightweight advantages of RidgeLoRA.
4.3 Ablation Study
Apart from comprehensive evaluation with various types of model and different tasks comparing with baselines, which provides solid evidences that the RidgeLoRA surpasses vanilla LoRA together with its state-of-the totth---- variants, we also conduct ablation study to demonstrate the validity of different parts of RidgeLoRA fine-grained level.
Ridge Enhanced Parallel Connections Since we modify the connection between the newly trained modules and the original matrix, in order to provide in-depth analysis about the necessity of series connection, we also conduct abla- tion study with different LoRA variants enhanced with a Ridge alongside. The experimental results can be found in the first group Table 5 and Table 6 where performance improvements can be observed each method comparing to itself without the Ridge. On the average score of Com- monsense Dataset, KaSA is observed get enhanced with Ridge by a 2.72% improvement. PiSSA is also enhanced, with an average improvement of 0.37% on Commonsense and 181 improvement on math datasets. These results prove that RidgeLoRA can better approximate full-rank weight update, thus leading better performances. It is noteworthy that here we prove that the series connection RidgeLoRA not only surpasses the parallel connection, but also outperforms the other LoRA variants enhanced by Ridge, validating the effectiveness of series connection.
GSM8K
MATH
Method
(Acc.)
(Acc.)
LoRA Ridge
55.39
9.14
DoRA + Ridge
54.12((1.887)
(.724(00644)
PISSA Ridge
56.36(10.07)
9.56
KaSA Ridge
51.42
7.98(10.78)
Table 6: Performances math datasets existing LoRA variants enhanced with parallel Ridge.
Weight Initialization Methods Here we remain the series connection setup and further compare the performances of different initialization methods of matrix A, namely Gaus-
GSMSK
MATH
Method
(Acc.)
(Acc.)
sian 44. Xavier 45). and Kaiming [46] initialization, the results can be found at the second group in Table and Ta We also include an initialization where matrix A only has diagonal values before training, termed as Diag- onal. From the results we can observe that randomized matrix initialization always leads better performances, which is demonstrated by the fact that Diagonal gets sur- passed by others by a margin at least 0.78%. As for the normal distribution initialized methods, we can observe that Gaussian, with a higher variance in our default setup, has better performances than Kaiming and Xavier,
Diagonal
54.04
8.98
Gaussianaesalt
56.44
9.76
Kaiming (A)
56.14
9.62
Kaiming (U)
55.84
9.46
Xavier (N)
55.46
9,08
Xavier (U)
55.09
8.92
Table 7: Performances on math datasets of RidgeLoRA with different initializa- tions.
8

---

Prabecins Norm
Nuchar Norm
My
Wy
Wio
10x
Wo
FFT
RidgeLoRA
PISSA
KiSA
LoRA
Figure 2: Analysis the weight update patterns of the Self-Attention module of Llama-3.1- 8B (GQA). Norm here denotes the norm of weight updates, error lines come from variances between different layers.
with a 0.64% improvement on GSM8K. Moreover, ini- tializing with uniform distributions won't 't lead good performances when compared to normal distributions, their performance drop can achieve up to 2.20% on OBQA taking Kaiming (U) as an example.
Pattern Analysis of Weight Updates As an auxiliary analysis, we also performed a visualization of the equivalent parameter updates of different methods. As is illustrated in Figure constrained with the low-rank nature, the update norms of other LoRA variants are always lower than those of FFT, especially the vanilla LoRA. Benefit from the full-rank nature of RidgeLoRA, its updates are comparable or sometimes larger than that of FFT, showcasing its good representation capability.
Ridge Squashing Loss Fusion As we fuse Ridge Squashing Loss with the original loss to facilitate the training process, in order prove its validity, we conduct ablation study with different hyper- parameter setups. From the evaluation results from Table 8 we can observe that on all eight datasets, negative value 3Rs tends to lead better model performances, with a gap of most 1.21% % in average score across two groups. Comparing to the baseline group (w/o LRS), selecting a negative values always bring about better performance, with its scores all above baseline accuracy. Furthermore, we conduct Student's 's t-test [49 to showcase that selecting negative value for 9RS significantly improves the performance, which means increasing the rank of weight updates is favorable for good performances. Details can be found at Appendix B.1
5 Conclusion
In this paper, we propose RidgeLoRA, a novel PEFT approach that enhances the vanilla LoRA architecture through two key innovations: replacing the conventional parallel structure with series connection and incorporating a diagonal ridge term. Combined with the well designed initializa- tion strategy and ridge squashing loss, RidgeLoRA achieves superior capability while maintaining computational efficiency. Through rigorous theoretical analysis and comprehensive experimental evaluations, demonstrated that RidgeLoRA consistently outperforms existing approaches. Our work opens new possibilities for PEFT and provides valuable insights into PEFT methods.
Limitations
While RidgeLoRA is supported by comprehensive experiments and rigorous mathematical derivations. However, several limitations remain: 1) Its potential in scenarios such as continual learning [50 51 and model editing (52) where vanilla LoRA commonly applied has not yet been explored. Future work could investigate how RidgeLoRA extends to these settings and whether it offers advantages such contexts. 2) Due to computational constraints, we do not include experiments on very large models (e.g., those with more than 70B parameters). Nonetheless, the reported results, along with detailed ablations, provide compelling demonstration of the method's effectiveness on a range of models.

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Rohan Anil, Sebastian Borgeaud, lean-Baptiss Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
[3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint (rXiv:2302.139777 2023.
[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini urXiv:2307.09280 2023.
[5] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[6] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.2178. 2024.
[7] Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.14337 2024.
[8] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprins 2.. 2024.
[9] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, Advances in Neural Information Processing Systems, 2022. URL Forum?id-TG3KACXED
editors, net/
[10] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efffiientt fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220 235. 2023. doi: 10. s42256 -23-
URL https: nature. www.articces/
[11] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine- tuning for large models: A comprehensive survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. UtL.htttps:/ooenrevvvee wtt.e//ewww..eeetrooooodddccecoe
[12] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representutions, 2022. httthtttt/::::eteeee
[13] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148 2023.
10

---

[14] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney. and John Patrick Cunningham. LoRA learns less and forgets less. Transactions Machine Learning Research, 2024. ISSN 2835-8856. h.R.iattpe://pppeeeeee.ee.e aloEru2qCG Featured Certification.
[15] Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228, 2024.
[16]
Shih- Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 4 International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 100 mlr prwsss/235/liuu24400
32121. PMLR, 21-27 Jul 2024. URL https::/ mlr html
[17] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https::/ 6ZBHIEtdP4
net/ mettforummicn
[18] Fan Wang. Juyong Jiang, Chansung Park, Sunghun Kim, and Jing Tang. KaSA: Knowledge- aware singular- value adaptation of large language models. In The Thirteenth International Conference on Learning Representations 2025.
http://ppeereveee.eeeeeee
id=0QqNieeivq
[19] Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.1822 2024.
[20] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Forty-first International Conference on Machine Learning. 2024. URL https //openreview
[21] Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, and Xiaolin Huang, Flat-loRA: Low-rank adaptation over a flat loss landscape. In Forty-second International Conference on Machine Learning, 2025. URL htpp:t////peeeeeeeee...
3Q 3xSNNNI
[22] Zhihua Zhang. The matrix ridge approximation: algorithms and applications. Machine Learning. 97(3)2227-258, 2014. doi: 10.10077/110944-133444144 URL, http:://ddo.. /trgg100.100 ott.:o...rr/10...1001..... 810994-013-5431-y
[23] Gene H Golub and Christian Reinsch. Singular value decomposition and least squares solutions. In Handbook for Automatic Computation: Volume II: Linear Algebra, pages 134-151. Springer, 1971.
[24] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-lim: Truncation-aware singular value decomposition for large language model compression. arXiv preprint XrXiv:2403.07378 2024.
[25] Patrick CHen, Hsiang-Fu Yu. Inderjit S Dhillon, and Cho-Jui Hsich. DRONE: Data-aware low-rank compression for large NLP models. In A. Beygelzimer, Y. Dauphin, P. Liang, and
J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL
https://openrevieet het:/forum?id=sthizz22
[26] Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression. arXiv preprint arXiv:2408.. 09632, 2024.
[27] Alex Gittens and Michael Mahoney. Revisiting the nystrom method for improved large-scale machine learning. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 567-575, Atlanta, Georgia, USA, 17-19 Jun 2013, PMLR. URI. attps://proceedingg mlr press/ v2ts httess//22//iittess1s. http//gttttnnnn.mmmmm html

---

[28] Petros Drineas. Ravi Kannan, and Michael W Mahoney. Fast monte carlo algorithms for matrices i: Approximating matrix multiplication. SIAM Journal on Computing. 36(1):132-157,
2006. doi: 10. 10.11377 10.1177/0009733994442666
[29] Commandant Benoit. Note une méthode de résolution des équations normales provenant de l'application de la méthode des moindres carrés un système d' éequation linéaires en nombre inféricur à celui des inconnues (procédé du commandant cholesky). Bulletin géodésique. 2(1): 67-77, 1924. htt.hhtps:
org///// 10. 1007/00F000003330000
[30] Shaowen Wang, Linxi Yu, and Jian Li. LoRA-GA: Low-rank adaptation with gradient approxi- mation. In The eiihty-eighth Annual Conference on Neural Information Processing Systems, 2024. httpp:/::////
ftrp::///ppppooroooooooooooooooo net/forumiidd-aaLWiiii
[31] Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-effie efficcien Ilm finetuning. arXiv preprint (tt::2400.090444. 2024.
[32] Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604 2024.
[33] Qiushi Huang. Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
id=Tw]TZZ9cRS
[34] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:23 2023.
[35] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL
[36] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang, Parameter-efffcient tie-tunnng for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024.
[37] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- tuning for transformer- based masked language-models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 12.18866//1122222all-eeee org/
URL
www.ootg.........eo 2022 ac1-222c1-sh
[38] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 4582-4599 Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18333/v12221. acl-long.353, URL
2021. ac121.cc1-10 acl- aong.ll..oo.33333 353/
[39] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
[40] Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 5:208-215, 2024.
[41] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[42] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrrik, and Graham Neubig. To- wards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL httppp///.ppppppeeeeeeee/////eea
12

---

[43] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 133):1112218, 1936.
[44] Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus- Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
[45] Xavier Glorot and Yoshua Bengio. Understanding the diffieulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statisties, volume 9 of Proceedings of Machine Leurning Reseurch, pages 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL httt:tttt:////oceeeeiii htt.:/iss........eee/esee////
mlr wwwpssee////ep..pp wttp://llooootte prot.prroo//////oooooolllaa html
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[47] Ky Fan. Maximum properties and inequalities for the eigenvalues of completely continuous operators. Proceedings of the National Academy of Sciences, 37(11):760--6) 1951.
[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpap:///ppaereeeeeeeeeeeeeeeee id=Bkg6RiCqy7
[49] Student. The probable error of mean. Biometrika, 6(1):1-25, 1908. ISSN 00063444, 14643510. URL http///www..oooor///trtaaaaabbba argg/ttablee2331555.
[50] Hongming Piao, Yichen Wu, Dapeng Wu, and Ying Wei. Federated continual learning via prompt-based dual knowledge transfer. In Forty-first International Conference on Machine Learning, 2024.
[51] Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning. arXiv preprint arXiv:2501.13196 2025.
[52] Hongming Piao, Hao Wang, Dapeng Wu, and Ying Wei. A Towards compositional model editing. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025.
[53] Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
[54] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088) 1986. hatpss///// http:://wwwwna.uureee// com Comm/articess// 0ateceeeee//33300 333533a0
[55] Bernard L Welch. The generalization of 'student's poroblet'"ppoblem problem when several different population varlances are involved, Biometrika, 34(1-2):28 35, 1947. doi: 10.1093/biomee3444.-22222 URL
https://doi. org/oo..//.o 10. w11.:///////////// 11t..ommmm.4.....225..
[56] Franklin E Sattertwwaite. An approximate distribution of estimates of variance components. Biometrics bulletin, 20-):100-144 1946. URL http: h/wp:////wwww// wwwp:www.....twooopt/w/tppttwttw org/ stable/30000111
[57] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-411 Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18653/v1/N19-11160 1423. URL http::s:://aaaaahlooggoo org/N19-1423/
[58] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. wttttttstss/pnneiee
fet//forum?dd:rJ4mmmn
13

---

[59] Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11622 364, 2019.
[60] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: enchancee bert with disentangled attention. In International Conference on Learning Representations, 2021. URL/httpas httttttt//pppreeeiiiii //tp///ppeeeeeeeee//e/ee/eeeeeee /ttpp//wweeeeeeeeeeeooooroeeeeee
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Siappr_filessppppee/01////111/ pttp:/ppppperr////p//
Inc., 2017. URL http:: httttps://prroceedingss neurips. wee/paper_filee files/ Paper pdf
[62] Joshua Ainslie, James Teerph Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895-4901, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18633/v/2003. emnlp-mmin.25 298.. URL http:::/cccaaaahhoolgg/// org. 2023. emnlp-main. 298.
[63] Lianmin Zheng, Wei-L.in Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748 8763. PMLR, 18-24 Jul 2021. URL htl/httss//proceedingn mlr. www.srresssvvee httlv//1///aaaadd/d11 html
[65] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https / //openreview..... www./eeooodddd
[66]
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Minneapolis,
Language Technologies, Volume (Long and Short Papers), pages Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/1//11-1 1300. URL /ttptttt////llaatboooooo.oo
1300/
[67] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1432-7439, 2020.
[68] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing und the 9th International Joint Conference on Natu- ral Language Processing FEMNLP-JJCNLA pages 4463 4433, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/1//D19-4445. URL https://aclarthollggs 1454/19-144/
[69] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, edi- tors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1479-1442 URL https: http://///caaaaooooooororrgg
1472/
14

---

[70] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun, ACM, 64(9):99-105 August 2021. ISSN 0001-0782. doi: 10.1 10.1145/3474381. URL https: org/10. 1145/3474381
[71] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
2018.
[72] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun' Jun'ichi ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium, October- November 2018, Association for Computational Linguisties. doi: 10.18653/11D118-12660 URL https://acclarthholog
[73] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 110446, 2021.
[74] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
[75] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 033344, 2021.
[76] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 07732, 2021.
[77] Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024. URI.
[78] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[79] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreevie: wtt.://ww..eeefoumiiii//ii.. Tc_Bc
[80] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326. 2019.
[81] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- languaee models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pages 292-305, Singapore, December 2023, Association for Computational Lin- guistics. doi: 10. 1 10011865/v 1/2/333/1/222..o/ 1ttp///.222.m...... mapp-mmi... URL https:: manlp-mainn. 20/
org 2023
[82] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Liveseu, and Steven Bethard, editors, Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013. A Association for Computational Linguistics. URL httpp:://cclatthollaaa org D
15

---

[83] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641. 2019.
doi: 10.1 lach/aaclaa_0299900 URI.https: htt.pttps:/aallannollgg/. 2oo
[84] William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL httphttts///claatttiie
org/ I05- 5002/
[85] Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpin and Lucia Specia. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors, Proceedings of the I Ith International Workshop on Semantic Evaluation (Semfival-2017), pages 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v L/S17-22001 https::::: /aa::/acclaaaa www.aaaannnnnlll.aooooaooollllla wtg/s//-220000
[86] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1112-111 1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v//N18- 1101. URL http:://aaaaattholoooo org Norg/N18-118 1101/
[87] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: (0.18633/11DDDE 1264. URL / //acl wwwlanntloogyy..oooooooooooolooe hwwptthhoogg..oooooooooo org/D16-1264/
[88] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Joaquin CeiiooneroCaander Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3 -78-3-540-33422-6.
[89] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop recognising textual entailment, volume 1. Citeseer, 2006.
[90] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Satoshi Sekine, Kentaro Inui, Ido Dagan, Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini, editors, Proceedings of the ACI-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague, June 2007, Association for Computational Linguistics. URL
[91] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 788):1, 2009.
16

---

A Theorems and Math Derivations
A.1 Proof of Theorem
Theorem A.I. Let K E Rdxd be rank-k matrix (k S d), D E Raxd be diagonal matrix, M R be an arbitrary matrix. Given that M has Singular Value Decomposition (SVD) M UEV Let A E R2(n-1)x4 be a matrix whose rows are indexed by ordered pairs (p, q) where P.q {1,2, n and + Each row (pp)) E R lxk has entries given by Ups Vas for j E (1, k.. Let E R=(n-1)x1 be vector whose entries Cpy Ej=k+1 Up588 Vaj. where
8y denotes the corresponding entries in E. We have that
min K.D
K + - MI & (1(I - A(A AAA
(6)
Proof. We construct an upper-bound objective function via a relaxation of the original problem. Perform the singular value decomposition (SVD) of M as UEVT, Define K U., VT, where Eg is a rank-k diagonal matrix which is left for optimization. Then we have
min K + D Ml I min |UEVV K.D
UEV
F,,D
min
-E- 1
Σ.D
which serves as a tractable upper bound of the original objective.
Subsequently, we conduct two steps to find the upper bound of (7) The first step is to get the analystic value of D to minimize 7 while fixing Σx Then D can be represented in terms of Σx. In the second step, we substitute D with its expression in terms of kg, and then minimize 7 with respect to Σ.
Derivation of the optimal D. Fix Σg. here we denote A = U-- VV We need to minimize || - ? with respect to D. Since D is diagonal, i.c., Dij = 0 where # j. We have that
|| - All ((Ds - -Ais)²
"(,, AA,)) Σ A2g
The objective above is minimized when Di, = A1,s. Therefore, problem reduces to minimizing
Σ ( -
with respect to the rank-k diagonal matrix Σz.
Derivation of the optimal Σg. Let Y = Ex. Since Σ and En are diagonal matrices, Y is also a diagonal matrix. Let = diag($1 82, sn) and Eg = diag(1 b.,,)). Note that there are exactly k non-zero diagonal entries in Ex. Let Jo c {I,...., n be the set of n k indices where 1 0. Let JR C (1,2,...,1) n. be the set of i indices where Lj + 0. The target objective can be rewritten as follows
kn).
2
min Jg.U9bsc4 p#g
E(UYV Jpo)
2=
min Jg.
Σ
Σ j=1
Uniyy Vas
(8)
For a fixed choice of the index set Jr. this is a linear least square problems with respect to variables (Libes..
Let denote the k vector whose elements are 8, - L, for j E TheThere are n(n 1) pairs of (p, q) with p * For each pair (p, q). let a(pg).Je be 1 x & row vector with entries Up/ Vas for
17

---

j E Jk, Cpq.Jo = EEEE, UpssyVaj. Let Ar, be an n(n x & matrix whose rows are (pp)).yy and let CJ2 be an nln 1) x vector with entries Coa.Jo
For fixed index set Js, the minimization objective can be rewritten as min
|| CJ2 ll
Jz.2
CJ.- The minimum The solution to this least squares problem is given by x² - -AA Asz) AJ, value for fixed Jj is
G(Ja) 11(I - -1 A1, A,,,
Each choice of JR. is an upper bound of the target objective. The final step is to find the optimal of & indices JR that minimize (J)+). There are (#) possible choices for the set Jg. The overall minimum value of the above objective function is
min (11
||(T A,, An, )c,,, )e,.11
The upper bound the theorem can be obtained by setting JR {1, 2, ...,k....
A.2 Weight Updates FFT
Here present two perspectives the weight updates during FFT. While low-rank matrices usually used to approximate full-rank matrices, we propose enhancing this approximation with ridge term, which provides better calibration to the original full-rank update scheme. There are two equivalent ways to view the weight updates the original matrix W:
Additive update (Parallel connection): W + AW;
Multiplicative update (series connection): WWW
The full equivalence of additive update and full-rank training is derived further at Appendix A.5 freezing the original matrix W, these two representations are equivalent, as we can establish bi-directional mapping between them:
Given any additive update AW, we can find its multiplicative counterpart AW" W-1(W + AW);
Given any multiplicative update AW', we can find its additive counterpart as AW W -W
In both cases, theoretically W WWW w- AW" is noteworthy that vanilla LoRA can be viewed as low-rank approximation to the parallel connection, where AW is constrained to be the product of low-rank matrices. We will also demonstrate the experimental results of series and parallel connections with the setup of RidgeLoRA..
A.3 Singular Value Decomposition with Math Bounded Error
According to Eckart- Eckart-Young.. SVD on a matrix W to obtain a low-rank matrix We with rank k. The decomposition error, calculated with Frobenius Norm [53], is equal to the sum of the squares of the singular values of the compressed part of the matrix, which can be formalized as
Theorem 43 when conducting low-rank decomposition with
min(den:doud dout))
W
Σ
1=A+1
where = UEVT and VT o, is the singular value calculated with SVD, rearranged by descending order. This theorem also gives math bound to the error of a low-rank decomposition approximating the original full-rank matrix, any matrix Wt with rank won't better approximate the matrix W. However, if breaking the low-rank restrictions with full-ranked ridge, though with limited parameters, the approximation error gets way lower compared SVD-based methods. This justifies the necessity of restoring the rank of low-parameter modules gain better performances.
18

---

A.4 Weight Updates of Vanilla LoRA
LoRA achieves efficient fine-tuning with the low-rank matrices, which is particularly effective when the target task requires capabilities that are closely related to those acquired during pre- traininng necessitating only minimal weight adjustments. the implementation of vanilla LoRA [[2], an input signal Xn is computed in the following way:
Xaut = (Wn + ar AB),
(10)
where Rdduxr B Brxdous, denotes the preset low rank of the two matrices and a acts as the scaling hyper-parameter. The low-rank setup freezes the original matrix W, making the weights be updated lightweight. The architecture enables the extra matrices: and B be absorbed by the pre-trained weights, thus feasible to reorganized into model with exactly the same structure as the original one. The weights can be reorganized follows:
W W -AAB.
(11)
Here we take look at the weight update AW, is item- wise weight difference between fine-tuned weight W and the original weight W:
AW W W AB, r
(12)
where A and are low-rank matrices, making this module short of representation capabilities. According to Biderman et al. [14], Shuttleworth et al. [15] LoRA can not fully calibrate full-rank training and may entail intruder dimensions for the existence of low-rank matrices.
A.5 Gradient Analysis of Full-Rank Training
X
W
X
Figure 3:
Schematic of matrix computation with input matrix X and output Xoat-
Here we analyze the matrix computation detail and the difference between the weight updates of FFT and LoRA training. Specifically, is depicted in Figure Xout is obtained by matrix computation from Xin, which results in the following computation path:
Xout Xii W 8Xout
(13)
X.T
OW
Theorem A.2. FFT is equivalent to adding a full-rank matrix through parallel connection while freezing the original matrix.
8C has been backpropagated [54] subsequent b Gradient of FFT: Suppose that the gradient aNout modules. Our goal is to compute the corresponding gradient w.r.t. the input, ax... through applying
the chain rule. Specifically, given the transformation Xout X W, the gradient can be calculated as:
OL OC 8Xout OL
gw aw aXout aw
0X
X
(14)
from which we can observe that the gradient is calculated by the product of input activation and gradients from subsequent modules, which can be further utilized by optimizers like AdamW [48] to compute the weight updates.
& Gradient of an Additive (Parallel) Matrix: Here we derive how parameters of an extra inserted parallel matrix W update during training, where the computation is transformed to the formalization below:
Xout Xmn AW),
(15)
19

---

where AW is full-rant while W is kept frozen during training. The gradient of the inserted matrix can be calculated from:
OC
dL
8x
a(W + AW)
9AW
DAW
0Xout
a(W
AW)
DAW
(16)
0C 0Xout
dC
I 0X
XT
0X
a(W +AW)
in
where the gradient is exactly the same conducting full-rank training. If the original matrix W frozen during training while the inserted matrix is initialized (0) ( (ojanxdaus Xdout to ensure the output at starting point calibrate the original oulputs, every optimization step of parallel added matrix is exactly the same as FFT, hereby leading the same resulls.
B Details and Analysis of Experiments
B.1 Hypothesis Test of .RS Selection
In order to complement the analysis of our ablation study in Section 4.3 we conduct t-test [49 on
the accuracy scores to analyze the impact of åRS and compare the statistical significance to prove the validity of the proposed ridge squashing loss. Data points across different datasets and 3RS values are from Table 8 In the hypothesis test, we refer to μ1 and #2 as the means of the two experimental groups, negative value group and positive value group, respectively, while po represents a reference data point in which ridge squashing loss is not in effect for comparison in our t-test. The results of the hypothesis tests, including the t-statistics and v-vuue for each dataset, are summarized Table 9
Jns
BoolQ (Ace.)
PIQA (Acc.)
SIQA (Acc.)
HellaSwag (Ace.)
WinoGrande (Ace.)
ARC-e (Acc.)
ARC-e (Acc.)
OBQA (Ace.)
Avg. (Acc.)
Negative Values
1.0
71.12
83.68
80.81
93.22
79.87
85.82
73.21
81.40
81.14
-5 x 10-1
71.43
83.08
81.27
93.24
81.53
86.66
71.76
82.00
81.37
-2 10
70.55
82,70
80.50
93.59
81.22
86.28
70.90
81.80
80.94
-1 10-1
71.40
83.41
80.81
93.54
81.45
86.28
72.01
81.60
81.31
-3x 10-2
71.00
83.62
80.66
93.59
82.16
85.98
70.82
80.80
81.08
-2
70,94
83.68
80.30
93.43
80.51
87.04
72.44
81.60
10
71.73
83.03
82.29
93.37
80.03
86.36
72.18
81.40
81.30
5 10
71.43
82.81
80.55
93.52
80.19
86.20
72.35
81.80
81.11
1 10
70.82
83.19
80,55
93.26
79.87
85.82
71.08
79.20
80.47
0 (wlo CRS)
70.61
82.21
80.25
93.49
80.35
85.77
71.33
82,20
80.78
Positive Values
1 1 10
70.61
82.21 21
80.04
92.71
80.03
85 .48
72.01
79.60
80.34
5
70.58
82.64
80.25
92.60
80.82
86.28
71.84
79.80
80.60
10
71.12
82.54
80.04
92.83
80.74
85.56
71.33
80.8
80.62
2 10
70.64
82.75
80.19
92.99
80.82
85.35
72.53
80.40
80.71
5 10
70.42
81.66
80.45
92.73
80.82
85.56
71.84
80.80
80.54
1 10
70.51
82.10
79.94
92.84
80.03
85.77
71.59
81.00
80.47
2 x 10
71.15
81.77
80.14
92.91
79.87
85.56
71.25
80.20
80.36
x 10
70.30
81.72
79.68
92.73
80.19
85.19
70.82
80.60
80.15
82.21
80.40
92.79
80.27
84.93
71.67
79.80
80.24
Table 8: Ablation study with different Ans values with Llama-2-7B base model on Commonsense Datasets. Hyper-parameters parameters, except for BRS, are set to align with the main experiment. We evaluate across positive and negative in comparison with the 3RS 0 test, i.e., the one without ÅRS, to obtain conclusions about how the loss takes effect. Observations from metrics above can be used for hypothesis test.
Test 1: Will fusing losses by a negative ins lead to better performance?
Here we conduct t-test with accuracy scores from negative value group and Ao to prove the validity of fusing losses by negative fRS. This a right-tailed mean test, the null hypothesis and the alternative hypothesis can be expressed as
Ho P1 $ μo
(Null Hypothesis)
H P1 > Ho
(Altemattve Hypothesis)
20

---

where the test statistic is given by f the variance, is the mean of all data points from the same group. The null hypothesis is rejected if t > f0,01-1 i.e., p < a, where the degree of freedom dfi is n - 1. We set the significance level p < 0.10. From the left of Table 9 the differences are significant across five of eight datasets, which proves the validity of LRS and indicates that setting 0RS negative value will improve the performance.
the value and the degree of freedom df di #0 where n denotes the number of data points and S is 91/vmi
Dataset
6 (1 > (o)
Pr-value
Significant
Dataset
fg (M2 < (g)
Py-value
Significant
BoolQ
4.4488
0.0021
/
BoolQ
-3,2042
0.0056
PIQA
8,3350
0,0000
PIQA
-58002
0.0000
SIQA
3.0429
0.0160
/
SIQA
-3,4093
0.0063
HellaSwag
-1.4335
0.8104
X
HellaSwag
-9.8413
0.0000
/
Winogrande
1,4475
0.0912
Winogrande
-1,1511
0.2722
ARC-Easy
3,8073
0.0052
ARC-Easy
-4.1315
0.0008
ARC-Chale ARC-Challenge
1.9867
0.0822
/
ARC- Challene
-0.6627
0.5188
X
OpenBookQQ -3.1967
0.9937
X
OpenBookQA
-2,8795
0.0129
Table 9: Hypothesis test results for μt > (40 (left) and μa < μ사 (right). Across six out cight datasets, the average accuracy scores from the negative group μu is significantly greater than μog- Meanwhile, μl is significantly higher than (22 in six out of eight datasets, showcasing the necessity of selecting negative ERS.
Test 2: Will fusing losses by a positive 3RS worsen the performance compared to the negative group?
In order to fully perform ablation study to facilitate hyper-pane parameter selection, we also conduct Welch's t-test [55] to make comparison between the positive and negative group for the uncertainty in the variance of the two groups This is a left-tailed mean test, where the hypotheses can be expressed
Ho μ2 p사
(Null Hypothesis)
Ha P2 < pn
(Alternative Hypothesis)
In this case, using the Welch-Sattertwwaiee formula are calculated as follows:
56
55
Z2
T
1/
S7
+
n2
R1
(17)
S2 )2
df2
[8] [8]/n) /n)]"
+ (S (85/n2)²
The null hypothesis is rejected if 12 < <ta,,l/ i..,, < a. From the right table of Table six out of eight datasets witnessed with high significance, which illustrates selecting negative åRS can be much better than picking a positive value for it. Furthermore, this proves that increasing the rank of the non-diagonal part of the weight update is more beneficial obtain better performances.
B.2 Results of Natural Language Understanding Benchmark
We also evaluate RidgeLoRA with BERT 57 -like encoder models on Natural Language Un-
derstanding datasets from the GLUE (58 benchmark. We selected encoder-basee discriminative
models, RoBERTa [59] and DeBERTaV3 (60), to evaluate RidgeLoRA on NLU tasks on the GLUE benchmarks. Here present the results from the NLU datasets in Table TO
From the results in the table, we can observe that the performance improvements hold for small encoder models whose model sizes are usually less than IB. The Ridge-enhanced model achieved best results almost all the datasets, with its average scores also the highest amongst its baselines. On datasets where Ridge doesn't perform that well, the accuracy drop compared to other well-performed LoRA variants usually less than 1 %, showcasing the scalability and good performance of RiAgeLoRA.
21

---

MNLI
SST-2
MRPC
CoLA (Mec.)
QNLI (Acc.)
RTE (Ace.)
STS-B (Pec.)
QQP
Avg.
Method
(Acc.)
(Acc.)
(Acc.)
(Acc.)
RoBERla-base
FFT
87.6
90.2
63.6
78.7
86.3
LoRA
87.2
94.4
87.3
61.8
92.9
90.7
85.4
DoRA
87.4
94.4
87.5
61.6
93.1
78.3
90.6
87.9
85.1
PISSA
87.1
94.0
88.0
62.8
92.2
74.4
90.3
89.3
84.8
KaSA
87.0
94.4
88.7
61.9
92.4
77.3
90.9
90.6
85.4
Ridgel
87.3
94.4
88.1
62.2
92.8
77.9
90.8
90.7
85.5
RoBEKIo-
FFT
90.2
96.4
90.9
68.0
94.7
86,6
92.4
92.2
88.9
LoRA
90.8
96.2
89.8
65.1
94.7
85,2
91.6
91.8
88.2
DoRA
90.8
96.2
90.0
64.6
94.7
83.8
91.7
91.7
87.9
PISSA
90.8
96.0
90.1
66.3
94.7
85.9
91.4
91.5
88.3
KaSA
89.7
96.1
90.7
67.3
94.9
84.8
91.2
91.8
88.3
Ridget
90.9
96.3
91.1
68.3
94.8
86.2
91.4
92.0
88.9
BeBE DeBERTO RTa-v 3-base
FFT
90.7
96.0
88.7
69.0
93.7
86.2
90.8
92.5
88.5
LoRA
89.5
95.5
89.0
68.2
94.0
84,5
90.1
91.2
87.8
DoRA
89.6
95.4
89.0
68.2
94.0
84,5
90.3
87.2
87.3
PISSA
89.7
95.5
88.5
71.6
93.8
83.4
90.6
91.5
88.1
KaSA
89.6
95.3
89.5
68.7
87.9
Ridge!
89.7
96.1
88.5
70.2
94.3
85.0
90.7
91.4
88.2
DeBERTa-Ys v.3-large
91.8
96.9
92.2
75.3
96.0
92.7
93.0
91.4
LoRA
91.8
95.8
89.7
71.9
95.7
90.5
92.1
91.6
89.9
DoRA
91.7
95.8
89.5
71.7
PISSA
91.6
88.5
71.3
KaSA
91.7
88.6
71.7
90.0
Ridge!
91.7
96.0
90.2
72.2
96.0
92.5
92.3
92.0
90.4
Table 10: Performance of NLU tasks with BERT-like models. The best results for each model on each dataset are highlighted with Gray background, and the highest scores achieved by low-rank methods are marked as Bold. denotes parallel connected module with Ridge.
B.3 Resource Consumption LoRA Variants
In addition the theoretical analysis as presented in Table T. we also conduct real-world of different LoRA variants on its memory cost and time consumed for initialization. The results are as reported in Table TT As is depicted, the proposed RidgeLoRA hardly increases the computation and memory requirements during training. Consistent with the theoretical results, conducting SVD on original weights in order initialize LoRA weights (like PiSSA or KaSA) takes way more time than LoRA, DoRA and RidgeLoRA. DoRA always consumes more FLOPs and memory for its decoupling the updates of magnitude and direction.
B.4 Details Descriptions of Experiments
Base Models As we select language generation tasks for evaluation, we choose our base models from state-of-the-art LLMSs and select Llama-2-7B 4 Llama-3.1-8B (6 and Mistral-v0.3-7B models. The reason for this selection is that we want to confirm the validity of RidgeLoRA models. Also, since the Llama-2 model follows the Multi-Head Attention (61 MHA) architecture, unlike the Grouped-Quers Attention (62) GQA) that is adopted by the latter two models, it is beneficial to further prove the generalizability of our effects considering this variation the architecture, For multi-modal understanding, following DoRA's [16] setup, adopt vicuna-7b-1.. 1.5 63 as
more
'It is noteworthy that following the experimental setups of previous LoRA variants, we have chosen the base versions of the models (not the instruction or the chat version) in all of our experiments, from which we assessed the capability of different tuning methods.
22

---

Method
Parameters
Weight Memory/MIB
Computation
Initialization
Zlama-2-711
LaRA
6,758,404,196
19,988,481
13556.79
14,121,9868118
0.7296
DoRA
6,759,76 763, 968
21,348,352
13562/22
14,339,184, 656,3844
0.9264
PISSA
66758,4040096
19,988,480
13556. 79
14,121,9860 986, 818,048
269,7828
KaSA
6,758,418.44
20,002,816
13556.79
14,122.016,178,
RidgeLoRA
6,757,773.536
19,357,920
13554.26
14,118,362,939.8
1.41366
LoRA
6,898,323,456
159,907, 840
14116.46
14,406,541 541,667.328
3.0530
DoRA
64
6.899. 6828
161, 22227711 712
14121.90 20.9
16, 146, 124,, 374,016
3.2561
PISSA
64
6 0.898,322, 456
159 9077 840
14116646
14,408, 541 6666,,228
270.26.8
28090,4009960
160,825,344
14.410.420,7155552
RidgeLoRA
64
6,885,306.592
146,890,976
1406-9,40
14,579.550,638,000
7,5198
Llama-3.
LoKA
8,051 2,52,668
20.971.3220
16144.41
15,962,216.1866,
1.2216
DoRA
8,052. 609 022
22.347.776
16149.91
16,230,68 1,5.4,66666 2,666
0.9259
PISSA
8,051,232.766
20,971,520
15,962.246.008
200.8085
KoSA
8,051.247. 104
20,985,856
16144.41
15,962, 2,4,2,2754,,,,. 784
200.6371
RidgeLaRA
8,051.429.600
21,168,352
16145.20
15,960,098,, 903998600,000
1.6857
LoRA
64
8. 1,199,033, .408
167,772,160
16731.61
16,262, 797,777
3.0631
DoRA
64
8,199,409, 6644
169, 1991148,416
16737.12
1.410,377.45,.33
3.3557
PISSA
8,198.033,
167,772,
16,262.88 79.777799
KoSA
64
8, 198,950,000
668 689 6664
16731.61
76,264,772,845.00
202,9946
RidgeLaRA
8. 290,890,208
960
16703.04
16,245,713,928,1,
7.5247
Mistral-r-..77
LoRA
7, 1,268,990 995,,072
20,971.520
14579.93
15, (1161,234,685,p00
0.8190
DoRA
7,270.371
22,347,776
14585.44
15,429,670. 1,,922
PISSA
7 26, 995 (072
20,971,520
14579.93
15,161, 2234,224448 952
201.1598
KoSA
7,269, 2269.009.440
20,985,856
14579.93
15,161, 26,
202.1278
RidgeLoRA
7,269,191 900944
21,168,352
14580.72
15,159,087,00 082,22222220444
1.6571
LoRA
64
7,418,795,711
167,772, 160
15167.14
15,461 882, 396 672
3,0997
DoRA
7,417,171
169,148, 416
17,609,360
3,2655
PISSA
64
7,415,795 712
167.772, 160
15167. 14
15,461, 882,899, 672
202.0486
KoSA
7,416.713.216
168,689.666 664
15167.14
15.463, 7,1.761144486
204.0085
RidgeLoRA
7,408.652.512
160,628,990 960
15138.56
15.444.702,527.66
8.2065
Table 11: Test results of the resources (time, computation and memory) consumed of different LoRA variants. This corroborates the theoretical results from Table
the language model and adopt the CL.IP 64 -based vision projector in LLaVA representations of images.
65 to obtain the
Datasets We used comprehensive datasets evaluate the performance of RidgeLLRRA in adapting the language models to different down-stream tasks. Starting from where LoRA is most widely used, perform supervised fine-tuning (SFT,99 the Commonsense Reasoning datasets and evaluate it on eight corresponding datasets. In order to further evaluate the performances of RidgeLoRA on long-form generation tasks, we also include Math & Code Problems, which require the model to have a strong reasoning and generation ability; Furthermore, besides language-only evaluation, we extend our experiments Natural Language Understanding datasets, where language models are aligned to visual projectors to understand images.
i. For the Commonsense Reasoning datasets, following previous works, conduct multi-task train- ing with the training split of eight related datasets, namely BoolQ (66), PIQA 677 SocialIQA
(88]
HellaSwag 669, WinoGrande (70) ARC-Easy, ARC-Challenge [71 and Openbook 72
These datasets require the large models to fully utilize the real-world commonsense knowledge to answer a question.
il. As for Math&Code problems, we evaluate LLMs with GSMSK 77 and MATH 74) for math, vamanEval (75) and MBPP (76) for code capability. We conduct supervised fine-tuning with MetaMath 777 and Code-Feedback for math and code, respectively, to ensure there is no data leakage.
ii. For the multi-modal understanding datasets, we include GQA ScienceQA SQA in Table 4.. TextvQA (80) VQAT in Table and POPE [8] to how well the trained model fits with the vision projector and understands the images.
iv. We also adopt the GLUE benchmark for the NLU tasks, which consists of the following datasets:
Feedbace
23

---

a) Single Sentence Classification Tasks: SST-2 82 or the Stanford Sentiment Treebank's s goal to predict the sentiment (positivelnegattive)
reviews on different movies, which is binary classification task. CoLA [3 or the Corpus of I Linguistic Acceptability consists of sentences each annotated with whether is grammatical English sentence.
b) Similarity or Paraphrase Tasks: MRPC [4] or the Microsoft Research Paraphrase Corpus is identify if sentence pair consists of sentences paraphrases of each other. QQP or Quora Question Pairs O is to determine whether two questions are semantically equivalent, question pairs are collected from the website Quora. STS-B 85 or the Semantic Textual Similarity Benchmark a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. The task evaluate how similar chunks of texts are with a score from 1 to 5.
c) Language Entailment Tasks: MNLI [6] or the Multi-Genre Natural Language Inference is crowdsourced dataset of sentence pairs with entailment annotations, sourced from diverse materials like speech, fiction, and reports, evaluated on both in-domain and cross-domain sections using private labels. QNLI or the Question Natural Language Inference consists of question-paragraph pairs from Wikipedia, originally from the SQuAD [87] and post processed when building GLUE. RTE or the Recognizing Textual Entailment binary entailment task with small training dataset, which consists of sentence pairs from four annual textual entailment challenges [88-11]
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are mainly presented in the Abstract, which are also summarized as three-fold contributions in Sec. [
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
Details of RidgeLoRA are proposed in Sec. Bl followed by theoretical analy- sis (extensive proof and theories can also be found in Appendix Ap.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The hyper parameter setups are as disclosed in Section
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: We use open-source datasets for evaluation and provide links to them in the appendix. Code is attached with the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
CooeeSumiiii issiimisssiiii mnylicyy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The hyper parameter setups are as disclosed in Section 4.1 which are consistent with the Code affiliated as supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars in Figure when conducting pattern analysis. We also include hypothesis testing in Appendix B. which serves as a strong indication of the significance of adding squashing loss. Following multiple previous LoRA variants, we report accuracies in our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
The resources consumed by different variants are included in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This research is in compliance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This research adopts only publicly available datasets and LLMs.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This research poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Following major previous works, all open-source datasets and LLMs included in this research are properly cited and discussed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Code is attached to supplementary materials. Guidance also included the repository.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
30

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs only adopted for grammar checking sentence level.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non -tandtaddrrd components.
Please refer to our LLM policy (https: for what should or should not be described.
31


Reviews:
[{"review_id": "HnNk0XXmFJ", "reviewer": "Reviewer_pZr9", "summary": "This paper presents RidgeLoRA, a PEFT method for LLMs that addresses the representation limitations of standard LoRA. By incorporating a full-rank ridge module through a diagonal matrix and redesigning the architecture from parallel to series connections, RidgeLoRA enables more accurate approximation of full-rank weight updates while maintaining efficiency. Experiments across multiple domains demonstrate that RidgeLoRA outperforms other LoRA variants and often matches or exceeds the performance of full-rank training.", "questions": "1. Could the authors provide a more detailed analysis of RidgeLoRA’s performance on models larger than 8B parameters?\n2. Could the authors provide case studies to illustrate RidgeLoRA’s superior performance?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n\n1. Extensive experiments are conducted across multiple domains and a range of large language models.\n2. The writing is clear and the paper is easy to follow.\n\nWeakness: \n\n1. The evaluation is limited to models with up to 8B parameters; the performance of RidgeLoRA on larger models (e.g., 70B) remains unclear.", "was_revised": true, "final_justification": "Thanks for your response. The rebuttal has addressed my concerns. I will keep my positive score.", "rebuttal": "We thank Reviewer pZr9 for the professional review. Our rebuttal is as follows.\n\n> Could the authors provide a more detailed analysis of RidgeLoRA’s performance on models larger than 8B parameters?\n\nWe appreciate that reviewers have pointed the scaling experiments out. Indeed, experiments with larger models are absolutely crucial. We have now included additional scaling experiments with Llama-2-13b to address this concern. The results further validate our method’s effectiveness on larger, frontier LLMs.\n\n|                         | **BoolQ** | **PIQA** | **SIQA** | **HellaSwag** | **WinoGrande** | **ARC-e** | **ARC-c** | **OBQA** | **Avg** |\n| ----------------------- | --------- | -------- | -------- | ------------- | -------------- | --------- | --------- | -------- | ------- |\n| Llama-2-13b (LoRA)      | 68.66     | 82.86    | 78.97    | 92.36         | 77.58          | 85.56     | 68.00     | 76.40    | 78.80   |\n| Llama-2-13b (DoRA)      | 70.62     | 84.71    | 78.81    | 93.17         | 79.24          | 88.17     | 74.91     | 77.80    | 80.93   |\n| Llama-2-13b (PiSSA)     | 71.90     | 85.85    | 82.14    | 95.71         | 82.56          | 88.01     | 73.63     | 80.80    | 82.57   |\n| Llama-2-13b (KaSA)      | 69.61     | 83.35    | 79.53    | 92.87         | 78.37          | 86.49     | 69.03     | 76.80    | 79.51   |\n| Llama-2-13b (RidgeLoRA) | 74.20     | 87.76    | 82.50    | 95.22         | 86.42          | 90.28     | 78.07     | 83.60    | 84.76   |\n\nFrom this table, it can be seen that as the model parameter scale increases, RidgeLoRA consistently outperforms other baselines, with an average improvement of 4% over vanilla LoRA.\n\n> Could the authors provide case studies to illustrate RidgeLoRA’s superior performance?\n\nThank you for the suggestion. Here we provide an illustrative case demonstrating RidgeLoRA’s superior reasoning capability compared to vanilla LoRA tuning.\n\nConsider the following question:\n\n```\nQuestion: A rooster lays an egg on the top of a slanted roof. Which way does the egg roll?\nA. To the left, because the roof is slanted that way.\nB. To the right, since the slope leads downward.\nC. It stays on the roof because it's balanced.\nD. It doesn’t roll at all—roosters don’t lay eggs.\n```\n\nThe Llama model fine-tuned with vanilla LoRA answers:\n\n```\nThe answer is B. To the right, since the slope leads downward. \n```\n\nIn contrast, the RidgeLoRA-enhanced Llama model correctly identifies the trick and answers:\n\n```\nThe answer is D. It doesn’t roll at all—roosters don’t lay eggs. Roosters don’t lay eggs, so this is a trick question.\n```\n\nThis example shows that RidgeLoRA can better capture nuanced reasoning and common-sense knowledge. We will include more case studies in other domains such as math and code problems in the revised version to further illustrate RidgeLoRA’s advantages."}, {"review_id": "7kyPKfgWvh", "reviewer": "Reviewer_WvKZ", "summary": "This work proposes a novel low-rank adaptation approach for fine-tuning LLMs. In particular, they propose an approach where the low-rank adapter layers are applied after the linear weights instead of in parallel, and a diagonal matrix is added to each low-rank matrix decomposition to approximate full-rank updates. Additionally, they introduce an additional loss term that encourages the adapter weights to have higher rank, to ensure that high-rank regions of the parameter space are explored throughout training. Experiments were conducted at the 7-8B-parameter scale across a variety of downstream tasks (e.g., commonsense reasoning, math and coding) with comparisons against other low-rank approaches and full fine-tuning.", "questions": "Please see my questions in the Weaknesses section.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Presents a novel approach that improves the representational capacity of low-rank adaptation while maintaining computational efficiency.\n- Evaluations are comprehensive, covering a variety of downstream tasks and ablation studies, and demonstrate that the proposed RidgeLoRA approach can result in performance improvements over existing low-rank approaches and occasionally performance on par with that of full fine-tuning.\n\nWeaknesses:\n- Evaluations are limited to the 7-8B model scale, and it is possible that the approximation quality and thus the gap vs. the full fine-tuning baseline shows different trends at other model scales.\n- Performance improvements are relatively small vs. other low-rank baselines with more recent LLMs (Llama-3.1-8B and Mistral-v0.3-7B).\n- The nuclear norm plot in Figure 2 appears to suggest that the weight updates still remain relatively low rank (similar to that of PiSSA) on GQA. How do these trends change across different datasets? On a related note, prior works suggest that the required rank r on the adapters can vary significantly across different downstream tasks (i.e., they have different intrinsic dimensionalities) [1]. Perhaps an interesting study would be to compare the same set of methods across tasks representative of different intrinsic dimensionalities to demonstrate that the proposed method is indeed more effective at approximating more challenging tasks that require higher rank updates.\n- It is not completely obvious to me whether the series connection necessarily has an advantage over the parallel approach in terms of predictive performance. The improvements observed in Table 6 seem very marginal. Perhaps the claim about the necessity of the serial connection requires additional validation.\n- Could the authors provide additional explanation as to why the numbers for Llama-3.1-8B and Mistral-v0.3-7B in Table 2 are often much worse than all of the low-rank methods? I am aware that the paper mentions this is similar to the trends observed in other papers, but am wondering if adding sufficient regularization can update the numbers to be more reflective of true full fine-tuning performance.\n\nReferences:\n[1] LoRA Learns Less and Forgets Less (Biderman et al., 2024)", "was_revised": true, "final_justification": "All of my remaining concerns have been addressed during the discussion period. I keep my positive view of the paper and maintain the original score of 5.", "rebuttal": "We appreciate the review of Reviewer WvKZ for its constructive suggestions. Our detailed rebuttal is as follows.\n\n> The nuclear norm plot in Figure 2 appears to suggest that the weight updates still remain relatively low rank (similar to that of PiSSA) on GQA.\n\nAnalyzing rank behavior across tasks with varying intrinsic dimensionality is indeed insightful. In response, we performed additional analysis using checkpoints from a diverse set of tasks (model weights trained on MetaMathQA, CommonSense, and Code-Feedback) and found that RidgeLoRA consistently yields higher nuclear norms compared to other LoRA variants. This suggests that RidgeLoRA is more capable of adapting to tasks that require higher-rank updates.\n\nDue to the constraints of the rebuttal format (e.g., no figures or external links), we are unable to include the full visualizations and detailed comparisons here. However, we will provide comprehensive nuclear norm plots and cross-task analysis in the final version of the paper.\n\n> It is not completely obvious to me whether the series connection necessarily has an advantage over the parallel approach in terms of predictive performance. The improvements observed in Table 6 seem very marginal. Perhaps the claim about the necessity of the serial connection requires additional validation.\n\nHere we include results as instructed by Reviewer S5wb to prove that the multiplicative structure (or serial connection) itself contributes a lot to the good performance.\n\n|                          | **BoolQ** | **PIQA** | **SIQA** | **HellaSwag** | **WinoGrande** | **ARC-e** | **ARC-c** | **OBQA** | **Avg** |\n| ------------------------ | --------- | -------- | -------- | ------------- | -------------- | --------- | --------- | -------- | ------- |\n| Llama-2-7b (Ridge)       | 71.43     | 83.08    | 81.27    | 93.24         | 81.53          | 86.66     | 71.76     | 82.00    | 81.37   |\n| Llama-2-7b (Freeze)      | 70.83     | 82.92    | 80.19    | 93.34         | 80.98          | 86.20     | 71.84     | 81.80    | 81.01   |\n| Llama-3.1-8B (Ridge)     | 73.90     | 89.34    | 83.67    | 95.79         | 86.98          | 93.52     | 83.87     | 87.00    | 86.76   |\n| Llama-3.1-8B (Freeze)    | 73.13     | 89.45    | 83.52    | 95.72         | 87.06          | 93.64     | 81.91     | 87.60    | 86.50   |\n| Mistral-v0.3-7B (Ridge)  | 74.63     | 91.35    | 81.88    | 96.38         | 88.63          | 93.18     | 82.59     | 90.80    | 87.43   |\n| Mistral-v0.3-7B (Freeze) | 73.35     | 91.46    | 83.27    | 96.34         | 88.56          | 92.97     | 81.06     | 90.4     | 87.18   |\n\nTo better isolate the effect of the multiplicative structure from the trainable ridge term $\\Sigma$, we conducted an additional experiment where we kept the ridge values **frozen** as the identity matrix (i.e., $\\Sigma = I$). This effectively removes the contribution of the trainable ridge term while preserving the multiplicative nature of the update.\n\nWe observe that although the performance of this is slightly weaker than the full RidgeLoRA with a learnable $\\Sigma$, it still significantly outperforms standard additive LoRA and most of its known variants. This result suggests that the **multiplicative structure itself contributes meaningfully to performance improvements**, even without additional trainable parameters from $\\Sigma$. In other words, the benefit of RidgeLoRA is not solely due to parameter count or flexibility, but also due to the structural advantage of multiplicative adaptation.\n\n> Could the authors provide additional explanation as to why the numbers for Llama-3.1-8B and Mistral-v0.3-7B in Table 2 are often much worse than all of the low-rank methods? I am aware that the paper mentions this is similar to the trends observed in other papers, but am wondering if adding sufficient regularization can update the numbers to be more reflective of true full fine-tuning performance.\n\nAs discussed in the paper, similar trends have been reported in prior work (e.g., DoRA, PiSSA, and KaSA), particularly in instruction tuning settings, where full fine-tuning (FFT) can be prone to overfitting or instability without careful regularization. In this context, LoRA can be seen as a form of regularization that constrains the parameter update space, which may help explain why it sometimes outperforms unconstrained FFT. Our work focuses specifically on the low-rank setups, and comprehensive results showcase that RidgeLoRA outperforms other LoRA variants.\n\nIn our experiments, we adopted commonly used FFT configurations without extensive hyper-parameter tuning. One key reason for this choice is fairness: although our primary focus is on comparing **low-rank adaptation methods**, we include FFT (training without regularization on weight updates) as a reference. To ensure meaningful comparisons, all methods within the same category (FFT or low-rank) are trained under consistent hyper-parameters. That is, all low-rank methods, including RidgeLoRA, share the same setup, and all FFT experiments follow a fixed default configuration."}, {"review_id": "mOOZUp0NQO", "reviewer": "Reviewer_S5wb", "summary": "The authors propose a PEFT method called RidgeLoRA which makes use of an extra trainable diagonal ridge term in addition to the low-rank factorization used in LoRA. They show their method performs better than some of the state-of-art PEFT methods like DoRA and PiSSA. There is also good ablation studies on the importance of the ridge term.", "questions": "- Why could RidgeLoRA use fewer parameters than LoRA for Llama-2-7B in Table 2 and Table 3 when it has extra diagonal parameters?\n\n- The use of ridge term also bears some resemblance to the method IA^3, which makes use of multiplicative diagonal scaling of the weight matrices as trainable parameters. The authors can consider including it in the discussion of related works. \n\nFew-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning\nLiu et al 2022", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- The authors propose an interesting PEFT method called RidgeLoRA, which makes use of multiplicative updates to the weights and an extra ridge term. They show that their method performs better than some of the state-of-art PEFT methods like DoRA and PiSSA. They also have good set of ablation studies on their design choices. Overall it is quite clearly written and a solid paper. \n\n- There is a set of good ablation studies performed to evaluate the different design choices of the method. Of particular importance is Table 6 where the authors show the importance of adding the ridge term, which helps other PEFT methods too. \n\n- For weaknesses there seems to be some mismatch in the comparison with LoRA. The current RidgeLoRA algorithm is a multiplicative update to W while LoRA is an additive update to W. It is not an apples-to-apples to comparison between the rank of a additive and the rank of a multiplicative update. So apart from the trainable ridge term \\Sigma, has the authors considered comparing the effect of additive VS multiplicative updates by setting \\Sigma = I? \n\n- I also find the use of the terms serial and parallel as illustrated in Figure 1 slightly confusing. It seems to me much more natural to describe the updates as additive or multiplicative to the weight matrix W. \n\n- Theorem 3.1 is related to the approximation property of adding a ridge term to matrices in general, and does not seem to have any direct relevance to finetuning performance. We all know that adding more parameters or increasing the rank should decrease the approximation error, so the theorem doesn't really contain any special insight on why the algorithm works.", "was_revised": true, "final_justification": "I am keeping my recommendation of borderline accept/accept since most reviewers agree that this is a solid paper, and the authors have sufficiently addressed our concerns. My minor complaints were over the use of terminologies and writing clarity issues, but I believe the authors would be able to fix them.", "rebuttal": "We sincerely appreciate Reviewer S5wb's insightful comments and suggestions. Our point-by-point responses are provided below.\n\n> The current RidgeLoRA algorithm is a multiplicative update to W while LoRA is an additive update to W. It is not an apples-to-apples to comparison between the rank of a additive and the rank of a multiplicative update. So apart from the trainable ridge term \\Sigma, has the authors considered comparing the effect of additive VS multiplicative updates by setting \\Sigma = I?\n\nThank you for this suggestion. To better isolate the effect of the multiplicative structure from the trainable ridge term $\\Sigma$, we conducted an additional experiment where we kept the ridge values **frozen** as the identity matrix (i.e., $\\Sigma = I$). This effectively removes the contribution of the trainable ridge term while preserving the multiplicative nature of the update.\n\n|                          | **BoolQ** | **PIQA** | **SIQA** | **HellaSwag** | **WinoGrande** | **ARC-e** | **ARC-c** | **OBQA** | **Avg** |\n| ------------------------ | --------- | -------- | -------- | ------------- | -------------- | --------- | --------- | -------- | ------- |\n| Llama-2-7b (Ridge)       | 71.43     | 83.08    | 81.27    | 93.24         | 81.53          | 86.66     | 71.76     | 82.00    | 81.37   |\n| Llama-2-7b (Freeze)      | 70.83     | 82.92    | 80.19    | 93.34         | 80.98          | 86.20     | 71.84     | 81.80    | 81.01   |\n| Llama-3.1-8B (Ridge)     | 73.90     | 89.34    | 83.67    | 95.79         | 86.98          | 93.52     | 83.87     | 87.00    | 86.76   |\n| Llama-3.1-8B (Freeze)    | 73.13     | 89.45    | 83.52    | 95.72         | 87.06          | 93.64     | 81.91     | 87.60    | 86.50   |\n| Mistral-v0.3-7B (Ridge)  | 74.63     | 91.35    | 81.88    | 96.38         | 88.63          | 93.18     | 82.59     | 90.80    | 87.43   |\n| Mistral-v0.3-7B (Freeze) | 73.35     | 91.46    | 83.27    | 96.34         | 88.56          | 92.97     | 81.06     | 90.4     | 87.18   |\n\nWe observe that although the performance of this is slightly weaker than the full RidgeLoRA with a learnable $\\Sigma$, it still significantly outperforms standard additive LoRA and most of its known variants. This result suggests that the **multiplicative structure itself contributes meaningfully to performance improvements**, even without additional trainable parameters from $\\Sigma$. In other words, the benefit of RidgeLoRA is not solely due to parameter count or flexibility, but also due to the structural advantage of multiplicative adaptation.\n\n> I also find the use of the terms serial and parallel as illustrated in Figure 1 slightly confusing. It seems to me much more natural to describe the updates as additive or multiplicative to the weight matrix W.\n\nThank you for the suggestion. We agree that describing the updates as *additive* or *multiplicative* to the weight matrix W may be more intuitive than using the terms *parallel* and *serial*. In the revision, we will revise the terminology and clarify the expressions—i.e., $X(W + AB)$ (additive) and $X(\\lambda \\Sigma + AB)W$ (multiplicative)—to improve clarity and readability.\n\n> Theorem 3.1 is related to the approximation property of adding a ridge term to matrices in general, and does not seem to have any direct relevance to finetuning performance. We all know that adding more parameters or increasing the rank should decrease the approximation error, so the theorem doesn't really contain any special insight on why the algorithm works.\n\nWe thank Reviewer S5wb for the thoughtful comment. While Theorem 3.1 primarily addresses the approximation error, this quantity is tightly connected to the model's ability to fit the parameter updates during finetuning. In particular, a lower approximation error indicates that the modified weight matrix more faithfully captures the intended update directions. Theorem 3.1 thus provides a theoretical foundation for why adding the ridge term leads to better representational fidelity. Rather than being a generic observation that more parameters help, the theorem quantitatively shows how the ridge term specifically improves the approximation in a non-trivial manner, helping explain the empirical gains we observe.\n\n> Why could RidgeLoRA use fewer parameters than LoRA for Llama-2-7B in Table 2 and Table 3 when it has extra diagonal parameters?\n\nWe thank Reviewer S5wb for carefully examining our work, here we further clarify the details of RidgeLoRA to avoid misunderstandings. As illustrated in Figure 1, the reason RidgeLoRA can use fewer parameters than LoRA despite introducing an additional diagonal term lies in the structural difference between the two methods.\n\nWhile vanilla LoRA adopts a parallel connection (or additive) structure of the form $X(W+AB)$, RidgeLoRA employs a *series connection* (or multiplicative) formalized as $X(\\lambda\\Sigma + AB)W$. This formulation not only introduces the ridge term $\\Sigma$, but also changes the shape of matrix B from $d_{out}\\times r$ to $d_{in} \\times r$, as reflected in the right part of Figure 1.\n\nBecause of this change, the number of trainable parameters in $B$ is reduced. As a result, even with the added diagonal matrix $\\Sigma$, which is rather lightweight compared with $B$, RidgeLoRA maintains a lower overall parameter size. Comparisons on the parameter size can also be found in the second column of Table 1.\n\n> The use of ridge term also bears some resemblance to the method IA^3, which makes use of multiplicative diagonal scaling of the weight matrices as trainable parameters. The authors can consider including it in the discussion of related works.\n\nWe sincerely thank Reviewer S5wb for pointing this out. IA^3[1] introduces trainable multiplicative diagonal scaling to adapt pre-trained models. As another PEFT method that is different from LoRA and its variants, IA^3 bears conceptual resemblance to RidgeLoRA’s use of the ridge term. We will include a discussion of IA^3 in the related work section and add the appropriate citation in the revised version.\n\n[1] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel.2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems 35 (2022), 1950–1965."}, {"review_id": "gJkSuasGlJ", "reviewer": "Reviewer_PKfP", "summary": "The paper proposes **RidgeLoRA**, a novel PEFT method that replaces LoRA’s parallel structure with a series connection and adds a diagonal ridge term to enhance expressiveness. Combined with improved initialization and a rank-promoting loss, RidgeLoRA achieves strong performance and efficiency, consistently outperforming prior approaches and offering new insights into PEFT design.", "questions": "Can the authors provide further empirical justification for why this change leads to improved representation and learning dynamics? For example, include further analysis or ablation comparing intermediate representations (e.g., gradients) between series and parallel designs to explain the observed performance gains.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a rigorous mathematical derivation of RidgeLoRA’s representational advantage over standard LoRA. The inclusion of Theorem 3.1 and detailed analysis underpins the claims with solid theory.\n\n2. The work relaxes the strict initialization constraints of LoRA, enabling broader and more effective initialization strategies.\n\n3. By mitigating the rank deficiency in LoRA through a lightweight ridge term, the paper tackles a core bottleneck in low-rank adaptation and shows improvements across strong baselines.\n\n4. Extensive empirical evaluations across a range of tasks (commonsense reasoning, math/code generation, and multi-modal understanding) using multiple backbone models (LLaMA-2, LLaMA-3.1, Mistral) validate RidgeLoRA’s effectiveness.\n\n5. RidgeLoRA is lightweight, and can be easily implemented in existing LoRA pipelines with minimal code changes. This makes it practical for both research and production use.\n\nWeaknesses:\n\n1. The paper acknowledges the absence of scaling experiments due to resource limitations, which slightly limits conclusions on its applicability to frontier LLMs.", "was_revised": true, "final_justification": "I believe this is a very good paper, though it doesn’t quite reach the level of having a groundbreaking impact. Therefore, I will maintain my decision to accept it.", "rebuttal": "We sincerely appreciate Reviewer PKfP's insightful review. Our point-by-point responses are provided as follows.\n\n> The paper acknowledges the absence of scaling experiments due to resource limitations, which slightly limits conclusions on its applicability to frontier LLMs.\n\nWe appreciate that reviewers have pointed the scaling experiments out. Indeed, experiments with larger models are absolutely crucial. We have now included additional scaling experiments with Llama-2-13b to address this concern. The results further validate our method’s effectiveness on larger, frontier LLMs.\n\n|                         | **BoolQ** | **PIQA** | **SIQA** | **HellaSwag** | **WinoGrande** | **ARC-e** | **ARC-c** | **OBQA** | **Avg** |\n| ----------------------- | --------- | -------- | -------- | ------------- | -------------- | --------- | --------- | -------- | ------- |\n| Llama-2-13b (LoRA)      | 68.66     | 82.86    | 78.97    | 92.36         | 77.58          | 85.56     | 68.00     | 76.40    | 78.80   |\n| Llama-2-13b (DoRA)      | 70.62     | 84.71    | 78.81    | 93.17         | 79.24          | 88.17     | 74.91     | 77.80    | 80.93   |\n| Llama-2-13b (PiSSA)     | 71.90     | 85.85    | 82.14    | 95.71         | 82.56          | 88.01     | 73.63     | 80.80    | 82.57   |\n| Llama-2-13b (KaSA)      | 69.61     | 83.35    | 79.53    | 92.87         | 78.37          | 86.49     | 69.03     | 76.80    | 79.51   |\n| Llama-2-13b (RidgeLoRA) | 74.20     | 87.76    | 82.50    | 95.22         | 86.42          | 90.28     | 78.07     | 83.60    | 84.76   |\n\nFrom this table, it can be seen that as the model parameter scale increases, RidgeLoRA consistently outperforms other baselines, with an average improvement of 4% over vanilla LoRA.\n\n> Can the authors provide further empirical justification for why this change leads to improved representation and learning dynamics? For example, include further analysis or ablation comparing intermediate representations (e.g., gradients) between series and parallel designs to explain the observed performance gains.\n\nWe thank Reviewer PKfP for this good suggestion that can definitely provide more evidence for the improved training efficiency. We tracked the gradient norm (`grad_norm`) over training steps for RidgeLoRA, PiSSA, KaSA, DoRA and vanilla LoRA. RidgeLoRA and PiSSA showcase higher gradient norms than other LoRA variants, indicating stronger learning dynamics. This supports that RidgeLoRA improves representation and training efficiency. We will add the curve of `grad_norm` from different variants to the revised version.\n\n| **step** | **LoRA** | **DoRA** | **KaSA** | **PiSSA** | **RidgeLoRA** |\n| -------- | -------- | -------- | -------- | --------- | ------------- |\n| 0        | 1.393    | 1.729    | 1.415    | 31.488    | 45.469        |\n| 1        | 1.425    | 1.763    | 1.447    | 32.165    | 46.546        |\n| 2        | 1.429    | 1.761    | 1.446    | 32.018    | 46.193        |\n| 3        | 1.394    | 1.726    | 1.42     | 31.466    | 44.341        |\n| 4        | 1.421    | 1.76     | 1.445    | 31.915    | 43.388        |\n| 5        | 1.424    | 1.762    | 1.447    | 31.504    | 40.92         |\n| 6        | 1.43     | 1.769    | 1.454    | 30.527    | 38.775        |\n| 7        | 1.428    | 1.763    | 1.45     | 30.193    | 36.892        |\n| 8        | 1.422    | 1.753    | 1.441    | 29.606    | 34.801        |\n| 9        | 1.454    | 1.78     | 1.472    | 27.18     | 32.838        |\n| 10       | 1.468    | 1.788    | 1.49     | 26.729    | 32.144        |\n| 11       | 1.489    | 1.809    | 1.507    | 26.221    | 30.791        |\n| 12       | 1.493    | 1.809    | 1.513    | 23.528    | 31.732        |\n| 13       | 1.58     | 1.875    | 1.593    | 23.203    | 31.284        |\n| 14       | 1.598    | 1.889    | 1.616    | 22.453    | 23.761        |\n| 15       | 1.648    | 1.918    | 1.655    | 22.676    | 30.052        |\n| ...      | ...      | ...      | ...      | ...       | ...           |\n| 995      | 0.302    | 0.286    | 0.295    | 1.273     | 1.127         |\n| 996      | 0.422    | 0.405    | 0.409    | 1.102     | 1.425         |\n| 997      | 0.332    | 0.321    | 0.315    | 0.938     | 1.184         |\n| 998      | 0.303    | 0.293    | 0.307    | 0.822     | 0.985         |\n| 999      | 0.273    | 0.279    | 0.273    | 1.524     | 0.943         |\n| 1000     | 0.344    | 0.336    | 0.340    | 1.316     | 1.335         |\n| 1001     | 0.397    | 0.393    | 0.397    | 1.015     | 1.084         |"}]

Author rebuttal:

Meta-review: This paper proposes RidgeLoRA, a parameter-efficient fine-tuning method for language models. RidgeLoRA introduces a diagonal component alongside LoRA weights and enhances their representation capacity through multiplicative updates. Both empirical and theoretical studies demonstrate its advantages over other PEFT methods. All reviewers recognize the paper’s contributions, and the authors have addressed most of their concerns in the rebuttal. I concur with the reviewers and recommend acceptance of the paper.

The authors are encouraged to incorporate the reviewers’ suggestions in the camera-ready version, such as adding the 13B experiment from the rebuttal and improving the clarity of certain terminology as noted by Reviewer S5wb.
Decision: Accept (spotlight)