PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Authors: Fanxu Meng
Zhaohui Wang
Muhan Zhang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: PEFT
LoRA
LLM
Finetune
SVD
Primary area: optimization_for_deep_networks
TL;DR: PiSSA reinitializes LoRA's parameters by applying SVD to the base model, which enables faster convergence and ultimately superior performance. Additionally, it can reduce quantization error compared to QLoRA.

Abstract: To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low-rank adaptation (LoRA) method approximates the model changes $\Delta W \in \mathbb{R}^{m \times n}$ through the product of two matrices $A \in \mathbb{R}^{m \times r}$ and $B \in \mathbb{R}^{r \times n}$, where $r \ll \min(m, n)$, $A$ is initialized with Gaussian noise, and $B$ with zeros. LoRA **freezes the original model $W$** and **updates the "Noise \& Zero" adapter**, which may lead to slow convergence. To overcome this limitation, we introduce **P**r**i**ncipal **S**ingular values and **S**ingular vectors **A**daptation (PiSSA). PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices $A$ and $B$ with the principal components of the original matrix $W$, and put the remaining components into a residual matrix $W^{res} \in \mathbb{R}^{m \times n}$ which is frozen during fine-tuning.
Compared to LoRA, PiSSA **updates the principal components** while **freezing the "residual" parts**, allowing faster convergence and enhanced performance. Comparative experiments of PiSSA and LoRA across 11 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently outperforms LoRA under identical experimental setups. On the GSM8K benchmark, Gemma-7B fine-tuned with PiSSA achieves an accuracy of 77.7\%, surpassing LoRA's 74.53\% by 3.25\%. Due to the same architecture, PiSSA is also compatible with quantization to further reduce the memory requirement of fine-tuning. Compared to QLoRA, QPiSSA (PiSSA with 4-bit quantization) exhibits smaller quantization errors in the initial stages. Fine-tuning LLaMA-3-70B on GSM8K, QPiSSA attains an accuracy of 86.05\%, exceeding the performances of QLoRA at 81.73\%. Leveraging a fast SVD technique, PiSSA can be initialized in only a few seconds, presenting a negligible cost for transitioning from LoRA to PiSSA.

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
Fanxu Meng Zhaohui Wang', Muhan Zhang 1.2*
Institute for Artificial Intelligence, Peking University
2 State Key Laboratory of General Artificial Intelligence, Peking University
Abstract
To parameter-efficiently fine-tune (PEFT) large language models (LLMs), the low- rank adaptation (LoRA) method approximates the model changes AW E Rmxx through the product of two matrices A E R and B gx where r < min (m, n.), A is initialized with Gaussian noise, and B with zeros. LoRA freezes the original model W and updates the "Noise & Zero" adapter, which may lead to slow convergence. To overcome this limitation, we introduce Principal Singular values and Singular vectors Adaptation (PiSSA). PiSSA shares the same architecture as LoRA, but initializes the adaptor matrices A and B with the princi- pal components of the original matrix W, and put the remaining components into a residual matrix Wree E RRmx which is frozen during fine-tuning. Compared to LoRA, PiSSA updates the principal components while freezing the "residual" parts, allowing faster convergence and enhanced performance. Comparative ex- periments of PiSSA and LoRA across 11 different models, ranging from 184M to 70B, encompassing 5 NLG and 8 NLU tasks, reveal that PiSSA consistently out- performs LoRA under identical experimental setups. On the GSMSK benchmark, Gemma-7B BB fine-tuned with PiSSA achieves accuracy of 77.7%, surpassing LoRA's 74.53% by 3.25%. Due to the same architecture, PiSSA also compat- ible with quantization to further reduce the memory requirement of fine-tuning Compared to QLoRA, QPiSSA (PiSSA with 4-bit quantization) exhibits smaller quantization errors in the initial stages. Fine-tuning I.LMMA3-70BB on GSM8K, QPiSSA attains an accuracy of 86.05% exceeding the performance of QLoRA at 81. 31.73%. Leveraging fast SVD technique, PiSSA can be initialized in only few seconds, presenting a negligible cost for transitioning from LoRA to PiSSA.
1 Introduction
Fine-tuning large language models (LLMs) is a highly effective technique for boosting their capabili- ties in various tasks 2 4, ensuring models to follow instructions [5] 7) and instilling models with desirable behaviors while eliminating undesirable ones 8 9.. However, the fine-tuning process for very large models is accompanied by prohibitive costs. For example, regular 16-bit fine-tuning of LLaMA 65B parameter model requires over 780 GB of GPU memory (10), and the VRAM consumption for training GPT-3 175B reaches 1.2TB I.. Consequently, various parameter-efficient fine-tuning (PEFT) 12 13 methods have been proposed reduce the number of parameters and memory usage required for fine-tuning. Due to the ability to maintain the performance of full fine- tuning without adding additional inference latency, Low-Rank Adaptation (LoRA) [11] has emerged as popular PEFT method.
*Correspondence to: Muhan Zhang <muhan @ pww..du..c.>>>>>.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pretrained Matrix
Pretrained Matrix
B
Residual Matrix
W IR
W E R MKK
UErr: Mara
21/2
A-N
m
x
(a) Full Fine-tuning
(b) LoRA
(c) PiSSA
The comparison among Full Fine-tuning, training with LoRA, and PiSSA. In this visualiza- tion, blue modules represent parts of the model whose parameters are frozen during training, while orange modules indicate components that require updates. QLoRA quantizes the pretrained matrix in LoRA to 4-bit, whereas QPiSSA quantizes the residual matrix in PiSSA.
Figure 1:
Table I: Comparison of similarities and differences between PiSSA and LoRA. this table, bold highlights the model's primary component, while underline denotes the residual component.
LoRA
PiSSA
Forward
Y = X(W + AW) = X(W + AB)
Y X X(Wres + W
X (W + AB)
A Upar) $1/2 [ir,rr)
R
A~ N(0,0²) E
Initialization
B
g1/2 (rra))
Vitri E 1
B=0E R
T U Stread
Gradient
0A
81.
X
0L 0Y
B
+
0L DA
X T
(LL BY
B
T
Principal
01 88
AL
Random Direction
AL
=
BL
Principal
Fine-tunes noise while freezing W.
Fine-tunes principal parts freezing W
Comparison
Slow convergence and underperformance QLoRA cannot reduce quantization error.
Fast convergence and better performance. QPiSSA can reduce quantization error.
LoRA I1 hypothesizes that the modifications to parameter matrices during fine-tuning exhibit low-rank properties. As depicted in Figure Ib for a pre-trained weight matrix W ggxx LoRA substitutes the updates with a low-rank decomposition AW = AB, where A E Rmxr and B E grx", and the rank r & min(m, n). For Y XW, the modified forward pass is as follows:
Y = X(W AW) = X(W + AB),
(I)
random Gaussian initialization is used for A and zero for B, making AB = at the beginning of training, thereby the injection of adapters does not affect the model's output initially. LoRA avoids the need to compute gradients or maintain the optimizer states for the original matrix W, instead optimizing the injected, significantly smaller low-rank matrices A, B. Thus, it could reduce the number of trainable parameters by 10,000 and the GPU memory requirement by 3x [I]. LoRA is capable of achieving comparable performance full parameter fine-tuning. By integrating the quantization of pre-trained matrices W. LoRA also enables reducing the average memory requirements by 16x [10]. Meanwhile, the adapters can still utilize higher precision weights, thus, the quantization usually does not significantly degrade the performance of LoRA.
(LL) B and aL = DE Compared to full fine-tuning, using LoRA initially does not change the output Y for the same input X, so the magnitude and direction of gradient are primarily determined by the values of A and B. Since A and B initialized with Gaussian noise and zeros in LoRA. the gradients could be small
According to Equation the gradients of and are 34 XT
BT
(Y

---

and uninformative for long time, leading to slow convergence in the tine-tuning process. We also observe this phenomenon empirically, LoRA often wastes much time around the initial point.
Our Principal Singular values and Singular vectors Adapter (PiSSA) diverges from LoRA and its successors by focusing not approximating AW, but W. We apply singular value decomposition (SVD) to matrix W. Based on the magnitude of the singular values, we partition W into two parts: the principal low-rank matrix Wpri, comprising few largest singular values, and the residual matrix Wres, which possesses the remaining smaller singular values (with a larger quantity, representing a possible long-tail tail distribution). The principal matrix W can be represented by the product of A E IR"* and B E R where < min n.. As depicted in Figure Ic A and B are initialized based on the principal singular values and singular vectors and are trainable. Conversely, W is initialized with the product of the residual singular values and singular vectors and remains frozen during fine-tuning. Since the principal singular vectors represent the directions in which the matrix W has the most significant stretching impact, by directly tuning these principal components, PiSSA is able to fit the training data faster and better (as demonstrated in Figure 2a). Moreover, the loss and gradient norm curves of PiSSA often demonstrate similar trend to those of full parameter fine- tuning in our experiments (Figure 4 indicating that fine-tuning the principal components matches the behavior of fine-tuning the full matrix to some degree.
Target
OL:BA
Losz
Lotid r84 2-tar
Losa:
PiSSAnt
3.780
Countizttin Ereee Reb..d wo (42
20
Leti 5-tter
P.SSArt4
115SAr128 5-iter
(a) PiSSA converges more rapidly.
(b) PiSSA reduces more quantization error.
Figure 2: We illustrate the two key advantages of PiSSA: converging faster and better, and reducing quantization error. In the left figure, we use toy example to show P'SSA's faster convergence, where we first train two-layer MLP classifying odd numbers of MNIST, and then fine-tune the model on even numbers. PiSSA finds the right direction more quickly and achieves a lower loss with the same number of steps. In the right figure, PiSSA reduces quantization error more effectively than LoftQ [4], with an optional 5-iteration SVD for further error reduction, as detailed in Appendix
Because the principal components Wpri are preserved in the adapter at full precision, an additional benefit of PiSSA that when applying quantization to the frozen part Wree we can significantly reduce the quantization error compared to QLoRA (which quantizes the entire W), as illustrated in Figure 26 Therefore, PiSSA is even more compatible with quantization than LoRA, making it a superior plug-and-play substitution for LoRA.
Our paper makes several significant contributions:
We analyze the initial gradient magnitude and direction in LoRA, demonstrating that A initially has a zero gradient and B has a random gradient, which slows down convergence and may lead convergence suboptimal local minima.
We propose PiSSA initialization, a novel method that approximates the optimization di- rection of full- full-parameter fine-tuning by adapting model's principal components. To our knowledge, PiSSA A is the first to apply SVD to the original model, using principal singu- lar values and vectors to initialize the adapter for fine-tuning, while keeping the residual components frozen. Experiments show that PiSSA converges faster and outperforms LoRA.
We combine PiSSA with NF4 quantization to propose QPiSSA, which reduces quantization error by about 20% compared QLoRA, while maintaining the fast convergence and high performance of PiSSA.
3

---

2
Related Works
The vast complexity and computational needs of large language models (LLMs) with billions of parameters present significant hurdles in adapting them for specific downstream tasks. Parameter Efficient Fine-Tuning (PEFT) [3] emerges as a compelling solution by minimizing the fine-tuning parameters and memory requirements while achieving comparable performance to full fine-tuning.
PEFT encompasses strategies like partial inne-tunig [15 16 17 [8
19
20
21
22 soft prompt
fine-tuming [23 24 25 26 27 [8 (2),, non non-linear adapter tinettunig [30 31
32
33
34
35 36 37
38
and low rank adapter based fine-tuning 440
42].
LoRA injects trainable adapters to the linear layers. After fine-tuning, these adaptations can be re-parameterized into the standard model structure, thus gaining widespread adoption due to their ability to maintain the model's ' original architecture while enabling efficient fine-tuning. Following LoRA, A [42] dynamically learns the rank size needed for LoRA in each layer of the model. DeltaLoRA 43 updates the original weights of the model using parameters from adapter layers, incorporates LoRA to prevent pruning introduces magnitude component to learn the scale of AW while utilizing the original AB as a direction component of AW. Unlike LoRA and its successors, which focus on learning low-rank approximations of weight updates, PiSSA directly tunes the essential low-rank parts of the model while keeping the noisier, high-rank, and nonessential parts frozen. Although our approach differs in philosophy from LoRA, it shares most of LoRA's structural benefits and can be extended by these methods to enhance its performance.
enhancing LoRA's representational capacity. LoSparse 44
from eliminating too many expressive neurons. DoRA 45
QLoRA 10 integrates LoRA with 4-bit NormalFloat (NF4) quantization, along with Double Quanti- zation and Paged Optimizers, enabling the fine-tuning GPU while preserving the performance of full 16-bit fine-tuning tasks. QA-LoRA [46] introduces group-wise operators to increase the degree of freedom in low-bit quantization. LoftQ [14] reduces quantization error by decomposing the quantization error matrix of QLoRA and retaining the principal components with an adapter. PiSSA can also be combined with quantization techniques, and we have found that PiSSA significantly reduces quantization error compared to QLoRA and LoftQ.
65B parameter model on single 48GB
3 PiSSA: Principal Singular Values and Singular Vectors Adaptation
This section formally presents Principal Singular values and Singular vectors Adaptation method. PiSSA computes the singular value decomposition (SVD) of matrices W within the self-attention and multilayer perceptron (MLP) layers. The (economy size) SVD of matrix W E Rm is given by W USVT, where U E [mm.min(mm,.) V R x=*mmin(mm. are the singular vectors with orthonormal columns, and VT is the transpose of V. S = diag(s) E xmii(m..)) where the operation diag(s) transforms s to a diagonal matrix S, and s E I2 min(m,))) represents the singular values arranged in descending order. When the top r singular values 5t:] are significantly larger than the remaining singular values S|r:), we denote the intrinsic rank of W as r. Consequently S, along with U and V. can be divided into two groups: the principal singular values and vectors- (Up.er). Strrar), Vi...]] }. and the residual singular values and vectors vectors-(U (UI.r..) S(r..:.. Vi.r. }. where the matrix slicing notations are the same as those in Torch and [: denotes the first r dimensions. The principal singular values and vectors utilized to initialize the injected adapter consisting of A E maxr and B E grxn;
A = Ut.or)
1/2 [[ree]] E Rmxx,
(2)
1// B [m)))] Vi.rr] E R"""".
The residual singular values and vectors are used to build the residual matrix which is frozen during fine-tuning:
Whe (..r)) Spren Virni m
As indicated by Equation 5 the integration of AB with the residual matrix also preserves the full capability of the pre-trained model in the beginning of fine-tuning:
= XW X(Wres + WPri X (W + +AB).
(5)

---

B and OL 8B A (0r) Since elements of (t,)) > elements of S|r:]- the trainable adapter W = AB contains the most essential directions of W. In the ideal case, training AB mirrors the process of fine-tuning the entire model despite using fewer parameters. The ability directly fine-tune the most essential part of a model enables PiSSA to converge faster and better. In contrast, LoRA initializes the adapters A and B with Gaussian noise and zeros while keeping W frozen. Consequently, the gradients are small or in random directions during the early stages of fine-tuning, possibly introducing much waste of gradient descent steps. Moreover, an inferior initialization might lead suboptimal local minimum, resulting in worse generalization performance.
Similar to LoRA, the gradients of A and B are also given by AL = X 00 2r
8A
Since PiSSA shares the identical architecture with LORA, it inherits most of LoRA's benefits. These include but are not limited to the capability of fine-tuning a model with a reduced number of trainable parameters, quantizing the residual model to decrease memory consumption during forward propagation in training, and easy deployment. The adapter's straightforward linear structure facilitates the integration of trainable matrices with the pre-trained weights upon deployment, thereby maintaining the original inference speed of a fully fine-tuned model. Employing the Fast SVD technique [47 allowed PiSSA finish initialization in several seconds (Appendix B which is negligible cost.
For storage efficiency, we can choose not to store the dense parameter matrix AW, but to store the low-rank matrices, A A and A instead. As shown in Appendix C leveraging solely the AA and A facilitates their seamless integration with the original pre-trained models. Finally, one pre-trained model can accommodate multiple AA, AB, fine-tuned by diverse PiSSA or LoRA procedures, which enables fast adaptation of the pre-trate trainee model to different downstream applications.
4 QPiSSA: An Extension Method with Lower Quantization Error
Quantization divides the value range of a matrix into several continuous regions, and maps all values falling inside region into the same "quantized" value. It is an effective technique reduce the memory consumption of forward propagation. At the same time, LoRA greatly reduces the backward memory requirement, making it highly suitable to use LoRA and quantization together, where the base model is quantized for memory-effcient forward propagation, and the LoRA adaptors are kept in full precision for accurate backward parameter updates. One representative previous work, QLoRA, quantizes the base model to Normal Float 4-bit (NF4) and initializes the full-precision A and B with Gaussian-Zeru initialization. Therefore, the overall error is given by:
Quantization Error of QLoRA |IW (mf44(W)= + AB) ||+ 1|W - nz AWW)
(6)
where |Mi| denotes the nuclear norm (also known the trace norm 48 defined as:
minffm. m,n)
|M = trace (VMM)) = Σ (M(M)
where (MM is the th singular value of M. As we can see, the quantization error of QLoRA the same as that of directly quantizing the base model. Our QPiSSA, however, does not quantize the base model but the residual model. Therefore, its error is given by:
Quantization Error of QPiSSA IIW (nf4(W + AB) ||- | nnf4(Wrex)ll I|.. (8)
Since the residual model has removed the largesingglla-valle components, Weres has narrower distribution than that of W, as can be seen in Figures [a and 3b (comparing the singular value distributions of W and WTess), as well Figures 3c and 3f (comparing the value distributions of W and WTe*), which is beneficial for reducing the quantization error. Moreover, given that NF4 is optimized for data with a normal distribution, discussed by Dettmers et al. [TO] we fit the values of W and Wres to a Gaussian distribution respectively. As illustrated in Figures 3e and 3r, Wres aligns more closely with a Gaussian distribution and exhibits smaller standard deviation, making it more suitable for applying NF4 than W. Both the above lead QPiSSA to achieve a significantly lower quantization error than QLoRA, shown in Figures 3d adland
Besides the advantage reducing quantization error, QPiSSA's gradient direction similar to that of PiSSA, resulting in significantly better fine-tuning performance compared to QLoRA.
5

---

smppar magttus
20
sigplar value magituu
R0
1:25
protabili dendily
75-
mu -.002e-06, std 0.0135)
1.5
10-
15
1.00
0.75
50
10-
5b
200
5b
100
125 In ne pat Singu values
1000
2000
4000
1000
2000
3000
4000
0.02
0.04
Original matrix W
(b) Residual matrix Wree
(c) The distribution of W
9..90-007
0.4.
0.055
0.050
Probabiitty danstor
0.3
sing Jiggllar www.
0.045
50
160
82
50
100
Singuat Values
0.0
1000
2000
3050
4000
1000
2000
3020
4000
-0.0
00
0.00
0.02
0.04
singaler vauu index
singular index
Data values
(d) Error matrix of QLoRA
(c) Error matrix of QPiSSA
(f) The distribution of Wres
Figure 3: Visualizations of LLaMA 2-7B's "layers[O].self attn.q aattn._ppro"""""""___" matrix, with distributions for the full model shown in Appendix Figures (a), (b), (d), and (e) display the singular values of W, Wres, W n. (4(4)W and Wres - -n -rff((Wees), respectively, Figures (c) and (f) show the data
distributions of W and Wres.
5 Experiments
The experiments were conducted on the NVIDIA A800-SXM4(80G) GPU. In our experiments, adopt the Alpaca [49] implementation strategy, using the Adam optimizer with a batch size of 128, learning rate of 2c-5, cosine annealing schedules, and warmup ratio of 0.03, without any weight decay. As discussed in Section B.3 of QLoRA [10], we compute the loss using only the responses from the instruction-followinn datasets. We ensure lora_aly alpha is always equal to lora r, set lora_des dropout to 0, and incorporate the adapters into all linear layers of the base model. We utilize the Float32 computation type for both the base model and the adapter in LoRA and PiSSA. For QLoRA, LoftQ, and QPiSSA, we use 4-bit NormalFloat [10] for the base model and Float32 for the adapter. B6 (50] used for full parameter fine-tuning save the resources (see Appendix D
5.1 Evaluating the Performance of PiSSA on both NL.G and NLU Tasks
We begin by comparing PiSSA, LoRA, and full-parameter fine-tuning natural language gener- ation (NLG) tasks. We fine-tuned LLaMA 2-7B [51), Mistral-7B-vO.1 (52). and Gemma-7B (53]
the GSM8K (54) and MATH [2] validation sets. Additionally, the models were fine-tuned on the Code- Feedback dataset (55] and evaluated for coding proficiency using the HumanEval (56) and MBPP (57) datasets. Furthermore, the models were trained on the WizardL Enso---rrruu dataset 7 and tested for conversational abilities the MT-Bench dataset [6]. All experiments were conducted using subsets containing 100K data points and were trained for only one epoch to reduce training overhead.
on the MetaMathQA dataset 2 to assess their mathematical problem-solving capabilities
As shown in Table 2 across all models and tasks, fine-tuning with PiSSA consistently surpasses the performance of fine- moe-tuning with LoRA. Further experiments demonstrated that this improvement is robust across various amounts of training data and epochs (Section 5.2 including both 4-bit and full precision (Section 5.3). different model sizes and types (Section 5.4 and varying proportions of trainable parameters (Section 5.5)
We also evaluate PiSSA's natural language understanding (NLU) capability on the GLUE bench- mark [59 with DeBERTa-v3-bass [60]. Table prresenssssetts the results across 8 tasks. PiSSA outperforms LoRA out of 8 NLU tasks, achieving an overall average improvement of 1.21 %. Upon reviewing the training loss on the exceptional dataset, MNLI, we observed that PiSSA's average loss of 0.17 was lower than LoRA's 0.24 in the final epoch. This indicates that the fitting ability of PiSSA remains stronger than that of LoRA.

---

Table Comparison of PiSSA and LoRA on NLG tasks, with results averaged over three runs and reported with standard deviations.
Model
Strategy
GSMSK
MATH
HvaanEval
MBPP
MT-Bench
Full (oRA(gaussian)
49.13:0.21
7.29#0.22
21.20+0.30
35.59:0.25
4.91
LLaMA 2-7B
42.85+0.12 12
5.50+0.33
18.35:0.31
35.5010.14
4.59:0.07
LoRA(kaiming)
43.2340.66 64
5.90=0.16 16
18,21
35.4740.37
4.56+0,04
PISSA
53.22+0.55
7.47=0.34
21.92+0.38
37.24+0.63
4.88+0.03
Full FT LoRA(gaussian)
69.9140.25
18.6420.35
45.31
51.4640.15
4.95=0.05
Mistral-7B
69.5020.42
20.0840.20
43.7840.34
58.4640.37
4.9020.05
LoRA(kaiming)
69.40+0.25
19.99=0.44
43.7440.14
58.3940.02
4, 4,9340.05
PiSSA
73.31=0.23
23.12+0.52
46.88+0.25
62.55+0.58
5.34+0.04
Full
72.09+0.32
22.71=0.34
47.0240.27
55.67+0.50
5.40+0.12
Gemma-7B
LoRA(gaussian)
75.11+0.64
30.4120.48
53.7040.25
65.5840.29
4. 4.98+0.02
LoRA(kaiming) PISSA 77.78+0.32
74.53+0.47
29.90=0.16
53.57+0.27
65,25+0.29
4.97+0.09
31.3320.33
54.31+0.28
66.1720.43
5.6420.10
Table 3: Comparison of PiSSA and LoRA on NLU tasks. LoRA and LoRA K denote LoRA with
Gaussian and Kaiming initialization for B, respectively. Results for full fine-tuning, BitFit [5],
HAdapter 30, PAdapter (6), LoRA G [1]] and AdaLoRA are from AdaLoRA [58]. averaged over
five runs. Remaining methods are averaged over three runs, with details in Appendix L
Method
Params
MNLI
SST2
MRPC
CoLA
QNLI
QQP
RTE
STSB
ALL
Full FT
184M
89.90
95.63
89.46
69.19
94.03
92.40
83.75
91.60
88.25
BitFit
0.IM
89.37
94.84
87.75
66.96
92.24
88.41
78.70
91.35
86.20
HAAdapter
1.22M
90.13
95.53
89.95
68.64
94.11
91.91
84.48
91.48
88.28
PAdapter
1.18M
90.33
95.61
89.46
68.77
94.29
92.04
85.20
91.54
88.41
LoRA
1.33M
90.65
94.95
89.95
69.82
93.87
91.99
85.20
91.60
88,50
LoRA K
1.33M
89.96
95.64
90.28
70.69
93.84
92.03
84.84
91.68
88.62
DoRA
1.27M
90.29
95.79
90.93
70.85
94.10
92.07
86.04
91.79
88.98
AdaLoRA
1.27M
90.76
96.10
90.69
71.45
94.55
92.23
88.09
91.84
89.46
PISSA
1.33M
90.37
96.22
91.50
73.12
94.43
92.33
88.69
92.00
89.83
5.2 Experiments using Full Data and More Epochs
In this section, finetune LLaMA 2-7B model the complete MetaMathQAA399K dataset for 3 epochs to ensure thorough saturation. The training loss and gradient norms is visualized to demonstrate quicker convergence and evaluated the GSM8K dataset every 1000 steps to demonstrate superior performance of PiSSA compared to LoRA. The results are depicted Figure Additionally, similar comparisons Mistral-7B and Gemma-7B are detailed in Appendix [
3.5
MON
LIAA
PRCC
LofA
RT
3.0-
3.25
3.20
200 staze
Tre 120 Stape
1.0
50.0
335
415
110
28: ak 2e
65.0
1000
2000
3300
4020
$300
6000
3000
8000
$000
Loss over steps.
Grad norm over steps.
Accuracy on GSM8K over training steps.
Figure 4: The loss, grad norm, and evaluation accuracy over the training steps of LoRA (indicated in blue), PiSSA (in orange), and full parameter fine-tuning (in red).
According to Figure 4a the loss of PiSSA reduces rapidly during the first 100 steps, and the grad norm (shown in Figure 4b PiSSA is significantly higher than that of LoRA, with a trend similar to full fine-tuning. Throughout the process, the loss of PiSSA remains lower than that of LoRA, indicating

---

that PiSSA converges to better local optimum. As shown in Figure 4c PISSA consistently achieves higher accuracy compared LoRA, and in cases also surpasses full parameters fine-tuning. We hypothesize that this is because PiSSA is a denoised version of full fine-tuning. Comparing the grad norm and loss curves of PiSSA and full fine-tuning, we can see that the larger grad norm of full fine-tuning does not bring lower loss, indicating that portion of the grad norm is spent on noisy directions not beneficial for loss reduction. This phenomenon consistent with Figure 2a
5.3 Conducting 4-bit Quantization Experiments
In this section, first compare the initial quantization error reduction ratio of PISSA, QLoRA, and
LoftQ. This ratio is defined as (1 [W-(n/4(W (oop4400"+BB)LLL >+AA)lll)) 100%, measuring the relative error
[W -n/ I(W)
decrease achieved by each mehod compared to directly quantizing the base model. The partial results
are presented in Table 4 and the complete results can be found in Table X Appendix
Table 4: The quantization error reduction ratio of QLoRA, LoftQ, and PiSSA across different layers.
Method
Rank
Gate
Up
Down
AVG
QLoRA
0
0
0
LLaMA 2-7B
loftQ PiSSA
128
16.5
15.9
12.4
12.4
12.3
128
27.9
27.2
18.7
18.6
15.8
13.6
13.6
19.4
QLoRA
0
0
LLaMA 3-8B
loftQ
128
16.4
29.8
28.8
16.1
11.9
11.7
11.7
18.1
PISSA
128
26.3
41.7
32.3
20.1
14.4
12.5
12.9
22.9
QLoRA
0
0
0
0
LoftQ
64
6.1
17.8
17.0
6.0
4.3
4.4
4.2
8.5
LLaMA 3-70B
PISSA
64
15.7
34.2
18.9
7.5
6.7
5.7
4.7
13.4
PISSA
128
23.2
49.0
10.1
8.8
8.2
In Table PiSSA reduces the quantization error by about 20% compared to directly quantizing
the base model. The reduction is more significant for lower-rank matrices. For instance, in the LLaMA-3-708B [6]. all "Key" projection layers validate that QLoRA, discussed in Section I does not reduce quantization error. In contrast, PiSSA significantly outperforms LoftQ in reducing quantization error, further discussed in Appendix H
reduction of 49%. The results in Table 4
0.39
0.6
a0
God Nort
QLoRA Lont gesSA
LoSA
FT
Training Less
3.30
PSSA
Training Grad
20
0.20
20
The Teai 200 Staps
40
80
160
1.
20
The Fins 100 Siaps
40
60
80
100
0.
1006
2000
1000
4200
5000
600n
7000
1000
5050
1000
2000
3000
4000
5200
6000
70RD
Boon
9000
(a) Loss over training steps.
(b) Grad norm over training steps.
82
QLaRA
LoRA
GAMAK Accurary (96)
LoftQ
QPISSA
PISSA
74
1000
2000
3000
4000
5000
6000
7000
8000
9000
(c) Accuracy on GSM8K over training steps.
Figure The loss, grad norm, and evaluation accuracy over the training steps of (Q)LoRA, (Q)PiSSA. LoftQ and full parameter fine-tuning.

---

The difference between QPiSSA and PiSSA is the quantization of the residual model to 4 bits. As introduced in Section the residual model has less influence on the optimal direction compared with the principal adapter, which is the same in both QPiSSA and PISSA. Therefore, besides reducing the quantization error, we expect QPiSSA to also converge faster than QLoRA and LoftQ. We train LLaMA 3-8B using LoRA/QLORA, PiSSA/QPISSA, LoftQ, and full fine-tuning on MetaMathQA- 395K for epochs, recording the loss, gradient norm, and accuracy on GSM8K, as shown in Figure 5
According to Figure 5 QPiSSA's loss reduction speed in the first 100 steps is even faster than PiSSA and full ane-tuning. Although LoftQ can reduce the quantization error, its loss convergence speed is not faster than LoRA and QLoRA, indicating that QPiSSA's ability to reduce the quantization error and its fast convergence might also be orthogonal capabilities. After sufficient training, QPiSSA's loss is also much lower than that of LoRA/QLoRA and LoftQ. The grad norm is significantly larger than those of LoR LoRA/QLoRA and LoftQ. In terms of fine-tuning performance, QPiSSA' A's accuracy is higher than that of A and LoftQ and even better than that of full-precisiog LoRA.
5.4 Experiments Across Various Sizes and Types of Models
In this section, we compare (Q)PiSSA and (Q)LoRA across 9 models, ranging from 7-70B param- eters, including L.L.MA 2-7/13B (5). LLLMA-3-8/70B [T], Mistral-7B [52], Gemma-7h (53
and Qwen1.5-7P5 62), 15-1.B 63 and MoE models: DeepSeek-MoF-116 16B [44] and Mixtral-
8x7B [65]- These models were fine-tuned on the MetaMathQA-100K and CodeFeedback- 100K dataset and evaluated on the GSM8K and HumanEval. DeepSeek-MoE-166 166. Mixtral-8x7BB, Yi-1.5- 34B, and LLaMA-3-700 A-BB were fine-tuned with QPiSSA and QLoRA, while the other models were using PiSSA and LoRA. From Figure 6 (Q)iSSA, compared to LQ),,, shows improved accuracy across various sizes and types of models, demonstrating its consistent advantage over LQ)..
GSM8K Accuracy (%)
0P:S5A
FSSSA
30-
Humee Dve Acaussy
co.
Di 1 -5
8
Doepdeek tod
L LaNkk..
d'aat 2:10
dwar 5.78
Micral 20
Letet ot
Gammuk 77
LLaNt 5-700
uaMA 1g
(LaMd 2.230
Demmat clatt-kdd
Oberr 0-771 5-77
Figure 6: Comparison of (Q)iSSA and (Q)LOAA across models from 7B to 70B
5.5 Experiments on Various Ranks
This section explores the impact of incrementally increasing the rank of PISSA/QPISSA and LORA/QLORA from I to 128, aiming to determine whether PISSA/QPISSA consistently outperforms LoRA/QLORA under different ranks. The training conducted using the MetaMathQA-
dataset for 1 epoch, while the validation is performed on the GSM8K and MATH datasets. The outcomes of
these experiments are depicted in Figure 7 with additional results presented in Appendix K
Figure 7 illustrates the quantization error reduction ratio across various ranks. In this figure, QLoRA shows no reduction in quantization error, while QPiSSA consistently outperforms LoftQ in reducing quantization error across all ranks, with a particularly notable advantage at lower ranks. In Figure 7b the final loss on the training set is shown for models trained with ranks ranging from 128. The results indicate that PiSSA and QPiSSA achieve better fit to the training data compared to LoRA, QLoRA, and LoftQ. In Figures and Figures 7d, we compare the accuracy of the fine-tuned models on the GSM8K and MATH validation sets under various ranks, finding that PiSSA consistently outperforms LoRA with the same amount of trainable parameters. Furthermore, as the rank increases, PiSSA will reach and surpass the performance of full- full-parameter fine-tuning.
9

---

Eroer tteelattion He
20
Lono OPSSA
MutuMathO Luxs
0.5
Pull FT
Left
25-
QPSSA
8
16
32
64
128
16
32
64
120
(a) Quantization error reduction ratio across ranks.
(b) Training loss under various ranks.
c5waa Aucuracy 190
50-
FT LoRA
LoRD QLoRA
MA Recumm Sat
Full FT
LonG QLaRA
LoRA
6-
40
30-
4-
20
16
32
16
(c) Accuracy GSM8K under various ranks.
(d) Accuracy on MATH under various ranks.
Figure 7: The comparison among (QLoRA, (Q)PiSSA, LoftQ, and full fine-tuning across ranks.
Conclusion
This paper presents PEFT technique that applies singular value decomposition (SVD) to the weight matrix of pre-trained models. The principal components obtained from the SVD are used to initialize low-rank adapter named PiSSA, while the residual components kept frozen, to achieve effective fine-tuning and parameter efficiency simultaneously. Through extensive experiments, we found that PiSSA and its 4-bit quantization version QPiSSA significantly outperform LoRA and QLoRA in both NLG and NLU tasks, across different training steps, various model sizes and types, and under various amount of trainable parameters. PiSSA provides novel direction for research PEFT by identifying and fine-tuning the principal components within the model, analogous to slicing and re-baking the richest slice of pizza. As PiSSA shares the architecture LoRA, it can be seamlessly used in existing LoRA pipelines as an efficient alternative initialization method.
7 Limitation
There are still some questions with PiSSA not addressed in this paper: 1) Besides language models, can PiSSA also be adapted to convolutional layers and enhance the performance of vision tasks? 2) Can PiSSA also benefit from some improvements LoRA, such as AdaLoRA (58) and DyLoRA [66] which adaptively adjust the rank? 3) Can provide more theoretical explanations for the advantages
of P over LoRA? We are actively exploring these questions. Nevertheless, we are excited to see the huge potential of PiSSA already demonstrated in existing experiments and look forward to more tests and suggestions from the community.
Acknowledgements:
This work is supported by the National Key R&D Program of China (2022ZD016030)
10

---

References
[1] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.0900 09583, 2023.
[2] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[3] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306. 08568, 2023.
[4] Raymond Li. Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.006
2023.
[5] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 11227730-27744, 2022.
[6] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-asaa-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[7] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2023.
[8] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 00862, 2022.
[9] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[10] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
[11] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 09685, 2021.
[12] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312. /2 /88, 2023.
[13] Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. /4608, 2024.
[14] Yixiao Li, Yifan Yu. Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fie-tuning-avee quantization for large language models. arXiv preprint arXiv:23 10.0866, 2023.
[15] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer- masked language-models, arXiv preprint arXiv:2 10199, 2021.
11

---

[16] Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver Steeg. Neural architecture search for parameter-effiiien fine-tuning of large pre-trained language models. arXiv preprini arXiv:2305. /6597, 2023.
[17] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schüite. Masking as an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004. 12406, 2020.
[18] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24205, 2021.
[19] Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:2110.0756)) 2021.
[20] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint arXív: 109.05687 2021.
[21] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff' pruning. arXiv preprint arXiv:2012.07463, 2020.
[22] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12799-12807, 2023.
[23] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00/21, 2021.
[24] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint
2021.
[25] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
[26] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
[27] Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.07904, 2021.
[28] Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. Attempt: Parameter-efficient multi-task task tuning via attentional mixtures of soft prompts. arXiv preprint wxXi:2205..1900 2022.
[29] Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multitask prompt tuning enables parameter-efficient transfer learning. arXiv preprint arXiv:2303.02861, 2023.
[30] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[31] Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via efficien transfer learning. arXiv preprint arXiv:2004.03889, 2020.
[32] Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, et al. Conditional adapters: Parameter-efficceet transfer learning with fast inference. Advances in Neural Information Processing Systems, 36, 2024.
[33] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. To- wards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2 10.04366, 2021.
12

---

[34] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010. / 1918, 2020.
[35] Hongyu Zhao, Hao Tan, and Hongyuan Mei. Tiny-attention adapter: Contexts are more important than the number of parameters. arXiv preprint wrXiv:2211.01977 2022.
[36] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint (rXiv:2005.00247, 2020.
[37] Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Mera: Merging pretrained adapters for few-shot learning. arXiv preprint arXiv:2308. 0ati:22088.9982,, 2023.
(38) Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint
2021.
[39] Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.00027. 2023.
[40] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Mcasuring the intrinsic dimension of objective landscapes. arXiv preprint /804.088000083 2018.
[41] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint 2rXiv:2012.132555 2020.
[42] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fine-tuning. In The Eleventh International Conference on Learning Representations, 2022.
[43] Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411, 2023.
[44] Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pages 20336-20350. PMLR, 2023.
[45] Shih-Yan Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molehanov, Yu-Chiang Frank Wang, K Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09) 03355, 2024.
[46] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-awae aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14711 2023.
[47] Nathan Halko, Per- Gunnar Martinsson, and Joel Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 23(2):217-288, 2011.
[48] Ky Fan. Maximum properties and inequalities for the eigenvalues of completely continuous operators. Proceedings of the National Academy of Sciences, 37(11) 1)7666776, 1951.
[49] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An fnslruction-folloiinng llama model. https://gt thub. com/ cow/tatsu- tatstaatssul// hatinllaassttttni ord, alpaca 2023.
[50] Shibo Wang and Pankaj Kanwar. Bfloat16: The secret to high performance on cloud tpus. Google Cloud Blog, 4, 2019.
13

---

[51] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288: 2023.
[52] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[53] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv22403.08225, 2024.
[54] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: 10.22110..4. 14.168 2021.
[55] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeeinteppreter: Integrating code generation with execution and refinement. arXiv preprini a402v: /4658, 2024.
[56] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov. Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Perbert-Vosss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
[57] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
[58] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficiee fine-tuning. arXiv preprint ar3iv:2303. 105 2023.
[59] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
[60] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre- praitngi with gradient-disentangle embedding sharing, 2021.
[61] AltMeta. Llama 3 model card. 2024.
[62]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint 2309 /6609, 2023.
[63] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.0 046522 2024.
14

---

[64] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.0mm 160066, 2024.
[65] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arX1v:2401.04000
2024.
[66] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558, 2022.
[67] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
[68] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
15

---

The Supplementary Material for The Paper "PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models."
In Section A we combined PISSA with two improved LoRA methods, and the experimental results show that these improvements can further enhance the effectiveness of PiSSA.
In Section B use fast singular value decomposition to initialize PiSSA. The results indicate that the performance of fast singular value decomposition approaches that of SVD decomposition in just several seconds. This ensures that the cost of converting from LoRA/QLORA to PiSSA/QPISSA is negligible.
we demonstrate that the trained PiSSA adapter can be losslessly converted to LoRA, allowing for integration with the original model, facilitating sharing, and enabling the use of multiple PiSSA adapters.
In Section C
In Section D
we explore the experimental effects of using different precisions.
In Section B
discuss the effects of QPiSSA during multiple rounds of SVD decom- position, which can significantly reduce quantization errors without increasing training or inference costs.
In Section
we compare the use of high, medium, and low singular values and vectors to initialize adapters. The experimental results show that initializing adapters with principal singular values and vectors yields the best fine-tuning performance.
In Section G we used a normal distribution function to fit all linear layers of multiple models and calculated their mu and sigma. The experimental results show that after using PiSSA for initialization, the distribution of the remaining models, as described in Section 3 of the main text, is indeed narrower than that of the original models.
In Section H we provide comprehensive comparison of quantization emror among QLoRA, LoftQ, and QPISSA theoretically explaining why QPiSSA reduce quantization errors.
Section | we combine QPiSSA with various quantization methods beyond Normal Float 4bit, including INT8 and GPTQ. QPiSSA effectively reduces quantization error in these formats, enhancing fine-tuning performance.
In Section we trained Mistral-7B and Gemma- 7B for a sufficient number of steps. The results indicate that PiSSA and LoRA are less prone to overfitting compared to full parameter fine-tuning.
detailed comparison of PiSSA and LoRA different ranks. It is evident that PiSSA consistently outperforms LoRA in terms of loss convergence, quantization error reduction, and final performance across different ranks.
In Section K we offer
In Section describe the detail setting for NLU task.
16

---

A Enhancing PiSSA with LoRA Improvement Methods
AdaLoRA introduces three improvements over LoRA:
Trainable parameters in AdaLoRA are changed to A, B, and E. A and B are Gaussian- initialized, and E is zero-initialized r-dimensional vector, making Adiag E) B AW, similar to singular value decomposition.
A regularization loss |AA" - - 1B8 B - 1| is used to make A and B gradually orthogonal during training, resembling the SVD of AW.
An initial large rank is set, and less important E values gradually masked during training, resulting in different final ranks for each layer, achieving better performance with the same number of parameters.
Despite the extensive use of SVD terms, AdaLoRA does not perform actual SVD on any matrix. In the PEFT domain, terms like low-rank decomposition, and singular value decomposition often appear. They generally refer to products of low-dimensional matrices approximating an ideal AW without actual matrix decomposition. To our knowledge, PiSSA is the first to perform SVD on the original model, fine-tuning the principal component while keeping the residual model frozen.
PiSSA and AdaLoR represent different improvements LoRA, making them combinable. There- fore, we additionally improved PiSSA based on LoRA's three innovations:
After extracting the principal singular values and vectors of W, S as independent trainable vector instead of multiplying it into U and V.
Since PiSSA's U and V are orthogonal at the beginning, maintaining their orthogonality through orthogonal regularization is very easy.
Although AdaLoRA oRA claims to dynamically reduce the number of trainable parameters, the initially large number of parameters is not truly pruned, resulting in more parameters being updated during actual training. Therefore, we did not use this improvement.
DoRA adds a learnable magnitude module to LoRA, normalizing W + AAB at each update step and multiplying its by the magnitude module. This allows A, B to learn the direction and the magnitude module to learn the magnitude AW. While this approach can improve fine-tuning performance, normalizing W + AB at each step results in slower fine-tuning speeds. In contrast, PiSSA only changes LoRA's initialization method, matching LoRA in training speed and converging faster, thereby reducing training costs.
Table GSM8K accuracy for LoRA and PiSSA when combined with LoRA improvement methods.
Model
Method
LoRA+
PiSSA+
Vanilla
71.01+0.199
76.75=0.036
LLaMA-3-8B
DoRA
72.38=0.188
77.51=0.257
AdaLoRA
72.31+0.202
78.5940.199
PiSSA, with its intrinsic principal singular values and orthogonal singular vectors, is very suitable for combination with AdaLoRA. According Table 5 The performance of the improved PiSSA surpasses all the other methods including PiSSA. From lines 1, and 2 of the table. it is evident that the performance of PiSSA combined with DoRA significantly surpasses that of DoRA alone and also exceeds the performance of PiSSA alone. Taking into account the combination experiments of PISSA with AdaLoRA, it can be inferred that PiSSA benefits from the enhancement techniques of LoRA, demonstrating the potential of PiSSA when integrated with other methods.
17

---

B Fast Singular Value Decomposition
In order speed up the decomposition of the pre-trained matrix W, we adopted the algorithm proposed by Halko et.al [47] (denoted as Fast SVD), which introduces randomness to achieve an approximate matrix decomposition. We compare the initialization time, error, and training loss between SVD and Fast SVD, with the results shown in Table ( Initialization time refers to the computation time taken to decompose the pre-trained parameter matrix W, measured seconds. Initialization error indicates the magnitude of the discrepancy introduced by Fast SVD compared to SVD after decomposing the matrix. Specifically, the error is the sum the absolute differences between the matrices decomposed by original SVD and Fast SVD. For the error, report the results of the self-attention module the table. Loss refers to the loss value at the end training. In Fast SVD, the parameter niter refers to the number of subspace iterations to conduct. A larger niter leads to increased decomposition time but results in smaller decomposition error. The symbol oo represents the experimental results with the SVD method.
Table Comparation between SVD and Fast SVD in terms initialization time, error and training loss.
Rank
Metric
Niter
1
2
4
8
32
64
128
5.05
8.75
5.07
8.42
5.55
8.47
6.80
11.89
2
4.38
4.71
4.79
4.84
5,06
5.79
7.70
16.75
Initialize
4
5.16
4.73
5.09
5.16
5.60
7.01
7.90
11.41
Time
4.72
5.11
5.14
5.40
5.94
7.80
10.09
14.81
6.24
6.57
6.80
7.04
9.99
14.59
22.67
00
434.92
434.15
434.30
435.42
435.25
437.22
434.48
435.84
1,30E-3
1.33E-3
1.55E-3
1.9E-3
1,98E-3
1,97-3
2.00E-3
1.9E-3
2
5,84E-4
1.25E-3
1.45E-3
1.43B-3
1,48E-3
1.55E-3
1,48E-3
1.33E-3
Initialize
4
6.01E-4
8.75F-4
6.75E-4
1.10R-3
1,05E-3
1.03E-3
1.08F-3
9.75F-4
Error
1.26F-4
2.34E-4
5.25E-4
7.25E-4
5.75E-4
8.25E-4
8.25E-4
7.75E-4
7,93E-5
2.25E-4
1,28F-4
6.50E-4
4.25E-4
6.50E-4
6.00E-4
4.75E-4
00
0.3629
0,3420
0.3237
0.3044
0.2855
0.2657
0.2468
0.2301
2
0.3467
0.3337
0.3172
0.2984
0.2795
0.2610
0.2435
0.2282
Training Loss
4
0.3445
0.3294
0.3134
0.2958
0.2761
0.2581
0.2414
0.2271
0.3425
0.3279
0.3122
0.2950
0.2753
0.2571
0.2406
0.2267
0.3413
0.3275
0.3116
0.2946
0,2762
0.2565
0.2405
0.2266
00
0.3412
0.3269
0.3116
0.2945
0.2762
0.2564
0.2403
0.2264
It can be observed that the computation time of the SVD is tens of times that of Fast SVD. In addition, SVD exhibits consistently high time consumption with minimal variation as the rank increases, while Fast SVD, although experiencing slight increase in computation time with higher ranks, remains significantly lower compared to SVD throughout. As the rank increases, the initialization error initially rises gradually, with slight decrease observed when the rank reaches 1 128. And at the same rank, increasing the niter Fast SVD leads to gradual reduction error. For training loss, we observed that as the rank increases, the training loss decreases gradually. At the same rank, with the increase of niter, the training loss of models initialized based on Fast SVD approaches that of models initialized based on SVD.
18

---

C Equivalently Converting PiSSA into LoRA
The advantage of PiSSA lies in its ability to significantly enhance training outcomes during the fine- tuning phase. After training, it allows for the direct sharing of the trained matrices A and B. However, we directly save A, B, users need perform singular value decomposition on the original model to get Wres, which requires additional time. When employing fast singular value decomposition, there can be slight inaccuracies too. More importantly, such a way necessitates altering the parameters of the original model, which can be inconvenient when using multiple adapters, especially when some adapters might disabled or activated. Therefore, we recommend converting the trained PiSSA A module cquivalently into a LoRA module, thereby climinating the need to modify the original model's parameters during sharing and usage. the initialization phase, PiSSA decomposes the original matrix into principal components and residual matrix: W W + AAB. Upon complction of training, the model adjusts the weights as follows: + AW = Wres + A'B.. Thus, the modification of the model weights by PiSSA is given by:
AW = A'B'
AB
(9)
B'
AA' A
(10)
AA
AB
where AA E R and AB E 22 Therefore, we can store and share the new adaptor AA and AB instead of A', B', which allows directly inserting the adaptor the original matrix and avoids breaking W. Since is typically small, the twice storage overhead is still acceptable. This modification allows for plug-and-play usage without the need for singular value decomposition, saving time and avoiding computational errors associated with the SVD, without necessitating changes to the original model parameters.
D Comparison of Fine- Tone-TuningTin in BF16 and FP32 Precision
In this section, we compare the effects of training with BFloat16 and Float32 precision. The comparing include four models: LLaMA---7BB Mistral-7B, Gemma-7B, and LLaMA-3-8B, each fine-tuned with all parameters in both BFloat16 and Float32 precision the MetaMathQAA395K dataset. The validation results conducted on the GSMSK dataset are shown in Figure 7
Table 7: Comparison of fine-tuning results of LLaMA-2-7B, Mistral-7B, Gemma- 7B, and LLaMA-3- 8B in BF16 and FP32 precision on MetaMathQA-395k dataset for 3 epochs.
Training Loss
GSM8K ACC (%)
MATH ACC (%)
Model
BF16
FP32
BF16
FP32
BF16
FP32
LLaMA-2-7B
0.1532
0.1316
63.15
68.31
13.14
20.38
Mistral-7B
0.1145
0.1306
73.09
65.88
26.44
23.66
Gemma-7B
0.1331
0.1382
75.21
75.97
29.18
28.64
LLaMA-3-8B
0.1271
0.1317
81.96
75.44
33.16
28.72
From Table 7 it is evident that the choice of precision greatly affects the experimental results. For example, the LLaMA-2-78 model shows a 5.16% higher performance on the GSM8K dataset when using FP32 compared to BF16. Conversely, the Mistral-7B and I.LaMA-3-BB 7.21% and 6.52% lower with FP32 than with BF16 separately. The Gemma-7B model shows similar performance with both precisions. Unfortunately, the experiments did not prove which precision is better. To reduce training costs, we use BF16 precision when fine-tuning all parameters. For methods with lower training costs, such as LoRA, PiSSA, we use FP32 precision. For QLoRA, QPiSSA and LoftQ, the base model was used NF4 precision, while the adapter layers used FP32 precision.
GSM8K
19

---

E Reducing Quantization Error through Multiple Iteration of SVD
Table 8
provides supplementary explanation of the results in Table When number of iterations LoftQ uses an N-bit quantized weight Q RN and low-rank approximations A E R
T
and B R"x"" to minimize the following objective by alternating between quantization and singular value decomposition:
min IW (Q AB' )|E, Q.A.B
(11)
where || || denotes the Frobenius norm, A and B are set to zero. Inspired by LoftQ, our QPiSSA
T-iter alternately minimize the following objective:
min W A,
IW (n.4(Wrrea) + ABT)
(12)
where A and B are initialized by the principal singular values and singular vectors. The process is summarized in Algorithm
Algorithm QPiSSA-T-iees 2
input Pre-trained weight W, target rank r, 4-bit quantization function nf4(-), alternating step T
1: Initialize Ao, Bo + SVD(W) by (2) and (3)
2: Initialize residual weight W f- W - Bo
3: for t= to
4: Update Ar, B, < SVD(W - nnf(WWWrr) rex)) by and
5: Update residual weight Wrea t W A
6: end for
output mf4(Wres). AT, BT
Table 8: PiSSA reduces more quantization error on various ranks and number of iterations.
Method
Rank
niter
O
Gate
Up
Down
AVG
QLoRA
0
0
0
loftQ
128
7.2
7.3
5.1
5.1
LLaMA -2-7B
PiSSA
128
1
19.0
18.1
8.9
8.9
8.2
5.9
6.0
10.7
loftQ
128
5
16.5
16.5
15.9
16.0
12.4
12.4
12.3
14.6
PISSA
128
5
27.9
27.2
18.7
18.6
15.8
13.6
13.6
19.4
QLoRA
0
0
0
0
LoftQ
64
4.3
11.0
9.9
3.9
2.7
2.5
2.6
5.3
PiSSA
64
16.4
2.9
3.3
loftQ
64
5
10.1
18.8
18.2
9.9
7.1
7.1
7.1
11.2
ILaMA -3-8B
PISSA
64
5
17.1
27.3
19.5
12.1
8.9
7.2
7.6
14.3
lofiQ
128
8.2
20.7
18.8
7.5
5.2
4.8
4.9
10.0
PiSSA
128
17.1
26.5
10.7
10.7
7.0
5.0
5.6
11.8
loftQ
128
5
16.4
29.8
28.8
16.1
11.9
11.7
11.7
18.1
PiSSA
128
5
26.3
41.7
32.3
20.1
14.4
12.5
12.9
22.9
QLoRA
0
0
LoftQ
64
1
2.4
11.6
9.2
1.9
1.8
1.7
1.3
4.3
PiSSA
64
1
12.3
25.0
9.0
4.1
4.2
3.2
2.2
8.6
LLaMA -3-70B
LofQ
64
4.4
4.2
PiSSA
64
34.2
18.9
7.5
5.7
4.7
PiSSA
128
1
17.7
36.6
15.7
6.7
5.8
4.5
3.8
13.0
PiSSA
128
5
23.2
49.0
30.5
12.5
10.1
8.8
8.2
20.3
According to Table multiple iterations can significantly reduce quantization error. For instance, using QPISSA-r64 with 5-iter on LLaMA-3-88 reduces the quantization error nearly twice as much as with 1-iter. In the main paper, we used 5 iterations Section and Section [.4 while iteration was used in Section 5.5
20

---

Conductive Experiments on Various SVD Components
To investigate the influence of singular values and vectors of varying magnitudes on the fine-tuning performance, we initialize the adapters injected into LLaMA 2-7B, Mistral-7B-vO.1 and Gemma-7H with principal, medium, and minor singular values and vectors. These models are then fine-tuned on the MetaMathQA dataset [2] and evaluated against the GSM8K [4] and MATH datasets [67], with outcomes depicted in Figures
0.32
Priddpa
Princ pal Hidde
30-
Princpa pal
Midde
GS Acamazyys
Minor
MATH Accuri
25-
Ninor
8.266
19
15
0.22
0.20
0.18
0.18
30
Gomma-78
Gamma-78
Gamma-75
Figure 8: Initializing with principal, medium, and minor singular values and vectors, the training loss on the MetaMathQA and the accuracy on the GSM8K and MATH validation sets are reported, respectively, for three models.
The results highlight that initializing adapters with principal singular values and vectors consistently leads to reduced training loss and enhanced accuracy on both the GSMSK and MATH validation datasets across all three models. This underscores the efficacy of our strategy in fine-tuning the model parameters based on the principal singular values.
G The Residual Matrices having a Narrower Distribution
To intuitively compare the distribution differences between quantized original and residual models, in Figure 3 we took LLaMA 2-7B's first Query layer as an example to illustrate the distribution of W and W res- However, using only one layer of one model is not statistically significant. In this section, we applied PiSSA initialization to LLaMA-2-7B. Mistral-7B, Gemma-7B, and LLaMA-3-88 and fit the values in every linear layer with Gaussian distribution and calculated their mu and sigma.
Bisa matices, pn Olumn
Barahtal to
302
(a) LLaMA-2-7B (b) Mistral-7B-vO.1
(c) Gemma-7B
(d) LLaMA-3-8B
Figure 9: Comparison of Loss and Ratio to the target A and target B for LoRA and PiSSA across the initial steps.
The results in Figure show that the residual models' means are closer to and the standard deviations are smaller after PiSSA initialization. Thus, Wrres indeed has narrower distribution than W in statistical sense. Nevertheless, the difference is not as large as that the first layer after averaging all layers, which we suspect is because middle layers in model tend to have more even eigenvalue distributions due redundancy and insufficient training.
21

---

H
Comparing the Quantization Error of QLoRA, LoftQ and QPiSSA
This section extends the discussion in Section by providing comprehensive comparison of the quan- tization errors associated with QLoRA, LoftQ, and QPiSSA. Using the (eaoooooss___atttg____ of LLaMA 2-7B as an example, we illustrate the singular values of critical matrices during the quantization process with QLoRA, LoftQ, and PiSSA in Figure I0 A larger sum of the singular values (nuclear norm) of the error matrix indicates greater quantization error.
anou ur value magnk.de
20-
snquua or voluo magemmmgnnco
20
1.25
15
15
1.00
10-
55
100
50
100
55
100
Pr nc pal Singuiar Valume
3000
index
index
(a) Original matrix W
(b) Quantized matrix n. (1WW)
(c) Residual matrix Wree
wwwpliee mugrttdd
2.4
0.4
3.4
0.4
0.055
0.050
angu anguarraie
0.3
0.225
0.045
0.2
300
50
0.2
300
angu
0.0
1000
2000
1000
4000
0
1000
2000
1050
4000
0.0
1000
2000
3000
4000
sing cler va us Indes
singuler inder
stng uler usu inder
(d) Error of QLoRA
(e) Error of LoftQ
(f) Error of PiSSA
Figure 10: Several important singular values for calculation the quantization error of QLoRA. LoftQ and PiSSA.
The quantization error of QLoRA, which quantizes the base model to Normal Float 4-bit (NF4) and initializes A and B with Gaussian-Zero initialization, is:
Quantization Error of QLoRA = ||W - (mf4(W) + AB) 1l. = | n.4(WW
(13)
As shown in Equation 13 LoRRA decomposes the original matrix in Figure 10a into the sum of a quantized matrix (Figure 10b) and an error matrix (Figure [Od). By comparing Figure 10a and Figure [Od]
that the magnitude of the error matrix is much smaller than that of the original matrix. Therefore, the benefit of preserving the principal components of the W matrix with the adapter greater than that of preserving the principal components of the error matrix with the adapter.
LoftQ 14, designed to preserve the principal components of the error matrix using the adapter, first performs singular value decomposition on the quantization error matrix of QLoRA:
Uer yerry = W - -nf4(W),
(14)
then uses the larger singular values initialize A and B, thereby reducing the quantization error to:
min(m.n)
LoftQer = I (nf4(W) + AB) || = || (rang) Vej
Σ
[
(15)
LoftQ eliminates only the largest r singular values |(or] (see Figure 10eh from the QLoRA error matrix (Figure 10dp.
Our PiSSA, however, does not quantify the base model but the residual model:
Quantization Error of PiSSA- = IIW - (nf4(WWres) + AB) ||- = || n2 r2f4(W (16)
where A and B are initialized following Equation 2 and 3 Since the residual model has removed the
lage-singular-vatee components, the value distribution of Weres can be better fitted by Student's t-distribution with higher degrees of freedom compared to W (as can be seen in Figure 11) and thus quantizing wree results in lower error using 4-bit NormalFloat (shown in Figure 10
22

---

df-1.90, loc-l 1e--66, scale-
df-2.01, locm- 9ocm-5.6e-05, scale=2e-3
80
Protabiiity density
60-
Probability density
150
40-
100
50
-0.04
-0.03
-0.02
-0.01
0.00
0.01
0.02
0.03
-0.04
-0.03
-0.02
0.00
0.01
0.02
0.03
0.04
Data
(a) The original matrix W
The residual matrix W,es
Figure 11: Fitting the original matrix and the residual matrix using Student's t-distribution.
I Combining QPiSSA with Various Quantization Methods
In addition NF4 quantization, QPiSSA also be combined with GPTQ and INT8 quantization. We posit that PiSSA effectively reduces quantization error for several reasons:
It reduces outlier values;
It makes the value distribution more Gaussian-like;
It preserves larger values in full precision, thereby narrowing the weight distribution in the quantized portion.
While INT8 also targets the reduction of outlier values (point 1). PiSSA has the potential to enhance this effect. The second point aligns well with NF4, and the third point is crucial as PiSSA uses an adaptor to retain significant portion of weights in full precision, maintaining the integrity of critical values.
Table 9: Quantization Error and Accuracy for PiSSA Combined with Various Quantization Methods. GPTQ quantizes each row w independently, adjusting one weight at time while updating all remain- ing, non-quantized weights. Therefore, the nuclear norm method used for calculating quantization error in the main paper is not applicable to GPTQ. Instead, we measure Perplexity on WikiText-2, where lower Perplexity indicates reduced quantization error.
Quantization Error
GSM8K Accuracy
Model
Dataset
QLoRA
PISSA
QLoRA
PISSA
NF4
324.8 (nuclear norm)
265.8 (nuclear norm)
70.7940.42
73.7620.20
LLaMA-3-8B
INT8
34.47 (nuclear norm)
28.21 (muclear norm)
71.6840.14 14
76.5420.32
GPTQ
20.79 (perplexity)
6.23 (perplexity)
70.18+0.42
74.58=0.22
As shown in Table QPiSSA combined with INT8 reduces quantization error by 18.16% % on LLaMA-3-8B, and significantly outperforms QLoRA using INT8. Furthermore, in row 3 of the table, the perplexity of LLaMA-3-8B increases to 20.79 after quantization with GPTQ-4bit on the dataset. However, when PiSSA is applied, the perplexity is reduced to 6.23. These results confirm the effectiveness of PiSSA in reducing quantization error, as discussed the main paper.
Overall, QPiSSA demonstrates a clear advantage over QLoRA when combined with various quanti- zation methods, retaining the fast convergence and superior performance characteristics of PiSSA while minimizing quantization error.
23

---

Evaluating PiSSA on Mixtral and Gemma with More Training Steps
This is the supplement for Section 5.2 We applied PISSA, LoRA, and full parameter fine-tuning on the full MetaMathQA-339 dataset using Mistral-7B and Gemma-7B models, training for 3 epochs. and 13 display the training loss, gradient norm, and evaluation accuracy on GSM8K.
Figures
Litta
LRRA
30.
PISZA
75
160
25
100
(a) Loss over training steps.
(b) Grad norm over steps.
(c) Accuracy GSMSK over training steps.
Figure 12: Fine-tuning Mistral-7B-vbl on the MetaMathQAA3955 dataset for 3 epochs: compari- son of full parameter fine-tuning (indicated by a dashed line), LoRA (in blue), and PiSSA (in orange).
ASSA
15.0
12.
3.8-
100
100
100
3.20
335
416-
25
1002
2020
1000
$920
5000
4020
1600
6000
4000
(a) Loss over training steps. (b) Grad norm over steps.
(c) Accuracy on GSM8K over training steps.
Figure 13: Fine-tuning Gemma-7B on the MetaMathQA-3955 dataset for 3 epochs: A comparison of full parameter fine-tuning (indicated by a dashed line), LoRA (in blue), and PiSSA (in orange).
13a the loss for full parameter fine-tuning decreases sharply with each epoch, indicating overtitiing to the training data. Notably, during the entire first epoch, the loss for full parameter fine-tumi tne-ttnnng on Mistral and Gemma significantly higher than for LoRA and PiSSA, suggesting that full parameter fine-tuning has weaker generalization capabilities compared to LoRA and PiSSA on Mistral-7B and Gemma-7B models. The gradient norm for the first epoch in Figure 13b fluctuates dramatically with each step, further indicating instability in the training process for full parameter fine-tuning. Consequently, as illustrated Figures 12c and 13c the performance of full parameter fine-tuning is markedly inferior to that of LoRA and PiSSA. These experiments demonstrate that using efffieter-efficient fine-tuning can prevent the over-fitting issue caused by over-parameters.
As shown in Figure 12a and
24

---

K
Supplementary Experiments on Various Ranks
K.1
Quantization Error for More Type of Layers
Figure 7a oalo shows the reduction ratio of quantization error for "q proj" layers. In Figure 14 we present the error reduction ratios for the remaining types of linear layers under different ranks.
OL
OLIRA
OLIRA
Lon0
LOFO
kank
gate_proj
down_pro)
ASSA
320
94
126
16
125
Figure 14: Comparison of quantization errors in QLoRA, LoftQ, and PiSSA across k_proj_ v_proj. o_proj and gate_proj up_proj, down_proo) layers.
From Figure 14 it can be observed that under different ranks, the reduction ratio of quantization for various linear layers in LLaMA-2-7B, including "k_proj"", "v_proj", "o_proj", "gate_proj", "up_proj"", and "down_proj"" layers, is consistently lower with PiSSA compared to LotfQ.
25

---

K.2 Evaluation Performance for More Model on Various Ranks
only validated the effectiveness of LLaMA-2-7B. In Figure 15 comparative results of Mistral-7B-v0.1, and Gemma-78 7B under different ranks.
Section 5.5
we also present the
Mistral-8Bww..1
Gemman78
0.25
0.375
Mooa/ath LUGO
0.24
VecNWNH 1000
0.23
0.350
0.325
0.22
0.300
0.22
0.10
0.205
0.18
Pank
(a) Final training loss across different ranks.
Fivetuning
PSCA
7B
73.0
72.5
COMA GOMACNN Co
03.
72
925
70-
64
33.0
60
124
Back
Amnc
Aank
(b) Rank Rise performance evaluated using pass@ I on the GSM8K dataset.
Finduring
LdAA
PSSK
2-78
Mistral-7Bvo.11
23
23
128
15
120
Park
Park
Blrk
(c) Rank-wise performance evaluated using pass@1 on the MATH dataset.
Figure 15: Fine-tuning LLaMA 2-7B, Mistral-7B-v0l and Gemma-7B on the MetaMathQA dataset: A comparison of full parameter fine-tuning (indicated by dashed line), LoRA (in blue), and PiSSA (in orange).
From Figure 15 PiSSA uses fewer trainable parameters compared to LoRA while achieving even surpassing full-parameter fine-tuning on ILaMA-2-7B and Mistral-7B. Remarkably, on Gemma-7B, PiSSA exceeds full- nall-parameter fine-tuning performance even rank=1. However, as the rank increases to 128, the performance of PiSSA begins to decline, indicating that PiSSA over- weerparameterizes earlier than LoRA. This parameterizaiinonnnn phenomenon does not occur on LLaMA-2-77 suggesting that increasing the rank further might enable PiSSA to achieve even higher performance on LLaMA-2-7B.
26

---

K.3 More Training Loss and Grad Norm under Various Ranks
In Figure 16 and we examining the loss and gradient norm during the training process of PiSSA and LoRA on LLaMA 2-7B, Mistral-7B-vO.. and Gemma-7B 7B using different ranks.
Pach-a0
1000 R01E0
Figure 16:
Comparison of training loss for LLaMA-2-7B, Mistral-7B, and Gemma-7B, organized into three rows, using LoRA and PISSA across ranks 2', E [o, 7], organized into eight columns.
1068
M0pe
Figure 17: Comparison grad norm for LLaMA-2-7B, Mistral-7B, and Gemma-7B 7B, organized into three rows, using LoRA and PISSA across ranks 2 E [0, 7]. organized into eight columns.
From Figure 16 PiSSA consistently shows faster initial loss reduction compared LoRA across various ranks. Additionally, the final loss remains lower than that of LoRA. This ad advantage is particularly pronounced when the rank is smaller. From Figure 17 the gradient norm of PiSSA remains consistently higher than that of LoRA throughout the training process, indicating its efficient fitting of the training data. closer look at the first few steps of LoRA's gradient norm reveals trend of rising and then falling. According to Section 3 LoRA's gradients are initially close to zero, leading to very slow model updates. This requires several steps to elevate LoRA's weights to higher level before subsequent updates. This phenomenon validates our assertion that LoRA wastes some training steps and therefore converges more slowly. demonstrates the robustness of the faster convergence property of PiSSA across various ranks.
27

---

L Experimental Settings on NLU
Datasets We evaluate the performance of PiSSA on GLUE benchmark, including single-sentence classification tasks (CoLA, SST), 5 pairwise text classification tasks (MNLI, RTE, QQP, MRPC and QNLI) and text similarity prediction task (STS-B). We report overall matched and mismatched accuracy on MNLI, Matthew's correlation on CoLA, Pearson correlation on STS-B -B,, and accuracy on other datasets.
Implementation Details To evaluate the performance of PiSSA intuitively, compared PiSSA LoRA with the same number of trainable parameters. DeBERTa-v3-bar has 184M trainable parameters. PiSSA and LoRA were applied to Wo. WK and Wy respectively, resulting in total of 1.33M trainable parameters.
The results for full fine-tune, BitFit 15]. HAdapter 30
LoRA with Gassian ini- tialization [ and AdaLoRA are sourced from AdaLoRA 58 based on five runs. The remaining results use the publicly available LoftQ [14] codebase and are averaged over three runs. In LoRA, the matrix is initialized zero, while the A matrix can be initialized using various methods, such as Gaussian initialization and Kaiming initialization [68]. The selection of the imitialization method can influence the final results. In this paper, we report the different results of LoRA based on Gaussian initialization and Kaiming initialization in the experiments, as shown in Table and Table []] For DoRA, we used the code from the PEFT package for deployment and conducted search on key hyperparameters. We set the rank of PiSSA in this experiment 16. We utilize A with linear learning rate schedule to optimize and tune learning rate (LR) from 1.-42e-4.3e-4.4e4..3
PAdapter
36
8 and selecte lora alpha in 8,
6e-4, 5e-5, 3e-5. Batch sizes (BS) are selected from 6, 8, 16, 32. The hyperparameter configurations PiSSA, DoRA and LoRA with Kaiming Initialization are shown in LoRA denotes LoRA with Kaiming initialization, and denotes LoRA alpha.
Table 10
Table 10: Hyperparameters PiSSA, DoRA and LoRA with Kaiming Initialization on GLUE.
PiSSA
DoRA
LoRA
Dataset
Epoch
BS
LR
Epoch
BS
LR
Epoch
BS
LR
MNLI
5
16
5c-4
10
32
2c-4
16
10
32
3e-4
8
SST-2
20
16
3e-5
10
16
4c-4
16
10
32
Ic-4
8
MRPC
20
2c-4
32
4e-4
4e-4
8
CoLA
20
le-4
20
8
le-4
6
30
32
4e-4
8
QNLI
10
32
le-4
16
10
16
2e-4
16
25
32
3e-4
8
QQP
10
16
le-4
10
16
1e-4
6
10
16
3e-4
8
RTE
.50
le-4
50
8
2c-4
6
50
4c-4
8
STS-B
20
3e-4
16
3e-4
4e-4
28

---

M
Comparison of Initial Gradient Subspaces
To compare the gradient subspaces of PiSSA and LoRA, we conducted two additional experiments to validate our analysis.
First, we trained LLaMA-3-8B the MetaMath dataset five times, initializing LoRA with different random seeds while using the same batch of 128 training examples to compute LoRA's gradients.
After performing dimensionality reduction two dimensions, the results are presented in Table [1
Table 11: Ablation results for LoRA and PiSSA across different seeds.
Method
Seed O
Seed 1
Seed 2
Seed 3
Seed
LoRA
[0,0] [0.1]
[0.0]
[0,0]
[0,0]
[0,0]
grad_A A
PiSSA
[0.1]
[0,1]
[0.1]
[0,1]
LoRA
[-0.99, 0.121
[0.95, 0.31]
[0.46, -0.891
[0.24, 0.97]
10.04, -0.991
grad B
PISSA
[1,0]
[1,0]
[1,0]
[1,0]
[1,0]
We observe that the gradient of matrix A remains consistently zero, while the gradient direction of matrix B varies across initializations. This behavior arises because matrix A's gradient depends on matrix B. which in LoRA is initialized to zero, resulting in a zero gradient for A. In contrast, matrix B is initialized from Gaussian distribution, leading to variation in its gradient direction across different seeds. In comparison, PiSSA's gradient direction remains consistent across all five training runs, as it solely depends on the original model and the training data. This experiment highlights the stability of PiSSA's optimization trajectory relative to LoRA's more variable directionality.
Next, quantitatively compared the effect of updating along the principal singular value direction versus a "random" direction during the early stages of fine-tuning. We trained LLaMA-3-8B on the MetaMathQA dataset using both PiSSA and LoRA, saving the parameters and gradients from the first 50 iterations. At the 50th step, the loss values for LoRA and PiSSA were 0.3677 and 0.2899, respectively. Using the parameters from the 50th step as the target point, we evaluated the movement in the first five steps relative to the target, computing how much progress was made towards the final point. We then divided this progress by the total target distance to obtain a ratio. These ratios are shown in Figure 18
0.9
LoRA PSSA
Left PSSA
LoRt PISSA
25-
20
Babe oN
20
15
15-
10
3.5
34
Stops
(a) Loss over steps.
(b) Ratio to target A over steps.
(c) Ratio to target A over steps.
Figure 18: Comparison of Loss and Ratio to the target A and target B for LoRA and PiSSA across the initial 5 steps.
The results reveal that after just five updates, PiSSA reduced the loss from 0.8884 to 0.3346, while LoRA's loss reduction was more modest, dropping only 0.5538. This demonstrates the advantage of updating along the principal singular value direction, which PiSSA leverages, leading to faster convergence. Further, in the first step, matrix A in LoRA exhibited zero gradient and therefore did not update. Over the next four steps, it moved only 15.94% towards the target direction. Similarly, matrix B in LoRA consistently moved less towards the target endpoint compared to PiSSA.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: 5
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwticccguie//eeieeeeeeii CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: eтror bars are not reported because it would be too computationally expensive
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::::////iis..eeeeeeee htt.://.......boooocttcccccsuued
Answer: [Yes]
Justification: the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We only use public available datasets.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
33

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We only use public available datasets and models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
34

---

Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "3cmHobmADo", "reviewer": "Reviewer_x9Pi", "summary": "The paper presents a method for creating lora-like fine-tuning adapters for base models. The idea is to initialize W, A and B matrices in LoRA from SVD decomposition. W is consists of less important ranks while A and B consist of the most important ranks. This implies that less important principle components of the model remain frozen while the most important ones get updated. Another benefit of this scheme is that A and B can be in high precision while W is in low precision, thus ensuring a reduced quantization error.", "questions": "Will the authors be releasing their code? I believe it is very important that they do as our community could greatly benefit from being able to reproduce the experiments here and build on top of it.", "limitations": "The limitations section in the paper is good enough for me.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I think this is a good paper with good results and a potential for significant impact as adapter tuning becomes more mainstream. The paper is written well, the method is intuitive, and the experimental results are thorough.", "weaknesses": "As authors note in the limitations section there are many more experiments they could have conducted, but those are future scope. I am happy with the current scope and experiments. Having said, that I wish the authors had included at least a few experiments comparing advanced LoRA adapter methods. DoRA is mentioned in related works, but I couldn't find any comparison with it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the originality of PiSSA, the quality of our experiments, and our contributions to the community. Here are the answers to your queries:\n\n**Q1: Comparison with advanced LoRA adapter methods, e.g., DoRA**\n\n**A1**: In our paper, we demonstrated PiSSA's effectiveness through large amount of experiments on 5 NLG and 8 NLU tasks, utilizing models ranging from 184M to 70B parameters. Our comparisons included 1k-9k+ training steps and ranging from 1 to 128 ranks, against methods such as LoRA, QLoRA, LoftQ, and full-parameter fine-tuning. PiSSA modifies only the initialization method used by LoRA, making it orthogonal to some LoRA enhancements. Initially, due to time and resource limitations, we did not compare PiSSA with DoRA. However, upon your request, we have now included these comparisons.\n\nDoRA adds a learnable magnitude module to LoRA, normalizing $W + AB$ at each update step and multiplying its by the magnitude module. This allows $A, B$ to learn the direction and the magnitude module to learn the magnitude of $\\Delta W$. While this approach can improve fine-tuning performance, normalizing $W + AB$ at each step results in slower fine-tuning speeds. In contrast, PiSSA only changes LoRA's initialization method, matching LoRA in training speed and converging faster, thereby reducing training costs.\n\nWe compared PiSSA with DoRA on LLaMA-3-8B for GSM8K and DeBERTa-v3-base for GLUE tasks, repeating each experiment three times and recording the results in **Table 1** and **Table 2**.\n\nFrom these tables, PiSSA significantly outperforms DoRA, as DoRA still uses zero and random initialization.\n\n**Q2: Open source schedule**\n\n**A2**: PiSSA has greatly benefited from the open-source community, and we are excited to contribute back. \n\nPiSSA only modifies the initialization method of LoRA, so it incurs no additional training costs, requires no extra environmental setup, and involves no changes to existing code. This allows for straightforward training and inference, just like LoRA, given the initialized parameters. We plan to release all PiSSA-initialized models from our paper, along with the initialization of future mainstream models, to facilitate easy adoption.\n\nFor users interested in customizing PiSSA configuration or modifying its core code, we have already integrated PiSSA  into HuggingFace’s PEFT library and provide the training scripts used in our paper to replicate the results. Additionally, we will release training logs and checkpoints from our experiments to aid in further comparisons and validations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhGhoDJPGd", "reviewer": "Reviewer_XzvL", "summary": "In this paper, the authors proposed a modified low-rank adaptation (LoRA) method for fine-tuning large pretrained models. Specifically, the proposed method initializes the A and B matrices with singular matrices and freezes the weight matrix to be the residual. Further, the authors provide a quantization step to reduce the memory consumption. The authors demonstrate the effectiveness of this proposed method on a lot of models and tasks, showing that this method converges faster than the conventional LoRA method.", "questions": "Please see the comments in the \"limitation\" block.", "limitations": "There is no negative societal impact as far as I can tell.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall, this paper is well-written. The author did a great job of presenting the new idea with details. The contents are organized well. \n- The authors comprehensively evaluated the proposed method, both quantitatively and qualitatively. \n- The proposed method is simple, yet effective.", "weaknesses": "- It would be great if the authors could provide more insight into the proposed method. For example, is it possible to compare the subspace of the gradient of conventional LoRA? \n- The proposed method utilizes the first r eigenvectors. How about other dimensions in the subspace?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of PiSSA as a simple yet effective method, and for your appreciation of the article's quantitative and qualitative analysis. Here are responses to the concerns raised:\n\n**Q1: Provide more insight, for example, comparing the subspace of the gradient of conventional LoRA?**\n\n**A1**: PiSSA potentially provides many insights for the PEFT area:\n\n1. To reduce fine-tuning parameters, LoRA and many of its variants add a low-rank adaptor $\\Delta W = AB$ to $W$, and initialize $\\Delta W$ with zero to ensure $W + AB$ initially does not change the model. However, PiSSA **for the first time** touches the frozen $W$ by decomposing $W$ into principal and residual components with SVD, which are used to initialize $\\Delta W$ and $W^{res}$ respectively. This idea has already inspired many follow-up works studying the optimal distribution plans for $\\Delta W$ and $W^{res}$, which contributes greatly to the community.\n2. By putting the principal components of $W$ into the adaptor $AB$, PiSSA can directly fine-tune the core functions of the model instead of fine-tuning the random/zero-initialized $AB$ as in LoRA. This leads to faster convergence and higher performance. From the gradient direction perspective, we have shown in lines 50-55 and lines 123-131 of our paper that the **initial gradient for LoRA is zero/random**, thus could **waste much time around the initial point**, while PiSSA optimizes along the gradient direction of the model's principal components, thereby approximating the effect of full parameter fine-tuning. The toy experiment in Figure 2a using the MNIST dataset and an MLP with 2 linear layers visually demonstrates that PiSSA and full fine-tuning **converge much faster** compared to LoRA. In the supplementary materials, the loss_landscape.gif shows this process dynamically. In LLM experiments, the advantage of PiSSA over LoRA and the similarities between PiSSA and full fine-tuning can also be observed from the loss and grad norm during the training process in Figure 4a and 4b.\n3. Compared to LoRA + quantization, PiSSA + quantization preserves the principal singular values and singular vectors in full precision while quantizing the residual part $W^{res}$ instead of the full $W$. Since the large singular values have been removed from $W^{res}$, its value distribution is much narrower than $W$, which makes quantization easier (the larger variance of the value distribution, the larger error quantization will introduce). This can be seen in Figure 3 and 5, where QPiSSA greatly **reduces the quantization error** and shows better performance than QLoRA.\n\nRegarding your question about comparing the subspace of gradients of PiSSA and LoRA, we further validated our analysis through two additional experiments. \n\n1. We trained LLaMA-3-8B on the MetaMath dataset 5 times, each time initializing LoRA with different random seeds while using the same batch of 128 training examples to calculate LoRA's gradients. After dimensionality reduction to two, we list the results in **Table 3 of General Response**. We can observe that the gradient of matrix $A$ is consistently zero, and the gradient direction of $B$ varies with each initialization. This occurs because the gradient of matrix $A$ depends on matrix $B$, which is initialized to zero in LoRA, resulting in a zero gradient for matrix $A$. Conversely, the gradient of matrix $B$ is influenced by matrix $A$, which is initialized from Gaussian, hence the varying gradient directions of matrix $B$ in each experiment. In contrast, the gradient direction in PiSSA remains consistent across all five training seeds and only depends on the original model and the training data. This experiment underscores the stability of PiSSA's optimization direction relative to LoRA.\n2. Furthermore, we quantitatively compared the effects of updating using the principal singular value direction v.s. “random” direction in the early stages of fine-tuning. We trained LLaMA-3-8B on the MetaMathQA dataset using PiSSA and LoRA, saving the parameters and gradients for the first 50 iterations. At the 50th step, the losses for LoRA and PiSSA dropped to 0.3677 and 0.2899, respectively. We took the parameters at the 50th step as the target point. Using the direction and distance from the initial parameters to the final step's parameters as a reference, we compared the first 5 steps for LoRA and PiSSA. For each step, we determined how much was moved in that direction, then divided by the target distance to obtain a ratio. The ratios are recorded in **Table 4 of General Response**. As shown in the first 2 rows of the table, after just 5 updates, PiSSA's loss reduced from 0.8884 to 0.3346, whereas LoRA only reduced to 0.5538, reflecting the advantage of the principal singular value direction over the zero-random direction in convergence speed. In rows 3-4 of the table, matrix $A$ in LoRA had a gradient of 0 at the first step and thus did not update. In the following 4 steps, it only moved 15.94% towards the target direction. In rows 5-6 of the table, matrix $B$ in LoRA always moved less towards the endpoint in the target direction compared to PiSSA.\n\n**Q2: Besides the first r eigenvectors. How about other dimensions in the subspace?**\n\nA2: Our PiSSA puts the first $r$ singular values and vectors in the adaptor $A,B$ and the remaining ones in the frozen $W^{res}$. In Figure 8 of the Appendix, we compared the training outcomes of using the principal, middle, and minor $r$ singular values and singular vectors to initialize $A,B$. The experimental results evident that training with the principal singular values results in lower loss and higher accuracy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CAfRDl34vr", "reviewer": "Reviewer_Bb8M", "summary": "The authors present PiSSA, a relatively simple change to the LoRA framework leading to a large amount of demonstrated benefits.  PiSSA proceeds by adjusting the initialization step of standard LoRA; rather than freezing the original weight matrix W and learning a low-rank perturbation W' = W + BA, PiSSA instead considers the SVD W = USV^T = U[:,r:]S[r:,r:]V[:,r:]^T + U[:,:r]S[:r,:r]V[:,:r]^T = W^{res} + AB, s.t. A=U[:,:r]S^{0.5}[:r,:r], B = S^{0.5}[:r,:r]V[:,:r]^T.  Due to this backwards compatibility, PiSSA enjoys the fine-tuning speed/memory savings of LoRA while also being easily implementable to existing PEFT packages.  Most importantly, the authors demonstrate across a large number of models, fine-tuning recipes, and benchmarks that PiSSA greatly improves on downstream accuracy compared to LoRA.  Furthermore, the authors show that this initialization + fine-tuning strategy also leads to significant decreases in quantization error, outperforming both QLoRA and the recently proposed LoftQ both in terms of training loss (e.g., faster loss reduction) and performance for GSM8K and MATH tasks.", "questions": "See weaknesses.\n\nMinor comment:\n\"loftQ\" in Table 3 <- \"LoftQ\"", "limitations": "The authors have addressed potential limitations of the demonstrated method.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**Originality** - the specific algorithm is an original, lightweight alteration to the LoRA framework.  In particular, the fact that there are no additional hyperparameters and, after the initialization phase, PiSSA is essentially swappable with LoRA implementations are all significant benefits of the method.  Furthermore, PiSSA's use of only fine-tuning the principle components in a LoRA fashion lends itself to strong intuition as to why the method leads to such strong downstream fine-tuning performance, and why LoRA is less successful in comparison.  The latter point is also true when considering quantization.\n\n**Quality** - the extensive experiments across models and fine-tuning evaluations well convey the benefits of this method over standard LoRA.  This is also true of the extensive quantization experiments.\n\n**Clarity** - for the most part, the paper is well written. However, there are specific points where the paper could improve clarity (see weaknesses below).\n\n**Significance** - the performance benefits and backwards compatibility lead to significant opportunity for future impact of this work.", "weaknesses": "# Clarity\nThere are several areas the paper could improve in clarify.  In particular, the literature review requires more granularity, e.g., \"AdaLoRA [42, 41, 43],\" only one of these works is AdaLoRA.  Furthermore, there is overlap between AdaLoRA and PiSSA in both their reliance on the SVD.  It is necessary to discuss exactly how AdaLoRA leverages SVDs, and how this differs from PiSSA.\n\nThere are several experimental details missing, e.g., are the GSM8K evals 8-shot?  What are the shots used for the other benchmarks?  Were the benchmarks run using the Eleuther eval harness?  Please include these details in the paper.\n\nPaper at times states it is similar to LoRA and, at other times, seeks to differentiate itself:\n- \"Unlike LoRA and its successors, which focus on learning low-rank approximations of weight updates, our PiSSA approach directly tunes the essential but low-rank parts of the model while keeping the noisier, high-rank, and nonessential parts frozen. Although our approach differs in philosophy from LoRA, it shares most of LoRA’s structural benefits and can be extended by these methods to enhance its performance\"\n- \"Since PiSSA shares the identical architecture with LoRA, it inherits most of LoRA’s benefits.\"\nThis was confusing on an initial pass of the paper.  A short, succinct list enumerating the similarities and differences early on may help readers understand exactly how PiSSA distinguishes itself from LoRA, but how it is able to inherit all of LoRA's PEFT benefits.\n\n# Quality\nThe authors consider the weight matrix of a single attention layer in Llama-2 in Figure 2:\n\"Since the residual model has removed the large-singular-value components, Wres has a narrower\n distribution than that of W, as can be seen in Figures 3a and 3b (comparing the singular value distributions of W and Wres), as well as Figures 3c and 3f (comparing the value distributions of W and Wres), which is beneficial for reducing the quantization error\" <- However, since this only looks at a single layer of a single model, it hard to accept this as evidence for a general claim.  Can you include a comparison of the distributions of mu/sigma for W and W^{res} (e.g., mean absolute differences between means and ratio between sigmas) across all layers for Llama2 and another model (e.g., Mistral) to show that, in general, W^{res} has a narrow distribution than W?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the originality, quality, and significance of our article. We also appreciate your suggestions for improving the writing. As we cannot modify the original text during the rebuttal period, we will incorporate your recommendations in the camera-ready version.\n\n**Q1: In the related work section, AdaLoRA [42, 41, 43] is mentioned, but only one of these works is AdaLoRA.**\n\nA1: Thank you for pointing this out. Due to the extensive content of PiSSA, we had to condense the text multiple times, leading to this issue. All these three papers dynamically adjust the rank of LoRA at each layer, so we used AdaLoRA as a collective term. We found similar issues in the subsequent DeltaLoRA [44, 45] citations. We will complete the citation information.\n\n**Q2: There is overlap between AdaLoRA and PiSSA in their reliance on the SVD. Discuss exactly how AdaLoRA differs from PiSSA.**\n\nA2: AdaLoRA introduces three improvements over LoRA:\n1. Trainable parameters in AdaLoRA are changed to $A, B$, and $E$. $A$ and $B$ are Gaussian-initialized, and $E$ is a zero-initialized $r$-dimensional vector, making $A diag(E) B = \\Delta W$, similar to singular value decomposition.\n2. A regularization loss $\\|AA^T-I\\|+\\|B^TB-I\\|$ is used to make $A$ and $B$ gradually orthogonal during training, resembling the SVD of $\\Delta W$.\n3. An initial large rank is set, and less important E values are gradually masked during training, resulting in different final ranks for each layer, achieving better performance with the same number of parameters.\n\nDespite the extensive use of SVD terms, AdaLoRA **does not perform actual SVD on any matrix**. In the PEFT domain, terms like low-rank decomposition, and singular value decomposition often appear. They generally refer to products of low-dimensional matrices approximating an ideal $\\Delta W$ without actual matrix decomposition. To our knowledge, PiSSA is the first to perform SVD on the original model, fine-tuning the principal component while keeping the residual model frozen. During the rebuttal, we have evaluated PiSSA against AdaLoRA on GSM8K and GLUE tasks. The results in **Table 1 and Table 2 of General Response** demonstrate that PiSSA outperforms AdaLoRA. \n\nAdditionally, PiSSA and AdaLoRA represent different improvements to LoRA, making them combinable. Therefore, we additionally improved PiSSA based on AdaLoRA's three innovations:\n\n1. After extracting the principal singular values and vectors of $W$, we use $S$ as an independent trainable vector instead of multiplying it into $U$ and $V$.\n2. Since PiSSA's $U$ and $V$ are orthogonal at the beginning, maintaining their orthogonality through orthogonal regularization is very easy.\n3. Although AdaLoRA claims to dynamically reduce the number of trainable parameters, the initially large number of parameters is not truly pruned, resulting in more parameters being updated during actual training. Therefore, we did not use this improvement.\n\nPiSSA, with its intrinsic principal singular values and orthogonal singular vectors, is very suitable for combination with AdaLoRA. According to **Table 1 of General Response,** The performance of the improved PiSSA surpasses all the other methods including PiSSA.\n\n**Q3: Are the GSM8K evaluations 8-shot?**\n\nA3: This article uses the zero-shot evaluation prompt for GSM8K and MATH following MetaMath [1], the zero-shot evaluation prompt for humanEval and MBPP following WizardCoder [2], and the zero-shot single answer grading prompt for MT-Bench [3]. We will provide detailed settings and corresponding code in the supplementary materials. \n\n[1] Yu, Longhui, et al. \"MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models.\" ICLR24.\n\n[2] Luo, Ziyang, et al. \"WizardCoder: Empowering Code Large Language Models with Evol-Instruct.\" ICLR24.\n\n[3] Zheng, Lianmin, et al. \"Judging llm-as-a-judge with mt-bench and chatbot arena.\" NeurIPS 2024.\n\n**Q4: A short, succinct list enumerating the similarities and differences early on may help readers understand.**\n\nA4: Thank you for the suggestion. PiSSA has the same structure as LoRA, making it easy for users to switch from LoRA to PiSSA for fine-tuning. However, PiSSA's different initialization leads to distinct optimization directions, faster convergence, better training results, and lower quantization errors. We will list the similarities and differences between PiSSA and LoRA in the introduction section of the camera-ready version for easier understanding.\n\n**Q5: Only looks at a single layer of a single model is hard to accept this as evidence for  $W_{res}$ having a narrower distribution than that of W.**\n\nA5: To intuitively compare the distribution differences between quantized original and residual models, we took LLaMA 2-7B's first Query layer as an example to illustrate the distribution of $W$ and $W_{res}$. As you suggested, using only one layer of one model is not statistically significant. In **Table 5 of General Response**, we applied PiSSA initialization to LLaMA-2-7B, Mistral-7B, Gemma-7B, and LLaMA-3-8B, and fit the values in every linear layer with Gaussian distribution and calculated their mu and sigma in the table. The results show that the residual models' means are closer to 0, and the standard deviations are smaller after PiSSA initialization. Thus, $W^{res}$ indeed has a narrower distribution than W in a statistical sense. Nevertheless, the difference is not as large as that in the first layer after averaging all layers, which we suspect is because middle layers in a model tend to have more even eigenvalue distributions due to redundancy and insufficient training. We will include this statistical result in the supplementary materials and reflect it in the main text.\n\n**Q6: Minor comment: \"loftQ\" in Table 3 <- \"LoftQ\".**\n\nA6: Thank you once again for your careful reading and helpful suggestion. Your feedback is greatly appreciated, and we will incorporate all your suggestions in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SG8jKysfgI", "reviewer": "Reviewer_m1t9", "summary": "The paper presents a novel initialization strategy for LoRA finetuning where A and B are initialized according to the top r singular values and the pre-trained weights are initialzed with the remaining components. \nThe authors show that this decomposition can help finetuning converge faster and behaves favorably compared to traditional quantization techniques such as NF4. \nThe authors conduct experiments on NLG and NLU tasks show show performance improvements of PiSSA over vanilla LoRA.", "questions": "- In line 174 the authors mention that $\\alpha=r$, why was this choice made? In [1] $\\alpha$ is set to a constant. Setting $\\alpha=r$ ultimately results in different learning rates for different r.\n- The advantage of of quantized PiSSA over QLoRA might come from outlier removal, was there any experiments done comparing QPiSSA to efficient quantization techniques such as GPTQ [2] or llm.int8 [3]?\n- In line 188 the authors mention that training for NLG experiments was only done for 1 epoch. It would be interesting to see if LoRA ultimately converges to a similar optimum after longer training, or whether the improved initialization also leads to a better optimum.\n- What do the entries in the legend of Figure 2b mean exactly?\n\n[1] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022\n\n[2] Frantar et al., GPTQ: Accurate post-training quantization for generative pre-trained transformers., ICLR 2023\n\n[3] Dettmers et al., LLM.int8(): 8-bit matrix multiplication for transformers at scale., NeurIPS 2022", "limitations": "The fact that PiSSA needs to store 2r ranks compared to r ranks for LoRA which is mentioned in Appendix C should be also mentioned under Limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Presentation**\n\nThe theoretical motivation of the paper is clear and the method is well motivated and explained.\n\n**Empirical results** \n\nThe authors provided several supporting ablations and experiments on improved quantization error compared to LoftQ and QLoRA, faster convergence etc.", "weaknesses": "**Significance of results**\n\nThe authors do not provide error bars in their results which questions whether the obtained results are actually statistically significant.\nError bars should be provided as well as significance tests should be performed. \n\n**Unsupported claims**\n\nThe authors claim that LoRA gradients point in random directions early during the fine-tuning stage which introduces wasteful update steps.\nWas this claim experimentally verified? If so, the authors should clarify how these results were obtained and elaborte on it in more detail in the paper.\n\n**Comparison to baselines**\n\nPiSSA is not compared to other commonly used initialization schemes, e.g., uniform gaussian [1], or kaiming init [2] for NLU and NLG experiments.\nFurther, the authors compare PiSSA only to standard LoRA, but there have been several improvements to LoRA, such as AdaLoRA [3], or DoRA [4].\nA comparison to those extensions would strengthen the paper. \nAlso, the authors may investigate whether their initialization could be applied to these extensions to further improve perormance.\nFinally, some related work uses a similar initialization by considering principal components [5].\n\n[1] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022\n\n[2] He et al., & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification., ICCV 2015\n\n[3] Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR 2023\n\n[4] Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation, ICML 2024\n\n[5] Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters, MLSys 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and constructive review. To answer your questions thoroughly, we have conducted numerous new experiments. We hope these efforts will address your concerns.\n\n**Q1: Need to provide error bars in the results**\n\n**A1**: The original paper includes extensive experiments cover 12 models, 13 tasks, 1k to 9k training steps, and 1 to 128 ranks. It compares PiSSA, QPiSSA with LoRA, QLoRA, LoftQ and Full fine-tuning. Running all these experiments multiple times is very time and resource-intensive. Despite this, PiSSA consistently demonstrates significant convergence speed and performance advantages, suggesting the improvement is statistically significant. The improvements of PiSSA over LoRA are notably highlighted in **Table 1** **and** **Table 2 of General Response**. The error range is very small compared to the improvement.\n\n**Q2: Comparison to Gaussian or Kaiming initialization**\n\n**A2**: **Table 1** and **Table 2** show that PiSSA outperforms both Gaussian and Kaiming initialization schemes, as they still use a zero-random initialization for the adaptor without touching the frozen $W$, while PiSSA for the first time moves the principal components out of $W$ to initialize the adaptor for directly fine-tuning the core model.\n\n**Q3: Comparison to AdaLoRA or DoRA**\n\n**A3**: Please refer to the response to **Q1 of Reviewer x9Pi**, and **Q2 of Reviewer Bb8M**. PiSSA outperforms both methods.\n\n**Q4: Combining AdaLoRA or DoRA with PiSSA**\n\n**A4**: Since PiSSA's structure is identical to LoRA's, it can be combined with other advanced LoRA variants. The specific new challenges introduced by combining PiSSA with different methods are, however, beyond the scope of this work. Nevertheless, we have evaluated the combination of PiSSA with AdaLoRA. Details can be found in **Q2 of Reviewer Bb8M**. PiSSA+AdaLoRA shows superior performance than other methods in **Table 1** including PiSSA.\n\nDue to time and resource constraints, we could not finish experiments combining DoRA and PiSSA. We are actively conducting these experiments, and will post the results in follow-up comment.\n\n**Q5: Comparing QPiSSA to GPTQ or int8**\n\n**A5**: QPiSSA is not a quantization technique, but applies NF4 quantization to PiSSA’s frozen weights; it can also be combined with GPTQ or int8. We believe PiSSA reduces quantization error due to the following reasons: 1. Reducing outlier values; 2. Making the value distribution closer to a Gaussian distribution; 3. Preserving larger values in full precision, which narrows the weight distribution of the quantized part. Int8 also focuses on the first point, and PiSSA could further improve it. The second point is friendly to NF4. The third point is also crucial, as PiSSA uses adaptor to preserve a great portion of weights in full precision. When compared to int8, QPiSSA+int8 reduces quantization error by **23.58%** on LLaMA-3-8B. In **Table 1**, rows 4 and 8 compare QLoRA and QPiSSA using int8, showing that QPiSSA still significantly outperforms QLoRA. We will post the results of combining PiSSA and GPTQ in the follow-up comment.\n\n**Q6: LoRA gradients point in random directions**\n\n**A6**: Please refer to the response to **Q1 of Reviewer XzvL**.\n\n**Q7: S-LoRA uses principal components initialization**\n\n**A7**: Although the term \"principal components\" appears in the third section of the S-LoRA paper, it is not a method focusing on adapter initialization or PEFT. S-LoRA is actually a system designed for the scalable serving of up to 2,000 LoRA adapters. In PEFT, terms like low-rank decomposition and SVD often appear. They mostly refer to using the product of low-dimensional matrices to approximate an ideal $\\Delta W$ through learning. These methods **do not actually perform decomposition** on any matrix. To our knowledge, PiSSA is the first study that directly performs SVD on the original model and extracts the principal singular values and vectors for fine-tuning.\n\n**Q8: Why is alpha set to r?**\n\n**A8**: Since PiSSA extracts the model's principal singular values and singular vectors for fine-tuning, the scaling factor (=alpha/r) in the adapter needs to be set to 1 to ensure adaptor + frozen weights equals the pre-trained weights initially. Nevertheless, it is also possible to set alpha to other values as in LoRA. In NLU tasks, we have experimented with changing this hyper-parameter to other values.\n\n**Q9: Can LoRA ultimately converges to a similar optimum after longer training?**\n\n**A9**: In Sections **5.2** and **5.3** of the original paper, we conducted experiments with larger datasets and longer training steps. And the results show that PiSSA/QPiSSA indeed achieve lower loss than LoRA/QLoRA, indicating convergence to a better optimum.\n\n**Q10: Meaning of ‘iter’ in Figure 2b**\n\n**A10**: The meaning of 'iter' is explained in Appendix E: simply put, it means iteratively executing PiSSA initialization multiple times, compressing the quantization error into the adapter each time to further reduce the quantization errors.\n\n**Q11: Convert to LoRA need 2r ranks**\n\n**A11**: There are three ways of using PiSSA. The first way is to save the adapter with rank $r$ and replace the original model with the residual model. However, sometimes people want to keep the original model unchanged. Thus the second way is to only save the adaptor with rank r without saving the residual model, and recover the residual model when needed by performing SVD again. Since fast singular value decomposition can complete PiSSA initialization within a few seconds to tens of seconds, the second way is also feasible in practice. The third way, which is what is discussed in Appendix C, saves a rank-2r adaptor to avoid SVD (trading-off space for time) and converts a PiSSA adaptor equivalently to a LoRA adaptor during inference. Therefore, we believe that the ability of PiSSA to equivalently convert to 2 times the rank of LoRA is not a limitation but an advantage, providing more flexibility for using PiSSA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
1. **Contribution**
    1. This paper analyzes the gradient of LoRA, showing that $A$ and $B$ initially have zero gradients and random gradient directions, respectively. This leads to slow convergence and might result in suboptimal local minimum points found.
    2. We propose the PiSSA initialization method, which approximates the optimization direction of full fine-tuning through fine-tuning the principal components of a model. To our best knowledge, PiSSA is the first study to perform SVD on the original model and use the principal singular values and singular vectors to initialize the adapter and fine-tune them, while the remaining parts are used to initialize the original model and are fixed during training. Experiments show that PiSSA converges faster and achieves better final results compared to LoRA.  
    3. We combine PiSSA with NF4 quantization to propose QPiSSA, which can reduce quantization error by about 20% compared to QLoRA, while maintaining the fast convergence and good performance of PiSSA.
2. **New experimental results**

    During this rebuttal period, we conducted numerous new experiments to address the reviewers' concerns. We have listed all the experimental results here and referenced them at the respective points of the reviewers' questions.
    
    **Table 1**. The comparisons of PiSSA with LoRA using Gaussian&Kaiming initialization, LoRA+llm.int8, DoRA, AdaLoRA, PiSSA+llm.int8, and PiSSA+AdaLoRA for GSM8K on LLaMA-3-8B. Each experiment was repeated three times, and the average values and standard deviations were recorded.
    
    | Method | Run 1 | Run 2 | Run 3 | GSM8K Average |
    | --- | --- | --- | --- | --- |
    | Full FT | 74.89 | 74.22 | 74.07 | 74.39±0.356 |
    | LoRA(gaussian-init) | 71.11 | 71.19 | 70.74 | 71.01±0.199 |
    | LoRA(kaiming init) | 72.25 | 71.57 | 71.95 | 71.92±0.279 |
    | LoRA+llm.int8 | 71.78 | 71.48 | 71.78 | 71.68±0.143 |
    | DoRA | 72.18 | 72.33 | 72.63 | 72.38±0.189 |
    | AdaLoRA | 72.48 | 72.42 | 72.02 | 72.31±0.202 |
    | PiSSA | **76.72** | **76.72** | **76.80** | **76.75±0.036** |
    | PiSSA+llm.int8 | **76.18** | **76.48** | **76.95** | **76.54±0.318** |
    | PiSSA+AdaLoRA | **78.77** | **78.32** | **78.69** | **78.59±0.199** |
    
    **Table 2**: Fine-tuning DeBERTa-v3-base with PiSSA, LoRA (using Gaussian & Kaiming initialization), DoRA, and AdaLoRA on the GLUE benchmark, including 8 subtasks MNLI, SST-2, CoLA, QQP, QNLI, RTE, MRPC, and STS-B. The results for PiSSA, DoRA, and LoRA (with Kaiming initialization) are averaged over three runs. The other results, taken from the AdaLoRA paper, are averaged over five runs, with only the mean values reported.
    
    | Method | Run 1 | Run 2 | Run 3 | GLUE Average |
    | --- | --- | --- | --- | --- |
    | Full FT | — | — | — | 88.245 |
    | LoRA(gaussian-init) | — | — | — | 88.503 |
    | LoRA(kaiming init) | 88.795| 88.665 | 88.395 | 88.618±0.167 |
    | DoRA | 89.186 | 88.955 | 88.810 | 88.984±0.155 |
    | AdaLoRA | — | — | — | 89.464 |
    | PiSSA | **89.915** | **89.783** | **89.711** | **89.803±0.084** |
    
    **Table 3**. The gradient direction of the first step for PiSSA and LoRA initialized with five different random seeds, using the same batch of 128 data points, was reduced to two dimensions. LoRA's gradient direction is zero and random, while PiSSA's direction is consistent, related to the principal singular value of the model.
    
    |  | Method | Seed 0 | Seed 1 | Seed 2 | Seed 3 | Seed 4 |
    | --- | --- | --- | --- | --- | --- | --- |
    | grad_A | LoRA | [0,0] | [0,0] | [0,0] | [0,0] | [0,0] |
    |  | PiSSA | **[0,1]** | **[0,1]** | **[0,1]** | **[0,1]** | **[0,1]** |
    | grad_B | LoRA | [-0.992,  0.122] | [ 0.9525,  0.304] | [ 0.4587, -0.888] | [ 0.241,  0.970] | [ 0.036, -0.999] |
    |  | PiSSA | **[1,0]** | **[1,0]** | **[1,0]** | **[1,0]** | **[1,0]** |
    
   **Table 4**. The ratio of the distance moved towards the target point by PiSSA and LoRA after 5 update steps.
    
    | Metric | Method | Step 1 | Step 2 | Step 3 | Step 4 | Step 5 |
    | --- | --- | --- | --- | --- | --- | --- |
    | Loss | LoRA | **0.8881** | 0.7943 | 0.6598 | 0.6021 | 0.5538 |
    |  | PiSSA | 0.8884 | **0.6476** | **0.4568** | **0.3657** | **0.3346** |
    | ratio_to_target_A | LoRA | 0.00% | 2.57% | 6.28% | 10.84% | 15.94% |
    |  | PiSSA | **4.31%** | **11.29%** | **17.30%** | **22.77%** | **27.71%** |
    | ratio_to_target_B | LoRA | 3.31% | 9.66% | 15.75% | 21.51% | 27.00% |
    |  | PiSSA | **4.29%** | **11.27%** | **17.29%** | **22.77%** | **27.72%** |
    
    **Table 5**. Using a normal distribution to fit the original model and the residuals model in PiSSA, then recoding the mean and standard deviation of the distribution.
    
    |  | mu | sigma |
    | --- | --- | --- |
    | LLaMA-2-7B | 5.4679e-06 | 0.0193 |
    | PiSSA-LLaMA-2-7B-r128 | **2.6775e-06** | **0.0172** |
    | Mistral-7B | 9.8105e-07 | 0.0033 |
    | PiSSA-Mistral-7B-r128 | **5.3738e-07** | **0.0029** |
    | gemma-7b | 1.3422e-06 | 0.0045 |
    | PiSSA-gemma-7b-r128 | **6.9983e-07** | **0.0040** |
    | LLaMA-3-8B | 5.6969e-06 | 0.0139 |
    | PiSSA-LLaMA-3-8B-r128 | **2.3858e-06** | **0.0118** |

Meta-review: The paper  presents a novel approach to parameter-efficient fine-tuning (PEFT) of large language models (LLMs). The proposed method, PiSSA, is an enhancement of the existing LoRA (Low-Rank Adaptation) method. PiSSA differentiates itself by initializing the adaptation matrices with the principal components obtained through SVD of the original model weights, as opposed to LoRA’s random initialization. This modification allows for more efficient fine-tuning. Additionally, PiSSA shows reduced quantization error when combined with 4-bit quantization (QPiSSA), making it a superior choice for resource-constrained environments. 
There were concerns related to the clarity of the presentation and the  statistical significance of the results, both of which the authors have addressed in their rebuttal. Given the novelty of the approach and its strong empirical performance, I recommend the paper for acceptance with a revision to address concerns raised through the reviewer discussions.
Decision: Accept (spotlight)