SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models

## Abstract

Abstract The rapid advancement in large language models (LLMs) comes with a significant increase in their parameter size, presenting challenges for adaptation and fine-tuning. Parameter-efficient fine-tuning (PEFT) methods are widely used to adapt LLMs for downstream tasks efficiently. In this paper, we propose Singular Values and Orthonormal Regularized Singular Vectors Adaptation, or SORSA, a novel PEFT method. We introduce a method to analyze the variation of the parameters by performing singular value decomposition (SVD) and discuss and analyze SORSA’s superiority in minimizing the alteration in the SVD aspect. Each SORSA adapter consists of two main parts: trainable principal singular weights W p = U p ​ diag ​ ( S p ) ​ V p ⊤ W_{p}=U_{p}\text{diag}(S_{p})V^{\top}_{p} , and frozen residual weights W r = U r ​ diag ​ ( S r ) ​ V r ⊤ W_{r}=U_{r}\text{diag}(S_{r})V^{\top}_{r} . These parts are initialized by performing SVD on pre-trained weights. Moreover, we implement and analyze an orthonormal regularizer, which we prove could decrease the condition number of W p W_{p} and allows the optimization to be more efficient. SORSA adapters could be merged during inference, thus eliminating any inference latency. After all, SORSA shows a faster convergence than PiSSA and LoRA in our experiments. On the GSM-8K benchmark, Llama 2 7B adapted using SORSA achieved 56.03% accuracy, surpassing LoRA (42.30%), Full FT (49.05%), and PiSSA (53.07%). On the MATH benchmark, SORSA achieved 10.36% accuracy, outperforming LoRA (5.50%), Full FT (7.22%), and PiSSA (7.44%). We conclude that SORSA offers a new perspective on parameter-efficient fine-tuning, demonstrating remarkable performance.
The code is available at https://github.com/Gunale0926/SORSA .

## 1 Introduction

Pre-trained large language models (LLMs) show remarkable generalization abilities, allowing them to perform various kinds of natural language processing (NLP) tasks (Peng et al., ; Touvron et al., ; Dubey et al., ; Radford et al., ; OpenAI, ). For specific downstream tasks, full parameter fine-tuning, which continues training all parameters of LLMs on downstream data, is widely used.

However, as the number of parameters in LLMs rapidly increases, full parameter fine-tuning becomes increasingly inefficient. For example, the estimated VRAM requirement for fully fine-tuning Llama 2 7B using Float32 could approach approximately 100 GB, making it unlikely to fully fine-tune the model on a single GPU with current technology. Additionally, the VRAM requirement for fully fine-tuning Llama 2 70B using Float32 exceeds 1 TB (Touvron et al., ; Anthony et al., ), thus rendering it unfeasible on a single GPU with current technology.

To address these challenges, several parameter-efficient fine-tuning (PEFT) methods (Houlsby et al., ; Lester et al., ; Hu et al., ) have been proposed. These methods enable the training of only a limited number of parameters, which significantly reduces VRAM requirements while achieving comparable or even superior performance to full fine-tuning. For instance, tuning Llama 2 7B in Float32 by LoRA (Hu et al., ) with a rank of 128 only takes approximately 60GB VRAM, which allows training on 1 $\times$ NVIDIA A100 (80GB), or even 3 $\times$ NVIDIA RTX 4090 (24GB).

Among those PEFT methods, LoRA (Hu et al., ) and its variants (Zhang et al., ; Meng et al., ; Liu et al., ; Dettmers et al., ) had become increasingly popular due to their: 1. Low training VRAM requirement 2. No inference latency 3. Versatility in different neuron network architectures.

In this paper, we propose a novel PEFT approach, Singular Values and Orthonormal Regularized Singular Vectors Adaptation, or SORSA. A SORSA adapter has two main parts: principal singular weights $W_{p}=U_{p}\text{diag}(S_{p})V^{\top}_{p}$, and residual weights $W_{r}=U_{r}\text{diag}(S_{r})V^{\top}_{r}$. These two parts are initialized by performing singular value decomposition (SVD) on pre-trained weight. Residual singular values and vectors will be merged into one matrix and frozen while training. We only train principal singular values and vectors with an orthonormal regularizer implemented to keep the orthonormality of $U_{p}$ and $V_{p}^{\top}$. The architecture of a SORSA adapter is illustrated in Figure 1.

Furthermore, we analyze the pattern of variation of singular values and vectors during parameter updating and discuss the different patterns of partial fine-tuning (FT), LoRA, SORSA without regularizer, and SORSA with regularizer concerning singular values and vectors’ updating.

We also provide a comprehensive gradient analysis that provides a mathematical foundation for SORSA. This analysis demonstrates several crucial properties of our method, including the convexity of the regularizer, Lipschitz continuity of the gradient, bounds on the hyperparameter $\gamma$. Moreover, we prove that SORSA improves the condition number of the optimization problem compared to unregularized approaches.

SORSA retains all the benefits of LoRA and its variants while demonstrating remarkable performance compared to PiSSA, LoRA, and full parameter fine-tuning in our experiments.

## 2 Related Works

Parameter-efficient fine-tuning (PEFT) methods have been developed to address the inefficiency of full parameter fine-tuning for large language models. These methods focus on adapting the model for downstream tasks while updating only a few parameters and keeping most of the model’s weights frozen. This approach significantly reduces the memory and computational requirements during training, especially VRAM.

Adapter-based PEFT is the first type of PEFT, which was initially designed by Houlsby et al. (). It introduces additional trainable non-linear blocks into the frozen pre-trained model, which could effectively tune the pre-trained model with a limited amount of trainable parameters. Its variants, e.g., Lin et al. (), reduce the number of adapter layers per block, and He et al. () focus on adding adapter modules parallel to existing layers. However, all adapter-based PEFT methods introduce inference latency due to their non-mergeable attribute.

Prompt-based PEFT is also a well-known type of PEFT, which was first proposed in Lester et al. (). There are several variants of this work, including Liu et al. (); Razdaibiedina et al. (). However, they have some inevitable shortcomings, such as potential performance limitations compared to full parameter fine-tuned models, additional inference latency due to expanding the length of the total input to the model, and the complexity of designing effective initialization.

LoRA (Hu et al., ) and its variants are the most popular type of PEFT methods. This type of PEFT is popular for its on-par or better performance than full parameter fine-tuning without introducing any inference latency. LoRA could be represented by equation $W=W_{0}+BA$, where $W_{0}\in\mathbb{R}^{m\times n}$ is the pre-trained weight, $A\in\mathbb{R}^{m\times r}$, using Gaussian initialization, and $B\in\mathbb{R}^{r\times n}$, using zero initialization, are low-rank matrices. Its variant, for example, AdaLoRA (Zhang et al., ) introduces an SVD decomposition and pruning for least significant singular values for more efficient parameter updating. DoRA (Liu et al., ) proposed a novel way to decompose weight into direction and magnitude by $W=\underline{m}\frac{W_{0}+\underline{BA}}{\|W_{0}+\underline{BA}\|_{c}}$, where $\underline{m}$ is initialized by $\underline{m}=\|W_{0}+\underline{BA}\|_{c}$, $\|\cdot\|_{c}$ denotes column-wise norm. The results show that DoRA has a better learning capacity than LoRA. However, DoRA introduced a calculation of norm in every training step, which makes it much more inefficient in training than LoRA. OLoRA (Büyükakyüz, ) uses QR decomposition to initialize the LoRA adapters $A$ and $B$, which initializes $B$ as an orthogonal matrix. They discusses the significance of orthonormality in neural networks’ weight (See Section 5 for more details). In their experiments, OLoRA demonstrates faster convergence than LoRA. PiSSA (Meng et al., ) decomposes pre-trained weight $W_{0}=U\text{diag}(S)V^{\top}$ by Singular Value Decomposition (SVD), and then splits $W_{0}$ into $W_{pri}$ and $W_{res}$: $W_{pri}=AB$ which is trainable, where $A=U_{[:r]}\text{diag}(S^{\frac{1}{2}}_{[:r]})$ and $B=\text{diag}(S^{\frac{1}{2}}_{[:r]})V^{\top}_{[:r]}$; $W_{res}=U_{[:r,:r]}\text{diag}(S_{[:r]})V^{\top}_{[:r,:]}$ which is frozen. PiSSA results a faster convergence speed and better fitting compared to LoRA. SORSA has a similar architecture to PiSSA, which both conducts SVD and replaces pre-trained weights with residual singular weights. SORSA inherits LoRA and its variants’ benefits, including low training VRAM requirement, no inference burden, and versatility in different architectures.

Other methods. There are also a few efficient adapting methods with unique techniques. For example, GaLore (Zhao et al., ) is a memory-efficient PEFT method that reduces VRAM usage by leveraging gradient accumulation and low-rank approximation. Despite its efficiency, GaLore’s complex implementation, involving perform SVD on gradients, adds computational complexity and may face scalability issues for very large models. LISA (Pan et al., ) uses a layer-wise importance sampling approach, prioritizing layers that significantly impact model performance, and selectively fine-tune essential parameters.

## 3 Singular Values and Vectors Analysis

### 3.1 Singular Value Decomposition

The geometric meaning of SVD can be summarized as follows: for every linear mapping $W:K^{m}\to K^{n}$, the singular value decomposition finds a set of orthonormal bases in both the original space and the image space such that $W$ maps the $i$-th basis vector of $K^{m}$ to a non-negative multiple of the $i$-th basis vector of $K^{n}$, and maps the remaining basis vectors of $K^{m}$ to the zero vector. In other words, the matrix $W$ can be represented as $\Sigma$ on these selected bases, where $\Sigma$ is a diagonal matrix with all non-negative diagonal elements.

According to the meaning of SVD, giving a matrix $W\in\mathbb{R}^{m\times n}$, let $k=\min(m,n)$, we could perform SVD to decompose $W$ by $W=U\text{diag}(S)V^{\top}$. Here, $U\in\mathbb{R}^{m\times k}$ is matrix of left singular vectors and have orthonormal columns, $V\in\mathbb{R}^{n\times k}$ is matrix of right singular vectors and have orthonormal columns, and $S\in\mathbb{R}^{k}$ are singular values $\sigma^{1},\sigma^{2}\ldots\sigma^{k}$ arranged in descending order. $\text{diag}(W)$ denotes a function to form a diagonal matrix from $W$.

### 3.2 Analysis Method

The study of DoRA (Liu et al., ) introduces an analysis method that focuses on the deviation of magnitude and direction ($\Delta M,\Delta D$) during training of full parameter fine-tuning and LoRA (Hu et al., ). They discovered that the distinction between full parameter fine-tuning and LoRA likely affects their learning ability difference. Inspired by their methods, we propose a novel method that analyzes the correlation between the deviation of singular values ($\Delta\Sigma$) and singular vectors ($\Delta D$) from pre-trained matrices during updating. Our analysis suggests a significant difference in singular values and vectors’ stability and an updating pattern of partial fine-tuning, LoRA, and SORSA.

The singular value and vector variations between pre-trained weight $W_{0}\in\mathbb{R}^{m\times n}$ and tuned weight $W_{t}\in\mathbb{R}^{m\times n}$, which $t$ denotes the training step, could be defined as follows:

$$ $\displaystyle\Delta\Sigma_{t}=\frac{\Sigma_{i=1}^{k}\left|\sigma_{t}^{i}-\sigma_{0}^{i}\right|}{k}$ (1) $$

Here, $\Delta\Sigma_{t}$ represents singular value variants between $W_{0}$ and $W_{t}$ at training step $t$. $\sigma^{i}_{t}$ denotes the $i$-th element in diagonal of $\Sigma_{t}$, where $\Sigma_{t}$ is decomposed from $W_{t}$ by performing SVD, $k=\min(m,n)$;

$$ $\displaystyle\Delta U_{t,j}=\left|\langle{\mathbf{u}}_{t}^{j},{\mathbf{u}}_{0}^{j}\rangle\right|$ (2) $$

$$ $\displaystyle\Delta V^{\top}_{t,i}=\left|\langle{\mathbf{v}}_{t}^{i},{\mathbf{v}}_{0}^{i}\rangle\right|$ (3) $$

$$ $\displaystyle\Delta D_{t}=1-\frac{1}{2k}\sum_{i=0}^{k}(\Delta U_{t,i}+\Delta V^{\top}_{t,i})$ (4) $$

Here, $k=\min(m,n)$; ${\mathbf{u}}_{t}^{j}$ denotes the $j$-th column vector of matrix $U_{t}$, and ${\mathbf{v}}_{t}^{i}$ denotes the $i$-th row vector of matrix $V_{t}^{\top}$; $\Delta D_{t}\in(0,1)$ represents variation of singular vectors between $W_{0}$ and $W_{t}$ at training step $t$; $U_{t}$ and $V_{t}^{\top}$ are decomposed from $W_{t}$ by performing SVD.

We adopt the analysis on Llama 2 7B (Touvron et al., ) using the first 100K data of MetaMathQA (Yu et al., ). We test partial fine-tuning, LoRA, and SORSA (with regularizer and without regularizer). See Section B.1 for training details of the analysis.

### 3.3 Analysis Result

This section analyzes the results of different training methods: partial fine-tuning, LoRA, and SORSA based on the data we collect. The analysis data is illustrated in Figure 2.

Figure: Figure 2: $\Delta D$ and $\Delta\Sigma$ of each trainable parameters during training steps. Numbers in the plot represent layer of the weight. Dots represent mean $\Delta D$ and $\Delta\Sigma$ at specific step.
Refer to caption: x2.png

The partial fine-tuning and LoRA methods analysis show that both methods exhibit significant adjustments in significant vectors $\Delta D$. This substantial alteration disrupts the characteristics of the pre-trained matrix and is likely to affect the model’s generalization ability. Moreover, the updates of all parameters in both partial fine-tuning and LoRA methods show a parallel updating pattern across weights in different layers, which emphasizes a restriction of these methods and may lead to a potentially disruptive modification of the generalization ability.

SORSA with regularizer shows $\Delta D$ and $\Delta\Sigma$ are in a much smaller range than other analysis methods. Moreover, different matrices demonstrated unrelated updating patterns, compared to all parallel-like other three methods, indicating that the updates in the SORSA are less constrained and potentially converge faster. In contrast, SORSA without the orthonormal regularizer exhibits a greater $\Delta D$ and $\Delta\Sigma$ alteration at each training step compared to SORSA with the regularizer. SORSA without regularizer also shows a linear-like updating pattern, similar to LoRA and partial FT. The result verifies the importance of regularizer in maintaining stability and minimizing the deviation from pre-trained matrices during the training process. These updating patterns indicate that SORSA maintains the characteristics of the pre-trained matrix better, thus potentially preserving the model’s generalization ability more effectively. This property allows SORSA adapters to be trained with higher learning rates without showing noticeable over-fitting compared to other tuning methods.

## 4 SORSA: Singular Value and Orthonormal Regularized Singular Vector Adaptation

According to the definition of SVD in Section 3.1, given a rank $r$ where $r\ll k$, we could perform the low-rank approximation by selecting the first $r$ items on the diagonal of $\Sigma$, which is the first $r$ most significant singular values, and also select the first $r$ columns of $U$ and first $r$ rows of $V^{\top}$, which correspond to the selected singular values. By performing SVD low-rank approximation, we could get a low-rank matrix that preserves the largest significant values and vectors, containing the matrix’s most significant data.

Therefore, for a pre-trained weight $W_{0}\in\mathbb{R}^{m\times n}$, we could split it based on its singular value into principal weight $W_{p}$ and residual weight $W_{r}$, where $W_{p}$ contains the most important part of information of the matrix, and $W_{r}$ contains the least significant part:

$$ $\displaystyle W_{p}=U_{[:,:r]}\text{diag}(S_{[:r]})V^{\top}_{[:r,:]}\in\mathbb{R}^{m\times n}$ (5) $$

$$ $\displaystyle W_{r}=U_{[:,r:]}\text{diag}(S_{[r:]})V^{\top}_{[r:,:]}\in\mathbb{R}^{m\times n}$ (6) $$

Here, $U$ represents the matrix of left singular vectors, $S$ represents the singular values, $\text{diag}(W)$ denotes a function to form a diagonal matrix from $W$, and $V$ represents the matrix of right singular vectors. We use PyTorch (Paszke et al., ) syntax to demonstrate matrix selection, where $[:,:r]$ denotes selecting the first $r$ columns of the matrix, and $[r:,:]$ denotes selecting the last $r$ rows of the matrix. We rewrite $U_{[:,:r]}$, $S_{[:r]}$ and $V^{\top}_{[:r,:]}$, which consist $W_{p}$ as $U_{p}$, $S_{p}$ and $V^{\top}_{p}$ for simplicity, and rewrite $U_{[:,r:]}$, $S_{[:r]}$ and $V^{\top}_{[r:,:]}$, which consist $W_{r}$ as $U_{r}$, $S_{r}$ and $V^{\top}_{r}$ correspondingly.

The initialization of $W_{r}$ in SORSA is the same as PiSSA (Meng et al., ). Nevertheless, unlike PiSSA which merge $\text{diag}(S_{p})$ with $U_{p}$ and $V^{\top}_{p}$ into $A$ and $B$ by $A=U_{p}\text{diag}(S_{p})^{\frac{1}{2}}$ and $B=\text{diag}(S_{p})^{\frac{1}{2}}V^{\top}_{p}$, SORSA remains $U_{p}$, $S_{p}$, and $V^{\top}_{p}$ in separate matrices. SORSA is defined by Equation 7, which is initially equivalent to the pre-trained weight $W_{0}$.

During training, $W_{r}$ remains frozen, and only $U_{p}$, $S_{p}$, and $V^{\top}_{p}$ are updated.

SORSA is defined as:

$$ $\displaystyle\text{SORSA}(x):=x(W_{r}+W_{p})=xW_{r}+xU_{p}\text{diag}(S_{p})V^{\top}_{p}$ (7) $$

In our implementation, we use a optimized version of SORSA equation which results a much faster computation speed, and could approach to the speed of LoRA. See Appendix A for more details.

We adopt an orthonormal regularizer similar to (Zhang et al., ) for $U_{p}$ and $V^{\top}_{p}$. We verify its importance and effectiveness in Section 5.

$$ $\displaystyle{\mathcal{L}}_{reg}=\|U^{\top}_{p}U_{p}-I\|_{F}+\|V^{\top}_{p}V_{p}-I\|_{F}$ (8) $$

Where ${\mathcal{L}}_{reg}$ is the orthonormal regularizer loss, the $U_{p}$ and $V^{\top}_{p}$ are each orthonormal vectors in columns and rows, respectively, after initialization due to SVD’s property. The regularizer could keep their orthonormality during training.

Therefore, parameter updating of $W_{p}$ in a SORSA adapter at training step $t$ could be expressed as:

$$ $\displaystyle W_{p,t+1}=$ $\displaystyle W_{p,t}-\eta_{t}\nabla_{W_{p,t}}{\mathcal{L}}_{train}-\gamma_{t}\nabla_{W_{p,t}}{\mathcal{L}}_{reg}$ (9) $$

At training step $t$, $\nabla_{W_{p,t}}{\mathcal{L}}_{train}$ denotes the gradient of ${\mathcal{L}}_{train}$ respect to $W_{p,t}$, and $\nabla_{W_{p,t}}{\mathcal{L}}_{reg}$ denotes the gradient of ${\mathcal{L}}_{reg}$ respect to the orthonormal regularizer loss $W_{p,t}$. $\eta_{t}$ and $\gamma_{t}$ are the learning rate for training loss and regularizer loss at step $t$, respectively.

We update the SORSA as Equation 10 for implementation simplicity.

$$ $\displaystyle W_{p,t+1}=$ $\displaystyle W_{p,t}-\eta_{t}\left(\nabla_{W_{p,t}}{\mathcal{L}}_{train}+\frac{\gamma}{\eta_{d}}\nabla_{W_{p,t}}{\mathcal{L}}_{reg}\right)$ (10) $$

$\eta_{d}$ is the maximum learning rate which is given from the scheduler. This implementation allows us to use only one optimizer and scheduler to deal with two different learning rates separately.

## 5 Gradient Analysis

In this section, we present a comprehensive mathematical analysis of the SORSA method which mainly focus on the effect of orthonormal regularization. Our investigation elucidates the key optimization properties of SORSA, providing a theoretical foundation for its advantages. We explore four critical aspects: the convexity of the regularizer, the Lipschitz continuity of the gradient, bounds on the hyperparameter $\gamma$, and the impact on the condition number of the optimization problem.

The proofs of the theorems and lemmas, along with additional mathematical details, are provided in Appendix C.

Our analysis reveals fundamental theoretical properties of SORSA, establishing its mathematical soundness and demonstrating its optimization advantages. We prove two key theorems that form the cornerstone of our theoretical framework:

###### Theorem 1 .

The regularizer ${\mathcal{L}}_{reg}$ is convex.

###### Theorem 2 .

The gradient of the regularizer ${\mathcal{L}}_{reg}$ is Lipschitz continuous.

These properties collectively ensure a well-behaved optimization landscape, facilitating stable and efficient training of SORSA adapters.

Building upon these foundational results, we further analyze the bounds of the hyperparameter $\gamma$, a critical factor in the performance of SORSA:

###### Theorem 3 .

For convergence of gradient descent, the learning rate $\eta_{d}$ and regularization parameter $\gamma$ must satisfy:

$$ $\displaystyle 0\leq\gamma<\frac{2}{k\eta_{d}}-L_{train}$ (11) $$

This theorem provides crucial guidance for practitioners, offering a clear criterion for selecting appropriate values of $\gamma$ to ensure convergence of the gradient descent process.

To demonstrate the superior optimization properties of SORSA, we present a novel analysis of its condition number, a critical factor in determining convergence speed and stability. Our theoretical investigation reveals a significant improvement in the condition number compared to unregularized approaches, providing a mathematical foundation for SORSA’s enhanced performance.

We begin this analysis by establishing a key lemma that bounds the effect of the orthonormal regularizer on the singular values of the weight matrix:

###### Lemma 4 .

Let $W_{p}^{\text{unreg}}=U_{p}^{\text{unreg}}\text{diag}(S_{p})^{\text{unreg}}(V_{p}^{\text{unreg}})^{\top}$ be the $W_{p}$ only training without using regularizer, and $W_{p}^{\text{reg}}=U_{p}^{\text{reg}}\text{diag}(S_{p})^{\text{reg}}(V_{p}^{\text{reg}})^{\top}$ be the $W_{p}$ training with the regularizer. For each singular value $\sigma_{i}$, the following bound holds:

$$ $\displaystyle(1-\epsilon)\sigma_{i}^{\text{unreg}}\leq\sigma_{i}^{\text{reg}}\leq(1+\epsilon)\sigma_{i}^{\text{unreg}}$ (12) $$

where $\epsilon$ is a small positive constant.

Lemma 4 provides a crucial connection between the regularizer and the singular values. Building on this result, we arrive at our main theorem regarding the condition number:

###### Theorem 5 .

The orthonormal regularizer in SORSA can improve the condition number of the optimization problem over the course of training under certain conditions. Specifically:

At initialization ($t=0$):

$$ $\displaystyle\frac{\kappa(W_{p}^{\text{reg}})}{\kappa(W_{p}^{\text{unreg}})}=1$ (13) $$

When training ($t>0$), while $U_{p}^{\text{unreg}}$ and $(V_{p}^{\text{unreg}})^{\top}$ are more and more deviating from orthonormal, $\kappa(U_{p}^{\text{unreg}})\cdot\kappa((V_{p}^{\text{unreg}})^{\top})$ will increase, and $(1+\delta_{1,t})(1+\delta_{2,t})(\frac{1+\epsilon_{t}}{1-\epsilon_{t}})$ will generally remain $1$ due to the regularization in orthonormality of $U_{p}^{\text{reg}}$ and $(V_{p}^{\text{reg}})^{\top}$. When $\kappa(U_{p}^{\text{unreg}})\cdot\kappa((V_{p}^{\text{unreg}})^{\top})$ becomes greater than $(1+\delta_{1,t})(1+\delta_{2,t})(\frac{1+\epsilon_{t}}{1-\epsilon_{t}})$, the ratio becomes strictly less than 1, indicating an improvement in the condition number.

This theorem quantifies the improvement in the condition number achieved by SORSA, offering an explanation for its fast convergence. The proof leverages the effects of the orthonormal regularization to establish a tight bound on the condition number ratio. This theorem could also shows that training with the regularizer, the distribution of $W_{p}$ will be more evenly distributed due to a smaller ratio between $\sigma_{\max}(W_{p})$ and $\sigma_{\min}(W_{p})$, which means a better training stability.

Moreover, as mentioned in Büyükakyüz (), orthonormal matrices in neuron networks could improve gradient flow (Saxe et al., ; Arjovsky et al., ) and enhanced optimization landscape (Huang et al., ; Wisdom et al., ), which could also explain SORSA’s superior performance in convergence.

In conclusion, these theorems provide a mathematical foundation for the SORSA method. These theoretical guarantees validate the empirical success of SORSA but also provide valuable insights for future developments in PEFT methods.

###### Theorem 1 .

###### Theorem 2 .

###### Theorem 3 .

###### Lemma 4 .

###### Theorem 5 .

## 6 Empirical Experiments

We conduct comparative experiments on different NLP tasks, including natural language generation (NLG) between SORSA, PiSSA (Meng et al., ), LoRA (Hu et al., ), and full parameter fine-tuning.

We conducted NLG tests on Llama 2 7B (Touvron et al., ), RWKV6 7B (Peng et al., ), Mistral 7B v0.1 (Jiang et al., ) Gemma 7B (Gemma Team et al., ). We trained the models using first 100K data in MetaMathQA (Yu et al., ), and evaluated the model on GSM-8K (Cobbe et al., ) and MATH (Hendrycks et al., ). We also trained the model on first 100K data in CodeFeedback Filtered Instruction (Zheng et al., ) dataset and evaluated it on HumanEval (Chen et al., ). The training process followed identical setups as the experiments conducted in PiSSA (Meng et al., ). All reported values are accuracy. See  Section B.2 for more details and hyperparameters of the training. We quoted some results of PiSSA, LoRA, and full parameter fine-tuning from Meng et al. (). For experiments, we conducted part of our experiments on a single NVIDIA A100-SXM4 (80GB) GPU, others on a single NVIDIA H100-SXM4 (80GB) GPU. See Table 1 for the results and Figure 3 for the loss and gradient norm comparison.

**Table 1: Comparing SORSA with other methods on NLG tasks. ^† denotes results from Meng et al. ().**
| Model | Method | Trainable<br>Parameters | GSM-8K | MATH | HumanEval |
| --- | --- | --- | --- | --- | --- |
| Llama 2 7B | Full FT | 6738M | 49.05^† | 7.22^† | 21.34^† |
| LoRA | 320M | 42.30^† | 5.50^† | 18.29^† |  |
| PiSSA | 320M | 53.07^† | 7.44^† | 21.95^† |  |
| SORSA | 320M | 56.03 | 10.36 | 24.39 |  |
| RWKV6 7B | LoRA | 176M | 8.04 | 7.38 | 15.24 |
| PiSSA | 176M | 32.07 | 9.42 | 17.07 |  |
| SORSA | 176M | 45.87 | 11.32 | 22.56 |  |
| Mistral 7B | Full FT | 7242M | 67.02^† | 18.60^† | 45.12^† |
| LoRA | 168M | 67.70^† | 19.68^† | 43.90^† |  |
| PiSSA | 168M | 72.86^† | 21.54^† | 46.95^† |  |
| SORSA | 168M | 73.09 | 21.86 | 47.56 |  |
| Gemma 7B | Full FT | 8538M | 71.34^† | 22.74^† | 46.95^† |
| LoRA | 200M | 74.90^† | 31.28^† | 53.66^† |  |
| PiSSA | 200M | 77.94^† | 31.94 ^† | 54.27^† |  |
| SORSA | 200M | 78.09 | 29.52 | 55.49 |  |

Figure: Figure 3: The training loss and gradient norm comparison between SORSA, PiSSA, and LoRA on MetaMathQA training of RWKV6 7B and Llama 2 7B. LoRA and PiSSA curves of Llama 2 7B are from Meng et al. ().
Refer to caption: x6.png

The results showed that across all models tested, SORSA generally outperformed other methods. For mathematical evaluations, on Llama 2 7B, SORSA scored 56.03% on GSM-8K and 10.36% on MATH, significantly outperforming other methods; For the RWKV6 7B model, SORSA achieved 45.87% accuracy on GSM-8K and 11.32% on MATH, surpassing both PiSSA and LoRA; On Mistral 7B, SORSA reached 73.09% on GSM-8K and 21.86% on MATH, slightly edging out competitors; With Gemma 7B, while SORSA’s MATH score of 29.52% was lower than LoRA and PiSSA, it still led on GSM-8K with 78.09%. Moreover, on coding evaluations, SORSA consistently achieved the highest HumanEval scores across other tuning methods, demonstrating its effectiveness on diverse natural language tasks. Noticeably, RWKV6 7B using LoRA performed extremely below our expectation in GSM-8K test. We believe this may be due to our prompt for mathematical tests (See Section B.2 for more details) are inconsistent with the model’s original prompt, causing LoRA to fail in learning this prompting behavior.

The Figure 3 reveals that while SORSA and PiSSA exhibit nearly identical loss curves at the beginning, and even a little bit higher than PiSSA on RWKV-6 training. However, when training step approximately $t>300$, SORSA continues to maintain the decrease of its loss steadily. In contrast, LoRA and PiSSA shows a deceleration in its loss reduction. This supports Theorem 5, confirming that SORSA consistently outperforms both LoRA and PiSSA, especially at later stages of training.

However, due to the limitation of computing resources, we only trained and benchmarked a small number of tasks.

## 7 Conclusion

In this paper, we introduced SORSA, a novel parameter-efficient fine-tuning (PEFT) method designed to enhance the adaptation of large language models (LLMs) for downstream tasks. SORSA utilizes singular value decomposition (SVD) to split pre-trained weights into principal and residual components, only training the principal singular values and vectors while freezing the residuals. We implemented an orthonormal regularizer to maintain the orthonormality of singular vectors during training, ensuring efficient parameter updates and preserving the integrity of singular values.

Our experiments demonstrated that SORSA outperforms existing PEFT methods, such as LoRA and PiSSA, in both convergence speed and accuracy on the NLG tasks. Specifically, Llama 2 7B, tuned with SORSA, achieved significant improvements in the GSM-8K and MATH benchmarks, highlighting the effectiveness of our approach.

We adopted singular values and vectors analysis, comparing SORSA with partial FT and LoRA, and we find SORSA’s superiority in preserving the pre-trained weight’s singular values and vectors during training. This suggests an explanation for SORSA’s supreme performance demonstrated in the experiment. We also show the significance of the orthonormal regularizer through analysis.

Our gradient analysis provided a mathematical foundation for SORSA, demonstrating its convexity, Lipschitz continuity, and the crucial role of the regularizer in improving the optimization landscape. This theoretical framework not only explains SORSA’s empirical superior performance but also offers valuable insights for future developments in adaptive learning algorithms.

SORSA retains the advantages of LoRA and variants, including low training VRAM requirements, no inference latency, and versatility across different neural network architectures. By offering a more efficient fine-tuning mechanism, SORSA presents a promising direction for future research and application in the field of LLMs.

Overall, SORSA gives a new perspective on parameter-efficient fine-tuning, showcasing exceptional efficiency and robust performance. It not only outperforms existing methods like LoRA and PiSSA in several downstream tasks but also maintains the practical benefits of low VRAM requirements, no inference latency, and ease of implementation. This innovative approach offers a promising direction for future research and practical applications in adapting pre-trained models, making it a pivotal development in the field.

## 8 Future Work

While SORSA demonstrates substantial improvements over existing PEFT methods, there are several avenues for future work to enhance its capabilities further and extend its applicability:

- •
Comparison Between Different Rank: Find how the performance of SORSA varies with different ranks. This investigation would help determine the optimal rank for various tasks and model sizes, potentially revealing a trade-off between computational efficiency and model performance.
- •
Extended Evaluation: Conduct broader evaluations of SORSA across different areas of benchmarks on models with more diverse sizes. This may able to provide a more comprehensive understanding of SORSA’s performance and versatility.
- •
Catastrophic Forgetting: Compare the performance of some general benchmarks like MMLU (Hendrycks et al., ) before and after models are adapted by SORSA in specific down-stream tasks in order to research whether SORSA could perform better than full-parameter fine-tuning, LoRA (Hu et al., ) or PiSSA (Meng et al., ) on preventing catastrophic forgetting.
- •
Quantization: Investigate the application of quantization techniques to SORSA similar to the approach of QLoRA (Dettmers et al., ) and QPiSSA (Meng et al., ). Quantization aims to reduce the memory footprint and computational requirements, making it feasible to deploy on devices with limited computational resources. By combining the benefits of quantization with the efficient fine-tuning capabilities of SORSA, quantization of SORSA may able to efficiently provides powerful models, which could eventually facilitate AI for a broader range of practical applications.

By pursuing these directions, future work can build on the foundation laid by SORSA, pushing the boundaries of PEFT and enhancing the adaptability and performance of large language models across a wide range of applications. This will enable the development of more versatile downstream models, potentially expanding the impact of Machine Learning into more fields and becoming an integral part of people’s everyday lives.

## 9 Impact Statement

In this paper, we introduced an innovative PEFT method in the area of Machine Learning. Our approach significantly streamlined the model’s tuning process, particularly for large-scale models, addressing both computational efficiency and environmental sustainability. As we push the boundaries of what is possible with Machine Learning, it is essential to consider the broader impacts of these advancements on both the environment and ethical standards within the field.

### 9.1 Environmental Impact

SORSA optimizes model tuning for large-scale models, reducing VRAM consumption by over 50% when adapting the Llama 3 70B Dubey et al. () model than full parameter fine-tuning. This significant reduction in hardware resource requirements also suggests a less energy consumption compared to full parameter fine-tuning methods. By enhancing efficiency, our approach could significantly contributes to reducing the carbon footprint of Machine Learning operations.

### 9.2 Ethical Concerns

The PEFT method, while efficient, raises critical ethical concerns regarding the security of built-in safety measures in AI models. As demonstrated in Lermen & Rogers-Smith (), subversive fine-tuning techniques can bypass safety training intended to prevent the generation of harmful content. The ease and affordability of such methods underscore the vulnerability of safety protocols. It is imperative to develop robust safeguards that keep pace with technological advancements, ensuring that efficiency gains in model tuning do not compromise the ethical use of AI.

## 10 Acknowledgment

We would like to extend our gratitude to Prof. Jianyong Wang from Tsinghua University and the Computer Science Department of the Advanced Project Research Laboratory of Tsinghua University High School for providing the endorsement to the research.

## Appendix A Faster SORSA Adapters

According to the definition of SORSA from Equation 7, because $\text{diag}(S_{p})$ is always a diagonal matrix, it is equivalent to:

$$ $\displaystyle\text{SORSA}(x)=xW_{r}+x(U_{p}\odot S_{p})V_{p}^{\top}$ (14) $$

Where $\odot$ denotes element-wise multiplication, and $\text{diag}(W)$ denotes the diagonal elements of the matrix $W$.

This transformation allows us to reduce the computational complexity of SORSA adapters. In the original form, we have to perform twice matrix multiplications. However, in the Equation 14, we only have one matrix multiplication, and one element-wise multiplication. The time complexity of element-wise multiplication is $O(m^{2})$, which is much less than matrix multiplication’s complexity $O(m^{3})$. Therefore, while $\lim_{m\to\infty}$, the computation speed of SORSA adapters will be same as LoRA and PiSSA.

We performed a benchmark using PyTorch (Paszke et al., ) on a NVIDIA H100 SXM4 (80GB) GPU backed with CUDA, and Apple M2Pro CPU, in order to test the computation time between these two methods. See Figure 4 to see our results.

Figure: Figure 4: Benchmark between two equations of SORSA
Refer to caption: x10.png

## Appendix B Experiments Details

### B.1 Analysis

For the singular values and vectors analysis in Section 3, we applied partial fine-tuning, LoRA and SORSA (with and without orthonormal regularizer) on Llama 2 7B (Touvron et al., ) model, training with the first 100K data in MetaMathQA (Yu et al., ) dataset. We only calculated the loss on the response part. The models are trained with TF32 & BF16 (Wang & Kanwar, ) mix precision. See Table 2 for hyperparameters.

We used AdamW (Loshchilov & Hutter, ) optimizer and cosine annealing scheduler in training. In the analysis, LoRA and SORSA were only applied to q_proj and v_proj matrices, respectively. For Partial FT, we set model’s q_proj and v_proj matrices to trainable.

We also found we should only perform SVD for analysis using CPU, in order to get the precise analysis data.

**Table 2: Hyperparameters for the analysis**
| Model | Llama 2 7B |  |  |  |
| --- | --- | --- | --- | --- |
| Method | Partial FT | LoRA | SORSA<br>(w/o reg) | SORSA |
| Training |  |  |  |  |
| Mix-Precision | TF32&BF16 | TF32&BF16 | TF32&BF16 | TF32&BF16 |
| Epoch | 1 | 1 | 1 | 1 |
| Batch Size | 128 | 128 | 128 | 128 |
| Max Length | 512 | 512 | 512 | 512 |
| Weight Decay | 0 | 0 | 0 | 0 |
| Warm-up Ratio | 0.03 | 0.03 | 0.03 | 0.03 |
| Learning Rate | 2e-5 | 2e-5 | 2e-5 | 3e-5 |
| Grad Clip | False | False | False | False |
| SORSA $\gamma$ | N/A | N/A | 0 | 5e-4 |
| Rank | N/A | 128 | 128 | 128 |

### B.2 NLG Experiments

For our NLG tasks, we adapted Llama 2 7B (Touvron et al., ), RWKV6 7B (Peng et al., ), Mistral 7B v0.1 (Jiang et al., ) Gemma 7B (Gemma Team et al., ). models by SORSA. For GSM-8K (Cobbe et al., ) and MATH (Hendrycks et al., ) evaluations, we trained those models with the first 100K data in MetaMathQA (Yu et al., ) dataset. For HumanEval (Chen et al., ) evaluation, we use the first 100K data in CodeFeedback Filtered Instruction(Zheng et al., ) dataset.

We used AdamW (Loshchilov & Hutter, ) optimizer and cosine annealing scheduler in training. SORSA adapters were applied on all linear matrices in every layer. We only calculated the loss on the response part. The models are loaded in FP32, and trained with TF32 & BF16 mix precision. In our experiments, we selected a higher learning rate for SORSA compared to other methods to counterbalance the negative effect of orthonormal regularizer on optimizing toward lower training loss. See Table 3 for hyperparameters. See LABEL:lst:math for the prompt we used in GSM-8K and MATH evaluations, and LABEL:lst:humaneval for the prompt we used for HumanEval tests.

**Table 3: Hyperparameters of training on models for GSM-8K and MATH**
| Model | Llama 2 7B | Mistral 7B | Gemma 7B | RWKV6 7B | RWKV6 7B |
| --- | --- | --- | --- | --- | --- |
| Method | SORSA | SORSA | SORSA | SORSA | LoRA<br>PiSSA |
| Training |  |  |  |  |  |
| Mix-Precision | TF32&BF16 | TF32&BF16 | TF32&BF16 | TF32&BF16 | TF32&BF16 |
| Epoch | 1 | 1 | 1 | 1 | 1 |
| Batch Size | 128 | 128 | 128 | 128 | 128 |
| Max Length | 512 | 512 | 512 | 512 | 512 |
| Weight Decay | 0 | 0 | 0 | 0 | 0 |
| Warm-up Ratio | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 |
| Learning Rate | 3e-5 | 3e-5 | 3e-5 | 3e-5 | 2e-5 |
| Grad Clip | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| SORSA $\gamma$ | 4e-4 | 4e-4 | 4e-4 | 4e-4 | N/A |
| Rank | 128 | 64 | 64 | 64 | 64 |
| Evaluating |  |  |  |  |  |
| Precision | BF16 | BF16 | BF16 | FP32 | FP32 |
| Sampling | False |  |  |  |  |
| Top-P | 1.0 |  |  |  |  |
| Max Length | GSM-8K: 1024<br>MATH: 2048<br>HumanEval: 2048 |  |  |  |  |

Figure: Listing 1: Prompt used for GSM-8K and MATH.

Figure: Listing 2: Prompt used for HumanEval evaluation.

## Appendix C Proofs

See 1

###### Proof.

We prove this in two steps:

First, we show that $f(U_{p})=\|U^{\top}_{p}U_{p}-I\|_{F}$ is convex. Then, we prove that $g(V_{p})=\|V_{p}V^{\top}_{p}-I\|_{F}$ is convex.

Since the sum of convex functions is convex, this will establish the convexity of ${\mathcal{L}}_{reg}$.

Let $U_{p},W\in\mathbb{R}^{m\times r}$. The Hessian of $f$ at $U_{p}$ in the direction $W$ is given by:

$$ $\displaystyle\begin{split}\nabla^{2}f(U_{p})[W,W]&=\lim_{\epsilon\to 0}\frac{1}{\epsilon^{2}}\Big(f(U_{p}+\epsilon W)-2f(U_{p})+f(U_{p}-\epsilon W)\Big)\\ &=\lim_{\epsilon\to 0}\frac{1}{\epsilon^{2}}\Big(\|(U_{p}+\epsilon W)^{\top}(U_{p}+\epsilon W)-I\|_{F}\\ &\quad-2\|U^{\top}_{p}U_{p}-I\|_{F}+\|(U_{p}-\epsilon W)^{\top}(U_{p}-\epsilon W)-I\|_{F}\Big)\\ &=\lim_{\epsilon\to 0}\frac{1}{\epsilon^{2}}\Big(\|U^{\top}_{p}U_{p}+\epsilon(U^{\top}_{p}W+W^{\top}U_{p})+\epsilon^{2}W^{\top}W-I\|_{F}\\ &\quad-2\|U^{\top}_{p}U_{p}-I\|_{F}+\|U^{\top}_{p}U_{p}-\epsilon(U^{\top}_{p}W+W^{\top}U_{p})+\epsilon^{2}W^{\top}W-I\|_{F}\Big)\\ &=2\|W^{\top}W\|_{F}\end{split}$ (15) $$

Since $\|W^{\top}W\|_{F}\geq 0$ for all $W$, we have $\nabla^{2}f(U_{p})[W,W]\geq 0$, which proves that $f$ is convex.

The proof for $g(V_{p})$ follows the same steps as for $f(U_{p})$, leading to the same conclusion.

Therefore, both $f(U_{p})$ and $g(V_{p})$ are convex, and consequently, ${\mathcal{L}}_{reg}$ is convex.
∎

See 2

###### Proof.

To prove Lipschitz continuity, we need to show that there exists a constant $L>0$ such that for any two pairs of matrices $(U_{1},V_{1})$ and $(U_{2},V_{2})$:

$$ $\displaystyle|{\mathcal{L}}_{reg}(U_{1},V_{1})-{\mathcal{L}}_{reg}(U_{2},V_{2})|\leq L(\|U_{1}-U_{2}\|_{F}+\|V_{1}-V_{2}\|_{F})$ (16) $$

Let’s break this down into two parts:

First, consider $|\|U_{1}^{\top}U_{1}-I\|_{F}-\|U_{2}^{\top}U_{2}-I\|_{F}|$:

$$ $\displaystyle\begin{split}|\|U_{1}^{\top}U_{1}-I\|_{F}-\|U_{2}^{\top}U_{2}-I\|_{F}|&\leq\|(U_{1}^{\top}U_{1}-I)-(U_{2}^{\top}U_{2}-I)\|_{F}\\ &=\|U_{1}^{\top}U_{1}-U_{2}^{\top}U_{2}\|_{F}\\ &=\|U_{1}^{\top}U_{1}-U_{1}^{\top}U_{2}+U_{1}^{\top}U_{2}-U_{2}^{\top}U_{2}\|_{F}\\ &\leq\|U_{1}^{\top}(U_{1}-U_{2})\|_{F}+\|(U_{1}^{\top}-U_{2}^{\top})U_{2}\|_{F}\\ &\leq\|U_{1}^{\top}\|_{F}\|U_{1}-U_{2}\|_{F}+\|U_{1}-U_{2}\|_{F}\|U_{2}\|_{F}\\ &\leq(\|U_{1}\|_{F}+\|U_{2}\|_{F})\|U_{1}-U_{2}\|_{F}\end{split}$ (17) $$

Here, we’ve used the triangle inequality and the sub-multiplicative property of the Frobenius norm.

Similarly for $V$:

$$ $\displaystyle|\|V_{1}^{\top}V_{1}-I\|_{F}-\|V_{2}^{\top}V_{2}-I\|_{F}|\leq(\|V_{1}\|_{F}+\|V_{2}\|_{F})\|V_{1}-V_{2}\|_{F}$ (18) $$

Combining these results:

$$ $\displaystyle\begin{split}|{\mathcal{L}}_{reg}(U_{1},V_{1})-{\mathcal{L}}_{reg}(U_{2},V_{2})|&\leq(\|U_{1}\|_{F}+\|U_{2}\|_{F})\|U_{1}-U_{2}\|_{F}\\ &\quad+(\|V_{1}\|_{F}+\|V_{2}\|_{F})\|V_{1}-V_{2}\|_{F}\\ &\leq\max(\|U_{1}\|_{F}+\|U_{2}\|_{F},\|V_{1}\|_{F}+\|V_{2}\|_{F})\\ &\quad\cdot(\|U_{1}-U_{2}\|_{F}+\|V_{1}-V_{2}\|_{F})\end{split}$ (19) $$

Let $L_{reg}=\max(\|U_{1}\|_{F}+\|U_{2}\|_{F},\|V_{1}\|_{F}+\|V_{2}\|_{F})$. This $L$ is finite because the Frobenius norms of $U$ and $V$ are bounded (they represent orthonormal matrices in the ideal case).

Therefore, we have shown that:

$$ $\displaystyle|{\mathcal{L}}_{reg}(U_{1},V_{1})-{\mathcal{L}}_{reg}(U_{2},V_{2})|\leq L_{reg}(\|U_{1}-U_{2}\|_{F}+\|V_{1}-V_{2}\|_{F})$ (20) $$

This proves that ${\mathcal{L}}_{reg}$ is Lipschitz continuous with Lipschitz constant $L_{reg}$.
∎

See 3

###### Proof.

For gradient descent to converge while the $\eta_{t}=\eta_{d}$, we need:

$$ $\displaystyle\eta_{d}<\frac{2}{L}=\frac{2}{L_{train}+\frac{\gamma}{\eta_{d}}L_{reg}}$ (21) $$

Rearranging this inequality:

$$ $\displaystyle\begin{split}\eta_{d}(L_{train}+\frac{\gamma}{\eta_{d}}L_{reg})&<2\\ \eta_{d}L_{train}+\gamma L_{reg}&<2\\ \gamma L_{reg}&<2-\eta_{d}L_{train}\\ \gamma&<\frac{2-\eta_{d}L_{train}}{L_{reg}}\end{split}$ (22) $$

We can assume that the regularizer’s gradients scale with $\eta_{d}$ , meaning that larger updating step (due to a larger $\eta_{d}$) will lead to greater deviations from orthonormality, which increases $L_{reg}$. Conversely, smaller steps lead to a more gradual progression towards orthonormality, which reduces $L_{reg}$. Therefore, we could assume $L_{reg}\propto\eta_{d}$. Moreover, the $\gamma$ must not be negative or the regularization term would impact negatively on the purposes it is supposed to have. Therefore, we can rewrite the inequality as:

$$ $\displaystyle 0\leq\gamma<\frac{2}{k\eta_{d}}-L_{train}$ (23) $$

This establishes the upper bound for $\gamma$.
∎

See 4

###### Proof.

First, let’s consider the effect of the orthonormal regularizer. The regularizer aims to make $U_{p}^{\top}U_{p}\approx I$ and $V_{p}V_{p}^{\top}\approx I$. We can quantify this approximation as:

$$ $\displaystyle\|\nabla_{W_{p}}{\mathcal{L}}_{reg}\|_{F}\leq\epsilon_{\nabla}$ (24) $$

where $\epsilon_{\nabla}>0$ is a small constant.

Now, let’s consider the difference between $W_{p}^{\text{unreg}}$ and $W_{p}^{\text{reg}}$:

$$ $\displaystyle\|W_{p}^{\text{reg}}-W_{p}^{\text{unreg}}\|_{F}=\gamma\|\nabla_{W_{p}}{\mathcal{L}}_{reg}\|_{F}=\gamma\epsilon_{\nabla}$ (25) $$

Now, we can use Weyl’s inequality (Weyl, ), which states that for matrices A and B:

$$ $\displaystyle|\sigma_{i}(A+B)-\sigma_{i}(A)|\leq\|B\|_{2}\leq\|B\|_{F}$ (26) $$

Applying this to our case, with $A=W_{p}^{\text{unreg}}$ and $B=W_{p}^{\text{reg}}-W_{p}^{\text{unreg}}$:

$$ $\displaystyle|\sigma_{i}^{\text{reg}}-\sigma_{i}^{\text{unreg}}|\leq\|W_{p}^{\text{reg}}-W_{p}^{\text{unreg}}\|_{F}\leq\gamma\epsilon_{\nabla}$ (27) $$

Let $\epsilon=\gamma\epsilon_{\nabla}$. Then we have:

$$ $\displaystyle-\epsilon\leq\sigma_{i}^{\text{reg}}-\sigma_{i}^{\text{unreg}}\leq\epsilon$ (28) $$

Rearranging this inequality gives us our desired bound:

$$ $\displaystyle(1-\epsilon)\sigma_{i}^{\text{unreg}}\leq\sigma_{i}^{\text{reg}}\leq(1+\epsilon)\sigma_{i}^{\text{unreg}}$ (29) $$

∎

See 5

###### Proof.

Let $W_{p}=U_{p}\text{diag}(S_{p})V_{p}^{\top}$ be the principal part of the singular value decomposition approximation of $W$. The condition number is given by:

$$ $\displaystyle\kappa(W_{p})=\frac{\sigma_{\max}(W_{p})}{\sigma_{\min}(W_{p})}$ (30) $$

where $\sigma_{\max}$ and $\sigma_{\min}$ are the maximum and minimum singular values of $W_{p}$.

At initialization ($t=0$):
Due to SVD initialization, $U_{p}$ and $V_{p}^{\top}$ are perfectly orthonormal, so:

$$ $\displaystyle\kappa(U_{p}^{\text{unreg}})=\kappa((V_{p}^{\text{unreg}})^{\top})=\kappa(U_{p}^{\text{reg}})=\kappa((V_{p}^{\text{reg}})^{\top})=1$ (31) $$

And $\epsilon_{0}=0$, $\delta_{1,0}=\delta_{2,0}=0$. Therefore:

$$ $\displaystyle\frac{\kappa(W_{p}^{\text{reg}})}{\kappa(W_{p}^{\text{unreg}})}=\frac{\kappa(\text{diag}(S_{p})^{\text{reg}})}{\kappa(\text{diag}(S_{p})^{\text{unreg}})}=1$ (32) $$

During training ($t>0$):
As training progresses, $U_{p}$ and $V_{p}^{\top}$ deviate from orthonormality in the unregularized case. We quantify this deviation:

$$ $\displaystyle\|U_{p}^{\top}U_{p}-I\|_{F}\leq\epsilon_{1,t}$ (33) $\displaystyle\|V_{p}^{\top}V_{p}-I\|_{F}\leq\epsilon_{2,t}$ (34) $$

where $\epsilon_{1,t},\epsilon_{2,t}>0$ increase over time t.

For the regularized matrices, we can bound their condition numbers:

$$ $\displaystyle\kappa(U_{p}^{\text{reg}})\leq 1+\delta_{1,t}$ (35) $\displaystyle\kappa(V_{p}^{\text{reg}})\leq 1+\delta_{2,t}$ (36) $$

where $\delta_{1,t},\delta_{2,t}$ are small positive numbers that remain bounded due to the regularization.

From the Lemma 4, we arrive at:

$$ $\displaystyle\begin{split}\frac{(1+\delta_{1,t})(1+\delta_{2,t})(\frac{1-\epsilon_{t}}{1+\epsilon_{t}})}{\kappa(U_{p}^{\text{unreg}})\cdot\kappa((V_{p}^{\text{unreg}})^{\top})}\leq\frac{\kappa(W_{p}^{\text{reg}})}{\kappa(W_{p}^{\text{unreg}})}\leq\frac{(1+\delta_{1,t})(1+\delta_{2,t})(\frac{1+\epsilon_{t}}{1-\epsilon_{t}})}{\kappa(U_{p}^{\text{unreg}})\cdot\kappa((V_{p}^{\text{unreg}})^{\top})}\end{split}$ (37) $$

As training continues, in the unregularized case, $\kappa(U_{p,\text{unreg}})$ and $\kappa(V_{p,\text{unreg}}^{\top})$ tend to increase as $U_{p}$ and $V_{p}^{\top}$ deviate further from orthonormality. On the other hand, $(1+\delta_{1,t})(1+\delta_{2,t})(\frac{1+\epsilon_{t}}{1-\epsilon_{t}})$ will approach to 1 because of the reinforcement in orthonormality. Therefore, $\exists c>0$, while $t>c$:

$$ $\displaystyle\kappa(U_{p}^{\text{unreg}})\cdot\kappa((V_{p}^{\text{unreg}})^{\top})>(1+\delta_{1,t})(1+\delta_{2,t})(\frac{1+\epsilon_{t}}{1-\epsilon_{t}})$ (38) $$

will hold true.

Therefore, while $t>c$, we have:

$$ $\displaystyle\frac{\kappa(W_{p}^{\text{reg}})}{\kappa(W_{p}^{\text{unreg}})}<1$ (39) $$

indicating an improvement in the condition number.
∎

###### Proof.

###### Proof.

###### Proof.

###### Proof.

###### Proof.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The same questions as given above in the section of Weaknesses.
soundness: 2
strengths: 1. The proposed method outperforms the other PEFT techniques such as LoRA and PiSSA in terms of accuracy on various benchmarks, showcasing its effectiveness. The results look very promising on a variety of experiments.
2. The authors also analyze the variation patterns of singular values and vectors during parameter updates and compare SORSA with other PEFT methods such as LoRA and partial fine-tuning.
summary: This paper introduces SORSA that is a parameter-efficient fine-tuning (PEFT) method designed to adapt large language models (LLMs) for downstream tasks. The experimental results demonstrate  that it can converge faster than PiSSA and LoRA and achieves the higher accuracy on benchmarks like GSM-8K and MATH.
weaknesses: 1. The noverty of this paper is limited. the initialization method is from Pissa [1], and updates in the form of singular value decomposition and the orthonormality regularizer are from AdaLoRA [2].
2. Some symbols in Theorem 3 and Theorem 5 are used without the previous definition, which can be confusing. It is better restate these theorems to make them more straightforward.
3.  Proof of Theorem 2 is not right. Line 1035-1036. "This L is finite because the Frobenius norms
of U and V are bounded (they represent orthonormal matrices in the ideal case)". This statement not ritght. You need to add the condition that Frobenius norms of U and V are bounded. $\mathcal{L}_{reg}$ is Lipshitz only when Frobenius norms of U and V are bounded.
4. Theorem 3 is questionable. Related workes on convergence of optimizers for transformers are not clear. They only give some preliminary results under some strict assumptions and settings. But Therorem 3 needs nothing. The proof of Theorem 3 is based on Eq (21), howerver, it has no basis. 
5. Due to my limited time, I haven't checked all the lemmas and theorems, but they do have some problems. I strongly recommend that all authors carefully examine the theoretical analysis to make sure it is correct.
6. In table 1，the result of RWKV6 7B on GSM-8K for LoRa is only 8.04%. This is very strange (too
low). Make sure the experiment setting is correct. Since SORSA is similar to Pissa and AdaLoRA. It is better to campare with the results of AdaLoRA as well.
[1] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal Singular Values and Singular
Vectors Adaptation of Large Language Models.
[2] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao.  Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: The paper should clarify its relationship to AdaLoRA and PiSSA in the "Related Works" section (or even the Introduction), which it currently does not.

There appear to be inconsistencies in notation with symbols like $V_{[:r]}$, $U_{[:r,:r]}$ and $V_{[:r,:]}$ in Section 2. Additionally, "range" notation is introduced only in Section 4.

Section 3.1 explains SVD, but this explanation is not well-written and is confusing. Given that SVD is a standard technique, it is unnecessary to explain it to the paper's target audience. Simply introducing the notation should suffice.

What is $\Sigma$ in Section 3.1? It is not defined. Is it the diagonal matrix of singular values? The paper already uses $S$ for the vector of singular values.

It may be clearer to use $\mathrm{diag}(S)$ instead of $\mathrm{diag}(W)$ at the end of page 3 and the symbol $W$ is already dedicated to weight matrices.

Section 3.2, titled "Analysis Method," does not actually describe any analysis method. Instead, it introduces metrics that express the deviation of the singular values and vectors of the updated weight matrices from those of the original pre-trained matrices. However, metrics alone do not constitute an analysis method. Moreover, these ad-hoc metrics and the conclusions drawn from their behavior in Figure 2 are speculative, with no analysis showing a direct correlation between these metrics and fine-tuning performance.

"$\Delta\Sigma_t$ represents singular value variants between $W_0$ and $W_t$"
should perhaps read
"$\Delta\Sigma_t$ represents the distance/difference between singular values of $W_0$ and $W_t$"

Section 3.3 compares the behavior of metrics from Section 3.2 for various fine-tuning methods, including SORSA. However, since SORSA has not yet been introduced, readers may not understand why certain behavior is expected. Therefore, this analysis should appear later in the paper, possibly in the appendix, as it is speculative and not essential for understanding the main advantages of the proposed technique.

"significant adjustments in significant vectors" should read "significant adjustments in singular vectors"

"parallel updating pattern across weights in different layers, which emphasizes a restriction of these methods"
I would say that it shows that there is not any restriction on updating parameters in all layers.

"indicating that the updates in the SORSA are less constrained"
Since the changes in singular values and vectors are smaller for SORSA, it suggests that the updates are actually more constrained, as SORSA uses orthonormality regularization as an additional constraint.

"matrix that preserves the largest significant values and vectors, containing the matrix’s most significant data"
Again, "significant" should likely be "singular". What does it mean "the matrix’s most significant data"? This is very poor description of what the largest singular values and the corresponding singular vectors represent.

"which consist $W_p$" should perhaps read "which constitute $W_p$" ... and similarly for $W_r$.

Equation (10) presents an implementation detail that seems unnecessary for inclusion in the main paper.

What is $k$ in Equation (11)? Without defining it, Equation (11) implies that $\gamma$ is smaller than any arbitrary number.

"... SORSA, we present a novel analysis of its condition number"
In (7), SORSA is presented as a function, so it does not have a condition number. The matrix $W_r+W_p$ does.

What are $\sigma_i^{unreg}$ and $\sigma_i^{reg}$ in Equation (12)? Are they the singular values of the fine-tuned models? The derivation of Equation (12) in the appendix is difficult to follow. Where does Equation (25) originate? What do $W_p^{unreg}$  and $W_p^{reg}$ represent in this equation—the parameters of the regularized and unregularized fine-tuned model? Equation (25) implies that the Frobenius norm of the differences between in trained matrices $W_p^{unreg}$  and $W_p^{reg}$ equals the scaled Frobenius norm of the gradient of the regularizer with respect to a $W_p$ matrix. Which $W_p$ matrix? What point is the gradient evaluated at? Equation (25) is completely unclear.

The symbols  $\delta_{1,t}$ and $\delta_{2,t}$ are not introduced in Theorem 5. Likewise, symbols $\kappa$, $\sigma_{max}$ and $\sigma_{min}$ are not introduced, though their meanings may be guessed.



**===End of the original review ====**

Here, I provide response to the Authors comments on the review as I was not able to respond by the official means:


The paper has been significantly improved in the current updated version. It is much easier to follow thanks to its improved structure and presentation. Additionally, results for AdaLoRA have been added, making the results more convincing. Accordingly, I am increasing my scores for the paper.

However, several issues remain. Some of these were already pointed out in the reviews but have not been addressed in the updated paper:

Multiple reviewers requested clarification on the relationship between SORSA, AdaLoRA, and PiSSA. Specifically, we asked for acknowledgment that SORSA is primarily a combination of ideas from these two methods. This relationship (especially with AdaLoRA) is still not clearly stated in the paper.

One reviewer pointed out that "SORSA (w/o reg) is essentially the same as PiSSA." I agree with this assessment, and it should be explicitly stated in the paper. While I understand that PiSSA merges singular values with the matrices of singular vectors, this correspond to only different parametrization of the same model and should not significantly affect optimization (as you confirm in your response). You mentioned that you avoided explicitly stating this to prevent confusion, but I believe that not clearly acknowledging the equivalence of "SORSA (w/o reg)" and "PiSSA" is actually more misleading.

I still find the "Singular Values and Vector Analysis" section speculative and of limited scientific value. As I noted in my previous review, "the ad-hoc metrics and the conclusions drawn from their behavior in Figure 2 are speculative, with no analysis showing a direct correlation between these metrics and fine-tuning performance." Why must we examine differences between weight matrices in terms of singular values and singular vectors separately? What do the patterns in Figure 2 teach us? Low-rank approximation using SVD minimizes the Frobenius norm of the difference between the original matrix and the approximating matrix, so why not simply measure similarity using the Frobenius norm? Wouldn't this metric be equally expressive? For example, the last graph in Figure 2 shows that in some layers, only singular values change while singular vectors remain constant. Does this observation have any meaningful implications? There is no analysis provided to support its relevance.

You state:

"Figure 2 shows that almost all FT, LoRA, and SORSA layers without a regularizer exhibit synchronized and linear-like updating in singular values and vectors. This shows that all layers are 'locked' with each other, which I interpret as 'restriction.' Although SORSA uses one additional regularizer during training, Figure 2 actually demonstrates its more 'free' updating (evident in how different layers can update more independently)."

What does it mean that "layers are 'locked' with each other"? Are you suggesting that updates are correlated across layers? There is no measurement of such correlations in the paper. Nor do you demonstrate that parameters are not updated independently in FT and LoRA. Figure 2 merely shows that all parameters across all layers are updated in FT and LoRA, while only some layers are effectively updated in SORSA. Perhaps stronger (L2 or L1) regularization toward the original weight matrix would achieve the same effect in LoRA.

I strongly suggest removing the section on the "optimized version of SORSA," even from the appendix. The optimization you describe is the only reasonable way to implement multiplication with a diagonal matrix, something anyone with basic algebra knowledge and programming skills would do by default. Admitting that you considered converting the diagonal matrix to a dense one and multiplying with it is embarrassing. Moreover, what you call element-wise multiplication in your equation is not truly element-wise, as the vector must first be broadcast. Finally, as I noted in my previous review, Figure 4 does not illustrate the quadratic vs. cubic complexity as claimed in the text.

The practical purpose of Equation (11), defining $\gamma$ remains unclear. I had hoped your derivation in the appendix would clarify this, but it did not. You state:

"$k$ will be a constant. This essentially implies $\gamma$ should be inversely proportional to $n_d$"

Here, $n_d$ is a constant hyperparameter (maximum learning rate), and you define $\gamma =k / n_d$, where $k$ is another hyperparameter. Why should $k$ be tuned instead of directly adjusting $\gamma$?

It is still unclear what the singular values in Equation (12) represent. Are they the corresponding values (regularized and unregularized) from the respective updates? They do not depend on any iteration index $t$ in the formula.

Minor Issues:

The figures, especially Figure 2, use fonts that are too small to read, although the authors promise to improve them.
Use diag(S) instead of diag(W), as W is already used to denote a weight matrix, not a vector.
 
The subsection titled "Analysis Method" should be renamed to something like "Measuring Similarity Between Singular Values and Vectors," as the current title does not accurately reflect its content.
soundness: 2
strengths: The paper reports superior performance for LLMs fine-tuned with SORSA compared to the PiSSA method. Additionally, the proposed approach is relatively easy to implement. The originality of this work lies in combining the ideas behind AdaLoRA and PiSSA, as described in the "Summary" section.
summary: The SORSA method proposed in this paper is a variant of LoRA—a technique for parameter-efficient fine-tuning of pre-trained large language models (LLMs). SORSA combines concepts from two previously proposed LoRA variants: AdaLoRA (Zhang et al., 2023) and PiSSA (Meng et al., 2024). In the original LoRA method, each weight matrix $W$ in an LLM is adapted as $W+AB$, where $AB$ represents the product of two low-rank matrices, $A$ and $B$, which are the trainable parameters. AdaLoRA adapts the weight matrices as $W+USV^T$, where $U$ and $V$ are randomly initialized low-rank matrices, and $S$ is a zero-initialized diagonal matrix. During training, $U$ and $V$ are regularized to form orthonormal bases in their columns, so that $USV^T$ resembles the singular value decomposition (SVD) of a matrix. Similar regularization has been shown to speed up training in many previous works.

PiSSA, on the other hand, directly decomposes each pre-trained weight matrix using SVD as $W=USV^T$. During fine-tuning, only the parameters associated with the largest singular values and the corresponding singular vectors (selected columns of $U$ and $V$) are updated, while the remaining parameters remain fixed. PiSSA does not use any orthogonality-enforcing regularization during fine-tuning. SORSA, the method proposed in this paper, essentially applies PiSSA but regularizes the fine-tuned vectors (selected columns of $U$ and $V$) to stay nearly orthonormal, as in AdaLoRA.
weaknesses: Firstly, the paper suffers from poor writing quality, with numerous grammatical errors and awkward English expressions (too many to list exhaustively in this review). The text is also poorly structured, with some mathematical symbols left undefined. The figures use such small font sizes that they are almost unreadable, even in the electronic version where one can zoom in. More detailed feedback is provided below. The paper appears to be written by an inexperienced author, so I would recommend seeking assistance from someone with more experience and strong English skills to review and refine the text.

Besides the standard full fine-tuning and LoRA, the paper compares the proposed SORSA method only with PiSSA. However, since SORSA builds on AdaLoRA’s concepts, I believe AdaLoRA (and possibly other related methods like OLoRA or DoRA) should also be included in the comparisons. Furthermore, the experimental results are limited compared to other referenced works (e.g., the PiSSA paper).

The author’s limited experience is further evident in presenting some trivial findings as if they were major contributions. For example, Section 4 refers to Appendix A, where readers are promised an optimized and highly efficient version of SORSA. However, Appendix A merely states the obvious: that multiplying by a diagonal matrix is more efficient than naively multiplying by a full matrix with zeroed-out off-diagonal elements. The appendix also includes an analysis of the speedup achieved by this "optimized" implementation, but it yields nonsensical results that do not align with the $O(N^2)$ versus $O(N^3)$ complexity of the optimized and naive approaches. Similarly, the statement that the regularizer is convex and Lipschitz continuous is rather obvious, although the corresponding derivations are reasonably included in an appendix.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Is there any insight into the "Grad Norm" figures in Figure 3?
soundness: 2
strengths: 1. While orthonormal regularization on incremental low-rank matrices has been used in existing works, the analysis of the condition number is novel to me, and the improved stability makes sense.
2. The approach is simple and effective, as it directly applies orthonormal regularization.
summary: The authors propose Singular Values and Orthonormal Regularized Singular Vectors Adaptation (SORSA) for parameter-efficient fine-tuning (PEFT). They use an analysis of singular values and vectors to show the limitation of existing works and propose to solve it via orthonormal regularization. Theoretical analysis shows that the condition number of the low-rank incremental matrix is improved via regularization, resulting in enhanced stability. The superior performance of SORSA is validated on three NLG tasks compared to other PEFT methods.
weaknesses: 1. The method is not well-motivated. The authors begin with an analysis of singular values and vectors, which shows a different updating pattern of SORSA compared to other methods. However, it is unclear how this is connected to the limitation of the generalization ability of LoRA and FT. I can only observe a limitation of the learning capacity of LoRA and FT. Additionally, it is not clear why the orthonormal regularization leads to a different updating pattern, as shown in Figure 2, and why this pattern can give an improvement. I suggest a further theoretical justification for this point.
2. There seems to be a misuse of terminology: the authors use FT to denote "partial fine-tuning" (for example, page 2, line 86). However, in the literature, FT often denotes "full fine-tuning" (for example, in DoRA [1]). If the analysis in Sec. 3.2 is truly inspired by DoRA, the authors may want to compare the updating patterns of "full fine-tuning, LoRA, and SORSA" instead of "partial fine-tuning, LoRA, and SORSA." If I misunderstood, could you provide a definition of partial fine-tuning early in the main paper?
3. The experimental comparison is not extensive enough in terms of benchmarks. Only NLG tasks are used to evaluate the method, and it's not clear whether SORSA works in other NLP tasks. More experiments on other tasks, such as the common GLUE benchmark [2] in natural language understanding (NLU), are expected.
4. It seems SORSA (w/o reg) is essentially the same as PiSSA as described in Related Work, so I encourage the authors to use consistent terminology throughout the paper. If they are different, please give a clear explanation of the differences. Also, in this case, quantitative ablation studies are missing since the only comparison between SORSA and SORSA (w/o reg) is in Figure 2. The performance of SORSA (w/o reg) should also be provided in Table 1.



[1] Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-
Ting Cheng, and Min-Hung Chen. DoRA: Weight-Decomposed Low-Rank Adaptation.
In Forty-first International Conference on Machine Learning, June 2024.

[2] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.
Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv
preprint arXiv:1804.07461, 2018.

**Minor comments**

1. Figure quality can be improved. For example, the font size should be larger for better readability.