MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

## Abstract

Abstract Low-rank adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for large language models (LLMs).
In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge.
Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters.
To achieve it,
we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix.
Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA.
We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.
Our code will be available at https://github.com/kongds/MoRA .

## 1 Introduction

As the size of language models increases, parameter-efficient fine-tuning (PEFT) Houlsby et al. () has emerged as a popular technique to adapt these models to specific downstream tasks.
Compared to Full Fine-Tuning (FFT), which updates all model parameters, PEFT modifies only a small part of the parameters. For example, it can achieve similar performance with FFT by updating less than 1% of the parameters in some tasks Hu et al. (), which significantly reduces the memory requirements for the optimizer and facilitates the storage and deployment of fine-tuned models.

Figure: (a) LoRA ($r=8$)
Refer to caption: extracted/5606540/lora.png

Among the existing PEFT methods, Low-Rank Adaptation (LoRA) Hu et al. () is particularly prevalent for LLMs. LoRA enhances performance over other PEFT methods such as prompt tuning Lester et al. () or adapters Houlsby et al. () by updating parameters via low-rank matrices. These matrices can be merged into the original model parameters, thereby avoiding additional computational costs during inference. There are numerous methods that aim to improve LoRA for LLMs. However,
most methods primarily validate their efficiency based on GLUE Wang et al. (), either by achieving better performance or by requiring fewer trainable parameters.
Recent methods Liu et al. (); Meng et al. (); Zhu et al. () leverage instruction tuning task such as Alpaca Wang et al. () or reasoning tasks like GSM8K Cobbe et al. () to better evaluate their performance on LLMs.
However, the diverse settings and datasets used in the evaluation complicate the understanding of their progression.

In this paper, we conduct a comprehensive evaluation of LoRA across various tasks under the same settings, including instruction tuning, mathematical reasoning, and continual pretraining. We find that LoRA-like methods demonstrate similar performance across these tasks and they perform comparably to FFT in instruction tuning but fall short in mathematical reasoning and continual pretraining. Among these tasks, instruction tuning primarily focuses on interacting with the format, rather than acquiring knowledge and capabilities, which are learned almost entirely during pretraining Zhou et al. (). We observe that LoRA is easily adapted to follow response formats in instruction tuning but struggles with other tasks that require enhancing knowledge and capabilities through fine-tuning.

One plausible explanation for this limitation observed with LoRA could be its reliance on low-rank updates Lialin et al. (). The low-rank update matrix, $\Delta W$, struggles to estimate the full-rank updates in FFT, particularly in memory-intensive tasks like continual pretraining that require memorizing domain-specific knowledge.
Since the rank of $\Delta W$ is significantly smaller than the full rank, this limitation restricts capacity to store new information via fine-tuning. Moreover, current variants of LoRA cannot alter the inherent characteristic of low-rank updates.
To validate this, we conducted a memorization task using pseudo-data to assess the performance of LoRA in memorizing new knowledge. We found that LoRA performed significantly worse than FFT, even with a large rank such as 256.

Given these observations, we introduce a method called MoRA, which employs a square matrix as opposed to low-rank matrices, aiming to maximize the rank in $\Delta W$ while maintaining the same number of trainable parameters. For instance, when utilizing 8 rank with the hidden size 4096, LoRA employs two low-rank matrices $A\in\mathbb{R}^{4096\times 8}$ and $B\in\mathbb{R}^{8\times 4096}$, with $rank(\Delta W)\leq 8$. Under same number of parameters, our method uses a square matrix $M\in\mathbb{R}^{256\times 256}$, with $rank(\Delta W)\leq 256$, as depicted in Figure 1.
Notably, our method exhibits a greater capacity than LoRA with a large rank.
To decrease the input dimension and increase the output dimension for $M$, we develop corresponding non-parameter operators.
Furthermore, these operators and $M$ can be substituted by a $\Delta W$, ensuring our method can be merged back into LLM like LoRA.

Our contributions are as follows:

- 1.
We introduce MoRA, a novel method that employs a square matrix instead of low-rank matrices in LoRA to achieve high-rank updating, while maintaining the same number of trainable parameters.
- 2.
We discuss four kinds of non-parameter operators of MoRA to reduce the input dimension and increase the output dimension for the square matrix, while ensures that the weight can be merged back into LLMs.
- 3.
We evaluate MoRA across five tasks: memory, instruction tuning, mathematical reasoning, continual pretraining, and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks, which demonstrates the effectiveness of high-rank updating.

## 2 Related Work

### 2.1 LoRA

LoRA is one of the most popular PEFT methods for fine-tuning LLM, owing to its broad applicability and robust performance in comparison to other methods.
To approximate the updated weight $\Delta W$ in FFT, LoRA employs two low-rank matrices for its decomposition. By adjusting the rank of these two matrices, LoRA can accordingly modify the trainable parameters. Benefit from it, LoRA can merge these matrices after fine-tuning without the inference latency compared to FFT.
There are many methods to further improve LoRA, particularly for the application in LLMs. DoRALiu et al. () further decomposes the original weight into magnitude and direction components and uses LoRA to update the direction component. LoRA+Hayou et al. () employs different learning rates for the two low-rank matrices to improve learning efficiency. ReLoRALialin et al. () integrates LoRA into the LLM during training to increase the rank of the final $\Delta W$.

### 2.2 Fine-Tuning with LLMs

Despite the impressive performance of LLMs with in-context learning, certain scenarios still necessitate fine-tuning, which can be broadly categorized into three types.
The first type, instruction tuning, aims to better align LLMs with end tasks and user preferences, without significantly enhancing the knowledge and capabilities of LLMs Zhou et al. (). This approach simplifies the process of dealing with varied tasks and understanding complex instructions.
The second type involves complex reasoning tasks such as mathematical problem-solving Collins et al. (); Imani et al. (); Yu et al. (), where general instruction tuning often falls short in handling complex, symbolic, multi-step reasoning tasks. To improve the reasoning abilities of LLMs, the majority of research focuses on creating corresponding training datasets, either by leveraging larger teacher models like GPT-4 Fu et al. (), or by rephrasing questions along a reasoning path Yu et al. ().
The third type, continual pretraining Cheng et al. (); Chen et al. (); Han et al. (); Liu et al. (), aims to enhance the domain-specific capabilities of LLMs. Unlike instruction tuning, it necessitates fine-tuning to augment the corresponding domain-specific knowledge and capabilities.

However, most variants of LoRA Kopiczko et al. (); Lialin et al. (); Dettmers et al. (); Zhu et al. () predominantly employ instruction tuning or text classification tasks from GLUE Wang et al. () to validate their efficacy on LLMs. Given that instruction tuning requires the least capacity for fine-tuning compared to other types, it may not accurately reflect the effectiveness of LoRA variants.
To better evaluate their methods, recent works Meng et al. (); Liu et al. (); Shi et al. (); Renduchintala et al. () have employed reasoning tasks to test their methods.
But the training sets used are often too small for LLMs to effectively learn reasoning. For instance, some methods Meng et al. (); Renduchintala et al. () utilize the GSM8K Cobbe et al. () with only 7.5K training samples. Compare to the SOTA method with 395K training samples Yu et al. (), this small training set achieves worse performance on reasoning and makes it hard to evaluate the effectiveness of these methods.

## 3 Analysis the Influence of Low-rank Updating

The key idea of LoRA Hu et al. () involves the use of low-rank updates to estimate full-rank updates in FFT. Formally, given a pretrained parameter matrix $W_{0}\in\mathbb{R}^{d\times k}$, LoRA employs two low-rank matrices to calculate the weight update $\Delta W$:

$$ $h=W_{0}x+\Delta Wx=W_{0}x+BAx$ (1) $$

where $A\in\mathbb{R}^{r\times k}$ and $B\in\mathbb{R}^{d\times r}$ represent the low-rank matrices in LoRA. To ensure that $\Delta W=0$ at the beginning of training, LoRA initializes $A$ with a Gaussian distribution and $B$ with zero. Due to the low-rank decomposition of $\Delta W$ into $BA$, the $rank(\Delta W)\leq r$.
The weight update in LoRA exhibits a markedly low rank, $r\ll\min(d,k)$, in comparison to the full-rank updating in FFT.
Low-rank updating by LoRA shows on-par performance with full-rank updating in some tasks such as text classification or instruction tuning Liu et al. (); Meng et al. ().
However, for tasks like complex reasoning or continual pretraining, LoRA tends to show worse performance Liu et al. ().

Based on these observations, we propose a hypothesis that low-rank updating is easy to leverage original knowledge and capabilities of LLM to solve task, but it is struggle to handle tasks that require enhancing knowledge and capabilities of LLM.

Figure: Figure 2: Performance of memorizing UUID pairs through fine-tuning with FFT and LoRA.
Refer to caption: extracted/5606540/fft-memory-lora.png

To substantiate this hypothesis, we examine the differences between LoRA and FFT in terms of memorizing new knowledge through fine-tuning. In order to circumvent leveraging the original knowledge of the LLM, we randomly generate 10K pairs of Universally Unique Identifiers (UUIDs), each pair comprising two UUIDs with 32 hexadecimal values. The task requires the LLM to generate the corresponding UUID based on the input UUID. For instance, given a UUID such as “205f3777-52b6-4270-9f67-c5125867d358”, the model should generate the corresponding UUID based on 10K training pairs.
This task can also be viewed as a question-answering task, while the knowledge indispensable for accomplishing it is exclusively from the training datasets rather than the LLM itself.

For the training settings, we employ LLaMA-2 7B as base model, utilizing 1,000 pairs per batch and conducting 100 epochs. For the LoRA, we apply low-rank matrices to all linear layers and search learning rate from $\{$1e-4,2e-4,3e-4$\}$ to enhance performances.
We conduct the experiment on LoRA using various ranks $r\in\{8,16,32,64,128,256\}$.
For the FFT, we directly use a learning rate of 3e-5.
Based on Figure 2, we observe low-rank updating are hard to memorizing new knowledge compared to FFT.
Although constantly increasing the rank of LoRA can alleviate this problem, the gap still exists.

In contrast to the memory task, we also evaluate the performance gap between LoRA and FFT on instruction tuning, which merely introduces new knowledge.
Similar to previous results Meng et al. (); Zhu et al. (), we also find that LoRA matches the performance of FFT with small rank $r=8$ in Table 1.
This indicates that LoRA can easily leverage the original knowledge of LLMs by fine-tuning like FFT.

## 4 Method

Based on the above analysis, we propose a new method to alleviate the negative effects of low-rank updating. The main idea of our method is to utilize the same trainable parameters as much as possible to achieve a higher rank in $\Delta W$.
Consider to the pretrained weight $W_{0}\in\mathbb{R}^{d\times k}$, LoRA uses two low-rank matrices $A$ and $B$ with $(d+k)r$ total trainable parameters for rank $r$.
Under same trainable parameters, a square matrix $M\in\mathbb{R}^{\hat{r}\times\hat{r}}$ where $\hat{r}=\lfloor\sqrt{(d+k)r}\rfloor$ can achieve the highest rank due to $r\ll\min(d,k)$.

To accomplish this, we need to reduce the input dimension and increase the output dimension for $M$. Formally,

$$ $\begin{split}h&=W_{0}x+f_{\text{decomp}}\left(Mf_{\text{comp}}\left(x\right) \right)\end{split}$ (2) $$

where $f_{\text{comp}}:\mathbb{R}^{k}\rightarrow\mathbb{R}^{\hat{r}}$ denotes the function that decreases the input dimension of $x$ from $k$ to $\hat{r}$, and $f_{\text{decomp}}:\mathbb{R}^{\hat{r}}\rightarrow\mathbb{R}^{d}$ represents the function that enhances the output dimension from $\hat{r}$ to $d$.
Furthermore, these two functions ought to be non-parameterized operators and expected to execute in linear time corresponding to the dimension. They should also have corresponding function, $f_{\overline{\text{comp}}}:\mathbb{R}^{\hat{r}\times\hat{r}}\rightarrow\mathbb
{R}^{\hat{r}\times k}$ and $f_{\overline{\text{decomp}}}:\mathbb{R}^{\hat{r}\times k}\rightarrow\mathbb{R}
^{d\times k}$, to transform $M$ into $\Delta W$. For any $x$, the following should hold:

$$ $f_{\text{decomp}}\left(Mf_{\text{comp}}\left(x\right)\right)=\Delta Wx,\forall x \in\mathbb{R}^{k}\\$ (3) $$

where $\Delta W=f_{\overline{\text{decomp}}}\left(f_{\overline{\text{comp}}}\left(M
\right)\right)$.
If Eq. 3 holds, $M$ can be losslessly expanded to $\Delta W$ based on $f_{\text{comp}}$ and $f_{\text{decomp}}$. This allows our method to merge back into the LLM like LoRA.

For the design of $f_{\text{comp}}$ and $f_{\text{comp}}$, we explore several methods to implement these functions.
One straightforward method is truncating the dimension and subsequently add it in corresponding dimension.
Formally, this can be represented as:

$$ $\begin{split}f_{\text{comp}}\left(x\right)&=x_{1:\hat{r}}\\ f_{\text{decomp}}\left(x\right)&=\begin{bmatrix}x\\ \mathbf{0}\end{bmatrix}\end{split}$ (4) $$

and the corresponding $\Delta W$ is:

$$ $\Delta W=\begin{bmatrix}M&\mathbf{0}\\ \mathbf{0}&\mathbf{0}\end{bmatrix}$ (5) $$

However, this method leads to a significant loss of information during compression and only modifies a segment of the output by appending a zero vector during decompression.
To improve it, we can share the rows and columns of $M$ to achieve a more efficient compression and decompression. Formally, this can be represented as:

$$ $\begin{split}f_{\text{comp}}\left(x\right)&=\begin{bmatrix}\sum_{j\in g_{i}}x_ {j}\end{bmatrix}_{i=1}^{r}\\ f_{\text{decomp}}\left(x\right)&=\begin{bmatrix}x_{\widetilde{g}^{\prime}_{i}} \end{bmatrix}_{i=1}^{d}\end{split}$ (6) $$

Here, $g$ and $g^{\prime}$ represent predefined groups that share the same row and column in $M$, respectively. The $j\in g_{i}$ indicates that the $j$-th dimension belongs to the $i$-th group in $g$. The term $\widetilde{g}^{\prime}_{i}$ is the reverse of $g^{\prime}_{i}$, referring to the $i$-th dimension associated with the $\widetilde{g}^{\prime}_{i}$-th group in $g^{\prime}$. The corresponding $\Delta W$ is as follows:

$$ $\Delta W_{i,j}=M_{\widetilde{g}^{\prime}_{i},\widetilde{g}_{j}}$ (7) $$

Sharing rows and columns can be efficient for larger ranks such as $r=128$ or $r=256$, as only a few rows or columns in $\Delta W$ share a common row or column.
For instance, considering to $\Delta W\in\mathbb{R}^{4096\times 4096}$ for $r=128$, which has $\hat{r}=1024$ and $M\in\mathbb{R}^{1024\times 1024}$. In this situation, only 4 rows or columns share the same row or column.
Conversely, for smaller ranks such as $r=8$, where $\hat{r}=256$, it requires average 16 rows or columns in a group to share the same row or column in $M$.
It can lead to inefficiencies due to the significant information loss during compression in Eq. 6.

To enhance performance for smaller ranks, we reshape $x$ instead of directly compressing it, to preserve the input information. In this context, $f_{\text{comp}}\left(x\right):\mathbb{R}^{k}\rightarrow\mathbb{R}^{n\times\hat
{r}}$ and $f_{\text{decomp}}:\mathbb{R}^{n\times\hat{r}}\rightarrow\mathbb{R}^{d}$.
Corresponding $f_{\text{comp}}$, $f_{\text{decomp}}$ and $\Delta W$ are as follows:

$$ $\begin{split}f_{\text{comp}}\left(x\right)&=\begin{bmatrix}x_{1:\hat{r}}&x_{ \hat{r}:2\hat{r}}&\cdots&x_{(n-1)\hat{r}:n\hat{r}}\end{bmatrix}\\ f_{\text{decomp}}\left(x\right)&=\text{concat}(x)\\ \Delta W&=\begin{bmatrix}M&\mathbf{0}&\cdots&\mathbf{0}\\ \mathbf{0}&M&\cdots&\mathbf{0}\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{0}&\mathbf{0}&\cdots&M\end{bmatrix}\end{split}$ (8) $$

where $\text{concat}(x)$ refers to concatenate the rows of $x$ into a vector.
For simplicity, we omit the padding and truncation operators in above functions and focus on the case where $d=k$. In comparison to sharing columns and rows, this method incurs additional computational overhead by reshaping $x$ into $\mathbb{R}^{n\times\hat{r}}$ instead of $\mathbb{R}^{\hat{r}}$. However, given that the size of $M$ is significantly smaller than $W_{0}$, this additional computation is very small for rank like 8. For instance, when fine-tuning the 7B model with rank of 8 ($\hat{r}=256$), this method is only 1.03 times slower than previous methods.

Inspired by RoPE Su et al. (), we can further refine this method by incorporating rotation operators into $f_{\text{comp}}$ to augment the expressiveness of $M$ by enable it to differentiate between various $x_{i\hat{r}:(i+1)\hat{r}}$ by rotating them.
We can modify Eq. 8 as follows:

$$ $\begin{split}f_{\text{comp}}\left(x\right)&=\begin{bmatrix}a^{1}&a^{2}&\cdots& a^{n-1}\end{bmatrix}\\ \Delta W&=\begin{bmatrix}P^{1}&\mathbf{0}&\cdots&\mathbf{0}\\ \mathbf{0}&P^{2}&\cdots&\mathbf{0}\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{0}&\mathbf{0}&\cdots&P^{n-1}\end{bmatrix}\end{split}$ (9) $$

where $a^{i}$ and $P^{i}$ represent the corresponding values of $x_{i\hat{r}:(i+1)\hat{r}}$ and $M$ post-rotation, respectively.
Following RoPE, we use a $\hat{r}\times\hat{r}$ block diagonal matrix to achieve the rotation.
However, our method use rotation information to enable $M$ distinguish the $x_{i\hat{r}:(i+1)\hat{r}}$ instead of token position in RoPE.
We can define $a^{i}$ and $P^{i}$ as follows:

$$ $\begin{split}a^{i}&=\begin{bmatrix}{R}_{\theta_{1},i}&\mathbf{0}&\cdots& \mathbf{0}\\ \mathbf{0}&{R}_{\theta_{2},i}&\cdots&\mathbf{0}\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{0}&\mathbf{0}&\cdots&{R}_{\theta_{\frac{\hat{r}}{2}},i}\end{bmatrix}x_ {i\hat{r}:(i+1)\hat{r}}\\ P^{i}&=M\begin{bmatrix}{R}_{\theta_{1},i}&\mathbf{0}&\cdots&\mathbf{0}\\ \mathbf{0}&{R}_{\theta_{2},i}&\cdots&\mathbf{0}\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{0}&\mathbf{0}&\cdots&{R}_{\theta_{\frac{\hat{r}}{2}},i}\end{bmatrix} \end{split}$ (10) $$

where $\theta_{j}=10000^{-2(j-1)/\hat{r}}$ and ${R}_{\theta_{j},i}\in\mathbb{R}^{2\times 2}$ is a rotation matrix:

$$ ${R}_{\theta_{j},i}=\begin{bmatrix}\cos i\theta_{j}&-\sin i\theta_{j}\\ \sin i\theta_{j}&\cos i\theta_{j}\end{bmatrix}\\$ (11) $$

## 5 Experiment

We evaluate our method on various tasks to understand the influence of high-rank updating.
In Section 5.1, we evaluate our method with LoRA and our method on memorizing UUID pairs to show the benefit of high-rank updating on memorizing.
In Section 5.2, we reproduce LoRA, LoRA variants and FFT on three fine-tuning tasks: instruction tuning, mathematical reasoning and continual pretraining.
In Section 5.3, we compare our method with LoRA and ReLoRA on pretraining by training transformer from scratch.

Figure: Figure 3: Performance of memorizing UUID pairs with LoRA and our method on rank 8 and 256.
Refer to caption: extracted/5606540/moravslora-memory-lora.png

**Table 1: Performance of FFT, LoRA, LoRA variants and our method on instruction tuning, mathematical reasoning and continual pretraining tasks.**
|  |  | Instruction Tuning | Mathematical Reasoning | Continual Pretraining |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Rank | MMLU 0 | MMLU 5 | GSM8K | MATH | BioMed. | Finance |
| FFT | - | 50.6 | 51.3 | 66.6 | 20.1 | 56.4 | 69.6 |
| LoRA | 8 | 50.2 | 51.5 | 64.6 | 15.1 | 52.3 | 64.0 |
| LoRA+ | 8 | 49.2 | 51.1 | 64.1 | 15.8 | 52.2 | 64.9 |
| ReLoRA | 8 | 49.3 | 50.2 | 61.5 | 14.5 | 46.3 | 61.0 |
| AsyLoRA | 8 | 50.3 | 52.2 | 64.5 | 15.0 | 52.5 | 63.5 |
| DoRA | 8 | 50.2 | 51.5 | 64.5 | 14.6 | 52.5 | 63.9 |
| MoRA (Ours) | 8 | 49.7 | 51.5 | 64.2 | 15.4 | 53.3 | 67.1 |
| LoRA | 256 | 49.7 | 50.8 | 67.9 | 19.9 | 54.1 | 67.3 |
| LoRA+ | 256 | 49.2 | 51.3 | 68.2 | 17.1 | 54.2 | 66.7 |
| ReLoRA | 256 | - | - | 64.0 | 18.1 | 52.9 | 57.9 |
| AsyLoRA | 256 | 50.1 | 52.0 | 66.9 | 19.3 | 54.1 | 66.9 |
| DoRA | 256 | 49.6 | 51.1 | 67.4 | 19.5 | 54.2 | 66.0 |
| MoRA (Ours) | 256 | 49.9 | 51.4 | 67.9 | 19.2 | 55.4 | 68.7 |

### 5.1 Memorizing UUID Pairs

We first compare our method with LoRA and FFT on memorizing UUID pairs to demonstrate improvements through high-rank updating.
Following the training settings in Section 3, we search learning rate from $\{$5e-5,1e-4,2e-4$\}$ and use decompress and compress functions in Eq. 8, sharing rows and columns in $M$.
Due to use one matrix $M$ instead of two matrices $A$ and $B$, we can directly initialize $M$ with zeros.
For the predefined groups $g$ and $g^{\prime}$, we group every adjacent $\hat{r}$ rows or columns together.
The training loss is presented in Figure3. Our method shows significant improvements over LoRA with the same number of trainable parameters, benefiting from high-rank updating. We also report character-level accuracy at various training steps in Table 2. MoRA requires fewer training steps to memorize these UUID pairs compared to LoRA.
Compared to FFT, MoRA with 256 rank can achieve similar performance and both method can memorize all UUID pairs in 500 steps.

**Table 2: Character-level accuracy of memorizing UUID pairs by generating the value of corresponding key in 300, 500, 700 and 900 training steps.**
|  | Rank | 300 | 500 | 700 | 900 |
| --- | --- | --- | --- | --- | --- |
| FFT | - | 42.5 | 100 | 100 | 100 |
| LoRA | 8 | 9.9 | 10.0 | 10.7 | 54.2 |
| MoRA | 8 | 10.1 | 15.7 | 87.4 | 100 |
| LoRA | 256 | 9.9 | 70.6 | 100 | 100 |
| MoRA | 256 | 41.6 | 100 | 100 | 100 |

### 5.2 Fine-tuning Tasks

#### 5.2.1 Setup

We evaluate our method across three fine-tuning tasks for large language models (LLMs): instruction tuning, mathematical reasoning, and continual pretraining.
For these tasks, we select high-quality corresponding datasets to test both LoRA and our method.
In instruction tuning, we utilize Tülu v2 Ivison et al. (), a blend of several high-quality instruction datasets, containing 326k filtered samples. We assess instruction performance using the MMLU Hendrycks et al. () in both zero-shot and five-shot settings. For mathematical reasoning, we employ the MetaMath Yu et al. () with its 395k samples to enhance mathematical reasoning capabilities and also use GSM8K Cobbe et al. () and MATH Hendrycks et al. () for further evaluation. In continual pretraining, we adapt an LLM to the biomedicine and finance using PubMed abstracts from the Pile Gao et al. () and finicial news, complemented by data preprocessing methods from AdaptLLM Cheng et al. () to boost performance. We report the average performance of corresponding tasks for continual pretraining. More details can be found in Appendix C.

Figure: (a) Pretraining loss at 250M models.
Refer to caption: extracted/5606540/250m-loss.png

#### 5.2.2 Baselines and Implements

For LoRA-like methods and MoRA, we conducted experiments at $r=8$ and $r=256$, and reproduce following methods across three tasks: FFT, LoRA, LoRA+ Hayou et al. (), AsyLoRA Zhu et al. (), ReLoRA Lialin et al. () and DoRA Liu et al. ().
LoRA+ enhances the learning rate of matrix $B$ in LoRA to facilitate efficient feature learning based on theoretical analysis.
We search the corresponding the hyperparameter $\lambda$ from $\{$2,4$\}$.
AsyLoRA also analyzes asymmetry in the $A$ and $B$ matrices, and we adopted their initialization strategy.
ReLoRA proposes a method to merge low-rank matrices into the model during training to increase the rank of $\Delta W$.
we search merge steps from {$1$k, $2$k} and use 50 steps restarts warmup.
DoRA leverages weight decomposition to enhance performance as a robust baseline.
For FFT, we follow the settings proposed by corresponding datasets.
For MoRA, we employed rotation operators as outlined in Eq. 9 to implement compression and decompression for $r=8$, and for $r=256$, we utilized shared rows and columns as specified in Eq. 6 and group every adjacent $\hat{r}$ rows or columns together.
The details hyperparameters about fine-tuning can be found in Appendix A.

#### 5.2.3 Results and Analysis

We present the results of fine-tuning tasks in Table 1.
We report the results of MMLU with zero-shot and 5-shot settings for instruction tuning, GSM8K and MATH for mathematical reasoning, and average performance on biomedical tasks and financial tasks for continual pretraining.

MoRA shows on par performances with LoRA on instruction tuning and mathematical reasoning.
Benefit from high-rank updating to memorize new knowledge, MoRA outperforms LoRA on both biomedical and financial domains for continual pretraining.

We also find that LoRA variants exhibit similar performances on these fine-tuning tasks as compared to LoRA. Although AsyLoRA achieves the best performance in instruction tuning, it demonstrates poor performance in mathematical reasoning. For ReLoRA, merging low-rank matrices during training can harm performance, particularly at the the high rank like 256.

Consider the difference between three tasks, they show different requirements for fine-tuning capabilities.
For instruction tuning, which does not learn new knowledge from fine-tuning, rank 8 is enough to achieve performance similar to FFT. For mathematical reasoning, rank 8 is unable to match FFT performance.
However, increasing the rank from 8 to 256 can eliminate the performance gap. For continual pretraining, LoRA with rank 256 still underperforms FFT.

### 5.3 Pretraining

**Table 3: Perplexity on C4 validation dataset.**
|  | 250M | 1.3B |
| --- | --- | --- |
| LoRA | 33.40 | 28.56 |
| MoRA (Ours) | 28.54 | 25.25 |
| ReLoRA | 32.19 | 27.80 |
| ReMoRA (Ours) | 26.74 | 23.34 |

To understand the influence of high-rank updating,
we train transformer from scratch on the C4 datasets Raffel et al. ().
For the model architeture, we train LLaMA-based model with RMSNorm Zhang and Sennrich (), SwiGLU Shazeer () and RoPE Su et al. () on 250M and 1.3B sizes.
For the hyperparameters, we use 10k steps, 1024 batch size, 512 sequence length and follow , using rank $r=128$ for LoRA and our methods and also keep modules without applying LoRA-like layernorm or embeddings unfreezed.
We compare our method with LoRA and ReLoRA.
To better show the difference between high-rank and low-rank updating, we reproduce ReLoRA and other methods without full-rank training warmup.
For MoRA, we use compression and decompression functions in Eq. 6 by sharing columns and rows.

We also combine merge-and-reint in ReLoRA with our method called ReMoRA by merging $M$ back into the original parameters during training to increase the rank of $\Delta W$.
However, if we directly merge $M$ with $g$ and $g^{\prime}$ in Eq. 6, the final rank of $\Delta W$ is unchanged due to the same expand pattern.
To solve this problem, we can change $g$ and $g^{\prime}$ after merging to ensure the rank of $\Delta W$ increasing.
More details about ReMoRA can be found in Appendix B.
For the hyperparameters corresponding to ReLoRA and ReMoRA, we merge every 2k steps and use 50 steps restarts warmup with optimizer reseting and jagged scheduler.

We show pretraining loss in Figure 4 and corresponding perplexity on C4 validation dataset in Table 3.
Our method show better performance on pretraining compared to LoRA and ReLoRA with same amount of trainable parameters.
Benefiting from high-rank updating, ReMoRA also achieves more improvements on MoRA compared to ReLoRA, which
demonstrates the effectiveness of merge-and-reint strategy in ReMoRA.

## 6 Analysis

Figure: Figure 5: The number of singular values $\textgreater 0.1$ in $\Delta W$ on the 250M pretraining model.
Refer to caption: extracted/5606540/svd.png

### 6.1 High-rank Updating

To demonstrate the impact of high-rank updating on the rank of $\Delta W$, we analyzed the spectrum of singular values for the learned $\Delta W$ on 250M pretraining 250M model.
We present the average count of singular values exceeding 0.1 across all layers for $\Delta W_{q}$, $\Delta W_{k}$, $\Delta W_{v}$, $\Delta W_{o}$, $\Delta W_{up}$, $\Delta W_{down}$, and $\Delta W_{gate}$ in Figure 5 following Lialin et al. ().
MoRA and ReMoRA exhibit a substantially higher number of significant singular values compared to LoRA and ReLoRA, highlighting the effectiveness of our methods in increasing the rank of $\Delta W$. We find the quantity of singular values shown in Figure 5 can be correlated with the perplexity metrics listed in Table 3. Moreover, MoRA, without merge-and-reint strategy in ReLoRA and ReMoRA, can achieve a lower perplexity than ReLoRA along with a higher significant singular values.

### 6.2 Influence of Decompression and Compression

To explore the impact of decompression and compression functions in MoRA, we report the performance on GSM8K using various methods: truncation, sharing, decoupling, and rotation in Table 4.
Among these methods, truncation shows the worst performance due to the significant information loss during compression.
Sharing can achieve better performance than truncation by leveraging the shared rows or columns to preserve the input information.
But in the case of $r=8$, sharing shows worse performance than decouple and rotation due to the large number of sharing rows or columns, as we discussed in Section 4.
Rotation is more efficient than decouple, due to the rotation information can help the square matrix to distinguish the input information.

**Table 4: Influence of decompression and compression functions on $r=8$ and $r=256$ on GSM8K.**
|  | $f_{comp}$, $f_{decomp}$ | $r=8$ | $r=256$ |
| --- | --- | --- | --- |
| Truncation | Eq. 4 | 59.5 | 66.6 |
| Sharing | Eq. 6 | 62.5 | 67.9 |
| Decouple | Eq. 8 | 63.6 | 67.8 |
| Rotation | Eq. 9 | 64.2 | 67.9 |

## 7 Conclusion

In this paper, we analyze the impact of low-rank updating through LoRA and observe that such updating struggles for memory-intensive tasks, which also limits current LoRA variants. To overcome this limitation, we introduce MoRA, a method that utilizes non-parameterized operators for high-rank updating. Within the MoRA framework, we explore various methods to implement decompression and compression functions. Performance comparisons indicate that MoRA matches LoRA in instruction tuning and mathematical reasoning, and exhibits superior performance in continual pretraining and memory tasks. Additionally, we conduct pretraining experiments to further demonstrate the effectiveness of high-rank updating and show superior results compared to ReLoRA.

## Appendix A Hyperparameters

We propose hyperparameters in Table 5.

**Table 5: Hyperparameters for fine-tuning on three datasets.**
| Dataset | Method | $r$ | $\alpha$ | LR | LR Scheduler | Warmup | Epochs | Batch size | $f_{\text{comp}}$, $f_{\text{decomp}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Tülu v2 | FFT | - | - | 2e-5 | cosine | 500 | 2 | 128 | - |
| LoRA-like | 8 | 16 | $\{$1e-4,2e-4$\}$ | cosine | 500 | 2 | 128 | - |  |
| MoRA | 8 | - | $\{$2e-4,3e-4$\}$ | cosine | 500 | 2 | 128 | Eq. 9 |  |
| LoRA-like | 256 | 128 | $\{$1e-4,2e-4$\}$ | cosine | 500 | 2 | 128 | - |  |
| MoRA | 256 | - | $\{$3e-5,5e-5$\}$ | cosine | 500 | 2 | 128 | Eq. 6 |  |
| MetaMath | FFT | - | - | 2e-5 | cosine | 300 | 3 | 128 | - |
| LoRA-like | 8 | 16 | $\{$1e-4,2e-4$\}$ | cosine | 300 | 3 | 128 | - |  |
| MoRA | 8 | - | $\{$2e-4,3e-4$\}$ | cosine | 300 | 3 | 128 | Eq. 9 |  |
| LoRA-like | 256 | 128 | $\{$1e-4,2e-4$\}$ | cosine | 300 | 3 | 128 | - |  |
| MoRA | 256 | - | $\{$3e-5,5e-5$\}$ | cosine | 300 | 3 | 128 | Eq. 6 |  |
| BioMed./Fiance | FFT | - | - | 3e-5 | linear | 150 | 3 | 128 | - |
| LoRA-like | 8 | 16 | $\{$3e-4,4e-4$\}$ | linear | 150 | 3 | 128 | - |  |
| MoRA | 8 | - | $\{$4e-4,5e-4$\}$ | linear | 150 | 3 | 128 | Eq. 9 |  |
| LoRA-like | 256 | 128 | $\{$3e-4,4e-4$\}$ | linear | 150 | 3 | 128 | - |  |
| MoRA | 256 | - | $\{$5e-5,7e-5$\}$ | linear | 150 | 3 | 128 | Eq. 6 |  |

## Appendix B Implementation of ReMoRA

We introduce detial implementation of ReMoRA in pretraining.
In this case, we simply define two kinds of $g$.
The first kind is grouping every adjacent $\hat{r}$ rows or columns together following the defined in fine-tuning, the first groups can be represented as $\{1,2,\ldots,\hat{r}\}$.
The second kind is grouping every neighboring $k$ of the rows or columns together, the first groups can be represented as $\{1,1+k,\ldots,1+\hat{r}k\}$.
We propose a example code about compression and decompression functions in Algorithm 1 and 2.
After merging, we can change the group type from 0 0 to $1$ or $1$ to 0 0.

Figure: Algorithm 1 Compression

Figure: Algorithm 2 Decompression

## Appendix C Downstream Tasks of Continual Pretraining

For biomedcine, we use PubMedQA Jin et al. (), RCT Dernoncourt and Lee (), USMLE Jin et al. (), and selecting biomedicine subjects from MMLU to evaluate the performance.
For finance, following BloombergGPT Wu et al. (),we use ConvFinQA Chen et al. (), NER Salinas Alvarado et al. (), Headline Sinha and Khandait (), FiQA SA Maia et al. () and FPB Malo et al. ().
We report the detail performance of these tasks following:

**Table 6: Performance on biomedical tasks.**
|  | $r$ | PubMedQA | USMLE | BioMMLU | RCT | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| FFT | - | 74.1 | 41.2 | 47.5 | 62.7 | 56.4 |
| LoRA | 8 | 73.1 | 34.9 | 45.3 | 54.9 | 51.9 |
| MoRA | 8 | 73.3 | 34.7 | 45.3 | 59.9 | 53.3 |
| LoRA | 256 | 73.8 | 39.7 | 46.0 | 56.9 | 54.1 |
| MoRA | 256 | 74.4 | 40.4 | 46.1 | 60.6 | 55.4 |

**Table 7: Performance on finicial tasks.**
|  | $r$ | ConvFinQA | FiQA SA | Headline | NER | FPB | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FFT | - | 44.4 | 78.8 | 82.3 | 68.1 | 74.3 | 69.6 |
| LoRA | 8 | 44.5 | 76.2 | 72.4 | 61.6 | 65.1 | 64.0 |
| MoRA | 8 | 45.8 | 76.6 | 76.3 | 68.9 | 68.2 | 67.1 |
| LoRA | 256 | 41.4 | 78.3 | 83.0 | 66.8 | 66.7 | 67.3 |
| MoRA | 256 | 47.7 | 76.3 | 83.4 | 68.0 | 68.1 | 68.7 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Why does MoRA rank 8 converge but LoRA rank 8 does not converge?

Important that the comparison with LoRA is with the same number of trainable parameters. How many total extra parameters does MoRA require, including the frozen compression and decompression stages?
soundness: 4
strengths: The authors motivate their approach by a clever memorization task. They show that vanilla LoRA with low rank (e.g. rank 8) does poorly on a contrived memorization task. They train on 10k unique IDs for 100 epochs where the LLM has to memorize the value associated with a key. This is a nice way to motivate the rest of the results.

The authors do rigorous benchmarking across training approaches, from instruction finetuning (IFT) and continual pretraining to pretraining from scratch. This is not so common among LoRA-style papers. The authors themselves bring up a valid critique that many LoRA-related papers focus on BERT style models and GLUE, which is not so relevant in the age of open source LLMs like Llama.

Very helpful to show the various effects of decompression approaches (Table 4)
Important to show the effect of training speed and memory usage - many LoRA studies leave this out.

This paper includes a final comparison of training latency and memory usage. Their method shows improvement over LoRA for the same number of trainable parameters, and is only 1.15x slower during training in certain settings.
summary: This paper develops an alternative to LoRA that uses a square matrix to achieve a high rank update for an equivalent number of trainable parameters. They show that their approach is competitive with LoRA using Llama 2 7B across various training regimes (and datasets), spanning instruction finetuning, continual pretraining, and even pretraining from scratch. This method seems quite promising and is grounded in strong empirical results.
weaknesses: 1. The authors note that “increasing rank alleviates this problem” of memorizing UUID pairs. Although MoRA converges faster than LoRA at rank 256, they both end up converging. What are the benefits of using MoRA  rank 8 vs. LoRA rank 256? According to Table 2, they get roughly the same performance. If MoRA is a strict pareto improvement over LoRA for the same number of parameters, the paper should state so unambiguously (the paper states this in a few separate places for specific tasks). If there are any caveats these should be stated clearly as well.

2. There is one recent study that is quite similar to MoRA: LoRA-XS (first appeared May 2024) “LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters” (Bałazy et al.) https://arxiv.org/abs/2405.17604. This is very similar to the approach taken by MoRA, and it should be cited as contemporaneous work. Another related (but less similar) study is “VeRA: Vector-Based Random Matrix Adaptation” (Kopiczko et al. 2023) https://arxiv.org/pdf/2310.11454

3. For different datasets, it would be very helpful to compare the total number of samples and tokens. The authors make a distinction between IFT with few samples (e.g. LIMA) and more samples, but don’t give enough detail to compare continual pretraining. How large are the biomedicine and finance corpora? The LoRA Learns Less and Forgets Less paper (https://arxiv.org/abs/2405.09673) makes a distinction between IFT and continual pretraining based on the total number of tokens (they find LoRA with rank 256 does worse than full finetuning when trained on >5B tokens). Are these data regimes similar?


### Minor comments:
What is the hardware use for the speed/memory usage? This would be informative for Table 5. Otherwise it is somewhat meaningless
The authors don’t define continual pretraining; presumably this means that the corpus is used to train with next token prediction.
A handful of paragraphs suffer from poor/confusing grammar

Typos:

130: Compare → Compared

200: f_comp → f_decomp

256:use → uses

321: adapt to the → adapt to 

342: 5.2.2 Baselines and Implements → Implementations

350: we search

355: The details parameters

362: grammar

369: grammar

399: leave modules layernorm → leave layernorm modules

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In sec 5.1, the search range of learning rate is from {5e-5,7e-5,1e-4}, does the results in Fig 3 are under the same learning rate setting?
2. Why are the baseline methods included in Section 5.2 excluded from Sections 5.1 and 5.3?
soundness: 3
strengths: 1. The paper provides a detailed analysis of the limitations of LoRA in tasks that require memorizing new knowledge.
2. The proposed high-rank updating method shows superior performance in memorization tasks compared to standard LoRA.
summary: This paper proposes a new parameter-efficient fine-tuning method, namely MoRA, designed to achieve high-rank updates to enhance the performance of large language models (LLMs) in memory-intensive tasks (tasks that require acquiring new knowledge for LLMs).
weaknesses: 1. In sec 5.1 and sec 5.3, MoRA is only compared with standard LoRA, more baselines(e.g., [1][2]) are needed to substantiate your hypothesis that high-rank updating benefits memory-intensive tasks.
2. It appears that the experiments in Section 5.2 are aimed at demonstrating MoRA's superiority in memory-intensive tasks and its comparable performance on other tasks. However, results indicate that MoRA underperforms other baselines in mathematical reasoning, which I consider memory-intensive, as increasing the rank from 8 to 256 significantly improves the performance of both LoRA and MoRA.
3. The writing needs significant refinement:

 *  Line 160: "we employ LLaMA-2 7B as base model, utilizing 1,000 pairs per batch and conducting 100 epochs".
 * Line 165: "we observe low-rank updating are hard to memorizing new knowledge compared to FFT".
 * Line 191: "enhances the output dimension".

[1] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

[2] DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the points outlined in the weaknesses section.
soundness: 3
strengths: 1. This method, MoRA, is well-motivated by identifying weaknesses in LoRA and other low-rank adaptation methods. An interesting and intuitive experiment on memorizing UUID pairs further highlights the necessity of developing methods to address this limitation. 

2. Experimental results on Continual Pretraining demonstrate MoRA's effectiveness in this scenario. 

3. The provided code is also well-written.
summary: This paper proposes a novel PEFT method for LLM, which leverages squared matrix to achieve updates with higher ranks compared to baseline methods such as LoRA. It reduces the input dimension and increases the output dimension to match those in the square matrix with a novel strategy. Experimental results across five downstream tasks show that the proposed method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.
weaknesses: 1. The primary advantage of MoRA lies in tasks requiring significant enhancement of LLM knowledge and capabilities, such as memorizing UUID pairs and continual pretraining (CP). However, it may be more practical to directly use full finetuning in such cases, particularly for CP, which generally demands high computational resources to achieve optimal performance. This raises questions about the suitability of applying PEFT in such high-cost scenarios. As shown in Table 2, MoRA cannot outperform full FT in CP tasks. Additionally, on downstream tasks outside of CP, MoRA does not exhibit a clear advantage over baseline methods and generally results in decreased performance compared with standard LoRA, which limits its practicality.

2. In the final update matrix $\Delta W$ obtained with MoRA, many entries are zero. This could be a drawback, as some of these parameters might be essential for specific downstream tasks, yet MoRA is unable to update them. This limitation potentially restricts MoRA's effectiveness for LLM updates across a diverse range of downstream tasks.

3. The method section could benefit from improved organization, especially as there are no flowcharts or diagrams to visually illustrate the MoRA method. Currently, certain lines, such as the definitions of $f_{comp}$ and $f_{decomp}$ are somewhat confusing and could be clarified for better comprehension.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see above.
soundness: 2
strengths: The authors observe that LoRA’s low-rank updates (e.g., using a rank as low as 8) may not fully leverage the model's learning potential. To address this, they propose restructuring LoRA updates by compressing the input dimension, increasing the rank of the update matrix, and then decompressing the output to match the original dimensions. This approach maintains the same number of trainable parameters while increasing the rank, which is an interesting idea. However, while this method enhances learning capabilities in terms of memorization, the performance improvements on downstream tasks do not directly translate as significantly.
summary: The authors suggest that low-rank updating mechanisms, as employed by LoRA, may limit the ability of large language models (LLMs) to effectively learn and memorize new knowledge, particularly in memory-intensive tasks. To address this limitation, they propose increasing the rank of the update matrix used in LoRA by employing a square matrix with a larger rank, while keeping the total number of trainable parameters constant. To achieve this, the authors introduce non-parametric operations that compress the input and decompress the output, ensuring dimensional compatibility for matrix multiplication. Additionally, they propose a rotation operation to preserve positional encodings during input compression, maintaining compatibility with matrix products.
weaknesses: 1. Unclear Compression and Decompression Operations: The compression and decompression operations described in equations (8) and following are not clearly explained. The rationale behind choosing these specific operations is unclear and requires further elaboration.

2. Rotation Operation: While the rotation operation is likely intended to preserve positional information after compressing the input, the explanation provided in the paper is not sufficiently clear. The authors need to clarify why rotation is added and how it enhances the matrix's ability to distinguish input information. The current description, especially the statement "rotation information can help the square matrix to distinguish the input information," is vague and needs more technical justification.

3. Performance Discrepancy in Table 1: In Table 1, MoRA with rank 8 achieves 100% accuracy, whereas LoRA with the same rank only achieves 52%. This discrepancy suggests that the LoRA model may have been undertrained or improperly tuned. The paper should address this discrepancy to ensure fair comparisons between methods.

4. Limited Improvement Across Tasks: MoRA shows clear improvements only in continual pre-training tasks, with little or no improvement in other downstream tasks. This suggests that MoRA is a task-specific method, which contradicts the goal of parameter-efficient fine-tuning methods that are expected to generalize across various tasks. This task-specific nature makes it challenging to justify its broad application without further experimentation, which defeats the purpose of an efficient fine-tuning method.

5. Uncertain Advantage of Rotation: The paper mentions that rotation helps the square matrix distinguish input information, but this benefit is not clearly demonstrated, particularly in higher-rank scenarios. At rank 256, the performance differences between the Sharing, Decouple, and Rotation methods are minimal. This raises the question of whether rotation is truly necessary at higher ranks, and the paper should clarify under what conditions (if any) rotation provides a real advantage.

6. For table 5, could you also provide the number of trainable parameters for both the methods under both ranks?