ResLoRA: Identity Residual Mapping in Low-Rank Adaption (2024)

Abstract: As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to finetune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at https://github.com/ microsoft/LMOps/tree/main/reslora.

heading: 1 Introduction
text: In recent years, large language models (LLMs) (Naveed et al., 2023) with hundreds of billions of parameters have shown remarkable performance on various tasks. Fine-tuning LLMs on specific datasets typically leads to better performance than merely giving instructions in the prompt during inference(Xu et al., 2023). However, the cost of it is often prohibitive due to the large number of parameters involved.
To address this problem, various parameterefficient fine-tuning (PEFT) methods have been proposed. PEFT methods freeze all parameters in the original model, and only tune a few parameters in the newly added modules. Among them, one of the most popular PEFT methods is LoRA(Hu et al., 2022), which stands for low-rank adaptation. LoRA uses two matrices parallel to the original
∗ Work done during internship at Microsoft. † Corresponding author.
frozen linear layer with few trainable parameters during training, and merges them together during inference. LoRA incurs no cost in terms of time and computation after merging, and has been mathematically proven (Zeng and Lee, 2023) to be effective, so it has a wide range of applications.
The basic LoRA method still has some limitations. Previous studies mainly focused on either dynamically adjusting the rank of LoRA modules in different layers of the model(Zhang et al., 2023a), or using fewer trainable parameters to achieve a similar effect as the original LoRA method(Valipour et al., 2022). However, they overlooked a potential problem: a long backward path hinders the updating of parameters in LoRA blocks.
As a prominent method, ResNet(He et al., 2016a,b) has proven to be widely efficient, and is also used in Transformer models(Vaswani et al., 2017), between different encoder and decoder blocks. Parallel to linears in these blocks, LoRA blocks can also benefit from the original shortcut design. However, unlike linears, LoRA blocks are more fine-grained. One LoRA block only corre-
8870
sponds to one linear, so the original shortcut is not perfectly suitable for LoRA blocks. Let’s use encoders of Transformer as an example. If we add LoRA blocks in query, key, or value linears, the previous gradient must go through the output linear and Softmax function when calculating backward, which may cause gradient vanishing or explosion.
In this paper, we present ResLoRA, a new framework that merges the shortcut of ResNet into LoRA blocks. To validate the efficiency of different residual structures, we propose three residual structures and conduct experiments on different models and tasks. However, the shortcut cannot be directly merged into the original network due to its nonplain structure, which undermines the advantage of LoRA. Considering this, we discuss different merging approaches to convert ResLoRA to the original LoRA blocks, in order to ensure that it can still be merged into the original modules during inference. After merging, our method introduces neither extra parameters nor computational complexity. Compared to LoRA, ResLoRA does not use any additional parameters, but achieves 1% to 20% improvement in performance during inference, and lower and faster convergence of loss during training. Moreover, our method can be easily applied to not only the basic LoRA method, but also the other variants of LoRA. Finally, we evaluate the robustness of the results on different experiments. To the best of our knowledge, ResLoRA is the first work that combines the residual path with the LoRA method. The effectiveness of our method is illustrated in Figure 1.
Overall, our contribution can be summarized as follows:
• We propose ResLoRA, a novel framework that improves LoRA. Compared with the original method, we use residual paths to accelerate the loss reduction process in the training stage, and can usually achieve significant improvements on test datasets.
• We investigate different merge approaches for ResLoRA, which can convert it to LoRA blocks, and finally merge them into the frozen original linear, without adding any cost in the inference stage. Benefits from the merge approaches, ResLoRA can be easily applied to other variants of the LoRA method.
• We evaluate the results in different models and tasks, to validate the robustness of our
improvement. Furthermore, we analyze the reasons why our method can obtain performance gains.

heading: 2 Related Works
text: Parameter-efficient fine-tuning (PEFT) Research on PEFT can be divided into three types. One line of research (Lester et al., 2021; Liu et al., 2023) is to add some special trainable vectors attached to the input sequence, which will increase the length of input and have a gap in results compared to fullfinetune. Another line of research is to add serialized modules in original modules both in training and inference stage, called Adapter(Houlsby et al., 2019; Zhang et al., 2023b,b). In contrast, LoRA method (Hu et al., 2022) adds new low-rank matrices parallel to original linear layer in training stage and merges them into the original model during inference, so there is no extra cost when inferring. Low-rank training method (LoRA) Recent studies on LoRA aim to achieve lower cost and better performance. Some researchers explore more flexible and appropriate ranks, such as DyLoRA(Valipour et al., 2022), ReLoRA(Lialin et al., 2023), LoHA(Hyeon-Woo et al., 2021) and LoKr(Yeh et al., 2023). AdaLoRA(Zhang et al., 2023a) design a method to dynamically allocate the rank of LoRA blocks in different layers based on their importance, which can reduce the unimportant rank of LoRA blocks. Other works focus on the combination of LoRA and other approaches, such as AdaMix(Wang et al., 2022) and QLoRA(Dettmers et al., 2023). Besides, LoRAHub(Huang et al., 2023) and LoRAMoE(Anonymous, 2024) focus on how to merge multiple LoRA blocks that are fine-tuned on different tasks respectively. Despite this, no one has focused on the potential barrier of gradient propagation in LoRA. Residual Network(ResNet) He et al. (2016a) and previous works(Srivastava et al., 2015) first introduced the residual network. This work solves the gradient vanishing or explosion and improves numerical stability during gradient updating. Considering that the extra shortcut path requires extra computational cost, some works(Ding et al., 2021) attempt to remove extra paths in the inference stage. Inspired by them, we first extend the main idea of ResNet to LoRA to achieve a faster and more stable training stage, and then design merging approaches to preserve the plain structure of LoRA, so as to
maintain the advantages of both LoRA and ResNet at the same time.

heading: 3 Method
text: In this section, we introduce our framework, which mainly consists of two parts: (1) ResLoRA blocks, which add various residual paths in LoRA blocks, mainly used in training stage; (2) merging approaches, which remove the residual paths to convert ResLoRA blocks to LoRA blocks, mainly used in inference stage.

heading: 3.1 LoRA Blocks
text: We start by revisiting the LoRA method. For an original matrix of the linear layer from a pre-trained model Wn ∈ Rp×q, where p and q denote the dimensions of output and input, the original equation can be written as hn = Wnxn, where x denotes the input vector, h denotes the output hidden vector, and n denotes the index of the layer. We define a LoRA block as an additional block parallel to an original matrix. A LoRA block contains two new matrices, down-projection A ∈ Rr×p and upprojection B ∈ Rq×r, which aim to decompose the high-rank matrix into the low-rank matrix. During training, the W is frozen and only the weights of A and B are updated; and during inference, the additional parameters are merged into the original parameters by Wn + BnAn, to ensure that no latency is introduced. Therefore, the LoRA method can be expressed as Equation 1:
hn = Wnxn +BnAnxn (1)
Figure 2a illustrates the structure of LoRA blocks. Because r ≪ min(p, q), the number of trainable parameters is significantly lower than full finetuning.

heading: 3.2 ResLoRA Blocks
text: Inspired by ResNet, we introduce residual paths in our method. Considering the different impacts of different structures and mechanisms, we design and implement three types of blocks, named input-shortcut (is), block-shortcut (bs), and middleshortcut (ms), respectively. Figure 2 shows the specific structures of each type. Input-shortcut structure means we directly use a shortcut between the input vectors of different LoRA blocks. Specifically, we add the input vector of the previous LoRA block to that of the current LoRA block. This implementation is inspired by
the original ResNet, which adds the previous input vectors to current input vectors. Unlike the forward path of ResNet, we don’t simply add the input of the LoRA block to the output, because this identity will not only affect the forward path of LoRA blocks, but also the forward path of the original linear layer. As a result, this simple design will create too large values of loss in the forward step to calculate the gradients, thus failing to train LoRA blocks. To avoid this, we only use a shortcut between LoRA blocks. Therefore, the output of a linear layer with the input-shortcut type of ResLoRA can be expressed as:
hn = Wnxn +BnAn(xn + xn−1) (2)
where n ∈ [1, L], and L is the number of layers in original model. If n = 0, we set xn−1 = xn to maintain the same order of magnitude between the first layer and further layers. The overall structure of this type can be seen in Figure 2b. Block-shortcut structure means we add shortcuts not to the input vectors, but to the weights of LoRA blocks. Although the input-shortcut structure implements the idea of residual paths, the extra forward path makes it impossible to convert to the original LoRA blocks directly, and merging approaches are required, which may incur performance losses. To obtain the advantages of both the LoRA method and the residual network, we design the block-shortcut structure, which is similar to the DenseNet structure(Huang et al., 2017). For the input vectors in the current layer, we use the current LoRA block and several previous LoRA blocks simultaneously to participate in the calculation. This forward path does not add any extra forward path for the input, but allows direct transfer of gradients to skip the middle layers, which can also reduce possible obstacles in backward calculation. The output of this structure can be expressed as follows:
hn = Wnxn + ( ∑m k=0Bn−kAn−k)xn (3)
where m ∈ [1, L] denotes the number of previous LoRA blocks to use. In the specific implementation, we set a hyper-parameter pre_num to control m. Besides, for each layer we set m = min(m,n − 1) to avoid out-of-index errors. Different values of m yield different results, and the details are presented in Section 4.5. The overall structure of this type is illustrated in Figure 2c.
Middle-shortcut structure means we add shortcuts to the intermediate results of LoRA blocks. LoRA blocks contain two matrices, A and B, and A is always closer to the input vectors than B. Because of the pre-existing shortcuts between Transformer blocks, A matrices are more likely to benefit from these shortcuts, and are less likely to encounter problems with gradient propagation. Hence we try a new structure so that B matrices can also benefit from the shortcuts. For each layer, the middle-shortcut structure ignores the modification of A matrices, and focuses on the shortcuts of B matrices: we add the previous output vectors of A matrices to the current output vector of A matrix, and then the sum is transferred to the current B matrix. In summary, the process can be expressed as:
hn = Wnxn +Bn( ∑m k=0An−kxn−k) (4)
Similar to the block-shortcut structure, this structure also uses a hyper-parameter pre_num to control the value of m. The overall structure of this type can be seen in Figure 2d.
Different types of structures attempt to add shortcuts to different positions, including input vectors, weights of LoRA blocks, and middle vectors, to find the optimal one. The main idea of these structures is the same as ResNet: use shortcuts to reduce the number of modules passed through in backward steps. In the following sections, we use ResLoRAis for the input-shortcut type of ResLoRA, ResLoRAbs for the block-shortcut type, and ResLoRAms for the middle-shortcut type.

heading: 3.3 Merging Approaches
text: While the additional shortcut brings benefits in ResLoRA, there are several issues. One of the most important issues is that a no-plain structure was created. The original LoRA blocks, which we call plain structure, can directly merge into the linear layer, because LoRA doesn’t require an extra forward path independent of the original layer. In other words, the forward path of LoRA blocks is similar to the original linear layers. However, ResLoRA uses an additional shortcut between ResLoRA blocks of different layers, which is not the same as the original forward path. Therefore, we need to design merging approaches to convert ResLoRA blocks to LoRA blocks.
How can we convert it? For the block-shortcut structure, the current ResLoRA blocks only require the weights of the previous ResLoRA blocks, and there is no extra forward path, so we can easily merge ResLoRA as follows:
W ∗n = Wn + ∑m k=0An−kBn−k (5)
where W ∗n denotes the weight of the linear layer after merging during inference. However, for the other two structures, direct merging is impossible, because the current ResLoRA blocks require input vectors from previous layers, which create an extra forward path and are different from the original linear layers. Suppose that we can express xn−1 = αxn, where α denotes a scaling factor. The previous input vectors can be easily converted to the current input vector, and the ResLoRA blocks can be converted to the LoRA blocks. For different xn−1, there is a different α that satisfies
xn−1 = αxn, so we cannot obtain a precise α. Our goal is to find an α∗ that satisfies xn−1 ≈ α∗xn. For example, for the input-shortcut structure, we can derive the following formula:
hn = Wnxn +BnAn(xn + xn−1)
≈ Wnxn +BnAn(xn + α∗xn) = Wnxn + (1 + α ∗)BnAnxn
(6)
Therefore, new weights of the current linear layer can be expressed as:
W ∗n = Wn + (1 + α ∗)BnAn (7)
The precision of α∗ is crucial for model inference because this factor directly determines whether the model merging is correct. Since the Frobenius norm, one of the most common matrix norms, can generally measure the size of a matrix(Ford, 2014), we design two approaches to estimate the value of α∗ using the Frobenius norm. Merge Based on Input. One approach is to directly calculate α∗ based on xn and xn−1. In the training stage, for each layer we use a sliding window to collect the most recent input vectors xn. After that in the inference stage, we calculate the Frobenius norms of all input vectors, and get the average of the Frobenius norms in each sliding window, where fn denotes the average of Frobenius norms in the n-th layer. We think of this number as representing the size of input vectors, and this mathematical relationship can be expressed as:
xn fn ≈ xn−1 fn−1
(8)
Based on this, we can get α∗ to be:
α∗ = fn−1 fn
(9)
Merge Based on Weights of ResLoRA Blocks. Another approach is to get α∗ based on the weights of previous ResLoRA blocks rather than input vectors. If we approximate the difference of functions between xn−1 and hn−2, we can assume that hn−2 is the representation of xn−1 on the orders of magnitude. Furthermore, if we only focus on the effect of weights and neglect the effect of input vectors, we can get the Frobenius norms of previous weights to represent the xn−1 on the orders of magnitude, where f∗n−2 denotes the Frobenius norms of weights of W ∗n−1, which corresponds to the linear
layer after merging. Therefore, this relationship between xn and xn−1 can be expressed as:
xn xn−1 ≈ f ∗ n−1 f∗n−2
(10)
Based on this, we can get α∗ to be:
α∗ = f∗n−2 f∗n−1
(11)
For ResLoRAms, we simply modify those merging approaches to adapt to the new structure. For merge based on input, we compute α∗ by:
α∗ = ∑m
k=1 α ∗ n−k (12)
where α∗n−k = fn−k/fn, and fn means the Frobenius norm of weights of Anxn. For merge based on weights of previous ResLoRA blocks, we calculate α in the same way and then merge A as:
A∗n = An + ∑m k=1 α ∗ n−kAn−k (13)
where A∗n denotes A after merging, and αn−1 denotes fn−1/fn.
By using these approaches, the ResLoRA blocks can be converted to LoRA blocks, which means no latency will appear in the inference stage. In what follows, we use mergebi for the merge based on input, and mergebw for the merge based on weights of ResLoRA blocks.

heading: 3.4 Mathematics Analyses
text: Although our method can intuitively solve the potential problem in the backward pass and accelerate the model training process, can we prove it mathematically? For simplicity, we choose the inputshortcut structure as an example. For a specific input x, if we want to update the weight of Bn−2 where n denotes the index of the layer, the gradient can be computed as follows:
∂L ∂Bn−2 = ∂L ∂hn ∂hn ∂xn−1 ∂xn−1 ∂Bn−2
(14)
where L is the value of the loss function for this input. For LoRA blocks, the sub-equation is:
∂hn ∂xn−1 = ∂hn ∂xn ∂xn ∂xn−1
= ∂(Wnxn +BnAnxn)
∂xn
∂xn ∂xn−1
(15)
However, for the input-shortcut structure of ResLoRA, the sub-equation is:
∂hn ∂xn−1 = ∂hn ∂xn ∂xn ∂xn−1
= ∂(Wnxn +BnAnxn)
∂xn
∂xn ∂xn−1
+ ∂(BnAnxn−1)
∂xn−1
(16)
Comparing Equation 16 with Equation 15, there is no factor before the extra term in Equation 16, so the gradient can avoid potential vanishing or explosion problems that appear in the factor ∂xn
∂xn−1 . Therefore, the training stage can benefit from ResLoRA blocks.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: To evaluate the effectiveness of ResLoRA, we conduct experiments on a wide range of models and tasks, including natural language generation (NLG), natural language understanding (NLU), and image generation. We present the details of the tasks in the following subsections respectively.
We compare our method with LoRA(Hu et al., 2022), AdaLoRA(Zhang et al., 2023a), LoHA(Hyeon-Woo et al., 2021) and LoKr(Yeh et al., 2023), which we detailedly describe in Section A. All details of experiments can be found in Section C.

heading: 4.2 Natural Language Generating
text: Models and Datasets. Considering that LoRA has been mainly used in LLMs recently, we choose LLaMA2-7B(Touvron et al., 2023), a popular opensource LLM, as the NLG model. We conduct experiments on five tasks, including mathematical and commonsense reasoning, which are the primary benchmarks to evaluate the general ability of LLMs. A summary of the datasets is presented in Section B. Main Results. We compare our method with various baseline methods. Table 1 shows the results of different tasks and methods. LoRAr=16 shows significant improvement over LoRAr=4 in all tasks, which means that a higher value of rank is effective on these tasks. Among the baseline methods, the original LoRA method performs the best, which may be because the variant methods of LoRA introduce several hyper-parameters. Compared with LoRA, ResLoRAis and ResLoRAbs perform better
on almost all tasks. For example, all three types of ResLoRA achieve higher accuracy on HellaSwag, which is 10.98%, 20.77%, and 36.85% higher than LoRA respectively.

heading: 4.3 Natural Language Understanding
text: Models and Datasets. We evaluate the proposed methods with RoBERTa-large(Liu et al., 2019) on the General Language Understanding Evaluation (GLUE, Wang et al. (2018)) benchmark, where the model and datasets are the same as Hu et al. (2022). RoBERTa is a competitive pre-trained model improved from BERT(Devlin et al., 2018). GLUE contains different types of tasks, and is widely used to evaluate the NLU ability of models. Main Results. Table 2 shows the results of NLU tasks. The conclusions are similar to those of NLG tasks. LoRAr=16 shows significant improvement over LoRAr=4 in all tasks, and LoRAr=4 shows the best results among all baseline methods. Compared with the baseline methods, our method shows significant improvement on almost all tasks. For all tasks except WNLI, our method demonstrates different degrees of performance enhancement. These experiments verify the general applicability of our method to the NLU tasks.

heading: 4.4 Text to Image
text: Models and Datasets. To verify the generalization of our method, we also conduct experiments on multi-modal tasks. We select the text-to-image task, which aims to generate the appropriate images based on input texts. The model we use is the popular Stable-Diffusion-v2(Rombach et al., 2022), one of the most popular image generation models. The dataset we select is Pinkney (2022), which contains images of a cartoon style. Our goal is
to let the model learn this cartoon style, which is greatly different from its original style. Main Results. Figure 3 shows the results of textto-image task. We use two prompts to generate images that are trained by LoRA and ResLoRAis respectively, and save the images from the training process. For both groups of images, ResLoRAis clearly generates more appropriate images in the later step. In Figure 3b, the 200-step result of ResLoRAis is already a vivid cartoon cat with legs, a tail, and clothes, whereas the result of the same step in LoRA is still an incomplete character. In Figure 3c, the 140-step result of ResLoRAis has been converted to the Pokemon style, while the result of LoRA is still a realistic bird, which is not our goal. In short, ResLoRAis shows better results and a faster training process for this task.

heading: 4.5 Ablation studies
text: Merge Approach. In Section 3.3, we propose two merge approaches to convert ResLoRA blocks to LoRA blocks, to avoid the extra cost in the inference stage. However, our approaches introduce unavoidable accuracy degradation. Nevertheless, our approaches achieve higher accuracy than LoRA and other variants.
Table 3 shows the original accuracy before merging, and the results after merging. The huge gap between the results before merging and the results with no merging approach confirms our point: merging approaches are essential for ResLoRA, because the additional residual path is introduced in the training stage, which causes the difference between training and inference. In spite of the different calculations, the results of the two approaches are similar in ResLoRAis. Our two merging approaches both have an accuracy degradation of about 1% compared to the results before merging, and have an accuracy improvement of about 10% compared to the results with no merging.
However, for ResLoRAms there is a large gap
between the results with and without merging. This may be because of more error accumulation and greater difficulty when merging more previous blocks. Therefore, it is worthwhile to continue to improve merging approaches. For the two modified approaches, mergebw shows worse results than mergebi. More detailed results in different datasets can be found in Section D. The Number of Previous ResLoRA Blocks. In ResLoRAbs and ResLoRAms, we intend to use not only one adjacent previous block but also multiple blocks to calculate the result. To verify the effectiveness of different numbers of previous blocks, we compare their results in Table 4. From the table, we observe that the value of pre_num affects the result significantly. Too large or too small a value is harmful, and the best results occur with a proper value of pre_num. Besides, the results show that it is workable to incorporate more previous blocks because more potential obstacles are skipped when training. The results of ResLoRAms are similar to those of ResLoRAbs.

heading: 4.6 Analysis
text: Extra residual paths can accelerate the training stage. As we showed in Section 3.4, extra residual paths can speed up the training stage and achieve better fitness of the models. But is that
really true? To verify this conclusion, we collect the loss curves.
Figure 4 shows the partial process of the loss when training with LoRA and ResLoRAbs with different pre_num, which denotes the number of previous ResLoRA blocks. The original LoRA method can be considered as ResLoRAbs with pre_num = 0. As we can see in the figure, the LoRA method has the largest loss and the ResLoRA with pre_num = −1 has the smallest loss. Moreover, as pre_num increases, the loss decreases faster and finally reaches the lower value. Extra residual paths produce more complex weights of LoRA matrices. Despite the mathematical reasoning in Section 3.4 that proves faster convergence when training, we still don’t know what exactly causes the higher accuracy. To explain this result, we save the model weights after sufficiently fine-tuning it, and compare the difference of trained matrices between the two methods. The result is displayed in Figure 5.
Considering the high computational complexity
of large matrices, we use the Frobenius norm to measure their complexity. In the figure, we subtract the F-norm of each LoRA block from each merged ResLoRAbs block, and show this difference through the heat map. Apparently, ResLoRA blocks contain elements with larger absolute values than LoRA blocks, which implies that the blocks can be trained more adequately using ResLoRA. This may be one of the reasons for the better performance of our method. More detailed results can be found in Section E.
Training cost analysis. Compared to LoRA, one of the disadvantages of ResLoRA is the extra time cost introduced during training, because the outputs of the previous layers are also involved in the computation. The results of the comparison can be found in Section F.
In fact, for each task we experimented on, we have trained for a sufficient number of epochs to ensure that the model has converged completely, and finally reported the best results among all epochs. This ensures that the results we report for LoRA are already the best results and more training time does not lead to better results. More details has been shown in Section F.

heading: 5 Conclusion
text: We develop a new improved framework ResLoRA for low-rank adaptation (LoRA). ResLoRA adds residual paths during training, and uses merging approaches to remove these paths during inference. Without any extra trainable parameters, ResLoRA can achieve better results in fewer training steps compared to the original LoRA and other baseline methods. We conduct experiments with three types of ResLoRA structures on NLG, NLU, and textto-image tasks, and the results of almost all tasks verify the effectiveness of our method.
Limitations
In this section, we discuss some limitations of our method and provide some suggestions for future work.
1. Despite adding no extra trainable parameters, the training cost of our method is higher than that of standard LoRA, because ResLoRA needs to use previous blocks when computing in one block. The more previous blocks are used, the higher the cost becomes. Therefore, it is important to balance the training cost and the performance.
2. During inference, we apply merging approaches to remove the extra residual paths. However, none of the merging approaches we proposed can achieve lossless merging, which compromises the final performance of the model. Designing more efficient approaches is desirable.
3. Our work is the first to combine residual paths with LoRA. Prior to this, many valuable works have been proposed, such as Zhang et al. (2023a); Dettmers et al. (2023); Lialin et al. (2023). Since there is no fundamental conflict between our ResLoRA and other methods, it is feasible to integrate these methods. We leave these investigations for future work.
Ethics Statement
This paper proposes a more efficient framework based on LoRA, which is helpful for customizing training models with low resources. We firmly believe that LLMs are not only a potential area of NLP, but also significant for all areas of artificial intelligence. Therefore, it is important to fine-tune a model efficiently. Although fine-tuning is a dualuse technology, we believe in the positive impact of our method.

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62276017, U1636211, 61672081), and the Fund of the State Key Laboratory of Software Development Environment (Grant No. SKLSDE-2021ZX-18).

heading: A Baseline
text: We compare our ResLoRA with the following methods:
• LoRA(Hu et al., 2022) is what we based on. It’s popular for its low memory requirement in the training stage and no latency in the inference stage.
• AdaLoRA(Zhang et al., 2023a) is an important variant of LoRA, which can adjust more ranks dynamically in important layers and modules in the training stage.
• LoHA(Hyeon-Woo et al., 2021) is another popular variant of LoRA, which uses the Hadamard product of two low-rank matrices to obtain a high-rank matrix, to perform the high-rank updating.
• LoKr(Yeh et al., 2023) is a method similar to LoHA. The difference is that LoKr replaces the Hadamard product in LoHA with the Kronecker product.

heading: B Details of NLG datasets
text: For mathematical tasks, we select four datasets. GSM8K(Cobbe et al., 2021) is a high-quality linguistically diverse dataset of grade school math word problems, which is widely used in open leaderboards. SVAMP(Patel et al., 2021) contains simple math word problems created by applying carefully chosen variations to examples sampled from existing datasets. MathQA(Amini et al., 2019) is an advanced dataset gathered by using a new representation language to annotate the AQuARAT dataset(Ling et al., 2017) with fully specified operational programs. MetaMathQA(Yu et al., 2023) uses LLMs to rewrite the question from multiple perspectives based on GSM8K(Cobbe et al., 2021) and MATH(Saxton et al., 2019).
For commonsense tasks, we select HellaSwag. HellaSwag(Zellers et al., 2019) is a challenging dataset, which contains questions to select the best endings to complete sentences. It has been considered as one of the most common datasets to judge the reasoning ability of LLMs.
We demonstrate the details of our NLG datasets in Table 5.

heading: C Experiments Details
text: We use the public Pytorch(Paszke et al., 2019) and Huggingface Transformers(Wolf et al., 2019)
libraries to implement the code of all methods, and conduct all the experiments using Tesla V100 GPUs. For all structures of the ResLoRA method, we set the input vectors of previous blocks to be none for the blocks that don’t have a previous one, such as blocks in the first layer of the model. The details of different models are presented respectively. For other baseline methods, we implement them via the PEFT(Mangrulkar et al., 2022) package from Huggingface. We only consider the linear layers as the target modules. Implementation Details of NLG Experiments. To reduce the GPU memory usage when training the LLM, we use the ZeRO-2 stage(Rajbhandari et al., 2020) to offload unnecessary parameters from GPU to CPU via Deepspeed(Rasley et al., 2020). For all experiments except LoRAr=16, we set the rank r = 4, the alpha α = 8, and the target module as query Wq and value Wv following the setup of Hu et al. (2022). For MetaMathQA, we randomly select 40k samples to train and test in the test dataset of GSM8K. Moreover, to let the LLM understand and follow the format of inputs and outputs, we use simple prompts for each task.
For all datasets, we report accuracy as the evaluation metric, which is higher the better. We report the results for ResLoRAis with the mergebw, for ResLoRAbs with pre_num = 4, and for ResLoRAms with pre_num = 4 after merging with the mergebi. More details can be found in our code. Implementation Details of NLU Experiments. We use the same baseline methods and report results with the same hyper-parameters in our method. The only difference is that there is no need to use Deepspeed because RoBERTa-large is not an LLM.
We report the F1 score for MRPC, the Matthew’s correlation coefficient for CoLA, the Pearson correlation for STS-B, and the accuracy for others. For all metrics, higher is better. We report the results for ResLoRAis with the mergebw, for ResLoRAbs with pre_num = 4, and for ResLoRAms with pre_num = 4 after merging with the mergebi. Implementation Details of Text-to-image Experiments. There are three parts in Stable-Diffusion, including the text encoder, the variational autoencoder(Kingma and Welling, 2013) and the UNet(Ronneberger et al., 2015). We apply our methods to U-Net. Considering the different sizes of matrices in adjacent layers in U-Net, we add LoRA blocks in all of {Wq,Wk,Wv,Wo}, but only en-
able residual path in blocks that have the same size as neighboring previous blocks. Moreover, we set the rank r = 16 and the alpha α = 32 to produce more prominent results.

heading: D Results for Merging Approach
text: We show the detailed results of merging experiments. Results of ResLoRAis are in Table 6, and the results of ResLoRAms are in Table 7. There is no need to apply extra merging approaches for ResLoRAbs.

heading: E Results of Analysis
text: We show the full version of Figure 4 in Figure 6, and the full version of Figure 5 in Figure 7.

heading: F Detailed comparison of training costs
text: We show the comparison of training cost between LoRA and ResLoRA in detail. From Table 8 and Table 9, it is obvious that in the last 10 epochs, all methods’ accuracy remains unchanged. This indicates that the trainable parameters of all methods have converged completely, and more training time for LoRA does not lead to better results.

heading: Future Work
text: Limitations
In this section, we discuss some limitations of our method and provide some suggestions for future work. 1. Despite adding no extra trainable parameters, the training cost of our method is higher than that of standard LoRA, because ResLoRA needs to use previous blocks when computing in one block. The more previous blocks are used, the higher the cost becomes. Therefore, it is important to balance the training cost and the performance. 2. During inference, we apply merging approaches to remove the extra residual paths. However, none of the merging approaches we proposed can achieve lossless merging, which compromises the final performance of the model. Designing more efficient approaches is desirable. 3. Our work is the first to combine residual paths with LoRA. Prior to this, many valuable works have been proposed, such as Zhang et al. (2023a); Dettmers et al. (2023); Lialin et al. (2023). Since there is no fundamental conflict between our ResLoRA and other methods, it is feasible to integrate these methods. We leave these investigations for future work