LoRA Learns Less and Forgets Less
Authors: Dan Biderman
Jacob Portes
Jose Javier Gonzalez Ortiz
Mansheej Paul
Philip Greengard
Connor Jennings
Daniel King
Sam Havens
Vitaliy Chiley
Jonathan Frankle
Cody Blakeney
John Patrick Cunningham
Venue: Accepted by TMLR ()

Abstract: Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning ($\approx$100K prompt-response pairs) and continued pretraining ($\approx$20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100$\times$ greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.

Published in Transactions on Machine Learning Research (08/2024)
LoRA Learns Less and Forgets Less
Dan Biderman 1.2, Jacob Portes?, Jose Javier Gonzalez Ortiz?, Mansheej Paul', Philip Greengard', Connor Jennings", Daniel King", Sam Havens?, Vitaliy Chiley?, Jonathan Frankle', Cody Blakeney", John P. Cunningham'
'Columbia University (db3236, pg2118, jpc2181) (pe2l81)@columiaa.ch @columbia.cdu
Databricks Mosaic Research (jacob.portes, j-gonzalez, mansheej.paul connor.jennings, daniel.king, hamehavens, vitaliy.chiley, jfrankle, sanaknncy/dtaabbbics..cm
Reviewed on OpenReview: http:://ppaarriieeeeee////e/ttep
Abstract
Low-Rank Adaptation (LoRA) is widely-used parameter-efficiett finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finctuning (=100K prompt-respones pairs) and continued pretraining (==20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's 's performance tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100x greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.
1 Introduction
Fineluning large language models (LLMs) with billions of weights requires non-trivial amount of GPU memory. Parameter-efficient finetuning methods reduce the memory footprint during training by freezing a pretrained LLM and only training a small number of additional parameters, often called adapters. Low-Rank Adaptation (LoRA; Hu et al. (221h) trains adapters that are low-rauk perturbations to selected weight, matrices.
LoRA is widely adopted for finctuning LLMs under hardware constraints, but the jury is still out whether it compromises performance compared to full finetuning. The two seminal methods papers the topic, introducing LoRA (Hu et al. 2021 and its more recent combination with model quantization (QLoRA; Dettmers et al. (2024)) reported that LoRA performs better or equivalent to full finetuning. More empirical work (Ghoss et al. 2024 Zhao et al. 2024b) reaches a similar conclusion; and this sentiment echoed in an array of industry blog posts as well (e.g.) Raschka (2023); Niederfahrenhors al. (2023)). At the same time, there is evidence that LoRA underperforms full finctuning (Ivison al.J [023) Zhuo al. 2024), and the need to improve upon LoRA has led to the development of enhanced LoRA variants (Hayou et al. 2024
Meng et al. 2024 Li et al. 2023b Shi et al. 2024) or alternative low-rank approximation methods (e.g Liu et al. (2024) Zhao et al. (2024a)). ). To shed light on this ongoing debate, we ask: under which conditions
does LoRA approximate full finetuning accuracy on challenging target domains, such as code
and math?
By training fewer parameters, LoRA is hypothesizod to constrain the finetuned model from diverging significantly from the base model (Sun al. 2023 Du al.] (204)). This potential characteristic is particularly helpful for LLM finetuning, form of continual learning where specializing in new domains

---

Published in Transactions on Machine Learning Research (08/2024)
come at the expense of base model capabilities (Wang et al. 2024 (a phenomenon known its extreme form as "catastrophic forgetting" |MoCloskey & Cohen| (1989) French (999)) To date, only a few studies have examined forgetting in modern LLMs (Kleiman et al. 2023 Kalajdzievski 2024 Vu et al.) 2022). To address this gap, we also ask: when performing continual learning on a new domain, to what extent docs LoRA mitigate forgetting of base model capabilities?
In this study, we compare LoRA and full finetuning for Llama-2 7B models across two challenging target domains, code and mathematics. Within each domain, explore two training regimes. The first regime is continued pretraining, which involves training on billions of unlabeled domain-speciin tokens, most commonly via full finetuning: here we use the StarCoder-Python (Li ct al. 2023a) and OpenWebMath (Paster et al. 2023 datasets (Table D
The second is instruction fineturing, the common scenario for LoRA involving question- answer datasets with tens to hundreds of millions of tokens. Here, we use Magicoder-EWi Eveeerr-vvl---eee prool-Isttruct- 110K
(Wei al. 2023) and MetaMathQQA Yuu al. 2023
We evaluate larget-domain performance (henceforth, learning) via challenging coding and math benchmarks (HumanEval; Chen et al. 20221) and GSM8K; Cobbe et al. (2021)). We evaluate source-domain forgetting performance on language understanding, world knowledge, and common-sense reasoning tasks (Zellers et al.
2019 Sakaguchi et al. 2019 Clark et al. 2018).
We find that with commonly used low-rank settings, LoRA substantially underperforms full finetuning, while typically requiring longer training (Sec. [.1).. continued pretraining, the performance gap between full finetuming and LoRA is not closed even with high ranks. In instruction finetuning, on the other hand, high ranks can match full finetuning performance.
Despite LoRA's limitations, we show that it consistently maintains better source-domain performance compared to full finetuning (Sec. 4.2 Furthermore, we characterize the tradeoff between learning and forgetting (Sec. 4.3 We then show that LoRA even with higher rank mitigates forgetting more aggressivcly than classic regularization techniques that aim to prevent overfitting, such as dropout (Srivastava) el al.] 2014 Goodfellow el al., 2013),, and weight decay [Goodfellow et al. 2016). Moreover, by analyzing the generated solutions HumanEval problems, we demonstrate that while full finetuning tends produce a limited set of solutions, LoRA produces wider range of solutions more akin those of the base model (Sun al.
2023
Du et al. 2024)
Why does LoRA underperform full finetuning? LoRA was originally motivated in part by the hypothesis that finetuning results in low-rank perturbations to the base model's weight matrix (Li et al.) 20188 Aghajanyan ct al. 2020: Hu et al., 2021). However, the tasks explored by these prior works are relatively easy for modern LLMss and certainly easier than the coding and math domains studied here. Thus, we perform singular value decomposition to show that full finetuning barely changes the spectrum of the base model's weight matrices, and yet the difference between the two (i.e. the perturbation) is high rank. The rank of the perturbation grows training progresses, with ranks 10-100x higher than typical LoRA configurations (Figure G
We conclude by proposing best practices for training models with LoRA. We find that LoRA is very sensitive to hyperparameters, including learning rales, choice of larget modules, ranks, and scaling factors; setting these properly is a prerequisite to approach full finetuning performance.
To summarize, we contribute the following results:
Full finetuning is more accurate and sample-efficient than LoRA in CPT for code and math; in instruction finetuning, higher ranks can close most of the gaps (Sec 4.1p.
LoRA forgets less of the source domain (Sec. 4.2 and 4.3)
LoRA forgets less than common regularization techniques; also helps maintaining the diversity of generations (Sec. 4.5
Full finetuning finds high rank weight perturbations (Sec. 4.6
A hyperparameter sensitivity analysis for LoRA, as well as practical recommendations (Sec. 4.7
Model checkpoints and LoRA adapters can be accessed at htttp://gtthai.ccaa//alllr/rrrrr
2

---

Published in Transactions on Machine Learning Research (08/2024)
Code
Math
CPT
StarCoder-Python (up to 20B tokens)
OpenWebMath (up to 20B tokens)
IFT
Eagicoolos-xoll-sstnnn--1111 (72.3M tokens)
MctaMathQA (103M tokens)
Table 1: Datasets and token counts for math and code experiments
2
Background
LoRA involves freezing a pretrained weight matrix W pretrained E Rdxk and learning only low-rank pertur- bation to it, denoted here as A, as follows:
W finetuned
W pretrrinedd
+
A AB A E dd B Rr
Most common implementations initialize Ag ~ NO 1), Bo= 0 and set the scalar ", = afr with a controllable hyperparameter a. The user chooses which W pretrained to adapt ("target modules"), the rank << d, k, and the hyperparameter a. By doing so, only x r + x & parameters are trained per module instoad of d k. which reduces the memory and FLOPS required for computing the gradient. As an example, applying 16 LoRA to 7B weight matrix with d = 4096 trains < 1% of the original parameter count. Appendix Sec. lays out the approximate memory savings by LoRA.
LoRA's introduction and first applications targeted only the Wg and We matrices in the self-attention module Hu et al. 2021b. Since then, it has become best practice to target all transformer modules Dettmers et al. 2024), i.e., (W("), wht) w!!), w!" H in the self-attention modules, and 11 [==1 in the feedforward modules for L layers in, say, a Llama architecture (Hu et al.
Raschka 2023
[ gatte Wip. [.ll down
2021
Touvron et al. 2023
3 Experimental Setup
We train on code and math datasets that have been shown to increase downstream performance. We motivate the training datasets and evaluation benchmarks below. training was done using the Databricks MosaicML composer streaming" and 1lm-foundry repositories, as well the HuggingFace peft library.
3.1 Datasets for Continued Pretraining (CPT) and Instruction Finetuning (IFT)
Coding CPT Starcoder-PPthon LLi et al. 2023a This dataset consists of permissively licensed repositories from GitHub, including Git commits, in 80+ programming languages. We chose the Python subset and sub-sampled to 20B tokens.
Math CPT Oatt Paster al. 2023 This dataset contains 14.7B tokens derived from mathematical web pages from Common Crawl, correctly formatted to preserve mathematical content such as LaTeX equations To match with the StarCoder-Python dataset, we trained on up to 20B tokens, repeating tokens beyond the first. 14.7B. An analysis this dataset shows that it contains a considerable amount of full English sentences
Coding IFT Magi of programming questions and answers. It reproduces the "Evol-Instrucl datasct of WizardCoder [Luo et al.
Wei et al. 2023) This dataset contains 72.97M tokens
httppe:/githhb. http:///hhhbbccm///aaaaaa///oppp
https://github.. http:///...cccc/occcccc///tttaaa
httpps://uuggingface.o http://nna..cccocaaaaaaa/////aae
that 75% the examples contain LaTex. The data is classified 99.7% English and "overwhelmingly English" by the langdetect and fasttext tools.
OOt.
selection examples, regex
3

---

Published in Transactions on Machine Learning Research (08/2024)
2023b by iteratively prompting an LLM (GPT-4) to increase the difficulty of set of question-answer pairs from Code Alpaca (Chaudhary 2023).
Math IFT MetaMathQA Yu et al. 2023] This dataset was built by bootstrapping mathematical word problems from the training sets GSM8K (Cobbe et al. 2021] and MATH (Hendrycks al. 2021 by rewriting the questions with variations using GPTT-3.5. This dataset contains 395K question-answe pairs and roughly 103M tokens 6
We quantify learning and forgetting via benchmarks reported on the Open LLM Leaderboard for state of the art open-source LLMs such as Llama (Touvron et al. 2023)
3.2 Measuring Learning with Coding and Math Benchmarks (target domain evaluation)
Coding HumanEval (Chen et al. 2021 This benchmark contains 164 problems that involve generating of Python program given a docstring and a function signature. A generation is considered correct if it. passes all supplied unit tests. We use the Code Generation LM Evaluation Harness (Ben Allal et al. 2022 configured output 50 generations per problem, and calculate "pass@1" " with softmax temperature=0.2 and top_p= 0.95 for 0-shot HumanEval.
Math GSM8K (Cobbe et al. 2021) This benchmark includes a collection of 8.5K grade-school math word problems. We evaluate on the test split of GSM8K (1,319 samples) as implemented in the LM Evaluation Harness Gao el al. 2023), with default generation parameters (lemperature=0, 5 few-shot, pass@1).
3.3
Forgetting Metrics (source domain evaluation)
We use the following benchmarks lo asses degradation of base model capabilities. HellaSwag Zellers el al. 2019 includes 70K problems, each describing an event with multiple possible continuations. The task is to pick the most plausible continuation, which requires making inferences about nuanced everyday situations. WinoGrande (Sakaguchi et al. 2019 also assesses commonsense reasoning. It includes 44K problems with sentences that require ambiguous pronoun resolution. ARC-Challenge (Clark al., 2018) consists of 7, 7,787 grade-school level, multiple-choice science questions, and tests complex reasoning and understanding of scientific concepts. These benchmarks involve multiple-choice questions that use the predicted logits for calculating accuracy, and do not require specifying further generation hyperparameters. All forgetting metrics were computed using the MosaicML. Gauntlet evaluation harness (Dohmann 2023
4 Results
4.1
Target-domain performance: LoRA at low ranks underperforms full finetuning
We compare LoRA and full finetuning after performing an exhaustive learning rate sweep for each method, which we found to be crucial (Dettmers et al. 2024). We include learning rate sweep results in Figure S1
We perform a sample-efficiency analysis i.e., compute the learning metrics as function of training samples seen for both LoRA and full finetuning. For IFT, we train separate models for 1, 4, 8, and 16 epochs. For CPT, we vary the number of training Lokens (0.25, 0.5, 1, 2, 4, 8, 16, 20 billion), using individual learning rate cooldown schedules. For each condition, we train one full finetuning model and three LoRA models with ranks 16, 64, 256 noting that most LoRA papers use a "low" rank of 8-64, (c.g., Dettmers al. (0024); Zhuo et al. (2024) The LoRA models target all transformer modules and use 2r, as known be best. practice (Raschka 2023 For further details experimental setup and hyperparameters, see Appendix A
The results appear in Fig. I We first note that for both programming and math, IFT improves evaluation scores much more than CPT, which is expected because the samples in each IFT dataset are more similar to the evaluation problems (e.g., for code, IFT achieves maximum HumanEval of 0.497 vs. 0.263 for CPT).
http:///wwaaaiaaarrrrrr///ccrtep 11m leaderboard 1eadeeerradddd
4

---

Published in Transactions on Machine Learning Research (08/2024)
Learning the target domain
A
Continued pretraining
B
Instruction finetuning
Llama-2-2 trained on Starcoder-Python
Llama-2-7B trained on Magicoder-Ewo-Hine Sagiioder-Evo--nsttuu-------
0.50
0.26
LoRA (r=16)
LoRA (r=44)
0.45
Accuracy HvmanEval
0.24
Acouracy in HumanEval
LoRA (r=25)
0.40
0.22
Full Finetuning
Code
0.35
0.20
0.18
0.25
0.16
0.20
0.15
base madel
base model
0.25
2
4
16
Epoch
Training Tokens (Billions)
C
D
Llama-2-7B trained on OpenWebMath
Llama-2-7B trained on MetaMathQA
0.30
0.28
0.26
Accuracy in GSMBK
Accuracy in GSMBK
0.5
0.24
Math
0.22
0.20
0.3
0.18
0.16
0.2
base model
0.14
model
0.25
2
8
16
Epoch
Training Tokens (Billions)
Figure 1: LoRA performance scales by rank and underperforms full finetuning code and math. (A) Starcoder-Python. (B) Eesrrodrr--oo---nsuutt--------
(C) OpenWebMath, (D) MetaMathQA. In (4) and (B) y-axis: HumanEval pass@1. In (C) and (D) y-axis: GSMSK strict match. In all panels, "base model" indicates Alama-2-7b without instruction finetuning. Note that 16 epochs are s=1.6B and x1.6B tokens, for Magicoder-Evol--nneee 110K and MetaMathQA.
For Code CPT (Fig. IA and Table S1 we identify a substantial gap between full finctuning and LoRA that grows with more data. The best LoRA model, with rank 256, peaks at 20B tokens with HumanEval=0.224 roughly matching full finetuning with 4B tokens (HumanEval=0.2213) Full finetuning reaches its peak HumanEval of 0.263 at 20B tokens. A clear ordering by rank emerges after the initial 1B CPT tokens.
HumanEval accuracy is clearly ordered by rank from the very first epoch. The more common 16 and r LoRA configurations have lower accuracy than full finetuning, with HumanEval scores of 0.358 and 0.417 at epoch 4, respectively). With a high LoRA rank (r 256), full finetuning performance be matched (LoRA=0.449) in epoch 4, full finetuning-0-997 in epoch 8).
For Code IFT (Fig. B and Table S5
5

---

Published in Transactions on Machine Learning Research (08/2024)
In Appendix Sec. F we perform a more sensitive HumanEval analysis, calculating pass@k function of k == 1,............... 256 with higher temperature of 0.8 for full finetuming and the LoRA models (at epoch 4). This analysis shows that full finetuning is superior to r = 256 for k < 64, after which the two are equal.
Math CPT (Fig. D and S3 results closely echo those of code CPT. Consistent patterns in GSM8K emerge at. 4B tokens. Full finetuning opens a gap in GSM8K which widens with more data. Similarly, LoRA performance is ordered by rank. The best LoRA (r 256) peaks at 16B tokens (GSM8K-0.203), underperforming full finetuning at 4B tokens (GSM8K=0.22)) and at its peak at 20B tokens (GSMSK=0.298).
LoRA closes much of the gap with full finetuning in the Math IFT (Fig. 1D and Table S7 dataset, while remaining less sample efficient. Both methods substantially improve upon the base model; LoRA (r - 256) peaks at 8 epochs (GSM8K=0.63) 634) while full finetuning achieves GSMSK-0.641 at epochs and peaks at 4 epochs, with GSMSK=0.642P" and achieve GSM8K-0.624 at epoch 4. We suggest that lower ranks are effective here because English mathematics problems involve a smaller domain shift from the pretraining data compared to coding ones.
F Unlike the code IFT dataset, r = 64 suffices to approach full finetuning
In summary, in CPT, LoRA underperforms full finetuning across all configurations. In IFT, and especially in code, high LoRA ranks are required to close the gap with full finctuning.
4.2 LoRA forgets less than full finetuning
Here, we investigate the extent of forgetting (defined in Sec. 3.2 as a function of training data in Fig. 2
Overall, we observe that (1) IFT induces more forgetting than than CPT, (2) programming induces more lorgetting than math, and (3) forgetting tends to worsen with training duration. Most importantly, LoRA forgets less than full finetuning, and the extent of forgetting is controlled by rank. In code - botth CPT and IFT full finetuning forgets substantially more than any LoRA configuration. In code CPT (Table S2 at 20B tokens, full finetuning scores 0.545 versus 0.617 by LoRA r = 256. In code IFT (Table (6).. full finetuning scores 0.414 versus 0.509 by LoRA r = 64. In math- - for both CPT and IFT - LoRA with r = 256 forgets nearly as much as full finetuning. In CPT (Table S4j, LoRA scores 0.616 (20B tokens) versus 0.613 of full finetuning (16B tokens). In IFT (Table S8), LoRA and full finetuing respectively degrade to 0.567 and 0.559 at epoch 16.
We note that the least forgeting occurs for the OpenWebMath dataset, which is dominated by English sentences (see 3.1 for details).
4.3 The Lorgetng Tradeoff
Tt is trivial that models that change less when finetuned to a new target domain will forget less of the source domain. The nontrivial question is: do LoRA and full finetuning differ in how they tradeoff learning and lorgetting? Can LoRA achieve similar target domain performance but with diminished lorgetting?
We form lergntngg Pareto curves by plotting the forgetting metric versus the learning metric for cach training duration (Fig. 3 As models train on more data, they learn more and forget more, traveling up and left in this space. As we increase LoRA ranks, we find that the curves shift up and left as well, again, learning more and forgetting more, doing so more consistently in IFT than CPT.
Each dataset presents a unique tradooff pattern which makes it difficult to conclude whether LoRA and full finetuning ofler fundamentally different.
tradeoffs. We will review each dataset next.
For Code CPT, though the full finetuning curve reaches much higher values of HumanEval, it appears to forget more for any given Eval value, which LoRA can reach if trained more tokens. This pattern does not hold for math CPT, where LoRA and full finetuning curves are roughly overlapping until full finetuning shoots up (in 4B tokens) to achieve much higher GSMSK scores without increased forgetting. In code IFT, LoRA+ r = 256 offers comparable HumanEval accuracy while strictly forgetting less. Lower ranks
We note that the original MctaMath paper reports a maximum accuracy of 0.665 when (fully) finetuning Llama-2-7B on the Math dataset. We attribute this small differences in hyperparameter;; they trained epochs with batch size of 128 using the AdamW optimizer, a learning rate of 2e-5, a learning rate warmup of 3%.
6

---

Published in Transactions on Machine Learning Research (08/2024)
Forgetting the source domain
A
Continued pretraining
B
Instruction finetuning
(HHollaSwaag + ARC-challenge WinoGrande)
Llama-2-7B trained on Starcoder-Python
(HellaaSwag + ARC-challenge WinoGrande)
Llama-2-7B trained on httpiodder-vvlllnssuurc.. 110K
bese model
base model
0.65
0.66
0.64
0.60
Code
0.62
0.55
0.60
0.50
LoRA (r=16)
0.58
LoRA (r=64)
LoRA (r=256)
0.45
Full Finetuning
0.54
0.25
0.5
8.0
16 0
20.0
1
4 Epoch
16
Training Tokens (Billions)
D
(HHllaaSwag + ARC-challenge + WinoGrande)
-larma-2-7B trained on OpenWebMath
(HellaaSwag ARC-challenge WinoGrande)
Llama-2-7B trained on MetaMathQA
0.67
base model
base madel
0.66
0.68
0.65
Math
0.62
0.64
0.60
0.63
0.56
0.25
0.5
8.0
16
2
8
16
Epoch
Training Tokens (Billions)
Figure 2: LoRA forgets less than full finetuning. In all panels, the y-axis shows the average of HellaSwag, ARC-Challenge and Winogrande for Llama-2-7B trained trained on: (A) StarCoder-Python (B) Magicoole-Foo--nsssuett-1.. (C) OpenWebMath (D) MetaMathOAA.
do not reach high values of HumanEval to compare to full finetuning, In math IFT, LoRA and full finctuning seem to lie on adjacent bearning-forgetting tradeoff curves, with full finetuning offering preferable tradeofIs.
With the caveats mentioned above, it seems that LoRA can offer preferable Marning-forgettinn tradeoffs for code, while full finetuning can offer preferable tradcoffs for math. Morcover the choice LoRA rank can knob to navigate the marning-forgetting tradeofIs.
4.4 For the Tülu-v2-mix dataset, LoRA is on par with full finetuning
far, we analyzed how LoRA and full finetuning specialize very specific domains. Often, code or math problems appear as part of larger IFT data mixtures that include multi-turn conversations and a varicty of other NLP tasks, such summarization, full finetuning models such popular dataset, the T2lu-v2-mix (Ivison al. 2023). The results
(e.g.. Wei et al. (2021)). ). We therefore finetuned LoRA and

---

Published in Transactions on Machine Learning Research (08/2024)
The learning-forgetting tradeoff
A
Continued pretraining
B
Instruction finetuning
Llama-2-78 trained Starcoder-Python
Llama-2-7B trained on Maggicooee-Eva---nstuu----10
0.50
0.26
0.45
0.24
Accuracy HumanEval
Accuracy HumanEvas
0.22
0.40
Code
0.20
0.35
LoRA (6=16)
0.18
LoRA (r=64)
0.30
LoRA (r=256)
0.16
Full Finetuning
0.25
0.20
0.54
0.56
0.58
0.60
0.62
0.64
0.45
0.50
0.55
0.65
(HelllaSaag ARC-challenge + WinoGrande)
(HellaSwag ARC-challenge + WinoGrande)
C
Llama-2-78 trained on OpenWebMath
Llama-2-78 trained on MetaMathQA
0.30
0,650
0.28
0.625
0.26
0.600
Accuracy in GSMBK
Accuracy in GSMBK
0.24
0.575
Math
0.22
0.550
0.20
0.525
0.18
0.500
0.16
0.475
0.14
0.450
0.56
0.58
0.60
0.62
0.64
0.58
0.57
0.58
0.59
0.60
0.61
0.62
0.63
(HellaSwag ARC-challenge WinoGrande)
(HellaSwag ARC-challenge WinoGrande)
Figure 3: LoRA vs. full finetuning trade-off for Llama-2 2-7B. Relative to full finetuning, LoRA learns less (lower values on the y-axis) and forgets less (higher values on the r-axis). Each dot is separate model, with marker size corresponding to training duration (from 0.25-20 billion tokens for CPT, and 1-16 epochs for IFT). Same data as Figures 2
presented the Appendix (Sec. C and Table S9 In summary, we find that both LoRA and full finetuning
meaningfully improve upon the base model, and that LoRA, even with lower ranks, can match full finetuning in chat quality measured by Multi-Turn Benchmark (MT-bench (Zheng et al.) 2024)), GSM8K (Cobbe
et al. 2021), and Massive Multitask Language Understanding (MMLU; Hendrycks et al. 2020 ). At longer training durations (6 epochs), LoRA also forgets less.
4.5
How strongly does LoRA constrain the finetuning process?
In this scction, we analyze Llama-2-7B models trained on the Magicoder-Evvo--nstruee datasct. We first compare the analyze the diversity of the generated text.
tradeoffs between LoRA and classic regularization techniques, and then

---

Published in Transactions on Machine Learning Research (08/2024)
Llama-2-7B trained on Magicoddr-Evo--nstucc-110
Leaming
Forgetting
(HellaSwag ARC-c WinoGrande)
0.5
0.65
Accuracy on HumanEval
0.60
0.4
0.55
0.3
Full Fineluning
0.50
Attention Dropout 0.1
Attention Dropout 0.05
0.2
0.45
Weight Dacay 5e-05
base model
Weight Decay 0. 0.0001
0.1
0.40
LaRA (rRA(r116)
LaRA (r=26))))))
0.35
Epochs (73M tokens each)
Epochs (73M tokens each)
LoRA forgets less than attention dropout and weight decay. Results from Llama-2-7B finetuned on Magicoder-Evol-Iinee Eerrroder-------ssuut--------- 110K. Left panel: learning measured by accuracy HumanEval. Right panel: forgetting as measured by the average of HellaSwag, ARC-Challenge and WinoGrande scores. The solid slateblue line shows that LoRA (r=256) learns as much as full finetuning, weight decay, and attention dropout, while forgetting much less.
Figure 4:
LoRA forgets less than attention dropout and weight decay We compare LoRA (r= 256, training all modules) to weight decay Goodfellow et al. 2016) with values 5e-5 le-1 and attention dropout (Srivastava et al. 2014) with values 0.05, 0.1. Both regularization techniques appear to learn and forget much as full finetuning, except that weight decay starts to generally deteriorate at longer training durations (epochs 8 and 16). LoRA, with the common r = 16, learns less and forgets less than all other models. LoRA y = 256, on the other hand, learns as much as the other methods while forgetting less.
LoRA helps maintain diversity of token generations. We scrutinize the generated solution strings for HumanEval problems. We calculate the unique number of output strings out of 50 generations (for base model, full finetuning, and LoRA) serving as coarse proxy for predictive diversity. In Figure 5 separately show the results for correct and incorrect answers. As in the reinforcement lcarning from human feedback literature (Du et al. 2024 Sun et al. 2023), we find that full finetuming results in fewer unique generations ("distribution collapse") compared to the base model, for both pass and fail generations, with LoRA in between the two. The above works also suggest that LoRA could even substitute a common Kullback- Kullback-Leibler divergence term that keeps the probabilitics of the generated text similar between the finetuned and basc model. We reiterate that exact string matching between generations not sensitive metric of predictive diversity, as generations can slightly vary in format and remain functionally identical.
4.6 Full finetuning code and math does not learn low-rank perturbations
In this section, we seek study whether we should expect. low-rank training be a good approximation to full finctuning, and if so, what is the necessary rank. Recall that full finctuning can be written as Wrinetuned W prettraiedd + A; here we compule the Singular Value Decomposition of all three terms in the equation. We focus continued pretraining for code, where there are drastic differences between LoRA and full finetuning. We analyze checkpoints obtained at 0.25, 0.5, 1, 2, 4, 8, 16, and 20 billion training tokens.
First, in Figure S8 we present results for the W, projection at layer 26 of Llama-2-7B (with dimensions d d, d = 4096). We show that the spectrum of the finetuned weight matrix very similar to that of the base weight matrix, both decaying slowly and requiring keeping & 50% of singular vectors (* 2000/4096) explain 90% of the variance in the weight matrix. Critically, the difference also has a similar spectrum the finetuned and base weight matrices (up multiplicative scaling). These results are in line with the
9

---

Published in Transactions on Machine Learning Research (08/2024)
HumanEval (samples=50, temp=0.2)
model base
model full
model lora
40
palso basee
30
10
passed
False
True
Tndl possed
30
10
20
40
20
40
Number Unique Generations
Number Unique Generations
Number of Unique Generations
Figure 5: LoRA maintains output token diversity relative to full finetuning.
analysis in Zeng & Lee (0224) showing that any transformer model can be well approximated with r = d/2. Additionally, we suggest. that there nothing extraordinary about the full finetuning spectra; similar spectra can be achieved by adding www-aanitudd Gaussian i.dd noise a weight matrix (Fig. S9
Next, ask when during training does the perturbation become high rank, and whether it mcaningfully varies between module lypes and layers. We estimate the rank needed to explain 90% of the variance in the matrix. The results appear in Figure 6 We find that: (1) The earliest checkpoint at 0.25B CPT tokens cxhibits A matriccs with a rank that is 10 100x larger than typical LoRA ranks; (2) the rank of increases when trained on more data; (3) MLP modules have higher ranks compared to attention modules; (4) first and last. layers seem be lower rank compared to middle layers.
4.7 Hyperparameter sensitivity analyses for LoRA
Our goal in this work was to optimally configure LoRA so that it has the best chances of matching full finetuning. This is nontrivial, as LoRA has a large number of hyperparameters to choose from: target modules, rank, scaling factors, and learning rates. We turn to analyze the importance of each, and provide some practical recommendations.
First, we found that the choice a 2r crucial for high ranks. Most common packages, e.g., Hugging Face's 's PEFT scale the LoRA matrices by a/r, effectively scaling down higher ranks (see also Kalajdzievski (2023) One might think that high learning rate values may compensate for fixed low a's, but doing so creates instabilities and often leads to inferior performance. To show this, we performed a joint hyperparameter sweep over a and learning rate for the Magicoder dataset training a r = 256 LoRA for 4 epochs (Fig. ??).
We find that a = 512 does much better than 256 or 32 across all learning rates.
Next, assess the relative contribution of target modules and rank, we trained Llama-2-78 models on epochs of the Magicoder dataset, sweeping over target modules (* (tttention", "ML.P", and "AlI", their union), ranks (r 16, 64, 256), setting 2r. Fig. shows that HumanEval performance increases with rank, and
htttp:://hugginggfac... hhttp://wwgggggga......//c/ctt/p co//bocss/pettt////ineee
10

---

Published in Transactions on Machine Learning Research (08/2024)
Target u_proj
Target k proj
Target v_proj
1800
1600
Effective Yark (mmttm600))
1400
1400
1200
1000
1000
1200
800
Effective
lonss
2000
000
200
200
30
15
20
25
10
15
25
30
5
15
25
30
Layer Number
Layer Number
Layer Number
Target O. pro)
Target down. pwo)
Target up pro
2600
bffecttew Kank fimex (0996)
1600
2000
1400
2400
1200
2200
Effective
CPI Tocers bilional
Ineers (en billered
2000
Iolenn billaneo
600
Number
Number
Number
Figure 6: Dynamics rank for Llama-2-7B trained on the Starcoder (CPT) data. each panel, the x-axis denotes layer number and the y-axis denotes rank needed to explain at least 90% of the variance (maximal dimensionality 4096). Colors denote CPT tokens, with lighter colors trained for longer.
Magicoder, rank=16
Magicoder, rank=64
Magicoder, rank=256
0.5
0.5
0.4
0.4
HumanEval pass(e)
HumanEval pass
HumanEval 1
0.3
0.3
0.3
0.2
0.2
model
Altontion, F216
Altantion, T=64
model
Attonitonn, rS256
0.1
MLP. r=18
0.1
MLP,, r:64
0.1
MLP, r:256
All, 11
All, me84
All. m225
finetuning
finetuning
finctuning
0.0
2 epoch
epoch
epoch
Figure 7: Targeting MLP or All modules is superior to training Attention modules alone. All Llama-2-7B checkpoints were trained on Magicoder for 1, 2 and 4 epochs with rank 16 (left), 64 (center) and 256 (right).
that targeting just "Attention" underperforms both "MLP" and "All", where in the latter, most gains are interestingly driven by the "MLP" modules. This is potential evidence that the MLP blocks are the primary loci for continual learning in LoRA, at least in our datasets.
For IFT, we find that LoRA is more sensitive to learning rates compared to full finetuning, and benefits from the highest learning rate that enables stable training for the chosen training duration (see Appendix See. B
11

---

Published in Transactions on Machine Learning Research (08/2024)
and Fig. S1 LoRA's best learning rates should be set one order of magnitude higher than full finetuning's s often ranging between 5e-5 and 5e-1 for these combinations of model architecture and dataset.
In Appendix Sec. ! we benchmark throughput and peak GPU memory of different LoRA configurations, showing that for standard implementations and a fixed batch size, LoRA tends to train slower than full finetuning.
To conclude, based on our main results and hyperparameter sweeps, we recommend: (a) using LoRA for
instruction finetuning and not continued pretraining: (b) if GPU memory allows, targeting "All" transformer modules with rank of 256, since ranks 16 - 64 tend not to suffice for code tasks; (c) using 2r, and (d) sweeping over learning rates between [le - 5, 5e -44, 4], picking the highest value that enables stable training.
5 Related Work
Extensions to LoRA LoRA has inspired many variants and extensions. One group of methods improves training with LoRA by focusing on initialization or scaling (Meng et al., 2024 Hayou et al.] 2024 Li
et al.] 2023b Kalajdzievski
2023 Nikdan el al. 2024), sequential training procedures (Xia al.) 2024),
architectural modifications (Shi et al.
2024). Other works propose alternative low-rank approximations
altogether (Liu et al. 2024 Zhao et al.
2024a Jiang et al. 2024a Kopiczko et al. 2023 In this study we
chose to analyze the classic LoRA sctup: while many of these proposed variations of LoRA seem promising,
we leave a rigorous comparison of these techniques to future work.
Benchmarking LoRA vs. Full Finetuning The original LoRA paper Hu el. al. (2021) | reported that LoRA matched full finetuning performance for RoBERTa (Liu et al. 2019 on GLUE (Wang et al. 2018). and
GPT-2 on E2E NLG Challenge (Novikova et al. 2017 and GPT-3 on WikisQL (Zhong et al. 2017), MNLI
Williams et al. 2017 and SAMSum (Gliwa et al. 2019 Many subsequent studies follow this template and report encoder model performance on tasks in GLUE such as SST-2 (Socher et al. 2013) and MNLI (Williams et al., 2017). Models such as RoBERTa are less than 340M parameters, however, and classification LLMs such as Llama-2-7B. Despite LoRA's popularity, only few studies have rigorously compared LoRA to full finetuming in this setting and with challenging domains such as code and math. Dettmers et al. (2024) for example found that QLoRA matched full finetuning MMLU (Hendrycks et al., (020) performance when finetuning Llama-1 7B, 13B, 33B and 65B on the Alpaca (Taori et al., 2023) and FLAN (Chung et al., 2024) datasets. Ivison et al. (0023) on the other hand found that QLoRA did not perform as well as full finetuning for Llama-2-7B, 13B and 70B models trained on the Tülü-v2- mix dataset when evaluated across MMLU, GSMSK, AlpacaEval (which uses LLM-as-a-judge;; (Dubois al. 2024h) and HumanEval. One recent notable study is Astraios, which found that LoRA at rank = 8 performed worse than full finetuning on 8 datasets and across 4 model sizes (up to 16 billion parameters), on 5 representative code tasks (Zhuo et al. 2024). Our study corroborates these results and shows that with higher ranks and proper hyperparameter choices, LoRA can perform much better.
tasks such as MNLI are quite trivial for modern billion-praamen
The conclusions have also been mixed with regards to the practical details surrounding LoRA target modules and rank: Raschka| (2023) and Dettmers et al. (2024) show that optimized LoRA configurations perform as well as full finetuning, and that performance is governed by choice of target modules but not rank. However, in that work, the scalar a was not modified with rank, and we found that increasing it to 2r was necessary to unlock improvements by rank. In contrast, Liu et al. (2024) shows that LoRA is sensitive to ranks. It is likely that some of these discrepancies are due to differences in finetuning datasets and evaluations.
Continual learning code and math. A growing body of work investigates ways of specializing LLMs to code and math. In code, models such as StarCoder (Li et al. 2023a Lozhkov et al.) 2024). DeepSeek Coder (Guo et al.] 20244, and SantaCoder (Allal et al. 2023) were pretrained from scratch on largo-scale code datasets. Alternatively, some works start with a generic pretrained base model, and combine continued pretraining on large code datasets followed by IFT on code problems (usually with full finetuning), e.g., Codex (Chen et al. 2021), Codo-Qwen (Bai et al.) 2023), CodeLlama (Roziere et al., (023).. Some perform only IFT on top of a base model, like MagiCoder Wei al. (2023), or WizardCoder (Luo et al. 2023b). OctoCoder (Muennighoff et al. 2023) performs IFT with LoRA.
12

---

Published in Transactions on Machine Learning Research (08/2024)
Similarly, much recent work aims to improve mathematical capabilities. Models like DeepSeek Math (Shao ett al. 2024) perform continued pretraining top of base model, while other methods focus on finetuning by generating high-quality synthetic math problems, scaling to millions of examples. Luo et al.
(2023a)
takes the Evol-Instruct approach to data generation (akin to the Magicoder datasct; Sec. 3.1 which it then uses lo train reward models for instruction quality and solution correctness, which are turn used for LLM finetuning. Other work develops Monte Carlo Tree Search methods to automatically supervise the intermediate reasoning steps while solving math problems (Luo et al. 2024), and Yue et al. (2024) generates questions and answers from the pretraining web corpus. Toshniwal et al. (0024) uses an LLM to synthesize Tode-Tnterpreter-aa can be verified against the official solutions. Singh et al. (2023) iterate over this procedure multiple times ("Sell-training") using an expectatioon-maxiiiallll approach. All reviewed methods meaningfully improve math capabilities.
solutions to the GSM8K and MATH benchmarks. The proposed solutions
Gearning-Forgettin tradeoffs Vu et al. 2022) shows that prompt tuning (ester et al. 2021 another paramcter-efficien finetuning method, can aid in mitigating forgetting for cross-lingual summarization tasks (using multilingual variants of the T5 model). With large Llama-style LLMs, it has been reported that code-finetune uned LLMs lose some of their capabilities language understanding and commonsense reasoning (Li et al., [2233a] Roziere al. 2023] Wei "replaying" sourco-domain data during continual learning, which can be done by storing the data in memory
2023). A common approach to mitigate forgetting involves
bulfer, generating it on the fly (Lesort et al. 2022 Scialom et al. 2022
Sun el al. 2019
6 Discussion
Does the difference between LoRA and full finetuning change with model size? Studies in the past have hinted at a relationship between the effectiveness of finetuning and model size (Aghajanyan
et al. 2020 Hu et al., 2021 Zhuo et al. 2024). While recent studies have successfully applied LoRA to 70B
parameter models (Ivison et al.) 2023 Yu et al. 2023 Niederfahrenhorst et al., 2023) Turgutlu 2024), and previous work shows that techniques like prompt luning become more effective for larger models (Vu et al.,
2022 we leave rigorous study of these intriguing scaling properties to future work.
Limitations of the spectral analysis. The observation that full finetuning tends to find high rank solutions does not rule out the possibility of low-rank solutions; rather, it shows that they are not typically found. An alternative interpretation is that the rank needed to reconstruct the weight matrix is higher than the rank needed for a downstream task. We also only presented SVD analysis for the continued pretraining setting. It is possible that a similar analysis for the instruction finetuning setting would reveal that the full finetuning does not tend to be as high rank.
7 Conclusion
This work sheds light on the downstream performance of billion parameter L.Mss trained with LoRA and full finctuning. Unlike most prior work, we use dopain-specific datasets in code and math, associated with sensitive evaluation metrics. We show that. LoRA, with commonly used low-rank settings, underperforms full finetuning across domains. We also show that LoRA keeps the finetuned model's behavior close to that of the base model, with diminished sourco-domain forgetting and more diverse generations at inference time. We show that LoRA mitigates forgetting more than classical regularization techniques, and also show that full finetuning finds weight perturbations that are far from being low-rank. We conclude by analyzing LoRA's increased sensitivity to hyperparameters and highlighting best practices.
Acknowledgements
We would like thank the editor and the three anonymous reviewers who provided high-quality feodback on this work. We are also grateful to Daniel Han and Damjan Kalajdzievski for carefully reading our work and pointing out the importance of setting a = 2r for training with high ranks.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Author Contributions
D.B. led this project by developing code, running experiments, analyzing results, and writing the manuscript.
J.P. ran experiments and assisted in the writing of the manuscript. J.G.O. wrote code and ran experiments.
P.G. advised the SVD analysis, C.J. ran experiments, and D.K. wrote code. M.P., S.H., V.C., J.F., C.B., and
J.P.C. advised this work.
14

---

Published in Transactions on Machine Learning Research (08/2024)
References
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, Xév preprint or Xiv::2012.. 1.2..1135 13255, 2020.
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars! Xiv preprint Xiv:29001 039988, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint orXiv:2309. 16609, 2023.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra.
A framework for the evaluation of code generation models. http:://gttbbb.co////gggeeerpppp com/biggoode-projjtt/ seccode-vvluatiion hhanness 2022.
Sahil Chaudhary. Code alpaca: An mnstruction-folowingg Ilama model for code generation. https://github ccm//aah11/81114//cdeealppppppp 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Flizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Tlya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling inst rnctionnnnnne language models. Journal of Machine Learning Research, 25(70):1 1:0, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXie, abs/1803.05444 2018. URL http::://api.ssmmnniiississ ort..lauu.rrg/orC : 39228816
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint Xrxiv:2110 1011.111110 14168, 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llss. Advances in Newral Information Processing Systems, 36, 2024.
Jeremy Dohmann. Blazingly fast IIm evaluation for in-context learning, February 2023. URL https: 7/www www..databrckk. wttrriiccckcccmboooo www.oooooooomoe///ooooooooooooee http:g///lo-eaaaaaiiooo---o----- Blog post, Mosaic Al Research.
Yuqing Du, Alexander Havrilla, Sainbayar Sukhbaatar, Pieter Abbeel, and Roberta Raileanu. A study on improving reasoning in language models. In Can't Believe It's Not Beller Workshop: Foilure Modes in the Age of Foundation Models, 2024. URL https://openreviiet net//oorrm
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neurul Information Processing Systems, 36, 2024.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):
1999.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL httts:://eeeeeeee org/reeordda1125668
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Di- nesh Manocha, et al. A closer look at the limitations of instruction tuning. or preprint 202.0:2402.05119 2024.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-amnolated dialogue dataset for abstractive summarization. arXiv preprint (:Xi::9111.12233 2019.
Tan Goodfellow, Yoshna Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
lan J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting gradient- neural networks. arXiv preprint arXiv:1312.62 62111 2013.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu,
YK Li, et al. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint Xiv:22401. 14196, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint Xiv:24402. 12354, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXic preprint orXiv:2009.0300 033000 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arv preprint Xri:i:21033.m 20o,2100.00874, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 006099555, 2021.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Tz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arxe preprint arXiv:3311.10002 2023.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for parfmeter-efficient fine-tuning, arXiv preprint arXiv:2405.11 12130, 2024a.
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T. Kwok. Forward- backward reasoning in large language models for mathematical verification, 2024b.
Damjan Kalajdzievski. A rank stabilization scaling factor for finc-tuning with lora. arXiv preprint arXiw:2312.087:9. 2023.
Damjan Kalajdzievski. Scaling laws for forgetting when fine-tuning large language models. arXie preprinl arXiv:2401.05605, 2024.
Anat Kleiman, Jonathan Frankle, Sham M Kakade, and Manshoej Paul. Predicting task forgetting in large language models. 2023. URL
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector- based random matrix adaptation. arXiv preprint 2xXi:22100.14455 2023.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Timothée Lesort, Oleksiy Ostapenko, Diganta Misra, Md Rifat Arefin, Pau Rodriguez, Laurent Char- lin, and Trina Rish. Challenging common assumptions about catastrophic lorgetting. or Xiv preprinl arXiv:2207.04543 2022.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter--fffcient prompt tuning. arXiv preprint ar Xr:::2104400899, 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint orxiv:1804.0085 2018.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Mucnnighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXie:2305.000 06161, 2023a.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampalziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-te cora-fine-tuniig sanee-tning--ware quantization for large language models. arXiv preprint orXiw:2310.0665, 2023b.
Shih- Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09953, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprinl Xrv: 11692, 2019.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next. generation. arXiv preprint Xro::22402 19173, 2024.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang, Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023a. URL httts: http://axvvvv...////// abs/ 09583
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. URL
06592
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardooder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.00 085688 2023b.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXie:2404.0090) 2024.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint XXi::3388.07124, 2023.
Artur Niederfahrenhorst, Kourosh Hakhamaneshi, and Rehaan Ahmad. Fine-tuning Ilms: Lora or full- paramcter? an in-depth analysis with llaaa 2, September 2023. URL httpp::t/ww....... anyscale. www.coocc/c////e com/blog/
Sine-tuning-1lmm-looe wewllmmsmoorroooo or wwwoor-oo-rul--l-oollleellllllee wwwllllaaaamtterrrreeeeeeaaaa an- www.-deethh-aallyy---tttt 11 Blog post.
Mahdi Nikdan, Soroush Tabesh, and Dan Alistarh. Rosa: Accurate parameter-officien finc-tuning via robust adapiation. arXiv preprint arXim:2401.0440 2024.
Jekaterina Novikova, Ondřej Dušck, and Verena Rieser. The e2e datasct: New challenges for end-to-end generation. arXiv preprint ar Xiv: 706. 09254, 2017.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXie preprinl arXiv:2310.0075 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 16. IEEE, 2020.
Sebastian Raschka.
Practical lips for finetuning 1lms using lora (low-rank adaptation), http:////p/ppatiiaaaaasssaoraorr finetunnnng 11ms#
2023. URL
MC2%A7 -orerambbb---ooo----ooo-e-------
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. Xiviv preprinl Xrx::2205. 12..122330 12393, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepsockmath: Pushing the limits of mathematical reasoning open language models, 2024. URL https://arxiv .rg. abbs/2402.0330
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Reslora: Identity residual mapping in low-rank adaption. arXiv preprinl wwXiu:2402.18032 2024.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solvvng with language models. arXiv preprint arXiv:2312. 06585, 2023.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky, Nya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929 1958, 2014.
Fan-Keng Sun, Hae Ho, and Hung-Y Yi Lee. Lamol: Language modeling for lifelong language learning, arXiv preprint XrXic:1909, 03329, 2019.
Simeng Sun, Dhawal Gupta, and Mohit Tyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization rlhf, 2023.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin llama model, 2023.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Giuman. Openmathinstruct-I: A 1.8 million math instruction tuning dataset, 2024. URL httpp::///axxv...o org/abs/ 2402. 10176]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xie preprint or arXin:2307.09988 2023.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Kerem Turgutlu. Efficient finetuning of Ilama with fsdp qdora, April 2024. URL https://www answer. ai/ 1ttpa//2220---22-----------0---0 -1ama-mmmaa html Blog post.
Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit lyyer, and Noah Constant. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. arXiv preprint Xrxi::2205.120 12647, 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi- task benchmark and analysis platform for natural language understanding. arXiv preprint orXiv:1804.07466 2018.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory method and application, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint ar Xix::209.00163 2021.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint Xrx::23312.020 02120, 2023.
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arviv preprint 2rvi:22212, 09561, 2022.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprinl arXiv: 1044. 05426, 2017.
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via
residual learning. arXiv preprint 0r1101440..4115, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 0203909.12280 12284, 2023.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web, 2024.
URL
03548
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation, 2024.
URL http8:
wwwpivv..rrgabb//21111 17513
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient I1m training by gradient low-rank projection. arXio preprint arXiv:2403.03507 2024a.
Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned 1lms that rival gpt-4, a technical report. arXiv preprint ar XrXi::2405.0 00732, 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXie preprint wxXiv1109.001000 2017.
Terry Yue Zhuo, Armel Zebave, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameter-efficies instruction tuning code large language models. arxiv preprint ar 0010:22400.00788, 2024.
19

---

Published in Transactions on Machine Learning Research (08/2024)
Appendix
A Experimental Setup
LoRA configuration for all experiments.
All experiments were done with the Databricks MosaicML composer, streaming and 1lm-foundry libraries in conjunction with the HageingFace peft library on 32xH100-80GB GPUs.
We targeted all trainable modules inside each of the L Llama transformer blocks:
(W w W: W:l), w Wip to achieve a constant scaling factor 7r = across ranks. We use lora dropout=0.05.
gate> up, down)) We used ranks of r 16,64,256 and set = 2r,
For both the Code CPT and Math CPT settings, we train the model once for 20B tokens. We then perform individual cooldowns using intermediate checkpoints as follows: We set target max training duration (e.g. billion tokens), and define the last 20% of max training duration the cooldown period. We then retrain from the latest available checkpoint prior to the cooldown period.
Code CPT Llama-2-7B trained the StarCoder-Python dataset.
seq len:
seq_len: 4096
optimizer: decoupled_ _acuupldd_loowwww lionw (betas=(0.9, 0.95))
learning_ rate: 1.0e-05 for LoRA and Full Finetuning
scheduler: inv inv_sqrt_wwitt _armuup (t s0ale= t_ 1o0mpp =_coollomm=000bba
alpha f decay=1, alpha cooldown=0). We note that this ends up looking very much like
trapezoidal schedule.
weight_decay: 1.0e-06
precision: amp bf16
global_train train til__trainnttchh_ size: 192
device_ microbatch size:
gradient _eriot___ippngg... norm (threshold=1)
num gpus: 32
LR Scheduler: Inverse square root with warmup twarmup 500 batches, tscale 500 batches,
Leooldown 5200 batches Offeceay = 1.0 fenoldown 0.0
Math CPT. Llama-2-7B trained on the OpenWebMath dataset.
max max__see___ len: 4096
optimizer: decoupled lioww (betas=[0.9,, 0.95])
learning_rate 1.0e-05 for full finetuning, 4.0e-05 for LoRA
scheduler: inv_sqrt with warmup (t scale=1000ba, warmup=1000ba, cccooloow==000ba
alpha f decay=1, alpha f cooldown-0). We note that this ends up looking very much like a trapezoidal schedule.
weight weight_decay: decay: 0
precision: amp_ bf16
global train batch size: 192
device train microbatch size:
gradient _ecppptt__liiiggg..___ norm (threshold=1)
num gpus: 32
Code IFT: Finetuning Llama-2-7b on the Magioodee--voo---ssnnn------0 dataset
max__seq__aa 4096
optimizer: decoupled lionw (betas=(0.9, 0.95))
learning_ratte rate: 20-4 for rank r 16, 64 and 1c-4 for 256 a 2r 512 (due instabilitics/loss
spikes at. 2e-4)
20

---

Published in Transactions on Machine Learning Research (08/2024)
scheduler: Lisine_withe warmup warmup (alpha f=0.01, t_ warmup=0..ddr)
Sright___cay:: 0
precision: amp bf16
global_train batch_ahtiii size: 192
device_trand microbatch size:
gradient _ciippigg_iipngg norm (threshold=1)
num gpus: 32
Math IFT: Finetuning Llama-2-7h on the MetaMathQA dataset
seq_len: 1024
optimizer: decoupled lionw (betas=[0.9,, 0.95))
learning_ rate: Full finetuning: 1e-5, LoRA: le-4 for 16, 64, 5e-5 for 256 due instabilities.
scheduler: cosine with warmup (alpha f=0.01, warmup=0.ldur)
weight_decay: decay: 0
precision: amp_bf16 bf16
global_ton train batch size: 768
device_ train_ microbatch size: size:
gradient _clppp_cllpp. norm (threshold=1)
num num_ 32
A.1 Training the input and output embedding layers.
Vanilla LoRA and other popular methods such QLoRA Dettmers et al. 2024 often do not train the input and output embedding layers. Recent open-source work T1 the other hand, shows that it might be beneficial to supplement LoRA with full finetuning of these two modules (additional ~ 200M parameters for a 7B model). We view this approach as a hybrid of LoRA and full finetuning, and therefore leave its empirical investigation for future work. Morcover, this hybrid approach involves furthor hyperparameter optimization: the input and output layers require tuning their own separale learning rates, which should typically be 2-10X smaller than the LoRA learning rates (training with a single learning rate results in instabilities).
B Learning rate searches
We perform a learning rate sensitivity analysis for Llama-2-7B, trained for two epochs on the code and math IFT datasets, and followed by HunanEval and GSM8K evaluation, respectively. Fig. [S] shows that LoRA improves monotonically with learning rate up to value at which training diverges, with best learning rates of 5 for code and 2 for math.
On both datasets, these best LoRA learning rates are underperformed by four alternative full finetuning learning rates. The best full finetuning learning rates are 5e-5 and le respectively, an order of magnitude smaller than LoRA. For LoRA, we cannot find alternative learning rates that achieve at least 90% of the best learning rate's performance. For full finetuning, there are two viable alternative learning rates for code and three for math.
Note that in these experiments, the LoRA models target all modules but the W gate: with a 32 which should preferably be higher for 64.
B.1 Learning rate sensitivity analysis across optimizers
We compared the AdamW and Decoupled LionW optimizers by training for two epochs of Magicoder- Instruct-110K using different learning rates. We found that Decoupled LionW performed better on HumanEval for both LoRA and full finetuning, and across learning rates, as seen in Fig. S2
https://unsloth attbbl/g//otttrrerriiinnggg see also the following blogpost. https://xwx. anyscale com/blog/ Fine-tuning- 11ms- -lor- full-uaaaee wallll-apaatte--ee in- harpphh-anslysis--ee with--iaa--tt--a----a-- Niederfahhrenhost 2023)
21

---

Published in Transactions on Machine Learning Research (08/2024)
A
1-ama-2-7b, Magiccoder-Evo-nnsstu------ (2ep)
Llama-2-7b, MetaMathQA (2ep)
0.6
Accuracy on HumanEval
Accuracy on GSM8K
0.5
0.3
0.4
0.3
0.2
base
0.2
base
Full
Full
LoRA (1364, modulessal))
Div.
(r=84, modules=all
Div.
DIv.
10
10"
10"
10
10"
Leaminn Rate
Learning Rate
Figure S1: LoRA is more sensitive to learning rates compared to full finetuning. Llama-2-7B
models (A) trained on Magicoder- Bovo--nssrruct-11000 (Wei et al. 2023) and evaluated on HumanEval,, (B)
trained on MelaMathQA (Yu et al., 2023) and evaluated on GSMSK. Experiments here are performed with
LionW; see Fig. S2 for a comparion to AdamW.
Model: Ilama.2-7b Dataset: Maggicoder-Evo--nsstuut------ (2ep)
Learning: HumanEval
Forgetting: Language Understanding
0.5
Language Undensanding
Megisoder paper
HimanFool
0.3
0.4
(rw64, modileseal)
Full Adam
LoRA Adam (r=64, modiles=al)
Magicader
paper LA
000
Diverged Diverged
0.3
Diverçed Diverged
10
10:5
10"*
10
10
20"5
10-4
10-1
Leamting
flate
Figure S2: Comparing LionW to AdamW across learning rates for two epochs of the Magicoder-Evol- Instruct-110K dataset. Left: HumanEval; Right: Average of "Language Understanding" benchmarks in the MosaicML evaluation gauntlet. Both mothods peak at the learning rate used in the original paper (Wei et al.
2023
B.2 The importance of the a parameter for LoRA
We found that the performance of all models was particularly sensitive to the LoRA hyperparameter. Figure S3 shows two experiments on two separate datasets Slagicoder-Evo--nsstreee and Open Web Math) for LoRA with rank 256. In both cases the best accuracy achieved when 2r.
22

---

Published in Transactions on Machine Learning Research (08/2024)
Llama-2-7B trained on Magicooeer-Evolinsttuue (4ep)
Llama-2-7B trained Open Web Math
0.30
0.4
0.25
HumanEval pass
0.3
0.20
GSMBK
0.15
0.2
0.10
0.1
setting
2566, d= 32
0.05
Baseline
256, == 256
LoRA In4o-05, 256.0 a:32.0 LoRA Ir4e-05, r5nk-266.0. a:512.0
0.0
m255, a 512
0.00
1c-05
0.0001 Leaminng Rate
5e-05
0.0005
0.25
26
Training Tokens (Billions)
(a) Jointly sweeping over LoRA a and learning rate. The optimal choice 2r (blue).
(b) Continued pretraining with two different choices of where 2r is best (blue).
Figure S3: LoRA performance is sensitive to the hyperparameter. We show that for Code IFT (a) and math CPT (b) an a that is scaled with rank such that a 2r leads to the highest accuracy.
23

---

Published in Transactions on Machine Learning Research (08/2024)
C Finetuning on the Tülu-v2-mix dataset
We finetuned Jlama-2-7b models on the Tülu-v2-mix (Lvison et al. 2023), dataset which contains a variety of finetuning datasets containing chain of thought reasoning, multi-turn assistant conversations, math and science problems, code, and more. There are roughly 326k samples this dataset.
As in all main experiments, we compared full finetuning and LoRA 16, 64, 256, targeting all transformer modules. For each of the four experimental conditions, trained model for up to 6 epochs and evaluated it after 2, 4, and 6 epochs. Different from the main experiments, the checkpoints evaluated are "hot" and are not cooled down for cach training duration.
As in the original paper (Ivison et al. 2023). we assess math capabilities with GSM8K Cobbe et al. 2022 STEM, humanities, and social science capabilities as the average of 57 subjects of the Massive Multitask Language Undorstanding (MMLU: Hendrycks al. (2020)): ) and conversational capabilitios with Multi- Turn Benchmark (MT-bench (Zheng al.) (0224) which includes 80 multi-turn conversations where the model responses are evaluated automatically by GPT-4. We also compute the same average forgetting score as in all other datasets in this paper.
Since datasets like Tülu-v2-mix are where LoRA is mostly used, we ask: can LoRA, even with a low rank, achieve full finetuiing-acuuracy both in specific domains and in general conversational capabilities?
C.1 Experimental setup
A After an initial learning rate sweep, we chose the following hyperparameters:
max_seq_len: 4096
optimizer: decoupled lionw (betas=[0.9, 0.95))
learning_raae Full finetuning: 5e-6; LoRA le-4
scheduler: cosine__iiie warmup (alpha f=0.01, t warmup=0.ldur)
weight_decay: 0
precision: amp_bf16
global_ train batch size: 192
device_trren tis__tain_mmiiione microbatch _iie: 6
gradient _ciiptt___ippggg...__ norm (threshold=1)
num gpus: 32
C.2 Results
First, we find that on MT-bench (Fig. S5 both LoRA and full finetuning meaningfully improve upon the base model (2.74), starting from the second epoch and improving only slightly when trained for longer. Crucially, within one standard crror of the mean of the full finetuning model (computed with 160 datapoints 80 questions x 2 turns). That is, one can achieve full finetuning conversational capabilities with r= 16. The caveat is that only 80 questions appear in this benchmark and that the variance, within model, is high.
all LoRA models
In GSMSK (Fig, S6ab, again, all models significantly improve upon the base model (0.145). Remarkably, even this specific domain, LoRA and full finetuning are overlapping, with the best model being LoRA 256 at epoch 4, which is followed by full finetuning at epoch 2. Here too, as in the other math datasets in the paper, there is an ordering by LoRA rank.
In MMLU (Fig. S6b full finetuning and LoRA are overlapping with LoRA r= 64 as the best model (epoch 4), followed by full finetuning at epoch 2. Here there is no ordering by rank.
As for forgetting (Fig. S7 we find an overall mild forgetting compared to the rest of the datasets in the paper. At two epochs, full finctuning does better than LoRA. The former starts to degrade at epoch At epoch 6, the findings of the main paper are replicated: full finetuning forgets the most and we find clear ordering of forgetting by rank.
24

---

Published in Transactions on Machine Learning Research (08/2024)
(HelllaSaag + -RR- RRC-challenge- WinoGrande)
Llama-2-78 trained Tülu-v2-mix
0.67
base modcl
0.66
0.65
0.64
LoRA (r=16)
0.63
LoRA (r=64)
LoRA (r-26)
Full Finetuning
0.62
2
3
6
Epochs
Figure S4: LoRA forgets less even on more diverse IFT dataset like Tülu-v2-mix. We plot the average forgetting score, same as in all other datasets, as a function of training duration.
Across all evaluations learning and forgetting full finetuning is the best model at epoch 2, and only degrades afterwards. LoRA, on the other hand, needs 4 epochs to train, mirroring the findings in the main part of the paper. LoRA r = 16 seems to offer competitive conversational capabilities, and minimal forgetting, but it underperforms in domain-specific knowledge like math. Future work should seck to understand why this the case.
Llama-2-7B trained Tülu-v2-mix
Average ITT-Bench Score
LoRA (r=16)
LoRA (r=64
LoRA (r2256)
Full Finetuning
4
Std. Error (Full Finetuning)
base model (from paper)
2
2
3
4 Epochs
5
6
Figure S5: Average MT-bench score with GPT-4 as judge, calculated over 80 questions with two turns each. Base model value reported in the MT-bench paper. We note that the Tülu paper reports 6.3 MT-bench value from full finetuning of Llama-2-76 base model, which is only slightly exceeding the standard error from our average score.
25

---

Published in Transactions on Machine Learning Research (08/2024)
Llama-2-7B trained on Tülu-v2-mix
Llama-2-7B trained on Tülu-v2-mix
0.51
0.325
0.50
0.300
Accuracy in GSMBK
MMLU (Average)
0.275
0.49
LoRA (r=16)
(r=16)
0.250
LoRA (r=64)
LoRA (r=64)
LoRA (r=256)
0.48
LoRA (F=256)
0.225
Fineluning
Full Fineluning
0.200
0.47
0.175
0.46
0.150
base model
base model
2
3
5
3
6
Epochs
Epochs
(a) Accuracy in GSMSK.
(b) Average of MMLU benchmarks.
Figure S6: On Tülu-v2-mix, LoRA and full finetuning both improve upon the base model and perform comparably.
Llama-2-77 trained on Tülu-v2-mix
3(HellaSwag ARC-challenge WinoGrandel
0.67
base model
0.66
0.65
0.64
LoRA (r=16
0.63
LoRA (r=44
LoRA (56)
Finetuning
0.62
3
Epochs
Figure S7: LoRA forgets less
diverse IFT dataset like Tilu-v2-mix. We plot the average forgetting score, same as in all other datascts, as a function of training duration.
26

---

Published in Transactions on Machine Learning Research (08/2024)
D Supplementary tables
Table S1: Searcoder-Pythoor Results (HumanEval pass@
Num. tokens (billions)
0.25
0.50
1
4
8
16
20
Condition
LoRA (r=16)
0.143
0.144
0.141
0.141
0.154
0.159
0.162
0.162
LoRA (r=64)
0.142
0.146
0.141
0.153
0.157
0.176
0.194
0.196
LoRA (r-256)
0.144
0.142
0.143
0.159
0.159
0.208
0.211
0.224
Full Fineluning
0.152
0.153
0.172
0.181
0.218
0.258
0.255
0.263
Table S2: Starcoder-Python Results (Forgetting Average)
Num. tokens (billions)
0.25
0.50
8
16
20
Condition
LoRA (r=16)
0.645
0.642
0.645
0.642
0.644
0.640
0.638
0.635
LoRA (r=64)
0.646
0.644
0.646
0.646
0.639
0.634
0.626
0.626
LoRA (r=256)
0.644
0.645
0.643
0.639
0.636
0.630
0.618
0.617
Full Finetuning
0.625
0.624
0.625
0.616
0.599
0.583
0.551
0.545
Table S3: OpenWebMath Results (GSM8K)
Num. tokens (billions)
0.25
0.50
1
4
8
16
20
Condition
LoRA (r=16)
0.162
0.157
0.161
0.155
0.165
0.156
0.152
0.158
LoRA (r=64)
0.163
0.167
0.150
0.166
0.164
0.168
0.179
0.163
LoRA (r=256)
0.162
0.161
0.140
0.170
0.193
0.196
0.203
0.202
Full Finetuning
0.155
0.152
0.165
0.158
0.224
0.238
0.283
0.293
27

---

Published in Transactions on Machine Learning Research (08/2024)
Table S4: OpenWebMath Results (Forgetting Average)
Num. tokens (billions)
0.25
0.50
4
8
16
20
Condition
LoRA (r=16)
0.640
0.641
0.646
0.641
0.643
0.641
0.636
0.637
ToRA (r=
0.640
0.640
0.638
0.637
0.643
0.634
0.634
0.627
LoRA (r=256)
0.638
0.638
0.637
0.634
0.633
0.620
0.620
0.616
Full Finetuning
0.634
0.634
0.640
0.630
0.629
0.619
0.613
0.618
Table $5: Magiooddr--roo--sstuu---01K Results (HumanEval pass@1)
Epoch Condition
2
4
8
16
LoRA (r=16)
0.197
0.275
0.358
0.338
0.324
LoRA (r=64)
0.249
0.339
0.417
0.392
0.405
LoRA (r-256)
0.299
0.385
0.498
0.437
0.466
Full Finetuning
0.302
0.464
0.470
0.497
0.416
Table S6: Magicoder-Evye -isrrrder--------nstt------ Results (Forgetting Average)
Epoch Condition
2
4
LoRA (r=16)
0.653
0.648
0.652
0.646
0.609
LoRA (r-64)
0.652
0.651
0.632
0.580
0.510
LoRA (r=256)
0.655
0.659
0.631
0.552
0.517
Full Finetuning
0.595
0.579
0.512
0.446
0.414
Table S7: MetaMathQA Results (GSM8K)
Epoch
2
4
8
16
Condition
LoRA (r-16)
0.447
0.528
0.580
0.578
0.569
LoRA (r=64)
0.527
0.588
0.624
0.624
0.595
LoRA (r=256)
0.557
0.607
0.625
0.634
0.594
Full Finetuning
0.604
0.641
0.642
0.619
0.599
Table S8: MetaMathQA Results (Forgetting Average)
Epoch Condition
2
4
16
LoRA (r=16)
0.628
0.617
0.616
0.616
0.596
LoRA (r=64)
0.617
0.609
0.608
0.586
0.568
LoRA (r=256)
0.613
0.607
0.599
0.584
0.567
Full Finetuning
0.598
0.599
0.590
0.572
0.559
28

---

Published in Transactions on Machine Learning Research (08/2024)
Table S9: vix Results
Table S10: Tülu-v2-mix MT-Bench
Table S11: Tüln-v2-mix MMLU
Epoch
2
4
6
Epoch
2
4
6
Condition
Condition
LoRA (r=16)
5.681
5.997
5.712
LoRA (r=16)
0.491
0.502
0.504
LoRA (r=64)
5.597
5.725
5.944
LoRA (r=64)
0.503
0.509
0.504
LoRA (r-256)
5.788
5.834
5.894
LoRA (r=256)
0.502
0.496
0.492
Full Finetuning
5.825
5.838
5.862
Full Finetuning
0.507
0.504
0.502
Table S12: Tülu-v2-mix GSM8K
Table S13: Tülu-v2-mix Forgetting Average
Epoch Condition
2
4
epoch condition
2
4
6
LoRA (r=16)
0.251
0.275
0,280
LoRA (r=16)
0.650
0.657
0.657
LoRA (r=64)
0.285
0.270
0.295
LoRA (r=64)
0.649
0.655
0.647
LoRA (r-256)
0.296
0.335
0.301
LoRA (r=256)
0.653
0.649
0.629
Full Finetuning
0.324
0.291
0.303
Full Finetuning
0.660
0.652
0.621
29

---

Published in Transactions on Machine Learning Research (08/2024)
E
Supplementary Figures for SVD Analysis
Singular Values
Cumulative Explained Variance
Base Rinetuned
DIfT.
Cumulative Explained Variance
Value
Finetuned Diff.
2000
4000
Index
Number Singular Values
Figure S8: SVD analysis for 4096 4096 matrix Wg at layer 26. Left: singular values for base weights, finetuned weights, and their difference. Right: cumulative explained variance. Notice that for all three matrices, rank > 1500 is needed to explain 90% of the variance.
Spectra of random matrices
MAD between spectra of random matrices
difference
A+ 0.10
log10 7 A AA+
0.6
-6-
-8-
0.2
-10
400
600
log10 (c)
(a) Spectrum of A and A + eB as well eB for 0.1. Notably, A, A, eB are high rank.
(b) Mean absolute difference between spectra of A and cB for various c.
Figure S9: Analyzing the spectra of the sum of two 1000 x 1000 Gaussian i.i.d matrices. A and B are 1000 1000 random matrices with i..dd standard normal Gaussian entries.
30

---

Published in Transactions on Machine Learning Research (08/2024)
F Solution Generation Diversity on HumanEval
For the bost set of Llama-2-7B models trained in MagicCoder we evaluate how their pass metric in the HumanEval benchmark increases as we increase the parameter which controls the acceplance criterion. The passåk metric (Chen et al. 2021 is defined as
pass@k = E
(1)
where n the number of generations, e number correct generations and determines the size of the sample set of generations considered for acceptance. Assuming we sample from the model outputs, i.e. sampling temperature 7 0, then incrcasing le will increase the diversity of generations, and incrcase the likelihood of a passing generation being present in random subset of size k.
reports passãk for the LoRA models trained in the MagicCoder dataset as well as the base Llama-2-77 model. For all models, as we increase k, the pass@k consistently and monotonically improves. Finetuned models scores are substantially higher than the base model. At 1, Full finetuning outperforms the LoRA model whose scores are ordered from largest to smallest rank, as expected. observe all models improving their passãk scores and the gap between them reducing when k > 16. We note that full finetuming is superior across all values of with temperature 0.8. This complements the results in which used temperature of 0.2 and pass@1, where the improvements upon 256 at epoch 4 are less clear.
Figure S10
increases we
Llama-2-7B Epoch=4
256 Generations with Temperature 0.8
0.8
pass@k Tess@k-Humaane HumanEval
Model
0.6
Base
Full finetuning
LoRA (r=16, All)
0.4
LoRA (r=64, All)
LoRA (r=256, All)
0.0
2
8
16 k
32
64
128
256
HumanEval pass@k for models trained on the Magicoder datasct. For every model, we sample 256 independent generations with temperature 0.8.
Figure S10:
31

---

Published in Transactions on Machine Learning Research (08/2024)
G Training Datasets
G.1
MetaMathQA (Math IFT)
The MetaMathQA dataset YYu et al. contains 395,000 samples that are bootsrapped from the GSM (Cobbe et al., 2021) and Math (Hendrycks
et al.
2021 training sets. These samples are augmented by GPT-3.5 using the following methods:
Answer Augmentation (155k samples, Yu et al. (2023) ): this method proposed by the MetaMathQA authors generates multiple reasoning paths for given mathetical question and filters for generated reasoning paths that contain the correct final answer.
Rephrasing (130k samples, (Yu et al. 2023) ): this method proposed by the MetaMathQA authors uses GPT-3.5 to rephrase questions. They check for the correctness rephrased questions by using few-shot Chain of Thought prompting to compare reasoning chains and proposed answers with ground truth answers.
Both Self-Verification. (Weng al., 2022) and FOBAR (Jiang al., 2024b) fall under the category of "backward reasoning," " where the question starts with given condition and requires reasoning backwards to solve for unknown variable. In order to generate new mathematical questions, numerical value in the original question is masked as variable X, and the question is rephrased accordingly.
Self- Verification (55k samples, Weng et al. [2022 ): the question is rephrased into declarative statement followed by a question about the masked variable
FOBAR (55k samples, Jiang al. (024bb): this approach is similar Self- Verification but directly appends the answer to the question using the template "If know the answer to the above question is Ac, what is the value of unknown variable x?"
MetaMathQA samples are organized by 4 columns: type, triginal_qqestionn query and response.
We include two full examples below:
Example G.1: MetaMathQA
type: CSH_Rephrased
Scott wants to prepare and freeze large portion of ratatouille. He purchases 5 pounds of eggplants and 4 pounds of succhini rate of 82.00 per pound from the farmens' market. Additionally, be needs 4 pounds of lomatoes that cost $3.50 per pound, 3 pounds of onions priced $1.00 per pound, and pound of basil sold for $2.50 per half pound. this quantity yields 4 quarts, what the cost per quart?
query:
Seagiial_queettoon Scott eggplants and pounds succhini $2.00 pound. He needs pounds tomatoes farmers' market he buys pounds of $1.00 pound he needs 3 pounds. Then needs pound basil which sold $2.50 per half pound. this yields quarts, how much does each quart cost?
make and freeze a large batch of ratatouille.
$3.50 a pound. The onions
response: The cost of the eggplants is pounds $2.00 per pound $10.00. The cost the zucchini is pounds $2.00 per pound = $8.00. The cost of the tomatoes 4 pounds x $3.50 per pound = $14.00. The cost of the onions is 3 pounds $1.00 per pound = $3,00. The cost of the basil is pound ($2.50 per half pound x 2) = $5.00. The total cost for all the ingredients is $10.00 + $8.00 $14.00 + $3.00 + $5.00 $40.00. Since this quantity yields 4 quarts, the cost per quart is $40.00 / quarts $10.00 per quart. # #### # 10 The answer is: 10
32

---

Published in Transactions on Machine Learning Research (08/2024)
Example G.2: MetaMathQA
type: KATH_SV
query; a school bus leaves school with X students on board, and one-half of the students get off the bus at each of the first three stops, The number of students remaining on the bus after the third stop is 6. What is the value of unknown variable X?
or original, question: If school bus leaves school with 48 students on board, and one-half of the students get off the bus at each of the first. three stops, how many students remain on the bus after the third stop?
response: To solve this problem, we need to determine the value of which represents the initial number of students on the bus. Let's break down the information given: Number of students remaining the bus after the third stop: 6 Number of students getting off the bus at each stop: 1/2 Number of stops: 3 We can up the equation as follows: Number of students on the bus (1/2)* 3 Number of students remaining the bus after the third stop (1/2)" 6 Let's simplify and solve for (1/8) To isolate we multiply both sides of the equation by 8: (1/8)
The value of
The answer 48
G.2 Magiccdee-Evo--Isstuu---0 (Code IFT)
Example G.3:
Angicoder-Evoo-Insttrree
instruction: Please amend subsequent Python script so includes 'whiie'' loop rather than existing 'for' loop, which iterates through the items an integer list. The script currently has bug where it attempts to print an object that is outside the bounds of the list. Fix this error and modify the script to use 'while' instead of 'for" loop. Ensure your script correctly handles empty lists.
"'python
Establish integer list
arr [1, 3, 4]
Determine the length of the list len(arr)
Traverse the list and output each individual element ratge( print (ern(()rr()) r..
response:
'<4 "python # Establigh integer list
[1,
Determine length
lan()
Initialize index at
Traverse the list and output each individual element while n: print [i]) +=
In the given code, tries to access 'arr [n] which is out of bounds python list indexing starts from 0 and enda at n-l for list of length Also, I used 'while' loop per request. By initializing 'i=0' before the loop and i += 1' within the loop, can iterate over the list. This iteration continues until 4 is less than 'n', the length of the list. The 'while" loop will proceed the list empty it correctly handles empty lists.
33

---

Published in Transactions on Machine Learning Research (08/2024)
G.3 Starcoder Python (Code CPT)
Example G.4: Starcoder-Python
<< "'yython
н*
function invoking
u*n
import base64
import logging
import requests
from flask import current_app
from requests import exceptions
logger logging. gettogger(__mmmee___
ORC_URL URL
"bbreshhtps:///
REPO URL
"Ahref htwp:///wwe....eeeooeee/////t/te
def get_request(ur1, parans)
***
get request
logger. request, connect url: Ys", url)
try:
response requests. get (url, parans-params)
return True, response
except exceptions. ConnectionError as
logger error
False, cconecction error'
except 10Error as err:
logger.error(err)
return False, 'I0 error'
rCr
more functions truncated...
34

---

Published in Transactions on Machine Learning Research (08/2024)
G.4 OpenWebMath (Math CPT)
Example G.5: Optn
url:
-2-aces-2 -eii--i--nnn--e- and-1- --uee---ee-nnee -n--a
fivercard-poker--andd pan-
text: # Probability of getting Aces, Kings and Queen in five card poker hand (Part IT) So reworked my formula in method after getting help with my original question Probability of getting Aces, Kings and 1 Queen five card poker hand. But still getting results that differ...ltthough they much much closer than before, but must still be making mistake somewhere in method Anyone know what is? Method SP capp 2K capp 1Q) P(Q[2A 'cap (())(())) $8= \frac [1] (12))frcc(( (choose 2) (46 (choose 1) (choose 3) (ffra (4 (choose 2) (4 \choose (52 (choose 5)88 $8= /frac (16ac(666112000) (172966666666)) (1500060
$8= 4 4.685642 10[-5]88 Method 2 $8 (f)rrrc (4 (choose 2 (4 (choose 2 (choose (52 (choose 5} (5444(3)541 $$5.540678 10{-5)88 Please make an effort to make the question self-contained and provide a link to your earlier question.
Sasha Oct 28 '12 at 19:56 think we would rather alve you edit. your initial question by adding your new progress. This avoids having loss of answer and keeps track of progress Jean-Sébastien Oct 28 '12 at 19:56 But there already answers to original question so those answers would not make sense now that I am using new formula for method 1. sonicboom Oct 28 '12 at 20:03 Conditional probability arguments can be delicate. Given that there are exactly two Kings, what's 's the $468 doing? That allows the possibility of more Kings.
André Nicolas Oct 28 '12 at 20:26 The $468 is because have already taken two kings from the pack leaving with 50. And now we have chosen 2 aces and we have to pick the other 1 card from the 50 remaining cards less the 4 aces?
sonicboom Oct 28 '12 at 20:42 show more comment $8 (1rccfffra((( (choose 2)44 \choose 1)) (choose 3) (frcc (4 (choose 2) [48 /choose (52 \choose 5)88 If you wrote this 88 (8rfrac[[4 \choose (48 (chouse 352 (chouse 5) Afrac (4 (choose (44 \choose 1)) (48 (chouse 3) (frcffrc( (44 Achoose (40 \choose (4) \choose 1388 it might be more obvious why they the same.
2014-03-07 11:01:44
35

---

Published in Transactions on Machine Learning Research (08/2024)
There is often some confusion about the memory gains that vanilla LoRA offers both in theory and in practice. In Appendix [H] we discuss some of the theoretical benefits of LoRA, and show how it can enable training both on GPUs with less memory and on fewer total GPUs (in the multi-GPU setting). In Appendix we show how LoRA in practice leads to memory savings relative to full finetuning, but can in fact lead to slower throughput for particular hardware and soft settings.
H Theoretical Memory Efficiency Gains with LoRA for Single and Multi-GPU Settings
Modern systems for training neural networks store and operate on the following objects (following the conventions in Rajbhandari et al. 2020)). Most memory requirements relate to model states, which include:
parameter weights
gradients
higher order optimization quantities such as optimizer momentum and variance in the Adam optimizer, and the momentum in the Lion optimizer
The remaining memory requirements come from the residual states:
activations (which depend on batch size and maximum sample sequence length) lemporary buffers for intermediate quantities in the forward and backward pass.
which will require more memory when increasing the batch size and maximum sequence lengths.
LoRA offers memory savings with respect to the model states, The next two sections describe these memory savings in the single GPU and multi-GPU setting with examples loosely inspired by |Rajbhandari al. (2020)
The data stored at single precision includes:
a "master copy" of the tuned parameter weights
the gradient
all optimizer states (both momentum and variance for Adam, and just momentum for Lion)
For simplicity, we do not consider mixed-precision training, which involves storing critical data at single precision (fp32; 4 bytes per number) while performing some computations at half precision (fp16 or bfloat16; bytes per number).
H.1 Training on a Single GPU
In the single GPU setup, the difference in memory requirements between LoRA and full finetuning is particularly drastic when using the Adam optimizer (Hu et al. 2021 Rajbhandari et al. 2020).
Storing the master weights in fp32 requires 4 bytes per parameter, while storing the gradient in fp32 requires bytes per tuned parameter. In order to maintain the optimizer state in fp32 for Adam, 8 bytes per tuned parameter are required; bytes for the momentum term, and bytes for the variance term. Let be the number of model parameters. Therefore, in the Adam full finetuning setting of v 7B parameter model, the total memory requirements are at least roughly 4 x V + 4 x & + v 112 GB.
The Lion optimizer only uses momentum term in the gradient calculation, and the variance term in Adam therefore disappears. In the Lion full finetuning setting of a v = 7B parameter model, the total memory requirements are therefore roughly x + x + x = 84 GB.
LoRA, on the other hand, does not calculate the gradients or maintain optimizer states (momentum and variance terms) for most of the porameters. Therefore the amount of memory used for these terms is drastically reduced.
36

---

Published in Transactions on Machine Learning Research (08/2024)
7B Training
GPU
8 GPUs
16 GPUs
32 GPUs
64 GPUs
Adam
112 GB
14 GB
7 GB
3.5 GB
1.75 GB
Adam + LoRA
15.12 GB
1.89 GB
0.945 GB
0.4725 GB
0.236 GB
Lion
84 GB
10.5 GB
5.25 GB
2.625 GB
1.3125 GB
Lion + LoRA
14.84 GB
1.855 GB
0.9275 GB
0,464 GB
0.232 GB
Table S14: Theoretical memory required to store the model and optimizer state during training for a 7B parameter model. Note that the numbers exclude memory needed to store activations. FSDP sharding the parameter and optimizer states across N devices results in less memory usage relative to LoRA. LoRA on the other hand enables training on GPUs with far less memory and also enables training without. needing as many GPUs to shard across.
A LoRA setting with Adam that only tunes matrices that are 1% of the total parameter count (e.g. v v= =BBB base model with 70M additional parameters used by LoRA) requires roughly 4 x 4 0.01) +4x v x 0.01+ + 8 x x 0.01 = 29.12 GB of memory. Theorctically this can be reduced further to 2 x v + 16 x v x 0.01 = 15.12 GB if the non- non-tuned parameter weights are stored in bfloal16. We use this assumption for the subsequent examples.
Note again that these numbers do not take into considcration sample batch size or sequence length, which affect the memory requirements of the aclivalions.
H.2 Training on a Multiple with Fully Sharded Data Parallelism
Past approaches for training LLMs across multiple GPUs include model parallelism, where different layers of the LLM are stored different GPUs. However this requires high communication overhead and has very poor throughput (Rajbhandari et al. 2020). Fully Sharded Data Parallelism (FSDP) shards the parameters, the gradient, and the optimizer states across GPUs. This incredibly efficient and actually is competitive with the memory savings offered by LoRA in certain settings.
FSDP sharding the parameter and optimizer states across N devices results less memory usage relative to LoRA. LoRA on the other hand enables training on GPUs with far less memory and also emanes training without needing as many GPUs to shard across.
For example, in the Adamn full finetuning setting of V = 7B parameter model on 8 GPUs with FSDP, the
total memory requirement for each GPU is roughly (4 x V + 4 x v + 8 w)/8 = 14 GB. This reduces further to 3.5 GB for FSDP with 32 GPUs (see Table $
The LoRA with Adam setup on GPUs (where V = 7B base model and there are 70M additional LoRA parameters) requires roughly (2 & + 16 x V x 0.01)/8 1.89 GB of memory per GPU. With 32 GPUs this decreases further to 0.4725 GB.
Standard industry level GPUs have on-device memory between 16 GB (e.g. V100s) and 80 GB (e.g. A100s and H100s). As Table $14 demonstrates, the per-GPU memory requirements for training 7B parameter model decrease drastically as the number of GPUs increase. The memory requirements for training a 7B model with Adam + LoRA on a single GPU are 15.12 GB, but the same per-GPU memory requirement for training a. 7B model with Adam but without LoRA on 8 GPUs is 14 GB. In this GPU scenario, the efficiency gains from LoRA disappear.
Table S15 applies similar calculations to 70B parameter model. Finetuning such large model on 8 GPUs is only possible using a technique like LoRA; where Adam requires 140 GB per GPU, Adam+ToRA requires 18.9 GB per GPU. The efficiency gains of LoRA relative to FSDP therefore depend on the model size and GPU availability /alliyyyyo costt considerations.
We do the same analysis for a 405B parameter model to highlight how LoRA is beneficial as model size scales (Table S16).
37

---

Published in Transactions on Machine Learning Research (08/2024)
70B Training
1 GPU
GPUs
16 GPUs
32 GPUs
64 GPUs
Adam
1.12 TB
140 GB
70 GB
35 GB
17.5 GB
Adam + LoRA
151.2 GB
18.9 GB
9.45 GB
4.725 GB
2.36 GB
Lion
840 GB
105 GB
52.5 GB
26.25 GB
13.125 GB
Lion + LoRA
148.4 GB
18.55 GB
9.275 GB
4.64 GB
2.32 GB
Table S15: Theoretical memory required to store the model and optimizer state during training for 70B parameter model.
405B Training
16
32
64
128
256
Adam
6480
810
405
202.5
101.25
50.625
25.3
Adam + LoRA
874.8
109.35
54.65
27.34
13.67
6.83
3.42
Lion
4860
607.5
303.75
151.875
75.94
37.97
18.98
Lion LoRA
858.6
107.325
53.66
26.83
13.42
6.71
3.35
Table S16: Thcorotical memory required store the model and optimizer state during training for 405B parameter model. Units are in gigabyles (GB)
LoRA Throughput and Memory Measurements
We report training efficiency comparisons between full finetuning and models trained with LoRA for various of choices of rank. We measured both the throughput (in tokens second) and peak active memory (in GB) for training runs representative of the experiments reported in the paper. We performed the runs using single node of 8x H100-800GB GPUs. We used a per-GPU micro batch size of and targeted all linear layer weights with LoRA (i.c. both Attention and MLP).
In Figure SS1 observe that there is significant gap between full finetuning and LoRA runs, related to the additional overheads of the LoRA computations. In general, LoRA leads to approximately 15% reduction in throughput for a given batch size. LoRA with higher ranks is slower than lower ranks across all batch sizes; this is particularly noticeable for rank r 512. Similarly, LoRA settings with higher batch sizes have slightly higher throughput relative to lower batch sizes. Some of the slowdown is intrinsically related to the overheads of performing LoRA, since in practice involves more computations intermediate activations. However, we note that we did not optimize the LoRA implementation and used the publicly available HuggingFace peft library, which might be amenable further optimizations that. could reduce the gap in throughput.
For peak memory, we notice that for small batch sizes, LoRA provides a substantial reduction in peak memory 40%). This is expected since the optimizer state significantly smaller when using parameter efficient methods. However, as batch size increases, the size intermediate activations increases proportionally, dominating the required memory. We limit the per GPU micro batch size to to prevent out of memory errors, so for batch sizes 64 and above, we perform gradient accumulation. This leads to the throughput and memory stabilizing for batch size 64 and above, with just around (~ 15% memory savings) for larger batch sizes.
38

---

Published in Transactions on Machine Learning Research (08/2024)
42500
Finetune
Full
40000
Peak Active Memory [GB]
LoRA (R=8)
(R+32)
Throughput [tokis]
37500
50
(R- 128)
(1=512)
32500
30
30000
27500
20
25000
32
128
256
18
64
128
256
Batch Size
Batch Size
Figure S11: Throughput and Memory Measurements for LoRA vs. full finetuning, (left) Training throughput measured in tokens per second across all 8 GPUs. (right) Peak active memory used by the training process in single GPU (max GPU memory is 80GB).
39


Reviews:
[{"review_id": "0AKEyrXuw4", "reviewer": "Reviewer_vfzD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- A large number of useful contributions are made throughout this paper. This is a very insightful paper that will be of great interest to the community.\n- The experimental setup is sound and the results are well presented. The authors fine-tune the models several times to reduce the variance in the results,\n  which is a good practice and will help to ensure the results are reliable. Additionally, the authors utilize open-access datasets and models, which will help\n  in the reproducibility of the results and foster further research in this area.\n- The hyper-parameter analysis for LoRA is a great addition. It's widely known that LoRA is sensitive to hyper-parameters, and the authors provide a\n  comprehensive analysis of the hyper-parameters that affect LoRA the most.\n  Interestingly, they find that selecting appropriate target modules is more important than rank itself. Future work may focus on understanding why this is the case.\n\n### Weaknesses\n\n- While the authors provide inference hyperparameters for HumanEval and GSM8K, they do not provide them for HellaSwag, Arc, and WinoGrande.\n  This information is vital for reproducibility and should be included in the paper.\n  Additionally, for HumanEval and GSM8K, the authors should provide top-p and top-k hyperparameters used, if any.\n- The analysis on sample diversity for LoRA could be improved. The authors utilize string match to measure diversity, which is a very limited metric,\n  especially for code generation tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide an in-depth analysis of a popular parameter-efficient fine-tuning method, LoRA.\nThey compare LoRA with full fine-tuning in both math and coding domains and find that LoRA tends to forget\nless the source domain, while learning less than full fine-tuning, which is more sample-efficient.\nThey also show that LoRA is a regularizer that promotes generation diversity.\nAnother interesting finding is that, contrary to popular belief, full-parameter fine-tuning finds high rank weight\nperturbations, which shakes one of the key motivations of utilizing LoRA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please include missing inference hyperparameters used for evaluation in the paper.\n- What is the impact on pass@k; k>1 for LoRA compared to full fine-tuning? Given the claim that LoRA promotes generation diversity,\n  it would be interesting to see how this affects the pass@k metric with ks greater than 1.\n  For this experiment, the authors should consider using a higher temperature parameter than 0.2.\n  [1] finds a temperature of 0.8 to be optimal for this task.\n  One way to show this is to graph the pass@k metric against k for both LoRA and full fine-tuning.\n\n[1] Chen et al., \"Evaluating Large Language Models Trained on Code\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lZYEK3XWun", "reviewer": "Reviewer_dMFP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and the key message is presented clearly, supported by strong empirical evidence.\n\n2. This paper conducts a deep analysis on LoRA and full-finetuning. One notable thing is that the authors have evaluated on many code and math datasets.\n\n3. The question that this paper tries to answer is important in practice. With the huge computational resources required to train LLMs, we often faces the problem as to what to choose between LoRA and full fine-tuning. I think this paper provides a valuable contribution on this front.\n\n\nWeaknesses:\n1. The authors have found that section 4.1 LoRA underperforms full fine-tuning on math and coding tasks. This seems to be expected, given that LoRA uses much less parameters. I wonder if this finding is novel or just verifying existing hypothesis?\n\n2. I think this paper lacks some discussion on the existing practice of fine-tuning on the match and code datasets. For example, one question would be when people fine-tune on these math or code datasets, what fine-tuning strategies do they often use? Are there tradeoffs that people have observed before or is this the first work? \n\n3. There is some ambiguity around the regularization here. In section 4.4, the authors mentioned that “we define regularization (loosely) as a training mechanism that keeps the finetuned LLM similar to the base LLM”. However, this does not seem to be the common meaning of regularization, which I think refers to techniques that helps the training trajectory easier. So my question is that why would people care about the regularization property as defined in the paper? It would also be good to add more clarification on what the authors mean by regularization in the paper. \n\n4. Beyond code and math tasks, I think another important and common fine-tuning direction is fine-tuning for chatting purposes. I wonder if the authors have done experiments on this. \n\n5. The paper concluded with some suggestions on practical takeaways for LoRA fine-tuning, i.e. Section 4.6. However, I find this part unrelated to the topic of this paper, which is to study the tradeoff between LoRA and full-finetuning. I would suggest the paper add more aspects on how practitioners can better choose between LoRA and full fine-tuning, given the observations from this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a comprehensive analysis on LoRA fine-tuning and full fine-tuning. Under various fine-tuning tasks, the authors make several interesting observations. On the downstream tasks, LoRA underperforms full fine-tuning. However, LoRA incurs less loss of the performance on pretraining tasks. Also, LoRA achieves stronger regularization compared to common techniques. The authors also analyzed the rank of perturbations from full fine-tuning and observed that they are much larger than the rank number used in LoRA, possibly explaining the gap between LoRA and full-finetuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses for details. For me the biggest concern is that this paper lacks a clear discussion on the existing studies comparing LoRA and full fine-tuning, e.g. performance and efficiency. I would suggest the authors describe the existing consensus regarding this tradeoff clearly in the introduction. If there is no study on this before, it should also be mentioned.\n\nAlso, the authors should clarify the meaning of the term “regularization” used in the paper. I think the current description is too vague and short.", "broader_impact_concerns": "I do not have concerns regarding the broader impacts of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jb3PQNQDI2", "reviewer": "Reviewer_eciD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The work investigates a large number of settings and benchmarks: continued pretraining vs instruction finetuning, benchmarks for Python code and maths generation, using multiple LoRA configurations per setting.\n- The authors attempt to understand why full finetuning learns more by investigating regularization effect of LoRA and looking at the rank of weight matrices post-training. Although the observation that \"less flexible models change less\" may seem trivial, this investigation provides additional insight into this specific finetuning method, and supporting evidence that LoRA finetuning is not always a good idea or easy drop-in replacement for full finetuning.\n- The authors provide practical recommendations for LoRA based finetuning (for IFT vs CPT), making this paper potentially useful reference material for practitioners - even if the used datasets are perhaps less common than some text datasets.\n\nWeaknesses:\n- The authors ask an important question in section 4.3: \"The nontrivial question is: do LoRA and full finetuning differ in how they tradeoff learning and forgetting? Can LoRA achieve similar target domain performance but with diminished forgetting?\". I don't believe that this specific question is answered in the paper as there is insufficient data to answer positively or negatively.\n\t- For code and math, it does like like we should use full finetuning (from section 4.1).\n\t- But: the authors do observe that there are outliers where LoRA performs as well while forgetting less. As far as I can tell, they do not draw conclusions from these observations however, nor make recommendations for when to use LoRA.\n\t- This work would have been stronger if full finetuning runs were run until they reach similar forgetting-or-learning performance, to enable direct comparison. At the moment the LoRA and full finetuning runs mostly live in different parts of the tradeoff curve. Of course, full finetuning is expensive so it is understandable that there are fewer points for this setup.\n\n- The authors show results for multiple ranks in the appendix, but as far as I can tell only for a single value for alpha for some Code runs, while keeping it to 2*rank for the maths runs. Given that there are many setup differences between code and maths runs, including LoRA hyperparameters, it may be less meaningful to compare LoRA performance between these two settings. That said, I'm not sure if using the same LoRA configuration in both domains is useful either.\n- The insights about forgetting and learning seem to depend strongly on the domain and dataset. The paper title makes the claim more general than it may be in practice. LoRA is often used in domains not related to maths and code generation, and we cannot say whether the same insights will hold there. Most LORA variations seem to look at GLUE or other text datasets. \"Forgetting\" is more difficult to test when going from text to text domains, but learning can probably be measured!\n\nI don't think restricting to \"just\" Llama-2 is a downside of this study, this is a large model that's expensive to train/finetune, and the authors try two very large versions. Obviously the insights would be more general if other models were tried as well but I don't think that's a prerequisite for acceptance.\n\nAdditionally:\n- I think the claims in the submission are accurate and convincing. My only (small) concern is that the general claim in the title has a caveat \"in code and maths generation tasks\". The authors provide clear evidence that LoRA learns less and forgets less, characterized by the learning/forgetting tradeoff curve. The study seems broad, looking at multiple datasets in domains in detail, even if it only tackles math and code (and not say, text generation in a specific domain). The authors make no claims that LoRA is as good as full finetuning, as only a handful of datapoints show this.  \n- I believe this study is interesting for TMLRs audience, especially the learning/forgetting tradeoff and the practical recommendations for LoRA based finetuning on small domains. Choosing whether to use LoRA or not based on specific points on the tradeoff curve could be useful in practice. Additionally, this study is one of the first to show that LoRA is not always a good drop-in replacement for full finetuning on specific domains.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work empirically investigate the effectiveness of Low-rank based finetuning of language models for code and maths domains. Authors look at performance on old domains under domain shift (forgetting) versus task performance+generation diversity (learning). The baseline each time is full finetuning of the LLM. Their investigation focuses on Llama 2 models of different sizes only.\n  \nThe authors find that LoRA underperforms full finetuning in code generation, but that it does mitigate forgetting, confirming the title of the paper. In some rare cases, LoRA-trained models sometimes perform just as well as full-finetuned models in the target domain, while forgetting less about the source domain. However, these outliers are not sufficient to say that LoRA learns just as much while forgetting less. \n  \nThe work also compares the regularization effect of LoRA to regularization techniques such as dropout, and find that LoRA compares favorably, also resulting in more diverse outputs than full finetuning.  \n\nThe authors hypothesize that the low-rank restriction proved effective in other tasks because the tasks were simple, and show hints that this holds by comparing the ranks of weights of LoRA-finetuned models to those of full finetuning models. Finally, they make concrete recommendations for finetuning with LoRA, namely to use it for instruction fine-tuning but to favor full finetuning for continued pretraining.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please provide more information about the experimental setup, especially the motivation for using different LoRA configurations between maths and code settings. Are hyperparameter choices motivated by the original works or benchmark datasets?\n- Please provide an answer for the question you ask in section 4.3, even if that answer is \"there is insufficient data to answer this question\".\n- For Figure 4C, we see performance on GSM8k decrease for most runs wrt the base model, for both full finetuning and LoRA. Does this indicate a mismatch between the GSM8k and WebMath dataset? It would be useful to provide more intuition for the lack of learning here as this figure seems to be the odd one out.\n- There are many knobs to tune for LLM finetuning, especially when switching from one domain to another. How were these parameters chosen? It would be beneficial to explain the training procedure in more detail if possible, including what is done with tokens+embeddings (if anything).\n- Related: Is the tokenizer frozen for code and math finetuning runs? In the domain shift setting, would a model benefit from adding new tokens or adapting the tokenizer, potentially reaching better performance or preventing forgetting if tokens for code/maths are added? See e.g. \"Efficient domain adaptation of language models via adaptive tokenization\", Sachidananda 2021. Some discussion on why these parameters are good under domain shift would strengthen the claims, IMO.", "broader_impact_concerns": "I dont believe there are ethical implications to this work that would require a more detailed impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts a comprehensive analysis on LoRA fine-tuning and full fine-tuning. Under code and math tasks, the authors empirically observed that LoRA learns less and forgets less. After author rebuttal, it received Accept, Accept, and Leaning Accept recommendations. 

All the reviewers agree that (1) the paper is well-written and the key message is presented clearly, supported by strong empirical evidence, (2) the question that this paper tries to answer is important in practice, and the paper is insightful and will be of great interest to the community. 

On the other hand, reviewers have asked many questions, and the authors have also promised many changes. However, the current draft has not reflected these. Therefore, the AC would like to recommend accept, depending on whether the promised changes have been made. 

The authors have promised adding additional new results, including:

1. the new analysis suggested by the reviewer, using temperature=0.8 and computing HumanEval pass@k for k=1, …, 256 for both LoRA (at different ranks, targeting all modules) and full finetuning.

2. the new experiment results on the Tülu-2 finetuning dataset, evaluating chat-quality with MT-Bench.

3. full finetuning experiments for shorter durations; we are currently running additional LoRA experiments with identical scaling factors across math and code (alpha=2r). 

4. rerunning the experiments in Fig. 4C with more tokens, to get closer to the regime in the Starcoder experiment. 

In terms of paper writing, the authors have promised to include the following revision:

1. detail the experimental setup and evaluation hyper-parameters used, and also link to the evaluation repositories we used, to ensure reproducibility.

2. revise Related Work section to review previous work studying continual learning on math and code datasets (via continued pretraining or instruction finetuning).

3. try to avoid the use of “regularization”, in favor of terminology that is more explicit about learning and forgetting in downstream tasks.

4. better streamline the connection between the core experiments and our hyperparameter sensitivity analyses and recommendations.

5. the introduction will review work comparing LoRA vs full finetuning, and make the point that the jury is still out.

6. Section 4.3 will state that our results cannot conclusively tell whether LoRA and full finetuning live on same or different tradeoff curves. It will also discuss potential avenues of future work given these results.
Decision: Accept (poster)