Package: loralib
LoRA: Low-Rank Adaptation of Large Language Models(For the radio communication technique, seeLoRa.)This repo contains the source code of the Python packageloraliband several examples of how to integrate it with PyTorch models, such as those in Hugging Face.
We only support PyTorch for now.
See our paper for a detailed description of LoRA.LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu ChenPaper:https://arxiv.org/abs/2106.09685Update 2/2023: LoRA is now supported by theState-of-the-art Parameter-Efficient Fine-Tuning (PEFT)library by Hugging Face.LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights.
This vastly reduces the storage requirement for large language models adapted to specific tasks and enables efficient task-switching during deployment all without introducing inference latency.
LoRA also outperforms several other adaptation methods including adapter, prefix-tuning, and fine-tuning.We obtain result comparable or superior to full finetuning on the GLUE benchmark usingRoBERTa (Liu et al., 2019)base and large andDeBERTa (He et al., 2020)XXL 1.5B, while only training and storing a fraction of the parameters. Click the numbers below to download the RoBERTa and DeBERTa LoRA checkpoints.RoBERTa baseFine-tuneRoBERTa baseLoRADeBERTa XXLFine-tuneDeBERTa XXLLoRA# of Trainable Params.125M0.8M1.5B4.7MMNLI (m-Acc/mm-Acc)87.687.5±.3/86.9±.391.7/91.991.9±.1/91.9±.2SST2 (Acc)94.895.1±.297.296.9±.2MRPC (Acc)90.289.7±.792.092.6±.6CoLA (Matthew's Corr)63.663.4±1.272.072.4±1.1QNLI (Acc)92.893.3±.396.096.0±.1QQP (Acc)91.990.8±.192.792.9±.1RTE (Acc)78.786.6±.793.994.9±.4STSB (Pearson/Spearman Corr)91.291.5±.2/91.3±.292.9/92.693.0±.2/92.9±.3Average86.4087.2491.0691.32Note: You still need the original pre-trained checkpoint fromHugging Faceto use the LoRA checkpoints.Fine-tuning numbers are taken fromLiu et al. (2019)andHe et al. (2020).  We include confidence intervals on results from our experiments. Please follow the instructions inexamples/NLU/to reproduce our results.On GPT-2, LoRA compares favorably to both full finetuning and other efficient tuning methods, such asadapter (Houlsby et al., 2019)andprefix tuning (Li and Liang, 2021). We evaluated on E2E NLG Challenge, DART, and WebNLG:Method# of Trainable ParamsE2E (BLEU)DART (BLEU)WebNLG (BLEU-U/S/A)GPT-2 M (Fine-Tune)354.92M68.246.030.4/63.2/47.6GPT-2 M (Adapter)0.37M66.342.445.1/54.5/50.2GPT-2 M (Prefix)0.35M69.745.744.1/63.1/54.4GPT-2 M (LoRA)0.35M70.4±.147.1±.246.7±.4/62.1±.2/55.3±.2GPT-2 L (Fine-Tune)774.03M68.546.541.7/64.6/54.2GPT-2 L (Adapter)0.88M69.1±.145.7±.149.8±.0/61.1±.0/56.0±.0GPT-2 L (Prefix)0.77M70.346.547.0/64.2/56.4GPT-2 L (LoRA)0.77M70.4±.147.5±.148.4±.3/64.0±.3/57.0±.1Non-LoRA baselines, except for adapter on GPT-2 large, are taken fromLi and Liang (2021). We include confidence intervals on results from our experiments.Download the GPT-2 LoRA checkpoints:GPT-2 Medium E2E(1.5 MB)GPT-2 Medium DART(1.5 MB)GPT-2 Medium WebNLG(1.5 MB)GPT-2 Large E2E(2.3 MB)GPT-2 Large DART(2.3 MB)GPT-2 Large WebNLG(2.3 MB)Please follow the instructions inexamples/NLG/to reproduce our result.Repository Overview(The initial release of this repo has been archived in the branch "snapshot-9-15-2021")There are several directories in this repo:loralib/contains the source code for the packageloralib, which needs to be installed to run the examples we provide;examples/NLG/contains an example implementation of LoRA in GPT-2 using our package, which can be used to reproduce the result in our paper;examples/NLU/contains an example implementation of LoRA in RoBERTa and DeBERTa using our package, which produces competitive results on the GLUE benchmark;See how we useloralibinGPT-2,RoBERTa, andDeBERTa v2QuickstartInstallingloralibis simplypipinstallloralib# Alternatively# pip install git+https://github.com/microsoft/LoRAYou can choose to adapt some layers by replacing them with counterparts implemented inloralib. We only supportnn.Linear,nn.Embedding, andnn.Conv2dfor now. We also support aMergedLinearfor cases where a singlenn.Linearrepresents more than one layers, such as in some implementations of the attentionqkvprojection (see Additional Notes for more).# ===== Before =====# layer = nn.Linear(in_features, out_features)# ===== After ======importloralibaslora# Add a pair of low-rank adaptation matrices with rank r=16layer=lora.Linear(in_features,out_features,r=16)Before the training loop begins, mark only LoRA parameters as trainable.importloralibasloramodel=BigModel()# This sets requires_grad to False for all parameters without the string "lora_" in their nameslora.mark_only_lora_as_trainable(model)# Training loopforbatchindataloader:...When saving a checkpoint, generate astate_dictthat only contains LoRA parameters.# ===== Before =====# torch.save(model.state_dict(), checkpoint_path)# ===== After =====torch.save(lora.lora_state_dict(model),checkpoint_path)When loading a checkpoint usingload_state_dict, be sure to setstrict=False.# Load the pretrained checkpoint firstmodel.load_state_dict(torch.load('ckpt_pretrained.pt'),strict=False)# Then load the LoRA checkpointmodel.load_state_dict(torch.load('ckpt_lora.pt'),strict=False)Now training can proceed as usual.Additional NotesWhile we focus on a simple yet effect setup, namely adapting only theqandvprojection in a Transformer, in our examples, LoRA can be apply to any subsets of pre-trained weights. We encourage you to explore different configurations, such as adapting the embedding layer by replacingnn.Embeddingwithlora.Embeddingand/or adapting the MLP layers. It's very likely that the optimal configuration varies for different model architectures and tasks.Some Transformer implementation uses a singlenn.Linearfor the projection matrices for query, key, and value. If one wishes to constrain the rank of the updates to the individual matrices, one has to either break it up into three separate matrices or uselora.MergedLinear. Make sure to modify the checkpoint accordingly if you choose to break up the layer.# ===== Before =====# qkv_proj = nn.Linear(d_model, 3*d_model)# ===== After =====# Break it up (remember to modify the pretrained checkpoint accordingly)q_proj=lora.Linear(d_model,d_model,r=8)k_proj=nn.Linear(d_model,d_model)v_proj=lora.Linear(d_model,d_model,r=8)# Alternatively, use lora.MergedLinear (recommended)qkv_proj=lora.MergedLinear(d_model,3*d_model,r=8,enable_lora=[True,False,True])Training bias vectors in tandem with LoRA might be a cost-efficient way to squeeze out extra task performance (if you tune the learning rate carefully). While we did not study its effect thoroughly in our paper, we make it easy to try inlora. You can mark some biases as trainable by passing "all" or "lora_only" tobias=when callingmark_only_lora_as_trainable. Remember to pass the correspondingbias=argument tolora_state_dictwhen saving a checkpoint.# ===== Before =====# lora.mark_only_lora_as_trainable(model) # Not training any bias vectors# ===== After =====# Training all bias vectors associated with modules we apply LoRA tolora.mark_only_lora_as_trainable(model,bias='lora_only')# Alternatively, we can train *all* bias vectors in the model, including LayerNorm biaseslora.mark_only_lora_as_trainable(model,bias='all')# When saving a checkpoint, use the same bias= ('all' or 'lora_only')torch.save(lora.lora_state_dict(model,bias='all'),checkpoint_path)Callingmodel.eval()will trigger the merging of LoRA parameters with the corresponding pretrained ones, which eliminates additional latency for subsequent forward passes. Callingmodel.train()again will undo the merge. This can be disabled by passingmerge_weights=Falseto LoRA layers.ContactPlease contact us or post an issue if you have any questions.For questions related to the packageloralib:Edward Hu (edward@edwardjhu.com)Phillip Wallis (phwallis@microsoft.com)Weizhu Chen (wzchen@microsoft.com)The GPT-2 example:Phillip Wallis (phwallis@microsoft.com)Yelong Shen (yeshe@microsoft.com)The RoBERTa/DeBERTa example:Lu Wang (luw@microsoft.com)AcknowledgementsWe thank in alphabetical order Jianfeng Gao, Jade Huang, Jiayuan Huang, Lisa Xiang Li, Xiaodong Liu, Yabin Liu, Benjamin Van Durme, Luis Vargas, Haoran Wei, Peter Welinder, and Greg Yang for providing valuable feedback.Citation@inproceedings{hu2022lora,title={Lo{RA}: Low-Rank Adaptation of Large Language Models},author={Edward J Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},booktitle={International Conference on Learning Representations},year={2022},url={https://openreview.net/forum?id=nZeVKeeFYf9}}ContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.