Package: mlora
m-LoRA: Efficient LLM Model Fine-Tune via Multi-LoRA Optimizationm-LoRA (a.k.a Multi-Lora Fine-Tune) is an open-source framework for fine-tuning Large Language Models (LLMs) using the efficient multiple LoRA/QLoRA methods. Key features of m-LoRA include:Efficient LoRA/QLoRA: Optimizes the fine-tuning process, significantly reducing GPU memory usage by leveraging a shared frozen-based model.Multiple LoRA Adapters: Support for concurrent fine-tuning of multiple LoRA/QLoRA adapters.ContentsUpdatesSupported ModelsOverviewGetting StartedInstallationContributingCopyrightUpdatesSupport multiple LLaMA2 fine-tuningSupport multiple ChatGLM fine-tuningSupport multiple LLaMA fine-tuningOn the way, BaichuanModelsModelModel sizeChatGLM6BChatGLM26B/12BChatGLM36BLLaMA7B//13B/33B/65BLLaMA-27B/13B/70BBaichuan7B/13BBaichuan27B/13BExample:Use our system  to improve the LLaMa-2 fine-tuning with less resourceshttps://www.kaggle.com/code/rraydata/multi-lora-example/notebookOverviewm-LoRAis a high-throughput LLM fine-tuning framework based on LoRA and QLoRA, compatible with HuggingFace-Transformers LLaMA Models and ChatGLM Models.This picture shows the basic principle of LoRA and Multi-LoRA.The system overview of m-LoRA is as follows.m-LoRA requiresPyTorchandNVIDIA CUDAcompatible GPUs.Main ContributionIntroduces the Multi-LoRA method, capable of enabling the sharing of pre-trained model weights during the fine-tuning process of large language models;Proposes a task scheduling algorithm to enhance the overall throughput of the task training process and reduce total training latency;Builds upon the above by implementing m-LoRA, a high-throughput large language model fine-tuning framework based on LoRA and QLoRA;Evaluates m-LoRA in experiments against existing systems, confirming that m-LoRA effectively utilizes system computing resources, thereby improving training throughput and reducing training latency compared to current systems.Experiment ResultsEnvironment: NVIDIA RTX A6000 with Intel Xeon Silver 4314 on Ubuntu 22.04.3Baseline: We utilized the widely adoptedAlpaca-LoRAas a foundation. On a single GPU, we independently ran multiple Alpaca-LoRA processes in parallel (marked asBaseline@Alpaca-Parallel) and sequentially (marked asBaseline@Alpaca-Seq), forming two baseline methods for the experiments. We test this on A100, and rest of results are based on the same GPU configure.Training Latency and ThroughputMethodLatencyThroughputBaseline@Alpaca-Seq10.51h608.41 token/sBaseline@Alpaca-Parallel9.85h649.30 token/sm-LoRA9.46h674.58 token/sWe conducted four identical fine-tuning jobs with same dataset and same hyper-parameters, incorporating two baselines and m-LoRA. During the experimental process, we collected the completion times for each task in the baseline methods and calculated the time taken by the slowest task as theTraining Latency. As shown in Table, m-LoRA exhibits lowerTraining Latencycompared to both baseline methods. Specifically, m-LoRA is 9.99% faster thanBaseline@Alpaca-Seqand 3.92% faster thanBaseline@Alpaca-Parallel.Video Memory UsageWe conducted several fine-tuning jobs with same dataset andbatch_size = {2,4, 6, 8}, incorporatingBaseline@Alpaca-Paralleland m-LoRA.Baseline@Alpaca-Paralleltriggered OOM error after 3 parallel tasks when batch size = 8, while m-LoRA can handle twice that amount.Batching StrategiesMethodTraining LatencyPeak Memory UsageAverage GPU UtilizationTraining ThroughputBaseline@Alpaca-Seq27.73h10.68GB79.39%653.35 token/sm-LoRA@M136.82h23.82GB96.52%672.54 token/sm-LoRA@M239.14h23.86GB96.41%671.28 token/sm-LoRA@M322.97h23.85GB95.22%674.41 token/sWe conducted four fine-tuning jobs with different dataset but same hyper-parameters, incorporatingBaseline@Alpaca-Seqand m-LoRA.During the experimental process, we collected following metrics:Training Latency= Job completion timeThroughput= The number of passed tokens in model forward process / training latencyMemory Usage= Peak video memory usageGPU Utilization= Average GPU utilizationAll metrics are computed for each job.M1, M2, M3represent three batch strategies of m-LoRA:Optimal-Fit, Trivial, and Fast-Fit.BASELINEdenotesBaseline@Alpaca-Seq.TheOptimal-Fitstrategy performs the best across all four metrics, while the other two strategies also outperform the baseline method other than training latency.Use Cases:Domain-Specific Fine-Tuning: This involves adapting a single model with various parameters particularly for one domain.Cross-Domain Fine-Tuning: This method leverages the base model to fine-tune multiple models, each intended for a different domain.QuickstartFirstly, you should clone this repository and install dependencies:# Clone Repositorygitclonehttps://github.com/TUDB-Labs/multi-lora-fine-tunecdmulti-lora-fine-tune# Install requirementspipinstall-rrequirements.txtThemlora.pycode is a starting point for finetuning on various datasets.
Basic command for finetuning a baseline model on theAlpaca Cleaneddataset:pythonmlora.py\--base_modelyahma/llama-7b-hf\--config./config/alpaca.json\--load_8bitYou can check the template finetune configuration intemplatefolder.For further detailed usage information, please use--helpoption:pythonmlora.py--helpDemo on ColabYou can run finetune on Colab by following this example:Google Colab Example. Make sure to switch the runtime environment to GPU before running it.InstallationYou can also install m-LoRA into your environment:# Optional but recommendedcondacreate-nmlora_envpython=3.8
condaactivatemlora_env# Install requirementspipinstall-rrequirements.txtAfter installation, you can use m-LoRA directly in your code:importmloraContributingWe welcome contributions to improve this repository! Please review the contribution guidelines before submitting pull requests or issues.Fork the repository.
Create a new branch for your feature or fix.
Submit a pull request with a detailed explanation of your changes.CitationPlease cite the repo if you use the code in this repo.@misc{m-LoRA,author={Zhengmao, Ye\textsuperscript{*} and Dengchun, Li\textsuperscript{*} and Jingqi, Tian and Tingfeng, Lan and Yanbo, Liang and Yexi, Jiang and Jie, Zuo and Hui, Lu and Lei, Duan and Mingjie, Tang},title={m-LoRA: Efficient LLM Model Fine-tune and Inference via Multi-Lora Optimization},year={2023},publisher={GitHub},howpublished={\url{https://github.com/TUDB-Labs/multi-lora-fine-tune}},note={\textsuperscript{*}: these authors contributed equally to this work.}}CopyrightCopyright © 2023 All Rights Reserved.This project is licensed under theApache 2.0 License.Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.