Package: lorahub
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA CompositionThe official repository which contains the code and pre-trained models for our paperLoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition.🔥 Updates[2023-8-29]: We released the full produce code atreproduce_bbh.py. Please checkout the script to reproduce our results![2023-8-03]: Integrated into Replicate, check out thedemo![2023-7-27]: We released ourcodeanddemo. Check it out![2023-7-26]: We released ourpaper.🏴󠁶󠁵󠁭󠁡󠁰󠁿 OverviewLow-rank adaptations (LoRA) are techniques for fine-tuning large language models on new tasks. We proposeLoraHub, a framework that allowscomposing multiple LoRA modulestrained on different tasks. The goal is to achieve good performance onunseen tasksusing just a few examples,without needing extra parameters or training. And we want to build a marketplace where users can share their trained LoRA modules, thereby facilitating the application of these modules to new tasks.The figure demostrates the zero-shot learning, few-shot in-context learning and few-shot lorahub learning (ours). Note that the Compose procedure is conducted per task rather than per example. Our method achieves similar inference throughput as zero-shot learning, yet approaches the performance of in-context learning on the BIG-Bench Hard (BBH) benchmark. The experimental results show the superior efficacy of our method in comparison to zero-shot learning while closely resembling the performance of in-context learning (ICL) in few-shot scenarios.The figure shows the pipeline of LoraHub Learning. Our method encompasses two stages: theComposestage and theAdaptstage. During the Compose stage, existing LoRA modules are integrated into one unified module, employing a set of weights, denoted asw, as coefficients. In the Adapt stage, the amalgamated LoRA module is evaluated on a few examples from the unseen task. Subsequently, agradient-free algorithmis applied to refinew. After executingKiterations, ahighly adapted LoRA moduleis produced, which can be incorporated with the LLM to perform the intended task.🌲 Project StructureOur code is organized as below:|--lorahub--algorithm.py# main code for lorahub learning and inference--constant.py# lora candidate module names|--example.py# usage code for demonstration purposeAnd you can use LoraHub learning by simply calling the following function:fromlorahub.algorithmimportlorahub_learninglorahub_learning(lora_module_list:List[str],# list of lora candidatesexample_inputs:List[str],example_outputs:List[str],max_inference_step:int,model_name_or_path=None,# if not given, we will use the model_name_or_path in lora configbatch_size=None,get_loss=default_get_loss,# The function to get the objective for optimiztion, use loss as default (can be changed to something like acc. or similarity)get_regular=default_l1_regularization,# The function to get regularization term for the weight, use 0.05*|w_i| as defaultseed=42)⚡️ QuickstartPrepare EnvironmentFirst, you should run the following commands to install the latest lib developed for LoraHub.pipinstalldatasetspipinstalltransformerspipinstallpeftpipinstallnevergradpipinstalltorchpipinstalltqdmpipinstallpandaspipinstallnumpyInstall LoraHubThe pypi package will be released in a few days. Please stay tuned!🏰 ResourceLoRA CandidatesOur methodology requires a compendium of LoRA modules trained on preceding tasks. For parity with Flan, we adopt the tasks utilized to instruct Flan-T5, thereby incorporating nearly 196 distinct tasks and their corresponding instructions viahttps://huggingface.co/datasets/conceptofmind/FLAN_2022. Following this, we created several LoRA modules as possible candidates. These LoRA modules can be accessed athttps://huggingface.co/models?search=lorahub.💬 CitationIf our work is useful for you, please consider citing our paper:@misc{huang2023lorahub,title={LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition},author={Chengsong Huang and Qian Liu and Bill Yuchen Lin and Tianyu Pang and Chao Du and Min Lin},year={2023},eprint={2307.13269},archivePrefix={arXiv},primaryClass={cs.CL}}