Package: llmtuner
👋 Join ourWeChat.[ English |中文]LLaMA Board: A One-stop Web UI for Getting Started with LLaMA FactoryPreview LLaMA Board at🤗 SpacesandModelScope, or launch it locally withCUDA_VISIBLE_DEVICES=0 python src/train_web.py.Here is an example of altering the self-cognition of an instruction-tuned language model within 10 minutes on a single GPU.https://github.com/hiyouga/LLaMA-Factory/assets/16256802/6ba60acc-e2e2-4bec-b846-2d88920d5ba1Table of ContentsFeaturesBenchmarkChangelogSupported ModelsSupported Training ApproachesProvided DatasetsRequirementGetting StartedProjects using LLaMA FactoryLicenseCitationAcknowledgementFeaturesVarious models: LLaMA, Mistral, Mixtral-MoE, Qwen, Yi, Gemma, Baichuan, ChatGLM, Phi, etc.Integrated methods: (Continuous) pre-training, supervised fine-tuning, reward modeling, PPO and DPO.Scalable resources: 32-bit full-tuning, 16-bit freeze-tuning, 16-bit LoRA, 2/4/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8.Advanced algorithms: DoRA, LongLoRA, LLaMA Pro, LoftQ, agent tuning.Practical tricks: FlashAttention-2, Unsloth, RoPE scaling, NEFTune, rsLoRA.Experiment monitors: LlamaBoard, TensorBoard, Wandb, MLflow, etc.BenchmarkCompared to ChatGLM'sP-Tuning, LLaMA-Factory's LoRA tuning offers up to3.7 times fastertraining speed with a better Rouge score on the advertising text generation task. By leveraging 4-bit quantization technique, LLaMA-Factory's QLoRA further improves the efficiency regarding the GPU memory.DefinitionsTraining Speed: the number of training samples processed per second during the training. (bs=4, cutoff_len=1024)Rouge Score: Rouge-2 score on the development set of theadvertising text generationtask. (bs=4, cutoff_len=1024)GPU Memory: Peak GPU memory usage in 4-bit quantized training. (bs=1, cutoff_len=1024)We adoptpre_seq_len=128for ChatGLM's P-Tuning andlora_rank=32for LLaMA-Factory's LoRA tuning.Changelog[24/02/28] We supported weight-decomposed LoRA (DoRA). Try--use_dorato activate DoRA training.[24/02/15] We supportedblock expansionproposed byLLaMA Pro. Seetests/llama_pro.pyfor usage.[24/02/05] Qwen1.5 (Qwen2 beta version) series models are supported in LLaMA-Factory. Check thisblog postfor details.Full Changelog[24/01/18] We supportedagent tuningfor most models, equipping model with tool using abilities by fine-tuning with--dataset glaive_toolcall.[23/12/23] We supportedunsloth's implementation to boost LoRA tuning for the LLaMA, Mistral and Yi models. Try--use_unslothargument to activate unsloth patch. It achieves 1.7x speed in our benchmark, checkthis pagefor details.[23/12/12] We supported fine-tuning the latest MoE modelMixtral 8x7Bin our framework. See hardware requirementhere.[23/12/01] We supported downloading pre-trained models and datasets from theModelScope Hubfor Chinese mainland users. Seethis tutorialfor usage.[23/10/21] We supportedNEFTunetrick for fine-tuning. Try--neftune_noise_alphaargument to activate NEFTune, e.g.,--neftune_noise_alpha 5.[23/09/27] We supported$S^2$-Attnproposed byLongLoRAfor the LLaMA models. Try--shift_attnargument to enable shift short attention.[23/09/23] We integrated MMLU, C-Eval and CMMLU benchmarks in this repo. Seethis exampleto evaluate your models.[23/09/10] We supportedFlashAttention-2. Try--flash_attnargument to enable FlashAttention-2 if you are using RTX4090, A100 or H100 GPUs.[23/08/12] We supportedRoPE scalingto extend the context length of the LLaMA models. Try--rope_scaling linearargument in training and--rope_scaling dynamicargument at inference to extrapolate the position embeddings.[23/08/11] We supportedDPO trainingfor instruction-tuned models. Seethis exampleto train your models.[23/07/31] We supporteddataset streaming. Try--streamingand--max_steps 10000arguments to load your dataset in streaming mode.[23/07/29] We released two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos (LLaMA-2/Baichuan) for details.[23/07/18] We developed anall-in-one Web UIfor training, evaluation and inference. Trytrain_web.pyto fine-tune models in your Web browser. Thank@KanadeSiinaand@codemayqfor their efforts in the development.[23/07/09] We releasedFastEdit⚡🩹, an easy-to-use package for editing the factual knowledge of large language models efficiently. Please followFastEditif you are interested.[23/06/29] We provided areproducible exampleof training a chat model using instruction-following datasets, seeBaichuan-7B-sftfor details.[23/06/22] We aligned thedemo APIwith theOpenAI'sformat where you can insert the fine-tuned model inarbitrary ChatGPT-based applications.[23/06/03] We supported quantized training and inference (akaQLoRA). Try--quantization_bit 4/8argument to work with quantized models.Supported ModelsModelModel sizeDefault moduleTemplateBaichuan27B/13BW_packbaichuan2BLOOM560M/1.1B/1.7B/3B/7.1B/176Bquery_key_value-BLOOMZ560M/1.1B/1.7B/3B/7.1B/176Bquery_key_value-ChatGLM36Bquery_key_valuechatglm3DeepSeek (MoE)7B/16B/67Bq_proj,v_projdeepseekFalcon7B/40B/180Bquery_key_valuefalconGemma2B/7Bq_proj,v_projgemmaInternLM27B/20Bwqkvintern2LLaMA7B/13B/33B/65Bq_proj,v_proj-LLaMA-27B/13B/70Bq_proj,v_projllama2Mistral7Bq_proj,v_projmistralMixtral8x7Bq_proj,v_projmistralPhi-1.5/21.3B/2.7Bq_proj,v_proj-Qwen1.8B/7B/14B/72Bc_attnqwenQwen1.50.5B/1.8B/4B/7B/14B/72Bq_proj,v_projqwenXVERSE7B/13B/65Bq_proj,v_projxverseYi6B/34Bq_proj,v_projyiYuan2B/51B/102Bq_proj,v_projyuan[!NOTE]Default moduleis used for the--lora_targetargument, you can use--lora_target allto specify all the available modules.For the "base" models, the--templateargument can be chosen fromdefault,alpaca,vicunaetc. But make sure to use thecorresponding templatefor the "chat" models.Please refer toconstants.pyfor a full list of models we supported.Supported Training ApproachesApproachFull-tuningFreeze-tuningLoRAQLoRAPre-Training:white_check_mark::white_check_mark::white_check_mark::white_check_mark:Supervised Fine-Tuning:white_check_mark::white_check_mark::white_check_mark::white_check_mark:Reward Modeling:white_check_mark::white_check_mark::white_check_mark::white_check_mark:PPO Training:white_check_mark::white_check_mark::white_check_mark::white_check_mark:DPO Training:white_check_mark::white_check_mark::white_check_mark::white_check_mark:[!NOTE]
Use--quantization_bit 4argument to enable QLoRA.Provided DatasetsPre-training datasetsWiki Demo (en)RefinedWeb (en)RedPajama V2 (en)Wikipedia (en)Wikipedia (zh)Pile (en)SkyPile (zh)The Stack (en)StarCoder (en)Supervised fine-tuning datasetsStanford Alpaca (en)Stanford Alpaca (zh)Alpaca GPT4 (en&zh)Self Cognition (zh)Open Assistant (multilingual)ShareGPT (zh)Guanaco Dataset (multilingual)BELLE 2M (zh)BELLE 1M (zh)BELLE 0.5M (zh)BELLE Dialogue 0.4M (zh)BELLE School Math 0.25M (zh)BELLE Multiturn Chat 0.8M (zh)UltraChat (en)LIMA (en)OpenPlatypus (en)CodeAlpaca 20k (en)Alpaca CoT (multilingual)OpenOrca (en)SlimOrca (en)MathInstruct (en)Firefly 1.1M (zh)Wiki QA (en)Web QA (zh)WebNovel (zh)Nectar (en)deepctrl (en&zh)Ad Gen (zh)ShareGPT Hyperfiltered (en)ShareGPT4 (en&zh)UltraChat 200k (en)AgentInstruct (en)LMSYS Chat 1M (en)Evol Instruct V2 (en)Glaive Function Calling V2 (en)Open Assistant (de)Dolly 15k (de)Alpaca GPT4 (de)OpenSchnabeltier (de)Evol Instruct (de)Dolphin (de)Booksum (de)Airoboros (de)Ultrachat (de)Preference datasetsHH-RLHF (en)Open Assistant (multilingual)GPT-4 Generated Data (en&zh)Nectar (en)Orca DPO (de)Please refer todata/README.mdfor details.Some datasets require confirmation before using them, so we recommend logging in with your Hugging Face account using these commands.pipinstall--upgradehuggingface_hub
huggingface-cliloginRequirementMandatoryMinimumRecommendpython3.83.10torch1.13.12.2.1transformers4.37.24.38.1datasets2.14.32.17.1accelerate0.27.20.27.2peft0.9.00.9.0trl0.7.110.7.11OptionalMinimumRecommendCUDA11.612.2deepspeed0.10.00.13.4bitsandbytes0.39.00.41.3flash-attn2.3.02.5.5Hardware Requirement*estimatedMethodBits7B13B30B65B8x7BFull16160GB320GB600GB1200GB900GBFreeze1620GB40GB120GB240GB200GBLoRA1616GB32GB80GB160GB120GBQLoRA810GB16GB40GB80GB80GBQLoRA46GB12GB24GB48GB32GBGetting StartedData Preparation (optional)Please refer todata/README.mdfor checking the details about the format of dataset files. You can either use a single.jsonfile or adataset loading scriptwith multiple files to create a custom dataset.[!NOTE]
Please updatedata/dataset_info.jsonto use your custom dataset. About the format of this file, please refer todata/README.md.Dependence Installation (optional)gitclonehttps://github.com/hiyouga/LLaMA-Factory.git
condacreate-nllama_factorypython=3.10
condaactivatellama_factorycdLLaMA-Factory
pipinstall-rrequirements.txtIf you want to enable the quantized LoRA (QLoRA) on the Windows platform, you will be required to install a pre-built version ofbitsandbyteslibrary, which supports CUDA 11.1 to 12.2.pipinstallhttps://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.40.0-py3-none-win_amd64.whlTo enable FlashAttention-2 on the Windows platform, you need to install the precompiledflash-attnlibrary, which supports CUDA 12.1 to 12.2. Please download the corresponding version fromflash-attentionbased on your requirements.Use ModelScope Hub (optional)If you have trouble with downloading models and datasets from Hugging Face, you can use LLaMA-Factory together with ModelScope in the following manner.exportUSE_MODELSCOPE_HUB=1# `set USE_MODELSCOPE_HUB=1` for WindowsThen you can train the corresponding model by specifying a model ID of the ModelScope Hub. (find a full list of model IDs atModelScope Hub)CUDA_VISIBLE_DEVICES=0pythonsrc/train_bash.py\--model_name_or_pathmodelscope/Llama-2-7b-ms\...# arguments (same as above)LLaMA Board also supports using the models and datasets on the ModelScope Hub.CUDA_VISIBLE_DEVICES=0USE_MODELSCOPE_HUB=1pythonsrc/train_web.pyTrain on a single GPU[!IMPORTANT]
If you want to train models on multiple GPUs, please refer toDistributed Training.Pre-TrainingCUDA_VISIBLE_DEVICES=0pythonsrc/train_bash.py\--stagept\--do_train\--model_name_or_pathpath_to_llama_model\--datasetwiki_demo\--finetuning_typelora\--lora_targetq_proj,v_proj\--output_dirpath_to_pt_checkpoint\--overwrite_cache\--per_device_train_batch_size4\--gradient_accumulation_steps4\--lr_scheduler_typecosine\--logging_steps10\--save_steps1000\--learning_rate5e-5\--num_train_epochs3.0\--plot_loss\--fp16Supervised Fine-TuningCUDA_VISIBLE_DEVICES=0pythonsrc/train_bash.py\--stagesft\--do_train\--model_name_or_pathpath_to_llama_model\--datasetalpaca_gpt4_en\--templatedefault\--finetuning_typelora\--lora_targetq_proj,v_proj\--output_dirpath_to_sft_checkpoint\--overwrite_cache\--per_device_train_batch_size4\--gradient_accumulation_steps4\--lr_scheduler_typecosine\--logging_steps10\--save_steps1000\--learning_rate5e-5\--num_train_epochs3.0\--plot_loss\--fp16Reward ModelingCUDA_VISIBLE_DEVICES=0pythonsrc/train_bash.py\--stagerm\--do_train\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_sft_checkpoint\--create_new_adapter\--datasetcomparison_gpt4_en\--templatedefault\--finetuning_typelora\--lora_targetq_proj,v_proj\--output_dirpath_to_rm_checkpoint\--per_device_train_batch_size2\--gradient_accumulation_steps4\--lr_scheduler_typecosine\--logging_steps10\--save_steps1000\--learning_rate1e-6\--num_train_epochs1.0\--plot_loss\--fp16PPO TrainingCUDA_VISIBLE_DEVICES=0pythonsrc/train_bash.py\--stageppo\--do_train\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_sft_checkpoint\--create_new_adapter\--datasetalpaca_gpt4_en\--templatedefault\--finetuning_typelora\--lora_targetq_proj,v_proj\--reward_modelpath_to_rm_checkpoint\--output_dirpath_to_ppo_checkpoint\--per_device_train_batch_size2\--gradient_accumulation_steps4\--lr_scheduler_typecosine\--top_k0\--top_p0.9\--logging_steps10\--save_steps1000\--learning_rate1e-5\--num_train_epochs1.0\--plot_loss\--fp16[!TIP]
Use--adapter_name_or_path path_to_sft_checkpoint,path_to_ppo_checkpointto infer the fine-tuned model.[!WARNING]
Use--per_device_train_batch_size=1for LLaMA-2 models in fp16 PPO training.DPO TrainingCUDA_VISIBLE_DEVICES=0pythonsrc/train_bash.py\--stagedpo\--do_train\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_sft_checkpoint\--create_new_adapter\--datasetcomparison_gpt4_en\--templatedefault\--finetuning_typelora\--lora_targetq_proj,v_proj\--output_dirpath_to_dpo_checkpoint\--per_device_train_batch_size2\--gradient_accumulation_steps4\--lr_scheduler_typecosine\--logging_steps10\--save_steps1000\--learning_rate1e-5\--num_train_epochs1.0\--plot_loss\--fp16[!TIP]
Use--adapter_name_or_path path_to_sft_checkpoint,path_to_dpo_checkpointto infer the fine-tuned model.Distributed TrainingUse Huggingface Accelerateaccelerateconfig# configure the environmentacceleratelaunchsrc/train_bash.py# arguments (same as above)Example config for LoRA trainingcompute_environment:LOCAL_MACHINEdebug:falsedistributed_type:MULTI_GPUdowncast_bf16:'no'gpu_ids:allmachine_rank:0main_training_function:mainmixed_precision:fp16num_machines:1num_processes:4rdzv_backend:staticsame_network:truetpu_env:[]tpu_use_cluster:falsetpu_use_sudo:falseuse_cpu:falseUse DeepSpeeddeepspeed--num_gpus8--master_port=9901src/train_bash.py\--deepspeedds_config.json\...# arguments (same as above)Example config for full-parameter training with DeepSpeed ZeRO-2{"train_batch_size":"auto","train_micro_batch_size_per_gpu":"auto","gradient_accumulation_steps":"auto","gradient_clipping":"auto","zero_allow_untested_optimizer":true,"fp16":{"enabled":"auto","loss_scale":0,"initial_scale_power":16,"loss_scale_window":1000,"hysteresis":2,"min_loss_scale":1},"zero_optimization":{"stage":2,"allgather_partitions":true,"allgather_bucket_size":5e8,"reduce_scatter":true,"reduce_bucket_size":5e8,"overlap_comm":false,"contiguous_gradients":true}}Merge LoRA weights and export modelpythonsrc/export_model.py\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_checkpoint\--templatedefault\--finetuning_typelora\--export_dirpath_to_export\--export_size2\--export_legacy_formatFalse[!WARNING]
Merging LoRA weights into a quantized model is not supported.[!TIP]
Use--export_quantization_bit 4and--export_quantization_dataset data/c4_demo.jsonto quantize the model after merging the LoRA weights.Inference with OpenAI-style APIpythonsrc/api_demo.py\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_checkpoint\--templatedefault\--finetuning_typelora[!TIP]
Visithttp://localhost:8000/docsfor API documentation.Inference with command linepythonsrc/cli_demo.py\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_checkpoint\--templatedefault\--finetuning_typeloraInference with web browserpythonsrc/web_demo.py\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_checkpoint\--templatedefault\--finetuning_typeloraEvaluationCUDA_VISIBLE_DEVICES=0pythonsrc/evaluate.py\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_checkpoint\--templatevanilla\--finetuning_typelora\--taskmmlu\--splittest\--langen\--n_shot5\--batch_size4PredictCUDA_VISIBLE_DEVICES=0pythonsrc/train_bash.py\--stagesft\--do_predict\--model_name_or_pathpath_to_llama_model\--adapter_name_or_pathpath_to_checkpoint\--datasetalpaca_gpt4_en\--templatedefault\--finetuning_typelora\--output_dirpath_to_predict_result\--per_device_eval_batch_size1\--max_samples100\--predict_with_generate\--fp16[!WARNING]
Use--per_device_train_batch_size=1for LLaMA-2 models in fp16 predict.[!TIP]
We recommend using--per_device_eval_batch_size=1and--max_target_length 128at 4/8-bit predict.Projects using LLaMA FactoryWang et al. ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation. 2023.[arxiv]Yu et al. Open, Closed, or Small Language Models for Text Classification? 2023.[arxiv]Luceri et al. Leveraging Large Language Models to Detect Influence Campaigns in Social Media. 2023.[arxiv]Zhang et al. Alleviating Hallucinations of Large Language Models through Induced Hallucinations. 2023.[arxiv]Wang et al. Know Your Needs Better: Towards Structured Understanding of Marketer Demands with Analogical Reasoning Augmented LLMs. 2024.[arxiv]Wang et al. CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning. 2024.[arxiv]Choi et al. FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMs. 2024.[arxiv]Zhang et al. AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts. 2024.[arxiv]Lyu et al. KnowTuning: Knowledge-aware Fine-tuning for Large Language Models. 2024.[arxiv]Yang et al. LaCo: Large Language Model Pruning via Layer Collaps. 2024.[arxiv]Bhardwaj et al. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. 2024.[arxiv]Yang et al. Enhancing Empathetic Response Generation by Augmenting LLMs with Small-scale Empathetic Models. 2024.[arxiv]Yi et al. Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding. 2024.[arxiv]Cao et al. Head-wise Shareable Attention for Large Language Models. 2024.[arxiv]Zhang et al. Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages. 2024.[arxiv]Kim et al. Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models. 2024.[arxiv]StarWhisper: A large language model for Astronomy, based on ChatGLM2-6B and Qwen-14B.DISC-LawLLM: A large language model specialized in Chinese legal domain, based on Baichuan-13B, is capable of retrieving and reasoning on legal knowledge.Sunsimiao: A large language model specialized in Chinese medical domain, based on Baichuan-7B and ChatGLM-6B.CareGPT: A series of large language models for Chinese medical domain, based on LLaMA2-7B and Baichuan-13B.MachineMindset: A series of MBTI Personality large language models, capable of giving any LLM 16 different personality types based on different datasets and training methods.[!TIP]
If you have a project that should be incorporated, please contact via email or create a pull request.LicenseThis repository is licensed under theApache-2.0 License.Please follow the model licenses to use the corresponding model weights:Baichuan2/BLOOM/ChatGLM3/DeepSeek/Falcon/Gemma/InternLM2/LLaMA/LLaMA-2/Mistral/Phi-1.5/2/Qwen/XVERSE/Yi/YuanCitationIf this work is helpful, please kindly cite as:@Misc{llama-factory,title={LLaMA Factory},author={hiyouga},howpublished={\url{https://github.com/hiyouga/LLaMA-Factory}},year={2023}}AcknowledgementThis repo benefits fromPEFT,QLoRAandFastChat. Thanks for their wonderful works.Star History