Package: xtuner
👋 join us on🔍 Explore our models onEnglish |简体中文🎉 News[2024/02]SupportGemmamodels![2024/02]SupportQwen1.5models![2024/01]SupportInternLM2models! The latest VLMLLaVA-Internlm2-7B/20Bmodels are released, with impressive performance![2024/01]SupportDeepSeek-MoEmodels! 20GB GPU memory is enough for QLoRA fine-tuning, and 4x80GB for full-parameter fine-tuning. Clickherefor details![2023/12]🔥 Support multi-modal VLM pretraining and fine-tuning withLLaVA-v1.5architecture! Clickherefor details![2023/12]🔥 SupportMixtral 8x7Bmodels! Clickherefor details![2023/11]SupportChatGLM3-6Bmodel![2023/10]SupportMSAgent-Benchdataset, and the fine-tuned LLMs can be applied byLagent![2023/10]Optimize the data processing to accommodatesystemcontext. More information can be found onDocs![2023/09]SupportInternLM-20Bmodels![2023/09]SupportBaichuan2models![2023/08]XTuner is released, with multiple fine-tuned adapters onHuggingFace.📖 IntroductionXTuner is an efficient, flexible and full-featured toolkit for fine-tuning large models.EfficientSupport LLM, VLM pre-training / fine-tuning on almost all GPUs. XTuner is capable of fine-tuning 7B LLM on a single 8GB GPU, as well as multi-node fine-tuning of models exceeding 70B.Automatically dispatch high-performance operators such as FlashAttention and Triton kernels to increase training throughput.Compatible withDeepSpeed🚀, easily utilizing a variety of ZeRO optimization techniques.FlexibleSupport various LLMs (InternLM,Mixtral-8x7B,Llama2,ChatGLM,Qwen,Baichuan, ...).Support VLM (LLaVA). The performance ofLLaVA-InternLM2-20Bis outstanding.Well-designed data pipeline, accommodating datasets in any format, including but not limited to open-source and custom formats.Support various training algorithms (QLoRA,LoRA, full-parameter fune-tune), allowing users to choose the most suitable solution for their requirements.Full-featuredSupport continuous pre-training, instruction fine-tuning, and agent fine-tuning.Support chatting with large models with pre-defined templates.The output models can seamlessly integrate with deployment and server toolkit (LMDeploy), and large-scale evaluation toolkit (OpenCompass,VLMEvalKit).🌟 DemosReady-to-use models and datasets from XTuner APIQLoRA Fine-tunePlugin-based ChatExamples of Plugin-based Chat 🔥🔥🔥🔥 SupportsModelsSFT DatasetsData PipelinesAlgorithmsInternLM2InternLMLlamaLlama2ChatGLM2ChatGLM3QwenBaichuanBaichuan2Mixtral 8x7BDeepSeek MoEGemma...MSAgent-BenchMOSS-003-SFT🔧Alpaca en/zhWizardLMoasst1Open-PlatypusCode AlpacaColorist🎨Arxiv GenTitleChinese LawOpenOrcaMedical Dialogue...Incremental Pre-trainingSingle-turn Conversation SFTMulti-turn Conversation SFTQLoRALoRAFull parameter fine-tune🛠️ Quick StartInstallationIt is recommended to build a Python-3.10 virtual environment using condacondacreate--namextuner-envpython=3.10-y
condaactivatextuner-envInstall XTuner via pippipinstall-Uxtuneror with DeepSpeed integrationpipinstall-U'xtuner[deepspeed]'Install XTuner from sourcegitclonehttps://github.com/InternLM/xtuner.gitcdxtuner
pipinstall-e'.[all]'Fine-tuneXTuner supports the efficient fine-tune (e.g., QLoRA) for LLMs. Dataset prepare guides can be found ondataset_prepare.md.Step 0, prepare the config. XTuner provides many ready-to-use configs and we can view all configs byxtunerlist-cfgOr, if the provided configs cannot meet the requirements, please copy the provided config to the specified directory and make specific modifications byxtunercopy-cfg${CONFIG_NAME}${SAVE_PATH}vi${SAVE_PATH}/${CONFIG_NAME}_copy.pyStep 1, start fine-tuning.xtunertrain${CONFIG_NAME_OR_PATH}For example, we can start the QLoRA fine-tuning of InternLM2-Chat-7B with oasst1 dataset by# On a single GPUxtunertraininternlm2_chat_7b_qlora_oasst1_e3--deepspeeddeepspeed_zero2# On multiple GPUs(DIST)NPROC_PER_NODE=${GPU_NUM}xtunertraininternlm2_chat_7b_qlora_oasst1_e3--deepspeeddeepspeed_zero2(SLURM)srun${SRUN_ARGS}xtunertraininternlm2_chat_7b_qlora_oasst1_e3--launcherslurm--deepspeeddeepspeed_zero2--deepspeedmeans usingDeepSpeed🚀 to optimize the training. XTuner comes with several integrated strategies including ZeRO-1, ZeRO-2, and ZeRO-3. If you wish to disable this feature, simply remove this argument.For more examples, please seefinetune.md.Step 2, convert the saved PTH model (if using DeepSpeed, it will be a directory) to HuggingFace model, byxtunerconvertpth_to_hf${CONFIG_NAME_OR_PATH}${PTH}${SAVE_PATH}ChatXTuner provides tools to chat with pretrained / fine-tuned LLMs.xtunerchat${NAME_OR_PATH_TO_LLM}--adapter{NAME_OR_PATH_TO_ADAPTER}[optionalarguments]For example, we can start the chat withInternLM2-Chat-7B with adapter trained from oasst1 dataset:xtunerchatinternlm/internlm2-chat-7b--adapterxtuner/internlm2-chat-7b-qlora-oasst1--prompt-templateinternlm2_chatLLaVA-InternLM2-7B:xtunerchatinternlm/internlm2-chat-7b--visual-encoderopenai/clip-vit-large-patch14-336--llavaxtuner/llava-internlm2-7b--prompt-templateinternlm2_chat--image$IMAGE_PATHFor more examples, please seechat.md.DeploymentStep 0, merge the HuggingFace adapter to pretrained LLM, byxtunerconvertmerge\${NAME_OR_PATH_TO_LLM}\${NAME_OR_PATH_TO_ADAPTER}\${SAVE_PATH}\--max-shard-size2GBStep 1, deploy fine-tuned LLM with any other framework, such asLMDeploy🚀.pipinstalllmdeploy
python-mlmdeploy.pytorch.chat${NAME_OR_PATH_TO_LLM}\--max_new_tokens256\--temperture0.8\--top_p0.95\--seed0🔥 Seeking efficient inference with less GPU memory? Try 4-bit quantization fromLMDeploy! For more details, seehere.EvaluationWe recommend usingOpenCompass, a comprehensive and systematic LLM evaluation library, which currently supports 50+ datasets with about 300,000 questions.🤝 ContributingWe appreciate all contributions to XTuner. Please refer toCONTRIBUTING.mdfor the contributing guideline.🎖️ AcknowledgementLlama 2QLoRALMDeployLLaVA🖊️ Citation@misc{2023xtuner,title={XTuner: A Toolkit for Efficiently Fine-tuning LLM},author={XTuner Contributors},howpublished={\url{https://github.com/InternLM/xtuner}},year={2023}}LicenseThis project is released under theApache License 2.0. Please also adhere to the Licenses of models and datasets being used.