Package: xllm
🦖 X—LLM: Cutting Edge & Easy LLM FinetuningCutting Edge & Easy LLM Finetuning using the most advanced methods (QLoRA, DeepSpeed, GPTQ, Flash Attention 2, FSDP,
etc)Developed
byBoris Zubarev|CV|LinkedIn|bobazooba@gmail.comWhy you should use X—LLM 🪄Are you usingLarge Language Models (LLMs)for your work and want to train them more efficiently with advanced
methods? Wish to focus on the data and improvements rather than repetitive and time-consuming coding for LLM training?X—LLMis your solution. It's a user-friendly library that streamlines training optimization, so you canfocus on
enhancing your models and data. Equipped withcutting-edge training techniques, X—LLM is engineered for efficiency
by engineers who understand your needs.X—LLMis ideal whether you'regearing up for productionor need afast prototyping tool.FeaturesHassle-free training for Large Language ModelsSeamless integration of new data and data processingEffortless expansion of the librarySpeed up your training, while simultaneously reducing model sizesEach checkpoint is saved to the 🤗 HuggingFace HubEasy-to-use integration with your existing projectCustomize almost any part of your training with easeTrack your training progress usingW&BSupported many 🤗 Transformers models
likeYi-34B,Mistal AI,Llama 2,Zephyr,OpenChat,Falcon,Phi,Qwen,MPTand many moreBenefit from cutting-edge advancements in LLM training optimizationQLoRA and fusingFlash Attention 2Gradient checkpointingbitsandbytesGPTQ (including post-training quantization)DeepSpeedFSDPAnd many moreQuickstart 🦖InstallationX—LLMis tested on Python 3.8+, PyTorch 2.0.1+ and CUDA 11.8.pipinstallxllmVersion which includedeepspeed,flash-attnandauto-gptq:pipinstall"xllm[train]"Defaultxllmversion recommended for local development,xllm[train]recommended for training.Training recommended environmentCUDA version:11.8Docker:huggingface/transformers-pytorch-gpu:latestFast prototyping ⚡fromxllmimportConfigfromxllm.datasetsimportGeneralDatasetfromxllm.experimentsimportExperiment# Init Config which controls the internal logic of xllm# QLoRA exampleconfig=Config(model_name_or_path="HuggingFaceH4/zephyr-7b-beta",apply_lora=True,load_in_4bit=True,)# Prepare the datatrain_data=["Hello!"]*100train_dataset=GeneralDataset.from_list(data=train_data)# Build Experiment from Config: init tokenizer and model, apply LoRA and so onexperiment=Experiment(config=config,train_dataset=train_dataset)experiment.build()# Run Experiment (training)experiment.run()# # [Optional] Fuse LoRA layers# experiment.fuse_lora()# [Optional] Or push LoRA weights to HuggingFace Hubexperiment.push_to_hub(repo_id="YOUR_NAME/MODEL_NAME")HowConfigcontrolsxllmMore about configLoRASimpleconfig=Config(model_name_or_path="openchat/openchat_3.5",apply_lora=True,)Advancedconfig=Config(model_name_or_path="openchat/openchat_3.5",apply_lora=True,lora_rank=8,lora_alpha=32,lora_dropout=0.05,raw_lora_target_modules="all",# Names of modules to apply LoRA. A comma-separated string, for example: "k,q,v" or "all".)QLoRATo train theQLoRAmodel, we need to load the backbone model usingbitsandbyteslibrary and int4 (or int8) weights.Simpleconfig=Config(model_name_or_path="01-ai/Yi-34B",apply_lora=True,load_in_4bit=True,prepare_model_for_kbit_training=True,)Advancedconfig=Config(model_name_or_path="01-ai/Yi-34B",apply_lora=True,load_in_4bit=True,prepare_model_for_kbit_training=True,llm_int8_threshold=6.0,llm_int8_has_fp16_weight=True,bnb_4bit_use_double_quant=True,bnb_4bit_quant_type="nf4",)Push checkpoints to the HuggingFace HubBefore that, you must log in toHuggingface Hubor add anAPI Tokento the environment variables.config=Config(model_name_or_path="HuggingFaceH4/zephyr-7b-beta",push_to_hub=True,hub_private_repo=True,hub_model_id="BobaZooba/AntModel-7B-XLLM-Demo-LoRA",save_steps=25,)Checkpoints will be saved locally and in Huggingface Hub eachsave_stepsIf you train a model withLoRA, then onlyLoRAweights will be savedReport to W&BBefore that, you must log in toW&Bor add anAPI Tokento the environment variables.config=Config(model_name_or_path="HuggingFaceH4/zephyr-7b-beta",report_to_wandb=True,logging_steps=5,wandb_project="xllm-demo",)Gradient checkpointingThis will help to useless GPU memoryduring training, that is, you will be able to learn more than without this
technique. The disadvantages of this technique is slowing down the forward step, that is,slowing down training.You will be training larger models (for example 7B in colab), but at the expense of training speed.config=Config(model_name_or_path="HuggingFaceH4/zephyr-7b-beta",use_gradient_checkpointing=True,)Flash Attention 2This speeds up training and GPU memory consumption, but it does not work with all models and GPUs. You also need to
installflash-attnfor this. This can be done using:pip install "xllm[train]"config=Config(model_name_or_path="meta-llama/Llama-2-7b-hf",use_flash_attention_2=True,)Recommended setupRecommendationsIncredibly effective method is LoRA (apply_lora). It allows for a tremendous reduction in training costs
and, moreover, helps very effectively combat catastrophic forgetting.Then, I advise usingload_in_4bitandprepare_model_for_kbit_trainingtogether. This also significantly reduces
memory consumption.Lastly, I would recommend applyuse_gradient_checkpointing. This method also greatly reduces memory consumption, but
at the expense of slowing down training.If your project allows, I suggest enablingpush_to_hubandhub_private_repo, also specifying the model name
inhub_model_idandsave_steps. Example: "BobaZooba/SupaDupaLlama-7B-LoRA". During training, every checkpoint of
your model will be saved in the HuggingFace Hub. If you specifiedapply_lora, then only the LoRA weights will be
saved, which you can later easily fuse with the main model, for example, usingxllm.If your GPU allows it adduse_flash_attention_2I also recommend usingreport_to_wandb, also specifyingwandb_project(the project name in W&B)
andwandb_entity(user or organization name in W&B).Note that forpush_to_hub, you need to log in to the HuggingFace Hub beforehand or specify the
token (HUGGING_FACE_HUB_TOKEN) in the .env file. Similarly, when usingreport_to_wandb, you will need to log in to
W&B. You can either specify the token (WANDB_API_KEY) in the .env file or you will be prompted to enter the token on
the command line.FeaturesQLoRAGradient checkpointingFlash Attention 2Stabilize trainingPush checkpoints to HuggingFace HubW&B reportconfig=Config(model_name_or_path="meta-llama/Llama-2-7b-hf",tokenizer_padding_side="right",# good for llama2warmup_steps=1000,max_steps=10000,logging_steps=1,save_steps=1000,per_device_train_batch_size=2,gradient_accumulation_steps=2,max_length=2048,stabilize=True,use_flash_attention_2=True,apply_lora=True,load_in_4bit=True,prepare_model_for_kbit_training=True,use_gradient_checkpointing=True,push_to_hub=False,hub_private_repo=True,hub_model_id="BobaZooba/SupaDupaLlama-7B-LoRA",report_to_wandb=False,wandb_project="xllm-demo",wandb_entity="bobazooba",)FuseThis operation is only for models with a LoRA adapter.You can explicitly specify to fuse the model after training.config=Config(model_name_or_path="HuggingFaceH4/zephyr-7b-beta",apply_lora=True,fuse_after_training=True,)Even when you are using QLoRaconfig=Config(model_name_or_path="HuggingFaceH4/zephyr-7b-beta",apply_lora=True,load_in_4bit=True,prepare_model_for_kbit_training=True,fuse_after_training=True,)Or you can fuse the model yourself after training.experiment.fuse_lora()DeepSpeedDeepSpeedis needed for training models onmultiple GPUs.DeepSpeedallows you
toefficiently manage the resources of several GPUs during training. For example, you
candistribute the gradients and the state of the optimizer to several GPUs, rather than storing a complete set of
gradients and the state of the optimizer on each GPU. Starting training usingDeepSpeedcan only happen from
thecommand line.train.pyfromxllmimportConfigfromxllm.datasetsimportGeneralDatasetfromxllm.cliimportcli_run_trainif__name__=='__main__':train_data=["Hello!"]*100train_dataset=GeneralDataset.from_list(data=train_data)cli_run_train(config_cls=Config,train_dataset=train_dataset)Run train (in thenum_gpusparameter, specify as many GPUs as you have)deepspeed--num_gpus=8train.py--deepspeed_stage2You also can pass other parametersdeepspeed--num_gpus=8train.py\--deepspeed_stage2\--apply_loraTrue\--stabilizeTrue\--use_gradient_checkpointingTrueNotebooksNameCommentLinkX—LLM PrototypingIn this notebook you will learn the basics of the libraryLlama2 & Mistral AI efficient fine-tuning7B model training in colab using QLoRA, bnb int4, gradient checkpointing and X—LLMProduction solution 🚀X—LLMenables not only to prototype models, but also facilitates the development of production-ready solutions through
built-in capabilities and customization.UsingX—LLMto train a model is easy and involves these few steps:Prepare— Get the data and the model ready by downloading and preparing them. Saves data locally
toconfig.train_local_path_to_dataandconfig.eval_local_path_to_dataif you are using eval datasetTrain— Use the data prepared in the previous step to train the modelFuse— If you used LoRA during the training, fuse LoRAQuantize— Optimize your model's memory usage by quantizing itRemember, these tasks inX—LLMstart from the command line. So, when you're all set to go, launching your full project
will look something like this:Example how to run your projectDownloading and preparing data and modelpython3MY_PROJECT/cli/prepare.py\--dataset_keyMY_DATASET\--model_name_or_pathmistralai/Mistral-7B-v0.1\--path_to_env_file./.envRun train using DeepSpeed on multiple GPUsdeepspeed--num_gpus=8MY_PROJECT/cli/train.py\--use_gradient_checkpointingTrue\--deepspeed_stage2\--stabilizeTrue\--model_name_or_pathmistralai/Mistral-7B-v0.1\--use_flash_attention_2False\--load_in_4bitTrue\--apply_loraTrue\--raw_lora_target_modulesall\--per_device_train_batch_size8\--warmup_steps1000\--save_total_limit0\--push_to_hubTrue\--hub_model_idMY_HF_HUB_NAME/LORA_MODEL_NAME\--hub_private_repoTrue\--report_to_wandbTrue\--path_to_env_file./.envFuse LoRApython3MY_PROJECT/cli/fuse.py\--model_name_or_pathmistralai/Mistral-7B-v0.1\--lora_hub_model_idMY_HF_HUB_NAME/LORA_MODEL_NAME\--hub_model_idMY_HF_HUB_NAME/MODEL_NAME\--hub_private_repoTrue\--force_fp16True\--fused_model_local_path./fused_model/\--path_to_env_file./.env[Optional] GPTQ quantization of the trained model with fused LoRApython3MY_PROJECT/cli/quantize.py\--model_name_or_path./fused_model/\--apply_loraFalse\--stabilizeFalse\--quantization_max_samples128\--quantized_model_path./quantized_model/\--prepare_model_for_kbit_trainingFalse\--quantized_hub_model_idMY_HF_HUB_NAME/MODEL_NAME_GPTQ\--quantized_hub_private_repoTrue\--path_to_env_file./.envRight now, theX—LLMlibrary lets you use only theSODA dataset. We've
set it up this way for demo purposes, but we're planning to add more datasets soon. You'll need to figure out how to
download and handle your dataset. Simply put, you take care of your data, andX—LLMhandles the rest. We've done it
this
way on purpose, to give you plenty of room to get creative and customize to your heart's content.You can customize your dataset in detail, adding additional fields. All of this will enable you to implement virtually
any task in the areas ofSupervised LearningandOffline Reinforcement Learning.At the same time, you always have an easy way to submit data for language modeling.ExamplefromxllmimportConfigfromxllm.datasetsimportGeneralDatasetfromxllm.cliimportcli_run_trainif__name__=='__main__':train_data=["Hello!"]*100train_dataset=GeneralDataset.from_list(data=train_data)cli_run_train(config_cls=Config,train_dataset=train_dataset)Build your own projectTo set up your own project usingX—LLM, you need to do two things:Implement your dataset (figure out how to download and handle it)AddX—LLM's command-line tools into your projectOnce that's done, your project will be good to go, and you can start running the steps you need (like prepare, train,
and so on).To get a handle on building your project withX—LLM, check out the materials below.Useful materialsGuide: here, we go into detail about everything the library can
doDemo project: here's a step-by-step example of how to useX—LLMand fit
it
into your own projectWeatherGPT: this repository features an example of how to utilize the xllm library. Included is a solution for a common type of assessment given to LLM engineers, who typically earn between $120,000 to $140,000 annuallyShurale: project with the finetuned 7B Mistal modelConfig 🔧TheX—LLMlibrary uses a single config setup for all steps like preparing, training and the other steps. It's
designed in a way that
lets you easily understand the available features and what you can adjust.Confighas control almost over every
single part of each step. Thanks to the config, you can pick your dataset, set your collator, manage the type of
quantization during training, decide if you want to use LoRA, if you need to push a checkpoint to theHuggingFace Hub,
and a
lot more.Config path:src.xllm.core.config.ConfigOrfromxllmimportConfigUseful materialsImportant config fields for different stepsHow do I choose the methods for training?Detailed description of all config fieldsCustomization options 🛠You have the flexibility to tweak many aspects of your model's training: data, how data is processed, trainer, config,
how the model is loaded, what happens before and after training, and so much more.We've got ready-to-use components for every part of thexllmpipeline. You can entirely switch out some components
like the dataset, collator, trainer, and experiment.
For some components like experiment and config, you have the option to just build on what's already there.Useful materialsHow to implement datasetHow to add CLI tools to your projectHow to implement collatorHow to implement trainerHow to implement experimentHow to extend configProjects using X—LLM 🏆Building something cool withX—LLM? Kindly reach out to me
atbobazooba@gmail.com. I'd love to hear from you.Hall of FameWrite to us so that we can add your project.Shurale7B-v1Shurale7B-v1-GPTQBadgeConsider adding a badge to your model card.[<imgsrc="https://github.com/BobaZooba/xllm/blob/main/static/images/xllm-badge.png"alt="Powered by X—LLM"width="175"height="32"/>](https://github.com/BobaZooba/xllm)Testing 🧪At the moment, we don't have Continuous Integration tests that utilize a GPU. However, we might develop these kinds of
tests in the future. It's important to note, though, that this would require investing time into their development, as
well as funding for machine maintenance.Future Work 🔮Add more teststAdd RunPod deploy functionAddDPOcomponentsAdd Callbacks toExperimentGPU CI using RunPodSolve DeepSpeed + push_to_hub checkpoints bugMake DeepSpeed Stage 3 + bitsandbytes works correcAdd multipackingAdd adaptive batch sizeFix caching in CIAdd sequence bucketingAdd more datasetsMaybe addtensor_parallelTale QuestTale Questis my personal project which was built usingxllmandShurale. It's an interactive text-based game
inTelegramwith dynamic AI characters, offering infinite scenariosYou will get into exciting journeys and complete fascinating quests. Chat
withGeorge Orwell,Tech Entrepreneur,Young Wizard,Noir Detective,Femme Fataleand many moreTry it now:https://t.me/talequestbotPlease support me here