Package: xturing
Build, customize and control your own personal LLMsxTuringprovides fast, efficient and simple fine-tuning of LLMs, such as LLaMA, GPT-J, Galactica, and more.
By providing an easy-to-use interface for fine-tuning LLMs to your own data and application, xTuring makes it
simple to build, customize and control LLMs. The entire process can be done inside your computer or in your
private cloud, ensuring data privacy and security.WithxTuringyou can,Ingest data from different sources and preprocess them to a format LLMs can understandScale from single to multiple GPUs for faster fine-tuningLeverage memory-efficient methods (i.e. INT4, LoRA fine-tuning) to reduce hardware costs by up to 90%Explore different fine-tuning methods and benchmark them to find the best performing modelEvaluate fine-tuned models on well-defined metrics for in-depth analysis🌟 What's new?We are excited to announce the latest enhancements to ourxTuringlibrary:LLaMA 2integration- You can use and fine-tune theLLaMA 2model in different configurations:off-the-shelf,off-the-shelf with INT8 precision,LoRA fine-tuning,LoRA fine-tuning with INT8 precisionandLoRA fine-tuning with INT4 precisionusing theGenericModelwrapper and/or you can use theLlama2class fromxturing.modelsto test and finetune the model.fromxturing.modelsimportLlama2model=Llama2()## orfromxturing.modelsimportBaseModelmodel=BaseModel.create('llama2')Evaluation- Now you can evaluate anyCausal Language Modelon any dataset. The metrics currently supported isperplexity.# Make the necessary importsfromxturing.datasetsimportInstructionDatasetfromxturing.modelsimportBaseModel# Load the desired datasetdataset=InstructionDataset('../llama/alpaca_data')# Load the desired modelmodel=BaseModel.create('gpt2')# Run the Evaluation of the model on the datasetresult=model.evaluate(dataset)# Print the resultprint(f"Perplexity of the evalution:{result}")INT4Precision- You can now use and fine-tune any LLM withINT4 PrecisionusingGenericLoraKbitModel.# Make the necessary importsfromxturing.datasetsimportInstructionDatasetfromxturing.modelsimportGenericLoraKbitModel# Load the desired datasetdataset=InstructionDataset('../llama/alpaca_data')# Load the desired model for INT4 bit fine-tuningmodel=GenericLoraKbitModel('tiiuae/falcon-7b')# Run the fine-tuningmodel.finetune(dataset)CPU inference- Now you can use just your CPU for inference of any LLM.CAUTION : The inference process may be sluggish because CPUs lack the required computational capacity for efficient inference.Batch integration- By tweaking the 'batch_size' in the .generate() and .evaluate() functions, you can expedite results. Using a 'batch_size' greater than 1 typically enhances processing efficiency.# Make the necessary importsfromxturing.datasetsimportInstructionDatasetfromxturing.modelsimportGenericLoraKbitModel# Load the desired datasetdataset=InstructionDataset('../llama/alpaca_data')# Load the desired model for INT4 bit fine-tuningmodel=GenericLoraKbitModel('tiiuae/falcon-7b')# Generate outputs on desired promptsoutputs=model.generate(dataset=dataset,batch_size=10)An exploration of theLlama LoRA INT4 working exampleis recommended for an understanding of its application.For an extended insight, consider examining theGenericModel working exampleavailable in the repository.⚙️ Installationpipinstallxturing🚀 Quickstartfromxturing.datasetsimportInstructionDatasetfromxturing.modelsimportBaseModel# Load the datasetinstruction_dataset=InstructionDataset("./alpaca_data")# Initialize the modelmodel=BaseModel.create("llama_lora")# Finetune the modelmodel.finetune(dataset=instruction_dataset)# Perform inferenceoutput=model.generate(texts=["Why LLM models are becoming so important?"])print("Generated output by the model:{}".format(output))You can find the data folderhere.CLI playground$xturingchat-m"<path-to-model-folder>"UI playgroundfromxturing.datasetsimportInstructionDatasetfromxturing.modelsimportBaseModelfromxturing.uiimportPlaygrounddataset=InstructionDataset("./alpaca_data")model=BaseModel.create("<model_name>")model.finetune(dataset=dataset)model.save("llama_lora_finetuned")Playground().launch()## launches localhost UI📚 TutorialsPreparing your datasetCerebras-GPT fine-tuning with LoRA and INT8Cerebras-GPT fine-tuning with LoRALLaMA fine-tuning with LoRA and INT8LLaMA fine-tuning with LoRALLaMA fine-tuningGPT-J fine-tuning with LoRA and INT8GPT-J fine-tuning with LoRAGPT-2 fine-tuning with LoRA📊 PerformanceHere is a comparison for the performance of different fine-tuning techniques on the LLaMA 7B model. We use theAlpaca datasetfor fine-tuning. The dataset contains 52K instructions.Hardware:4xA100 40GB GPU, 335GB CPU RAMFine-tuning parameters:{'maximum sequence length':512,'batch size':1,}LLaMA-7BDeepSpeed + CPU OffloadingLoRA + DeepSpeedLoRA + DeepSpeed + CPU OffloadingGPU33.5 GB23.7 GB21.9 GBCPU190 GB10.2 GB14.9 GBTime/epoch21 hours20 mins20 minsContribute to this by submitting your performance results on other GPUs by creating an issue with your hardware specifications, memory consumption and time per epoch.📎 Fine-tuned model checkpointsWe have already fine-tuned some models that you can use as your base or start playing with.
Here is how you would load them:fromxturing.modelsimportBaseModelmodel=BaseModel.load("x/distilgpt2_lora_finetuned_alpaca")modeldatasetPathDistilGPT-2 LoRAalpacax/distilgpt2_lora_finetuned_alpacaLLaMA LoRAalpacax/llama_lora_finetuned_alpacaSupported ModelsBelow is a list of all the supported models viaBaseModelclass ofxTuringand their corresponding keys to load them.ModelKeyBloombloomCerebrascerebrasDistilGPT-2distilgpt2Falcon-7BfalconGalacticagalacticaGPT-JgptjGPT-2gpt2LlaMAllamaLlaMA2llama2OPT-1.3BoptThe above mentioned are the base variants of the LLMs. Below are the templates to get theirLoRA,INT8,INT8 + LoRAandINT4 + LoRAversions.VersionTemplateLoRA<model_key>_loraINT8<model_key>_int8INT8 + LoRA<model_key>_lora_int8** In order to load any model'sINT4+LoRAversion, you will need to make use ofGenericLoraKbitModelclass fromxturing.models. Below is how to use it:model=GenericLoraKbitModel('<model_path>')Themodel_pathcan be replaced with you local directory or any HuggingFace library model likefacebook/opt-1.3b.📈 RoadmapSupport forLLaMA,GPT-J,GPT-2,OPT,Cerebras-GPT,GalacticaandBloommodelsDataset generation using self-instructionLow-precision LoRA fine-tuning and unsupervised fine-tuningINT8 low-precision fine-tuning supportOpenAI, Cohere and AI21 Studio model APIs for dataset generationAdded fine-tuned checkpoints for some models to the hubINT4 LLaMA LoRA fine-tuning demoINT4 LLaMA LoRA fine-tuning with INT4 generationSupport for aGeneric modelwrapperSupport forFalcon-7BmodelINT4 low-precision fine-tuning supportEvaluation of LLM modelsINT3, INT2, INT1 low-precision fine-tuning supportSupport for Stable Diffusion🤝 Help and SupportIf you have any questions, you can create an issue on this repository.You can also join ourDiscord serverand start a discussion in the#xturingchannel.📝 LicenseThis project is licensed under the Apache License 2.0 - see theLICENSEfile for details.🌎 ContributingAs an open source project in a rapidly evolving field, we welcome contributions of all kinds, including new features and better documentation. Please read ourcontributing guideto learn how you can get involved.