Package: llama-recipes
Llama 2 Fine-tuning / Inference Recipes and ExamplesThe 'llama-recipes' repository is a companion to theLlama 2 model. The goal of this repository is to provide examples to quickly get started with fine-tuning for domain adaptation and how to run inference for the fine-tuned models. For ease of use, the examples use Hugging Face converted versions of the models. See steps for conversion of the modelhere.Llama 2 is a new technology that carries potential risks with use. Testing conducted to date has not — and could not — cover all scenarios. In order to help developers address these risks, we have created theResponsible Use Guide. More details can be found in our research paper as well. For downloading the models, follow the instructions onLlama 2 repo.Table of ContentsQuick startModel ConversionFine-tuningSingle GPUMulti GPU One NodeMulti GPU Multi NodeInferenceRepository OrganizationLicense and Acceptable Use PolicyQuick StartLlama 2 Jupyter Notebook: This jupyter notebook steps you through how to finetune a Llama 2 model on the text summarization task using thesamsum. The notebook uses parameter efficient finetuning (PEFT) and int8 quantization to finetune a 7B on a single GPU like an A10 with 24GB gpu memory.InstallationLlama-recipes provides a pip distribution for easy install and usage in other projects. Alternatively, it can be installed from source.Install with pippip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipesInstall from sourceTo install from source e.g. for development use this command. We're using hatchling as our build backend which requires an up-to-date pip as well as setuptools package.pip install -U pip setuptools
pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 -e .For development and contributing to llama-recipes please install all optional dependencies:pip install -U pip setuptools
pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 -e .[tests,auditnlg,vllm]Install with optional dependenciesLlama-recipes offers the installation of optional packages. There are three optional dependency groups.
To run the unit tests we can install the required dependencies with:pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes[tests]For the vLLM example we need additional requirements that can be installed with:pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes[vllm]To use the sensitive topics safety checker install with:pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes[auditnlg]Optional dependencies can also be combines with [option1,option2].⚠️Note⚠️  Some features (especially fine-tuning with FSDP + PEFT) currently require PyTorch nightlies to be installed. Please make sure to install the nightlies if you're using these features followingthis guide.NoteAll the setting defined inconfig filescan be passed as args through CLI when running the script, there is no need to change from config files directly.NoteIn case need to run PEFT model with FSDP, please make sure to use the PyTorch Nightlies.For more in depth information checkout the following:Single GPU Fine-tuningMulti-GPU Fine-tuningLLM Fine-tuningAdding custom datasetsInferenceFAQsWhere to find the models?You can find llama v2 models on HuggingFace hubhere, where models withhfin the name are already converted to HuggingFace checkpoints so no further conversion is needed. The conversion step below is only for original model weights from Meta that are hosted on HuggingFace model hub as well.Model conversion to Hugging FaceThe recipes and notebooks in this folder are using the Llama 2 model definition provided by Hugging Face's transformers library.Given that the original checkpoint resides under models/7B you can install all requirements and convert the checkpoint with:## Install HuggingFace Transformers from sourcepipfreeze|greptransformers## verify it is version 4.31.0 or highergitclonegit@github.com:huggingface/transformers.gitcdtransformers
pipinstallprotobuf
pythonsrc/transformers/models/llama/convert_llama_weights_to_hf.py\--input_dir/path/to/downloaded/llama/weights--model_size7B--output_dir/output/pathFine-tuningFor fine-tuning Llama 2 models for your domain-specific use cases recipes for PEFT, FSDP, PEFT+FSDP have been included along with a few test datasets. For details seeLLM Fine-tuning.Single and Multi GPU FinetuneIf you want to dive right into single or multi GPU fine-tuning, run the examples below on a single GPU like A10, T4, V100, A100 etc.
All the parameters in the examples and recipes below need to be further tuned to have desired results based on the model, method, data and task at hand.Note:To change the dataset in the commands below pass thedatasetarg. Current options for dataset aregrammar_dataset,alpaca_datasetandsamsum_dataset. A description of the datasets and how to add custom datasets can be found inDataset.md. Forgrammar_dataset,alpaca_datasetplease make sure you use the suggested instructions fromhereto set them up.Default dataset and other LORA config has been set tosamsum_dataset.Make sure to set the right path to the model in thetraining config.Single GPU:#if running on multi-gpu machineexportCUDA_VISIBLE_DEVICES=0python-mllama_recipes.finetuning--use_peft--peft_methodlora--quantization--model_name/patht_of_model_folder/7B--output_dirPath/to/save/PEFT/modelHere we make use of Parameter Efficient Methods (PEFT) as described in the next section. To run the command above make sure to pass thepeft_methodarg which can be set tolora,llama_adapterorprefix.Noteif you are running on a machine with multiple GPUs please make sure to only make one of them visible usingexport CUDA_VISIBLE_DEVICES=GPU:idMake sure you setsave_modelparameter to save the model. Be sure to check the other training parameter intrain configas well as others in the config folder as needed. All parameter can be passed as args to the training script. No need to alter the config files.Multiple GPUs One Node:NOTEplease make sure to use PyTorch Nightlies for using PEFT+FSDP. Also, note that int8 quantization from bit&bytes currently is not supported in FSDP.torchrun--nnodes1--nproc_per_node4examples/finetuning.py--enable_fsdp--use_peft--peft_methodlora--model_name/patht_of_model_folder/7B--pure_bf16--output_dirPath/to/save/PEFT/modelHere we use FSDP as discussed in the next section which can be used along with PEFT methods. To make use of PEFT methods with FSDP make sure to passuse_peftandpeft_methodargs along withenable_fsdp. Here we are usingBF16for training.Flash Attention and Xformer Memory Efficient KernelsSettinguse_fast_kernelswill enable using of Flash Attention or Xformer memory-efficient kernels based on the hardware being used. This would speed up the fine-tuning job. This has been enabled inoptimumlibrary from HuggingFace as a one-liner API, please read morehere.torchrun--nnodes1--nproc_per_node4examples/finetuning.py--enable_fsdp--use_peft--peft_methodlora--model_name/patht_of_model_folder/7B--pure_bf16--output_dirPath/to/save/PEFT/model--use_fast_kernelsFine-tuning using FSDP OnlyIf you are interested in running full parameter fine-tuning without making use of PEFT methods, please use the following command. Make sure to change thenproc_per_nodeto your available GPUs. This has been tested withBF16on 8xA100, 40GB GPUs.torchrun--nnodes1--nproc_per_node8examples/finetuning.py--enable_fsdp--model_name/patht_of_model_folder/7B--dist_checkpoint_root_foldermodel_checkpoints--dist_checkpoint_folderfine-tuned--use_fast_kernelsFine-tuning using FSDP on 70B ModelIf you are interested in running full parameter fine-tuning on the 70B model, you can enablelow_cpu_fsdpmode as the following command. This option will load model on rank0 only before moving model to devices to construct FSDP. This can dramatically save cpu memory when loading large models like 70B (on a 8-gpu node, this reduces cpu memory from 2+T to 280G for 70B model). This has been tested withBF16on 16xA100, 80GB GPUs.torchrun--nnodes1--nproc_per_node8examples/finetuning.py--enable_fsdp--low_cpu_fsdp--pure_bf16--model_name/patht_of_model_folder/70B--batch_size_training1--dist_checkpoint_root_foldermodel_checkpoints--dist_checkpoint_folderfine-tunedMulti GPU Multi Node:sbatchmulti_node.slurm# Change the num nodes and GPU per nodes in the script before running.You can read more about our fine-tuning strategieshere.Repository OrganizationThis repository is organized in the following way:configs: Contains the configuration files for PEFT methods, FSDP, Datasets.docs: Example recipes for single and multi-gpu fine-tuning recipes.datasets: Contains individual scripts for each dataset to download and process. Note: Use of any of the datasets should be in compliance with the dataset's underlying licenses (including but not limited to non-commercial uses)examples: Contains examples script for finetuning and inference of the Llama 2 model as well as how to use them safely.inference: Includes modules for inference for the fine-tuned models.model_checkpointing: Contains FSDP checkpoint handlers.policies: Contains FSDP scripts to provide different policies, such as mixed precision, transformer wrapping policy and activation checkpointing along with any precision optimizer (used for running FSDP with pure bf16 mode).utils: Utility files for:train_utils.pyprovides training/eval loop and more train utils.dataset_utils.pyto get preprocessed datasets.config_utils.pyto override the configs received from CLI.fsdp_utils.pyprovides FSDP  wrapping policy for PEFT methods.memory_utils.pycontext manager to track different memory stats in train loop.LicenseSee the License filehereand Acceptable Use Policyhere