Package: llm-foundry
LLM FoundryThis repository contains code for training, finetuning, evaluating, and deploying LLMs for inference withComposerand theMosaicML platform. Designed to be easy-to-use, efficientandflexible, this codebase enables rapid experimentation with the latest techniques.You'll find in this repo:llmfoundry/- source code for models, datasets, callbacks, utilities, etc.scripts/- scripts to run LLM workloadsdata_prep/- convert text data from original sources to StreamingDataset formattrain/- train or finetune HuggingFace and MPT models from 125M - 70B parameterstrain/benchmarking- profile training throughput and MFUinference/- convert models to HuggingFace or ONNX format, and generate responsesinference/benchmarking- profile inference latency and throughputeval/- evaluate LLMs on academic (or custom) in-context-learning tasksmcli/- launch any of these workloads usingMCLIand theMosaicML platformTUTORIAL.md- a deeper dive into the repo, example workflows, and FAQsMPTMosaic Pretrained Transformers (MPT) are GPT-style models with some special features -- Flash Attention for efficiency, ALiBi for context length extrapolation, and stability improvements to mitigate loss spikes. As part of MosaicML's Foundation series, we have open-sourced several MPT models:ModelContext LengthDownloadCommercial use?MPT-30B8192https://huggingface.co/mosaicml/mpt-30bYesMPT-30B-Instruct8192https://huggingface.co/mosaicml/mpt-30b-instructYesMPT-30B-Chat8192https://huggingface.co/mosaicml/mpt-30b-chatNoMPT-7b-8k8192https://huggingface.co/mosaicml/mpt-7b-8kYesMPT-7b-8k-Chat8192https://huggingface.co/mosaicml/mpt-7b-8k-chatNoMPT-7B2048https://huggingface.co/mosaicml/mpt-7bYesMPT-7B-Instruct2048https://huggingface.co/mosaicml/mpt-7b-instructYesMPT-7B-Chat2048https://huggingface.co/mosaicml/mpt-7b-chatNoMPT-7B-StoryWriter65536https://huggingface.co/mosaicml/mpt-7b-storywriterYesTo try out these models locally,follow the instructionsinscripts/inference/README.mdto prompt HF models using ourhf_generate.pyorhf_chat.pyscripts.MPT CommunityWe've been overwhelmed by all the amazing work the community has put into MPT! Here we provide a few links to some of them:ReplitLM:replit-code-v1-3bis a 2.7B Causal Language Model focused on Code Completion. The model has been trained on a subset of the Stack Dedup v1.2 dataset covering 20 languages such as Java, Python, and C++LLaVa-MPT: Visual instruction tuning to get MPT multimodal capabilitiesggml: Optimized MPT version for efficient inference on consumer hardwareGPT4All: locally running chat system, now with MPT support!Q8MPT-Chat: 8-bit optimized MPT for CPU by our friends at IntelTutorial videos from the community:Using MPT-7B with Langchainby@jamesbriggsMPT-7B StoryWriter IntrobyAItrepreneurFine-tuning MPT-7B on a single GPUby@AIology2022How to Fine-tune MPT-7B-Instruct on Google Colabby@VRSENSomething missing? Contribute with a PR!Latest NewsBlog: LLM Training and Inference with Intel Gaudi2 AI AcceleratorsBlog: Training LLMs at Scale with AMD MI250 GPUsBlog: Training LLMs with AMD MI250 GPUs and MosaicMLBlog: Announcing MPT-7B-8K: 8K Context Length for Document UnderstandingBlog: Training LLMs with AMD MI250 GPUs and MosaicMLBlog: MPT-30B: Raising the bar for open-source foundation modelsBlog: Introducing MPT-7BBlog: Benchmarking LLMs on H100Blog: Blazingly Fast LLM EvaluationBlog: GPT3 Quality for $500kBlog: Billion parameter GPT training made easyHardware and Software RequirementsThis codebase has been tested with PyTorch 2.1 with NVIDIA A100s and H100s.
This codebase may also work on systems with other devices, such as consumer NVIDIA cards and AMD cards, but we are not actively testing these systems.
If you have success/failure using LLM Foundry on other systems, please let us know in a Github issue and we will update the support matrix!DeviceTorch VersionCuda VersionStatusA100-40GB/80GB2.1.012.1:white_check_mark: SupportedH100-80GB2.1.012.1:white_check_mark: SupportedMosaicML Docker ImagesWe highly recommend using our prebuilt Docker images. You can find them here:https://hub.docker.com/orgs/mosaicml/repositories.Themosaicml/pytorchimages are pinned to specific PyTorch and CUDA versions, and are stable and rarely updated.Themosaicml/llm-foundryimages are built with new tags upon every commit to themainbranch.
You can select a specific commit hash such asmosaicml/llm-foundry:1.13.1_cu117-f678575or take the latest one usingmosaicml/llm-foundry:1.13.1_cu117-latest.Please Note:Themosaicml/llm-foundryimages do not come with thellm-foundrypackage preinstalled, just the dependencies. You will still need topip install llm-foundryeither from PyPi or from source.Docker ImageTorch VersionCuda VersionLLM Foundry dependencies installed?mosaicml/pytorch:2.1.0_cu121-python3.10-ubuntu20.042.1.012.1 (Infiniband)Nomosaicml/llm-foundry:2.1.0_cu121-latest2.1.012.1 (Infiniband)Yes (flash attention v1. Warning: Support for flash attention v1 has been deprecated.)mosaicml/llm-foundry:2.1.0_cu121_flash2-latest2.1.012.1 (Infiniband)Yes (flash attention v2. Note: We recommend using flash attention v2.)mosaicml/llm-foundry:2.1.0_cu121_aws-latest2.1.012.1 (EFA)Yes (flash attention v1. Warning: Support for flash attention v1 has been deprecated.)mosaicml/llm-foundry:2.1.0_cu121_flash2_aws-latest2.1.012.1 (EFA)Yes (flash attention v2. Note: We recommend using flash attention v2.)InstallationThis assumes you already have PyTorch, CMake, and packaging installed. If not, you can install them withpip install cmake packaging torch.To get started, clone the repo and set up your environment. Instructions to do so differ slightly depending on whether you're using Docker.With Docker (recommended)Westronglyrecommend working with LLM Foundry inside a Docker container (see our recommended Docker image above). If you are doing so, follow these steps to clone the repo and install the requirements.gitclonehttps://github.com/mosaicml/llm-foundry.gitcdllm-foundry
pipinstall-e".[gpu-flash2]"# or `pip install -e .` if no NVIDIA GPU.# Note: Currently, `pip install -e ".[gpu-flash2]"` installs Flash Attention v2, and `pip install -e ".[gpu]"` installs Flash Attention v1.#       However, once the support for Flash Attention v1 is removed, both of these commands will install Flash Attention v2.Without Docker (not recommended)If you choose not to use Docker, you should create and use a virtual environment.gitclonehttps://github.com/mosaicml/llm-foundry.gitcdllm-foundry# Creating and activate a virtual environmentpython3-mvenvllmfoundry-venvsourcellmfoundry-venv/bin/activate

pipinstallcmakepackagingtorch# setup.py requires these be installedpipinstall-e".[gpu-flash2]"# or `pip install -e .` if no NVIDIA GPU.# Note: Currently, `pip install -e ".[gpu-flash2]"` installs Flash Attention v2, and `pip install -e ".[gpu]"` installs Flash Attention v1.#       However, once the support for Flash Attention v1 is removed, both of these commands will install Flash Attention v2.TransformerEngine and amp_fp8 supportNVIDIA H100 GPUs have FP8 support; this additionally requires the following installations:pipinstallflash-attn==1.0.7--no-build-isolation
pipinstallgit+https://github.com/NVIDIA/TransformerEngine.git@v0.10Seeherefor more details on enabling TransformerEngine layers and amp_fp8.AMD (BETA support)Inour testing of AMD GPUs, the env setup includes:gitclonehttps://github.com/mosaicml/llm-foundry.gitcdllm-foundry# Creating and activate a virtual environmentpython3-mvenvllmfoundry-venv-amdsourcellmfoundry-venv-amd/bin/activate# installspipinstallcmakepackagingtorch
pipinstall-e.# This installs some things that are not needed but they don't hurtpip3installtorchtorchvisiontorchaudio--index-urlhttps://download.pytorch.org/whl/rocm5.4.2Lastly, install the ROCm enabled flash attention (instructionshere).Notes:attn_impl: tritondoes not work.We don't yet have a Docker image where everything works perfectly. You might need to up/downgrade some packages (in our case, we needed to downgrade tonumpy==1.23.5) before everything works without issue.Intel GaudiSupport for LLM Foundry on Intel Gaudi devices is experimental, please use the branchhabana_alphaand see theREADME on that branchwhich hasinstall instructions and known issues.For training and inference performance results on Intel Gaudi2 accelerators, see our blog:https://www.databricks.com/blog/llm-training-and-inference-intel-gaudi2-ai-acceleratorsQuickstartNoteMake sure to go through the installation steps above before trying the quickstart!Here is an end-to-end workflow for preparing a subset of the C4 dataset, training an MPT-125M model for 10 batches,
converting the model to HuggingFace format, evaluating the model on the Winograd challenge, and generating responses to prompts.(Remember this is a quickstart just to demonstrate the tools -- To get good quality, the LLM must be trained for longer than 10 batches 😄)cdscripts# Convert C4 dataset to StreamingDataset formatpythondata_prep/convert_dataset_hf.py\--datasetc4--data_subseten\--out_rootmy-copy-c4--splitstrain_smallval_small\--concat_tokens2048--tokenizerEleutherAI/gpt-neox-20b--eos_text'
'# Train an MPT-125m model for 10 batchescomposertrain/train.py\train/yamls/pretrain/mpt-125m.yaml\data_local=my-copy-c4\train_loader.dataset.split=train_small\eval_loader.dataset.split=val_small\max_duration=10ba\eval_interval=0\save_folder=mpt-125m# Convert the model to HuggingFace formatpythoninference/convert_composer_to_hf.py\--composer_pathmpt-125m/ep0-ba10-rank0.pt\--hf_output_pathmpt-125m-hf\--output_precisionbf16\# --hf_repo_for_upload user-org/repo-name# Evaluate the model on a subset of taskscomposereval/eval.py\eval/yamls/hf_eval.yaml\icl_tasks=eval/yamls/copa.yaml\model_name_or_path=mpt-125m-hf# Generate responses to promptspythoninference/hf_generate.py\--name_or_pathmpt-125m-hf\--max_new_tokens256\--prompts\"The answer to life, the universe, and happiness is"\"Here's a quick recipe for baking chocolate chip cookies: Start by"Note: thecomposercommand used above to train the model refers to theComposerlibrary's distributed launcher.If you have a write-enabledHuggingFace auth token, you can optionally upload your model to the Hub! Just export your token like this:exportHUGGING_FACE_HUB_TOKEN=your-auth-tokenand uncomment the line containing--hf_repo_for_upload ...in the above call toinference/convert_composer_to_hf.py.Learn more about LLM Foundry!Check outTUTORIAL.mdto keep learning about working with LLM Foundry. The tutorial highlights example workflows, points you to other resources throughout the repo, and answers frequently asked questions!Contact UsIf you run into any problems with the code, please file Github issues directly to this repo.If you want to train LLMs on the MosaicML platform, reach out to us atdemo@mosaicml.com!