Package: composer
Supercharge your Model TrainingDeep Learning Framework for Training at Scale[Website]-[Getting Started]-[Docs]-[We're Hiring!]👋 WelcomeComposer is an open-source deep learning training library byMosaicML. Built on top of PyTorch, the Composer library makes it easier to implement distributed training workflows on large-scale clusters.We built Composer to beoptimized for scalability and usability, integrating best practices for efficient, multi-node training. By abstracting away low-level complexities like parallelism techniques, distributed data loading, and memory optimization, you can focus on training modern ML models and running experiments without slowing down.We recommend using Composer to speedup your experimentation workflow if you’re training neural networks of any size, including:Large Language Models (LLMs)Diffusion modelsEmbedding models (e.g. BERT)Transformer-based modelsConvolutional Neural Networks (CNNs)Composer is heavily used by the MosaicML research team to train state-of-the-art models like MPT, and we open-sourced this library to enable the ML community to do the same. This framework is used by organizations in both the tech industry and the academic sphere and is continually updated with new features, bug fixes, and stability improvements for production workloads.🔑 Key FeaturesWe designed Composer from the ground up for modern deep learning workloads. Gone are the days of AlexNet and ResNet, when state-of-the-art models could be trained on a couple of desktop GPUs. Today, developing the latest and greatest deep learning models often requires cluster-scale hardware — but with Composer’s help, you’ll hardly notice the difference.The heart of Composer is our Trainer abstraction: a highly optimized PyTorch training loop designed to allow both you and your model to iterate faster. Our trainer has simple ways for you to configure your parallelization scheme, data loaders, metrics, loggers, and more.ScalabilityWhether you’re training on 1 GPU or 512 GPUs, 50MB or 10TB of data - Composer is built to keep your workflow simple.FSDP: For large models that are too large to fit on GPUs, Composer has integrated PyTorchFullyShardedDataParallelisminto our trainer and made it simple to efficiently parallelize custom models. We’ve found FSDP is competitive performance-wise with much more complex parallelism strategies. Alternatively, Composer also supports standard PyTorch distributed data parallelism (DDP) and Deepspeed execution.Elastic sharded checkpointing: Save on eight GPUs, resume on sixteen. Composer supports elastic sharded checkpointing, so you never have to worry if your sharded saved state is compatible with your new hardware setup.Data streaming:Working with large datasets? Download datasets from cloud blob storage on the fly by integrating with MosaicMLStreamingDatasetduring model training.CustomizabilityOther high-level deep learning trainers provide simplicity at the cost of rigidity. When you want to add your own features, their abstractions get in your way. Composer, on the other hand, provides simple ways for you to customize our Trainer to your needs.Fig. 1:Composer’s training loop has a series of events that occur at each stage in the training process. Callbacks are functions that users write to run at specific events. For example, ourLearning Rate Monitor Callbacklogs the learning rate at every BATCH_END event.Callbacks: Composer’s callback system allows you to insert custom logic at any point in the training loop. We’ve written callbacks to monitor memory usage, log and visualize images, and estimate your model’s remaining training time, to name a few. This feature is popular among researchers who want to implement and experiment with custom training techniques.Speedup algorithms: We draw from the latest research to create a collection of algorithmic speedups. Stack these speedups into MosaicML recipes to boost your training speeds. Our team has open-sourced the optimal combinations of speedups for different types of models.8x speedup: Stable Diffusion$200k original SD2 cost —> $50k (Blog)7x speedup: ResNet-50 on ImageNet3h33m —> 25m on 8xA100 (Blog)8.8x speedup: BERT-Base Pretraining10h —> 1.13h on 8xA100 (Blog)5.4x speedup: DeepLab v3 on ADE20K3h30m —> 39m on 8xA100 (Blog)Better workflowsComposer is built to automate away low-level pain points and headaches so you can focus on the important (and fun) parts of deep learning and iterate faster.Auto-resumption: Failed training run? Have no fear — just re-run your code, and Composer will automatically resume from your latest saved checkpoint.CUDA OOM Prevention: Say goodbye to out-of-memory errors.  Set your microbatch size to “auto”, and Composer will automatically select the biggest one that fits on your GPUs.Time Abstractions:Ever messed up your conversion between update steps, epochs, samples, and tokens? Specify your training duration with custom units (epochs, batches, samples, and tokens) in your training loop with ourTimeclass.IntegrationsIntegrate with the tools you know and love for experiment tracking and data streaming.Cloud integrations: Our Checkpointing and logging features have first-class support for remote storage and loading from Cloud bucket (OCI, GCP, AWS S3).Experiment tracking:Weights and Biases, MLFlow, CometML, and neptune.ai — the choice is yours, easily log your data to your favorite platform.🚀 Getting Started📍PrerequisitesComposer is designed for users who are comfortable with Python and have basic familiarity with deep learning fundamentals and PyTorch.Software requirements:A recent version of PyTorch.Hardware requirements:System with CUDA-compatible GPUs (AMD + RoCM coming soon!). Composer can run on CPUs, but for full benefits, we recommend using it on hardware accelerators.💾 InstallationComposer can be installed withpip:pipinstallmosaicmlTo simplify the environment setup for Composer, we also provide a set ofpre-built Docker images. Wehighly recommendyou use our Docker images.🏁 Quick StartHere is a code snippet demonstrating our Trainer on the MNIST dataset.importtorchimporttorch.nnasnnimporttorch.nn.functionalasFfromtorchvisionimportdatasets,transformsfromtorch.utils.dataimportDataLoaderfromcomposerimportTrainerfromcomposer.modelsimportComposerClassifierfromcomposer.algorithmsimportLabelSmoothing,CutMix,ChannelsLastclassModel(nn.Module):"""Toy convolutional neural network architecture in pytorch for MNIST."""def__init__(self,num_classes:int=10):super().__init__()self.num_classes=num_classesself.conv1=nn.Conv2d(1,16,(3,3),padding=0)self.conv2=nn.Conv2d(16,32,(3,3),padding=0)self.bn=nn.BatchNorm2d(32)self.fc1=nn.Linear(32*16,32)self.fc2=nn.Linear(32,num_classes)defforward(self,x):out=self.conv1(x)out=F.relu(out)out=self.conv2(out)out=self.bn(out)out=F.relu(out)out=F.adaptive_avg_pool2d(out,(4,4))out=torch.flatten(out,1,-1)out=self.fc1(out)out=F.relu(out)returnself.fc2(out)transform=transforms.Compose([transforms.ToTensor()])dataset=datasets.MNIST("data",train=True,download=True,transform=transform)train_dataloader=DataLoader(dataset,batch_size=128)trainer=Trainer(model=ComposerClassifier(module=Model(),num_classes=10),train_dataloader=train_dataloader,max_duration="2ep",algorithms=[LabelSmoothing(smoothing=0.1),CutMix(alpha=1.0),ChannelsLast(),],)trainer.fit()Next, check out ourGetting Started Colabfor a walk-through of Composer’s main features. In this tutorial, we will cover the basics of the Composer Trainer:DataloaderTrainerOptimizer and SchedulerLoggingTraining a baseline modelSpeeding up training📚 Learn moreOnce you’ve completed the Quick Start, you can go through the below tutorials or ourdocumentationto further familiarize yourself with Composer.If you have any questions, please feel free to reach out to us on ourCommunity Slack!Here are some resources actively maintained by the Composer community to help you get started:ResourceDetailsTraining BERTs with Composer and 🤗A Colab Notebook showing how to train BERT models with Composer and 🤗!Pretraining and Finetuning an LLM TutorialA tutorial from MosaicML’s LLM Foundry, using MosaicML Composer, StreamingDataset, and MCLI on training and evaluating LLMs.Migrating from PyTorch LightningA tutorial is to illustrating a path from working in PyTorch Lightning to working in Composer.Finetuning and Pretraining HuggingFace ModelsWant to use Hugging Face models with Composer? No problem. Here, we’ll walk through using Composer to fine-tune a pretrained Hugging Face BERT model.Building Speedup MethodsA Colab Notebook showing how to build new training modifications on top of Composer🛠️ For Best Results, Use with the MosaicML EcosystemComposer can be used on its own, but for the smoothest experience we recommend using it in combination with other components of the MosaicML ecosystem:MosaicML platform(MCLI)- Our proprietary Command Line Interface (CLI) and Python SDK for orchestrating, scaling, and monitoring the GPU nodes and container images executing training and deployment. Used by our customers for training their own Generative AI models.To get started,sign up hereto apply for access and check out ourTrainingandInferenceproduct pagesMosaicML LLM Foundry- This open source repository contains code for training, finetuning, evaluating, and preparing LLMs for inference withComposer. Designed to be easy to use, efficient and flexible, this codebase is designed to enable rapid experimentation with the latest techniques.MosaicML StreamingDataset- Open-source library for fast, accurate streaming from cloud storage.MosaicML Diffusion- Open-source code to train your own Stable Diffusion model on your own data.  Learn more via our blogs: (Results,Speedup Details)MosaicML Examples- This repo contains reference examples for using theMosaicML platformto train and deploy machine learning models at scale. It's designed to be easily forked/copied and modified.🏆 Project ShowcaseHere are some projects and experiments that used Composer. Got something to add? Share in ourCommunity Slack!MPT Foundation Series:Commercially usable open source LLMs, optimized for fast training and inference and trained with Composer.MPT-7B BlogMPT-7B-8k BlogMPT-30B BlogMosaic Diffusion Models: see how we trained a stable diffusion model from scratch for <$50kreplit-code-v1-3b: A 2.7B Causal Language Model focused onCode Completion,trained by Replit on the MosaicML platform in 10 days.BabyLLM:the first LLM to support both Arabic and English. This 7B model was trained by MetaDialog on the world’s largest Arabic/English dataset to improve customer support workflows (Blog)BioMedLM: a domain-specific LLM for Bio Medicine built by MosaicML andStanford CRFM💫 ContributorsComposer is part of the broader Machine Learning community, and we welcome any contributions, pull requests, or issues!To start contributing, see ourContributingpage.P.S.:We're hiring!❓FAQWhat is the best tech stack you recommend when training large models?We recommend that users combine components of the MosaicML ecosystem for the smoothest experience:ComposerStreamingDatasetMCLI(MosaicML platform)How can I get community support for using Composer?You can join ourCommunity Slack!How does Composer compare to other trainers like NeMo Megatron and PyTorch Lightning?We built Composer to be optimized for both simplicity and efficiency. Community users have shared that they enjoy Composer for its capabilities and ease of use compared to alternative libraries.How do I use Composer to train graph neural networks (GNNs), or Generative Adversarial Networks (GANs), or models for reinforcement learning (RL)?We recommend you use alternative libraries for if you want to train these types of models - a lot of assumptions we made when designing Composer are suboptimal for GNNs, RL, and GANs✍️ Citation@misc{mosaicml2022composer,
    author = {The Mosaic ML Team},
    title = {composer},
    year = {2021},
    howpublished = {\url{https://github.com/mosaicml/composer/}},
}