Package: deepspeed
Latest NewsDeepSpeed empowers ChatGPT-like model training with a single click, offering 15x speedup over SOTA RLHF systems with unprecedented cost reduction at all scales;learn how.[2024/01]DeepSpeed-FastGen: Introducting Mixtral, Phi-2, and Falcon support with major performance and feature enhancements.[2023/11]Llama 2 Inference on 4th Gen Intel® Xeon® Scalable Processor with DeepSpeed[Intel version][2023/11]DeepSpeed ZeRO-Offload++: 6x Higher Training Throughput via Collaborative CPU/GPU Twin-Flow[2023/11]DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference[English] [中文] [日本語][2023/10]DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs[English] [中文] [日本語][2023/09] Announcing the DeepSpeed4Science Initiative: Enabling large-scale scientific discovery through sophisticated AI system technologies [DeepSpeed4Science website] [Tutorials] [White paper] [Blog] [中文] [日本語]More news[2023/08]DeepSpeed ZeRO-Inference: 20x faster inference through weight quantization and KV cache offloading[2023/08]DeepSpeed-Chat: Llama/Llama-2 system support, efficiency boost, and training stability improvements[2023/08]DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models[中文] [日本語][2023/06]ZeRO++: A leap in speed for LLM and chat model training with 4X less communication[English] [中文] [日本語]Extreme Speed and Scale for DL Training and InferenceDeepSpeedenables world's most powerful language models likeMT-530BandBLOOM. It is an easy-to-use deep learning optimization software suite that powers unprecedented scale and speed for both training and inference. With DeepSpeed you can:Train/Inference dense or sparse models with billions or trillions of parametersAchieve excellent system throughput and efficiently scale to thousands of GPUsTrain/Inference on resource constrained GPU systemsAchieve unprecedented low latency and high throughput for inferenceAchieve extreme compression for an unparalleled inference latency and model size reduction with low costsDeepSpeed's four innovation pillarsDeepSpeed-TrainingDeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. These innovations such as ZeRO, 3D-Parallelism, DeepSpeed-MoE, ZeRO-Infinity, etc. fall under the training pillar. Learn more:DeepSpeed-TrainingDeepSpeed-InferenceDeepSpeed brings together innovations in parallelism technology such as tensor, pipeline, expert and ZeRO-parallelism, and combines them with high performance custom inference kernels, communication optimizations and heterogeneous memory technologies to enable inference at an unprecedented scale, while achieving unparalleled latency, throughput and cost reduction. This systematic composition of system technologies for inference falls under the inference pillar. Learn more:DeepSpeed-InferenceDeepSpeed-CompressionTo further increase the inference efficiency, DeepSpeed offers easy-to-use and flexible-to-compose compression techniques for researchers and practitioners to compress their models while delivering faster speed, smaller model size, and significantly reduced compression cost. Moreover, SoTA innovations on compression like ZeroQuant and XTC are included under the compression pillar. Learn more:DeepSpeed-CompressionDeepSpeed4ScienceIn line with Microsoft's mission to solve humanity's most pressing challenges, the DeepSpeed team at Microsoft is responding to this opportunity by launching a new initiative calledDeepSpeed4Science, aiming to build unique capabilities through AI system technology innovations to help domain experts to unlock today's biggest science mysteries. Learn more:DeepSpeed4Science websiteandtutorialsDeepSpeed Software SuiteDeepSpeed LibraryTheDeepSpeedlibrary (this repository) implements and packages the innovations and technologies in DeepSpeed Training, Inference and Compression Pillars into a single easy-to-use, open-sourced repository. It allows for easy composition of multitude of features within a single training, inference or compression pipeline. The DeepSpeed Library is heavily adopted by the DL community, and has been used to enable some of the most powerful models (seeDeepSpeed Adoption).Model Implementations for Inference (MII)Model Implementations for Inference (MII)is an open-sourced repository for making low-latency and high-throughput inference accessible to all data scientists by alleviating the need to apply complex system optimization techniques themselves. Out-of-box, MII offers support for thousands of widely used DL models, optimized using DeepSpeed-Inference, that can be deployed with a few lines of code, while achieving significant latency reduction compared to their vanilla open-sourced versions.DeepSpeed on AzureDeepSpeed users are diverse and have access to different environments. We recommend to try DeepSpeed on Azure as it is the simplest and easiest method. The recommended method to try DeepSpeed on Azure is through AzureMLrecipes. The job submission and data preparation scripts have been made availablehere. For more details on how to use DeepSpeed on Azure, please follow theAzure tutorial.DeepSpeed AdoptionDeepSpeed is an important part of Microsoft’s newAI at Scaleinitiative to enable next-generation AI capabilities at scale, where you can find more
informationhere.DeepSpeed has been used to train many different large-scale models, below is a list of several examples that we are aware of (if you'd like to include your model please submit a PR):Megatron-Turing NLG (530B)Jurassic-1 (178B)BLOOM (176B)GLM (130B)xTrimoPGLM (100B)YaLM (100B)GPT-NeoX (20B)AlexaTM (20B)Turing NLG (17B)METRO-LM (5.4B)DeepSpeed has been integrated with several different popular open-source DL frameworks such as:DocumentationTransformers with DeepSpeedAccelerate with DeepSpeedLightning with DeepSpeedMosaicML with DeepSpeedDetermined with DeepSpeedMMEngine with DeepSpeedBuild Pipeline StatusDescriptionStatusNVIDIAAMDCPUPyTorch NightlyIntegrationsMiscInstallationThe quickest way to get started with DeepSpeed is via pip, this will install
the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA
versions. DeepSpeed includes several C++/CUDA extensions that we commonly refer
to as our 'ops'.  By default, all of these extensions/ops will be built
just-in-time (JIT) usingtorch's JIT C++ extension loader that relies on
ninjato build and
dynamically link them at runtime.RequirementsPyTorchmust be installedbeforeinstalling DeepSpeed.For full feature support we recommend a version of PyTorch that is >= 1.9 and ideally the latest PyTorch stable release.A CUDA or ROCm compiler such asnvccorhipccused to compile C++/CUDA/HIP extensions.Specific GPUs we develop and test against are listed below, this doesn't mean your GPU will not work if it doesn't fall into this category it's just DeepSpeed is most well tested on the following:NVIDIA: Pascal, Volta, Ampere, and Hopper architecturesAMD: MI100 and MI200PyPIWe regularly push releases toPyPIand encourage users to install from there in most cases.pipinstalldeepspeedAfter installation, you can validate your install and see which extensions/ops
your machine is compatible with via the DeepSpeed environment report.ds_reportIf you would like to pre-install any of the DeepSpeed extensions/ops (instead
of JIT compiling) or install pre-compiled ops via PyPI please see ouradvanced
installation instructions.WindowsWindows support is partially supported with DeepSpeed. On Windows you can build wheel with following steps, currently only inference mode is supported.Install pytorch, such as pytorch 1.8 + cuda 11.1Install visual cpp build tools, such as VS2019 C++ x64/x86 build toolsLaunch cmd console with Administrator privilege for creating required symlink foldersRunpython setup.py bdist_wheelto build wheel indistfolderFeaturesPlease checkoutDeepSpeed-Training,DeepSpeed-InferenceandDeepSpeed-Compressionpages for full set of features offered along each of these three pillars.Further ReadingAll DeepSpeed documentation, tutorials, and blogs can be found on our website:deepspeed.aiDescriptionGetting StartedFirst steps with DeepSpeedDeepSpeed JSON ConfigurationConfiguring DeepSpeedAPI DocumentationGenerated DeepSpeed API documentationTutorialsTutorialsBlogsBlogsContributingDeepSpeed welcomes your contributions! Please see ourcontributingguide for more details on formatting, testing,
etc.Thanks so much to all of our amazing contributors!Contributor License AgreementThis project welcomes contributions and suggestions. Most contributions require you to
agree to a Contributor License Agreement (CLA) declaring that you have the right to, and
actually do, grant us the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need
to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply
follow the instructions provided by the bot. You will only need to do this once across
all repos using our CLA.Code of ConductThis project has adopted theMicrosoft Open Source Code of
Conduct. For more information see theCode of Conduct FAQor contactopencode@microsoft.comwith any additional questions or comments.PublicationsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He. (2019) ZeRO: memory optimizations toward training trillion parameter models.arXiv:1910.02054andIn Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20).Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. (2020) DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters.In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial).Minjia Zhang, Yuxiong He. (2020) Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping.arXiv:2010.13369andNeurIPS 2020.Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He. (2021) ZeRO-Offload: Democratizing Billion-Scale Model Training.arXiv:2101.06840andUSENIX ATC 2021.[paper][slides][blog]Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He. (2021) 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed.arXiv:2102.02888andICML 2021.Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He. (2021) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning.arXiv:2104.07857andSC 2021.[paper][slides][blog]Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He. (2021) 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed.arXiv:2104.06069andHiPC 2022.Conglong Li, Minjia Zhang, Yuxiong He. (2021) The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models.arXiv:2108.06084andNeurIPS 2022.Yucheng Lu, Conglong Li, Minjia Zhang, Christopher De Sa, Yuxiong He. (2022) Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam.arXiv:2202.06009.Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He. (2022) DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI ScalearXiv:2201.05596andICML 2022.[pdf][slides][blog]Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro. (2022) Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language ModelarXiv:2201.11990.Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, Yuxiong He. (2022) Extreme Compression for Pre-trained Transformers Made Simple and Efficient.arXiv:2206.01859andNeurIPS 2022.Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He. (2022) ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers.arXiv:2206.01861andNeurIPS 2022[slides][blog]Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He. (2022) DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale.arXiv:2207.00032andSC 2022.[paper][slides][blog]Zhewei Yao, Xiaoxia Wu, Conglong Li, Connor Holmes, Minjia Zhang, Cheng Li, Yuxiong He. (2022) Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers.arXiv:2211.11586.Conglong Li, Zhewei Yao, Xiaoxia Wu, Minjia Zhang, Yuxiong He. (2022) DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing.arXiv:2212.03597ENLSP2023 Workshop at NeurIPS2023Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He. (2023) Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases.arXiv:2301.12017andICML2023.Syed Zawad, Cheng Li, Zhewei Yao, Elton Zheng, Yuxiong He, Feng Yan. (2023) DySR: Adaptive Super-Resolution via Algorithm and System Co-design.ICLR:2023.Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, Yuxiong He. (2023) Scaling Vision-Language Models with Sparse Mixture of Experts.arXiv:2303.07226andFinding at EMNLP2023.Quentin Anthony, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He, Aamir Shafi, Mustafa Abduljabbar, Hari Subramoni, Dhabaleswar Panda. (2023) MCR-DL: Mix-and-Match Communication Runtime for Deep LearningarXiv:2303.08374and will appear at IPDPS 2023.Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele. (2023) A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts TrainingarXiv:2303.06318and will appear at ICS 2023.Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He. (2023) ZeRO++: Extremely Efficient Collective Communication for Giant Model TrainingarXiv:2306.10209andML for Sys Workshop at NeurIPS2023[blog]Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, Yuxiong He. (2023) ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank CompensationarXiv:2303.08302andENLSP2023 Workshop at NeurIPS2023[slides]Pareesa Ameneh Golnari, Zhewei Yao, Yuxiong He. (2023) Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important?arXiv:2305.09847Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, Yuxiong He. (2023) DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All ScalesarXiv:2308.01320.Xiaoxia Wu, Zhewei Yao, Yuxiong He. (2023) ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point FormatsarXiv:2307.09782andENLSP2023 Workshop at NeurIPS2023[slides]Zhewei Yao, Xiaoxia Wu, Conglong Li, Minjia Zhang, Heyang Qin, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He. (2023) DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal AttentionarXiv:2309.14327Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang, Conglong Li, Shiyang Chen, Chengming Zhang, Masahiro Tanaka, Xiaoxia Wu, Jeff Rasley, Ammar Ahmad Awan, Connor Holmes, Martin Cai, Adam Ghanem, Zhongzhu Zhou, Yuxiong He, et al. (2023) DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System TechnologiesarXiv:2310.04610[blog]Zhewei Yao, Reza Yazdani Aminabadi, Stephen Youn, Xiaoxia Wu, Elton Zheng, Yuxiong He. (2023) ZeroQuant-HERO: Hardware-Enhanced Robust Optimized Post-Training Quantization Framework for W8A8 TransformersarXiv:2310.17723VideosDeepSpeed KDD 2020 TutorialOverviewZeRO + large model training17B T-NLG demoFastest BERT training + RScan tuningDeepSpeed hands on deep dive:part 1,part 2,part 3FAQMicrosoft Research WebinarRegistration is free and all videos are available on-demand.ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed.DeepSpeed on AzureMLLarge Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference[slides]Community TutorialsDeepSpeed: All the tricks to scale to gigantic models (Mark Saroufim)Turing-NLG, DeepSpeed and the ZeRO optimizer (Yannic Kilcher)Ultimate Guide To Scaling ML Models (The AI Epiphany)