Package: byteps
BytePSBytePS is a high performance and general distributed training framework. It supports TensorFlow, Keras, PyTorch, and MXNet, and can run on either TCP or RDMA network.BytePS outperforms existing open-sourced distributed training frameworks by a large margin. For example, on BERT-large training, BytePS can achieve ~90% scaling efficiency with 256 GPUs (see below), which is much higher thanHorovod+NCCL. In certain scenarios, BytePS can double the training speed compared with Horovod+NCCL.NewsBytePS paperhas been accepted to OSDI'20. The code to reproduce the end-to-end evaluation is availablehere.Supportgradient compression.v0.2.4Fix compatibility issue with tf2 + standalone kerasAdd support for tensorflow.kerasImprove robustness of broadcastv0.2.3Add DistributedDataParallel module for PyTorchFix the problem of different CPU tensor using the same nameAdd skip_synchronize api for PyTorchAdd the option for lazy/non-lazy initv0.2.0Largely improve RDMA performance by enforcing page aligned memory.Add IPC support for RDMA. Now support colocating servers and workers without sacrificing much performance.Fix a hanging bug in BytePS server.Fix RDMA-related segmentation fault problem during fork() (e.g., used by PyTorch data loader).New feature: Enable mixing use of colocate and non-colocate servers, along with a smart tensor allocation strategy.New feature: Addbpslaunchas the command to launch tasks.Add support for pip install:pip3 install bytepsPerformanceWe show our experiment on BERT-large training, which is based on GluonNLP toolkit. The model uses mixed precision.We use Tesla V100 32GB GPUs and set batch size equal to 64 per GPU. Each machine has 8 V100 GPUs (32GB memory) with NVLink-enabled. Machines are inter-connected with 100 Gbps RDMA network. This is the same hardware setup you can get onAWS.BytePS achieves ~90% scaling efficiency for BERT-large with 256 GPUs. The code is availablehere. As a comparison, Horovod+NCCL has only ~70% scaling efficiency even after expert parameter tunning.With slower network, BytePS offers even more performance advantages -- up to 2x of Horovod+NCCL. You can find more evaluation results atperformance.md.Goodbye MPI, Hello CloudHow can BytePS outperform Horovod by so much? One of the main reasons is that BytePS is designed for cloud and shared clusters, and throws away MPI.MPI was born in the HPC world and is good for a cluster built with homogeneous hardware and for running a single job. However, cloud (or in-house shared clusters) is different.This leads us to rethink the best communication strategy, as explained inhere. In short, BytePS only uses NCCL inside a machine, while re-implements the inter-machine communication.BytePS also incorporates many acceleration techniques such as hierarchical strategy, pipelining, tensor partitioning, NUMA-aware local communication, priority-based scheduling, etc.Quick StartWe provide astep-by-step tutorialfor you to run benchmark training tasks. The simplest way to start is to use ourdocker images. Refer toDocumentationsfor how tolaunch distributed jobsand moredetailed configurations. After you can start BytePS, readbest practiceto get the best performance.Below, we explain how to install BytePS by yourself. There are two options.Install by pippip3 install bytepsBuild from source codeYou can try out the latest features by directly installing from master branch:git clone --recursive https://github.com/bytedance/byteps
cd byteps
python3 setup.py installNotes for above two options:BytePS assumes that you have already installed one or more of the following frameworks: TensorFlow / PyTorch / MXNet.BytePS depends on CUDA and NCCL. You should specify the NCCL path withexport BYTEPS_NCCL_HOME=/path/to/nccl. By default it points to/usr/local/nccl.The installation requires gcc>=4.9. If you are working on CentOS/Redhat and have gcc<4.9, you can tryyum install devtoolset-7before everything else. In general, we recommend using gcc 4.9 for best compatibility (how to pin gcc).RDMA support: During setup, the script will automatically detect the RDMA header file. If you want to use RDMA, make sure your RDMA environment has been properly installed and tested before install (install on Ubuntu-18.04).ExamplesBasic examples are provided under theexamplefolder.To reproduce the end-to-end evaluation in our OSDI'20 paper, find the code at thisrepo.Use BytePS in Your CodeThough being totally different at its core, BytePS is highly compatible with Horovod interfaces (Thank you, Horovod community!). We chose Horovod interfaces in order to minimize your efforts for testing BytePS.If your tasks only rely on Horovod's allreduce and broadcast, you should be able to switch to BytePS in 1 minute. Simply replaceimport horovod.tensorflow as hvdbyimport byteps.tensorflow as bps, and then replace allhvdin your code bybps. If your code invokeshvd.allreducedirectly, you should also replace it bybps.push_pull.Many of our examples were copied from Horovod and modified in this way. For instance, compare the MNIST example forBytePSandHorovod.BytePS also supports other native APIs, e.g., PyTorch Distributed Data Parallel and TensorFlow Mirrored Strategy. SeeDistributedDataParallel.mdandMirroredStrategy.mdfor usage.Limitations and Future PlansBytePS does not support pure CPU training for now. One reason is that thecheap PS assumptionof BytePS do not hold for CPU training. Consequently, you need CUDA and NCCL to build and run BytePS.We would like to have below features, and there is no fundamental difficulty to implement them in BytePS architecture. However, they are not implemented yet:Sparse model trainingFault-toleranceStraggler-mitigationPublications[OSDI'20] "A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters". Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, Chuanxiong Guo.[SOSP'19] "A Generic Communication Scheduler for Distributed DNN Training Acceleration". Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, Chuanxiong Guo. (Code is atbytescheduler branch)