CrossBow: Scaling Deep Learning on Multi-GPU Servers

With the widespread availability of servers with 4 or more GPUs, scalability in terms of the number of GPUs in a server when training deep learning models becomes a paramount concern. Systems such as TensorFlow and MXNet train using synchronous stochastic gradient descent—an input batch is partitioned across the GPUs, each computing a partial gradient. The gradients are then combined to update the model parameters before proceeding to the next batch. For many deep learning models, this introduces a scalability challenge: to keep multiple GPUs fully utilised, the batch size must be sufficiently large, but a large batch size slows down model convergence due to the less frequent model updates, thus prolonging the time to reach a desired level of accuracy. This paper introduces CrossBow, a new single-server multiGPU deep learning system that avoids the above trade-off. CrossBow trains multiple model replicas concurrently on each GPU, thereby avoiding under-utilisation of GPUs evenwhen the preferred batch size is small. For this, CrossBow must (i) decide on an appropriate number of model replicas per GPU and (ii) employ an efficient and scalable synchronisation scheme within and across GPUs. CrossBow automatically tunes the number of replicas per GPU at runtime to maximise training throughput for a given batch size. We designed a novel synchronisation scheme that eliminates dependencies among model replicas, enabling high throughput and scalability. Our experiments show that CrossBow outperforms TensorFlow on a 4-GPU server by 2.5× with ResNet-32. 1 BATCH SIZES IN DEEP-LEARNING SYSTEMS Large-scale deep learning models are prominent in many application fields, including computer vision [5, 9, 12], speech recognition [14], and natural language processing [13]. Models are trained using labelled data samples to make accurate predictions. When training with stochastic gradient descent (SGD), model parameters are iteratively refined based on the how well they can predict the label of each sample. A model update, termed a gradient, is typically computed over a batch of samples. The goal of training is to reach a desired level of accuracy as fast as possible. Training can effectively exploit the parallelism of a GPU, and multi-GPU servers have become widely available: public cloud providers offer VMs with up to 16 GPUs [2], and a 10-GPU server costs less than $40,000. Deep learning systems such as TensorFlow [1], MXNet [3], CNTK [10], and Caffe [6] must therefore scale well with multiple GPUs in a single server. As the number and capability of GPUs in a single server increases, this becomes increasingly challenging because systems must fully utilise the parallelism of all GPUs during training without introducing bottlenecks. Current systems [1, 3] must use large batch sizes in order to scale to multiple GPUs. They commonly employ synchronous stochastic gradient descent (S-SGD) [8] to parallelise model training on a multi-GPU server. Assuming a batch size b across k GPUs, parallel training with S-SGD involves two types of computational tasks for learning and synchronisation: learning tasks on each GPU first compute a partial gradient usingb/k samples; synchronisation tasks then update the model with aggregate values before the next batch is considered. To scale S-SGD with the number of GPUs k , the batch size b must increase as well. When b increases, however, eventually statistical efficiency [15] decreases due to the less frequent model updates, resulting in a higher time-to-accuracy. Systems attempt to compensate for this effect during training: they may proportionally increase the learning rate [4], or they may adapt the batch size during training [11]. These techniques, however, do not avoid the fundamental issue that current systems couple the batch size b with the number of GPUs k , preventing the training of models with small batch using many GPUs. We ask the question how to design for a deep-learning system that supports the efficient training with small batch sizes while scaling to many GPUs. 2 CROSSBOW: MULTIPLE REPLICAS PER GPU Our design for a new deep-learning system, CrossBow, exploits the idea to train multiple model replicas concurrently on each GPU, thus fully utilising all GPU resources even with small batch sizes. For this, CrossBow overcomes two challenges: (i) it automatically selects the optimal number of model replicas per GPU (§2.1); and (ii) it frequently synchronises a large number of model replicas within and across GPUs without limiting concurrency (§2.2). 2.1 Selecting the number of model replicas When training multiple model replicas on a single GPU, the number of replicas must be chosen carefully for a given batch size: when training too few replicas, the GPU is under-utilised, wasting resources; choosing too many means that learning tasks are sequentialised partially on the GPU, leading to a slow-down. In addition, the synchronisation of many replicas reduces statistical efficiency, particularly for large models. Finding a sweet-spot is challenging because it depends on the features of the model (e.g. its parameters), the GPU (e.g. the number of cores), and the software stack used for training (e.g. the specific employed kernels). CrossBow estimates the best number of replicas per GPU based on the observed training throughput, i.e. the data volume of processed batches per second. CrossBow thus detects underand over-utilisation of the GPU during training: it observes a decrease in throughput and adapts the number of model replicas accordingly. CrossBow starts with a single replica per GPU, m=1, and increasesm as long as a significant increase in throughput is observed. The increase must be above a given threshold (empirically set to 10%) to compensate for the potential loss in statistical efficiency due to the larger number of replicas. In practice, CrossBow discovers the best number of replicas to use after a few seconds. 2.2 Synchronising model replicas Deep learning systems such as TensorFlow and MXNet adopt synchronous SGD (S-SGD) to coordinate model replicas. It requires all replicas to synchronise at the same time, creating a global barrier between successive learning tasks. This approach is ill-suited for CrossBow for two reasons: (i) since CrossBow trains multiple replicas per GPU, the overall number of replicas in the system is significantly higher, typically by a factor of 2 or more. This drastically increases the chances of stragglers, which would slow down the rest of the replicas due to the global barrier. The effect would be even more pronounced if different generations of GPUs (with different performance characteristics) were used; and (ii) the cost of synchronisingwithin a single GPU is significantly lower than across GPUs. By requiring a global synchronisation step, S-SGD does not exploit this difference. To address these shortcomings, we develop a new synchronisation technique that eliminates any global barrier and reduces dependencies among tasks to increase concurrency and, hence, training throughput. Our approach works as follows: each GPU stores a reference copy of the model locally. After a model replica has been updated by all its assigned learning tasks, the CrossBow scheduler issues a synchronisation task that computes the weighted difference between the model replica and the local reference model and applies it to the replica. Once all model replicas across all GPUs have completed their assigned learning tasks, all differences are aggregated and sent to one GPU where a new global model is generated and then copied to all other GPUs. This ensures that all reference models are consistent. As done by elastic averaging SGD (EA-SGD) [16], CrossBow uses a moving average to update the values of models, which yields faster convergence. A key property of our approach is that all replica synchronisation tasks are independent of each other. Once a replica has synchronised with its local reference model, it is ready to accept the next learning task without waiting for other synchronisation tasks to complete. Dependencies between successive synchronisation tasks are set on a per replica basis. This fine-grained concurrency control has several benefits: (i) by avoiding a global synchronisation step, it allows CrossBow to scale to a large number of GPUs and model replicas; (ii) it is not impacted by stragglers (e.g. due to heterogeneous hardware). CrossBow’s task scheduler maintains a pool of model replicas and assigns learning tasks opportunistically as a new model replica becomes available. This ensures that model replicas executing on a faster GPU receive more tasks; and (iii) by decoupling intraand inter-device synchronisation, it is possible to optimise performance further by taking advantage of locality. In our current implementation, we first aggregate the differences for all model replicas hosted on the same GPU before propagating them to the GPU responsible for generating the new model, thus reducing GPU-to-GPU transfers. TensorFlow and Caffe2 [4] support several inter-GPU synchronisation mechanisms to aggregate gradients and broadcast models. While they overlap data movement with gradient computation, as each operation is handled by distinct parts of the GPU runtime (the copy and compute engines), CrossBow enables efficient multiplexing of tasks on the compute engine. DimmWitted [15] explores the problem of model replica synchronisation on multi-core CPUs. 0 100 200 300 400 1 2 4 T im e t o a c c u ra c y ( s e c ) Number of GPUs Crossbow