###### Abstract  
The recent unprecedented success of deep learning (DL) in various fields is underlied by its use of large-scale data and models. Training a large-scale deep neural network (DNN) model with large-scale data, however, is time-consuming. To speed up the training of massive DNN models, data-parallel distributed training based on the parameter server (PS) has been widely applied. In general, a synchronous PS-based training suffers from the synchronization overhead, especially in heterogeneous environments. To reduce the synchronization overhead, asynchronous PS-based training employs the asynchronous communication between PS and workers so that PS processes the request of each worker independently without waiting. Despite the performance improvement of asynchronous training, however, it inevitably incurs the difference among the local models of workers, where such a difference among workers may cause slower model convergence. Fro addressing this problem, in this work, we propose a novel asynchronous PS-based training algorithm, SHAT that considers (1) the scale of distributed training and (2) the heterogeneity among workers for successfully reducing the difference among the local models of workers. The extensive empirical evaluation demonstrates that (1) the model trained by SHAT converges to the higher accuracy up to 5.22% than state-of-the-art algorithms, and (2) the model convergence of SHAT is robust under various heterogeneous environments. distributed deep learning; data parallelism; PS-based distributed training; heterogeneous environments +
Footnote †: journal: Medical Computing  
that all workers always have the exactly same models across the entire training, which helps to speed up the model convergence. While the synchronous PS-based distributed training [12; 13], in general, has a good convergence rate thanks to its model synchronization across all workers, the synchronization overhead becomes larger as the number of workers and the size of a model increase, which may degrade the training performance. According to [14; 15], the synchronization overhead is more than 70% of the entire training when the VGG-16 model training with more than 8 workers. This performance issue can be more serious in heterogeneous environments, where there are workers with different training speeds [16; 17]. To reduce the synchronization overhead, a number of distributed training algorithms adopting asynchronous communication have been studied [6; 7; 8; 10; 11; 18; 19]. These algorithms aim to improve the training performance by reducing the synchronization overhead while maintaining the degradation of the model accuracy as small as possible. In general, the model accuracy and training performance are in a trade-off relationship; that is, reducing the overhead by asynchronous communication improves the training performance, but it inevitably results in a significant difference among the local models of workers. Such a difference can delay the convergence of the global model since the gradients computed from the local models of different workers may interfere with each other [7; 11; 14]. In addition, the difference among the local models of workers tends to become larger as the scale of distributed training increases (i.e., a larger number of workers), especially in heterogeneous environments. Therefore, in asynchronous distributed training, it is important to reduce the difference among the local models of workers by considering the scale of distributed training and the heterogeneity among workers. However, existing asynchronous training algorithms [6; 7; 8; 9; 10] do not consider the scale of distributed training and the heterogeneity among workers when they update the local model of each worker. By these limitations, in large-scale distributed heterogeneous environments, existing asynchronous training algorithms often have the problem of the delayed model convergence [11; 14].