Distributed Training Across the World

Traditional synchronous distributed training is performed inside a cluster since it requires high bandwidth and low latency network ( e.g . 25Gb Ethernet or Infiniband ) . However , in many application scenarios , training data are often distributed across many geographic locations , where physical distance is long and latency is high . Traditional synchronous distributed training can not scale well under such limited network conditions . In this work , we aim to scale distributed learning under high-latency network . To achieve this , we propose Delayed and Temporally Sparse ( DTS ) update that enables synchronous training to tolerate extreme network conditions without compromising accuracy . We benchmark our algorithms on servers deployed across three continents in the world : London ( Europe ) , Tokyo ( Asia ) , Oregon ( North America ) and Ohio ( North America ) . Under such challenging settings , DTS achieves 90× speedup over traditional methods without loss of accuracy on ImageNet . 1 INTRODUCTION . Deep neural networks have demonstrated much success in solving large-scale machine learning problems ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ) . However , training deep neural networks may take days or even weeks to converge . In order to enable training in a reasonable time , distributed training is an important technique and gains increasing attention ( Li et al. , 2014 ; Dean et al. , 2012 ; Recht et al. , 2011 ; Goyal et al. , 2017 ; Jia et al. , 2018 ) . To maintain a good scalability , a low latency and high bandwidth network is essential for most modern distributed systems . Existing frameworks ( Chen et al. , 2015 ; Xing et al. , 2015 ; Moritz et al. , 2015 ; Abadi et al. , 2015 ; Akiba et al. , 2017 ; Paszke et al. , 2017 ; Sergeev & Del Balso , 2018 ) all require high-end network infrastructure such as 25Gbps Ethernet or Infiniband where bandwidth is as large as 10 to 100 Gbps and latency is as small as 1 us . 16x4 = 64 cards , batch size 32 latency ( ms ) speed SSGD DTS ECD-PSGD ASGD FedAvg 0 151 0.79 0.7935 0.7994 1 141 0.73 0.7907 0.7829 5 130 0.68 0.7878 0.7677 10 119 0.62 0.7816 0.7607 50 62.2 0.32 0.7866 0.70 0.68 0.7441 100 34.2 0.18 0.7853 0.7370 500 7.46 0.04 0.7812 0.5179 1000 3.73 0.02 0.7721 0.3764 16x4 = 64 cards , batch size 32-1 latency ( ms ) spee SSGD DTS ECD-PSGD 0 151 0.79 0.7935 1 141 0.73 0.7907 5 130 0.68 0.7878 10 119 0.62 0.7816 50 62.2 0.32 0.7866 0.70 100 34.2 0.18 0.7853 500 7.46 0.04 0.7812 1000 3.73 0.02 0.7721 Sc al ab ili ty 0.00 0.23 0.45 0.68 0.90 Network Latency ( ms ) 0 1 5 10 50 100 500 1000 SSGD Bandwidth is easy to increase ( e.g . stacking hardware ) but latency is hard to improve ( physical limits ) . For example , if we have two servers located at Shanghai and Boston respectively , even at the speed of light and direct air distance , it still takes 78ms∗ to send and receive a packet . In real world scenario , the latency can be only worse ( around 700ms ) because indirect routing between internet service providers ( ISP ) and queuing delay in switches . Such high latency cause severe scalability† issue for distributed training . As shown in Fig . 1 , traditional distributed training algorithm scales poorly under such large latency . In many scenarios , the training data involv s privacysensitive infor ation , such as personal medical history ( Jochems et al. , 2016 ; 2017 ) and keyboard input history ( McMahan et al. , 2 16 ; Konen et al. , 2016 ; Bonawitz et al. , 2019 ) , thus can not be centralized to a data center due to security and legacy concerns ( GDPR , 2016 ) . When datasets are distributed across many ∗11 , 725km × 2/ ( 3× 108m/s ) = 78.16ms . Information collected from Google Maps . †If a system can achieve M times speedup on N machines , the scalability is defined as M/N . Under review as a conference paper at ICLR 2020 Latency Bandwidth Scalability Inside 3 25600 819.2 Across 158750 29 8.192 London TokyoOregon Ohio 102ms 210ms 97ms 70ms Table 1-1 Latency Scalability Across the world 3 2274 Inside a cluster 479000 23.32 Latency Speed 23 479,000 2,274 3 Inside a cluster Across the world 159,666x longer 98x worse µs µs img/s img/s Figure 2 : Network conditions across different continents ( Left ) and the comparison between connections inside a cluster ( Right ) . Different from training inside a data center , long-distance distributed training suffers from high latency , which proposes a severe challenge to scale across the world . locations across public clouds , private clusters and even edge devices , it is impossible to setup a low latency network under long-distance connections and thereby hurts the scalability of training . In this work , we enable scalable distributed training across different geographical locations with long distance and high latency network connection . We propose Delayed Update to tolerate latency by putting off the synchronization barrier to a later iteration ; we also propose Temporally Sparse Update to amortize the latency and alleviate congestion by reducing the synchronization frequency . To ensure no loss of accuracy , we design a novel error compensation to overcome the staleness for both vanilla and momentum SGD . We focus on the widely adopted synchronous update using data parallelism . As shown in Fig . 1 , DTS can maintain high scalability even when latency is as high as 1000ms . This result is also better than existing state of art technologies such as ECD-PSGD ( results copied directly from their original paper ( Tang et al. , 2018a ) ) We benchmark our DTS under a challenging settings : training a deep neural network on four AWS P3 instances located in different continents of the world ( Fig . 2 ) . The measured latency is as large as 277ms ( compared to internal latency 2us ) . In this case , the naive distributed synchronous SGD ( SSGD ) can only achieve a poor scalability of 0.008 . It means in this setting , distributed training with 100 servers is even slower than single machine ( 0.8 v.s . 1.0 ) . Meanwhile DTS achieves scalability of 0.72 without compromising the accuracy . In conclusion , our contributions are listed below : • We propose delayed update to tolerate the latency and temporally sparse update to amortize the latency . While preserving the accuracy , delayed update tolerates up to 6 seconds latency and temporally sparse update reduced the traffic congestion by 20× . • We theoretically justify the convergence rate of our proposed algorithms . We show that both algorithms can be as fast original SGD while scaling well under high latency . • With servers and data distributed in four different countries across the world , we can train ResNet-50 on ImageNet with scalability . To our best knowledge , DTS is the first work that can achieve scalable synchronous distributed training under such high latency . 2 RELATED WORK . Distributed Learning becomes ever more important as the sizes of both datasets and models increase . Many studies have been made to explore the efficiency , both at the algorithm level ( Li et al. , 2014 ; Dean et al. , 2012 ; Recht et al. , 2011 ; Goyal et al. , 2017 ; Jia et al. , 2018 ) and at the framework level ( Chen et al. , 2015 ; Akiba et al. , 2017 ; Abadi et al. , 2015 ; Paszke et al. , 2017 ; Sergeev & Del Balso , 2018 ) . In most of distributed algorithms , each node performs computation and exchange updates through network . A key component to improve scalability is to reduce the communication-tocomputation ratio . The communication cost is determined by latency and bandwidth . Conventional studies focus on reducing bandwidth requirements as the latency of internal traffic is usually low . Gradient quantization / compression has been proposed to reduce the data to be transferred . Onebit gradient gradient ( Seide et al. , 2014 ) achieves 10× speedup using 20 GPUs on text-to-speech task . QSGD ( Alistarh et al. , 2016 ) and Terngrad ( Wen et al. , 2017 ) further improves the trade-off between accuracy and gradient precision . But the theoretical limit of the quantization can not go beyond 32 . To overcome the limitation , gradient sparsification using predefined static threshold ( Strom , 2015 ; Aji & Under review as a conference paper at ICLR 2020 Heafield , 2017 ) and dynamic compression ratio ( Chen et al. , 2018 ) demonstrate that 99 % gradients can be pruned with negligible degradation on model performance . ( Sattler et al. , 2018 ) combines both quantization and compression to push the ratio to a new level . DGC ( Lin et al. , 2017 ) and DoubleSequeeze ( Tang et al. , 2019 ) explore how to compensate the error to preserve the accuracy better . The convergence of quantization and compression is also discussed in ( Tang et al. , 2019 ; Jiang & Agrawal , 2018 ; Alistarh et al. , 2018 ) . However , latency remains an issue , especially when servers are not in the same physical location . Learning on decentralized data has become increasingly popular recently as the growing awareness of data privacy ( GDPR , 2016 ) . For example , Federated Learning ( Google , 2017 ) aim to jointly train a model without centralizing the data and have been used to train models for medical treatments across multiple hospitals ( Jochems et al. , 2016 ) , analyze patient survival situations from various countries ( Jochems et al. , 2017 ) and build predictive keyboards to improve typing experience ( McMahan et al. , 2016 ; Google , 2017 ; Bonawitz et al. , 2019 ) . However , existing federated learning works do not scale well under high latency network . An orthogonal exploration is Decentralized Training where only partial synchronization is performed in each update , such as AD-PSGD ( Lian et al. , 2017 ) and D2 ( Tang et al. , 2018b ) . None of them has been evaluated on large learning tasks yet . Asynchronous SGD ( ASGD ) is derived from ( Tsitsiklis et al. , 1986 ) and has advantages in unstable latency and fault tolerance . Different from synchronous SGD ( SSGD ) , ASGD relaxes synchronization by allowing training on inconsistent models and powers many successful applications such as HOGWILD ! ( Recht et al. , 2011 ) , BUCKWILD ! ( De Sa et al. , 2015 ) and dist-belief ( Dean et al. , 2012 ) . However , most of them are implemented through parameter server ( Dean et al. , 2012 ; Chilimbi et al. , 2014 ; Li et al. , 2014 ) and leads to problems like communication congestion and resource congestion when scaling up . Moreover , training on inconsistent models leads to different behaviors compared to training on a single device . Though there are studies discussing the convergences ( Recht et al. , 2011 ; De Sa et al. , 2015 ) and showing that ASGD converges as good as SSGD ( Lian et al. , 2015 ) , synchronous SGD ( SSGD ) is usually preferred in practice ( Goyal et al. , 2017 ; Sun et al. , 2019 ) because it has consistent behaviors when increasing the number of machines and therefore is easy to develop and deploy .

Summary: The paper presents an approach - Delayed and Temporally Sparse Update (DTS) - to do distributed training on nodes that are very distant from each other in terms of latency. The approach relies on delayed updates to tolerate latencies and temporally sparse updates to reduce traffic. The approach is implemented in a synchronous stochastic gradient decent (SGD) scheme.