Delay-Tolerant Local SGD for Efficient Distributed Training

1 INTRODUCTION Data-parallel synchronous SGD is currently the workhorse algorithm for large-scale distributed deep learning tasks with many workers ( e.g . GPUs ) , where each worker calculates the stochastic gradient on local data and synchronizes with the other workers in one training iteration ( Goyal et al. , 2017 ; You et al. , 2017 ; Huo et al. , 2020 ) . However , high communication overheads make it inefficient to train large deep neural networks ( DNNs ) with a large number of workers . Generally speaking , the communication overheads come in two forms : 1 ) high communication delay due to the unstable network or a large number of communication hops , and 2 ) large communication budget caused by the large size of the DNN models with limited network bandwidth . Although communication delay is not a prominent problem for the data center environment , it can severely degrade training efficiency in practical scenarios , e.g . when the workers are geo-distributed or placed under different networks ( Ethernet , cellular networks , Wi-Fi , etc . ) in federated learning ( Konečnỳ et al. , 2016 ) . Existing works to address the communication inefficiency of synchronous SGD can be roughly classified into three categories : 1 ) pipelining ( Pipe-SGD ( Li et al. , 2018 ) ) ; 2 ) gradient compression ( Aji & Heafield , 2017 ; Stich et al. , 2018 ; Alistarh et al. , 2018 ; Yu et al. , 2018 ; Vogels et al. , 2019 ) ; and 3 ) periodic averaging ( also known as Local SGD ) ( Stich , 2019 ; Lin et al. , 2018a ) . In pipelining , the model update uses stale information such that the next iteration does not wait for the synchronization of information in the current iteration to update the model . As the synchronization barrier is removed , pipelining can overlap computation with communication to achieve delay tolerance . Gradient compression reduces the amount of data transferred in each iteration by condensing the gradient with a compressor C ( · ) . Representative methods include scalar quantization ( Alistarh et al. , 2017 ; Wen et al. , 2017 ; Bernstein et al. , 2018 ) , gradient sparsification ( Aji & Heafield , 2017 ; Stich et al. , 2018 ; Alistarh et al. , 2018 ) , and vector quantization ( Yu et al. , 2018 ; Vogels et al. , 2019 ) . Periodic averaging reduces the frequency of communication by synchronizing the workers every p ( larger than 1 ) iterations . Periodic averaging is also shown to be effective for federated learning ( McMahan et al. , 2017 ) . In summary , exiting works handle the high communication delay with pipelining and use gradient compression and periodic averaging to reduce the communication budget . However , all existing methods fail to address both . It is also unclear how the three communication-efficient techniques introduced above can be used jointly without hurting the convergence of SGD . In this paper , we propose a novel framework Overlap Local Computation with Compressed Communication ( i.e. , OLCO3 ) to make distributed training both delay-tolerant AND communication efficient by enabling and improving the combination of the above three communicationefficient techniques . In Table 1 , we compare OLCO3 with the aforementioned works and two succeeding state-of-the-art delay-tolerant methods CoCoD-SGD ( Shen et al. , 2019 ) and OverlapLocalSGD ( Wang et al. , 2020 ) . Under the periodic averaging framework , we use p to denote the number of local SGD iterations per communication round , and s to denote the number of communication rounds that the information used in the model update has been outdated for . Let the computation time of one SGD iteration be Tcomput , then we can pipeline the communication and the computation when the communication delay time is less than sp · Tcomput . For simplicity , we define the delay tolerance of a method as T = sp . Local SGD has to use up-to-date information for the model update ( s = 0 , p ≥ 1 , T = sp = 0 ) . CoCoD-SGD and OverlapLocalSGD combine pipelining and periodic averaging by using stale results from last communication round ( s = 1 , p ≥ 1 , T = sp = p ) , while our OLCO3 supports various staleness ( s ≥ 1 , p ≥ 1 , T = sp ) and all other features in Table 1 . The main contributions of this paper are summarized as follows : • We propose the novel OLCO3 method , which achieves extreme communication efficiency by addressing both the high communication delay and large communication budget issues . • OLCO3 introduces novel staleness compensation and compression compensation techniques . Convergence analysis shows that OLCO3 achieves the same convergence rate as SGD . • Extensive experiments on deep learning tasks show that OLCO3 significantly outperforms ex- isting delay-tolerant methods in both the communication efficiency and model accuracy . 2 BACKGROUNDS & RELATED WORKS SGD and Pipelining . In distributed training , we minimize the global loss function f ( · ) = 1 K ∑K k=1 fk ( · ) , where fk ( · ) is the local loss function at worker k ∈ [ K ] . At iteration t , vanilla synchronous SGD updates the model xt ∈ Rd with learning rate ηt via xt+1 = xt − ηt K ∑K k=1∇Fk ( xt ; ξ ( k ) t ) , where ξ ( k ) t is the stochastic sampling variable and ∇Fk ( xt ; ξ ( k ) t ) is the corresponding stochastic gradient at worker k. Throughout this paper , we assume that the stochastic gradient is an unbiased estimator by default , i.e. , E ξ ( k ) t ∇Fk ( xt ; ξ ( k ) t ) = ∇fk ( xt ) . Pipe-SGD ( Li et al. , 2018 ) parallelizes the communication and computation of SGD via pipelining . At iteration t , worker k computes stochastic gradient ∇Fk ( xt ; ξ ( k ) t ) at current model xt and communicates to get the averaged stochastic gradient 1K ∑K k=1∇Fk ( xt ; ξ ( k ) t ) . Instead of waiting the communication to finish , Pipe-SGD concurrently updates the current model with stale averaged stochastic gradient via xt+1 = xt − ηtK ∑K k=1∇Fk ( xt−s ; ξ ( k ) t−s ) . Note that Pipe-SGD is different from asynchronous SGD ( Ho et al. , 2013 ; Lian et al. , 2015 ) which computes stochastic gradient using stale model and does not parallelize the computation and communication of a worker . A problem of Pipe-SGD is that its performance deteriorates severely under high communication delay ( large s ) . Pipelining with Periodic Averaging . CoCoD-SGD ( Shen et al. , 2019 ) utilizes periodic averaging to reduce the number of communication rounds and parallelizes the local model update and global model averaging by concurrently conducting xt = 1 K K∑ k=1 x ( k ) t and x ( k ) t+p = x ( k ) t − t+p−1∑ τ=t ητ∇Fk ( x ( k ) τ ; ξ ( k ) τ ) . ( 1 ) in which x ( k ) t denotes the local model at worker k as the local models on different workers are no longer consistent in non-communicating iterations . When the operations in Eq . ( 1 ) finishes , the local model is updated via x ( k ) t+p ← xt + x ( k ) t+p− x ( k ) t and t← t+ p. CoCoD-SGD can tolerate delay up to p SGD iterations ( i.e. , one communication round in periodic averaging ) . OverlapLocalSGD ( Wang et al. , 2020 ) improves CoCoD-SGD by heuristically pulling x ( k ) t+p back to the xt after the operations in Eq . ( 1 ) via x ( k ) t+p ← ( 1 − α ) x ( k ) t+p + αxt where 0 ≤ α < 1 . The motivation is to reduce the inconsistency in the local models across workers . OverlapLocalSGD also develops a momentum variant , which maintains a slow momentum buffer for xt following SlowMo ( Wang et al. , 2019 ) . As both CoCoD-SGD and OverlapLocalSGD communicates the non-compressed local model update , they suffer from a large communication budget in each communication round . Gradient Compression . The gradient vector v ∈ Rd can be sent with a much smaller communication budget by applying a compressor C ( · ) . Specifically , Scalar quantization rounds 32-bit floatingpoint gradient components to low-precision values of only several bits . One important such algorithm is scaled SignSGD ( called SignSGD in this paper ) ( Bernstein et al. , 2018 ; Karimireddy et al. , 2019 ) which uses C ( v ) = ‖v‖1d sign ( v ) to compress v to 1 bit . Gradient sparsification only communicates large gradient components . Vector quantization uses a codebook where each code is a vector and quantizes the gradient vector as a linear combination of the vector codes . With the local error feedback technique ( Seide et al. , 2014 ; Lin et al. , 2018b ; Wu et al. , 2018 ; Karimireddy et al. , 2019 ; Zheng et al. , 2019 ) , which adds the previous compression error ( i.e. , v − C ( v ) ) to current gradient before compression , gradient compression can achieve comparable performance as full-precision training . Local error feedback also works for both one-way compression ( compress the communication from worker to sever ) ( Karimireddy et al. , 2019 ) and two-way compression ( compress the communication between worker and server ) ( Zheng et al. , 2019 ) . Challenges . Simultaneously achieving communication compression with pipelining and periodic averaging requires careful algorithm design because 1 ) pipelining introduces staleness , and 2 ) stateof-the-art vector quantization methods usually require an additional round of communication to solve the compressor C ( · ) , which is unfavorable in high communication delay scenarios . 3 THE PROPOSED FRAMEWORK : OLCO3 In this section , we will introduce our new delay-tolerant and communication-efficient training framework OLCO3 . We discuss two variants of OLCO3 : OLCO3-TC for two-way compression in masterslave communication mode , and OLCO3-VQ adopting commutative vector quantization for both the master-slave and ring all-reduce communication modes . Note that one-way compression in just a special case of OLCO3-TC and we omit it for conciseness . We use “ line x ” to refer to the x-th line of Algorithm 1 . The key differences between OLCO3-TC and OLCO3-VQ are marked in red color . 3.1 OLCO3-TC FOR TWO-WAY COMPRESSION Motivation . OLCO3-TC is presented in the green part of Algorithm 1 for efficient master-slave distributed training . Naively pipelining local computation with compressed communication will break the update rule of momentum SGD for the averaged model xt = 1K ∑K k=1 x ( k ) t , leading to non- convergence . Therefore , we consider an auxiliary variable x̃t : = 1K ∑K k=1 x ( k ) t − 1K ∑K k=1 e ( k ) t −et , where e ( k ) t is the local compression error at worker k and et is the compression error at the server . If x̃t can follow the update rule of momentum SGD , then the real trained model xt will gradually approach x̃t as the training converges because the gradient and errors e ( k ) t , et → 0 . Pipelining . For non-communicating iterations , we perform the local update following Local SGD ( line 4 ) . A communicating iteration takes place every p iterations . To pipeline the communication Algorithm 1 Overlap Local Computation with Compressed Communication ( OLCO3 ) on worker k ∈ [ K ] . Green part : OLCO3-TC ; Yellow part : OLCO3-VQ . Best view in color . 1 : Input : period p ≥ 1 , staleness s ≥ 0 , number of iterations T , number of workers K , learning rate { ηt } T−1t=0 , and compression scheme C ( · ) . 2 : Initialize : Local model x ( k ) 0 = x0 , local error e ( k ) 0 = 0 , server error e0 = 0 , local momentum buffer m ( k ) 0 = 0 , and momentum constant 0 < µ < 1 . Variables with negative subscripts are 0 . 3 : for t = 0 , 1 , · · · , T − 1 do 4 : m ( k ) t+1 = µm ( k ) t +∇Fk ( x ( k ) t ; ξ ( k ) t ) , x ( k ) t+1 = x ( k ) t − ηtm ( k ) t+1 // Momentum Local SGD . 5 : if ( t+ 1 ) mod p = 0 then 6 : Maintain or reset the momentum buffer . 7 : ∆ ( k ) t+1 = x ( k ) t+1−p − x ( k ) t+1 + e ( k ) t // Compression compensation . 8 : e ( k ) t+1 = e ( k ) t+2 = · · · = e ( k ) t+p = ∆ ( k ) t+1 − C ( ∆ ( k ) t+1 ) // Compression . 9 : Invoke the communication thread in parallel which does : 10 : ( 1 ) Send C ( ∆ ( k ) t+1 ) to and receive C ( ∆t+1 ) from the server node . 11 : ( 2 ) Server : ∆t+1 = 1K ∑K k=1 C ( ∆ ( k ) t+1 ) + et ; et+1 = et+2 = · · · = et+p = ∆t+1 − C ( ∆t+1 ) . 12 : Block until C ( ∆t+1−sp ) is ready . 13 : xt+1 = xt+1−p − C ( ∆t+1−sp ) 14 : x ( k ) t+1 ← xt+1 − ∑s−1 i=0 C ( ∆ ( k ) t+1−ip ) // Staleness compensation . 15 : ∆ ( k ) t+1 = x ( k ) t+1−p − x ( k ) t+1 + e ( k ) t−sp // Compression compensation . 16 : Invoke the communication thread in parallel which does : 17 : ( 1 ) e ( k ) t+1 = e ( k ) t+2 = · · · = e ( k ) t+p = ∆ ( k ) t+1 − C ( ∆ ( k ) t+1 ) // Compression . 18 : ( 2 ) Average 1K ∑K k=1 C ( ∆ ( k ) t+1 ) by ring all-reduce or master-slave communication . 19 : Block until 1K ∑K k=1 C ( ∆ ( k ) t+1−sp ) and e ( k ) t+1−sp is ready . 20 : xt+1 = xt+1−p − 1K ∑K k=1 C ( ∆ ( k ) t+1−sp ) 21 : x ( k ) t+1 ← xt+1 − ∑s−1 i=0 ∆ ( k ) t+1−ip // Staleness compensation . 22 : end if 23 : end for 24 : Output : averaged model xT = 1K ∑K k=1 x ( k ) T and computation , we compress the local update ∆ ( k ) t+1 ( line 7 ) for efficient communication , and at the same time , try to update the model with a stale compressed global update C ( ∆t+1−sp ) ( line 13 ) that has been outdated for s communication rounds ( i.e. , the staleness is s ) . The momentum buffer can be maintained or reset to zero every p iteration ( line 6 ) . If the delay tolerance T = sp is larger than the actual communication delay , the blocking in line 12 becomes a no-op and there will be no synchronization barrier . The server compresses the sum of the compressed local updates from all workers ( line 11 ) and sends it back , making OLCO3-TC an efficient two-way compression method . Compensation . To make the update of the auxiliary variable x̃t follow momentum SGD , we propose to 1 ) compensate staleness with all compressed local updates with staleness ∈ [ 0 , s − 1 ] ( line 14 ) , which requires no communication and allows less stale local update to affect the local model , and 2 ) maintain a local error ( line 8 ) and add it to the next local update before compression ( line 7 ) to compensate the compression error . With the two compensation techniques in OLCO3-TC , Lemma 1 shows that the update rule of x̃t follows momentum SGD with averaged momentum 1K ∑K k=1 m ( k ) t . Lemma 1 . For OLCO3-TC , let x̃t : = 1K ∑K k=1 x ( k ) t − 1K ∑K k=1 e ( k ) t − et−sp , then we have x̃t = x̃t−1 − ηt−1K ∑K k=1 m ( k ) t . Note that there is a “ gradient mismatch ” problem as the local momentum m ( k ) t is computed at the local model x ( k ) t but used in the update rule of the auxiliary variable x̃t ( Karimireddy et al. , 2019 ; Xu et al. , 2020 ) . However , our analysis shows that it does not affect the convergence rate . We have also considered OLCO3 for one-way compression ( i.e. , OLCO3-OC ) as a special case of OLCO3-TC . In OLCO3-OC , the compressor at the server side is identity function and the server error et is 0 . For OLCO3-OC , the auxiliary variable x̃t also follows momentum SGD as stated in Lemma 2 . Lemma 2 . For OLCO3-OC , let x̃t : = 1K ∑K k=1 x ( k ) t − 1K ∑K k=1 e ( k ) t , then we have x̃t = x̃t−1 − ηt−1 K ∑K k=1 m ( k ) t . We can see that the delay tolerance of both OLCO3-TC and OLCO3-OC are T = sp ( s ≥ 1 , p ≥ 1 ) . They have a memory overhead of O ( sd ) for storing information with staleness ∈ [ 0 , s − 1 ] . For most compression schemes such as SignSGD , the computation complexity of C ( · ) is O ( d ) . 3.2 OLCO3-VQ FOR COMMUTATIVE VECTOR QUANTIZATION OLCO3-TC and OLCO3-OC work for compressed communication in the master-slave communication paradigm . In contrast , OLCO3-VQ ( the yellow part of Algorithm 1 ) works for both the master-slave and ring all-reduce communication paradigms . Ring all-reduce minimizes communication congestion by shifting from centralized aggregation in master-slave communication ( Yu et al. , 2018 ) . OLCO3-VQ relies on a state-of-the-art vector quantization scheme , PowerSGD ( Vogels et al. , 2019 ) , which satisfies commutability for compression , i.e. , C ( v1 ) + C ( v2 ) = C ( v1 + v2 ) . However , directly using PowerSGD breaks the delay tolerance of OLCO3 as its compressor C ( · ) needs communication and introduces synchronization barriers . Specifically , PowerSGD invokes communication across all workers to compute a transformation matrix , which is used to project the local updates to the compressed form . Pipelining with Communication-Dependent Compressor . To make OLCO3-VQ delay-tolerant , we further propose a novel compression compensation technique with the stale local error ( line 15 ) . This is in contrast to OLCO3-TC and OLCO3-OC , which use immediate compressed results to calculate the up-to-date local error . As this technique removes the dependency on immediate compressed results , we can move the whole compression and averaging process to the communication thread ( lines 17 and 18 ) . For staleness compensation , OLCO3-VQ uses all uncompressed local updates with staleness ∈ [ 0 , s−1 ] instead of compressed local updates in OLCO3-TC . With the two compensation techniques , Lemma 3 shows that for OLCO3-VQ , the auxiliary variable x̃t associated with the stale local error also follows the momentum SGD update rule . Lemma 3 . For OLCO3-VQ , let x̃t : = 1K ∑K k=1 x ( k ) t − 1K ∑K k=1 e ( k ) t−sp , then we have x̃t = x̃t−1 − ηt−1 K ∑K k=1 m ( k ) t . 4 THEORETICAL RESULTS In this section , we provide the convergence results of the OLCO3 variants for both SGD and momentum SGD maintaining momentum ( line 6 of Algorithm 1 ) with common assumptions . As OLCO3OC is a special case of OLCO3-TC , we only analyze OLCO3-TC and OLCO3-VQ . The detailed proofs of Theorems 1 , 2 , 3 , and 4 can be found in Appendix D , E , F , and G respectively . The detailed proofs of Lemma 1 , 2 , and 3 can be found in Appendix C. We use f∗ to denote the optimal loss . Assumption 1 . ( L-Lipschitz Smoothness ) Both the local ( fk ( · ) ) and global ( f ( · ) = 1K ∑K k=1 fk ( · ) ) loss functions are L-smooth , i.e. , ‖∇f ( x ) −∇f ( y ) ‖2 ≤ L‖x− y‖2 , ∀x , y ∈ Rd , ( 2 ) ‖∇fk ( x ) −∇fk ( y ) ‖2 ≤ L‖x− y‖2 , ∀k ∈ [ K ] , ∀x , y ∈ Rd . ( 3 ) Assumption 2 . ( Local Bounded Variance ) The local stochastic gradient ∇Fk ( x ; ξ ) has a bounded variance , i.e. , Eξ∼Dk‖∇Fk ( x ; ξ ) − ∇fk ( x ) ‖22 ≤ σ2 , ∀k ∈ [ K ] , ∀x ∈ Rd . Note that Eξ∼Dk∇Fk ( x ; ξ ) = ∇fk ( x ) . Assumption 3 . ( Bounded Variance across Workers ) The L2 norm of the difference of the local and global full gradient is bounded , i.e. , ‖∇fk ( x ) −∇f ( x ) ‖22 ≤ κ2 , ∀k ∈ [ K ] , ∀x ∈ Rd . κ = 0 leads to i.i.d . data distributions across workers . Assumption 4 . ( Bounded Full Gradient ) The second moment of the global full gradient is bounded , i.e. , ‖∇f ( x ) ‖22 ≤ G2 , ∀x ∈ Rd . Assumption 5 . ( Karimireddy et al. , 2019 ) The compression function C ( · ) : Rd → R is a δapproximate compressor for 0 < δ ≤ 1 if for all v ∈ Rd , ‖C ( v ) − v‖22 ≤ ( 1− δ ) ‖v‖22 . 4.1 SGD Theorem 1 . For OLCO3-VQ with vanilla SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ min { 16L ( s+1 ) p , 1 9L } , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( f ( x0 ) − f∗ ) ηT + 9ηLσ2 K + 12η2L2 ( s+ 1 ) pσ2 [ 1+ ( 4 ) 14 ( 1− δ ) δ2 ( s+ 1 ) p ] + 36η2L2 ( s+ 1 ) 2p2κ2 ( 1 + 5 ( 1− δ ) δ2 ) + 168 ( 1− δ ) δ2 η2L2 ( s+ 1 ) 2p2G2 . If we set the learning rate η = O ( K 12T− 12 ) and the communication interval p = O ( K− 34T 14 ( s + 1 ) −1 ) , the convergence rate will be O ( K− 12T− 12 ) . The O ( K− 12T− 12 ) rate is the same as synchronous SGD and Local SGD , and achieves linear speedup regarding the number of workers K. Theorem 2 . For OLCO3-TC with vanilla SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ min { 16L ( s+1 ) p , 1 9L } and let h ( δ ) = 1−δ δ2 ( 1 + 4 ( 2−δ ) δ2 ) , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( f ( x0 ) − f∗ ) ηT + 9ηLσ2 K + 12η2pσ2 ( s+ 1 + 80h ( δ ) p ) + 12η2p2κ2 ( 3 ( s+ 1 ) 2 + 80h ( δ ) ) + 960η2p2G2h ( δ ) . ( 5 ) If we set the learning rate η = O ( K 12T− 12 ) and the communication interval p = O ( K− 34T 14 ( s + 1 ) −1 ) , the convergence rate will be O ( K− 12T− 12 ) . When the data distributions across workers are i.i.d . ( i.e. , κ = 0 ) , if we choose the learning rate η = O ( K 12T− 12 ) and the communication interval p = min { O ( K− 32T 12 ( s + 1 ) −1 ) , O ( K− 34T 14 ) } ( p = O ( K− 34T 14 ) for a enough large T ) instead , the convergence rate will still be O ( K− 12T− 12 ) . Therefore , OLCO3-TC can tackle a larger communication interval p ( O ( K− 3 4T 1 4 ) ) than OLCO3VQ ( O ( K− 34T 14 ( s+ 1 ) −1 ) ) in the i.i.d . setting . But they are the same in the non-i.i.d . setting . 4.2 MOMENTUM SGD Theorem 3 . For OLCO3-VQ with Momentum SGD and under Assumptions 1 , 2 , 3 , 4 , 5 , if the learning rate η ≤ min { 1−µ√ 72L ( s+1 ) p , 1−µ9L } and let g ( µ , δ , s , p ) = 15 ( 1−µ ) 2 + 60 ( 1−δ ) ( s+1 ) 2p2 δ2 , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( 1− µ ) ( f ( x0 ) − f∗ ) ηT + 9Lησ2 ( 1− µ ) K ( 6 ) + 4η2L2 ( 1− µ ) 2 [ ( 4 ( s+ 1 ) p+ g ( µ , δ , s , p ) ) σ2 + ( 12 ( s+ 1 ) 2p2 + g ( µ , δ , s , p ) ) κ2 + g ( µ , δ , s , p ) G2 ] . Theorem 4 . For OLCO3-TC with Momentum SGD and under Assumptions 1 , 2 , 3 , 4 , 5 , if the learning rate η ≤ min { 1−µ√ 72L ( s+1 ) p , 1−µ9L } and h ( δ ) = 1−δ δ2 ( 1 + 4 ( 2−δ ) δ2 ) , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( 1− µ ) ( f ( x0 ) − f∗ ) ηT + 9Lησ2 ( 1− µ ) K + 6η2L2 ( 1− µ ) 2 [ σ2 ( 9 ( 1− µ ) 2 + 2 ( s+ 1 ) p+ 168h ( δ ) p2 ) + κ2 ( 9 ( 1− µ ) 2 + 6 ( s+ 1 ) 2p2 + 168h ( δ ) p2 ) +G2 ( 9 ( 1− µ ) 2 + 168h ( δ ) p2 ) ] . ( 7 ) The same convergence rate and communication interval p are achieved as in Section 4.1 . 5 EXPERIMENTS We compare the following methods : 1 ) Local SGD ( baseline , NO delay tolerance with T = 0 ) ; 2 ) Pipe-SGD ; 3 ) CoCoD-SGD ; 4 ) OverlapLocalSGD with hyperparameters following Wang et al . ( 2020 ) ; 5 ) OLCO3-OC with SignSGD compression ; 6 ) OLCO3-VQ with PowerSGD compressoin ; 7 ) OLCO3-TC with SignSGD compression . The momentum buffer is maintained ( line 6 of Algorithm 1 ) by default . We do not report the results of Pipe-SGD as it does not converge for the large delay tolerance T we experimented . We train ResNet-110 ( He et al. , 2016 ) with 8 workers for the CIFAR-10 ( Krizhevsky et al. , 2009 ) image classification task , and report the mean and standard deviation of the test accuracy over 3 runs in both the i.i.d . and non-i.i.d . setting . We also train ResNet-50 with 16 workers for the ImageNet ( Russakovsky et al. , 2015 ) image classification task . More detailed descriptions of the experiment configurations can be found in Appendix A.1 . Delay Tolerance with Lower Communication Budget . The training curves of ResNet-110 on CIFAR-10 and ResNet-50 on ImageNet are shown in Figure 1 . We use s = 1 because CoCoD-SGD and OverlapLocalSGD do not support s ≥ 2 . Compared with other delay-tolerant methods , the communication budget of the OLCO3 variants is significantly smaller due to compressed communication . OLCO3 is also robust to communication delay with a large T = sp . Therefore , OLCO3 features extreme communication efficiency with compressed communication , delay tolerance , and low communication frequency due to periodic averaging . Better Model Performance . The two plots in the first row of Figure 1 show that OLCO3-OC and OLCO3-TC outperforms other delay-tolerant methods and are comparable to Local SGD regarding the model accuracy . The performance of OLCO3-VQ is similar to CoCoD-SGD but inferior to OverlapLocalSGD . However , in the non-i.i.d . results reported in Table 2 , all OLCO3 variants outperform existing delay-tolerant methods in accuracy . This is in line with the theoretical results in Theorems 1 , 2 , 3 , and 4 , which show that OLCO3-TC can tackle a larger p than OLCO3-VQ in the i.i.d . setting but the two methods are similar in the non-i.i.d . setting . In the non-i.i.d . setting , all OLCO3 variants perform very close to Local SGD . On average , OLCO3-OC and OLCO3-TC improve the test accuracy of CoCoD-SGD and OverlapLocalSGD by 2.0 % and 0.8 % , respectively . OLCO3-VQ improves CoCoD-SGD and OverlapLocalSGD by 1.6 % and 0.4 % . These results empirically confirm that the staleness compensation and compression compensation techniques in OLCO3 are effective . Varying Delay Tolerance . We vary the delay tolerance T with staleness fixed at s = 1 in the left plot of Figure 2 . The goal is to check the robustness of OLCO3 to the different period p. The results show that OLCO3-OC and OLCO3-TC always outperform other delay-tolerant methods , and have more comparable performance to Local SGD . Note that both the OLCO3-OC and OLCO3TC provide a significantly smaller communication budget according to Figure 1 . OLCO3-VQ also outperforms CoCoD-SGD with a much smaller communication budget . Varying Staleness . We vary the staleness s of OLCO3 in the right plot of Figure 2 under fixed delay tolerance T . Local SGD only supports s = 0 with no delay tolerance , and CoCoD-SGD and OverlapLocalSGD only support s = 1 , so there is only one result for them in the figure . When increasing the staleness beyond 2 for OLCO3 , the deterioration of the model performance is very small , especially for OLCO3-VQ . This suggests that the staleness compensation techniques in OLCO3 are effective . The performance peaks at s = 2 because an appropriate staleness may introduce some noise that helps generalization . In comparison , we can not tune staleness s for better performance in CoCoD-SGD and OverlapLocalSGD . 6 CONCLUSION In this work , we proposed a new OLCO3 framework to achieve extreme communication efficiency with high delay tolerance and a low communication budget in distributed training . OLCO3 uses novel staleness compensation and compression compensation techniques , and the theoretical results show that it converges as fast as vanilla synchronous SGD . Experimental results show that OLCO3 significantly outperforms existing delay-tolerant methods in terms of the communication budget and model performance . REFERENCES Alham Fikri Aji and Kenneth Heafield . Sparse communication for distributed gradient descent . arXiv preprint arXiv:1704.05021 , 2017 . Dan Alistarh , Demjan Grubic , Jerry Li , Ryota Tomioka , and Milan Vojnovic . Qsgd : Communication-efficient sgd via gradient quantization and encoding . In Advances in Neural Information Processing Systems , pp . 1709–1720 , 2017 . Dan Alistarh , Torsten Hoefler , Mikael Johansson , Nikola Konstantinov , Sarit Khirirat , and Cédric Renggli . The convergence of sparsified gradient methods . In Advances in Neural Information Processing Systems , pp . 5973–5983 , 2018 . Jeremy Bernstein , Yu-Xiang Wang , Kamyar Azizzadenesheli , and Animashree Anandkumar . signSGD : Compressed optimisation for non-convex problems . In Jennifer Dy and Andreas Krause ( eds . ) , Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp . 560–569 , Stockholmsmässan , Stockholm Sweden , 10–15 Jul 2018 . PMLR . URL http : //proceedings.mlr.press/v80/ bernstein18a.html . Priya Goyal , Piotr Dollár , Ross Girshick , Pieter Noordhuis , Lukasz Wesolowski , Aapo Kyrola , Andrew Tulloch , Yangqing Jia , and Kaiming He . Accurate , large minibatch sgd : Training imagenet in 1 hour . arXiv preprint arXiv:1706.02677 , 2017 . Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image recognition . In Proceedings of the IEEE conference on computer vision and pattern recognition , pp . 770–778 , 2016 . Qirong Ho , James Cipar , Henggang Cui , Seunghak Lee , Jin Kyu Kim , Phillip B Gibbons , Garth A Gibson , Greg Ganger , and Eric P Xing . More effective distributed ml via a stale synchronous parallel parameter server . In Advances in neural information processing systems , pp . 1223–1231 , 2013 . Zhouyuan Huo , Bin Gu , and Heng Huang . Large batch training does not need warmup . arXiv preprint arXiv:2002.01576 , 2020 . Sai Praneeth Karimireddy , Quentin Rebjock , Sebastian Stich , and Martin Jaggi . Error feedback fixes signsgd and other gradient compression schemes . In International Conference on Machine Learning , pp . 3252–3261 , 2019 . Jakub Konečnỳ , H Brendan McMahan , Felix X Yu , Peter Richtárik , Ananda Theertha Suresh , and Dave Bacon . Federated learning : Strategies for improving communication efficiency . arXiv preprint arXiv:1610.05492 , 2016 . Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images . 2009 . Youjie Li , Mingchao Yu , Songze Li , Salman Avestimehr , Nam Sung Kim , and Alexander Schwing . Pipe-sgd : A decentralized pipelined sgd framework for distributed deep net training . In Advances in Neural Information Processing Systems , pp . 8045–8056 , 2018 . Xiangru Lian , Yijun Huang , Yuncheng Li , and Ji Liu . Asynchronous parallel stochastic gradient for nonconvex optimization . In Advances in Neural Information Processing Systems , pp . 2737–2745 , 2015 . Tao Lin , Sebastian U Stich , Kumar Kshitij Patel , and Martin Jaggi . Don ’ t use large mini-batches , use local sgd . arXiv preprint arXiv:1808.07217 , 2018a . Yujun Lin , Song Han , Huizi Mao , Yu Wang , and Bill Dally . Deep gradient compression : Reducing the communication bandwidth for distributed training . In International Conference on Learning Representations , 2018b . URL https : //openreview.net/forum ? id=SkhQHMW0W . Ilya Loshchilov and Frank Hutter . Sgdr : Stochastic gradient descent with warm restarts . arXiv preprint arXiv:1608.03983 , 2016 . Brendan McMahan , Eider Moore , Daniel Ramage , Seth Hampson , and Blaise Aguera y Arcas . Communication-efficient learning of deep networks from decentralized data . In Artificial Intelligence and Statistics , pp . 1273–1282 . PMLR , 2017 . Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al . Pytorch : An imperative style , highperformance deep learning library . In Advances in Neural Information Processing Systems , pp . 8024–8035 , 2019 . Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei . ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision ( IJCV ) , 115 ( 3 ) :211–252 , 2015. doi : 10.1007/s11263-015-0816-y . Frank Seide , Hao Fu , Jasha Droppo , Gang Li , and Dong Yu . 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns . In Fifteenth Annual Conference of the International Speech Communication Association , 2014 . Shuheng Shen , Linli Xu , Jingchang Liu , Xianfeng Liang , and Yifei Cheng . Faster distributed deep net training : Computation and communication decoupled stochastic gradient descent . arXiv preprint arXiv:1906.12043 , 2019 . Sebastian U. Stich . Local SGD converges fast and communicates little . In International Conference on Learning Representations , 2019 . URL https : //openreview.net/forum ? id= S1g2JnRcFX . Sebastian U Stich , Jean-Baptiste Cordonnier , and Martin Jaggi . Sparsified sgd with memory . In Advances in Neural Information Processing Systems , pp . 4447–4458 , 2018 . Thijs Vogels , Sai Praneeth Karimireddy , and Martin Jaggi . Powersgd : Practical low-rank gradient compression for distributed optimization . In Advances in Neural Information Processing Systems , pp . 14259–14268 , 2019 . Jianyu Wang , Vinayak Tantia , Nicolas Ballas , and Michael Rabbat . Slowmo : Improving communication-efficient distributed sgd with slow momentum . arXiv preprint arXiv:1910.00643 , 2019 . Jianyu Wang , Hao Liang , and Gauri Joshi . Overlap local-sgd : An algorithmic approach to hide communication delays in distributed sgd . In ICASSP 2020-2020 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ) , pp . 8871–8875 . IEEE , 2020 . Wei Wen , Cong Xu , Feng Yan , Chunpeng Wu , Yandan Wang , Yiran Chen , and Hai Li . Terngrad : Ternary gradients to reduce communication in distributed deep learning . In Advances in neural information processing systems , pp . 1509–1519 , 2017 . Jiaxiang Wu , Weidong Huang , Junzhou Huang , and Tong Zhang . Error compensated quantized sgd and its applications to large-scale distributed optimization . arXiv preprint arXiv:1806.08054 , 2018 . An Xu , Zhouyuan Huo , and Heng Huang . Training faster with compressed gradient . arXiv preprint arXiv:2008.05823 , 2020 . Yang You , Igor Gitman , and Boris Ginsburg . Scaling sgd batch size to 32k for imagenet training . arXiv preprint arXiv:1708.03888 , 6 , 2017 . Mingchao Yu , Zhifeng Lin , Krishna Narra , Songze Li , Youjie Li , Nam Sung Kim , Alexander Schwing , Murali Annavaram , and Salman Avestimehr . Gradiveq : Vector quantization for bandwidth-efficient gradient aggregation in distributed cnn training . In Advances in Neural Information Processing Systems , pp . 5123–5133 , 2018 . Shuai Zheng , Ziyue Huang , and James Kwok . Communication-efficient distributed blockwise momentum sgd with error-feedback . In Advances in Neural Information Processing Systems , pp . 11446–11456 , 2019 . A ADDITIONAL EXPERIMENTAL RESULTS A.1 EXPERIMENTAL SETTING All experiments are implemented with PyTorch ( Paszke et al. , 2019 ) and run on a cluster of Nvidia Tesla P40 GPUs . Each node is connected by 40Gbps Ethernet and equipped with 4 GPUs . CIFAR . We train the ResNet-110 ( He et al. , 2016 ) model with 8 workers on CIFAR-10 ( Krizhevsky et al. , 2009 ) image classification task . We report the mean and standard deviation metrics over 3 runs . The base learning rate is 0.4 and the total batch size is 512 . The momentum constant is 0.9 and the weight decay is 1× 10−4 . The model is trained for 200 epochs with a learning rate decay of 0.1 at epoch 100 and 150 . We linearly warm up the learning rate from 0.05 to 0.4 in the beginning 5 epochs . For OLCO3 with staleness s ∈ { 2 , 4 , 8 } , we set the base learning rate to 0.2 due to increased staleness . The rank of PowerSGD is 4 . Random cropping , random flipping , and standardization are applied as data augmentation techniques . We also train ResNet-56 to explore more combinations of s and p in Appendix A.4 with the same other settings . ImageNet . We train the ResNet-50 model with 16 workers on ImageNet ( Russakovsky et al. , 2015 ) image classification tasks . The model is trained for 120 epochs with a cosine learning rate scheduling ( Loshchilov & Hutter , 2016 ) . The base learning rate is 0.4 and the total batch size is 2048 . The momentum constant is 0.9 and the weight decay is 1 × 10−4 . We linearly warm up the learning rate from 0.025 to 0.4 in the beginning 5 epochs . The rank of PowerSGD is 50 . Random cropping , random flipping , and standardization are applied as data augmentation techniques . The Non-i.i.d . Setting . Similar to ( Wang et al. , 2020 ) , we randomly choose fraction α of the whole data , sort the data by the class , and evenly assign them to all workers in order . For the rest fraction ( 1− α ) of the whole data , we randomly and evenly distribute them to all workers ( Figure 3 ) . When 0 < α ≤ 1 is large , the data distribution across workers is non-i.i.d and highly skewed . When α = 0 , it becomes i.i.d . data distribution across workers . In our non-i.i.d . experiments , we choose α = 0.8 . A.2 TRAINING CURVE A.3 TEST ACCURACY Again , Figure 6 empirically confirms the theoretical results in Theorems 1 , 2 , 3 , and 4 that OLCO3TC can handle a larger period p than OLCO3 and that this gap increases with the staleness s in the i.i.d . setting . Note that in the right plot of Figure 2 , the gap between OLCO3-TC and OLCO3-VQ does not increase with s because the period p is decreasing ( the delay tolerance T = sp is fixed ) . B ASSUMPTIONS Assumption 1 . ( L-Lipschitz Smoothness ) Both the local ( fk ( · ) ) and global ( f ( · ) = 1K ∑K k=1 fk ( · ) ) loss functions are L-smooth , i.e. , ‖∇f ( x ) −∇f ( y ) ‖2 ≤ L‖x− y‖2 , ∀x , y ∈ Rd , ( 8 ) ‖∇fk ( x ) −∇fk ( y ) ‖2 ≤ L‖x− y‖2 , ∀k ∈ [ K ] , ∀x , y ∈ Rd . ( 9 ) Assumption 2 . ( Local Bounded Variance ) The local stochastic gradient ∇Fk ( x ; ξ ) has a bounded variance , i.e. , Eξ∼Dk‖∇Fk ( x ; ξ ) −∇fk ( x ) ‖22 ≤ σ2 , ∀k ∈ [ K ] , ∀x ∈ Rd . ( 10 ) Note that Eξ∼Dk∇Fk ( x ; ξ ) = ∇fk ( x ) . Assumption 3 . ( Bounded Variance across Workers ) The L2 norm of the difference of the local and global full gradient is bounded , i.e. , ‖∇fk ( x ) −∇f ( x ) ‖22 ≤ κ2 , ∀k ∈ [ K ] , ∀x ∈ Rd , ( 11 ) where κ = 0 leads to i.i.d . data distributions across workers . Assumption 4 . ( Bounded Full Gradient ) The second moment of the global full gradient is bounded , i.e. , ‖∇f ( x ) ‖22 ≤ G2 , ∀x ∈ Rd . ( 12 ) Assumption 5 . ( δ-approximate compressor ) The compression function C ( · ) : Rd → R is a δapproximate compressor for 0 < δ ≤ 1 if for all v ∈ Rd , ‖C ( v ) − v‖22 ≤ ( 1− δ ) ‖v‖22 . ( 13 ) C BASIC LEMMAS Lemma 1 . For OLCO3-TC , let x̃t : = 1K ∑K k=1 x ( k ) t − 1K ∑K k=1 e ( k ) t − et−sp , then we have x̃t = x̃t−1 − ηt−1 K K∑ k=1 m ( k ) t . ( 14 ) Proof . For t = np where n is some integer , x̃np = 1 K K∑ k=1 x ( k ) np − 1 K K∑ k=1 e ( k ) np − e ( n−s ) p = xnp − 1 K K∑ k=1 s−1∑ i=0 C ( ∆ ( k ) ( n−i ) p ) − 1 K K∑ k=1 e ( k ) np − e ( n−s ) p = x ( n−1 ) p − C ( ∆ ( n−s ) p ) − 1 K K∑ k=1 s−1∑ i=0 C ( ∆ ( k ) ( n−i ) p ) − 1 K K∑ k=1 e ( k ) np − e ( n−s ) p = 1 K K∑ k=1 ( x ( k ) ( n−1 ) p + s−1∑ i=0 C ( ∆ ( k ) ( n−1−i ) p ) ) − C ( ∆ ( n−s ) p ) − 1 K K∑ k=1 s−1∑ i=0 C ( ∆ ( k ) ( n−i ) p ) − 1 K K∑ k=1 e ( k ) np − e ( n−s ) p = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 ∆ ( k ) np + 1 K K∑ k=1 C ( ∆ ( k ) ( n−s ) p ) − C ( ∆ ( n−s ) p ) − e ( n−s ) p = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 ∆ ( k ) np − e ( n−s ) p−1 = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 np−1∑ τ= ( n−1 ) p ητm ( k ) τ+1 − 1 K K∑ k=1 e ( k ) np−1 − e ( n−s ) p−1 = x̃np−1 − 1 K K∑ k=1 ηnp−1m ( k ) np . ( 15 ) For t 6= np , x̃t = 1 K K∑ k=1 x ( k ) t − 1 K K∑ k=1 e ( k ) t − et−sp = 1 K K∑ k=1 x ( k ) t − 1 K K∑ k=1 e ( k ) t−1 − et−sp−1 = x̃t−1 − ηt−1 K K∑ k=1 m ( k ) t . ( 16 ) Lemma 2 . For OLCO3-OC , let x̃t : = 1K ∑K k=1 x ( k ) t − 1K ∑K k=1 e ( k ) t , then we have x̃t = x̃t−1 − ηt−1 K K∑ k=1 m ( k ) t . ( 17 ) Proof . For t = np where n is some integer , x̃np = 1 K K∑ k=1 x ( k ) np − 1 K K∑ k=1 e ( k ) np = xnp − 1 K K∑ k=1 s−1∑ i=0 C ( ∆ ( k ) ( n−i ) p ) − 1 K K∑ k=1 e ( k ) np = x ( n−1 ) p − 1 K K∑ k=1 C ( ∆ ( k ) ( n−s ) p ) − 1 K K∑ k=1 s−1∑ i=0 C ( ∆ ( k ) ( n−i ) p ) − 1 K K∑ k=1 e ( k ) np = 1 K K∑ k=1 ( x ( k ) ( n−1 ) p + s−1∑ i=0 C ( ∆ ( k ) ( n−1−i ) p ) ) − 1 K K∑ k=1 C ( ∆ ( k ) ( n−s ) p ) − 1 K K∑ k=1 s−1∑ i=0 C ( ∆ ( k ) ( n−i ) p ) − 1 K K∑ k=1 e ( k ) np = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 C ( ∆ ( k ) np ) − 1 K K∑ k=1 e ( k ) np = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 ∆ ( k ) np = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 np−1∑ τ= ( n−1 ) p ητm ( k ) τ+1 − 1 K K∑ k=1 e ( k ) np−1 = x̃np−1 − 1 K K∑ k=1 ηnp−1m ( k ) np . ( 18 ) For t 6= np , x̃t = 1 K K∑ k=1 x ( k ) t − 1 K K∑ k=1 e ( k ) t = 1 K K∑ k=1 x ( k ) t − 1 K K∑ k=1 e ( k ) t−1 = x̃t−1 − ηt−1 K K∑ k=1 m ( k ) t . ( 19 ) Lemma 3 . For OLCO3-VQ , let x̃t : = 1K ∑K k=1 x ( k ) t − 1K ∑K k=1 e ( k ) t−sp , then we have x̃t = x̃t−1 − ηt−1 K K∑ k=1 m ( k ) t . ( 20 ) Proof . For t = np where n is some integer , x̃np = 1 K K∑ k=1 x ( k ) np − 1 K K∑ k=1 e ( k ) ( n−s ) p = xnp − 1 K K∑ k=1 s−1∑ i=0 ∆ ( k ) ( n−i ) p − 1 K K∑ k=1 e ( k ) ( n−s ) p = x ( n−1 ) p − 1 K K∑ k=1 C ( ∆ ( k ) ( n−s ) p ) − 1 K K∑ k=1 s−1∑ i=0 ∆ ( k ) ( n−i ) p − 1 K K∑ k=1 e ( k ) ( n−s ) p = 1 K K∑ k=1 ( x ( k ) ( n−1 ) p + s−1∑ i=0 ∆ ( k ) ( n−1−i ) p ) − 1 K K∑ k=1 C ( ∆ ( k ) ( n−s ) p ) − 1 K K∑ k=1 s−1∑ i=0 ∆ ( k ) ( n−i ) p − 1 K K∑ k=1 e ( k ) ( n−s ) p = 1 K K∑ k=1 ( x ( k ) ( n−1 ) p + s−1∑ i=0 ∆ ( k ) ( n−1−i ) p ) − 1 K K∑ k=1 ∆ ( k ) ( n−s ) p − 1 K K∑ k=1 s−1∑ i=0 ∆ ( k ) ( n−i ) p = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 ∆ ( k ) np = 1 K K∑ k=1 x ( k ) ( n−1 ) p − 1 K K∑ k=1 np−1∑ τ= ( n−1 ) p ητm ( k ) τ+1 − 1 K K∑ k=1 e ( k ) np−1−sp = x̃np−1 − 1 K K∑ k=1 ηnp−1m ( k ) np . ( 21 ) For t 6= np , x̃t = 1 K K∑ k=1 x ( k ) t − 1 K K∑ k=1 e ( k ) t−sp = 1 K K∑ k=1 x ( k ) t − 1 K K∑ k=1 e ( k ) t−1−sp = x̃t−1 − ηt−1 K K∑ k=1 m ( k ) t . ( 22 ) D PROOF OF THEOREM 1 Lemma 4 . For OLCO3-VQ with vanilla SGD and under Assumptions 2 , 3 , 4 , and 5 , the local error satisfies E‖e ( k ) t ‖22 ≤ 12 ( 1− δ ) δ2 p2η2 ( σ2 + κ2 +G2 ) . ( 23 ) Proof . First we have E‖∇F ( x ( k ) t ; ξ ( k ) t ) ‖22 ≤ 3E‖∇F ( x ( k ) t ; ξ ( k ) t ) −∇fk ( x ( k ) t ) ‖2 + 3E‖∇fk ( x ( k ) t ) −∇f ( x ( k ) t ) ‖22 + 3E‖∇f ( x ( k ) t ) ‖22 ≤ 3σ2 + 3κ2 + 3G2 . ( 24 ) Let St = b tpc , E‖e ( k ) t ‖22 = E‖e ( k ) Stp ‖22 = E‖C ( ∆ ( k ) Stp ) −∆ ( k ) Stp‖ 2 2 ≤ ( 1− δ ) E‖∆ ( k ) Stp ‖22 = ( 1− δ ) E‖ Stp−1∑ t′= ( St−1 ) p η∇F ( x ( k ) t′ ; ξ ( k ) t′ ) + e ( k ) ( St−s−1 ) p‖ 2 2 ≤ ( 1− δ ) ( 1 + ρ ) E‖e ( k ) ( St−s−1 ) p‖ 2 2 + ( 1 + δ ) ( 1 + 1 ρ ) E‖ Stp−1∑ t′= ( St−1 ) p η∇F ( x ( k ) t′ ; ξ ( k ) t′ ) ‖ 2 2 ≤ ( 1− δ ) ( 1 + ρ ) E‖e ( k ) ( St−s−1 ) p‖ 2 2 + 3 ( 1 + δ ) ( 1 + 1 ρ ) p2η2 ( σ2 + κ2 +G2 ) . ( 25 ) Therefore , E‖e ( k ) t ‖22 ≤ 3 ( 1− δ ) ( 1 + 1 ρ ) p2η2 ( σ2 + κ2 +G2 ) bSts c−1∑ i=0 [ ( 1− δ ) ( 1 + ρ ) ] i ≤ 3 ( 1− δ ) ( 1 + 1ρ ) 1− ( 1− δ ) ( 1 + ρ ) p2η2 ( σ2 + κ2 +G2 ) . ( 26 ) Let ρ = δ2 ( 1−δ ) such that 1 + 1 ρ = 2−δ δ ≤ 2 δ , then E‖e ( k ) t ‖22 ≤ 12 ( 1−δ ) δ2 p 2η2 ( σ2 + κ2 +G2 ) . Lemma 5 . For OLCO3-VQ with vanilla SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ 16L ( s+1 ) p , we have 1 KT T−1∑ t=0 K−1∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 3η2 ( s+ 1 ) pσ2 ( 1 + 12 ( 1− δ ) δ2 ( s+ 1 ) p ) + 9η2 ( s+ 1 ) 2p2κ2 ( 1 + 4 ( 1− δ ) δ2 ) + 36 ( 1− δ ) δ2 η2 ( s+ 1 ) 2p2G2 . ( 27 ) Proof . Let St = b tpc , 1 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 = 1 K K∑ k=1 E‖ 1 K K∑ k′=1 ( − s−1∑ i=0 ∆ ( k′ ) ( St−i ) p − t−1∑ t′=Stp η∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) ) − ( − s−1∑ i=0 ∆ ( k ) ( St−i ) p − t−1∑ t′=Stp η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ) − 1 K K∑ k′=1 e ( k ′ ) t−sp‖22 = 1 K K∑ k=1 E‖ − 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p η∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) + t−1∑ t′= ( St−s ) p η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) − 1 K K∑ k′=1 e ( k ′ ) t−sp − 1 K K∑ k′=1 s−1∑ i=0 e ( k ′ ) ( St−i−s−1 ) p + s−1∑ i=0 e ( k ) ( St−i−s−1 ) p‖ 2 2 ≤ 2η 2 K K∑ k=1 E‖ − 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) + t−1∑ t′= ( St−s ) p ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ‖ 2 2 + 2 K K∑ k=1 E‖ − 1 K K∑ k′=1 e ( k ′ ) t−sp − 1 K K∑ k′=1 s−1∑ i=0 e ( k ′ ) ( St−i−s−1 ) p + s−1∑ i=0 e ( k ) ( St−i−s−1 ) p‖ 2 2 . ( 28 ) The first term is bounded by 2η2 K K∑ k=1 E‖ − 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) + t−1∑ t′= ( St−s ) p ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ‖ 2 2 ≤ 2η 2 K K∑ k=1 E ∥∥∥∥∥∥ t−1∑ t′= ( St−s ) p ( − 1 K K∑ k′=1 ( ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) −∇fk′ ( x ( k′ ) t′ ) ) + ( ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) −∇fk ( x ( k ) t′ ) ) ) ∥∥∥∥∥∥ 2 2 + 2η2 K K∑ k=1 E‖ t−1∑ t′= ( St−s ) p ( − 1 K K∑ k′=1 ∇fk′ ( x ( k ′ ) t′ ) +∇fk ( x ( k ) t′ ) ) ‖ 2 2 = 2η2 K K∑ k=1 t−1∑ t′= ( St−s ) p E‖ − 1 K K∑ k′=1 ( ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) −∇fk′ ( x ( k′ ) t′ ) ) + ( ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) −∇fk ( x ( k ) t′ ) ) ‖ 2 2 + 2η2 K K∑ k=1 E‖ t−1∑ t′= ( St−s ) p ( − 1 K K∑ k′=1 ∇fk′ ( x ( k ′ ) t′ ) +∇fk ( x ( k ) t′ ) ) ‖ 2 2 ≤ 2η 2 K K∑ k=1 t−1∑ t′= ( St−s ) p E‖∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) −∇fk ( x ( k ) t′ ) ‖ 2 2 + 2η2 K K∑ k=1 t−1∑ t′= ( St−s ) p ( t− ( St − s ) p ) E‖ − 1 K K∑ k′=1 ∇fk′ ( x ( k ′ ) t′ ) +∇fk ( x ( k ) t′ ) ‖ 2 2 ≤ 2η2 ( s+ 1 ) pσ2 + 2η 2 ( s+ 1 ) p K t−1∑ t′=t− ( s+1 ) p K∑ k=1 E‖ − 1 K K∑ k′=1 ∇fk′ ( x ( k ′ ) t′ ) +∇fk ( x ( k ) t′ ) ‖ 2 2 , ( 29 ) where the third inequality follows 1K ∑K k=1 ‖ 1 K ∑K k′=1 ak′ − ak‖22 = 1 K ∑K k=1 ‖ak‖22 − ‖ 1K ∑K k=1 ak‖22 ≤ 1 K ∑K k=1 ‖ak‖22 , and 1 K K∑ k=1 E‖ − 1 K K∑ k′=1 ∇fk′ ( xk ′ t ) +∇fk ( x ( k ) t ) ‖22 = 3 K K∑ k=1 E [ ‖∇fk ( x ( k ) t ) −∇fk ( x̃t ) ‖22 + ‖∇fk ( x̃t ) −∇f ( x̃t ) ‖22 + ‖∇f ( x̃t ) − 1 K K∑ k′=1 ∇fk′ ( xk ′ t ) ‖22 ] ≤ 3 K K∑ k=1 E [ L2‖x̃t − x ( k ) t ‖22 + κ2 + 1 K K∑ k′=1 ‖∇fk′ ( x̃t ) −∇fk′ ( xk ′ t ) ‖22 ] ≤ 6L 2 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 + 3κ2 . ( 30 ) The second term is bounded by 2 K K∑ k=1 E‖ − 1 K K∑ k′=1 e ( k ′ ) t−sp − 1 K K∑ k′=1 s−1∑ i=0 e ( k ′ ) ( St−i−s−1 ) p + s−1∑ i=0 e ( k ) ( St−i−s−1 ) p‖ 2 2 ≤ 2 ( 1 + s ) K K∑ k=1 E‖ 1 K K∑ k′=1 e ( k ′ ) ( St−s ) p‖ 2 2 + 2 ( 1 + 1s ) K K∑ k=1 E‖ − 1 K K∑ k′=1 s−1∑ i=0 e ( k ′ ) ( St−i−s−1 ) p + s−1∑ i=0 e ( k ) ( St−i−s−1 ) p‖ 2 2 ≤ 2 ( 1 + s ) K K∑ k=1 E‖e ( k ) ( St−s ) p‖ 2 2 + 2 ( 1 + 1s ) K K∑ k=1 E‖ s−1∑ i=0 e ( k ) ( St−i−s−1 ) p‖ 2 2 ≤ 2 ( 1 + s ) K K∑ k=1 E‖e ( k ) ( St−s ) p‖ 2 2 + 2 ( 1 + s ) K K∑ k=1 s−1∑ i=0 E‖e ( k ) ( St−i−s−1 ) p‖ 2 2 = 2 ( s+ 1 ) K K∑ k=1 s∑ i=0 E‖e ( k ) ( St−i−s ) p‖ 2 2 ≤ 24 ( 1− δ ) δ2 ( s+ 1 ) 2p2η2 ( σ2 + κ2 +G2 ) , ( 31 ) where the last inequality follows Lemma 4 . Combine the bounds of the first term and the second term , we have 1 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 2η2 ( s+ 1 ) pσ2 + 2η2 ( s+ 1 ) p t−1∑ t′=t− ( s+1 ) p ( 6L2 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 + 3κ2 ) + 24 ( 1− δ ) δ2 ( s+ 1 ) 2p2η2 ( σ2 + κ2 +G2 ) ≤ 2η2 ( s+ 1 ) pσ2 ( 1 + 12 ( 1− δ ) δ2 ( s+ 1 ) p ) + 6η2 ( s+ 1 ) 2p2κ2 ( 1 + 4 ( 1− δ ) δ2 ) + 24 ( 1− δ ) δ2 η2 ( s+ 1 ) 2p2G2 + 12η2L2 ( s+ 1 ) p t−1∑ t′=t− ( s+1 ) p 1 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 . ( 32 ) Sum the above inequality from t = 0 to t = T − 1 and divide it by T , 1 KT T−1∑ t=0 K−1∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 2η2 ( s+ 1 ) pσ2 ( 1 + 12 ( 1− δ ) δ2 ( s+ 1 ) p ) + 6η2 ( s+ 1 ) 2p2κ2 ( 1 + 4 ( 1− δ ) δ2 ) + 24 ( 1− δ ) δ2 η2 ( s+ 1 ) 2p2G2 + 12η2L2 ( s+ 1 ) 2p2 · 1 KT T−1∑ t=0 K−1∑ k=1 E‖x̃t − x ( k ) t ‖22 . ( 33 ) Therefore , 1 KT T−1∑ t=0 K−1∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 2η2 ( s+ 1 ) pσ2 ( 1 + 12 ( 1−δ ) δ2 ( s+ 1 ) p ) + 6η 2 ( s+ 1 ) 2p2κ2 ( 1 + 4 ( 1−δ ) δ2 ) + 24 ( 1−δ ) δ2 η 2 ( s+ 1 ) 2p2G2 1− 12η2L2 ( s+ 1 ) 2p2 . ( 34 ) If we choose η ≤ 16L ( s+1 ) p , 1 KT T−1∑ t=0 K−1∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 3η2 ( s+ 1 ) pσ2 ( 1 + 12 ( 1− δ ) δ2 ( s+ 1 ) p ) + 9η2 ( s+ 1 ) 2p2κ2 ( 1 + 4 ( 1− δ ) δ2 ) + 36 ( 1− δ ) δ2 η2 ( s+ 1 ) 2p2G2 . ( 35 ) Theorem 1 . For OLCO3-VQ with vanilla SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ min { 16L ( s+1 ) p , 1 9L } , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( f ( x0 ) − f∗ ) ηT + 9ηLσ2 K + 12η2L2 ( s+ 1 ) pσ2 ( 1 + 14 ( 1− δ ) δ2 ( s+ 1 ) p ) + 36η2L2 ( s+ 1 ) 2p2κ2 ( 1 + 5 ( 1− δ ) δ2 ) + 168 ( 1− δ ) δ2 η2L2 ( s+ 1 ) 2p2G2 . ( 36 ) Proof . According to Assumption 1 , Etf ( x̃t+1 ) − f ( x̃t ) ≤ Et〈∇f ( x̃t ) , x̃t+1 − x̃t〉+ L 2 Et‖x̃t+1 − x̃t‖22 = −η〈∇f ( x̃t ) , 1 K K∑ k=1 ∇fk ( x ( k ) t ) 〉+ Lη2 2 Et‖ 1 K K∑ k=1 ∇Fk ( x ( k ) t ; ξ ( k ) t ) ‖22 . ( 37 ) For the first term , −〈∇f ( x̃t ) , 1 K K∑ k=1 ∇fk ( x ( k ) t ) 〉 = −‖∇f ( x̃t ) ‖22 − 〈∇f ( x̃t ) , 1 K K∑ k=1 ( ∇fk ( x ( k ) t ) −∇fk ( x̃t ) ) 〉 ≤ −1 2 ‖∇f ( x̃t ) ‖22 + 1 2 ‖ 1 K K∑ k=1 ( ∇fk ( x ( k ) t ) −∇fk ( x̃t ) ) ‖22 ≤ −1 2 ‖∇f ( x̃t ) ‖22 + L2 2K K∑ k=1 ‖x̃t − x ( k ) t ‖22 , ( 38 ) where the first equality follows that∇f ( x̃t ) = 1K ∑K k=1∇fk ( x̃t ) . For the second term , Et‖ 1 K K∑ k=1 ∇Fk ( x ( k ) t ; ξ ( k ) t ) ‖22 = Et‖ 1 K K∑ k=1 ( ∇Fk ( x ( k ) t ; ξ ( k ) t ) −∇fk ( x ( k ) t ) ) + 1 K K∑ k=1 ( ∇fk ( x ( k ) t ) −∇fk ( x̃t ) ) +∇f ( x̃t ) ‖22 ≤ 3Et‖ 1 K K∑ k=1 ( ∇Fk ( x ( k ) t ; ξ ( k ) t ) −∇fk ( x ( k ) t ) ) ‖22 + 3‖ 1 K K∑ k=1 ( ∇fk ( x ( k ) t ) −∇fk ( x̃t ) ) ‖22 + 3‖∇f ( x̃t ) ‖22 ≤ 3σ 2 K + 3L2 K K∑ k=1 ‖x̃t − x ( k ) t ‖22 + 3‖∇f ( x̃t ) ‖22 . ( 39 ) Combine them and we have Et ( x̃t+1 ) − f ( x̃t ) ≤ − η 2 ( 1− 3ηL ) ‖∇f ( x̃t ) ‖22 + ηL2 2 ( 1 + 3ηL ) 1 K K∑ k=1 ‖x̃t − x ( k ) t ‖22 + 3η2Lσ2 2K . ( 40 ) If we choose η ≤ 19L , Et ( x̃t+1 ) − f ( x̃t ) ≤ − η 3 ‖∇f ( x̃t ) ‖22 + 2ηL2 3 1 K K∑ k=1 ‖x̃t − x ( k ) t ‖22 + 3η2Lσ2 2K . ( 41 ) Then for the averaged parameters 1K ∑K k=1 x ( k ) t , ‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 2‖∇f ( 1 K K∑ k=1 x ( k ) t ) −∇f ( x̃t ) ‖22 + 2‖∇f ( x̃t ) ‖22 ≤ 2L2‖ 1 K K∑ k=1 e ( k ) t−sp‖22 + 2‖∇f ( x̃t ) ‖22 ≤ 2L2 K K∑ k=1 ‖e ( k ) t−sp‖22 + 2‖∇f ( x̃t ) ‖22 ≤ 6 [ f ( x̃t ) − Etf ( x̃t+1 ) ] η + 9ηLσ2 K + 4L2 K K∑ k=1 ‖x̃t − x ( k ) t ‖22 + 2L2 K K∑ k=1 ‖e ( k ) t−sp‖22 . ( 42 ) Take total expectation , sum from t = 0 to t = T − 1 , and rearrange , 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 [ f ( x̃0 ) − Ef ( x̃T ) ] ηT + 9ηLσ2 K + 2L2 KT T−1∑ t=0 K∑ k=1 E‖e ( k ) t−sp‖22 + 4L2 KT T−1∑ t=0 K∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 6 [ f ( x̃0 ) − Ef ( x̃T ) ] ηT + 9ηLσ2 K + 24 ( 1− δ ) δ2 p2η2L2 ( σ2 + κ2 +G2 ) + 12η2L2 ( s+ 1 ) pσ2 ( 1 + 12 ( 1− δ ) δ2 ( s+ 1 ) p ) + 36η2L2 ( s+ 1 ) 2p2κ2 ( 1 + 4 ( 1− δ ) δ2 ) + 144 ( 1− δ ) δ2 η2L2 ( s+ 1 ) 2p2G2 ≤ 6 ( f ( x0 ) − f∗ ) ηT + 9ηLσ2 K + 12η2L2 ( s+ 1 ) pσ2 ( 1 + 14 ( 1− δ ) δ2 ( s+ 1 ) p ) + 36η2L2 ( s+ 1 ) 2p2κ2 ( 1 + 5 ( 1− δ ) δ2 ) + 168 ( 1− δ ) δ2 η2L2 ( s+ 1 ) 2p2G2 , ( 43 ) where the second inequality follows Lemma 4 and 5 . E PROOF OF THEOREM 2 Lemma 6 . For OLCO3-TC with vanilla SGD and under Assumptions 2 , 3 , 4 , and 5 , the local error satisfies E‖e ( k ) t ‖22 ≤ 12 ( 1− δ ) δ2 p2η2 ( σ2 + κ2 +G2 ) . ( 44 ) Proof . Same as the proof of Lemma 4 , except that e ( k ) ( St−s−1 ) p is replaced with e ( k ) ( St−1 ) p. Lemma 7 . For OLCO3-TC with vanilla SGD and under Assumptions 2 , 3 , 4 , and 5 , the server error satisfies E‖et‖22 ≤ 96 ( 2− δ ) ( 1− δ ) δ4 p2η2 ( σ2 + κ2 +G2 ) . ( 45 ) Proof . Let St = b tpc , E‖ 1 K K∑ k=1 C ( ∆ ( k ) Stp ) ‖ 2 2 ≤ 2E‖ 1 K K∑ k=1 C ( ∆ ( k ) Stp ) − 1 K K∑ k=1 ∆ ( k ) Stp ‖22 + 2E‖ 1 K K∑ k=1 ∆ ( k ) Stp ‖22 ≤ 2 K K∑ k=1 E‖C ( ∆ ( k ) Stp ) −∆ ( k ) Stp ‖22 + 2 K K∑ k=1 E‖∆ ( k ) Stp‖ 2 2 ≤ 2 ( 2− δ ) K K∑ k=1 E‖∆ ( k ) Stp‖ 2 2 . ( 46 ) Following the proof of Lemma 4 we have E‖∆ ( k ) Stp‖ 2 2 ≤ 3 ( 1+ 1ρ ) 1− ( 1−δ ) ( 1+ρ ) p 2η2 ( σ2+κ2+G2 ) . Therefore , E‖et‖22 = E‖eStp‖22 ≤ ( 1− δ ) E‖ 1 K K∑ k=1 C ( ∆ ( k ) Stp ) + e ( St−1 ) p‖ 2 ≤ ( 1− δ ) ( 1 + 1 ρ ) E‖ 1 K K∑ k=1 C ( ∆ ( k ) Stp ) ‖ 2 2 + ( 1− δ ) ( 1 + ρ ) E‖e ( St−1 ) p‖ 2 2 ≤ 2 ( 2− δ ) ( 1− δ ) ( 1 + 1 ρ ) 1 K K∑ k=1 E‖∆ ( k ) Stp‖ 2 2 + ( 1− δ ) ( 1 + ρ ) E‖e ( St−1 ) p‖ 2 2 ≤ 2 ( 2− δ ) ( 1− δ ) ( 1 + 1 ρ ) 3 ( 1 + 1ρ ) 1− ( 1− δ ) ( 1 + ρ ) p2η2 ( σ2 + κ2 +G2 ) + ( 1− δ ) ( 1 + ρ ) E‖e ( St−1 ) p‖ 2 2 ≤ 6 ( 2− δ ) ( 1− δ ) ( 1 + 1ρ ) 2 [ 1− ( 1− δ ) ( 1 + ρ ) ] 2 p2η2 ( σ2 + κ2 +G2 ) . ( 47 ) Let ρ = δ2 ( 1−δ ) such that 1+ 1 ρ = 2−δ δ ≤ 2 δ , then E‖e ( k ) t ‖22 ≤ 96 ( 2−δ ) ( 1−δ ) δ4 p 2η2 ( σ2 +κ2 +G2 ) . Lemma 8 . For OLCO3-TC with vanilla SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ 16L ( s+1 ) p and let h ( δ ) = 1−δ δ2 ( 1 + 4 ( 2−δ ) δ2 ) , we have 1 KT T−1∑ t=0 K−1∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 3η2pσ2 ( s+ 1 + 72h ( δ ) p ) + 9η2p2κ2 ( ( s+ 1 ) 2 + 24h ( δ ) ) + 216h ( δ ) η2p2G2 . ( 48 ) Proof . Let St = b tpc , 1 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 = 1 K K∑ k=1 E‖ 1 K K∑ k′=1 ( − s−1∑ i=0 C ( ∆ ( k ′ ) ( St−i ) p ) − t−1∑ t′=Stp η∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) ) − ( − s−1∑ i=0 C ( ∆ ( k ) ( St−i ) p ) − t−1∑ t′=Stp η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ) − 1 K K∑ k′=1 e ( k ′ ) t − et−sp‖22 , ( 49 ) where s−1∑ i=0 C ( ∆ ( k ) ( St−i ) p ) = s−1∑ i=0 [ ∆ ( k ) ( St−i ) p − e ( k ) ( St−i ) p ] = s−1∑ i=0 [ ( St−i ) p−1∑ t′= ( St−i−1 ) p η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) + e ( k ) ( St−i−1 ) p − e ( k ) ( St−i ) p ] = s−1∑ i=0 ( St−i ) p−1∑ t′= ( St−i−1 ) p ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) + e ( k ) ( St−s ) p − e ( k ) Stp . ( 50 ) Therefore 1 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 = 1 K K∑ k=1 E‖ − 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p η∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) + t−1∑ t′= ( St−s ) p η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) − 1 K K∑ k′=1 ( e ( k ′ ) ( St−s ) p − e ( k′ ) Stp ) + ( e ( k ) ( St−s ) p − e ( k ) Stp ) − 1 K K∑ k′=1 e ( k ′ ) t − et−sp‖22 ≤ 2η 2 K K∑ k=1 E‖ − 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p η∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) + t−1∑ t′= ( St−s ) p η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ‖ 2 2 + 2 K K∑ k=1 E‖ − 1 K K∑ k′=1 e ( k ′ ) ( St−s ) p + e ( k ) ( St−s ) p − e ( k ) Stp − e ( St−s ) p‖ 2 2 , ( 51 ) where the first term can be bounded following Eqs . ( 29,30 ) . The second term satisfies 2 K K∑ k=1 E‖ − 1 K K∑ k′=1 e ( k ′ ) ( St−s ) p + e ( k ) ( St−s ) p − e ( k ) Stp − e ( St−s ) p‖ 2 2 ≤ 6 K K∑ k=1 E‖ − 1 K K∑ k′=1 e ( k ′ ) ( St−s ) p + e ( k ) ( St−s ) p‖ 2 2 + 6 K K∑ k=1 E‖e ( k ) Stp‖ 2 2 + 6 K K∑ k=1 E‖e ( St−s ) p‖ 2 2 ≤ 6 K K∑ k=1 E‖e ( k ) ( St−s ) p‖ 2 2 + 6 K K∑ k=1 E‖e ( k ) Stp‖ 2 2 + 6 K K∑ k=1 E‖e ( St−s ) p‖ 2 2 ≤ 1− δ δ2 ( 1 + 4 ( 2− δ ) δ2 ) · 144p2η2 ( σ2 + κ2 +G2 ) , ( 52 ) where the last inequality follows Lemmas 6 and 7 . Let h ( δ ) = 1−δδ2 ( 1+ 4 ( 2−δ ) δ2 ) . Combine the above two inequalities and we have 1 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 2η2 ( s+ 1 ) pσ2 + 2η2 ( s+ 1 ) p t−1∑ t′=t− ( s+1 ) p ( 6L2 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 + 3κ2 ) + 144h ( δ ) p2η2 ( σ2 + κ2 +G2 ) ≤ 2η2pσ2 ( s+ 1 + 72h ( δ ) p ) + 6η2p2κ2 ( ( s+ 1 ) 2 + 24h ( δ ) ) + 144h ( δ ) p2η2G2 + 12η2L2 ( s+ 1 ) p t−1∑ t′=t− ( s+1 ) p 1 K K∑ k=1 E‖x̃t − x ( k ) t ‖22 . ( 53 ) Following Eqs . ( 33,34,35 ) , 1 KT T−1∑ t=0 K−1∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 3η2pσ2 ( s+ 1 + 72h ( δ ) p ) + 9η2p2κ2 ( ( s+ 1 ) 2 + 24h ( δ ) ) + 216h ( δ ) η2p2G2 . ( 54 ) Theorem 2 . For OLCO3-TC with vanilla SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ min { 16L ( s+1 ) p , 1 9L } and let h ( δ ) = 1−δ δ2 ( 1 + 4 ( 2−δ ) δ2 ) , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( f ( x0 ) − f∗ ) ηT + 9ηLσ2 K + 12η2pσ2 ( s+ 1 + 80h ( δ ) p ) + 12η2p2κ2 ( 3 ( s+ 1 ) 2 + 80h ( δ ) ) + 960η2p2G2h ( δ ) . ( 55 ) Proof . Following the proof of Theorem 1 , we have the same inequality as Eq . ( 41 ) : Etf ( x̃t+1 ) − f ( x̃t ) ≤ − η 3 ‖∇f ( x̃t ) ‖22 + 2ηL2 3 1 K K∑ k=1 ‖x̃t − x ( k ) t ‖22 + 3η2Lσ2 2K . ( 56 ) Then for the averaged parameters 1K ∑K k=1 x ( k ) t , ‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 2‖∇f ( 1 K K∑ k=1 x ( k ) t ) −∇f ( x̃t ) ‖22 + 2‖∇f ( x̃t ) ‖22 ≤ 2L2‖ 1 K K∑ k=1 e ( k ) t + et−sp‖22 + 2‖∇f ( x̃t ) ‖22 ≤ 4L 2 K K∑ k=1 ‖e ( k ) t ‖22 + 4L2 K ‖et−sp‖22 + 2‖∇f ( x̃t ) ‖22 ≤ 6 [ f ( x̃t ) − Etf ( x̃t+1 ) ] η + 9ηLσ2 K + 4L2 K K∑ k=1 ‖x̃t − x ( k ) t ‖22 + 4L2 K K∑ k=1 ‖e ( k ) t ‖22 + 4L2 K ‖et−sp‖22 . ( 57 ) Take total expectation , sum from t = 0 to t = T − 1 , and rearrange , 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 [ f ( x̃0 ) − Ef ( x̃T ) ] ηT + 9ηLσ2 K + 4L2 KT T−1∑ t=0 K∑ k=1 E‖e ( k ) t−sp‖22 + 4L2 KT T−1∑ t=0 E‖et−sp‖22 + 4L2 KT T−1∑ t=0 K∑ k=1 E‖x̃t − x ( k ) t ‖22 ≤ 6 [ f ( x̃0 ) − Ef ( x̃T ) ] ηT + 9ηLσ2 K + 12η2pσ2 ( s+ 1 + 72h ( δ ) p+ 4 ( 1− δ ) δ2 p+ 32 ( 2− δ ) ( 1− δ ) δ4 p ) + 12η2p2κ2 ( 3 ( s+ 1 ) 2 + 72h ( δ ) + 4 ( 1− δ ) δ2 + 32 ( 2− δ ) ( 1− δ ) δ4 ) + 12η2p2G2 ( 72h ( δ ) + 4 ( 1− δ ) δ2 + 32 ( 2− δ ) ( 1− δ ) δ4 ) ≤ 6 ( f ( x̃0 ) − f∗ ) ηT + 9ηLσ2 K + 12η2pσ2 ( s+ 1 + 80h ( δ ) p ) + 12η2p2κ2 ( 3 ( s+ 1 ) 2 + 80h ( δ ) ) + 960η2p2G2h ( δ ) , ( 58 ) where the second inequality follows Lemmas 6 , 7 and 8 . F PROOF OF THEOREM 3 We first define two virtual variables zt and pt satisfying pt = { µ 1−µ ( x̃t − x̃t−1 ) , t ≥ 1 0 , t = 0 ( 59 ) and zt = x̃t + pt . ( 60 ) Then the update rule of zt satisfies zt+1 − zt = ( x̃t+1 − x̃t ) + µ 1− µ ( x̃t+1 − x̃t ) − µ 1− µ ( x̃t − x̃t−1 ) = − η K K∑ k=1 m ( k ) t+1 − µ 1− µ η K K∑ k=1 m ( k ) t+1 + µ 1− µ η K K∑ k=1 m ( k ) t = − η ( 1− µ ) K K∑ k=1 ( m ( k ) t+1 − µm ( k ) t ) = − η ( 1− µ ) K K∑ k=1 ∇fk ( x ( k ) t ; ξ ( k ) t ) , ( 61 ) which exists for OLCO3-OC , OLCO3-VQ and OLCO3-TC . Lemma 9 . For OLCO3 with Momentum SGD , we have E‖m ( k ) t ‖22 ≤ 3 ( σ2 + κ2 +G2 ) ( 1− µ ) 2 . ( 62 ) Proof . E‖m ( k ) t ‖22 = E‖ t−1∑ t′=0 µt−1−t ′ ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ‖ 2 2 = ( t−1∑ t′=0 µt−1−t ′ ) 2E‖ t−1∑ t′=0 µt−1−t ′∑t−1 t′=0 µ t−1−t′ ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ‖ 2 2 ≤ ( t−1∑ t′=0 µt−1−t ′ ) 2E‖∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) ‖ 2 2 ≤ 3 ( σ2 + κ2 +G2 ) ( 1− µ ) 2 . ( 63 ) Lemma 10 . For OLCO3 with Momentum SGD , we have E‖pt‖2 ≤ 3µ2η2 ( σ2 + κ2 +G2 ) ( 1− µ ) 4 . ( 64 ) Proof . E‖pt‖2 = µ2 ( 1− µ ) 2 E‖x̃t − x̃t−1‖2 = µ2η2 ( 1− µ ) 2 E‖ 1 K K∑ k=1 m ( k ) t ‖2 ≤ µ2η2 ( 1− µ ) 2K K∑ k=1 E‖m ( k ) t ‖22 ≤ 3µ 2η2 ( σ2 + κ2 +G2 ) ( 1− µ ) 4 . ( 65 ) Lemma 11 . For OLCO3-VQ with Momentum SGD and under Assumptions 2 , 3 , 4 , and 5 , the local error satisfies E‖e ( k ) t ‖22 ≤ 12 ( 1− δ ) ( 1− µ ) 2δ2 p2η2 ( σ2 + κ2 +G2 ) . ( 66 ) Proof . Let St = b tpc , E‖e ( k ) t ‖22 = E‖e ( k ) Stp ‖22 = E‖C ( ∆ ( k ) Stp ) −∆ ( k ) Stp‖ 2 2 ≤ ( 1− δ ) E‖∆ ( k ) Stp ‖22 = ( 1− δ ) E‖ ST p−1∑ t′= ( St−1 ) p ηm ( k ) t′ + e ( k ) ( St−s−1 ) p‖ 2 2 ≤ ( 1− δ ) ( 1 + ρ ) E‖e ( k ) ( St−s−1 ) p‖ 2 2 + ( 1− δ ) ( 1 + 1 ρ ) 3η2p2 ( σ2 + κ2 +G2 ) ( 1− µ ) 2 . ( 67 ) Therefore , E‖e ( k ) t ‖22 ≤ 3 ( 1− δ ) ( 1 + 1 ρ ) p2η2 ( σ2 + κ2 +G2 ) ( 1− µ ) 2 bSts c−1∑ i=0 [ ( 1− δ ) ( 1 + ρ ) ] i ≤ 3 ( 1− δ ) ( 1 + 1ρ ) 1− ( 1− δ ) ( 1 + ρ ) p2η2 ( σ2 + κ2 +G2 ) ( 1− µ ) 2 . ( 68 ) Let ρ = δ2 ( 1−δ ) such that 1 + 1 ρ = 2−δ δ ≤ 2 δ , then E‖e ( k ) t ‖22 ≤ 12 ( 1−δ ) ( 1−µ ) 2δ2 p 2η2 ( σ2 + κ2 +G2 ) . Lemma 12 . For OLCO3-VQ with Momentum SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ 1−µ√ 72L ( s+1 ) p , we have 1 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 ≤ 4η2σ2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + ( s+ 1 ) p+ 12 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) + 12η2κ2 ( 1− µ ) 2 ( 1 ( 1− µ ) 2 + ( s+ 1 ) 2p2 + 4 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) + 12η2G2 ( 1− µ ) 2 ( 1 ( 1− µ ) 2 + 4 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) . ( 69 ) Proof . Let St = b tpc , 1 K K∑ k=1 E‖zt − x ( k ) t ‖22 = 1 K K∑ k=1 E‖ 1 K K∑ k′=1 ( − s−1∑ i=0 ∆ ( k′ ) ( St−i ) p − ( x ( k′ ) Stp − x ( k ′ ) t ) ) − ( − s−1∑ i=0 ∆ ( k ) ( St−i ) p − ( x ( k ) Stp − x ( k ) t ) − 1 K K∑ k′=1 e ( k ′ ) t−sp‖22 = 1 K K∑ k=1 E‖ 1 K K∑ k′=1 ( ηm ( k ′ ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1 + t−1∑ t′= ( St−s ) p η∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) t−1−t′∑ τ=0 µτ ) − ( ηm ( k ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1 + t−1∑ t′= ( St−s ) p η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) t−1−t′∑ τ=0 µτ ) + 1 K K∑ k′=1 e ( k ′ ) t−sp + 1 K K∑ k′=1 s−1∑ i=0 e ( k ′ ) ( St−i−s−1 ) p − s−1∑ i=0 e ( k ) ( St−i−s−1 ) p‖ 2 2 ≤ 3η 2 K K∑ k=1 E‖ 1 K K∑ k′=1 m ( k ′ ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1 −m ( k ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1‖22 + 3η2 K K∑ k=1 E‖ 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) t−1−t′∑ τ=0 µτ − t−1∑ t′= ( St−s ) p ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) t−1−t′∑ τ=0 µτ‖22 + 3 K K∑ k=1 E‖ 1 K K∑ k′=1 e ( k ′ ) t−sp + 1 K K∑ k′=1 s−1∑ i=0 e ( k ′ ) ( St−i−s−1 ) p + s−1∑ i=0 e ( k ) ( St−i−s−1 ) p‖ 2 2 . ( 70 ) The first term 3η2 K K∑ k=1 E‖ 1 K K∑ k′=1 m ( k ′ ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1 −m ( k ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1‖22 ≤ 3η 2 K K∑ k=1 E‖m ( k ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1‖22 ≤ 3η2 ( 1− µ ) 2K K∑ k=1 E‖m ( k ) ( St−s ) p‖ 2 2 ≤ 9η2 ( σ2 + κ2 +G2 ) ( 1− µ ) 4 , ( 71 ) where the last inequality follows Lemma 9 . Following Eq . ( 29 ) , the second term can be bounded by 3η2 K K∑ k=1 E‖ 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) t−1−t′∑ τ=0 µτ − t−1∑ t′= ( St−s ) p ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) t−1−t′∑ τ=0 µτ‖22 = 3η2 K K∑ k=1 E‖ t−1∑ t′= ( St−s ) p [ 1 K K∑ k′=1 ( ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) −∇fk′ ( x ( k′ ) t′ ) ) − ( ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) −∇fk ( x ( k ) t′ ) ) ] t−1−t′∑ τ=0 µτ‖22 + 3η2 K K∑ k=1 E‖ t−1∑ t′= ( St−s ) p ( 1 K K∑ k′=1 ∇fk′ ( x ( k ′ ) t′ ) −∇fk ( x ( k ) t′ ) ) t−1−t′∑ τ=0 µτ‖22 ≤ 3η 2 ( 1− µ ) 2K K∑ k=1 t−1∑ t′= ( St−s ) p E‖∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) −∇fk ( x ( k ) t′ ) ‖ 2 2 + 3η2 ( 1− µ ) 2K ( t− ( St − s ) p ) K∑ k=1 t−1∑ t′= ( St−s ) p E‖ 1 K K∑ k′=1 ∇fk′ ( x ( k ′ ) t′ ) −∇fk ( x ( k ) t′ ) ‖ 2 2 ≤ 3η 2 ( s+ 1 ) pσ2 ( 1− µ ) 2 + 3η2 ( s+ 1 ) p ( 1− µ ) 2 t−1∑ t′=t− ( s+1 ) p ( 6L2 K K∑ k=1 E‖zt′ − x ( k ) t′ ‖ 2 2 + 3κ 2 ) , ( 72 ) where the last inequality follows Eq . ( 30 ) . Combine the bounds of the first and second term with Lemma 11 and Eq . ( 31 ) , 1 K K∑ k=1 E‖zt − x ( k ) t ‖22 ≤ 9η 2 ( σ2 + κ2 +G2 ) ( 1− µ ) 4 + 3η2 ( s+ 1 ) pσ2 ( 1− µ ) 2 + 3η2 ( s+ 1 ) p ( 1− µ ) 2 t−1∑ t′=t− ( s+1 ) p ( 6L2 K K∑ k=1 E‖zt′ − x ( k ) t′ ‖ 2 2 + 3κ 2 ) + 3 ( s+ 1 ) K K∑ k=1 s∑ i=0 E‖e ( k ) ( St−i−s ) p‖ 2 2 ≤ 9η 2 ( σ2 + κ2 +G2 ) ( 1− µ ) 4 + 3η2 ( s+ 1 ) pσ2 ( 1− µ ) 2 + 9η2 ( s+ 1 ) 2p2κ2 ( 1− µ ) 2 + 36 ( 1− δ ) η2 ( s+ 1 ) 2p2 ( σ2 + κ2 +G2 ) ( 1− µ ) 2δ2 + 18η2L2 ( s+ 1 ) p ( 1− µ ) 2 t−1∑ t′=t− ( s+1 ) p 1 K K∑ k=1 E‖zt′ − x ( k ) t′ ‖ 2 2 . ( 73 ) Sum the above inequality from t = 0 to t = T − 1 and divide it by T , 1 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 ≤ 3η2σ2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + ( s+ 1 ) p+ 12 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) + 9η2κ2 ( 1− µ ) 2 ( 1 ( 1− µ ) 2 + ( s+ 1 ) 2p2 + 4 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) + 9η2G2 ( 1− µ ) 2 ( 1 ( 1− µ ) 2 + 4 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) + 18η2L2 ( s+ 1 ) 2p2 ( 1− µ ) 2 1 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 . ( 74 ) If we choose η ≤ 1−µ√ 72L ( s+1 ) p , 1 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 ≤ 4η2σ2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + ( s+ 1 ) p+ 12 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) + 12η2κ2 ( 1− µ ) 2 ( 1 ( 1− µ ) 2 + ( s+ 1 ) 2p2 + 4 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) + 12η2G2 ( 1− µ ) 2 ( 1 ( 1− µ ) 2 + 4 ( 1− δ ) ( s+ 1 ) 2p2 δ2 ) . ( 75 ) Theorem 3 . For OLCO3-VQ with Momentum SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ min { 1−µ√ 72L ( s+1 ) p , 1−µ9L } and let g ( µ , δ , s , p ) = 15 ( 1−µ ) 2 + 60 ( 1−δ ) ( s+1 ) 2p2 δ2 , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( 1− µ ) ( f ( x0 ) − f∗ ) ηT + 9Lησ2 ( 1− µ ) K + 4η2L2 ( 1− µ ) 2 [ ( 4 ( s+ 1 ) p+ g ( µ , δ , s , p ) ) σ2 + ( 12 ( s+ 1 ) 2p2 + g ( µ , δ , s , p ) ) κ2 + g ( µ , δ , s , p ) G2 ] . ( 76 ) Proof . Following the proof of Theorem 1 and the update rule Eq . ( 61 ) , we have a similar inequality as Eq . ( 41 ) by choosing η ≤ 1−µ9L : Etf ( zt+1 ) − f ( zt ) ≤ η 1− µ ( −1 2 ‖∇f ( zt ) ‖22 + L2 2K K∑ k=1 ‖zt − x ( k ) t ‖22 ) + Lη2 2 ( 1− µ ) 2 ( 3σ2 K + 3L2 K K∑ k=1 ‖zt − x ( k ) t ‖22 + 3‖∇f ( zt ) ‖22 ) = − η 2 ( 1− µ ) ( 1− 3Lη 1− µ ) ‖∇f ( zt ) ‖22 + L2η 2 ( 1− µ ) K ( 1 + 3Lη 1− µ ) K∑ k=1 ‖zt − x ( k ) t ‖22 + 3Lη2σ2 2 ( 1− µ ) 2K ≤ − η 3 ( 1− µ ) ‖∇f ( zt ) ‖22 + 2ηL2 3 ( 1− µ ) K K∑ k=1 ‖zt − x ( k ) t ‖22 + 3Lη2σ2 2 ( 1− µ ) 2K . ( 77 ) Then for the averaged parameters 1K ∑K k=1 x ( k ) t , ‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 2‖∇f ( 1 K K∑ k=1 x ( k ) t ) −∇f ( zt ) ‖22 + 2‖∇f ( zt ) ‖22 ≤ 2L2‖ 1 K K∑ k=1 e ( k ) t−sp − pt‖22 + 2‖∇f ( zt ) ‖22 ≤ 4L2‖ 1 K K∑ k=1 e ( k ) t−sp‖22 + 4L2‖pt‖22 + 2‖∇f ( zt ) ‖22 . ( 78 ) Therefore 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( 1− µ ) [ f ( z0 ) − f ( zT ) ] ηT + 9Lησ2 ( 1− µ ) K + 4L2 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 + 4L2 T T−1∑ t=0 E‖ 1 K K∑ k=1 e ( k ) t−sp‖22 + 4L2 T T−1∑ t=0 E‖pt‖22 ≤ 6 ( 1− µ ) ( f ( x0 ) − f∗ ) ηT + 9Lησ2 ( 1− µ ) K + 4η2L2 ( 1− µ ) 2 [ ( 4 ( s+ 1 ) p+ g ( µ , δ , s , p ) ) σ2 + ( 12 ( s+ 1 ) 2p2 + g ( µ , δ , s , p ) ) κ2 + g ( µ , δ , s , p ) G2 ] . ( 79 ) where the last inequality follows Lemmas 10 , 11 and 12 and g ( µ , δ , s , p ) = 15 ( 1−µ ) 2 + 60 ( 1−δ ) ( s+1 ) 2p2 δ2 . G PROOF OF THEOREM 4 Lemma 13 . For OLCO3-TC with Momentum SGD and under Assumptions 2 , 3 , 4 , and 5 , the local error satisfies E‖e ( k ) t ‖22 ≤ 12 ( 1− δ ) ( 1− µ ) 2δ2 p2η2 ( σ2 + κ2 +G2 ) . ( 80 ) Proof . Same as the proof of Lemma 11 , except that e ( k ) ( St−s−1 ) p is replaced with e ( k ) ( St−1 ) p. Lemma 14 . For OLCO3-TC with vanilla SGD and under Assumptions 2 , 3 , 4 , and 5 , the server error satisfies E‖et‖22 ≤ 96 ( 2− δ ) ( 1− δ ) ( 1− µ ) 2δ4 p2η2 ( σ2 + κ2 +G2 ) . ( 81 ) Proof . Let St = b tpc . Following the proof of Lemma 11 we have E‖∆ ( k ) Stp ‖22 ≤ 3 ( 1+ 1ρ ) 1− ( 1−δ ) ( 1+ρ ) p2η2 ( σ2+κ2+G2 ) ( 1−µ ) 2 . Therefore , E‖et‖22 ≤ 2 ( 2− δ ) ( 1− δ ) ( 1 + 1 ρ ) 1 K K∑ k=1 E‖∆ ( k ) Stp‖ 2 2 + ( 1− δ ) ( 1 + ρ ) E‖e ( St−1 ) p‖ 2 2 ≤ 2 ( 2− δ ) ( 1− δ ) ( 1 + 1 ρ ) 3 ( 1 + 1ρ ) 1− ( 1− δ ) ( 1 + ρ ) p2η2 ( σ2 + κ2 +G2 ) ( 1− µ ) 2 + ( 1− δ ) ( 1 + ρ ) E‖e ( St−1 ) p‖ 2 2 ≤ 6 ( 2− δ ) ( 1− δ ) ( 1 + 1ρ ) 2 [ 1− ( 1− δ ) ( 1 + ρ ) ] 2 ( 1− µ ) 2 p2η2 ( σ2 + κ2 +G2 ) , ( 82 ) where the first inequality follows the proof of Lemma 7 . Let ρ = δ2 ( 1−δ ) such that 1+ 1 ρ = 2−δ δ ≤ 2 δ , then E‖e ( k ) t ‖22 ≤ E‖e ( k ) t ‖22 ≤ 96 ( 2−δ ) ( 1−δ ) ( 1−µ ) 2δ4 p 2η2 ( σ2 + κ2 +G2 ) . Lemma 15 . For OLCO3-TC with Momentum SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ 1−µ√ 72L ( s+1 ) p , we have 1 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 ≤ 3η2σ2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + ( s+ 1 ) p+ 72h ( δ ) p2 ) + 3η2κ2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + 3 ( s+ 1 ) 2p2 + 72h ( δ ) p2 ) + 3η2G2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + 72h ( δ ) p2 ) , ( 83 ) where h ( δ ) = 1−δδ2 ( 1 + 4 ( 2−δ ) δ2 ) . Proof . Let St = b tpc , 1 K K∑ k=1 E‖zt − x ( k ) t ‖22 = 1 K K∑ k=1 E‖ 1 K K∑ k′=1 ( − s−1∑ i=0 C ( ∆ ( k ′ ) ( St−i ) p ) − ( x ( k′ ) Stp − x ( k ′ ) t ) ) − ( − s−1∑ i=0 C ( ∆ ( k ) ( St−i ) p ) − ( x ( k′ ) Stp − x ( k ) t ) − 1 K K∑ k′=1 e ( k ) t − et−sp‖22 , ( 84 ) where s−1∑ i=0 C ( ∆ ( k ) ( St−i ) p ) + ( x ( k ) Stp − x ( k ) t ) = s−1∑ i=0 [ ∆ ( k ) ( St−i ) p − e ( k ) ( St−i ) p ] + ( x ( k ) Stp − x ( k ) t ) = s−1∑ i=0 [ m ( k ) ( St−i−1 ) p p−1∑ τ=0 µτ+1 + ( St−i ) p−1∑ t′= ( St−i−1 ) p η∇F ( x ( k ) t′ ; ξ ( k ) t′ ) ( St−i ) p−1−t′∑ τ=0 µτ + e ( k ) ( St−i−1 ) p − e ( k ) ( St−i ) p ] + m ( k ) Stp t−1−Stp∑ τ=0 µτ+1 + t−1∑ t′=Stp η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) t−1−t′∑ τ=0 µτ = m ( k ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1 + t−1∑ t′= ( St−s ) p η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) t−1−t′∑ τ=0 µτ + e ( k ) ( St−s ) p − e ( k ) Stp . ( 85 ) Therefore , 1 K K∑ k=1 E‖zt − x ( k ) t ‖22 = 1 K K∑ k=1 E‖ 1 K K∑ k′=1 ηm ( k ′ ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1 − ηm ( k ) ( St−s ) p t−1− ( St−s ) p∑ τ=0 µτ+1 + 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p η∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) t−1−t′∑ τ=0 µτ − t−1∑ t′= ( St−s ) p η∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) t−1−t′∑ τ=0 µτ + 1 K K∑ k′=1 e ( k ′ ) ( St−s ) p − e ( k ) ( St−s ) p + e ( k ) Stp + et−sp‖22 = 3η2 ( 1− µ ) 2K K∑ k=1 E‖m ( k ) ( St−s ) p‖ 2 2 + 3 K K∑ k=1 E‖ 1 K K∑ k′=1 e ( k ′ ) ( St−s ) p − e ( k ) ( St−s ) p + e ( k ) Stp + et−sp‖22 + 3η2 K K∑ k=1 E‖ 1 K K∑ k′=1 t−1∑ t′= ( St−s ) p ∇Fk′ ( x ( k ′ ) t′ ; ξ ( k′ ) t′ ) t−1−t′∑ τ=0 µτ − t−1∑ t′= ( St−s ) p ∇Fk ( x ( k ) t′ ; ξ ( k ) t′ ) t−1−t′∑ τ=0 µτ‖22 , ( 86 ) where the first term is bounded following Lemma 9 and the third term is bounded following Eq . ( 72 ) . The second term 3 K K∑ k=1 E‖ 1 K K∑ k′=1 e ( k ′ ) ( St−s ) p − e ( k ) ( St−s ) p + e ( k ) Stp + et−sp‖22 ≤ 9 K K∑ k=1 E‖e ( k ) ( St−s ) p‖ 2 2 + 9 K K∑ k=1 E‖e ( k ) Stp‖ 2 2 + 9 K K∑ k=1 E‖e ( St−s ) p‖ 2 2 ≤ 1− δ ( 1− µ ) 2δ2 ( 1 + 4 ( 2− δ ) δ2 ) · 216p2η2 ( σ2 + κ2 +G2 ) . ( 87 ) Combine these bounds , 1 K K∑ k=1 E‖zt − x ( k ) t ‖22 ≤ 9η2 ( σ2 + κ2 +G2 ) ( 1− µ ) 4 + 1− δ ( 1− µ ) 2δ2 ( 1 + 4 ( 2− δ ) δ2 ) · 216p2η2 ( σ2 + κ2 +G2 ) + 3η2 ( s+ 1 ) pσ2 ( 1− µ ) 2 + 3η2 ( s+ 1 ) p ( 1− µ ) 2 t−1∑ t′=t− ( s+1 ) p ( 6L2 K K∑ k=1 E‖zt′ − x ( k ) t′ ‖ 2 2 + 3κ 2 ) = 18η2L2 ( s+ 1 ) p ( 1− µ ) 2 t−1∑ t′=t− ( s+1 ) p 1 K K∑ k=1 E‖zt′ − x ( k ) t′ ‖ 2 2 ( 88 ) Sum the above inequality from t = 0 to t = T − 1 , divide it by T , and choose η ≤ 1−µ√ 72L ( s+1 ) p , 1 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 ≤ 3η2σ2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + ( s+ 1 ) p+ 72h ( δ ) p2 ) + 3η2κ2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + 3 ( s+ 1 ) 2p2 + 72h ( δ ) p2 ) + 3η2G2 ( 1− µ ) 2 ( 3 ( 1− µ ) 2 + 72h ( δ ) p2 ) . ( 89 ) Theorem 4 . For OLCO3-TC with Momentum SGD and under Assumptions 1 , 2 , 3 , 4 , and 5 , if the learning rate η ≤ min { 1−µ√ 72L ( s+1 ) p , 1−µ9L } and let h ( δ ) = 1−δ δ2 ( 1 + 4 ( 2−δ ) δ2 ) , then 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ ≤ 6 ( 1− µ ) ( f ( x0 ) − f∗ ) ηT + 9Lησ2 ( 1− µ ) K + 6η2L2 ( 1− µ ) 2 [ σ2 ( 9 ( 1− µ ) 2 + 2 ( s+ 1 ) p+ 168h ( δ ) p2 ) + κ2 ( 9 ( 1− µ ) 2 + 6 ( s+ 1 ) 2p2 + 168h ( δ ) p2 ) +G2 ( 9 ( 1− µ ) 2 + 168h ( δ ) p2 ) ] . ( 90 ) Proof . Following the proof of Theorem 3 , Etf ( zt+1 ) −f ( zt ) ≤ − η 3 ( 1− µ ) ‖∇f ( zt ) ‖22 + 2ηL2 3 ( 1− µ ) K K∑ k=1 ‖zt−x ( k ) t ‖22 + 3Lη2σ2 2 ( 1− µ ) 2K , ( 91 ) ‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 2‖∇f ( 1 K K∑ k=1 x ( k ) t ) −∇f ( zt ) ‖22 + 2‖∇f ( zt ) ‖22 ≤ 2L2‖ 1 K K∑ k=1 e ( k ) t + et−sp − pt‖22 + 2‖∇f ( zt ) ‖22 ≤ 6L2‖ 1 K K∑ k=1 e ( k ) t−sp‖22 + 6L2‖et−sp‖22 + 6L2‖pt‖22 + 2‖∇f ( zt ) ‖22 . ( 92 ) Therefore 1 T T−1∑ t=0 E‖∇f ( 1 K K∑ k=1 x ( k ) t ) ‖22 ≤ 6 ( 1− µ ) [ f ( z0 ) − f ( zT ) ] ηT + 9Lησ2 ( 1− µ ) K + 4L2 KT T−1∑ t=0 K∑ k=1 E‖zt − x ( k ) t ‖22 + 6L2 T T−1∑ t=0 E‖ 1 K K∑ k=1 e ( k ) t−sp‖22 + 6L2 T T−1∑ t=0 E‖et−sp‖22 + 6L2 T T−1∑ t=0 E‖pt‖22 ≤ 6 ( 1− µ ) ( f ( x0 ) − f∗ ) ηT + 9Lησ2 ( 1− µ ) K + 6η2L2 ( 1− µ ) 2 [ σ2 ( 9 ( 1− µ ) 2 + 2 ( s+ 1 ) p+ 168h ( δ ) p2 ) + κ2 ( 9 ( 1− µ ) 2 + 6 ( s+ 1 ) 2p2 + 168h ( δ ) p2 ) +G2 ( 9 ( 1− µ ) 2 + 168h ( δ ) p2 ) ] , ( 93 ) where the last inequality follows Lemmas 10 , 13 , 14 and 15 .

Summary: Review: This paper studies distributed training of neural networks. The major obstacles in distributed training are communication costs and communication delays. In the literature there exists different methods which attempt to overcome these two issues but, as far as the authors claim, none of the existing algorithms succeeds in dealing with both these aspects at the same time. The authors propose a novel distributed method OLCO_3 which is designed to address both communication costs and communication delays. In particular, the authors propose two variants of the method: OLCO_3 TC which comprises pipelining and compensation, and OLCO_3 VQ which comprises pipelining with communication-dependent compressor. Both the versions of OLCO_3 are analyzed from a theoretical perspective: under some assumptions the authors conduct a theoretical analysis of the convergence of the proposed schemes. Finally, the method in its two variants is benchmarked and compared with state-of-the-art distributed algorithms.