Preserved central model for faster bidirectional compression in distributed settings

1 Introduction . Large scale distributed machine learning is widely used in many modern applications [ 1 , 6 , 28 ] . The training is distributed over a potentially large number N of workers that communicate either with a central server [ see 17 , 22 , on federated learning ] , or using peer-to-peer communication [ 9 , 34 , 32 ] . In this work , we consider a setting using a central server that aggregates updates from remote nodes . Formally , we have a number of features d ∈ N∗ , and a convex cost function F : Rd → R. We want to solve the following distributed convex optimization problem using stochastic gradient algorithms [ 25 , 5 ] : minw∈Rd F ( w ) with F ( w ) = 1N ∑N i=1 Fi ( w ) , where ( Fi ) N i=1 is a local risk function ( empirical risk or expected risk in a streaming framework ) . This applies to both instances of distributed and federated learning . An important issue of those frameworks is the high communication cost between the workers and the central server [ 16 , Sec . 3.5 ] . This cost is a concern from several points of view . First , exchanging information can be the bottleneck in terms of speed . Second , the data consumption and the bandwidth usage of training large distributed models can be problematic ; and furthermore , the energetic and environmental impact of those exchanges is a growing concern . Over the last few years , new algorithms were introduced , compressing messages in the upload communications ( i.e. , from remote devices to the central server ) in order to reduce the size of those exchanges [ 29 , 3 , 36 , 2 , 35 , 31 , 30 , 23 , 18 ] . More recently , a new trend has emerged to also compress the downlink communication : this is bidirectional compression . The necessity for bidirectional compression can depend on the situation . For example , a single uplink compression could be sufficient in asymmetric regimes in which broadcasting a message to N workers ( “ one to N ” ) is faster than aggregating the information coming from each node ( “ N to one ” ) . However , in other regimes , e.g . with few machines , where the bottleneck is the transfer time of a heavy model ( up to several GB in modern Deep Learning architectures ) the downlink communication can not be disregarded , as the upload and download speed are of the same order [ 24 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Furthermore , in a situation in which participants have to systematically download an update ( e.g. , on their smartphones ) to participate in the training , participants would prefer to receive a small size update ( compressed ) rather than a heavier one . To encompass all situations , we consider algorithms for which the information exchanged is compressed in both directions . To perform downlink communication , existing bidirectional algorithms [ 33 , 38 , 26 , 19 , 24 , 14 , 37 , 11 ] first aggregate all the information they have received , compress them and then carry out the broadcast . Both the main “ global ” model and the “ local ” ones perform the same update with this compressed information . Consequently , the model hold on the central server and the one used on the local workers ( to query the gradient oracle ) are identical . However , this means that the model on the central server has been artificially degraded : instead of using all the information it has received , it is updated with the compressed information . Here , we focus on preserving ( instead of degrading ) the central model : the update made on its side does not depend on the downlink compression . This implies that the local models are different from the central model . The local gradients are thus measured on a “ perturbed model ” ( or “ perturbed iterate ” ) : such an approach requires a more involved analysis and the algorithm must be carefully designed to control the deviation between the local and global models [ 21 ] . For example , algorithms directly compressing the model or the update would simply not converge . We propose MCM - Model Compression with Memory - a new algorithm that 1 ) preserves the central model , and 2 ) uses a memory scheme to reduce the variance of the local model . We prove that the convergence of this method is similar to the one of algorithms using only unidirectional compression . Potential Impact . Proposing an analysis that handles perturbed iterates is the key to unlock three major challenges of distributed learning run with bidirectionally compressed gradients . First , we show that it is possible to improve the convergence rate by sending different randomized models to the different workers , this is Rand-MCM . Secondly , this analysis also paves the way to deal with partially participating machines : the adaptation of Rand-MCM to this framework is straightforward ; while adapting existing algorithms [ 26 ] to partial participation is not practical . Thirdly , this framework is also promising in terms of business applications , e.g. , in the situation of learning with privacy guarantees and with a trusted central server . We detail those three possible extensions in Subsection 4.1 . Broader impact . This work is aligned with a global effort to make the usage of large scale Federated Learning sustainable by minimizing its environmental impact . Though the impact of such algorithms is expected to be positive , at least on environmental concerns , cautiousness is still required , as a rebound effect may be observed [ 12 ] : having energetically cheaper and faster algorithms may result in an increase of such applications , annihilating the gain made by algorithmic progress . Contributions . We make the following contributions : 1 . We propose a new algorithm MCM , combining a memory process to the “ preserved ” update . To convey the key steps of the proof , we also introduce an auxiliary hypothetical algorithm , Ghost . 2 . For those algorithms , we carefully control the variance of the local models w.r.t . the global one . We provide a contraction equation involving the control on the local model ’ s variance and show that MCM achieves the same rate of convergence as single compression in strongly-convex , convex and non-convex regimes . We give a comparisons of MCM ’ s rates with existing algorithms in Table 2 . 3 . We propose a variant , Rand-MCM incorporating diversity into models shared with the local workers and show that it improves convergence for quadratic functions . This is the first algorithm for double compression to focus on a preserved central model . We underline , both theoretically and in practice , that we get the same asymptotic convergence rate for simple and double compression - which is a major improvement . Our approach is one of the first to allow for worker dependent model , and to naturally adapt to worker dependent compression levels . The rest of the paper is organized as follows : in Section 2 we present the problem statement and introduce MCM and Rand-MCM . Theoretical results on these algorithms are successively presented in Sections 3 and 4 . Finally , we present experiments supporting the theory in Section 5 . 2 Problem statement . We consider the minimization problem described in Section 1 . In the convex case , we assume there exists an optimal parameter w∗ , and denote F∗ = F ( w∗ ) . We use ‖·‖ to denote the Euclidean norm . To solve this problem , we rely on a stochastic gradient descent ( SGD ) algorithm . A stochastic gradient gik+1 is provided at iteration k in N to the device i in J1 , NK . This gradient oracle can be computed on a mini-batch of size b . This function is then evaluated at point wk . In the classical centralized framework ( without compression ) , for a learning rate γ , SGD corresponds to : wk+1 = wk − γ 1 N N∑ i=1 gik+1 ( wk ) . ( 1 ) We now describe the framework used for compression . 2.1 Bidirectional compression framework . Bidirectional compression consists in compressing communications in both directions between the central server and remote devices . We use two different compression operators , respectively Cup and Cdwn to compress the message in each direction . Roughly speaking , the update in eq . ( 1 ) becomes : wk+1 = wk − γCdwn ( 1 N N∑ i=1 Cup ( gik+1 ( wk ) ) ) . However , this approach has a major drawback . The central server receives and aggregates information 1 N ∑N i=1 Cup ( gik+1 ( wk ) ) . But in order to be able to broadcast it back , it compresses it , before applying the update . We refer to this strategy as the “ degraded update ” approach . Its major advantage is simplicity , and it was used in all previous papers performing double compression . Yet , it appears to be a waste of valuable information . In this paper , we update the global model wk+1 independently of the downlink compression : { wk+1 = wk − γ 1N ∑N i=1 Cup ( gik+1 ( ŵk ) ) . ŵk+1 = Cdwn ( wk+1 ) ( 2 ) However , bluntly compressing wk+1 in eq . ( 2 ) hinders convergence , thus the second part of the update needs to be refined by adding a memory mechanism . We now describe both communication stages of the real MCM , which is entirely defined by the following uplink and downlink equations . Downlink Uplink { Ωk+1 = wk+1 −Hk , ŵk+1 = Hk + Cdwn ( Ωk+1 ) Hk+1 = Hk + αdwnCdwn ( Ωk+1 ) . ∀i ∈ J1 , NK , ∆ik = gik+1 ( ŵk ) − hik wk+1 = wk − γN ∑N i=1 Cup ( ∆ik ) + hik hik+1 = h i k + αupCup ( ∆ik ) . ( 3 ) Downlink Communication . We introduce a downlink memory term ( Hk ) k , which is available on both workers and central server . The difference Ωk+1 between the model and this memory is compressed and exchanged , then the local model is reconstructed from this information . The memory is then updated as defined on left part of eq . ( 3 ) , with a learning rate αdwn . Introducing this memory mechanism is crucial to control the variance of the local model ŵk+1 . To the best of our knowledge MCM is the first algorithm that uses such a memory mechanism for downlink compression . This mechanism was introduced by Mishchenko et al . [ 23 ] for the uplink compression but with the other purpose of mitigating the impact of heterogeneity , while we use it here to avoid divergence of the local model ’ s variance . Uplink Communication . The motivation to introduce an uplink memory term hik for each device i ∈ J1 , NK is different , and better understood . Indeed , for the uplink direction , this mechanism is only necessary ( and then crucial ) to handle heterogeneous workers [ i.e. , with different data distributions , see e.g . 24 ] . Here , the difference ∆ik between the stochastic gradient g i k+1 at the local model ŵk ( as defined in eq . ( 3 ) ) and the memory term is compressed and exchanged . The memory is then updated as defined on right part of eq . ( 3 ) with a rate αdwn . Remark 1 ( Rate αdwn ) . It is necessary to use αdwn < 1 . Otherwise , the compression noise tends to propagate and is amplified , because of the multiplicative nature of the compression . In Figure 1 we compare MCM , with 3 other strategies : compressing only the update , compressing wk − ŵk−1 , ( i.e. , αdwn = 1 ) , and compressing the model ( i.e. , Hk = 0 ) , showing that only MCM converges . Remark 2 ( Memory vs Error Feedback ) . Error feedback is another technique , introduced by Seide et al . [ 29 ] . In the context of double compression , it has been shown to improve convergence for a restrictive class of contracting compression operators ( which are generally biased ) by Zheng et al . [ 38 ] , Tang et al . [ 33 ] . However , we note several differences to our approach . ( 1 ) For unbiased operators - as considered in Dore , it did not lead to any theoretical improvement [ Remark 2 in Sec . 4.1. , 19 ] . ( 2 ) Moreover , only a fraction ( namely ( 1 + ωdwn ) −1 ) of the “ error ” wk+1 − ŵk+1 can be preserved in the EF term ( see line 18 in algo 1 in Liu et al. ) . It is thus impossible to recover the central preserved model as a function of the degraded model and the EF term . ( 3 ) [ 38 ] consider a biased operator and the same compression level for uplink and downlink compression . They also rely on stronger assumptions on the gradient ( uniformly bounded ) and only tackle the homogeneous case . In Table 1 we summarize the main algorithms for compression in distributed training . As downlink communication can be more efficient than uplink , we consider distinct operators Cdwn , Cup and allow the corresponding compressions levels to be distinct : those quantities are defined in Assumption 1 . Assumption 1 . There exists constants ωup , ωdwn ∈ R∗+ , such that the compression operators Cup and Cdwn satisfy the two following properties for all w in Rd : E [ Cup/dwn ( w ) ] = w , and E [ ‖Cup/dwn ( w ) − w‖2 ] ≤ ωup/dwn‖w‖2 . The higher is ω , the more aggressive the compression is . We only consider unbiased operators , that encompass sparsification , quantization and sketching . References and a discussion on those operators , and possible extensions of our results to biased operators are provided in Appendix A.1 . Remark 3 ( Related work on Perturbed iterate analysis ) . The theory of perturbed iterate analysis was introduced by Mania et al . [ 21 ] to deal with asynchronous SGD . More recently , it was used by Stich and Karimireddy [ 30 ] , Gorbunov et al . [ 11 ] to analyze the convergence of algorithms with uplink compressions , error feedback and asynchrony . Using gradients at randomly perturbed points can also be seen as a form of randomized smoothing [ 27 ] , a point we discuss in Appendix A.2 . 2.2 The randomization mechanism , Rand-MCM In this subsection , we describe the key feature introduced in Rand-MCM : randomization . It consists in performing an independent compression for each device instead of performing a single one for all of them . As a consequence , each worker holds a different model centered around the global one . This introduces some supplementary randomness that stabilizes the algorithm . Formally , we will consider N mutually independent compression operators Cdwn , i instead of a single one Cdwn , and the central server will send to the device i at iteration k + 1 the compression of the difference between its model and the local memory on worker i : Cdwn , i ( wk+1 −Hik ) . The tradeoffs associated with this modification are discussed in Section 4 . The pseudocode of Rand-MCM is given in Algorithm 1 in Appendix A . It incorporates all components described above : 1 ) the bidirectional compression , 2 ) the model update using the non-degraded point , 3 ) the two memories , 4 ) the up and down compression operators , 5 ) the randomization mechanism .

Summary: The paper proposes a bidirectional compression algorithm for parameter-server-based distributed training. Two variants, MCM and Rand-MCM, are proposed to achieve the same improvement as the uplink-compression-only compression algorithm, where the downlink-compression-only impacts local models while the global model is preserved. The Rand-MCM also introduces the concept of (simulated) random smoothing via model compression.