Preserved central model for faster bidirectional compression in distributed settings

1 Introduction . Large scale distributed machine learning is widely used in many modern applications [ 1 , 6 , 28 ] . The training is distributed over a potentially large number N of workers that communicate either with a central server [ see 17 , 22 , on federated learning ] , or using peer-to-peer communication [ 9 , 34 , 32 ] . In this work , we consider a setting using a central server that aggregates updates from remote nodes . Formally , we have a number of features d ∈ N∗ , and a convex cost function F : Rd → R. We want to solve the following distributed convex optimization problem using stochastic gradient algorithms [ 25 , 5 ] : minw∈Rd F ( w ) with F ( w ) = 1N ∑N i=1 Fi ( w ) , where ( Fi ) N i=1 is a local risk function ( empirical risk or expected risk in a streaming framework ) . This applies to both instances of distributed and federated learning . An important issue of those frameworks is the high communication cost between the workers and the central server [ 16 , Sec . 3.5 ] . This cost is a concern from several points of view . First , exchanging information can be the bottleneck in terms of speed . Second , the data consumption and the bandwidth usage of training large distributed models can be problematic ; and furthermore , the energetic and environmental impact of those exchanges is a growing concern . Over the last few years , new algorithms were introduced , compressing messages in the upload communications ( i.e. , from remote devices to the central server ) in order to reduce the size of those exchanges [ 29 , 3 , 36 , 2 , 35 , 31 , 30 , 23 , 18 ] . More recently , a new trend has emerged to also compress the downlink communication : this is bidirectional compression . The necessity for bidirectional compression can depend on the situation . For example , a single uplink compression could be sufficient in asymmetric regimes in which broadcasting a message to N workers ( “ one to N ” ) is faster than aggregating the information coming from each node ( “ N to one ” ) . However , in other regimes , e.g . with few machines , where the bottleneck is the transfer time of a heavy model ( up to several GB in modern Deep Learning architectures ) the downlink communication can not be disregarded , as the upload and download speed are of the same order [ 24 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Furthermore , in a situation in which participants have to systematically download an update ( e.g. , on their smartphones ) to participate in the training , participants would prefer to receive a small size update ( compressed ) rather than a heavier one . To encompass all situations , we consider algorithms for which the information exchanged is compressed in both directions . To perform downlink communication , existing bidirectional algorithms [ 33 , 38 , 26 , 19 , 24 , 14 , 37 , 11 ] first aggregate all the information they have received , compress them and then carry out the broadcast . Both the main “ global ” model and the “ local ” ones perform the same update with this compressed information . Consequently , the model hold on the central server and the one used on the local workers ( to query the gradient oracle ) are identical . However , this means that the model on the central server has been artificially degraded : instead of using all the information it has received , it is updated with the compressed information . Here , we focus on preserving ( instead of degrading ) the central model : the update made on its side does not depend on the downlink compression . This implies that the local models are different from the central model . The local gradients are thus measured on a “ perturbed model ” ( or “ perturbed iterate ” ) : such an approach requires a more involved analysis and the algorithm must be carefully designed to control the deviation between the local and global models [ 21 ] . For example , algorithms directly compressing the model or the update would simply not converge . We propose MCM - Model Compression with Memory - a new algorithm that 1 ) preserves the central model , and 2 ) uses a memory scheme to reduce the variance of the local model . We prove that the convergence of this method is similar to the one of algorithms using only unidirectional compression . Potential Impact . Proposing an analysis that handles perturbed iterates is the key to unlock three major challenges of distributed learning run with bidirectionally compressed gradients . First , we show that it is possible to improve the convergence rate by sending different randomized models to the different workers , this is Rand-MCM . Secondly , this analysis also paves the way to deal with partially participating machines : the adaptation of Rand-MCM to this framework is straightforward ; while adapting existing algorithms [ 26 ] to partial participation is not practical . Thirdly , this framework is also promising in terms of business applications , e.g. , in the situation of learning with privacy guarantees and with a trusted central server . We detail those three possible extensions in Subsection 4.1 . Broader impact . This work is aligned with a global effort to make the usage of large scale Federated Learning sustainable by minimizing its environmental impact . Though the impact of such algorithms is expected to be positive , at least on environmental concerns , cautiousness is still required , as a rebound effect may be observed [ 12 ] : having energetically cheaper and faster algorithms may result in an increase of such applications , annihilating the gain made by algorithmic progress . Contributions . We make the following contributions : 1 . We propose a new algorithm MCM , combining a memory process to the “ preserved ” update . To convey the key steps of the proof , we also introduce an auxiliary hypothetical algorithm , Ghost . 2 . For those algorithms , we carefully control the variance of the local models w.r.t . the global one . We provide a contraction equation involving the control on the local model ’ s variance and show that MCM achieves the same rate of convergence as single compression in strongly-convex , convex and non-convex regimes . We give a comparisons of MCM ’ s rates with existing algorithms in Table 2 . 3 . We propose a variant , Rand-MCM incorporating diversity into models shared with the local workers and show that it improves convergence for quadratic functions . This is the first algorithm for double compression to focus on a preserved central model . We underline , both theoretically and in practice , that we get the same asymptotic convergence rate for simple and double compression - which is a major improvement . Our approach is one of the first to allow for worker dependent model , and to naturally adapt to worker dependent compression levels . The rest of the paper is organized as follows : in Section 2 we present the problem statement and introduce MCM and Rand-MCM . Theoretical results on these algorithms are successively presented in Sections 3 and 4 . Finally , we present experiments supporting the theory in Section 5 . 2 Problem statement . We consider the minimization problem described in Section 1 . In the convex case , we assume there exists an optimal parameter w∗ , and denote F∗ = F ( w∗ ) . We use ‖·‖ to denote the Euclidean norm . To solve this problem , we rely on a stochastic gradient descent ( SGD ) algorithm . A stochastic gradient gik+1 is provided at iteration k in N to the device i in J1 , NK . This gradient oracle can be computed on a mini-batch of size b . This function is then evaluated at point wk . In the classical centralized framework ( without compression ) , for a learning rate γ , SGD corresponds to : wk+1 = wk − γ 1 N N∑ i=1 gik+1 ( wk ) . ( 1 ) We now describe the framework used for compression . 2.1 Bidirectional compression framework . Bidirectional compression consists in compressing communications in both directions between the central server and remote devices . We use two different compression operators , respectively Cup and Cdwn to compress the message in each direction . Roughly speaking , the update in eq . ( 1 ) becomes : wk+1 = wk − γCdwn ( 1 N N∑ i=1 Cup ( gik+1 ( wk ) ) ) . However , this approach has a major drawback . The central server receives and aggregates information 1 N ∑N i=1 Cup ( gik+1 ( wk ) ) . But in order to be able to broadcast it back , it compresses it , before applying the update . We refer to this strategy as the “ degraded update ” approach . Its major advantage is simplicity , and it was used in all previous papers performing double compression . Yet , it appears to be a waste of valuable information . In this paper , we update the global model wk+1 independently of the downlink compression : { wk+1 = wk − γ 1N ∑N i=1 Cup ( gik+1 ( ŵk ) ) . ŵk+1 = Cdwn ( wk+1 ) ( 2 ) However , bluntly compressing wk+1 in eq . ( 2 ) hinders convergence , thus the second part of the update needs to be refined by adding a memory mechanism . We now describe both communication stages of the real MCM , which is entirely defined by the following uplink and downlink equations . Downlink Uplink { Ωk+1 = wk+1 −Hk , ŵk+1 = Hk + Cdwn ( Ωk+1 ) Hk+1 = Hk + αdwnCdwn ( Ωk+1 ) . ∀i ∈ J1 , NK , ∆ik = gik+1 ( ŵk ) − hik wk+1 = wk − γN ∑N i=1 Cup ( ∆ik ) + hik hik+1 = h i k + αupCup ( ∆ik ) . ( 3 ) Downlink Communication . We introduce a downlink memory term ( Hk ) k , which is available on both workers and central server . The difference Ωk+1 between the model and this memory is compressed and exchanged , then the local model is reconstructed from this information . The memory is then updated as defined on left part of eq . ( 3 ) , with a learning rate αdwn . Introducing this memory mechanism is crucial to control the variance of the local model ŵk+1 . To the best of our knowledge MCM is the first algorithm that uses such a memory mechanism for downlink compression . This mechanism was introduced by Mishchenko et al . [ 23 ] for the uplink compression but with the other purpose of mitigating the impact of heterogeneity , while we use it here to avoid divergence of the local model ’ s variance . Uplink Communication . The motivation to introduce an uplink memory term hik for each device i ∈ J1 , NK is different , and better understood . Indeed , for the uplink direction , this mechanism is only necessary ( and then crucial ) to handle heterogeneous workers [ i.e. , with different data distributions , see e.g . 24 ] . Here , the difference ∆ik between the stochastic gradient g i k+1 at the local model ŵk ( as defined in eq . ( 3 ) ) and the memory term is compressed and exchanged . The memory is then updated as defined on right part of eq . ( 3 ) with a rate αdwn . Remark 1 ( Rate αdwn ) . It is necessary to use αdwn < 1 . Otherwise , the compression noise tends to propagate and is amplified , because of the multiplicative nature of the compression . In Figure 1 we compare MCM , with 3 other strategies : compressing only the update , compressing wk − ŵk−1 , ( i.e. , αdwn = 1 ) , and compressing the model ( i.e. , Hk = 0 ) , showing that only MCM converges . Remark 2 ( Memory vs Error Feedback ) . Error feedback is another technique , introduced by Seide et al . [ 29 ] . In the context of double compression , it has been shown to improve convergence for a restrictive class of contracting compression operators ( which are generally biased ) by Zheng et al . [ 38 ] , Tang et al . [ 33 ] . However , we note several differences to our approach . ( 1 ) For unbiased operators - as considered in Dore , it did not lead to any theoretical improvement [ Remark 2 in Sec . 4.1. , 19 ] . ( 2 ) Moreover , only a fraction ( namely ( 1 + ωdwn ) −1 ) of the “ error ” wk+1 − ŵk+1 can be preserved in the EF term ( see line 18 in algo 1 in Liu et al. ) . It is thus impossible to recover the central preserved model as a function of the degraded model and the EF term . ( 3 ) [ 38 ] consider a biased operator and the same compression level for uplink and downlink compression . They also rely on stronger assumptions on the gradient ( uniformly bounded ) and only tackle the homogeneous case . In Table 1 we summarize the main algorithms for compression in distributed training . As downlink communication can be more efficient than uplink , we consider distinct operators Cdwn , Cup and allow the corresponding compressions levels to be distinct : those quantities are defined in Assumption 1 . Assumption 1 . There exists constants ωup , ωdwn ∈ R∗+ , such that the compression operators Cup and Cdwn satisfy the two following properties for all w in Rd : E [ Cup/dwn ( w ) ] = w , and E [ ‖Cup/dwn ( w ) − w‖2 ] ≤ ωup/dwn‖w‖2 . The higher is ω , the more aggressive the compression is . We only consider unbiased operators , that encompass sparsification , quantization and sketching . References and a discussion on those operators , and possible extensions of our results to biased operators are provided in Appendix A.1 . Remark 3 ( Related work on Perturbed iterate analysis ) . The theory of perturbed iterate analysis was introduced by Mania et al . [ 21 ] to deal with asynchronous SGD . More recently , it was used by Stich and Karimireddy [ 30 ] , Gorbunov et al . [ 11 ] to analyze the convergence of algorithms with uplink compressions , error feedback and asynchrony . Using gradients at randomly perturbed points can also be seen as a form of randomized smoothing [ 27 ] , a point we discuss in Appendix A.2 . 2.2 The randomization mechanism , Rand-MCM In this subsection , we describe the key feature introduced in Rand-MCM : randomization . It consists in performing an independent compression for each device instead of performing a single one for all of them . As a consequence , each worker holds a different model centered around the global one . This introduces some supplementary randomness that stabilizes the algorithm . Formally , we will consider N mutually independent compression operators Cdwn , i instead of a single one Cdwn , and the central server will send to the device i at iteration k + 1 the compression of the difference between its model and the local memory on worker i : Cdwn , i ( wk+1 −Hik ) . The tradeoffs associated with this modification are discussed in Section 4 . The pseudocode of Rand-MCM is given in Algorithm 1 in Appendix A . It incorporates all components described above : 1 ) the bidirectional compression , 2 ) the model update using the non-degraded point , 3 ) the two memories , 4 ) the up and down compression operators , 5 ) the randomization mechanism .

Summary: The paper proposes MCM ~ algorithm to perform bidirectional compression in distributed setting. The authors claim similar convergence guarantees as vanilla setting.  They introduce the notion of sending different models to different clients while keeping the global model preserved.