Smoothness Matrices Beat Smoothness Constants: Better  Communication Compression Techniques for Distributed Optimization

1 Introduction . With the desire to build and train high quality machine learning models comes an increased appetite for larger models , both in terms of the number of parameters encoding them , and in the amount of data required to train them . In the big data regime , the data is partitioned among many parallel machines , which then cooperatively train a single global model , usually orchestrated by a central server . Distributed training is cast as the distributed optimization problem min x∈Rd f ( x ) +R ( x ) , f ( x ) : = 1n n∑ i=1 fi ( x ) , ( 1 ) where d is the number of parameters of model x ∈ Rd , n is the number of machines participating in the training , fi ( x ) is the loss associated with the data stored on machine i ∈ [ n ] : = { 1 , 2 , . . . , n } , f ( x ) is the empirical loss , and R ( x ) is a regularizer . Ample research over the past two decades has shown that first-order methods are highly scalable and as a result are the methods of choice for distributed optimization problems [ Liu and Zhang , 2020 ] . In particular , a substantial amount of work 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . has been devoted to speeding up the training process by developing efficient methods empowered with techniques such as compressed communication , variance reduction and acceleration . Compressed communication . In distributed training , compute nodes have to communicate with each other , often via a central server , in order to be able to maintain consensus and jointly train a global model . However , communication of the information pertaining to local progress , which is typically contained in gradient ( s ) distilled from local data , is almost invariably the key bottleneck in distributed training systems [ Xu et al. , 2020 ] . One popular way to address this issue is to reduce the number of bits encoding the vector/tensor to be transferred via the help of a lossy compression operator . Numerous unbiased gradient compression operators have been proposed for this purpose , including several types of sparsifications [ Wang et al. , 2018 , Mishchenko et al. , 2020 , Alistarh et al. , 2018 ] and quantizations [ Alistarh et al. , 2017 , Zhang et al. , 2017 , Horváth et al. , 2019a , Wu et al. , 2018 ] . Variance reduction . A marked issue that needs to be addressed by successful distributed optimization methods has to do with the ( potential ) “ dissimilarity ” of the local loss functions f1 , . . . , fn , which in turn is due to the heterogeneity of the training data defining these functions . The higher the dissimilarity , the harder it is for the devices to find the minimizer of ( 1 ) . This issue exists even in the unregularized case ( R ≡ 0 ) . Indeed , while in this case 1n ∑ i∇fi ( x∗ ) = 0 if x∗ is a minimizer of f , this does not mean that the individual gradients , ∇f1 ( x∗ ) , . . . , ∇fn ( x∗ ) , are all zero . This shows that local gradient information alone is not enough for any node to “ realize ” that a solution has been found , which encourages further , in this case unnecessary , iterations . If unaddressed properly , an algorithm is forced to use smaller learning rates , and this leads to unnecessarily slow convergence . On the other hand , when a fixed learning rate is used , the rate is fast , but convergence stops in a potentially large neighborhood1 of the optimum x∗ . This issue is exacerbated further by the extra noise coming from gradient compression . Indeed , this noise prevents methods such as Distributed Compressed Gradient Descent ( DCGD ) [ Khirirat et al. , 2018 ] from converging to x∗ with a constant learning rate even in the interpolation regime characterized by the identities ∇fi ( x∗ ) = 0 for all i. Fortunately , these issues can be resolved via carefully designed variance reduction techniques [ Gower et al. , 2020 ] . In particular , the first variance reduction mechanism for removing the variance coming from compression operators in distributed training is due to Mishchenko et al . [ 2019 ] , embodied in their DIANA algorithm . The method was initially analyzed for ternary quantization only [ Wen et al. , 2017 ] , and later generalized to handle a general class of unbiased compression operators [ Horváth et al. , 2019b , Gorbunov et al. , 2020b ] . Acceleration . To speed up distributed training even further , it is often possible to employ Nesterov ’ s acceleration technique [ Nesterov , 1983 , 2004 ] in concert with gradient compression and variance reduction . For instance , Li et al . [ 2020 ] developed the ADIANA method , which adds acceleration on top of a variant of DIANA that relies on the computation of full-batch gradients on all nodes . The resulting method offers provable speedups in convex and strongly convex regimes . Another example is the method ECLK of Qian et al . [ 2020 ] , which employs compressed communication via any ( possibly biased ) compressor satisfying a certain contraction property in combination with a slightly different variance reduction technique known as error compensation [ Stich and Karimireddy , 2019 , Karimireddy et al. , 2019 ] , while acceleration is offered by a loopless variant of the accelerated method Katyusha [ Allen-Zhu , 2017 , Kovalev et al. , 2020 ] . Further tricks . Numerous other techniques are often used to improve some other aspects of distributed training , including implementing multiple local gradient steps before communication [ Stich , 2020 , Karimireddy et al. , 2020 , Woodworth et al. , 2020a ] , asynchronous communication protocols [ Agarwal and Duchi , 2011 , Lian et al. , 2015 , Recht et al. , 2011 ] , in-network aggregation [ Sapio et al. , 2021 ] , and performing the distributed training in a decentralized peer-to-peer manner without the reliance on an orchestrating server [ Koloskova et al. , 2019 , Alghunaim et al. , 2019 ] . However , in this work , we do not explore these directions and focus on the three techniques described before , namely , compressed communication , variance reduction and acceleration . 2 Mining for Smoothness Information . 2.1 . One size fits all . Arguably , one of the most ubiquitous , if not the most ubiquitous , assumptions used in the literature on first-order optimization methods is that of L-smoothness [ Nesterov , 2004 ] . A 1In theR ≡ 0 case , this neighborhood is proportional to the variance of the local gradients at the optimum : 1n ∑n i=1 ‖∇fi ( x ∗ ) ‖2 . differentiable function φ : Rd → R is said to be L-smooth if there exists a constant L ≥ 0 such that φ ( x ) ≤ φ ( y ) + 〈∇φ ( x ) , x− y〉+ L2 ‖x− y‖ 2 ( 2 ) holds for all x , y ∈ Rd . However , most works in the area of finite-sum distributed optimization use it very crudely : they assume that all local loss functions fi as well as their average , f = 1n ∑ i fi , share the same smoothness constant L [ Tang et al. , 2019 , Woodworth et al. , 2020b , Stich , 2020 ] . This is crude because much information is lost . Indeed , assuming that each fi is Li-smooth , it is well known that f is Lf -smooth with Lf satisfying Lf ≤ 1n ∑ i Li . In the light of this , the above assumption is crude as it effectively replaces the values L1 , . . . , Ln and Lf with a single parameter L satisfying L ≥ max { L1 , . . . , Ln } . Since the stepsizes and convergence rates of first-order methods depend on the smoothness constant ( s ) employed , convergence analysis relying on such crude approximation may be significantly suboptimal , and the methods too slow when implemented following the theory . 2.2 . “ According to the work of their hands ” ( Lam 3:64 ) . Significant theoretical and practical improvement can often be obtained when taking account of all the smoothness constants involved , avoiding the practice of replacing them all with a single crude bound . Such analyses are more rare , but fairly common . For example , [ Richtárik and Takáč , 2016a , Hanzely and Richtárik , 2019a ] . 2.3 . “ Like treasure hidden in a field , which a man found and covered up ” ( Mat 13:44 ) . The starting point of this paper is the observation that there is a hitherto untapped richness of smoothness information that can be used to construct better distributed optimization algorithms and obtain better theory . This information is available , but hidden from sight , and is based on the notion of matrix smoothness . Definition 1 ( Matrix Smoothness ) . We say that a differentiable function φ : Rd → R is L-smooth if there exists a symmetric positive semidefinite matrix L 0 such that φ ( x ) ≤ φ ( y ) + 〈∇φ ( y ) , x− y〉+ 12‖x− y‖ 2 L , ∀x , y ∈ Rd . ( 3 ) The standard L-smoothness condition ( 2 ) is obtained as a special case of ( 3 ) for matrices of the form L = LI , where I is the identity matrix . In particular , if function fi appearing in ( 1 ) is often the average loss over the training data stored on node i , i.e. , fi ( x ) = 1 mi ∑mi m=1 φim ( Aimx ) , ( 4 ) where Aim ∈ Rdim×d is a data matrix , and φim : Rdim → R is a differentiable function ( e.g. , the loss over all but the last linear layer of a NN ) . The following simple result from Qu and Richtárik [ 2016b ] , used therein in the context of randomized coordinate descent methods , states that if the loss functions φim are smooth in the standard scalar sense , then fi is smooth in the matrix sense . Lemma 1 . If each φim is λim-smooth , then the function fi defined in ( 4 ) is Li-smooth with Li = 1 mi ∑mi m=1 λimA > imAim . ( 5 ) In cases where the local functions fi are of the form ( 4 ) 2—and it is clear this structure is ubiquitous— there is a lot of potentially useful information contained in the matrix smoothness “ constant ” Li . If we were to use the scalar smoothness constant of fi instead , we would be effectively tossing this richness away , and replacing it with Li = λmax ( Li ) ; the largest eigenvalue of Li . This seems wasteful . As we show in this work , it is . However , we offer a fix . 3 Motivation and Contributions . To the best of our knowledge , none of the current distributed optimization methods , including the methods DCGD [ Khirirat et al. , 2018 ] , DIANA [ Mishchenko et al. , 2019 ] and ADIANA [ Li et al. , 2020 ] discussed in Section 1 , are capable of exploiting the inherently rich data-dependent smoothness structure of the local losses beyond standard smoothness constants . To this effect , we impose the following assumption throughout the paper : Assumption 1 . The functions fi : Rd → R are differentiable , convex , lower bounded3 and Lismooth . Moreover , f is L-smooth with ( standard ) smoothness constant L : = λmax ( L ) . 2Our theoretical results hold for general loss functions fi and do not assume the structure ( 4 ) 3Lower boundedness of fi ( x ) can be dropped if Li 0 is positive definite . This part of the assumption is not a restriction in applications as all loss function are lower bounded . In this paper , we argue that when training supervised models , smoothness matrices ( see Definition 1 ) — information-rich generalizations of the classical and ubiquitous smoothness constants—can and should be exploited for further dramatic gains , both in theory and practice . 3.1 . Unbiased diagonal sketches . We study unbiased diagonal sketches , defined as follows : Definition 2 ( Unbiased diagonal sketch ) . Let S be a random subset of the set of coordinates/features of the model x ∈ Rd we wish to train , i.e. , S ⊆ [ d ] : = { 1 , 2 , . . . , d } . Let S be proper , i.e. , pj : = Prob ( j ∈ S ) > 0 for all coordinates j ∈ [ d ] . We now define a random diagonal matrix ( sketch ) C = CS ∈ Rd×d via C = Diag ( c1 , . . . , cd ) , cj = { 1/pj if j∈S , 0 otherwise . ( 6 ) Note that given a vector x = ( x1 , . . . , xd ) ∈ Rd , we have ( Cx ) j = { xj/pj if j∈S 0 if j /∈S . So , we can control the sparsity level of the product Cx by engineering the properties of the random set S. Also note that E [ Cx ] = x for all x . 3.2 . Data-dependent sparsification operators . In order to further alleviate the communication burden inherent in distributed optimization , we further propose data-dependent sparsification operators that can take full advantage of the smoothness matrices Li associated with the local losses fi . To the best of our knowledge , this is in sharp contrast with the design of all existing tractable compression techniques used in distributed training , which are proposed independently of the training data , and typically based on intuitive or information-theoretic principles . With each node i we associate an unbiased diagonal matrix Ci of the form ( 6 ) . We use this and the smoothness matrix of fi to define a sparsification technique , described next . Definition 3 ( Data-dependent sparsification ) . In situations when the i-th node wished to communicate local gradient ∇fi ( x ) , we ask the node to send the sparse ( =compressed ) vector CiL† 1/2 i ∇fi ( x ) to the server instead . The server then constructs ( =decompresses ) an unbiased estimator of∇fi ( x ) as : gi ( x ) = L 1/2 i CiL †1/2 i ∇fi ( x ) , ( 7 ) where L† 1/2 i denotes the square root of the Moore-Penrose pseudoinverse of Li . Notable differences of our proposed communication protocol when compared with standard sparsification techniques are : i ) we use the smoothness matrix Li , ii ) the compressed vector CiL †1/2 i ∇fi ( x ) is not unbiased , iii ) we devise a separate decompression mechanism ( 7 ) , also involving Li , and this enforces effective unbiasedness . 3.3 . Matrix-smoothness-aware redesign of 3 methods . To showcase the power of our approach , we demonstrate how our matrix-smoothness-aware sparsification technique ( 7 ) can be adapted to DCGD , DIANA and ADIANA , in each case leading to significant communication savings . By doing so , we show that matrix smoothness can be effectively used to speed up communication compression , variance reduction and acceleration , respectively . This results in three novel methods : DCGD+ , DIANA+ , and ADIANA+ ; see Table 1 . 3.4 . Dramatic improvements in complexity results . We perform complexity analyses for our methods and derive convergence rates under matrix smoothness4 ( see Assumption 3 ) and strong convexity assumptions ( see Theorems 2 , 3 and 4 ) . We show that new methods always outperform the originals/baselines , and often dramatically so . To illustrate the potential of our sparsification technique ( 7 ) embedded in the new methods , let all machines i ∈ [ n ] use sketches Ci induced by independent5 samplings Si with probabilities pi ; j : = Prob ( j ∈ Si ) . Then we show that , with optimized probabilities pi ; j , DCGD+ can be O ( min ( n , d ) ) times faster then DCGD ( see Remark 3 ) and DIANA+ can be O ( min ( n , d ) ) times faster than DIANA ( see Remark 4 ) , depending on matrices Li . For the accelerated method , we highlight improvements when condition numbers of subproblems are O ( nd2 ) . We show that ADIANA+ can be faster than the original ADIANA by a factor of O ( √ d ) in high compression regime , and by a factor of O ( √ min ( n , d ) ) in low compression regime ( see Remark 5 ) . Main theoretical results are summarized in Table 2 . 3.5 . Single node case . Specializing our theory to the single machine setting ( n = 1 ) , we design new non-distributed algorithms providing an alternative viewpoint to randomized coordinate descent methods ( see Appendix E ) . 3.6 . Lower bounds . Using matrices as linear compression operators , we further investigate the trade-off between communicated bits and variance induced by the compression ( see Appendix F ) . 3.7 . Experiments . We conduct numerical experiments using LibSVM datasets [ Chang and Lin , 2011 ] , confirming the effectiveness and superiority of our sparsification protocol ( 7 ) over the standard sparsification scheme ( see Section 6 and Section C ) .

Summary: This paper analyzes several optimization concepts: compression, variance reduction, and acceleration, using the assumption of smoothness matrices rather than the smoothness parameter. Sketching matrices are designed as sum_{i\in S} L^{1/2}_i c_i * L^{1/2,dagger}_i where L_i is the "smoothness matrix" associated with the ith data sample. Taking this additional information into account, which for generalized linear models is not too hard to compute, it is suggested that there can be a factor of min(n,d) improvement in unaccelerated methods and sqrt(min(m,n)) benefit in accelerated methods.