Towards Tight Communication Lower Bounds for Distributed Optimisation

PN i=1 fi ( x ) . This problem arises naturally in large-scale distributed optimisation , where a standard solution is to apply variants of ( stochastic ) gradient descent . We focus on the communication complexity of this problem : our main result provides the first fully unconditional bounds on total number of bits which need to be sent and received by the N machines to solve this problem under point-to-point communication , within a given error-tolerance . Specifically , we show that ⌦ ( Nd log d/N '' ) total bits need to be communicated between the machines to find an additive ✏-approximation to the minimum of PN i=1 fi ( x ) . The result holds for both deterministic and randomised algorithms , and , importantly , requires no assumptions on the algorithm structure . The lower bound is tight under certain restrictions on parameter values , and is matched within constant factors for quadratic objectives by a new variant of quantised gradient descent , which we describe and analyse . Our results bring over tools from communication complexity to distributed optimisation , which has potential for further applications . 1 Introduction . The ability to efficiently distribute large-scale optimisation over several computing nodes has been one of the key enablers of recent progress in machine learning , and the last decade has seen significant attention dedicated to efficient distributed optimisation . One specific area of focus has been on reducing the communication cost of distributed machine learning , i.e . the total number of bits sent and received by machines in order to jointly optimise an objective function . To this end , communicationefficient variants are known for most classical optimisation algorithms , and in fact entire families of communication-compression methods have been introduced in the last decade . We consider a standard setting in which N machines communicate by sending point-to-point binary messages to each other . Given dimension d , and a domain D ✓ Rd , each machine i is given an input function fi : D ! R , corresponding to a subset of the data , and the machines need to jointly minimise the empirical risk PN i=1 fi ( x ) with either deterministic or probabilistic guarantees on the output , within `` additive error tolerance . That is , at least one node needs to output z 2 [ 0 , 1 ] d such that NX i=1 fi ( z ) inf x2 [ 0,1 ] d NX i=1 fi ( x ) + `` . ( 1 ) This setting models data-parallel optimisation , and covers virtually all practical settings , from largescale regression , to the training of deep neural networks . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The key parameters governing communication complexity are the problem dimension d , the solution accuracy `` , and the number of machines N . Most communication-efficient approaches can be linked to ( at least ) one of the following strategies : dimensionality-reduction methods , such as the sparsification of model updates [ 3 , 14 , 15 , 17 , 18 ] , or projection [ 10 , 34 , 35 ] , which attempt to reduce the dependency on the parameter d , quantisation methods [ 2 , 11 , 27 , 31 ] , whose rough goal is to improve the dependency on the accuracy `` , and communication-reduction methods such as reducing the frequency of communication [ 5 , 30 , 36 ] relative to the number of optimisation steps , or communicating via point-to-point messages via , e.g. , gossiping [ 19 , 24 ] . Although these methods use a diverse range of algorithmic ideas , upon close inspection , they all appear to have a worst-case total communication cost of at least Nd log ( d/ '' ) bits , even for simple convex d-dimensional problems , and even for variations of the above standard setting . For instance , even though dimensionality-reduction or quantisation methods might send asymptotically less than d bits per algorithm iteration , they have to compensate for this in the worst case by running for asymptotically more iterations . ( See e.g . [ 2 ] for a simple example of this trade-off . ) It is therefore natural to ask whether this complexity threshold is inherent , or whether it can be circumvented via improved algorithmic techniques . This is our motivating question . A partial answer is given by the foundational work of Tsitsiklis and Luo [ 32 ] , who gave a lower bound of ⌦ ( d log ( d/ '' ) ) in the case where two nodes communicate to optimise over quadratic functions . Their argument works by counting the total number of possible `` -approximate solutions in the input volume : communication has to be at least the logarithm of this number . Subsequent work has considered more complex input functions , e.g . [ 41 ] , or stronger notions of approximation [ 33 ] . The original argument generalises directly to N nodes under the strong assumption that each node has to return the correct output : in this case , communication complexity is asymptotically N times the 2-node cost [ 11 ] . In this context , it is surprisingly still unknown whether the ⌦ ( Nd log ( d/ '' ) ) communication threshold is actually inherent for distributed optimisation in the standard case where only a single node needs to return the output . This question is not just of theoretical interest , since there are many practical settings in large-scale optimisation , such as federated learning [ 20 ] or the parameter server setting [ 23 ] , where only a single node coordinates the optimisation , and knows the final answer . This raises the question whether more communication-efficient algorithms are possible in such settings , in which communication cost is a key concern . At the same time , it is also not clear under which conditions algorithms achieving this asymptotic complexity exist . 1.1 Contribution . In this paper , we take a significant step towards addressing these questions . Our main result is the first unconditional lower bound on the communication complexity of distributed optimisation in the setting discussed above , showing that it is impossible to obtain a significant improvement in communication cost even if only one node ( the “ coordinator ” ) learns the final output . Specifically , even if the input functions fi at the nodes are promised to be quadratic functions x 7 ! 0kx x⇤k22 for some constant 0 > 0 , then any deterministic or randomised algorithm where at least one node learns a solution to ( 1 ) requires ⌦ ⇣ Nd log d N '' ⌘ total bits to be communicated for = 0N , as long as parameters satisfy d/N2 '' = ⌦ ( 1 ) . We emphasise that the lower bound requires no assumptions on the structure of the algorithm or amount of local computation . We also note that in most practical settings , the parameter dependency requirement is satisfied , as the number of parameters d is significantly larger than the number of machines N multiplied by the error tolerance `` – moreover , a non-trivial dependence between , d , N and `` is required for the lower bound to hold . We discuss this in detail below . Our results start from the classic idea of linking communication complexity with the number of quadratic functions with distinct minima in the domain [ 32 ] . To extend this approach to randomised ( stochastic ) algorithms and to the multi-node case N > 2 , we build new connections to results and techniques from communication complexity [ 21 ] . Such connections have not to our knowledge been explored in the context of ( real-valued ) optimisation tasks . Our work thus provides a template and a basic toolkit for applying communication complexity results to distributed optimisation . As further applications , we improve the main lower bound to ⌦ ( Nd log ( d/ '' ) ) for the deterministic case if some node is required to output both the approximate minimiser z and the value of the sumPN i=1 fi ( z ) , as well as prove stronger lower bounds in the more challenging non-convex case ( see Section 5 and Appendices A and B . ) To complement this lower bound , we show that for strongly convex and strongly smooth functions , distributed optimisation can be done using deterministic quantised gradient descent with O ⇣ Nd log log d '' ⌘ total bits communicated , where PN i=1 fi is ↵-strongly convex and -strongly smooth , and = /↵ is the condition number . This is , to our knowledge , the first tight upper bound for communication cost of quantised gradient descent on quadratic functions , as well as the first upper bound that does not require all-to-all broadcast . In particular , for constant condition number , this matches our main lower bound when d N , e.g . d = ⌦ ( N2+ ) for constant > 0 . Our algorithm builds on prior quantised gradient descent implementations [ 2 , 25 ] , however , to achieve a tight bound , we need to ( a ) ensure that our gradient quantisation is sufficiently parsimonious , using O ( d log ) bits per gradient , and ( b ) avoid all-to-all exchange of gradients . For ( a ) , we specialise a recent lattice-based quantisation scheme which allows arbitrary centring of iterates [ 11 ] , and for ( b ) , we use two-stage quantisation approach , where the nodes first send their quantised gradients to the coordinator , and the coordinator then broadcasts the carefully quantised sum back to nodes . In Appendix D , we further show that this running time can be improved using randomisation when d/N '' is small , using a simple sub-sampling approach . 1.2 Discussion . Implications . While the focus of our work is theoretical , our results show that current practical algorithmic approaches are already close to worst-case optimal . Specifically , we show that it is impossible to obtain algorithms with communication cost e.g . O ( Nd+ d log d/ '' ) by “ pipelining ” communication costs across algorithm iterations , performing additional local optimisation steps , or by introducing entirely new algorithmic techniques . At the same time , our upper bound conceptually shows that , for quadratic functions , carefully quantised gradient descent can cost asymptotically the same as broadcasting the solution , while the lower bound shows that this cost is inherent . Specifically , broadcasting a single d-dimensional point from [ 0 , 1 ] d within accuracy ( `` / ) 1/2 , as required for `` -approximation of the sum PN i=1 fi ( z ) , to N nodes costs ⌦ ( Nd log ( d/ '' ) ) bits [ 11 ] , and thus the communication cost of our algorithm is tight . The lower bound shows that little can be gained by avoiding this broadcast . Extensions . Following Tsitsiklis and Luo [ 32 ] and Magnússon et al . [ 25 ] , we have assumed above that the range of the input functions is [ 0 , 1 ] d , and the global objective is the sum of input functions . However , the results apply even with some modifications to the setting . First , we can consider a case where the global objective is the average 1/N PN i=1 fi instead of sum . In this case , the lower bound holds with = 0 , and the upper bound holds as stated , with being the smoothness parameter of the average . Second , the results can be extended for any compact convex domain of input functions , with the precise bounds depending on the volume and diameter of the domain for the lower and upper bound , respectively . For example , one can easily verify that , for inputs defined over the unit sphere , the bounds still hold , but without the factor d inside the logarithm . Limitations . We note that there still remains a small gap between our upper and lower bounds . To illustrate this , consider optimisation of the average of quadratic functions x 7 ! kx x⇤k22 defined over the unit hypercube [ 0 , 1 ] d ; in this case , the bounds take the form ⌦ ⇣ Nd log d N '' ⌘ and O ⇣ Nd log d '' ⌘ . Moreover , the lower bound requires the parameter dependency d/N2 '' = ⌦ ( 1 ) to hold . We note that we can not fully get rid of this requirement : specifically , as we show in Appendix D , we can use a simple input sub-sampling approach to show that , if we have d/ '' = O ( N ) for < 1 , then problem ( 1 ) can be solved with O ( N d log d/ '' ) total bits communicated using a randomised algorithm , asymptotically less than dictated by the general lower bound . A second question our current techniques do not address is the precise dependency on the condition number . Our lower bound techniques do no benefit from large , so new ideas would be required to address this question . On the upper bound side , the linear dependency on appears to be inherent for our quantised gradient descent algorithm . However , very recent progress on quantised second-order methods [ 1 , 15 ] shows that it is possible to improve this dependency in general , by leveraging second-order information together with quantisation . Specifically , Alimisis et al . [ 1 ] , Islamov et al . [ 15 ] provide complex quantised variants of Newton-type algorithms , which can achieve linear-in-d communication cost per iteration , under certain assumptions . Thus , these algorithms can asymptotically reach the optimal Nd log ( d/✏ ) complexity threshold implied by our lower bounds , within logarithmic factors in and other terms , for a wider range of inputs . This comes at the relative cost of a more complex algorithm , and significant additional local computation .

Summary: The paper shows lower bounds for distribution optimization. For the standard distributed optimization setting with N machines each of which holds a d-dimensional function f_i and the goal is to minimize \sum_{i=1}^N f_i(x), the paper shows that \Omega(Nd log(d/N\eps)) bits of communication are necessary to obtain an \eps sub-optimal solution. The result holds both for randomized and deterministic algorithms.   The paper also shows a new upper bound derived using a variant of quantized gradient descent. The upper bound nearly matches the lower bound for well-conditioned problems (when the overall problem has a constant condition number).  The lower bounds are shown via interesting applications of standard communication complexity machinery. The upper bound follows via relatively simple modifications to existing quantization techniques.