New Bounds For Distributed Mean Estimation and Variance Reduction

∑n v=1 xv , while min- imizing total communication cost . DME is a fundamental construct in distributed machine learning , and there has been considerable work on variants of this problem , especially in the context of distributed variance reduction for stochastic gradients in parallel SGD . Previous work typically assumes an upper bound on the norm of the input vectors , and achieves an error bound in terms of this norm . However , in many real applications , the input vectors are concentrated around the correct output µ , but µ itself has large norm . In such cases , previous output error bounds perform poorly . In this paper , we show that output error bounds need not depend on input norm . We provide a method of quantization which allows distributed mean estimation to be performed with solution quality dependent only on the distance between inputs , not on input norm , and show an analogous result for distributed variance reduction . The technique is based on a new connection with lattice theory . We also provide lower bounds showing that the communication to error trade-off of our algorithms is asymptotically optimal . As the lattices achieving optimal bounds under ` 2-norm can be computationally impractical , we also present an extension which leverages easy-to-use cubic lattices , and is loose only up to a logarithmic factor in d. We show experimentally that our method yields practical improvements for common applications , relative to prior approaches . 1 Introduction . Several problems in distributed machine learning and optimization can be reduced to variants distributed mean estimation problem , in which n machines must cooperate to jointly estimate the mean of their d-dimensional inputs µ = 1n ∑n v=1 xv as closely as possible , while minimizing communication . In particular , this construct is often used for distributed variance reduction : here , each machine receives as input an independent probabilistic estimate of a d-dimensional vector ∇ , and the aim is for all machines to output a common estimate of ∇ with lower variance than the individual inputs , minimizing communication . Without any communication restrictions , the ideal output would be the mean of all machines ’ inputs . While variants of these fundamental problems have been considered since seminal work by Tsitsiklis & Luo ( 1987 ) , the task has seen renewed attention recently in the context of distributed machine learning . In particular , variance reduction is a key component in data-parallel distributed stochastic gradient descent ( SGD ) , the standard way to parallelize the training of deep neural networks , e.g . Bottou ( 2010 ) ; Abadi et al . ( 2016 ) , where it is used to estimate the average of gradient updates obtained in parallel at the nodes . Thus , several prior works proposed efficient compression schemes to solve variance reduction or mean estimation , see e.g . Suresh et al . ( 2017 ) ; Alistarh et al . ( 2017 ) ; Ramezani-Kebrya et al . ( 2019 ) ; Gandikota et al . ( 2019 ) , and Ben-Nun & Hoefler ( 2019 ) for a general survey of practical distribution schemes . These schemes seek to quantize nodes ’ inputs coordinatewise to one of a limited collection of values , in order to then efficiently encode and transmit these quantized values . A trade-off then arises between the number of bits sent , and the added variance due of quantization . Since the measure of output quality is variance , it appears most natural to evaluate this with respect to input variance , in order to show that variance reduction is indeed achieved . Surprisingly , however , we are aware of no previous works which do so ; all existing methods give bounds on output variance in terms of the squared input norm . This is clearly suboptimal when the squared norm is higher than the variance , i.e. , when inputs are not centered around the origin . In some practical scenarios this causes output variance to be higher than input variance , as we demonstrate in Section 4 . Contributions . In this paper , we provide the first bounds for distributed mean estimation and variance reduction which are still tight when inputs are not centered around the origin . Our results are based on new lattice-based quantization techniques , which may be of independent interest , and come with matching lower bounds , and practical extensions . More precisely , our contributions are as follows : • For distributed mean estimation , we show that , to achieve a reduction of a factor q in the input ‘ variance ’ ( which we define to be the maximum squared distance between inputs ) , it is necessary and sufficient for machines to communicate Θ ( d log q ) bits . • For variance reduction , we show tight Θ ( d logn ) bounds on the worst-case communication bits required to achieve optimal Θ ( n ) -factor variance reduction by n nodes over d-dimensional input , and indeed to achieve any variance reduction at all . We then show how incorporating error detection into our quantization scheme , we can also obtain tight bounds on the bits required in expectation . • We show how to efficiently instantiate our lattice-based quantization framework in practice , with guarantees . In particular , we devise a variant of the scheme which ensures close-to-optimal communication-variance bounds even for the standard cubic lattice , and use it to obtain improvements relative to the best known previous methods for distributed mean estimation , both on synthetic and real-world tasks . 1.1 Problem Definitions and Discussion . MeanEstimation is defined as follows : we have n machines v , and each receives as input a vector xv ∈ Rd . We also assume that all machines receive a common value y , with the guarantee that for any machines u , v , ‖xu−xv‖ ≤ y . Our goal is for all machines to output the same value EST ∈ Rd , which is an unbiased estimator of the mean µ = 1n ∑ v∈M xv , i.e . E [ EST ] = µ , with variance as low as possible . Notice that the input specification is entirely deterministic ; any randomness in the output arises only from the algorithm used . In the variant of VarianceReduction , we again have a set of n machines , and now an unknown true vector ∇ . Each machine v receives as input an independent unbiased estimator xv of ∇ ( i.e. , E [ xv ] = ∇ ) with variance E [ ‖xv −∇‖2 ] ≤ σ2 . Machines are assumed to have knowledge of σ . Our goal is for all machines to output the same value EST ∈ Rd , which is an unbiased estimator of ∇ , i.e. , E [ EST ] = ∇ , with low variance . Since the input is random , output randomness now stems from this input randomness as well as any randomness in the algorithm . VarianceReduction is common for instance in the context of gradient-based optimization of machine learning models , where we assume that each machine v processes local samples in order to obtain a stochastic gradient g̃v , which is an unbiased estimator of the true gradient ∇ , with variance bound σ2 . If we directly averaged the local stochastic gradients g̃v , we could obtain an unbiased estimator of the true gradient G with variance bound σ2/n , which can lead to faster convergence . Input Variance Assumption . The parameter y replaces the usual MeanEstimation assumption of a known bound M on the norms of input vectors . Note that , in the worst case , we can always set y = 2M and obtain the same asymptotic upper bounds as in e.g . Suresh et al . ( 2017 ) ; our results are therefore at least as good as previous approaches in all cases , but provide significant improvement when inputs are not centered around the origin . The reason for this change is to allow stronger bounds in scenarios where we expect inputs to be closer to each other than to the origin . In particular , it allows our MeanEstimation problem to more effectively generalize VarianceReduction . Parameter y is a deterministic analogue of the parameter σ for VarianceReduction ; both y and σ provide a bound on the distance of inputs from their mean , rather than from the origin . Accordingly , input variance σ2 for a VarianceReduction instance corresponds ( up to constant factors ) to y2 for a MeanEstimation instance . For consistency of terminology , we therefore refer to y2 as the input variance of the instance ( despite such inputs being deterministic ) . It is common in machine learning applications of VarianceReduction to assume that an estimate of the variance σ2 is known ( Alistarh et al. , 2017 ; Gandikota et al. , 2019 ) . To study both problems in a common framework , we make the analogous assumption about MeanEstimation , and assume knowledge of the input variance y2 . Even if the relevant bounds y or σ are not known a priori , they can usually be estimated in practice . Relationship Between Problems . If one allows unrestricted communication , the natural solution to both problems is to average the inputs . This is an exact solution to MeanEstimation with variance 0 , and is also an asymptotically optimal solution to VarianceReduction , of variance at most σ 2 n . 1 However , doing so would require the exchange of infi- nite precision real numbers . So , it is common to instead communicate quantized values of bounded bit-length ( Alistarh et al. , 2017 ) , which will engender additional variance caused by random choices within the quantization method . The resulting estimates will therefore have variance V arquant for MeanEstimation , and σ 2 n + V arquant for VarianceReduction . We will show a trade-off between bits of communication and output variance for both problems ; in the case of VarianceReduction , though , there is an ‘ upper limit ’ to this trade-off , since we can not go below Ω ( σ 2 n ) total output variance . The other major difference between the two problems is that in MeanEstimation , distances between inputs are bounded by y with certainty , whereas in VarianceReduction they are instead bounded by O ( σ ) only in expectation . This causes extra complications for quantization , and introduces a gap between average and worst-case communication cost . Distributed Model . We aim to provide a widely applicable method for distributed mean estimation , and therefore we avoid relying on the specifics of particular distributed models . Instead , we assume that the basic communication structures we use ( stars and binary trees ) can be constructed without significant overhead . This setting is supported by machine learning applications , which have very high input dimension ( i.e. , d n ) , and so the costs of synchronization or construction of an overlay ( which do not depend on d , and are generally poly-logarithmic in n ) , will be heavily dominated by the communication costs incurred subsequently during mean estimation . They also need only be incurred once , even if mean estimation or variance reduction is to be performed many times ( e.g . during distributed SGD ) . For these reasons , we do not include these model-specific setup costs in our stated complexities ; any implementation should take them into separate consideration . For simplicity , we will present our algorithms within a basic synchronous fault-free messagepassing model , in which machines can send arbitrary messages to any other machine , but they could naturally be extended to asynchronous and shared-memory models of communication . Our aim will be to minimize the number of bits sent and received by any machine during the algorithm ; we do not consider other measures such as round complexity . Vector Norms . When dealing with vectors in Rd , we will use names in bold , e.g . x , y . We will state most of our results in such a way that they will apply to any of the three 1For specific classes of input distribution , and for non-asymptotic concentration results , however , better estimators of ∇ are known ; see e.g . Joly et al . ( 2017 ) . most commonly-used norms on Rd in applications : ` 1 norm ‖x‖1 : = ∑d i=1 |xi| , ` 2 norm ‖x‖2 : = √∑d i=1 x 2 i , and ` ∞ norm ‖x‖∞ : = maxdi=1 ‖xi‖ . Throughout the paper we will therefore use the general notation ‖·‖ , which should be considered to be fixed as one of these norms , other than for statements specific to particular norms . Definitions which depend on norms , such as variance Var [ x ] : = E [ ‖x−E [ x ] ‖2 ] , are therefore assumed to also be under the appropriate norm .

Summary: The paper considers a particular setting of distributed mean estimation problem, where each party has a vector of potentially large $l_2$ norm, yet this vectors are fairly close to each other. The goal is to communicate as few bits as possible and estimate the mean of the vectors. Previous approaches had the dependence on the size of the ball containing all the vectors, which gives bad bounds if vectors are long (but close to each other).