New Bounds For Distributed Mean Estimation and Variance Reduction

∑n v=1 xv , while min- imizing total communication cost . DME is a fundamental construct in distributed machine learning , and there has been considerable work on variants of this problem , especially in the context of distributed variance reduction for stochastic gradients in parallel SGD . Previous work typically assumes an upper bound on the norm of the input vectors , and achieves an error bound in terms of this norm . However , in many real applications , the input vectors are concentrated around the correct output µ , but µ itself has large norm . In such cases , previous output error bounds perform poorly . In this paper , we show that output error bounds need not depend on input norm . We provide a method of quantization which allows distributed mean estimation to be performed with solution quality dependent only on the distance between inputs , not on input norm , and show an analogous result for distributed variance reduction . The technique is based on a new connection with lattice theory . We also provide lower bounds showing that the communication to error trade-off of our algorithms is asymptotically optimal . As the lattices achieving optimal bounds under ` 2-norm can be computationally impractical , we also present an extension which leverages easy-to-use cubic lattices , and is loose only up to a logarithmic factor in d. We show experimentally that our method yields practical improvements for common applications , relative to prior approaches . 1 Introduction . Several problems in distributed machine learning and optimization can be reduced to variants distributed mean estimation problem , in which n machines must cooperate to jointly estimate the mean of their d-dimensional inputs µ = 1n ∑n v=1 xv as closely as possible , while minimizing communication . In particular , this construct is often used for distributed variance reduction : here , each machine receives as input an independent probabilistic estimate of a d-dimensional vector ∇ , and the aim is for all machines to output a common estimate of ∇ with lower variance than the individual inputs , minimizing communication . Without any communication restrictions , the ideal output would be the mean of all machines ’ inputs . While variants of these fundamental problems have been considered since seminal work by Tsitsiklis & Luo ( 1987 ) , the task has seen renewed attention recently in the context of distributed machine learning . In particular , variance reduction is a key component in data-parallel distributed stochastic gradient descent ( SGD ) , the standard way to parallelize the training of deep neural networks , e.g . Bottou ( 2010 ) ; Abadi et al . ( 2016 ) , where it is used to estimate the average of gradient updates obtained in parallel at the nodes . Thus , several prior works proposed efficient compression schemes to solve variance reduction or mean estimation , see e.g . Suresh et al . ( 2017 ) ; Alistarh et al . ( 2017 ) ; Ramezani-Kebrya et al . ( 2019 ) ; Gandikota et al . ( 2019 ) , and Ben-Nun & Hoefler ( 2019 ) for a general survey of practical distribution schemes . These schemes seek to quantize nodes ’ inputs coordinatewise to one of a limited collection of values , in order to then efficiently encode and transmit these quantized values . A trade-off then arises between the number of bits sent , and the added variance due of quantization . Since the measure of output quality is variance , it appears most natural to evaluate this with respect to input variance , in order to show that variance reduction is indeed achieved . Surprisingly , however , we are aware of no previous works which do so ; all existing methods give bounds on output variance in terms of the squared input norm . This is clearly suboptimal when the squared norm is higher than the variance , i.e. , when inputs are not centered around the origin . In some practical scenarios this causes output variance to be higher than input variance , as we demonstrate in Section 4 . Contributions . In this paper , we provide the first bounds for distributed mean estimation and variance reduction which are still tight when inputs are not centered around the origin . Our results are based on new lattice-based quantization techniques , which may be of independent interest , and come with matching lower bounds , and practical extensions . More precisely , our contributions are as follows : • For distributed mean estimation , we show that , to achieve a reduction of a factor q in the input ‘ variance ’ ( which we define to be the maximum squared distance between inputs ) , it is necessary and sufficient for machines to communicate Θ ( d log q ) bits . • For variance reduction , we show tight Θ ( d logn ) bounds on the worst-case communication bits required to achieve optimal Θ ( n ) -factor variance reduction by n nodes over d-dimensional input , and indeed to achieve any variance reduction at all . We then show how incorporating error detection into our quantization scheme , we can also obtain tight bounds on the bits required in expectation . • We show how to efficiently instantiate our lattice-based quantization framework in practice , with guarantees . In particular , we devise a variant of the scheme which ensures close-to-optimal communication-variance bounds even for the standard cubic lattice , and use it to obtain improvements relative to the best known previous methods for distributed mean estimation , both on synthetic and real-world tasks . 1.1 Problem Definitions and Discussion . MeanEstimation is defined as follows : we have n machines v , and each receives as input a vector xv ∈ Rd . We also assume that all machines receive a common value y , with the guarantee that for any machines u , v , ‖xu−xv‖ ≤ y . Our goal is for all machines to output the same value EST ∈ Rd , which is an unbiased estimator of the mean µ = 1n ∑ v∈M xv , i.e . E [ EST ] = µ , with variance as low as possible . Notice that the input specification is entirely deterministic ; any randomness in the output arises only from the algorithm used . In the variant of VarianceReduction , we again have a set of n machines , and now an unknown true vector ∇ . Each machine v receives as input an independent unbiased estimator xv of ∇ ( i.e. , E [ xv ] = ∇ ) with variance E [ ‖xv −∇‖2 ] ≤ σ2 . Machines are assumed to have knowledge of σ . Our goal is for all machines to output the same value EST ∈ Rd , which is an unbiased estimator of ∇ , i.e. , E [ EST ] = ∇ , with low variance . Since the input is random , output randomness now stems from this input randomness as well as any randomness in the algorithm . VarianceReduction is common for instance in the context of gradient-based optimization of machine learning models , where we assume that each machine v processes local samples in order to obtain a stochastic gradient g̃v , which is an unbiased estimator of the true gradient ∇ , with variance bound σ2 . If we directly averaged the local stochastic gradients g̃v , we could obtain an unbiased estimator of the true gradient G with variance bound σ2/n , which can lead to faster convergence . Input Variance Assumption . The parameter y replaces the usual MeanEstimation assumption of a known bound M on the norms of input vectors . Note that , in the worst case , we can always set y = 2M and obtain the same asymptotic upper bounds as in e.g . Suresh et al . ( 2017 ) ; our results are therefore at least as good as previous approaches in all cases , but provide significant improvement when inputs are not centered around the origin . The reason for this change is to allow stronger bounds in scenarios where we expect inputs to be closer to each other than to the origin . In particular , it allows our MeanEstimation problem to more effectively generalize VarianceReduction . Parameter y is a deterministic analogue of the parameter σ for VarianceReduction ; both y and σ provide a bound on the distance of inputs from their mean , rather than from the origin . Accordingly , input variance σ2 for a VarianceReduction instance corresponds ( up to constant factors ) to y2 for a MeanEstimation instance . For consistency of terminology , we therefore refer to y2 as the input variance of the instance ( despite such inputs being deterministic ) . It is common in machine learning applications of VarianceReduction to assume that an estimate of the variance σ2 is known ( Alistarh et al. , 2017 ; Gandikota et al. , 2019 ) . To study both problems in a common framework , we make the analogous assumption about MeanEstimation , and assume knowledge of the input variance y2 . Even if the relevant bounds y or σ are not known a priori , they can usually be estimated in practice . Relationship Between Problems . If one allows unrestricted communication , the natural solution to both problems is to average the inputs . This is an exact solution to MeanEstimation with variance 0 , and is also an asymptotically optimal solution to VarianceReduction , of variance at most σ 2 n . 1 However , doing so would require the exchange of infi- nite precision real numbers . So , it is common to instead communicate quantized values of bounded bit-length ( Alistarh et al. , 2017 ) , which will engender additional variance caused by random choices within the quantization method . The resulting estimates will therefore have variance V arquant for MeanEstimation , and σ 2 n + V arquant for VarianceReduction . We will show a trade-off between bits of communication and output variance for both problems ; in the case of VarianceReduction , though , there is an ‘ upper limit ’ to this trade-off , since we can not go below Ω ( σ 2 n ) total output variance . The other major difference between the two problems is that in MeanEstimation , distances between inputs are bounded by y with certainty , whereas in VarianceReduction they are instead bounded by O ( σ ) only in expectation . This causes extra complications for quantization , and introduces a gap between average and worst-case communication cost . Distributed Model . We aim to provide a widely applicable method for distributed mean estimation , and therefore we avoid relying on the specifics of particular distributed models . Instead , we assume that the basic communication structures we use ( stars and binary trees ) can be constructed without significant overhead . This setting is supported by machine learning applications , which have very high input dimension ( i.e. , d n ) , and so the costs of synchronization or construction of an overlay ( which do not depend on d , and are generally poly-logarithmic in n ) , will be heavily dominated by the communication costs incurred subsequently during mean estimation . They also need only be incurred once , even if mean estimation or variance reduction is to be performed many times ( e.g . during distributed SGD ) . For these reasons , we do not include these model-specific setup costs in our stated complexities ; any implementation should take them into separate consideration . For simplicity , we will present our algorithms within a basic synchronous fault-free messagepassing model , in which machines can send arbitrary messages to any other machine , but they could naturally be extended to asynchronous and shared-memory models of communication . Our aim will be to minimize the number of bits sent and received by any machine during the algorithm ; we do not consider other measures such as round complexity . Vector Norms . When dealing with vectors in Rd , we will use names in bold , e.g . x , y . We will state most of our results in such a way that they will apply to any of the three 1For specific classes of input distribution , and for non-asymptotic concentration results , however , better estimators of ∇ are known ; see e.g . Joly et al . ( 2017 ) . most commonly-used norms on Rd in applications : ` 1 norm ‖x‖1 : = ∑d i=1 |xi| , ` 2 norm ‖x‖2 : = √∑d i=1 x 2 i , and ` ∞ norm ‖x‖∞ : = maxdi=1 ‖xi‖ . Throughout the paper we will therefore use the general notation ‖·‖ , which should be considered to be fixed as one of these norms , other than for statements specific to particular norms . Definitions which depend on norms , such as variance Var [ x ] : = E [ ‖x−E [ x ] ‖2 ] , are therefore assumed to also be under the appropriate norm .

Summary: This paper studies the problem of mean estimation of n vectors in R^d in a distributed setting. There are n machines. Each has one data point, a vector x_v. The goal is to compute the mean of x_v’s for v = 1, …, n with as few bits of communication as possible.