Distributed Estimation with Multiple Samples per User: Sharp Rates and Phase Transition

Ô m for small m. However , for large m we observe an interesting phase transition : the dependence of the error rate on the communication constraint ¸ changes from 1/ Ô 2¸ to 1/ Ô ¸ . 1 Introduction . 1.1 Background and motivation . We consider the problem of density estimation in a distributed setting where each of n users locally holds a dataset of m independent observations from an unknown discrete probability distribution , and a central server seeks to leverage their data in order to estimate this underlying distribution . This abstract formulation encompasses an array of crucial building blocks in areas such as federated learning , IoT , or measurement from deployed sensors , and has ( along with some of its variants ) received significant attention in recent years . Our focus in this work is to understand how a bandwidth constraint at each user affects this learning task . Namely , to characterize the fundamental tradeoffs between communication , local sample size , and estimation error rate . Previous work on distributed estimation has focused primarily on the single-sample case where each user has exactly one draw from an unknown underlying distribution p. We consider the more general , more practical , and technically more involved setting where each of the n users holds m Ø 1 samples from a k-ary distribution , and can transmit ¸ bits to a central server . Our goal is to understand the minimax rates of estimating p in ¸1 distance . Before detailing our results and discussing their relation with prior work , we provide the necessary background and notation . 1.2 Problem setting and notation . Let k be a positive integer . Let k = { p = ( p1 , . . . , pk ) œ Rk : p Ø 0 , ÎpÎ1 = 1 } denote the set of all probability distributions over a discrete domain of size k , which we assume without loss of generality is [ k ] : = { 1 , 2 , . . . , k } . For p , q œ k , the total variation ( TV ) distance between 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . p and q is TV ( p , q ) : = supS™ [ k ] ( p ( S ) ≠ q ( S ) ) . Here for a set S ™ [ k ] , p ( S ) : = q xœS px . Each of n users observes m i.i.d . samples from the same ( unknown ) distribution p œ k. We write Xi = ( X ( 1 ) i , X ( 2 ) i , . . . , X ( m ) i ) for the samples of the ith user , and Xn = ( X1 , X2 , . . . , Xn ) œ [ k ] nm for the collection of all nm samples . We denote the empirical frequencies of samples at user i by Mi , where , for x œ [ k ] , Mi ( x ) : = 1m qm j=1 1 { X ( j ) i = x } . Each user can only send an ¸-bit message , capturing their bandwidth constraints . In particular , user i is allowed to send a message Yi œ { 0 , 1 } ¸ to the central server through a channel Wi œ W¸ , that is , a Markov kernel Wi : { 0 , 1 } ¸ ◊ [ k ] m æ [ 0 , 1 ] , where Wi ( y | x ) = Pr ( Yi = y | Xi = x ) , ’ x œ [ k ] m , y œ { 0 , 1 } ¸ . Intuitively Wi can be viewed as a randomized function that maps an input Xi to some ¸-bit string Yi , which represents the messaging scheme used by the user . We consider two standard types of protocols , noninteractive and interactive . In noninteractive protocols , all the channels Wis are chosen in parallel , and the messages are sent independently of each other with no shared random seed available . In ( sequentially ) interactive protocols , the choice of Wi may depend on the previous messages Y i≠1 : = ( Y1 , Y2 , . . . , Yi≠1 ) ; that is , the message sent by the ith user can depend on the messages sent by the previous users , and on a common random seed U ( independent of the observations ) available to all users.12 Distribution estimation . Upon observing the messages Y n : = ( Y1 , Y2 , . . . , Yn ) , the server estimates the underlying distribution p with an estimator ‚p : ( { 0 , 1 } ¸ ) n æ k , with goal to minimize the expected total variation ( TV ) distance between p and ‚p . In this work , we are interested in studying the minimax risk for distribution estimation under ¸-bit communication constraints , defined as R ( ¸ , k , n , m ) : = min W nœWn ¸ min p̂ max pœ k E [ TV ( ‚p ( Y n ) , p ) ] ( 1 ) where the minimum is taken over all ( possibly interactive ) protocols . 1.3 Prior results and our contribution . For the setting when m = 1 , i.e. , each user only observes one sample , the problem has been studied by various works in the literature [ 11 , 1 , 3 ] , where it has been established that R ( ¸ , k , n , 1 ) = AÚ k n ‚ Ú k2 n2¸ B . ( 2 ) In this work , we consider a natural setting where each user has access to m > 1 i.i.d . samples . As a baseline , it is immediate to see that , absent any communication constraint ( i.e. , ¸ = Œ ) , the error rate would behave as in the centralized case , R ( Œ , k , n , m ) = AÚ k mn B . ( 3 ) This raises the natural question of how the minimax risk behaves between those two extremes , and of whether the exponential decrease of the risk as a function of the communication constraint ¸ is preserved . We point out that this setting is substantially more challenging to analyze , and requires significantly different techniques and new ideas than the m = 1 case . Indeed , as m grows , even the case ¸ = 1 is not entirely clear : given 1 π m π k samples and only one bit of information to send , should each user send that bit of information about one of their samples only ? Or about the ( quite imprecise ) empirical distribution they can form with their own m samples ? Now , what about the case m ∫ k , where each user now can compute by themselves an accurate estimate of the unknown p , but still lack the bandwith to send enough information about it to the server ? 1Formally , this is captured by defining channels as Wi : Y ◊ [ k ] m ◊Yi≠1 ◊U æ [ 0 , 1 ] , where Y : = { 0 , 1 } ¸ . 2We note that some work further distinguish between private-coin noninteractive protocols , where each user randomizes their message independently as described above , and public-coin noninteractive protocols , where a common random seed is available to all users . In this work , we design private-coin noninteractive protocols for our upper bounds ; as our lower bounds hold even for sequentially interactive protocols , this only makes our results stronger . We are able to obtain tight bounds for most parameter regimes , and show that the minimax rates exhibit an interesting phase transition as m and ¸ vary ( with respect to the domain size k ) . We state the results under cases where m < k/2¸ and m Ø k/2¸ in Theorem 1.1 and 1.2 respectively . Theorem 1.1 . When m < k/2¸ and n/ log ( n ) > k2¸ log m , the minimax rate satisfies R ( ¸ , k , n , m ) = AÚ k mn ‚ Ú k2 mn2¸ B . Moreover , this bound is achieved by a noninteractive communication protocol . We observe that , keeping the same communication budget for each user , the minimax risk is reduced by a factor of Ô m compared to the one-sample case ( 2 ) . In this regime , increasing the communication budget ¸ still reduces the estimation error exponentially , as it did in the one-sample case . Our second set of results shows that this is no longer the case as m increases : Theorem 1.2 . When m Ø k/2¸ and n¸/ log ( n ) > min ( m log ( k/m + 1 ) , k ) log m , the minimax rate satisfies R ( ¸ , k , n , m ) = Y __ ] __ [ O 3Ò k mn ‚ Ò k log ( k/m+1 ) n¸ 4 , when m < k , O 3Ò k mn ‚ Ò k2 mn¸ 4 , when m Ø k. Moreover , this bound is achieved by a noninteractive communication protocol . Unlike the previous case m < k/2¸ , the risk now decreases at a rate 1/ Ô ¸ instead of 1/ Ô 2¸ , an exponential slowdown . As stated , however , Theorem 1.2 only provides an upper bound on the minimax rate , from which it would still be conceivable that the true rate behaves as 1/ Ô 2¸ . We rule out this possibility , confirming the phase transition suggested by Theorem 1.2 as m grows : Theorem 1.3 . When m Ø k/2¸ and n > ( k/¸ ) 2 , the minimax rate satisfies R ( ¸ , k , n , m ) = Y __ ] __ [ 3Ò k mn ‚ Ò k n¸ log k 4 , when m < k log k , 3Ò k mn ‚ Ò k2 mn¸ 4 , when m Ø k log k. Note that the lower bound of Theorem 1.3 matches the upper bound from Theorem 1.2 up to either constant or logarithmic factors . Interestingly , our results imply that , keeping other parameters fixed , increasing m from k/2¸ to k can at best improve the rate by logarithmic factors in k. About the restriction on n. Our bounds for rates are tight up to logarithmic factors in k , for sufficiently large n. The constraint on the range of n in Theorems 1.1 to 1.3 may appear restrictive at first . However , note that since Theorem 1.1 focuses on the regime m < k/2¸ , the bound becomes vacuous for n Æ k ( as then k 2 mn2¸ Ø 1 ) , and thus we can always assume n Ø k. Thus , the requirement n/ log n Ø k2¸ log m is only restrictive for ¸ π log log m+log log n. The assumption in Theorem 1.2 is more consequential ; but it can be shown that n · ¸ must be at least linear in k under some conditions . Indeed , even if each user were given the distribution up to ¸1 error Á < 1 , a standard packing argument would require n¸ Ø k log 1Á bits of communication . Finally , the requirement n > ( k/¸ ) 2 ( instead of the natural n > k/¸ we just argued ) in Theorem 1.3 is indeed not innocuous , and is an artifact of our proof method . For m = k log k , note that if we want to estimate the distribution up to TV error Á < ¸/ ( k log k ) , the above lower bound implies n = ! k/ ( Á2¸ log k ) `` > ( k/¸ ) 2 . Hence Theorem 1.3 demonstrates a phase transition in this case . When Á > ¸/ ( k log k ) , the phase transition is less clear given Theorem 1.3 alone . However , observe that by the packing argument discussed above , we need at least n Ø k log 1Á /¸ users even when each user knows the exact distribution , and hence for any fixed m. This shows that the rate O ( k2/ ( mn2¸ ) ) with exponential dependence on ¸ can not always hold . We see removing these restrictions as an interesting future direction . Our algorithm . To motivate our algorithm , we consider the simplest distribution estimation problem : under communication constraints “ For nm coin tosses with each user observing m coin tosses and sending 1 bit to the center , how do we best estimate the bias of the coin ? ” A key component of our general algorithm is an algorithm for this simplest of settings , which turns out to be surprisingly non-trivial . One may easily come up with simple solutions . One idea is to let each user send 0 if there are more heads than tails and 1 otherwise . This algorithm works well when the coin is almost fair , e.g. , Pr ( head ) œ ( 1/2 ≠ 1/Ôm , 1/2 + 1/Ôm ) , but it is ineffective if the coin is strongly biased . Another idea might be to only let each user indicate whether head appeared or not . This would lead to much better performance for strongly biased coins , e.g. , Pr ( head ) < 1/m , but worse accuracy for fair coins . We elaborate further on this in Section 2.1.1 . In Section 2.1 , we propose an algorithm which estimates the bias of the coin with accuracy O ( 1/mn ) with 1-bit of communication from each user under mild regularity conditions . Note that this is the best rate that can be achieve even when the server has access to all clean samples . Our proposed algorithm not only combines the best of the two naïve algorithms , but further tackles the regime where both solutions fail . It first forms local estimates of the bias of lower accuracy , and then boosts the accuracy by combining local estimates . For the first part , we use a scheme similar to the Gray coding based scheme proposed for Gaussian mean estimation in [ 8 ] , but we need to modify it significantly as the tail probabilities of the underlying distribution can change with the unknown bias . Using this algorithm as a primitive , we form a general algorithm in steps . The first step is to convert the bias estimation algorithm to that for estimating a t-ary distribution using 1-bit communication per user . We do so by partitioning the overall domain [ k ] into t ¥ k/2¸ parts of size at most 2¸ : we then estimate the distribution induced on the parts , using the 1-bit algorithm above by setting t to equal the number of parts . ( Users assigned to this task use their m samples , but only one bit of communication . ) That being done , we estimate the k/2¸ conditional distributions given the sample lies in each of these parts . Since each part has size at most 2¸ , the user can communicate one exact value in these parts using ¸ bits , which allows us to perform this task under our ¸-bit communication constraint , keeping only one of the m samples and transmitting its exact value within a given part . ( Users assigned to this task use their ¸ bits of communication , but only one of their m samples . ) Finally , combining the two types of estimates – of the induced distribution over parts , and of the conditional distribution within each part – allows us to form our overall estimate of the distribution .

Summary: This paper is about distributed density estimation of an unknown probability distribution $p$, where the samples from that distribution are held by different users. In the model considered, there is a central server that collects messages from the users who hold the samples from $p$. Apart from the number of users $n$ and the domain size $k$, the problem is parametrized by the number of samples $m$ that each user holds and the maximum message size of $\ell$ bits. The goal is to minimize the total communication complexity (i.e the number of bits sent to the server). Interestingly, the authors show a phase transition of the error of the estimation with regard to $m$ happening at $m=k/2^\ell$. Specifically, it is shown that the error scales with $\frac{1}{\sqrt{\ 2^\ell}}$ for $m<k/2^\ell$ and with $\frac{1}{\sqrt{\ell}}$ otherwise. The existence of this phase transition is supported by both upper and lower bounds, although they are not shown for the entire parameter regime. However, both bounds are shown for sufficiently large $n$ and they are also tight up to logarithmic factors. The algorithm uses the estimation of a cointoss ($k=2$) bernoulli distribution as a primitive and recursively estimates the mass of certain parts of the domain in this way.