Distributed Estimation with Multiple Samples per User: Sharp Rates and Phase Transition

Ô m for small m. However , for large m we observe an interesting phase transition : the dependence of the error rate on the communication constraint ¸ changes from 1/ Ô 2¸ to 1/ Ô ¸ . 1 Introduction . 1.1 Background and motivation . We consider the problem of density estimation in a distributed setting where each of n users locally holds a dataset of m independent observations from an unknown discrete probability distribution , and a central server seeks to leverage their data in order to estimate this underlying distribution . This abstract formulation encompasses an array of crucial building blocks in areas such as federated learning , IoT , or measurement from deployed sensors , and has ( along with some of its variants ) received significant attention in recent years . Our focus in this work is to understand how a bandwidth constraint at each user affects this learning task . Namely , to characterize the fundamental tradeoffs between communication , local sample size , and estimation error rate . Previous work on distributed estimation has focused primarily on the single-sample case where each user has exactly one draw from an unknown underlying distribution p. We consider the more general , more practical , and technically more involved setting where each of the n users holds m Ø 1 samples from a k-ary distribution , and can transmit ¸ bits to a central server . Our goal is to understand the minimax rates of estimating p in ¸1 distance . Before detailing our results and discussing their relation with prior work , we provide the necessary background and notation . 1.2 Problem setting and notation . Let k be a positive integer . Let k = { p = ( p1 , . . . , pk ) œ Rk : p Ø 0 , ÎpÎ1 = 1 } denote the set of all probability distributions over a discrete domain of size k , which we assume without loss of generality is [ k ] : = { 1 , 2 , . . . , k } . For p , q œ k , the total variation ( TV ) distance between 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . p and q is TV ( p , q ) : = supS™ [ k ] ( p ( S ) ≠ q ( S ) ) . Here for a set S ™ [ k ] , p ( S ) : = q xœS px . Each of n users observes m i.i.d . samples from the same ( unknown ) distribution p œ k. We write Xi = ( X ( 1 ) i , X ( 2 ) i , . . . , X ( m ) i ) for the samples of the ith user , and Xn = ( X1 , X2 , . . . , Xn ) œ [ k ] nm for the collection of all nm samples . We denote the empirical frequencies of samples at user i by Mi , where , for x œ [ k ] , Mi ( x ) : = 1m qm j=1 1 { X ( j ) i = x } . Each user can only send an ¸-bit message , capturing their bandwidth constraints . In particular , user i is allowed to send a message Yi œ { 0 , 1 } ¸ to the central server through a channel Wi œ W¸ , that is , a Markov kernel Wi : { 0 , 1 } ¸ ◊ [ k ] m æ [ 0 , 1 ] , where Wi ( y | x ) = Pr ( Yi = y | Xi = x ) , ’ x œ [ k ] m , y œ { 0 , 1 } ¸ . Intuitively Wi can be viewed as a randomized function that maps an input Xi to some ¸-bit string Yi , which represents the messaging scheme used by the user . We consider two standard types of protocols , noninteractive and interactive . In noninteractive protocols , all the channels Wis are chosen in parallel , and the messages are sent independently of each other with no shared random seed available . In ( sequentially ) interactive protocols , the choice of Wi may depend on the previous messages Y i≠1 : = ( Y1 , Y2 , . . . , Yi≠1 ) ; that is , the message sent by the ith user can depend on the messages sent by the previous users , and on a common random seed U ( independent of the observations ) available to all users.12 Distribution estimation . Upon observing the messages Y n : = ( Y1 , Y2 , . . . , Yn ) , the server estimates the underlying distribution p with an estimator ‚p : ( { 0 , 1 } ¸ ) n æ k , with goal to minimize the expected total variation ( TV ) distance between p and ‚p . In this work , we are interested in studying the minimax risk for distribution estimation under ¸-bit communication constraints , defined as R ( ¸ , k , n , m ) : = min W nœWn ¸ min p̂ max pœ k E [ TV ( ‚p ( Y n ) , p ) ] ( 1 ) where the minimum is taken over all ( possibly interactive ) protocols . 1.3 Prior results and our contribution . For the setting when m = 1 , i.e. , each user only observes one sample , the problem has been studied by various works in the literature [ 11 , 1 , 3 ] , where it has been established that R ( ¸ , k , n , 1 ) = AÚ k n ‚ Ú k2 n2¸ B . ( 2 ) In this work , we consider a natural setting where each user has access to m > 1 i.i.d . samples . As a baseline , it is immediate to see that , absent any communication constraint ( i.e. , ¸ = Œ ) , the error rate would behave as in the centralized case , R ( Œ , k , n , m ) = AÚ k mn B . ( 3 ) This raises the natural question of how the minimax risk behaves between those two extremes , and of whether the exponential decrease of the risk as a function of the communication constraint ¸ is preserved . We point out that this setting is substantially more challenging to analyze , and requires significantly different techniques and new ideas than the m = 1 case . Indeed , as m grows , even the case ¸ = 1 is not entirely clear : given 1 π m π k samples and only one bit of information to send , should each user send that bit of information about one of their samples only ? Or about the ( quite imprecise ) empirical distribution they can form with their own m samples ? Now , what about the case m ∫ k , where each user now can compute by themselves an accurate estimate of the unknown p , but still lack the bandwith to send enough information about it to the server ? 1Formally , this is captured by defining channels as Wi : Y ◊ [ k ] m ◊Yi≠1 ◊U æ [ 0 , 1 ] , where Y : = { 0 , 1 } ¸ . 2We note that some work further distinguish between private-coin noninteractive protocols , where each user randomizes their message independently as described above , and public-coin noninteractive protocols , where a common random seed is available to all users . In this work , we design private-coin noninteractive protocols for our upper bounds ; as our lower bounds hold even for sequentially interactive protocols , this only makes our results stronger . We are able to obtain tight bounds for most parameter regimes , and show that the minimax rates exhibit an interesting phase transition as m and ¸ vary ( with respect to the domain size k ) . We state the results under cases where m < k/2¸ and m Ø k/2¸ in Theorem 1.1 and 1.2 respectively . Theorem 1.1 . When m < k/2¸ and n/ log ( n ) > k2¸ log m , the minimax rate satisfies R ( ¸ , k , n , m ) = AÚ k mn ‚ Ú k2 mn2¸ B . Moreover , this bound is achieved by a noninteractive communication protocol . We observe that , keeping the same communication budget for each user , the minimax risk is reduced by a factor of Ô m compared to the one-sample case ( 2 ) . In this regime , increasing the communication budget ¸ still reduces the estimation error exponentially , as it did in the one-sample case . Our second set of results shows that this is no longer the case as m increases : Theorem 1.2 . When m Ø k/2¸ and n¸/ log ( n ) > min ( m log ( k/m + 1 ) , k ) log m , the minimax rate satisfies R ( ¸ , k , n , m ) = Y __ ] __ [ O 3Ò k mn ‚ Ò k log ( k/m+1 ) n¸ 4 , when m < k , O 3Ò k mn ‚ Ò k2 mn¸ 4 , when m Ø k. Moreover , this bound is achieved by a noninteractive communication protocol . Unlike the previous case m < k/2¸ , the risk now decreases at a rate 1/ Ô ¸ instead of 1/ Ô 2¸ , an exponential slowdown . As stated , however , Theorem 1.2 only provides an upper bound on the minimax rate , from which it would still be conceivable that the true rate behaves as 1/ Ô 2¸ . We rule out this possibility , confirming the phase transition suggested by Theorem 1.2 as m grows : Theorem 1.3 . When m Ø k/2¸ and n > ( k/¸ ) 2 , the minimax rate satisfies R ( ¸ , k , n , m ) = Y __ ] __ [ 3Ò k mn ‚ Ò k n¸ log k 4 , when m < k log k , 3Ò k mn ‚ Ò k2 mn¸ 4 , when m Ø k log k. Note that the lower bound of Theorem 1.3 matches the upper bound from Theorem 1.2 up to either constant or logarithmic factors . Interestingly , our results imply that , keeping other parameters fixed , increasing m from k/2¸ to k can at best improve the rate by logarithmic factors in k. About the restriction on n. Our bounds for rates are tight up to logarithmic factors in k , for sufficiently large n. The constraint on the range of n in Theorems 1.1 to 1.3 may appear restrictive at first . However , note that since Theorem 1.1 focuses on the regime m < k/2¸ , the bound becomes vacuous for n Æ k ( as then k 2 mn2¸ Ø 1 ) , and thus we can always assume n Ø k. Thus , the requirement n/ log n Ø k2¸ log m is only restrictive for ¸ π log log m+log log n. The assumption in Theorem 1.2 is more consequential ; but it can be shown that n · ¸ must be at least linear in k under some conditions . Indeed , even if each user were given the distribution up to ¸1 error Á < 1 , a standard packing argument would require n¸ Ø k log 1Á bits of communication . Finally , the requirement n > ( k/¸ ) 2 ( instead of the natural n > k/¸ we just argued ) in Theorem 1.3 is indeed not innocuous , and is an artifact of our proof method . For m = k log k , note that if we want to estimate the distribution up to TV error Á < ¸/ ( k log k ) , the above lower bound implies n = ! k/ ( Á2¸ log k ) `` > ( k/¸ ) 2 . Hence Theorem 1.3 demonstrates a phase transition in this case . When Á > ¸/ ( k log k ) , the phase transition is less clear given Theorem 1.3 alone . However , observe that by the packing argument discussed above , we need at least n Ø k log 1Á /¸ users even when each user knows the exact distribution , and hence for any fixed m. This shows that the rate O ( k2/ ( mn2¸ ) ) with exponential dependence on ¸ can not always hold . We see removing these restrictions as an interesting future direction . Our algorithm . To motivate our algorithm , we consider the simplest distribution estimation problem : under communication constraints “ For nm coin tosses with each user observing m coin tosses and sending 1 bit to the center , how do we best estimate the bias of the coin ? ” A key component of our general algorithm is an algorithm for this simplest of settings , which turns out to be surprisingly non-trivial . One may easily come up with simple solutions . One idea is to let each user send 0 if there are more heads than tails and 1 otherwise . This algorithm works well when the coin is almost fair , e.g. , Pr ( head ) œ ( 1/2 ≠ 1/Ôm , 1/2 + 1/Ôm ) , but it is ineffective if the coin is strongly biased . Another idea might be to only let each user indicate whether head appeared or not . This would lead to much better performance for strongly biased coins , e.g. , Pr ( head ) < 1/m , but worse accuracy for fair coins . We elaborate further on this in Section 2.1.1 . In Section 2.1 , we propose an algorithm which estimates the bias of the coin with accuracy O ( 1/mn ) with 1-bit of communication from each user under mild regularity conditions . Note that this is the best rate that can be achieve even when the server has access to all clean samples . Our proposed algorithm not only combines the best of the two naïve algorithms , but further tackles the regime where both solutions fail . It first forms local estimates of the bias of lower accuracy , and then boosts the accuracy by combining local estimates . For the first part , we use a scheme similar to the Gray coding based scheme proposed for Gaussian mean estimation in [ 8 ] , but we need to modify it significantly as the tail probabilities of the underlying distribution can change with the unknown bias . Using this algorithm as a primitive , we form a general algorithm in steps . The first step is to convert the bias estimation algorithm to that for estimating a t-ary distribution using 1-bit communication per user . We do so by partitioning the overall domain [ k ] into t ¥ k/2¸ parts of size at most 2¸ : we then estimate the distribution induced on the parts , using the 1-bit algorithm above by setting t to equal the number of parts . ( Users assigned to this task use their m samples , but only one bit of communication . ) That being done , we estimate the k/2¸ conditional distributions given the sample lies in each of these parts . Since each part has size at most 2¸ , the user can communicate one exact value in these parts using ¸ bits , which allows us to perform this task under our ¸-bit communication constraint , keeping only one of the m samples and transmitting its exact value within a given part . ( Users assigned to this task use their ¸ bits of communication , but only one of their m samples . ) Finally , combining the two types of estimates – of the induced distribution over parts , and of the conditional distribution within each part – allows us to form our overall estimate of the distribution .

Summary: The paper studies the problem of distributed learning of a discrete distribution.  Formally, there are $n$ participants each of whom receives $m$ samples from a discrete distribution $\mathcal{D}$  on $[k]$.  The participants, after seeing their own samples, each send at most $l$ bits of information to a centralized server.  The goal is for the centralized server to output a distribution $\mathcal{D'}$ that is close to $\mathcal{D}$ in TV distance.  The main results of the paper are matching upper and lower bounds in various regimes of $n,m,k, l$, generalizing previous works that had only studied the case of $m = 1$.  Furthermore, the upper bounds are obtained using non-interactive algorithms with no shared randomness while the lower bound holds even against sequential algorithms that use shared randomness.  The main takeaway is a supposed phase transition when $m \sim k/2^l$ where the "value" of additional bits goes from exponential when $m < k/ 2^l$ to polynomial when $m > k/2^l$.