Pointwise Bounds for Distribution Estimation under Communication Constraints

⇣ max ⇣kpk1/2 n2b , 1 n ⌘⌘ when n is sufficiently large , hence it is governed by the half-norm of p instead of the ambient dimension d. For the achievability result , we propose a two-round sequentially interactive estimation scheme that achieves this error rate uniformly over all p. This two-round scheme extends to ` q loss with q 1 , and hence gives pointwise upper bounds on ` q error . We also develop a new local minimax lower bound with ( almost ) matching ` 2 error , showing that any interactive scheme must admit a ⌦ ⇣kpk ( 1+ ) /2 n2b ⌘ ` 2 error for any > 0 . Our upper and lower bounds together indicate that the H1/2 ( p ) , log ( kpk1/2 ) bits of communication is both sufficient and necessary to achieve the optimal ( centralized ) performance , where H1/2 ( p ) is the Rényi entropy of order 2 . Therefore , under the ` 2 loss , the correct measure of the local communication complexity at p is its Rényi entropy . 1 Introduction . Learning a distribution from its samples has been a fundamental task in unsupervised learning dating back to the late nineteenth century [ 1 ] . This task , especially under distributed settings , has gained growing popularity in the recent years as data is increasingly generated “ at the edge ” by countless sensors , smartphones , and other devices . When data is distributed across multiple devices , communication cost and bandwidth often become a bottleneck hampering the training of highaccuracy machine learning models [ 2 ] . This is even more so for federated learning and analytics type settings [ 3 ] which rely on wireless mobile links for communication . To resolve this issue , several communication-efficient distribution learning schemes have been recently proposed and studied in the literature ( see Section 1.2 for a thorough discussion ) . On the positive side , the state-of-the-art schemes are known to be worst-case ( minimax ) optimal as they have been shown to achieve the information-theoretic lower bounds on the global minimax error [ 4–8 ] . On the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . negative side , however , the ` 2 estimation error achieved by these schemes scales as O ( dn2b ) under a b-bit communication constraint on each sample , where d is the alphabet size of the unknown discrete distribution p. This suggests that without additional assumptions on p the error scales linearly in d , i.e . the introduction of communication constraints introduces a penalty d on the estimation accuracy . This is true even if we allow for interaction between clients [ 8 , 9 ] . A recent work [ 10 ] has moved a step forward from the “ global minimax regime ” by restricting the target distribution p to be s-sparse and showing that the ` 2 error can be reduced to O ( s log dn2b ) in this case , i.e . the error depends on the sparsity s rather than the ambient dimension d. However , their scheme heavily relies on the s-sparse assumption and requires knowing s beforehand . Therefore , when s is unknown and we need to work with a ( potentially loose ) upper bound on s , the estimation error increases accordingly . In addition , little is known when the target distribution deviates slightly from being exactly s-sparse . In this paper , we argue that all these results can be overly pessimistic , as worst-case notions of complexity and schemes designed to optimize these worst-case notions can be too conservative . Instead , we seek a measure of local complexity that captures the hardness of estimating a specific instance p. Ideally , we want a scheme that adapts to the hardness of the problem instead of being tuned to the worst-case scenario ; that is , a scheme achieving smaller error when p is “ simpler. ” Our contributions Motivated by these observations , in this work we consider the local minimax complexity of distribution estimation and quantify the hardness of estimating a specific p under communication constraints . In particular , under the ` 2 loss , we show that the local complexity of estimating p is captured by its half-norm1 kpk 1 2 : we propose a two-round interactive scheme that uniformly achieves the O ⇣ max ⇣kpk1/2 n2b , 1 n ⌘⌘ error under ` 2 loss2 which requires no prior information on p. On the impossibility side , we also show that for any ( arbitrarily ) interactive scheme , the local minimax error ( which is formally defined in Theorem 2.4 ) must be at least ⌦ ⇣ max ⇣kpk1+ /2 n2b , 1 n ⌘⌘ for any > 0 when n is sufficiently large . These upper and the lower bounds together indicate that kpk1/2 plays a fundamental role in distributed estimation and that the dlog ( kpk1/2 ) e bits of communication is both sufficient and necessary to achieve the optimal ( centralized ) performance when n is large enough . Indeed , this quantity is exactly the Rényi entropy of order 2 , i.e . H1/2 ( p ) , log ( kpk1/2 ) , showing that under the ` 2 loss , the correct measure of the local communication complexity at p is the Rényi entropy of p. Compared to the global minimax results where the error scales as O ( dn2b ) , we see that when we move toward the local regime , the linear dependency on d in the convergence rate is replaced by kpk1/2 . This dimension independent convergence is also empirically verified by our experiments ( see Section 3 for more details ) . Note that kpk1/2 < d , so our proposed scheme is also globally minimax optimal . Moreover , since kpk1/2 < kpk0 , our scheme achieves the O ( s n2b ) convergence rate under the s-sparse model [ 10 ] , which improves the O ( s log ( d/s ) n2b ) upper bound in [ 10 ] by further shaving off the additional log ( d/s ) term ( though admittedly , their scheme is designed under the more stringent non-interactive setting ) . As another immediate corollary , our pointwise upper bounds indicate that 1 bit suffices to attain the performance of the centralized model when the target distribution is highly skewed , such as the ( truncated ) Geometric distributions and Zipf distributions with degree greater than two . Our techniques Our proposed two-round interactive scheme is based on a local refinement approach , where in the first round , a standard ( global ) minimax optimal estimation scheme is applied to localize p. In the second round , we use additional ⇥ ( n ) samples ( clients ) , together with the information obtained from the previous round , to locally refine the estimate . The localizationrefinement procedure enables us to tune the encoders ( in the second round ) to the target distribution p and hence attain the optimal pointwise convergence rate uniformly . On the other hand , our lower bound is based on the quantized Fisher information framework introduced in [ 11 ] . However , in order to obtain a local bound around p , we develop a new approach that 1Here we generalize the notion of q-norm kpkq , ⇣Pd i=1 p q i ⌘1/q to q 2 [ 0 , 1 ] . 2Our scheme also guarantees pointwise upper bounds on ` 1 or general ` q errors . See Theorem 2.2. first finds the best parametric sub-model containing p , and then upper bounds its Fisher information in a neighborhood around p. To the best of our knowledge , this is the first impossibility result that allows to capture the local complexity of high dimensional estimation under information constraints and can be of independent interest , e.g . to derive pointwise lower bounds for other estimation models under general information constraints . 1.1 Notation and Setup . The general distributed statistical task we consider in this paper can be formulated as follows . Each one of the n clients has local data Xi ⇠ p , where p 2 Pd and Pd , n p 2 Rd+ P j pj = 1 o is the collection of all d-dimensional discrete distributions . The i-th client then sends a message Yi 2 Y to the server , who upon receiving Y n aims to estimate the unknown distribution p. At client i , the message Yi is generated via a sequentially interactive protocol ; that is , samples are communicated sequentially by broadcasting the communication to all nodes in the system including the server . Therefore , the encoding function Wi of the i-th client can depend on all previous messages Y1 , ... , Yi 1 . Formally , it can be written as a randomized mapping ( possibly using shared randomness across participating clients and the server ) of the form Wi ( ·|Xi , Y i 1 ) , and the b-bit communication constraint restricts |Y| 2b . As a special case , when Wi depends only on Xi and is independent of the other messages Y i 1 for all i ( i.e . Wi ( ·|Xi , Y i 1 ) = Wi ( ·|Xi ) ) , we say the corresponding protocol is non-interactive . Finally , we call the tuple ( Wn , p̂ ( Y n ) ) an estimation scheme , where p̂ ( Y n ) is an estimator of p. We use ⇧seq and ⇧ind to denote the collections of all sequentially interactive and non-interactive schemes respectively . Our goal here is to design a scheme ( Wn , p̂ ( Y n ) ) to minimize the ` 2 ( or ` 1 ) estimation error : r ( ` 2 , p , ( Wn , p̂ ) ) , E [ kp p̂ ( Y n ) k22 ] , for all p 2 Pd , as well as characterizing the best error achievable by any scheme in ⇧seq . We note that while our impossibility bounds hold for any scheme in ⇧seq , the particular scheme we propose uses only one round of interaction.For ` 1 error , we replace k·k22 in the above expectations with k·k1 . In this work , we mainly focus on the regime 1 ⌧ d ⌧ n , aiming to characterize the statistical convergence rates when n is sufficiently large . 1.2 Related works . Estimating discrete distributions is a fundamental task in statistical inference and has a rich literature [ 12–15 ] . Under communication constraints , the optimal convergence rate for discrete distribution estimation was established in [ 4–8 , 16 ] for the non-interactive setting , and [ 8 , 9 ] for the general ( blackboard ) interactive model . The recent work [ 10 ] considers the same task under the s sparse assumption for the distribution under communication or privacy constraints . However , all these works study the global minimax error and focus on minimizing the worst-case estimation error . Hence the resultant schemes are tuned to minimize the error in the worst-case which may be too pessimistic for most real-world applications . A slightly different but closely related problem is distributed estimation of distributions [ 16–20 ] and heavy hitter detection under local differential privacy ( LDP ) constraints [ 21–24 ] . Although originally designed for preserving user privacy , some of these schemes can be made communication efficient . For instance , the scheme in [ 20 ] suggests that one can use 1 bit communication to achieve O ( d/n ) ` 2 error ( which is global minimax optimal ) ; the non-interactive tree-based schemes proposed in [ 22 , 23 ] can be cast into a 1-bit frequency oracle , but since the scheme is optimized with respect to ` 1 error , directly applying their frequency oracle leads to a sub-optimal convergence O ⇣ s ( log d+logn ) n ⌘ in ` 2 . In the line of distributed estimation under LDP constraint , the recent work [ 25 ] studies local minimax lower bounds and shows that the local modulus of continuity with respect to the variation distance governs the rate of convergence under LDP . However , the notion of the local minimax error in [ 25 ] is based on the two-point method , so their characterized lower bounds may not be uniformly attainable in general high-dimensional settings . Organization The rest of the paper is organized as follows . In Section 2 , we present our main results , including pointwise upper bounds and an ( almost ) matching local minimax lower bound . We provide examples and experiments in Section 3 to demonstrate how our pointwise bounds improve upon previous global minimax results . In Section 4 , we introduce our two-round interactive scheme that achieves the pointwise bound . The analysis of this scheme , as well as the proof of the upper bounds , is given in Section 5 . Finally , in Section 6 we provide the proof of the local minimax lower bound .

Summary: The paper studies distributed estimation of a discrete, $d$-dimensional law $p$ under communication constraints (wherein each client may only send $b$ bits) in a sequential model (wherein clients are ordered, and are aware of messages sent by earlier clients when preparing their message).   The authors describe what is essentially a two-stage scheme that first obtains a crude estimate of the law using a minimax scheme. This estimate yields a sense of which elements of the law are large, and subsequent communications are designed that provide more samples for such elements, the precise weighting of which depends on the error metric being controlled. As a result, if the number of clients is large enough ($n \gg d^3\log d$ for squared-$\ell_2$ error), the resulting estimate for the squared-$\ell_2$ criterion has error decaying as $\max( \\| p\\|_{1/2}2^{-b} , 1)/n$. Both stages of the scheme use the grouping idea of Han et al [4].  The above scheme is complemented by a lower bound for the $\ell_2$ criterion, which shows that under the same large $n$ condition, a matching lower bound but with a $(1+\delta)/2$ norm instead of the $1/2$ norm, lending a measure of tightness to the achievability analysis. This is derived via a Fisher information technique due to Barnes et al, and a nice construction of a parametric family of discrete laws in which the lowest $d-h$ masses are fixed. Under some auxiliary conditions, the resulting bound depends linearly on $h^2 p_{(h)},$ where $p_{(h)}$ is the $h$th biggest entry of $p$, and this quantity is characterised in terms of the $1/2$ and $(1+\delta)/2$ norms.  Together, these results almost characterise the bit-rate needed by each client to ensure optimal estimation, specifically the $1/2$-Renyi entropy of $p.$ This immediately leads to improved rates for spase, and for highly skewed laws.