Pointwise Bounds for Distribution Estimation under Communication Constraints

⇣ max ⇣kpk1/2 n2b , 1 n ⌘⌘ when n is sufficiently large , hence it is governed by the half-norm of p instead of the ambient dimension d. For the achievability result , we propose a two-round sequentially interactive estimation scheme that achieves this error rate uniformly over all p. This two-round scheme extends to ` q loss with q 1 , and hence gives pointwise upper bounds on ` q error . We also develop a new local minimax lower bound with ( almost ) matching ` 2 error , showing that any interactive scheme must admit a ⌦ ⇣kpk ( 1+ ) /2 n2b ⌘ ` 2 error for any > 0 . Our upper and lower bounds together indicate that the H1/2 ( p ) , log ( kpk1/2 ) bits of communication is both sufficient and necessary to achieve the optimal ( centralized ) performance , where H1/2 ( p ) is the Rényi entropy of order 2 . Therefore , under the ` 2 loss , the correct measure of the local communication complexity at p is its Rényi entropy . 1 Introduction . Learning a distribution from its samples has been a fundamental task in unsupervised learning dating back to the late nineteenth century [ 1 ] . This task , especially under distributed settings , has gained growing popularity in the recent years as data is increasingly generated “ at the edge ” by countless sensors , smartphones , and other devices . When data is distributed across multiple devices , communication cost and bandwidth often become a bottleneck hampering the training of highaccuracy machine learning models [ 2 ] . This is even more so for federated learning and analytics type settings [ 3 ] which rely on wireless mobile links for communication . To resolve this issue , several communication-efficient distribution learning schemes have been recently proposed and studied in the literature ( see Section 1.2 for a thorough discussion ) . On the positive side , the state-of-the-art schemes are known to be worst-case ( minimax ) optimal as they have been shown to achieve the information-theoretic lower bounds on the global minimax error [ 4–8 ] . On the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . negative side , however , the ` 2 estimation error achieved by these schemes scales as O ( dn2b ) under a b-bit communication constraint on each sample , where d is the alphabet size of the unknown discrete distribution p. This suggests that without additional assumptions on p the error scales linearly in d , i.e . the introduction of communication constraints introduces a penalty d on the estimation accuracy . This is true even if we allow for interaction between clients [ 8 , 9 ] . A recent work [ 10 ] has moved a step forward from the “ global minimax regime ” by restricting the target distribution p to be s-sparse and showing that the ` 2 error can be reduced to O ( s log dn2b ) in this case , i.e . the error depends on the sparsity s rather than the ambient dimension d. However , their scheme heavily relies on the s-sparse assumption and requires knowing s beforehand . Therefore , when s is unknown and we need to work with a ( potentially loose ) upper bound on s , the estimation error increases accordingly . In addition , little is known when the target distribution deviates slightly from being exactly s-sparse . In this paper , we argue that all these results can be overly pessimistic , as worst-case notions of complexity and schemes designed to optimize these worst-case notions can be too conservative . Instead , we seek a measure of local complexity that captures the hardness of estimating a specific instance p. Ideally , we want a scheme that adapts to the hardness of the problem instead of being tuned to the worst-case scenario ; that is , a scheme achieving smaller error when p is “ simpler. ” Our contributions Motivated by these observations , in this work we consider the local minimax complexity of distribution estimation and quantify the hardness of estimating a specific p under communication constraints . In particular , under the ` 2 loss , we show that the local complexity of estimating p is captured by its half-norm1 kpk 1 2 : we propose a two-round interactive scheme that uniformly achieves the O ⇣ max ⇣kpk1/2 n2b , 1 n ⌘⌘ error under ` 2 loss2 which requires no prior information on p. On the impossibility side , we also show that for any ( arbitrarily ) interactive scheme , the local minimax error ( which is formally defined in Theorem 2.4 ) must be at least ⌦ ⇣ max ⇣kpk1+ /2 n2b , 1 n ⌘⌘ for any > 0 when n is sufficiently large . These upper and the lower bounds together indicate that kpk1/2 plays a fundamental role in distributed estimation and that the dlog ( kpk1/2 ) e bits of communication is both sufficient and necessary to achieve the optimal ( centralized ) performance when n is large enough . Indeed , this quantity is exactly the Rényi entropy of order 2 , i.e . H1/2 ( p ) , log ( kpk1/2 ) , showing that under the ` 2 loss , the correct measure of the local communication complexity at p is the Rényi entropy of p. Compared to the global minimax results where the error scales as O ( dn2b ) , we see that when we move toward the local regime , the linear dependency on d in the convergence rate is replaced by kpk1/2 . This dimension independent convergence is also empirically verified by our experiments ( see Section 3 for more details ) . Note that kpk1/2 < d , so our proposed scheme is also globally minimax optimal . Moreover , since kpk1/2 < kpk0 , our scheme achieves the O ( s n2b ) convergence rate under the s-sparse model [ 10 ] , which improves the O ( s log ( d/s ) n2b ) upper bound in [ 10 ] by further shaving off the additional log ( d/s ) term ( though admittedly , their scheme is designed under the more stringent non-interactive setting ) . As another immediate corollary , our pointwise upper bounds indicate that 1 bit suffices to attain the performance of the centralized model when the target distribution is highly skewed , such as the ( truncated ) Geometric distributions and Zipf distributions with degree greater than two . Our techniques Our proposed two-round interactive scheme is based on a local refinement approach , where in the first round , a standard ( global ) minimax optimal estimation scheme is applied to localize p. In the second round , we use additional ⇥ ( n ) samples ( clients ) , together with the information obtained from the previous round , to locally refine the estimate . The localizationrefinement procedure enables us to tune the encoders ( in the second round ) to the target distribution p and hence attain the optimal pointwise convergence rate uniformly . On the other hand , our lower bound is based on the quantized Fisher information framework introduced in [ 11 ] . However , in order to obtain a local bound around p , we develop a new approach that 1Here we generalize the notion of q-norm kpkq , ⇣Pd i=1 p q i ⌘1/q to q 2 [ 0 , 1 ] . 2Our scheme also guarantees pointwise upper bounds on ` 1 or general ` q errors . See Theorem 2.2. first finds the best parametric sub-model containing p , and then upper bounds its Fisher information in a neighborhood around p. To the best of our knowledge , this is the first impossibility result that allows to capture the local complexity of high dimensional estimation under information constraints and can be of independent interest , e.g . to derive pointwise lower bounds for other estimation models under general information constraints . 1.1 Notation and Setup . The general distributed statistical task we consider in this paper can be formulated as follows . Each one of the n clients has local data Xi ⇠ p , where p 2 Pd and Pd , n p 2 Rd+ P j pj = 1 o is the collection of all d-dimensional discrete distributions . The i-th client then sends a message Yi 2 Y to the server , who upon receiving Y n aims to estimate the unknown distribution p. At client i , the message Yi is generated via a sequentially interactive protocol ; that is , samples are communicated sequentially by broadcasting the communication to all nodes in the system including the server . Therefore , the encoding function Wi of the i-th client can depend on all previous messages Y1 , ... , Yi 1 . Formally , it can be written as a randomized mapping ( possibly using shared randomness across participating clients and the server ) of the form Wi ( ·|Xi , Y i 1 ) , and the b-bit communication constraint restricts |Y| 2b . As a special case , when Wi depends only on Xi and is independent of the other messages Y i 1 for all i ( i.e . Wi ( ·|Xi , Y i 1 ) = Wi ( ·|Xi ) ) , we say the corresponding protocol is non-interactive . Finally , we call the tuple ( Wn , p̂ ( Y n ) ) an estimation scheme , where p̂ ( Y n ) is an estimator of p. We use ⇧seq and ⇧ind to denote the collections of all sequentially interactive and non-interactive schemes respectively . Our goal here is to design a scheme ( Wn , p̂ ( Y n ) ) to minimize the ` 2 ( or ` 1 ) estimation error : r ( ` 2 , p , ( Wn , p̂ ) ) , E [ kp p̂ ( Y n ) k22 ] , for all p 2 Pd , as well as characterizing the best error achievable by any scheme in ⇧seq . We note that while our impossibility bounds hold for any scheme in ⇧seq , the particular scheme we propose uses only one round of interaction.For ` 1 error , we replace k·k22 in the above expectations with k·k1 . In this work , we mainly focus on the regime 1 ⌧ d ⌧ n , aiming to characterize the statistical convergence rates when n is sufficiently large . 1.2 Related works . Estimating discrete distributions is a fundamental task in statistical inference and has a rich literature [ 12–15 ] . Under communication constraints , the optimal convergence rate for discrete distribution estimation was established in [ 4–8 , 16 ] for the non-interactive setting , and [ 8 , 9 ] for the general ( blackboard ) interactive model . The recent work [ 10 ] considers the same task under the s sparse assumption for the distribution under communication or privacy constraints . However , all these works study the global minimax error and focus on minimizing the worst-case estimation error . Hence the resultant schemes are tuned to minimize the error in the worst-case which may be too pessimistic for most real-world applications . A slightly different but closely related problem is distributed estimation of distributions [ 16–20 ] and heavy hitter detection under local differential privacy ( LDP ) constraints [ 21–24 ] . Although originally designed for preserving user privacy , some of these schemes can be made communication efficient . For instance , the scheme in [ 20 ] suggests that one can use 1 bit communication to achieve O ( d/n ) ` 2 error ( which is global minimax optimal ) ; the non-interactive tree-based schemes proposed in [ 22 , 23 ] can be cast into a 1-bit frequency oracle , but since the scheme is optimized with respect to ` 1 error , directly applying their frequency oracle leads to a sub-optimal convergence O ⇣ s ( log d+logn ) n ⌘ in ` 2 . In the line of distributed estimation under LDP constraint , the recent work [ 25 ] studies local minimax lower bounds and shows that the local modulus of continuity with respect to the variation distance governs the rate of convergence under LDP . However , the notion of the local minimax error in [ 25 ] is based on the two-point method , so their characterized lower bounds may not be uniformly attainable in general high-dimensional settings . Organization The rest of the paper is organized as follows . In Section 2 , we present our main results , including pointwise upper bounds and an ( almost ) matching local minimax lower bound . We provide examples and experiments in Section 3 to demonstrate how our pointwise bounds improve upon previous global minimax results . In Section 4 , we introduce our two-round interactive scheme that achieves the pointwise bound . The analysis of this scheme , as well as the proof of the upper bounds , is given in Section 5 . Finally , in Section 6 we provide the proof of the local minimax lower bound .

Summary: This paper considers the estimation of a discrete distribution P on an alphabet of size d under communication constraints. Each local client holds a single observation from P and transmits no more than b bits to the central server. A two-round interactive scheme is proposed and achieves nearly optimal estimation error. The estimation error depends on the Rényi entropy of P instead of the global parameter d. The minimax lower bound is also derived in this paper.