Optimal Rates for Nonparametric Density Estimation under Communication Constraints

1 Introduction . Estimating distributions from samples is a fundamental statistical task . Modern applications , such as those arising in federated learning or the Internet of Things ( IoT ) , often limit access to the true data samples . One common limitation in large scale distributed systems is communication constraints , which require that each data sample must be compressed to a small number of bits . Most prior work on communication-constrained estimation has focused on parametric problems such as Gaussian mean estimation and discrete distribution estimation . In this work , we study nonparametric density estimation under communication constraints where independent samples from an unknown distribution ( whose density f lies in a function class ) are distributed across players ( one sample per player ) , and each player can only send ` bits about its sample to a central referee ; the referee outputs an estimate of f based on these ` -bit messages . This problem has been considered before by [ 5 ] for densities in Hölder classHsL ( [ 0 , 1 ] ) ( functions supported on [ 0 , 1 ] satisfying |f ( x ) − f ( y ) | ≤ L|x− y|s for every x , y ∈ [ 0 , 1 ] ) , which is a relatively simple class for which the normalized histogram with uniform bins is known to be optimal in the centralized setting . This suggests a natural method for distributed setting – quantize each data sample into uniform bins and use the optimal estimator for distributed discrete distribution estimation . Indeed , [ 5 ] shows that this is optimal for distributed estimation of densities from the Hölder class under communication constraints . However , this simple estimator does not seem to extend to the richer Sobolev class and the most general Besov classes . In particular , the following question is largely open : How to quantize samples to estimate densities from Besov classes under communication constraints ? 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We resolve this question both when the density belongs to a Besov class with known parameters ( nonadaptive setting ) , and when the density belongs to a Besov class where only upper and lower bounds on parameters are known ( adaptive setting ) . Specifically , our proposed estimators exploit the sparsity of wavelet basis for the Besov class , and use vector quantization followed by the distributed simulation technique introduced in [ 3 ] for distributed parametric estimation . We also establish lower bounds that prove the optimality of our estimators ( up to logarithmic terms in the adaptive setting ) . 1.1 Problem setup . X1 , . . . , Xn are independent samples from an unknown distribution with density f supported on X : = [ 0 , 1 ] and belonging to the Besov space B ( p , q , s ) ( see Section 2 for details ) . There are n distributed users ( players ) , with player-i having access to sample Xi . Each player can only transmit ` bits about its sample to a central server ( referee ) whose goal , upon observing ` -bit messages from n players , is to estimate f . We consider an interactive setting , where the current player observes the messages from previous players and can use them to design its message.1 That is , in round i , player-i chooses a communication-constrained channel ( randomized mapping ) Wi : X → { 0 , 1 } ` as a function of prior messages Y1 , . . . , Yi−1 and randomness U available to all players ; it then passes Xi through Wi to generate Yi ∈ { 0 , 1 } ` . The referee observes the messages Y1 , . . . , Yn and outputs an estimate f̂ of f . We term such an f̂ an ( n , ` ) -estimate ; let En , ` denote the set of all ( n , ` ) -estimates . Our goal is to design estimators that achieve the minmax expected Lr loss defined , for r ≥ 1 , by L∗r ( n , ` , p , q , s ) : = inf f̂∈En , ` sup f∈B ( p , q , s ) Ef [ ∥∥f̂ − f∥∥r r ] . ( 1 ) For upper bounds on L∗r ( n , ` , p , q , s ) , we consider algorithms that use the more restricted noninteractive protocols , where the channel Wi of player-i is not allowed to depend on the messages Y1 , . . . , Yi−1 or on the common randomness U , but it may depend on the private randomness Ui available at player-i , where U1 , . . . , Un are independent of each other and jointly of U . Noninteractive protocols are easier to implement and result in much simpler engineering for the distributed system . 1.2 Our results and techniques . Our first result is an information-theoretic lower bound on L∗r . Theorem 1.1 . For any p , q , s , r , there exist constants C = C ( p , q , s , r ) > 0 , α = α ( p , q , s , r ) > 0 such that L∗r ( n , ` , p , q , s ) ≥ C · max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } , r ≤ ( s+ 1 ) p , max { n− rs 2s+1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 ( logn2 ` ) −α } , r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , max { ( n logn ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 ( logn2 ` ) −α } , r ≥ ( 2s+ 1 ) p. We emphasize that this lower bound even applies to interactive protocols as defined in Section 1.1 . When the parameters p , q , s of the Besov space are known , we design a noninteractive estimator that achieves the optimal rate when r ≤ p. Theorem 1.2 . For any r ≥ 1 and p , q , s with r ≤ p , there exists a constant C = C ( p , q , s , r ) and an ( n , ` ) -estimate f̂ formed using a noninteractive protocol such that sup f∈B ( p , q , s ) Ef [ ∥∥f̂ − f∥∥r r ] ≤ C max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } . We finally design an adaptive , noninteractive estimator that only requires bounds on s , and no further knowledge of p and q . Moreover , this estimator achieves ( up to logarithmic factors ) the optimal rate for all parameter values . Theorem 1.3 . For any N ∈ N , r ≥ 1 , and p , q , s with 1/p < s < N , there exist constants C = C ( p , q , s , r ) , α = α ( p , q , s , r ) and an ( n , ` ) -estimate f̂ formed using a noninteractive protocol 1Since our lower bounds rely on general results in [ 2 ] , we borrow the notation from that paper . such that sup f∈B ( p , q , s ) Ef [ ∥∥f̂ − f∥∥r r ] ≤ C logα n · max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } , r ≤ ( s+ 1 ) p , max { n− rs 2s+1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } , r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , max { n− r ( s−1/p+1/r ) 2 ( s−1/p ) +1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } , r ≥ ( 2s+ 1 ) p , where the protocol only requires knowledge of N ( an upper bound on s ) . In summary , for all r ≥ 1 , the minmax Lr loss of estimating B ( p , q , s ) ( up to logarithmic factors ) is L∗r ( n , ` , p , q , s ) max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } r ≤ ( s+ 1 ) p , max { n− rs 2s+1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , max { n− r ( s−1/p+1/r ) 2 ( s−1/p ) +1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } r ≥ ( 2s+ 1 ) p. ( 2 ) It it worth noting that the first term of the maximum in all cases is the standard , unconstrained nonparametric rate ( cf . [ 10 ] , or the discussion below ) , while the second term reflects the convergence slowdown due to the communication constraints . The effect of communication constraints disappears when ` is sufficiently large . In particular , we get back the centralized rates when ` satisfies ` ≥ ( 1 2s+1 ) logn , if r ≤ ( s+ 1 ) p , ( 2s ( 1−1/r ) ( 2s+1 ) ( s−1/p+1/r ) − 1 2s+1 ) logn , if r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , ( 1 2 ( s−1/p ) +1 ) logn , if r ≥ ( 2s+ 1 ) p. For the standard case of L2 loss , with , say p ≥ 2 , the minmax rate becomes the more interpretable quantity L∗2 ( n , ` , p , q , s ) max { n− 2s 2s+1 , ( n2 ` ) − 2s2s+2 } , where we see that the ` -bit communication constraint reduces the exponent of the convergence rate from 2s2s+1 to 2s 2s+2 . Note that the difference becomes less perceptible as ` grows , or s tends to∞ . Finally , from ( 2 ) we observe qualitative changes at r = ( s + 1 ) p and r = ( 2s + 1 ) p , where the rate exponent changes slope . This phenomenon , whose analogue is observed in the unconstrained setting [ 10 ] as well as under local privacy constraints [ 7 ] , is sometimes referred to as an elbow effect . Quantize , simulate and infer . A conceptually simple technique for distributed estimation under communication constraints ( “ simulate-and-infer ” ) was proposed in [ 3 ] , which uses communication to simulate samples from the unknown distribution , and provides an optimal rate estimator for discrete distribution estimation under communication constraints . A natural approach for nonparametric estimation would be to quantize the samples to the available number of bits ( ` ) and use this quantized sample to estimate the distribution . However , it is unclear if this approach gives optimal rates . Instead , in our approach , we quantize without inducing any “ loss of information. ” Specifically , we form an approximately sufficient statistic ( based on wavelets ) that can be represented using finite bits and does not result in rate-loss . The number of bits could still be more than ` , and therefore , we use simulate-and-infer to generate samples from the statistic . Thus , loss of information due to communication constraints only happens in the last step , when we use multiple samples to simulate a sample from the sufficient statistic ; the quantization part is just for efficient finite representation . Sobolev spaces and Fourier bases . A first approach that we tried for Sobolev spaces was to use the Fourier basis , a natural choice for a Sobolev space . However , all our attempts led to a suboptimal performance either in the dependence on ` ( we were not able to get the exponential 2− ` dependence ) or the exponent of n ( i.e. , when we tried to get 2− ` dependence , this resulted in a suboptimal exponent of n ) . Somewhat surprisingly , the more general wavelet-based approach described above gives us tight bounds for Sobolev space as well , since Sobolev space S ( β ) = B ( 2 , 2 , β ) . Thus , it seems that it is necessary to use the wavelet representation even for Sobolev spaces to get an appropriately “ small ” statistic – sparsity of wavelets is very useful for inference under communication constraints . Organization . After discussing prior works ( Section 1.3 ) and preliminaries ( Section 2 ) , we describe our estimation algorithms in Section 3 . In Section 3.1 , we present the nonadaptive single-level estimator ( comprising Algorithm 2 and Algorithm 3 ) , which achieves guarantees of Theorem 1.2 ( proof in Appendix B ) . In Section 3.2 , we present the adaptive multi-level estimator ( comprising Algorithm 4 and Algorithm 5 ) , which achieves guarantees of Theorem 1.3 ( proof in Appendix C ) . Finally , in Section 4 , we briefly discuss our approach to proving lower bounds of Theorem 1.1 , with details deferred to Appendix D. Appendices can be found in the Supplementary .

Summary: This problem of estimating a density, assumed to lie in a Besov class, under communication constraints in which each sample must be encoded using a fixed number $\ell$ of bits before computing the density estimate. Both the non-adaptive and adaptive cases are considered, with matching upper and lower bounds (up to polylogarithmic factors) obtained in both cases. The matching bounds also indicate that the rate is the same for interactive and non-interactive protocols. The proposed methods utilizes 3 main ingredients: 1. The fact that at most a constant number of wavelets at resolution J are non-zero on any interval of width $2^{-J}$. 2. A particular randomized scheme that quantizes the values of the non-zero wavelets at a sample point without introducing bias. 3. The DISTRSIM protocol proposed by [3], which maps a quantized sample to a smaller IID sample of higher resolution.