Optimal Rates for Nonparametric Density Estimation under Communication Constraints

1 Introduction . Estimating distributions from samples is a fundamental statistical task . Modern applications , such as those arising in federated learning or the Internet of Things ( IoT ) , often limit access to the true data samples . One common limitation in large scale distributed systems is communication constraints , which require that each data sample must be compressed to a small number of bits . Most prior work on communication-constrained estimation has focused on parametric problems such as Gaussian mean estimation and discrete distribution estimation . In this work , we study nonparametric density estimation under communication constraints where independent samples from an unknown distribution ( whose density f lies in a function class ) are distributed across players ( one sample per player ) , and each player can only send ` bits about its sample to a central referee ; the referee outputs an estimate of f based on these ` -bit messages . This problem has been considered before by [ 5 ] for densities in Hölder classHsL ( [ 0 , 1 ] ) ( functions supported on [ 0 , 1 ] satisfying |f ( x ) − f ( y ) | ≤ L|x− y|s for every x , y ∈ [ 0 , 1 ] ) , which is a relatively simple class for which the normalized histogram with uniform bins is known to be optimal in the centralized setting . This suggests a natural method for distributed setting – quantize each data sample into uniform bins and use the optimal estimator for distributed discrete distribution estimation . Indeed , [ 5 ] shows that this is optimal for distributed estimation of densities from the Hölder class under communication constraints . However , this simple estimator does not seem to extend to the richer Sobolev class and the most general Besov classes . In particular , the following question is largely open : How to quantize samples to estimate densities from Besov classes under communication constraints ? 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We resolve this question both when the density belongs to a Besov class with known parameters ( nonadaptive setting ) , and when the density belongs to a Besov class where only upper and lower bounds on parameters are known ( adaptive setting ) . Specifically , our proposed estimators exploit the sparsity of wavelet basis for the Besov class , and use vector quantization followed by the distributed simulation technique introduced in [ 3 ] for distributed parametric estimation . We also establish lower bounds that prove the optimality of our estimators ( up to logarithmic terms in the adaptive setting ) . 1.1 Problem setup . X1 , . . . , Xn are independent samples from an unknown distribution with density f supported on X : = [ 0 , 1 ] and belonging to the Besov space B ( p , q , s ) ( see Section 2 for details ) . There are n distributed users ( players ) , with player-i having access to sample Xi . Each player can only transmit ` bits about its sample to a central server ( referee ) whose goal , upon observing ` -bit messages from n players , is to estimate f . We consider an interactive setting , where the current player observes the messages from previous players and can use them to design its message.1 That is , in round i , player-i chooses a communication-constrained channel ( randomized mapping ) Wi : X → { 0 , 1 } ` as a function of prior messages Y1 , . . . , Yi−1 and randomness U available to all players ; it then passes Xi through Wi to generate Yi ∈ { 0 , 1 } ` . The referee observes the messages Y1 , . . . , Yn and outputs an estimate f̂ of f . We term such an f̂ an ( n , ` ) -estimate ; let En , ` denote the set of all ( n , ` ) -estimates . Our goal is to design estimators that achieve the minmax expected Lr loss defined , for r ≥ 1 , by L∗r ( n , ` , p , q , s ) : = inf f̂∈En , ` sup f∈B ( p , q , s ) Ef [ ∥∥f̂ − f∥∥r r ] . ( 1 ) For upper bounds on L∗r ( n , ` , p , q , s ) , we consider algorithms that use the more restricted noninteractive protocols , where the channel Wi of player-i is not allowed to depend on the messages Y1 , . . . , Yi−1 or on the common randomness U , but it may depend on the private randomness Ui available at player-i , where U1 , . . . , Un are independent of each other and jointly of U . Noninteractive protocols are easier to implement and result in much simpler engineering for the distributed system . 1.2 Our results and techniques . Our first result is an information-theoretic lower bound on L∗r . Theorem 1.1 . For any p , q , s , r , there exist constants C = C ( p , q , s , r ) > 0 , α = α ( p , q , s , r ) > 0 such that L∗r ( n , ` , p , q , s ) ≥ C · max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } , r ≤ ( s+ 1 ) p , max { n− rs 2s+1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 ( logn2 ` ) −α } , r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , max { ( n logn ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 ( logn2 ` ) −α } , r ≥ ( 2s+ 1 ) p. We emphasize that this lower bound even applies to interactive protocols as defined in Section 1.1 . When the parameters p , q , s of the Besov space are known , we design a noninteractive estimator that achieves the optimal rate when r ≤ p. Theorem 1.2 . For any r ≥ 1 and p , q , s with r ≤ p , there exists a constant C = C ( p , q , s , r ) and an ( n , ` ) -estimate f̂ formed using a noninteractive protocol such that sup f∈B ( p , q , s ) Ef [ ∥∥f̂ − f∥∥r r ] ≤ C max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } . We finally design an adaptive , noninteractive estimator that only requires bounds on s , and no further knowledge of p and q . Moreover , this estimator achieves ( up to logarithmic factors ) the optimal rate for all parameter values . Theorem 1.3 . For any N ∈ N , r ≥ 1 , and p , q , s with 1/p < s < N , there exist constants C = C ( p , q , s , r ) , α = α ( p , q , s , r ) and an ( n , ` ) -estimate f̂ formed using a noninteractive protocol 1Since our lower bounds rely on general results in [ 2 ] , we borrow the notation from that paper . such that sup f∈B ( p , q , s ) Ef [ ∥∥f̂ − f∥∥r r ] ≤ C logα n · max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } , r ≤ ( s+ 1 ) p , max { n− rs 2s+1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } , r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , max { n− r ( s−1/p+1/r ) 2 ( s−1/p ) +1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } , r ≥ ( 2s+ 1 ) p , where the protocol only requires knowledge of N ( an upper bound on s ) . In summary , for all r ≥ 1 , the minmax Lr loss of estimating B ( p , q , s ) ( up to logarithmic factors ) is L∗r ( n , ` , p , q , s ) max { n− rs 2s+1 , ( n2 ` ) − rs2s+2 } r ≤ ( s+ 1 ) p , max { n− rs 2s+1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , max { n− r ( s−1/p+1/r ) 2 ( s−1/p ) +1 , ( n2 ` ) − r ( s−1/p+1/r ) 2 ( s−1/p ) +2 } r ≥ ( 2s+ 1 ) p. ( 2 ) It it worth noting that the first term of the maximum in all cases is the standard , unconstrained nonparametric rate ( cf . [ 10 ] , or the discussion below ) , while the second term reflects the convergence slowdown due to the communication constraints . The effect of communication constraints disappears when ` is sufficiently large . In particular , we get back the centralized rates when ` satisfies ` ≥ ( 1 2s+1 ) logn , if r ≤ ( s+ 1 ) p , ( 2s ( 1−1/r ) ( 2s+1 ) ( s−1/p+1/r ) − 1 2s+1 ) logn , if r ∈ ( ( s+ 1 ) p , ( 2s+ 1 ) p ) , ( 1 2 ( s−1/p ) +1 ) logn , if r ≥ ( 2s+ 1 ) p. For the standard case of L2 loss , with , say p ≥ 2 , the minmax rate becomes the more interpretable quantity L∗2 ( n , ` , p , q , s ) max { n− 2s 2s+1 , ( n2 ` ) − 2s2s+2 } , where we see that the ` -bit communication constraint reduces the exponent of the convergence rate from 2s2s+1 to 2s 2s+2 . Note that the difference becomes less perceptible as ` grows , or s tends to∞ . Finally , from ( 2 ) we observe qualitative changes at r = ( s + 1 ) p and r = ( 2s + 1 ) p , where the rate exponent changes slope . This phenomenon , whose analogue is observed in the unconstrained setting [ 10 ] as well as under local privacy constraints [ 7 ] , is sometimes referred to as an elbow effect . Quantize , simulate and infer . A conceptually simple technique for distributed estimation under communication constraints ( “ simulate-and-infer ” ) was proposed in [ 3 ] , which uses communication to simulate samples from the unknown distribution , and provides an optimal rate estimator for discrete distribution estimation under communication constraints . A natural approach for nonparametric estimation would be to quantize the samples to the available number of bits ( ` ) and use this quantized sample to estimate the distribution . However , it is unclear if this approach gives optimal rates . Instead , in our approach , we quantize without inducing any “ loss of information. ” Specifically , we form an approximately sufficient statistic ( based on wavelets ) that can be represented using finite bits and does not result in rate-loss . The number of bits could still be more than ` , and therefore , we use simulate-and-infer to generate samples from the statistic . Thus , loss of information due to communication constraints only happens in the last step , when we use multiple samples to simulate a sample from the sufficient statistic ; the quantization part is just for efficient finite representation . Sobolev spaces and Fourier bases . A first approach that we tried for Sobolev spaces was to use the Fourier basis , a natural choice for a Sobolev space . However , all our attempts led to a suboptimal performance either in the dependence on ` ( we were not able to get the exponential 2− ` dependence ) or the exponent of n ( i.e. , when we tried to get 2− ` dependence , this resulted in a suboptimal exponent of n ) . Somewhat surprisingly , the more general wavelet-based approach described above gives us tight bounds for Sobolev space as well , since Sobolev space S ( β ) = B ( 2 , 2 , β ) . Thus , it seems that it is necessary to use the wavelet representation even for Sobolev spaces to get an appropriately “ small ” statistic – sparsity of wavelets is very useful for inference under communication constraints . Organization . After discussing prior works ( Section 1.3 ) and preliminaries ( Section 2 ) , we describe our estimation algorithms in Section 3 . In Section 3.1 , we present the nonadaptive single-level estimator ( comprising Algorithm 2 and Algorithm 3 ) , which achieves guarantees of Theorem 1.2 ( proof in Appendix B ) . In Section 3.2 , we present the adaptive multi-level estimator ( comprising Algorithm 4 and Algorithm 5 ) , which achieves guarantees of Theorem 1.3 ( proof in Appendix C ) . Finally , in Section 4 , we briefly discuss our approach to proving lower bounds of Theorem 1.1 , with details deferred to Appendix D. Appendices can be found in the Supplementary .

Summary: The paper addresses learning densities in Bezov space under communication constraints, primarily in a non-interactive setting. In fact, Theorems 1.1 and 1.3 together suggest that interaction may not help much in achieving a better performance. The proposed algorithms leverage wavelet-induced sparsity, vector quantization, and distributed (sample) simulation, but not the standard Fourier basis.