Title: Unified Lower Bounds for Interactive High-dimensional Estimation under Information Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: statistical estimation; interactivity; local differential privacy; communication constraint
Abstract: We consider distributed parameter estimation using interactive protocols subject to local information constraints such as bandwidth limitations, local differential privacy, and restricted measurements. We provide a unified framework enabling us to derive a variety of (tight) minimax lower bounds for different parametric families of distributions, both continuous and discrete, under any $\ell_p$ loss. Our lower bound framework is versatile and yields “plug-and-play” bounds that are widely applicable to a large range of estimation problems, and, for the prototypical case of the Gaussian family, circumvents limitations of previous techniques. In particular, our approach recovers bounds obtained using data processing inequalities and Cramér–Rao bounds, two other alternative approaches for proving lower bounds in our setting of interest. Further, for the families considered, we complement our lower bounds with matching upper bounds.

Unified lower bounds for interactive high-dimensional estimation under information constraints

Abstract

We consider distributed parameter estimation using interactive protocols subject to local information constraints such as bandwidth limitations, local differential privacy, and restricted measurements. We provide a unified framework enabling us to derive a variety of (tight) minimax lower bounds for different parametric families of distributions, both continuous and discrete, under any glyph[lscript] p loss. Our lower bound framework is versatile and yields 'plug-and-play' bounds that are widely applicable to a large range of estimation problems. In particular, our approach recovers bounds obtained using data processing inequalities and Cramér-Rao bounds, two other alternative approaches for proving lower bounds in our setting of interest. Further, for the families considered, we complement our lower bounds with matching upper bounds.

1 Introduction

We consider the problem of parameter estimation under local information constraints , where the estimation algorithm has access to only limited information about each sample. These constraints can be of various types, including communication constraints, where each sample must be described using a few ( e.g. , constant number of) bits; (local) privacy constraints, where each sample is obtained from a different user and the users seek to reveal as little as possible about their specific data; as well as many others, e.g. , noisy communication channels, or limited types of data access such as linear measurements. Such problems have received significant attention in recent years, motivated by applications such as data analytics in distributed systems and federated learning.

Our main focus is on information-theoretic lower bounds for the minimax error rates (or, equivalently, the sample complexity) of these problems. Several recent works have provided bounds that apply to specific constraints or work for specific parametric estimation problems, sometimes without allowing for interactive protocols. Indeed, handling interactive protocols is technically challenging, and several results in prior work exhibit flaws in their analysis. In particular, even the most basic Gaussian mean estimation problem using interactive communication remains, surprisingly, open.

We present general, 'plug-and-play' lower bounds for parametric estimation under information constraints that can be used for any local information constraint and allows for interactive protocols. Our abstract bound requires very simple (and natural) assumptions to hold for the underlying parametric family; in particular, we do not require technical 'regularity' conditions that are common in asymptotic statistics.

We apply our general bound to canonical problems of high-dimensional mean estimation and distribution estimation, under privacy and communication constraints, for the entire family of glyph[lscript] p loss functions for p ≥ 1 . In addition, we provide complementary schemes that show that our lower bounds are tight for most settings of interest.

1.1 Our results

Our main contribution is a general approach to establish lower bounds in distributed informationconstrained parameter estimation. The setup is described in detail in Section 2 and is illustrated in Fig. 1. In short, independent samples X n = ( X 1 , . . . , X n ) are generated from an unknown distribution p from a parametric family P Θ = { p θ , θ ∈ Θ } of distributions. Only limited information Y i about datum X i is available to the algorithm. The goal is to estimate the underlying parameter θ associated with p . Furthermore, we consider interactive estimation, wherein Y i can depend on Y 1 , . . . , Y i -1 . Our general lower bound, which we develop in Section 3, takes the following form: Consider a collection of distributions { p z } z ∈{-1 , +1 } k ⊆ P Θ contained in the parametric family. This collection represents a 'difficult subproblem' that underlies the parametric estimation problem being considered; such constructions are often used when deriving information-theoretic lower bounds ( e.g. , in Assouad's method [33]). Note that each coordinate of z represents, in essence, a different 'direction' of uncertainty for the parameter space. The difficulty of the estimation problem can be related to the difficulty of determining a randomly chosen z (or most of the coordinates of z ), denoted Z , by observing samples from p z . Once Z = z is fixed, n independent samples X n = ( X 1 , . . . , X n ) are generated from p z and the limited information Y i about X i is passed to an estimator.

Our most general result, stated as Theorem 1, is an upper bound for the average discrepancy, an average distance quantity related to average probability of error in determining coordinates of Z by observing the limited information Y n = ( Y 1 , . . . , Y n ) . Our bounding term reflects the underlying information constraints using a quantity that captures how 'aligned' we can make our information about the sample to the uncertainty in different coordinates of Z . Importantly, our results hold under minimal assumptions. In particular, in contrast to many previous works, our results do not require any 'bounded ratio' assumption on the collection { p z } z ∈{-1 , +1 } k , which would ask that the density function change by at most a constant factor if we modify one coordinate of z . When we impose additional structure for p z - such as orthogonality of the random changes in density when we modify different coordinates of z and, more stringently, independence and subgaussianity of these changes - we get concrete bounds which are readily applicable to different problems. These plug-and-play bounds are stated as consequences of our main result in Theorem 2. The interested reader can also directly consider the applications to local privacy (Corollary 1) or communication constraints (Corollary 2).

We demonstrate the versatility of the framework by showing that it readily yields tight (and in some cases nearly tight) bounds for parameter estimation (both in the sparse and dense cases) for several fundamental families of continuous and discrete distributions, several families of information constraints such as communication and local differential privacy (LDP), and for the family of glyph[lscript] p loss functions for p ≥ 1 , all when interactive protocols are allowed. To complement our lower bounds, we provide algorithms (protocols) which attain the stated rates, thus establishing optimality of our results. 1 We discuss these results in Section 5, where we provide the corresponding statements. In terms of the applications, our contributions are two-fold:

We obtain several results from a diverse set of prior works in a unified fashion as simple corollaries of our main result: As discussed further in Section 1.2, the lower bounds for mean estimation for product Bernoulli under glyph[lscript] 2 loss and those for estimation of discrete distributions under glyph[lscript] 1 and glyph[lscript] 2 losses, for both communication and LDP constraints, were known from previous work. However, our approach allows us to easily recover those results and extend them to arbitrary glyph[lscript] p losses, with interaction allowed, in a unified fashion.

Our bounds also yield new lower bounds for some canonical problems. The prototypical example being mean estimation for high-dimensional Gaussian distributions under information constraints. As discussed in the next section, while some prior work claimed lower bounds for this problem, their arguments appear to be flawed - at a high level, due to the

'bounded ratio' assumption their techniques rely on, which Gaussian distributions do not satisfy, and which our framework does not require. To the best of our knowledge our work is the first to obtain those lower bounds for interactive mean estimation of high-dimensional Gaussian distributions under communication or local privacy constraints.

1.2 Previous and related work

There is a significant amount of work in the literature dedicated to parameter estimation under various constraints and settings. Here, we restrict our discussion to works that are most relevant to the current paper, with a focus on the interactive setting (either the sequential or blackboard model; see Section 2 for definitions).

The work arguably closest to ours is the recent work [5], which focuses on density estimation and goodness-of-fit testing of discrete distributions, under the glyph[lscript] 1 metric, for sequentially interactive protocols under general local information constraints (including, as special cases, local privacy and communication constraints, as in the present paper). This work can be seen as a significant generalization of the techniques of [5], allowing us to obtain lower bounds for estimation in a variety of settings, notably high-dimensional parameter estimation.

Among other works on high-dimensional mean estimation under communication constraints, [17, 10] consider communication-constrained Gaussian mean estimation in the blackboard communication model, under glyph[lscript] 2 loss. The protocols for the upper bounds in these works do not require interactivity and are complemented with lower bounds which show that the bounds are tight up to constant factors in the dense case and up to logarithmic factors in the sparse case. However, the proof of the lower bound in [10] seems to present a gap (specifically, in the truncation argument of [10, Theorem 4.3]), as confirmed in personal communication with the authors. Correcting the issue in the truncation argument would lead to a result significantly weaker than the claimed lower bound, and it is unclear whether this can be fixed using the techniques from that paper.

In this work, we present interactive protocols for the sparse case which improve over the noninteractive protocols and strenghten the upper bounds by a logarithmic factor in the interactive case (see Remark 1). Further, using our general framework, we establish a nearly-matching lower bound for the problem, recovering the rate lower bound originally claimed in [10] up to a logarithmic factor. In a slightly different setting, [26] considers the mean estimation problem for product Bernoulli distributions when the mean vector is 1 -sparse, under glyph[lscript] 2 loss. The lower bound in [26], too, allows sequentially interactive protocols. In the blackboard communication model, [19] and [18] obtained tight bounds for mean estimation and density estimation under glyph[lscript] 2 and glyph[lscript] 1 loss, respectively.

Turning to local privacy, [23] provide upper bounds (as well as some partial lower bounds) for onedimensional Gaussian mean estimation under LDP under the glyph[lscript] 2 loss, in the sequentially interactive model. Recent works of [7] and [8] obtain lower bounds for mean estimation in the blackboard communication model and under LDP, respectively, for both Gaussian and product Bernoulli distributions; as well as density estimation for discrete distributions. Their approach is based on the classic Cramér-Rao bound and, as such, is tied inherently to the use of the glyph[lscript] 2 loss. In a recent independent work, [25] extended these methods to obtain lower bounds under general glyph[lscript] p loss under communication constraints, which are tight for Gaussian mean estimation under noninteractive protocols. [14], by developing a locally private counterpart of some of the well-known information-theoretic tools for establishing statistical lower bounds (namely, Le Cam, Fano, and Assouad), establish tight or nearly tight bounds for several mean estimation problems in the LDP setting.

More recently, drawing on machinery from the communication complexity literature, [13] develop a methodology for proving lower bounds under LDP constraints in the blackboard communication model. They obtain lower bounds for mean estimation of product Bernoulli distributions under general glyph[lscript] p losses which match ours (in the high-privacy regime, i.e. , small ε ). Similar to the results under communication constraints [17, 10], their approach relies heavily on the assumption that the distributions on each coordinate are independent, which fails to generalize to discrete distributions. Further, their bounds are tailored to the LDP constraints and do not seem to extend to arbitrary information constraints. Finally, while [13] also claims tight bounds for mean estimation of Gaussian and sparse Gaussian distributions under glyph[lscript] 2 loss, their argument invokes the analogous (flawed) result from [10] and thus it is unclear whether the stated lower bound can be shown using their techniques.

Finally, we mention that very recently, following the appearance of [5], an updated version of [19] appeared online as [21], which has similar results as ours for the high-dimensional mean estimation problem under communication constraints. Both our work and [21] build upon the framework presented for the discrete setting in [5]. Moreover, their work still need the 'bounded ratio', and hence their lower bound for sparse Gaussian family only works for noninteractive protocols.

Notation. Hereafter, we write log and ln for the binary and natural logarithms, respectively. For distributions p 1 , p 2 over X , denote their Kullback-Leibler divergence (in nats) by D( p 1 ‖ p 2 ) , and their

Hellinger distance by d H ( p 1 , p 2 ) := ( 1 2 ∫ ( √ d p 1 d λ -√ d p 2 d λ ) 2 d λ ) 1 / 2 , where we assume p 1 , p 2 glyph[lessmuch] λ for some underlying measure λ on X . Further, we denote the Shannon entropy of a random variable X by H ( X ) and the mutual information between X and Y by I ( X ; Y ) ; we will sometimes write H ( p ) for the entropy of a random variable with distribution p . We refer the reader to [12] for details on these notions and their properties, which will be used throughout. Given two functions f, g , we write f glyph[lessorsimilar] g if there exists an absolute constant C > 0 such that f ( x ) ≤ Cg ( x ) for all x ; and f glyph[equivasymptotic] g if f glyph[lessorsimilar] g and f glyph[greaterorsimilar] g both hold. Finally, we use the standard asymptotic notation O ( f ) , Ω( f ) , Θ( f ) .

Organization. In Section 2, we formalize our setting of interactive inference under local information constraints. The general lower bound framework and results are presented in Section 3, where we provide implications of the general result under additional structures and specific information constraints such as local privacy (LDP) and communication constraints. Finally, we use our framework to readily derive lower bounds for a wide range of applications in Section 5. Due to space constraints, all proofs, as well as our upper bounds (algorithms) are provided in the Supplement, where we also discuss how our techniques compare with other existing approaches for proving lower bounds under information constraints, namely, those based on strong data processing inequalities (SDPI) or on the van Trees inequality.

2 The setup

We consider standard parametric estimation problems. For some Θ ⊆ R d , let P Θ = { p θ , θ ∈ Θ } be a family of distributions over some measurable space ( X , X ) , namely each p θ is a distribution over ( X , X ) . Suppose n independent samples X n = ( X 1 , . . . , X n ) from an unknown p θ ∈ P Θ are obtained. The goal in parametric estimation is to design estimators ˆ θ : X n → Θ , and form estimates ˆ θ ( X n ) of θ using independent samples X n from p θ . We illustrate our results using two specific distribution families: discrete probability mass functions (pmfs) and high-dimensional product distributions with unknown mean vectors. We will describe the precise minimax setting in detail later in this section.

We are interested in an information-constrained setting, where we do not have direct access to the samples X n from p θ . Instead, we can only obtain limited information about each datapoint X i . Following [3], we model these information constraints by specifying an allowed set of channels W with input alphabet X and some output space Y . 2 Each sample X i is passed through a channel from W , chosen appropriately, and its output Y i is the observation we get. This setting is quite general and captures as special cases the popular communication and privacy constraints, as we will describe momentarily.

We now formally describe the setting, which is illustrated in Fig. 1. n i.i.d. samples X 1 , . . . , X n from an unknown distribution p θ ∈ P Θ are observed by players (users) where player t observes X t . Player t ∈ [ n ] selects a channel W t ∈ W and sends the message Y t to a referee, where Y t is drawn from the probability measure W t ( · | X t , Y 1 , . . . , Y t -1 ) . The referee observes Y n := ( Y 1 , . . . , Y n ) and seeks to estimate the parameter θ .

The freedom allowed in the choice of W t at the players gives rise to various communication protocols. We focus on interactive protocols , where channels are chosen by one player at a time, and they can use all previous messages to make this choice. We describe this class of protocols below, where we further allow each player t to have a different set of constraints W t ( e.g. , a different communication budget), and W t must be in W t . For simplicity of exposition, and as these already encapsulate most

X

1

X

2

X

3

. . .

X

n

-

2

X

n

-

1

X

n

W

1

W

2

W

3

. . .

W

n

-

2

W

n

-

1

W

n

Y

1

Y

2

Y

3

. . .

Y

n

-

2

Y

n

-

1

Y

n

p

Referee

output

of the difficulties, we focus here on the case of sequentially interactive protocols, which has been widely considered in the literature and captures many settings of interest. However, we emphasize that our results extend to the more general class of fully interactive protocols (see Supplement).

Definition 1 (Sequentially Interactive Protocols) . Let X 1 , . . . , X n be i.i.d. samples from p θ , θ ∈ Θ . A sequentially interactive protocol Π using W n = ( W 1 , . . . , W n ) involves mutually independent random variables U, U 1 , . . . , U n (independent of the input X 1 , . . . , X n ) and mappings g t : ( U, U t ) ↦→ W t ∈ W t for selecting the channel in round t ∈ [ n ] . In round t , player t uses the channel W t to produce the message (output) Y t according to the probability distribution W t ( · | X t , Y 1 , . . . , Y t -1 ) . The messages Y n = ( Y 1 , . . . , Y n ) received by the referee and the public randomness U (available to all players) constitute the transcript of the protocol Π ; the private randomness U 1 , . . . , U n (where U t is local to player t ) is not part of the transcript. In other words, the channel at player t as a (randomized) mapping W : X × Y t -1 →Y , which depends on input x ∈ X and the previous t -1 messages y t -1 ∈ Y t -1 outputs some y ∈ Y .

For concreteness, we now instantiate this definition for the two aforementioned types of information constraints, communication and (local) privacy.

be the family of channels with input alphabet X and output alphabet the set of all glyph[lscript] -bit strings. This captures the constraint where the message from each player can be at most glyph[lscript] bits long, and corresponds to the choice W n = ( W comm ,glyph[lscript] , . . . , W comm ,glyph[lscript] ) . Note that allowing a different communication budget to each player can be done by setting W n = ( W comm ,glyph[lscript] 1 , . . . , W comm ,glyph[lscript] n ) .

Local differential privacy constraints For ε > 0 and t ≥ 1 , a channel W : X × Y t -1 → Y is ε -locally differentially private (LDP) [16, 15, 24] if

We denote by W priv ,ε the set of all ε -LDP channels. For sequentially interactive protocols, the ε -LDP condition is captured by setting W n = ( W priv ,ε , . . . , W priv ,ε ) . As before, one can allow different privacy parameters for each player by setting W n = ( W priv ,ε 1 , . . . , W priv ,ε n ) .

Finally, we formalize the interactive parametric estimation problem for the family P Θ = { p θ , θ ∈ Θ } . We consider the problem of estimating θ under glyph[lscript] p loss. For p ∈ [1 , ∞ ) , the glyph[lscript] p distance between

u, v ∈ R d is glyph[lscript] p ( u, v ) = ‖ u -v ‖ p = ( ∑ d i =1 | u i -v i | p ) 1 /p . This definition extends in a natural way to p = ∞ by taking the limit. 3

Definition 2 (Sequentially Interactive Estimates) . Fix d ∈ N and p ∈ [0 , ∞ ] . Given a family P Θ of distributions on X , with Θ ⊂ R d , an estimate for P Θ consists of a sequentially interactive protocol Π with transcript ( Y n , U ) and estimator ˆ θ : ( Y n , U ) ↦→ ˆ θ ( Y n , U ) ∈ Θ . The referee observes the transcript ( Y n , U ) and forms the estimate ˆ θ ( Y n , U ) of the unknown θ . Further, for n ∈ N and γ ∈ (0 , 1) , (Π , ˆ θ ) constitutes an ( n, γ ) -estimator for P Θ using W under glyph[lscript] p loss if for every θ ∈ Θ the transcript ( Y n , U ) of Π satisfies

Note that the expectation is over the input X n ∼ p n θ for the protocol Π and the randomness of Π .

3 Main result: The information contraction bound

Our main result is a unified framework to bound the information revealed about the unknown θ by the transcript of the messages obtained via the constraints defined by the channel family W . The framework is versatile and provides tight bounds for several families of continuous and discrete distributions, several families of information constraints such as communication and local differential privacy, and for the family of glyph[lscript] p loss functions for p ≥ 1 .

Our approach at a high-level proceeds as below: We first consider the 'pertubation space' Z := {-1 , +1 } k , for some suitable k . We associate with each z ∈ Z a parameter θ z ∈ Θ , and refer to p θ z simply as p z . These distributions are designed in a way that the distance between θ z and θ z ′ is large when the Hamming distance between z and z ′ is large. With this, the difficulty of estimating θ will be captured in the difficulty of estimating the associated z . This will make our approach compatible with the standard Assouad's method for deriving lower bounds ( cf. [33]).

Then, we let Z = ( Z 1 , . . . , Z k ) be a random variable over Z . Under some assumptions on the distribution of Z , we will bound the information between the individual Z i s and the transcript ( Y n , U ) induced by a family of channels W . Combining the two steps above provides us with the desired lower bounds. Formally, let Z := {-1 , +1 } k for some k and { p z } z ∈Z (where p z = p θ z ) be a collection of distributions over X , indexed by z ∈ Z . For z ∈ Z , denote by z ⊕ i ∈ Z the vector obtained by flipping the sign of the i th coordinate of z . To bound the information that can be obtained about the underlying z from the observations, we make the following assumptions:

Assumption 1 (Densities Exist) . For every z ∈ Z and i ∈ [ k ] it holds that p z ⊕ i glyph[lessmuch] p z , and there exist measurable functions φ z,i : X → R such that d p z ⊕ i d p z = 1 + φ z,i .

The functions φ z,i capture the change in density when the coordinate i is flipped. In our applications below, we will have discrete distributions or continuous densities, and the Radon-Nikodym derivatives above can be replaced with the corresponding ratios between the pmfs and pdfs, respectively.

Assumption 2 (Orthogonality) . There exists some α 2 ≥ 0 such that, for all z ∈ Z and distinct i, j ∈ [ k ] , E p z [ φ z,i φ z,j ] = 0 and E p z [ φ 2 z,i ] ≤ α 2 .

Note that from Assumption 1 we have that E p z [ φ z,i ] = 0 for each i . In conjunction with Assumption 2 this implies that for any fixed z ∈ Z , the family (1 , φ z, 1 , . . . , φ z,k ) is orthogonal and uniformly bounded in L 2 ( X , p z ) . Taken together, Assumption 1 and Assumption 2 roughly say that the densities can be decomposed into uncorrelated 'perturbations' across coordinates of Z . In later sections, we will show that for several families, such as discrete distributions, product Bernoulli distributions, and spherical Gaussians, well-known constructions for lower bounds satisfy these assumptions.

Our first bound given in (3) only requires Assumption 1; by imposing the additional structure of Assumption 2, we obtain the more specialized bound given in (4). Interestingly, (3) can be strengthened further when the following subgaussianity assumption holds.

Assumption 3 (Subgaussianity) . There exists some σ ≥ 0 such that, for all z ∈ Z , the random vector φ z ( X ) := ( φ z,i ( X )) i ∈ [ k ] ∈ R k is σ 2 -subgaussian for X ∼ p z . 4

Let Z = ( Z 1 , . . . , Z k ) be a random variable over Z such that Pr[ Z i = 1] = τ for all i ∈ [ k ] and the Z i s are all independent; we denote this distribution by Rad( τ ) ⊗ k . Our main result is an upper bound on the average amount of information that can be obtained about a coordinate of Z from the transcript ( Y n , U ) of a sequentially interactive protocol, as a function of the information constraint channels and φ Z,i s. This result only requires Assumption 1, and is presented below in its most general form, suited to applications beyond those discussed in the current paper.

Theorem 1 (Information contraction bound: Technical form) . Fix τ ∈ (0 , 1 / 2] . Let Π be a sequentially interactive protocol using W n , and let Z be a random variable on Z with distribution Rad( τ ) ⊗ k . Let ( Y n , U ) be the transcript of Π when the input X 1 , . . . , X n is i.i.d. with common distribution p Z , with density function p Y n Z . Then, under Assumption 1,

We now instantiate this result, invoking Assumptions 2 and 3, to give simple 'plug-and-play' bounds which can be applied readily to several inference problems and information constraints.

Theorem 2. Fix τ ∈ (0 , 1 / 2] . Let Π be a sequentially interactive protocol using W n , and let Z be a random variable on Z with distribution Rad( τ ) ⊗ k . Let ( Y n , U ) be the transcript of Π when the input X 1 , . . . , X n is i.i.d. with common distribution p Z . Then, under Assumptions 1 and 2, we have

Moreover, if Assumption 3 holds as well, we have

where I ( p z ; W ) denotes the mutual information I ( X ; Y ) between the input X ∼ p z and the output Y of the channel W with X as input.

As an illustrative and important corollary, we now derive the implications of this theorem for communication and privacy constraints. For both constraints our tight (or nearly tight) bounds in Section 5 follow directly from these corollaries.

Corollary 1 (Local privacy constraints) . For W = W priv ,ε and any family of distributions { p z , z ∈ {-1 , +1 } k } satisfying Assumptions 1 and 2, with the notation of Theorem 2, we have

Moreover, if Assumption 3 holds as well, we have

Corollary 2 (Communication constraints) . For any family of channels W with finite output space Y and any family of distributions { p z , z ∈ {-1 , +1 } k } satisfying Assumptions 1 and 2, with the notation of Theorem 2, we have

Moreover, if Assumption 3 holds as well, we have

4 An Assouad-type bound

In the previous section we provided an upper bound on 1 k ∑ k i =1 d TV ( p Y n + i , p Y n -i ) . We now prove a lower bound for this quantity in terms of the parameter estimation task we set out to solve. This is an 'Assouad's lemma-type' bound, which when combined with Theorem 2 will establish the bounds for n ; and, reorganizing, the minimax rate lower bounds. To state the result, we require the following assumption, which relates the glyph[lscript] p distance between parameters θ z s to the distance between z s.

Assumption 4 (Additive loss) . Fix p ∈ [1 , ∞ ) . For every z, z ′ ∈ Z ⊂ {1 , +1 } k

glyph[negationslash]

where d Ham ( z, z ′ ) := ∑ k i =1 1 { z i = z ′ i } denotes the Hamming distance.

Lemma 1 (Assouad-type bound) . Let p ≥ 1 and assume that { p z , z ∈ Z} , τ ∈ [0 , 1 / 2] satisfy Assumption 4. Let Z be a random variable on Z = {-1 , +1 } k with distribution Rad( τ ) ⊗ k . Suppose that (Π , ˆ θ ) constitutes an ( n, γ ) -estimator of P Θ using W n under glyph[lscript] p loss (see Definition 2) and Pr Z [ p Z ∈ P Θ ] ≥ 1 -τ/ 4 . Then the transcript ( Y n , U ) of Π satisfies

where p Y n + i := E [ p Y n Z ∣ ∣ Z i = +1 ] , p Y n -i := E [ p Y n Z ∣ ∣ Z i = -1 ] .

5 Applications

Wenowconsider three distribution families: product Bernoulli distributions and Gaussian distributions with identity covariance matrix (and s -sparse mean vectors), and discrete distributions (multinomials), to illustrate the generality and efficacy of our bounds. We describe these three families below, before addressing each of them in their respective subsection. Due to space constraints, we present the proof of the upper bounds in Appendix C, and proof of the lower bounds in Appendix G.

Sparse Product Bernoulli ( B d,s ). Let 1 ≤ s ≤ d , Θ = { θ ∈ [ -1 , 1] d : ‖ θ ‖ 0 ≤ s } , and X = {-1 , 1 } d . Let P Θ := B d,s be the family of d -dimensional s -sparse product Bernoulli distributions over X . Namely, for θ = ( θ 1 , . . . , θ d ) ∈ Θ , the distribution p θ is equal to ⊗ d j =1 Rad( 1 2 ( θ j +1)) : a distribution on {-1 , +1 } d such that the marginal distributions are independent, and for which the mean of the j th marginal is θ j .

Sparse Gaussian ( G d,s ). Let 1 ≤ s ≤ d , Θ = { θ ∈ [ -1 , 1] d : ‖ θ ‖ 0 ≤ s } , and X = R d . Let P Θ := G d,s be the family of d -dimensional spherical Gaussian distributions with bounded s -sparse mean. That is, for θ ∈ Θ , p θ = G ( θ, I ) with mean θ and covariance matrix I . We note that this general formulation assumes ‖ θ ‖ ∞ ≤ 1 (from the choice of Θ ). 5

Discrete distributions ( ∆ d ). Let Θ = { θ ∈ [0 , 1] d : ∑ d i =1 θ i = 1 } ⊆ R d and X = { 1 , 2 , . . . , d } . Let P Θ := ∆ d , where ∆ d is the standard ( d -1) -simplex of all probability mass functions over X . Namely, the distribution p θ is a distribution on X , where, for j ∈ [ d ] , the probability assigned to the element j is p θ ( j ) = θ j . For a unified presentation, we view θ as the mean vector of the 'categorical distribution,' namely the distribution of vector ( 1 { X = x } , x ∈ X ) for X with distribution p θ .

We now define our measure of interest, the minimax error rate of mean estimation.

Definition 3 (Minimax rate of mean estimation) . Let P be a family of distributions parameterized by Θ ⊆ R d . For p ∈ [1 , ∞ ] , n ∈ N , and a family of channels W , the minimax error rate of mean estimation for P using W n under glyph[lscript] p loss, denoted E p ( P , W , n ) , is the least γ ∈ (0 , 1] such that there exists an ( n, γ ) -estimator for P using W under glyph[lscript] p loss (see Definition 2).

We obtain lower bounds on the minimax rate of mean estimation for the different families above by specializing our general bound. Importantly, our methodology is not specific to glyph[lscript] p losses, and can be used for arbitrary additive losses such as (squared) Hellinger or, indeed, for any loss function for which an analogue of Lemma 1 can be derived.

Product Bernoulli family. We first establish the following bounds for B d,s under privacy and communication constraints.

Theorem 3 (Product Bernoulli) . Fix p ∈ [1 , ∞ ) . For 4 log d ≤ s ≤ d , ε ∈ (0 , ∞ ) , and glyph[lscript] ≥ 1 ,

and

For p = ∞ , we have the upper bounds

while the lower bounds given in Eqs. (10) and (11) hold for p = ∞ , too. 6

Remark 1 . Previous work had shown, in the simpler noninteractive model, a rate lower bound scaling as √ ds/ ( nglyph[lscript] ) log(2 d/s ) for the specific case of glyph[lscript] 2 loss (see, for instance, [21, Theorem 7] for the sparse Gaussian case, which implies the Bernoulli one). An analogous phenomenon was observed for local privacy ( e.g. , [14]). Thus, by removing this logarithmic factor from the upper bound, our result establishes the first (to the best of our knowledge) separation between interactive and noninteractive protocols for sparse mean estimation under communication or local privacy constraints.

Remark 2 . Although we stated for simplicity the lower bounds of Theorem 3 in the case where all n players have a similar local constraints ( i.e. , same privacy parameter ε , or same bandwidth constraint glyph[lscript] ), it is immediate to check from the application of Theorem 2 that the result extends to different constraints for each player; replacing n ( ε 2 ∧ ε ) and nglyph[lscript] in the statement by ∑ n t =1 ε 2 t ∧ ε t and ∑ n t =1 glyph[lscript] t , respectively. A similar remark applies to the Gaussian and discrete families.

Gaussian family. We derive a lower bound for E p ( G d,s , W , n ) under local privacy (captured by W = W priv ,ε ) and communication (captured by W = W comm ,glyph[lscript] ) constraints. 7 Recall that for product Bernoulli mean estimation we had optimal bounds for both privacy and communication constraints for all finite p . For Gaussians, we will obtain tight bounds for privacy constraints for ε ∈ (0 , 1] . However, for communication constraints and privacy constraints when ε ≥ 1 , our bounds for Gaussian distributions lose a (single) logarithmic factor in some parameter regimes.

Theorem 4 (Gaussian distributions) . Fix p ∈ [1 , ∞ ) . For 4 log d ≤ s ≤ d , under LDP constraints, when ε ∈ (0 , 1] ,

and when ε > 1 ,

Under communication constraints,

The same bound as in Theorem 3 hold for p = ∞ .

We emphasize that, as discussed in Sections 1.1 and 1.2, to the best of our knowledge our results provides the first lower bounds for interactive Gaussian mean estimation under these constraints.

Discrete distribution estimation. We are able to derive a lower bound for E p (∆ d , W , n ) , the minimax rate for discrete density estimation, under local privacy and communication constraints. In the interest of space, we focus here on two important corollaries; first, for the case of total variation distance ( glyph[lscript] 1 ), where combining it with known upper bounds we obtain optimal bounds for all ε > 0 . In particular, for ε ∈ (0 , 1] (high-privacy regime) we retrieve the lower bound established in [5], which matches the upper bound from [4]. For ε > 1 (low-privacy regime), our bound matches the upper bound for the noninteractive case, established in [32, 1], showing that even in this low-privacy regime interactivity cannot lead to better rates, except maybe up to constant factors.

Corollary 3 (Total variation distance) . For ε > 0 , we have

For the case of glyph[lscript] 2 estimation, we also obtain order-optimal bounds:

Corollary 4 ( glyph[lscript] 2 density estimation) . For ε > 0 , we have

For glyph[lscript] ≥ 1 ,

6 Acknowledgment

The authors would like to thank Yanjun Han, and the anonymous reviewers for helpful comments on an earlier version of this paper.

References

Jayadev Acharya, Ziteng Sun, and Huanyu Zhang. Hadamard response: Estimating distributions privately, efficiently, and with little communication. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of Machine Learning Research , volume 89 of Proceedings of Machine Learning Research , pages 1120-1129. PMLR, 16-18 Apr 2019. URL http: //proceedings.mlr.press/v89/acharya19a.html .

Jayadev Acharya, Clément L. Canonne, and Himanshu Tyagi. Distributed signal detection under communication constraints. volume 125 of Proceedings of Machine Learning Research , pages 41-63. PMLR, July 2020. URL http://proceedings.mlr.press/v125/ acharya20b.html .

Jayadev Acharya, Clément L. Canonne, and Himanshu Tyagi. Inference under information constraints I: Lower bounds from chi-square contraction. IEEE Trans. Inform. Theory , 66 (12):7835-7855, 2020. ISSN 0018-9448. doi: 10.1109/TIT.2020.3028440. URL https: //doi.org/10.1109/TIT.2020.3028440 . Preprint available at arXiv:abs/1812.11476.

For glyph[lscript] ≥ 1 ,

Jayadev Acharya, Clément L. Canonne, Cody Freitag, Ziteng Sun, and Himanshu Tyagi. Inference under information constraints III: local privacy constraints. IEEE J. Sel. Areas Inf. Theory , 2(1):253-267, 2021.

Jayadev Acharya, Clément L. Canonne, Yuhan Liu, Ziteng Sun, and Himanshu Tyagi. Interactive inference under information constraints. IEEE Trans. Inform. Theory , 68(1):502-516, 2022. ISSN 0018-9448. doi: 10.1109/tit.2021.3123905. URL https://doi.org/10.1109/tit. 2021.3123905 .

Ziv Bar-Yossef, T. S. Jayram, Ravi Kumar, and D. Sivakumar. An information statistics approach to data stream and communication complexity. J. Comput. System Sci. , 68(4):702-732, 2004. ISSN 0022-0000. doi: 10.1016/j.jcss.2003.11.006. URL https://doi.org/10.1016/j. jcss.2003.11.006 .

Leighton P. Barnes, Yanjun Han, and Ayfer Özgür. Fisher information for distributed estimation under a blackboard communication protocol. In ISIT , pages 2704-2708. IEEE, 2019.

Leighton Pate Barnes, Wei-Ning Chen, and Ayfer Özgür. Fisher information under local differential privacy. IEEE J. Sel. Areas Inf. Theory , 1(3):645-659, 2020.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . OUP Oxford, 2013.

Mark Braverman, Ankit Garg, Tengyu Ma, Huy L. Nguyen, and David P. Woodruff. Communication lower bounds for statistical estimation problems via a distributed data processing inequality. In Symposium on Theory of Computing Conference, STOC'16 , pages 1011-1020. ACM, 2016.

Clément L. Canonne, Gautam Kamath, Audra McMillan, Jonathan Ullman, and Lydia Zakynthinou. Private identity testing for high-dimensional distributions. In Advances in Neural Information Processing Systems 33 , 2020. To appear. Preprint available at arXiv:abs/1905.11947.

Thomas M. Cover and Joy A. Thomas. Elements of information theory . Wiley-Interscience [John Wiley & Sons], Hoboken, NJ, second edition, 2006. ISBN 978-0-471-24195-9; 0-471-24195-4.

John Duchi and Ryan Rogers. Lower bounds for locally private estimation via communication complexity. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 1161-1191, Phoenix, USA, June 2019. PMLR.

John C. Duchi, Michael I. Jordan, and Martin J. Wainwright. Minimax optimal procedures for locally private estimation. J. Amer. Statist. Assoc. , 113(521):182-201, 2018. ISSN 0162-1459.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography , volume 3876 of Lecture Notes in Comput. Sci. , pages 265-284. Springer, Berlin, 2006.

Alexandre V. Evfimievski, Johannes Gehrke, and Ramakrishnan Srikant. Limiting privacy breaches in privacy preserving data mining. In PODS , pages 211-222. ACM, 2003.

Ankit Garg, Tengyu Ma, and Huy L. Nguyen. On communication cost of distributed statistical estimation and dimensionality. In Advances in Neural Information Processing Systems 27 , pages 2726-2734, 2014.

Yanjun Han, Pritam Mukherjee, Ayfer Özgür, and Tsachy Weissman. Distributed statistical estimation of high-dimensional and non-parametric distributions. In Proceedings of the 2018 IEEE International Symposium on Information Theory (ISIT'18) , pages 506-510, 2018.

Yanjun Han, Ayfer Özgür, and Tsachy Weissman. Geometric lower bounds for distributed parameter estimation under communication constraints. In Proceedings of the 31st Conference on Learning Theory, COLT 2018 , volume 75 of Proceedings of Machine Learning Research , pages 3163-3188. PMLR, 2018.

Yanjun Han, Ayfer Özgür, and Tsachy Weissman. Geometric Lower Bounds for Distributed Parameter Estimation under Communication Constraints. ArXiv e-prints , abs/1802.08417v1, February 2018. First version ( https://arxiv.org/abs/1802.08417v1 ).

Yanjun Han, Ayfer Ozgur, and Tsachy Weissman. Geometric lower bounds for distributed parameter estimation under communication constraints. IEEE Transactions on Information Theory , 67(12), 2021.

T. S. Jayram. Hellinger strikes back: a note on the multi-party information complexity of AND. In Approximation, randomization, and combinatorial optimization , volume 5687 of Lecture Notes in Comput. Sci. , pages 562-573. Springer, Berlin, 2009. doi: 10.1007/978-3-642-03685-9\ _42. URL https://doi.org/10.1007/978-3-642-03685-9_42 .

Matthew Joseph, Janardhan Kulkarni, Jieming Mao, and Steven Z. Wu. Locally private gaussian estimation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 2984-2993. Curran Associates, Inc., 2019.

Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM J. Comput. , 40(3):793-826, 2011. ISSN 0097-5397.

Septimia Sarbu and Abdellatif Zaidi. On learning parametric distributions from quantized samples. In Proceedings of the 2021 IEEE International Symposium on Information Theory (ISIT'21) , June 2021.

Ohad Shamir. Fundamental limits of online and distributed algorithms for statistical learning and estimation. In Advances in Neural Information Processing Systems 27 , pages 163-171, 2014.

Ramon van Handel. Probability in High Dimensions . December 2016. URL https://web. math.princeton.edu/~rvan/APC550.pdf .

Martin J. Wainwright. Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting. IEEE Trans. Inform. Theory , 55(12):5728-5741, 2009. ISSN 0018-9448. doi: 10.1109/TIT.2009.2032816. URL https://doi.org/10.1109/TIT.2009.2032816 .

Stanley L. Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association , 60(309):63-69, 1965. doi: 10.1080/ 01621459.1965.10480775.

Yihong Wu. Lecture notes on: Information-theoretic methods for high-dimensional statistics, 2020. URL http://www.stat.yale.edu/~yw562/teaching/it-stats.pdf .

Aolin Xu and Maxim Raginsky. Information-theoretic lower bounds on Bayes risk in decentralized estimation. IEEE Transactions on Information Theory , 63(3):1580-1600, 2017.

Min Ye and Alexander Barg. Optimal schemes for discrete distribution estimation under locally differential privacy. IEEE Trans. Inform. Theory , 64(8):5662-5676, 2018. ISSN 0018-9448. doi: 10.1109/TIT.2018.2809790. URL https://doi.org/10.1109/TIT.2018.2809790 .

Bin Yu. Assouad, Fano, and Le Cam. In Festschrift for Lucien Le Cam , pages 423-435. Springer, 1997. doi: 10.1007/978-1-4612-1880-7_29. URL http://dx.doi.org/10.1007/ 978-1-4612-1880-7_29 .

Yuchen Zhang, John Duchi, Michael I. Jordan, and Martin J. Wainwright. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. In Advances in Neural Information Processing Systems 26 , pages 2328-2336, 2013.

A Fully interactive model

In this appendix, we describe how to extend our results, presented in the sequentially interactive model, to the more general interactive setting. We first formally define this setting and the corresponding notion of protocols. Hereafter, we use ∗ for the Kleene star operation, i.e. , V ∗ = ⋃ ∞ n =0 V n .

Definition 4 (Interactive Protocols) . Let X 1 , . . . , X n be i.i.d. samples from p θ , θ ∈ Θ , and W ∗ be a collection of sequences of pairs of channel families and players; that is, each element of W ∗ is a sequence ( W t , j t ) t ∈ N where j t ∈ [ n ] . An interactive protocol Π using W ∗ comprises a random variable U (independent of the input X 1 , . . . , X n ) and, for each t ∈ N , mappings

with the constraint that (( W 1 , N 1 ) , . . . , ( W t , N t )) must be consistent with some sequence from W ∗ ; that is, there exists (( W s , j s )) s ∈ N ∈ W ∗ such that W s ∈ W s and N s = j s for all 1 ≤ s ≤ t . These two mappings respectively indicate (i) whether the protocol is to stop (symbol ⊥ ), and, if not, which player is to speak at round t ∈ N , and (ii)) which channel this player selects at this round.

In round t , if N t = ⊥ , the protocol ends. Otherwise, player N t (as selected by the protocol, based on the previous messages) uses the channel W t to produce the message (output) Y t according to the probability measure W t ( · | X N t ) . We further require that T := inf { t ∈ N : N t = ⊥ } is finite a.s. The messages Y T = ( Y 1 , . . . , Y T ) received by the referee and the public randomness U constitute the transcript of the protocol Π .

In other terms, the channel used by the player N t speaking at time t is a Markov kernel

with Y t ⊆ Y ; and, for player j ∈ [ n ] , the allowed subsequences ( W t , j t ) t ∈ N : j t = j capture the possible sequences of channels allowed to the player. As an example, if we were to require that any single player can speak at most once, then for every j ∈ [ n ] and every ( W t , j t ) t ∈ N ∈ W n , we would have ∑ ∞ t =1 1 { j t = j } ≤ 1 .

In the interactive model, we can then capture the constraint that each player must communicate at most glyph[lscript] bits in total by letting W n be the set of sequences ( W comm ,glyph[lscript] t t , j t ) t ∈ N such that

In the simpler sequentially interactive model, this condition simply becomes the choice of W n = ( W comm ,glyph[lscript] , . . . , W comm ,glyph[lscript] ) .

A.1 Lower Bounds under Full Interactive Model

Next we discuss how our technique extends to the full interactive model. For any full interactive protocol Π , let Y ∗ ∈ Y ∗ be the message sequence generated by the protocol. Then, for all y ∗ ∈ Y ∗ , we have

The following lemma states that if X n are generated from a product distribution, the distribution of the transcript satisfies a property similar to the 'cut-and-paste' property from [6].

Lemma 2 ([20]) . If X n ∼ p = ⊗ n t =1 p t , the transcript of the protocol satisfies

where g t ( y ∗ , x t ) = ∏ ∞ j =1 W j ( y j | x t , y j -1 ) 1 { σ j ( y j -1 ) = t } .

Hence, when X n ∼ p ⊗ n z we have

Here we can define a similar notion of 'channel' for a communication protocol Π for the i th player when the underlying distribution is p z by setting glyph[negationslash]

Then we have, for all t ∈ [ n ] ,

We proceed to prove a bound similar to Theorem 1 in terms of the 'channel' defined in Eq. (20), as stated below.

Theorem 5 (Information contraction bound) . Fix τ ∈ (0 , 1 / 2] . Let Π be a fully interactive protocol using W n , and let Z be a random variable on Z with distribution Rad( τ ) ⊗ k . Let ( Y ∗ , U ) be the transcript of Π when the input X 1 , . . . , X n is i.i.d. with common distribution p Z . Then, under Assumption 1,

We can see the bound is in identical form to Theorem 1 except that we replace each player's channel with the ˜ W j, p z ( y ∗ | X ) we defined. Other similar bounds in Section 3 can also be derived under additional assumptions and specific constraints. We present the proof for Theorem 5 below and omit the detailed statements and proof for other bounds.

Proof. Analogously to Eq. (36), we can get

For all z ∈ {-1 , +1 } k and i, t , by the definition of Hellinger distance and Eq. (19), we have glyph[negationslash]

glyph[negationslash]

Proceeding from above, we get under Assumption 1, glyph[negationslash]

glyph[negationslash]

Plugging the above bound into Eq. (21), we can obtain the bound in Theorem 5 by taking the maximum over all z ∈ {-1 , +1 } k and all possible channel sequences.

glyph[negationslash]

B A measure change bound

We here provide a variant of Talagrand's transportation-cost inequality which is used in deriving Eq. (5) (under Assumption 3) in the second part of Theorem 2. We note that this type of result is not novel, and can be derived from standard arguments in the literature (see, e.g., [9, Chapter 8] or [27, Chapter 4]). However, the lemma below is specifically tailored for our purposes, and we provide the proof for completeness. A similar bound was derived in [2], where Gaussian mean testing under communication constraints was considered.

Lemma 3 (A measure change bound) . Consider a random variable X taking values in X and with distribution P . Let Φ: X → R k be such that the random vector Φ( X ) is σ 2 -subgaussian. Then, for any function a : X → [0 , ∞ ) such that E [ a ( X )] < ∞ , we have

Proof. By an application of Gibb's variational principle ( cf. [9, Corollary 4.14]) the following holds: For a random variable Z and distributions P and Q on the underlying probability space satisfying Q glyph[lessmuch] P (that is, such that Q is absolutely continuous with respect to P ), we have

To apply this bound, set P to be the distribution of X and let Q glyph[lessmuch] P be defined using its density (Radon-Nikodym derivative) with respect to P given by

Now, note that for any unit vector v , we have, setting Z = v ᵀ Φ( X ) and using the σ 2 -subgaussianity of Φ( X ) , that

In particular, for λ = 1 σ √ 2D( Q ‖ P ) , we get

Applying this to the unit vector v := E Q [Φ( X )] ‖ E Q [Φ( X )] ‖ 2 then yields

To conclude, it then suffices to observe that

The proof is completed by combining the bounds above, as E Q [Φ( X )] = E P [Φ( X ) a ( X ))] E P [ a ( X )] .

C Upper bounds

We now describe and analyze the interactive algorithms for the estimation tasks we consider.

C.1 Product Bernoulli Distributions

Recall that B d,s , the family of d -dimensional s -sparse product Bernoulli distributions, is defined as

We now provide the interactive protocols achieving the upper bounds of Theorem 3 for sparse product Bernoulli mean estimation under LDP and communication constraints .

Our protocols has two ingredients described below:

Estimating non-zero mean coordinates. In this step we will start with S 0 = [ d ] , the set of all possible coordinates. Then we will iteratively prune the set S 0 → S 1 → . . . → S T , such that | S T | = 3 s (this step is skipped if s ≥ d/ 3 ) is a good estimate for the set of coordinates with non-zero mean.

Estimating the non-zero means. We then estimate the means of the coordinates in S T , which is equivalent to solving a dense mean estimation problem in 3 s dimensions.

In the next two sections, we provide the details of the algorithm that matches the lower bounds obtained in Section 5 for interactive protocols under LDP and communication constraints respectively.

C.1.1 LDP constraints

In this subsection, we will focus on the case ε ∈ (0 , 1] (high-privacy regime). For the case ε > 1 , we rely a privatization of the communication-limited algorithm, which will be discussed at the end of Appendix C.1.2. Our protocol for Bernoulli mean estimation under LDP constraints is described in Algorithm 1. As stated above, in each round t = 1 , . . . , T , for each j ∈ S t -1 a new group of players apply the well known binary Randomized Response (RR) mechanism [29, 24] to their j th coordinate. Using these messages we then guess a set of coordinates with highest possible means (in absolute value) and prune the set to S t . This is done in Lines 2-6 of Algorithm 1.

In Lines 7-12, the algorithm uses the same approach to estimate the means of coordinates within S T and sets remaining coordinates to zero.

The privacy guarantee follows immediately from that of the RR mechanism, and further, this only requires one bit of communication per player.

Algorithm 1 LDP protocol for mean estimation for the product of Bernoulli family

Require: n players, dimension d , sparsity parameter s , privacy parameter ε .

1: Set T := log 3 d 3 s , α := e ε 1+ e ε , S 0 = [ d ] , N 0 := n 6 d .

2: for t = 1 , 2 , . . . , T do

3: for j ∈ S t -1 do

4: Get a group of new players G t,j of size N t = N 0 · 2 t .

5: Player i ∈ G t,j , upon observing X i ∈ {-1 , +1 } d sends the message Y i ∈ {-1 , +1 } such that

6: Set M t,j := ∑ i ∈ G t,j Y i . Let S t ⊆ S t -1 be the set of the | S t -1 | / 3 indices with the largest | M t,j | .

7: for j ∈ S T do

8: Get a group of new players G T,j , j ∈ S T of size N T +1 = N 0 · 2 T .

9: Player i ∈ G T,j , sends the message Y i ∈ {-1 , +1 } according to Eq. (23) and M T,j := ∑ i ∈ G T,j Y i

10: for j ∈ [ d ] do 11:

12: return ̂ µ .

The performance guarantee of Algorithm 1 is stated below, which matches the lower bounds obtained in Section 5.

Proposition 1. Fix p ∈ [1 , ∞ ] . For n ≥ 1 and ε ∈ (0 , 1] , Algorithm 1 is an ( n, γ ) -estimator using W ε under glyph[lscript] p loss for B d,s with γ = O (√ pds 2 /p nε 2 ) for p ≤ 2 log s and γ = O (√ d log s nε 2 ) for p > 2 log s .

Proof. The total number of players used by Algorithm 1 uses is

To prove the utility guarantee, we bound the estimation error in the estimated set S T and the error outside the set S T in the following lemma.

Lemma 4. Let S T be the subset obtained from the first stage of Algorithm 1. Then,

The proposition follows directly from the lemma. Indeed, for p > 2 log s , by monotonicity of glyph[lscript] p norms we have ‖ µ -ˆ µ ‖ p ≤ ‖ µ -ˆ µ ‖ p ′ for all p ′ ≤ p , and thus choosing p ′ := 2 log s is sufficient to obtain the stated bound.

Proof of Lemma 4. We prove the bound on each term individually. The first term captures the performance of our estimator within coordinates in S T and the second term states that we do not 'prune' too many coordinates with high non-zero means.

Bounding the first term. For j / ∈ S T , we output ̂ µ j = 0 . Therefore,

Since µ is s -sparse, it will suffice to show that for all j with | µ j | > 0 ,

Let

Note that for ε ∈ (0 , 1] , we have 2 α -1 ≥ e -1 e +1 ε . Therefore, if | µ j | ≤ H , then Eq. (24) holds since Pr[ j / ∈ S ] ≤ 1 . We hereafter assume | µ j | > H , and let µ j = β j H with β j > 1 . Let E t,j be the event that coordinate j is removed in round t given that j ∈ S t -1 . Then we have

Weproceed to bound each Pr[ E t,j ] separately. Note that for i ∈ G t,j , Y i ∈ {-1 , +1 } and by Eq. (23)

Let a t,j be the number of coordinates j ′ with µ j ′ = 0 and | M t,j ′ | ≥ 1 2 N t (2 α -1) β j H . Since we select the | S t -1 | / 3 coordinates with the largest magnitude of the sum, for j / ∈ S t to happen at least one of the following must occur: (i) a t,j > 1 3 | S t -1 | -s , or (ii) M t,j < 1 2 N t (2 α -1) β j H .

By Hoeffding's inequality, we have

Let p t,j := e -5 · 2 t β 2 j . Similarly, for any j ′ such that µ j ′ = 0 ,

Since all coordinates are independent, a t,j is binomially distributed with mean at most 2 p t,j | S t -1 | . By Markov's inequality, we get

recalling that | S t -1 | = d 3 t -1 ≥ 9 s . By a union bound and summing over t ∈ [ T ] , we get

Not that for x > 0 , x p e -x 2 ≤ ( p 2 e ) p/ 2 . Hence

for some absolute constant C > 0 , completing the proof.

Bounding the second term. Note that S T is a random variable itself. We show that the bound holds for any realization of S T . We need the following result which follows from standard moment bounds on binomial distributions.

Fact 1. Let p ≥ 1 , m ∈ N , 0 ≤ q ≤ 1 , and N ∼ Bin( m,q ) . Then, E [ | N -mq | p ] ≤ 2 -p/ 2 m p/ 2 p p/ 2 .

Applying this with m = N T ≥ n 6 d , the transformation from Bernoulli to {-1 , +1 } , and the scaling by 2 α -1 , yields for j ∈ S T , and using Eq. (25)

Upon summing over j ∈ S T , we obtain

C.1.2 Communication constraints

In Algorithm 2 we propose a protocol to estimate the mean of product Bernoulli distributions under glyph[lscript] -bit communication constraints. As mentioned in the previous subsection, the ε -LDP algorithm with ε > 1 will follow from a simple modification of the communication-constrained one; we discuss how to privatize the latter to obtain the former at the end of the section. As in the LDP case when ε ∈ (0 , 1] , in 2-10 the algorithm iteratively prunes an initial set S 0 = [ d ] to obtain a set S T of size max { 3 s, glyph[lscript] } , which denotes the set of potential non-zero coordinates. We then estimate the mean of coordinates in S T . If glyph[lscript] > 3 s , then we can directly send the values of all coordinates in S T and use it for estimation; otherwise, when 3 s > glyph[lscript] , we again partition S T into sets of size glyph[lscript] and each player sends the bits of its sample in this set. This is done in Lines 11-18. We state the performance of Algorithm 2 below.

Proposition 2. Fix p ∈ [1 , ∞ ] . For n ≥ 1 and glyph[lscript] ≤ d , we have Algorithm 2 is an ( n, γ ) -estimator using W glyph[lscript] under glyph[lscript] p loss for B d,s with γ = O (√ pds 2 /p nglyph[lscript] + ( p +log(2 glyph[lscript]/s )) s 2 /p n ) for p ≤ 2 log s and

When glyph[lscript] ≤ 3 s , the bound we get is γ glyph[lessorsimilar] √ pds 2 /p nglyph[lscript] . The analysis is almost identical to the case under LDP constraints, since in both cases, the information we get about coordinate j are samples from a Rademacher distribution with mean (2 α -1) µ j . There are only two differences. (i) α = 1 instead of

Require: n players, dimension d , sparsity parameter s , communication bound glyph[lscript] .

1: Set T := log 3 ( d/ max { 3 s, glyph[lscript] } ) , S 0 := [ d ] , N 0 := nglyph[lscript] 18 d .

2: for t = 1 , 2 , . . . , T do

3: Set P := d 3 t -1 glyph[lscript] , and partition S t -1 into P subsets S t -1 , 1 , . . . , S t -1 ,P , each of size glyph[lscript] .

4: for j = 1 , 2 , . . . , P do

5: Get a group of new players G t,j of size N t = N 0 · 2 t .

6:

7:

Player

}

t

x

S

∈

-

For

x

1

i

,j

∈

8:

t

Set

S

⊆

.

S

S

t

9: if glyph[lscript] ≤ 3 s then

10: Partition S T into 3 s/glyph[lscript] subsets of size glyph[lscript] each, S T,j , j ∈ [3 s/glyph[lscript] ] .

11: for j = 1 , . . . , 3 s/glyph[lscript] do

12: Get a new group G T +1 ,j of players of size nglyph[lscript]/ (6 s ) .

13: Player i ∈ G T +1 ,j , sends the message Y i = { ( X i ) x } x ∈ S T,j .

14: For x ∈ S T,j , let M T +1 ,x = ∑ i ∈ G T +1 ,j ( X i ) x . Set

t

-

1

t,j

G

,j

, let

,

upon

M

t,x observing

X

i

∈

{-

∈

-

∑

t,j

X

i

)

x

.

i

G

∈

:=

(

1

to be the set of indices with the largest

|

1

,

+1

M

t,x

{

(

X

i

)

x

15: For x / ∈ S T , set ̂ µ x = 0 .

16: if glyph[lscript] > 3 s then ,

17: Get n/ 2 new players G T +1 and for i ∈ G T +1 , player i sends Y i = { ( X i ) x } x ∈ S T glyph[triangleright] This can be done since | S T | = glyph[lscript] if glyph[lscript] > 3 s .

18: For x ∈ S T , let M T +1 ,x = ∑ i ∈ G T +1 ,j ( X i ) x . Set S T +1 ⊆ S T to be the set of indices with the largest | M T +1 ,x | and | S T +1 | = 3 s . For all x ∈ S T +1 , set

and for all x / ∈ S T +1 , ̂ µ x = 0 .

19: return ̂ µ .

Θ ( ε 2 ) . (ii) There is a factor of glyph[lscript] more players in the corresponding groups. Combing both factors, we can obtain the desired bound by replacing ε 2 by glyph[lscript] . We omit the detailed proof in this case.

When glyph[lscript] > 3 s , after T glyph[equivasymptotic] log( d/glyph[lscript] ) rounds, we can find a subset S T of size glyph[lscript] which contains most of the coordinates with large biases. The protocol then asks new players to send all coordinates within S T using glyph[lscript] bits. In this case, it would be enough to prove Lemma 5 since for the coordinates outside S T , we can show the error is small following exactly the same steps as the proof for bouding the first term in Lemma 4 as we explained in the case when glyph[lscript] ≤ 3 s .

Lemma 5. Let S T be the subset obtained from the first stage of Algorithm 2, we have

Proof. Similar to Lemma 4, we will prove that the statement is true for any realization of S T , which is a stronger statement than the claim.

}

|

d

sends and

|

S

t

|

the

=

|

S

message

t

-

1

|

/

3

.

Y

i

=

Fix S T +1 . For each j ∈ S T +1 , M T +1 ,j is binomially distributed with mean µ j and n/ 2 trials. By similar computations as Lemma 4, we have

glyph[negationslash]

Next we show for all j ∈ S T such that µ j = 0 ,

If | µ j | ≤ H ′ := 8 √ ln 2 glyph[lscript] s n , Eq. (27) always holds since Pr[ j / ∈ S ] ≤ 1 . Hence we hereafter assume that | µ j | > H ′ , and write µ j = β j H ′ for some β j > 1 .

Let a T +1 ,j be the number of coordinates j ′ with µ j ′ = 0 and | M T +1 ,j ′ | ≥ n 2 · β j H ′ 2 . Then since S T +1 contains the top 3 s coordinates with the largest magnitude of the sum, we have j / ∈ S T +1 happens only if at least one of the following occurs (i) a T +1 ,j > 2 s , or (ii) M T +1 ,j < n 2 · β j H ′ 2 . By Hoeffding's inequality, we have

Similarly, for any j ′ such that µ j ′ = 0 ,

Since all coordinates are independent, a T +1 ,j is binomially distributed with mean at most 2 p T +1 ,j glyph[lscript] , and therefore, by Markov's inequality,

the last step since β j > 1 . By a union bound, we have

Using the inequality x p a -x 2 ≤ ( p 2 e ln a ) p/ 2 which holds for all x > 0 , we get overall

establishing Eq. (27). Combining Eq. (26) and Eq. (27) concludes the proof Lemma 5 since there are at most s unbiased coordinates.

Algorithm under LDP with ε > 1 To get a ε -LDP algorithm in the regime ε > 1 (low-privacy regime), we perform the following changes to obtain a private algorithm from Algorithm 2:

Each user independently flips each coordinate of their local sample to get Z i where, for all x ∈ [ d ] , ( Z i ) x = ( X i ) x with probability e e +1 and ( Z i ) x = 1 -( X i ) x with probability 1 e +1 (note that this corresponds to applying Randomized Response independently to each bit with privacy parameter 1 ).

Users then follow Algorithm 2 with the setting glyph[lscript] = glyph[floorleft] ε glyph[floorright] and local data { Z i } i ∈ [ n ] , and obtain estimate ̂ µ .

The final estimate is then e +1 e -1 ̂ µ .

The privacy guarantee of the algorithm comes from the fact that Algorithm 2 sends at most glyph[lscript] = glyph[floorleft] ε glyph[floorright] coordinates of each Z i , and for any S with | S | ≤ glyph[floorleft] ε glyph[floorright]

The utility guarantee follows from observing that µ Z = e -1 e +1 µ and hence any glyph[lscript] p error guarantee will be preserved up to a constant.

C.2 Gaussian Mean Estimation

Recall that G d,s denotes the family of d -dimensional spherical Gaussian distributions with s -sparse mean in [ -1 , 1] d , i.e. ,

We will prove the following results for LDP and communication constraints, respectively.

Proposition 3. Fix p ∈ [1 , ∞ ] . For n ≥ 1 and ε ∈ (0 , 1] , there exists an ( n, γ ) -estimator using W ε under glyph[lscript] p loss for G d,s with γ = O (√ pds 2 /p nε 2 ) for p ≤ 2 log s and γ = O (√ d log s nε 2 ) for p > 2 log s .

Proposition 4. Fix p ∈ [1 , ∞ ] . For n ≥ 1 and glyph[lscript] ≤ d , there exists an ( n, γ ) -estimator using W glyph[lscript] under glyph[lscript] p loss for G d,s with γ = O (√ pds 2 /p nglyph[lscript] + ( p +log(2 glyph[lscript]/s )) s 2 /p n ) for p ≤ 2 log s and γ = O (√ d log s nglyph[lscript] + log glyph[lscript] n ) for p > 2 log s .

We reduce the problem of Gaussian mean estimation to that of Bernoulli mean estimation and then invoke Propositions 1 and 2 from the previous section. At the heart of the reduction is a simple idea that was used in, e.g. , [10, 2, 11]: the sign of a Gaussian random variable already preserves sufficient information about the mean. Details follow.

Let p ∈ G d,s with mean µ ( p ) = ( µ ( p ) 1 , . . . , µ ( p ) d ) . For X ∼ p , let Y = (sign( X i )) i ∈ [ d ] ∈ {-1 , +1 } d be a random variable indicating the signs of the d coordinates of X . By the independence of the coordinates of X , note that Y is distributed as a product Bernoulli distribution (in B d ) with mean vector ν ( p ) given by

and, since | µ ( p ) i | ≤ 1 , we have ν ( p ) ∈ [ -η, η ] d , where η := Erf ( 1 / 2 ) ≈ 0 . 623 . Moreover, it is immediate to see that each player, given a sample from p , can convert it to a sample from the corresponding product Bernoulli distribution. We now show that a good estimate for ν ( p ) yields a good estimate for µ ( p ) .

Lemma 6. Fix any p ∈ [1 , ∞ ) , and p ∈ G d . For ̂ ν ∈ [ -η, η ] d , define ̂ µ ∈ [ -1 , 1] d by ̂ µ i := √ 2 Erf -1 ( ̂ ν i ) , for all i ∈ [ d ] . Then

Proof. By computing the maximum of its derivative, 8 we observe that the function Erf -1 is √ eπ 2 -Lipschitz on [ -η, η ] . By the definition of ̂ µ and recalling Eq. (29), we then have

where we used the fact that ν, ̂ ν ∈ [ -η, η ] .

As previously discussed, combining Lemma 6 with Propositions 1 and 2 (with γ ′ := √ 2 eπ γ ) immediately implies Propositions 3 and 4 for p ∈ [1 , ∞ ] .

Remark 3 . Note that for the Gaussian family, we also consider the linear measurement constraint. Under linear measurement constraints, we can use the linear measurement matrix to obtain r out of d coordinates and perform the above reduction to product of Bernoulli family. The obtained bound will be same as that under communication constraints.

D Relation to other lower bound methods

We now discuss how our techniques compare with other existing approaches for proving lower bounds under information constraints. Specifically, we clarify the relationship between our technique and the approach using strong data processing inequalities (SDPI) as well as that based on van Trees inequality (a generalization of the Cramér-Rao bound).

D.1 Strong data processing inequalities

We note first that the bound in Eq. (5) can be interpreted as a strong data processing inequality. Indeed, the average discrepancy on the left-side of inequality can be viewed as the average information Y n reveals about each bit of Z . Here the information is measured in terms of total variation distance. The information quantity on the right-side denotes the information between the input X n and the output Y n of the channels. Since the Markov relation Z n -X n -Y n holds, the inequality is thus a strong data processing inequality with strong data processing constant roughly σ 2 /k . Such strong data processing inequalities were used to derive lower bounds for statistical estimation under communication constraints in [34, 10, 31]. We note that our approach recovers these bounds, and further applies to arbitrary constraints captured by W .

D.2 Connection to the van Trees inequality

The average information bound in (3), in fact, allows us to recover bounds similar to the van Trees inequality-based bounds developed in [7] and [8].

For Θ ⊂ R k and a parametric family 9 P Θ = { p θ , θ ∈ Θ } , recall that the Fisher information matrix J ( θ ) is a k × k matrix given by, under some mild regularity conditions,

In particular, the diagonal entries equal

For our application, given a channel W ∈ W , we consider the family P W Θ := { p θ W , θ ∈ Θ } of distributions induced on the output of the channel W when the input distributions are from P Θ . We denote the Fisher information matrix for this family by J W ( θ ) , which we compute next under a refined version of our Assumption 1 described below.

Let θ be a point in the interior of Θ and p θ be differentiable at θ . Weset θ z := θ + γ 2 z , z ∈ {-1 , +1 } k , and make the following assumption about the structure of the parametric family of distribution: For all z ∈ {-1 , +1 } k and i ∈ [ k ] ,

where E p z [ ξ γ z,i ( X ) 2 ] and E p z [ ψ γ z,i ( X ) 2 ] are assumed to be uniformly bounded for γ sufficiently small; for concreteness, we assume E p z [ ψ γ z,i ( X ) 2 ] ≤ c 2 for a constant c , for all γ sufficiently small. Let ξ z,i ( x ) := lim γ → 0 ξ γ z,i ( x ) , for all x .

In applications, we expect the dependence of ξ γ z,i on γ to be 'mild,' and, in essence, the assumption above provides a linear expansion of the term α z,i φ z,i from Assumption 1 as a function of the perturbation parameter γ . Assuming that the densities are differentiable as a function of θ , for the distribution p θ W of the output of a channel W with input X ∼ p θ , we get

where we used Eq. (30), the fact that lim γ → 0 θ z = θ , the fact that E p z [ ψ γ z,i ( X ) W ( y | X ) ] ≤ c √ E p z [ W ( y | X ) 2 ] ≤ c , and the dominated convergence theorem. Thus, we get

Our information contraction bound will be seen later (Section 5) to yield lower bounds for expected estimation error. For concreteness, we give a preview of a version here. We assume for simplicity that W t = W for all t and consider the glyph[lscript] 2 loss function for the dense ( τ = 1 / 2 ) case. By following the proof of Lemma 1 below, given an ( n, γ ) -estimator ˆ θ = ˆ θ ( Y n , U ) of P Θ using W n under glyph[lscript] 2 loss, we can find an estimator ˆ Z = ˆ Z ( Y n , U ) such that glyph[negationslash]

glyph[negationslash]

Upon setting γ := 4 γ/ k , we get that the left-side of Eq. (3) is bounded below by 1 / 4 . For the same γ and under Eq. (30), the right-side evaluates to whereby

where we used ( a + b ) 2 ≤ 2( a 2 + b 2 ) and

Therefore, Eq. (3) yields

This bound is, in effect, the same as the van Trees inequality with Tr ( J W ( θ ) ) replaced by

In fact, in view of Eq. (31), Tr ( J W ( θ ) ) = lim γ → 0 g ( γ ) =: g (0) . Thus, our general lower bound will recover van Trees inequality-based bounds when Eq. (30) holds and g ( γ ) ≈ g (0) . We note that Eq. (30) holds for all the families considered in this paper (see Eq. (40) for product Bernoulli, Eq. (42) for Gaussian, and Eq. (50) for discrete distributions). We close this discussion by noting that results in Section 3 are obtained by deriving bounds for g ( γ ) which apply for all γ and, therefore, also for g (0) = Tr ( J W ( θ ) ) .

E Missing proofs in Section 3

E.1 Proof of Theorem 1

Consider Z = ( Z 1 , . . . , Z k ) ∈ {-1 , 1 } k where Z 1 , . . . , Z k are i.i.d. with Pr[ Z i = 1] = τ . For a fixed i ∈ [ k ] , let glyph[negationslash]

glyph[negationslash]

the partial mixtures of message distributions conditioned on Z i . We will rely on the following lemma, which relates the desired average discrepancy between the p Y n + i and p Y n -i 's to the sum of n 'local' discrepancy measures (in the form of Hellinger distances between local messages). Each local measure can then be easily bounded in terms of the density p z and the channel W to get the desired bound.

Lemma 7. With the notation of Theorem 1, we have

where p z W denotes the distribution of Y ∼ W ( · | X ) when X ∼ p z .

The proof of the lemma is rather involved and constitutes the core of the argument. We defer it to the end of the section and show first how it implies Theorem 1. For all z and W , we have

Moreover, under Assumption 1; for any W ∈ W t and y ∈ Y ,

Plugging this back into (33), we get

Combining this with Lemma 7 concludes the proof of Theorem 1.

Proof of Lemma 7. Our first step is to use the Cauchy-Schwarz inequality, followed by an inequality relating total variation and Hellinger distances:

where the last inequality uses joint convexity of squared Hellinger distance, and the final identity is due to independence of each coordinate of Z and symmetry of Hellinger whereby E Z [ d H ( p Y n Z , p Y n Z ⊕ i ) 2 | Z i = +1 ] = E Z [ d H ( p Y n Z , p Y n Z ⊕ i ) 2 | Z i = -1 ] .

In order to bound the resulting terms of the sum, we will rely on the so-called cut-paste property of Hellinger distance [6]. Before doing so, we will require an additional piece of notation: for fixed z ∈ Z , i ∈ [ k ] , t ∈ [ n ] , let p Y n t ← z ⊕ i denote the message distribution where player t gets a sample from p z ⊕ i and all other players get samples from p z . That is, for all y n ∈ Y n , the density of p Y n t ← z ⊕ i with respect to the underlying product measure µ ⊗ n is given by glyph[negationslash]

The following lemma, due to [22], allows us to relate d H ( p Y n z , p Y n z ⊕ i ) , the distance between message distributions when all players get observations from p z , or all from p z ⊕ i , to the distances d H ( p Y n z , p Y n t ← z ⊕ i ) where only one of the n players gets a sample from p z ⊕ i .

Lemma 8 ([22, Theorem 7]) . There exists c H > 0 such that for all z ∈ Z and i ∈ [ k ] ,

Moreover, one can take c H = 2 ∏ ∞ t =1 1 1 -2 -t < 7 .

Combining Eq. (34) and Lemma 8, we get

In view of bounding the RHS of (36) term by term, fix j ∈ [ n ] and z ∈ Z . Recalling the expression of p Y n t ← z ⊕ i from (35), unrolling the definition of Hellinger distance, and recalling (35), we have glyph[negationslash]

where the second-to-last identity uses the observation that, for any fixed y t ∈ Y t ,

which in turn follows upon taking marginal integrals for each coordinate. We then get from the pointwise inequality ∑ k i =1 ∫ Y t -1 f i,t ( y t -1 , y t ) d µ ≤ sup y ′ ∈Y t -1 ∑ k i =1 ∫ Y f i,t ( y ′ , y t ) d µ that

the second identity follows upon taking marginal integrals, and by replacing f i,t by its definition; and the second inequality using that { W y ′ : y ′ ∈ Y t -1 } ⊆ W t , so that we are taking a supremum over a larger set.

Plugging this back into (36) and upper bounding the inner expectation by a maximum concludes the proof of the lemma.

E.2 Proof of Theorem 2

Our starting point is Eq. (3) which holds under Assumption 1. We will bound the right-handside of Eq. (3) under assumptions of orthogonality and subgaussianity to prove the two bounds in Theorem 2.

First, under orthogonality (Assumption 2), we apply Bessel's inequality to Eq. (3). For a fixed z ∈ Z , write ψ z,i = φ z,i √ E p z [ φ 2 z,i ] , and complete (1 , ψ z, 1 , . . . , ψ z,k ) to get an orthonormal basis B for

L 2 ( X , p z ) . Fix any W ∈ W and y ∈ Y , and, for brevity, define a : X → R as a ( x ) = W ( y | x ) . Then, we have

where for the second identity we used the assumption that 〈 E [ a ] , ψ z,i 〉 = 0 for all i ∈ [ k ] (since 1 and ψ z,i are orthogonal). This establishes Eq. (4).

Turning to Eq. (5), suppose that Assumption 3 holds. Fix z ∈ Z , and consider any W ∈ W and y ∈ Y . Upon applying Lemma 4 of the Supplement (See Supplement (Appendix B) for the precise statement and proof) to the σ 2 -subgaussian random vector φ z ( X ) and with a ( x ) set to W ( y | x ) ∈ [0 , 1] , we get that

Integrating over y ∈ Y , this gives

which yields the claimed bound.

E.3 Proof of Corollary 1

For any W ∈ W priv ,ε , the ε -LDP condition from Eq. (2) can be seen to imply that, for every y ∈ Y ,

By taking expectation over x 3 then again either over x 1 or x 2 (all distributed according to p z ), this yields

Squaring and taking the expectation on both sides, we obtain

Dividing by E p z [ W ( y | X )] , summing over y ∈ Y , and using ∫ Y E p z [ W ( y | X )] d µ = 1 gives

thus establishing (6). For the bound of e ε , observe that, for all y ∈ Y ,

Hence

The bound (7) (under Assumption 3) will follow from (5), and the relation between differential privacy and KL divergence. Indeed, the mutual information I ( p z ; W ) can be rewritten as the expected (over X ∼ p Z ) KL divergence between the distribution p W,X := W ( · | X ) over Y induced by the channel W on input X , and the distribution p W Z := E X ′ ∼ p z [ W ( · | X ′ )] over Y induced by the input distribution p z and the channel W :

but the ε -LDP condition from Eq. (2) guarantees that the log-likelihood ratio in the inner expectation is (almost surely) at most ε , so that I ( p z ; W ) ≤ ε for every z and W ∈ W priv ,ε . This yields (7).

E.4 Proof of Corollary 2

In view of (4), to establish (8), it suffices to show that Var p z [ W ( y | X )] E p z [ W ( y | X )] ≤ 1 for every y ∈ Y . Since W ( y | x ) ∈ (0 , 1] for all x ∈ X and y ∈ Y , so that

The second bound (under Assumption 3) will follow from (5). Indeed, recalling that the entropy of the output of a channel is bounded below by the mutual information between input and the output, we have I ( p z ; W ) ≤ H ( p z W ) , where p z W := E p z [ W ( · | X )] is the distribution over Y induced by the input distribution p z and the channel W . Using the fact that the entropy of a distribution over Y is at most log |Y| in (5) gives (9).

F Missing proofs in Section 4

F.1 Proof of Lemma 1

Given an ( n, γ ) -estimator (Π , ˆ θ ) , define an estimate ˆ Z for Z as

By the triangle inequality,

Since (Π , ˆ θ ) is an ( n, γ ) -estimator under glyph[lscript] p loss for P Θ ,

glyph[negationslash]

glyph[negationslash]

where Eq. (38) follows from Assumption 4 and Pr[ p Z ∈ P Θ ] ≥ 1 -τ/ 4 . Next, for p ∈ [1 , ∞ ) , by Assumption 4, ∥ ∥ θ Z -θ ˆ Z ∥ ∥ p p ≥ 4 p γ p τk ∑ k i =1 1 { Z i = ˆ Z i } . Combining with Eq. (39) this shows that 1 τk ∑ k i =1 Pr [ Z i = ˆ Z i ] ≤ 3 4 .

glyph[negationslash]

glyph[negationslash]

Furthermore, since the Markov relation Z i -( Y n , U ) -ˆ Z i holds for all i , we can lower bound Pr [ Z i = ˆ Z i ] using the standard relation between total variation distance and hypothesis testing as

follows, using that τ ≤ 1 / 2 in the second inequality:

glyph[negationslash]

Summing over 1 ≤ i ≤ k and combining it with the previous bound, we obtain glyph[negationslash]

and reorganizing proves the result.

G Missing statements and proofs in Section 5

G.1 Proof of Theorem 3

Fix p ∈ [1 , ∞ ) . Let k = d , Z = {-1 , +1 } d , and τ = s 2 d ; and suppose that, for some γ ∈ (0 , 1 / 8] , there exists an ( n, γ ) -estimator for B d,s under glyph[lscript] p loss. We fix a parameter γ ∈ (0 , 1 / 2] , which will be chosen as a function of γ, d, p later. Consider the set of 2 d product Bernoulli distributions { p z } z ∈Z , where µ ( p z ) = µ z := 1 2 γ ( z + 1 d ) (so the sparsity of the mean vector is equal to the number of positive coordinates of z ). We have, for z ∈ Z ,

It follows for z ∈ Z and i ∈ [ d ] that

glyph[negationslash]

where φ z,i ( x ) := -γz i x i 1+ 1 2 γ (1+ z i ) x i . We can verify that, for i = j ,

so that Assumptions 1 and 2 are satisfied for α 2 := 2 γ 2 . Moreover, using, e.g. , Hoeffding's lemma ( cf. [9]), for γ < 1 , the random vector φ z ( X ) = ( φ z,i ( X )) i ∈ [ d ] is γ 2 (1 -γ 2 ) 2 -subgaussian. Thus, Assumption 3 holds as well, and we can invoke both parts of Theorem 2.

Let ‖ z ‖ + := |{ i ∈ [ d ] | z i = 1 }| , so that ‖ µ z ‖ 0 = ∑ d i =1 1 2 (1 + z i ) = ‖ z ‖ + . The next claim, which follows from standard bounds for binomial random variables, states that when Z ∼ Rad( τ ) ⊗ d , µ Z is s -sparse with high probability.

Hence the construction satisfies Pr Z [ p Z ∈ B d,s ] ≤ 1 -τ/ 4 , as required in Lemma 1.

We now choose γ = γ ( p ) := 4 γ ( s/ 2) 1 /p ∈ (0 , 1 / 2] , which implies that Assumption 4 holds since

Therefore, we can apply Lemma 1 as well. For W priv ,ε , we prove the two parts of the lower bound separately, depending on whether ε ≤ 1 . First, upon combining the bounds obtained by Corollary 1 and Lemma 1 (specifically, for the former, (6)), we get

whereby, upon recalling that α 2 = 2 γ 2 , and using the value of γ = γ ( p ) above, it follows that

Thus, E p ( B d,s , W priv ,ε , n ) = Ω (√ ds 2 /p nε 2 ) for ε ∈ (0 , 1] . For the second part of the bound, which dominates for ε > 1 , observe that Assumption 3 holds with σ 2 := γ 2 (1 -γ 2 ) 2 ≤ 2 γ 2 ; allowing us to apply the second part of Corollary 1, (7), which as before combined with Lemma 1 yields

and again from the setting of γ we get E p ( B d,s , W priv ,ε , n ) = Ω (√ ds 2 /p nε ) .

Similarly, for W comm ,glyph[lscript] , again since Assumption 3 holds with σ 2 ≤ 2 γ 2 , upon combining the bounds obtained by Corollary 2 and Lemma 1, we get

which gives E p ( B d,s , W comm ,glyph[lscript] , n ) = Ω ( √ ds 2 /p nglyph[lscript] ∧ 1 ) . Finally, note that for glyph[lscript] ≥ d , the lower bound follows from the minimax rate in the unconstrained setting, which can be seen to be Ω (√ s 2 /p log(2 d/s ) /n ) [28, 30]. This completes the proof.

This handles the case p ∈ [1 , ∞ ) . For p = ∞ , the lower bounds immediately follow from plugging p = log s in the previous expressions, as discussed in Footnote 3.

G.2 Proof of Theorem 4

We denote the mean by µ instead of θ , denote the estimator by ˆ µ , and consider the minimax error rate E p ( G d,s , W , n ) of mean estimation for P Θ = G d,s using W under glyph[lscript] p loss.

Proof of Theorem 4. Let ϕ denote the probability density function of the standard Gaussian distribution G ( 0 , I ) . Fix p ∈ [1 , ∞ ) . Let k = d , Z = {-1 , +1 } d , and τ = s 2 d ; and suppose that, for some γ ∈ (0 , 1 / 8] , there exists an ( n, γ ) -estimator for G d,s under glyph[lscript] p loss. We fix a parameter γ := γ ( p ) := 4 γ ( s/ 2) 1 /p ∈ (0 , 1 / 2] , and consider the set of distributions { p z } z ∈Z of all 2 d spherical Gaussian distributions with mean µ z := γ ( z + 1 d ) , where z ∈ Z . Again, note that ‖ µ z ‖ 0 = ∑ d i =1 1 { z i = 1 } = ‖ z ‖ + , and Fact 2 applies here too. Then by the definition of Gaussian density, for z ∈ Z ,

Therefore, for z ∈ Z and i ∈ [ d ] , we have

glyph[negationslash]

where φ z,i ( x ) := 1 -e -2 γx i z i e 2 γ 2 z i . By using the Gaussian moment-generating function, for i = j ,

so that Assumptions 1 and 2 are satisfied for α 2 := e 4 γ 2 -1 . By our choice of γ and the assumption on γ , one can check that Assumption 4 holds:

Moreover, similar to the product of Bernoulli case, using Fact 2, we can show that Pr Z [ p Z ∈ G d,s ] ≤ 1 -τ/ 4 . This allows us to apply Lemma 1.

G.2.1 Privacy constraints for ε ∈ (0 , 1)

For W priv ,ε , upon combining the bounds obtained by Corollary 1 and Lemma 1, we get

whereby, upon noting that α 2 = e 4 γ 2 -1 ≤ 8 γ 2 holds since γ ≤ 1 / 2 , and using the value of γ = γ ( p ) above, it follows that

Thus, E p ( G d,s , W priv ,ε , n ) = Ω (√ ds 2 /p nε 2 ∧ 1 ) . This establishes the lower bounds for W priv ,ε . (Recall that the bound for p = ∞ then follows from setting p = log d .)

G.2.2 Communication constraints, and privacy constraints for ε ≥ 1

For these cases, to prove a lower bound with the desired dependence on ε or glyph[lscript] , we will need to use the tighter bounds in Corollaries 1 and 2 which hold only under Assumption 3. This, however, leads to an issue: the random vector φ z ( X ) = ( φ z,i ( X )) i ∈ [ d ] is not subgaussian, due to the one-sided exponential growth, and therefore Assumption 3 does not hold.

To overcome this and still obtain a linear dependence on glyph[lscript] (or ε ) (instead of the suboptimal 2 glyph[lscript] (or e ε )), we will consider instead the class of 'truncated' Gaussian distributions, whose corresponding φ functions are subgaussian; and argue that these truncated distributions are close enough to the original Gaussian distributions such a lower bound in the truncated case implies one in the original Gaussian case.

In particular, we consider the following collection of truncated Gaussian distributions. For z ∈ Z , let p z be the density function of a spherical Gaussian distribution with mean µ z as defined in Eq. (41). For a truncation bound B , let p z,B be the distribution of X ∼ p z conditioned on the event that ‖ X ‖ ∞ ≤ B . That is, we have, for x ∈ R d ,

where C z = 1 / Pr X ∼ p z [ ‖ X ‖ ∞ ≤ B ] . Then the following bound follows from standard Gaussian concentration bound on each dimension and a union bound over all dimensions.

Let p Y n z,B be the distribution of the messages obtained by executing the protocol when each user gets a sample from p z,B and let the corresponding mixtures be denoted by p Y n + i,B and p Y n -i,B . Then we have

The third inequality follows from data processing inequality and the fourth inequality follows from subadditivity of TV distance.

Combining this with Lemma 1, for any protocol that correctly learns the Gaussian family, we must have

Next we show that the φ functions corresponding to p z,B 's are subgaussian and establish the corresponding upper bounds on the average information bound above. Note that

By the inequality | ab -1 | ≤ | a | · | b -1 | + | a -1 | , we have have, for all z ∈ Z ,

Moreover, for all z ∈ Z , for γ ≤ 1 3 B ,

Hence, applying the inequality | ab -1 | ≤ | a | · | b -1 | + | a -1 | again on Eq. (44), we have for γ ≤ 1 3 B ,

Thus, we get that for all z ∈ Z , i ∈ [ d ] , φ B z,i is subgaussian with proxy σ B = 12 γB + 10 d 7 n 7 .

Under communication constraints, applying Corollary 2, we get

To conclude, we observe that by plugging our setting of γ = γ ( p ) in the above inequality, we must have

in order to satisfy Eq. (43), hence proving the desired lower bound. The lower bound for LDP with ε > 1 follows similarly by applying Corollary 1.

G.3 Detailed results for discrete family

We derive a lower bound for E p (∆ d , W , n ) , the minimax rate for discrete density estimation, under local privacy and communication constraints.

Theorem 6. Fix p ∈ [1 , ∞ ) . For ε > 0 , and glyph[lscript] ≥ 1 , we have

and

In particular, for n ( ( e ε -1) 2 ∧ e ε ) ≥ d 2 and n (2 glyph[lscript] ∧ d ) ≥ d 2 , the first term of the corresponding lower bounds dominates. Before turning to the proof of this theorem, we note that Corollary 3 and Corollary 4 are direct corollaries of the theorem.

We now establish Theorem 6.

Proof of Theorem 6. Fix p ∈ [1 , ∞ ) , and suppose that, for some γ ∈ (0 , 1 / 16] , there exists an ( n, γ ) -estimator for ∆ d under glyph[lscript] p loss. Set

and assume, without loss of generality, that D is even. By definition, we then have γ ∈ (0 , 1 / (16 D 1 -1 /p )] and D ≤ d ; we can therefore restrict ourselves to the first D elements of the domain, embedding ∆ D into ∆ d , to prove our lower bound.

Let k = D 2 , Z = {-1 , +1 } D/ 2 , and τ = 1 2 ; and suppose that, for some γ ∈ (0 , 1 / (16 D 1 -1 /p )] , there exists an ( n, γ ) -estimator for ∆ D under glyph[lscript] p loss. (We will use the fact that γ ≤ 1 / (16 D 1 -1 /p ) for Eq. (49) to be a valid distribution with positive mass, as we will need | γ | ≤ 1 D ; and to bound α 2 later on, as we will require | γ | ≤ 1 2 D .) Define γ = γ ( p ) as

which implies γ ∈ [0 , 1 / (2 D )] . Consider the set of D -ary distributions P γ Discrete = { p z } z ∈Z defined as follows. For z ∈ Z , and x ∈ X = [ D ]

For z ∈ Z and i ∈ [ D/ 2] , we have

where

glyph[negationslash]

Once again, we can verify that for i = j

so that Assumptions 1 and 2 are satisfied for α 2 := 16 γ 2 D (using that Dγ ≤ 1 / 2 to simplify the bound). 10 Thus, we can invoke the first part of Theorem 2. Note that Assumption 4 holds, since

For W priv ,ε , by combining the bounds obtained by Corollary 1 and Lemma 1, we get

whereby, upon recalling the value of α 2 and using the setting of γ = γ ( p ) from Eq. (48), it follows that

Thus we obtain the bound Eq. (46) as claimed.

Similarly, for W comm ,glyph[lscript] , upon combining the bounds obtained by Corollary 2 and Lemma 1 and recalling that |Y| = 2 glyph[lscript] , we get

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses parametric estimation under a communication setup. This setup adds variation to classic parametric estimation and focuses on the setup $\\theta \\to X^n \\to Y^n$ where the goal is to estimate $\\theta$ given $Y^n$, which is generated in an interactive, sequential manner with $Y_i$ possibly dependent on other signals. \n\nThe main result of this paper is Theorem 1, which gives an information contraction bound. With additional assumptions on orthogonality and subgaussianity, the paper's next result is Theorem 2, which gives two corollary information contraction bounds expressed in terms of variance, mean and mutual information terms relating to channels.\n\nThe authors then provide several examples of tightness (and near tightness) for these bounds by looking at minimax rates for examples such as product Bernoulli, sparse Gaussian, and discrete distributions under information or communication constraints.\n\nSTRENGTHS:\n**Originality**: The paper builds upon the classic framework of parametric estimation under an interactive framework. While the problem setup itself is not original, this paper contributes new findings to minimax theory for these setups, particularly with communication and privacy constraints.\n\n**Quality**: The paper is nicely written, and the authors explain the setting and results well. No glaring issues were found with regards to technicalities.\n\n**Clarity**: Overall, the paper is cleanly written, bar some comments stated in below sections. \n\n**Significance**: The interactive parametric estimation problem considered by the paper is an interesting one and is directly relevant to the community.\n\nWEAKNESSES:\nOverall, the paper itself is quite nicely written and the topic is interesting. Some comments regarding weaknesses include:\n\n1. The paper is somewhat structured reversely in many scenarios, where the paper references equations that appear later on in the paper. This may cause a little bit of confusion for readers.\n\n2. It may be helpful to highlight the difference in technical contributions (for example, proof techniques) between this work, [4], and [16], considering the problem setup is quite similar and both this paper and [16] \"build upon the framework presented for the discrete setting in [4]\". What are some scenarios where the minimax results in this paper can significantly improve upon results in [16]?\n\nQUESTIONS:\n1. While the paper focuses on *interactive* protocols, it is curious whether similar techniques can be applied to the general parametric estimation model (i.e., in the setting where Y = X) and achieve tighter lower bounds than previous works, considering the techniques used in this paper do not depend on \"bounded ratio\" type assumptions as claimed in the paper. Would it be possible to use results in the paper to achieve significant improvement to classic minimax bounds for sparse/non-sparse Gaussians with non-identity covariance $\\mathbb{I}$ such as in the cases where covariance matrix eigenvalues are largely varying (i.e., on a different order)? \n\n2. How much are we sacrificing by using the inequality in (30) within the supplementary material? Would not dropping the second term in the denominator give improved results?\n\n3. How limiting is the assumption of Assumption 4 and its corresponding Lemma 1? In other words, how easy is it to find $\\theta$'s that satisfy this assumption?\n\n4. It may be beneficial to add some references to the parts of the paper where it is claimed that previous works regarding bounded ratio type assumptions are \"flawed\" (for example, maybe previous papers that analyze this flaw).\n\n\nSome additional suggestions/typos/confusions:\n\na. The term $S$ is not clearly defined in Equation (2). It may help to add some description of what this means.\n\nb. Line 193 seems to be incorrectly formatted.\n\nLIMITATIONS:\nThere does not appear to be any negative societal impact associated with this paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This work investigates the distributed parameter estimation problem under local information constraints such as communication constraints, local privacy constraints, and restricted measurements. The authors focus on information-theoretic lower bounds for the minimax error rates of these problems and present "plug-and-play" lower bounds that can be applied to various estimation problems. In addition, for most cases, the lower bounds are complemented with matching upper bounds.

STRENGTHS:
This work presents a unified framework for deriving a variety of minimax lower bounds for various families of distributions under local information constraints for the distributed parameter estimation problem. Additionally, matching upper bounds have been provided to demonstrate that the lower bounds are tight for most settings.

WEAKNESSES:
It would be beneficial to see a table displaying the lower/upper bounds for the various cases considered in this work.

QUESTIONS:
no

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies parameter estimation problem in a setting with the following two important assumptions:\n1. Raw samples X_i's are generated from a certain parametric distribution, however, they are not directly observed. Instead, Y_i's which are generated from channels subject to certain constraints are observed. These constraints can encapsulate various problems of interest, notably communication constraints and local privacy. \n2. Interaction is allowed. That is, Y_i's are generated with memory. \n\nThe main result is a general abstract upper bound on the sensitivity of the transcript (i.e., public view Y_i's and common randomness if any) of the interactive protocol. \nThis, upon nontrivial specification, yields a suite of tight or nearly tight impossibility results for problems mentioned above.\n\nSTRENGTHS:\nI'm not an expert in this field. \nThough the main text is somewhat dense, I found it pretty accessible. \nMost importantly, the results seem pretty satisfactory.\n\nWEAKNESSES:\nI don't see obvious weaknesses. \nPlease instead see my questions below (most of which are very minor).\n\nQUESTIONS:\nMajor questions:\n1. Is it possible to explain why in the main bounds (Theorem 1 & 2), TV distance was used? Is this the only reasonable choice? If so, why?\nIf not, is using other distance/info measure beneficial in any way, e.g., simplifying the proof, improving leading constant factors, etc.?\n\n2. I saw that most examples in this paper are regarding product (or \"product-like\") distributions (e.g., product binary distribution, isotropic Gaussian). And my shallow understanding is that this seems to be a limitation of the technique. \nThis question may go beyond the scope of the paper, but I'm wondering if it is possible to obtain similar results when certain structured correlation is present. \nE.g., Gaussian with general covariance, or simple temporal correlation such as Markovian structure. \n\nOther comments:\n\n1. Line 28, \"local information constraint\". I'm not familiar with this line of work and this may be a silly question. Could the authors offer some examples of non-local/global constraints?\n\n2. I feel like already in line 44, the buzz word \"Assouad\" should have been mentioned. This is a rather standard tool and a general audience can smell that when reading upon line 44. \n\n3. Mysteriously the line breaks at 193. \n\n4. In Equation (2), the Gothic letter under the first sup was defined in footnote 2. It took me a while to find it. Perhaps upgrade this footnote to the main text?\n\n5. To information theorists the notation y^t (perhaps more commonly: y_1^t) for (y_1, ..., y_t) is common but I'm not sure how standard it is to a general reader, especially given that this can be confused with the t-th power of a number. \nI encourage the authors define this notation somewhere. \n\n6. The notation p_Z^{Y^n} denotes the density of (Y^n, U), not just Y^n. Somehow U is dropped. Is there a reason for doing so?\n\n7. In Equation (3), the integral is against y w.r.t. mu. Sometimes it might be slightly more clear to explicitly write $\\mathrm{d}\\\\, \\mu(y)$. \nAlso, this measure mu was defined in footnote 2, which is somewhat hidden. \n\n8. A typo on line 359, double commas following \"i.e.\".\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies distributed parameter estimation under local information constraints. In this problem, independent samples $X_1,..., X_n$ are generated from an unknown distribution $p_\\theta$ from a parametric family $P_\\Theta$ of distributions. The samples are not accessible directly, rather available is only partial inofmration $Y_1, ..., Y_n$  generated by passing the original samples through classical channels $W_i$. The goal is to estimate the underlying parameter $\\theta$ by accessing $Y_1, ..., Y_n$. The channels can be interactive, dependig on the past observations. They also local information constraints,  including communication constraints, and privacy constraints. \n\nThis paper derives lower bounds on the sample compelxity of this problem in a general framework. Moreover, this work explores applications of the general bound to problems of high-dimensional mean estimation and distribution estimation, under privacy and communication constraints, for the entire family of $l_p$ loss functions for $p \\geq 1$.\n\nSTRENGTHS:\nThe problem formulation and the related results are substantially general and can be used in various problems involving different constraints. \nThe paper is rigorous and well written.\n\nWEAKNESSES:\nIt is not clear how significant the generalization is compared to previous works (i.e., reference [4]). Is the paper generalizing [4] to $l_p$ loss functions? If that is the case, I am not sure how important that is beyond a mathematical curiosity. \n\nMinor comment: it looks like that the symbol $\\wedge$ is not defined.\n\nQUESTIONS:\nAs another point, the current approach focuses on a finite perturbation subset of $\\Theta$. How does this approach work for continuous parameter space $\\Theta$ ? Do we need to make sure that the Z space is a good representation of $\\Theta$?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work considers the problem of proving sample complexity lower bounds for parameter/density estimation in a distributed setting with information constraints (local privacy, communication constraints etc). What distinguishes this paper from prior work is that it focuses on the scenario where there is interaction between the users who share their data. Thus, while the data-points are initially generated in an i.i.d. fashion, when the time comes for the data-points to be subjected to the information constraints, the results depend on previous users' actions.\n\nThe authors develop general tools for obtaining lower bounds in this setting, where the main challenge involves dealing with the dependencies implied by the interactions between users. The main tool developed is a lemma that is essentially reminiscent of Assouad's method that is adapted to the setting of sequentially interactive distributed protocols with information constraints. The resulting framework manages to capture various existing techniques such as strong data-processing inequalities and bounds derived using the distributed version of the Bayesian Cramer-Rao bound know as the Van Trees inequality. The paper also includes applications of these techniques to a number of estimation problems, while the derived lower bounds are in most cases complemented by nearly-matching upper bounds. Finally, there's a description of how the techniques could be adapted to work in the fully interactive setting as well.\n\nSTRENGTHS:\nThis is a quite strong paper with technical contributions that improve our understanding of statistical estimation tasks in the distributed setting. The work provides both techniques (which are presented in a quite general way) and applications (which confirm the effectiveness of said techniques). The applications include the fundamental tasks of mean estimation for binary product distributions and Gaussian distributions under the $\\ell_p$-norm, as well as density estimation for discrete distributions. The work is also notable for identifying mistakes in previous papers where some of these results were claimed and using the developed framework to fix the proofs. Finally, the writing is very clear and the work is clearly positioned within the overall literature.\n\nWEAKNESSES:\nNo obvious weaknesses.\n\nQUESTIONS:\nI have one question. The applications given in this paper within the context of parameter estimation for high-dimensional distributions all involve independent marginals (binary product distributions, spherical Gaussians). Additionally, the lemma the proofs of these statements rely on (lemma 3) is an analogue of Assouad's method. Assouad's method assumes that the family of distributions considered as part of proving the lower bound are parameterized by the vertices of the boolean hypercube, with each coordinate being drawn independently. This frequently ends up limiting the applicability of the method to problems involving distributions with independent marginals. Is this a limitation of the work, or is there evidence the method can lead to tight lower bounds even with non-product distributions (e.g., for Gaussian covariance estimation)?\n\nLIMITATIONS:\nThis is a purely theoretical work with a focus on lower bounds. Thus, there is no apparent societal impact (positive or negative).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}