SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ziteng Sun
Google Research, New York
zitengsun@google.com
Himanshu Tyagi
Indian Institute of Science, Bangalore
htyagi@iisc.ac.in
We consider distributed parameter estimation using interactive protocols subject to local information constraints such as bandwidth limitations, local differential privacy, and restricted measurements. We provide a unified framework enabling us to derive a variety of (tight) minimax lower bounds for different parametric families of distributions, both continuous and discrete, under any  $\ell_p$  loss. Our lower bound framework is versatile and yields "plug-and-play" bounds that are widely applicable to a large range of estimation problems. In particular, our approach recovers bounds obtained using data processing inequalities and Cramér-Rao bounds, two other alternative approaches for proving lower bounds in our setting of interest. Further, for the families considered, we complement our lower bounds with matching upper bounds.
We consider the problem of parameter estimation under local information constraints, where the estimation algorithm has access to only limited information about each sample. These constraints can be of various types, including communication constraints, where each sample must be described using a few (e.g., constant number of) bits; (local) privacy constraints, where each sample is obtained from a different user and the users seek to reveal as little as possible about their specific data; as well as many others, e.g., noisy communication channels, or limited types of data access such as linear measurements. Such problems have received significant attention in recent years, motivated by applications such as data analytics in distributed systems and federated learning.
Our main focus is on information-theoretic lower bounds for the minimax error rates (or, equivalently, the sample complexity) of these problems. Several recent works have provided bounds that apply to specific constraints or work for specific parametric estimation problems, sometimes without allowing for interactive protocols. Indeed, handling interactive protocols is technically challenging, and several results in prior work exhibit flaws in their analysis. In particular, even the most basic Gaussian mean estimation problem using interactive communication remains, surprisingly, open.
We present general, "plug-and-play" lower bounds for parametric estimation under information constraints that can be used for any local information constraint and allows for interactive protocols. Our abstract bound requires very simple (and natural) assumptions to hold for the underlying parametric family; in particular, we do not require technical "regularity" conditions that are common in asymptotic statistics.
We apply our general bound to canonical problems of high-dimensional mean estimation and distribution estimation, under privacy and communication constraints, for the entire family of  $\ell_p$  loss functions for  $p\geq 1$ . In addition, we provide complementary schemes that show that our lower bounds are tight for most settings of interest.
Our main contribution is a general approach to establish lower bounds in distributed information-constrained parameter estimation. The setup is described in detail in Section 2 and is illustrated in Fig. 1. In short, independent samples  $X^n = (X_1, \ldots, X_n)$  are generated from an unknown distribution  $\mathbf{p}$  from a parametric family  $\mathcal{P}_{\Theta} = \{\mathbf{p}_{\theta}, \theta \in \Theta\}$  of distributions. Only limited information  $Y_i$  about datum  $X_i$  is available to the algorithm. The goal is to estimate the underlying parameter  $\theta$  associated with  $\mathbf{p}$ . Furthermore, we consider interactive estimation, wherein  $Y_i$  can depend on  $Y_1, \ldots, Y_{i-1}$ . Our general lower bound, which we develop in Section 3, takes the following form: Consider a collection of distributions  $\{\mathbf{p}_z\}_{z \in \{-1, +1\}^k} \subseteq \mathcal{P}_{\Theta}$  contained in the parametric family. This collection represents a "difficult subproblem" that underlies the parametric estimation problem being considered; such constructions are often used when deriving information-theoretic lower bounds (e.g., in Assouad's method [33]). Note that each coordinate of  $z$  represents, in essence, a different "direction" of uncertainty for the parameter space. The difficulty of the estimation problem can be related to the difficulty of determining a randomly chosen  $z$  (or most of the coordinates of  $z$ ), denoted  $Z$ , by observing samples from  $\mathbf{p}_z$ . Once  $Z = z$  is fixed,  $n$  independent samples  $X^n = (X_1, \ldots, X_n)$  are generated from  $\mathbf{p}_z$  and the limited information  $Y_i$  about  $X_i$  is passed to an estimator.
Our most general result, stated as Theorem 1, is an upper bound for the average discrepancy, an average distance quantity related to average probability of error in determining coordinates of  $Z$  by observing the limited information  $Y^{n} = (Y_{1},\dots ,Y_{n})$ . Our bounding term reflects the underlying information constraints using a quantity that captures how "aligned" we can make our information about the sample to the uncertainty in different coordinates of  $Z$ . Importantly, our results hold under minimal assumptions. In particular, in contrast to many previous works, our results do not require any "bounded ratio" assumption on the collection  $\{\mathbf{p}_z\}_{z\in \{-1, + 1\} ^k}$ , which would ask that the density function change by at most a constant factor if we modify one coordinate of  $z$ . When we impose additional structure for  $\mathbf{p}_z$  - such as orthogonality of the random changes in density when we modify different coordinates of  $z$  and, more stringently, independence and subgaussianity of these changes - we get concrete bounds which are readily applicable to different problems. These plug-and-play bounds are stated as consequences of our main result in Theorem 2. The interested reader can also directly consider the applications to local privacy (Corollary 1) or communication constraints (Corollary 2).
We demonstrate the versatility of the framework by showing that it readily yields tight (and in some cases nearly tight) bounds for parameter estimation (both in the sparse and dense cases) for several fundamental families of continuous and discrete distributions, several families of information constraints such as communication and local differential privacy (LDP), and for the family of  $\ell_p$  loss functions for  $p \geq 1$ , all when interactive protocols are allowed. To complement our lower bounds, we provide algorithms (protocols) which attain the stated rates, thus establishing optimality of our results. We discuss these results in Section 5, where we provide the corresponding statements. In terms of the applications, our contributions are two-fold:
1. We obtain several results from a diverse set of prior works in a unified fashion as simple corollaries of our main result: As discussed further in Section 1.2, the lower bounds for mean estimation for product Bernoulli under  $\ell_2$  loss and those for estimation of discrete distributions under  $\ell_1$  and  $\ell_2$  losses, for both communication and LDP constraints, were known from previous work. However, our approach allows us to easily recover those results and extend them to arbitrary  $\ell_p$  losses, with interaction allowed, in a unified fashion.
2. Our bounds also yield new lower bounds for some canonical problems. The prototypical example being mean estimation for high-dimensional Gaussian distributions under information constraints. As discussed in the next section, while some prior work claimed lower bounds for this problem, their arguments appear to be flawed – at a high level, due to the
<sup>1</sup>Up to a logarithmic factor in the case of  $\ell_{\infty}$  loss, or, for some of our bounds, with a mild restriction on  $n$  being large enough.
"bounded ratio" assumption their techniques rely on, which Gaussian distributions do not satisfy, and which our framework does not require. To the best of our knowledge our work is the first to obtain those lower bounds for interactive mean estimation of high-dimensional Gaussian distributions under communication or local privacy constraints.
1.2 Previous and related work
There is a significant amount of work in the literature dedicated to parameter estimation under various constraints and settings. Here, we restrict our discussion to works that are most relevant to the current paper, with a focus on the interactive setting (either the sequential or blackboard model; see Section 2 for definitions).
The work arguably closest to ours is the recent work [5], which focuses on density estimation and goodness-of-fit testing of discrete distributions, under the  $\ell_1$  metric, for sequentially interactive protocols under general local information constraints (including, as special cases, local privacy and communication constraints, as in the present paper). This work can be seen as a significant generalization of the techniques of [5], allowing us to obtain lower bounds for estimation in a variety of settings, notably high-dimensional parameter estimation.
Among other works on high-dimensional mean estimation under communication constraints, [17, 10] consider communication-constrained Gaussian mean estimation in the blackboard communication model, under  $\ell_2$  loss. The protocols for the upper bounds in these works do not require interactivity and are complemented with lower bounds which show that the bounds are tight up to constant factors in the dense case and up to logarithmic factors in the sparse case. However, the proof of the lower bound in [10] seems to present a gap (specifically, in the truncation argument of [10, Theorem 4.3]), as confirmed in personal communication with the authors. Correcting the issue in the truncation argument would lead to a result significantly weaker than the claimed lower bound, and it is unclear whether this can be fixed using the techniques from that paper.
In this work, we present interactive protocols for the sparse case which improve over the noninteractive protocols and strengthen the upper bounds by a logarithmic factor in the interactive case (see Remark 1). Further, using our general framework, we establish a nearly-matching lower bound for the problem, recovering the rate lower bound originally claimed in [10] up to a logarithmic factor. In a slightly different setting, [26] considers the mean estimation problem for product Bernoulli distributions when the mean vector is 1-sparse, under  $\ell_2$  loss. The lower bound in [26], too, allows sequentially interactive protocols. In the blackboard communication model, [19] and [18] obtained tight bounds for mean estimation and density estimation under  $\ell_2$  and  $\ell_1$  loss, respectively.
Turning to local privacy, [23] provide upper bounds (as well as some partial lower bounds) for one-dimensional Gaussian mean estimation under LDP under the  $\ell_2$  loss, in the sequentially interactive model. Recent works of [7] and [8] obtain lower bounds for mean estimation in the blackboard communication model and under LDP, respectively, for both Gaussian and product Bernoulli distributions; as well as density estimation for discrete distributions. Their approach is based on the classic Cramér-Rao bound and, as such, is tied inherently to the use of the  $\ell_2$  loss. In a recent independent work, [25] extended these methods to obtain lower bounds under general  $\ell_p$  loss under communication constraints, which are tight for Gaussian mean estimation under noninteractive protocols. [14], by developing a locally private counterpart of some of the well-known information-theoretic tools for establishing statistical lower bounds (namely, Le Cam, Fano, and Assouad), establish tight or nearly tight bounds for several mean estimation problems in the LDP setting.
More recently, drawing on machinery from the communication complexity literature, [13] develop a methodology for proving lower bounds under LDP constraints in the blackboard communication model. They obtain lower bounds for mean estimation of product Bernoulli distributions under general  $\ell_p$  losses which match ours (in the high-privacy regime, i.e., small  $\varepsilon$ ). Similar to the results under communication constraints [17, 10], their approach relies heavily on the assumption that the distributions on each coordinate are independent, which fails to generalize to discrete distributions. Further, their bounds are tailored to the LDP constraints and do not seem to extend to arbitrary information constraints. Finally, while [13] also claims tight bounds for mean estimation of Gaussian and sparse Gaussian distributions under  $\ell_2$  loss, their argument invokes the analogous (flawed) result from [10] and thus it is unclear whether the stated lower bound can be shown using their techniques.
Finally, we mention that very recently, following the appearance of [5], an updated version of [19] appeared online as [21], which has similar results as ours for the high-dimensional mean estimation problem under communication constraints. Both our work and [21] build upon the framework presented for the discrete setting in [5]. Moreover, their work still need the "bounded ratio", and hence their lower bound for sparse Gaussian family only works for noninteractive protocols.
Notation. Hereafter, we write  $\log$  and  $\ln$  for the binary and natural logarithms, respectively. For distributions  $\mathbf{p}_1, \mathbf{p}_2$  over  $\mathcal{X}$ , denote their Kullback-Leibler divergence (in nats) by  $\mathrm{D}(\mathbf{p}_1\|\mathbf{p}_2)$ , and their Hellinger distance by  $\mathrm{d_H}(\mathbf{p}_1,\mathbf{p}_2) := \left(\frac{1}{2}\int\left(\sqrt{\frac{\mathrm{d}\mathbf{p}_1}{\mathrm{d}\lambda}} - \sqrt{\frac{\mathrm{d}\mathbf{p}_2}{\mathrm{d}\lambda}}\right)^2\mathrm{d}\lambda\right)^{1/2}$ , where we assume  $\mathbf{p}_1,\mathbf{p}_2 \ll \lambda$  for some underlying measure  $\lambda$  on  $\mathcal{X}$ . Further, we denote the Shannon entropy of a random variable  $X$  by  $H(X)$  and the mutual information between  $X$  and  $Y$  by  $I(X;Y)$ ; we will sometimes write  $H(\mathbf{p})$  for the entropy of a random variable with distribution  $\mathbf{p}$ . We refer the reader to [12] for details on these notions and their properties, which will be used throughout. Given two functions  $f,g$ , we write  $f \lesssim g$  if there exists an absolute constant  $C > 0$  such that  $f(x) \leq Cg(x)$  for all  $x$ ; and  $f \asymp g$  if  $f \lesssim g$  and  $f \gtrsim g$  both hold. Finally, we use the standard asymptotic notation  $O(f),\Omega(f),\Theta(f)$ .
Organization. In Section 2, we formalize our setting of interactive inference under local information constraints. The general lower bound framework and results are presented in Section 3, where we provide implications of the general result under additional structures and specific information constraints such as local privacy (LDP) and communication constraints. Finally, we use our framework to readily derive lower bounds for a wide range of applications in Section 5. Due to space constraints, all proofs, as well as our upper bounds (algorithms) are provided in the Supplement, where we also discuss how our techniques compare with other existing approaches for proving lower bounds under information constraints, namely, those based on strong data processing inequalities (SDPI) or on the van Trees inequality.
2 The setup
We consider standard parametric estimation problems. For some  $\Theta \subseteq \mathbb{R}^d$ , let  $\mathcal{P}_{\Theta} = \{\mathbf{p}_{\theta}, \theta \in \Theta\}$  be a family of distributions over some measurable space  $(\mathcal{X}, \mathfrak{X})$ , namely each  $\mathbf{p}_{\theta}$  is a distribution over  $(\mathcal{X}, \mathfrak{X})$ . Suppose  $n$  independent samples  $X^n = (X_1, \ldots, X_n)$  from an unknown  $\mathbf{p}_{\theta} \in \mathcal{P}_{\Theta}$  are obtained. The goal in parametric estimation is to design estimators  $\hat{\theta}: \mathcal{X}^n \to \Theta$ , and form estimates  $\hat{\theta}(X^n)$  of  $\theta$  using independent samples  $X^n$  from  $\mathbf{p}_{\theta}$ . We illustrate our results using two specific distribution families: discrete probability mass functions (pmfs) and high-dimensional product distributions with unknown mean vectors. We will describe the precise minimax setting in detail later in this section.
We are interested in an information-constrained setting, where we do not have direct access to the samples  $X^n$  from  $\mathbf{p}_{\theta}$ . Instead, we can only obtain limited information about each datapoint  $X_{i}$ . Following [3], we model these information constraints by specifying an allowed set of channels  $\mathcal{W}$  with input alphabet  $\mathcal{X}$  and some output space  $\mathcal{Y}$ . Each sample  $X_{i}$  is passed through a channel from  $\mathcal{W}$ , chosen appropriately, and its output  $Y_{i}$  is the observation we get. This setting is quite general and captures as special cases the popular communication and privacy constraints, as we will describe momentarily.
We now formally describe the setting, which is illustrated in Fig. 1.  $n$  i.i.d. samples  $X_{1},\ldots ,X_{n}$  from an unknown distribution  $\mathbf{p}_{\theta}\in \mathcal{P}_{\Theta}$  are observed by players (users) where player  $t$  observes  $X_{t}$ . Player  $t\in [n]$  selects a channel  $W_{t}\in \mathcal{W}$  and sends the message  $Y_{t}$  to a referee, where  $Y_{t}$  is drawn from the probability measure  $W_{t}(\cdot \mid X_{t},Y_{1},\dots ,Y_{t - 1})$ . The referee observes  $Y^{n}\coloneqq (Y_{1},\dots ,Y_{n})$  and seeks to estimate the parameter  $\theta$ .
The freedom allowed in the choice of  $W_{t}$  at the players gives rise to various communication protocols. We focus on interactive protocols, where channels are chosen by one player at a time, and they can use all previous messages to make this choice. We describe this class of protocols below, where we further allow each player  $t$  to have a different set of constraints  $\mathcal{W}_t$  (e.g., a different communication budget), and  $W_{t}$  must be in  $\mathcal{W}_t$ . For simplicity of exposition, and as these already encapsulate most
Formally, a channel is a Markov kernel  $W: \mathfrak{Y} \times \mathcal{X} \to [0,1]$ , which we assume to be absolutely continuous with respect to some underlying measure  $\mu$  on  $(\mathcal{V}, \mathfrak{V})$ . When clear from the context, we will drop the reference to the  $\sigma$ -algebras  $\mathfrak{X}$  and  $\mathfrak{Y}$ ; in particular, in the case of finite  $\mathcal{X}$  or  $\mathcal{Y}$ .
Figure 1: The information-constrained distributed model. In the interactive setting,  $W_{t}$  can depend on the previous messages  $Y_{1},\ldots ,Y_{t - 1}$  (dotted, upwards arrows).
of the difficulties, we focus here on the case of sequentially interactive protocols, which has been widely considered in the literature and captures many settings of interest. However, we emphasize that our results extend to the more general class of fully interactive protocols (see Supplement).
Definition 1 (Sequentially Interactive Protocols). Let  $X_{1},\ldots ,X_{n}$  be i.i.d. samples from  $\mathbf{p}_{\theta}$ ,  $\theta \in \Theta$ . A sequentially interactive protocol  $\Pi$  using  $\mathcal{W}^n = (\mathcal{W}_1,\dots ,\mathcal{W}_n)$  involves mutually independent random variables  $U,U_{1},\ldots ,U_{n}$  (independent of the input  $X_{1},\ldots ,X_{n}$ ) and mappings  $g_{t}\colon (U,U_{t})\mapsto W_{t}\in \mathcal{W}_{t}$  for selecting the channel in round  $t\in [n]$ . In round  $t$ , player  $t$  uses the channel  $W_{t}$  to produce the message (output)  $Y_{t}$  according to the probability distribution  $W_{t}(\cdot \mid X_{t},Y_{1},\dots ,Y_{t - 1})$ . The messages  $Y^{n} = (Y_{1},\dots ,Y_{n})$  received by the referee and the public randomness  $U$  (available to all players) constitute the transcript of the protocol  $\Pi$ ; the private randomness  $U_{1},\ldots ,U_{n}$  (where  $U_{t}$  is local to player  $t$ ) is not part of the transcript. In other words, the channel at player  $t$  as a (randomized) mapping  $W\colon \mathcal{X}\times \mathcal{Y}^{t - 1}\to \mathcal{V}$ , which depends on input  $x\in \mathcal{X}$  and the previous  $t - 1$  messages  $y^{t - 1}\in \mathcal{Y}^{t - 1}$  outputs some  $y\in \mathcal{V}$ .
For concreteness, we now instantiate this definition for the two aforementioned types of information constraints, communication and (local) privacy.
Communication constraints Let  $\mathcal{Y} := \{0,1\}^* = \bigcup_{m=0}^{\infty} \{0,1\}^m$ . For  $\ell \geq 1$  and  $t \geq 1$ , let
$$
\mathcal {W} ^ {\operatorname {c o m m}, \ell} := \{W: \mathcal {X} \times \mathcal {Y} ^ {*} \rightarrow \{0, 1 \} ^ {\ell} \} \tag {1}
$$
be the family of channels with input alphabet  $\mathcal{X}$  and output alphabet the set of all  $\ell$ -bit strings. This captures the constraint where the message from each player can be at most  $\ell$  bits long, and corresponds to the choice  $\mathcal{W}^n = (\mathcal{W}^{\mathrm{comm},\ell},\ldots ,\mathcal{W}^{\mathrm{comm},\ell})$ . Note that allowing a different communication budget to each player can be done by setting  $\mathcal{W}^n = (\mathcal{W}^{\mathrm{comm},\ell_1},\ldots ,\mathcal{W}^{\mathrm{comm},\ell_n})$ .
Local differential privacy constraints For  $\varepsilon > 0$  and  $t \geq 1$ , a channel  $W \colon \mathcal{X} \times \mathcal{Y}^{t-1} \to \mathcal{Y}$  is  $\varepsilon$ -locally differentially private (LDP) [16, 15, 24] if
$$
\sup  _ {S \in \mathfrak {Y}} \sup  _ {y ^ {t - 1} \in \mathcal {Y} ^ {t - 1}} \frac {W (S \mid x _ {1} , y ^ {t - 1})}{W (S \mid x _ {2} , y ^ {t - 1})} \leq e ^ {\varepsilon}, \quad \forall x _ {1}, x _ {2} \in \mathcal {X}. \tag {2}
$$
We denote by  $\mathcal{W}^{\mathrm{priv},\varepsilon}$  the set of all  $\varepsilon$ -LDP channels. For sequentially interactive protocols, the  $\varepsilon$ -LDP condition is captured by setting  $\mathcal{W}^n = (\mathcal{W}^{\mathrm{priv},\varepsilon},\ldots,\mathcal{W}^{\mathrm{priv},\varepsilon})$ . As before, one can allow different privacy parameters for each player by setting  $\mathcal{W}^n = (\mathcal{W}^{\mathrm{priv},\varepsilon_1},\ldots,\mathcal{W}^{\mathrm{priv},\varepsilon_n})$ .
Finally, we formalize the interactive parametric estimation problem for the family  $\mathcal{P}_{\Theta} = \{\mathbf{p}_{\theta},\theta \in \Theta \}$ . We consider the problem of estimating  $\theta$  under  $\ell_p$  loss. For  $p\in [1,\infty)$ , the  $\ell_p$  distance between
$u, v \in \mathbb{R}^d$  is  $\ell_p(u, v) = \| u - v\|_p = \left(\sum_{i=1}^d |u_i - v_i|^p\right)^{1/p}$ . This definition extends in a natural way to  $p = \infty$  by taking the limit.
Definition 2 (Sequentially Interactive Estimates). Fix  $d \in \mathbb{N}$  and  $p \in [0, \infty]$ . Given a family  $\mathcal{P}_{\Theta}$  of distributions on  $\mathcal{X}$ , with  $\Theta \subset \mathbb{R}^d$ , an estimate for  $\mathcal{P}_{\Theta}$  consists of a sequentially interactive protocol  $\Pi$  with transcript  $(Y^n, U)$  and estimator  $\hat{\theta}: (Y^n, U) \mapsto \hat{\theta}(Y^n, U) \in \Theta$ . The referee observes the transcript  $(Y^n, U)$  and forms the estimate  $\hat{\theta}(Y^n, U)$  of the unknown  $\theta$ . Further, for  $n \in \mathbb{N}$  and  $\gamma \in (0, 1)$ ,  $(\Pi, \hat{\theta})$  constitutes an  $(n, \gamma)$ -estimator for  $\mathcal{P}_{\Theta}$  using  $\mathcal{W}$  under  $\ell_p$  loss if for every  $\theta \in \Theta$  the transcript  $(Y^n, U)$  of  $\Pi$  satisfies
$$
\mathbb {E} _ {\mathbf {P} _ {\theta} ^ {n}} \left[ \ell_ {p} (\theta , \hat {\theta} (Y ^ {n}, U)) ^ {p} \right] ^ {1 / p} \leq \gamma .
$$
Note that the expectation is over the input  $X^n\sim \mathbf{p}_\theta^n$  for the protocol  $\Pi$  and the randomness of  $\Pi$ .
3 Main result: The information contraction bound