SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 4 (Interactive Protocols). Let  $X_{1},\ldots ,X_{n}$  be i.i.d. samples from  $\mathbf{p}_{\theta}$ ,  $\theta \in \Theta$ , and  $\mathcal{W}^*$  be a collection of sequences of pairs of channel families and players; that is, each element of  $\mathcal{W}^*$  is a sequence  $(\mathcal{W}_t,j_t)_{t\in \mathbb{N}}$  where  $j_{t}\in [n]$ . An interactive protocol  $\Pi$  using  $\mathcal{W}^*$  comprises a random variable  $U$  (independent of the input  $X_{1},\ldots ,X_{n}$ ) and, for each  $t\in \mathbb{N}$ , mappings
$$
\sigma_ {t} \colon Y _ {1}, \dots , Y _ {t - 1}, U \mapsto N _ {t} \in [ n ] \cup \{\bot \}
$$
$$
g _ {t} \colon Y _ {1}, \dots , Y _ {t - 1}, U \mapsto W _ {t}
$$
with the constraint that  $((W_1, N_1), \ldots, (W_t, N_t))$  must be consistent with some sequence from  $\mathcal{W}^*$ ; that is, there exists  $((\mathcal{W}_s, j_s))_{s \in \mathbb{N}} \in \mathcal{W}^*$  such that  $W_s \in \mathcal{W}_s$  and  $N_s = j_s$  for all  $1 \leq s \leq t$ . These two mappings respectively indicate (i) whether the protocol is to stop (symbol  $\perp$ ), and, if not, which player is to speak at round  $t \in \mathbb{N}$ , and (ii)) which channel this player selects at this round.
In round  $t$ , if  $N_{t} = \bot$ , the protocol ends. Otherwise, player  $N_{t}$  (as selected by the protocol, based on the previous messages) uses the channel  $W_{t}$  to produce the message (output)  $Y_{t}$  according to the probability measure  $W_{t}(\cdot \mid X_{N_{t}})$ . We further require that  $T := \inf \{ t \in \mathbb{N} : N_{t} = \bot \}$  is finite a.s. The messages  $Y^{T} = (Y_{1},\dots ,Y_{T})$  received by the referee and the public randomness  $U$  constitute the transcript of the protocol  $\Pi$ .
In other terms, the channel used by the player  $N_{t}$  speaking at time  $t$  is a Markov kernel
$$
W _ {t} \colon \mathfrak {Y} _ {t} \times \mathcal {X} \times \mathcal {Y} ^ {t - 1} \rightarrow [ 0, 1 ],
$$
with  $\mathcal{V}_t\subseteq \mathcal{V}$ ; and, for player  $j\in [n]$ , the allowed subsequences  $(\mathcal{W}_t,j_t)_{t\in \mathbb{N}:j_t = j}$  capture the possible sequences of channels allowed to the player. As an example, if we were to require that any single player can speak at most once, then for every  $j\in [n]$  and every  $(\mathcal{W}_t,j_t)_{t\in \mathbb{N}}\in \mathcal{W}^n$ , we would have  $\sum_{t = 1}^{\infty}\mathbb{1}\{j_t = j\} \leq 1$ .
In the interactive model, we can then capture the constraint that each player must communicate at most  $\ell$  bits in total by letting  $\mathcal{W}^n$  be the set of sequences  $(\mathcal{W}_t^{\mathrm{comm},\ell_t},j_t)_{t\in \mathbb{N}}$  such that
$$
\forall j \in [ n ], \quad \sum_ {t = 1} ^ {\infty} \ell_ {t} \cdot \mathbb {1} \left\{j _ {t} = j \right\} \leq \ell .
$$
In the simpler sequentially interactive model, this condition simply becomes the choice of  $\mathcal{W}^n = (\mathcal{W}^{\mathrm{comm},\ell},\dots,\mathcal{W}^{\mathrm{comm},\ell})$
A.1 Lower Bounds under Full Interactive Model
Next we discuss how our technique extends to the full interactive model. For any full interactive protocol  $\Pi$ , let  $Y^{*} \in \mathcal{V}^{*}$  be the message sequence generated by the protocol. Then, for all  $y^{*} \in \mathcal{V}^{*}$ , we have
$$
\operatorname * {P r} _ {X ^ {n} \sim \mathbf {p}} \left[ Y ^ {*} = y ^ {*} \right] = \mathbb {E} _ {X ^ {n} \sim \mathbf {p}} \left[ \prod_ {t = 1} ^ {\infty} W _ {t} \left(y _ {t} \mid X _ {\sigma_ {t} (y ^ {t - 1})}, y ^ {t - 1}\right) \right].
$$
The following lemma states that if  $X^n$  are generated from a product distribution, the distribution of the transcript satisfies a property similar to the "cut-and-paste" property from [6].
Lemma 2 ([20]). If  $X^n\sim \mathbf{p} = \otimes_{t = 1}^n\mathbf{p}_t$  , the transcript of the protocol satisfies
$$
\Pr_ {X ^ {n} \sim \mathbf {p}} [ Y ^ {*} = y ^ {*} ] = \prod_ {t = 1} ^ {n} \mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {t}} [ g _ {t} (y ^ {*}, X _ {t}) ], \tag {19}
$$
where  $g_{t}(y^{*},x_{t}) = \prod_{j = 1}^{\infty}W_{j}(y_{j}\mid x_{t},y^{j - 1})\mathbb{1}\left\{\sigma_{j}(y^{j - 1}) = t\right\}$
Hence, when  $X^n\sim \mathbf{p}_z^{\otimes n}$  we have
$$
\mathbf {p} _ {z} ^ {y ^ {*}} := \Pr_ {X ^ {n} \sim \mathbf {p} _ {z} ^ {\otimes n}} [ Y ^ {*} = y ^ {*} ] = \prod_ {t = 1} ^ {n} \mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} [ g _ {t} (y ^ {*}, X _ {t}) ].
$$
Here we can define a similar notion of "channel" for a communication protocol  $\Pi$  for the  $i$ th player when the underlying distribution is  $\mathbf{p}_z$  by setting
$$
\tilde {W} _ {t, \mathbf {p} _ {z}} \left(y ^ {*} \mid x\right) = g _ {t} \left(y ^ {*}, x\right) \left(\prod_ {j \neq t} \mathbb {E} _ {X _ {j} \sim \mathbf {p} _ {z}} \left[ g _ {j} \left(y ^ {*}, X _ {j}\right) \right]\right). \tag {20}
$$
Then we have, for all  $t\in [n]$
$$
\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} \left[ \tilde {W} _ {t, \mathbf {p} _ {z}} (y ^ {*} \mid X _ {t}) \right] = \Pr_ {X ^ {n} \sim \mathbf {p} _ {z} ^ {\otimes n}} [ Y ^ {*} = y ^ {*} ].
$$
We proceed to prove a bound similar to Theorem 1 in terms of the "channel" defined in Eq. (20), as stated below.
Theorem 5 (Information contraction bound). Fix  $\tau \in (0,1/2]$ . Let  $\Pi$  be a fully interactive protocol using  $\mathcal{W}^n$ , and let  $Z$  be a random variable on  $\mathcal{Z}$  with distribution  $\mathrm{Rad}(\tau)^{\otimes k}$ . Let  $(Y^{*},U)$  be the transcript of  $\Pi$  when the input  $X_{1},\ldots,X_{n}$  is i.i.d. with common distribution  $\mathbf{p}_{Z}$ . Then, under Assumption 1,
$$
\begin{array}{l} \left(\frac {1}{k} \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {T V}} \Big (\mathbf {p} _ {+ i} ^ {Y ^ {*}}, \mathbf {p} _ {- i} ^ {Y ^ {*}} \Big)\right) ^ {2} \\ \leq \frac {7}{k} \alpha^ {2} \sum_ {j = 1} ^ {n} \max _ {z \in \mathcal {Z}} \max _ {(\mathcal {W} _ {t}, j _ {t}) _ {t \in \mathbb {N}} \in \mathcal {W} ^ {n}} \sum_ {i = 1} ^ {k} \int_ {y ^ {*} \in \mathcal {Y} ^ {*}} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \phi_ {z , i} (X) \tilde {W} _ {j , \mathbf {p} _ {z}} (y ^ {*} \mid X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \tilde {W} _ {j , \mathbf {p} _ {z}} (y ^ {*} \mid X) \right]} d \mu , \\ \end{array}
$$
where  $\mathbf{p}_{+i}^{Y^*} := \mathbb{E}\left[\mathbf{p}_Z^{Y^*} \mid Z_i = 1\right]$ ,  $\mathbf{p}_{-i}^{Y^*} := \mathbb{E}\left[\mathbf{p}_Z^{Y^*} \mid Z_i = 1\right]$ .
We can see the bound is in identical form to Theorem 1 except that we replace each player's channel with the  $\tilde{W}_{j,\mathbf{P}_z}(y^*\mid X)$  we defined. Other similar bounds in Section 3 can also be derived under additional assumptions and specific constraints. We present the proof for Theorem 5 below and omit the detailed statements and proof for other bounds.
Proof. Analogously to Eq. (36), we can get
$$
\frac {1}{k} \left(\sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {T V}} \left(\mathbf {p} _ {+ i} ^ {Y ^ {*}}, \mathbf {p} _ {- i} ^ {Y ^ {*}}\right)\right) ^ {2} \leq 1 4 \sum_ {t = 1} ^ {n} \mathbb {E} _ {Z} \left[ \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {Z} ^ {Y ^ {*}}, \mathbf {p} _ {t \leftarrow Z ^ {\oplus} i} ^ {Y ^ {*}}\right) ^ {2} \right] \tag {21}
$$
For all  $z \in \{-1, +1\}^k$  and  $i, t$ , by the definition of Hellinger distance and Eq. (19), we have
$$
\begin{array}{l} 2\mathrm{d}_{\mathrm{H}}\Bigl(\mathbf{p}_{z}^{Y^{*}},\mathbf{p}_{t\leftarrow z^{\oplus i}}^{Y^{*}}\Bigr)^{2} = \int_{y^{*}\in \mathcal{Y}^{*}}\prod_{\substack{1\leq j\leq n\\ j\neq t}}\mathbb{E}_{X_{j}\sim \mathbf{p}_{z}}[g_{j}(y^{*},X_{j})]\Bigl( \sqrt{\mathbb{E}_{X_{t}\sim\mathbf{p}_{z}\oplus i}[g_{t}(y^{*},X_{t})]} -\sqrt{\mathbb{E}_{X_{t}\sim\mathbf{p}_{z}}[g_{t}(y^{*},X_{t})]}\Bigr)^{2}\mathrm{d}\mu \\ \leq \int_ {y ^ {*} \in \mathcal {Y} ^ {*}} \left(\prod_ {j \neq t} \mathbb {E} _ {X _ {j} \sim \mathbf {p} _ {z}} [ g _ {j} (y ^ {*}, X _ {j}) ]\right) \left(\frac {(\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} [ g _ {t} (y ^ {*} , X _ {t}) ] - \mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z} \oplus i} [ g _ {t} (y ^ {*} , X _ {t}) ]) ^ {2}}{\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} [ g _ {t} (y ^ {*} , X _ {t}) ]}\right) d \mu , \\ \end{array}
$$
Proceeding from above, we get under Assumption 1,
$$
\begin{array}{l} 2 \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {Y ^ {*}}, \mathbf {p} _ {t \leftarrow z ^ {\oplus i}} ^ {Y ^ {*}}\right) ^ {2} \leq \alpha^ {2} \int_ {y ^ {*} \in \mathcal {Y} ^ {*}} \left(\prod_ {j \neq t} \mathbb {E} _ {X _ {j} \sim \mathbf {p} _ {z}} \left[ g _ {j} \left(y ^ {*}, X _ {j}\right) \right]\right) \left(\frac {\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} \left[ \phi_ {z , i} \left(X _ {t}\right) g _ {t} \left(y ^ {*} , X _ {t}\right) \right] ^ {2}}{\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} \left[ g _ {t} \left(y ^ {*} , X _ {t}\right) \right]}\right) \mathrm {d} \mu \\ = \alpha^ {2} \int_ {y ^ {*} \in \mathcal {Y} ^ {*}} \frac {\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} \left[ \phi_ {z , i} (X _ {t}) g _ {t} (y ^ {*} , X _ {t}) \prod_ {j \neq t} \mathbb {E} _ {X _ {j} \sim \mathbf {p} _ {z}} [ g _ {j} (y ^ {*} , X _ {j}) ] \right] ^ {2}}{\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} \left[ g _ {t} (y ^ {*} , X _ {t}) \prod_ {j \neq t} \mathbb {E} _ {X _ {j} \sim \mathbf {p} _ {z}} [ g _ {j} (y ^ {*} , X _ {j}) ] \right]} d \mu \\ = \alpha^ {2} \int_ {y ^ {*} \in \mathcal {Y} ^ {*}} \frac {\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} \left[ \phi_ {z , i} \left(X _ {t}\right) \tilde {W} _ {t , \mathbf {p} _ {z}} \left(y ^ {*} \mid X\right) \right] ^ {2}}{\mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z}} \left[ \tilde {W} _ {t , \mathbf {p} _ {z}} \left(y ^ {*} \mid X\right) \right]} d \mu . \\ \end{array}
$$
Plugging the above bound into Eq. (21), we can obtain the bound in Theorem 5 by taking the maximum over all  $z \in \{-1, +1\}^k$  and all possible channel sequences.
B A measure change bound
We here provide a variant of Talagrand's transportation-cost inequality which is used in deriving Eq. (5) (under Assumption 3) in the second part of Theorem 2. We note that this type of result is not novel, and can be derived from standard arguments in the literature (see, e.g., [9, Chapter 8] or [27, Chapter 4]). However, the lemma below is specifically tailored for our purposes, and we provide the proof for completeness. A similar bound was derived in [2], where Gaussian mean testing under communication constraints was considered.
Lemma 3 (A measure change bound). Consider a random variable  $X$  taking values in  $\mathcal{X}$  and with distribution  $P$ . Let  $\Phi \colon \mathcal{X} \to \mathbb{R}^k$  be such that the random vector  $\Phi(X)$  is  $\sigma^2$ -subgaussian. Then, for any function  $a \colon \mathcal{X} \to [0, \infty)$  such that  $\mathbb{E}[a(X)] < \infty$ , we have
$$
\frac {\| \mathbb {E} [ \Phi (X) a (X) ] \| _ {2} ^ {2}}{\mathbb {E} [ a (X) ] ^ {2}} \leq 2 \sigma^ {2} \frac {\mathbb {E} [ a (X) \ln a (X) ]}{\mathbb {E} [ a (X) ]} + 2 \sigma^ {2} \ln \frac {1}{\mathbb {E} [ a (X) ]}.
$$
Proof. By an application of Gibb's variational principle (cf. [9, Corollary 4.14]) the following holds: For a random variable  $Z$  and distributions  $P$  and  $Q$  on the underlying probability space satisfying  $Q \ll P$  (that is, such that  $Q$  is absolutely continuous with respect to  $P$ ), we have
$$
\lambda \mathbb {E} _ {Q} [ Z ] \leq \ln \mathbb {E} _ {P} \left[ e ^ {\lambda Z} \right] + \mathrm {D} (Q \| P).
$$
To apply this bound, set  $P$  to be the distribution of  $X$  and let  $Q \ll P$  be defined using its density (Radon-Nikodym derivative) with respect to  $P$  given by
$$
\frac {\mathrm {d} Q}{\mathrm {d} P} = \frac {a (X)}{\mathbb {E} _ {P} [ a (X) ]}.
$$
Now, note that for any unit vector  $v$ , we have, setting  $Z = v^{\mathsf{T}}\Phi(X)$  and using the  $\sigma^2$ -subgaussianity of  $\Phi(X)$ , that
$$
\lambda \mathbb {E} _ {Q} \left[ v ^ {\intercal} \Phi (X) \right] \leq \ln \mathbb {E} _ {P} \left[ e ^ {\lambda v ^ {\intercal} \Phi (X)} \right] + \mathrm {D} (Q \| P) \leq \frac {\sigma^ {2} \lambda^ {2}}{2} + \mathrm {D} (Q \| P).
$$
In particular, for  $\lambda = \frac{1}{\sigma}\sqrt{2\mathrm{D}(Q\|P)}$  we get
$$
\mathbb {E} _ {Q} [ v ^ {\intercal} \Phi (X) ] \leq \sigma \sqrt {2 \mathrm {D} (Q \| P)}.
$$
Applying this to the unit vector  $v := \frac{\mathbb{E}_Q[\Phi(X)]}{\|\mathbb{E}_Q[\Phi(X)]\|_2}$  then yields
$$
\left\| \mathbb {E} _ {Q} [ \Phi (X) ] \right\| _ {2} \leq \sigma \sqrt {2 \mathrm {D} (Q \| P)}.
$$
To conclude, it then suffices to observe that
$$
\mathrm {D} (Q \| P) = \frac {\mathbb {E} _ {P} [ a (X) \ln a (X) ]}{\mathbb {E} _ {P} [ a (X) ]} + \ln \frac {1}{\mathbb {E} _ {P} [ a (X) ]}.
$$
The proof is completed by combining the bounds above, as  $\mathbb{E}_Q[\Phi (X)] = \frac{\mathbb{E}_P[\Phi(X)a(X))]}{\mathbb{E}_P[a(X)]}$ .
C Upper bounds
We now describe and analyze the interactive algorithms for the estimation tasks we consider.
C.1 Product Bernoulli Distributions
Recall that  $\mathcal{B}_{d,s}$ , the family of  $d$ -dimensional  $s$ -sparse product Bernoulli distributions, is defined as
$$
\mathcal {B} _ {d, s} := \left\{\bigotimes_ {j = 1} ^ {d} \operatorname {R a d} \left(\frac {1}{2} \left(\mu_ {j} + 1\right)\right): \mu \in [ - 1, 1 ] ^ {d}, \| \mu \| _ {0} \leq s \right\}. \tag {22}
$$
We now provide the interactive protocols achieving the upper bounds of Theorem 3 for sparse product Bernoulli mean estimation under LDP and communication constraints.
Our protocols has two ingredients described below:
- Estimating non-zero mean coordinates. In this step we will start with  $S_0 = [d]$ , the set of all possible coordinates. Then we will iteratively prune the set  $S_0 \to S_1 \to \ldots \to S_T$ , such that  $|S_T| = 3s$  (this step is skipped if  $s \geq d/3$ ) is a good estimate for the set of coordinates with non-zero mean.
- Estimating the non-zero means. We then estimate the means of the coordinates in  $S_T$ , which is equivalent to solving a dense mean estimation problem in 3s dimensions.
In the next two sections, we provide the details of the algorithm that matches the lower bounds obtained in Section 5 for interactive protocols under LDP and communication constraints respectively.
C.1.1 LDP constraints
In this subsection, we will focus on the case  $\varepsilon \in (0,1]$  (high-privacy regime). For the case  $\varepsilon > 1$ , we rely a privatization of the communication-limited algorithm, which will be discussed at the end of Appendix C.1.2. Our protocol for Bernoulli mean estimation under LDP constraints is described in Algorithm 1. As stated above, in each round  $t = 1,\dots,T$ , for each  $j \in S_{t - 1}$  a new group of players apply the well known binary Randomized Response (RR) mechanism [29, 24] to their  $j$ th coordinate. Using these messages we then guess a set of coordinates with highest possible means (in absolute value) and prune the set to  $S_{t}$ . This is done in Lines 2-6 of Algorithm 1.
In Lines 7-12, the algorithm uses the same approach to estimate the means of coordinates within  $S_T$  and sets remaining coordinates to zero.
The privacy guarantee follows immediately from that of the RR mechanism, and further, this only requires one bit of communication per player.
Algorithm 1 LDP protocol for mean estimation for the product of Bernoulli family
Require:  $n$  players, dimension  $d$ , sparsity parameter  $s$ , privacy parameter  $\varepsilon$ .
1: Set  $T \coloneqq \log_3\frac{d}{3s}$ ,  $\alpha \coloneqq \frac{e^{\varepsilon}}{1 + e^{\varepsilon}}$ ,  $S_0 = [d]$ ,  $N_0 \coloneqq \frac{n}{6d}$ .
2: for  $t = 1,2,\ldots ,T$  do
3: for  $j\in S_{t - 1}$  do
4: Get a group of new players  $G_{t,j}$  of size  $N_{t} = N_{0} \cdot 2^{t}$ .
5: Player  $i \in G_{t,j}$ , upon observing  $X_i \in \{-1, +1\}^d$  sends the message  $Y_i \in \{-1, +1\}$  such that
$$
Y _ {i} = \left\{ \begin{array}{l l} \left(X _ {i}\right) _ {j} & \text {w . p .} \alpha , \\ - \left(X _ {i}\right) _ {j} & \text {w . p .} 1 - \alpha . \end{array} \right. \tag {23}
$$
6: Set  $M_{t,j} \coloneqq \sum_{i \in G_{t,j}} Y_i$ . Let  $S_t \subseteq S_{t-1}$  be the set of the  $|S_{t-1}|/3$  indices with the largest  $|M_{t,j}|$ .
7: for  $j\in S_T$  do
8: Get a group of new players  $G_{T,j}, j \in S_T$  of size  $N_{T+1} = N_0 \cdot 2^T$ .
9: Player  $i \in G_{T,j}$ , sends the message  $Y_i \in \{-1, +1\}$  according to Eq. (23) and  $M_{T,j} := \sum_{i \in G_{T,j}} Y_i$ .
10: for  $j\in [d]$  do
$$
\widehat {\mu} _ {j} = \left\{ \begin{array}{l l} \frac {M _ {j , T}}{(2 \alpha - 1) N _ {T + 1}} & \text {i f} j \in S _ {T}, \\ 0 & \text {o t h e r w i s e}. \end{array} \right.
$$
12: return  $\widehat{\mu}$
The performance guarantee of Algorithm 1 is stated below, which matches the lower bounds obtained in Section 5.
Proposition 1. Fix  $p \in [1, \infty]$ . For  $n \geq 1$  and  $\varepsilon \in (0, 1]$ , Algorithm 1 is an  $(n, \gamma)$ -estimator using  $\mathcal{W}_{\varepsilon}$  under  $\ell_p$  loss for  $\mathcal{B}_{d,s}$  with  $\gamma = O\left(\sqrt{\frac{pds^2/p}{n\varepsilon^2}}\right)$  for  $p \leq 2\log s$  and  $\gamma = O\left(\sqrt{\frac{d\log s}{n\varepsilon^2}}\right)$  for  $p > 2\log s$ .
Proof. The total number of players used by Algorithm 1 uses is
$$
\sum_ {t = 1} ^ {T + 1} \left| S _ {t - 1} \right| \cdot N _ {t} = \left| S _ {0} \right| \cdot N _ {0} \cdot \sum_ {t = 1} ^ {T + 1} \frac {2 ^ {t}}{3 ^ {t - 1}} \leq 6 \left| S _ {0} \right| \cdot N _ {0} = n.
$$
To prove the utility guarantee, we bound the estimation error in the estimated set  $S_{T}$  and the error outside the set  $S_{T}$  in the following lemma.
Lemma 4. Let  $S_T$  be the subset obtained from the first stage of Algorithm 1. Then,
$$
\max  \left\{\mathbb {E} \left[ \sum_ {j \notin S _ {T}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right], \mathbb {E} \left[ \sum_ {j \in S _ {T}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right] \right\} = O \left(s \left(\frac {p d}{n \varepsilon^ {2}}\right) ^ {p / 2}\right).
$$
The proposition follows directly from the lemma. Indeed, for  $p > 2\log s$ , by monotonicity of  $\ell_p$  norms we have  $\| \mu - \hat{\mu} \|_p \leq \| \mu - \hat{\mu} \|_{p'}$  for all  $p' \leq p$ , and thus choosing  $p' := 2\log s$  is sufficient to obtain the stated bound.
Proof of Lemma 4. We prove the bound on each term individually. The first term captures the performance of our estimator within coordinates in  $S_{T}$  and the second term states that we do not "prune" too many coordinates with high non-zero means.
Bounding the first term. For  $j \notin S_T$ , we output  $\widehat{\mu}_j = 0$ . Therefore,
$$
\mathbb {E} \left[ \sum_ {j \notin S _ {T}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right] = \sum_ {j} \mathbb {E} [ | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \cdot \mathbb {1} \{j \notin S _ {T} \} ] = \sum_ {j} | \mu_ {j} | ^ {p} \cdot \Pr [ j \notin S _ {T} ].
$$
Since  $\mu$  is  $s$ -sparse, it will suffice to show that for all  $j$  with  $|\mu_j| > 0$
$$
\left| \mu_ {j} \right| ^ {p} \cdot \Pr [ j \notin S _ {T} ] = O \left(\left(\frac {p d}{n \varepsilon^ {2}}\right) ^ {p / 2}\right). \tag {24}
$$
$$
H := 2 0 \sqrt {\frac {d}{n (2 \alpha - 1) ^ {2}}}.
$$
Note that for  $\varepsilon \in (0,1]$ , we have  $2\alpha - 1 \geq \frac{e - 1}{e + 1}\varepsilon$ . Therefore, if  $|\mu_j| \leq H$ , then Eq. (24) holds since  $\operatorname*{Pr}[j \notin S] \leq 1$ . We hereafter assume  $|\mu_j| > H$ , and let  $\mu_j = \beta_jH$  with  $\beta_j > 1$ . Let  $E_{t,j}$  be the event that coordinate  $j$  is removed in round  $t$  given that  $j \in S_{t-1}$ . Then we have
$$
\Pr [ j \notin S _ {T} ] \leq \sum_ {t = 1} ^ {T} \Pr [ E _ {t, j} ].
$$
We proceed to bound each  $\operatorname*{Pr}[E_{t,j}]$  separately. Note that for  $i\in G_{t,j}$ ,  $Y_{i}\in \{-1, + 1\}$  and by Eq. (23)
$$
\mathbb {E} \left[ Y _ {i} \right] = (2 \alpha - 1) \cdot \mu_ {j} = (2 \alpha - 1) \beta_ {j} H. \tag {25}
$$
Let  $a_{t,j}$  be the number of coordinates  $j'$  with  $\mu_{j'} = 0$  and  $|M_{t,j'}| \geq \frac{1}{2} N_t(2\alpha - 1)\beta_jH$ . Since we select the  $|S_{t-1}|/3$  coordinates with the largest magnitude of the sum, for  $j \notin S_t$  to happen at least one of the following must occur: (i)  $a_{t,j} > \frac{1}{3} |S_{t-1}| - s$ , or (ii)  $M_{t,j} < \frac{1}{2} N_t(2\alpha - 1)\beta_jH$ .
By Hoeffding's inequality, we have
$$
\Pr \left[ M _ {t, j} <   \frac {1}{2} N _ {t} (2 \alpha - 1) \beta_ {j} H \right] \leq \exp \left(- \frac {1}{8} N _ {t} ((2 \alpha - 1) \beta_ {j} H) ^ {2}\right) <   \exp \left(- 5 \cdot 2 ^ {t} \beta_ {j} ^ {2}\right).
$$
Let  $p_{t,j} \coloneqq e^{-5\cdot 2^t\beta_j^2}$ . Similarly, for any  $j'$  such that  $\mu_{j'} = 0$ ,
$$
\Pr \left[ | M _ {t, j ^ {\prime}} | \geq \frac {1}{2} N _ {t} (2 \alpha - 1) \beta_ {j} H \right] \leq 2 p _ {t, j}.
$$
Since all coordinates are independent,  $a_{t,j}$  is binomially distributed with mean at most  $2p_{t,j}|S_{t-1}|$ . By Markov's inequality, we get
$$
\Pr \left[ a _ {t, j} > \frac {1}{3} | S _ {t - 1} | - s \right] \leq \frac {\mathbb {E} [ a _ {t , j} ]}{| S _ {t - 1} | / 3 - s} \leq p _ {t, j},
$$
recalling that  $|S_{t - 1}| = d3^{t - 1} \geq 9s$ . By a union bound and summing over  $t \in [T]$ , we get
$$
\operatorname * {P r} [ j \notin S _ {T} ] \leq \sum_ {t = 1} ^ {T} \operatorname * {P r} [ E _ {t, j} ] \leq \sum_ {t = 1} ^ {T} 3 p _ {t, j} = 3 \sum_ {t = 1} ^ {T} \exp \left(- 2 ^ {t} \cdot 5 \beta_ {j} ^ {2}\right) \leq 6 \exp \left(- 5 \beta_ {j} ^ {2}\right).
$$
Not that for  $x > 0$ ,  $x^{p}e^{-x^{2}} \leq \left(\frac{p}{2e}\right)^{p/2}$ . Hence
$$
| \mu_ {j} | ^ {p} \cdot \operatorname * {P r} [ j \notin S _ {T} ] \leq 6 H ^ {p} \beta_ {j} ^ {p} e ^ {- 5 \beta_ {j} ^ {2}} \leq \left(C \frac {p d}{n \varepsilon^ {2}}\right) ^ {p / 2},
$$
for some absolute constant  $C > 0$ , completing the proof.
Bounding the second term. Note that  $S_T$  is a random variable itself. We show that the bound holds for any realization of  $S_T$ . We need the following result which follows from standard moment bounds on binomial distributions.
Fact 1. Let  $p \geq 1$ ,  $m \in \mathbb{N}$ ,  $0 \leq q \leq 1$ , and  $N \sim \mathrm{Bin}(m, q)$ . Then,  $\mathbb{E}[|N - mq|^p] \leq 2^{-p/2} m^{p/2} p^{p/2}$ .
Applying this with  $m = N_T \geq \frac{n}{6d}$ , the transformation from Bernoulli to  $\{-1, +1\}$ , and the scaling by  $2\alpha - 1$ , yields for  $j \in S_T$ , and using Eq. (25)
$$
\mathbb {E} \left[ | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right] \leq \left(\frac {p}{(n / 6 d) (2 \alpha - 1) ^ {2}}\right) ^ {p / 2}.
$$
Upon summing over  $j \in S_T$ , we obtain
$$
\mathbb {E} \left[ \sum_ {j \in S _ {T}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right] \leq 3 s \cdot \left(\frac {6 (e + 1) ^ {2} d}{(e - 1) ^ {2} n \varepsilon^ {2}}\right) ^ {p / 2} \leq 3 \cdot 6 ^ {p} \cdot s \left(\frac {p d}{n \varepsilon^ {2}}\right) ^ {p / 2}.
$$
C.1.2 Communication constraints
In Algorithm 2 we propose a protocol to estimate the mean of product Bernoulli distributions under  $\ell$ -bit communication constraints. As mentioned in the previous subsection, the  $\varepsilon$ -LDP algorithm with  $\varepsilon > 1$  will follow from a simple modification of the communication-constrained one; we discuss how to privatize the latter to obtain the former at the end of the section. As in the LDP case when  $\varepsilon \in (0,1]$ , in 2-10 the algorithm iteratively prunes an initial set  $S_0 = [d]$  to obtain a set  $S_T$  of size  $\max \{3s,\ell\}$ , which denotes the set of potential non-zero coordinates. We then estimate the mean of coordinates in  $S_T$ . If  $\ell > 3s$ , then we can directly send the values of all coordinates in  $S_T$  and use it for estimation; otherwise, when  $3s > \ell$ , we again partition  $S_T$  into sets of size  $\ell$  and each player sends the bits of its sample in this set. This is done in Lines 11-18. We state the performance of Algorithm 2 below.