SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 2. Fix  $p \in [1, \infty]$ . For  $n \geq 1$  and  $\ell \leq d$ , we have Algorithm 2 is an  $(n, \gamma)$ -estimator using  $\mathcal{W}_{\ell}$  under  $\ell_p$  loss for  $\mathcal{B}_{d,s}$  with  $\gamma = O\left(\sqrt{\frac{pds^2/p}{n\ell} + \frac{(p + \log(2\ell/s))s^{2/p}}{n}}\right)$  for  $p \leq 2\log s$  and  $\gamma = O\left(\sqrt{\frac{d\log s}{n\ell} + \frac{\log\ell}{n}}\right)$  for  $p > 2\log s$ .
When  $\ell \leq 3s$ , the bound we get is  $\gamma \lesssim \sqrt{\frac{pds^{2/p}}{n\ell}}$ . The analysis is almost identical to the case under LDP constraints, since in both cases, the information we get about coordinate  $j$  are samples from a Rademacher distribution with mean  $(2\alpha - 1)\mu_j$ . There are only two differences. (i)  $\alpha = 1$  instead of
Algorithm 2  $\ell$ -bit protocol for estimating product of Bernoulli family
Require:  $n$  players, dimension  $d$ , sparsity parameter  $s$ , communication bound  $\ell$ .  
1: Set  $T := \log_3(d / \max\{3s, \ell\})$ ,  $S_0 := [d]$ ,  $N_0 := \frac{n\ell}{18d}$ .  
2: for  $t = 1, 2, \ldots, T$  do  
3: Set  $P := \frac{d}{3^{t-1}\ell}$ , and partition  $S_{t-1}$  into  $P$  subsets  $S_{t-1,1}, \ldots, S_{t-1,P}$ , each of size  $\ell$ .  
4: for  $j = 1, 2, \ldots, P$  do  
5: Get a group of new players  $G_{t,j}$  of size  $N_t = N_0 \cdot 2^t$ .  
6: Player  $i \in G_{t,j}$ , upon observing  $X_i \in \{-1, +1\}^d$  sends the message  $Y_i = \{(X_i)_x\}_{x \in S_{t-1,j}}$ .  
7: For  $x \in S_{t-1,j}$ , let  $M_{t,x} := \sum_{i \in G_{t,j}} (X_i)_x$ .  
8: Set  $S_t \subseteq S_{t-1}$  to be the set of indices with the largest  $|M_{t,x}|$  and  $|S_t| = |S_{t-1}|/3$ .  
9: if  $\ell \leq 3s$  then  
10: Partition  $S_T$  into  $3s/\ell$  subsets of size  $\ell$  each,  $S_{T,j}, j \in [3s/\ell]$ .  
11: for  $j = 1, \ldots, 3s/\ell$  do  
12: Get a new group  $G_{T+1,j}$  of players of size  $n\ell/(6s)$ .  
13: Player  $i \in G_{T+1,j}$ , sends the message  $Y_i = \{(X_i)_x\}_{x \in S_{T,j}}$ .  
14: For  $x \in S_{T,j}$ , let  $M_{T+1,x} = \sum_{i \in G_{T+1,j}} (X_i)_x$ . Set  $\widehat{\mu}_x := \frac{6s}{n\ell} M_{T+1,x}$ .  
15: For  $x \notin S_T$ , set  $\widehat{\mu}_x = 0$ .  
16: if  $\ell > 3s$  then,  
17: Get  $n/2$  new players  $G_{T+1}$  and for  $i \in G_{T+1}$ , player  $i$  sends  $Y_i = \{(X_i)_x\}_{x \in S_T} \triangleright$  This can be done since  $|S_T| = \ell$  if  $\ell > 3s$ .  
18: For  $x \in S_T$ , let  $M_{T+1,x} = \sum_{i \in G_{T+1,j}} (X_i)_x$ . Set  $S_{T+1} \subseteq S_T$  to be the set of indices with the largest  $|M_{T+1,x}|$  and  $|S_{T+1}| = 3s$ . For all  $x \in S_{T+1}$ , set  $\widehat{\mu}_x := \frac{2}{n} M_{T+1,x}$ ,  
    and for all  $x \notin S_{T+1}, \widehat{\mu}_x = 0$ .  
19: return  $\widehat{\mu}$ .
$\Theta \big(\varepsilon^2\big)$ . (ii) There is a factor of  $\ell$  more players in the corresponding groups. Combining both factors, we can obtain the desired bound by replacing  $\varepsilon^2$  by  $\ell$ . We omit the detailed proof in this case.
When  $\ell > 3s$ , after  $T \asymp \log(d / \ell)$  rounds, we can find a subset  $S_T$  of size  $\ell$  which contains most of the coordinates with large biases. The protocol then asks new players to send all coordinates within  $S_T$  using  $\ell$  bits. In this case, it would be enough to prove Lemma 5 since for the coordinates outside  $S_T$ , we can show the error is small following exactly the same steps as the proof for bounding the first term in Lemma 4 as we explained in the case when  $\ell \leq 3s$ .
Lemma 5. Let  $S_T$  be the subset obtained from the first stage of Algorithm 2, we have
$$
\mathbb {E} \left[ \sum_ {j \in S _ {T}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right] = O \left(s \left(\frac {p + \log \frac {2 \ell}{s}}{n}\right) ^ {p / 2}\right).
$$
Proof. Similar to Lemma 4, we will prove that the statement is true for any realization of  $S_T$ , which is a stronger statement than the claim.
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {j \in S _ {T}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right] = \mathbb {E} \left[ \sum_ {j \in S _ {T}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \mathbb {1} \{j \in S _ {T + 1} \} \right] + \mathbb {E} \left[ \sum_ {j \in S _ {T}} | \mu_ {j} | ^ {p} \mathbb {1} \{j \notin S _ {T + 1} \} \right] \\ \leq \mathbb {E} \left[ \sum_ {j \in S _ {T + 1}} | \mu_ {j} - \widehat {\mu} _ {j} | ^ {p} \right] + \sum_ {j \in S _ {T}} | \mu_ {j} | ^ {p} \Pr [ j \notin S _ {T + 1} ]. \\ \end{array}
$$
Fix  $S_{T+1}$ . For each  $j \in S_{T+1}$ ,  $M_{T+1,j}$  is binomially distributed with mean  $\mu_j$  and  $n/2$  trials. By similar computations as Lemma 4, we have
$$
\mathbb {E} \left[ \sum_ {j \in S _ {T + 1}} \left| \mu_ {j} - \widehat {\mu} _ {j} \right| ^ {p} \right] = O \left(s \left(\frac {p}{n}\right) ^ {p / 2}\right). \tag {26}
$$
Next we show for all  $j \in S_T$  such that  $\mu_j \neq 0$ ,
$$
\left| \mu_ {j} \right| ^ {p} \Pr [ j \notin S _ {T + 1} ] \leq 2 \left(\frac {p \vee 6 4 \ln \frac {2 \ell}{s}}{n}\right) ^ {p / 2}. \tag {27}
$$
If  $|\mu_j| \leq H' := 8\sqrt{\frac{\ln 2\ell}{n}}$ , Eq. (27) always holds since  $\operatorname{Pr}[j \notin S] \leq 1$ . Hence we hereafter assume that  $|\mu_j| > H'$ , and write  $\mu_j = \beta_j H'$  for some  $\beta_j > 1$ .
Let  $a_{T+1,j}$  be the number of coordinates  $j'$  with  $\mu_{j'} = 0$  and  $|M_{T+1,j'}| \geq \frac{n}{2} \cdot \frac{\beta_j H'}{2}$ . Then since  $S_{T+1}$  contains the top  $3s$  coordinates with the largest magnitude of the sum, we have  $j \notin S_{T+1}$  happens only if at least one of the following occurs (i)  $a_{T+1,j} > 2s$ , or (ii)  $M_{T+1,j} < \frac{n}{2} \cdot \frac{\beta_j H'}{2}$ .
ByHoeffding'sinequality,wehave
$$
\operatorname * {P r} \left[ M _ {T + 1, j} <   \frac {n}{2} \cdot \frac {\beta_ {j} H ^ {\prime}}{2} \right] \leq \exp \left(- \frac {1}{2} \cdot \frac {n}{2} \cdot \left(\frac {\beta_ {j} H ^ {\prime}}{2}\right) ^ {2}\right) = \left(\frac {2 \ell}{s}\right) ^ {- 4 \beta_ {j} ^ {2}} := p _ {T + 1, j}.
$$
Similarly, for any  $j'$  such that  $\mu_{j'} = 0$ ,
$$
\Pr \left[ | M _ {T + 1, j ^ {\prime}} | \geq \frac {n}{2} \cdot \frac {\beta_ {j} H ^ {\prime}}{2} \right] \leq 2 p _ {T + 1, j}.
$$
Since all coordinates are independent,  $a_{T + 1,j}$  is binomially distributed with mean at most  $2p_{T + 1,j}\ell$ , and therefore, by Markov's inequality,
$$
\Pr \left[ a _ {T + 1, j} > 2 s \right] \leq \frac {2 p _ {T + 1 , j} \ell}{2 s} \leq \left(\frac {2 \ell}{s}\right) ^ {1 - 4 \beta_ {j} ^ {2}} \leq \left(\frac {2 \ell}{s}\right) ^ {- 3 \beta_ {j} ^ {2}}
$$
the last step since  $\beta_{j} > 1$ . By a union bound, we have
$$
\operatorname * {P r} [ j \notin S _ {T} ] \leq \operatorname * {P r} [ a _ {T + 1, j} > 2 s ] + \operatorname * {P r} \left[ M _ {T + 1, j} <   \frac {1}{4} \frac {n}{2} \cdot \frac {\beta_ {j} H ^ {\prime}}{2} \right] \leq 2 \left(\frac {2 \ell}{s}\right) ^ {- 3 \beta_ {j} ^ {2}}.
$$
Using the inequality  $x^p a^{-x^2} \leq \left( \frac{p}{2e\ln a} \right)^{p/2}$  which holds for all  $x > 0$ , we get overall
$$
| \mu_ {j} | ^ {p} \cdot \Pr [ j \notin S _ {T} ] \leq 2 H ^ {\prime p} \beta_ {j} ^ {p} \left(\frac {2 \ell}{s}\right) ^ {- 4 \beta_ {j} ^ {2}} \leq 2 \left(\frac {p}{e n}\right) ^ {p / 2},
$$
establishing Eq. (27). Combining Eq. (26) and Eq. (27) concludes the proof Lemma 5 since there are at most  $s$  unbiased coordinates.
Algorithm under LDP with  $\varepsilon > 1$  To get a  $\varepsilon$ -LDP algorithm in the regime  $\varepsilon > 1$  (low-privacy regime), we perform the following changes to obtain a private algorithm from Algorithm 2:
- Each user independently flips each coordinate of their local sample to get  $Z_{i}$  where, for all  $x \in [d]$ ,  $(Z_{i})_{x} = (X_{i})_{x}$  with probability  $\frac{e}{e + 1}$  and  $(Z_{i})_{x} = 1 - (X_{i})_{x}$  with probability  $\frac{1}{e + 1}$  (note that this corresponds to applying Randomized Response independently to each bit with privacy parameter 1).
- Users then follow Algorithm 2 with the setting  $\ell = \lfloor \varepsilon \rfloor$  and local data  $\{Z_i\}_{i \in [n]}$ , and obtain estimate  $\widehat{\mu}$ .
The final estimate is then  $\frac{e + 1}{e - 1}\widehat{\mu}$
The privacy guarantee of the algorithm comes from the fact that Algorithm 2 sends at most  $\ell = \lfloor \varepsilon \rfloor$  coordinates of each  $Z_{i}$ , and for any  $S$  with  $|S|\leq \lfloor \varepsilon \rfloor$
$$
\frac {\operatorname * {P r} [ \left\{(Z _ {i}) _ {x} \right\} _ {x \in S} \mid X _ {i} ]}{\operatorname * {P r} [ \left\{(Z _ {i}) _ {x} \right\} _ {x \in S} \mid X _ {i} ^ {\prime} ]} = \prod_ {x \in S} \frac {\operatorname * {P r} [ \left(Z _ {i}\right) _ {x} \mid (X _ {i}) _ {x} ]}{\operatorname * {P r} [ \left(Z _ {i}\right) _ {x} \mid (X _ {i} ^ {\prime}) _ {x} ]} \leq e ^ {\lfloor \varepsilon \rfloor}.
$$
The utility guarantee follows from observing that  $\mu_{Z} = \frac{e - 1}{e + 1}\mu$  and hence any  $\ell_p$  error guarantee will be preserved up to a constant.
C.2 Gaussian Mean Estimation
Recall that  $\mathcal{G}_{d,s}$  denotes the family of  $d$ -dimensional spherical Gaussian distributions with  $s$ -sparse mean in  $[-1,1]^d$ , i.e.,
$$
\mathcal {G} _ {d, s} = \left\{\mathcal {G} (\mu , \mathbb {I}): \| \mu \| _ {\infty} \leq 1, \| \mu \| _ {0} \leq s \right\}. \tag {28}
$$
We will prove the following results for LDP and communication constraints, respectively.
Proposition 3. Fix  $p \in [1, \infty]$ . For  $n \geq 1$  and  $\varepsilon \in (0, 1]$ , there exists an  $(n, \gamma)$ -estimator using  $\mathcal{W}_{\varepsilon}$  under  $\ell_p$  loss for  $\mathcal{G}_{d,s}$  with  $\gamma = O\left(\sqrt{\frac{pds^2/p}{n\varepsilon^2}}\right)$  for  $p \leq 2\log s$  and  $\gamma = O\left(\sqrt{\frac{d\log s}{n\varepsilon^2}}\right)$  for  $p > 2\log s$ .
Proposition 4. Fix  $p \in [1, \infty]$ . For  $n \geq 1$  and  $\ell \leq d$ , there exists an  $(n, \gamma)$ -estimator using  $\mathcal{W}_{\ell}$  under  $\ell_p$  loss for  $\mathcal{G}_{d,s}$  with  $\gamma = O\left(\sqrt{\frac{pds^2/p}{n\ell} + \frac{(p + \log(2\ell/s))s^{2/p}}{n}}\right)$  for  $p \leq 2\log s$  and  $\gamma = O\left(\sqrt{\frac{d\log s}{n\ell} + \frac{\log\ell}{n}}\right)$  for  $p > 2\log s$ .
We reduce the problem of Gaussian mean estimation to that of Bernoulli mean estimation and then invoke Propositions 1 and 2 from the previous section. At the heart of the reduction is a simple idea that was used in, e.g., [10, 2, 11]: the sign of a Gaussian random variable already preserves sufficient information about the mean. Details follow.
Let  $\mathbf{p} \in \mathcal{G}_{d,s}$  with mean  $\mu(\mathbf{p}) = (\mu(\mathbf{p})_1, \ldots, \mu(\mathbf{p})_d)$ . For  $X \sim \mathbf{p}$ , let  $Y = (\mathrm{sign}(X_i))_{i \in [d]} \in \{-1, +1\}^d$  be a random variable indicating the signs of the  $d$  coordinates of  $X$ . By the independence of the coordinates of  $X$ , note that  $Y$  is distributed as a product Bernoulli distribution (in  $\mathcal{B}_d$ ) with mean vector  $\nu(\mathbf{p})$  given by
$$
\nu (\mathbf {p}) _ {i} = 2 \Pr_ {X \sim \mathbf {p}} [ X _ {i} > 0 ] - 1 = \operatorname {E r f} \left(\frac {\mu (\mathbf {p}) _ {i}}{\sqrt {2}}\right), \quad i \in [ d ], \tag {29}
$$
and, since  $|\mu(\mathbf{p})_i| \leq 1$ , we have  $\nu(\mathbf{p}) \in [-\eta, \eta]^d$ , where  $\eta := \operatorname{Erf}\left(1 / \sqrt{2}\right) \approx 0.623$ . Moreover, it is immediate to see that each player, given a sample from  $\mathbf{p}$ , can convert it to a sample from the corresponding product Bernoulli distribution. We now show that a good estimate for  $\nu(\mathbf{p})$  yields a good estimate for  $\mu(\mathbf{p})$ .
Lemma 6. Fix any  $p \in [1, \infty)$ , and  $\mathbf{p} \in \mathcal{G}_d$ . For  $\widehat{\nu} \in [-\eta, \eta]^d$ , define  $\widehat{\mu} \in [-1, 1]^d$  by  $\widehat{\mu}_i := \sqrt{2} \operatorname{Erf}^{-1}(\widehat{\nu}_i)$ , for all  $i \in [d]$ . Then
$$
\| \mu (\mathbf {p}) - \widehat {\mu} \| _ {p} \leq \sqrt {\frac {e \pi}{2}} \cdot \| \nu (\mathbf {p}) - \widehat {\nu} \| _ {p}.
$$
Proof. By computing the maximum of its derivative, we observe that the function  $\operatorname{Erf}^{-1}$  is  $\frac{\sqrt{e\pi}}{2}$ -Lipschitz on  $[- \eta, \eta]$ . By the definition of  $\widehat{\mu}$  and recalling Eq. (29), we then have
$$
\| \mu (\mathbf {p}) - \widehat {\mu} \| _ {p} ^ {p} = \sum_ {i = 1} ^ {d} | \mu (\mathbf {p}) _ {i} - \widehat {\mu} _ {i} | ^ {p} = 2 ^ {p / 2} \cdot \sum_ {i = 1} ^ {d} \left| \operatorname {E r f} ^ {- 1} \left(\nu_ {i}\right) - \operatorname {E r f} ^ {- 1} \left(\widehat {\nu} _ {i}\right) \right| ^ {p} \leq \left(\frac {e \pi}{2}\right) ^ {p / 2} \cdot \sum_ {i = 1} ^ {d} | \nu_ {i} - \widehat {\nu} _ {i} | ^ {p},
$$
where we used the fact that  $\nu, \widehat{\nu} \in [-\eta, \eta]^d$ .
${}^{8}$  Specifically,we have that  $\mathop{\max }\limits_{{x \in  \left\lbrack  {-\eta ,\eta }\right\rbrack  }}{\operatorname{Erf}}^{-1}\left( x\right)  = 1/\sqrt{2}$  by definition of  $\eta$  and monotonicity of Erf. Recalling then that,for all  $x \in  \left\lbrack  {-\eta ,\eta }\right\rbrack  ,{\left( {\operatorname{Erf}}^{-1}\right) }^{\prime }\left( x\right)  = \frac{1}{{\operatorname{Erf}}^{\prime }\left( {{\operatorname{Erf}}^{-1}\left( x\right) }\right) } = \frac{\sqrt{\pi }}{2}{e}^{\left( {{\operatorname{Erf}}^{-1}\left( x\right) }\right) }^{2} \leq  \frac{\sqrt{\pi }}{2}{e}^{\frac{1}{2}}$  ,we get the Lipschitzness claim.
As previously discussed, combining Lemma 6 with Propositions 1 and 2 (with  $\gamma' := \sqrt{\frac{2}{e\pi}}\gamma$ ) immediately implies Propositions 3 and 4 for  $p \in [1,\infty]$ .
Remark 3. Note that for the Gaussian family, we also consider the linear measurement constraint. Under linear measurement constraints, we can use the linear measurement matrix to obtain  $r$  out of  $d$  coordinates and perform the above reduction to product of Bernoulli family. The obtained bound will be same as that under communication constraints.
D Relation to other lower bound methods
We now discuss how our techniques compare with other existing approaches for proving lower bounds under information constraints. Specifically, we clarify the relationship between our technique and the approach using strong data processing inequalities (SDPI) as well as that based on van Trees inequality (a generalization of the Cramér-Rao bound).
D.1 Strong data processing inequalities
We note first that the bound in Eq. (5) can be interpreted as a strong data processing inequality. Indeed, the average discrepancy on the left-side of inequality can be viewed as the average information  $Y^n$  reveals about each bit of  $Z$ . Here the information is measured in terms of total variation distance. The information quantity on the right-side denotes the information between the input  $X^n$  and the output  $Y^n$  of the channels. Since the Markov relation  $Z^n - X^n - Y^n$  holds, the inequality is thus a strong data processing inequality with strong data processing constant roughly  $\sigma^2 / k$ . Such strong data processing inequalities were used to derive lower bounds for statistical estimation under communication constraints in [34, 10, 31]. We note that our approach recovers these bounds, and further applies to arbitrary constraints captured by  $\mathcal{W}$ .
D.2 Connection to the van Trees inequality
The average information bound in (3), in fact, allows us to recover bounds similar to the van Trees inequality-based bounds developed in [7] and [8].
For  $\Theta \subset \mathbb{R}^k$  and a parametric family  $^9\mathcal{P}_{\Theta} = \{\mathbf{p}_\theta ,\theta \in \Theta \}$ , recall that the Fisher information matrix  $J(\theta)$  is a  $k\times k$  matrix given by, under some mild regularity conditions,
$$
J (\theta) _ {i, j} = - \mathbb {E} _ {\mathbf {p} _ {\theta}} \left[ \frac {\partial^ {2} \log \mathbf {p} _ {\theta}}{\partial \theta_ {i} \partial \theta_ {j}} (X) \right], \quad i, j \in [ k ].
$$
In particular, the diagonal entries equal
$$
J (\theta) _ {i, i} = \mathbb {E} _ {\mathbf {p} _ {\theta}} \left[ \left(\frac {1}{\mathbf {p} _ {\theta} (X)} \cdot \frac {\partial \mathbf {p} _ {\theta}}{\partial \theta_ {i}} (X)\right) ^ {2} \right], \quad i \in [ k ].
$$
For our application, given a channel  $W \in \mathcal{W}$ , we consider the family  $\mathcal{P}_{\Theta}^{W} \coloneqq \{\mathbf{p}_{\theta}^{W}, \theta \in \Theta\}$  of distributions induced on the output of the channel  $W$  when the input distributions are from  $\mathcal{P}_{\Theta}$ . We denote the Fisher information matrix for this family by  $J^{W}(\theta)$ , which we compute next under a refined version of our Assumption 1 described below.
Let  $\theta$  be a point in the interior of  $\Theta$  and  $\mathbf{p}_{\theta}$  be differentiable at  $\theta$ . We set  $\theta_{z} := \theta + \frac{\gamma}{2} z, z \in \{-1, +1\}^{k}$ , and make the following assumption about the structure of the parametric family of distribution: For all  $z \in \{-1, +1\}^{k}$  and  $i \in [k]$ ,
$$
\frac {\mathrm {d} \mathbf {p} _ {z ^ {\oplus i}}}{\mathrm {d} \mathbf {p} _ {z}} = 1 + \gamma \xi_ {z, i} ^ {\gamma} + \gamma^ {2} \psi_ {z, i} ^ {\gamma}, \tag {30}
$$
where  $\mathbb{E}_{\mathbf{p}_z}\left[\xi_{z,i}^\gamma (X)^2\right]$  and  $\mathbb{E}_{\mathbf{p}_z}\left[\psi_{z,i}^\gamma (X)^2\right]$  are assumed to be uniformly bounded for  $\gamma$  sufficiently small; for concreteness, we assume  $\mathbb{E}_{\mathbf{p}_z}\left[\psi_{z,i}^\gamma (X)^2\right] \leq c^2$  for a constant  $c$ , for all  $\gamma$  sufficiently small. Let  $\xi_{z,i}(x) \coloneqq \lim_{\gamma \to 0} \xi_{z,i}^\gamma (x)$ , for all  $x$ .
<sup>9</sup>We assume that each distribution  $\mathbf{p}_{\theta}$  has a density with respect to a common measure  $\nu$ , and, with a slight abuse of notation, denote the density of  $\mathbf{p}_{\theta}$  also by  $\mathbf{p}_{\theta}(X)$ .
In applications, we expect the dependence of  $\xi_{z,i}^{\gamma}$  on  $\gamma$  to be "mild," and, in essence, the assumption above provides a linear expansion of the term  $\alpha_{z,i}\phi_{z,i}$  from Assumption 1 as a function of the perturbation parameter  $\gamma$ . Assuming that the densities are differentiable as a function of  $\theta$ , for the distribution  $\mathbf{p}_{\theta}^{W}$  of the output of a channel  $W$  with input  $X \sim \mathbf{p}_{\theta}$ , we get
$$
\begin{array}{l} \frac {\partial \mathbf {p} _ {\theta} ^ {W} (y)}{\partial \theta_ {i}} = z _ {i} \lim  _ {\gamma \rightarrow 0} \frac {\mathbf {p} _ {\theta_ {z}} ^ {W} (y) - \mathbf {p} _ {\theta_ {z} \oplus i} ^ {W} (y)}{\gamma} \\ = z _ {i} \lim  _ {\gamma \rightarrow 0} \mathbb {E} _ {\mathbf {p} _ {z}} \left[\left(\xi_ {z, i} ^ {\gamma} (X) + \gamma \psi_ {z, i} ^ {\gamma} (X)\right) W (y \mid X) \right] \\ = z _ {i} \mathbb {E} _ {\mathbf {p} _ {\theta}} \left[ \xi_ {z, i} W (y \mid X) \right], \\ \end{array}
$$
where we used Eq. (30), the fact that  $\lim_{\gamma \to 0} \theta_z = \theta$ , the fact that  $\mathbb{E}_{\mathbf{p}_z}\left[\psi_{z,i}^\gamma(X)W(y \mid X)\right] \leq c\sqrt{\mathbb{E}_{\mathbf{p}_z}[W(y \mid X)^2]} \leq c$ , and the dominated convergence theorem. Thus, we get
$$
\operatorname {T r} \left(J ^ {W} (\theta)\right) = \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \frac {\mathbb {E} _ {\mathbf {p} _ {\theta}} \left[ \xi_ {z , i} (X) W (y \mid X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {\theta}} [ W (y \mid X) ]} \mathrm {d} \mu . \tag {31}
$$
Our information contraction bound will be seen later (Section 5) to yield lower bounds for expected estimation error. For concreteness, we give a preview of a version here. We assume for simplicity that  $\mathcal{W}_t = \mathcal{W}$  for all  $t$  and consider the  $\ell_2$  loss function for the dense  $(\tau = 1/2)$  case. By following the proof of Lemma 1 below, given an  $(n,\gamma)$ -estimator  $\hat{\theta} = \hat{\theta}(Y^n,U)$  of  $\mathcal{P}_{\Theta}$  using  $\mathcal{W}^n$  under  $\ell_2$  loss, we can find an estimator  $\hat{Z} = \hat{Z}(Y^n,U)$  such that
$$
\gamma^ {2} \sum_ {i = 1} ^ {k} \Pr \Big [ \hat {Z} _ {i} \neq Z _ {i} \Big ] = \mathbb {E} \Big [ \big \| \theta_ {Z} - \theta_ {\hat {Z}} \big \| _ {2} ^ {2} \Big ] \leq 4 \gamma^ {2},
$$
$$
\frac {1}{k} \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {T V}} \Big (\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}} \Big) \geq 1 - \frac {2}{k} \sum_ {i = 1} ^ {k} \operatorname * {P r} \Big [ \hat {Z} _ {i} \neq Z _ {i} \Big ] \geq 1 - \frac {8 \gamma^ {2}}{k \gamma^ {2}}.
$$
Upon setting  $\gamma \coloneqq 4\gamma /\sqrt{k}$ , we get that the left-side of Eq. (3) is bounded below by  $1 / 4$ . For the same  $\gamma$  and under Eq. (30), the right-side evaluates to
$$
\begin{array}{l} \frac {4 \gamma^ {2} n}{k} \max  _ {z \in \mathcal {Z}} \max  _ {W \in \mathcal {W}} \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \left(\xi_ {z , i} ^ {\gamma} (X) + \gamma \psi_ {z , i} ^ {\gamma} (X)\right) W (y \mid X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} d \mu \\ \leq \frac {8 \gamma^ {2} n}{k} \max _ {z \in \mathcal {Z}} \max _ {W \in \mathcal {W}} \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \xi_ {z , i} ^ {\gamma} (X) W (y \mid X) \right] ^ {2} + \gamma^ {2} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ \psi_ {z , i} ^ {\gamma} (X) W (y \mid X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \mathrm {d} \mu \\ \leq \frac {1 2 8 \gamma^ {2} n}{k ^ {2}} \left(\max _ {z \in \mathcal {Z}} \max _ {W \in \mathcal {W}} \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \xi_ {z , i} ^ {\gamma} (X) W (y \mid X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \mathrm {d} \mu + c ^ {2} \gamma^ {2}\right), \\ \end{array}
$$
where we used  $(a + b)^2 \leq 2(a^2 + b^2)$  and
$$
\int_ {\mathcal {Y}} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \psi_ {z , i} ^ {\gamma} (X) W (y \mid X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \mathrm {d} \mu \leq \int_ {\mathcal {Y}} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ \psi_ {z, i} ^ {\gamma} (X) ^ {2} W (y \mid X) \right] \mathrm {d} \mu = \mathbb {E} _ {\mathbf {p} _ {z}} \left[ \psi_ {z, i} ^ {\gamma} (X) ^ {2} \right] \leq c ^ {2}.
$$
Therefore, Eq. (3) yields
$$
\gamma^ {2} \geq \frac {k ^ {2}}{2 5 6 \cdot n \left(\max  _ {z \in \mathcal {Z}} \max  _ {W \in \mathcal {W}} \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \xi_ {z , i} ^ {\gamma} (X) W (y | X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y | X) ]} \mathrm {d} \mu + c ^ {2}\right)}.
$$
This bound is, in effect, the same as the van Trees inequality with  $\mathrm{Tr}\big(J^W (\theta)\big)$  replaced by
$$
g (\gamma) := \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} [ \phi_ {z , i} (X) W (y \mid X) ] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} d \mu .
$$
In fact, in view of Eq. (31),  $\operatorname{Tr}\big(J^{W}(\theta)\big) = \lim_{\gamma \to 0}g(\gamma) =: g(0)$ . Thus, our general lower bound will recover van Trees inequality-based bounds when Eq. (30) holds and  $g(\gamma) \approx g(0)$ . We note that Eq. (30) holds for all the families considered in this paper (see Eq. (40) for product Bernoulli, Eq. (42) for Gaussian, and Eq. (50) for discrete distributions). We close this discussion by noting that results in Section 3 are obtained by deriving bounds for  $g(\gamma)$  which apply for all  $\gamma$  and, therefore, also for  $g(0) = \operatorname{Tr}\big(J^{W}(\theta)\big)$ .
E Missing proofs in Section 3
Consider  $Z = (Z_{1},\ldots ,Z_{k})\in \{-1,1\}^{k}$  where  $Z_{1},\ldots ,Z_{k}$  are i.i.d. with  $\operatorname*{Pr}[Z_i = 1] = \tau$ . For a fixed  $i\in [k]$ , let
$$
\mathbf{p}_{+i}^{Y^{n}}:= \mathbb{E}_{Z}\Big[\mathbf{p}_{Z}^{Y^{n}}\mid Z_{i} = +1\Big] = \sum_{z:z_{i} = +1}\Big(\prod_{j\neq i}\tau^{\frac{1 + z_{j}}{2}}(1 - \tau)^{\frac{1 - z_{j}}{2}}\Big)\mathbf{p}_{z}^{Y^{n}}
$$