SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {p} _ {- i} ^ {Y ^ {n}} := \mathbb {E} _ {Z} \left[ \mathbf {p} _ {Z} ^ {Y ^ {n}} \mid Z _ {i} = - 1 \right] = \sum_ {z: z _ {i} = - 1} \left(\prod_ {j \neq i} \tau^ {\frac {1 + z _ {j}}{2}} \left(1 - \tau\right) ^ {\frac {1 - z _ {j}}{2}}\right) \mathbf {p} _ {z} ^ {Y ^ {n}},
$$
the partial mixtures of message distributions conditioned on  $Z_{i}$ . We will rely on the following lemma, which relates the desired average discrepancy between the  $\mathbf{p}_{+i}^{Y^n}$  and  $\mathbf{p}_{-i}^{Y^n}$ 's to the sum of  $n$  "local" discrepancy measures (in the form of Hellinger distances between local messages). Each local measure can then be easily bounded in terms of the density  $\mathbf{p}_z$  and the channel  $W$  to get the desired bound.
Lemma 7. With the notation of Theorem 1, we have
$$
\left(\frac {1}{k} \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {T V}} \left(\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}}\right)\right) ^ {2} \leq \frac {1 4}{k} \sum_ {t = 1} ^ {n} \max  _ {z \in \mathcal {Z}} \max  _ {W \in \mathcal {W} _ {t}} \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {W}, \mathbf {p} _ {z \oplus i} ^ {W}\right) ^ {2}, \tag {32}
$$
where  $\mathbf{p}_z^W$  denotes the distribution of  $Y\sim W(\cdot \mid X)$  when  $X\sim \mathbf{p}_z$
The proof of the lemma is rather involved and constitutes the core of the argument. We defer it to the end of the section and show first how it implies Theorem 1. For all  $z$  and  $W$ , we have
$$
\begin{array}{l} \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {W}, \mathbf {p} _ {z ^ {\oplus i}} ^ {W}\right) ^ {2} = \frac {1}{2} \int_ {y \in \mathcal {Y}} \left(\sqrt {\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} - \sqrt {\mathbb {E} _ {\mathbf {p} _ {z ^ {\oplus i}}} [ W (y \mid X) ]}\right) ^ {2} \mathrm {d} \mu \\ = \frac {1}{2} \int_ {\mathcal {Y}} \left(\frac {\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ] - \mathbb {E} _ {\mathbf {p} _ {z} \oplus i} [ W (y \mid X) ]}{\sqrt {\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} + \sqrt {\mathbb {E} _ {\mathbf {p} _ {z} \oplus i} [ W (y \mid X) ]}}\right) ^ {2} d \mu \\ \leq \frac {1}{2} \int_ {\mathcal {Y}} \frac {\left(\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ] - \mathbb {E} _ {\mathbf {p} _ {z} \oplus i} [ W (y \mid X) ]\right) ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} d \mu . \tag {33} \\ \end{array}
$$
Moreover, under Assumption 1; for any  $W \in \mathcal{W}_t$  and  $y \in \mathcal{V}$ ,
$$
\mathbb {E} _ {\mathbf {p} _ {z} \oplus i} [ W (y \mid X) ] = \mathbb {E} _ {\mathbf {p} _ {z}} \left[ \frac {\mathrm {d} \mathbf {p} _ {z} \oplus i}{\mathrm {d} \mathbf {p} _ {z}} \cdot W (y \mid X) \right] = \mathbb {E} _ {\mathbf {p} _ {z}} \left[ \left(1 + \phi_ {z, i} (X)\right) \cdot W (y \mid X) \right].
$$
Plugging this back into (33), we get
$$
\mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {W}, \mathbf {p} _ {z ^ {\oplus i}} ^ {W}\right) ^ {2} \leq \frac {1}{2} \int_ {y} \frac {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ \phi_ {z , i} (X) W (y \mid X) \right] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \mathrm {d} \mu .
$$
Combining this with Lemma 7 concludes the proof of Theorem 1.
Proof of Lemma 7. Our first step is to use the Cauchy-Schwarz inequality, followed by an inequality relating total variation and Hellinger distances:
$$
\begin{array}{l} \frac {1}{k} \left(\sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {T V}} \left(\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}}\right)\right) ^ {2} \leq \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {T V}} \left(\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}}\right) ^ {2} \\ \leq 2 \sum_ {i = 1} ^ {k} d _ {H} \left(\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}}\right) ^ {2} \\ \leq 2 \sum_ {i = 1} ^ {k} \mathbb {E} _ {Z} \left[ \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {Z} ^ {Y ^ {n}}, \mathbf {p} _ {Z ^ {\oplus i}} ^ {Y ^ {n}}\right) ^ {2} \mid Z _ {i} = + 1 \right] \\ = 2 \sum_ {i = 1} ^ {k} \mathbb {E} _ {Z} \left[ \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {Z} ^ {Y ^ {n}}, \mathbf {p} _ {Z \oplus i} ^ {Y ^ {n}}\right) ^ {2} \right], \tag {34} \\ \end{array}
$$
where the last inequality uses joint convexity of squared Hellinger distance, and the final identity is due to independence of each coordinate of  $Z$  and symmetry of Hellinger whereby  $\mathbb{E}_Z\left[\mathrm{d_H}\big(\mathbf{p}_Z^{Y^n},\mathbf{p}_{Z^{\oplus i}}^{Y^n}\big)^2\mid Z_i = +1\right] = \mathbb{E}_Z\left[\mathrm{d_H}\big(\mathbf{p}_Z^{Y^n},\mathbf{p}_{Z^{\oplus i}}^{Y^n}\big)^2\mid Z_i = -1\right]$ .
In order to bound the resulting terms of the sum, we will rely on the so-called cut-paste property of Hellinger distance [6]. Before doing so, we will require an additional piece of notation: for fixed  $z \in \mathcal{Z}, i \in [k], t \in [n]$ , let  $\mathbf{p}_{t \leftarrow z \oplus i}^{Y^n}$  denote the message distribution where player  $t$  gets a sample from  $\mathbf{p}_{z \oplus i}$  and all other players get samples from  $\mathbf{p}_z$ . That is, for all  $y^n \in \mathcal{V}^n$ , the density of  $\mathbf{p}_{t \leftarrow z \oplus i}^{Y^n}$  with respect to the underlying product measure  $\mu^{\otimes n}$  is given by
$$
\frac {\mathrm {d} \mathbf {p} _ {t \leftarrow z \oplus i} ^ {Y ^ {n}}}{\mathrm {d} \mu^ {\otimes n}} (y ^ {n}) = \mathbb {E} _ {X _ {t} \sim \mathbf {p} _ {z \oplus i}} \left[ W ^ {y ^ {t - 1}} \left(y _ {t} \mid X _ {t}\right) \right] \cdot \prod_ {j \neq t} \mathbb {E} _ {X _ {j} \sim \mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X _ {j}\right) \right]. \tag {35}
$$
The following lemma, due to [22], allows us to relate  $\mathrm{d_H}\big(\mathbf{p}_z^{Y^n},\mathbf{p}_{z^{\oplus i}}^{Y^n}\big)$ , the distance between message distributions when all players get observations from  $\mathbf{p}_z$ , or all from  $\mathbf{p}_{z^{\oplus i}}$ , to the distances  $\mathrm{d_H}\big(\mathbf{p}_z^{Y^n},\mathbf{p}_{t\leftarrow z^{\oplus i}}^{Y^n}\big)$  where only one of the  $n$  players gets a sample from  $\mathbf{p}_{z^{\oplus i}}$ .
Lemma 8 ([22, Theorem 7]). There exists  $c_{\mathrm{H}} > 0$  such that for all  $z \in \mathcal{Z}$  and  $i \in [k]$ ,
$$
\mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {Y ^ {n}}, \mathbf {p} _ {z ^ {\oplus i}} ^ {Y ^ {n}}\right) ^ {2} \leq c _ {\mathrm {H}} \sum_ {t = 1} ^ {n} \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {Y ^ {n}}, \mathbf {p} _ {t \leftarrow z ^ {\oplus i}} ^ {Y ^ {n}}\right) ^ {2}.
$$
Moreover, one can take  $c_{\mathrm{H}} = 2 \prod_{t=1}^{\infty} \frac{1}{1 - 2^{-t}} < 7$ .
Combining Eq. (34) and Lemma 8, we get
$$
\begin{array}{l} \frac {1}{k} \left(\sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {T V}} \left(\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}}\right)\right) ^ {2} \leq 1 4 \sum_ {i = 1} ^ {k} \sum_ {t = 1} ^ {n} \mathbb {E} _ {Z} \left[ \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {Z} ^ {Y ^ {n}}, \mathbf {p} _ {t \leftarrow Z ^ {\oplus i}} ^ {Y ^ {n}}\right) ^ {2} \right] \\ = 1 4 \sum_ {t = 1} ^ {n} \mathbb {E} _ {Z} \left[ \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {Z} ^ {Y ^ {n}}, \mathbf {p} _ {t \leftarrow Z ^ {\oplus i}} ^ {Y ^ {n}}\right) ^ {2} \right]. \tag {36} \\ \end{array}
$$
In view of bounding the RHS of (36) term by term, fix  $j \in [n]$  and  $z \in \mathcal{Z}$ . Recalling the expression of  $\mathbf{p}_{t \leftarrow z^{\oplus i}}^{Y^n}$  from (35), unrolling the definition of Hellinger distance, and recalling (35), we have
$$
\begin{array}{l} 2 \sum_ {i = 1} ^ {k} d _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {Y ^ {n}}, \mathbf {p} _ {t \leftarrow z ^ {\oplus i}} ^ {Y ^ {n}}\right) ^ {2} \\ = \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y} ^ {n}} \left(\sqrt {\frac {\mathrm {d} \mathbf {p} _ {z} ^ {Y ^ {n}}}{\mathrm {d} \mu^ {\otimes n}}} - \sqrt {\frac {\mathrm {d} \mathbf {p} _ {t \leftarrow z ^ {\oplus i}} ^ {Y ^ {n}}}{\mathrm {d} \mu^ {\otimes n}}}\right) ^ {2} \mathrm {d} \mu^ {\otimes n} \\ = \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y} ^ {n}} \prod_ {j \neq t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \underbrace {\left(\sqrt {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {t - 1}} \left(y _ {t} \mid X\right) \right]} - \sqrt {\mathbb {E} _ {\mathbf {p} _ {z} \oplus i} \left[ W ^ {y ^ {t - 1}} \left(y _ {t} \mid X\right) \right]}\right) ^ {2}} _ {:= f _ {i, t} \left(y ^ {t - 1}, y _ {t}\right)} \mathrm {d} \mu^ {\otimes n} \\ = \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y} ^ {t - 1}} \prod_ {j <   t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \int_ {\mathcal {Y}} f _ {i, t} \left(y ^ {t - 1}, y _ {t}\right) \int_ {\mathcal {Y} ^ {n - t}} \prod_ {j > t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] d \mu^ {\otimes (t - 1)} d \mu d \mu^ {\otimes (n - t)} \\ = \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y} ^ {t - 1}} \prod_ {j <   t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \int_ {\mathcal {Y}} f _ {i, t} \left(y ^ {t - 1}, y _ {t}\right) \left(\int_ {\mathcal {Y} ^ {n - t}} \prod_ {j > t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \mathrm {d} \mu^ {\otimes (n - t)}\right) \mathrm {d} \mu^ {\otimes (t - 1)} \mathrm {d} \mu \\ = \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y} ^ {t - 1}} \prod_ {j <   t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \int_ {\mathcal {Y}} f _ {i, t} \left(y ^ {t - 1}, y _ {t}\right) \mathrm {d} \mu \mathrm {d} \mu^ {\otimes (t - 1)} \\ = \int_ {\mathcal {Y} ^ {t - 1}} \prod_ {j <   t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} f _ {i, t} \left(y ^ {t - 1}, y _ {t}\right) \mathrm {d} \mu \mathrm {d} \mu^ {\otimes (t - 1)}, \\ \end{array}
$$
where the second-to-last identity uses the observation that, for any fixed  $y^{t}\in \mathcal{V}^{t}$
$$
\int_ {\mathcal {Y} ^ {n - t}} \prod_ {j > t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \mathrm {d} \mu^ {\otimes (n - t)} = 1,
$$
which in turn follows upon taking marginal integrals for each coordinate. We then get from the pointwise inequality  $\sum_{i=1}^{k} \int_{\mathcal{Y}^{t-1}} f_{i,t}(y^{t-1}, y_t) \, \mathrm{d}\mu \leq \sup_{y' \in \mathcal{Y}^{t-1}} \sum_{i=1}^{k} \int_{\mathcal{Y}} f_{i,t}(y', y_t) \, \mathrm{d}\mu$  that
$$
\begin{array}{l} 2 \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {Y ^ {n}}, \mathbf {p} _ {t \leftarrow z ^ {\oplus i}} ^ {Y ^ {n}}\right) ^ {2} \leq \int_ {\mathcal {Y} ^ {t - 1}} \prod_ {j <   t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \sup  _ {y ^ {\prime} \in \mathcal {Y} ^ {t - 1}} \sum_ {i = 1} ^ {k} \left(\int_ {\mathcal {Y}} f _ {i, t} \left(y ^ {\prime}, y _ {t}\right) \mathrm {d} \mu\right) \mathrm {d} \mu^ {\otimes (t - 1)} \\ = \left(\sup  _ {y ^ {\prime} \in \mathcal {Y} ^ {t - 1}} \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} f _ {i, t} \left(y ^ {\prime}, y _ {t}\right) \mathrm {d} \mu\right) \int_ {\mathcal {Y} ^ {t - 1}} \prod_ {j <   t} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {j - 1}} \left(y _ {j} \mid X\right) \right] \mathrm {d} \mu^ {\otimes (t - 1)} \\ = \sup  _ {y ^ {\prime} \in \mathcal {Y} ^ {t - 1}} \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \left(\sqrt {\mathbb {E} _ {\mathbf {p} _ {z}} \left[ W ^ {y ^ {\prime}} (y \mid X) \right]} - \sqrt {\mathbb {E} _ {\mathbf {p} _ {z} \oplus i} \left[ W ^ {y ^ {\prime}} (y \mid X) \right]}\right) ^ {2} d \mu \\ \leq \sup  _ {W \in \mathcal {W} _ {t}} \sum_ {i = 1} ^ {k} \int_ {\mathcal {Y}} \left(\sqrt {\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} - \sqrt {\mathbb {E} _ {\mathbf {p} _ {z} \oplus i} [ W (y \mid X) ]}\right) ^ {2} d \mu \\ = 2 \cdot \sup  _ {W \in \mathcal {W} _ {t}} \sum_ {i = 1} ^ {k} \mathrm {d} _ {\mathrm {H}} \left(\mathbf {p} _ {z} ^ {W}, \mathbf {p} _ {z ^ {\oplus i}} ^ {W}\right) ^ {2}. \tag {37} \\ \end{array}
$$
the second identity follows upon taking marginal integrals, and by replacing  $f_{i,t}$  by its definition; and the second inequality using that  $\left\{ W^{y'} : y' \in \mathcal{Y}^{t-1} \right\} \subseteq \mathcal{W}_t$ , so that we are taking a supremum over a larger set.
Plugging this back into (36) and upper bounding the inner expectation by a maximum concludes the proof of the lemma.
Our starting point is Eq. (3) which holds under Assumption 1. We will bound the right-hand-side of Eq. (3) under assumptions of orthogonality and subgaussianity to prove the two bounds in Theorem 2.
First, under orthogonality (Assumption 2), we apply Bessel's inequality to Eq. (3). For a fixed  $z \in \mathcal{Z}$ , write  $\psi_{z,i} = \frac{\phi_{z,i}}{\sqrt{\mathbb{E}_{\mathbf{P}z}\left[\phi_{z,i}^2\right]}},$  and complete  $(1, \psi_{z,1}, \ldots, \psi_{z,k})$  to get an orthonormal basis  $\mathcal{B}$  for  $L^2(\mathcal{X}, \mathbf{p}_z)$ . Fix any  $W \in \mathcal{W}$  and  $y \in \mathcal{Y}$ , and, for brevity, define  $a: \mathcal{X} \to \mathbb{R}$  as  $a(x) = W(y \mid x)$ . Then, we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \mathbb {E} [ \phi_ {z, i} (X) a (X) ] ^ {2} \leq \alpha^ {2} \sum_ {i = 1} ^ {k} \mathbb {E} [ \psi_ {z, i} (X) a (X) ] ^ {2} = \alpha^ {2} \sum_ {i = 1} ^ {k} \langle a, \psi_ {z, i} \rangle^ {2} = \alpha^ {2} \sum_ {i = 1} ^ {k} \langle a - \mathbb {E} [ a ], \psi_ {z, i} \rangle^ {2} \\ \leq \alpha^ {2} \sum_ {\psi \in \mathcal {B}} \langle a - \mathbb {E} [ a ], \psi \rangle^ {2} = \alpha^ {2} \operatorname {V a r} [ a (X) ], \\ \end{array}
$$
where for the second identity we used the assumption that  $\langle \mathbb{E}[a],\psi_{z,i}\rangle = 0$  for all  $i\in [k]$  (since 1 and  $\psi_{z,i}$  are orthogonal). This establishes Eq. (4).
Turning to Eq. (5), suppose that Assumption 3 holds. Fix  $z \in \mathcal{Z}$ , and consider any  $W \in \mathcal{W}$  and  $y \in \mathcal{Y}$ . Upon applying Lemma 4 of the Supplement (See Supplement (Appendix B) for the precise statement and proof) to the  $\sigma^2$ -subgaussian random vector  $\phi_z(X)$  and with  $a(x)$  set to  $W(y \mid x) \in [0,1]$ , we get that
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \mathbb {E} _ {\mathbf {p} _ {z}} [ \phi_ {z, i} (X) W (y \mid X) ] ^ {2} = \| \mathbb {E} _ {\mathbf {p} _ {z}} [ \phi_ {z} (X) W (y \mid X) ] \| _ {2} ^ {2} \\ \leq 2 \sigma^ {2} \mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ] \cdot \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) \log \frac {W (y \mid X)}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \right] \\ \end{array}
$$
Integrating over  $y \in \mathcal{V}$ , this gives
$$
\begin{array}{l} \int_ {\mathcal {Y}} \frac {\sum_ {i = 1} ^ {k} \mathbb {E} _ {\mathbf {p} _ {z}} [ \phi_ {z , i} (X) W (y \mid X) ] ^ {2}}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \mathrm {d} \mu \leq 2 \sigma^ {2} \cdot \int_ {\mathcal {Y}} \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) \log \frac {W (y \mid X)}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \right] \mathrm {d} \mu \\ = 2 \sigma^ {2} I (\mathbf {p} _ {z}; W), \\ \end{array}
$$
which yields the claimed bound.
E.3 Proof of Corollary 1
For any  $W \in \mathcal{W}^{\mathrm{priv},\varepsilon}$ , the  $\varepsilon$ -LDP condition from Eq. (2) can be seen to imply that, for every  $y \in \mathcal{V}$ ,
$$
W (y \mid x _ {1}) - W (y \mid x _ {2}) \leq \left(e ^ {\varepsilon} - 1\right) W (y \mid x _ {3}), \quad \forall x _ {1}, x _ {2}, x _ {3} \in \mathcal {X}.
$$
By taking expectation over  $x_{3}$  then again either over  $x_{1}$  or  $x_{2}$  (all distributed according to  $\mathbf{p}_z$ ), this yields
$$
\left| W (y \mid x) - \mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ] \right| \leq \left(e ^ {\varepsilon} - 1\right) \mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ], \quad \forall x \in \mathcal {X}.
$$
Squaring and taking the expectation on both sides, we obtain
$$
\operatorname {V a r} _ {\mathbf {p} _ {z}} [ W (y \mid X) ] \leq \left(e ^ {\varepsilon} - 1\right) ^ {2} \mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ] ^ {2}.
$$
Dividing by  $\mathbb{E}_{\mathbf{p}_z}[W(y\mid X)]$ , summing over  $y\in \mathcal{V}$ , and using  $\int_{\mathcal{V}}\mathbb{E}_{\mathbf{p}_z}[W(y\mid X)]\mathrm{d}\mu = 1$  gives
$$
\int_ {\mathcal {Y}} \frac {\operatorname {V a r} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]} \mathrm {d} \mu \leq (e ^ {\varepsilon} - 1) ^ {2} \int_ {\mathcal {Y}} \mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ] \mathrm {d} \mu = (e ^ {\varepsilon} - 1) ^ {2},
$$
thus establishing (6). For the bound of  $e^{\varepsilon}$ , observe that, for all  $y \in \mathcal{V}$
$$
\operatorname {V a r} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) \right] \leq \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) ^ {2} \right] \leq e ^ {\varepsilon} \min  _ {x \in \mathcal {X}} W (y \mid x) \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) \right].
$$
$$
\int_ {\mathcal {Y}} \frac {\operatorname {V a r} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]}{\mathbb {E} _ {\mathbf {p} _ {z}} [ W (y \mid X) ]}   \mathrm {d} \mu \leq e ^ {\varepsilon} \int_ {\mathcal {Y}} \min  _ {x \in \mathcal {X}} W (y \mid x)   \mathrm {d} \mu \leq e ^ {\varepsilon} \cdot \min  _ {x \in \mathcal {X}} \int_ {\mathcal {Y}} W (y \mid x)   \mathrm {d} \mu = e ^ {\varepsilon}.
$$
The bound (7) (under Assumption 3) will follow from (5), and the relation between differential privacy and KL divergence. Indeed, the mutual information  $I(\mathbf{p}_z;W)$  can be rewritten as the expected (over  $X\sim \mathbf{p}_Z$ ) KL divergence between the distribution  $\mathbf{p}^{W,X}\coloneqq W(\cdot \mid X)$  over  $\mathcal{V}$  induced by the channel  $W$  on input  $X$ , and the distribution  $\mathbf{p}_Z^W\coloneqq \mathbb{E}_{X'\sim \mathbf{p}_Z}[W(\cdot \mid X')]$  over  $\mathcal{V}$  induced by the input distribution  $\mathbf{p}_z$  and the channel  $W$ :
$$
I (\mathbf {p} _ {z}; W) = \mathbb {E} _ {X \sim \mathbf {p} _ {z}} \left[ \mathrm {D} \big (\mathbf {p} ^ {W, X} \| \mathbf {p} _ {z} ^ {W} \big) \right] = \mathbb {E} _ {X \sim \mathbf {p} _ {z}} \left[ \mathbb {E} _ {Y \sim \mathbf {p} ^ {W, X}} \left[ \ln \frac {W (Y \mid X)}{\mathbb {E} _ {X ^ {\prime} \sim \mathbf {p} _ {z}} [ W (Y \mid X ^ {\prime}) ]} \right] \right];
$$
but the  $\varepsilon$ -LDP condition from Eq. (2) guarantees that the log-likelihood ratio in the inner expectation is (almost surely) at most  $\varepsilon$ , so that  $I(\mathbf{p}_z;W)\leq \varepsilon$  for every  $z$  and  $W\in \mathcal{W}^{\mathrm{priv},\varepsilon}$ . This yields (7).
In view of (4), to establish (8), it suffices to show that  $\frac{\operatorname{Var}_{\mathbf{p}_z}[W(y|X)]}{\mathbb{E}_{\mathbf{p}_z}[W(y|X)]} \leq 1$  for every  $y \in \mathcal{V}$ . Since  $W(y \mid x) \in (0,1]$  for all  $x \in \mathcal{X}$  and  $y \in \mathcal{Y}$ , so that
$$
\operatorname {V a r} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) \right] \leq \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) ^ {2} \right] \leq \mathbb {E} _ {\mathbf {p} _ {z}} \left[ W (y \mid X) \right].
$$
The second bound (under Assumption 3) will follow from (5). Indeed, recalling that the entropy of the output of a channel is bounded below by the mutual information between input and the output, we have  $I(\mathbf{p}_z;W)\leq H(\mathbf{p}_z^W)$ , where  $\mathbf{p}_z^W\coloneqq \mathbb{E}_{\mathbf{p}_z}[W(\cdot \mid X)]$  is the distribution over  $\mathcal{V}$  induced by the input distribution  $\mathbf{p}_z$  and the channel  $W$ . Using the fact that the entropy of a distribution over  $\mathcal{V}$  is at most  $\log |\mathcal{V}|$  in (5) gives (9).
F Missing proofs in Section 4
Given an  $(n,\gamma)$ -estimator  $(\Pi ,\hat{\theta})$ , define an estimate  $\hat{Z}$  for  $Z$  as
$$
\hat{Z} := \operatorname *{argmin}_{z\in \mathcal{Z}}\bigg\|\theta_{z} - \hat{\theta} (Y^{n},U)\bigg\|_{p}.
$$
$$
\left\| \theta_ {Z} - \theta_ {\hat {Z}} \right\| _ {p} \leq \left\| \theta_ {Z} - \hat {\theta} (Y ^ {n}, U) \right\| _ {p} + \left\| \theta_ {\hat {Z}} - \hat {\theta} (Y ^ {n}, U) \right\| _ {p} \leq 2 \left\| \hat {\theta} (Y ^ {n}, U) - \theta_ {Z} \right\| _ {p}.
$$
Since  $(\Pi, \hat{\theta})$  is an  $(n, \gamma)$ -estimator under  $\ell_p$  loss for  $\mathcal{P}_{\Theta}$ ,
$$
\begin{array}{l} \mathbb {E} _ {Z} \left[ \mathbb {E} _ {\mathbf {p} _ {Z}} \left[ \left\| \theta_ {Z} - \theta_ {\hat {Z}} \right\| _ {p} ^ {p} \right] \right] \leq 2 ^ {p} \gamma^ {p} \Pr [ \mathbf {p} _ {Z} \in \mathcal {P} _ {\Theta} ] + \max _ {z \neq z ^ {\prime}} \| \theta_ {z} - \theta_ {z ^ {\prime}} \| _ {p} ^ {p} \Pr [ \mathbf {p} _ {Z} \notin \mathcal {P} _ {\Theta} ] \\ \leq 2 ^ {p} \gamma^ {p} + 4 ^ {p} \gamma^ {p} \frac {1}{\tau} \cdot \frac {\tau}{4} (38) \\ \leq \frac {3}{4} 4 ^ {p} \gamma^ {p}, (39) \\ \end{array}
$$
where Eq. (38) follows from Assumption 4 and  $\operatorname*{Pr}[\mathbf{p}_Z\in \mathcal{P}_{\Theta}]\geq 1 - \tau /4$  Next, for  $p\in [1,\infty)$  by Assumption 4,  $\left\| \theta_{Z} - \theta_{\hat{Z}}\right\| _p^p\geq \frac{4^p\gamma^p}{\tau k}\sum_{i = 1}^k\mathbb{1}\bigg\{Z_i\neq \hat{Z}_i\bigg\}$  . Combining with Eq. (39) this shows that  $\frac{1}{\tau k}\sum_{i = 1}^{k}\operatorname *{Pr}\Big[Z_i\neq \hat{Z}_i\Big]\leq \frac{3}{4}$
Furthermore, since the Markov relation  $Z_{i} - (Y^{n},U) - \hat{Z}_{i}$  holds for all  $i$ , we can lower bound  $\operatorname*{Pr}\left[Z_i\neq \hat{Z}_i\right]$  using the standard relation between total variation distance and hypothesis testing as
follows, using that  $\tau \leq 1 / 2$  in the second inequality:
$$
\begin{array}{l} \Pr \left[ Z _ {i} \neq \hat {Z} _ {i} \right] \geq \tau \Pr \left[ \hat {Z} _ {i} = - 1 \mid Z _ {i} = 1 \right] + (1 - \tau) \Pr \left[ \hat {Z} _ {i} = 1 \mid Z _ {i} = - 1 \right] \\ \geq \tau \left(\Pr \left[ \hat {Z} _ {i} = - 1 \mid Z _ {i} = 1 \right] + \Pr \left[ \hat {Z} _ {i} = 1 \mid Z _ {i} = - 1 \right]\right) \\ \geq \tau \Big (1 - \mathrm {d} _ {\mathrm {T V}} \left(\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}}\right) \Big). \\ \end{array}
$$
Summing over  $1 \leq i \leq k$  and combining it with the previous bound, we obtain
$$
\frac {3}{4} \geq \frac {1}{\tau k} \sum_ {i = 1} ^ {k} \Pr \left[ Z _ {i} \neq \hat {Z} _ {i} \right] \geq 1 - \frac {1}{k} \sum_ {i = 1} ^ {k} d _ {\mathrm {T V}} \left(\mathbf {p} _ {+ i} ^ {Y ^ {n}}, \mathbf {p} _ {- i} ^ {Y ^ {n}}\right)
$$
and reorganizing proves the result.
G Missing statements and proofs in Section 5
Fix  $p \in [1, \infty)$ . Let  $k = d$ ,  $\mathcal{Z} = \{-1, +1\}^d$ , and  $\tau = \frac{s}{2d}$ ; and suppose that, for some  $\gamma \in (0, 1/8]$ , there exists an  $(n, \gamma)$ -estimator for  $\mathcal{B}_{d,s}$  under  $\ell_p$  loss. We fix a parameter  $\gamma \in (0, 1/2]$ , which will be chosen as a function of  $\gamma, d, p$  later. Consider the set of  $2^d$  product Bernoulli distributions  $\{\mathbf{p}_z\}_{z \in \mathcal{Z}}$  where  $\mu(\mathbf{p}_z) = \mu_z := \frac{1}{2}\gamma(z + 1_d)$  (so the sparsity of the mean vector is equal to the number of positive coordinates of  $z$ ). We have, for  $z \in \mathcal{Z}$ ,
$$
\mathbf {p} _ {z} (x) = \frac {1}{2 ^ {d}} \prod_ {i = 1} ^ {d} \left(1 + \frac {1}{2} \gamma (z _ {i} + 1) x _ {i}\right), \qquad x \in \mathcal {X}.
$$
It follows for  $z \in \mathcal{Z}$  and  $i \in [d]$  that
$$
\mathbf {p} _ {z ^ {\oplus i}} (x) = \frac {1 + \frac {1}{2} \gamma \left(1 - z _ {i}\right) x _ {i}}{1 + \frac {1}{2} \gamma \left(1 + z _ {i}\right) x _ {i}} \mathbf {p} _ {z} (x) = \left(1 - \gamma \frac {z _ {i} x _ {i}}{1 + \frac {1}{2} \gamma \left(1 + z _ {i}\right) x _ {i}}\right) \mathbf {p} _ {z} (x) = \left(1 + \phi_ {z, i} (x)\right) \mathbf {p} _ {z} (x) \tag {40}
$$
where  $\phi_{z,i}(x) := -\frac{\gamma z_i x_i}{1 + \frac{1}{2} \gamma (1 + z_i) x_i}$ . We can verify that, for  $i \neq j$ ,