SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We bound these three terms separately. To bound the first one, observe that condition on  $\mathcal{E}_1\cap \mathcal{E}_2$ $\sum_{i = 1}^{k}Z_{(i)}\geq \tilde{Z}_{(1)}^{(j)}\geq 0.9k\sqrt{\frac{\log N}{\pi\log 2}} -\frac{k}{\sqrt{M}}\gamma .$  As a result,
$$
\begin{array}{l} \operatorname * {P r} \left\{\mathcal {E} _ {1} \cap \mathcal {E} _ {2} \cap \mathcal {E} _ {M} \right\} \cdot \mathbb {E} \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \middle | \mathcal {E} _ {1} \cap \mathcal {E} _ {2} \cap \mathcal {E} _ {M} \right] \\ \geq \frac {0 . 9 k \sqrt {\frac {\log N}{\pi \log 2}} - \frac {k}{\sqrt {M}} \gamma}{\sqrt {M (1 + \gamma)}} \cdot \left(1 - \left(0. 1 + e ^ {- \xi^ {2} / 2} + 2 e ^ {- M \gamma^ {2} / 4}\right)\right). \tag {102} \\ \end{array}
$$
To bound the second term, observe that under  $\mathcal{E}_2$
$$
\sum_ {i = 1} ^ {k} Z _ {(i)} \geq - \frac {k}{\sqrt {M}} \xi ,
$$
so we have
$$
\begin{array}{l} \Pr \left\{\mathcal {E} _ {2} \cap \mathcal {E} _ {1} ^ {c} \cap \mathcal {E} _ {M} \right\} \cdot \mathbb {E} \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \middle | \mathcal {E} _ {2} \cap \mathcal {E} _ {1} ^ {c} \cap \mathcal {E} _ {M} \right] \\ \geq \Pr \left\{\mathcal {E} _ {2} \cap \mathcal {E} _ {1} ^ {c} \cap \mathcal {E} _ {M} \right\} \cdot \left(- \frac {k}{\sqrt {M ^ {2} (1 - \gamma)}} \xi\right) \\ \geq \Pr \left\{\mathcal {E} _ {1} ^ {c} \right\} \cdot \left(- \frac {k}{\sqrt {M ^ {2} (1 - \gamma)}} \xi\right) \\ \geq 0. 1 \cdot \left(- \frac {\xi \sqrt {k}}{\sqrt {M ^ {2} (1 - \gamma)}}\right). \tag {103} \\ \end{array}
$$
For the third term, it holds that
$$
\begin{array}{l} \operatorname * {P r} \left\{\mathcal {E} _ {2} ^ {c} \cup \mathcal {E} _ {M} ^ {c} \right\} \cdot \mathbb {E} \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \middle | \mathcal {E} _ {2} ^ {c} \cup \mathcal {E} _ {M} ^ {c} \right] \\ \geq \Pr \left\{\mathcal {E} _ {2} ^ {c} \cup \mathcal {E} _ {M} ^ {c} \right\} \cdot \inf  _ {a \in \mathbb {S} ^ {M - 1}} \sum_ {i = 1} ^ {k} a _ {(i)} \\ \geq - \Pr \left\{\mathcal {E} _ {2} ^ {c} \cup \mathcal {E} _ {M} ^ {c} \right\} \cdot \frac {k}{\sqrt {M}} \\ \geq - \left(e ^ {- \xi^ {2} / 2} + e ^ {- M \gamma^ {2} / 4}\right) \cdot \frac {k}{\sqrt {M}} \tag {104} \\ \end{array}
$$
Combining (102), (103), and (104) together, we arrive at
$$
\begin{array}{l} \mathbb {E} _ {Z _ {1}, \dots , Z _ {M} \stackrel {\text {i . i . d}} {\sim}} \mathcal {N} (0, 1) \left[ \frac {\sum_ {i = 1} ^ {k} Z _ {(i)}}{\sqrt {\sum_ {i = 1} ^ {M} Z _ {i} ^ {2}}} \right] \\ \geq \frac {0 . 9 k \left(\sqrt {\frac {\log N}{\pi \log 2}}\right) - \frac {k}{\sqrt {M}} \gamma}{\sqrt {M (1 + \gamma)}} \cdot \left(1 - \left(0. 1 + e ^ {- \xi^ {2} / 2} + 2 e ^ {- M \gamma^ {2} / 4}\right)\right) - 0. 1 \cdot \left(\frac {\xi \sqrt {k}}{\sqrt {M ^ {2} (1 - \gamma)}}\right) \\ - \left(e ^ {- \xi^ {2} / 2} + e ^ {- M \gamma^ {2} / 4}\right) \cdot \frac {k}{\sqrt {M}}. \\ \end{array}
$$
Finally, setting  $\gamma = O\left(\frac{1}{\sqrt{M}}\right)$  and  $\xi = O(1)$  yields the desired lower bound
$$
C _ {d, M, k} = \Omega \left(\frac {k \log N}{\sqrt {M}}\right).
$$
H Additional Experimental Results
In Figure 2, we provide additional empirical results by sweeping the number of users  $n$  from 2,000 to 10,000 on the left and sweeping the dimension  $d$  from 200 to 1,000 on the right.
I Additional Details on Prior LDP Schemes
For completeness, we provide additional details on prior LDP mean estimation schemes in this section, including PrivUnit [4], SQKR [6], FT21 [12], and MMRC [30]. We skip prior work analyzing compression-privacy-utility tradeoffs that do not specifically focus on the distributed mean estimation problem [19, 20] or others that study frequency estimation [6, 11, 30].
I.1 PrivUnit [4]
[2] considered the mean estimation problem under DP constraint (without communication constraint) when  $\mathcal{X} = \mathbb{S}^{d - 1} = \{v\in \mathbb{R}^d:\| v\| _1 = 1\}$ . Since there is no communication constraint, they assumed canonical protocol where the random encoder is  $f:\mathbb{S}^{d - 1}\to \mathbb{R}^d$  and the decoder is a simple additive aggregator
$$
g _ {n} (f (v _ {1}), \dots , f (v _ {n})) = \frac {1}{n} \sum_ {i = 1} ^ {n} f (v _ {i}).
$$
The authors showed that PrivUnit is an exact optimal among the family of unbiased locally private procedures.
Recall that given an input vector  $v \in \mathbb{S}^{d - 1}$ , the local randomized  $\mathrm{PrivUnit}(p,q)$  has the following distribution up to normalization:
$$
\operatorname {P r i v U n i t} (p, q) \sim \left\{ \begin{array}{l l} Z | \langle Z, v \rangle \geq \gamma & \text {w . p .} p \\ Z | \langle Z, v \rangle <   \gamma & \text {w . p .} 1 - p \end{array} \right.
$$
where  $Z$  has a uniform distribution on  $\mathbb{S}^{d - 1}$ . Let  $S_{\gamma}$  be the surface area of hypersphere cap  $\{z \in \mathbb{S}^{d - 1} | \langle z, v \rangle \geq \gamma \}$ , with  $S_{-1}$  representing the surface area of the  $d$  dimensional hypersphere. We denoted  $q = \operatorname*{Pr}[Z_1 \leq \gamma] = (S_{-1} - S_{\gamma}) / S_{-1}$  (convention from [4, 2]). The normalization factor is required to obtain unbiasedness.
[2] also introduced PrivUnitG, which is a Gaussian approximation of PrivUnit. In this approach,  $Z$  is sampled from an i.i.d.  $\mathcal{N}(0,1 / d)$  distribution. This simplifies the process of determining more accurate parameters  $p,q$ , and  $\gamma$ . Consequently, in practical applications, PrivUnitG surpasses PrivUnit in performance owing to superior parameter optimization.
I.2 SQKR [6]
Next, we outline the encoder and decoder of SQKR in this section. The encoding function mainly consists of three steps: (1) computing Kashin's representation, (2) quantization, and (3) sampling and privatization.
Compute Kashin's representation A tight frame is a set of vectors  $\{u_j\}_{j=1}^N \in \mathbb{R}^d$  that satisfy Parseval's identity, i.e.  $\|v\|_2^2 = \sum_{j=1}^N \langle u_j, v \rangle^2$  for all  $v \in \mathbb{R}^d$ . We say that the expansion  $v = \sum_{j=1}^N a_j u_j$  is a Kashin's representation of  $x$  at level  $K$  if  $\max_j |a_j| \leq \frac{K}{\sqrt{N}} \|v\|_2$  [23]. [27] shows that if  $N > (1 + \mu)d$  for some  $\mu > 0$ , then there exists a tight frame  $\{u_j\}_{j=1}^N$  such that for any  $x \in \mathbb{R}^d$ , one can find a Kashin's representation at level  $K = \Theta(1)$ . This implies that we can represent the local vector  $v$  with coefficients  $\{a_j\}_{j=1}^N \in [-c/\sqrt{d}, c/\sqrt{d}]^N$  for some constants  $c$  and  $N = \Theta(d)$ .
Quantization In the quantization step, each client quantizes each  $a_{j}$  into a 1-bit message  $q_{j} \in \left\{-c / \sqrt{d}, c / \sqrt{d}\right\}$  with  $\mathbb{E}[q_j] = a_j$ . This yields an unbiased estimator of  $\{a_{j}\}_{j=1}^{N}$ , which can be described in  $N = \Theta(d)$  bits. Moreover, due to the small range of each  $a_{j}$ , the variance of  $q_{j}$  is bounded by  $O(1 / d)$ .
Sampling and privatization To further reduce  $\{q_j\}$  to  $k = \min (\lceil \varepsilon \rceil ,b)$  bits, client  $i$  draws  $k$  independent samples from  $\{q_j\}_{j = 1}^N$  with the help of shared randomness, and privatizes its  $k$  bits message via  $2^{k}$ -RR mechanism[36], yielding the final privatized report of  $k$  bits, which it sends to the server.
Upon receiving the report from client  $i$ , the server can construct unbiased estimators  $\hat{a}_j$  for each  $\{a_j\}_{j=1}^N$ , and hence reconstruct  $\hat{v} = \sum_{j=1}^N \hat{a}_j u_j$ , which yields an unbiased estimator of  $v$ . In [6], it is shown that the variance of  $\hat{v}$  can be controlled by  $O(d / \min(\varepsilon^2, \varepsilon, b))$ .
I.3 FT21 [12] and MMRC [30]
Both FT21 and MMRC aim to simulate a given  $\varepsilon$ -LDP scheme. More concretely, consider an  $\varepsilon$ -LDP mechanism  $q(\cdot | v)$  that we wish to compress, which in our case, PrivUnit. A number of candidates  $u_{1}, \dots, u_{N}$  are drawn from a fixed reference distribution  $p(u)$  (known to both the client and the server), which in our case, uniform distribution on the sphere  $\mathbb{S}^{d-1}$ . Under FT21 [12], these candidates are generated from an (exponentially strong) PRG, with seed length  $\ell = \text{polylog}(d)$ . The client then performs rejection sampling and sends the seed of the sampled candidates to the server. See Algorithm 2 for an illustration.
On the other hand, under MRC [30] the LDP mechanism is simulated via a minimal random coding technique [15]. Specifically, the candidates are generated via shared randomness, and the client performs an importance sampling and sends the index of the sampled one to the server, as illustrated in Algorithm 3. It can be shown that when the target mechanism is  $\varepsilon$ -LDP, the communication costs of both strategies are  $\Theta(\varepsilon)$  bits. It is also worth noting that both strategies will incur some bias (though the bias can be made exponentially small as one increases the communication cost), and [30] provides a way to correct the bias when the target mechanism is PrivUnit (or general cap-based mechanisms).
Algorithm 3 Simulating LDP mechanisms via importance sampling [30]
Inputs:  $\varepsilon$ -LDP mechanism  $q(\cdot | v)$ , ref. distribution  $p(\cdot)$ , # of candidates  $M$
Draw samples  $u_{1}, \dots, u_{M}$  from  $p(u)$  using the shared source of randomness.
for  $k\in \{1,\dots ,M\}$  do
$w(k)\gets q(u_k|v) / p(u_k).$
$\pi_{\mathrm{MRC}}(\cdot)\gets w(\cdot) / \sum_kw(k).$
Draw  $k^{*}\gets \pi_{\mathrm{MRC}}$
Output:  $k^{*}$