SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} O \left(\frac {B}{n} + \frac {d B \log (B / \delta)}{n ^ {2}} + \frac {d (\log (1 / \delta) + \varepsilon) \log (B / \delta)}{n ^ {2} \varepsilon^ {2}}\right) \\ = O \left(\frac {d}{n 2 ^ {b}} + \frac {d ^ {2} \log (d / \delta)}{n ^ {2} 2 ^ {b}} + \frac {d (\log (1 / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right). \\ \end{array}
$$
Algorithm 3 Subsampled Recursive Hadamard Response
Input: user data  $x_{1},\ldots ,x_{n}\in \{0,1\}^{d}$  (where  $d$  is a power of two), DP parameters  $(\varepsilon ,\delta)$ , communication budget  $b$ .
Output: frequency estimate  $\hat{\pi}$
Set  $B \coloneqq d / 2^{b - 1}$  and partition each one-hot vector  $x_{i}$  into  $2^{b - 1}$  chunks:  $x_{i}^{(1)},\dots,x_{i}^{(2^{b} - 1)} \in \{0,1\}^{B}$ .
Compute the Hadamard transform of each chunk:  $y_{i}^{(\ell)} = H_{B}\cdot x_{i}^{(\ell)}$
for coordinate  $j \in [B]$  do
Draw  $Z_{i,j}\stackrel {\mathrm{i.i.d.}}{\sim}\operatorname {Bern}\left(\frac{1}{B}\right)$
Send  $(y_{i}^{(1)}(j),\dots,y_{i}^{(2^{b - 1})}(j))$  to the server.
Server computes the average:  $\forall \ell \in [2^{b - 1}], j \in [B]$ ,
$$
\hat {y} ^ {(\ell)} (j) := \frac {B}{n} \sum_ {i: Z _ {i j} = 1} y _ {i} ^ {(\ell)} (j) + N (0, \sigma^ {2}),
$$
where  $\sigma^2$  is computed according to Theorem A.2.
Server performs the inverse Hadamard transform  $\hat{\pi}^{(\ell)} = H_B\cdot \hat{y}^{(\ell)}$ , for  $\ell = 1,\dots,B$
Return:  $\hat{\pi} = \left(\left(\hat{\pi}^{(1)}\right)^{\mathsf{T}},\dots,\left(\hat{\pi}^{(2^{b - 1})}\right)^{\mathsf{T}}\right)$
Notice that when  $n = \tilde{\Omega}(d)$ , the error can be simplified to
$$
O \left(\frac {d}{n 2 ^ {b}} + \frac {d (\log (1 / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right),
$$
which matches the order-optimal estimation error (up to a log  $d$  factor) subject to a  $b$ -bit constraint [55, 3, 2] and  $(\varepsilon, \delta)$ -DP constraint [15, 7].
B Proof of Theorem 4.1
It is trivial to see that the average communication cost is  $d \cdot \gamma = b$  bits. To compute the  $\ell_2^2$  estimation error, observe that
$$
\begin{array}{l} \mathbb {E} \left[ \| \hat {\mu} _ {x ^ {n}} - \mu_ {x ^ {n}} \| _ {2} ^ {2} \right] \\ = \sum_ {j = 1} ^ {d} \mathbb {E} \left[ \left(\frac {1}{n \gamma} \sum_ {i} x _ {i} (j) \cdot Z _ {i, j} + N (0, \sigma^ {2}) - \frac {1}{n} \sum_ {i} x _ {i} (j)\right) ^ {2} \right] \\ = \sum_ {j = 1} ^ {d} \frac {1}{n ^ {2}} \mathbb {E} \left[ \left(\frac {1}{\gamma} \sum_ {i} x _ {i} (j) \cdot Z _ {i, j} - \sum_ {i} x _ {i} (j)\right) ^ {2} \right] + d \sigma^ {2} \\ = \sum_ {j = 1} ^ {d} \frac {1}{n ^ {2}} \mathbb {E} \left[ \left(\frac {1}{\gamma} \sum_ {i} x _ {i} (j) \cdot Z _ {i, j}\right) ^ {2} \right] - \frac {1}{n ^ {2}} \left(\sum_ {i} x _ {i} (j)\right) ^ {2} + d \sigma^ {2} \\ = \sum_ {j = 1} ^ {d} \frac {1}{n ^ {2}} \mathbb {E} \left[ \frac {1}{\gamma^ {2}} \sum_ {i} x _ {i} ^ {2} (j) \cdot Z _ {i, j} ^ {2} + \frac {1}{\gamma^ {2}} \sum_ {i \neq i ^ {\prime}} x _ {i} (j) x _ {i ^ {\prime}} (j) Z _ {i, j} Z _ {i ^ {\prime}, j} \right] - \frac {1}{n ^ {2}} \left(\sum_ {i} x _ {i} (j)\right) ^ {2} + d \sigma^ {2} \\ = \sum_ {j = 1} ^ {d} \frac {1}{n ^ {2}} \left(\frac {1}{\gamma} \sum_ {i} x _ {i} ^ {2} (j) + \sum_ {i \neq i ^ {\prime}} x _ {i} (j) x _ {i ^ {\prime}} (j)\right) - \frac {1}{n ^ {2}} \left(\sum_ {i} x _ {i} (j)\right) ^ {2} + d \sigma^ {2} \\ = \sum_ {j = 1} ^ {d} \frac {1}{n ^ {2}} \left(\frac {1}{\gamma} - 1\right) \left(\sum_ {i} x _ {i} ^ {2} (j)\right) + d \sigma^ {2} \\ \leq \frac {d c ^ {2}}{n \gamma} + d \sigma^ {2}, \\ \end{array}
$$
which yields the inequality of (2). Next, we analyze the privacy of Algorithm 1. We first the following two lemmas for subsampling and the Gaussian mechanism:
Lemma B.1 ([66, 82]). If  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -DP, then  $\mathcal{M}'$  that applies  $\mathcal{M} \circ$  PoissonSample satisfies  $(\varepsilon', \delta')$ -DP with  $\varepsilon' = \log(1 + \gamma(e^{\varepsilon} - 1))$  and  $\delta' = \gamma \delta$ .
Lemma B.2 ([15]). For any  $\varepsilon, \delta \in (0,1)$ , the Gaussian output perturbation mechanism with  $\sigma^2 \coloneqq \frac{\Delta^2 2 \log(1.25 / \delta)}{\varepsilon^2}$  satisfies  $(\varepsilon, \delta)$ -DP, where  $\Delta$  is the  $\ell_2$  sensitivity of the target function.
Now, we use the above two lemmas to analyze the per-coordinate privacy leakage of Algorithm 1. For simplicity, we analyze the sum of  $x_{i}(j)$ 's instead (and normalized it in the last step). Let  $S_{j}(x^{n}) \coloneqq \sum_{i = 1}^{n}(x_{i}(j))$ , then clearly the sensitivity of  $S_{j}(x^{n})$  is  $2c$ , so Lemma B.2 implies  $S_{j}(x^{n}) + N(0,\sigma_{1}^{2})$  satisfies  $(\varepsilon_1,\delta_1)$ -DP if we set  $\sigma_1^2 = \frac{2c^2\log(1.25 / \delta_1)}{\varepsilon_1^2}$  (assuming  $\varepsilon_{1} < 1$ ). Next, if applying subsampling before computing the sum, i.e.,
$$
S _ {j} \circ \text {P o i s s o n S a m p l e} _ {\gamma} (x ^ {n}) := \sum_ {i = 1} ^ {n} x _ {i} (j) Z _ {i, j},
$$
where  $Z_{i,j}\stackrel {\mathrm{i.i.d.}}{\sim}\operatorname {Bern}(\gamma)$  as defined in Algorithm 1, then by Lemma B.1,
$$
S _ {j} \circ \text {P o i s s o n S a m p l e} _ {\gamma} (x ^ {n}) + N \left(0, \sigma_ {1} ^ {2}\right)
$$
satisfies  $(\varepsilon_2, \delta_2)$ -DP with  $\varepsilon_2 \coloneqq \log (1 + \gamma (e^{\varepsilon_1} - 1)) = C_1 \gamma \varepsilon_1$  (since we assume  $\epsilon_1 < 1$ ) and  $\delta_2 \coloneqq \gamma \delta_1$ . Equivalently, we have
$$
\left\{ \begin{array}{l} \varepsilon_ {1} = \tilde {C} _ {1} \frac {1}{\gamma} \varepsilon_ {2} \\ \delta_ {1} = \frac {1}{\gamma} \delta_ {2}. \end{array} \right. \tag {10}
$$
Now, since we have established the per-coordinate privacy leakage, we apply the following composition theorem to account for the total privacy budgets.
Theorem B.3 ([39]). For any  $\varepsilon > 0$ ,  $\delta \in [0,1]$  and  $\tilde{\delta} \in (0,1]$ , the class of  $(\varepsilon, \delta)$ -DP mechanisms satisfies  $(\tilde{\varepsilon}_{\tilde{\delta}}, d\delta + \tilde{\delta})$ -DP under  $d$ -fold adaptive composition, for
$$
\tilde {\varepsilon} _ {\tilde {\delta}} = d \varepsilon \left(e ^ {\varepsilon} - 1\right) + \varepsilon \sqrt {2 d \log (1 / \tilde {\delta})}.
$$
According Theorem B.3, Algorithm 1 satisfies  $(\varepsilon, \delta)$ -DP for
$$
\varepsilon = d \varepsilon_ {2} \left(e ^ {\varepsilon_ {2}} - 1\right) + \varepsilon_ {2} \sqrt {2 d \log (1 / \tilde {\delta})}, \tag {11}
$$
and  $\delta = d\delta_{2} + \tilde{\delta}$  (where  $\tilde{\delta}$  is a free parameter that we can optimize).
Consequently, for a pre-specified (total) privacy budget  $(\varepsilon, \delta)$ , we set parameters as follows. Let  $\tilde{\delta} = \frac{\delta}{2}$  and  $\delta_1 = \frac{1}{\gamma} \delta_2 = \frac{1}{2d\gamma} \delta$ . Let  $\varepsilon_2 \leq 1$  so that  $e^{\varepsilon_2} - 1 \leq 2\varepsilon_2$  holds. Then (11) implies Algorithm 1 is
$$
\varepsilon = 2 d \varepsilon_ {2} ^ {2} + \varepsilon_ {2} \sqrt {2 d \log (1 / \tilde {\delta})} \geq d \varepsilon_ {2} (e ^ {\varepsilon_ {2}} - 1) + \varepsilon_ {2} \sqrt {2 d \log (1 / \tilde {\delta})}.
$$
Solving the above quadratic (in-)equality for  $\varepsilon_{2}$ , it suffices that
$$
\varepsilon_ {2} = \min \left(1, \frac {- \sqrt {2 d \log (2 / \delta)} + \sqrt {2 d \log (2 / \delta) + 8 \varepsilon d}}{4 d}\right) = O \left(\min \left(1, \frac {\varepsilon}{\sqrt {d (\log (1 / \delta) + \varepsilon)}}\right)\right).
$$
Consequently, we set  $\varepsilon_1 = \frac{\tilde{C}_1}{\gamma}\varepsilon_2 = O\left(\min \left(1,\frac{\varepsilon}{\gamma\sqrt{d(\log(1 / \delta) + \varepsilon}}\right)\right)$  (note that we require  $\varepsilon_{1} = O(1)$  so that (10) holds).
Plugging  $(\varepsilon_{1},\delta_{1})$  into  $\sigma_1^2$  , we need to choose
$$
\sigma_ {1} ^ {2} := \frac {2 c ^ {2} \log (1 . 2 5 / \delta_ {1})}{\varepsilon_ {1} ^ {2}} = \Omega \left(\max \left(c ^ {2} \log (d / \delta), \frac {\gamma^ {2} c ^ {2} d (\log (1 / \delta) + \varepsilon) \log (d / \delta)}{\varepsilon^ {2}}\right)\right).
$$
Finally, as we are interested in estimating the (subsampled) mean instead of the sum, we will normalize the private sum by
$$
\hat {\mu} _ {j} (x ^ {n}) = \frac {1}{n \gamma} \left(S _ {j} \circ \mathsf {P o i s s o n S a m p l e} _ {\gamma} (x ^ {n}) + N (0, \sigma_ {1} ^ {2})\right) = \frac {1}{n \gamma} S _ {j} \circ \mathsf {P o i s s o n S a m p l e} _ {\gamma} (x ^ {n}) + N (0, \sigma^ {2}),
$$
$$
\sigma^ {2} = \Theta \left(\max \left(\frac {c ^ {2} \log (d / \delta)}{n ^ {2} \gamma^ {2}}, \frac {c ^ {2} d (\log (1 / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right)\right).
$$
Plugging in  $\sigma^2$  above and  $\gamma = b / d$  yields the desired accuracy in Theorem 4.1.
Since we will reuse the above result, we summarize it into the following lemma:
Lemma B.4. Let  $f_{i}:\mathbb{R}^{d\times m}\mapsto \mathbb{R}^{D}$  for  $i = 1,\dots,B$  be  $n$  functions with sensitivity bounded by  $\Delta$  (where the number of inputs  $m$  can be a random variable). Then
$$
\left(f _ {1} \circ \text {P o i s s o n S a m p l e} _ {\gamma} \left(x ^ {n}\right) + N \left(0, \sigma^ {2}\right), \dots , f _ {B} \circ \text {P o i s s o n S a m p l e} _ {\gamma} \left(x ^ {n}\right) + N \left(0, \sigma^ {2}\right)\right)
$$
satisfies  $(\varepsilon, \delta)$ -DP, if
$$
\sigma^ {2} \geq O \left(\max \left(\Delta^ {2} \log (B / \delta), \frac {\gamma^ {2} \Delta^ {2} B (\log (1 / \delta) + \varepsilon) \log (B / \delta)}{\varepsilon^ {2}}\right)\right).
$$
C Omitted details of dimension-free communication cost
C.1 Proof of Theorem 4.4
To prove Theorem 4.4, it suffices to prove the following  $\ell_{\infty}$  version:
Theorem C.1. Let  $x_{1},\ldots ,x_{n}\in \{-c,c\}^{d}$ ,  $d^{\prime} = \min \left(nb,\frac{n^{2}\varepsilon^{2}}{(\log(1 / \delta) + \varepsilon)\log(d / \delta)}\right)$ , and
$$
\sigma^ {2} = O \left(\frac {c ^ {2} \log (1 / \delta)}{n ^ {2} \gamma^ {2}} + \frac {c ^ {2} d ^ {\prime} (\log \left(d ^ {\prime} / \delta\right) + \varepsilon) \log \left(d ^ {\prime} / \delta\right)}{n ^ {2} \varepsilon^ {2}}\right). \tag {12}
$$
Then Algorithm 2 is  $(\varepsilon, \delta)$ -DP and yields an unbiased estimator on  $\mu$ . In addition, the (average) per-client communication cost is  $\gamma d' = b$  bits, and the  $\ell_2^2$  estimation error is at most
$$
O \left(c ^ {2} d ^ {2} \log \left(\frac {d}{\delta}\right) \max  \left(\frac {1}{n b}, \frac {\left(\log (1 / \delta) + \varepsilon\right)}{n ^ {2} \varepsilon^ {2}}\right)\right). \tag {13}
$$
With a slight abuse of notation, we let  $\mu_{\mathcal{I}} \in \mathbb{R}^d$  be such that
$$
\mu_ {\mathcal {J}} (j) = \left\{ \begin{array}{l l} 0, & \text {i f} j \notin \mathcal {J} \\ \frac {d \mu_ {j}}{d ^ {\prime}}, & \text {e l s e .} \end{array} \right.
$$
Note that  $\mu_{\mathcal{I}}$  is an unbiased estimate of  $\mu$  if  $\mathcal{I}$  is selected uniformly at random. Then the  $\ell_2^2$  error can be controlled by
$$
\begin{array}{l} \mathbb {E} \left[ \| \mu - \hat {\mu} \| _ {2} ^ {2} \right] \stackrel {\mathrm {(a)}} {=} \mathbb {E} \left[ \| \mu - \mu_ {\mathcal {I}} \| _ {2} ^ {2} \right] + \mathbb {E} \left[ \| \mu_ {\mathcal {I}} - \hat {\mu} \| _ {2} ^ {2} \right] \\ \stackrel {(b)} {\leq} \mathbb {E} \left[ \| \mu - \mu_ {\mathcal {I}} \| _ {2} ^ {2} \right] + \frac {d ^ {2}}{d ^ {\prime 2}} O \left(\max  \left(\frac {d ^ {\prime 2} c ^ {2}}{n b}, \frac {d ^ {\prime 3} c ^ {2} \log (d / \delta)}{n ^ {2} b ^ {2}}, \frac {c ^ {2} d ^ {\prime 2} (\log (1 / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right)\right) \\ = \mathbb {E} \left[ \| \mu - \mu_ {\mathcal {I}} \| _ {2} ^ {2} \right] + O \left(\max  \left(\frac {d ^ {2} c ^ {2}}{n b}, \frac {d ^ {2} d ^ {\prime} c ^ {2} \log (d / \delta)}{n ^ {2} b ^ {2}}, \frac {c ^ {2} d ^ {2} (\log (1 / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right)\right) \\ \stackrel {(c)} {\leq} \frac {d ^ {2} c ^ {2}}{d ^ {\prime}} + O \left(\max  \left(\frac {d ^ {2} c ^ {2}}{n b}, \frac {d ^ {2} d ^ {\prime} c ^ {2} \log (d / \delta)}{n ^ {2} b ^ {2}}, \frac {c ^ {2} d ^ {2} (\log (1 / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right)\right), \\ \end{array}
$$
where (a) holds since  $\mu_{\mathcal{I}}$  is an unbiased estimate of  $\mu$  and conditioned on  $\mathcal{I}$ ,  $\hat{\mu}$  is an unbiased estimate of  $\mu_{\mathcal{I}}$ ; (b) follows from Theorem 4.1; (c) holds due to the following fact:
$$
\mathbb {E} \left[ \| \mu - \mu_ {\mathcal {I}} \| _ {2} ^ {2} \right] \leq \sum_ {j \in \mathcal {J}} \mu_ {\mathcal {I}} (j) ^ {2} + \sum_ {j \in [ d ]} \mu_ {j} ^ {2} \leq \frac {d ^ {2} c ^ {2}}{d ^ {\prime}} + d c ^ {2} \leq \frac {2 d ^ {2} c ^ {2}}{d ^ {\prime}}.
$$
Therefore, by setting  $d' = \min \left( nb, \frac{n^2 \varepsilon^2}{(\log(1/\delta) + \varepsilon) \log(d/\delta)} \right)$  we ensure the first term in (c) is always smaller than the second term, and the second term can be simplified as follows:
$$
\begin{array}{l} O \left(c ^ {2} d ^ {2} \max  \left(\frac {1}{n b}, \frac {d ^ {\prime} \log (d / \delta)}{n ^ {2} b ^ {2}}, \frac {(\log (1 / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right)\right) \\ \leq O \left(c ^ {2} d ^ {2} \max  \left(\frac {1}{n b}, \frac {n b \log (d / \delta)}{n ^ {2} b ^ {2}}, \frac {\left(\log (1 / \delta) + \varepsilon\right) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}\right)\right) \\ \leq O \left(c ^ {2} d ^ {2} \log (d / \delta) \max \left(\frac {1}{n b}, \frac {(\log (1 / \delta) + \varepsilon)}{n ^ {2} \varepsilon^ {2}}\right)\right). \\ \end{array}
$$
Finally, applying the same trick of Kashin's representation, we can transform the  $\ell_{\infty}$  geometry to  $\ell_2$  (similar to Corollary 4.3), hence proving Theorem 4.4.
D Proof of Theorem A.1
Let  $\pi := \frac{1}{n} \sum_{i} x_{i}$  and  $\pi^{(\ell)}$  be defined in the same way as  $x_{i}^{(\ell)}$  for  $\ell \in [B]$ . Then our goal is to bound  $\left|\pi^{(\ell)}(j) - \hat{\pi}^{(\ell)}(j)\right|$ , for all  $\ell \in [2^{b-1}]$  and  $j \in [B]$ .
To this end, let  $y^{(\ell)} \coloneqq H_B \cdot \pi^{(\ell)}$  (so it holds that  $\pi^{(\ell)} = \frac{1}{B} H_B \cdot y^{(\ell)}$ ). Then we have
$$
\begin{array}{l} \mathbb {E} \left[ \left| \pi^ {(\ell)} (j) - \hat {\pi} ^ {(\ell)} (j) \right| \right] \stackrel {(a)} {\leq} \sqrt {\mathbb {E} \left[ \left(\pi^ {(\ell)} (j) - \hat {\pi} ^ {(\ell)} (j)\right) ^ {2} \right]} \\ = \sqrt {\mathbb {E} \left[ \left(\frac {1}{B} H _ {B} \cdot \left(y ^ {(\ell)} - \hat {y} ^ {(\ell)}\right) (j)\right) ^ {2} \right]}. \tag {14} \\ \end{array}
$$
Next, observe that due to the subsampling step, for all  $\ell \in [2^{b - 1}]$  and  $j\in [B]$
$$
\hat {y} ^ {(\ell)} (j) = \frac {B}{n} \sum_ {i = 1} ^ {n} \langle (H _ {B}) _ {j}, x _ {i} ^ {(\ell)} \rangle \cdot Z _ {i j} + N (0, \sigma^ {2}),
$$
where recall that  $Z_{ij} \stackrel{\mathrm{i.i.d.}}{\sim} \operatorname{Ber}(1/B)$ . Therefore,  $\hat{y}^{(\ell)}(j)$  is an unbiased estimator of  $y^{(\ell)}(j)$ . In addition, since we choose  $Z_{ij}$  independently in Algorithm 3,  $\hat{y}^{(\ell)}(j)$ 's are independent for different  $j$ 's, so we have
$$
\begin{array}{l} \mathbb {E} \left[ \left(\hat {y} ^ {(\ell)} (j) - y ^ {(\ell)} (j)\right) ^ {2} \right] = \operatorname {V a r} \left(\hat {y} ^ {(\ell)} (j)\right) \\ = \sigma^ {2} + \frac {B ^ {2}}{n ^ {2}} \sum_ {i = 1} ^ {n} \left\langle \left(H _ {B}\right) _ {j}, x _ {i} ^ {(\ell)} \right\rangle^ {2} \mathsf {V a r} \left(Z _ {i j}\right) \\ \leq \sigma^ {2} + \frac {B}{n ^ {2}} \sum_ {i = 1} ^ {n} \langle (H _ {B}) _ {j}, x _ {i} ^ {(\ell)} \rangle^ {2} \\ = \sigma^ {2} + \frac {B}{n ^ {2}} \underbrace {\sum_ {i = 1} ^ {n} \mathbb {1} _ {\{x _ {i} \in \ell - \text {t h c h u n k} \}}} _ {:= C _ {\ell}}, \tag {15} \\ \end{array}
$$
and for all  $j \neq j'$
$$
\mathbb {E} \left[ \left(\hat {y} ^ {(\ell)} (j) - y ^ {(\ell)} (j)\right) \cdot \left(\hat {y} ^ {(\ell)} \left(j ^ {\prime}\right) - y ^ {(\ell)} \left(j ^ {\prime}\right)\right) \right] = 0. \tag {16}
$$
Therefore, we continue bounding (14) as follows:
$$
\begin{array}{l} \sqrt {\mathbb {E} \left[ \left(\frac {1}{B} H _ {B} \cdot (y ^ {(\ell)} - \hat {y} ^ {(\ell)}) (j)\right) ^ {2} \right]} = \sqrt {\frac {1}{B ^ {2}} \mathbb {E} \left[ \langle (H _ {B}) _ {j} , (\hat {y} ^ {(\ell)} - y ^ {(\ell)}) \rangle^ {2} \right]} \\ = \sqrt {\frac {1}{B ^ {2}} \mathbb {E} \left[ \left(\sum_ {k = 1} ^ {B} (H _ {B}) _ {j k} \cdot (\hat {y} ^ {(\ell)} (k) - y ^ {(\ell)} (k))\right) ^ {2} \right]} \\ \stackrel {\mathrm {(a)}} {=} \sqrt {\frac {1}{B ^ {2}} \mathbb {E} \left[ \sum_ {k = 1} ^ {B} \left(\hat {y} ^ {(\ell)} (k) - y ^ {(\ell)} (k)\right) ^ {2} \right]} \\ \stackrel {\mathrm {(b)}} {=} \sqrt {\frac {C _ {\ell}}{n ^ {2}} + \frac {\sigma^ {2}}{B}} \\ \stackrel {\mathrm {(c)}} {\leq} \sqrt {\frac {1}{n} + \frac {\sigma^ {2}}{B}}, \\ \end{array}
$$
where (a) holds since each entry of  $H_{B}$  takes value in  $\{-1, 1\}$  and by (16), (b) holds due to (15), and (c) holds because  $C_{\ell} \leq n$  for all  $\ell$ .
Finally, to bound the  $\ell_2^2$  error, observe that the above analysis ensures that
$$
\mathbb {E} \left[ \left(\pi^ {(\ell)} (j) - \hat {\pi} ^ {(\ell)} (j)\right) ^ {2} \right] \leq \frac {C _ {\ell (j)}}{n ^ {2}} + \frac {\sigma^ {2}}{B},
$$
where  $\ell(j) \in [2^{b-1}]$  is the index of the chuck containing  $j$ . Therefore, summing over  $j \in [d]$ , we must have
$$
\mathbb {E} \left[ \left\| \pi^ {(\ell)} - \hat {\pi} ^ {(\ell)} \right\| _ {2} ^ {2} \right] \leq \sum_ {j = 1} ^ {d} \frac {C _ {\ell (j)}}{n ^ {2}} + \frac {d \sigma^ {2}}{B} = \frac {B}{n} + \frac {d \sigma^ {2}}{B},
$$
$$
\sum_ {j} C _ {\ell (j)} = \sum_ {\ell = 1} ^ {2 ^ {b - 1}} \sum_ {j ^ {\prime} \in \ell - \text {t h c h u n k}} \sum_ {i = 1} ^ {n} \mathbb {1} _ {\{i \in \ell - \text {t h c h u n k} \}} = B \sum_ {\ell = 1} ^ {2 ^ {b - 1}} \sum_ {i = 1} ^ {n} \mathbb {1} _ {\{i \in \ell - \text {t h c h u n k} \}} = B \cdot n.
$$
E Proof of Theorem A.2
Let  $f_{j}(x^{n}) \coloneqq (\pi^{(1)}(j),\dots,\pi^{(2^{b - 1})}(j))$ , for  $j = 1,\ldots,B$ . Then the  $\ell_2$  sensitivity of  $f_{j}$  is  $\Delta = \frac{B}{n}$ . Set the sampling rate  $\gamma = \frac{1}{B}$  and the proof is complete by Lemma B.4.
F Algorithm of Shuffled SQKR
Algorithm 4 Shuffled SQKR
Input: users' data  $x_{1},\ldots ,x_{n}$  , local-DP parameter  $\varepsilon_0$  , communication parameters  $b_{0},T$    
Output: mean estimator  $\hat{\mu}$    
for round  $k\in [T]$  do   
for user  $i\in [n]$  do Sample  $s(i,1),\dots,s(i,b_0)\stackrel {\mathrm{i.i.d.}}{\sim}\mathrm{Unif}[d]$  Sample  $Z\sim \mathrm{Bern}\left(\frac{e^{\varepsilon_0}}{e^{\varepsilon_0} + 2^{b_0} - 1}\right)$  if Z=1 then Set  $Y(i,1),\ldots ,Y(i,b_0)\gets x_i(s(i,1)),\ldots ,x_i(s(i,b_0))$  else Sample  $Y(i,1),\ldots ,Y(i,b_0)\stackrel {\mathrm{i.i.d.}}{\sim}\mathrm{Unif}\{-c,c\}$  end if Send  $Y(i,1),\dots,Y(i,b_0)$  and  $s(i,1),\dots,s(i,b_0)$  to shuffler   
end for   
Shuffler samples a permutation  $\pi \sim \mathrm{Unif}\{f:[n]\to [n]$  bijective}   
for  $j\in [b_0]$  do Shuffler sends  $Y(\pi (1),j),\dots,Y(\pi (n),j)$  and  $s(\pi (1),j),\dots,s(\pi (n),j)$  to server   
end for  $\hat{\mu}^{(k)}\gets \frac{d}{nb_0}\frac{e^{\varepsilon_0} + 2^{b_0} - 1}{e^{\varepsilon_0} - 1}\sum_{i = 1}^{n}\sum_{j = 1}^{b_0}Y(\pi (i),j)e_{s(\pi (i),j)}$    
end for   
Return  $\hat{\mu} := \frac{1}{T}\sum_{k = 1}^{T}\hat{\mu}^{(k)}$
G Proof of Theorem 5.3
Each round  $x^n \mapsto \hat{\mu}^{(k)}$  of Algorithm 4 implements the private-coin SQKR scheme of [32], achieving the communication cost and error as stated in Lemma 5.2.
Lemma G.1 (SQKR [32]). For all  $\varepsilon_0 > 0, b_0 > 0$ , the random mapping  $x_i \mapsto Y(i,1), \ldots, Y(i,b_0), s(i,1), \ldots, s(i,b_0)$  in Algorithm 4 is  $(\varepsilon_0,0)$ -LDP and has output that can be communicated in  $b_0\log(d)$  bits, and the  $\hat{\mu}^{(k)}$  computed from  $Y(i,1), \ldots, Y(i,b_0), s(i,1), \ldots, s(i,b_0)$  is an unbiased estimator of  $\mu$  satisfying
$$
\max  _ {x ^ {n}} \mathbb {E} \left[ \left\| \mu (x ^ {n}) - \hat {\mu} ^ {(k)} (x ^ {n}) \right\| _ {2} ^ {2} \right] = O \left(\frac {C ^ {2} d}{n \min  \left(\varepsilon_ {0} ^ {2} , \varepsilon_ {0} , b _ {0}\right)}\right). \tag {17}
$$
We now characterize the error performance of Algorithm 4 for general choices of parameters that satisfy privacy and communication constraints.
Proposition G.2. For all  $\varepsilon > 0, b > 0, n > 0$ , with any arbitrary choice of
$$
\delta_ {1} \in \left(e ^ {- n / 1 6 e}, 1 \right] \tag {18}
$$
$$
\delta_ {2} \in (0, 1 ], \tag {19}
$$
there exists a choice of parameters  $\varepsilon_0, b_0, T$  such that Algorithm 4 is  $(\varepsilon, T\delta_1 + \delta_2)$ -DP, uses no more than  $b$  bits of communication, and produces  $\hat{\mu}$  such that
$$
\left. \max  _ {x ^ {n}} \mathbb {E} \left[ \| \mu - \hat {\mu} \| _ {2} ^ {2} \right] = O \left(\max  \left(\frac {C ^ {2} d \log (d) b _ {0}}{n b}, \frac {C ^ {2} d \log \left(1 / \delta_ {1}\right) (\log \left(1 / \delta_ {2}\right) + \varepsilon)}{n ^ {2} \varepsilon^ {2}}\right)\right). \right. \tag {20}
$$
Proof. For arbitrary choice of
$$
b _ {0} <   \log \left(\frac {n}{1 6 \log (2)}\right), \tag {21}
$$
it suffices to choose
$$
T = \left\lfloor \frac {b}{\left(\log_ {2} (d) + 1\right) b _ {0}} \right\rfloor \tag {22}
$$
$$
\varepsilon_ {0} = O \left(\min  \left(1, \frac {\varepsilon \sqrt {n}}{\sqrt {T \log \left(1 / \delta_ {1}\right) (\log \left(1 / \delta_ {2}\right) + \varepsilon)}}\right)\right). \tag {23}
$$
The fact that Algorithm 4 uses less than  $b$  bits is immediate from the choice of  $T$ .
Applying Lemma G.1, by construction the mapping from each  $x_{i}$  to  $Y(i,1),\ldots ,Y(i,b_0)$  is  $(\varepsilon_0,0)$ -LDP. By assumption
$$
\delta_ {1} > e ^ {- n / 1 6 e}, \tag {24}
$$
the inequality
$$
1 <   \log \left(\frac {n}{1 6 \log \left(2 / \delta_ {1}\right)}\right) \tag {25}
$$
is satisfied. Then the choice of
$$
\varepsilon_ {0} \leq 1 \tag {26}
$$
also satisfies  $\varepsilon_0\leq \log \left(\frac{n}{16\log(2 / \delta)}\right)$ , so by Lemma 5.1 the mapping  $x^n\mapsto \hat{\mu}^{(k)}$  is  $(\varepsilon_{1},\delta_{1})$ -DP, where
$$
\varepsilon_ {1} = O \left(\frac {\varepsilon_ {0} \sqrt {\log \left(1 / \delta_ {1}\right)}}{\sqrt {n}}\right). \tag {27}
$$
Since the output of Algorithm 4 is a function of  $(\hat{\mu}^{(1)},\dots ,\hat{\mu}^{(T)})$  , by B.3 it suffices to have
$$
\varepsilon_ {1} = O \left(\min  \left(1, \frac {\varepsilon}{\sqrt {T \left(\log \left(1 / \delta_ {2}\right) + \varepsilon\right)}}\right)\right) \tag {28}
$$
for Algorithm 4 to be  $(\varepsilon, T\delta_{1} + \delta_{2})$ -DP. The first inequality follows from the assumption of  $\delta_{1} > e^{-n / 16e}$  and choice of  $\varepsilon_0 = O(1)$ , and the second from choice of
$$
\varepsilon_ {0} = O \left(\frac {\varepsilon \sqrt {n}}{\sqrt {T \log \left(1 / \delta_ {1}\right) (\log \left(1 / \delta_ {2}\right) + \varepsilon)}}\right). \tag {29}
$$
Since  $\varepsilon_0\leq 1\leq b$  we have  $\min (\varepsilon_0^2,\varepsilon_0,b) = \varepsilon_0^2$  Applying Lemma G.1,
$$
\begin{array}{l} \max  _ {x ^ {n}} \mathbb {E} \left[ \| \mu - \hat {\mu} \| _ {2} ^ {2} \right] = \frac {1}{T} \max  _ {x ^ {n}} \mathbb {E} \left[ \left\| \mu - \hat {\mu} ^ {(1)} \right\| _ {2} ^ {2} \right] (30) \\ = O \left(\frac {d}{T n \varepsilon_ {0} ^ {2}}\right) (31) \\ = O \left(\max  \left(\frac {d}{T n}, \frac {d \log \left(1 / \delta_ {1}\right) (\log \left(1 / \delta_ {2}\right) + \varepsilon)}{n ^ {2} \varepsilon^ {2}}\right)\right). (32) \\ \end{array}
$$
Substituting the choice of  $T$  gives the desired result.
To show Theorem 5.3, it suffices to choose
$$
b _ {0} = 1 \tag {33}
$$
$$
\delta_ {1} = \frac {\delta}{2 T} \tag {34}
$$
$$
\delta_ {2} = \frac {\delta}{2}, \tag {35}
$$
which requires  $n > 16e\log (2)\approx 30.14$  due to (21), and apply the previous proposition.
H Using Kashin's Representation