SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
O \left(\underbrace {\frac {d C ^ {2}}{n b} + \frac {C ^ {2} d ^ {2} \log (1 / \delta)}{n ^ {2} b ^ {2}}} _ {(\alpha)} + \underbrace {\frac {C ^ {2} d (\log (d / \delta) + \varepsilon) \log (d / \delta)}{n ^ {2} \varepsilon^ {2}}} _ {(\beta)}\right). \tag {4}
$$
The first term  $(\alpha)$  in the estimation error in Corollary 4.3 is the error due to compression, and the second term  $(\beta)$  is the error due to privatization (which is order-optimal under  $(\varepsilon, \delta)$ -DP up to an additional  $\log(d / \delta)$  factor as we discuss in Section 4.2). In particular, if we ignore the polylogarithmic terms and assume  $\varepsilon = O(1)$ , the privatization error  $(\beta)$  can be simplified to  $\tilde{O}\left(\frac{dC^2}{n^2\varepsilon^2}\right)$ , which dominates the total  $\ell_2^2$  error when  $b = \tilde{\Omega}_{\delta}\left(\max\left(n\varepsilon^2, \sqrt{d}\varepsilon\right)\right)$ , i.e. in this regime the total  $\ell_2^2$  error is order-wise equal to the optimal centralized DP error  $(\beta)$ . This implies that no more than  $b = \tilde{\Omega}_{\delta}\left(\max\left(n\varepsilon^2, \sqrt{d}\varepsilon\right)\right)$  bits per client are needed to achieve the order-optimal  $\ell_2^2$  error.
In the next section, we introduce a modification to Algorithm 1, which allows the removal of the  $\Omega\left(\sqrt{d}\varepsilon\right)$  term in the communication cost.
4.1 Dimension-free communication cost
In order to remove the dependence on the dimension  $d$  in the communication cost  $b = \tilde{\Omega}_{\delta}\left(\max \left(n\varepsilon^{2},\sqrt{d}\varepsilon\right)\right)$  from the previous section, we need to improve the performance of our scheme in the small-sample regime  $n\varepsilon^2 = o(\sqrt{d}\varepsilon)$ . Equivalently, we want to be able to achieve the centralized DP performance by using only  $b = \tilde{\Omega}_{\delta}\left(n\varepsilon^{2}\right)$  bits per client when  $n\varepsilon = o(\sqrt{d})$ . Assuming  $\varepsilon \approx 1$ , note that this implies that the total communication bandwidth of the system  $nb = o(d)$ , i.e. the server can receive information about at most  $nb = n^2\varepsilon^2 = o(d)$  coordinates. We show that in this regime the performance of the scheme can be improved by a priori restricting the server's attention to a subset of the coordinates.
We make the following modification to Algorithm 1: before performing Algorithm 1, the server randomly selects  $d' \approx O\left(\min(d, n^2 \varepsilon^2)\right)$  coordinates and only requires clients to run Algorithm 1 on them. We present the modified scheme in Algorithm 2 and summarize its performance in Theorem 4.4.
Similarly, we can obtain the following  $\ell_2$  mean estimation via Kashin's representations:
Theorem 4.4 ( $\ell_2$  mean estimation.). Let  $x_{1},\ldots ,x_{n}\in \mathcal{B}_{2}(C)$  (i.e.,  $\| x_i\| _2\leq C$  for all  $i\in [n]$ ),  $\varepsilon, \delta >0$ , and  $d^{\prime} = \min \left(d,nb,\frac{n^{2}\varepsilon^{2}}{(\log(1 / \delta) + \varepsilon)\log(d / \delta)}\right)$ . Then there exists a
$$
\sigma^ {2} = O \left(\frac {C ^ {2} \log (1 / \delta)}{d ^ {\prime} n ^ {2} \gamma^ {2}} + \frac {C ^ {2} d ^ {\prime} (\log (1 / \delta) + \varepsilon) \log \left(d ^ {\prime} / \delta\right)}{d n ^ {2} \varepsilon^ {2}}\right). \tag {5}
$$
such that Algorithm 2 is  $(\varepsilon, \delta)$ -DP. In addition, the (average) per-client communication cost is  $\gamma d = b$  bits, and the  $\ell_2^2$  estimation error is at most
$$
O \left(\max  \left(\frac {C ^ {2} d \log (d / \delta)}{n b}, \frac {C ^ {2} d \log (d / \delta) (\log (1 / \delta) + \varepsilon)}{n ^ {2} \varepsilon^ {2}}\right)\right). \tag {6}
$$
Corollary 4.5. As long as  $b = \Omega \left( \frac{n \varepsilon^2}{\log(1 / \delta) + \varepsilon} \right)$ , the  $\ell_2^2$  error of mean estimation is
$$
O \left(\frac {C ^ {2} d \log (d / \delta) (\log (1 / \delta) + \varepsilon)}{n ^ {2} \varepsilon^ {2}}\right).
$$
As suggested by Corollary 4.5, we see that when  $\varepsilon = O(1)$ ,  $b = \tilde{\Omega}\left(n\varepsilon^{2}\right)$  bits per client are sufficient to achieve the order-optimal  $\tilde{O}_{\delta}\left(\frac{c^{2}d}{n^{2}\varepsilon^{2}}\right)$  error (even in the small sample regime  $n \leq \sqrt{d}$ ), i.e. the communication cost of the scheme is independent of the dimension  $d$ .
4.2 Lower bounds
In this section, we argue that the estimation error in Theorem 4.4 is optimal up to an  $\log (d / \delta)$  factor. Specifically, Theorem 5.3 of [31] shows that any  $b$ -bit unbiased compression scheme will incur  $\Omega \left(\frac{C^2d}{nb}\right)$  error for the  $\ell_2$  mean estimation problem (even when privacy is not required). This matches the first term in (6) up to a logarithmic factor.
On the other hand, the centralized Gaussian mechanism (under a central  $(\varepsilon ,\delta)$  -DP) achieves  $O\left(\frac{C^2d\log(1 / \delta)}{n^2\varepsilon^2}\right)$  MSE [15] which is orderoptimal in most parameter regimes; see the lower bounds in Theorem 3.1 of [28] or Proposition 23 of [30]). Hence, we can conclude that
Algorithm 2 CSGM with Coordinate Pre-selection
Input: users' data  $x_{1},\ldots ,x_{n}$  coordinate selection  $d^{\prime}\leq d$  sampling parameters  $\gamma \coloneqq b / d^{\prime}$  DP parameters  $(\varepsilon ,\delta)$
Randomly select  $d^{\prime}$  coordinates  $\mathcal{I}\coloneqq$ $\{j_1,\dots ,j_{d'}\} \subset [d]$
Pre-processing  $x_{i}$  by restricting it on  $\mathcal{J}$ :
$$
x _ {i} (\mathcal {J}) := \left(x _ {i} \left(j _ {1}\right), \dots , x _ {i} \left(j _ {\mid \mathcal {J} \mid}\right)\right).
$$
Apply CSGM (Algorithm 1) on  $x_{i}(\mathcal{J}), i \in [n]$ :  $\hat{\mu}_{\mathcal{J}} \gets \mathrm{CSGM}(x_i(\mathcal{J}), i \in [n])$ .
for  $j\in [d]$  do
$$
i f j \in \mathcal {J} \text {t h e n}
$$
$$
\hat {\mu} _ {j} = \hat {\mu} _ {\mathcal {I}} (j).
$$
$$
\hat {\mu} _ {j} = 0.
$$
Return:  $\hat{\mu} \coloneqq \left( \frac{d}{d'} \hat{\mu}_1, \frac{d}{d'} \hat{\mu}_2, \dots, \frac{d}{d'} \hat{\mu}_d \right)$ .
the total communication received by the server has to be at least  $\Omega (n^{2}\varepsilon^{2})$  bits in order to achieve the same error as the Gaussian mechanism. Therefore, the (average) per-client communication cost has to be at least  $\Omega (n\varepsilon^2)$  bits. Hence we conclude that Algorithm 2 is optimal (up to a logarithmic factor).
For completeness, we state the communication lower bound in the following theorem:
Theorem 4.6 (Communication lower bound for mean estimation under central DP). Let  $x_{1}, \ldots, x_{n} \in \mathcal{B}_{2}(C)$ . Let  $Y_{1}, \ldots, Y_{n}$  be any  $b$ -bit local reports generated from a (possibly interactive) compressor and be unbiased in the sense that  $\mathbb{E}\left[\sum_{i}Y_{i}\right] = \sum_{i}x_{i}$ . Then if  $\mathbb{E}\left[\left\| \frac{1}{n}\sum_{i}Y_{i} - \frac{1}{n}\sum_{i}x_{i}\right\|_{2}^{2}\right] \leq O\left(\frac{C^{2}d\log(1 / \delta)}{n^{2}\varepsilon^{2}}\right)$ , it holds that  $b = \Omega \left(\frac{n\varepsilon^2}{\log(1 / \delta)}\right)$ .
Finally, we remark that the logarithmic gap between the upper and lower bounds may be due to the specific composition theorem (Theorem III.3 of [39]) we use in our proof, which is simpler to work with but possibly slightly weaker. However, in our experiments, we compute and account for all privacy budgets with Rényi DP [69, 82], and hence can obtain better constants compared to our theoretical analysis.
5 Achieving the Optimal Trade-off via Shuffling
In Section 4 and Section A, we see that the communication cost can be reduced to  $(\tilde{O}\left(n\varepsilon^{2}\right)$  for mean estimation and  $\tilde{O}\left(\log \left(\lceil n\varepsilon^{2}\rceil\right)\right)$  for frequency estimation) while still achieving the order-wise optimal error, as long as the server is trusted. On the other hand, when the server is untrusted, [33, 31] show that optimal error under  $(\varepsilon ,\delta)$ -DP can be achieved with secure aggregation. However, the communication cost of these schemes is  $\tilde{O}\left(n^{2}\varepsilon^{2}\right)$  bits per client for mean estimation and  $\tilde{O}\left(n\varepsilon\right)$  bits per client for frequency estimation. This corresponds to a factor of  $n$  increase for mean estimation and an exponential increase for frequency estimation. In this section, we investigate whether the
optimal communication-accuracy-privacy trade-off from the previous sections can be achieved when the server is not fully trusted.
In this section, we show that if there exists a secure shuffler that randomly permutes clients' locally privatized messages and releases them to the server, we can achieve the nearly optimal (within a log  $d$  factor) central-DP error in mean estimation with  $\tilde{O}\left(n\varepsilon^{2}\right)$  bits of communication. We note that a similar result has been proven in a concurrent work [54]. Specifically, we present a mean estimation scheme that combines a local-DP mechanism with privacy amplification via shuffling by building on the following recent result [40, 43]:
Lemma 5.1 ([43]). Let  $\mathcal{M}_i$  be an independent  $(\varepsilon_0,0)$ -LDP mechanism for each  $i \in [n]$  with  $\varepsilon_0 \leq 1$  and  $\pi$  be a random permutation of  $[n]$ . Then for any  $\delta \in [0,1]$  such that  $\varepsilon_0 \leq \log \left(\frac{n}{16\log(2 / \delta)}\right)$ , the mechanism  $S: (x_1, \ldots, x_n) \mapsto (\mathcal{M}_1(x_{\pi(1)}), \ldots, \mathcal{M}_n(x_{\pi(n)}))$  is  $(\varepsilon, \delta)$ -DP for some  $\varepsilon$  such that  $\varepsilon = O\left(\varepsilon_0\frac{\sqrt{\log(1 / \delta)}}{\sqrt{n}}\right)$ .
Privacy analysis. With the above amplification lemma, we only need to design the local randomizers  $\mathcal{M}_i$  that satisfy  $\varepsilon_0$ -LDP. Note that the above lemma is only tight when  $\varepsilon_0 = O(1)$ , thus restricting the (amplified) central  $\varepsilon = O(1 / \sqrt{n})$ , i.e. to be very small. To accommodate larger  $\varepsilon$ , users can send different portions of their messages to the server in separate shuffling rounds. Equivalently, we repeat the shuffled LDP mechanism for  $T = O\left(\lceil n\varepsilon^2\rceil\right)$  rounds while ensuring that in each round, clients communicate an independent piece of information about their sample to the server. More precisely, within each round, each client applies the local randomizers  $\mathcal{M}_i$  with a per-round local privacy budget  $\varepsilon_0 = O(1)$  and sends an independent message to the server. This results in (amplified) central  $O(1 / \sqrt{n})$ -DP per round, which after composition over  $T = O\left(\lceil n\varepsilon^2\rceil\right)$  rounds leads to  $\varepsilon$ -DP for the overall scheme as suggested by the composition theorem [58]). We detail the algorithm in Algorithm 4 in Appendix F.
Communication costs. The communication cost of the above  $T$ -round scheme can be computed as follows. As shown in [32], the optimal communication cost of an  $\varepsilon_0$ -LDP mean estimation is  $O\left(\lceil \varepsilon_0\rceil\right)$  bits. In addition, the (private-coin) SQKR scheme proposed in [32] uses  $O\left(\lceil \varepsilon_0\rceil\log d\right)$  bits of communication (we state the formal performance guarantee for this scheme in Lemma 5.2), where compression is done by subsampling coordinates and privatization is performed with Randomized Response. Therefore, since the per-round  $\varepsilon_0 = O(1)$ , the total per-client communication cost is  $O\left(n\varepsilon^2\log d\right)$ , matching the optimal communication bounds in Section 4 within a log  $d$  factor.
Lemma 5.2 (SQKR [32]). For all  $\varepsilon_0 > 0, b_0 > 0$ , there exists a  $(\varepsilon_0, 0)$ -LDP mechanism using  $b_0 \log(d)$  bits such that  $\hat{\mu}$  is unbiased and satisfies  $\mathbb{E}\left[\|\mu(x^n) - \hat{\mu}(x^n)\|_2^2\right] = O\left(\frac{c^2 d}{n \min\left(\varepsilon_0^2, \varepsilon_0, b_0\right)}\right)$ .
Finally, we summarize the performance guarantee for the overall scheme (Algorithm 4) in the following theorem.
Theorem 5.3 ( $\ell_2$  mean estimation). Let  $x_1, \ldots, x_n \in \mathcal{B}_2(C)$  (i.e.,  $\|x_i\|_2 \leq C$  for all  $i \in [n]$ ). For all  $\varepsilon > 0, b > 0, n > 30$ , and  $\delta \in (\delta_{\min}, 1]$  where  $\delta_{\min} = O\left(\frac{be^{-n}}{\log(d)}\right)$ , Algorithm 4 combined with Kashin's representation and randomized rounding is  $(\varepsilon, \delta)$ -DP, uses no more than  $b$  bits of communication, and achieves
$$
\mathbb {E} \left[ \| \mu (x ^ {n}) - \hat {\mu} (x ^ {n}) \| _ {2} ^ {2} \right] = O \left(C ^ {2} d \max  \left(\frac {\log (d)}{n b}, \frac {\log (b / \delta) (\log (1 / \delta) + \varepsilon)}{n ^ {2} \varepsilon^ {2}}\right)\right).
$$
Remark 5.4. As opposed to previous schemes Algorithm 1-3, the shuffled SQKR requires some condition on  $\delta$ , i.e.,  $\delta \in [\delta_{\min}, 1]$  due to the specific shuffling lemma we used. In practice, however,  $\delta_{\min}$  is small due to the exponential dependence on  $n$ . The order-wise optimal error of  $O\left(\frac{C^2d}{n^2\min(\varepsilon^2,\varepsilon)}\right)$  is achieved, up to logarithmic factors, when  $b = \Omega_{\delta}$  ( $n\log(d)\min(\varepsilon^2,\varepsilon)$ ).
Remark 5.5. We note that similar ideas of private mean estimation based on shuffling have been studied before, see, for instance, [53]. However, these papers do not use the above privacy budget splitting trick over multiple rounds, so their result is only optimal when  $\varepsilon$  is very small. The above scheme can be viewed as a multi-message shuffling scheme [35, 51], and in particular, can be regarded as a generalization of the scalar mean estimation scheme [35] to  $d$ -dim mean estimation.
Figure 1: MSEs of CSGM (Algorithm 1) and shuffle LDP schemes.
In this section, we empirically evaluate our mean estimation scheme (CSGM) from Section 4, examine its privacy-accuracy-communication trade-off, and compare it with other DP mechanisms (including the shuffling-based mechanism introduced in Section 5).
Setup. For a given dimension  $d$ , and number of samples  $n$ , we generate local vectors  $X_{i} \in \mathbb{R}^{d}$  as follows: let  $X_{i}(j) \stackrel{\mathrm{i.i.d.}}{\sim} \frac{1}{\sqrt{d}} (2 \cdot \operatorname{Ber}(0.8) - 1)$  where  $\operatorname{Ber}(0.8)$  is a Bernoulli random variable with bias  $p = 0.8$ . This ensures  $\| X_{i}\|_{\infty} \leq 1 / \sqrt{d}$  and  $\| X_{i}\|_{2} \leq 1$ , and in addition, the empirical mean  $\mu(X^n) \coloneqq \frac{1}{n} \sum_{i} X_i$  does not converge to 0. Note that as our goal is to construct an unbiased estimator, we did not project our final estimator back to the  $\ell_{\infty}$  or  $\ell_2$  space as the projection step may introduce bias. Therefore, the  $\ell_2$  estimation error can be greater than 1. We account for the privacy budget with Rényi DP [69] and the privacy-amplification by subsampling lemma in [82] and convert Rényi DP to  $(\varepsilon, \delta)$ -DP via [30].
Privacy-accuracy-communication trade-off of CSGM. In the first experiment (left of Figure 1), we apply Algorithm 1 with different sampling rates  $\gamma$ , which leads to different communication budgets ( $b = \gamma d$ ). Note that when  $\gamma = 1$ , the scheme reduces to the central Gaussian mechanism without compression. In Figure 1, we see that with a fixed communication budget, CSGM approximates the central (uncompressed) Gaussian mechanism in the high privacy regime (small  $\varepsilon$ ) and starts deviating from it when  $\varepsilon$  exceeds a certain value. In addition, that value of  $\varepsilon$  depends only on sample size  $n$  and the communication budget  $b$  and not the dimension  $d$  as predicted by our theory: recall that the compression error dominates the total error, and hence the performance starts to deviate from the (uncompressed) Gaussian mechanism when  $b = o(n\varepsilon^2)$ , a condition that is independent of  $d$ . Observe, for example, that when  $b = 50$  bits, the Gaussian mechanism starts outperforming CSGM at  $\varepsilon \geq 0.5$  for both  $d = 500$  and  $d = 5000$ . Hence, for  $\varepsilon \approx 0.5$  CSGM is able to provide 10X compression when  $d = 500$ , but 100X compression when  $d = 5000$  without impacting MSE.
Comparison with local and shuffle DP. Next, we compare the CSGM with local and shuffled DP for  $d = 10^3$  and  $n = 500$ . For local DP, we consider the private-coin SQKR scheme introduced in Section 5 which uses  $(\lceil \log d \rceil + 1) T = 11T$  bits for  $T$  shuffling rounds and DJW [37] which is known to be order-optimal when  $\varepsilon = O(1)$  (but is not communication-efficient). For shuffle-DP, we apply the amplification lemma in [43] to find the corresponding local  $\varepsilon_0$  (see Section 5 for more details) and simulate both SQKR and DJW as the local randomizers.
The MSEs of all mechanisms are reported in the right of Figure 1. Our results suggest that for a fixed communication budget (say, 10 bits), the practical performance of CSGM significantly outperforms shuffled-DP mechanisms, including the shuffled SQKR and DJW, even though they have the same order-wise guarantees theoretically. In addition, the amplification gain of single-round shuffling diminishes fast as  $\varepsilon$  increases. Indeed, when  $\varepsilon \geq 0.8$ , we observe no amplification gain compared to the pure local DP.
While we have characterized the (nearly) orderwise optimal trade-offs under central and shuffled DP, it remains unclear whether the pre-constants can be further sharpened, potentially enhancing the practical applicability of these approaches. Furthermore, our analysis is based on a classical privacy amplification technique. By employing more advanced accounting methods, it might be possible to shave off the logarithmic factors.
This work was supported in part by NSF Award # CCF-2213223.
[73] C. Studer, W. Yin, and R. G. Baraniuk. Signal representations with minimum  $\ell_{\infty}$ -norm. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1270-1277. IEEE, 2012.
[74] A. T. Suresh, F. X. Yu, S. Kumar, and H. B. McMahan. Distributed mean estimation with limited communication. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 3329-3337. JMLR.org, 2017.
[75] R. Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[76] T. Wang, J. Zhao, X. Yang, and X. Ren. Locally differentially private data collection and analysis. arXiv preprint arXiv:1906.01777, 2019.
[77] J. Wangni, J. Wang, J. Liu, and T. Zhang. Gradient sparsification for communication-efficient distributed optimization. In Advances in Neural Information Processing Systems, pages 1299-1309, 2018.
[78] S. L. Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309):63-69, 1965.
[79] W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in neural information processing systems, pages 1509-1519, 2017.
[80] M. Ye and A. Barg. Optimal schemes for discrete distribution estimation under local differential privacy. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 759-763, June 2017.
[81] W. Zhu, P. Kairouz, B. McMahan, H. Sun, and W. Li. Federated heavy hitters discovery with differential privacy. In International Conference on Artificial Intelligence and Statistics, pages 3837-3847. PMLR, 2020.
[82] Y. Zhu and Y.-X. Wang. Poisson subsampled Rényi differential privacy. In International Conference on Machine Learning, pages 7634-7642. PMLR, 2019.
A Distributed Frequency Estimation
In this section, we consider the frequency estimation problem for federated analytics. Recall that for the frequency estimation task, each client's private data  $x_{i} \in \{0,1\}^{d}$  satisfies  $\| x_{i}\|_{0} = 1$ , and the goal is to estimate  $\pi \coloneqq \frac{1}{n}\sum_{i}x_{i}$  by minimizing the  $\ell_2$  (or  $\ell_1,\ell_\infty$ ) error  $\mathbb{E}\left[\| \pi -\hat{\pi} (Y^n)\| _2^2\right]$  subject to communication and  $(\varepsilon ,\delta)$ -DP constraints. When the context is clear, we sometimes use  $x_{i}$  to denote, by abuse of notation, the index of the item, i.e.,  $x_{i} \in [d]$ .
To fully make use of the  $\ell_0$  structure of the problem, a standard technique is applying a Hadamard transform to convert the  $\ell_0$  geometry to an  $\ell_{\infty}$  one and then leveraging the recursive structure of Hadamard matrices to efficiently compress local messages.
Specifically, for a given  $b$ -bit constraint, we partition each local item  $x_{i}$  into  $2^{b-1}$  chunks  $x_{i}^{(1)}, \ldots, x_{i}^{(2^b - 1)} \in \{0, 1\}^B$ , where  $B := d / 2^{b-1}$  and  $x_{i}^{(j)} = x_{i}[B \cdot (j - 1) : B \cdot j - 1]$ . Note that since  $x_{i}$  is one-hot, only one chunk of  $x_{i}^{(j)}$  is non-zero. Then, client  $i$  performs the following Hadamard transform for each chunk:  $y_{i}^{(\ell)} = H_{B} \cdot x_{i}^{(\ell)}$ , where  $H_{B}$  is defined recursively as follows:
$$
H _ {2 ^ {n}} = \frac {1}{\sqrt {2}} \left[ \begin{array}{c c} H _ {2 ^ {n - 1}}, & H _ {2 ^ {n - 1}} \\ H _ {2 ^ {n - 1}}, & - H _ {2 ^ {n - 1}} \end{array} \right], \text {a n d} H _ {0} = [ 1 ].
$$
Each client then generates a sampling vector  $Z_{ij} \stackrel{\mathrm{i.i.d.}}{\sim} \operatorname{Bern}\left(\frac{1}{B}\right)$  via shared randomness that is also known by the server, and commits  $(y_i^{(1)}(j), \dots, y_i^{(2^b - 1)}(j))$  as its local report. Since  $(y_i^{(1)}(j), \dots, y_i^{(2^b - 1)}(j))$  only contains a single non-zero entry that can be  $\frac{1}{\sqrt{B}}$  or  $-\frac{1}{\sqrt{B}}$ , the local report can be represented in  $b$  bits ( $b - 1$  bits for the location of the non-zero entry and 1 bit for its sign).
From the local reports, the server can compute an unbiased estimator by summing them together (with proper normalization) and performing an inverse Hadamard transform. Moreover, with an adequate injection of Gaussian noise, the frequency estimator satisfies  $(\varepsilon, \delta)$ -DP.
The idea has been used in previous literature under local DP [19, 6, 3, 32], but in order to obtain the order-optimal trade-off under central-DP, one has to combine Hadamard transform with a random subsampling step and incorporate the privacy amplification due to random compression in the analysis. In Algorithm 3, we provide a summary of the resultant scheme which builds on the Recursive Hadamard Response (RHR) mechanism from [32], which was originally designed for communication-efficient frequency estimation under local DP.
In the following theorem, we control the  $\ell_{\infty}$  error of Algorithm 3.
Theorem A.1. Let  $\hat{\pi} (x^n)$  be the output of Algorithm 3. Then it holds that for all  $j\in [d]$
$$
\mathbb {E} \left[ | \pi (j) - \hat {\pi} (j) | \right] \leq \sqrt {\frac {\sum_ {i} \mathbb {1} _ {\{x _ {i} \in [ B \cdot (j - 1) : B \cdot j - 1 ] \}}}{n ^ {2}} + \frac {\sigma^ {2}}{B}}, \tag {7}
$$
and the  $\ell_2^2$  and  $\ell_{1}$  errors are bounded by
$$
\mathbb {E} \left[ \| \pi - \hat {\pi} \| _ {2} ^ {2} \right] \leq \frac {B}{n} + \frac {d \sigma^ {2}}{B}, a n d \tag {8}
$$
$$
\mathbb {E} \left[ \| \pi - \hat {\pi} \| _ {1} \right] \leq \sqrt {\frac {d B}{n} + \frac {d ^ {2} \sigma^ {2}}{B}}. \tag {9}
$$
Theorem A.2. For any  $\varepsilon, \delta > 0$ , Algorithm 3 is  $(\varepsilon, \delta)$ -DP, if
$$
\sigma^ {2} \geq O \left(\frac {B ^ {2} \log (B / \delta)}{n ^ {2}} + \frac {B (\log (1 / \delta) + \varepsilon) \log (B / \delta)}{n ^ {2} \varepsilon^ {2}}\right).
$$
By combining Theorem A.1 and Theorem A.2, we conclude that Algorithm 3 achieves  $(\varepsilon, \delta)$ -DP with  $\ell_2^2$  error