SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 2. While our randomizers in this section pay an additive term that does not decrease with  $n$  (e.g.  $\log^2(d)/k$  for SRHT), this term is negligible in most settings of interest. Indeed, using Theorem 3 and the fact that  $\mathsf{Err}_{n,d}(\mathsf{PrivUnit}G) = c_{d,\varepsilon} d/n\varepsilon$ , we get that the final error of our SRHT algorithm is roughly  $c_{d,\varepsilon} d/n\varepsilon(1 + o(1)) + O(\log^2(d)/k)$ . This implies that in the high-dimensional setting the bias term is negligible.
However, to cover the whole spectrum of parameters, we develop a nearly unbiased versions of these algorithms in Appendix A. In particular, we show in Theorem 6 that our unbiased version has error
$$
\operatorname {E r r} _ {n, d} (\operatorname {P r i v U n i t} G) \cdot \left(1 + O \left(\frac {\varepsilon + \log k}{k} + \sqrt {\frac {\log (n d / k)}{k}}\right)\right).
$$
3 Efficient Server Runtime via Correlated Sampling
One downside of the algorithms in the previous section is that server runtime can be costly: indeed, as each client uses an independent transformation, the server has to apply the inverse transformation
matrix multiplication) for each client, resulting in runtime  $O(nd\log d)$ . In this section, we propose a new protocol that significantly reduces server runtime to  $O(n\log^3 d + d\log d + nk)$  while retaining similar error guarantees. The protocol uses correlated transformations between users which allows the server to apply an inverse transformation only a small number of times. However, clients cannot use the same transformation as this will result in large bias.
The protocol works as follows: the server samples  $U \in \mathbb{R}^{d \times d}$  from the Randomized Hadamard transform:  $U = HD$  where  $H \in \mathbb{R}^{d \times d}$  is the Hadamard transform, and  $D \in \mathbb{R}^{d \times d}$  is a diagonal matrix where each diagonal entry is independently sampled from the Rademacher distribution. Then, client  $i \in [n]$ , samples a random sampling matrix  $S_{i} \in \mathbb{R}^{k \times d}$ , and uses  $U$  to define the transform  $W_{i} \in \mathbb{R}^{k \times d}$ :
$$
W _ {i} = \sqrt {\frac {d}{k}} S _ {i} U. \tag {3}
$$
We describe the full details of the client and server algorithms for correlated ProjUnit in Algorithm 3 and Algorithm 4, and denote them  $\mathcal{R}_{\mathrm{CPU}}$  and  $\mathcal{A}_{\mathrm{CPU}}$ , respectively. We have the following guarantee. We defer the proof to Appendix C.
Algorithm 3 Correlated ProjUnit (client)
Require: Input vector  $v\in \mathbb{R}^d$
1: Randomly sample diagonal  $D$  from the Rademacher distribution based on predefined seed
2: Sample  $S \in \mathbb{R}^{k \times d}$  where each row is chosen uniformly at random without replacement from standard basis vectors  $\{e_1, \ldots, e_d\}$
3: Project the input vector  $v_{p} = SHDv$  where  $H \in \mathbb{R}^{d \times d}$  is the Hadamard matrix
4:Normalize  $u = \frac{v_p}{\|v_p\|_2}$
5: Let  $\hat{u} = \mathrm{PrivUnitG}(u)$  (as in Algorithm 8)
6: Send  $\hat{u}$ , and (an encoding of)  $S$  to server
Algorithm 4 Correlated ProjUnit (server)
1:Receive  $\hat{u}_1,\dots ,\hat{u}_1$  from clients with (encodings of) transforms  $S_{1},\ldots ,S_{n}$
2: Sample diagonal matrices  $D$  from Rademacher distribution based on predefined seed
3: Let  $U = HD$  where  $H \in \mathbb{R}^{d \times d}$  is the Hadamard matrix
4: Return the estimate  $\hat{\mu} = \frac{1}{n} U^{\top}\sum_{i = 1}^{n}S_{i}^{\top}\hat{u}_{i}$
Theorem 4. Let  $k \leq d$ . Then for all unit vectors  $v_{1},\ldots ,v_{n} \in \mathbb{R}^{d}$ , setting  $\hat{\mu} = \mathcal{A}_{\mathrm{CPU}}(\mathcal{R}_{\mathrm{CPU}}(v_1),\dots,\mathcal{R}_{\mathrm{CPU}}(v_n))$ , the local randomizers  $\mathcal{R}_{\mathrm{CPU}}$  are  $\varepsilon$ -DP and
$$
\mathbb {E} \left[ \left\| \hat {\mu} - \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right\| _ {2} ^ {2} \right] \leq \operatorname {E r r} _ {n, d} (\operatorname {P r i v U n i t} G) \left(1 + O \left(\frac {\varepsilon + \log k}{k}\right)\right) + O \left(\frac {\log^ {2} d}{k}\right).
$$
Moreover, server runtime is  $O(n \log(d) \log^2(nd) + d \log d + nk)$ .
We conclude the paper with several experiments that demonstrate the performance of our proposed algorithms, comparing them to existing algorithms in the literature. We conduct our experiments in two different settings: the first is synthetic data, where we aim to test our algorithms and understand their performance for our basic task of private mean estimation, comparing them to other algorithms. In our second setting, we seek to evaluate the performance of our algorithms for private federated learning which requires private mean estimation as a subroutine for DP-SGD.
4.1 Private mean estimation
In our synthetic-data experiment, we study the basic private mean estimation problem, aspiring to investigate the following aspects:
Figure 2: Performance of ProjUnit, FastProjUnit and their correlated versions with  $90\%$  confidence intervals as a function of  $k$  for  $d = 32768$ ,  $\mathrm{NumRep} = 30$ ,  $n = 50$ , and  $\varepsilon = 10$
1. Utility of ProjUnit algorithms as a function of the communication budget
2. Utility of our low-communication algorithms compared to the optimal utility and other existing low-communication algorithms
3. Run-time complexity of our algorithms compared to existing algorithms
Our experiments<sup>3</sup> measure the error of different algorithms for estimating the mean of a dataset. To this end, we sample unit vectors  $v_{1}, \ldots, v_{n} \in \mathcal{R}^{d}$  by normalizing samples from the normal distribution  $\mathsf{N}(\mu, 1/d)$  (where  $\mu \in \mathbb{R}^{d}$  is a random unit vector), and apply a certain privacy protocol NumRep times to estimate the mean  $\sum_{i=1}^{n} v_{i}/n$ , producing mean squared errors  $e_{1}, \ldots, e_{\text{NumRep}}$ . Our final error estimate is then the mean  $\frac{1}{\text{NumRep}} \sum_{i=1}^{\text{NumRep}} e_{i}$ . We test the performance of several algorithms: ProjUnit (Subsection 2.1), FastProjUnit (Subsection 2.2), FastProjUnit-corr (Section 3), PrivUnitG [7], CompPrivUnitG [7, 24], PrivHS [19], RePrivHS [19, 24], SQKR [14].
In Figure 2, we plot the error for our ProjUnit algorithms as a function of the communication budget  $k$ . We consider a high-dimensional regime where  $d = 2^{15}$  with a small number of users  $n = 50$  and a bounded communication budget  $k \in [1, 2000]$ . Our plot shows that our ProjUnit algorithms obtain the same near-optimal error as PrivUnitG for  $k$  as small as 1000. Moreover, the plots show that the correlated versions of our ProjUnit algorithms obtain nearly the same error.
To translate this into concrete numbers, the transform  $W$  can be communicated using a small seed ( $\sim 128$  bits) in practice, or using  $k\log d + \log^2 d \sim 20400$  bits or less than  $3\mathrm{kB}$  for  $d = 10^6$ . Sending the  $k$ -dimensional vector of 32-bit floats would need an additional  $4\mathrm{kB}$ . Thus the total communication cost is between 4 and  $8\mathrm{kB}$ . This can be further reduced by using a compressed version of PrivUnitG in the projected space, which requires the client to send a 128-bit seed. In this setting, the communication cost is a total of 256 bits. Thus in the sequel, we primarily focus on the  $k = 1000$  version of our algorithms.
(b)
Figure 3: Run-time (in seconds) as a function of the dimension for (a)  $\varepsilon = 10$  and (b)  $\varepsilon = 16$ . The plots for some algorithms are not complete as they did not finish within the cut-off time.
3The code is also available online on https://github.com/apple/ml-projunit
Figure 4: Test accuracy on the MNIST dataset with  $90\%$  confidence intervals as a function of epoch for  $\varepsilon = 10.0$ .
Next, we compare the performance of our low-communication algorithms against existing low-communication algorithms: PrivHS and SQKR. In Figure 1, we plot the error as a function of the privacy parameter for each algorithm using the best choice of  $k$  (bound on communication) for each algorithm. In particular, we choose  $k = \varepsilon$  for SQKR, num repetitions  $R = \varepsilon /2$  for repeated PrivHS,  $k = 1000$  for ProjUnit and FastProjUnit. Moreover, in this experiment we set  $n = 1$  and NumRep  $= 50$  to estimate the variance of each method. The figure shows that PrivHS and SQKR, while having low-communication complexity, suffer a significantly worse utility than (near-optimal) PrivUnitG. On the other side, both our ProjUnit algorithms obtain nearly the same error as PrivUnitG with a bounded communication budget of  $k = 1000$ .
In our third experiment in Figure 3, we plot the runtime of each algorithm as a function of the privacy parameter. Here, we use  $n = 1$ , NumRep = 10 and measure the run-time of each method for different values of the dimension  $d$  and privacy parameter  $\varepsilon$ , allowing each method to run for 1 hour before interrupting. As expected from our theoretical analysis, the runtime of ProjUnit using random rotations is noticeably slower than the (high communication cost) PrivUnitG. However, our SRHT-based FastProjUnit is substantially faster and has a comparable run-time to PrivUnitG. Moreover, for large  $\varepsilon$  and  $d$ , the run-time of compressed PrivUnitG becomes too costly compared to our algorithms due to the  $e^{\varepsilon}d$  time complexity.
4.2 Private federated learning
Having demonstrated the effectiveness of our methods for private mean estimation, in this section we illustrate the improvements offered by our algorithms for private federated learning. Similarly to the experimental setup in [13], we consider the MNIST [31] dataset and train a convolutional network (see Table 2) using DP-SGD [1] with 10 epochs and different sub-routines for privately estimating the mean of gradients at each batch. In order to bound the sensitivity, we clip the gradients to have  $\ell_2$ -norm 1, and run DP-SGD with batch size of 600, step-size equal to 0.1, and momentum of 0.5.
Figure 4 shows our results for this experiment, where we plot the test accuracy as a function of the epoch for each method. The plots demonstrate that our ProjUnit algorithms obtain similar performance to PrivUnitG, and better performance than the Gaussian mechanism or PrivHS. For the Gaussian mechanism, we set  $\delta = 10^{-5}$  and add noise to satisfy  $(\varepsilon, \delta)$ -DP using the analysis in [8]. We did not run SQKR in this experiment as it is not sufficiently computationally efficient for this experiment and has substantially worse performance in the experiments of the previous section. We also did not run the MVU mechanism [13] as their experiments show that it is worse than the Gaussian mechanism which has worse performance than our methods.
Finally, our private algorithms obtain accuracy roughly  $91\%$ , whereas the same model trained without privacy obtains around  $98\%$ . This degradation in accuracy is mostly due to the choice of the optimization algorithm (DP-SGD with clipping); indeed, even without adding any noise, DP-SGD with clipping achieves around  $91\%$  accuracy, suggesting that other private optimization algorithms with different clipping strategies (e.g. [34, 6]) may tighten this gap further. As this is not the main focus of our work, we leave this investigation to future work.
HLN is supported by NSF CAREER grant 1750716 and NSF grant 2311649.
[30] Pravesh K. Kothari and Raghu Meka. Almost optimal pseudorandom generators for spherical caps: Extended abstract. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing, STOC '15, page 247-256, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450335362. doi: 10.1145/2746539.2746611. URL https://doi.org/10.1145/2746539.2746611.
[31] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database, 1998. URL http://yann.lecun.com/exdb/mnist. ATT Labs [Online].
[32] P. Mayekar and H. Tyagi. Limits on gradient compression for stochastic optimization. In 2020 IEEE International Symposium on Information Theory (ISIT), pages 2658-2663, 2020. doi: 10.1109/ISIT44484.2020.9174075.
[33] Noam Nisan. Pseudorandom generators for space-bounded computation. Combinatorica, 12: 449-461, 1992.
[34] Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X. Yu, Sashank J. Reddi, and Sanjiv Kumar. AdaCliP: Adaptive clipping for private SGD. arXiv:1908.07643 [cs.LG], 2020.
[35] Tamás Sarlós. Improved approximation algorithms for large matrices via random projections. In Proceedings of the 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 143–152, 2006.
[36] Jeanette P. Schmidt, Alan Siegel, and Aravind Srinivasan. Chernoff-hoeffding bounds for applications with limited independence. SIAM Journal on Discrete Mathematics, 8(2):223-250, 1995. doi: 10.1137/S089548019223872X.
[37] Ananda Theertha Suresh, Felix X. Yu, Sanjiv Kumar, and H. Brendan McMahan. Distributed mean estimation with limited communication. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
[38] Shay Vargaftik, Ran Ben-Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzenmacher. Drive: One-bit distributed mean estimation. Proceedings of the 34nd Annual Conference on Advances in Neural Information Processing Systems (NeurIPS), 2021.
[39] Shay Vargaftik, Ran Ben-Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben Itzhak, and Michael Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.