SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Optimal projection dimension  $k^*$ . Since by Proposition 4.3 we know that  $N_{+} \asymp F_{\| u \|, p} k$  roughly, the term  $\left[\frac{2e^{\epsilon / N_{+}}}{(e^{\epsilon / N_{+}} - 1)^{2}} + \frac{4e^{2\epsilon / N_{+}}}{(e^{\epsilon / N_{+}} - 1)^{4}}\right]$  would increase with  $k$ . Therefore, there exists an optimal  $k^*$  that minimizes the estimation variance. When  $k$  is sufficiently large, we have
$$
\frac {V _ {R R}}{k} \approx \frac {\theta (\pi - \theta)}{k} + \frac {2 \pi^ {2} F _ {\parallel u \parallel , p} ^ {2} k}{\epsilon^ {2}} + \frac {8 \pi^ {2} F _ {\parallel u \parallel , p} ^ {4} k ^ {3}}{\epsilon^ {4}},
$$
using the approximation that  $e^x - 1 \approx x$  when  $x$  is small. To find the optimal  $k$  minimizing this expression, we compute the derivative and set it as zero:
$$
- \frac {\theta (\pi - \theta)}{k ^ {2}} + \frac {2 \pi^ {2} F _ {\parallel u \parallel , p} ^ {2}}{\epsilon^ {2}} + \frac {2 4 \pi^ {2} F _ {\parallel u \parallel , p} ^ {4} k ^ {2}}{\epsilon^ {4}} = 0 \Rightarrow k ^ {*} \asymp \frac {\epsilon \theta (\pi - \theta)}{F _ {\parallel u \parallel , p}}.
$$
This analysis suggests the optimal  $k^*$  of DP-SignRP-RR is larger when: (1)  $\epsilon$  is large; (2)  $F_{\| u\|,p}$  is small, which is typically true when the norm of the data is relatively large.
4.2 Smooth Flipping Probability: the Benefit of Robustness
Next, we improve DP-SignRP-RR based on a novel tool which we call "smooth flipping probability". The idea is inspired by the "smooth sensitivity" for DP noise addition [66], which essentially says that, the farther  $u$  is from a high "local sensitivity" region, the less noise is needed for  $u$  to achieve DP. The local sensitivity at a specific data point  $u$  is defined as  $LS(u) = \max_{u' \in Nb(u)} \| f(u) - f(u') \|_2$ , which differs from Definition 2.3 in that local sensitivity only considers the local neighbors of  $u$ .
What is the "local perturbation" for flipping the SignRP? Consider one projection  $x_{j} = W[;, j]^{T}u$  and  $s_{j} = \text{sign}(x_{j})$ . For a neighbor  $u'$  of  $u$ , denote  $x_{j}' = W[;, j]^{T}u'$  and  $s_{j}' = \text{sign}(x_{j}')$ . By  $\beta$ -adjacency,  $s_{j}'$  is possible to be different from  $s_{j}$  only if  $|x_{j}| < \beta \max_{i=1,\dots,p} |W_{ij}|$ . This implies:
- When  $|x_{j}| < \beta \max_{i = 1,\dots ,p}|W_{ij}|$ , the "local sign flipping" is required, since  $s_j^\prime$  may differ from  $s_j$ . We can apply the standard RR: keep  $s_j$  with prob.  $\frac{e^{\varepsilon / k}}{e^{\varepsilon / k} + 1}$  and flip otherwise.
- When  $|x_j| \geq \beta \max_{i=1,\dots,p} |W_{ij}|$ , there does not exist  $u' \in Nb(u)$  such that  $s_j' \neq s_j$ . Thus, the "local sign flipping" is not needed. However, if we do not perturb  $s_j$ , DP would be violated since  $s_j'$  might need random flipping. Instead, we use a "smooth flipping probability" which applies less perturbation for points far away from the "high perturbation region", i.e., the regime where  $|x_j| < \beta \max_{i=1,\dots,p} |W_{ij}|$ . Specifically, we define  $L_j = \left\lceil \frac{|x_j|}{\beta \max_{i=1,\dots,p} |W_{ij}|} \right\rceil$ . The probability of keeping the sign  $s_j$  is  $P_j^{(u)} = \frac{e^{\epsilon_j'}}{e^{\epsilon_j} + 1}$  where  $\epsilon_j' = \frac{L_j}{k} \epsilon$ . It can be shown (e.g., in the proof of Theorem 4.5) that this flipping probability is "smooth": for  $\forall u, u'$  that are neighbors,  $P_j^{(u)} \leq e^{\epsilon/k} P_j^{(u')}$ , which justifies its name.
Algorithm 4: DP-SignRP-RR-smooth using smooth flipping probability
1 Input: Data  $u\in [-1,1]^p$ $\epsilon >0$  Number of projections  $k$
2 Output: Differentially private sign random projections
3 Apply RP by  $x = W^T u$ , where  $W \in \mathbb{R}^{p \times k}$  is a random  $N(0,1)$  matrix
4 Compute  $L_{j} = \left\lceil \frac{|x_{j}|}{\beta\max_{i = 1,\ldots,p}|W_{ij}|}\right\rceil$  for  $j = 1,\dots,k$
5 Compute  $\tilde{s}_j = \left\{ \begin{array}{ll} sign(x_j), & \text{with prob.}\frac{e^{\epsilon_j'}}{e^{\epsilon_j'} + 1}\\ -sign(x_j), & \text{with prob.}\frac{1}{e^{\epsilon_j'} + 1} \end{array} \right.$  for  $j = 1,\dots,k$  with  $\epsilon_j^\prime = \frac{L_j}{k}\epsilon$
6 Return  $\tilde{s}$  as the DP-SignRP of  $u$
The concrete steps of DP-SignRP-RR-smooth are summarized in Algorithm 4. Note that, the above two cases can be unified into one, since  $\frac{e^{\varepsilon / k}}{e^{\varepsilon / k} + 1}$  is the smooth flipping probability with  $L_{j} = 1$ . In Figure 1, we provide an illustrative example of the smooth flipping probability (red) and the "local flipping probability" (green) discussed above. The global upper bound (black dash) corresponds to the standard randomized response. The local flipping probability is only non-zero at  $L = 1$ . The proposed smooth flipping probability is non-zero for all  $L > 0$ , but keeps shrinking exponentially as  $L$  gets larger, i.e., as the projected value is farther from 0.
Theorem 4.5. Algorithm 4 achieves  $\epsilon$ -DP.
Figure 1: The smooth flipping prob., local flipping prob., and the global upper bound.  $\epsilon = 1$ ,  $k = 1$ .  $L$  is from Algorithm 4.
Comparison with DP-SignRP-RR (Algorithm 3). In terms of algorithm design and privacy, DP-SignRP-RR-smooth (Algorithm 4) has two advantages over the standard DP-SignRP-RR method:
1. DP-SignRP-RR-smooth does not require (assume) a lower bound  $m$  on the data norms;
2. DP-SignRP-RR-smooth achieves  $\epsilon$ -DP, while DP-SignRP-RR only guarantee  $(\epsilon, \delta)$ -DP.
Note that Algorithm 3 can also achieve  $\epsilon$ -DP if we set the flipping probability to be  $\frac{1}{e^{\epsilon / k} + 1}$  (for all projected values). This corresponds to the global upper bound (black dotted line) in Figure 1. Thus, DP-SignRP-RR-smooth requires much smaller flipping probability than DP-SignRP-RR.
DP-SignRP with Rademacher projection. Similar to DP-RP, the flipping probabilities also depend on the projection matrix  $W$  through  $N_{+}$  (for DP-SignRP-RR) and  $L$  (for DP-SignRP-RR-smooth). In Appendix B.2, we analytically show that Rademacher RP requires smaller flipping probability than that of Gaussian RP for both variants.
4.3 DP-SignOPORP with Smooth Flipping Probability
We incorporate the smooth flipping technique to develop DP-SignOPORP, as given in Algorithm 5 with two variants. DP-SignOPORP-RR is based on the standard randomized response (RR), while DP-SignOPORP-RR-smooth is an improved version with our proposed smooth flipping probability.
Algorithm 5: DP-SignOPORP-RR and DP-SignOPORP-RR-smooth
1 Input: Data  $u\in [-1,1]^p$ $\epsilon >0$  Number of projections  $k$
2 Output: Differentially private sign OPORP
3 Apply Algorithm 2 with a random Rademacher projection vector to get the OPORP  $x$
DP-SignOPORP-RR:
4 Compute  $\tilde{s}_j = \left\{ \begin{array}{ll} sign(x_j), & \text{with prob. $\frac{e^\epsilon}{e^\epsilon + 1}$ for $j = 1, \ldots, k$}\\ -sign(x_j), & \text{with prob. $\frac{1}{e^\epsilon + 1}$ for $j = 1, \ldots, k$} \end{array} \right.$
DP-SignOPORP-RR-smooth:
5 Compute  $L_{j} = \lceil \frac{|x_{j}|}{\beta}\rceil$  for  $j = 1,\dots,k$
6 Compute  $\tilde{s}_j = \left\{ \begin{array}{ll} sign(x_j), & \text{with prob.}\frac{e^{\epsilon_j'}}{e^{\epsilon_j'} + 1}\\ -sign(x_j), & \text{with prob.}\frac{1}{e^{\epsilon_j'} + 1} \end{array} \right.$  for  $j = 1,\dots,k$  with  $\epsilon_j^\prime = L_j\epsilon_j$
7 For  $\tilde{s}_j = 0$ , assign a random coin in  $\{-1,1\}$
8 Return  $\tilde{s}$  as the DP-SignRP of  $u$
Benefit of binning. In Algorithm 5, the key difference compared with DP-SignRP methods is that, the  $\frac{1}{N_{+}}$  (in Algorithm 3) or  $\frac{1}{k}$  (in Algorithm 4) is removed from the flipping probability formulas, which is a significant advantage in privacy. This improvement is a result of the binning step. By Definition 2.2, two neighboring data  $u^{\prime}$  and  $u$  differ in one coordinate. For SignRP, since each projected data is an aggregation of the whole data vector, when we switch from  $u$  to  $u^{\prime}$ , (in principle) all  $k$  projections are possible to change. In contrast, in OPORP, since each data entry appears in only one bin,  $u^{\prime}$  will only cause exactly one projected output to change, which enhances privacy.
Theorem 4.6. Both variants in Algorithm 5 are  $\epsilon$ -DP.
Proof. The proof follows the idea of the proofs of DP-SignRP methods, but we need to additionally consider the empty bins. Denote  $x$  and  $x'$  as the OPORP from Algorithm 2 using a same random vector  $w$ , and let  $s = \text{sign}(x)$ ,  $s' = \text{sign}(x')$ . Suppose  $u$  and  $u'$  differ in dimension  $i$ , and dimension  $i$  is assigned to the  $j^*$ -th bin with  $j^* = \lceil \pi(i) / (p / k) \rceil$ , where  $\pi : [p] \mapsto [p]$  is the permutation in OPORP. For any  $y \in \{-1, 1\}^k$ , it is easy to see that to compute  $\log \frac{\text{Pr}(\tilde{s} = y)}{\text{Pr}(\tilde{s}' = y)}$ , it suffices to look at the  $j^*$ -th output sample because other probabilities cancel out. For the  $j^*$ -th sample, when  $s_{j^*} \neq 0$  and  $s_{j^*}' \neq 0$ , by the same arguments as in the proof of Theorem 4.5, we know that  $e^{-\epsilon} \leq \frac{\text{Pr}(s_{j^*}) = a}{\text{Pr}(s_{j^*}') = a} \leq e^\epsilon$ ,  $a \in \{-1, 1\}$ , for both variants (DP-SignOPORP-RR and DP-SignOPORP-RR-smooth). When one of  $s_{j^*}$  and  $s_{j^*}'$  equals 0, it is also easy to see that (assume  $s_{j^*} = 1$  and  $s_{j^*}'$  is a random coin)
$$
1 \leq \frac {P r (s _ {j ^ {*}}) = 1}{P r (s _ {j ^ {*}} ^ {\prime}) = 1} = \frac {2 e ^ {\epsilon}}{e ^ {\epsilon} + 1} \leq e ^ {\epsilon}, \quad e ^ {- \epsilon} \leq \frac {P r (s _ {j ^ {*}}) = - 1}{P r (s _ {j ^ {*}} ^ {\prime}) = - 1} = \frac {2}{e ^ {\epsilon} + 1} \leq 1.
$$
This holds for both variants. In particular, this case corresponds to  $L_{j^*} = 1$  for the smooth flipping probability. This proves the theorem.
Multiple repetitions. In Algorithm 5 Line 7, if OPORP generates an empty bin (i.e.,  $x_{j} = 0$ ), we must assign a random sign to it in order to maintain DP. This tends to undermine the utility since a random coin does not provide any useful information about the data. To avoid too many empty bins, one simple way is to repeat the DP-SignOPORP (with smaller  $k$ ) for  $t > 1$  times, and concatenate the output vectors [12]. For example, if the target  $k = 256$ , we may run Algorithm 5 for  $t = 4$  times, each time with  $256 / 4 = 64$  projected values and privacy budget  $\epsilon / 4$ . We still obtain a length-256 bit vector with  $\epsilon$ -DP by the composition theorem. In the experiments, we will see that this strategy may improve the overall performance of DP-SignOPORP.
We present experiments on retrieval and classification tasks to demonstrate the effectiveness of our proposed methods. We compare the following algorithms:
- Raw-data-G-OPT: adding optimal Gaussian noise with sensitivity  $\beta$  to the original data, which is the most basic DP method for privatizing the original data.
- DP-RP-G (Algorithm 1 + Theorem 3.1): Gaussian DP-RP with the Gaussian mechanism.
- DP-RP-G-OPT (Algorithm 1 + Theorem 3.2): DP-RP with Gaussian random projection matrix and the optimal Gaussian noise.
- DP-OPORP (Theorem 3.3): one permutation + one random projection with Rademacher projection vector and optimal Gaussian noise.
- DP-SignOPROP-RR and DP-SignOPORP-RR-smooth (Algorithm 5): signed OPORP with standard randomized response and with smooth flipping probability, respectively.
As noted in Section 4.3, due to binning, DP-SignOPORP is substantially better than DP-SignRP. Thus we present results for DP-SignOPORP in our experiments (and only  $\beta = 1$  for conciseness).
Similarity Search. We first test the methods in similarity search problems, using two standard image retrieval datasets, MNIST [53] and CIFAR [51]. For both datasets, we treat the training set as the database, and use the samples from the test set as the queries. We set the ground truth neighbors for each query as the top-50 sample vectors with the highest cosine similarity to the query. To search with DP-RP (and DP-OPORP), we estimate the cosine between a query and a sample point by the cosine between their corresponding output noisy projected values. We report only the precisions (and please see Appendix for other metrics), averaged over 10 repetitions.
Figure 2: Retrieval recall and precision on MNIST and CIFAR,  $\beta = 1$ ,  $\delta = 10^{-6}$ .
Figure 2 shows that, as a result of the optimal Gaussian mechanism, DP-RP-G-OPT substantially outperforms the prior method in the literature, DP-RP-G, at all  $\epsilon$ . DP-OPORP achieve better performance than DP-RP-G-OPT and is recommended as the best (full-precision) DP-RP variant. These two methods provide much higher precision and recall than the optimal Gaussian mechanism applied to the original data (Raw-data-G-OPT).
In Figure 3, we plot the precision and recall of DP-SignOPORP-RR (with standard randomized response), and DP-SignOPORP-RR-s (with smooth flipping probability). We present the results with repetition  $t = 2,4$  which yield good overall performance. As we can see, (i) the proposed smooth flipping probability considerably improves the standard randomized response technique, and (ii)
Figure 3: Retrieval on MNIST with DP-SignOPORP-RR and DP-SignOPORP-RR-smooth. For DP-OPORP, DP-SignOPORP-RR, and Raw-data-G-OPT,  $\delta = 10^{-6}$ .
Figure 4: SVM test accuracy on WEBSPAM. For  $(\epsilon, \delta)$ -DP methods,  $\delta = 10^{-6}$ .
DP-SignOPORP in general provides better search accuracy than DP-OPORP when  $\epsilon < 10\sim 15$  This range of  $\epsilon$  is common in the use cases of DP for providing reasonable privacy protection. Also, we note that smaller  $t$  (repetitions) typically performs better with smaller  $\epsilon$  but worse when  $\epsilon$  is large.
Classification. We evaluate the performance of DP-SignOPORP in classification problems trained by SVM [20], on the WEBSPAM dataset [11]. We apply the "max normalization", i.e., divide each data column by its largest magnitude such that the data entries are bounded in  $[-1, 1]$ . The SVM results are presented in Figure 4. For this task, we plot DP-SignOPORP with  $t = 1$ . We again observe the advantage of DP-SignOPORP over DP-OPORP, and the advantage of the proposed smooth flipping probability over the standard randomized response. Specifically, when  $\epsilon \approx 5$ , the test accuracy of Raw-data-G-OPT is around  $60\%$  (for WEBSPAM, this is almost the same as random guessing), but DP-SignOPORP can achieve  $95\%$  classification accuracy with  $k = 1024$ .
For a database  $U \in \mathbb{R}^{n \times p}$  consisting of  $n$  data points, our goal is to make its random projections (RP) and SignRP differentially private—the DP guarantee we have derived holds independently and simultaneously for every data vector  $u \in U$ . There are works on randomized algorithms that assume the randomness of the algorithm are "internal" and also kept private [7, 73, 25]. We assume the projection matrix  $W$  is known/public, which is stronger in privacy and enables broader applications of the released data. Also, a "known projection matrix" is typical in the local DP setting [47, 18], since the projection matrix is created and shared among the users by the data aggregator.
In our study, we first revisit the prior work [49] on the  $(\epsilon, \delta)$ -DP random projection based on Gaussian noise addition ("DP-RP-G"). Then we incorporate the optimal Gaussian mechanism and propose an improved method DP-RP-G-OPT. We also develop DP-OPORP based on the optimal Gaussian mechanism. We then propose algorithms for privatizing the (1-bit) DP-SignRP: "DP-SignRP-RR" based on the standard "randomized response" (RR), and "DP-SignRP-RR-smooth" based on a proposed concept of "smooth flipping probability" of SignRP. We extend the idea of smooth flipping probability to OPORP (a variant of count-sketch) and demonstrate the advantage of DP-SignOPORP.
Experiments on retrieval and classification tasks justify the effectiveness of our proposed methods. In particular, DP-SignOPORP outperforms other methods especially when  $\epsilon$  is not large (e.g.,  $\epsilon \leq 5$ ). These results also verify the advantage of using the smooth flipping probability.
The authors would like to thank several researchers from Microsoft Research: Janardhan Kulkarni, Yin Tat Lee, Sergey Yekhanin, for the helpful discussions.
[1] Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 308-318, Vienna, Austria, 2016.
[2] Dimitris Achlioptas. Database-friendly random projections: Johnson-lindenstrauss with binary coins. J. Comput. Syst. Sci., 66(4):671-687, 2003.
[3] Eman Abdullah AlOmar, Wajdi Aljedaani, Murtaza Tamjeed, Mohamed Wiem Mkaouer, and Yasmine N. El-Glaly. Finding the needle in a haystack: On the automatic identification of accessibility user reviews. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 387:1-387:15, Virtual Event / Yokohama, Japan, 2021.
[4] Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 403-412, Stockholm, Sweden, 2018.
[5] Arnaud Berlioz, Arik Friedman, Mohamed Ali Käafar, Roksana Boreli, and Shlomo Berkovsky. Applying differential privacy to matrix factorization. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys), pages 107-114, Vienna, Austria, 2015.
[6] Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: Applications to image and text data. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 245-250, San Francisco, CA, 2001.
[7] Jeremiah Blocki, Avrim Blum, Anupam Datta, and Or Sheffet. The johnson-lindenstrauss transform itself preserves differential privacy. In Proceedings of the 53rd Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 410-419, New Brunswick, NJ, 2012.
[8] Avrim Blum, Cynthia Dwork, Frank McSherry, and Kobbi Nissim. Practical privacy: the SuLQ framework. In Proceedings of the Twenty-fourth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS), pages 128-138, Baltimore, MD, 2005.
[9] Petros Boufounos and Richard G. Baraniuk. 1-bit compressive sensing. In Proceedings of the 42nd Annual Conference on Information Sciences and Systems (CISS), pages 16-21, Princeton, NJ, 2008.
[10] Emmanuel J. Candès, Justin K. Romberg, and Terence Tao. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory, 52(2):489-509, 2006.
[11] Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011.
[12] Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. Theor. Comput. Sci., 312(1):3-15, 2004.
[13] Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing (STOC), pages 380-388, Montreal, Canada, 2002.
[14] Kamalika Chaudhuri and Claire Monteleoni. Privacy-preserving logistic regression. In Advances in Neural Information Processing Systems (NIPS), pages 289–296, Vancouver, Canada, 2008.
[84] Jun Zhang, Graham Cormode, Cecilia M. Procopiuc, Divesh Srivastava, and Xiaokui Xiao. Privbayes: private data release via bayesian networks. In International Conference on Management of Data (SIGMOD), pages 1423-1434, Snowbird, UT, 2014.
[85] Jun Zhang, Zhenjie Zhang, Xiaokui Xiao, Yin Yang, and Marianne Winslett. Functional mechanism: Regression analysis under differential privacy. Proc. VLDB Endow., 5(11):1364-1375, 2012.
[86] Shan Zhang, Lei Wang, Naila Murray, and Piotr Koniusz. Kernelized few-shot object detection with efficient integral aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19185-19194, New Orleans, LA, 2022.
[87] Zhaoqi Zhang, Panpan Qi, and Wei Wang. Dynamic malware analysis with feature engineering and feature learning. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), pages 1210-1217, New York, NY, 2020.
[88] Fuheng Zhao, Dan Qiao, Rachel Redberg, Divyakant Agrawal, Amr El Abbadi, and Yu-Xiang Wang. Differentially private linear sketches: Efficient implementations and applications. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[89] Argyrios Zymnis, Stephen P. Boyd, and Emmanuel J. Candès. Compressed sensing with quantized measurements. IEEE Signal Process. Lett., 17(2):149-152, 2010.
1.1 Random Projections (RP) and Sign Random Projections (SignRP) 2
1.2 Count-Sketch and OPORP: Efficient RP-type Alternatives 2
2 Background on Differential Privacy 3
3 Revisiting Differentially Private Random Projection Methods 3
3.1 Gaussian Noise Mechanism for DP-RP 3
3.2 DP-OPORP: A More Efficient Alternative 4
4 DP-SignRP: Differentially Private Sign Random Projections 5
4.1 DP-SignRP-RR by Randomized Response 5
4.1.1 The flipping probability and calculation of  $N_{+}$  5
4.1.2 Utility in angle estimation by DP-SignRP-RR 6
4.2 Smooth Flipping Probability: the Benefit of Robustness 6
4.3 DP-SignOPORP with Smooth Flipping Probability 8
A Analytic DP-RP and the Laplace Mechanism 18
A.1 Analytic DP-RP-G Approach: Bounding the Sensitivity 18
A.2 The Laplace Noise Mechanism for DP-RP 18
B Comparison of Different Projection Matrices and the Benefits of Rademacher RP 20
B.1 Rademacher Projection for DP-RP 20
B.2 Rademacher Projection for DP-SignRP 21
C Comparison of DP-RP and DP-OPORP on Inner Product Estimation 22
D More Experiment Results 26
E Deferred Proofs 28
E.1 Proof of Lemma 4.2 31
E.2 Proof of Proposition 4.3 31
E.3 Proof of Theorem 4.4 32
E.4 Proof of Theorem 4.5 32
A Analytic DP-RP and the Laplace Mechanism
A.1 Analytic DP-RP-G Approach: Bounding the Sensitivity
According to Theorem 3.1, one will choose  $\sigma$  based on the observed  $\Delta_{2}$  which is calculated according to (3) for a given realization of the projection matrix  $W$ . For the convenience of theoretical analysis, it is often helpful to derive the criterion for choosing  $\sigma$  directly in terms of the input parameters  $\epsilon$ ,  $\delta$ ,  $p$ , and  $k$ , to understand and better compare the noise level. We call the corresponding algorithm to be "Analytic DP-RP-G". This is achieved by bounding  $\Delta_{2}$  in high probability.
Lemma A.1 ([52]). Let  $Y_{1}, \ldots, Y_{n}$  be iid standard Gaussian random variables, and denote  $Z_{i} = \sum_{i=1}^{n} Y_{i}^{2}$ . Then for any  $t > 0$ ,
$$
P r (Z \geq n + 2 \sqrt {n t} + 2 t) \leq \exp (- t).
$$
Lemma A.2 (Bounding  $\Delta_2$ ). Let  $W \in \mathbb{R}^{p \times k}$  be a random matrix with iid  $N(0,1)$  entries, and  $\Delta_2$  be defined in (3). Then for any  $0 < \delta < 1$ ,
$$
P r \left(\Delta_ {2} \geq \beta \sqrt {1 + 2 \sqrt {\log (p / \delta) / k} + 2 \log (p / \delta) / k}\right) \leq \delta .
$$
Proof. Let  $e_i$  be the unit base vector with the  $i$ -th dimension being 1 and all other dimensions being zero. For each  $i = 1, \dots, p$ ,  $W^T e_i$  follows iid chi-square distribution with  $k$  degree of freedom. Applying Lemma A.1 we have that for any  $\delta' > 0$ ,
$$
P r \left(\| W ^ {T} e _ {i} \| ^ {2} \geq \beta^ {2} (k + 2 \sqrt {k \log (1 / \delta^ {\prime})} + 2 \log (1 / \delta^ {\prime}))\right) \leq \delta^ {\prime}, \forall i = 1, \dots , p.
$$
Applying union bound gives
$$
P r \left(\frac {1}{k} \beta^ {2} \max  _ {i = 1, \dots , p} \| W ^ {T} e _ {i} \| ^ {2} \geq 1 + 2 \sqrt {\log (1 / \delta^ {\prime}) / k} + 2 \log (1 / \delta^ {\prime}) / k\right) \leq p \delta^ {\prime}.
$$
Setting  $\delta = p\delta'$  and taking the square root on both sides completes the proof.