SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 9 also illustrates when it might be a good strategy to directly add noise to the original data. For example, when  $p = 1000$ , the ratio can be below 1 if  $\sigma < 0.1$ . One can numerically verify that, (in Figure 10) in order for  $\sigma < 0.1$  at  $\Delta_2 = \beta = 1$ , we need  $\epsilon > 100$ . In other words, adding noise to the raw data might be plausible when  $\epsilon > 100$ . In the literature, however, many DP applications typically require a much smaller  $\epsilon$ , such as  $\epsilon \in [0.1, 20]$  (e.g., [43, 48]). Therefore, DP-RP and DP-OPORP is much better (i.e., has much smaller inner product estimation variance) than adding noise to the raw data in common privacy regimes.
D More Experiment Results
We provide the complete set of plots of our experimental results. In Figure 11 and Figure 12, we report the precision@10 and recall@100 curves of DP-RP variants and DP-OPORP on MNIST and CIFAR, respectively. In Figure 13, we report the test accuracy on the Webspam dataset of these methods. From all plots, we see that DP-RP-G-OPT-B and DP-OPORP perform equally the best on all the tasks, significantly better than the strategy of adding Gaussian noise to the raw data.
In Figure 14, we report the recall@100 metric of DP-SignOPORP methods in addition to the precision@10 metric shown in the main paper. For completeness, we also include the SVM test accuracy in Figure 15, which is the same as Figure 4 in the main context.
Figure 11: Retrieval recall and precision on MNIST,  $\beta = 1$ ,  $\delta = 10^{-6}$ .
Figure 12: Retrieval recall and precision on CIFAR,  $\beta = 1$ ,  $\delta = 10^{-6}$ .
Figure 13: SVM classification on WEBSPAM,  $\beta = 1$ ,  $\delta = 10^{-6}$ .
Figure 14: Retrieval on MNIST with DP-SignOPORP-RR and DP-SignOPORP-RR-smooth (in the caption, "-s" stands for "-smooth". For DP-OPORP and Raw-data-G-OPT, we let  $\delta = 10^{-6}$
Figure 15: SVM classification on Webspam with DP-SignOPORP-RR and DP-SignOPORP-RR-s. For DP-OPORP and Raw-data-G-OPT, we let  $\delta = 10^{-6}$ .
E Deferred Proofs
The following lemma on Gaussian random variables will be used in our proof for Lemma 4.2, and may also be of independent interest.
Lemma E.1. Let  $\left( \begin{array}{c}X\\ Y \end{array} \right)\sim N\left( \begin{array}{cc}\sigma_x^2 & \rho \sigma_x\sigma_y\\ \rho \sigma_x\sigma_y & \sigma_y^2 \end{array} \right)$ . Denote  $r = \sigma_x / \sigma_y$ . Then we have:
1.  $Pr(|X| > |Y|) = \frac{1}{\pi} \left[ \tan^{-1} \left( \frac{r - \rho}{\sqrt{1 - \rho^2}} \right) + \tan^{-1} \left( \frac{r + \rho}{\sqrt{1 - \rho^2}} \right) \right]$ . When  $r \leq 1$ , the maximum is achieved at  $\rho = 0$ , i.e.,  $\max_{\rho} Pr(|X| < |Y|) = \frac{2}{\pi} \tan^{-1}(r)$ .
2. The conditional expectation:
$$
\mathbb {E} \left[ | X | | | X | > | Y | \right] = \sigma_ {x} \sqrt {\frac {\pi}{2}} \cdot \frac {\frac {r - \rho}{\sqrt {1 + r ^ {2} - 2 r \rho}} + \frac {r + \rho}{\sqrt {1 + r ^ {2} + 2 r \rho}}}{\tan^ {- 1} \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}}\right) + \tan^ {- 1} \left(\frac {r + \rho}{\sqrt {1 - \rho^ {2}}}\right)}.
$$
3. The conditional tail probability: for any  $r > 0$ ,  $\rho \in (-1,1)$ , for any  $t > 0$
$$
P r (| X | > t | | X | > | Y |) \leq \exp \left(- \frac {t ^ {2}}{2 \sigma_ {x} ^ {2}}\right).
$$
Proof. The bivariate normal density function is
$$
\begin{array}{l} f (x, y) = \frac {1}{2 \pi \sigma_ {x} \sigma_ {y} \sqrt {1 - \rho^ {2}}} \exp \left(- \frac {\frac {x ^ {2}}{\sigma_ {x} ^ {2}} - \frac {2 \rho x y}{\sigma_ {x} \sigma_ {y}} + \frac {y ^ {2}}{\sigma_ {y} ^ {2}}}{2 (1 - \rho^ {2})}\right) \\ = \frac {1}{2 \pi \sigma_ {x} \sigma_ {y} \sqrt {1 - \rho^ {2}}} \exp \left(- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}\right) \exp \left(- \frac {\left(\frac {y}{\sigma_ {y}} - \rho \frac {x}{\sigma_ {x}}\right) ^ {2}}{2 (1 - \rho^ {2})}\right). \\ \end{array}
$$
Therefore, we have  $\mathbb{E}\left[|X||X| > |Y|\right] = \frac{A}{P}$ , with
$$
A = \int_ {- \infty} ^ {\infty} \frac {| x |}{\sqrt {2 \pi} \sigma_ {x} \sqrt {1 - \rho^ {2}}} \exp \left(- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}\right) d x \int_ {- | x |} ^ {| x |} \frac {1}{\sqrt {2 \pi} \sigma_ {y}} \exp \left(- \frac {\left(\frac {y}{\sigma_ {y}} - \rho \frac {x}{\sigma_ {x}}\right) ^ {2}}{2 (1 - \rho^ {2})}\right) d y,
$$
$$
P = \int_ {- \infty} ^ {\infty} \frac {1}{\sqrt {2 \pi} \sigma_ {x} \sqrt {1 - \rho^ {2}}} \exp \left(- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}\right) d x \int_ {- | x |} ^ {| x |} \frac {1}{\sqrt {2 \pi} \sigma_ {y}} \exp \left(- \frac {\left(\frac {y}{\sigma_ {y}} - \rho \frac {x}{\sigma_ {x}}\right) ^ {2}}{2 (1 - \rho^ {2})}\right) d y.
$$
Note that  $P = \Pr(|X| > |Y|)$  in the first statement of the theorem. Our calculation will use the following two identities involving the Gaussian functions [67]:
$$
\int_ {0} ^ {\infty} \phi (a x) \Phi (b x) d x = \frac {1}{2 \pi | a |} \left(\frac {\pi}{2} + \tan^ {- 1} \left(\frac {b}{| a |}\right)\right), \tag {16}
$$
$$
\int_ {0} ^ {\infty} x \phi (a x) \Phi (b x) d x = \frac {1}{2 \sqrt {2 \pi}} \left(1 + \frac {b}{\sqrt {1 + b ^ {2}}}\right), \tag {17}
$$
where  $\phi (x)$  and  $\Phi (x)$  are the pdf and cdf of the standard Gaussian distribution.
With a proper change of random variables, we can compute  $A$  as
$$
\begin{array}{l} A = \int_ {- \infty} ^ {\infty} \frac {| x |}{\sqrt {2 \pi} \sigma_ {x} \sqrt {1 - \rho^ {2}}} \exp \left(- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}\right) d x \int_ {\frac {\frac {| x |}{\sigma_ {y}} - \rho \frac {x}{\sigma_ {x}}}{\sqrt {1 - \rho^ {2}}}} ^ {\frac {| x |}{\sigma_ {y}} - \rho \frac {x}{\sigma_ {x}}} \sqrt {1 - \rho^ {2}} \frac {1}{\sqrt {2 \pi}} e ^ {- s ^ {2}} d s \\ = \int_ {- \infty} ^ {\infty} \frac {| x |}{\sqrt {2 \pi} \sigma_ {x}} \exp \left(- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}\right) \left[ \Phi \left(\frac {\frac {| x |}{\sigma_ {y}} - \rho \frac {x}{\sigma_ {x}}}{\sqrt {1 - \rho^ {2}}}\right) - \Phi \left(\frac {\frac {- | x |}{\sigma_ {y}} - \rho \frac {x}{\sigma_ {x}}}{\sqrt {1 - \rho^ {2}}}\right) \right] d x \\ = \int_ {- \infty} ^ {\infty} \frac {\sigma_ {x} | t |}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \left[ \Phi \left(\frac {\frac {\sigma_ {x}}{\sigma_ {y}} | t | - \rho t}{\sqrt {1 - \rho^ {2}}}\right) - \Phi \left(- \frac {\frac {\sigma_ {x}}{\sigma_ {y}} | t | - \rho t}{\sqrt {1 - \rho^ {2}}}\right) \right] d t \\ := A _ {1} - A _ {2}. \\ \end{array}
$$
For the first term we have
$$
\begin{array}{l} A _ {1} = \sigma_ {x} \left[ \int_ {0} ^ {\infty} \frac {t}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(\frac {\frac {\sigma_ {x}}{\sigma_ {y}} - \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t + \int_ {- \infty} ^ {0} \frac {- t}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(- \frac {\frac {\sigma_ {x}}{\sigma_ {y}} + \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t \right] \\ = \sigma_ {x} \left[ \int_ {0} ^ {\infty} \frac {t}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(\frac {\frac {\sigma_ {x}}{\sigma_ {y}} - \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t + \int_ {0} ^ {\infty} \frac {s}{\sqrt {2 \pi}} \exp \left(- \frac {s ^ {2}}{2}\right) \Phi \left(\frac {\frac {\sigma_ {x}}{\sigma_ {y}} + \rho}{\sqrt {1 - \rho^ {2}}} s\right) d s \right] \\ = \sigma_ {x} \left[ \frac {1}{2 \sqrt {2 \pi}} \left(1 + \frac {\frac {\frac {\sigma_ {x}}{\sigma_ {y}} - \rho}{\sqrt {1 - \rho^ {2}}}}{\sqrt {1 + \frac {(\frac {\sigma_ {x}}{\sigma_ {y}} - \rho) ^ {2}}{1 - \rho^ {2}}}}\right) + \frac {1}{2 \sqrt {2 \pi}} \left(1 + \frac {\frac {\frac {\sigma_ {x}}{\sigma_ {y}} + \rho}{\sqrt {1 - \rho^ {2}}}}{\sqrt {1 + \frac {(\frac {\sigma_ {x}}{\sigma_ {y}} + \rho) ^ {2}}{1 - \rho^ {2}}}}\right) \right] \\ = \sigma_ {x} \left[ \frac {1}{\sqrt {2 \pi}} + \frac {1}{2 \sqrt {2 \pi}} \left(\frac {r - \rho}{\sqrt {1 + r ^ {2} - 2 r \rho}} + \frac {r + \rho}{\sqrt {1 + r ^ {2} + 2 r \rho}}\right) \right], \\ \end{array}
$$
where we denote  $r = \frac{\sigma_x}{\sigma_y}$  and use (17). Similarly, we have that
$$
\begin{array}{l} A _ {2} = \sigma_ {x} \left[ \int_ {0} ^ {\infty} \frac {t}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(- \frac {r + \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t + \int_ {- \infty} ^ {0} \frac {- t}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t \right] \\ = \sigma_ {x} \left[ \int_ {0} ^ {\infty} \frac {t}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(- \frac {r + \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t + \int_ {0} ^ {\infty} \frac {s}{\sqrt {2 \pi}} \exp \left(- \frac {s ^ {2}}{2}\right) \Phi \left(- \frac {r - \rho}{\sqrt {1 - \rho^ {2}}} s\right) d s \right] \\ = \sigma_ {x} \left[ \frac {1}{\sqrt {2 \pi}} - \frac {1}{2 \sqrt {2 \pi}} \left(\frac {r - \rho}{\sqrt {1 + r ^ {2} - 2 r \rho}} + \frac {r + \rho}{\sqrt {1 + r ^ {2} + 2 r \rho}}\right) \right]. \\ \end{array}
$$
$$
A (\rho , r) = A _ {1} - A _ {2} = \frac {\sigma_ {x}}{\sqrt {2 \pi}} \left(\frac {r - \rho}{\sqrt {1 + r ^ {2} - 2 r \rho}} + \frac {r + \rho}{\sqrt {1 + r ^ {2} + 2 r \rho}}\right). \tag {18}
$$
To compute  $P$ , by doing a similar change of variables, we have
$$
\begin{array}{l} P = \int_ {- \infty} ^ {\infty} \frac {1}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \left[ \Phi \left(\frac {r | t | - \rho t}{\sqrt {1 - \rho^ {2}}}\right) - \Phi \left(- \frac {r | t | - \rho t}{\sqrt {1 - \rho^ {2}}}\right) \right] d t \\ := P _ {1} - P _ {2}. \\ \end{array}
$$
Using (16), we obtain
$$
\begin{array}{l} P _ {1} = \int_ {0} ^ {\infty} \frac {1}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t + \int_ {- \infty} ^ {0} \frac {1}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(- \frac {r + \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t \\ = \int_ {0} ^ {\infty} \frac {1}{\sqrt {2 \pi}} \exp \left(- \frac {t ^ {2}}{2}\right) \Phi \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}} t\right) d t + \int_ {0} ^ {\infty} \frac {1}{\sqrt {2 \pi}} \exp \left(- \frac {s ^ {2}}{2}\right) \Phi \left(\frac {r + \rho}{\sqrt {1 - \rho^ {2}}} s\right) d s \\ = \frac {1}{2 \pi} \left(\frac {\pi}{2} + \tan^ {- 1} \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}}\right)\right) + \frac {1}{2 \pi} \left(\frac {\pi}{2} + \tan^ {- 1} \left(\frac {r + \rho}{\sqrt {1 - \rho^ {2}}}\right)\right) \\ = \frac {1}{2} + \frac {1}{2 \pi} \left[ \tan^ {- 1} \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}}\right) + \tan^ {- 1} \left(\frac {r + \rho}{\sqrt {1 - \rho^ {2}}}\right) \right], \\ \end{array}
$$
$$
P _ {2} = \frac {1}{2} - \frac {1}{2 \pi} \left[ \tan^ {- 1} \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}}\right) + \tan^ {- 1} \left(\frac {r + \rho}{\sqrt {1 - \rho^ {2}}}\right) \right],
$$
$$
P (\rho , r) = P _ {1} - P _ {2} = \frac {1}{\pi} \left[ \tan^ {- 1} \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}}\right) + \tan^ {- 1} \left(\frac {r + \rho}{\sqrt {1 - \rho^ {2}}}\right) \right] \tag {19}
$$
Therefore, we know that
$$
\mathbb {E} \left[ | X | | | X | > | Y | \right] = \frac {A (\rho , r)}{P (\rho , r)} = \sigma_ {x} \sqrt {\frac {\pi}{2}} \cdot \frac {\frac {r - \rho}{\sqrt {1 + r ^ {2} - 2 r \rho}} + \frac {r + \rho}{\sqrt {1 + r ^ {2} + 2 r \rho}}}{\tan^ {- 1} \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}}\right) + \tan^ {- 1} \left(\frac {r + \rho}{\sqrt {1 - \rho^ {2}}}\right)},
$$
with  $r = \sigma_x / \sigma_y$ . We now investigate the derivative of  $P$ . By some algebra, we can show that
$$
\frac {\partial P (\rho , r)}{\partial \rho} = \frac {2 r \rho (r ^ {2} - 1)}{(1 + r ^ {2} - 2 r \rho) (1 + r ^ {2} + 2 r \rho) \sqrt {1 - \rho^ {2}}}.
$$
When  $0 < r \leq 1$ ,  $\frac{\partial P(\rho, r)}{\partial \rho} \geq 0$  when  $\rho \leq 0$  and  $\frac{\partial P(\rho, r)}{\partial \rho} \leq 0$  when  $\rho > 0$ . Therefore,  $\max_{\rho} P(\rho, r) = P(0, r) = \frac{2}{\pi} \tan^{-1}(r)$ .
Tail bound. By our previous calculations, the conditional distribution of  $X$  given  $|X| > |Y|$  is
$$
f (x | | X | > | Y |) = \frac {\frac {1}{\sqrt {2 \pi}} \exp \left(- \frac {x ^ {2}}{2}\right) \left[ \Phi \left(\frac {r | x | - \rho x}{\sqrt {1 - \rho^ {2}}}\right) - \Phi \left(- \frac {r | x | - \rho x}{\sqrt {1 - \rho^ {2}}}\right) \right]}{P}, \quad x \in \mathbb {R},
$$
with  $P = \Pr(|X| > |Y|)$  in (19) the normalizing constant to make the integral equal to 1.
The conditional tail probability can be computed as follows. For some  $t > 0$ , by symmetry,
$$
\begin{array}{l} P r (| X | > t, | X | > | Y |) \\ = 2 \int_ {t} ^ {\infty} \frac {1}{\sqrt {2 \pi} \sigma_ {x}} \exp \left(- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}\right) \left[ \Phi \left(\frac {r | x | - \rho x}{\sqrt {1 - \rho^ {2}}}\right) - \Phi \left(- \frac {r | x | - \rho x}{\sqrt {1 - \rho^ {2}}}\right) \right] d x \\ = 2 \int_ {\frac {t}{\sigma_ {x}}} ^ {\infty} \frac {1}{\sqrt {2 \pi}} \exp \left(- \frac {x ^ {2}}{2}\right) \left[ \Phi \left(\frac {r | x | - \rho x}{\sqrt {1 - \rho^ {2}}}\right) - \Phi \left(- \frac {r | x | - \rho x}{\sqrt {1 - \rho^ {2}}}\right) \right] d x \\ := 2 \left(\tilde {P} _ {1} - \tilde {P} _ {2}\right). \\ \end{array}
$$
For  $\tilde{P}_1$ , using polar coordinates we have
$$
\begin{array}{l} \tilde {P} _ {1} = \frac {1}{2 \pi} \int_ {\frac {t}{\sigma_ {x}}} ^ {\infty} e ^ {- \frac {x ^ {2}}{2}} d x \int_ {- \infty} ^ {\frac {r - \rho}{\sqrt {1 - \rho^ {2}}} x} e ^ {- \frac {y ^ {2}}{2}} d y \\ = \frac {1}{2 \pi} \int_ {- \frac {\pi}{2}} ^ {\tan^ {- 1} \left(\frac {r - \rho}{\sqrt {1 - \rho^ {2}}}\right)} d \theta \int_ {\frac {t}{\sigma_ {x}} \cos (\theta)} ^ {\infty} e ^ {- \frac {r ^ {2}}{2}} r d r \\ = \frac {1}{2 \pi} \int_ {- \frac {\pi}{2}} ^ {\tan^ {- 1} (\frac {r - \rho}{\sqrt {1 - \rho^ {2}}})} \exp \left(- \frac {t ^ {2}}{2 \sigma_ {x} ^ {2} \cos^ {2} (\theta)}\right) d \theta . \\ \end{array}
$$
$$
\tilde {P} _ {2} = \frac {1}{2 \pi} \int_ {- \frac {\pi}{2}} ^ {\tan^ {- 1} (- \frac {r + \rho}{\sqrt {1 - \rho^ {2}}})} \exp \left(- \frac {t ^ {2}}{2 \sigma_ {x} ^ {2} \cos^ {2} (\theta)}\right) d \theta .
$$
$$
\begin{array}{l} P r (| X | > t, | X | > | Y |) = \frac {1}{\pi} \int_ {\tan^ {- 1} (- \frac {r + \rho}{\sqrt {1 - \rho^ {2}}})} ^ {\tan^ {- 1} (\frac {r - \rho}{\sqrt {1 - \rho^ {2}}})} \exp \left(- \frac {t ^ {2}}{2 \sigma_ {x} ^ {2} \cos^ {2} (\theta)}\right) d \theta \\ = \frac {1}{\pi} \int_ {- \tan^ {- 1} (\frac {r + \rho}{\sqrt {1 - \rho^ {2}}})} ^ {\tan^ {- 1} (\frac {r - \rho}{\sqrt {1 - \rho^ {2}}})} \exp \left(- \frac {t ^ {2}}{2 \sigma_ {x} ^ {2} \cos^ {2} (\theta)}\right) d \theta \\ \leq e ^ {- \frac {t ^ {2}}{2 \sigma_ {x} ^ {2}}} \frac {1}{\pi} \int_ {- \tan^ {- 1} (\frac {r + \rho}{\sqrt {1 - \rho^ {2}}})} ^ {\tan^ {- 1} (\frac {r - \rho}{\sqrt {1 - \rho^ {2}}})} d \theta , \\ \end{array}
$$
since  $\cos^2 (\theta)\in [0,1]$ . Notice that  $P$  in (19) can be written as  $P = \frac{1}{\pi}\int_{-\tan^{-1}(\frac{r + \rho}{\sqrt{1 - \rho^2}})}^{\tan^{-1}(\frac{r - \rho}{\sqrt{1 - \rho^2}})}d\theta$ . Hence, we know that the conditional tail probability is
$$
\begin{array}{l} P r (| X | > t | | X | > | Y |) = \frac {P r (| X | > t , | X | > | Y |)}{P r (| X | > | Y |)} \\ \leq \exp \left(- \frac {t ^ {2}}{2 \sigma_ {x} ^ {2}}\right), \quad \forall r > 0, \rho \in (- 1, 1). \\ \end{array}
$$
At the boundaries  $\rho = 1$ ,  $\rho = -1$ , one can verify  $Pr(|X| > |Y|) = 0$ . This concludes the proof.
E.1 Proof of Lemma 4.2
Proof. By Lemma E.1 and independence, we know that  $Pr(\max_{i=1,\ldots,p}|X| > |Y|)$  reaches its maximum when  $X_i$  is independent of  $Y$ , i.e., when  $\rho(X_i,Y) = 0, \forall i = 1,\dots,p$ . Since  $|X_i|$  follows a half-normal distribution with cdf being  $\operatorname{erf}\left(\frac{x}{\sqrt{2}\sigma_x}\right)$ , we have
$$
P r (\max _ {i = 1, \ldots , p} | X _ {i} | \leq t) = \operatorname {e r f} \left(\frac {t}{\sqrt {2} \sigma_ {x}}\right) ^ {p} = \left[ 2 \Phi \left(\frac {x}{\sigma_ {x}}\right) - 1 \right] ^ {p},
$$
and probability density function  $g(x) = 2p[\Phi(\frac{x}{\sigma_x}) - 1]^{p-1} \frac{1}{\sqrt{2\pi} \sigma_x} e^{-\frac{x^2}{2\sigma_x^2}}$ . When  $Y$  is independent of all  $X_i$ 's (which gives the upper bound), we have
$$
\begin{array}{l} P r (\max  _ {i = 1, \dots , p} | X | > | Y |) = \int_ {0} ^ {\infty} 2 p \left[ \Phi \left(\frac {x}{\sigma_ {x}}\right) - 1 \right] ^ {p - 1} \frac {1}{\sqrt {2 \pi} \sigma_ {x}} e ^ {- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}} P r (| Y | <   x) d x \\ = \int_ {0} ^ {\infty} 2 p \left[ \Phi \left(\frac {x}{\sigma_ {x}}\right) - 1 \right] ^ {p - 1} \frac {1}{\sqrt {2 \pi} \sigma_ {x}} e ^ {- \frac {x ^ {2}}{2 \sigma_ {x} ^ {2}}} \operatorname {e r f} \left(\frac {x}{\sqrt {2} \sigma_ {y}}\right) d x \\ = \int_ {0} ^ {\infty} 2 p [ 2 \Phi (t) - 1 ] ^ {p - 1} [ 2 \Phi (r t) - 1 ] \phi (t) d t, \\ \end{array}
$$
with a proper change of variables. This gives an upper bound as shown above.
E.2 Proof of Proposition 4.3
Proof. Consider a single Gaussian projection vector  $w$  with iid  $N(0,1)$  entries. Since  $w^T u = \sum_{i=1}^{p} u_i w_i$  and each  $w_i \sim N(0,1)$ , we know that  $\binom{\beta w_i}{x} \sim N\left(\frac{\beta^2}{\rho_i \beta \|u\|} \frac{\rho_i \beta \|u\|}{\|u\|^2}\right)$  where  $\rho_i = \frac{u_i}{\|u\|}$  is the correlation coefficient. Since  $|w^T (u - u')| \leq \beta \max_{i=1,\dots,p} |w_i|$  by Definition 2.2 of  $\beta$ -neighboring (and more generally, when  $\|u - u'\|_1 \leq \beta$ ), we have
$$
P r(\max_{u^{\prime}\in Nb(u)}|w^{T}(u - u^{\prime})|\geq |w^{T}u|) = P r(\beta \max_{i = 1,\ldots ,p}|w_{i}|\geq |w^{T}u|).
$$
Note that,  $\beta \max_{i=1,\ldots,p} |w_i| \geq |w^T u|$  is a necessary condition for the event that there exists a neighbor such that  $\text{sign}(w^T u) \neq \text{sign}(w^T u')$ . Denote  $I = \mathbb{1}\{\beta \max_{i=1,\ldots,p} |w_i| \geq |w^T u|\}$ . Applying Lemma 4.2 with  $r = \beta / \|u\| \leq 1$  yields
$$
\begin{array}{l} \mathbb {E} [ I ] = \Pr (\beta \max  _ {i = 1, \dots , p} | w _ {i} | \geq | w ^ {T} u |) \\ \leq F _ {\| u \|, p} = \int_ {0} ^ {\infty} 2 p [ 2 \Phi (t) - 1 ] ^ {p - 1} [ 2 \Phi (r t) - 1 ] \phi (t) d t \tag {20} \\ \end{array}
$$
as given by (5). Let  $I_{j}$  be the corresponding indicator function w.r.t. each column in the projection matrix  $W$ . Denote  $N_{+} = \sum_{j=1}^{k} I_{j}$ , and by the above reasoning, we know that  $|S| \leq N_{+}$  where  $S$  is defined in the theorem. Since the columns of  $W$  are independent,  $N_{+}$  follows a Binomial  $(k, \mathbb{E}[I])$  distribution with  $k$  trials and success probability  $\mathbb{E}[I]$  bounded as above. Therefore, with probability  $1 - \delta$ , we have  $N_{+} \leq B^{-1}(\delta; F_{\|u\|,p}, k)$ , where  $B^{-1}$  is the inverse cdf of the Binomial distribution. This proves the claim as desired.
Proof. Let  $s = \text{sign}(W^T u) \in \{-1, +1\}^k$ ,  $s' = \text{sign}(W^T u') \in \{-1, +1\}^k$ . We denote the collision probability of non-private SignRP as
$$
P _ {S R P} = P r (s _ {1 j} = s _ {2 j}) = 1 - \frac {\cos^ {- 1} (\rho)}{\pi} = 1 - \frac {\theta}{\pi}.
$$
Hence, the collision probability of DP-SignRP-RR can be computed as
$\tilde{P} \coloneqq Pr(\tilde{s}_{1j} = \tilde{s}_{2j}) = Pr(s_{1j} = s_{2j}$ , both change sign or not change sign)
$$
\begin{array}{l} + P r \left(s _ {1 j} \neq s _ {2 j}, \text {o n e s i g n c h a n g e s}\right) \\ = P _ {S R P} \left[ \left(\frac {e ^ {\epsilon^ {\prime}}}{e ^ {\epsilon^ {\prime}} + 1}\right) ^ {2} + \left(\frac {1}{e ^ {\epsilon^ {\prime}} + 1}\right) ^ {2} \right] + 2 \left(1 - P _ {S R P}\right) \frac {e ^ {\epsilon^ {\prime}}}{\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}} \\ = P _ {S R P} \frac {\left(e ^ {\epsilon^ {\prime}} - 1\right) ^ {2}}{\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}} + \frac {2 e ^ {\epsilon^ {\prime}}}{\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}}, \\ \end{array}
$$
which increases linearly in  $P_{SRP}$ . Thus, it holds that
$$
\mathbb {E} [ \hat {P} _ {R R} ] = \frac {(e ^ {\epsilon^ {\prime}} + 1) ^ {2}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {2}} \tilde {P} - \frac {2 e ^ {\epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {2}} = P _ {S R P} = 1 - \frac {\theta}{\pi},
$$
which implies  $\mathbb{E}[\hat{\theta}_{RR}] = \pi \left(1 - (1 - \frac{\theta}{\pi})\right) = \theta$ . To compute the variance, we first estimate  $\theta = \cos^{-1}(\rho)$  by
$$
\hat {\theta} = \pi (1 - \hat {P} _ {R R}).
$$
Then according to the Central Limit Theorem (CLT), for the sample mean of iid Bernoulli's, as  $k \to \infty$ , we have
$$
\frac {1}{k} \sum_ {j = 1} ^ {k} \mathbb {1} \{\tilde {s} _ {1 j} = \tilde {s} _ {2 j} \} \rightarrow N (\tilde {P}, \frac {\tilde {P} (1 - \tilde {P})}{k}).
$$
As a result, we have  $\hat{\theta} \to N(\theta, \frac{V_{RR}}{k})$ , where
$$
\begin{array}{l} V _ {R R} = \frac {\pi^ {2} \left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {4}}{\left(e ^ {\epsilon^ {\prime}} - 1\right) ^ {4}} \left[ \left(1 - \frac {\theta}{\pi}\right) \frac {\left(e ^ {\epsilon^ {\prime}} - 1\right) ^ {2}}{\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}} + \frac {2 e ^ {\epsilon^ {\prime}}}{\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}} \right] \left[ \frac {e ^ {2 \epsilon^ {\prime}} + 1}{\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}} - \left(1 - \frac {\theta}{\pi}\right) \frac {\left(e ^ {\epsilon^ {\prime}} - 1\right) ^ {2}}{\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}} \right] \\ = \frac {\pi^ {2} (e ^ {\epsilon^ {\prime}} + 1) ^ {4}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {4}} \Big [ (1 - \frac {\theta}{\pi}) \frac {(e ^ {\epsilon^ {\prime}} - 1) ^ {2}}{(e ^ {\epsilon^ {\prime}} + 1) ^ {2}} + \frac {2 e ^ {\epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} + 1) ^ {2}} \Big ] \Big [ \frac {\theta}{\pi} \frac {(e ^ {\epsilon^ {\prime}} - 1) ^ {2}}{(e ^ {\epsilon^ {\prime}} + 1) ^ {2}} + \frac {2 e ^ {\epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} + 1) ^ {2}} \Big ] \\ = \frac {\pi^ {2} \theta}{\pi} (1 - \frac {\theta}{\pi}) + (1 - \frac {\theta}{\pi}) \frac {2 e ^ {\epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {2}} + \frac {\theta}{\pi} \frac {2 e ^ {\epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {2}} + \frac {4 e ^ {2 \epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {4}} \\ = \theta (\pi - \theta) + \frac {2 \pi^ {2} e ^ {\epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {2}} + \frac {4 \pi^ {2} e ^ {2 \epsilon^ {\prime}}}{(e ^ {\epsilon^ {\prime}} - 1) ^ {4}}. \\ \end{array}
$$
We conclude the proof by replacing  $\epsilon' = \epsilon / N_{+}$ .
E.4 Proof of Theorem 4.5
Proof. Let us consider a single projection vector  $w_{j} = W_{[.,j]}$ . Denote  $x_{j} = w_{j}^{T}u$  and  $x_{j}^{\prime} = w_{j}^{T}u^{\prime}$  for a neighboring data  $u^{\prime}$  of  $u$ , and  $s_j = \text{sign}(x_j)$ ,  $s_j' = \text{sign}(x_j')$ . Also, let  $L_{j} = \lceil \frac{|x_{j}|}{\beta \max_{i=1,\dots,p} |W_{ij}|} \rceil$  and  $L_{j}' = \lceil \frac{|x_{j}'|}{\beta \max_{i=1,\dots,p} |W_{ij}|} \rceil$ . W.l.o.g., we can assume  $s_j = 1$  by the symmetry of random projection and the symmetry of DP. Consider two cases:
- Case I:  $L_{j} \geq 2$ . In this case, we know that  $s_{j}^{\prime} = s_{j}$ , i.e., the change from  $u$  to  $u^{\prime}$  will not change the sign of the projection. Thus, in Algorithm 4, we have
$$
\frac {P r (\tilde {s} _ {j} = 1)}{P r (\tilde {s} _ {j} ^ {\prime} = 1)} = \exp (\frac {L _ {j} - L _ {j} ^ {\prime}}{k} \epsilon) \frac {\exp (\frac {L _ {j} ^ {\prime}}{k} \epsilon) + 1}{\exp (\frac {L _ {j}}{k} \epsilon) + 1}.
$$
By the definition of  $\beta$ -adjacency,  $|L_j - L_j'|$  equals either 0 or 1. When  $L_j = L_j'$ ,  $\frac{Pr(\tilde{s}_j = 1)}{Pr(\tilde{s}_j' = 1)} = 1$ . When  $L_j - L_j' = 1$ , we have
$$
\frac {P r (\tilde {s} _ {j} = 1)}{P r (\tilde {s} _ {j} ^ {\prime} = 1)} = \frac {\exp \left(\frac {L _ {j}}{k} \epsilon\right) + \exp \left(\frac {1}{k} \epsilon\right)}{\exp \left(\frac {L _ {j}}{k} \epsilon\right) + 1}.
$$
Hence, we have  $1 \leq \frac{Pr(\tilde{s}_j = 1)}{Pr(\tilde{s}_j' = 1)} \leq e^{\frac{\epsilon}{k}}$  by the numeric identity  $1 \leq \frac{a + c}{b + c} \leq \frac{a}{b}$  for  $a \geq b > 0$  and  $c > 0$ . Thus, by symmetry,  $e^{-\frac{\epsilon}{k}} \leq \frac{Pr(\tilde{s}_j = 1)}{Pr(\tilde{s}_j' = 1)} \leq e^{\frac{\epsilon}{k}}$ . On the other hand,