SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With the high probability bound on the  $l_{2}$ -sensitivity  $\Delta_{2}$ , we have the following result.
Theorem A.3 (Analytic DP-RP-G). For any  $\epsilon > 0$  and  $0 < \delta < \frac{1}{2}$ , Algorithm 1 achieves  $(\epsilon, \delta)$ -DP when  $\sigma \geq \frac{\beta \sqrt{1 + 2 \sqrt{\log(2p / \delta) / k} + 2 \log(2p / \delta) / k}}{\epsilon} \sqrt{2(\log(2 / \delta) + \epsilon)}$ .
Proof. By Lemma A.2, we know that with probability  $\delta / 2$ ,  $\Delta_2$  is bounded by  $\beta \sqrt{1 + 2\sqrt{\log(2p / \delta) / k} + 2\log(2p / \delta) / k}$ . In this event, by Theorem 3.1, Algorithm 1 is  $(\epsilon, \delta / 2)$ -DP when  $\sigma \geq \Delta_2 \frac{\sqrt{2(\log(2 / \delta) + \epsilon)}}{\epsilon}$ . Therefore, the approach is  $(\epsilon, \delta)$ -DP.
We remark that, since this analytic version is built upon a high probability bound on the sensitivity, in most cases, the noise variance in Theorem A.3 will be slightly larger than the one computed by the realized/exact sensitivity in Theorem 3.1.
A.2 The Laplace Noise Mechanism for DP-RP
We briefly discuss the Laplace mechanism for DP-RP (and why we did not use it).
Definition A.1 ( $l_{1}$ -sensitivity). Let  $Nb(u)$  denote the neighbor set of  $u$ , and  $\mathcal{N}(\mathcal{U}) = \{(u, u') : u' \in Nb(u), u, u' \in \mathcal{U}\}$  be the collection of all possible neighboring data pairs. The  $l_{1}$ -sensitivity of a function  $f: \mathcal{U} \mapsto \mathbb{R}^{k}$  is defined as  $\triangle_1 = \max_{(u, u') \in \mathcal{N}(\mathcal{U})} \| f(u) - f(u') \|_1$ .
Theorem A.4 (Laplace mechanism [30]). Given any function  $f: \mathcal{U} \mapsto \mathbb{R}^k$  and  $\epsilon > 0$ , the randomized algorithm  $\mathcal{M}(u) = f(u) + (Z_1, \dots, Z_k)$  is  $\epsilon$ -DP, where  $Z_1, \dots, Z_k$  follow iid Laplace  $(\Delta_1 / \epsilon)$ . The centered Laplace distribution with scale  $\lambda$  has density function  $g(x) = \frac{1}{2\lambda} e^{-\frac{|x|}{\lambda}}$ .
Algorithm 6: DP-RP-L
1 Input: Data  $u\in [-1,1]^p$  , privacy parameters  $\epsilon >0$  , number of projections  $k$
2 Output:  $\epsilon$ -differentially private random projections  $\tilde{x} \in \mathbb{R}^k$
3 Apply RP  $x = \frac{1}{\sqrt{k}} W^T u$ , where  $W \in \mathbb{R}^{p \times k}$  is a random  $N(0,1)$  matrix
4 Compute sensitivity  $\Delta_{1}$  by Definition A.1
5 Generate iid random noise vector  $L\in \mathbb{R}^k$  following Laplace  $\left(\frac{\Delta_1}{\epsilon}\right)$
6 Return  $\tilde{x} = x + L$
By Theorem A.4, one may add  $Laplace(\lambda)$  noise to the projected data with  $\lambda = \Delta_1 / \epsilon$  to achieve  $\epsilon$ -DP. For completeness, we detail the approach in Algorithm 6, which is the same as Algorithm 1 except for the noise added.  $\Delta_1$  is the  $l_1$ -sensitivity for a given  $W$  given in Definition A.1:  $\Delta_1 = \frac{1}{\sqrt{k}} \beta \max_{i=1,\dots,p} \| W_{[i,:]} \|_1$ . Each term in the maximum operator is a sum of  $k$  iid standard half-normal random variables ( $Y = |X|$  where  $X$  is standard normal). The readers might be interested in a comparison between the noise level of the two noise addition mechanisms, i.e., DP-RP-G and DP-RP-L. To observe the dependence of the noise level  $\lambda$  on  $\epsilon, k$  and  $p$ , we derive a tail bound on the  $l_1$ -sensitivity. We first establish the following lemma.
Lemma A.5 (Half-normal tail bound). Let  $X_{1}, \ldots, X_{n}$  be iid  $N(0,1)$  random variables and let  $Y_{i} = |X_{i}|$ ,  $\forall i$ . Denote  $Z = \sum_{i=1}^{n} Y_{i}$ . Then for any  $t > 0$ , it holds that
$$
P r (Z \geq \sqrt {2 n ^ {2} \log 2 + 2 n t}) \leq \exp (- t).
$$
Proof. By the Markov inequality, we write (with some  $\zeta > 0$ )
$$
P r (Z \geq t) = P r (\exp (\zeta \sum_ {i = 1} ^ {n} Y _ {i}) \geq \exp (\zeta t)) \leq \frac {\mathbb {E} [ \exp (\zeta \sum_ {i = 1} ^ {n} Y _ {i} ]}{\exp (\zeta t)}.
$$
By the independence of  $Y_{i}$  and the moment generating function of half-normal distribution, we have
$$
\begin{array}{l} \mathbb {E} \left[ \right. \exp \left( \right.\zeta \sum_ {i = 1} ^ {n} Y _ {i} \left. \right] = \prod_ {i = 1} ^ {n} \mathbb {E} \left[ \exp \left(\zeta Y _ {i}\right)\right] = \exp \left(\frac {n \zeta^ {2}}{2}\right) (2 \Phi (\zeta)) ^ {n} = \exp \left(\frac {n \zeta^ {2}}{2} + n \log 2 \Phi (\zeta)\right) \\ \leq \exp (\frac {n \zeta^ {2}}{2} + n \log 2). \\ \end{array}
$$
$$
P r (Z \geq t) \leq \exp (\frac {n \zeta^ {2}}{2} - t \zeta + n \log 2),
$$
which is minimized at  $\zeta = \frac{t}{n}$ , leading to
$$
P r (Z \geq t) \leq \exp \left(- \frac {t ^ {2}}{2 n} + n \log 2\right).
$$
Taking  $t = \sqrt{2n^2\log 2 + 2nt}$  completes the proof.
Using Lemma A.5, we obtain a high probability bound on the  $l_{1}$ -sensitivity as follows. The proof is omitted since it is similar to the proof of Lemma A.2.
Lemma A.6 (Bounding  $\Delta_1$ ). Let  $W \in \mathbb{R}^{p \times k}$  be a random matrix with iid  $N(0,1)$  entries, and  $\Delta_1$  be defined in Definition A.1. Then for any  $0 < \delta < 1$ ,
$$
P r \left(\Delta_ {1} \geq \beta \sqrt {2 k \log 2 + 2 \log (p / \delta)}\right) \leq \delta .
$$
It is important to point out that, unlike in the Gaussian mechanism, simply inserting the upper bound in Lemma A.6 to replace the (exact)  $\Delta_{1}$  in the noise level of  $Laplace(\Delta_1 / \epsilon)$  in Algorithm 6 does not provide pure  $\epsilon$ -DP. In other words, we must compute the actual  $\Delta_{1}$  by Definition A.1 for a realization
of the projection matrix  $W$ . This is because, using the high probability bound of  $\Delta_{1}$  would introduce a failure probability (i.e., when the actual  $\Delta_{1}$  is larger than the bound) which violates  $\epsilon$ -DP.
Comparison with DP-RP-G-OPT. Lemma A.6 provides an estimate of the noise level required by the Laplace mechanism. We can roughly compare DP-RP-L with DP-RP-OPT in terms of the noise variances, which equal  $\sigma^2$  for Gaussian and  $2\lambda^2$  for Laplace, respectively. Lemma A.6 shows that in most cases, the Laplace noise requires  $\lambda = \mathcal{O}\left(\frac{\sqrt{k}}{\epsilon}\right)$ . Recall that in DP-RP-G and DP-RP-G-OPT, the Gaussian noise level  $\sigma = \mathcal{O}\left(\frac{1}{\epsilon}\right)$  when  $\epsilon \to 0$  and  $\sigma = \mathcal{O}\left(\frac{1}{\sqrt{\epsilon}}\right)$  when  $\epsilon \to \infty$ . Thus, the Gaussian noise would be smaller with relatively small  $\epsilon$ , and the Laplace noise would be smaller only when  $\epsilon = \Omega(k)$ , which is usually too large to be useful since in practice  $k$  is usually hundreds to thousands. As a result, we would expect DP-RP-G methods to perform better than DP-RP-L in common scenarios. Therefore, in this work, we mainly focused on the Gaussian mechanism for DP-RP noise addition.
B Comparison of Different Projection Matrices and the Benefits of Rademacher RP
Besides Gaussian random projection, we can also adopt other types of projection matrices which might even work better for DP. The following distributions of  $w_{ij}$  are popular:
- The uniform distribution,  $\sqrt{3} \times \text{unif}[-1, 1]$ . The  $\sqrt{3}$  factor is placed here to have  $\mathbb{E}(w_{ij}^2) = 1$  by following the convention in the practice of random projections.
- The "very sparse" distribution, as used in [56]:
$$
w _ {i j} = \sqrt {s} \times \left\{ \begin{array}{r l} - 1 & \text {w i t h p r o b .} \quad 1 / (2 s) \\ 0 & \text {w i t h p r o b .} \quad 1 - 1 / s, \\ + 1 & \text {w i t h p r o b .} \quad 1 / (2 s) \end{array} \right. \tag {7}
$$
which generalizes [2] (for  $s = 1$  and  $s = 3$ ). Note that when  $s = 1$ , it is also called the "symmetric Bernoulli" distribution or the "Rademacher" distribution.
Next, we compare these various types of projection matrices and show that Rademacher (symmetric Bernoulli) random projection is superior to Gaussian random projection for both DP-RP and DP-SignRP in that less perturbation is required to achieve the same privacy level.
B.1 Rademacher Projection for DP-RP
From Theorem 3.1 and Theorem 3.2, it is clear that the noise magnitude of Gaussian noise in DP-RP directly depends on the  $l_{2}$ -sensitivity  $\triangle_{2}$ , which, according to (3), equals the largest row norm of the projection matrix  $W$ . Among the above mentioned distributions, the dense Rademacher projection  $(s = 1$  in (7)) has  $\triangle_{2} = \frac{1}{\sqrt{k}}\beta \times \sqrt{k} = \beta$  which is independent of  $p$ . This could be much smaller than the dense Gaussian projection (i.e., DP-RP-G-OPT).
Figure 5: The  $l_{2}$ -sensitivity  $\triangle_2$  (3) for different types of random projection matrices against the data dimensionality  $p$ , at  $k = 256$  and  $k = 512$ , respectively.  $\beta = 1$ .
In Figure 5, we numerically simulate the  $\triangle_2$  of different projection matrices, which shows that the Rademacher projection produces the smallest sensitivity. This, when plugged into the optimal Gaussian mechanism (Theorem 3.2), leads to smaller Gaussian noise variance needed.
B.2 Rademacher Projection for DP-SignRP
From our analysis, it is clear that the flipping probability of DP-SignRP (both DP-SignRP-RR and DP-SignRP-RR-smooth) essentially depends on how concentrated the projected data is around zero. Particularly,  $N_{+}$  in Algorithm 3, as given in Proposition 4.3, is a high probability upper bound on a Binomial random variable with success probability  $Pr(\beta \max_{i=1,\dots,p} |w_i| \geq |w^T u|)$  with  $w \sim N(0,1)$ . In Algorithm 4,  $L_j = \lceil \frac{|w_j^T u|}{\beta \max_{i=1,\dots,p} |W_{ij}|} \rceil$ . For both quantities, a smaller value leads to a smaller sign flipping probability and thus better utility.
$N_{+}$  in DP-SignRP-RR. We first consider the  $N_{+}$  in Algorithm 3, which determines the flipping probability  $\frac{1}{e^{\varepsilon / N_{+} + 1}}$ . Particularly,  $N_{+}$  in Algorithm 3, as given in Proposition 4.3, is a high probability upper bound on a Binomial random variable with success probability
$$
P _ {+} = \Pr \left(\beta \max  _ {i = 1, \dots , p} | w _ {i} | \geq | w ^ {T} u |\right), \tag {8}
$$
where  $w$  is the  $p$ -dimensional projection vector. When  $w_{i}$  is sampled from the Rademacher distribution, i.e.,  $w_{i} \in \{-1, + 1\}$  with equal probabilities, the probability calculation can be simplified:
$$
P _ {+, b} = \Pr \left(\beta \max  _ {i = 1, \dots , p} | w _ {i} | \geq \left| \sum_ {i = 1} ^ {p} w _ {i} u _ {i} \right|\right) = \Pr \left(\beta \geq \left| \sum_ {i = 1} ^ {p} w _ {i} u _ {i} \right|\right) \approx 2 \Phi \left(\frac {\beta}{\| u \|}\right) - 1. \tag {9}
$$
Based on the central limit theorem, the normal approximation (9) is accurate unless  $p$  is very small. Recall that, when  $w_{i}$ 's are sampled from the Gaussian distribution, we can calculate an upper bound in (20), which is re-written as below:
$$
P _ {+, g} = \Pr \left(\beta \max  _ {i = 1, \dots , p} | w _ {i} | \geq \left| \sum_ {i = 1} ^ {p} w _ {i} u _ {i} \right|\right) \leq \int_ {0} ^ {\infty} 2 p [ 2 \Phi (t) - 1 ] ^ {p - 1} [ 2 \Phi (\beta t / \| u \|) - 1 ] \phi (t) d t. \tag {10}
$$
Next, we provide a simulation study to justify the approximation and compare different distributions in terms of their impact on the probability (8), for  $\beta = 1$  as well as  $\beta = 0.1$ . For simplicity, we simulate the data as a  $p$ -dimensional vector of uniform random numbers sampled from unif[-1, 1]. We experiment with five different choices of  $w$ : the standard Gaussian, the uniform, the "very sparse" distribution (7) with  $s = 1$ ,  $s = 3$ , and  $s = 10$ . We vary  $p$  from 10 to 1000. For each case, we repeat the simulations  $10^{7}$  times to ensure sufficient accuracy. Figure 6 verifies that the two approximations (9) and (10) are accurate. In Figure 7, we provide the curves for more types of projection matrices. From both figures, we clearly see that using the Rademacher projection can considerably reduce (8) compared with Gaussian (and other) projections, leading to smaller  $N_{+}$  value. This typically implies better utility.
$L_{j}$  in DP-SignRP-RR-smooth. Similarly, we numerically evaluate the  $L_{j}$  in Algorithm 4. We run Algorithm 4 with  $k = 512$ , which gives  $512L_{j}$  values. In Figure 8, we plot the proportion (or the approximated distribution) of the values of  $L_{j}$  among  $k$  projections. As we see, Rademacher projection produces least number of small  $L_{j}$  values, and largest number of higher  $L_{j}$  values. As the smooth flipping probability equals  $\frac{1}{\exp\left(\frac{L_j}{k}\epsilon\right) + 1}$ , larger  $L_{j}$  leads to smaller probability of sign flipping. Hence, Rademacher is again the best choice for the projection matrix.
Figure 6: Simulations for evaluating (9) and (10), using two choices for  $w$ : the Gaussian distribution and the Rademacher distribution (i.e., (7) with  $s = 1$ ). We plot the two upper bounds (9) and (10) as black dashed curves, which both overlap with their corresponding simulations.
Figure 7: Simulations (same as in Figure 7) for evaluating (8), using five different choices for  $w$ : the Gaussian, the uniform, the "very sparse" distribution (7) with  $s = 1,3$  and 10.  $s = 1$  is the Rademacher distribution. The data vector is simulated by sampling each entry from  $unif[-1,1]$ .
C Comparison of DP-RP and DP-OPORP on Inner Product Estimation
In this section, we theoretically compare DP-RP and DP-OPORP. In Section B, we have shown that Rademacher projection requires lowest noise magnitude. Thus, we consider DP-RP with Rademacher projections here. For clarity, we summarize the algorithm in Algorithm 7. We name it "DP-RP-G-OPT-B", where "G" stands for the Gaussian noise mechanism and "B" stands for "symmetric Bernoulli" projections.
Algorithm 7: DP-RP-G-OPT-B
1 Input: Data  $u\in [-1,1]^p$  , privacy parameters  $\epsilon >0$ $\delta \in (0,1)$  , number of projections  $k$
2 Output:  $(\epsilon, \delta)$ -differentially private random projections  $\tilde{x} \in \mathbb{R}^k$
3 Apply  $\mathrm{RP}x = \frac{1}{\sqrt{k}} W^T u$ , where  $W \in \mathbb{R}^{p \times k}$  is a random Rademacher matrix
4 Generate iid random noise vector  $G \in \mathbb{R}^k$  following  $N(0, \sigma^2)$  where  $\sigma$  is obtained by Theorem 3.2 with  $\triangle_2 = \beta$
5 Return  $\tilde{x} = x + G$
In our analysis, for simplicity we assume the data are normalized, i.e., the data vector has  $l_{2}$  norm equal to 1. In this case, the inner product is also the cosine. The baseline method is the most straightforward: we add optimal Gaussian noise to each dimension of the original data (Raw-data-G-OPT). For this strategy, the sensitivity is also  $\triangle_{2} = \beta$ . This means, when we compare all three methods: Raw-data-G-OPT, DP-RP-G-OPT-B, and DP-OPORP, the noise level  $\sigma$  is the same. This makes it convenient to conduct the comparisons, from which we can gain valuable insights.
Figure 8: Simulations for evaluating  $L_{j}$  in Algorithm 4, using different choices for  $w$ .  $s = 1$  is the Rademacher distribution. Left:  $p = 100$ , right:  $p = 1000$ . The  $y$ -axis is the proportion (normalized histogram) of the values of all the  $L_{j}$ ,  $j = 1, \dots, k$  computed using  $k = 512$  projected samples.
Theorem C.1 (Raw-data-G-OPT, i.e., adding optimal Gaussian noise on raw data). Let  $\sigma$  be the solution to (4) with  $\triangle_2 = \beta$ . For any  $u, v \in \mathcal{U}$ , let  $\tilde{u}_i = u_i + a_i$  and  $\tilde{v}_i = v_i + b_i$  be the DP noisy vectors, with  $a_i, b_i \sim N(0, \sigma^2)$  i.i.d. Then, denote  $\hat{g}_{org} = \sum_{i=1}^{p} \tilde{u}_i \tilde{v}_i$ . We have
$$
\mathbb {E} \left[ \hat {g} _ {o r g} \right] = \sum_ {i = 1} ^ {p} u _ {i} v _ {i}, \quad \operatorname {V a r} \left(\hat {g} _ {o r g}\right) = \sigma^ {2} \sum_ {i = 1} ^ {p} \left(u _ {i} ^ {2} + v _ {i} ^ {2}\right) + p \sigma^ {4}. \tag {11}
$$
Proof. To add Gaussian noise to the original data, it suffices to find the sensitivity, which, by Definition 2.2, is  $\triangle_2 = \beta$ . Thus, the approach is  $(\epsilon, \delta)$ -DP according to the optimal Gaussian mechanism (Theorem 3.2). To compute the mean and variance, consider some  $i \in [p]$ . We have
$$
\mathbb {E} \left[ \left(u _ {i} + a _ {i}\right) \left(v _ {i} + b _ {i}\right) \right] = \mathbb {E} \left[ u _ {i} v _ {i} + a _ {i} v _ {i} + b _ {i} u _ {i} + a _ {i} b _ {i} \right] = u _ {i} v _ {i}.
$$
Thus, taking the sum implies  $\mathbb{E}[\hat{g}_{org}] = \sum_{i=1}^{p} u_i v_i$ . For the variance,
$$
\mathbb {E} \left[ \left(u _ {i} + a _ {i}\right) \left(v _ {i} + b _ {i}\right) \right] ^ {2} = \mathbb {E} [ u _ {i} v _ {i} + a _ {i} v _ {i} + b _ {i} u _ {i} + a _ {i} b _ {i} ] ^ {2} = u _ {i} ^ {2} v _ {i} ^ {2} + \sigma^ {2} \left(u _ {i} ^ {2} + v _ {i} ^ {2}\right) + \sigma^ {4},
$$
$$
V a r \left(\left(u _ {i} + a _ {i}\right) \left(v _ {i} + b _ {i}\right)\right) = \sigma^ {2} \left(u _ {i} ^ {2} + v _ {i} ^ {2}\right) + \sigma^ {4}.
$$
Therefore, by independence,
$$
V a r \left(\hat {g} _ {o r g}\right) = V a r \left(\sum_ {i = 1} ^ {p} \left(u _ {i} + a _ {i}\right) \left(v _ {i} + b _ {i}\right)\right) = \sigma^ {2} \sum_ {i = 1} ^ {p} \left(u _ {i} ^ {2} + v _ {i} ^ {2}\right) + p \sigma^ {4},
$$
For DP-RP-G-OPT-B and DP-OPORP, we have the following results.
Theorem C.2 (DP-RP-G-OPT-B inner product estimation). Let  $\sigma$  be the solution to (4) with  $\triangle_2 = \beta$ . In Algorithm 7, let  $W \in \{-1,1\}^{p \times k}$  be a Rademacher random matrix. Denote  $x = \frac{1}{\sqrt{k}} W^T u$ ,  $y = \frac{1}{\sqrt{k}} W^T v$ , and  $a, b$  are two random Gaussian noise vectors following  $N(0,\sigma^2)$ . Let  $\hat{g}_{rp} = \sum_{j=1}^{k}(x_j + a_j)(y_j + b_j)$ . Then,  $\mathbb{E}[\hat{g}_{rp}] = \sum_{i=1}^{p}u_i v_i$ , and
$$
V a r \left(\hat {g} _ {r p}\right) = \sigma^ {2} \sum_ {i = 1} ^ {p} \left(u _ {i} ^ {2} + v _ {i} ^ {2}\right) + k \sigma^ {4} + \frac {1}{k} \left(\sum_ {i = 1} ^ {p} u _ {i} ^ {2} \sum_ {i = 1} ^ {p} v _ {i} ^ {2} + \left(\sum_ {i = 1} ^ {p} u _ {i} v _ {i}\right) ^ {2} - 2 \sum_ {i = 1} ^ {p} u _ {i} ^ {2} v _ {i} ^ {2}\right). \tag {12}
$$
Proof. The conditional mean and variance can be computed as
$$
\mathbb {E} \left[ \sum_ {j = 1} ^ {k} \left(x _ {j} + a _ {j}\right) \left(y _ {j} + b _ {j}\right) | x _ {j}, y _ {j}, j = 1, \dots , k \right] = \sum_ {j = 1} ^ {k} x _ {j} y _ {j},
$$
$$
V a r \left(\sum_ {j = 1} ^ {k} \left(x _ {j} + a _ {j}\right) \left(y _ {j} + b _ {j}\right) | x _ {j}, y _ {j}, j = 1, \dots , k\right) = \sigma^ {2} \sum_ {j = 1} ^ {k} \left(x _ {j} ^ {2} + y _ {j} ^ {2}\right) + k \sigma^ {4},
$$
where the variance calculation follows from Theorem C.1. Hence, we have
$$
\mathbb {E} \left[ \sum_ {j = 1} ^ {k} \left(x _ {j} + a _ {j}\right) \left(y _ {j} + b _ {j}\right) \right] = \mathbb {E} \left[ \sum_ {j = 1} ^ {k} x _ {j} y _ {j} \right] = \sum_ {i = 1} ^ {p} u _ {i} v _ {i},
$$
$$
\begin{array}{l} V a r \left(\hat {g} _ {r p}\right) = \mathbb {E} \left[ \sigma^ {2} \sum_ {j = 1} ^ {k} \left(x _ {j} ^ {2} + y _ {j} ^ {2}\right) + k \sigma^ {4} \right] + V a r \left(\sum_ {j = 1} ^ {k} x _ {j} y _ {j}\right) \\ = \sigma^ {2} \sum_ {i = 1} ^ {p} \left(u _ {i} ^ {2} + v _ {i} ^ {2}\right) + k \sigma^ {4} + \frac {1}{k} \left(\sum_ {i = 1} ^ {p} u _ {i} ^ {2} \sum_ {i = 1} ^ {p} v _ {i} ^ {2} + \left(\sum_ {i = 1} ^ {p} u _ {i} v _ {i}\right) ^ {2} - 2 \sum_ {i = 1} ^ {p} u _ {i} ^ {2} v _ {i} ^ {2}\right). \tag {13} \\ \end{array}
$$
In the above calculation, the formula of  $Var\left(\sum_{j=1}^{k} x_j y_j\right)$  is from the result in [56] with  $s = 1$  for Rademacher distribution.
Algorithm 8: DP-OPORP
1 Input: Data  $u \in [-1, 1]^p$ , privacy parameters  $\epsilon > 0$ ,  $\delta \in (0, 1)$ , number of projections  $k$
2 Output: Differentially private OPORP
3 Apply Algorithm 2 with a random Rademacher projection vector to obtain the OPORP  $x$
4 Set sensitivity  $\Delta_2 = \beta$
5 Generate iid random vector  $G \in \mathbb{R}^k$  following  $N(0, \sigma^2)$  where  $\sigma$  is computed by Theorem 3.2
6 Return  $\tilde{x} = x + G$
Theorem C.3 (DP-OPORP inner product estimation). Let  $\sigma$  be the solution to (4) with  $\triangle_2 = \beta$ . Let  $w \in \{-1, 1\}^p$  be a Rademacher random vector. In Algorithm 8, let  $x$  and  $y$  be the OPORP of  $u$  and  $v$ , and  $a, b$  be two random Gaussian noise vectors following  $N(0, \sigma^2)$ . Denote  $\hat{g}_{oporp} = \sum_{j=1}^{k} (x_j + a_j)(y_j + b_j)$ . Then,  $\mathbb{E}[\hat{g}_{oporp}] = \sum_{i=1}^{p} u_i v_i$ , and
$$
\operatorname {V a r} \left(\hat {g} _ {\text {o p o r p}}\right) = \sigma^ {2} \sum_ {i = 1} ^ {p} \left(u _ {i} ^ {2} + v _ {i} ^ {2}\right) + k \sigma^ {4} + \frac {1}{k} \left(\sum_ {i = 1} ^ {p} u _ {i} ^ {2} \sum_ {i = 1} ^ {p} v _ {i} ^ {2} + \left(\sum_ {i = 1} ^ {p} u _ {i} v _ {i}\right) ^ {2} - 2 \sum_ {i = 1} ^ {p} u _ {i} ^ {2} v _ {i} ^ {2}\right) \frac {p - k}{p - 1}. \tag {14}
$$
Proof. The proof is similar to that of Theorem C.2, with the help of the result in [57].
The variance reduction factor  $\frac{p - k}{p - 1}$  can be quite beneficial when  $p$  is not very large. Also, see [57] for the normalized estimators for both OPORP and VSRP (very sparse random projections). The normalization steps can substantially reduce the estimation variance.
Comparison. For the convenience of comparison, let us assume that the data are row-normalized, i.e.,  $\| u\| ^2 = 1$  for all  $u\in \mathcal{U}$ . Let  $\rho = \sum_{i = 1}^{p}u_{i}v_{i}$ . We have
$$
V a r \left(\hat {g} _ {o r g}\right) = 2 \sigma^ {2} + p \sigma^ {4},
$$
$$
V a r \left(\hat {g} _ {r p}\right) = 2 \sigma^ {2} + k \sigma^ {4} + \frac {1}{k} \left(1 + \rho^ {2} - 2 \sum_ {i = 1} ^ {p} u _ {i} ^ {2} v _ {i} ^ {2}\right),
$$
$$
V a r \left(\hat {g} _ {o p o r p}\right) = 2 \sigma^ {2} + k \sigma^ {4} + \frac {1}{k} \left(1 + \rho^ {2} - 2 \sum_ {i = 1} ^ {p} u _ {i} ^ {2} v _ {i} ^ {2}\right) \frac {p - k}{p - 1}.
$$
For high-dimensional data (large  $p$ ), we see that  $\hat{g}_{rp}$  and  $\hat{g}_{oporp}$  has roughly the same variance, approximately  $2\sigma^2 + k\sigma^4 + \frac{1}{k}$ . We would like to compare this with  $Var(\hat{g}_{org}) = 2\sigma^2 + p\sigma^4$  the variance for adding noise directly to the original data.
Let's define the ratio of the variances:
$$
R = \frac {2 \sigma^ {2} + p \sigma^ {4}}{2 \sigma^ {2} + k \sigma^ {4} + \frac {1}{k}} \sim \frac {p \sigma^ {4}}{k \sigma^ {4}} = \frac {p}{k} \quad (\text {i f} p \text {i s l a r g e o r} \sigma \text {i s h i g h}) \tag {15}
$$
to illustrate the benefit of RP-type algorithms (DP-RP and DP-OPORP) in protecting the privacy of the (high-dimensional) data. If  $\frac{p}{k} = 100$ , then it is possible that the ratio of the variances can be roughly 100. This would be a huge advantage. Figure 9 plots the ratio  $R$  for  $p = 1000$  and  $p = 10000$  as well as a series of  $k / p$  values, with respect to  $\sigma$ .
Figure 9: We plot the ratio of variances in (15) for  $p = 1000$  and  $p = 10000$ . We choose  $k$  values with  $k / p \in \{0.01, 0.05, 0.1, 0.5\}$ . Then for any  $\sigma$  value, we are able to compute the ratio  $R$ . For larger  $\sigma$ , we have  $R \sim \frac{p}{k}$  as expected. See Figure 10 for the relationship among  $\sigma, \Delta$ , and  $\epsilon$  (and  $\delta$ ).
Figure 10: Left panel: the optimal Gaussian noise  $\sigma$  versus  $\epsilon$  for a series of  $\Delta_2$  values, by solving the nonlinear equation (4) in Theorem 3.2, for  $\delta = 10^{-6}$ . Right panel: the optimal Gaussian noise  $\sigma$  versus  $\Delta_2$  for a series of  $\epsilon$  values.