SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Smooth Flipping Probability for Differentially Private Sign Random Projection Methods
Ping Li, Xiaoyun Li
LinkedIn Ads
700 Bellevue Way NE, Bellevue, WA 98004, USA
{pingli98, lixiaoyun996}@gmail.com
We develop a series of differential privacy (DP) algorithms from a family of random projection (RP) and sign random projection (SignRP) methods. We first show how to improve the previous DP-RP approach using the "optimal Gaussian mechanism". Then, we propose a series of DP-SignRP algorithms that leverage the robustness of the "sign flipping probability" of random projections. That is, given  $x = \sum_{i=1}^{p} u_i w_i$  where  $u$  is a  $p$ -dimensional data vector and  $w$  is a symmetric random vector,  $\text{sign}(x)$  only has a fairly small probability to be flipped if there is a small modification on data  $u$ , depending on the specific distribution of  $w$ . This robustness leads to our novel design of "smooth flipping probability" for SignRP-type algorithms with better utility than using the standard randomized response mechanism. Retrieval and classification experiments demonstrate that, among the presented DP-RP algorithms, DP-SignOPORP (where OPORP is an improvement over the celebrated count-sketch algorithms), performs the best in general.
In the industrial practice, DP methods were not very popular for machine learning or search, largely because the performance typically would drop substantially if DP is applied. Since our proposed new DP algorithms have significantly improved the performance, it is anticipated that our work will motivate a wide adoption of DP in practice. Finally, we stress that, since our methods are applied to the original data (i.e., feature vectors), the privacy of downstream tasks is naturally protected.
Protecting data privacy has become an urgent need and a trending research topic recently. Among many notions of privacy, the "differential privacy" (DP) [30] has gained tremendous attention in both the research community and industrial applications, and has been widely applied to numerous tasks such as frequency estimation [32, 26], clustering [36, 34], regression and classification [14, 85], DP-SGD [1, 35], principle component analysis [40], empirical risk minimization [15], graph analysis [33], matrix completion [8, 45], etc. In a machine learning model, there are several choices on when to deploy differential privacy (DP): (a) at the data collection/processing stage [84, 81, 5, 19]; (b) during the model training stage [1, 78]; and (c) on the model output (or summary statistics) [29]. Among them, applying DP as early as at the data collection stage provides strong protection in the sense that the subsequent operations or outputs will become private by the post-processing property of DP. Moreover, when the data holder needs to release the data to third-parties, a privacy-preserving data publishing mechanism (at an early stage) becomes necessary.
In this paper, we focus on the differential privacy of the broad family of random projection (RP) methods, including RP, sign random projections (SignRP) and count-sketch-type algorithm. Our algorithms can be applied to data publishing and processing for downstream machine learning tasks.
1.1 Random Projections (RP) and Sign Random Projections (SignRP)
In practice, data compression and dimension reduction techniques can be crucial when dealing with massive (high-dimensional) data. The random projection (RP) method is an important and fundamental dimensionality reduction algorithm. Denote  $u \in \mathbb{R}^p$  as the data vector with  $p$  features. With some  $k$  (and typically  $k \ll p$ ), we define the random projection of  $u$  as
$$
X = \frac {1}{\sqrt {k}} W ^ {T} u, W \in \mathbb {R} ^ {p \times k}. \tag {1}
$$
The entries of the random matrix  $W$  typically follow the Gaussian distribution or Gaussian-like distribution such as the Rademacher (symmetric Bernoulli) distribution. It has been well-understood that, with a sufficient number of projections  $(k)$ , the distance between two vectors (using the same projection  $W$ ) is preserved within a small multiplicative error with high probability. The dimensionality reduction and geometry preserving properties make RP widely useful in numerous applications, such as distance estimation, nearest neighbor search, clustering, classification, compressed sensing, permutation recovery, etc. [46, 44, 22, 6, 2, 37, 24, 76, 10, 28, 56, 38, 23, 21, 60, 68, 75, 82, 83].
While RP is able to reduce the dimensionality, storing and transmitting projected data might still be expensive for large datasets. One can further compress the RPs by quantization/discretization, where we only use a few bits to represent the projected values [41, 9, 27, 89, 54, 58, 50, 55, 61, 62]. In this paper, we will consider the differential privacy of the extreme case of sign (1-bit) random projection (SignRP), also known as the "SimHash" [13, 24], where only the sign of the projected data is stored. The signs still preserve the angle/cosine information. SignRP has been widely used for approximate near neighbor (ANN) search [39] by building hash tables from the bits [44, 13, 71]. RP and SignRP can also be extended to non-linear random features [69, 63, 64].
1.2 Count-Sketch and OPORP: Efficient RP-type Alternatives
The celebrated count-sketch [12] can be viewed as a highly efficient RP-type algorithm, because, as an option, it only requires "one permutation + one random projection" (OPORP) [57], as opposed to  $k$  projections in RP. Applications of count-sketch include graph embedding [80], word & image embedding [16, 87, 3, 72, 86], model & communication compression [79, 17, 70, 42, 59], etc. The recent work by [57] improved the original count-sketch in two aspects: (i) using a fixed-length binning scheme makes the algorithms more convenient and also reduces the variance by a factor of  $\frac{p - k}{p - 1}$ ; (ii) the projected data (i.e., vectors in  $k$  dimensions) should be normalized before they are used to estimate the original cosine  $\rho$ , which also substantially reduces the estimation variance, essentially from  $(1 + \rho^2) / k$  (un-normalized) to  $(1 - \rho^2)^2 / k$  (normalized).
- In Section 3, for DP-RP, we first revisit the prior work [49] on the  $(\epsilon, \delta)$ -DP random projection based on Gaussian noise addition (called DP-RP-G). Then we incorporate the optimal Gaussian mechanism and propose an improved method DP-RP-G-OPT. We also develop DP-OPORP based on the optimal Gaussian mechanism.
- In Section 4, we propose two algorithms for privatizing the (1-bit) DP-SignRP. The first method, DP-SignRP-RR, is based on the standard "randomized response" (RR). Then, we propose an improved method named DP-SignRP-RR-smooth based on a proposed concept of "smooth flipping probability", thanks to the robustness brought by the "aggregate-and-sign" operations of SignRP. Finally, we extend the idea of smooth flipping probability to OPORP and show that DP-SignOPORP is more advantageous in terms of privacy.
- In Section 5, we conduct retrieval and classification experiments on benchmark datasets. For full-precision methods, the proposed DP-OPORP substantially improves prior method DP-RP-G. For the 1-bit RP variants, the suggested DP-SignOPORP can outperform DP-RP and DP-OPORP especially when  $\epsilon$  is not large (e.g.,  $\epsilon \leq 5$ ). Our results also verify the advantage of smooth flipping probability over the classic bit flipping strategy.
2 Background on Differential Privacy
Throughout the paper,  $\| \cdot \|_1$  and  $\| \cdot \|_2$  are the  $l_1$  and  $l_2$  norms, respectively, and  $\| \cdot \|$  will denote the  $l_2$  norm if there is no risk of confusion. Let  $u \in \mathcal{U}$  be the data vector of a user where  $\mathcal{U}$  denotes the data domain. In this paper, we consider real-valued data vectors with bounded range, i.e.,  $\mathcal{U} = [-1, 1]^p$  for convenience. We also assume  $\| u \| > 0$ , i.e., there is no all-zero data sample. The formal definition of differential privacy (DP) is given as follows.
Definition 2.1 (Differential Privacy [30]). For a randomized algorithm  $\mathcal{M}:\mathcal{U}\mapsto \operatorname {Range}(\mathcal{M})$ , if for any two adjacent datasets  $u$  and  $u^{\prime}$ , it holds that
$$
P r [ \mathcal {M} (u) \in O ] \leq e ^ {\epsilon} P r [ \mathcal {M} \left(u ^ {\prime}\right) \in O ] + \delta \tag {2}
$$
for  $\forall O\subset \mathsf{Range}(\mathcal{M})$  and some  $\epsilon ,\delta \geq 0$  , then algorithm  $\mathcal{M}$  is called  $(\epsilon ,\delta)$  -differentially private. If  $\delta = 0$ $\mathcal{M}$  is called  $\epsilon$  -differentially private.
In this work, we follow the standard setup in the literature of DP sketching/Hashing (e.g., [7, 49, 73, 74, 88, 25, 65]) that neighboring data vectors  $u$  and  $u'$  only differ in one dimension, which is usually referred to as the "attribute-level DP"—an adversary cannot detect a small change in any attribute of  $u$  based on the output of the algorithm.
Definition 2.2 ( $\beta$ -adjacency). Let  $u \in [-1,1]^p$  be a data vector. A vector  $u' \in [-1,1]^p$  is said to be  $\beta$ -adjacent to  $u$  if  $u'$  and  $u$  differ in one dimension  $i$ , and  $|u_i - u_i'| \leq \beta$ .
Data vectors  $u$  and  $u'$  satisfying Definition 2.2 are called  $\beta$ -adjacent or  $\beta$ -neighboring. In the literature (e.g., [49]),  $\beta = 1$  is typically used. We allow a general  $\beta$  to accommodate the practical needs. Moreover, the concept of sensitivity is important for the design of DP noise addition mechanisms.
Definition 2.3 ( $l_{2}$ -sensitivity). Let  $Nb(u)$  denote the neighbor set of  $u$ , and  $\mathcal{N}(\mathcal{U}) = \{(u, u') : u' \in Nb(u), u, u' \in \mathcal{U}\}$  be the collection of all possible neighboring data pairs. The  $l_{2}$ -sensitivity of a function  $f: \mathcal{U} \mapsto \mathbb{R}^{k}$  is defined as  $\triangle_2 = \max_{(u, u') \in \mathcal{N}(\mathcal{U})} \| f(u) - f(u') \|_2$ .
The following composition theorem of DP is also useful in our analysis.
Theorem 2.1 (Composition Theorem [31]). Let  $\mathcal{M}_j: \mathcal{U} \to \mathbb{R}$  be an  $(\epsilon_j, \delta_j)$ -DP algorithm for  $j = 1, \dots, k$ . Then  $\mathcal{M}(u) = (\mathcal{M}_1(u), \dots, \mathcal{M}_k(u))$  is  $(\sum_{j=1}^k \epsilon_i, \sum_{j=1}^k \delta_j)$ -DP.
3 Revisiting Differentially Private Random Projection Methods
We first revisit the DP noise addition mechanism for random projections and show how to apply the optimal Gaussian mechanism [4] to improve prior methods. We also propose a DP algorithm based on a variant of count-sketch which is computationally much more efficient than dense projections.
3.1 Gaussian Noise Mechanism for DP-RP
Let  $\mathcal{N} = \{u, u' \in \mathcal{U} : u \in Nb(u)\}$  be the collection of all possible  $\beta$ -neighboring data pairs, where  $Nb(u)$  is the neighbor set of  $u$ . For a  $p$ -by- $k$  projection matrix  $W$ , the  $l_2$ -sensitivity of the RP (1), according to Definition 2.3, can be precisely computed by
$$
\triangle_ {2} = \frac {1}{\sqrt {k}} \beta \max  _ {(u, u ^ {\prime}) \in \mathcal {N} (\mathcal {U})} \| W ^ {T} u - W ^ {T} u ^ {\prime} \| _ {2} = \frac {1}{\sqrt {k}} \beta \max  _ {i = 1, \dots , p} \| W _ {[ i,: ]} \| _ {2}, \tag {3}
$$
where  $W_{[i,:]}$  denotes the  $i$ -th row of  $W$ . As such, the DP noise level depends on the choice of projection matrix  $W$ . A comparison of Rademacher vs. Gaussian projection in DP-RP method can be found in Section B.1. With this calculation, the general Gaussian noise mechanism for DP-RP is summarized in Algorithm 1. We present the algorithms for a single data point  $u$ , and the procedure is applied to every data vector in the database (with same  $W$  but independent noise). After random projection, we simply add a random Gaussian noise vector following iid  $N(0,\sigma^2)$  to the projected data. The following "DP-RP-G" result is known in the literature.
Theorem 3.1 (DP-RP-G [49]). Let  $\triangle_2$  be defined in (3). For any  $\epsilon >0$  and  $0 < \delta < \frac{1}{2}$ , DP-RP-G in Algorithm 1 is  $(\epsilon, \delta)$ -DP if  $\sigma \geq \triangle_2 \frac{\sqrt{2(\log(1 / \delta) + \epsilon)}}{\epsilon}$ .
Algorithm 1: DP-RP-G and DP-RP-G-OPT
1 Input: Data  $u\in [-1,1]^p$  , privacy parameters  $\epsilon >0$ $\delta \in (0,1)$  , number of projections  $k$
2 Output:  $(\epsilon, \delta)$ -differentially private random projections  $\tilde{x} \in \mathbb{R}^k$
3 Apply  $\mathrm{RP}x = \frac{1}{\sqrt{k}} W^T u$ , where  $W \in \mathbb{R}^{p \times k}$  has iid entries from  $N(0,1)$  or Rademacher
4 Compute the sensitivity  $\triangle_2$  by (3)
5 Generate the random noise vector  $G \in \mathbb{R}^k$  whose entries are iid samples from  $N(0, \sigma^2)$  where  $\sigma$  is obtained by Theorem 3.1 (DP-RP-G) or Theorem 3.2 (DP-RP-G-OPT)
6 Return  $\tilde{x} = x + G$
Remark 3.1. Note that in implementation,  $\triangle_2$  is computed by (3) using the realization of the projection matrix  $W$ . In Appendix A, we provide a high probability bound on  $\triangle_2$  and more analysis and discussion on the Laplace mechanism for DP-RP.
In Theorem 3.1 (as well as the classical Gaussian mechanism, e.g., [31]), the analysis on the noise level is based on upper bounding the tail of Gaussian distribution. It can be improved by computing the exact tail probabilities. [4, Theorem 8] proposed an optimal Gaussian mechanism based on this idea, which allows us to improve Theorem 3.1 and obtain the DP-RP-G-OPT method as follows.
Theorem 3.2 (DP-RP-G-OPT). Suppose  $\triangle_2$  is defined as (3). For any  $\epsilon >0$  and  $0 < \delta < 1$ , DP-RP-G-OPT in Algorithm 1 achieves  $(\epsilon ,\delta)$ -DP if  $\sigma \geq \sigma^{*}$  where  $\sigma^{*}$  is the solution to the equation
$$
\Phi \left(\frac {\triangle_ {2}}{2 \sigma} - \frac {\epsilon \sigma}{\triangle_ {2}}\right) - e ^ {\epsilon} \Phi \left(- \frac {\triangle_ {2}}{2 \sigma} - \frac {\epsilon \sigma}{\triangle_ {2}}\right) = \delta , \tag {4}
$$
where  $\Phi (\cdot)$  is the cdf of the standard normal distribution.
Remark 3.2. For both Theorem 3.2 and Theorem 3.1, the Gaussian noise level is  $\sigma = \mathcal{O}\left(\frac{\triangle_2}{\epsilon}\right)$  when  $\epsilon \to 0$ , and  $\sigma = \mathcal{O}\left(\frac{\triangle_2}{\sqrt{\epsilon}}\right)$  when  $\epsilon \to \infty$ , which is rate optimal [4]. Essentially, Theorem 3.2 reduces the noise variance of Theorem 3.1 by analyzing the tail probability exactly.
3.2 DP-OPORP: A More Efficient Alternative
The standard random projections (1) require  $k$  projections. We can reduce the cost from  $O(kp)$  to  $O(p)$  by count-sketch: "binning + Rademacher RP". Basically, we split the data entries into  $k$  bins, and apply RP in each bin to generate  $k$  samples. [88] studied noise injection mechanism for count-sketch [12]. Recently, [57] proposed OPORP (One Permutation + One Random Projection), an improved variant of count-sketch using fixed-length binning and the normalized estimator. The steps are summarized in Algorithm 2. Note that the output  $x_{i}$  can be  $l_{2}$  normalized to reduce variance.
Algorithm 2: OPORP: count-sketch with fixed-length binning
1 Input: Data vector  $u\in \mathbb{R}^p$  ; Number of projected samples  $k$
2 Output:  $k$  OPORP samples
3 Apply a permutation  $\Pi :[p]\mapsto [p]$  to  $u$  to get  $u_{\Pi}$
4 Split the  $p$  permuted data columns into  $k$  consecutive length-  $p / k$  bins:  $u_{\Pi} = [u_{\Pi}^{(1)},\dots,u_{\Pi}^{(k)}]$
5 Generate a vector  $w \in \mathbb{R}^p$  following Rademacher distribution, denoted as  $w = [w^{(1)}, \dots, w^{(k)}]$ .
6 Return  $k$  projected samples by  $x_{i} = w^{(i)^{T}}u_{\Pi}^{(i)}$ , for  $i = 1,\dots,k$ .
For OPORP, the sensitivity  $\triangle_2 = \beta$ , due to the binning and Rademacher projection: changing one coordinate of  $u$  by  $\beta$  leads to a change of  $\beta$  in term of  $l_2$  distance between the projected vectors.
Theorem 3.3 (DP-OPORP). Adding noise from iid  $N(0,\sigma^2)$  to OPORP (the output of Algorithm 2) achieves  $(\epsilon, \delta)$ -DP, where  $\sigma$  is the solution to (4) with  $\triangle_2 = \beta$ .
The advantages of DP-RP and DP-OPORP over the approach of adding Gaussian noise to raw data for inner product estimation are theoretically justified in Appendix C.
4 DP-SignRP: Differentially Private Sign Random Projections
We now propose our main algorithms that output SignRP with DP guarantees. Firstly, we analyze the standard randomized response (RR) technique, DP-SignRP-RR, under Gaussian random projections. Next, we propose the concept of "smooth flipping probability" and develop DP-SignRP-RR-smooth as an improvement. Finally, we propose DP-SignOPORP that combines all the techniques.
4.1 DP-SignRP-RR by Randomized Response
Algorithm 3: DP-SignRP-RR
1 Input: Data  $u\in [-1,1]^p$ $\epsilon >0$ $0 <   \delta <  1$  Number of projections  $k$  ; norm lower bound  $m$
2 Output: Differentially private sign random projections
3 Apply RP by  $x = \frac{1}{\sqrt{k}} W^T u$ , where  $W \in \mathbb{R}^{p \times k}$  is a random  $N(0,1)$  matrix
4 Let  $N_{+}(m,\delta ,k,p)$  be computed as in Proposition 4.3
5 Compute  $\tilde{s}_j = \left\{ \begin{array}{ll} sign(x_j), & \text{with prob.}\frac{e^{\epsilon'}}{e^{\epsilon'} + 1}\\ -sign(x_j), & \text{with prob.}\frac{1}{e^{\epsilon'} + 1} \end{array} \right.$  for  $j = 1,\dots ,k$  with  $\epsilon^{\prime} = \epsilon /N_{+}(m,\delta ,k,p)$
6 Return  $\tilde{s}$  as the DP-SignRP of  $u$
We develop DP-SignRP-RR based on the classic randomized response (RR) mechanism [77, 31]. As summarized in Algorithm 3, after we apply random projection  $x = W^{T}u$ , we take  $s = \text{sign}(x)$ . Then, for each  $s_{j}$ , we keep the sign with probability  $\frac{e^{\epsilon'}}{e^{\epsilon'} + 1}$  and flip the sign with probability  $\frac{1}{e^{\epsilon'} + 1}$ . Here  $\epsilon' = \epsilon / N_{+}$ , where  $N_{+}(m, \delta, k, p)$  is an upper bound on the number of different signs (among  $k$  signs) of  $x = W^{T}u$  and  $x = W^{T}u'$  for any  $\beta$ -adjacent  $(u, u')$ , which will be derived later in Proposition 4.3. Here,  $m$  is a lower bound on the  $l_{2}$  norm of the data, i.e.,  $\| u \| \geq m$  for all  $u \in \mathcal{U}$ . The final output is  $\tilde{s}$  after perturbing  $s$  by the above procedure.
Theorem 4.1. Algorithm 3 is  $(\epsilon, \delta)$ -DP.
Proof. For any data point  $u \in \mathcal{U}$  and its  $\beta$ -adjacent neighbor  $u'$ , denote  $s = \text{sign}(W^T u) \in \{-1, +1\}^k$ ,  $s' = \text{sign}(W^T u') \in \{-1, +1\}^k$ , and let  $\tilde{s}$  and  $\tilde{s}'$  be the corresponding randomized sign vectors output by Algorithm 3. Denote  $S = \{i : s_j \neq s_j'\}$  and  $S^c = [k] \setminus S$ . For any vector  $y \in \{-1, +1\}^k$ , define  $S_0 = \{j \in S : s_j = y_j\}$ ,  $S_1 = \{j \in S : s_j \neq y_j\}$ ,  $S_0^c = \{j \in S^c : s_j = y_j\}$  and  $S_1^c = \{j \in S^c : s_j \neq y_j\}$ . By Proposition 4.3, we know that the event  $\{|S| \leq N_+(m, \delta, k, p)\}$  happens with probability at least  $1 - \delta$ . In this event, we have
$$
\begin{array}{l} \log \frac {P r (\tilde {s} = y)}{P r (\tilde {s} ^ {\prime} = y)} = \log \frac {\prod_ {j \in S _ {0} ^ {c}} \frac {e ^ {\epsilon^ {\prime}}}{e ^ {\epsilon^ {\prime}} + 1} \prod_ {j \in S _ {1} ^ {c}} \frac {1}{e ^ {\epsilon^ {\prime}} + 1} \prod_ {j \in S _ {0}} \frac {e ^ {\epsilon^ {\prime}}}{e ^ {\epsilon^ {\prime}} + 1} \prod_ {j \in S _ {1}} \frac {1}{e ^ {\epsilon^ {\prime}} + 1}}{\prod_ {j \in S _ {0} ^ {c}} \frac {e ^ {\epsilon^ {\prime}}}{e ^ {\epsilon^ {\prime}} + 1} \prod_ {j \in S _ {1} ^ {c}} \frac {1}{e ^ {\epsilon^ {\prime}} + 1} \prod_ {j \in S _ {0}} \frac {1}{e ^ {\epsilon^ {\prime}} + 1} \prod_ {j \in S _ {1}} \frac {e ^ {\epsilon^ {\prime}}}{e ^ {\epsilon^ {\prime}} + 1}} \\ \leq \log \frac {\prod_ {j \in S} \frac {e ^ {\epsilon^ {\prime}}}{e ^ {\epsilon^ {\prime}} + 1}}{\prod_ {j \in S} \frac {1}{e ^ {\epsilon^ {\prime}} + 1}} = | S | \epsilon^ {\prime} \leq N _ {+} \epsilon^ {\prime} = \epsilon . \\ \end{array}
$$
Since this event occurs with probability at least  $1 - \delta$ , the overall procedure is  $(\epsilon, \delta)$ -DP.
4.1.1 The flipping probability and calculation of  $N_{+}$
In the proof of Theorem 4.1,  $N_{+}$  is a upper bound on  $S = \{i:s_j\neq s_j'\}$  where  $s = \mathrm{sign}(W^T u)$  and  $s^\prime = \mathrm{sign}(W^T u^\prime)$  for neighboring data  $(u,u^{\prime})$  , which determines the flipping probability thus the utility. Next, we analyze  $N_{+}$  . The following is a useful lemma; all proofs are placed in Appendix E.
Lemma 4.2. Let  $X_{1},\ldots ,X_{p}$  be iid  $N(0,\sigma_x^2)$  variables. Let  $Y\sim N(0,\sigma_y^2)$  be another Gaussian random variable with arbitrary dependence structure with  $X_{i}$ 's. Let  $r = \sigma_x / \sigma_y\leq 1$ . Then
$$
P _ {+} (r, p) := P r \left(\max  _ {i = 1, \dots , p} | X | > | Y |\right) \leq \int_ {0} ^ {\infty} 2 p [ 2 \Phi (t) - 1 ] ^ {p - 1} [ 2 \Phi (r t) - 1 ] \phi (t) d t, \tag {5}
$$
where  $\phi (x)$  and  $\Phi (x)$  are the standard Gaussian pdf and cdf, respectively.
Equipped with Lemma 4.2, we can derive  $N_{+}$  in Algorithm 3, an upper bound on the number of projected signs that are possible to change when  $u$  is replaced by any neighboring data vector  $u'$ .
Proposition 4.3 (Bound  $N_{+}$ ). Suppose  $u \in [-1,1]^p$  and  $\beta \leq \| u\|$ . Denote  $r = \frac{\beta}{\|u\|}$  and  $F_{\|u\|,p} = P_{+}(\frac{\beta}{\|u\|},p)$  in (5). Denote  $s = \text{sign}(W^T u)$  and  $s' = \text{sign}(W^T u')$  for  $\beta$ -neighboring  $(u,u')$ , and  $S = \{i : s_j \neq s_j'\}$ . Then with probability  $1 - \delta$ ,  $|S| \leq N_{+}(\|u\|, \delta, k, p) = B^{-1}(\delta; F_{\|u\|,p}, k)$ , where  $B^{-1}(\delta; F_{\|u\|,p}, k)$  represents the inverse cdf of Binomial distribution with rate  $F_{\|u\|,p}$  and  $k$  trials.
Since  $F_{\| u\| ,p} = P_{+}(r,p)$  is an increasing function in  $r = \beta /\| u\|$ ,  $N_{+}$  would be smaller if  $\beta /\| u\|$  is smaller. Therefore, in DP-SignRP-RR, the probability of flipping the true SignRP,  $\frac{1}{e^{\epsilon / N_{+}} + 1}$ , would be smaller when the data norm (or, its lower bound  $m$ ) is large compared with  $\beta$ .
4.1.2 Utility in angle estimation by DP-SignRP-RR
Define the DP-SignRP-RR estimator of the angle between two data points  $u$  and  $v$  as
$$
\hat {\theta} _ {R R} = \pi \left(1 - \hat {P} _ {R R}\right), \quad \text {w h e r e} \hat {P} _ {R R} = \frac {\left(e ^ {\epsilon^ {\prime}} + 1\right) ^ {2}}{\left(e ^ {\epsilon^ {\prime}} - 1\right) ^ {2}} \frac {1}{k} \sum_ {j = 1} ^ {k} \mathbb {1} \left\{\tilde {s} _ {1 j} = \tilde {s} _ {2 j} \right\} - \frac {2 e ^ {\epsilon^ {\prime}}}{\left(e ^ {\epsilon^ {\prime}} - 1\right) ^ {2}}. \tag {6}
$$
Theorem 4.4. Let  $\rho = \cos(u, v)$  and  $\theta = \cos^{-1}(\rho)$ . Run Algorithm 3 with  $N_{+}$  given in Proposition 4.3 and define the DP-SignRP-RR angle estimator by (6). We have  $\mathbb{E}[\hat{\theta}_{RR}] = \theta$ . As  $k \to \infty$ ,  $\hat{\theta}_{RR} \to N(\theta, \frac{V_{RR}}{k})$ , with
$$
V _ {R R} = \theta (\pi - \theta) + \frac {2 \pi^ {2} e ^ {\epsilon / N _ {+}}}{(e ^ {\epsilon / N _ {+}} - 1) ^ {2}} + \frac {4 \pi^ {2} e ^ {2 \epsilon / N _ {+}}}{(e ^ {\epsilon / N _ {+}} - 1) ^ {4}}.
$$
Theorem 4.4 says that  $\hat{\theta}_{RR}$  is an unbiased estimator of  $\theta$  and asymptotically normal. Compared with  $\frac{\theta(\pi - \theta)}{k}$ , the variance of the cosine estimator from non-DP SignRP [41, 13], we see that  $\hat{\theta}_{RR}$  incurs an extra variance (i.e., utility loss) of  $\frac{\pi^2}{k} \left[ \frac{2e^{\epsilon / N_+}}{(e^{\epsilon / N_+} - 1)^2} + \frac{4e^{2\epsilon / N_+}}{(e^{\epsilon / N_+} - 1)^4} \right]$ . This quantity increases as  $\epsilon$  gets smaller, illustrating the utility-privacy trade-off of DP-SignRP-RR.