Title: Smooth Flipping Probability for Differential Private Sign Random Projection Methods
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy

Random Projection
Abstract: We develop a series of differential privacy (DP) algorithms from a family of random projection (RP) and sign random projection (SignRP) methods. We first show how to improve the previous DP-RP approach using the ``optimal Gaussian mechanism''. Then, we propose a series of DP-SignRP algorithms that leverage the robustness of the ``sign flipping probability'' of random projections. That is, given $x = \sum_{i=1}^p u_i w_{i}$ where $u$ is a $p$-dimensional data vector and $w$ is a symmetric random vector,  $sign(x)$ only has a fairly small probability to be flipped if there is a small modification on data $u$, depending on the specific distribution of $w$. This robustness leads to our novel design of ``smooth flipping probability'' for SignRP-type algorithms with better utility than using the standard randomized response mechanism. Retrieval and classification experiments demonstrate that,  among the presented DP-RP  algorithms,  \textbf{DP-SignOPORP} (where OPORP  is an improvement over the celebrated  count-sketch algorithms), performs the best in general.

In the industrial practice, DP methods were not very popular for machine learning or search, largely because the performance typically would  drop substantially if DP is applied. Since our proposed new DP algorithms have significantly improved the  performance, it is anticipated that our work will motivate a wide adoption of DP in practice. Finally, we   stress that, since our  methods are applied to the original data (i.e., feature vectors), the privacy of  downstream tasks is naturally protected.

Smooth Flipping Probability for Differentially Private Sign Random Projection Methods

Abstract

We develop a series of differential privacy (DP) algorithms from a family of random projection (RP) and sign random projection (SignRP) methods. We first show how to improve the previous DP-RP approach using the 'optimal Gaussian mechanism'. Then, we propose a series of DP-SignRP algorithms that leverage the robustness of the 'sign flipping probability' of random projections. That is, given x = ∑ p i =1 u i w i where u is a p -dimensional data vector and w is a symmetric random vector, sign ( x ) only has a fairly small probability to be flipped if there is a small modification on data u , depending on the specific distribution of w . This robustness leads to our novel design of 'smooth flipping probability' for SignRPtype algorithms with better utility than using the standard randomized response mechanism. Retrieval and classification experiments demonstrate that, among the presented DP-RP algorithms, DP-SignOPORP (where OPORP is an improvement over the celebrated count-sketch algorithms), performs the best in general.

In the industrial practice, DP methods were not very popular for machine learning or search, largely because the performance typically would drop substantially if DP is applied. Since our proposed new DP algorithms have significantly improved the performance, it is anticipated that our work will motivate a wide adoption of DP in practice. Finally, we stress that, since our methods are applied to the original data (i.e., feature vectors), the privacy of downstream tasks is naturally protected.

1 Introduction

Protecting data privacy has become an urgent need and a trending research topic recently. Among many notions of privacy, the 'differential privacy' (DP) [30] has gained tremendous attention in both the research community and industrial applications, and has been widely applied to numerous tasks such as frequency estimation [32, 26], clustering [36, 34], regression and classification [14, 85], DPSGD[1, 35], principle component analysis [40], empirical risk minimization [15], graph analysis [33], matrix completion [8, 45], etc. In a machine learning model, there are several choices on when to deploy differential privacy (DP): (a) at the data collection/processing stage [84, 81, 5, 19]; (b) during the model training stage [1, 78]; and (c) on the model output (or summary statistics) [29]. Among them, applying DP as early as at the data collection stage provides strong protection in the sense that the subsequent operations or outputs will become private by the post-processing property of DP. Moreover, when the data holder needs to release the data to third-parties, a privacy-preserving data publishing mechanism (at an early stage) becomes necessary.

In this paper, we focus on the differential privacy of the broad family of random projection (RP) methods, including RP, sign random projections (SignRP) and count-sketch-type algorithm. Our algorithms can be applied to data publishing and processing for downstream machine learning tasks.

1.1 Random Projections (RP) and Sign Random Projections (SignRP)

In practice, data compression and dimension reduction techniques can be crucial when dealing with massive (high-dimensional) data. The random projection (RP) method is an important and fundamental dimensionality reduction algorithm. Denote u ∈ R p as the data vector with p features. With some k (and typically k ≪ p ), we define the random projection of u as

The entries of the random matrix W typically follow the Gaussian distribution or Gaussian-like distribution such as the Rademacher (symmetric Bernoulli) distribution. It has been well-understood that, with a sufficient number of projections ( k ), the distance between two vectors (using the same projection W ) is preserved within a small multiplicative error with high probability. The dimensionality reduction and geometry preserving properties make RP widely useful in numerous applications, such as distance estimation, nearest neighbor search, clustering, classification, compressed sensing, permutation recovery, etc. [46, 44, 22, 6, 2, 37, 24, 76, 10, 28, 56, 38, 23, 21, 60, 68, 75, 82, 83].

While RP is able to reduce the dimensionality, storing and transmitting projected data might still be expensive for large datasets. One can further compress the RPs by quantization/discretization, where we only use a few bits to represent the projected values [41, 9, 27, 89, 54, 58, 50, 55, 61, 62]. In this paper, we will consider the differential privacy of the extreme case of sign (1-bit) random projection (SignRP), also known as the 'SimHash' [13, 24], where only the sign of the projected data is stored. The signs still preserve the angle/cosine information. SignRP has been widely used for approximate near neighbor (ANN) search [39] by building hash tables from the bits [44, 13, 71]. RP and SignRP can also be extended to non-linear random features [69, 63, 64].

1.2 Count-Sketch and OPORP: Efficient RP-type Alternatives

The celebrated count-sketch [12] can be viewed as a highly efficient RP-type algorithm, because, as an option, it only requires 'one permutation + one random projection' (OPORP) [57], as opposed to k projections in RP. Applications of count-sketch include graph embedding [80], word & image embedding [16, 87, 3, 72, 86], model & communication compression [79, 17, 70, 42, 59], etc. The recent work by [57] improved the original count-sketch in two aspects: (i) using a fixed-length binning scheme makes the algorithms more convenient and also reduces the variance by a factor of p -k p -1 ; (ii) the projected data (i.e., vectors in k dimensions) should be normalized before they are used to estimate the original cosine ρ , which also substantially reduces the estimation variance, essentially from (1 + ρ 2 ) /k (un-normalized) to (1 -ρ 2 ) 2 /k (normalized).

The main contributions of this work include the following:

In Section 3, for DP-RP, we first revisit the prior work [49] on the ( ϵ, δ ) -DP random projection based on Gaussian noise addition (called DP-RP-G). Then we incorporate the optimal Gaussian mechanism and propose an improved method DP-RP-G-OPT. We also develop DP-OPORP based on the optimal Gaussian mechanism.

In Section 4, we propose two algorithms for privatizing the (1-bit) DP-SignRP. The first method, DP-SignRP-RR, is based on the standard 'randomized response' (RR). Then, we propose an improved method named DP-SignRP-RR-smooth based on a proposed concept of 'smooth flipping probability' , thanks to the robustness brought by the 'aggregate-and-sign' operations of SignRP. Finally, we extend the idea of smooth flipping probability to OPORP and show that DP-SignOPORP is more advantageous in terms of privacy.

In Section 5, we conduct retrieval and classification experiments on benchmark datasets. For full-precision methods, the proposed DP-OPORP substantially improves prior method DP-RP-G. For the 1-bit RP variants, the suggested DP-SignOPORP can outperform DP-RP and DP-OPORP especially when ϵ is not large (e.g., ϵ ≤ 5 ). Our results also verify the advantage of smooth flipping probability over the classic bit flipping strategy.

2 Background on Differential Privacy

Throughout the paper, ∥ · ∥ 1 and ∥ · ∥ 2 are the l 1 and l 2 norms, respectively, and ∥ · ∥ will denote the l 2 norm if there no risk of confusion. Let u ∈ U be the data vector of a user where U denotes the data domain. In this paper, we consider real-valued data vectors with bounded range, i.e., U = [ -1 , 1] p for convenience. We also assume ∥ u ∥ > 0 , i.e., there is no all-zero data sample. The formal definition of differential privacy (DP) is given as follows.

Definition 2.1 (Differential Privacy [30]) . For a randomized algorithm M : U ↦→ Range ( M ) , if for any two adjacent datasets u and u ′ , it holds that

for ∀ O ⊂ Range ( M ) and some ϵ, δ ≥ 0 , then algorithm M is called ( ϵ, δ ) -differentially private. If δ = 0 , M is called ϵ -differentially private.

In this work, we follow the standard setup in the literature of DP sketching/hashing (e.g., [7, 49, 73, 74, 88, 25, 65]) that neighboring data vectors u and u ′ only differ in one dimension, which is usually referred to as the 'attribute-level DP'-an adversary cannot detect a small change in any attribute of u based on the output of the algorithm.

Definition 2.2 ( β -adjacency) . Let u ∈ [ -1 , 1] p be a data vector. A vector u ′ ∈ [ -1 , 1] p is said to be β -adjacent to u if u ′ and u differ in one dimension i , and | u i -u ′ i | ≤ β .

Data vectors u and u ′ satisfying Definition 2.2 are called β -adjacent or β -neighboring. In the literature (e.g., [49]), β = 1 is typically used. We allow a general β to accommodate the practical needs. Moreover, the concept of sensitivity is important for the design of DP noise addition mechanisms.

Definition 2.3 ( l 2 -sensitivity) . Let Nb ( u ) denote the neighbor set of u , and N ( U ) = { ( u, u ′ ) : u ′ ∈ Nb ( u ) , u, u ′ ∈ U} be the collection of all possible neighboring data pairs. The l 2 -sensitivity of a function f : U ↦→ R k is defined as △ 2 = max ( u,u ′ ) ∈N ( U ) ∥ f ( u ) -f ( u ′ ) ∥ 2 .

The following composition theorem of DP is also useful in our analysis.

Theorem 2.1 (Composition Theorem [31]) . Let M j : U → R be an ( ϵ j , δ j ) -DP algorithm for j = 1 , ..., k . Then M ( u ) = ( M 1 ( u ) , ..., M k ( u )) is ( ∑ k j =1 ϵ i , ∑ k j =1 δ j ) -DP.

3 Revisiting Differentially Private Random Projection Methods

We first revisit the DP noise addition mechanism for random projections and show how to apply the optimal Gaussian mechanism [4] to improve prior methods. We also propose a DP algorithm based on a variant of count-sketch which is computationally much more efficient than dense projections.

3.1 Gaussian Noise Mechanism for DP-RP

Let N = { u, u ′ ∈ U : u ∈ Nb ( u ) } be the collection of all possible β -neighboring data pairs, where Nb ( u ) is the neighbor set of u . For a p -byk projection matrix W , the l 2 -sensitivity of the RP (1), according to Definition 2.3, can be precisely computed by

where W [ i, :] denotes the i -th row of W . As such, the DP noise level depends on the choice of projection matrix W . A comparison of Rademacher vs. Gaussian projection in DP-RP method can be found in Section B.1. With this calculation, the general Gaussian noise mechanism for DP-RP is summarized in Algorithm 1. We present the algorithms for a single data point u , and the procedure is applied to every data vector in the database (with same W but independent noise). After random projection, we simply add a random Gaussian noise vector following iid N (0 , σ 2 ) to the projected data. The following 'DP-RP-G' result is known in the literature.

Theorem 3.1 (DP-RP-G [49]) . Let △ 2 be defined in (3). For any ϵ > 0 and 0 < δ < 1 2 , DP-RP-G in Algorithm 1 is ( ϵ, δ ) -DP if σ ≥ △ 2 √ 2(log(1 /δ )+ ϵ ) ϵ .

Algorithm 1: DP-RP-G and DP-RP-G-OPT

1 Input: Data u ∈ [ -1 , 1] p , privacy parameters ϵ > 0 , δ ∈ (0 , 1) , number of projections k

2 Output: ( ϵ, δ ) -differentially private random projections ˜ x ∈ R k

3 Apply RP x = 1 √ k W T u , where W ∈ R p × k has iid entries from N (0 , 1) or Rademacher

4 Compute the sensitivity △ 2 by (3)

5 Generate the random noise vector G ∈ R k whose entries are iid samples from N (0 , σ 2 ) where σ is obtained by Theorem 3.1 (DP-RP-G) or Theorem 3.2 (DP-RP-G-OPT)

6 Return ˜ x = x + G

Remark 3.1. Note that in implementation, △ 2 is computed by (3) using the realization of the projection matrix W . In Appendix A, we provide a high probability bound on △ 2 and more analysis and discussion on the Laplace mechanism for DP-RP.

In Theorem 3.1 (as well as the classical Gaussian mechanism, e.g., [31]), the analysis on the noise level is based on upper bounding the tail of Gaussian distribution. It can be improved by computing the exact tail probabilities. [4, Theorem 8] proposed an optimal Gaussian mechanism based on this idea, which allows us to improve Theorem 3.1 and obtain the DP-RP-G-OPT method as follows.

Theorem 3.2 (DP-RP-G-OPT) . Suppose △ 2 is defined as (3). For any ϵ > 0 and 0 < δ < 1 , DP-RP-G-OPT in Algorithm 1 achieves ( ϵ, δ ) -DP if σ ≥ σ ∗ where σ ∗ is the solution to the equation

where Φ( · ) is the cdf of the standard normal distribution.

Remark 3.2. For both Theorem 3.2 and Theorem 3.1, the Gaussian noise level is σ = O ( △ 2 ϵ ) when ϵ → 0 , and σ = O ( △ 2 √ ϵ ) when ϵ →∞ , which is rate optimal [4]. Essentially, Theorem 3.2 reduces the noise variance of Theorem 3.1 by analyzing the tail probability exactly.

3.2 DP-OPORP: A More Efficient Alternative

The standard random projections (1) require k projections. We can reduce the cost from O ( kp ) to O ( p ) by count-sketch: 'binning + Rademacher RP'. Basically, we split the data entries into k bins, and apply RP in each bin to generate k samples. [88] studied noise injection mechanism for count-sketch [12]. Recently, [57] proposed OPORP (One Permutation + One Random Projection), an improved variant of count-sketch using fixed-length binning and the normalized estimator. The steps are summarized in Algorithm 2. Note that the output x i can be l 2 normalized to reduce variance.

Algorithm 2: OPORP: count-sketch with fixed-length binning

1 Input: Data vector u ∈ R p ; Number of projected samples k

2 Output: k OPORP samples

3 Apply a permutation Π : [ p ] ↦→ [ p ] to u to get u Π

4 Split the p permuted data columns into k consecutive lengthp/k bins: u Π = [ u (1) Π , ..., u ( k ) Π ]

5 Generate a vector w ∈ R p following Rademacher distribution, denoted as w = [ w (1) , ..., w ( k ) ]

6 Return k projected samples by x i = w ( i ) T u ( i ) Π , for i = 1 , ..., k .

For OPORP, the sensitivity △ 2 = β , due to the binning and Rademacher projection: changing one coordinate of u by β leads to a change of β in term of l 2 distance between the projected vectors.

Theorem 3.3 (DP-OPORP) . Adding noise from iid N (0 , σ 2 ) to OPORP (the output of Algorithm 2) achieves ( ϵ, δ ) -DP, where σ is the solution to (4) with △ 2 = β .

The advantages of DP-RP and DP-OPORP over the approach of adding Gaussian noise to raw data for inner product estimation are theoretically justified in Appendix C.

4 DP-SignRP: Differentially Private Sign Random Projections

We now propose our main algorithms that output SignRP with DP guarantees. Firstly, we analyze the standard randomized response (RR) technique, DP-SignRP-RR, under Gaussian random projections. Next, we propose the concept of 'smooth flipping probability' and develop DP-SignRP-RR-smooth as an improvement. Finally, we propose DP-SignOPORP that combines all the techniques.

4.1 DP-SignRP-RR by Randomized Response

Algorithm 3: DP-SignRP-RR

1 Input: Data u ∈ [ -1 , 1] p ; ϵ > 0 , 0 < δ < 1 ; Number of projections k ; norm lower bound m

2 Output: Differentially private sign random projections

3 Apply RP by x = 1 √ k W T u , where W ∈ R p × k is a random N (0 , 1) matrix

4 Let N + ( m,δ,k, p ) be computed as in Proposition 4.3

6 Return ˜ s as the DP-SignRP of u

We develop DP-SignRP-RR based on the classic randomized response (RR) mechanism [77, 31]. As summarized in Algorithm 3, after we apply random projection x = W T u , we take s = sign ( x ) .

Then, for each s j , we keep the sign with probability e ϵ ′ e ϵ ′ +1 and flip the sign with probability 1 e ϵ ′ +1 . Here ϵ ′ = ϵ/N + , where N + ( m,δ,k, p ) is an upper bound on the number of different signs (among k signs) of x = W T u and x = W T u ′ for any β -adjacent ( u, u ′ ) , which will be derived later in Proposition 4.3. Here, m is a lower bound on the l 2 norm of the data, i.e., ∥ u ∥ ≥ m for all u ∈ U . The final output is ˜ s after perturbing s by the above procedure.

Theorem 4.1. Algorithm 3 is ( ϵ, δ ) -DP.

̸

Proof. For any data point u ∈ U and its β -adjacent neighbor u ′ , denote s = sign ( W T u ) ∈ {-1 , +1 } k , s ′ = sign ( W T u ′ ) ∈ {-1 , +1 } k , and let ˜ s and ˜ s ′ be the corresponding randomized sign vectors output by Algorithm 3. Denote S = { i : s j = s ′ j } and S c = [ k ] \ S . For any vector y ∈ {-1 , +1 } k , define S 0 = { j ∈ S : s j = y j } , S 1 = { j ∈ S : s j = y j } , S c 0 = { j ∈ S c : s j = y j } and S c 1 = { j ∈ S c : s j = y j } . By Proposition 4.3, we know that the event {| S | ≤ N + ( m,δ,k, p ) } happens with probability at least 1 -δ . In this event, we have

Since this event occurs with probability at least 1 -δ , the overall procedure is ( ϵ, δ ) -DP.

4.1.1 The flipping probability and calculation of N +

In the proof of Theorem 4.1, N + is a upper bound on S = { i : s j = s ′ j } where s = sign ( W T u ) and s ′ = sign ( W T u ′ ) for neighboring data ( u, u ′ ) , which determines the flipping probability thus the utility. Next, we analyze N + . The following is a useful lemma; all proofs are placed in Appendix E.

̸

Lemma 4.2. Let X 1 , ..., X p be iid N (0 , σ 2 x ) variables. Let Y ∼ N (0 , σ 2 y ) be another Gaussian random variable with arbitrary dependence structure with X i 's. Let r = σ x /σ y ≤ 1 . Then

where ϕ ( x ) and Φ( x ) are the standard Gaussian pdf and cdf, respectively.

̸

̸

Equipped with Lemma 4.2, we can derive N + in Algorithm 3, an upper bound on the number of projected signs that are possible to change when u is replaced by any neighboring data vector u ′ .

̸

Proposition 4.3 (Bound N + ) . Suppose u ∈ [ -1 , 1] p and β ≤ ∥ u ∥ . Denote r = β ∥ u ∥ and F ∥ u ∥ ,p = P + ( β ∥ u ∥ , p ) in (5). Denote s = sign ( W T u ) and s ′ = sign ( W T u ′ ) for β -neighboring ( u, u ′ ) , and S = { i : s j = s ′ j } . Then with probability 1 -δ , | S | ≤ N + ( ∥ u ∥ , δ, k, p ) = B -1 ( δ ; F ∥ u ∥ ,p , k ) , where B -1 ( δ ; F ∥ u ∥ ,p , k ) represents the inverse cdf of Binomial distribution with rate F ∥ u ∥ ,p and k trials.

Since F ∥ u ∥ ,p = P + ( r, p ) is an increasing function in r = β/ ∥ u ∥ , N + would be smaller if β/ ∥ u ∥ is smaller. Therefore, in DP-SignRP-RR, the probability of flipping the true SignRP, 1 e ϵ/N + +1 , would be smaller when the data norm (or, its lower bound m ) is large compared with β .

4.1.2 Utility in angle estimation by DP-SignRP-RR

Define the DP-SignRP-RR estimator of the angle between two data points u and v as

Theorem 4.4. Let ρ = cos( u, v ) and θ = cos -1 ( ρ ) . Run Algorithm 3 with N + given in Proposition 4.3 and define the DP-SignRP-RR angle estimator by (6). We have E [ ˆ θ RR ] = θ . As k → ∞ , ˆ θ RR → N ( θ, V RR k ) , with

Theorem 4.4 says that ˆ θ RR is an unbiased estimator of θ and asymptotically normal. Compared with θ ( π -θ ) k , the variance of the cosine estimator from non-DP SignRP [41, 13], we see that ˆ θ RR incurs an extra variance (i.e., utility loss) of π 2 k [ 2 e ϵ/N + ( e ϵ/N + -1) 2 + 4 e 2 ϵ/N + ( e ϵ/N + -1) 4 ] . This quantity increases as ϵ gets smaller, illustrating the utility-privacy trade-off of DP-SignRP-RR.

Optimal projection dimension k ∗ . Since by Proposition 4.3 we know that N + ≍ F ∥ u ∥ ,p k roughly, the term [ 2 e ϵ/N + ( e ϵ/N + -1) 2 + 4 e 2 ϵ/N + ( e ϵ/N + -1) 4 ] would increase with k . Therefore, there exists an optimal k ∗ that minimizes the estimation variance. When k is sufficiently large, we have

using the approximation that e x -1 ≈ x when x is small. To find the optimal k minimizing this expression, we compute the derivative and set it as zero:

This analysis suggests the optimal k ∗ of DP-SignRP-RR is larger when: (1) ϵ is large; (2) F ∥ u ∥ ,p is small, which is typically true when the norm of the data is relatively large.

4.2 Smooth Flipping Probability: the Benefit of Robustness

Next, we improve DP-SignRP-RR based on a novel tool which we call 'smooth flipping probability' . The idea is inspired by the 'smooth sensitivity' for DP noise addition [66], which essentially says that, the farther u is from a high 'local sensitivity' region, the less noise is needed for u to achieve DP. The local sensitivity at a specific data point u is defined as LS ( u ) = max u ′ ∈ Nb ( u ) ∥ f ( u ) -f ( u ′ ) ∥ 2 , which differs from Definition 2.3 in that local sensitivity only considers the local neighbors of u .

What is the 'local perturbation' for flipping the SignRP? Consider one projection x j = W [: , j ] T u and s j = sign ( x j ) . For a neighbor u ′ of u , denote x ′ j = W [: , j ] T u ′ and s ′ j = sign ( x ′ j ) . By β -adjacency, s ′ j is possible to be different from s j only if | x j | < β max i =1 ,...,p | W ij | . This implies:

When | x j | < β max i =1 ,...,p | W ij | , the 'local sign flipping' is required, since s ′ j may differ from s j . We can apply the standard RR: keep s j with prob. e ϵ/k e ϵ/k +1 and flip otherwise.

′ ′

When | x j | ≥ β max i =1 ,...,p | W ij | , there does not exist u ∈ Nb ( u ) such that s j = s j . Thus, the 'local sign flipping' is not needed. However, if we do not perturb s j , DP would be violated since s ′ j might need random flipping. Instead, we use a 'smooth flipping probability' which applies less perturbation for points far away from the 'high perturbation region', i.e., the regime where | x j | < β max i =1 ,...,p | W ij | . Specifically, we define L j = ⌈ | x j | β max i =1 ,...,p | W ij | ⌉ . The probability of keeping the sign s j is P ( u ) j = e ϵ ′ j e ϵ ′ j +1 where ϵ ′ j = L j k ϵ . It can be shown (e.g., in the proof of Theorem 4.5) that this flipping probability is 'smooth': for ∀ u, u ′ that are neighbors, P ( u ) j ≤ e ϵ/k P ( u ′ ) j , which justifies its name.

Algorithm 4: DP-SignRP-RR-smooth using smooth flipping probability

1 Input: Data u ∈ [ -1 , 1] ; ϵ > 0 ; Number of projections k

2 Output: Differentially private sign random projections

3 Apply RP by x = W T u , where W ∈ R p × k is a random N (0 , 1)

4 Compute L j = ⌈ | x j | β max i =1 ,...,p | W ij | ⌉ for j = 1 , ..., k

5 Compute ˜ s j =    sign ( x j ) , with prob. e ϵ ′ j e ϵ ′ j +1 -sign ( x j ) , with prob. 1 ϵ ′ j for j = 1 , ..., k , with ϵ ′ j = L j k

6 Return ˜ s as the DP-SignRP of

The concrete steps of DP-SignRP-RR-smooth are summarized in Algorithm 4. Note that, the above two cases can be unified into one, since e ϵ/k e ϵ/k +1 is the smooth flipping probability with L j = 1 . In Figure 1, we provide an illustrative example of the smooth flipping probability (red) and the 'local flipping probability' (green) discussed above. The global upper bound (black dash) corresponds to the standard randomized response. The local flipping probability is only non-zero at L = 1 . The proposed smooth flipping probability is non-zero for all L > 0 , but keeps shrinking exponentially as L gets larger, i.e., as the projected value is farther from 0 .

Theorem 4.5. Algorithm 4 achieves ϵ -DP.

1

2

3

4

5

6

7

8

L

0

0.1

0.2

0.3

Flipping Probability

smooth flip prob.

local flip prob.

global flip prob.

Comparison with DP-SignRP-RR (Algorithm 3). In terms of algorithm design and privacy, DP-SignRP-RR-smooth (Algorithm 4) has two advantages over the standard DP-SignRP-RR method:

DP-SignRP-RR-smooth does not require (assume) a lower bound m on the data norms;

DP-SignRP-RR-smooth achieves ϵ -DP, while DP-SignRP-RR only guarantee ( ϵ, δ ) -DP.

Note that Algorithm 3 can also achieve ϵ -DP if we set the flipping probability to be 1 e ϵ/k +1 (for all projected values). This corresponds to the global upper bound (black dotted line) in Figure 1. Thus, DP-SignRP-RR-smooth requires much smaller flipping probability than DP-SignRP-RR.

DP-SignRP with Rademacher projection. Similar to DP-RP, the flipping probabilities also depend on the projection matrix W through N + (for DP-SignRP-RR) and L (for DP-SignRP-RR-smooth). In Appendix B.2, we analytically show that Rademacher RP requires smaller flipping probability than that of Gaussian RP for both variants.

̸

4.3 DP-SignOPORP with Smooth Flipping Probability

We incorporate the smooth flipping technique to develop DP-SignOPORP, as given in Algorithm 5 with two variants. DP-SignOPORP-RR is based on the standard randomized response (RR), while DP-SignOPORP-RR-smooth is an improved version with our proposed smooth flipping probability.

Algorithm 5: DP-SignOPORP-RR and DP-SignOPORP-RR-smooth

1 Input: Data u ∈ [ -1 , 1] p ; ϵ > 0 ; Number of projections k

2 Output: Differentially private sign OPORP

3 Apply Algorithm 2 with a random Rademacher projection vector to get the OPORP DP-SignOPORP-RR:

4 Compute ˜ s j = { sign ( x j ) , with prob. e ϵ e ϵ +1 -sign ( x j ) , with prob. 1 e ϵ +1 for j = 1 , ..., k

DP-SignOPORP-RR-smooth:

5 Compute L j = ⌈ | x j | β ⌉ for j = 1 , ..., k

6 Compute ˜ s j =    sign ( x j ) , with prob. e ϵ ′ j e ϵ ′ j +1 -sign ( x j ) , with prob. 1 e ϵ ′ j +1 for j = 1 , ..., k , with ϵ ′ j = L j ϵ

7 For ˜ s j = 0 , assign a random coin in {-1 , 1 }

8 Return ˜ s as the DP-SignRP of u

Benefit of binning. In Algorithm 5, the key difference compared with DP-SignRP methods is that, the 1 N + (in Algorithm 3) or 1 k (in Algorithm 4) is removed from the flipping probability formulas, which is a significant advantage in privacy. This improvement is a result of the binning step. By Definition 2.2, two neighboring data u ′ and u differ in one coordinate. For SignRP, since each projected data is an aggregation of the whole data vector, when we switch from u to u ′ , (in principle) all k projections are possible to change. In contrast, in OPORP, since each data entry appears in only one bin, u ′ will only cause exactly one projected output to change, which enhances privacy.

Theorem 4.6. Both variants in Algorithm 5 are ϵ -DP.

̸

Proof. The proof follows the idea of the proofs of DP-SignRP methods, but we need to additionally consider the empty bins. Denote x and x ′ as the OPORP from Algorithm 2 using a same random vector w , and let s = sign ( x ) , s ′ = sign ( x ′ ) . Suppose u and u ′ differ in dimension i , and dimension i is assigned to the j ∗ -th bin with j ∗ = ⌈ π ( i ) / ( p/k ) ⌉ , where π : [ p ] ↦→ [ p ] is the permutation in OPORP. For any y ∈ {-1 , 1 } k , it is easy to see that to compute log Pr (˜ s = y ) Pr (˜ s ′ = y ) , it suffices to look at the j ∗ -th output sample because other probabilities cancel out. For the j ∗ -th sample, when s j ∗ = 0 and s ′ j ∗ = 0 , by the same arguments as in the proof of Theorem 4.5, we know that e -ϵ ≤ Pr ( s j ∗ )= a Pr ( s ′ j ∗ )= a ≤ e ϵ , a ∈ {-1 , 1 } , for both variants (DP-SignOPORP-RR and DP-SignOPORP-RR-smooth). When one of s j ∗ and s ′ j ∗ equals 0 , it is also easy to see that (assume s j ∗ = 1 and s ′ j ∗ is a random coin)

This holds for both variants. In particular, this case corresponds to L j ∗ = 1 for the smooth flipping probability. This proves the theorem.

Multiple repetitions. In Algorithm 5 Line 7, if OPORP generates an empty bin (i.e., x j = 0 ), we must assign a random sign to it in order to maintain DP. This tends to undermine the utility since a random coin does not provide any useful information about the data. To avoid too many empty bins, one simple way is to repeat the DP-SignOPORP (with smaller k ) for t > 1 times, and concatenate the output vectors [12]. For example, if the target k = 256 , we may run Algorithm 5 for t = 4 times, each time with 256 / 4 = 64 projected values and privacy budget ϵ/ 4 . We still obtain a length-256 bit vector with ϵ -DP by the composition theorem. In the experiments, we will see that this strategy may improve the overall performance of DP-SignOPORP.

x

̸

5 Experiments

We present experiments on retrieval and classification tasks to demonstrate the effectiveness of our proposed methods. We compare the following algorithms:

Raw-data-G-OPT: adding optimal Gaussian noise with sensitivity β to the original data, which is the most basic DP method for privatizing the original data.

DP-RP-G (Algorithm 1 + Theorem 3.1): Gaussian DP-RP with the Gaussian mechanism.

DP-RP-G-OPT (Algorithm 1 + Theorem 3.2): DP-RP with Gaussian random projection matrix and the optimal Gaussian noise.

DP-OPORP (Theorem 3.3): one permutation + one random projection with Rademacher projection vector and optimal Gaussian noise.

DP-SignOPROP-RR and DP-SignOPORP-RR-smooth (Algorithm 5): signed OPORP with standard randomized response and with smooth flipping probability, respectively.

As noted in Section 4.3, due to binning, DP-SignOPORP is substantially better than DP-SignRP. Thus we present results for DP-SignOPORP in our experiments (and only β = 1 for conciseness).

Similarity Search. We first test the methods in similarity search problems, using two standard image retrieval datasets, MNIST [53] and CIFAR [51]. For both datasets, we treat the training set as the database, and use the samples from the test set as the queries. We set the ground truth neighbors for each query as the top-50 sample vectors with the highest cosine similarity to the query. To search with DP-RP (and DP-OPORP), we estimate the cosine between a query and a sample point by the cosine between their corresponding output noisy projected values. We report only the precisions (and please see Appendix for other metrics), averaged over 10 repetitions.

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 64

DP-OPORP

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 128

DP-OPORP

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 256

DP-OPORP

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

precision@10

CIFAR: k = 64

DP-OPORP

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

precision@10

CIFAR: k = 128

DP-OPORP

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

precision@10

CIFAR: k = 256

DP-OPORP

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

Figure 2 shows that, as a result of the optimal Gaussian mechanism, DP-RP-G-OPT substantially outperforms the prior method in the literature, DP-RP-G, at all ϵ . DP-OPORP achieve better performance than DP-RP-G-OPT and is recommended as the best (full-precision) DP-RP variant. These two methods provide much higher precision and recall than the optimal Gaussian mechanism applied to the original data (Raw-data-G-OPT).

In Figure 3, we plot the precision and recall of DP-SignOPORP-RR (with standard randomized response), and DP-SignOPORP-RR-s (with smooth flipping probability). We present the results with repetition t = 2 , 4 which yield good overall performance. As we can see, (i) the proposed smooth flipping probability considerably improves the standard randomized response technique, and (ii)

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 128

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 256

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 512

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

test accuracy (%)

WEBSPAM: k = 256

DP-SignOPORP-RR t=1

DP-SignOPORP-RR-s t=1

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

test accuracy (%)

WEBSPAM: k = 512

DP-SignOPORP-RR t=1

DP-SignOPORP-RR-s t=1

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

100

test accuracy (%)

WEBSPAM: k = 1024

DP-SignOPORP-RR t=1

DP-SignOPORP-RR-s t=1

DP-OPORP

Raw-data-G-OPT

DP-SignOPORP in general provides better search accuracy than DP-OPORP when ϵ < 10 ∼ 15 . This range of ϵ is common in the use cases of DP for providing reasonable privacy protection. Also, we note that smaller t (repetitions) typically performs better with smaller ϵ but worse when ϵ is large.

Classification. We evaluate the performance of DP-SignOPORP in classification problems trained by SVM [20], on the WEBSPAM dataset [11]. We apply the 'max normalization', i.e., divide each data column by its largest magnitude such that the data entries are bounded in [ -1 , 1] . The SVM results are presented in Figure 4. For this task, we plot DP-SignOPORP with t = 1 . We again observe the advantage of DP-SignOPORP over DP-OPORP, and the advantage of the proposed smooth flipping probability over the standard randomized response. Specifically, when ϵ ≈ 5 , the test accuracy of Raw-data-G-OPT is around 60% (for WEBSPAM, this is almost the same as random guessing), but DP-SignOPORP can achieve 95% classification accuracy with k = 1024 .

6 Conclusion

For a database U ∈ R n × p consisting of n data points, our goal is to make its random projections (RP) and SignRP differentially private-the DP guarantee we have derived holds independently and simultaneously for every data vector u ∈ U . There are works on randomized algorithms that assume the randomness of the algorithm are 'internal' and also kept private [7, 73, 25]. We assume the projection matrix W is known/public, which is stronger in privacy and enables broader applications of the released data. Also, a 'known projection matrix' is typical in the local DP setting [47, 18], since the projection matrix is created and shared among the users by the data aggregator.

In our study, we first revisit the prior work [49] on the ( ϵ, δ ) -DP random projection based on Gaussian noise addition ('DP-RP-G'). Then we incorporate the optimal Gaussian mechanism and propose an improved method DP-RP-G-OPT. We also develop DP-OPORP based on the optimal Gaussian mechanism. We then propose algorithms for privatizing the (1-bit) DP-SignRP: 'DP-SignRP-RR' based on the standard 'randomized response' (RR), and 'DP-SignRP-RR-smooth' based on a proposed concept of 'smooth flipping probability' of SignRP. We extend the idea of smooth flipping probability to OPORP (a variant of count-sketch) and demonstrate the advantage of DP-SignOPORP.

Experiments on retrieval and classification tasks justify the effectiveness of our proposed methods. In particular, DP-SignOPORP outperforms other methods especially when ϵ is not large (e.g., ϵ ≤ 5 ). These results also verify the advantage of using the smooth flipping probability.

References

Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS) , pages 308-318, Vienna, Austria, 2016.

Dimitris Achlioptas. Database-friendly random projections: Johnson-lindenstrauss with binary coins. J. Comput. Syst. Sci. , 66(4):671-687, 2003.

Eman Abdullah AlOmar, Wajdi Aljedaani, Murtaza Tamjeed, Mohamed Wiem Mkaouer, and Yasmine N. El-Glaly. Finding the needle in a haystack: On the automatic identification of accessibility user reviews. In Proceedings of the Conference on Human Factors in Computing Systems (CHI) , pages 387:1-387:15, Virtual Event / Yokohama, Japan, 2021.

Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 403-412, Stockholmsmässan, Stockholm, Sweden, 2018.

Arnaud Berlioz, Arik Friedman, Mohamed Ali Kâafar, Roksana Boreli, and Shlomo Berkovsky. Applying differential privacy to matrix factorization. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys) , pages 107-114, Vienna, Austria, 2015.

Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: Applications to image and text data. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , pages 245-250, San Francisco, CA, 2001.

Jeremiah Blocki, Avrim Blum, Anupam Datta, and Or Sheffet. The johnson-lindenstrauss transform itself preserves differential privacy. In Proceedings of the 53rd Annual IEEE Symposium on Foundations of Computer Science (FOCS) , pages 410-419, New Brunswick, NJ, 2012.

Avrim Blum, Cynthia Dwork, Frank McSherry, and Kobbi Nissim. Practical privacy: the SuLQ framework. In Proceedings of the Twenty-fourth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) , pages 128-138, Baltimore, MD, 2005.

Petros Boufounos and Richard G. Baraniuk. 1-bit compressive sensing. In Proceedings of the 42nd Annual Conference on Information Sciences and Systems (CISS) , pages 16-21, Princeton, NJ, 2008.

Emmanuel J. Candès, Justin K. Romberg, and Terence Tao. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory , 52(2):489-509, 2006.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):27, 2011.

Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. Theor. Comput. Sci. , 312(1):3-15, 2004.

Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the Thiry-Fourth Annual ACM Symposium on Theory of Computing (STOC) , pages 380-388, Montreal, Canada, 2002.

Kamalika Chaudhuri and Claire Monteleoni. Privacy-preserving logistic regression. In Advances in Neural Information Processing Systems (NIPS) , pages 289-296, Vancouver, Canada, 2008.

Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Differentially private empirical risk minimization. J. Mach. Learn. Res. , 12:1069-1109, 2011.

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL) , pages 1870-1879, Vancouver, Canada, 2017.

Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing Neural Networks with the Hashing Trick. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , pages 2285-2294, Lille, France, 2015.

Graham Cormode, Somesh Jha, Tejas Kulkarni, Ninghui Li, Divesh Srivastava, and Tianhao Wang. Privacy at scale: Local differential privacy in practice. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD) , pages 1655-1658, Houston, TX, 2018.

Graham Cormode, Tejas Kulkarni, and Divesh Srivastava. Marginal release under local differential privacy. In Proceedings of the 2018 International Conference on Management of Data (SIGMOD) , pages 131-146, Houston, TX, 2018.

Corinna Cortes and Vladimir Vapnik. Support-vector networks. Mach. Learn. , 20(3):273-297, 1995.

George E. Dahl, Jack W. Stokes, Li Deng, and Dong Yu. Large-scale malware classification using random projections and neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3422-3426, Vancouver, Canada, 2013.

Sanjoy Dasgupta. Experiments with random projection. In Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence (UAI) , pages 143-151, Stanford, CA, 2000.

Sanjoy Dasgupta and Yoav Freund. Random projection trees and low dimensional manifolds. In Proceedings of the 40th Annual ACM Symposium on Theory of Computing (STOC) , pages 537-546, Victoria, Canada, 2008.

Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the Twentieth Annual Symposium on Computational Geometry (SCG) , pages 253-262, Brooklyn, NY, 2004.

Charlie Dickens, Justin Thaler, and Daniel Ting. Order-invariant cardinality estimators are differentially private. In Advances in Neural Information Processing Systems (NeurIPS) , New Orleans, LA, 2022.

Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. In Advances in Neural Information Processing Systems (NIPS) , pages 3571-3580, Long Beach, CA, 2017.

Wei Dong, Moses Charikar, and Kai Li. Asymmetric distance estimation with sketches for similarity search in high-dimensional spaces. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , pages 123-130, 2008.

David L. Donoho. Compressed sensing. IEEE Trans. Inf. Theory , 52(4):1289-1306, 2006.

Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the 41st Annual ACM Symposium on Theory of Computing (STOC) , pages 371-380, Bethesda, MD, 2009.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Theory of Cryptography Conference (TCC) , pages 265-284, New York, NY, 2006.

Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. , 9(3-4):211-407, 2014.

Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. RAPPOR: randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security (CCS) , pages 1054-1067, Scottsdale, AZ, 2014.

Chenglin Fan, Ping Li, and Xiaoyun Li. Private graph all-pairwise-shortest-path distance release with improved error rate. In Advances in Neural Information Processing Systems (NeurIPS) , New Orleans, LA, 2022.

Chenglin Fan, Ping Li, and Xiaoyun Li. k -median clustering via metric embedding: Towards better initialization with differential privacy. In Advances in Neural Information Processing Systems (NeurIPS) , New Orleans, LA, 2023.

Huang Fang, Xiaoyun Li, Chenglin Fan, and Ping Li. Improved convergence of differential private sgd with gradient clipping. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR) , Kigali, Rwanda, 2023.

Dan Feldman, Amos Fiat, Haim Kaplan, and Kobbi Nissim. Private coresets. In Proceedings of the 41st Annual ACM Symposium on Theory of Computing (STOC) , pages 361-370, Bethesda, MD, 2009.

Xiaoli Zhang Fern and Carla E. Brodley. Random projection for high dimensional data clustering: Acluster ensemble approach. In Proceedings of the Twentieth International Conference (ICML) , pages 186-193, Washington, DC, 2003.

Yoav Freund, Sanjoy Dasgupta, Mayank Kabra, and Nakul Verma. Learning the structure of manifolds using random projections. In Advances in Neural Information Processing Systems (NIPS) , pages 473-480, Vancouver, Canada, 2007.

Jerome H. Friedman, Forest Baskett, and Leonard J. Shustek. An algorithm for finding nearest neighbors. IEEE Trans. Computers , 24(10):1000-1006, 1975.

Jason Ge, Zhaoran Wang, Mengdi Wang, and Han Liu. Minimax-optimal privacy-preserving sparse PCA in distributed systems. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1589-1598, Playa Blanca, Lanzarote, Canary Islands, Spain, 2018.

Michel X. Goemans and David P. Williamson. Improved approximation algorithms for maximumcut and satisfiability problems using semidefinite programming. J. ACM , 42(6):1115-1145, 1995.

Farzin Haddadpour, Belhal Karimi, Ping Li, and Xiaoyun Li. Fedsketch: Communicationefficient and private federated learning via sketching. arXiv preprint arXiv:2008.04975 , 2020.

Justin Hsu Marco Gaboardi Andreas Haeberlen and Sanjeev Khanna. Differential privacy: An economic method for choosing epsilon. arXiv preprint arXiv:1402.3329 , 2014.

Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposium on the Theory of Computing (STOC) , pages 604-613, Dallas, TX, 1998.

Prateek Jain, Om Dipakbhai Thakkar, and Abhradeep Thakurta. Differentially private matrix completion revisited. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 2220-2229, Stockholmsmässan, Stockholm, Sweden, 2018.

William B. Johnson and Joram Lindenstrauss. Extensions of Lipschitz mapping into Hilbert space. Contemporary Mathematics , 26:189-206, 1984.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. Extremal mechanisms for local differential privacy. In Advances in Neural Information Processing Systems (NeurIPS) , pages 2879-2887, Montreal, Canada, 2014.

Christopher T Kenny, Shiro Kuriwaki, Cory McCartan, Evan TR Rosenman, Tyler Simko, and Kosuke Imai. The use of differential privacy for census data and its impact on redistricting: The case of the 2020 us census. Science advances , 7(41):eabk3283, 2021.

Krishnaram Kenthapadi, Aleksandra Korolova, Ilya Mironov, and Nina Mishra. Privacy via the johnson-lindenstrauss transform. J. Priv. Confidentiality , 5(1), 2013.

Karin Knudson, Rayan Saab, and Rachel Ward. One-bit compressive sensing with norm estimation. IEEE Trans. Inf. Theory , 62(5):2748-2758, 2016.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, University of Toronto , 2009.

Béatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics , pages 1302-1338, 2000.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE , 86(11):2278-2324, 1998.

Cong Leng, Jian Cheng, and Hanqing Lu. Random subspace for binary codes learning in large scale image retrieval. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , pages 1031-1034, Gold Coast, Australia, 2014.

Ping Li. Sign-full random projections. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI) , pages 4205-4212, Honolulu, HI, 2019.

Ping Li, Trevor Hastie, and Kenneth Ward Church. Very sparse random projections. In roceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , pages 287-296, Philadelphia, PA, 2006.

Ping Li and Xiaoyun Li. OPORP: one permutation + one random projection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) , pages 1303-1315, Long Beach, CA, 2023.

Ping Li, Michael Mitzenmacher, and Anshumali Shrivastava. Coding for random projections. In Proceedings of the 31th International Conference on Machine Learning (ICML) , pages 676-684, Beijing, China, 2014.

Ping Li and Weijie Zhao. Gcwsnet: Generalized consistent weighted sampling for scalable and accurate training of neural networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM) , pages 1188-1198, Atlanta, GA, 2022.

Xiaoyun Li and Ping Li. Generalization error analysis of quantized compressive learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 15124-15134, Vancouver, Canada, 2019.

Xiaoyun Li and Ping Li. Random projections with asymmetric quantization. In Advances in Neural Information Processing Systems (NeurIPS) , pages 10857-10866, Vancouver, Canada, 2019.

Xiaoyun Li and Ping Li. One-sketch-for-all: Non-linear random features from compressed linear measurements. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 2647-2655, Virtual Event, 2021.

Xiaoyun Li and Ping Li. Quantization algorithms for random Fourier features. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 6369-6380, Virtual Event, 2021.

Xiaoyun Li and Ping Li. Signrff: Sign random fourier features. In Advances in Neural Information Processing Systems (NeurIPS) , pages 17802-17817, New Orleans, LA, 2022.

Xiaoyun Li and Ping Li. Differentially private one permutation hashing and bin-wise consistent weighted sampling. arXiv preprint arXiv:2306.07674 , 2023.

Kobbi Nissim, Sofya Raskhodnikova, and Adam D. Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the 39th Annual ACM Symposium on Theory of Computing (STOC) , pages 75-84, San Diego, CA, 2007.

Donald Bruce Owen. A table of normal integrals: A table. Communications in StatisticsSimulation and Computation , 9(4):389-419, 1980.

Stephan Rabanser, Stephan Günnemann, and Zachary C. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In Advances in Neural Information Processing Systems (NeurIPS) , pages 1394-1406, Vancouver, Canada, 2019.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems (NIPS) , pages 1177-1184, Vancouver, Canada, 2007.

Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. FetchSGD: Communication-efficient federated learning with sketching. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 8253-8265, Virtual Event, 2020.

Anshumali Shrivastava and Ping Li. In defense of minhash over simhash. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 886-894, Reykjavik, Iceland, 2014.

Karan Singhal, Hakim Sidahmed, Zachary Garrett, Shanshan Wu, John Rush, and Sushant Prakash. Federated reconstruction: Partially local federated learning. In Advances in Neural Information Processing Systems (NeurIPS) , virtual, 2021.

Adam D. Smith, Shuang Song, and Abhradeep Thakurta. The flajolet-martin sketch itself preserves differential privacy: Private counting with minimal space. In Advances in Neural Information Processing Systems , virtual, 2020.

Nina Mesing Stausholm. Improved differentially private euclidean distance approximation. In Proceedings of the 40th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems (PODS) , pages 42-56, Virtual Event, China, 2021.

Tyler M. Tomita, James Browne, Cencheng Shen, Jaewon Chung, Jesse Patsolic, Benjamin Falk, Carey E. Priebe, Jason Yim, Randal C. Burns, Mauro Maggioni, and Joshua T. Vogelstein. Sparse projection oblique randomer forests. J. Mach. Learn. Res. , 21:104:1-104:39, 2020.

Santosh S Vempala. The random projection method , volume 65. American Mathematical Soc., 2005.

Stanley L Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association , 60(309):63-69, 1965.

Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Trans. Inf. Forensics Secur. , 15:3454-3469, 2020.

Kilian Q. Weinberger, Anirban Dasgupta, John Langford, Alexander J. Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML) , pages 1113-1120, Montreal, Canada, 2009.

Jun Wu, Jingrui He, and Jiejun Xu. DEMO-Net: Degree-specific graph neural networks for node and graph classification. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 406-415, Anchorage, AK, 2019.

Bin Yang, Issei Sato, and Hiroshi Nakagawa. Bayesian differential privacy on correlated data. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD) , pages 747-762, Melbourne, Australia, 2015.

Hang Zhang and Ping Li. Optimal estimator for unlabeled linear regression. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 11153-11162, Virtual Event, 2020.

Hang Zhang and Ping Li. One-step estimator for permuted sparse recovery. In Proceedings of the International Conference on Machine Learning (ICML) , pages 41244-41267, Honolulu, HI, 2023.

Jun Zhang, Graham Cormode, Cecilia M. Procopiuc, Divesh Srivastava, and Xiaokui Xiao. Privbayes: private data release via bayesian networks. In International Conference on Management of Data (SIGMOD) , pages 1423-1434, Snowbird, UT, 2014.

Jun Zhang, Zhenjie Zhang, Xiaokui Xiao, Yin Yang, and Marianne Winslett. Functional mechanism: Regression analysis under differential privacy. Proc. VLDB Endow. , 5(11):13641375, 2012.

Shan Zhang, Lei Wang, Naila Murray, and Piotr Koniusz. Kernelized few-shot object detection with efficient integral aggregation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19185-19194, New Orleans, LA, 2022.

Zhaoqi Zhang, Panpan Qi, and Wei Wang. Dynamic malware analysis with feature engineering and feature learning. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI) , pages 1210-1217, New York, NY, 2020.

Fuheng Zhao, Dan Qiao, Rachel Redberg, Divyakant Agrawal, Amr El Abbadi, and Yu-Xiang Wang. Differentially private linear sketches: Efficient implementations and applications. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Argyrios Zymnis, Stephen P. Boyd, and Emmanuel J. Candès. Compressed sensing with quantized measurements. IEEE Signal Process. Lett. , 17(2):149-152, 2010.

Contents

A Analytic DP-RP and the Laplace Mechanism

A.1 Analytic DP-RP-G Approach: Bounding the Sensitivity

According to Theorem 3.1, one will choose σ based on the observed ∆ 2 which is calculated according to (3) for a given realization of the projection matrix W . For the convenience of theoretical analysis, it is often helpful to derive the criterion for choosing σ directly in terms of the input parameters ϵ , δ , p , and k , to understand and better compare the noise level. We call the corresponding algorithm to be 'Analytic DP-RP-G'. This is achieved by bounding ∆ 2 in high probability.

Lemma A.1 ([52]) . Let Y 1 , ..., Y n be iid standard Gaussian random variables, and denote Z i = ∑ n i =1 Y 2 i . Then for any t > 0 ,

Lemma A.2 (Bounding ∆ 2 ) . Let W ∈ R p × k be a random matrix with iid N (0 , 1) entries, and ∆ 2 be defined in (3). Then for any 0 < δ < 1 ,

Proof. Let e i be the unit base vector with the i -th dimension being 1 and all other dimensions being zero. For each i = 1 , ..., p , W T e i follows iid chi-square distribution with k degree of freedom. Applying Lemma A.1 we have that for any δ ′ > 0 ,

Applying union bound gives

Setting δ = pδ

′ and taking the square root on both sides completes the proof.

With the high probability bound on the l 2 -sensitivity ∆ 2 , we have the following result.

Theorem A.3 (Analytic DP-RP-G) . For any ϵ > 0 and 0 < δ < 1 2 , Algorithm 1 achieves ( ϵ, δ ) -DP when σ ≥ β √ 1+2 √ log(2 p/δ ) /k +2log(2 p/δ ) /k √ 2(log(2 /δ )+ ϵ ) ϵ .

Proof. By Lemma A.2, we know that with probability δ/ 2 , ∆ 2 is bounded by β √ 1 + 2 √ log(2 p/δ ) /k +2log(2 p/δ ) /k . In this event, by Theorem 3.1, Algorithm 1 is ( ϵ, δ/ 2) -DP when σ ≥ ∆ 2 √ 2(log(2 /δ )+ ϵ ) . Therefore, the approach is ( ϵ, δ ) -DP.

We remark that, since this analytic version is built upon a high probability bound on the sensitivity, in most cases, the noise variance in Theorem A.3 will be slightly larger than the one computed by the realized/exact sensitivity in Theorem 3.1.

A.2 The Laplace Noise Mechanism for DP-RP

We briefly discuss the Laplace mechanism for DP-RP (and why we did not use it).

Definition A.1 ( l 1 -sensitivity) . Let Nb ( u ) denote the neighbor set of u , and N ( U ) = { ( u, u ′ ) : u ′ ∈ Nb ( u ) , u, u ′ ∈ U} be the collection of all possible neighboring data pairs. The l 1 -sensitivity of a function f : U ↦→ R k is defined as △ 1 = max ( u,u ′ ) ∈N ( U ) ∥ f ( u ) -f ( u ′ ) ∥ 1 .

Theorem A.4 (Laplace mechanism [30]) . Given any function f : U ↦→ R k and ϵ > 0 , the randomized algorithm M ( u ) = f ( u ) + ( Z 1 , ..., Z k ) is ϵ -DP, where Z 1 , ..., Z k follow iid Laplace (∆ 1 /ϵ ) . The centered Laplace distribution with scale λ has density function g ( x ) = 1 2 λ e -| x | λ .

Algorithm 6: DP-RP-L

1 Input: Data u ∈ [ -1 , 1] p , privacy parameters ϵ > 0 , number of projections k

2 Output: ϵ -differentially private random projections ˜ x ∈ R k

3 Apply RP x = 1 √ k W T u , where W ∈ R p × k is a random N (0 , 1) matrix

4 Compute sensitivity ∆ 1 by Definition A.1

5 Generate iid random noise vector L ∈ R k following Laplace ( ∆ 1 ϵ )

6 Return ˜ x = x + L

By Theorem A.4, one may add Laplace ( λ ) noise to the projected data with λ = ∆ 1 /ϵ to achieve ϵ -DP. For completeness, we detail the approach in Algorithm 6, which is the same as Algorithm 1 except for the noise added. ∆ 1 is the l 1 -sensitivity for a given W given in Definition A.1: ∆ 1 = 1 √ k β max i =1 ,...,p ∥ W [ i, :] ∥ 1 . Each term in the maximum operator is a sum of k iid standard halfnormal random variables ( Y = | X | where X is standard normal). The readers might be interested in a comparison between the noise level of the two noise addition mechanisms, i.e., DP-RP-G and DP-RP-L. To observe the dependence of the noise level λ on ϵ , k and p , we derive a tail bound on the l 1 -sensitivity. We first establish the following lemma.

Lemma A.5 (Half-normal tail bound) . Let X 1 , ..., X n be iid N (0 , 1) random variables and let Y i = | X i | , ∀ i . Denote Z = ∑ n i =1 Y i . Then for any t > 0 , it holds that

Proof. By the Markov inequality, we write (with some ζ > 0 )

By the independence of Y i and the moment generating function of half-normal distribution, we have

Hence, we have

which is minimized at ζ = t n , leading to

Taking t = √ 2 n 2 log 2 + 2 nt completes the proof.

Using Lemma A.5, we obtain a high probability bound on the l 1 -sensitivity as follows. The proof is omitted since it is similar to the proof of Lemma A.2.

Lemma A.6 (Bounding ∆ 1 ) . Let W ∈ R p × k be a random matrix with iid N (0 , 1) entries, and ∆ 1 be defined in Definition A.1. Then for any 0 < δ < 1 ,

It is important to point out that, unlike in the Gaussian mechanism, simply inserting the upper bound in Lemma A.6 to replace the (exact) ∆ 1 in the noise level of Laplace (∆ 1 /ϵ ) in Algorithm 6 does not provide pure ϵ -DP. In other words, we must compute the actual ∆ 1 by Definition A.1 for a realization

of the projection matrix W . This is because, using the high probability bound of ∆ 1 would introduce a failure probability (i.e., when the actual ∆ 1 is larger than the bound) which violates ϵ -DP.

Comparison with DP-RP-G-OPT. Lemma A.6 provides an estimate of the noise level required by the Laplace mechanism. We can roughly compare DP-RP-L with DP-RP-OPT in terms of the noise variances, which equal σ 2 for Gaussian and 2 λ 2 for Laplace, respectively. Lemma A.6 shows that in most cases, the Laplace noise requires λ = O ( √ k ϵ ) . Recall that in DP-RP-G and DP-RP-G-OPT, the Gaussian noise level σ = O ( 1 ϵ ) when ϵ → 0 and σ = O ( 1 √ ϵ ) when ϵ →∞ . Thus, the Gaussian noise would be smaller with relatively small ϵ , and the Laplace noise would be smaller only when ϵ = Ω( k ) , which is usually too large to be useful since in practice k is usually hundreds to thousands. As a result, we would expect DP-RP-G methods to perform better than DP-RP-L in common scenarios. Therefore, in this work, we mainly focused on the Gaussian mechanism for DP-RP noise addition.

B Comparison of Different Projection Matrices and the Benefits of Rademacher RP

Besides Gaussian random projection, we can also adopt other types of projection matrices which might even work better for DP. The following distributions of w ij are popular:

The uniform distribution, √ 3 × unif [ -1 , 1] . The √ 3 factor is placed here to have E ( w 2 ij ) = 1 by following the convention in the practice of random projections.

The 'very sparse' distribution, as used in [56]:

which generalizes [2] (for s = 1 and s = 3 ). Note that when s = 1 , it is also called the 'symmetric Bernoulli' distribution or the 'Rademacher' distribution.

Next, we compare these various types of projection matrices and show that Rademacher (symmetric Bernoulli) random projection is superior to Gaussian random projection for both DP-RP and DPSignRP in that less perturbation is required to achieve the same privacy level.

B.1 Rademacher Projection for DP-RP

From Theorem 3.1 and Theorem 3.2, it is clear that the noise magnitude of Gaussian noise in DP-RP directly depends on the l 2 -sensitivity △ 2 , which, according to (3), equals the largest row norm of the projection matrix W . Among the above mentioned distributions, the dense Rademacher projection ( s = 1 in (7)) has △ 2 = 1 √ k β × √ k = β which is independent of p . This could be much smaller than the dense Gaussian projection (i.e., DP-RP-G-OPT).

10

200

400

600

800

1000

p

1

1.1

1.2

1.3

= 1

k = 256

uniform

s = 1

s = 3

s = 10

10

200

400

600

800

1000

p

1

1.1

1.2

1.3

= 1

k = 512

uniform

s = 1

s = 3

s = 10

In Figure 5, we numerically simulate the △ 2 of different projection matrices, which shows that the Rademacher projection produces the smallest sensitivity. This, when plugged into the optimal Gaussian mechanism (Theorem 3.2), leads to smaller Gaussian noise variance needed.

B.2 Rademacher Projection for DP-SignRP

From our analysis, it is clear that the flipping probability of DP-SignRP (both DP-SignRP-RR and DP-SignRP-RR-smooth) essentially depends on how concentrated the projected data is around zero. Particularly, N + in Algorithm 3, as given in Proposition 4.3, is a high probability upper bound on a Binomial random variable with success probability Pr ( β max i =1 ,...,p | w i | ≥ | w T u | ) with w ∼ N (0 , 1) . In Algorithm 4, L j = ⌈ | w T j u | β max i =1 ,...,p | W ij | ⌉ . For both quantities, a smaller value leads to a smaller sign flipping probability and thus better utility.

N + in DP-SignRP-RR. We first consider the N + in Algorithm 3, which determines the flipping probability 1 e ϵ/N + +1 . Particularly, N + in Algorithm 3, as given in Proposition 4.3, is a high probability upper bound on a Binomial random variable with success probability

where w is the p -dimensional projection vector. When w i is sampled from the Rademacher distribution, i.e., w i ∈ {-1 , +1 } with equal probabilities, the probability calculation can be simplified:

Based on the central limit theorem, the normal approximation (9) is accurate unless p is very small. Recall that, when w i 's are sampled from the Gaussian distribution, we can calculate an upper bound in (20), which is re-written as below:

Next, we provide a simulation study to justify the approximation and compare different distributions in terms of their impact on the probability (8), for β = 1 as well as β = 0 . 1 . For simplicity, we simulate the data as a p -dimensional vector of uniform random numbers sampled from unif [ -1 , 1] . We experiment with five different choices of w : the standard Gaussian, the uniform, the 'very sparse' distribution (7) with s = 1 , s = 3 , and s = 10 . We vary p from 10 to 1000. For each case, we repeat the simulations 10 7 times to ensure sufficient accuracy. Figure 6 verifies that the two approximations (9) and (10) are accurate. In Figure 7, we provide the curves for more types of projection matrices. From both figures, we clearly see that using the Rademacher projection can considerably reduce (8) compared with Gaussian (and other) projections, leading to smaller N + value. This typically implies better utility.

L j in DP-SignRP-RR-smooth. Similarly, we numerically evaluate the L j in Algorithm 4. We run Algorithm 4 with k = 512 , which gives 512 L j values. In Figure 8, we plot the proportion (or the approximated distribution) of the values of L j among k projections. As we see, Rademacher projection produces least number of small L j values, and largest number of higher L j values. As the smooth flipping probability equals 1 exp( L j k ϵ )+1 , larger L j leads to smaller probability of sign flipping. Hence, Rademacher is again the best choice for the projection matrix.

10

200

400

600

800

1000

p

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Prob

= 1

Approx

10

200

400

600

800

1000

p

0

0.02

0.04

0.06

0.08

Prob

= 0.1

Approx

10

200

400

600

800

1000

p

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Prob

= 1

uniform

s = 1

s = 3

s = 10

10

200

400

600

800

1000

p

0

0.02

0.04

0.06

0.08

Prob

= 0.1

uniform

s = 1

s = 3

s = 10

C Comparison of DP-RP and DP-OPORP on Inner Product Estimation

In this section, we theoretically compare DP-RP and DP-OPORP. In Section B, we have shown that Rademacher projection requires lowest noise magnitude. Thus, we consider DP-RP with Rademacher projections here. For clarity, we summarize the algorithm in Algorithm 7. We name it 'DP-RPG-OPT-B', where 'G' stands for the G aussian noise mechanism and 'B' stands for 'symmetric B ernoulli' projections.

Algorithm 7: DP-RP-G-OPT-B

1

Input: Data u ∈ [ - 1 , 1] p , privacy parameters ϵ > 0 , δ ∈ (0 , 1) , number of projections k

2 Output: ( ϵ, δ ) -differentially private random projections ˜ x ∈ R k

3 Apply RP x = 1 √ k W T u , where W ∈ R p × k is a random Rademacher matrix

4 Generate iid random noise vector G ∈ R k following N (0 , σ 2 ) where σ is obtained by Theorem 3.2 with △ 2 = β

5 Return ˜ x = x + G

In our analysis, for simplicity we assume the data are normalized, i.e., the data vector has l 2 norm equal to 1. In this case, the inner product is also the cosine. The baseline method is the most straightforward: we add optimal Gaussian noise to each dimension of the original data (Raw-dataG-OPT). For this strategy, the sensitivity is also △ 2 = β . This means, when we compare all three methods: Raw-data-G-OPT, DP-RP-G-OPT-B, and DP-OPORP, the noise level σ is the same. This makes it convenience to conduct the comparisons, from which we can gain valuable insights.

1

2

4

6

8

10

L

0

0.1

0.2

0.3

0.4

0.5

Proportion

= 1

p = 100

uniform

s = 1

s = 3

s = 10

1

5

10

15

20

L

0

0.05

0.1

0.15

Proportion

= 1

p = 1000

uniform

s = 1

s = 3

s = 10

Theorem C.1 (Raw-data-G-OPT, i.e., adding optimal Gaussian noise on raw data) . Let σ be the solution to (4) with △ 2 = β . For any u, v ∈ U , let ˜ u i = u i + a i and ˜ v i = v i + b i be the DP noisy vectors, with a i , b i ∼ N (0 , σ 2 ) i.i.d. Then, denote ˆ g org = ∑ p i =1 ˜ u i ˜ v i . we have

Proof. To add Gaussian noise to the original data, it suffices to find the sensitivity, which, by Definition 2.2, is △ 2 = β . Thus, the approach is ( ϵ, δ ) -DP according to the optimal Gaussian mechanism (Theorem 3.2). To compute the mean and variance, consider some i ∈ [ p ] . We have

Thus, taking the sum implies E [ˆ g org ] = ∑ p i =1 u i v i . For the variance,

which leads to

Therefore, by independence,

which proves the claim.

For DP-RP-G-OPT-B and DP-OPORP, we have the following results.

Theorem C.2 (DP-RP-G-OPT-B inner product estimation) . Let σ be the solution to (4) with △ 2 = β . In Algorithm 7, let W ∈ {-1 , 1 } p × k be a Rademacher random matrix. Denote x = 1 √ k W T u , y = 1 √ k W T v , and a , b are two random Gaussian noise vectors following N (0 , σ 2 ) . Let ˆ g rp = ∑ k j =1 ( x j + a j )( y j + b j ) . Then, E [ˆ g rp ] = ∑ p i =1 u i v i , and

Proof. The conditional mean and variance can be computed as

where the variance calculation follows from Theorem C.1. Hence, we have

In the above calculation, the formula of V ar ( ∑ k j =1 x j y j ) is from the result in [56] with s = 1 for Rademacher distribution.

Algorithm 8: DP-OPORP

1 Input: Data u ∈ [ -1 , 1] p , privacy parameters ϵ > 0 , δ ∈ (0 , 1) , number of projections k

2 Output: Differentially private OPORP

3 Apply Algorithm 2 with a random Rademacher projection vector to obtain the OPORP x

4 Set sensitivity ∆ 2 = β

5 Generate iid random vector G ∈ R k following N (0 , σ 2 ) where σ is computed by Theorem 3.2

6 Return ˜ x = x + G

Theorem C.3 (DP-OPORP inner product estimation) . Let σ be the solution to (4) with △ 2 = β . Let w ∈ {-1 , 1 } p be a Rademacher random vector. In Algorithm 8, let x and y be the OPORP of u and v , and a , b be two random Gaussian noise vectors following N (0 , σ 2 ) . Denote ˆ g oporp = ∑ k j =1 ( x j + a j )( y j + b j ) . Then, E [ˆ g oporp ] = ∑ p i =1 u i v i , and

Proof. The proof is similar to that of Theorem C.2, with the help of the result in [57].

The variance reduction factor p -k p -1 can be quite beneficial when p is not very large. Also, see [57] for the normalized estimators for both OPORP and VSRP (very sparse random projections). The normalization steps can substantially reduce the estimation variance.

Comparison. For the convenience of comparison, let us assume that the data are row-normalized, i.e., ∥ u ∥ 2 = 1 for all u ∈ U . Let ρ = ∑ p i =1 u i v i . We have

For high-dimensional data (large p ), we see that ˆ g rp and ˆ g oporp has roughly the same variance, approximately 2 σ 2 + kσ 4 + 1 k . We would like to compare this with V ar (ˆ g org ) = 2 σ 2 + pσ 4 the variance for adding noise directly to the original data.

Let's define the ratio of the variances:

to illustrate the benefit of RP-type algorithms (DP-RP and DP-OPORP) in protecting the privacy of the (high-dimensional) data. If p k = 100 , then it is possible that the ratio of the variances can be roughly 100. This would be a huge advantage. Figure 9 plots the ratio R for p = 1000 and p = 10000 as well as a series of k/p values, with respect to σ .

10 -2

10 -1

10 0

10 1

10 2

10 -3

10 -2

10 -1

10 0

10 1

10 2

Ratio

k/p = 0.01

k/p = 0.05

k/p = 0.1

k/p = 0.5

p = 1000

10 -2

10 -1

10 0

10 1

10 2

10 -2

10 -1

10 0

10 1

10 2

Ratio

k/p = 0.01

k/p = 0.05

k/p = 0.1

k/p = 0.5

p = 10000

10 -1

10 0

10 1

10 2

10 -2

10 -1

10 0

10 1

10 2

10 3

10 4

2

= 0.1

2

= 1

2

= 10

2

= 100

= 1e-6

10 -1

10 0

10 1

10 2

2

10 -2

10 -1

10 0

10 1

10 2

10 3

10 4

= 0.1

= 1

= 10

= 100

= 1e-6

Figure 9 also illustrates when it might be a good strategy to directly add noise to the original data. For example, when p = 1000 , the ratio can be below 1 if σ < 0 . 1 . One can numerically verify that, (in Figure 10) in order for σ < 0 . 1 at ∆ 2 = β = 1 , we need ϵ > 100 . In other words, adding noise to the raw data might be plausible when ϵ > 100 . In the literature, however, many DP applications typically require a much smaller ϵ , such as ϵ ∈ [0 . 1 , 20] (e.g., [43, 48]). Therefore, DP-RP and DP-OPORP is much better (i.e., has much smaller inner product estimation variance) than adding noise to the raw data in common privacy regimes.

D More Experiment Results

We provide the complete set of plots of our experimental results. In Figure 11 and Figure 12, we report the precision@10 and recall@100 curves of DP-RP variants and DP-OPORP on MNIST and CIFAR, respectively. In Figure 13, we report the test accuracy on the Webspam dataset of these methods. From all plots, we see that DP-RP-G-OPT-B and DP-OPORP perform equally the best on all the tasks, significanly better than the strategy of adding Gaussian noise to the raw data.

In Figure 14, we report the recall@100 metric of DP-SignOPORP methods in addition to the precision@10 metric shown in the main paper. For completeness, we also include the SVM test accuracy in Figure 15, which is the same as Figure 4 in the main context.

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 64

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 128

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 256

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

recall@100

MNIST: k = 64

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

recall@100

MNIST: k = 128

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

recall@100

MNIST: k = 256

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

precision@10

CIFAR: k = 64

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

precision@10

CIFAR: k = 128

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

precision@10

CIFAR: k = 256

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.1

0.2

0.3

0.4

recall@100

CIFAR: k = 64

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.1

0.2

0.3

0.4

recall@100

CIFAR: k = 128

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.1

0.2

0.3

0.4

recall@100

CIFAR: k = 256

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

test accuracy (%)

WEBSPAM: k = 256

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G

0

5

10

15

20

60

70

80

90

test accuracy (%)

WEBSPAM: k = 512

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

test accuracy (%)

WEBSPAM: k = 1024

DP-OPORP

DP-RP-G-OPT-B

DP-RP-G-OPT

DP-RP-G

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 128

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 256

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

precision@10

MNIST: k = 512

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

recall@100

MNIST: k = 128

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

0.8

recall@100

MNIST: k = 256

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

0

0.2

0.4

0.6

recall@100

MNIST: k = 512

DP-SignOPORP-RR t=2

DP-SignOPORP-RR t=4

DP-SignOPORP-RR-s t=2

DP-SignOPORP-RR-s t=4

DP-OPORP

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

test accuracy (%)

WEBSPAM: k = 256

DP-SignOPORP-RR t=1

DP-SignOPORP-RR-s t=1

DP-OPORP

DP-RP-G-OPT-B

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

test accuracy (%)

WEBSPAM: k = 512

DP-SignOPORP-RR t=1

DP-SignOPORP-RR-s t=1

DP-OPORP

DP-RP-G-OPT-B

Raw-data-G-OPT

0

5

10

15

20

60

70

80

90

100

test accuracy (%)

WEBSPAM: k = 1024

DP-SignOPORP-RR t=1

DP-SignOPORP-RR-s t=1

DP-OPORP

DP-RP-G-OPT-B

Raw-data-G-OPT

E Deferred Proofs

The following lemma on Gaussian random variables will be used in our proof for Lemma 4.2, and may also be of independent interest.

1. Pr ( | X | > | Y | ) = 1 π [ tan -1 ( r -ρ √ 1 -ρ 2 ) +tan -1 ( r + ρ √ 1 -ρ 2 )] . When r ≤ 1 , the maximum is achieved at ρ = 0 , i.e., max ρ Pr ( | X | < | Y | ) = 2 π tan -1 ( r ) . 2. The conditional expectation:

3. The conditional tail probability: for any r > 0 , ρ ∈ ( -1 , 1) , for any t > 0 ,

Proof. The bivariate normal density function is

Therefore, we have E [ | X | ∣ ∣ | X | > | Y | ] = A P , with

Note that P = Pr ( | X | > | Y | ) in the first statement of the theorem. Our calculation will use the following two identities involving the Gaussian functions [67]:

where ϕ ( x ) and Φ( x ) are the pdf and cdf of the standard Gaussian distribution.

With a proper change of random variables, we can compute A as

For the first term we have

where we denote r = σ x σ y and use (17). Similarly, we have that

Therefore, we obtain

To compute P , by doing a similar change of variables, we have

Using (16), we obtain

which leads to

Therefore, we know that

with r = σ x /σ y . We now investigate the derivative of P . By some algebra, we can show that

When 0 < r ≤ 1 , ∂P ( ρ,r ) ∂ρ ≥ 0 when ρ ≤ 0 and ∂P ( ρ,r ) ∂ρ ≤ 0 when ρ > 0 . Therefore, max ρ P ( ρ, r ) = P (0 , r ) = 2 π tan -1 ( r ) .

Tail bound. By our previous calculations, the conditional distribution of X given | X | > | Y | is

with P = Pr ( | X | > | Y | ) in (19) the normalizing constant to make the integral equal to 1. The conditional tail probability can be computed as follows. For some t > 0 , by symmetry,

For ˜ P 1 , using polar coordinates we have

Similarly,

Therefore, we obtain

since cos 2 ( θ ) ∈ [0 , 1] . Notice that P in (19) can be written as P = 1 π ∫ tan -1 ( r -ρ √ 1 -ρ 2 ) -tan -1 ( r + ρ √ 1 -ρ 2 ) dθ . Hence, we know that the conditional tail probability is

At the boundaries ρ = 1 , ρ = -1 , one can verify Pr ( | X | > | Y | ) = 0 . This concludes the proof.

E.1 Proof of Lemma 4.2

Proof. By Lemma E.1 and independence, we know that Pr (max i =1 ,...,p | X | > | Y | ) reaches its maximum when X i is independent of Y , i.e., when ρ ( X i , Y ) = 0 , ∀ i = 1 , ..., p . Since | X i | follows a half-normal distribution with cdf being erf( x √ 2 σ x ) , we have

2

and probability density function g ( x ) = 2 p [Φ( x σ x ) -1] p -1 1 √ 2 πσ x e -x 2 σ 2 x . When Y is independent of all X i 's (which gives the upper bound), we have

with a proper change of variables. This gives an upper bound as shown above.

E.2 Proof of Proposition 4.3

Proof. Consider a single Gaussian projection vector w with iid N (0 , 1) entries. Since w T u = ∑ p i =1 u i w i and each w i ∼ N (0 , 1) , we know that ( βw i x ) ∼ N ( β 2 ρ i β ∥ u ∥ ρ i β ∥ u ∥ ∥ u ∥ 2 ) where ρ i = u i ∥ u ∥ is the correlation coefficient. Since | w T ( u -u ′ ) | ≤ β max i =1 ,...,p | w i | by Definition 2.2 of β -neighboring (and more generally, when ∥ u -u ′ ∥ 1 ≤ β ), we have

̸

Note that, β max i =1 ,...,p | w i | ≥ | w T u | is a necessary condition for the event that there exists a neighbor such that sign ( w T u ) = sign ( w T u ′ ) . Denote I = ✶ { β max i =1 ,...,p | w i | ≥ | w T u |} . Applying Lemma 4.2 with r = β/ ∥ u ∥ ≤ 1 yields as given by (5). Let I j be the corresponding indicator function w.r.t. each column in the projection matrix W . Denote N + = ∑ k j =1 I j , and by the above reasoning, we know that | S | ≤ N + where S is defined in the theorem. Since the columns of W are independent, N + follows a Binomial ( k, E [ I ]) distribution with k trials and success probability E [ I ] bounded as above. Therefore, with probability 1 -δ , we have N + ≤ B -1 ( δ ; F ∥ u ∥ ,p , k ) , where B -1 is the inverse cdf of the Binomial distribution. This proves the claim as desired.

E.3 Proof of Theorem 4.4

Proof. Let s = sign ( W T u ) ∈ {-1 , +1 } k , s ′ = sign ( W T u ′ ) ∈ {-1 , +1 } k . We denote the collision probability of non-private SignRP as

Hence, the collision probability of DP-SignRP-RR can be computed as

̸

which increases linearly in P SRP . Thus, it holds that

which implies E [ ˆ θ RR ] = π ( 1 -(1 -θ π ) ) = θ . To compute the variance, we first estimate θ = cos -1 ( ρ ) by

Then according to the Central Limit Theorem (CLT), for the sample mean of iid Bernoulli's, as k →∞ , we have

As a result, we have ˆ θ → N ( θ, V RR k ) , where

We conclude the proof by replacing ϵ ′ = ϵ/N + .

E.4 Proof of Theorem 4.5

Proof. Let us consider a single projection vector w j = W [: ,j ] . Denote x j = w T j u and x ′ j = w T j u ′ for a neighboring data u ′ of u , and s j = sign ( x j ) , s ′ j = sign ( x ′ j ) . Also, let L j = ⌈ | x j | β max i =1 ,...,p | W ij | ⌉ and L ′ j = ⌈ | x ′ j | β max i =1 ,...,p | W ij | ⌉ . W.l.o.g., we can assume s j = 1 by the symmetry of random projection and the symmetry of DP. Consider two cases:

Case I: L j ≥ 2 . In this case, we know that s ′ j = s j , i.e., the change from u to u ′ will not change the sign of the projection. Thus, in Algorithm 4, we have

By the definition of β -adjacency, | L j -L ′ j | equals either 0 or 1 . When L j = L ′ j , Pr (˜ s j =1) Pr (˜ s ′ j =1) = 1 . When L j -L ′ j = 1 , we have

Hence, we have 1 ≤ Pr (˜ s j =1) Pr (˜ s ′ j =1) ≤ e ϵ k by the numeric identity 1 ≤ a + c b + c ≤ a b for a ≥ b > 0 and c > 0 . Thus, by symmetry, e -ϵ k ≤ Pr (˜ s j =1) Pr (˜ s ′ j =1) ≤ e ϵ k . On the other hand,

Similarly, when L j = L ′ j , the ratio equals 1 . When L j = L ′ j -1 , we have Pr (˜ s j = -1) Pr (˜ s ′ j = -1) ≤ exp( L ′ j k ϵ -L j k ϵ ) = e ϵ k . By symmetry we obtain e -ϵ k ≤ Pr (˜ s j = -1) Pr (˜ s ′ j = -1) ≤ e ϵ k .

Case II: L j = 1 . In this case, s j might be different from s ′ j . First, if L ′ j = 2 , then the above analysis also applies that Pr (˜ s j =1) Pr (˜ s ′ j =1) and Pr (˜ s j = -1) Pr (˜ s ′ j = -1) are both lower and upper bounded by e -ϵ k and e ϵ k , respectively. It suffices to examine the case when L ′ j = 1 . In this case, if s ′ j = s j = 1 then the probability ratios simply equal 1. If s ′ j = -1 , we have

Combining two cases, we have that log Pr (˜ s j = t ) Pr (˜ s ′ j = t ) ≤ ϵ k , for t = -1 , 1 , and for all j = 1 , ..., k . That is, each single perturbed sign achieves ϵ k -DP. Since the k projections are independent, by Theorem 2.1, we know that the output bit vector ˜ s = [˜ s 1 , ..., ˜ s k ] is ϵ -DP as claimed.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a series of differentially private (DP) Random Projection and Signed Random Projection algorithms. By using randomized responses and the idea of smooth sensitivity, the new proposed algorithms are shown to outperform previous results while still maintaining the same amount of privacy guarantee.\n\nSTRENGTHS:\n- The idea of using the exact tail probabilities of Gaussian to compute the necessary noise level (Theorem 3.2) is pretty interesting. This might imply that we can improve the empirical performance of most Gaussian Mechanism-based algorithms by using this simple fix.\n\n- \"Local perturbation\" is also an interesting idea to investigate since a lot of the time, we only need to maintain the privacy for specific sets of datasets rather than for any datasets in the data universe.\n\n- The appendix has plenty of good and thorough discussions that improve the readability and completeness of the paper.\n\n- The proofs and analyses are fairly well-organized and well-written.\n\nWEAKNESSES:\n- Seems like we do not have a utility guarantee for the smooth flipping probability algorithms? Since this is the main focus of the paper, the result would be stronger if the author can show that the algorithm performs well both in practice and in theory.\n\n- It would be interesting to see if the new DP-random projection algorithms can perform well in deep learning tasks. For example, if we use the DP-random projection to quantize the gradient, how does that affect our results? I feel like this would better demonstrate the power of the projection algorithms.\n\n- The runtime should probably be reported in the experiments section since it's a fairly important aspect of these types of algorithms.\n\nQUESTIONS:\n- How does the author numerically evaluate $\\sigma$ in Theorem 3.2? Do we have a closed form solution for that?\n\n- I'm also curious how the author compute $N_{+}$ in practice since it looks like a pretty complicated bound?\n\n- Since Proposition 4.4 suggests that $N_{+}$ is smaller when the data norm is large compared to $\\beta$, can we just scale up the data in practice?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors study the important problem of preserving privacy in matrix data. They study a family of fundamental operations on data which is Random Projection. This is useful for dimensionality reduction, NN search etc. Based on RP one can also obtained signed RP (SRP) which are useful (as in  SimHash) to find near duplicates and estimate similarity of vectors, etc. The authors present DP algorithms for this family of applications (RP, SRP). Their algorithm improve over prior work ([47]) showing methods of reducing the amount of noise using a series of techniques including smooth sensitivity and better upper bounds on sensitivity. Their experiments show improved results.
All in all this is an interesting paper on improving the accuracy of an important building block in data analysis with privacy.

STRENGTHS:
+ Important problem : computing with privacy random projection based sketches
+ Improved empirical results for a practical algorithm 
+ theoretically sound and non-trivial results

WEAKNESSES:
- It would be interesting to know how much head room is there if the projection matrix itself is kept secrete. There is no comparison with methods that omit the matrix W.

QUESTIONS:
1) Can you discuss more the comparison with methods like Blocki et al that keep W secrete (or even provide empirical results for comparing which such methods)? Is it possible to improve the results significantly?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
The paper introduces a series of differential privacy (DP) random projections (RP)  algorithms based on the robustness of the sign flipping probability in RP. This approach is promising as it seeks to enhance the existing methods in both theoretical guarantees and empirical performance.

STRENGTHS:
1. The analysis of the probability of smooth sign flipping is a novel contribution, offering a fresh perspective for DP RP algorithms.

2. The presentation of the paper is generally clear, providing a concise explanation of the methodology and algorithms developed. The authors effectively explain the motivation behind their research and the rationale for leveraging sign flipping probability.

3. The experimental evaluation of the proposed algorithms is adequate, highlighting the improvements achieved by DP-SignOPORP and iDP-SignRP over existing algorithms in the standard differential privacy setting. The comparison with other methods in the literature provides valuable insights into the performance of the proposed algorithms.

WEAKNESSES:
NA

QUESTIONS:
Could you provide more intuition on the scale of the smoothness in terms of sign flipping probability?

LIMITATIONS:
The authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of computing differentially private sketches of high dimensional vectors via random projections. The contributions are:\n1. Algorithm DP-RP-G-OPT: An algorithm that uses the Gaussian mechanism to privatize random projections using Gaussian and Rademacher matrices, but adds more carefully calibrated noise by using the CDF of the Gaussian distribution as opposed to the usual tail bounds. It outputs a k-dimensional DP sketch of a p dimensional input vector, it seems to be working in the local DP setting as it operates on a single data vector.\n2. Algorithm DP-OPORP: An algorithm that appends the OPORP mechanism of Li and Li ([53] in the paper) with the Gaussian mechanism to produce a DP variant of the former. It produces a k dimensional sketch of the p dimensional input vector but uses only $p$-many multiplication operations instead of $kp$-many.\n3. Algorithm DP-SignRP: An algorithm that produces a 1-bit sketch of the input vector in a differentially private way. The algorithm only accepts input vectors with some lower bound $m$ on the norm which must be passed as an argument. They also use this algorithm to produce a DP sign estimator.\n4. Algorithms DP-SignRP-RR and DP-SignRP-RR-Smooth: The authors aim to produce an improved version of DP-SignRP by leveraging the stability of the sign of the projected output under small changes in the input. They also use the binning of OPORP to reduce the sensitivity of the algorithm to the input so as to be able to add less privatizing noise.\n\nThe authors go on to test their methods empirically on similarity search and classification problems.\n\nSTRENGTHS:\n1. The authors want to leverage the stability of the sign operator under random projects to boost the privacy guarantee over what one might get naively, in some sense they want to appeal to the local sensitivity of the mapping when the input vector is far away from the decision boundary. This idea may be fruitful.\n2. The authors conduct some experiments to test their methods.\n\nWEAKNESSES:\n1. The privacy model here is not clear - the authors define and talk about differential privacy in the central model (where the stability guarantee holds over adjacent data sets) but their algorithms take as input only one data point. It seems that instead all the guarantees are meant to hold in the non-interactive local model of differential privacy.\n2. The methods DP-RP-G-OPT and DP-OPORP are not very novel - the first achieves no asymptotic improvements over prior work and seems to improve the performance by some unspecified constant. The second simply appends the OPORP algorithm from prior work by the Gaussian mechanism.\n3. The significance DP-SignRP-RR is not clear - there is a lower bound $m$ on the input vector.\n4. The quality of the writing needs to be improved. The most novel contribution seems to be DP-Sign-OPORP-smooth-RR which is relegated to the end.\n\nQUESTIONS:\n1. Can you please elaborate on the model of differential privacy that you are operating in? What is the significance of this for the CountSketch application that you describe in the beginning?\n2. Why do we need to assume a lower bound on the norm of the input vector for DP-SignRP-RR? Does this impact its applicability/parameter regimes? How should we set the value of $m$? In particular it seems like $m<\\sqrt{p}$ is necessary to avoid the domain becoming empty.\n\nLIMITATIONS:\nThis is a theoretical work and I do not feel that potential societal impact needs to be discussed here.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}