Title: On Computing Pairwise Statistics with Local Differential Privacy
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

local differential privacy

pairwise statistics
Abstract: We study the problem of computing pairwise statistics, i.e., ones of the form $\binom{n}{2}^{-1} \sum_{i \ne j} f(x_i, x_j)$, where $x_i$ denotes the input to the $i$th user, with differential privacy (DP) in the local model. This formulation captures important metrics such as Kendall's $\tau$ coefficient, Area Under Curve, Gini's mean difference, Gini's entropy, etc. We give several novel and generic algorithms for the problem, leveraging techniques from DP algorithms for linear queries.

On Computing Pairwise Statistics with Local Differential Privacy

Abstract

We study the problem of computing pairwise statistics, i.e., ones of the form ( n 2 ) -1 ∑ i = j f ( x i , x j ) , where x i denotes the input to the i th user, with differential privacy (DP) in the local model. This formulation captures important metrics such as Kendall's τ coefficient, Area Under Curve, Gini's mean difference, Gini's entropy, etc. We give several novel and generic algorithms for the problem, leveraging techniques from DP algorithms for linear queries.

1 Introduction

Differential privacy (DP) [DMNS06] is a widely studied and used notion for quantifying the privacy protections of algorithms in data analytics and machine learning. It has witnessed many practical deployments [EPK14, Sha14, Gre16, App17, DKY17, Abo18, RE19, TM20, KT18, RSP + 21]. On a high level, DP dictates that the output of a (randomized) algorithm remains statistically indistinguishable if the input of any single user is modified; the degree of statistical indistinguishability is quantified by the privacy parameter ε > 0 .

An important setting of DP is the local model [EGS03, KLN + 11], where each of n users holds an input that they wish to keep private. An analyst wishes to compute some known function of the users inputs. The analyst and the users engage in an interactive protocol at the end of which the analyst is supposed to compute an estimate to the value of the function on the user inputs. When the aforementioned statistical indistinguishability property is enforced on the algorithm's transcript, the algorithm is said to be ε -local DP (see Section 2 for a formal definition). Non-interactive algorithms refer to the setting where each user sends a single (DP) message to the analyst, who is then supposed to output an estimate of the desired value without any further interaction with the users. As usual in the interactive local DP setting, we assume a broadcast model where every communication is visible to all users and to the analyst (and subject to the DP constraint). The number of rounds of an interactive protocol refers to the number of back-and-forths between the set of users and the analyst.

While the local DP setting offers a compelling trust model compared to the central DP model (where an analyst is assumed to have access to the raw user data and the privacy guarantee is only enforced at its output), the local setting is often limited by lower bounds on the error incurred by private protocols (e.g., [BNO08, CSS12]). In this work, we study some basic tasks in analytics and learning, and give local DP protocols with significantly smaller error than what was previously known.

Quadratic Form Computation. Throughout the paper, we consider the following task:

Definition 1 (Quadratic Form Computation) . Given a matrix W ∈ R k × k . Each user i receives x i ∈ [ k ] . The goal is to compute the quadratic form h T x Wh x where h x ∈ Z k ≥ 0 denotes the normalized histogram of the input, i.e., ( h x ) b := 1 n · |{ i ∈ [ n ] | x i = b }| .

An algorithm computing quadratic forms immediately implies an algorithm for computing pairwise statistics (also known as U -statistics of degree 2 ) defined as F ( X ) = 1 ( n 2 ) ∑ 1 ≤ i<j ≤ n f ( x i , x j ) , where f : X 2 → R is a symmetric function (called kernel ), and x i is the input to the i th user. The family of pairwise statistics is notable in that it contains several statistical quantities that are widely used in different areas, including the Area Under the Curve (AUC), Kendall's τ coefficient, the Gini's mean difference, and the Gini's diversity index (aka Gini-Simpson index or Gini's impurity). Computing quadratic forms has been studied in the context of local DP by Bell et al. [BBGK20] who gave algorithms for functions f that are Lipschitz and for estimating the AUC of a classifier; the latter was improved recently [CM23]. While these problems are simple in central DP (as we can directly add noise to the function value), we note that a clear challenge for computing pairwise statistics (and hence quadratic forms) in local DP is that each summand depends on data points held by two different users.

Linear Queries. Computation of quadratic forms is a natural 'degree-2' variant of the so-called linear queries (defined below), a well-studied problem in the DP literature.

Definition 2 (Linear Queries) . Given a workload matrix W ∈ R k × k , the goal is to compute Wh x , where h x ∈ Z k ≥ 0 denotes the normalized histogram of the input.

There is a long line of work on privately answering linear queries. In both the central and local models, the optimal errors achievable by DP algorithms are (mostly) well-understood for various types of errors, such as the ℓ 2 2 -error or the ℓ ∞ -error [HT10, BDKT12, LMH + 15, NTZ16, BUV18, BBNS19, ENU20, Nik23]. Recall that the MSE of an estimate ˆ z of a real value z is defined as MSE(ˆ z, z ) := E ˆ z [(ˆ z -z ) 2 ] . For our purpose, we consider the following natural measure of accuracy:

Definition 3 (mMSE) . Given a mechanism M that answers linear queries for a workload matrix W ∈ R k × k , its maximum mean square error (mMSE) is defined as mMSE( M ; W ) := max x ∈ [ k ] n max j ∈ [ k ] MSE( M ( x ) j , ( Wh x ) j ) .

In a recent seminal work, Edmonds et al. [ENU20] provide a nearly optimal characterization of the error achievable by non-interactive ε -local DP algorithms. To describe their result, we need some additional definitions of norms on matrices and related quantities:

▷ 1 → 2 norm: For any matrix A , ∥ A ∥ 1 → 2 is the maximum ( ℓ 2 -)norm of its columns.

▷ ℓ ∞ -norm: For any matrix A , ∥ A ∥ ∞ is the maximum absolute value among its entries.

▷ Factorization norm: For W ∈ R k × k , let γ 2 ( W ) := min L T R = W ∥ L ∥ 1 → 2 ∥ R ∥ 1 → 2 .

▷ Approximate-Factorization norm 1 : For W ∈ R k × k and α ≥ 0 , let γ 2 ( W ; α ) := min ∥ ˜ W -W ∥ ∞ ≤ α γ 2 ( ˜ W ) .

Finally, we define ζ ( W,n ) := min α ≥ 0 ( γ 2 ( W ; α ) + αε √ n ) . (Note that ζ ( W,n ) ≤ γ 2 ( W ) as we can pick α = 0 .) The result of Edmonds et al. [ENU20] states that this quantity, up to polylogarithmic factors, governs the best error achievable for linear queries in the non-interactive local DP model 2 :

Theorem 4 ([ENU20]) . For any workload matrix W , there is a non-interactive ε -local DP mechanism for linear queries with mMSE at most O ( ζ ( W,n ) 2 ε 2 n ) . Furthermore, any non-interactive ε -local DP mechanism must incur mMSE at least ˜ Ω ( ζ ( W,n ) 2 ε 2 n ) .

Given such a tight and generic characterization for linear queries, it is natural to ask:

Can we characterize the error of computing the quadratic form for any matrix W with local DP?

1.1 Our Contributions

In this work, we answer this question by establishing connections between the problems of computing quadratic forms and linear queries. Leveraging the wealth of knowledge on the latter, we obtain several new (and general) upper and lower bounds for the former.

Non-Interactive Local DP. First, in the non-interactive setting, we give the following upper and lower bounds. Together, they show that the error one can expect for computing quadratic forms is essentially the same as that for computing linear queries (as provided in Theorem 4).

Theorem 5 (Non-interactive Algorithm) . For any W ∈ R k × k , there is a non-interactive ε -local DP mechanism for estimating quadratic forms on W with MSE at most O ( ζ ( W,n ) 2 (log k ) ε 2 n ) .

Theorem 6 (Non-interactive Lower Bound) . For any symmetric W ∈ R k × k , any non-interactive ε -local DP mechanism for estimating quadratic forms on W must incur MSE at least ˜ Ω ( ζ ( W,n ) 2 ε 2 n ) .

Our results above are generic and can be applied to any pairwise statistics. To demonstrate the power of the algorithm, we now state implications for several classes of statistics that have been studied in the privacy literature. Due to space constraints, we defer their definitions to Section 5.

Corollary 7. There is a non-interactive ε -local DP mechanism for computing pairwise statistics for:

▷ Any G -Lipschitz function f : [0 , 1] → R , with MSE O ( G 2 log n ε 2 n ) ,

▷ Kendall's τ coefficient, with MSE O ( (log k ) 5 ε 2 n ) ,

▷ AUC under the balancedness assumption 3 , with MSE O ( (log k ) 3 ε 2 n ) ,

▷ Gini's diversity index, with MSE O ( log k ε 2 n ) .

For O (1) -Lipschitz functions, which includes several well-known metrics such as Gini mean difference, we improve upon the algorithm of Bell et al. [BBGK20] whose MSE is O ( 1 ε √ n ) . We are not aware of any previous bounds on Kendall's τ coefficient before; the metric was mentioned in [BBGK20] without any error guarantee given. For AUC, our bound is the same as in [BBGK20] but worse than a follow-up work [CM23] by a log k factor; nonetheless, we stress that our result is derived as a corollary of a generic algorithm without relying too deeply on AUC (except for the γ 2 -norm of its matrix). For Gini's diversity index, our bound is again worse than that from Bravo et al. [BHBFG + 22] by log k factor, but their algorithm requires the use of public randomness; we do not use any public randomness. (Throughout this work, when we refer to the local model without further specification, we assume no public randomness, aka private-coin model.)

Remark. Since our non-interactive algorithm only requires a vector-summation primitive, we can also apply protocols for vector summation in the shuffle model (e.g., [CSU + 19, BBGN20, GMPV20]) to obtain an ( ε, δ ) -DP protocol in the shuffle model, reducing the MSE by a factor of n for each setting in Theorem 5 and Corollary 7.

Interactive Local DP. Finally, we also provide an interactive algorithm whose MSE does not depend on γ 2 ( W ) , as long as n is sufficiently large:

Theorem 8 (Interactive Algorithm) . For any W ∈ R k × k , there is a three-round ε -DP algorithm for estimating quadratic forms on W such that, for n ≥ ( γ 2 ( W ) · log k ∥ W ∥ ∞ · ε ) O (1) , the MSE is O ( ∥ W ∥ 2 ∞ ε 2 n ) .

We note that the dependence O ( ∥ W ∥ 2 ∞ ε 2 n ) is the best possible: even when the k = 2 and W is binary, the problem is as hard as binary summation, which is known to require MSE at least Ω ( 1 ε 2 n ) even for an arbitrary number of rounds of interactions [CSS12], and we can rescale this hard instance to get any desired ℓ ∞ -norm.

Combining the non-interactive lower bound in Theorem 6 and the interactive upper bound in Theorem 8, our results imply that there exists a matrix W ∈ R k × k , ε > 0 , n ∈ N such that interactive algorithms are provably more accurate than non-interactive ones for privately computing the quadratic form on W . This places computing pairwise statistics as one of the few problems (and perhaps the most natural) that are known to separate interactive and non-interactive local DP. Due to space constraints, we defer further discussion of this to the Appendix.

1.2 Technical Overview

We now give a brief technical overview of our proofs.

Black-Box Reduction from Linear Queries (Lower Bound). As mentioned earlier, our results are shown through connections between computing quadratic forms and linear queries. We will start with black-box reductions between the two. Suppose that we have a non-interactive ε -local DP algorithm A for computing the quadratic form on W . We will construct a non-interactive ε -local DP algorithm A ′ for computing linear queries of W . This will allow us to establish our lower bound (Theorem 6) as a consequence of the lower bound for linear queries (Theorem 4).

For simplicity of this overview, instead of considering the full quadratic form, suppose that A works on 2 n users with inputs x 1 , . . . , x n , y 1 , . . . , y n and computes an estimate of h T y Wh x . On input x 1 , . . . , x n , algorithm A works as follows:

▷ Each user i runs the randomizer of A on x i to get a response o i and sends it to the analyst.

▷ For each j ∈ [ k ] , the analyst simulates running the randomizer of A on y 1 = · · · = y n = j to get responses o ′ 1 , . . . , o ′ n . Then, the analyst lets ˆ z j be the estimator of A based on the responses o 1 , . . . , o n , o ′ 1 , . . . , o ′ n .

▷ The analyst then outputs (ˆ z 1 , . . . , ˆ z k ) as its estimate.

In other words, the randomized responses from A are used as an 'oracle' for A ′ to compute the different linear queries. The key observation here is that, when we set y 1 = · · · = y n = j , A produces an estimate for h T y Wh x = 1 T j Wh x = ( Wh x ) j as desired. Note that this reduction only works in the non-interactive setting: if we were in the interactive setting, the responses on x 1 , . . . , x n (of protocol A ) would have been dependent on those of y 1 , . . . , y n . Therefore, the first step of the reduction would have been impossible.

While this encapsulates the high-level ideas of our proof, there are some details that needs to be handled. E.g., if A outputs the quadratic form h T x ∪ y Wh x ∪ y , there are 'cross terms' of the form h T x Wh x that need to be removed. We formally describe and analyze the full reduction in Section 4.

Black-Box Reduction to Linear Queries (Algorithm). We can also give a reduction in the reverse direction, although this results in an additional round of communication. Specifically, let A ′ be a non-interactive ε -local DP algorithm for computing linear queries of W . We can construct a two-round (2 ε ) -local DP algorithm A for computing the quadratic form on W as follows.

▷ First Round: Run A ′ on all users to compute an estimate (ˆ z 1 , . . . , ˆ z k ) for Wh x .

▷ Second Round: Each user j ∈ [ n ] , sends o j = ˆ z x j + κ j to the analyst where κ j is (appropriately calibrated) Laplace noise. The analyst then outputs 1 n ( o 1 + · · · + o n ) .

Again, we omit some details for simplicity, such as the fact that each ˆ z j may not be bounded a priori, which may make the second step violate DP. However, these are relatively straightforward to handle.

If there were no noise, we would have o j = 1 T x j Wh x and thus 1 n ( o 1 + · · · + o n ) = h T x Wh x as desired. Furthermore, it is not hard to see that the error from κ 1 , . . . , κ n is dominated by the error from A ′ in the first step. In other words, we get an error similar to the one in Theorem 5 here, but the protocol is interactive (two-round). Making the protocol non-interactive requires us to step away from the black-box approach and open up the linear query algorithm (Theorem 4).

White-Box Algorithms. For simplicity, in this section we describe protocols with accuracy that depends on γ 2 ( W ) , which can be larger than ζ ( W,n ) . The desired error dependence on ζ ( W,n ) can be obtained from this bound via a reduction, as shown in Lemma 10.

To understand our algorithm, we first describe the matrix mechanism for linear queries (cf. [ENU20]). That algorithm works as follows: factorize W = L T R . Then, user i sends o R i = R 1 x i + z R i

where z R i is a appropriately selected random noise. In other words, each user privatizes W 1 x i and sends it to the analyst. Finally, the analyst outputs L T ( 1 n ( o R 1 + · · · + o R n ) ) . This is exactly equal to Wh x + L T Z R where Z R := 1 n ( z R 1 + · · · + z R n ) . It is possible to select the noise in such a way that Z R is ( ∥ R ∥ 1 → 2 / ε √ n ) -sub-Gaussian. This leads to an mMSE of O ( γ 2 ( W ) 2 ε 2 n ) .

This suggests a natural approach for quadratic forms: in addition to sending (a privatized version of) R 1 x i , the user sends a privatized version of L 1 x i , i.e., o L i = L 1 x i + z L i where z L i is a random noise, to the analyst. The analyst then outputs 〈 1 n ( o L 1 + · · · + o L n ) , 1 n ( o R 1 + · · · + o R n ) 〉 . Letting Z L := 1 n ( z L 1 + · · · + z L n ) , the output can be written as h T x Wh x + 〈 Lh x , Z R 〉 + 〈 Rh x , Z L 〉 + 〈 Z L , Z R 〉 . There are three error terms: 〈 Lh x , Z R 〉 , 〈 Rh x , Z L 〉 , and 〈 Z L , Z R 〉 . Similar to linear queries analysis, it is not hard to see that the first two terms contribute O ( γ 2 ( W ) 2 ε 2 n ) to the MSE . Unfortunately, the last term is problematic for us: a simple calculation shows that it contributes O ( ℓ ε 4 n 2 ) to the MSE , where ℓ denotes the number of rows of L, R . A priori, this term can be quite large as there is no obvious bound on ℓ . In fact, if W is full rank (which is the case for most popular pairwise statistics), then we know that ℓ must be at least k . This leads to an undesired error term O ( k ε 4 n 2 ) , which dominates the first term for small-to-moderate values of n , i.e., when n ≤ k ε 2 γ 2 ( W ) 2 .

To overcome this, we observe that, if we only look for approximate factorization (in the same sense as γ 2 ( W ; α ) defined above), then it is always possible to reduce ℓ via dimensionality reduction techniques (e.g., [DG03]). Namely, we may pick a (e.g., random Gaussian) matrix A and replace L, R with AL,AR respectively. Selecting the number of rows of A appropriately then yields Theorem 5.

Additional Interactive Algorithms. To describe our algorithm, let us start by instantiating the above black-box two-round reduction using the matrix mechanism. In this context, the reduction yields the following algorithm:

▷ First Round: Each user i sends o L i = L 1 x i + z L i to the analyst, where z L i is appropriately selected random noise. The analyst then broadcasts O L = 1 n ( o L 1 + · · · + o L n ) to each user.

▷ Second Round: Each user j ∈ [ n ] sends o j = 〈 O L , R 1 x j 〉 + κ j to the analyst, where κ j is appropriately calibrated Laplace noise. The analyst then outputs 1 n ( o 1 + · · · + o n ) .

It is not hard to see that the κ j noise terms together contribute at most O ( 1 / ε 2 n ) to the MSE . Therefore, the main noise comes from the first step. Similar to the previous discussion, this noise can be written as 〈 Rh x , Z L 〉 where Z L := 1 n ( z L 1 + · · · + z L n ) . The contribution of this noise to the MSE is then O ( γ 2 ( W ) 2 ε 2 n ) . The γ 2 2 ( W ) term shows up in the error because ∥ Rh x ∥ 2 can be as large as ∥ R ∥ 1 → 2 . The idea motivating our improvement is simple: Can we replace the Rh x term with a term that is much smaller?

This brings us to the following strategy. We will use an additional round at the beginning to compute a rough estimate µ of Rh x . Using the so-called projection mechanism [BBNS19] 4 , it is possible to compute µ such that ∥ Rh x -µ ∥ ≤ ( γ 2 ( W ) · (log k ) /ε ) O (1) n Ω(1) . The subspace orthogonal to µ can be processed in a similar manner as before, but now the error term will just be 〈 Rh x -µ, Z L 〉 . Since ∥ Rh x -µ ∥ is now much smaller (approaches 0 as n →∞ ), this gives us the improved error bound. We note that the direction of µ can be handled by having each user i directly send 〈 o L i , µ 〉 plus an appropriately calibrated noise. This summarizes the high-level idea of our approach.

Due to space constraints, we focus on the non-interactive algorithms in the main body and defer the proof of the interactive algorithm to the Appendix.

2 Preliminaries

Differential Privacy. For ε ≥ 0 , an algorithm M is ϵ -DP if for every pair X,X ′ of inputs that differ on one user's input and for every possible output o , Pr[ M ( X ) = o ] ≤ e ε · Pr[ M ( X ′ ) = o ] .

An algorithm A in the local DP model consists of a randomizer R and an analyst, computed as A ( X ) = analyst( R ( x 1 ) , . . . , R ( x n )) for input X = { x 1 , . . . , x n } . A is said to be (non-interactive) ε -local DP if R is ε -DP.

A real-valued random variable Z is σ -sub-Gaussian iff E [exp( Z 2 /σ 2 )] ≤ 2 . A R d -valued random variable Z is σ -sub-Gaussian iff ⟨ θ, Z ⟩ is σ -sub-Gaussian for all unit vectors θ ∈ R d .

Theorem 9 ([BBNS19]) . For any C, ε > 0 , there is a (non-interactive) ε -local DP algorithm VRand ε,C that takes in x ∈ R d such that ∥ x ∥ 2 ≤ C and outputs Y ∈ R d such that E [ Y ] = x and Y -x is σ -sub-Gaussian for σ = O ( C/ε ) .

Error: Factorization vs Approximate-Factorization. We show that it suffices to give errors in terms of γ 2 ( W ) instead of ζ ( W ; n ) ; this will be convenient for our subsequent proofs. Due to space constraints, the proof of the following statement is deferred to Appendix A.

Lemma 10. Suppose that, for all W ∈ R k × k , there is a non-interactive ε -local DP protocol A for quadratic form on W with MSE O ( c ( n, ε, k ) · γ 2 ( W ) 2 ε 2 n ) where c ( n, ε, k ) ≥ Ω(1) . Then there is also a non-interactive ε -local DP protocol A ′ with MSE O ( c ( n, ε, k ) · ζ ( W ; n ) 2 ε 2 n ) .

3 Non-Interactive Algorithm

In this section, we prove Theorem 5. To do so, let us start by defining (approximate) rank-restricted factorization norms 5 , which are the same as γ 2 ( W ) , γ 2 ( W ; α ) except we now restrict the number of rows of L, R to be at most ℓ :

▷ For W ∈ R k × k , ℓ ∈ N , let γ ℓ 2 ( W ) := min L T R = W ; L,R ∈ R ℓ × k ∥ L ∥ 1 → 2 ∥ R ∥ 1 → 2 .

▷ For W ∈ R k × k , ℓ ∈ N and α ≥ 0 , let γ ℓ 2 ( W ; α ) := min ∥ ˜ W -W ∥ ∞ ≤ α γ ℓ 2 ( ˜ W ) .

We can now use the approximate rank-restricted factorization to perform the algorithm as outlined in Section 1.2 with an error term that depends on ℓ (and α ):

Lemma 11. For any W ∈ R k × k , ℓ ∈ N , and α ≥ 0 , there is a non-interactive ε -local DP algorithm that estimates the quadratic form on W to within an MSE of O ( α 2 + γ ℓ 2 ( W ; α ) 2 · ( 1 ε 2 n + ℓ ε 4 n 2 )) .

Proof. By definition of γ ℓ 2 ( W ; α ) , there exists ˜ W ∈ R k × k , L, R ∈ R ℓ × k such that ∥ ˜ W -W ∥ ∞ ≤ α and ˜ W = L T R where, w.l.o.g., by rescaling, ∥ L ∥ 1 → 2 = ∥ R ∥ 1 → 2 = √ γ ℓ 2 ( W ) =: C .

Algorithm Description. Let ¯ ε = ε/ 2 . The algorithm works as follows:

▷ Each user i ∈ [ n ] sends y L i ← VRand ¯ ε,C ( L 1 x i ) and y R i ← VRand ¯ ε,C ( R 1 x i ) to the analyst (where VRand · ( · ) is from Theorem 9).

As each user runs an ( ε/ 2) -DP randomizer twice, the algorithm is ε -DP.

Utility Analysis. Let z L i = y L i -L 1 x i and z R i = y R i -R 1 x i . From Theorem 9, z L i , z R i are zero-mean and σ -sub-Gaussian for σ = O ( C/ε ) . Let Z L := 1 n ∑ i ∈ [ n ] z L i , Z R := 1 n ∑ i ∈ [ n ] z R i ; we then have that these are zero-mean and σ ′ -sub-Gaussian for σ ′ = σ/ √ n . The MSE of the protocol is given by

Rank-Restricted Approximate Factorization via JL. We next show that w.l.o.g. we can take ℓ to be quite small in Lemma 11 above.

As mentioned earlier, this lemma is proved by applying Johnson-Lindenstrauss (JL) dimensionality reduction to each column of L, R . We summarize a simplified version of the JL lemma below.

Lemma13 (Johnson-Lindenstrauss Lemma (e.g., [DG03])) . Let β ∈ (0 , 1) and U = { u 1 , . . . u m } ⊆ R d . For some ℓ ≤ O ( β -2 log m ) , there exists a matrix A ∈ R ℓ × d such that for all u, v ∈ U , we have (1 -β ) ∥ u -v ∥ 2 2 ≤ ∥ Au -Av ∥ 2 2 ≤ (1 + β ) ∥ u -v ∥ 2 2 .

We are now ready to prove Lemma 12.

Proof of Lemma 12. By definition of γ 2 ( W ) , there exists L, R ∈ R d × k for some d ∈ N such that W = L T R where ∥ L ∥ 1 → 2 = ∥ R ∥ 1 → 2 = √ γ 2 ( W ) =: C .

Let L 1 , . . . , L k (resp. R 1 , . . . , R k ) be the columns of L (resp. R ). Consider U = { 0 , L 1 , . . . , L k , R 1 , . . . , R k , -L 1 , . . . , -L k , -R 1 , . . . , -R k } and β = 0 . 5 α/C 2 ; let ℓ = O ( β -2 log k ) = O ( γ 2 ( W ) 2 log k α 2 ) and A ∈ R ℓ × d be as guaranteed by Lemma 13.

Let ˜ L = AL, ˜ R = AR , and ˜ W = ˜ L T ˜ R . For all i ∈ [ k ] , we have ∥ ˜ L i ∥ 2 2 ≤ (1 + β ) ∥ L i ∥ 2 2 and ∥ ˜ R i ∥ 2 2 ≤ (1 + β ) ∥ R i ∥ 2 2 . Therefore, ∥ ˜ L ∥ 1 → 2 , ∥ ˜ R ∥ 1 → 2 ≤ O ( C ) , i.e., γ ℓ 2 ( ˜ W ) ≤ O ( γ 2 ( W )) . Moreover, for each i, j ∈ [ k ] , we have

We end this section by proving Theorem 5, which is a simple combination of Lemma 11 and Lemma 12.

Proof of Theorem 5. Pick 6 α = γ 2 ( W ) ε √ n and apply Lemma 12: There exists ℓ = O ( γ 2 ( W ) 2 log k α 2 ) = O ( (log k ) ε 2 n ) such that γ ℓ 2 ( W ; α ) ≤ O ( γ 2 ( W )) . Plugging this back into Lemma 11 then gives us a non-interactive ε -local DP protocol with MSE

Applying Lemma 10 then concludes the proof.

4 Lower Bounds for Non-Interactive Algorithms

In this section we formalize the reduction from linear queries to computing quadratic forms, as outlined in Section 1.2. The properties of the reduction are stated in the theorem below.

Theorem 14. Let W ∈ R k × k be symmetric. Suppose that there is a non-interactive ε -local DP mechanism for computing the quadratic form on W with MSE at most α ( ε, n ) . Then there is a noninteractive ε -local DP protocol for computing the linear queries of W with mMSE O ( α ( ε/ 2 , n ) + α ( ε/ 2 , 2 n )) .

Theorem 14 and the lower bound in Theorem 4 immediately imply Theorem 6. In the proof below, we use subscripts ε, n to denote the privacy loss parameter and the number of users in the protocol.

Proof of Theorem 14. Let A ε,n be the ε -local DP protocol for computing the quadratic form on W , and let R ε,n denote its randomizer. We construct an algorithm A ′ ε,n for linear queries with workload matrix W . On input x 1 , . . . , x n , proceed as follows:

▷ Run the protocol of A ε/ 2 ,n on x 1 , . . . , x n to compute an estimate ˆ z for h T x A h x .

▷ In the same round as above, run the randomizer R ε/ 2 , 2 n of A ε/ 2 , 2 n on x 1 , . . . , x n to get the responses R ε/ 2 , 2 n ( x 1 ) , . . . , R ε/ 2 , 2 n ( x n ) .

▷ For each j ∈ [ k ] , do the following:

▷ Run the randomizer R of A on y 1 = · · · = y n = j to get the responses R ε/ 2 , 2 n ( y 1 ) , . . . , R ε/ 2 , 2 n ( y n ) .

▷ Compute the estimator z ′ j of A on the 2 n responses R ε/ 2 , 2 n ( x 1 ) , . . . , R ε/ 2 , 2 n ( x n ) , R ε/ 2 , 2 n ( y 1 ) , . . . , R ε/ 2 , 2 n ( y n ) .

▷ Set ˆ z j = 2 z ′ j -1 2 ˆ z -1 2 1 T j W 1 j .

▷ Output (ˆ z 1 , . . . , ˆ z j ) as the estimates for the linear queries.

Since ( ε/ 2) -local DP randomizers are run on each input x i twice, the basic composition theorem implies that this is a ε -local DP algorithm as desired.

For each j ∈ [ k ] , we now compute the MSE of ˆ z j . First, observe that

where y denotes the dataset with n copies of j .

Therefore, we can bound the MSE of ˆ z j as follows:

where the last inequality follows from the guarantees of A . Thus, the mMSE of A ′ is at most O ( α ( ε/ 2 , 2 n ) + α ( ε/ 2 , n )) , as desired.

5 Upper Bounds for Specific Metrics

In this section, we obtain concrete upper bounds for many well-known U-statistics of degree 2. For a kernel f : X → R , let W f ∈ R X×X denote the matrix defined by W f x,x ′ = f ( x, x ′ ) . Our proof of Corollary 7 proceeds by providing an upper bound on γ 2 ( W f ) for each U-statistic with kernel f ; the bounds immediately follow from Theorem 5. Similar to before, let k denote |X| .

The following (non-trivial) facts about the factorization norm are useful to keep in mind:

Fact 15. The factorization norm γ 2 satisfies the following properties:

[TJ89] For any A,B , we have γ 2 ( A + B ) ≤ γ 2 ( A ) + γ 2 ( B ) .

[LSS08] For any A,B , γ 2 ( A ⊗ B ) = γ 2 ( A ) · γ 2 ( B ) .

̸

Gini's diversity index. For X ⊆ R , the kernel f ( x, x ′ ) = 1 [ x = x ′ ] captures Gini's diversity index . From Fact 15(1), we have γ 2 ( W f ) ≤ γ 2 ( 1 k × k ) + γ 2 ( I k × k ) ≤ 1 + 1 where the inequality γ 2 ( 1 k × k ) ≤ 1 is from the factorization L = R = 1 k and γ 2 ( I k × k ) ≤ 1 is from L = R = I k × k .

Kendall's τ coefficient and AUC. For X = A × B ⊆ R 2 with x i = ( y i , z i ) , the kernel f (( y i , z i ) , ( y j , z j )) = sgn( y i -y j ) · sgn( z i -z j ) yields Kendall's τ coefficient . Let U m ∈ {-1 , 1 } m denote the matrix that has +1 on all entries above the main diagonal (inclusive) and -1 elsewhere. It is known that γ 2 ( U m ) = Θ(log m ) . Wecan bound γ 2 ( W f ) for Kendall's τ coefficient by observing that W f = U A ⊗ U B . From Fact 15(2), γ 2 ( W f ) ≤ γ 2 ( U A ) · γ 2 ( U B ) ≲ (log | A | ) (log | B | ) ≤ (log k ) 2 .

AUC for binary classification is defined in a similar manner as Kendall's tau coefficient, except that (i) B = { 0 , 1 } and (ii) the normalization constant being 1 n + n -instead if 1 ( n 2 ) where n + (resp., n -)

denotes the number of 1-labeled (resp., 0-labeled) examples. The AUC result follows from the above since | B | = 2 in this case. We remark that, for the AUC case, we also have to split the privacy budget and use half of it to estimate n + so that we can renormalize correctly. It is not hard to see that this renormalization procedure results in at most an additive factor of O ( 1 ε 2 n ) in the MSE , under the balancedness assumption that n -, n + ≥ Ω( n ) .

Lipschitz Losses. Let X = [0 , 1] and let f : X → R be any function such that | f ( x ) -f ( x ′ ) | ≤ G · | x -x ′ | ; we call f G -Lipschitz . This class includes U-statistics such as the Gini's mean difference , which is given by the kernel f ( x i , x j ) = | x i -x j | , which is 1-Lipschitz.

Similar to [BBGK20], we use a discrete case where X = [ k ] as a subroutine. Our algorithm for this is stated below. Note that Corollary 16 immediately implies the bound for the continuous case: given any function f : [0 , 1] → R , we may select k to be sufficiently large, e.g., k = Θ( εn 2 ) , and discretize the function over the points 1 /k, 2 /k, . . . , k/k . Defining g : [ k ] → R by g ( i ) = f ( i/k ) allows us to use Corollary 16 with Lipschitz constant G/k . This leads to a MSE of O ( G 2 log( εn 2 ) ε 2 n ) . The MSE resulting from the discretization error is then at most G 2 n 2 2 < G 2 2 .

k ε n

Corollary 16 (Discrete Lipschitz) . Assuming X = [ k ] and that f is G -Lipschitz. There is a noninteractive ε -local DP algorithm for computing pairwise statistics for f with MSE O ( G 2 k 2 (log k ) ε 2 n ) .

Again, the above corollary follows from Theorem 5 and the following factorization.

Lemma 17. Assuming that f : [ k ] → R is G -Lipschitz, then γ 2 ( W f ) ≤ O ( Gk ) .

Proof. We assume w.l.o.g. that k = 2 q -1 for some q ∈ N and ∥ W f ∥ ∞ ≤ Lk ; otherwise, we may shift W f (and the answer) without incurring any additional error. We arrange [ k ] into a balanced binary search tree T of depth q -1 naturally (where the root is 2 q -1 and the leaves are 1 , 3 , . . . , k ). Let P ( j ) denote the path from node j to the root (inclusive) in T , and let ℓ ( j ) denote the depth of j (where the root has depth 0). Furthermore, let parent( j ) denote the parent of j in T . For notational convenience, let parent(2 q -1 ) = ⊥ and let f ( i, ⊥ ) = 0 for all i ∈ [ k ] .

We construct L, R ∈ R k × k as follows.

For i, j ∈ [ k ] , we have ( L T R ) i,j = ∑ t ∈ P ( j ) ( f ( i, t ) -f ( i, parent( t ))) = f ( i, j ) . Thus, L T R = W .

Furthermore, ∥ R ∥ 2 1 → 2 = 1 + ( 5 6 ) 2 + · · · + ( 5 6 ) 2( q -1) ≲ 1 . Meanwhile, we can bound ∥ L ∥ 2 1 → 2 by

where the first inequality follows since f is G -Lipschitz. Thus γ 2 ( W f ) ≤ O ( Gk ) .

6 Conclusion and Open Questions

In this work, we systematically study the problem of privately computing pairwise statistics. We give a non-interactive local DP algorithm and a nearly-matching lower bound for the problem. Furthermore, we show that, for some metrics, improvements can be made if interaction is allowed.

There are several immediate questions from our work. For example, is it possible to remove the log k multiplicative factor in our non-interactive algorithm (Theorem 5)? Similarly, can the second additive term in our interactive algorithm be removed? As also suggested by [BBGK20], an intriguing research direction is to study more complicated statistics such as the 'higher-degree' ones (e.g., those involving triplets instead of pairs). It would be interesting to see if techniques from linear queries and from our work can be applied to these problems.

References

[Abo18] John M Abowd. The US Census Bureau adopts differential privacy. In KDD , pages 2867-2867, 2018.

[ACG + 16] Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In SIGSAC , pages 308-318, 2016.

[App17] Apple Differential Privacy Team. Learning with privacy at scale. Apple Machine Learning Journal , 2017.

[BBGK20] James Bell, Aurélien Bellet, Adrià Gascón, and Tejas Kulkarni. Private protocols for U-statistics in the local model and beyond. In AISTATS , volume 108, pages 1573-1583, 2020.

[BBGN20] Borja Balle, James Bell, Adrià Gascón, and Kobbi Nissim. Private summation in the multi-message shuffle model. In CCS , pages 657-676, 2020.

[BBNS19] Jaroslaw Blasiok, Mark Bun, Aleksandar Nikolov, and Thomas Steinke. Towards instance-optimal private query release. In SODA , pages 2480-2497, 2019.

[BDKT12] Aditya Bhaskara, Daniel Dadush, Ravishankar Krishnaswamy, and Kunal Talwar. Unconditional differentially private mechanisms for linear queries. In STOC , pages 1269-1284, 2012.

[BHBFG + 22] Gecia Bravo-Hermsdorff, Róbert Busa-Fekete, Mohammad Ghavamzadeh, Andres Munoz Medina, and Umar Syed. Private and communication-efficient algorithms for entropy estimation. NeurIPS , 35:15382-15393, 2022.

[BNO08] Amos Beimel, Kobbi Nissim, and Eran Omri. Distributed private data analysis: Simultaneously solving how and what. In CRYPTO , pages 451-468. Springer, 2008.

[Bub15] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

[BUV18] Mark Bun, Jonathan R. Ullman, and Salil P. Vadhan. Fingerprinting codes and the price of approximate differential privacy. SIAM J. Comput. , 47(5):1888-1938, 2018.

[CM23] Graham Cormode and Igor L. Markov. Federated calibration and evaluation of binary classifiers. Proc. VLDB Endow. , 16(11):3253-3265, 2023.

[CSS12] TH Hubert Chan, Elaine Shi, and Dawn Song. Optimal lower bound for differentially private multi-party aggregation. In ESA , pages 277-288, 2012.

[CSU + 19] Albert Cheu, Adam D. Smith, Jonathan R. Ullman, David Zeber, and Maxim Zhilyaev. Distributed differential privacy via shuffling. In EUROCRYPT , pages 375-403, 2019.

[DG03] Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Struct. Algorithms , 22(1):60-65, 2003.

[DKY17] Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. In NeurIPS , pages 3571-3580, 2017.

[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In TCC , pages 265-284, 2006.

[EGS03] Alexandre Evfimievski, Johannes Gehrke, and Ramakrishnan Srikant. Limiting privacy breaches in privacy preserving data mining. In PODS , pages 211-222, 2003.

[ENU20] Alexander Edmonds, Aleksandar Nikolov, and Jonathan R. Ullman. The power of factorization mechanisms in local and central differential privacy. In STOC , pages 425-438, 2020.

[EPK14] Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. RAPPOR: Randomized aggregatable privacy-preserving ordinal response. In CCS , pages 1054-1067, 2014.

[GMPV20] Badih Ghazi, Pasin Manurangsi, Rasmus Pagh, and Ameya Velingker. Private aggregation from fewer anonymous messages. In EUROCRYPT , pages 798-827, 2020.

[Gre16] Andy Greenberg. Apple's 'differential privacy' is about collecting your data - but not your data. Wired, June , 13, 2016.

[HT10] Moritz Hardt and Kunal Talwar. On the geometry of differential privacy. In STOC , pages 705-714, 2010.

[KLN + 11] Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam D. Smith. What can we learn privately? SIAM J. Comput. , 40(3):793-826, 2011.

[KT18] Krishnaram Kenthapadi and Thanh T. L. Tran. Pripearl: A framework for privacypreserving analytics and reporting at linkedin. In CIKM , pages 2183-2191, 2018.

[LMH + 15] Chao Li, Gerome Miklau, Michael Hay, Andrew McGregor, and Vibhor Rastogi. The matrix mechanism: optimizing linear counting queries under differential privacy. VLDB J. , 24(6):757-781, 2015.

[LMSS07] Nati Linial, Shahar Mendelson, Gideon Schechtman, and Adi Shraibman. Complexity measures of sign matrices. Comb. , 27(4):439-463, 2007.

[LSS08] Troy Lee, Adi Shraibman, and Robert Spalek. A direct product theorem for discrepancy. In CCC , pages 71-80, 2008.

[Nik23] Aleksandar Nikolov. Private query release via the johnson-lindenstrauss transform. In Nikhil Bansal and Viswanath Nagarajan, editors, SODA , pages 4982-5002, 2023.

[NTZ16] Aleksandar Nikolov, Kunal Talwar, and Li Zhang. The geometry of differential privacy: The small database and approximate cases. SIAM J. Comput. , 45(2):575-616, 2016.

[RE19] Carey Radebaugh and Ulfar Erlingsson. Introducing TensorFlow Privacy: Learning with Differential Privacy for Training Data, March 2019. blog.tensorflow.org .

[RSP + 21] Ryan Rogers, Subbu Subramaniam, Sean Peng, David Durfee, Seunghyun Lee, Santosh Kumar Kancha, Shraddha Sahay, and Parvez Ahammad. Linkedin's audience engagements API: A privacy preserving data analytics system at scale. J. Priv. Confiden. , 11(3), 2021.

[Sha14] Stephen Shankland. How Google tricks itself to protect Chrome user privacy. CNET, October , 2014.

[TJ89] Nicole Tomczak-Jaegermann. Banach-Mazur distances and finite-dimensional operator ideals. Pure and Applied Mathematics , 38:395, 1989.

[TM20] Davide Testuggine and Ilya Mironov. PyTorch Differential Privacy Series Part 1: DP-SGD Algorithm Explained, August 2020. medium.com .

A Additional Preliminaries

We use d tv , d KL to denote the total variation (TV) distance and the Kullback-Leibler (KL) divergence between two distributions. Pinsker's inequality states that

for all distributions P, Q .

Interactive Local DP

For interactive local DP, we consider protocols that proceed in rounds. In each round, the analyst may send a message (which may depend on what the analyst has received in the previous rounds) to each user, who then replies back with a (randomized) response. The DP guarantee is then enforced on the view of the analyst.

Clipping and Resulting Error

Clipping is a standard technique in DP to achieve bounded sensitivity (cf. [ACG + 16]). In our interactive algorithm, we will use clipping on real values to bound their sensitivity. More specifically, for τ ≥ 0 , let clip τ : R → R denote the function:

There could be additional errors resulting from clipping. For the purpose of bounding such terms, we will use the following simple lemma:

Lemma 18. Let X denote any random variable over R and any τ ≥ 0 . Then, we have

Proof. We can bound the LHS term by

where in the second step we used the Cauchy-Schwarz inequality.

Factorization vs Approximate-Factorization: Proof of Lemma 10

Proof of Lemma 10. Let W be any matrix. By definition of ζ , there exists ˜ W such that ζ ( W ; n ) = γ 2 ( ˜ W ) + ∥ ˜ W -W ∥ ∞ · ( ε √ n ) . The new protocol A ′ simply runs A but on ˜ W . Let ˆ z denote the output of A ′ . Its MSE can be bounded by

Projection Mechanism

As outlined in Section 1.2, we will use the projection mechanism as a subroutine for our algorithm. For W ∈ R ℓ × k , we define W ∆ to denote the set { Wx | x ∈ R k , ∥ x ∥ 1 ≤ 1 } . The guarantees of the algorithm are summarized below.

Theorem 19 ([BBNS19]) . For any workload W ∈ R ℓ × k , there is a non-interactive ε -local DP mechanism M such that

Moreover, the output M ( W ) always belongs to W ∆ .

We provide the proof below for completeness.

Proof of Theorem 19. Let C := ∥ W ∥ 1 → 2 . The algorithm works as follows:

▷ Each user i ∈ [ n ] sends y i := VRand ε,C ( W 1 x i ) to the analyst (where VRand · ( · ) is from Theorem 9).

▷ The analyzer computes Y := 1 n ∑ i ∈ [ n ] y i and then outputs µ := argmin u ∈ W ∆ ∥ u -Y ∥ 2 .

The privacy guarantee of the algorithm follows from Theorem 9.

Let z i = y i -W 1 x i . From Theorem 9, z i is zero-mean and σ -sub-Gaussian for σ = O ( C/ε ) . Let Z := 1 n ∑ i ∈ [ n ] z i ; we then have that it is zero-mean and σ ′ -sub-Gaussian for σ ′ = σ/ √ n . By the definition of sub-Gaussian random variable and a union bound, for every β > 0 , with probability 1 -β the following holds:

where W j denote the j th column of W .

It is well known 7 that if we define µ as we did (i.e., as projection of Y on W ∆ ), then we have

for all w ∈ W ∆ .

Plugging in w = Wh x , we have

where the equality follows from the fact W ∆ is the convex hull of W 1 , . . . , W k , -W 1 , . . . , -W k . Putting together (2) and the above then yields

B On the Lower Bound for Linear Queries from [ENU20]

In this section, we provide details on how we can interpret the bounds of [ENU20] as stated in the form of Theorem 4. The lower bound in [ENU20] is originally for the ℓ ∞ -error; we make the observation below that a simple modification of their proof can be made so that it applies to mMSE . Note that, in addition to getting a stronger result in terms of error metric (because a lower bound on

mMSE implies the same lower bound on ℓ ∞ -error), we also get a quantitatively stronger bound that does not depend on k because we avoid having to take a union bound over k queries (which was required for the proof for ℓ ∞ -error in [ENU20]).

In [ENU20, Section 3.4], it was shown that w.l.o.g. it suffices to consider 'symmetric' workload matrix W (i.e., ones that can be written as [ W ′ , -W ′ ] for some W ′ ). We will thus do so throughout the rest of this section. Following their notation, we also assume that W ∈ R k ×X , i.e., X is the input space and there are k linear queries. We also recall the following two lemmas from their paper.

Lemma 20 ([ENU20, Lemma 11]) . Let ε ∈ (0 , 1] . For any distribution λ 1 , . . . , λ m , µ 1 , . . . , µ m on X , distribution π over [ m ] and ε -local DP randomizer R , we have 8

where M ∈ R m ×X is a matrix such that M v,x = λ v ( x ) -µ v ( x ) .

We note that in [ENU20, Lemma 11], the LHS is not exactly the same as in our version above. However, inspecting the very first inequality from their proof shows that their bound goes through the quantity on the LHS of Lemma 20.

The next lemma, which describes the properties of the hard distributions, is arguably the main technical contribution of the lower bound in [ENU20].

Lemma 21 ([ENU20, Lemma 21]) . Let W ∈ R k ×X be any symmetric workload matrix and α > 0 . There exist 9 ξ ≥ 0 , and distributions ˜ λ 1 , . . . , ˜ λ k , ˜ µ 1 , . . . , ˜ µ k on X and ˜ π over [ k ] such that

(i) ( W ˜ λ v ) i = 0 for all i, v ∈ [ k ] .

(ii) for all v ∈ supp(˜ π ) , ( W ˜ µ v ) v ≳ ξ + α log( ∥ W ∥ ∞ /α ) . 10

(iii) Let ˜ M be defined similarly as in Lemma 20. Then ∥ ˜ M ∥ ℓ ∞ → L 2 (˜ π ) ≲ ξ γ 2 ( W,α ) .

It will in fact be easier to work with the following version of the lemma, which is implicit in the proof of [ENU20, Theorem 22]:

Lemma 22. Let W ∈ R k ×X be any symmetric workload matrix and α > 0 . There exist distributions ˜ λ 1 , . . . , ˜ λ k , ˜ µ 1 , . . . , ˜ µ k on X and ˜ π over [ k ] such that

(i) ( W ˜ λ v ) i = 0 for all i, v ∈ [ k ] .

(ii) for all v ∈ supp(˜ π ) , ( W ˜ µ v ) v ≳ α log( ∥ W ∥ ∞ /α ) .

(iii) Let ˜ M be defined similarly as in Lemma 20. Then ∥ ˜ M ∥ ℓ ∞ → L 2 (˜ π ) ≲ α γ 2 ( W,α ) .

Proof Sketch. This is constructed by taking ˜ λ 1 , . . . , ˜ λ k , ˜ µ 1 , . . . , ˜ µ k and ˜ π from Lemma 21. Then, replace each ˜ µ v by the mixture (1 -β ) · ˜ λ v + β · ˜ µ v where β = min { 1 , α/ξ } , for all v ∈ [ k ] . Both the lower bound for ( W ˜ µ v ) v and the upper bound for ∥ ˜ M ∥ ℓ ∞ → L 2 (˜ π ) scale linearly with β , yielding the desired bounds.

We can now prove the following, which is a more qualitative version of the lower bound in Theorem 4.

Theorem 23. Let W ∈ R be any symmetric workload matrix. Any non-interactive ε -local DP  

mechanism must incur an mMSE at least Ω  ζ ( W,n ) 2 ε 2 n · 1 log ( ε 2 n ∥ W ∥ 2 ∞ ζ ( W,n ) 2 ) 2  .

Proof. For notational convenience, we write ζ as a shorthand for ζ ( W,n ) . Let α = C 1 ζ ε √ n where C 1 ∈ (0 , 0 . 1) is a sufficiently small constant. Suppose for the sake of contradiction that there exists a non-interactive ε -local DP protocol (whose randomizer is R ) with mMSE at most ( α ′ ) 2 for α ′ = C 2 α log( ∥ W ∥ ∞ /α ) where C 2 ∈ (0 , 0 . 1) is a sufficiently small constant.

Recall the definition of ζ ; we must have ζ ≤ γ 2 ( W, ζ 2 ε √ n ) + ζ 2 . This gives

Since γ 2 ( W, · ) is an increasing function, we thus have

Let ˜ λ 1 , . . . , ˜ λ k , ˜ µ 1 , . . . , ˜ µ k and ˜ π be as in Lemma 22. By Lemma 20 and Lemma 22(iii), we have

On the other hand, since the protocol has mMSE at most ( α ′ ) 2 , we may use the following algorithm to distinguish R ( λ v ) n from R ( µ v ) n : let the analyst computes an estimate for Wh x (using the n samples from R ( · ) provided). If the estimate has absolute value less than 10 α ′ , then output λ v ; otherwise, output µ v . From Lemma 22(i)(ii), it is not hard to see that, for C 2 that is sufficiently small, this algorithm is correct with probability at least 2/3. This means that d tv ( R ( λ v ) n , R ( µ v ) n ) ≥ 1 / 3 . Pinsker's inequality (i.e., eq. (1)) then yields d KL ( R ( λ v ) n ∥ R ( µ v ) n ) ≳ 1 . Comparing this with the above eq. (4), we get αε √ n γ 2 ( W,α ) ≳ 1 , which contradicts eq. (3) when C 1 is sufficiently small.

C Three-Round Algorithm

In this section, we describe and analyze our interactive algorithm. The error guarantees of the algorithm are stated below. (Note that Theorem 24 is a more precise version of Theorem 8.)

Theorem 24. For any W ∈ R k × k and n ≥ ˜ Ω ( γ 2 ( W ) 4 log k ∥ W ∥ 4 ∞ ε 2 ) , there is a three-round ε -DP algorithm for estimating quadratic forms on W such that the MSE is O ( ∥ W ∥ 2 ∞ ε 2 n ) .

Proof. Throughout the proof, we assume that n ≥ Q · γ 2 ( W ) 4 log k ∥ W ∥ 4 ∞ ε 2 · log ( 10 γ 2 ( W ) log k ∥ W ∥ ∞ ε ) , where Q is a sufficiently large constant. Recall from the definition of γ 2 ( W ) that there must exist L, R such that W = L T R and ∥ L ∥ 1 → 2 = ∥ R ∥ 1 → 2 = √ γ 2 ( W ) =: C .

Algorithm Description. Let ¯ ε = ε/ 4 and τ = 4 ∥ W ∥ ∞ . The algorithm works as follows:

▷ First Round: Run the ¯ ε -local DP protocol from Theorem 19 to get an estimate µ R of Rh x .

▷ Second Round:

▷ The analyzer forwards µ R to all users.

▷ Each user i ∈ [ n ] sends the following to the analyst:

▷ y L i ← VRand ¯ ε,C ( L 1 x i ) (where VRand · ( · ) is from Theorem 9).

▷ a i ←⟨ L 1 x i , µ R ⟩ + κ i where κ i ∼ Lap(2 ∥ W ∥ ∞ / ¯ ε ) back to the analyst.

▷ Third Round:

▷ The analyst then computes Y L := 1 n ∑ i ∈ [ n ] y L i and forwards it to the users.

▷ Each user i ∈ [ n ] sends v i ← clip τ ( 〈 Y L , R 1 x i -µ R 〉 ) + z i where z i ∼ Lap(2 τ/ ¯ ε ) .

Finally, the analyst outputs 1 n ( ∑ i ∈ [ n ] a i ) + 1 n ( ∑ i ∈ [ n ] v i ) .

Privacy Analysis. Each user i 's input is used four times:

▷ To produce µ R . This step is ¯ ε -DP by Theorem 19.

▷ To produce y L i . This step is ¯ ε -DP by Theorem 9.

▷ To produce a i . From the guarantee of Theorem 19, µ R belong to R ∆ . As a result, | ⟨ L 1 x i , µ R ⟩ | ≤ ∥ W ∥ ∞ . In other words, the sensitivity of ⟨ L 1 x i , µ R ⟩ (as a function of x i ) is at most 2 ∥ W ∥ ∞ . Since κ i is sampled from Lap(2 ∥ W ∥ ∞ / ¯ ε ) , this step is also ¯ ε -DP.

▷ To produce v i . Due to clipping, the sensitivity of clip τ ( 〈 Y L , R 1 x i -µ R 〉 ) (as a function of x i ) is at most 2 τ . Thus, since we are adding a noise z i drawn from Lap(2 τ/ ¯ ε ) , this step is also ¯ ε -DP.

Hence, by the basic composition theorem, the entire algorithm is ε -local DP as desired.

Utility Analysis. First, notice that, for a fixed µ R ∈ R ∆ , we have

Let z L i = y L i -L 1 x i . From Theorem 9, z L i is zero-mean and σ -sub-Gaussian for σ = O ( C/ε ) . Let Z L := 1 n ∑ i ∈ [ n ] z L i ; we then have that Z L is zero-mean and σ ′ -sub-Gaussian for σ ′ = σ/ √ n . Note that Y L = Lh x + Z L . Furthermore, let us write θ i as a shorthand for clip τ ( 〈 Y L , R 1 x i -µ R 〉 ) -〈 Y L , R 1 x i -µ R 〉 .

The MSE can be bounded by

where the last step is by applying the Cauchy-Schwarz inequality to the last term.

To handle the middle term in eq. (5), note that Z L is independent of µ R and, as stated earlier, is σ ′ -sub-Gaussian. Thus, we have

where the last inequality is due to our assumption on n .

As for the last term in eq. (5), notice that

Since µ R belongs to R ∆ , we have 1 2 ( R 1 x i -µ R ) ∈ R ∆ , which implies that | ⟨ Lh x , R 1 x i -µ R ⟩ | ≤ 2 ∥ W ∥ ∞ and ∥ R 1 x i -µ R ∥ 2 ≤ 2 C . As such, we have

where the last inequality is due to our assumption on n and from γ 2 ( W ) ≥ ∥ W ∥ ∞ .

Meanwhile, since τ = 4 ∥ W ∥ ∞ , we have

where the last inequality is again due to our assumption on n and from γ 2 ( W ) ≥ ∥ W ∥ ∞ .

Thus, we may apply Lemma 18 to conclude that

Combining eqs. (5) to (7), the MSE of the estimate is at most O ( ∥ W ∥ 2 ∞ ε 2 n ) as desired.

C.1 On Separating Non-Interactive and Interactive Local DP

We end by observing that our interactive local DP algorithm (Theorem 24) together with the noninteractive lower bound (Theorem 6 and particularly, the more quantitative version, Theorem 23) gives an asymptotic separation on the MSE achievable by ε -local DP interactive algorithms and those that are non-interactive, as long as we pick W together with n ∈ N such that the following holds:

where the first condition is from Theorem 24 (and letting ε = 1 ) and the second condition ensures that the error from Theorem 24 is asymptotically larger than the lower bound from Theorem 23.

It is simple to verify that it suffices to pick W such that ∥ W ∥ ∞ = 1 , γ 2 ( W, 0 . 1) , γ 2 ( W ) = Θ( k c ) for any constant c > 0 and pick n to precisely satisfy the bound in the first condition. Again, it is not hard to construct such a matrix e.g., by taking W to be a random ( k × k ) matrix where each entry is an i.i.d. Rademacher random variable, which yields γ 2 ( W, 0 . 1) , γ 2 ( W ) = Θ( √ k ) w.h.p. [LMSS07].

Peer reviews:
comments: SUMMARY:
In this paper, the authors analyzed the problem of privately computing the quadratic form in the model of differential privacy, and provide non-interactive local DP algorithm with MSE upper/lower bounds with gap log(k). The paper further develops results for an interactive algorithm for the same problem and proves analogous bounds.

STRENGTHS:
The bound results for both non-interactive and interactive local DP algorithms are generic. The authors provided a complete analysis on the non-interactive algorithm along with its bound results.

WEAKNESSES:
Compared to the linear queries result, the MSE bounds of the non-interactive local DP mechanism for estimating quadratic forms reveal a noticeable gap depending on $k$, the dimension of matrix W.

QUESTIONS:
1. In addition to the discussions in the paper, can the author provide some intuitions on if the upper bound of the non-interactive local DP mechanism for estimating quadratic forms should depend on k?

2. Can the author provide some intuition on why the definition of $\gamma_2(W,\alpha)$ uses infinity norm instead of other norms? Is there any complexity results on other norms?

LIMITATIONS:
Compared to the similar sample complexity results for computing the statistical queries in the models of differential privacy, the sample complexity results for computing the pairwise statistics in local DP reveals a gap that depends on the dimension of the matrix $W$.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper studies the computing of pairwise statistics with local differential privacy by considering the quadratic form computation.  In order to obtain the lower boudn and uppoer bounds, it proposes the inter-reductions between quadratic forms and linear queries.

STRENGTHS:
Studying pair statistics with local differential privacy is intersting.

WEAKNESSES:
(1)  The presentation needs some improvement.   There are many grammmatical mistakes in many  places.  The preliminary part is quite messy: a formal definition of central differetial privacy is given but the definition of local DP is informal.   Randomized response mechanisms are not defined in the paper.
(2)  I don't see the significance of the two reductions in Section 1.2.    Either the reductions are trivial or the subtlety has not been made explicit yet.

QUESTIONS:
Q1:  Could you pls explain the logic behind the sentence in Line 150?

Q2: Could you explain more the matrix mechanism especially the factorization? (Line 159)

Q3:  I don't see the reasoning in Line 288 especially the second line there about "cross terms".

LIMITATIONS:
This paper highly depends on the results in ENU20. I don't see any significant technical contribution in this paper
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of computing Quadratic Forms $h_x^TWh_x$ under Local Differential Privacy, where $h_x$ is the normalized histogram representation of a vector $x\\in [k]^n$. In particular, reductions to and from the problem of computing linear queries are established, through which algorithms and tight bound results (in mean squared error) consequently follow from those in linear queries. The reduction from linear queries is built on the observation that the $j$th entry $(Wh_x)_j$ of the linear query can be obtained from three quadratic forms:\n\n($h_{x\\cup 1_j}^TWh_{x\\cup 1_j}$), ($h_{x}^TWh_{x}$) and ($h_{1_j}^TWh_{1_j}$). \n\nThe reduction to linear queries is built on two observations: 1) $h_x^T W h_x^T$ is a linear query on $h_x$ with weights $Wh_x$ (also a linear query); 2) $h_x^T W h_x^T$ is an inner product of $Lh_x$ and $Rh_x$, where $W=L^T R$. The second approach can be further refined, where using a JL projection on $L$ and $R$ allows the magnitudes of the noise terms associated with privatizing $Lh_x$ and $Rh_x$ to be reduced to $O(\\log n)$ from $O(k)$.\n\nSTRENGTHS:\n- This paper presents a generic framework for estimating pairwise statistics that can be expressed as quadratic forms (although utility in some statistics can be improved by exploiting their specific properties).\n- The ideas used are interesting and natural, the presented analysis is solid.\n- They discuss both interactive and non-interactive settings, and show that computing pairwise statistics separate interactive and non-interactive local DP.\n\nWEAKNESSES:\n- In the proof of Theorem 5, the condition on $\\alpha$ requires $\\varepsilon \\ge 1/\\sqrt{n}$, but $l=O(\\log(k) \\varepsilon^2 n)$ requires $\\varepsilon < 1$ for $l< n$? These place $\\varepsilon$ in a restrictive range.\n- Although the main paper focuses on presenting reductions in the non-interactive setting, there are places where the discussions of interactive and non-interactive appear together, which caused some distraction for me. I think all of the discussion on the interactive setting can be moved to its own section toward the end.\n\nQUESTIONS:\n- In the proof of Theorem 14, line 286 requires $W$ to be symmetric?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the computation of quadratic forms of histograms under local differential privacy (LDP). Previously, special cases of this problem have been studied, but this paper presents a general theory analogous to the existing theory for linear queries. The problem is studied both in the standard LDP setting and using \"interactive\", multi-round protocols where each user outputs an LDP value in every round. The 9-page submission focuses on the standard, non-interactive setting. Lower and upper bounds are presented that are tight up to polylogarithmic factors. A sketch of interactive protocols (with better utility) is presented, with details in supplementary material.\n\nSTRENGTHS:\n- Identifies a natural problem for which special cases have been studied before and presents a general theory\n- The results are tight (up to polylogarithmic factors) for *every* quadratic form\n- The techniques for multi-round LDP protocols are particularly interesting, and separate single- and multi-round LDP for a natural problem\n- The mechanisms are simple to describe\n- The paper is very well-written\n\nWEAKNESSES:\n- Though LDP has been deployed in practice, its general usefulness has sometimes been questioned, since it tends to require very large data sizes to get good utility\n\nQUESTIONS:\n- Can you comment on potential practical use of your results? Are there large factors hidden in big-O notation, and in particular what kind of data size is needed to get good utility for the examples in Corollary 7?\n- The shuffle model is sometimes used to amplify privacy of LDP protocols. What results in the this model are implied by your results?\n- In line 36, I suppose $\\mathcal{X}$ should be $\\mathcal{X}^2$?\n\nLIMITATIONS:\nSince there are no experiments, I would have like at least some discussion of the extent to which the protocols proposed might be practical, or whether the contribution is considered purely theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}