Title: Fast Optimal Locally Private Mean Estimation via Random Projections
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy

mean estimation

private federated learning

communication complexity
Abstract: We study the problem of locally private mean estimation of high-dimensional vectors in the Euclidean ball. Existing algorithms for this problem either incur sub-optimal error or have high communication and/or run-time complexity. We propose a new algorithmic framework, namely ProjUnit, for private mean estimation that yields algorithms that are computationally efficient, have low communication complexity, and incur optimal error up to a $1+o(1)$-factor. Our framework is deceptively simple: each randomizer projects its input to a random low-dimensional subspace and then runs an optimal algorithm such a PrivUnitG in the lower dimensional space. We analyze the error of the algorithm in terms of properties of the random projection ensemble, and study two instantiations. We conduct several experiments for private mean estimation and private federated learning which demonstrate that our algorithms obtain nearly the same utility as optimal algorithms while having significantly lower communication and computational cost.

Fast Optimal Locally Private Mean Estimation via Random Projections

Abstract

We study the problem of locally private mean estimation of high-dimensional vectors in the Euclidean ball. Existing algorithms for this problem either incur suboptimal error or have high communication and/or run-time complexity. We propose a new algorithmic framework, ProjUnit, for private mean estimation that yields algorithms that are computationally efficient, have low communication complexity, and incur optimal error up to a 1 + o (1) -factor. Our framework is deceptively simple: each randomizer projects its input to a random low-dimensional subspace, normalizes the result, and then runs an optimal algorithm such as PrivUnitG in the lower-dimensional space. In addition, we show that, by appropriately correlating the random projection matrices across devices, we can achieve fast server run-time. We mathematically analyze the error of the algorithm in terms of properties of the random projections, and study two instantiations. Lastly, our experiments for private mean estimation and private federated learning demonstrate that our algorithms empirically obtain nearly the same utility as optimal ones while having significantly lower communication and computational cost.

1 Introduction

Distributed estimation of the mean, or equivalently the sum, of vectors v 1 , . . . , v n ∈ R d is a fundamental problem in distributed optimization and federated learning. For example, in the latter, each of n devices may compute some update to parameters of a machine learning model based on its local data, at which point a central server wishes to apply all updates to the model, i.e. add ∑ n i =1 v i to the vector of parameters. The typical desire to keep local data private necessitates methods for computing this sum while preserving privacy of the local data on each individual device, so that the central server essentially only learns the noisy sum and (almost) nothing about each individual summand v i [20, 11].

The gold standard for measuring privacy preservation is via the language of differential privacy [21]. In this work, we study this problem in the setting of local differential privacy (LDP). We consider (oneround) protocols for which there exists some randomized algorithm R : R d →M (called the local randomizer ), such that each device i sends R ( v i ) to the server. We say the protocol is ε -differentially private if for any v, v ′ ∈ R d and any event S ⊆ M , Pr( R ( v ) ∈ S ) ≤ e ε Pr( R ( v ′ ) ∈ S ) . If ε = 0 then the distribution of R ( v ) is independent of v , and hence the output of R ( · ) reveals nothing about the local data (perfect privacy); meanwhile if ε = ∞ then the distributions of R ( v ) and R ( v ′ ) can be arbitrarily far, so that in fact one may simply set R ( x ) = x and reveal local data in the clear (total lack of privacy). Thus, ε ≥ 0 is typically called the privacy loss of a protocol.

Kunal Talwar

Apple Inc.

There has been much previous work on private protocols for estimating the mean µ := 1 n ∑ n i =1 v i in the LDP setting. Henceforth we assume each v i lives on the unit Euclidean sphere 1 S d -1 ⊂ R d . Let ˆ µ be the estimate computed by the central server based on the randomized messages R ( v 1 ) , . . . , R ( v n ) it receives. Duchi and Rogers [18] showed that the asymptotically optimal expected mean squared error E ∥ µ -ˆ µ ∥ 2 2 achievable by any one-round protocol must be at least Ω( d n min( ε,ε 2 ) ) , which is in fact achieved by several protocols [19, 9, 14, 24]. These protocols however achieved empirically different errors, with some having noticeably better constant factors than others.

Recent work of Asi et al. [7] sought to understand the optimal error achievable by any protocol. Let R be any local randomizer satisfying ε -DP, and A be an aggregation algorithm for the central server such that it computes its mean estimate as ˆ µ := A ( R ( v 1 ) , . . . , R ( v n )) . Furthermore, suppose that the protocol is unbiased , so that E ˆ µ = µ for any inputs v 1 , . . . , v n . Lastly, let A PrivUnit ε , R PrivUnit ε denote the PrivUnit ε protocol of [9] (parameterized to satisfy ε -DP 2 ). Let Err n,d ( A , R ) denote

Asi et al. [7] proved the remarkable theorem that for any n, d :

Thus, PrivUnit is not only asymptotically optimal, but in fact actually optimal in a very strong sense (at least, amongst unbiased protocols).

While this work thus characterizes the optimal error achievable for ε -LDP mean estimation, there are other desiderata that are important in practice. The most important amongst them are the device runtime (the time to compute R ( v ) ), the server runtime (the time to compute A on ( R ( v 1 ) , . . . , R ( v n )) ), and the communication cost ( ⌈ log 2 |M|⌉ bits for each device to send its R ( v ) to the server). The known error-optimal algorithms (PrivUnit [9] and PrivUnitG [7]) either require communicating d floats or have a slower device runtime of Ω( e ε d ) . As mean estimation is often used as a subroutine in high-dimensional learning settings, this communication cost can be prohibitive and this has led to a large body of work on reducing communication cost [2, 27, 14, 26, 24, 12]. Server runtimes of these optimal algorithms are also slow, scaling as nd , whereas one could hope for nearly linear time ˜ O ( n + d ) (see Table 1).

Chen et al. [14] recently studied this tradeoff and proposed an algorithm called SQKR, which has an optimal communication cost of ε bits and device runtime only O ( d log 2 d ) . However, this algorithm incurs error that is suboptimal by a constant factor, which can be detrimental in practice. Indeed our experiments in Section 4 demonstrate the significance of such constants as the utility of these algorithms does not match that of optimal algorithms even empirically, resulting for example in 10% degradation in accuracy for private federated learning over MNIST with ε = 10 .

Feldman and Talwar [24] give a general approach to reducing communication via rejection sampling. When applied to PrivUnitG, it yields a natural algorithm that we call Compressed PrivUnitG. While it yields optimal error and near-optimal communication, it requires device run time that is O ( e ε d ) . These algorithms are often used for large d (e.g. in the range 10 5 -10 7 ) corresponding to large model sizes. The values of ε are often in the range 4 -12 or more, which may be justifiable due to privacy being improved by aggregation or shuffling [10, 15, 22, 25]. For this range of values, the Θ( e ε d ) device runtime is prohibitively large and natural approaches to reduce this in Feldman and Talwar [24] lead to increased error. To summarize, in the high-dimensional setting, communication-efficient local randomizers are forced to choose between high device runtime or suboptimal error (see Table 1).

Another related line of work is non-private communication efficient distributed mean estimation where numerous papers have recently studied the problem due to its importance in federated learning [37, 4, 29, 2, 26, 23, 32, 38, 39]. Similarly to our paper, multiple works in this line of work have used random rotations to design efficient algorithms [37-39]. However, the purpose of these works is to

develop better quantization schemes for real-valued vectors to reduce communication to (1+ o (1)) · d bits. This is different from our goal, which is to send k ≪ d parameters while still obtaining the statistically optimal bounds up to a 1 + o (1) factor. Moreover, in order to preserve privacy, our algorithms require new techniques to handle the norms of the projected vectors, and post-process them using different normalization schemes, in order to guarantee that the projection error after post-processing is negligible.

1.1 Contributions

Our main contribution is a new framework, ProjUnit, for private mean estimation which results in nearoptimal, efficient, and low-communication algorithms. Our algorithm obtains the same optimal utility as PrivUnit and PrivUnitG but with a significantly lower polylogarithmic communication complexity, and device runtime that is O ( d log d ) and server runtime ˜ O ( n + d ) . We also implement our algorithms and show that both the computational cost and communication cost are small empirically as well. Figure 1 plots the error as a function of ε for several algorithms and demonstrates the superiority of our algorithms compared to existing low-communication algorithms (see more details in Section 4). Moreover, we show that the optimal error bound indeed translates to fast convergence in our private federated learning simulation.

At a high level, each local randomizer in our algorithm first projects the vector to a randomly chosen lower-dimensional subspace, and then runs an optimal local randomizer in this lowerdimensional space. At first glance, this is reminiscent of the use of random projections in the Johnson-Lindenstrauss (JL) transform or the use of various embeddings in prior work (such as [14]). However, unlike the JL transform and embeddings in prior work, in our application, each point uses its own projection matrix . The JL transform is designed to preserve distances between points, not the points themselves. In our application, a random projection is used to obtain a low-dimensional unbiased estimator for a point; however the variance of this estimator is quite large (of the order of d/k ). Our main observation is that while large, this variance is small compared to the variance added due to privacy when k is chosen appropriately. This fact allows us to use the projections as a pre-processing step. With some care needed to control the norm of the projected vector which is no longer fixed, we then run a local randomizer in the lower dimensional space. We analyze the expected squared error of the whole process and show that as long as the random projection ensemble satisfies certain specific properties, the expected squared error of our algorithm is within (1 + o (1)) factors of the optimal; the o (1) term here falls with the projection dimension k . The required properties are easily shown to be satisfied by random orthogonal projections. We further show that more structured projection ensembles, that allow for faster projection algorithms, also satisfy the required properties, and this yields even faster device runtime.

Although these structured projections result in fast device runtime, they still incur an expensive computational cost for the server which needs to apply the inverse transformation for each client, resulting in runtime O ( nd log d ) . Specifically, each device sends a privated version ˆ v i of W i v i , and the server must then compute ∑ i W ⊤ ˆ v i . To address this, we use correlated transformations in order to reduce server runtime while maintaining optimal accuracy up to 1 + o (1) factors. In our correlated ProjUnit protocol, the server pre-defines a random transformation W , which all devices then use to define W i = S i W where S i is a sampling matrix. The advantage is then that ∑ i W ⊤ i ˆ v i is replaced with ∑ i W ⊤ ˆ v i = W ⊤ ( ∑ i S i ˆ v i ) , which can be computed more quickly as it requires only a single matrix-vector multiplication. The main challenge with correlated transformations is that the correlated client transformations result in increased variance. However, we show that the independence in choosing the sampling matrices S i is sufficient to obtain optimal error.

Finally, we note without correlating projections each client using its own projection would imply that each projection needs to be communicated to the server. Doing this naively would require communicating kd real values completely defeating the benefits of our protocol. However the projection matrix does not depend on the input and therefore can be communicated cheaply using a seed to an appropriate pseudorandom generator.

2 A random projection framework for low-communication private algorithms

In this section we propose a new algorithm, namely ProjUnit, which has low communication complexity and obtains near-optimal error (namely, up to a 1+ o (1) factor of optimum). The starting point

2

4

6

8

10

12

14

10 4

10 5

FastProjUnit

ProjUnit

PrivUnitG

PrivHS

RePrivHS

SQKR

Squared Error

ε

of our algorithms is a randomized projection map in R k × d which we use to project the input vectors to a lower-dimensional space. The algorithm then normalizes the vector as a necessary pre-processing step. Finally, the local randomizer applies PrivUnitG [7] (see Algorithm 8 in Appendix) over the normalized projected vector and sends the response to the server. The server then applies the inverse transformation and aggregates the responses in order to estimate the mean. We present the full details of the client and server algorithms in Algorithm 1 and Algorithm 2.

To analyze this algorithm, we first present our general framework for an arbitrary distribution over projections. In the next sections we utilize different instances of the framework using different random projections such as random rotations and more structured transforms.

Algorithm 1 ProjUnit (client)

Require: Input vector v ∈ R d , Distribution over projections W .

1: Randomly sample transform W ∈ R k × d from W

2: Project the input vector v p = Wv

3: Normalize u = v p ∥ v p ∥ 2

4: Let ˆ u = PrivUnitG ( u ) (as in Algorithm 8)

5: Send ˆ u and (encoding of) W to server

The following theorem states the privacy and utility guarantees of ProjUnit for a general distribution over transformation W that satisfies certain properties. For ease of notation, let R PU denote the ProjUnit local randomizer of the client (Algorithm 1), and A PU denote the server aggregation of

Algorithm 2 ProjUnit (server)

1: Receive ˆ u 1 , . . . , ˆ u 1 from clients with (encodings of) transforms W 1 , . . . , W n

2: Return the estimate ˆ µ = 1 n ∑ n i =1 W ⊤ i ˆ u i

ProjUnit (Algorithm 2). To simplify notation, we let

denote the error of the PrivUnitG ε -DP protocol where A PrivUnitG ε , R PrivUnitG ε denote the PrivUnitG protocol with optimized parameters (see Algorithm 8) and c d,ε = O (1) is a constant [7]. We defer the proof to Appendix B.1.

Theorem 1. Let k ≤ d and assume that the transformations W i ∈ R k × d are independently chosen from a distribution W that satisfies:

Bounded operator norm: E [ ∥ ∥ W ⊤ i ∥ ∥ 2 ] ≤ d/k + β W .

Bounded bias: ∥ ∥ ∥ E [ W ⊤ i W i v ∥ W i v ∥ 2 ] -v ∥ ∥ ∥ 2 ≤ √ α W for all unit vectors v ∈ R d .

Then for all unit vectors v 1 , . . . , v n ∈ R d , setting ˆ µ = A PU ( R PU ( v 1 ) , . . . , R PU ( v n )) , the local randomizers R PU are ε -DP and

2.1 ProjUnit using Random Rotations

Building on the randomized projection framework of the previous section, in this section we instantiate it with a random rotation matrix. In particular, we sample W ∈ R k × d with the structure

where U ∈ R d × d is a random rotation matrix such that U ⊤ U = I , and S ∈ R k × d is a sampling matrix where each row has a single 1 in a uniformly random location (without repetitions). The following theorem states our guarantees for this distribution.

Theorem 2. Let k ≤ d and W ∈ R k × d be a random rotation matrix sampled as described in (1) . Then for all unit vectors v 1 , . . . , v n ∈ R d , setting ˆ µ = A PU ( R PU ( v 1 ) , . . . , R PU ( v n )) , the local randomizers R PU are ε -DP and

The proof follows directly from Theorem 1 and the following proposition which proves certain properties of random rotations. We defer the proof to Appendix B.2.

Proposition 1. Let W ∈ R k × d be a random rotation matrix sampled as described in (1) . Then

Bounded operator norm: ∥ W ⊤ ∥ ≤ √ d k .

Bounded bias: for every unit vector v ∈ R d : ∥ ∥ ∥ E W ⊤ Wv ∥ Wv ∥ -v ∥ ∥ ∥ = O (1 /k ) .

We also have similar analysis for Gaussian transforms with an additional O ( √ k/d ) factor in the first term. We include the analysis in Appendix D.

2.2 Fast ProjUnit using the SRHT

While the random rotation based randomizer enjoys near-optimal error and low communication complexity, its runtime complexity is somewhat unsatisfactory as it requires calculating Wv for W ∈ R k × d , taking time O ( kd ) . In this section, we propose a ProjUnit algorithm using the Subsampled Randomized Hadamard transform (SRHT), which is closely related to the fast JL transform [3]. We show that this algorithm has the same optimality and low-communication guarantees as the random rotations version, and additionally has an efficient implementation with O ( d log d ) client runtime.

The SRHT ensemble contains matrices W H ∈ R k × d with the following structure:

where S ∈ R k × d is a sampling matrix where each row has a single 1 in a uniformly random location sampled without replacement, H ∈ R d × d is the Hadamard matrix, and D ∈ R d × d is a diagonal matrix where D ii are independent samples from the Rademacher distribution, that is, D ii ∼ Unif {-1 , +1 } . The main advantage of the SRHT is that there exist efficient algorithms for matrix-vector multiplication with H .

The following theorem presents our main guarantees for the SRHT-based ProjUnit algorithm.

Theorem 3. Let k ≤ d and W be sampled from the SRHT ensemble as described in (2) . Then for all unit vectors v 1 , . . . , v n ∈ R d , setting ˆ µ = A PU ( R PU ( v 1 ) , . . . , R PU ( v n )) , the local randomizers R PU are ε -DP and

Remark 1. The communication complexity of SRHT-based ProjUnit can be reduced to O ( k log d + log 2 d ) . To see this, note that ˆ u is a k -dimensional vector. Moreover, the matrix W = √ d/k · SHD can be sent in O ( k log d ) as follows: S has k rows, each with a single entry that contains 1 , hence we can send the indices for each row using log d bits for each row. Moreover, H is the Hadamard transform and need not be sent. Finally, D is a diagonal matrix with entries D ii ∼ Unif {-1 , +1 } . By standard techniques [36, 5], we only need the entries of D to be O (log( d )) -wise independent for Proposition 2 to hold. Thus O (log 2 d ) bits suffice to communicate a sampled D .

The proof of the theorem builds directly on the following two properties of the SHRT.

Proposition 2. Let W be sampled from the SRHT ensemble as described in (2) . Then we have

Bounded operator norm: E [∥ ∥ W ⊤ ∥ ∥ ] = E [ ∥ W ∥ ] ≤ √ d/k.

Bounded bias: for every unit vector v ∈ R d : ∥ ∥ ∥ E W ⊤ Wv ∥ Wv ∥ -v ∥ ∥ ∥ = O (log( d ) / √ k ) .

Theorem 3 now follows directly from the bounds of Theorem 1 using α W = O (log 2 ( d ) /k ) . We defer the proof to Appendix B.3.

Remark 2. While our randomizers in this section pay an additive term that does not decrease with n (e.g. log 2 ( d ) /k for SRHT), this term is negligible in most settings of interest. Indeed, using Theorem 3 and the fact that Err n,d ( PrivUnitG ) = c d,ε d/nε , we get that the final error of our SRHT algorithm is roughly c d,ε d/nε (1 + o (1)) + O (log 2 ( d ) /k ) . This implies that in the high-dimensional setting the bias term is negligible.

However, to cover the whole spectrum of parameters, we develop a nearly unbiased versions of these algorithms in Appendix A. In particular, we show in Theorem 6 that our unbiased version has error

3 Efficient Server Runtime via Correlated Sampling

One downside of the algorithms in the previous section is that server runtime can be costly: indeed, as each client uses an independent transformation, the server has to apply the inverse transformation

(matrix multiplication) for each client, resulting in runtime O ( nd log d ) . In this section, we propose a new protocol that significantly reduces server runtime to O ( n log 3 d + d log d + nk ) while retaining similar error guarantees. The protocol uses correlated transformations between users which allows the server to apply an inverse transformation only a small number of times. However, clients cannot use the same transformation as this will result in large bias.

The protocol works as follows: the server samples U ∈ R d × d from the Randomized Hadamard transform: U = HD where H ∈ R d × d is the Hadamard transform, and D ∈ R d × d is a diagonal matrix where each diagonal entry is independently sampled from the Rademacher distribution. Then, client i ∈ [ n ] , samples a random sampling matrix S i ∈ R k × d , and uses U to define the transform W i ∈ R k × d :

We describe the full details of the client and server algorithms for correlated ProjUnit in Algorithm 3 and Algorithm 4, and denote them R CPU and A CPU , respectively. We have the following guarantee. We defer the proof to Appendix C.

Algorithm 3 Correlated ProjUnit (client)

Require: Input vector v ∈ R d .

1: Randomly sample diagonal D from the Rademacher distribution based on predefined seed

2: Sample S ∈ R k × d where each row is chosen uniformly at random without replacement from standard basis vectors { e 1 , . . . , e d }

3: Project the input vector v p = SHDv where H ∈ R d × d is the Hadamard matrix

4: Normalize u = v p ∥ v p ∥ 2

5: Let ˆ u = PrivUnitG ( u ) (as in Algorithm 8)

6: Send ˆ u , and (an encoding of) S to server

Algorithm 4 Correlated ProjUnit (server)

1: Receive ˆ u 1 , . . . , ˆ u 1 from clients with (encodings of) transforms S 1 , . . . , S n

2: Sample diagonal matrices D from Rademacher distribution based on predefined seed

3: Let U = HD where H ∈ R d × d is the Hadamard matrix

4: Return the estimate ˆ µ = 1 n U ⊤ ∑ n i =1 S ⊤ i ˆ u i

Theorem 4. Let k ≤ d . Then for all unit vectors v 1 , . . . , v n ∈ R d , setting ˆ µ = A CPU ( R CPU ( v 1 ) , . . . , R CPU ( v n )) , the local randomizers R CPU are ε -DP and

Moreover, server runtime is O ( n log( d ) log 2 ( nd ) + d log d + nk ) .

4 Experiments

We conclude the paper with several experiments that demonstrate the performance of our proposed algorithms, comparing them to existing algorithms in the literature. We conduct our experiments in two different settings: the first is synthetic data, where we aim to test our algorithms and understand their performance for our basic task of private mean estimation, comparing them to other algorithms. In our second setting, we seek to evaluate the performance of our algorithms for private federated learning which requires private mean estimation as a subroutine for DP-SGD.

4.1 Private mean estimation

In our synthetic-data experiment, we study the basic private mean estimation problem, aspiring to investigate the following aspects:

10 2

10 3

1.5 × 10 1

1.6 × 10 1

1.7 × 10 1

1.8 × 10 1

1.9 × 10 1

2×10 1

2.1 × 10 1

PrivUnitG

FastProjUnit

FastProjUnit-corr

ProjUnit

Squared Error

k

Utility of ProjUnit algorithms as a function of the communication budget

Utility of our low-communication algorithms compared to the optimal utility and other existing low-communication algorithms

Run-time complexity of our algorithms compared to existing algorithms

Our experiments 3 measure the error of different algorithms for estimating the mean of a dataset. To this end, we sample unit vectors v 1 , . . . , v n ∈ R d by normalizing samples from the normal distribution N ( µ, 1 /d ) (where µ ∈ R d is a random unit vector), and apply a certain privacy protocol NumRep times to estimate the mean ∑ n i =1 v i /n , producing mean squared errors e 1 , . . . , e NumRep . Our final error estimate is then the mean 1 NumRep ∑ NumRep i =1 e i . We test the performance of several algorithms: ProjUnit (Subsection 2.1), FastProjUnit (Subsection 2.2), FastProjUnit-corr (Section 3), PrivUnitG [7], CompPrivUnitG [7, 24], PrivHS [19], RePrivHS [19, 24], SQKR [14].

In Figure 2, we plot the error for our ProjUnit algorithms as a function of the communication budget k . We consider a high-dimensional regime where d = 2 15 with a small number of users n = 50 and a bounded communication budget k ∈ [1 , 2000] . Our plot shows that our ProjUnit algorithms obtain the same near-optimal error as PrivUnitG for k as small as 1000 . Moreover, the plots show that the correlated versions of our ProjUnit algorithms obtain nearly the same error.

To translate this into concrete numbers, the transform W can be communicated using a small seed ( ∼ 128 bits) in practice, or using k log d + log 2 d ∼ 20400 bits or less than 3kB for d = 10 6 . Sending the k -dimensional vector of 32-bit floats would need an additional 4kB. Thus the total communication cost is between 4 and 8kB. This can be further reduced by using a compressed version of PrivUnitG in the projected space, which requires the client to send a 128-bit seed. In this setting, the communication cost is a total of 256 bits. Thus in the sequel, we primarily focus on the k = 1000 version of our algorithms.

10 1

10 2

10 3

10 4

10 5

10 6

10

3

10

2

10

1

10 0

10 1

10 2

10 3

PrivUnitG

ProjUnit

FastProjUnit

PrivHS

SQKR

CompPrivUnitG

Time (seconds)

d

10 1

10 2

10 3

10 4

10 5

10 6

10

3

10

2

10

1

10 0

10 1

10 2

10 3

PrivUnitG

ProjUnit

FastProjUnit

PrivHS

SQKR

CompPrivUnitG

Time (seconds)

d

(a)

(b)

0

2

4

6

8

0.70

0.75

0.80

0.85

0.90

MNIST (epsilon = 10)

PrivUnitG

FastProjUnit (k = 1000)

FastProjUnit-corr (k = 1000)

Test accuracy

Epoch

Next, we compare the performance of our low-communication algorithms against existing lowcommunication algorithms: PrivHS and SQKR. In Figure 1, we plot the error as a function of the privacy parameter for each algorithm using the best choice of k (bound on communication) for each algorithm. In particular, we choose k = ε for SQKR, num repetitions R = ε/ 2 for repeated PrivHS, k = 1000 for ProjUnit and FastProjUnit. Moreover, in this experiment we set n = 1 and NumRep = 50 to estimate the variance of each method. The figure shows that PrivHS and SQKR, while having low-communication complexity, suffer a significantly worse utility than (nearoptimal) PrivUnitG. On the other side, both our ProjUnit algorithms obtain nearly the same error as PrivUnitG with a bounded communication budget of k = 1000 .

In our third experiment in Figure 3, we plot the runtime of each algorithm as a function of the privacy parameter. Here, we use n = 1 , NumRep = 10 and measure the run-time of each method for different values of the dimension d and privacy parameter ε , allowing each method to run for 1 hour before interrupting. As expected from our theoretical analysis, the runtime of ProjUnit using random rotations is noticeably slower than the (high communication cost) PrivUnitG. However, our SRHT-based FastProjUnit is substantially faster and has a comparable run-time to PrivUnitG. Moreover, for large ε and d , the run-time of compressed PrivUnitG becomes too costly compared to our algorithms due to the e ε d time complexity.

4.2 Private federated learning

Having demonstrated the effectiveness of our methods for private mean estimation, in this section we illustrate the improvements offered by our algorithms for private federated learning. Similarly to the experimental setup in [13], we consider the MNIST [31] dataset and train a convolutional network (see Table 2) using DP-SGD [1] with 10 epochs and different sub-routines for privately estimating the mean of gradients at each batch. In order to bound the sensitivity, we clip the gradients to have ℓ 2 -norm 1 , and run DP-SGD with batch size of 600 , step-size equal to 0 . 1 , and momentum of 0 . 5 .

Figure 4 shows our results for this experiment, where we plot the test accuracy as a function of the epoch for each method. The plots demonstrate that our ProjUnit algorithms obtain similar performance to PrivUnitG, and better performance than the Gaussian mechanism or PrivHS. For the Gaussian mechanism, we set δ = 10 -5 and add noise to satisfy ( ε, δ ) -DP using the analysis in [8]. We did not run SQKR in this experiment as it is not sufficiently computationally efficient for this experiment and has substantially worse performance in the experiments of the previous section. We also did not run the MVU mechanism [13] as their experiments show that it is worse than the Gaussian mechanism which has worse performance than our methods.

Finally, our private algorithms obtain accuracy roughly 91%, whereas the same model trained without privacy obtains around 98%. This degradation in accuracy is mostly due to the choice of the optimization algorithm (DP-SGD with clipping); indeed, even without adding any noise, DP-SGD with clipping achieves around 91% accuracy, suggesting that other private optimization algorithms with different clipping strategies (e.g. [34, 6]) may tighten this gap further. As this is not the main focus of our work, we leave this investigation to future work.

Acknowledgments

HLN is supported by NSF CAREER grant 1750716 and NSF grant 2311649.

References

Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 23rd Annual ACM Conference on Computer and Communications Security (CCS) , pages 308-318, 2016.

Naman Agarwal, Ananda Theertha Suresh, Felix Xinnan X Yu, Sanjiv Kumar, and Brendan McMahan. cpsgd: Communication-efficient and differentially-private distributed sgd. In Proceedings of the 31nd Annual Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Nir Ailon and Bernard Chazelle. The fast Johnson-Lindenstrauss transform and approximate nearest neighbors. SIAM Journal on Computing , 39(1):302-322, 2009.

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30, pages 1709-1720. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 6c340f25839e6acdc73414517203f5f0-Paper.pdf .

N. Alon and J. H. Spencer. The Probabilistic Method . Wiley-Interscience, second edition, 2000.

Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 383-392, 2021.

Hilal Asi, Vitaly Feldman, and Kunal Talwar. Optimal algorithms for mean estimation under local differential privacy. In Proceedings of the 39th International Conference on Machine Learning (ICML) , 2022.

Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 394-403. PMLR, 2018.

Abhishek Bhowmick, John Duchi, Julien Freudiger, Gaurav Kapoor, and Ryan Rogers. Protection against reconstruction and its applications in private federated learning. arXiv:1812.00984 [stat.ML] , 2018.

Andrea Bittau, Úlfar Erlingsson, Petros Maniatis, Ilya Mironov, Ananth Raghunathan, David Lie, Mitch Rudominer, Ushasree Kode, Julien Tinnes, and Bernhard Seefeld. Prochlo: Strong privacy for analytics in the crowd. In Proceedings of the 26th Symposium on Operating Systems Principles , SOSP '17, page 441-459, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450350853. doi: 10.1145/3132747.3132769. URL https: //doi.org/10.1145/3132747.3132769 .

Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy-preserving machine learning. In Proceedings of the Annual ACM SIGSAC Conference on Computer and Communications Security (CCS) , pages 1175-1191, 2017.

Kamalika Chaudhuri, Claire Monteleoni, and Anand D. Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12:1069-1109, 2011.

Kamalika Chaudhuri, Chuan Guo, and Mike Rabbat. Privacy-aware compression for federated data analysis. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence , volume 180, pages 296-306. PMLR, 2022. URL https://proceedings.mlr. press/v180/chaudhuri22a.html .

Wei-Ning Chen, Peter Kairouz, and Ayfer Özgür. Breaking the communication-privacy-accuracy trilemma. In Proceedings of the 33rd Annual Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Albert Cheu, Adam Smith, Jonathan Ullman, David Zeber, and Maxim Zhilyaev. Distributed differential privacy via shuffling. In Yuval Ishai and Vincent Rijmen, editors, Advances in Cryptology - EUROCRYPT 2019 , pages 375-403, Cham, 2019. Springer International Publishing. ISBN 978-3-030-17653-2.

Michael B. Cohen, Jelani Nelson, and David P. Woodruff. Optimal approximate matrix product in terms of stable rank. In Proceedings of the 41st International Colloquium on Automata, Languages and Programming (ICALP) , pages 11:1-11:14, 2016. Full version at https: //arxiv.org/abs/1507.02268v3 .

K.R. Davidson and Stanislaw Szarek. Local operator theory, random matrices and banach spaces. Handbook on the Geometry of Banach spaces, Vol. 1 , pages 317-366, 01 2003.

John Duchi and Ryan Rogers. Lower bounds for locally private estimation via communication complexity. In Proceedings of the 32nd Annual Conference on Learning Theory (COLT) , pages 1161-1191, 2019.

John C. Duchi, Michael I. Jordan, and Martin J. Wainwright. Minimax optimal procedures for locally private estimation. Journal of the American Statistical Association , 113(521):182-215, 2018.

Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology (EUROCRYPT 2006) , 2006.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Theory of Cryptography Conference , pages 265-284, 2006.

Ulfar Erlingsson, Vitaly Feldman, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Abhradeep Thakurta. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth ACM-SIAM Symposium on Discrete Algorithms (SODA) , 2019.

Fartash Faghri, Iman Tabrizian, Ilia Markov, Dan Alistarh, Daniel M. Roy, and Ali RamezaniKebrya. Adaptive gradient quantization for data-parallel sgd. In Advances in Neural Information Processing Systems , volume 33, 2020.

Vitaly Feldman and Kunal Talwar. Lossless compression of efficient private local randomizers. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 3208-3219. PMLR, 2021.

Vitaly Feldman, Audra McMillan, and Kunal Talwar. Hiding among the clones: A simple and nearly optimal analysis of privacy amplification by shuffling. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 954-964, 2022. doi: 10.1109/ FOCS52979.2021.00096. arXiv:2012.12803 [cs.LG].

Venkata Gandikota, Daniel Kane, Raj Kumar Maity, and Arya Mazumdar. vqsgd: Vector quantized stochastic gradient descent. arXiv preprint arXiv:1911.07971 , 2019.

Antonious M. Girgis, Deepesh Data, Suhas Diggavi, Peter Kairouz, and Ananda Theertha Suresh. Shuffled model of federated learning: Privacy, communication and accuracy trade-offs, 2020.

Parikshit Gopalan, Daniek Kane, and Raghu Meka. Pseudorandomness via the discrete fourier transform. pages 903-922, 10 2015. doi: 10.1109/FOCS.2015.60.

Jakub Koneˇ cn` y and Peter Richtárik. Randomized distributed mean estimation: Accuracy vs. communication. Frontiers in Applied Mathematics and Statistics , 4:62, 2018.

Pravesh K. Kothari and Raghu Meka. Almost optimal pseudorandom generators for spherical caps: Extended abstract. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing , STOC '15, page 247-256, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450335362. doi: 10.1145/2746539.2746611. URL https://doi.org/10.1145/2746539.2746611 .

Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database, 1998. URL http://yann.lecun.com/exdb/mnist . ATT Labs [Online].

P. Mayekar and H. Tyagi. Limits on gradient compression for stochastic optimization. In 2020 IEEE International Symposium on Information Theory (ISIT) , pages 2658-2663, 2020. doi: 10.1109/ISIT44484.2020.9174075.

Noam Nisan. Pseudorandom generators for space-bounded computation. Combinatorica , 12: 449-461, 1992.

Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X. Yu, Sashank J. Reddi, and Sanjiv Kumar. AdaCliP: Adaptive clipping for private SGD. arXiv:1908.07643 [cs.LG] , 2020.

Tamás Sarlós. Improved approximation algorithms for large matrices via random projections. In Proceedings of the 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS) , pages 143-152, 2006.

Jeanette P. Schmidt, Alan Siegel, and Aravind Srinivasan. Chernoff-hoeffding bounds for applications with limited independence. SIAM Journal on Discrete Mathematics , 8(2):223-250, 1995. doi: 10.1137/S089548019223872X.

Ananda Theertha Suresh, Felix X. Yu, Sanjiv Kumar, and H. Brendan McMahan. Distributed mean estimation with limited communication. In Proceedings of the 34th International Conference on Machine Learning (ICML) , 2017.

Shay Vargaftik, Ran Ben-Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzenmacher. Drive: One-bit distributed mean estimation. Proceedings of the 34nd Annual Conference on Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Shay Vargaftik, Ran Ben-Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben Itzhak, and Michael Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning. In Proceedings of the 39th International Conference on Machine Learning (ICML) , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an efficient distributed mean estimation framework, which first reduces local data dimension using (different) random projections and then applies an existing private processor (Asi et al [7]) on the projected data before sending them to the server. Further speedups of the framework are also presented, using a more efficient projection matrix and correlated sampling. Utility bounds of the framework and its speedups are derived, and experimental results on both synthesis and real-world data sets are presented.\n\nSTRENGTHS:\n[1] Accelerating private distributed mean estimation is a useful topic. \n\n[2] The paper (especially, narrative) is beautifully written. \n\n[3] Utility bounds are derived.\n\nWEAKNESSES:\n[1] Novelty of the proposed framework is a bit limited, as it basically applies random projection (in a non-traditional but still considerably straightforward way) on PrivUnitG (Asi et al [7]) to speed up the latter. \n\n[2] Comparison of efficiency to the naive PrivUnitG in Table 2 is missing and should be added. Importantly, I suspect the proposed framework does not significantly improve over PrivUnitG, because it simply applies PrivUniteG on fewer features -- this is partly evidenced in the experimental results (Figure 3), where PrivUnitG appears more efficient. \n\n[3] There is a potential error in the proof of Theorem 1 that may make the result invalid. \n\nIn line 486 (supplementary), equation (i) is argued to follow because $E[\\hat{u}] = u$, but such argument does not make sense to me. Consider a simple case when n = 1, then the equation basically argues $E[||A + B||_2^2] = E[||A||_2^2] + E[||B||_2^2]$. I don't see how this can fly... Can authors elaborate? \n\n\n---------- Update ----------\n\nI've read the author responses and my concerns [2][3] are well-addressed. I have raised my score accordingly.\n\nQUESTIONS:\nCould authors respond to my comments [3] and [2] in the Weakness section?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a fast algorithm for private mean estimation under communication constraints. The proposed scheme primarily relies on random projection, and the algorithm enhances speed through the utilization of Hadamard matrix and sampling techniques.\n\nSTRENGTHS:\n- Theorem 1 offers a theoretical analysis of the utility achieved when random projection is employed.\n\n- It is intriguing to observe that the combination of SRHT (Hadamard matrix + Rademacher diagonal matrix) achieves results close to the optimal.\n\n- The idea of enhancing performance through correlated sampling appears to be a compelling approach.\n\nWEAKNESSES:\n1. I believe the current work primarily focuses on the tradeoff between utility and complexity, where the communication budget is relatively higher compared to other algorithms like SQKR. For instance, SQKR has a communication budget of epsilon (~10), while the proposed scheme has k on the order of 10^2 to 10^3 (the communication budget k\\log^2 d is even larger).\n\n2. In Figure 1, comparing the proposed scheme with a communication budget of k\\log^2 d to SQKR with a communication budget of epsilon may appear unfair (despite being the optimal choices for their respective algorithms). To properly compare with SQKR, I suggest the authors provide two separate figures: one with a communication budget of k and another with a communication budget of k\\log^2 d.\n\nQUESTIONS:\n- Could you please specify the exact value of k used in Figure 1?\n\n- As both SQKR and the current work utilize Kashin's representation, which can be approximated using an orthogonal matrix, it might be worth exploring the possibility of a unified framework that encompasses both algorithms.\n\n- Does the LDP constraint automatically hold due to the inclusion of PrivUnit in the projected space?\nEven in correlated sampling?\n\nLIMITATIONS:\nAlthough the proposed work is impressive in terms of complexity, the communication budget is order-wise suboptimal compared to previous work (e.g., SQKR).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of locally private mean estimation, and design algorithms with optimal (upto $1+o(1)$ factor) error with better communication complexity and runtime. The key idea behind improved communication complexity and runtime is (a certain) random projection based pre-processing step.\n\nSTRENGTHS:\n1. The problem of private mean estimation is an important problem in itself and moreover, a lot of interest is also derived from its application to federated learning. Hence, there has been a lot of work recently on devising \"better\" algorithms for it in various settings. The paper identifies gaps in our understanding of the problem, mainly on obtaining optimal error with better communication complexity and runtime, and make solid contributions towards them.\n\n2. The paper is written well. Instead of presenting the final result first, it gradually builds towards conveying the challenges and key ideas.\n\n3. The experiments are encouraging, testing various aspects and comparison with many known methods. Further, the experiments section is well-organized and outlined.\n\nWEAKNESSES:\n1. The underlying key technique -- random-projection to smaller dimensions -- is simple and has been used in the prior works for this problem (albeit without the privUnit). Further, even improvements gained via Subsampled Randomized Hadamard Transform follow standard ideas used in prior works. I encourage the authors to highlight if there are significant differences between this paper and the prior works.\n\n2. While I enjoyed reading Section 2 where the authors gradually build towards the main results and various instantiations, I found Section 3 to be short. It does not provide intuition on how correlated sample is beneficial. Similarly, Remark 2 references unbiased versions of the proposed procedures, deferred to Appendix A.  I encourage the authors to expand on these, if possible. \n\nMinor: Table 2 referenced is missing in the main text.\n\nQUESTIONS:\nPlease respond to the weaknesses. Another simple questions is: why not use same $W$ in every client? Then the server won't have apply the inverse transformation (matrix multiplication) for each client. Is there some fundamental reason this isn't done?\n\nLIMITATIONS:\nThe paper focuses on obtaining optimal (upto upto $1+o(1)$ factor) error for worst-case instances (minimax risk). Consequently, methods which achieve optimal error but with (large) constant factor are deemed inferior. However, in practice, the instances are usually not worst-case but are rather structured (eg: sparse) and therein non-optimal methods may fare better (unlesss there is something fundamental about the achieving worse error regardless of the instance). Therefore, more instance-specific guarantees may be needed to reflect improvement in practice. This is not to say that worst-case error (which is studied in this paper) isn't interesting and important.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of performing mean estimation for worst-case data under local differential privacy. The precise setting considered involves $n$ users, each of whom has one data vector $v_i$ (which, without loss of generality, can be assumed to exist within the unit ball). The users must privatize their data and send them to a central server which tries to estimate $\\frac{1}{n} \\sum_i v_i$ as accurately as possible using the privatized data. The older work of Asi et. al. showed that the PrivUnit algorithm of Bhowmick et. al. attains the optimal error rate among all unbiased estimators. However, implementing that approach leads to concerns regarding the number of bits that need to be exchanged, as well as the runtime, both from the perspective of the clients and the server. The present work tries to address these concerns by using an approach relying on random projections. In particular, the users embed their data into a low-dimensional sub-space of the original space and privatize the resulting vectors. This results in a significantly smaller number of bits being exchanged, thus addressing the issue of potential constraints in communication. In greater detail, the authors give two different algorithms, both of which fit into the above framework. The first algorithm involves each user randomly sampling their own projection matrix, which they must communicate to the server. On the other hand, the second algorithm involves the server choosing a matrix which they communicate to the clients, and then each client has to transform the matrix using their own sampling matrix. For both approaches, the overhead in the error is small ($1 + o(1)$ specifically), while the latter approach manages to significantly reduce not just the communication between servers, but also the runtime for both the clients and the server. The theoretical results are complemented by experiments which support them.\n\nSTRENGTHS:\nThe paper is very well written and gives a complete picture, covering both theoretical aspects and empirical evaluation. Additionally, the coverage of the prior work in the introduction is quite thorough, so the results are clearly positioned within the overall literature. Considering the computational aspects (in addition to the statistical ones) of performing mean estimation under local differential privacy is a very well motivated question, which the paper does a good job of addressing by giving algorithms that have significant improvements compared to prior work. For that reason, I believe that it is of interest to the NeurIPS community.\n\nWEAKNESSES:\nNo obvious weaknesses.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nNo obvious negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}