Title: Privacy Amplification via Compression: Achieving the Optimal Privacy-Accuracy-Communication Trade-off in Distributed Mean Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy

Federated Learning

Communication
Abstract: Privacy and communication constraints are two major bottlenecks in federated learning (FL) and analytics (FA). We study the optimal accuracy of mean and frequency estimation (canonical models for FL and FA respectively) under joint communication and $(\varepsilon, \delta)$-differential privacy (DP) constraints. We consider both the central and the multi-message shuffled DP models. We show that in order to achieve the optimal $\ell_2$ error under $(\varepsilon, \delta)$-DP, it is sufficient for each client to send $\Theta\left( n \min\left(\varepsilon, \varepsilon^2\right)\right)$ bits for FL %{\color{blue}(assuming the dimension $d \gg n \min\left(\varepsilon, \varepsilon^2\right)$)} 
and $\Theta\left(\log\left( n\min\left(\varepsilon, \varepsilon^2\right) \right)\right)$ bits for FA to the server, where $n$ is the number of participating clients.  Without compression, each client needs $O(d)$ bits and $O\left(\log d\right)$ bits for the mean and frequency estimation problems respectively (where $d$ corresponds to the number of trainable parameters in FL or the domain size in FA), meaning that we can get significant savings in the regime $ n \min\left(\varepsilon, \varepsilon^2\right)  = o(d)$, which is often the relevant regime in practice. 

We propose two different ways to leverage compression for privacy amplification and achieve the optimal privacy-communication-accuracy trade-offs. In both cases, each client communicates only partial information about its sample and we show that privacy is amplified by randomly selecting the part contributed by each client. In the first method, the random selection is revealed to the server, which results in a central DP guarantee with optimal privacy-communication-accuracy trade-offs.  In the second method, the random data parts from the clients are  shuffled by a secure shuffler resulting in a multi-message shuffling scheme with the same optimal trade-offs. As a result, we establish the optimal three-way trade-offs between privacy, communication, and accuracy for both the central DP and multi-message shuffling frameworks.

Privacy Amplification via Compression: Achieving the Optimal Privacy-Accuracy-Communication Trade-offs in Distributed Mean Estimation

Abstract

Privacy and communication constraints are two major bottlenecks in federated learning (FL) and analytics (FA). We study the optimal accuracy of mean and frequency estimation (canonical models for FL and FA respectively) under joint communication and ( ε, δ ) -differential privacy (DP) constraints. We consider both the central and the multi-message shuffled DP models. We show that in order to achieve the optimal ℓ 2 error under ( ε, δ ) -DP, it is sufficient for each client to send Θ ( n min ( ε, ε 2 )) bits for FL and Θ ( log ( n min ( ε, ε 2 ))) bits for FA to the server, where n is the number of participating clients. Without compression, each client needs O ( d ) bits and O (log d ) bits for the mean and frequency estimation problems respectively (where d corresponds to the number of trainable parameters in FL or the domain size in FA), meaning that we can get significant savings in the regime n min ( ε, ε 2 ) = o ( d ) , which is often the relevant regime in practice.

We propose two different ways to leverage compression for privacy amplification and achieve the optimal privacy-communication-accuracy trade-offs. In both cases, each client communicates only partial information about its sample and we show that privacy is amplified by randomly selecting the part contributed by each client. In the first method, the random selection is revealed to the server, which results in a central DP guarantee with optimal privacy-communication-accuracy trade-offs. In the second method, the random data parts from the clients are shuffled by a secure shuffler resulting in a multi-message shuffling scheme with the same optimal tradeoffs. As a result, we establish the optimal three-way trade-offs between privacy, communication, and accuracy for both the central DP and multi-message shuffling frameworks.

1 Introduction

In the basic setting of federated learning (FL) [68, 64, 61] and analytics (FA), a server wants to execute a specific learning or analytics task on raw data that is kept on clients' devices. Consider, for example, model updates in FL or histogram estimation in FA, both of which can be modeled as a distributed mean estimation problem. Clients communicate targeted messages to the server and the privacy of the users' data is ensured (in terms of explicit differential privacy (DP) [38] guarantees) by adding carefully calibrated noise to the computed mean at the server before releasing it to downstream modules (e.g., the server computes the average model update and corrupts it with the addition of

noise). This is called the trusted server or central DP model, as it entrusts the central server with privatization and is one of the most common ways in which federated learning and analytics are implemented today 1 .

In this paper, we ask the following question: given that the server needs to privatize the mean, can the clients communicate 'less information' to the server? More precisely, can we leverage the fact that the server only needs to output a noisy (approximate) estimate of the mean to reduce the communication load without sacrificing accuracy? In recent years, there has been significant interest in the central DP model [1] as well as communication efficiency and privacy for FL and FA under different models, including local DP [78, 63, 58, 80, 17, 6, 16, 32], shuffle [40, 43] and distributed DP [9, 59, 8, 33, 34]; however, this basic question has remained open.

One natural way to reduce communication is to have clients communicate only partial information about their samples. For example, in the case of model updates, each client can update only a subset of the model coefficients. In histogram estimation, information about a client's sample can be 'split' into multiple parts, and the client can communicate only one part. However, this results in less information at the server, or effectively fewer samples to estimate the target quantity, e.g., each model coefficient is now updated only by a subset of the clients. A quick calculation reveals that this increases the sensitivity of the estimate to each user's sample and therefore requires the addition of larger noise at the server to achieve the same privacy level. Hence, reducing communication reduces accuracy for the same privacy guarantee.

Wecircumvent this challenge with a simple but insightful observation: when each client communicates only partial information about its sample, we can amplify privacy by randomly selecting the part contributed by each client. This random selection is hidden from a downstream module which has only access to the estimate revealed by the server, which leads to privacy amplification. Privacy amplification by subsampling has been studied in [66, 14] but usually refers to the selection of a random subset of the clients (from a larger pool of available clients). In our case, it is the "piece of information" that is randomly selected at each client. We call this gain privacy amplification via compression as it emerges from the fact that each client does not fully communicate its sample. We use it to establish the optimal communication-privacy-accuracy trade-off for central DP. Note that this same type of gain cannot be leveraged in the local DP model where the server is untrusted. The server needs to know the random selection at each client to construct an estimate. Indeed, in the local DP model, the privacy-accuracy trade-off is known to be significantly worse than the central DP model (see Table 1).

This naturally leads to a follow-up question: can we leverage privacy amplification via compression and achieve the same three-way trade-off by using secure aggregation [33] and shuffling [40] type models which hide information from the server? For secure aggregation, the three-way trade-off has been studied in [31] and the communication cost is significantly larger than the communication cost for central DP proved in this paper (see Table 1). For shuffling, the optimal communication cost has been posed as an open problem in [31]. We resolve this problem by showing that the optimal central DP trade-off can also be achieved with a multi-message shuffling scheme, establishing the optimal communication cost. (We note that a similar result has been concurrently and independently proved in [54] under the shuffle DP setting.) As before, our scheme leverages a privacy amplification gain. Each client communicates partial information about its sample; the identity of the message is erased by the secure shuffler, and hence the untrusted server does not know which part is contributed by each client. We show that to achieve the optimal trade-off, it is critical for each client to split its information into multiple messages and employ multiple shuffling rounds by carefully splitting the privacy budget across different rounds. In contrast, the linearity of secure aggregation requires all participating clients to communicate consistent information (same parts), hence precluding privacy amplification by compression. See Table 1 for a detailed comparison.

Our contributions. We study distributed mean and frequency estimation as canonical building blocks for FL and FA. We consider both the central DP and the multi-message shuffling models. We characterize the order-optimal privacy-accuracy-communication trade-offs for distributed mean estimation and provide an achievable scheme for frequency estimation (in Appendix A) under the central DP model. Our results reveal that privacy and communication efficiency can be achieved simultaneously with no additional penalty for accuracy. In particular, we show that ˜ O ( n min ( ε, ε 2 ))

and ˜ O ( log ( n min ( ε, ε 2 ))) bits of (per-client) communication are sufficient to achieve the orderoptimal error under ( ε, δ ) -privacy for mean and frequency estimation respectively, where n is the number of participating clients. Without compression, each client needs O ( d ) bits and log d bits for the mean and frequency estimation problems respectively (where d is the number of trainable parameters in FL or the domain size in FA), which means that we can get significant savings in the regime nε 2 = o ( d ) (assuming ε = O (1) ). We note that this is often the relevant regime not only for cross-silo but also for cross-device FL/FA. For instance, in practical FL, d usually ranges from 10 6 to 10 9 , and n , the per-epoch sample size, is usually much smaller (e.g., of the order of 10 3 to 10 5 ). For distributed mean estimation, we show that the central DP trade-off can also be achieved with a multi-message shuffling scheme (within a log d factor in communication cost). Hence our paper establishes the three-way trade-off between privacy, communication, and accuracy for both the central DP and multi-message shuffling frameworks, both of which were open problems in the prior literature. Compared with local DP where 1 bit is sufficient when ε = O (1) , this shows that central/shuffling DP has a larger communication cost but can achieve much smaller error (by a factor of n ) and hence is usually preferable in practical applications. Compared with distributed DP where the server aggregates local (encoded) messages with secure multi-party computation (e.g., [23, 8, 34]), we can improve the communication cost by a factor of n , therefore showing that the communication cost can be reduced with a trusted server or shuffler. We summarize the comparisons of our main results to local and distributed DP in Table 1.

Notation. Throughout this paper, we use [ m ] to denote the set of { 1 , ..., m } for any m ∈ N . Random variables (vectors) ( X 1 , ..., X m ) are denoted as X m . We also make use of Bachmann-Landau asymptotic notation, i.e., O,o, Ω , ω, and Θ .

2 Problem Formulation

We first present the distributed mean estimation (DME) [74] problem under differential privacy. Note that DME is closely related to federated learning with SGD (or similar stochastic optimization methods, such as FedAvg [68]), where in each iteration, the server updates the global model by a noisy mean of the local model updates. This noisy estimate is typically obtained by using a DME scheme, and thus one can easily build a distributed DP-SGD scheme (and hence a private FL scheme) from a differentially private DME scheme. Moreover, as shown in [49], as long as we have an unbiased estimate of the gradient at each round, the convergence rates of SGD (or DP-SGD) depend on the ℓ 2 estimation error.

Distributed mean estimation. Consider n clients each with local data x i ∈ R d that satisfies ∥ x i ∥ 2 ≤ C for some constant C > 0 (one can think of x i as a clipped local gradient). A server wants to learn an estimate ˆ µ of the mean µ ( x n ) ≜ 1 n ∑ i x i from x n = ( x 1 , . . . , x n ) after communicating with the n clients. Toward this end, each client locally compresses x i into a b -bit message Y i = enc i ( x i ) ∈ Y through a local encoder enc i : X ↦→ Y (where |Y| ≤ 2 b and sends it to the central server, which upon receiving Y n = ( Y 1 , . . . , Y n ) computes an estimate ˆ µ = dec ( Y n ) that satisfies the following differential privacy:

Definition 2.1 (Differential Privacy) . The mechanism ˆ µ is ( ε, δ ) -differentially private if for any neighboring datasets x n := ( x 1 , ..., x i , ..., x n ) , x ′ n := ( x 1 , ..., x ′ i , ..., x n ) , and measurable S ⊆ Y ,

where the probability is taken over the randomness of ˆ µ .

Our goal is to design schemes that minimize the ℓ 2 2 estimation error:

subject to b -bit communication and ( ε, δ ) -DP constraints.

Distributed frequency estimation. Similarly, frequency estimation can also be formulated as a mean estimation problem but with sparse (one-hot) vectors. Let each user i hold an item x i in a size d domain X . The server aims to estimate the histogram of the n items. Without loss of generality, we can assume that X := { e 1 , ..., e d } ∈ { 0 , 1 } d (where e j is the j -th standard basis vector in R d ), i.e., each item is expressed as a one-hot vector. Then, the histogram of the n items can be expressed as π ( x n ) := ∑ i ∈ [ n ] x i . Similar to the mean estimation problem, clients locally compute and then send Y i = enc i ( x i ) ∈ Y (for some Y such that |Y| ≤ 2 b ), and the central server computes the estimate ˆ π = dec ( y n ) . Our goal is to design schemes that minimize the ℓ 2 2 or ℓ 1 error 2 :

subject to communication and DP constraints (where ∥·∥ can be ℓ 1 or ℓ 2 2 ).

3 Related Works

Federated learning and distributed mean estimation. Federated learning [64, 68, 60] emerges as a decentralized machine learning framework that provides data confidentiality by retaining clients' raw data on edge devices. In FL, communication between clients and the central server can quickly become a bottleneck [68], so previous works have focused on compressing local model updates via gradient quantization [68, 10, 48, 74, 79, 77, 24], sparsification [18, 56, 41]. To further enhance data security, FL is often combined with differential privacy [38, 1, 9]. Among these works, [ ? ] also employs gradient sparsification (or gradient subsampling) to reduce the problem dimensionality. However, the sparsification takes place after the aggregation of local gradients, so the randomness introduced during sparsification cannot be leveraged to amplify the differential privacy guarantee. As a result, this approach leads to a suboptimal trade-off between privacy and communication compared to our scheme.

Note that in this work, we consider FL (or more specifically, the distributed mean estimation) under a central -DP setting where the server is trusted, which is different from the local DP model [63, 37, 70, 76, 22, 32] and the distributed DP model with secure aggregation [23, 21, 59, 8, 33, 34].

A key step in our mean estimation scheme is pre-processing the local data via Kashin's representation [67]. While various compression schemes, based on quantization, sparsification, and dithering have been proposed in the recent literature, Kashin's representation has also been explored in a few works for communication efficiency [47, 73, 29, 71] and for LDP [42] and is particularly powerful in the case of joint communication and privacy constraints as it helps spread the information in a vector evenly in every dimension.

Distributed frequency estimation and heavy hitters. Distributed frequency estimation (a.k.a. histogram estimation) is another canonical task that has been heavily studied under a distributed setting with DP. Prior works either focus on 1) the local DP model with or without communication constraints, e.g., [20, 19, 25, 26, 57] (under an ℓ ∞ loss for heavy hitter estimation) and [58, 80, 76, 6, 5, 32, 46, 72, 45] (under an ℓ 1 or ℓ 2 loss), or 2) the central DP model without communication constraints [38, 52, 65, 27, 13, 81, 36]. As suggested in [37, 3, 2, 4, 16], compared to central DP, local DP models usually incur much larger estimation errors and can significantly decrease the utility. In this work, we consider central DP but with explicit communication constraints.

Local DP with shuffling. A recent line of works [40, 35, 12, 43, 50, 51] considers shuffle -DP, showing that one can significantly boost the central DP guarantees by randomly shuffling local (privatized) messages. In this work, we show that the same shuffling technique can be used to achieve the optimal central DP error with nearly optimal communication cost. Therefore, we can obtain the same level of central DP with small communication costs while weakening the security assumption: achieving the optimal communication cost (under central DP) only requires a secure shuffler (as opposed to a fully trusted central server).

4 Distributed Mean Estimation

In this section, we present a mean estimation scheme that achieves the optimal ˜ O δ ( C 2 d n 2 ε 2 ) error under ( ε, δ ) -DP while only using ˜ O ( nε 2 ) bits of per-client communication.

We first consider a slightly simpler, discrete setting with ℓ ∞ geometry (as opposed to the ℓ 2 mean estimation stated in Section 2): assume each client observes x i ∈ {-c, c } d where c > 0 is a constant, and a central server aims to estimate the mean µ ( x n ) := 1 n ∑ n i =1 x i by minimizing the ℓ 2 2 error subject to the privacy and communication constraints. We argue later that solutions to the above ℓ ∞ problem can be used for ℓ 2 mean estimation by applying Kashin's representation.

To solve the aforementioned ℓ ∞ mean estimation problem, first observe that each client's local data can be expressed in d bits since each coordinate of x i can only take values in { c, -c } . To reduce the communication load to o ( d ) bits, each client adopts the following subsampling strategy: for each coordinate j ∈ [ d ] , client i chooses to send x i ( j ) to the server with probability γ . We assume that this subsampling step is performed with a seed shared by the client and the server 3 , hence the server knows which coordinates are communicated by each client. Therefore upon receiving the client messages, it can compute the mean of each coordinate and privatize it by adding Gaussian noise. The key observation we leverage is that the randomness in the compression algorithm can be used to amplify privacy or equivalently reduce the magnitude of the Gaussian noise that is needed for privatization. Note that such randomness needs to be kept private from an adversary as the privacy guarantee of the scheme relies on it.

We summarize the scheme in Algorithm 1 and state its privacy and utility guarantees in the following theorem.

Algorithm 1 Coordinate Subsampled Gaussian Mechanism (CSGM)

Theorem 4.1 ( ℓ ∞ mean estimation.) . Let x 1 , ..., x n ∈ {-c, c } d and ε, δ > 0 . There exists a

(1) such that Algorithm 1 is ( ε, δ ) -DP and the ℓ 2 2 estimation error of ˆ µ is at most

In addition, the (average) per-client communication cost is γd = b bits, and Algorithm 1 yields an unbiased estimator of µ .

Remark 4.2 (Unbiasedness) . Note that for mean estimation, we usually want the final mean estimator to be unbiased since standard convergence analyses of SGD [49] require an unbiased estimate of the true gradient in each optimization

Input: users' data x 1 , ..., x n , sampling parameters γ := b/d , DP parameters ( ε, δ ) .

Output: mean estimator ˆ µ .

for user

i

∈

[

n

]

do for

coordinate

j

∈

[

d

]

do

end if end for end for for coordinate j ∈ [ d ] do

Server computes the average ˆ µ j := 1 nγ ∑ i : Z ij =1 x i ( j ) + N (0 , σ 2 ) , where σ 2 is computed according to (1) in Theorem 4.1.

end for Return: ˆ µ := (ˆ µ 1 , ˆ µ 2 , ..., ˆ µ d )

.

round. Given that our proposed mean estimation schemes (Algorithm 1 and Algorithm 2 in the next section) are all unbiased, we can combine them with SGD/federated averaging and readily apply [49] to obtain a convergence guarantee for the resulting communication-efficient DP-SGD.

For the ℓ 2 mean estimation task formulated in Section 2, we pre-process local vectors by first computing their Kashin's representations and then performing randomized rounding [62, 75, 42, 32]. Specifically, if x i has ℓ 2 norm bounded by C , then its Kashin's representation (with respect to a tight frame K ∈ R d × D where D = Θ( d ) ) ˜ x i has bounded ℓ ∞ norm: ∥ ˜ x i ∥ ∞ ≤ c = O ( C √ d ) and satisfies x i = K ˜ x i . This allows us to convert the ℓ 2 geometry to an ℓ ∞ geometry. Furthermore, by randomly rounding each coordinate of ˜ x i to {-c, c } (see for example [32]), we can readily apply Algorithm 1 and obtain the following result for ℓ 2 mean estimation as a corollary:

Corollary 4.3 ( ℓ 2 mean estimation) . Let x 1 , ..., x n ∈ B 2 ( C ) (i.e., ∥ x i ∥ 2 ≤ C for all i ∈ [ n ] ). Then for any ε, δ > 0 , Algorithm 1 combined with Kashin's representation and randomized rounding yields an ( ε, δ ) -DP unbiased estimator for µ with ℓ 2 2 estimation error bounded by

The first term ( α ) in the estimation error in Corollary 4.3 is the error due to compression, and the second term ( β ) is the error due to privatization (which is order-optimal under ( ε, δ ) -DP up to an additional log( d/δ ) factor as we discuss in Section 4.2). In particular, if we ignore the polylogarithmic terms and assume ε = O (1) , the privatization error ( β ) can be simplified to ˜ O ( dC 2 n 2 ε 2 ) , which dominates the total ℓ 2 2 error when b = ˜ Ω δ ( max ( nε 2 , √ dε )) , i.e. in this regime the total ℓ 2 2 error is order-wise equal to the optimal centralized DP error ( β ) . This implies that no more than b = ˜ Ω δ ( max ( nε 2 , √ dε )) bits per client are needed to achieve the order-optimal ℓ 2 2 error.

In the next section, we introduce a modification to Algorithm 1, which allows the removal of the Ω ( √ dε ) term in the communication cost.

4.1 Dimension-free communication cost

In order to remove the dependence on the dimension d in the communication cost b = ˜ Ω δ ( max ( nε 2 , √ dε )) from the previous section, we need to improve the performance of our scheme in the small-sample regime nε 2 = o ( √ dε ) . Equivalently, we want to be able to achieve the centralized DP performance by using only b = ˜ Ω δ ( nε 2 ) bits per client when nε = o ( √ d ) . Assuming ε ≈ 1 , note that this implies that the total communication bandwidth of the system nb = o ( d ) , i.e. the server can receive information about at most nb = n 2 ε 2 = o ( d ) coordinates. We show that in this regime the performance of the scheme can be improved by a priori restricting the server's attention to a subset of the coordinates.

We make the following modification to Algorithm 1: before performing Algorithm 1, the server randomly selects d ′ ≈ O ( min( d, n 2 ε 2 ) ) coordinates and only requires clients to run Algorithm 1 on them. We present the modified scheme in Algorithm 2 and summarize its performance in Theorem 4.4.

Similarly, we can obtain the following ℓ 2 mean estimation via Kashin's representations:

),

such that Algorithm 2 is ( ε, δ ) -DP. In addition, the (average) per-client communication cost is γd = b bits, and the ℓ 2 2 estimation error is at most

Corollary 4.5. As long as b = Ω ( nε 2 log(1 /δ )+ ε ) , the ℓ 2 2 error of mean estimation is

As suggested by Corollary 4.5, we see that when ε = O (1) , b = ˜ Ω ( nε 2 ) bits per client are sufficient to achieve the order-optimal ˜ O δ ( c 2 d n 2 ε 2 ) error (even in the small sample regime n ≤ √ d ), i.e. the communication cost of the scheme is independent of the dimension d .

4.2 Lower bounds

In this section, we argue that the estimation error in Theorem 4.4 is optimal up to an log ( d/δ ) factor. Specifically, Theorem 5.3 of [31] shows that any b -bit unbiased compression scheme will incur Ω ( C 2 d nb ) error for the ℓ 2 mean estimation problem (even when privacy is not required). This matches the first term in (6) up to a logarithmic factor.

On the other hand, the centralized Gaussian mechanism (under a central ( ε, δ ) -DP) achieves O ( C 2 d log(1 /δ ) n 2 ε 2 ) MSE [15] (which is orderoptimal in most parameter regimes; see the lower bounds in Theorem 3.1 of [28] or Proposition 23 of [30]). Hence, we can conclude that

Algorithm 2 CSGMwith Coordinate Pre-selection

Input: users' data x 1 , ..., x n , coordinate selection d ′ ≤ d , sampling parameters γ := b/d ′ , DP parameters ( ε, δ ) .

Output: mean estimator ˆ µ .

Randomly select

d

′

{

1

d

′

j

, ..., j

} ⊂

for user

i

.

[

d

]

[

n

]

∈

do

Pre-processing x i by restricting it on J :

end for

Apply CSGM (Algorithm 1) on x i ( J ) , i ∈ [ n ] : ˆ µ J ← CSGM ( x i ( J ) , i ∈ [ n ]) .

for if

j

∈

j

ˆ

µ

else

ˆ

j

µ

= 0

end if end for

Return:

[

d

]

∈ J

do then

j

= ˆ

J

µ

(

j

)

.

:=

.

d

d

d

d

d

d

′

′

′

1

2

d

.

µ

ˆ

µ

ˆ

,

µ

ˆ

, ...,

µ

ˆ

the total communication received by the server has to be at least Ω( n 2 ε 2 ) bits in order to achieve the same error as the Gaussian mechanism. Therefore, the (average) per-client communication cost has to be at least Ω( nε 2 ) bits. Hence we conclude that Algorithm 2 is optimal (up to a logarithmic factor).

For completeness, we state the communication lower bound in the following theorem:

Theorem 4.6 (Communication lower bound for mean estimation under central DP) . Let x 1 , ..., x n ∈ B 2 ( C ) . Let Y 1 , ..., Y n be any b -bit local reports generated from a (possibly interactive) compressor and be unbiased in the sense that E [ ∑ i Y i ] = ∑ i x i . Then if E [ ∥ ∥ 1 n ∑ i Y i -1 n ∑ i x i ∥ ∥ 2 2 ] ≤ O ( C 2 d log(1 /δ ) n 2 ε 2 ) , it holds that b = Ω ( nε 2 log(1 /δ ) ) .

Finally, we remark that the logarithmic gap between the upper and lower bounds may be due to the specific composition theorem (Theorem III.3 of [39]) we use in our proof, which is simpler to work with but possibly slightly weaker. However, in our experiments, we compute and account for all privacy budgets with Rényi DP [69, 82], and hence can obtain better constants compared to our theoretical analysis.

5 Achieving the Optimal Trade-off via Shuffling

In Section 4 and Section A, we see that the communication cost can be reduced to ( ˜ O ( nε 2 ) for mean estimation and ˜ O ( log ( ⌈ nε 2 ⌉ )) for frequency estimation) while still achieving the order-wise optimal error, as long as the server is trusted . On the other hand, when the server is untrusted, [33, 31] show that optimal error under ( ε, δ ) -DP can be achieved with secure aggregation. However, the communication cost of these schemes is ˜ O ( n 2 ε 2 ) bits per client for mean estimation and ˜ O ( nε ) bits per client for frequency estimation. This corresponds to a factor of n increase for mean estimation and an exponential increase for frequency estimation. In this section, we investigate whether the coordinates

J

:=

(

)

optimal communication-accuracy-privacy trade-off from the previous sections can be achieved when the server is not fully trusted.

In this section, we show that if there exists a secure shuffler that randomly permutes clients' locally privatized messages and releases them to the server, we can achieve the nearly optimal (within a log d factor) central-DP error in mean estimation with ˜ O ( nε 2 ) bits of communication. We note that a similar result has been proven in a concurrent work [54]. Specifically, we present a mean estimation scheme that combines a local-DP mechanism with privacy amplification via shuffling by building on the following recent result [40, 43]:

Lemma 5.1 ([43]) . Let M i be an independent ( ε 0 , 0) -LDP mechanism for each i ∈ [ n ] with ε 0 ≤ 1 and π be a random permutation of [ n ] . Then for any δ ∈ [0 , 1] such that ε 0 ≤ log ( n 16 log(2 /δ ) ) , the mechanism S : ( x 1 , . . . , x n ) ↦→ ( M 1 ( x π (1) ) , . . . , M n ( x π ( n ) )) is ( ε, δ ) -DP for some ε such that ε = O ( ε 0 √ log(1 /δ ) √ n ) .

Privacy analysis. With the above amplification lemma, we only need to design the local randomizers M i that satisfy ε 0 -LDP. Note that the above lemma is only tight when ε 0 = O (1) , thus restricting the (amplified) central ε = O (1 / √ n ) , i.e. to be very small. To accommodate larger ε , users can send different portions of their messages to the server in separate shuffling rounds. Equivalently, we repeat the shuffled LDP mechanism for T = O ( ⌈ nε 2 ⌉ ) rounds while ensuring that in each round, clients communicate an independent piece of information about their sample to the server. More precisely, within each round, each client applies the local randomizers M i with a per-round local privacy budget ε 0 = O (1) and sends an independent message to the server. This results in (amplified) central O (1 / √ n ) -DP per round, which after composition over T = O ( ⌈ nε 2 ⌉ ) rounds leads to ε -DP for the overall scheme as suggested by the composition theorem [58]). We detail the algorithm in Algorithm 4 in Appendix F.

Communication costs. The communication cost of the above T -round scheme can be computed as follows. As shown in [32], the optimal communication cost of an ε 0 -LDP mean estimation is O ( ⌈ ε 0 ⌉ ) bits. In addition, the (private-coin) SQKR scheme proposed in [32] uses O ( ⌈ ε 0 ⌉ log d ) bits of communication (we state the formal performance guarantee for this scheme in Lemma 5.2), where compression is done by subsampling coordinates and privatization is performed with Randomized Response. Therefore, since the per-round ε 0 = O (1) , the total per-client communication cost is O ( nε 2 log d ) , matching the optimal communication bounds in Section 4 within a log d factor.

Lemma 5.2 (SQKR [32]) . For all ε 0 > 0 , b 0 > 0 , there exists a ( ε 0 , 0) -LDP mechanism using b 0 log( d ) bits such that ˆ µ is unbiased and satisfies E [ ∥ µ ( x n ) -ˆ µ ( x n ) ∥ 2 2 ] = O ( c 2 d n min ( ε 2 0 ,ε 0 ,b 0 ) ) .

Finally, we summarize the performance guarantee for the overall scheme (Algorithm 4) in the following theorem.

Theorem 5.3 ( ℓ 2 mean estimation) . Let x 1 , ..., x n ∈ B 2 ( C ) (i.e., ∥ x i ∥ 2 ≤ C for all i ∈ [ n ] ). For all ε > 0 , b > 0 , n > 30 , and δ ∈ ( δ min , 1] where δ min = O ( be -n log( d ) ) , Algorithm 4 combined with Kashin's representation and randomized rounding is ( ε, δ ) -DP, uses no more than b bits of communication, and achieves

Remark 5.4. As opposed to previous schemes Algorithm 1-3, the shuffled SQKR requires some condition on δ , i.e., δ ∈ [ δ min , 1] due to the specific shuffling lemma we used. In practice, however, δ min is small due to the exponential dependence on n . The order-wise optimal error of O ( C 2 d n 2 min( ε 2 ,ε ) ) is achieved, up to logarithmic factors, when b = Ω δ ( n log( d ) min ( ε 2 , ε )) .

Remark 5.5. We note that similar ideas of private mean estimation based on shuffling have been studied before, see, for instance, [53]. However, these papers do not use the above privacy budget splitting trick over multiple rounds, so their result is only optimal when ε is very small. The above scheme can be viewed as a multi-message shuffling scheme [35, 51], and in particular, can be regarded as a generalization of the scalar mean estimation scheme [35] to d -dim mean estimation.

CSGM (n = 500,d = 500,6 =1e-6)

Meanestimation(n=500,d=1000,6=1e-6)

101

y= 0.1 (50 bits)

103

y = 0.2 (100 bits)

100

y= 0.5 (250 bits)

y= 0.8 (400 bits)

102

CSGM (10 bits)

CSGM (20 bits)

10-1

y= 1.0 (500 bits)

101

CSGM (50 bits)

CSGM (100 bits)

CSGM (200 bits)

M

E

S

10-2

MSI

E

100

CSGM (1000 bits)

CSGM (500 bits)

CSGM (800 bits)

Central Gaussian

10-3

10-1

SQKR

Shuffled SQKR (1 round)

DJW

10-4

10-2

Shuffled DjW

Shuffled SQKR (5 rounds)

Shuffled SQKR (10 rounds)

10-5

10-3

Shuffled SQKR (20 rounds)

Shuffled SQKR (50 rounds)

Shuffled SQKR (100 rounds)

0.0

0.5

1.0

1.5

Privacy (ε)

2.0

2.5

3.0

3.5

4.0

0.25

0.50

0.75

Privacy(ε)

1.00

1.25

1.50

1.75

2.00

6 Experiments

In this section, we empirically evaluate our mean estimation scheme (CSGM) from Section 4, examine its privacy-accuracy-communication trade-off, and compare it with other DP mechanisms (including the shuffling-based mechanism introduced in Section 5).

Setup. For a given dimension d , and number of samples n , we generate local vectors X i ∈ R d as follows: let X i ( j ) i.i.d. ∼ 1 √ d (2 · Ber (0 . 8) -1) where Ber (0 . 8) is a Bernoulli random variable with bias p = 0 . 8 . This ensures ∥ X i ∥ ∞ ≤ 1 / √ d and ∥ X i ∥ 2 ≤ 1 , and in addition, the empirical mean µ ( X n ) := 1 n ∑ i X i does not converge to 0 . Note that as our goal is to construct an unbiased estimator, we did not project our final estimator back to the ℓ ∞ or ℓ 2 space as the projection step may introduce bias. Therefore, the ℓ 2 estimation error can be greater than 1 . We account for the privacy budget with Rényi DP [69] and the privacy-amplification by subsampling lemma in [82] and convert Rényi DP to ( ε, δ ) -DP via [30].

Privacy-accuracy-communication trade-off of CSGM. In the first experiment (left of Figure 1), we apply Algorithm 1 with different sampling rates γ , which leads to different communication budgets ( b = γd ). Note that when γ = 1 , the scheme reduces to the central Gaussian mechanism without compression. In Figure1, we see that with a fixed communication budget, CSGM approximates the central (uncompressed) Gaussian mechanism in the high privacy regime (small ε ) and starts deviating from it when ε exceeds a certain value. In addition, that value of ε depends only on sample size n and the communication budget b and not the dimension d as predicted by our theory: recall that the compression error dominates the total error, and hence the performance starts to deviate from the (uncompressed) Gaussian mechanism when b = o ( nε 2 ) , a condition that is independent of d . Observe, for example, that when b = 50 bits, the Gaussian mechanism starts outperforming CSGM at ε ≥ 0 . 5 for both d = 500 and d = 5000 . Hence, for ε ≈ 0 . 5 CSGM is able to provide 10X compression when d = 500 , but 100X compression when d = 5000 without impacting MSE.

Comparison with local and shuffle DP. Next, we compare the CSGM with local and shuffled DP for d = 10 3 and n = 500 . For local DP, we consider the private-coin SQKR scheme introduced in Section 5 which uses ( ⌈ log d ⌉ +1) T = 11 T bits for T shuffling rounds and DJW [37] which is known to be order-optimal when ε = O (1) (but is not communication-efficient). For shuffle-DP, we apply the amplification lemma in [43] to find the corresponding local ε 0 (see Section 5 for more details) and simulate both SQKR and DJW as the local randomizers.

The MSEs of all mechanisms are reported in the right of Figure 1. Our results suggest that for a fixed communication budget (say, 10 bits), the practical performance of CSGM significantly outperforms shuffled-DP mechanisms, including the shuffled SQKR and DJW, eventhough they have the same order-wise guarantees theoretically. In addition, the amplification gain of single-round shuffling diminishes fast as ε increases. Indeed, when ε ≥ 0 . 8 , we observe no amplification gain compared to the pure local DP.

7 Limitations and Future Work

While we have characterized the (nearly) orderwise optimal trade-offs under central and shuffled DP, it remains unclear whether the pre-constants can be further sharpened, potentially enhancing the practical applicability of these approaches. Furthermore, our analysis is based on a classical privacy amplification technique. By employing more advanced accounting methods, it might be possible to shave off the logarithmic factors.

References

M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

J. Acharya, C. L. Canonne, and H. Tyagi. Inference under information constraints II: Communication constraints and shared randomness. arXiv preprint arXiv:1905.08302 , 2019.

J. Acharya, C. L. Canonne, and H. Tyagi. Inference under information constraints: Lower bounds from chi-square contraction. In Conference on Learning Theory , pages 3-17. PMLR, 2019.

J. Acharya, C. L. Canonne, and H. Tyagi. General lower bounds for interactive high-dimensional estimation under information constraints. arXiv preprint arXiv:2010.06562 , 2020.

J. Acharya and Z. Sun. Communication complexity in locally private distribution estimation and heavy hitters. In International Conference on Machine Learning , pages 51-60, 2019.

J. Acharya, Z. Sun, and H. Zhang. Hadamard response: Estimating distributions privately, efficiently, and with little communication. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1120-1129, 2019.

J. Acharya, Z. Sun, and H. Zhang. Differentially private Assouad, Fano, and Le Cam. In Algorithmic Learning Theory , pages 48-78. PMLR, 2021.

N. Agarwal, P. Kairouz, and Z. Liu. The Skellam mechanism for differentially private federated learning. Advances in Neural Information Processing Systems , 34:5052-5064, 2021.

N. Agarwal, A. T. Suresh, F. X. X. Yu, S. Kumar, and B. McMahan. cpSGD: Communicationefficient and differentially-private distributed sgd. In Advances in Neural Information Processing Systems , pages 7564-7575, 2018.

D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efficient sgd via gradient quantization and encoding. In Advances in Neural Information Processing Systems 30 , pages 1709-1720, 2017.

J. Allen, B. Ding, J. Kulkarni, H. Nori, O. Ohrimenko, and S. Yekhanin. An algorithmic framework for differentially private data analysis on trusted processors. Advances in Neural Information Processing Systems , 32, 2019.

V. Balcer and A. Cheu. Separating local & shuffled differential privacy via histograms. arXiv preprint arXiv:1911.06879 , 2019.

V. Balcer and S. Vadhan. Differential privacy on finite computers. arXiv preprint arXiv:1709.05396 , 2017.

B. Balle, G. Barthe, and M. Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in Neural Information Processing Systems , 31, 2018.

B. Balle and Y.-X. Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning , pages 394-403. PMLR, 2018.

L. P. Barnes, W.-N. Chen, and A. Ozgur. Fisher information under local differential privacy. arXiv preprint arXiv:2005.10783 , 2020.

L. P. Barnes, Y. Han, and A. Ozgur. Lower bounds for learning distributions under communication constraints via fisher information, 2019.

L. P. Barnes, H. A. Inan, B. Isik, and A. Ozgur. rTop-k: A statistical estimation approach to distributed SGD, 2020.

R. Bassily, K. Nissim, U. Stemmer, and A. Thakurta. Practical locally private heavy hitters. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 2285-2293, Red Hook, NY, USA, 2017. Curran Associates Inc.

R. Bassily and A. Smith. Local, private, efficient protocols for succinct histograms. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing , STOC '15, page 127-135, New York, NY, USA, 2015. Association for Computing Machinery.

J. H. Bell, K. A. Bonawitz, A. Gascón, T. Lepoint, and M. Raykova. Secure single-server aggregation with (poly) logarithmic overhead. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security , pages 1253-1269, 2020.

A. Bhowmick, J. Duchi, J. Freudiger, G. Kapoor, and R. Rogers. Protection against reconstruction and its applications in private federated learning. arXiv preprint arXiv:1812.00984 , 2018.

K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:1611.04482 , 2016.

M. Braverman, A. Garg, T. Ma, H. L. Nguyen, and D. P. Woodruff. Communication lower bounds for statistical estimation problems via a distributed data processing inequality. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 10111020, 2016.

M. Bun, J. Nelson, and U. Stemmer. Heavy hitters and the structure of local privacy. In Proceedings of the 37th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems , SIGMOD/PODS '18, page 435-447, New York, NY, USA, 2018. Association for Computing Machinery.

M. Bun, J. Nelson, and U. Stemmer. Heavy hitters and the structure of local privacy. ACM Transactions on Algorithms (TALG) , 15(4):1-40, 2019.

M. Bun and T. Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference , pages 635-658. Springer, 2016.

T. T. Cai, Y. Wang, and L. Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. The Annals of Statistics , 49(5):2825-2850, 2021.

S. Caldas, J. Koneˇ cny, H. B. McMahan, and A. Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.07210 , 2018.

C. L. Canonne, G. Kamath, and T. Steinke. The discrete gaussian for differential privacy. arXiv preprint arXiv:2004.00010 , 2020.

W.-N. Chen, C. A. C. Choo, P. Kairouz, and A. T. Suresh. The fundamental price of secure aggregation in differentially private federated learning. In International Conference on Machine Learning , pages 3056-3089. PMLR, 2022.

W.-N. Chen, P. Kairouz, and A. Ozgur. Breaking the communication-privacy-accuracy trilemma. Advances in Neural Information Processing Systems , 33, 2020.

W.-N. Chen, A. Ozgur, G. Cormode, and A. Bharadwaj. The communication cost of security and privacy in federated frequency estimation. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 4247-4274. PMLR, 25-27 Apr 2023.

W.-N. Chen, A. Ozgur, and P. Kairouz. The Poisson binomial mechanism for unbiased federated learning with secure aggregation. In International Conference on Machine Learning , pages 3490-3506. PMLR, 2022.

A. Cheu, A. Smith, J. Ullman, D. Zeber, and M. Zhilyaev. Distributed differential privacy via shuffling. In Annual International Conference on the Theory and Applications of Cryptographic Techniques , pages 375-403. Springer, 2019.

G. Cormode and A. Bharadwaj. Sample-and-threshold differential privacy: Histograms and applications. In International Conference on Artificial Intelligence and Statistics , pages 14201431. PMLR, 2022.

J. C. Duchi, M. I. Jordan, and M. J. Wainwright. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , pages 429-438. IEEE, 2013.

C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

C. Dwork, G. N. Rothblum, and S. Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 51-60. IEEE, 2010.

Ú. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, K. Talwar, and A. Thakurta. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2468-2479. SIAM, 2019.

F. Farokhi. Gradient sparsification can improve performance of differentially-private convex machine learning. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 1695-1700. IEEE, 2021.

V. Feldman, C. Guzman, and S. Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1265-1277. SIAM, 2017.

V. Feldman, A. McMillan, and K. Talwar. Hiding among the clones: A simple and nearly optimal analysis of privacy amplification by shuffling. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 954-964. IEEE, 2022.

V. Feldman, A. McMillan, and K. Talwar. Stronger privacy amplification by shuffling for Rényi and approximate differential privacy. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 4966-4981. SIAM, 2023.

V. Feldman, J. Nelson, H. Nguyen, and K. Talwar. Private frequency estimation via projective geometry. In International Conference on Machine Learning , pages 6418-6433. PMLR, 2022.

V. Feldman and K. Talwar. Lossless compression of efficient private local randomizers. arXiv preprint arXiv:2102.12099 , 2021.

J.-J. Fuchs. Spread representations. In 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR) , pages 814-817. IEEE, 2011.

V. Gandikota, D. Kane, R. K. Maity, and A. Mazumdar. vqSGD: Vector quantized stochastic gradient descent, 2019.

S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

B. Ghazi, N. Golowich, R. Kumar, R. Pagh, and A. Velingker. On the power of multiple anonymous messages. arXiv preprint arXiv:1908.11358 , 2019.

B. Ghazi, R. Kumar, P. Manurangsi, and R. Pagh. Private counting from anonymous messages: Near-optimal accuracy with vanishing communication overhead. In International Conference on Machine Learning , pages 3505-3514. PMLR, 2020.

A. Ghosh, T. Roughgarden, and M. Sundararajan. Universally utility-maximizing privacy mechanisms. SIAM Journal on Computing , 41(6):1673-1693, 2012.

A. Girgis, D. Data, S. Diggavi, P. Kairouz, and A. T. Suresh. Shuffled model of differential privacy in federated learning. In International Conference on Artificial Intelligence and Statistics , pages 2521-2529. PMLR, 2021.

A. M. Girgis and S. Diggavi. Multi-message shuffled privacy in federated learning. arXiv preprint arXiv:2302.11152 , 2023.

Y. Han, A. Özgür, and T. Weissman. Geometric lower bounds for distributed parameter estimation under communication constraints. In Conference On Learning Theory , pages 31633188. PMLR, 2018.

R. Hu, Y. Gong, and Y. Guo. Federated learning with sparsification-amplified privacy and adaptive optimization. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 1463-1469. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.

Z. Huang, Y. Qiu, K. Yi, and G. Cormode. Frequency estimation under multiparty differential privacy: One-shot and streaming. Proc. VLDB Endow. , 15(10):2058-2070, jun 2022.

P. Kairouz, K. Bonawitz, and D. Ramage. Discrete distribution estimation under local privacy. In Proceedings of The 33rd International Conference on Machine Learning , volume 48, pages 2436-2444, New York, New York, USA, 20-22 Jun 2016.

P. Kairouz, Z. Liu, and T. Steinke. The distributed discrete gaussian mechanism for federated learning with secure aggregation. arXiv preprint arXiv:2102.06387 , 2021.

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 , 2019.

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

B. Kashin. Section of some finite-dimensional sets and classes of smooth functions (in russian) izv. Acad. Nauk. SSSR , 41:334-351, 1977.

S. P. Kasiviswanathan, H. K. Lee, K. Nissim, S. Raskhodnikova, and A. Smith. What can we learn privately? SIAM Journal on Computing , 40(3):793-826, 2011.

J. Koneˇ cn` y, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016.

A. Korolova, K. Kenthapadi, N. Mishra, and A. Ntoulas. Releasing search queries and clicks privately. In Proceedings of the 18th international conference on World wide web , pages 171-180, 2009.

N. Li, W. Qardaji, and D. Su. On sampling, anonymization, and differential privacy or, kanonymization meets differential privacy. In Proceedings of the 7th ACM Symposium on Information, Computer and Communications Security , pages 32-33, 2012.

Y. Lyubarskii and R. Vershynin. Uncertainty principles and vector quantization. IEEE Transactions on Information Theory , 56(7):3491-3501, 2010.

H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Arcas. Communication-efficient learning of deep networks from decentralized data (2016). arXiv preprint arXiv:1602.05629 , 2016.

I. Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF) , pages 263-275. IEEE, 2017.

T. T. Nguyên, X. Xiao, Y. Yang, S. C. Hui, H. Shin, and J. Shin. Collecting and analyzing data from smart device users with local differential privacy, 2016.

M. Safaryan, E. Shulgin, and P. Richtárik. Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor. arXiv preprint arXiv:2002.08958 , 2020.

A. Shah, W.-N. Chen, J. Balle, P. Kairouz, and L. Theis. Optimal compression of locally differentially private mechanisms. In International Conference on Artificial Intelligence and Statistics , pages 7680-7723. PMLR, 2022.

C. Studer, W. Yin, and R. G. Baraniuk. Signal representations with minimum ℓ ∞ -norm. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1270-1277. IEEE, 2012.

A. T. Suresh, F. X. Yu, S. Kumar, and H. B. McMahan. Distributed mean estimation with limited communication. In Proceedings of the 34th International Conference on Machine Learning Volume 70 , ICML'17, page 3329-3337. JMLR.org, 2017.

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

T. Wang, J. Zhao, X. Yang, and X. Ren. Locally differentially private data collection and analysis. arXiv preprint arXiv:1906.01777 , 2019.

J. Wangni, J. Wang, J. Liu, and T. Zhang. Gradient sparsification for communication-efficient distributed optimization. In Advances in Neural Information Processing Systems , pages 12991309, 2018.

S. L. Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association , 60(309):63-69, 1965.

W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in neural information processing systems , pages 1509-1519, 2017.

M. Ye and A. Barg. Optimal schemes for discrete distribution estimation under local differential privacy. In 2017 IEEE International Symposium on Information Theory (ISIT) , pages 759-763, June 2017.

W. Zhu, P. Kairouz, B. McMahan, H. Sun, and W. Li. Federated heavy hitters discovery with differential privacy. In International Conference on Artificial Intelligence and Statistics , pages 3837-3847. PMLR, 2020.

Y. Zhu and Y.-X. Wang. Poission subsampled Rényi differential privacy. In International Conference on Machine Learning , pages 7634-7642. PMLR, 2019.

A Distributed Frequency Estimation

In this section, we consider the frequency estimation problem for federated analytics. Recall that for the frequency estimation task, each client's private data x i ∈ { 0 , 1 } d satisfies ∥ x i ∥ 0 = 1 , and the goal is to estimate π := 1 n ∑ i x i by minimizing the ℓ 2 (or ℓ 1 , ℓ ∞ ) error E [ ∥ π -ˆ π ( Y n ) ∥ 2 2 ] subject to communication and ( ε, δ ) -DP constraints. When the context is clear, we sometimes use x i to denote, by abuse of notation, the index of the item, i.e., x i ∈ [ d ] .

To fully make use of the ℓ 0 structure of the problem, a standard technique is applying a Hadamard transform to convert the ℓ 0 geometry to an ℓ ∞ one and then leveraging the recursive structure of Hadamard matrices to efficiently compress local messages.

Specifically, for a given b -bit constraint, we partition each local item x i into 2 b -1 chunks x (1) i , ..., x (2 b -1) i ∈ { 0 , 1 } B , where B := d/ 2 b -1 and x ( j ) i = x i [ B · ( j -1) : B · j -1] . Note that since x i is one-hot, only one chunk of x ( j ) i is non-zero. Then, client i performs the following Hadamard transform for each chunk: y ( ℓ ) i = H B · x ( ℓ ) i , where H B is defined recursively as follows:

Each client then generates a sampling vector Z ij i.i.d. ∼ Bern ( 1 B ) via shared randomness that is also known by the server, and commits ( y (1) i ( j ) , ..., y (2 b -1 ) i ( j )) as its local report. Since ( y (1) i ( j ) , ..., y (2 b -1 ) i ( j )) only contains a single non-zero entry that can be 1 √ B or -1 √ B , the local report can be represented in b bits ( b -1 bits for the location of the non-zero entry and 1 bit for its sign).

From the local reports, the server can compute an unbiased estimator by summing them together (with proper normalization) and performing an inverse Hadamard transform. Moreover, with an adequate injection of Gaussian noise, the frequency estimator satisfies ( ε, δ ) -DP.

The idea has been used in previous literature under local DP [19, 6, 3, 32], but in order to obtain the order-optimal trade-off under central -DP, one has to combine Hadamard transform with a random subsampling step and incorporate the privacy amplification due to random compression in the analysis. In Algorithm 3, we provide a summary of the resultant scheme which builds on the Recursive Hadamard Response (RHR) mechanism from [32], which was originally designed for communication-efficient frequency estimation under local DP.

In the following theorem, we control the ℓ ∞ error of Algorithm 3.

Theorem A.1. Let ˆ π ( x n ) be the output of Algorithm 3. Then it holds that for all j ∈ [ d ] , and the ℓ 2 2 and ℓ 1 errors are bounded by

Theorem A.2. For any ε, δ > 0 , Algorithm 3 is ( ε, δ ) -DP, if

By combining Theorem A.1 and Theorem A.2, we conclude that Algorithm 3 achieves ( ε, δ ) -DP with ℓ 2 2 error

Algorithm 3 Subsampled Recursive Hadamard Response

Input: user data x 1 , ..., x n ∈ { 0 , 1 } d (where d is a power of two), DP parameters ( ε, δ ) , communication budget b .

Output: frequency estimate ˆ π

Set B := d/ 2 b -1 and partition each one-hot vector x i into 2 b -1 chunks: x (1) i , ..., x (2 b -1) i ∈ { 0 , 1 } B .

for user i ∈ [ n ] do

Compute the Hadamard transform of each chunk: y ( ℓ ) i = H B · x ( ℓ ) i .

end if end for

end for

Server computes the average: ∀ ℓ ∈ [2 b -1 ] , j ∈ [ B ] ,

is computed according to Theorem A.2.

where σ 2

Server performs the inverse Hadamard transform ˆ π ( ℓ ) = H B · ˆ y ( ℓ ) , for ℓ = 1 , ..., B .

Notice that when n = ˜ Ω( d ) , the error can be simplified to

which matches the order-optimal estimation error (up to a log d factor) subject to a b -bit constraint [55, 3, 2] and ( ε, δ ) -DP constraint [15, 7].

B Proof of Theorem 4.1

It is trivial to see that the average communication cost is d · γ = b bits. To compute the ℓ 2 2 estimation error, observe that

̸

̸

which yields the inequality of (2). Next, we analyze the privacy of Algorithm 1. We first the following two lemmas for subsampling and the Gaussian mechanism:

Lemma B.1 ([66, 82]) . If M is ( ε, δ ) -DP, then M ′ that applies M◦ PoissonSample satisfies ( ε ′ , δ ′ ) -DP with ε ′ = log (1 + γ ( e ε -1)) and δ ′ = γδ .

Lemma B.2 ([15]) . For any ε, δ ∈ (0 , 1) , the Gaussian output perturbation mechanism with σ 2 := ∆ 2 2 log(1 . 25 /δ ) ε 2 satisfies ( ε, δ ) -DP, where ∆ is the ℓ 2 sensitivity of the target function.

Now, we use the above two lemmas to analyze the per-coordinate privacy leakage of Algorithm 1. For simplicity, we analyze the sum of x i ( j ) 's instead (and normalized it in the last step). Let S j ( x n ) := ∑ n i =1 ( x i ( j )) , then clearly the sensitivity of S j ( x n ) is 2 c , so Lemma B.2 implies S j ( x n ) + N (0 , σ 2 1 ) satisfies ( ε 1 , δ 1 ) -DP if we set σ 2 1 = 2 c 2 log(1 . 25 /δ 1 ) ε 2 1 (assuming ε 1 < 1 ). Next, if applying subsampling before computing the sum, i.e.,

where Z i,j i.i.d. ∼ Bern ( γ ) as defined in Algorithm 1, then by Lemma B.1,

satisfies ( ε 2 , δ 2 ) -DP with ε 2 := log (1 + γ ( e ε 1 -1)) = C 1 γε 1 (since we assume ϵ 1 < 1 ) and δ 2 := γδ 1 . Equivalently, we have

Now, since we have established the per-coordinate privacy leakage, we apply the following composition theorem to account for the total privacy budgets.

Theorem B.3 ([39]) . For any ε > 0 , δ ∈ [0 , 1] and ˜ δ ∈ (0 , 1] , the class of ( ε, δ ) -DP mechanisms satisfies (˜ ε ˜ δ , dδ + ˜ δ ) -DP under d -fold adaptive composition, for

According Theorem B.3, Algorithm 1 satisfies ( ε, δ ) -DP for

and δ = dδ 2 + ˜ δ (where ˜ δ is a free parameter that we can optimize).

Consequently, for a pre-specified (total) privacy budget ( ε, δ ) , we set parameters as follows. Let ˜ δ = δ 2 and δ 1 = 1 γ δ 2 = 1 2 dγ δ . Let ε 2 ≤ 1 so that e ε 2 -1 ≤ 2 ε 2 holds. Then (11) implies Algorithm 1 is

Solving the above quadratic (in-)equality for ε 2 , it suffices that

Consequently, we set ε 1 = ˜ C 1 γ ε 2 = O ( min ( 1 , ε γ √ d (log(1 /δ )+ ε ) )) (note that we require ε 1 = O (1) so that (10) holds).

Plugging ( ε 1 , δ 1 ) into σ 2 1 , we need to choose

Finally, as we are interested in estimating the (subsampled) mean instead of the sum, we will normalize the private sum by

where

Plugging in σ 2 above and γ = b/d yields the desired accuracy in Theorem 4.1. □

Since we will reuse the above result, we summarize it into the following lemma:

Lemma B.4. Let f i : R d × m ↦→ R D for i = 1 , ..., B be n functions with sensitivity bounded by ∆ (where the number of inputs m can be a random variable). Then

satisfies ( ε, δ ) -DP, if

C Omitted details of dimension-free communication cost

C.1 Proof of Theorem 4.4

To prove Theorem 4.4, it suffices to prove the following ℓ ∞ version:

Then Algorithm 2 is ( ε, δ ) -DP and yields an unbiased estimator on µ . In addition, the (average) per-client communication cost is γd ′ = b bits, and the ℓ 2 2 estimation error is at most

With a slight abuse of notation, we let µ J ∈ R d be such that

Note that µ J is an unbiased estimate of µ if J is selected uniformly at random. Then the ℓ 2 2 error can be controlled by

where (a) holds since µ J is an unbiased estimate of µ and conditioned on J , ˆ µ is an unbiased estimate of µ J ; (b) follows from Theorem 4.1; (c) holds due to the following fact:

Therefore, by setting d ′ = min ( nb, n 2 ε 2 (log(1 /δ )+ ε ) log( d/δ ) ) we ensure the first term in (c) is always smaller than the second term, and the second term can be simplified as follows:

Finally, applying the same trick of Kashin's representation, we can transform the ℓ ∞ geometry to ℓ 2 (similar to Corollary 4.3), hence proving Theorem 4.4. □

D Proof of Theorem A.1

Let π := 1 n ∑ i x i and π ( ℓ ) be defined in the same way as x ( ℓ ) i for ℓ ∈ [ B ] . Then our goal is to bound ∣ ∣ π ( ℓ ) ( j ) -ˆ π ( ℓ ) ( j ) ∣ ∣ , for all ℓ ∈ [2 b -1 ] and j ∈ [ B ] .

To this end, let y ( ℓ ) := H B · π ( ℓ ) (so it holds that π ( ℓ ) = 1 H B · y ( ℓ ) ). Then we have

Next, observe that due to the subsampling step, for all ℓ ∈ [2 b -1 ] and j ∈ [ B ] ,

where recall that Z ij i.i.d. ∼ Ber (1 /B ) . Therefore, ˆ y ( ℓ ) ( j ) is an unbiased estimator of y ( ℓ ) ( j ) . In addition, since we choose Z ij independently in Algorithm 3, ˆ y ( ℓ ) ( j ) 's are independent for different j 's, so we have

̸

and for all j = j ′

Therefore, we continue bounding (14) as follows:

where (a) holds since each entry of H B takes value in {-1 , 1 } and by (16), (b) holds due to (15), and (c) holds because C ℓ ≤ n for all ℓ .

Finally, to bound the ℓ 2 2 error, observe that the above analysis ensures that

where ℓ ( j ) ∈ [2 b -1 ] is the index of the chuck containing j . Therefore, summing over j ∈ [ d ] , we must have

since

E Proof of Theorem A.2

Let f j ( x n ) := ( π (1) ( j ) , ..., π (2 b -1 ) ( j )) , for j = 1 , ..., B . Then the ℓ 2 sensitivity of f j is ∆ = B n . Set the sampling rate γ = 1 B and the proof is complete by Lemma B.4. □

F Algorithm of Shuffled SQKR

Algorithm 4 Shuffled SQKR

G Proof of Theorem 5.3

Each round x n ↦→ ˆ µ ( k ) of Algorithm 4 implements the private-coin SQKR scheme of [32], achieving the communication cost and error as stated in Lemma 5.2.

Lemma G.1 (SQKR [32]) . For all ε 0 > 0 , b 0 > 0 , the random mapping x i ↦→ Y ( i, 1) , . . . , Y ( i, b 0 ) , s ( i, 1) , . . . , s ( i, b 0 ) in Algorithm 4 is ( ε 0 , 0) -LDP and has output that can be communicated in b 0 log( d ) bits, and the ˆ µ ( k ) computed from Y ( i, 1) , . . . , Y ( i, b 0 ) , s ( i, 1) , . . . , s ( i, b 0 ) is an unbiased estimator of µ satisfying

We now characterize the error performance of Algorithm 4 for general choices of parameters that satisfy privacy and communication constraints.

Proposition G.2. For all ε > 0 , b > 0 , n > 0 , with any arbitrary choice of

there exists a choice of parameters ε 0 , b 0 , T such that Algorithm 4 is ( ε, T δ 1 + δ 2 ) -DP, uses no more than b bits of communication, and produces ˆ µ such that

Proof. For arbitrary choice of

it suffices to choose

The fact that Algorithm 4 uses less than b bits is immediate from the choice of T .

Applying Lemma G.1, by construction the mapping from each x i to Y ( i, 1) , . . . , Y ( i, b 0 ) is ( ε 0 , 0) -LDP. By assumption

also satisfies ε 0 ≤ log ( n 16 log(2 /δ ) ) , so by Lemma 5.1 the mapping x n ↦→ ˆ µ ( k ) is ( ε 1 , δ 1 ) -DP, where

Since the output of Algorithm 4 is a function of ( ˆ µ (1) , . . . , ˆ µ ( T ) ) , by B.3 it suffices to have

for Algorithm 4 to be ( ε, T δ 1 + δ 2 ) -DP. The first inequality follows from the assumption of δ 1 > e -n/ 16 e and choice of ε 0 = O (1) , and the second from choice of

Since ε 0 ≤ 1 ≤ b , we have min( ε 2 0 , ε 0 , b ) = ε 2 0 . Applying Lemma G.1,

Substituting the choice of T gives the desired result.

To show Theorem 5.3, it suffices to choose

which requires n > 16 e log(2) ≈ 30 . 14 due to (21), and apply the previous proposition.

the inequality is satisfied. Then the choice of

H Using Kashin's Representation

We first introduce the idea of a tight frame in Kashin's representation. A tight frame is a set of vectors { u j } D j =1 ⊂ R d that satisfy Parseval's identity, i.e. ∥ x ∥ 2 2 = ∑ D j =1 ⟨ u j , x ⟩ 2 for all x ∈ R d .

A frame can be viewed as a generalization of the notion of an orthogonal basis in R d for D > d . To increase robustness, we wish the information to be spread evenly across different coefficients, which motivates the following definition of a Kashin's representation:

Definition H.1. For a set of vectors { u j } D j =1 , we say the expansion

is a Kashin's representation of the vector x at level K .

By Theorem 3.5 and Theorem 4.1 in [67], we have the following lemma:

LemmaH.2. There exists a tight frame U = [ u 1 , ..., u D ] with (1) D = Θ( d ) and (2) level K = O (1) .

The above lemma implies that for each x i ∈ R d such that ∥ x i ∥ 2 ≤ 1 , one can always represent each x i with coefficients y i ∈ [ -γ 0 / √ d, γ 0 / √ d ] γ 1 d for some γ 0 , γ 1 > 0 and x i = Uy i .

I Rényi-DP for Shuffled SQKR

In this section we restate some results for RDP which are useful for privacy accounting in experiments.

Following the proof of Corollary 4.3 in [44], applying Theorem 4.1 in the same paper yields the following.

Lemma I.1. Let M i be an independent ( ε 0 , 0) -LDP mechanism for each i ∈ [ n ] with ε 0 ≤ 1 and π be a random permutation of [ n ] . Then for any α < n 16 ε 0 exp( ε 0 ) , the mechanism

is ( ε 1 ( α ) , δ ) -RDP with

where

For small ε 0 , the result below is useful.

Lemma I.2 ([40]) . Under the same assumptions as Lemma I.1, S is ( ε ( α ) , δ ) -RDP

Applying Lemma G.1, by construction the mapping from each x i to y ( i, 1) , . . . , y ( i, b 0 ) is ( ε 0 , 0) -LDP. By Lemma I.1, respectively Lemma I.2, the mapping x n ↦→ ˆ µ ( k ) is ( ε 1 ( α ) , α ) -RDP where ε 1 ( α ) is given by (36), respectively (39). By composition, Algorithm 4 is ( Tε ( α ) , α ) -RDP.

We can convert this bound back to ( ε, δ ) -DP using Proposition 12 from [30].

Proposition I.3. For all δ > 0 , Algorithm 4 is ( ε, δ ) -DP where

where

and σ, δ min are given by (37) , (38) respectively.

J Additional Experiments

Here experiments are done with the same setup as in Section 6, with local vectors X i ( j ) i.i.d. ∼ 1 √ d (2 · Ber (0 . 8) -1) . We set δ = 10 -6 .

Mean Estimation,d=2000,ε=0.5

SQKR

Shuffled SQKR (T =1 round)

Shuffled SQKR(T =5rounds)

Shuffled SQKR (T = 10 rounds)

Shuffled SQKR (T = 20 rounds)

102

Shuffled SQKR(T=50 rounds)

Shuffled SQKR (T = 100 rounds)

MSE

101

102

103

n

Figure 2 illustrates separation between Algorithm 4 and LDP schemes. Algorithm 4 achieves error decreasing quadratically with n as guaranteed by Theorem 5.3. With only one round of shuffling, there is separation from the LDP scheme only when n is sufficiently large, and thus order-optimal error performance only occurs for large n (or equivalently small ε ). This problem is avoided with multiple rounds of shuffling.

Meanestimation,n=1o0,ε=0.5

Without,d=2500

107

Without,d=3600

?

Without,d=4900

￥

Without,d=6400

With,d=2500

106

With,d=3600

With, d =4900

With,d=6400

105

MSE

104

IXIXIX+1X1XX1XX1X1Y1838X

103

102

(XXXXXX00000WX

100

101

Communication constraintb[bits]

Figure 3 compares the performance of CSGM with and without coordinate pre-selection. In this regime coordinate pre-selection improves performance for all b . As predicted by Corollary 4.3 and Corollary 4.5, the MSE decreases with b but is effectively constant for sufficiently high b where the

privacy term dominates. We can determine the communication cost needed for order-optimal central DP error performance to be the b at which the MSE is within some fixed constant factor away from the limiting value. We see that the communication cost increases with dimension d with the vanilla CSGM scheme, but a dimension-free communication cost is achieved with coordinate pre-selection.

Peer reviews:
comments: SUMMARY:
This paper studies the distributed mean estimation problem with communication constraints and central DP. Communication constraints are met by subsampling a Kashin representation of the target vector, and \eps-DP is achieved by adding Gaussian noise. Privacy in the absence of a trusted server can be achieved using a LDP mechanism along with a secure shuffler.

After discussions with the authors, I have increased my score.

STRENGTHS:
The work claims to be the first to study the communication-privacy-accuracy tradeoff for DME problem with central DP. It provides a simple scheme that achieves order-optimal error bounds.

WEAKNESSES:
While this is the first work to study the problem, the solution was well-known. Unlike the LDP constraints, there is no tension between the communication cost and privacy in this case.  Therefore, in my opinion, the novelty of this work is limited. The results are a straightforward amalgamation of well-known techniques that are independently known to be optimal.

QUESTIONS:
The results are intuitively correct and are technically sound.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the optimal rates of the tradeoff among privacy (central DP), utility ($\\ell_2$ estimation error), and communication cost (in bits) for the distributed mean and frequency estimation settings. The authors used Kashin's representation and Coordinate Subsampled Gaussian Mechanism to achieve the optimal rates. They also propose a many-round procedure with a shuffling mechanism to achieve the optimal rate if there is a secure shuffler instead of a secure central server. Simulation results are provided to supper the theoretical results of the distributed mean estimation setting.\n\nSTRENGTHS:\n1. This paper provides the optimal rates for the distributed mean estimation given specific privacy and communication constraints, which is a fundamental problem in distributed learning (with a central server).\n2. The authors proposed the Coordinate Subsampled Gaussian Mechanism and used it to realize the optimal rates.\n3. The theoretical results are supported by the simulation results.\n\nWEAKNESSES:\n1. In Theorem 5.3, the rate is $O(C^2 d \\ldots)$ while in the appendix Lemma G.1 and Proposition G.2, the rates are $O(c^2 d)$. As Theorem 5.3 is also using Kashin's representation, I think it should be $c=O(\\frac{C}{\\sqrt{d}})$ as in Line 211. Therefore, there could be something incorrect in the result of Theorem 5.3. However, removing $d$ from the current result also seems to be strange to me.\n2. The authors derived the theoretical results for the histogram estimation while not showing corresponding simulation results. They claimed these results in the abstract and introduction as contributions while only showing them in the appendix, which may not be very appropriate.\n3. This paper uses the name 'federated learning' many times while the problem being solved is more restricted to the distributed learning  (with a central server).\n\n\nMinor weaknesses:\n1. Citation issues: [33] is cited as 'in submission' while it is already published in AISTATS 2023. [59] and [60] are duplicates.\n2. Line 91. $10^6-10^9$ -> $10^6$ to $10^9$.\n3. In Theorem 4.1, $\\sigma$ should use $\\Omega$ notation since we need it large enough to guarantee privacy. Similar misuses are in other results. This can be seen in the transition from $\\sigma_1^2$ to $\\sigma^2$ in Line 630 where the former used $\\Omega$ while the latter used $O$.\n4. Line 358. $\\lceil \\log d\\rceil + 1)T$ should be $(\\lceil \\log d\\rceil + 1)T$\n5. Line 361. The period is missing in the end of this line.\n6. The citation of Theorem B.3 is missing.\n7. Line 628. $e_2^\\epsilon$ should be $e^{\\epsilon_2}$.\n8. Line 642. The second term in this equality is random which must be incorrect. Nevertheless, the final result is unaffected. \n\n---\nI have read the rebuttal which addressed my questions and the weakness concerns.\n\nQUESTIONS:\n1. Line 212. What does the dot product in $x_i = K \\cdot \\tilde x_i$ mean? Is it a matrix multiplied by a vector? \n2. Are the notations of $(\\alpha)$ and $(\\beta)$ in Equation (4) in Line 217 misplaced? I think $(\\alpha)$ is only the first part of Equation (4) and $(\\beta)$ is the remaining two parts. Nevertheless, this seems not to affect the analysis in Section 4.1.\n3. Line 354. For the statement of $d=5000$, is it from the previous claim based on the theoretical results '$b$ is independent of $d$'? Or is the result of $d=5000$ missing in this manuscript?\n4. Line 622 in Section B. Should the sensitivity of $S_j(x^n)$ be $2c$? Should $Z_{i,j}$ be i.i.d. from $\\mathrm{Bern}(\\gamma)$ instead of $\\mathrm{Bern}(1/\\gamma)$?\n5. Line 630. Should $\\log(d/\\delta)$ be $\\log(d\\gamma/\\delta)$? Is $\\gamma$ missing here (and elsewhere) intentionally?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new randomization scheme for differentially private mean estimation. It is based on each user choosing randomly the coordinates to be sent in their messages. Similar scheme has been proposed in reference paper [55] (Hu et al., 2020), however Hu et al. do not sparsify the individual data-element-wise vectors, only mini-batch gradients.\n\nIn the proposed scheme, data-element-wise vectors are sparsified randomly and the server averages these sparse vectors and adds normally distributed noise. A classical differential privacy analysis (meaning it uses some classical (eps,delta)-amplification results) is carried out resulting in optimal communication - privacy - accuracy tradeoffs (solving also an open problem in the untrusted server - setting) when these quantities are measured asymptotically.\n\nAlgorithms both in trusted server setting and untrusted server (via multimessage shuffling) are given.\n\nThe privacy analysis is essentially for the $\\ell_\\infty$-mean estimation, and the $\\ell_2$-mean estimation bounds are obtained using the so called Kashin's representation which turns $\\ell_2$-constraints into $\\ell_\\infty$-constraints in an optimal way.\n\nSTRENGTHS:\n- solves an open problem and given optimal tradeoffs between privacy, accuracy and communication both in trusted and untrusted setting\n- Generally well written\n- extensive list of references and a comprehensive 'related works' section. I think the paper serves even as a good starting point for diving into this topic.\n\nWEAKNESSES:\n- I think some parts are too densely written. The Kashin's representation is not introduced properly I think, those lines 208 to 214 give simply to little information. Like what is that tight frame $K$? I think there is some (still quite short) introduction in reference [34]. I think it would help a lot to have some more material on Kashin's representation, e.g. in the appendix. You use it repeatedly to get the $\\ell_2$ mean estimation bounds anyways.\n\n- Related to the above comment, and for further improving writing: in lines 645-646 in the appendix you write \"applying the same trick of Kashin's representation\"... what are you referring to with 'the same' ? Here also would help expanding / elaborating a bit.\n\n- I think the paper is not entirely explicit about the difficulty of accurate privacy accounting in case of $\\ell_2$-mean estimation (see the question comment below in the Questions - section)\n\n- It is not entirely clear how well the method fits to reducing communication in FL training of ML models if one would need to randomly sparsify individual gradients (see comment in the Questions - section)\n\nQUESTIONS:\n-  You mention that the privacy analysis could possibly be improved, and the log factors shaved off. Somehow I get the impression that you could elaborate on this more... Looking at the trusted server setting, I get the impression that the $\\ell_\\infty$-mean estimation algorithm one could perhaps analyse tightly (with RDP or directly with approximative DP). In a sense I think you do that in the experimental example. However, again with the $\\ell_\\infty$-constraints, how would you go about involving the Kashin's representations in the analysis? I get the impression that, e.g., in reference [34], the privacy bounds are simply big-O bounds as there are some universal constant involved when transforming between the constraints. Do you think it would be possible to analyse $\\ell_2$-mean estimation algorithm using this coordinate subsampling tightly?\n\n- How would this fit to federated learning? I mean wouldn't the sparsity be lost quickly in case each device/user has model updates constructed using several data-elements? Wouldn't sparsification similar to the that of [55] be then required to reduce the communication?\n\nMinor comments:\n- Shouldn't \"$\\ell_2$ error\" in Table 1 be actually \"$\\ell_2^2$ error ?\n- In bibliography, please use the Arxiv reference for the reference [33] instead of \"in submission\" \n- The reference paper [55] (which is quite central here) has been published in IJCAI, you only mention the Arxiv reference.\n\nLIMITATIONS:\nYes, I think the limitations with the analysis are adequately discussed in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studies the federated learning problem within the central differential privacy model, where a trusted central server gathers information from local clients. The primary focus is on minimizing communication costs while ensuring privacy and maintaining accuracy guarantees. To address this challenge, the paper introduces a novel privacy amplification framework. The framework operates by having each local client transmit only a random subsample of its full parameters to the central server, effectively reducing the communication overhead. Simultaneously, the randomness introduced in the subsampling process amplifies the privacy guarantee. Notably, in the specific case of distributed mean estimation, this framework significantly reduces the communication cost from an order of $O(d)$ to $\tilde O(n\min\\{\epsilon, \epsilon^2\\})$, showing a significant improvement over state-of-art result.

STRENGTHS:
This paper provides a novel privacy amplification method in central DP setting that optimizes the communication-privacy-accuracy three-way trade-off. Furthermore, the proposed distributed mean estimation also has significant implications. For example, upon setting $x_i$ to be local gradients, this algorithm immediately implies a private distributed SGD algorithm with reduced communication cost.

WEAKNESSES:
The privacy amplification method is restricted to the central DP model and does not apply to LDP model.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10