Title: Breaking the Communication-Privacy-Accuracy Tradeoff with $f$-Differential Privacy
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential privacy

federated data analytics

discrete valued-mechanism

distributed mean estimation
Abstract: We consider a federated data analytics problem in which a server coordinates the collaborative data analysis of multiple users with privacy concerns and limited communication capability. The commonly adopted compression schemes introduce information loss into local data while improving communication efficiency, and it remains an open problem whether such discrete-valued mechanisms provide any privacy protection. In this paper, we study the local differential privacy guarantees of discrete-valued mechanisms with finite output space through the lens of $f$-differential privacy (DP). More specifically, we advance the existing literature by deriving tight $f$-DP guarantees for a variety of discrete-valued mechanisms, including the binomial noise and the binomial mechanisms that are proposed for privacy preservation, and the sign-based methods that are proposed for data compression, in closed-form expressions. We further investigate the amplification in privacy by sparsification and propose a ternary stochastic compressor. By leveraging compression for privacy amplification, we improve the existing methods by removing the dependency of accuracy (in terms of mean square error) on communication cost in the popular use case of distributed mean estimation, therefore breaking the three-way tradeoff between privacy, communication, and accuracy.

Breaking the Communication-Privacy-Accuracy Tradeoff with f -Differential Privacy

Abstract

We consider a federated data analytics problem in which a server coordinates the collaborative data analysis of multiple users with privacy concerns and limited communication capability. The commonly adopted compression schemes introduce information loss into local data while improving communication efficiency, and it remains an open problem whether such discrete-valued mechanisms provide any privacy protection. In this paper, we study the local differential privacy guarantees of discrete-valued mechanisms with finite output space through the lens of f -differential privacy (DP). More specifically, we advance the existing literature by deriving tight f -DP guarantees for a variety of discrete-valued mechanisms, including the binomial noise and the binomial mechanisms that are proposed for privacy preservation, and the sign-based methods that are proposed for data compression, in closed-form expressions. We further investigate the amplification in privacy by sparsification and propose a ternary stochastic compressor. By leveraging compression for privacy amplification, we improve the existing methods by removing the dependency of accuracy (in terms of mean square error) on communication cost in the popular use case of distributed mean estimation, therefore breaking the three-way tradeoff between privacy, communication, and accuracy.

1 Introduction

Nowadays, the massive data generated and collected for analysis, and consequently the prohibitive communication overhead for data transmission, are overwhelming the centralized data analytics paradigm. Federated data analytics is, therefore, proposed as a new distributed computing paradigm that enables data analysis while keeping the raw data locally on the user devices [1]. Similarly to its most notable use case, i.e., federated learning (FL) [2, 3], federated data analytics faces two critical challenges: data privacy and communication efficiency. On one hand, the local data of users may contain sensitive information, and privacy-preserving mechanisms are needed. On the other hand, the user devices are usually equipped with limited communication capabilities, and compression mechanisms are often adopted to improve communication efficiency.

Differential privacy (DP) has become the gold standard for privacy measures due to its rigorous foundation and simple implementation. One classic technique to ensure DP is adding Gaussian or Laplacian noises to the data [4]. However, they are prone to numerical errors on finite-precision computers [5] and may not be suitable for federated data analytics with communication constraints due to their continuous nature. With such consideration, various discrete noises with privacy guarantees have been proposed, e.g., the binomial noise [6], the discrete Gaussian mechanism [7], and the Skellam mechanism [8]. Nonetheless, the additive noises in [7] and [8] assume infinite range, which renders them less communication-efficient without appropriate clipping. Unfortunately, clipping

usually ruins the unbiasedness of the mechanism. [9] develops a Poisson binomial mechanism (PBM) that does not rely on additive noise. In PBM, each user adopts a binomial mechanism, which takes a continuous input and encodes it into the success probability of a binomial distribution. The output of the binomial mechanism is shared with a central server which releases the aggregated result that follows the Poisson binomial distribution. However, [9] focuses on distributed DP in which the server only observes the output of the aggregated results instead of the data shared by each individual user, and therefore, requires a secure computation function (e.g., secure aggregation [3]).

In addition to discrete DP mechanisms, existing works have investigated the fundamental tradeoff between communication, privacy, and accuracy under the classic ( ϵ, δ ) -DP framework (e.g., [10, 11, 12, 13]). Notably, in the case of distributed mean estimation, [13] incorporates Kashin's representation and proposed Subsampled and Quantized Kashin's Response (SQKR), which achieves order-optimal mean square error (MSE) that has a linear dependency on the dimension of the private data d . SQKR first computes Kashin's representation of the private data and quantizes each coordinate into a 1-bit message. Then, k coordinates are randomly sampled and privatized by the 2 k -Random Response mechanism [14]. SQKR achieves an order-optimal three-way tradeoff between privacy, accuracy, and communication. Nonetheless, it does not account for the privacy introduced during sparsification.

Intuitively, as compression becomes more aggressive, less information will be shared by the users, which naturally leads to better privacy protection. However, formally quantifying the privacy guarantees of compression mechanisms remains an open problem. In this work, we close the gap by investigating the local DP guarantees of discrete-valued mechanisms, based on which a ternary stochastic compressor is proposed to leverage the privacy amplification by compression and advance the literature by achieving a better communication-privacy-accuracy tradeoff. More specifically, we focus on the emerging concept of f -DP [15] that can be readily converted to ( ϵ, δ ) -DP and R´ enyi differential privacy [16] in a lossless way while enjoying better composition property [17].

Our contributions . In this work, we derive the closed-form expressions of the tradeoff function between type I and type II error rates in the hypothesis testing problem for a generic discrete-valued mechanism with a finite output space, based on which f -DP guarantees of the binomial noise (c.f. Section 4.1) and the binomial mechanism (c.f. Section 4.2) that covers a variety of discrete differentially private mechanisms and compression mechanisms as special cases are obtained. Our analyses lead to tighter privacy guarantees for binomial noise than [6] and extend the results for the binomial mechanism in [9] to local DP. To the best of our knowledge, this is the first work that investigates the f -DP guarantees of discrete-valued mechanisms, and the results could possibly inspire the design of better differentially private compression mechanisms.

Inspired by the analytical results, we also leverage the privacy amplification of the sparsification scheme and propose a ternary stochastic compressor (c.f. Section 5). By accounting for the privacy amplification of compression, our analyses reveal that given a privacy budget µ -GDP (which is a special case of f -DP) with µ < √ 4 dr/ (1 -r ) (in which r is the ratio of non-zero coordinates in expectation for the sparsification scheme), the MSE of the ternary stochastic compressor only depends on µ in the use case of distributed mean estimation (which is the building block of FL). In this sense, we break the three-way tradeoff between communication overhead, privacy, and accuracy by removing the dependency of accuracy on the communication overhead. Different from existing works which suggest that, in the high privacy regime, the error introduced by compression is dominated by the error introduced for privacy, we show that the error caused by compression could be translated into enhancement in privacy. Compared to SQKR [13], the proposed scheme yields better privacy guarantees given the same MSE and communication cost. For the scenario where each user i observes x i ∈ {-c, c } d for some constant c > 0 , the proposed scheme achieves the same privacy guarantee and MSE as those of the classic Gaussian mechanism in the large d regime, which essentially means that the improvement in communication efficiency is achieved for free. We remark that the regime of large d is often of interest in practical FL in which d is the number of training parameters.

2 Related Work

Recently, there is a surge of interest in developing differentially private data analysis techniques, which can be divided into three categories: central differential privacy (CDP) that assumes a trusted central server to perturb the collected data [18], distributed differential privacy that relies on secure aggregation during data collection [3], and local differential privacy (LDP) that avoids the need for

the trusted server by perturbing the local data on the user side [19]. To overcome the drawbacks of the Gaussian and Laplacian mechanisms, several discrete mechanisms have been proposed. [18] introduces the one-dimensional binomial noise, which is extended to the general d -dimensional case in [6] with more comprehensive analysis in terms of ( ϵ, δ ) -DP. [20] analyzes the LDP guarantees of discrete Gaussian noise, while [7] further considers secure aggregation. [8] studies the R´ enyi DP guarantees of the Skellam mechanism. However, both the discrete Gaussian mechanism and the Skellam mechanism assume infinite ranges at the output, which makes them less communication efficient without appropriate clipping. Moreover, all the above three mechanisms achieve differential privacy at the cost of exploding variance for the additive noise in the high-privacy regimes.

Another line of studies jointly considers privacy preservation and compression. [10, 11] propose to achieve DP by quantizing, sampling, and perturbing each entry, while [12] proposes a vector quantization scheme with local differential privacy. However, the MSE of these schemes grows with d 2 . [13] investigates the three-way communication-privacy-accuracy tradeoff and incorporates Kashin's representation to achieve order-optimal estimation error in mean estimation. [21] proposes to first sample a portion of coordinates, followed by the randomized response mechanism [22]. [23] and [24] further incorporate shuffling for privacy amplification. [25] proposes to compress the LDP schemes using a pseudorandom generator, while [26] utilizes minimal random coding. [27] proposes a privacy-aware compression mechanism that accommodates DP requirement and unbiasedness simultaneously. However, they consider pure ϵ -DP, which cannot be easily generalized to the relaxed variants. [9] proposes the Poisson binomial mechanism with R´ enyi DP guarantees. Nonetheless, R´ enyi DP lacks the favorable hypothesis testing interpretation and the conversion to ( ϵ, δ ) -DP is lossy. Moreover, most of the existing works focus on privatizing the compressed data or vice versa, leaving the privacy guarantees of compression mechanisms largely unexplored. [28] proposes a numerical accountant based on fast Fourier transform [29] to evaluate ( ϵ, δ ) -DP of general discrete-valued mechanisms. Recently, an independent work [30] studies privacy amplification by compression for central ( ϵ, δ )-DP and multi-message shuffling frameworks. In this work, we consider LDP through the lens of f -DP and eliminate the need for a trusted server or shuffler.

Among the relaxations of differential privacy notions [31, 16, 32], f -DP [15] is a variant of ϵ -DP with hypothesis testing interpretation, which enjoys the property of lossless conversion to ( ϵ, δ ) -DP and tight composition [33]. As a result, it leads to favorable performance in distributed/federated learning [34, 35]. However, to the best of our knowledge, none of the existing works study the f -DP of discrete-valued mechanisms. In this work, we bridge the gap by deriving tight f -DP guarantees of various compression mechanisms in closed form, based on which a ternary stochastic compressor is proposed to achieve a better communication-privacy-accuracy tradeoff than existing methods.

3 Problem Setup and Preliminaries

3.1 Problem Setup

We consider a set of N users (denoted by N ) with local data x i ∈ R d . The users aim to share x i 's with a central server in a privacy-preserving and communication-efficient manner. More specifically, the users adopt a privacy-preserving mechanism M to obfuscate their data and share the perturbed results M ( x i ) 's with the central server. In the use case of distributed/federated learning, each user has a local dataset S . During each training step, it computes the local stochastic gradients and shares the obfuscated gradients with the server. In this sense, the overall gradient computation and obfuscation mechanism M takes the local dataset S as the input and outputs the obfuscated result M ( S ) . Upon receiving the shared M ( S ) 's, the server estimates the mean of the local gradients.

3.2 Differential Privacy

Formally, differential privacy is defined as follows.

Definition 1 ( ( ϵ, δ ) -DP [18]) . A randomized mechanism M is ( ϵ, δ ) -differentially private if for all neighboring datasets S and S ′ and all O ⊂ O in the range of M , we have

(1)

in which S and S ′ are neighboring datasets that differ in only one record, and ϵ, δ ≥ 0 are the parameters that characterize the level of differential privacy.

3.3 f -Differential Privacy

Assuming that there exist two neighboring datasets S and S ′ , from the hypothesis testing perspective, we have the following two hypotheses

Let P and Q denote the probability distribution of M ( S ) and M ( S ′ ) , respectively. [15] formulates the problem of distinguishing the two hypotheses as the tradeoff between the achievable type I and type II error rates. More precisely, consider a rejection rule 0 ≤ ϕ ≤ 1 (which rejects H 0 with a probability of ϕ ), the type I and type II error rates are defined as α ϕ = E P [ ϕ ] and β ϕ = 1 -E Q [ ϕ ] , respectively. In this sense, f -DP characterizes the tradeoff between type I and type II error rates. The tradeoff function and f -DP are formally defined as follows.

Definition 2 (tradeoff function [15]) . For any two probability distributions P and Q on the same space, the tradeoff function T ( P, Q ) : [0 , 1] → [0 , 1] is defined as T ( P, Q )( α ) = inf { β ϕ : α ϕ ≤ α } , where the infimum is taken over all (measurable) rejection rule ϕ .

Definition 3 ( f -DP [15]) . Let f be a tradeoff function. With a slight abuse of notation, a mechanism M is f -differentially private if T ( M ( S ) , M ( S ′ )) ≥ f for all neighboring datasets S and S ′ , which suggests that the attacker cannot achieve a type II error rate smaller than f ( α ) .

f -DP can be converted to ( ϵ, δ ) -DP as follows.

Lemma 1. [15] A mechanism is f ( α ) -differentially private if and only if it is ( ϵ, δ ) -differentially private with

Finally, we introduce a special case of f -DP with f ( α ) = Φ(Φ -1 (1 -α ) -µ ) , which is denoted as µ -GDP. More specifically, µ -GDP corresponds to the tradeoff function of two normal distributions with mean 0 and µ , respectively, and a variance of 1.

4 Tight f -DP Analysis for Existing Discrete-Valued Mechanisms

In this section, we derive the f -DP guarantees for a variety of existing differentially private discretevalued mechanisms in the scalar case (i.e., d = 1 ) to illustrate the main ideas. The vector case will be discussed in Section 6. More specifically, according to Definition 3, the f -DP of a mechanism M is given by the infimum of the tradeoff function over all neighboring datasets S and S ′ , i.e., f ( α ) = inf S,S ′ inf ϕ { β ϕ ( α ) : α ϕ ≤ α } . Therefore, the analysis consists of two steps: 1) we obtain the closed-form expressions of the tradeoff functions, i.e., inf ϕ { β ϕ ( α ) : α ϕ ≤ α } , for a generic discrete-valued mechanism (see Section A in the supplementary material); and 2) given the tradeoff functions, we derive the f -DP by identifying the mechanism-specific infimums of the tradeoff functions over all possible neighboring datasets. We remark that the tradeoff functions for the discrete-valued mechanisms are essentially piece-wise functions with both the domain and range of each piece determined by both the mechanisms and the datasets, which renders the analysis for the second step highly non-trivial.

4.1 Binomial Noise

In this subsection, we consider the binomial noise (i.e., Algorithm 1) proposed in [6], which serves as a communication-efficient alternative to the classic Gaussian noise. More specifically, the output of stochastic quantization in [6] is perturbed by a binomial random variable.

Algorithm 1 Binomial Noise [6]

Input : x i ∈ [0 , 1 , · · · , l ] , i ∈ N , number of trials M , success probability p . Privatization: Z i ≜ x i + Binom ( M,p ) .

Theorem 1. Let ˜ Z = Binom ( M,p ) , the binomial noise mechanism in Algorithm 1 is f bn ( α ) -differentially private with

in which

Given that P ( ˜ Z = k ) = ( M k ) p k (1 -p ) M -k , it can be readily shown that when p = 0 . 5 , both β + ϕ, inf ( α ) and β -ϕ, inf ( α ) are maximized, and f ( α ) = β + ϕ, inf ( α ) = β -ϕ, inf ( α ) .

Fig. 1 shows the impact of M when l = 8 , which confirms the result in [6] that a larger M provides better privacy protection (recall that given the same α , a larger β α indicates that the attacker makes mistakes in the hypothesis testing more likely and therefore corresponds to better privacy protection). Note that the output of Algorithm 1 Z i ∈ { 0 , 1 , ..., M + l } , which reqiures a communication overhead of log 2 ( M + l +1) bits. We can readily convert f ( α ) -DP to ( ϵ, δ ) -DP by utilizing Lemma 1.

Remark 1. The results derived in this work improve [6] in two aspects: (1) Theorem 1 in [6] requires Mp (1 -p ) ≥ max(23log(10 d/δ ) , 2 l/s ) > max(23log(10) , 2 l/s ) , in which 1 /s ∈ N is some scaling factor. When p = 1 / 2 , it requires M ≥ 212 . More specifically, for M = 500 , [6] requires δ > 0 . 044 . Our results imply that there exists some ( ϵ, δ ) such that Algorithm 1 is ( ϵ, δ ) -DP as long as M > l . For M = 500 , δ can be as small as 4 . 61 × 10 -136 . (2) Our results are tight, in the sense that no relaxation is applied in our derivation. As an example, when M = 500 and p = 0 . 5 , Theorem 1 in [6] gives (3 . 18 , 0 . 044) -DP while Theorem 1 in this paper yields (1 . 67 , 0 . 039) -DP.

4.2 Binomial Mechanism

Algorithm 2 Binomial Mechanism [9]

In this subsection, we consider the binomial mechanism (i.e., Algorithm 2). Different from Algorithm 1 that perturbs the data with noise following the binomial distribution with the same success probability, the binomial mechanism encodes the input x i into the success probability of the binomial distribution. We establish the privacy guarantee of Algorithm 2 as follows.

in which

Remark 2 ( Comparison to [9] ) . The binomial mechanism is part of the Poisson binomial mechanism proposed in [9]. More specifically, in [9], each user i shares the output of the binomial mechanism

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

f(α)

M=10,p=0.5

M=50,p=0.5

M=100,p=0.5

M=200,p=0.5

M=500,p=0.5

α

Z i with the server, in which p i ( x i ) = 1 2 + θ c x i and θ is some design parameter. It can be readily verified that p max = 1 -p min in this case. The server then aggregates the result through ¯ x = c MNθ ( ∑ i ∈N Z i -MN 2 ) . [9] requires secure aggregation and considers the privacy leakage of releasing ¯ x , while we complement it by showing the LDP, i.e., the privacy leakage of releasing Z i for each user. In addition, we eliminate the constraint θ ∈ [0 , 1 4 ] , and the results hold for any selection of p i ( x i ) . Moreover, the privacy guarantees in Theorem 2 are tight since no relaxation is involved. Fig. 2 shows the impact of M on the privacy guarantee. In contrast to binomial noise, the privacy of the binomial mechanisms improves as M (and equivalently communication overhead) decreases, which implies that it is more suitable for communication-constrained scenarios. We also derive the f -DP of the Poisson binomial mechanism, which are presented in Section C in the supplementary material.

In the following, we present two existing compressors that are special cases of the binomial mechanism.

Example 1. We first consider the following stochastic sign compressor proposed in [36].

Definition 4 ( Two-Level Stochastic Compressor [36]) . For any given x ∈ [ -c, c ] , the compressor sto -sign outputs

where A > c is the design parameter that controls the level of stochasticity.

With a slight modification (i.e., mapping the output space from { 0 , 1 } to {-1 , 1 } ), sto -sign ( x, A ) can be understood as a special case of the binomial mechanism with M = 1 and p i ( x i ) = A + x i 2 A . In this case, we have p max = A + c 2 A and p min = A -c 2 A . Applying the results in Theorem 2 yields

Combining (9) with (3) suggests that the sto -sign compressor ensures (ln( A + c A -c ) , 0) -DP.

Example 2. The second sign-based compressor that we examine is CLDP ∞ ( · ) [23].

Definition 5 ( CLDP ∞ ( · ) [23]) . For any given x ∈ [ -c, c ] , the compressor CLDP ∞ ( · ) outputs CLDP ∞ ( ϵ ) , which is given by

CLDP ∞ ( ϵ ) can be understood as a special case of sto -sign ( x, A ) with A = c ( e ϵ +1) e ϵ -1 . In this case, according to (9), we have

Combining the above result with (3) suggests that CLDP ∞ ( ϵ ) ensures ( ϵ, 0) -DP, which recovers the result in [23]. It is worth mentioning that CLDP ∞ ( ϵ ) can be understood as the composition of sto -sign with A = c followed by the randomized response mechanism [22], and is equivalent to the one-dimensional case of the compressor in [13]. Moreover, the one-dimensional case of the schemes in [10, 11] can also be understood as special cases of sto -sign .

5 The Proposed Ternary Compressor

The output of the binomial mechanism with M = 1 lies in the set { 0 , 1 } , which coincides with the sign-based compressor. In this section, we extend the analysis to the ternary case, which can be understood as a combination of sign-based quantization and sparsification (when the output takes value 0, no transmission is needed since it does not contain any information) and leads to improved communication efficiency. More specifically, we propose the following ternary compressor.

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

f(α)

M=16

M=32

M=64

M=128

M=256

α

Definition 6 ( Ternary Stochastic Compressor ) . For any given x ∈ [ -c, c ] , the compressor ternary outputs ternary ( x, A, B ) , which is given by

where B > A > c are the design parameters that control the level of sparsity.

For the ternary stochastic compressor in Definition 6, we establish its privacy guarantee as follows.

Theorem 3. The ternary stochastic compressor is f ternary ( α ) -differentially private with

Remark 3 (Privacy amplification by sparsification) . It can be observed from (9) and (13) that f ternary ( α ) > f sto -sign when α ∈ [ A -c 2 B , 1 -A + c 2 B ] , and f ternary ( α ) = f sto -sign , otherwise. Fig. 3 shows f ternary ( α ) and f sto -sign for c = 0 . 1 , A = 0 . 25 , B = 0 . 5 ,

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

f(α)

sto-sign

ternary

(ln(2), 0.05)-DP

α

and the shaded gray area corresponds to the improvement in privacy. It can be observed that communication efficiency and privacy are improved simultaneously. It is worth mentioning that, if we convert the privacy guarantees to ( ϵ, 0) -DP, we have ϵ = ln( 7 3 ) for both compressors. However, the ternary compressor ensures (ln(2) , 0 . 05) -DP (i.e., f ternary ( α ) ≥ max { 0 , 0 . 95 -2 α, 0 . 5(0 . 95 -α ) } ) while the sto -sign compressor does not. We note that for the same A , as B increases (i.e., communication cost decreases), f ternary ( α ) approaches f ( α ) = 1 -α (which corresponds to perfect privacy).

In the following, we present a special case of the proposed ternary stochastic compressor.

Example 3. The ternary-based compressor proposed in [37] is formally defined as follows.

Definition 7 ( ternarize ( · ) [37]) . For any given x ∈ [ -c, c ] , the compressor ternarize ( · ) outputs ternarize ( x, B ) = sign ( x ) with probability | x | /B and ternarize ( x, B ) = 0 otherwise, in which B > c is the design parameter.

̸

ternarize ( x, B ) can be understood as a special case of ternary ( x, A, B ) with A = | x | . According to Theorem 3, f ternary ( α ) = 1 -c B -α for α ∈ [0 , 1 -c B ] and f ternary ( α ) = 0 for α ∈ [1 -c B , 1] . Combining the above result with (3), we have δ = c B and ϵ = 0 , i.e., ternarize ( · ) provides perfect privacy protection ( ϵ = 0 ) with a violation probability of δ = c B . Specifically, the attacker cannot distinguish x i from x ′ i if the output of ternarize ( · ) = 0 (perfect privacy protection), while no differential privacy is provided if the output of ternarize ( · ) = 0 (violation of the privacy guarantee).

Remark 4. It is worth mentioning that, in [37], the users transmit a scaled version of ternarize ( · ) and the scaling factor reveals the magnitude information of x i . Therefore, the compressor in [37] is not differentially private.

6 Breaking the Communication-Privacy-Accuracy Tradeoff

In this section, we extend the results in Section 5 to the vector case in two different approaches, followed by discussions on the three-way tradeoff between communication, privacy, and accuracy. The results in Section 4 can be extended similarly. Specifically, in the first approach, we derive the µ -GDP in closed form, while introducing some loss in privacy guarantees. In the second approach, a tight approximation is presented. Given the results in Section 5, we can readily convert f -DP in the scalar case to Gaussian differential privacy in the vector case as follows.

Theorem 4. Given a vector x i = [ x i, 1 , x i, 2 , · · · , x i,d ] with | x i,j | ≤ c, ∀ j . Applying the ternary compressor to the j -th coordinate of x i independently yields µ -GDP with µ = -2Φ -1 ( 1 1+( A + c A -c ) d ) .

Remark 5. Note that || x i || 2 ≤ c is a sufficient condition for | x i,j | ≤ c, ∀ j . In the proof of Theorem 4, we first convert f ternary ( α ) -DP to ( ϵ, 0) -DP for the scalar case, and then obtain ( dϵ, 0) -DP

for the d -dimensional case, followed by the conversion to GDP. One may notice that some loss in privacy guarantee is introduced since the extreme case | x i,j | = c, ∀ j actually violates the condition || x i || 2 ≤ c . To address this issue, following a similar method in [13, 38, 9], one may introduce Kashin's representation to transform the l 2 geometry of the data into the l ∞ geometry. More specifically, [39] shows that for D > d , there exists a tight frame U such that for any x ∈ R d , one can always represent each x i with y i ∈ [ -γ 0 / √ d, -γ 0 / √ d ] D for some γ 0 and x i = Uy i .

In Theorem 4, some loss in privacy guarantees is introduced when we convert f -DP to µ -GDP. In fact, since each coordinate of the vector is processed independently, the extension from the scalar case to the d -dimensional case may be understood as the d -fold composition of the mechanism in the scalar case. The composed result can be well approximated or numerically obtained via the central limit theorem for f -DP in [15] or the Edgeworth expansion in [33]. In the following, we present the result for the ternary compressor by utilizing the central limit theorem for f -DP.

Theorem 5. For a vector x i = [ x i, 1 , x i, 2 , · · · , x i,d ] with | x i,j | ≤ c, ∀ j , the ternary compressor with B ≥ A > c is f ternary ( α ) -DP with

in which

Given the above results, we investigate the communication-privacy-accuracy tradeoff and compare the proposed ternary stochastic compressor with the state-of-the-art method SQKR in [13] and the classic Gaussian mechanism. According to the discussion in Remark 5, given the l 2 norm constraint, Kashin's representation can be applied to transform it into the l ∞ geometry. Therefore, for ease of discussion, we consider the setting in which each user i stores a vector x i = [ x i, 1 , x i, 2 , · · · , x i,d ] with | x i,j | ≤ c = C √ d , ∀ j , and || x i || 2 ≤ C .

Ternary Stochastic Compressor : Let Z i,j = ternary ( x i,j , A, B ) , then E [ BZ i,j ] = x i,j and V ar ( BZ i,j ) = AB -x 2 i,j . In this sense, applying the ternary stochastic compressor to each coordinate of x i independently yields an unbiased estimator with a variance of ABd -|| x i || 2 2 . The privacy guarantee is given by Theorem 5, and the communication overhead is (log 2 ( d ) + 1) A B d bits in expectation.

SQKR : In SQKR, each user first quantizes each coordinate of x i to {-c, c } with 1-bit stochastic quantization. Then, it samples k coordinates (with replacement) and privatizes the k bit message via the 2 k Random response mechanism with ϵ -LDP [14]. The SQKR mechanism yields an unbiased estimator with a variance of d k ( e ϵ +2 k -1 e ϵ -1 ) 2 C 2 - || x i || 2 2 . The privacy guarantee is ϵ -LDP, and the corresponding communication overhead is (log 2 ( d ) + 1) k bits.

Gaussian Mechanism : We apply the Gaussian mechanism (i.e., adding independent zero-mean Gaussian noise n i,j ∼ N (0 , σ 2 ) to x i,j ), followed by a sparsification probability of 1 -A/B as in ternary ( x i,j , A, B ) , which gives Z Gauss i,j = B A ( x i,j + n i,j ) with probability A/B and Z Gauss i,j = 0 , otherwise. It can be observed that E [ Z Gauss i,j ] = x i,j and V ar ( Z Gauss i,j ) = B A σ 2 +( B A -1) x 2 i,j . Therefore, the Gaussian mechanism yields an unbiased estimator with a variance of B A σ 2 d +( B A -1) || x i || 2 2 . By utilizing the post-processing property, it can be shown that the above Gaussian mechanism is 2 √ dc σ -GDP [15], and the communication overhead is (log 2 ( d ) + 32) A B d bits in expectation.

Discussion : It can be observed that for SQKR, with a given privacy guarantee ϵ -LDP, the variance (i.e., MSE) depends on k (i.e., the communication overhead). When e ϵ ≪ 2 k (which corresponds to the high privacy regime), the variance grows rapidly as k increases. For the proposed ternary stochastic compressor, it can be observed that both the privacy guarantee (in terms of µ -GDP) and the variance depend on AB . Particularly, with a given privacy guarantee µ < √ 4 dr/ (1 -r ) for r = A/B , the variance is given by (4 d/µ 2 +1) C 2 -|| x i || 2 2 , which remains the same regardless of the communication overhead. In this sense, we essentially remove the dependency of accuracy on the communication overhead and therefore break the three-way tradeoff between communication

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

f(α)

Ternary-ε = 1

Ternary-ε = 2

Ternary-ε = 5

SQKR-ε = 1

SQKR-ε = 2

SQKR-ε = 5

0

1

2

3

4

5

0

5

10

15

20

25

30

MSE

Gaussian-Sparsity Ratio 0.2

Gaussian-Sparsity Ratio 0.4

Gaussian-Sparsity Ratio 0.6

Gaussian-Sparsity Ratio 0.8

Gaussian-Sparsity Ratio 1.0

Ternary-Sparsity Ratio 0.2

Ternary-Sparsity Ratio 0.4

Ternary-Sparsity Ratio 0.6

Ternary-Sparsity Ratio 0.8

Ternary-Sparsity Ratio 1.0

0

2

4

6

8

10

0

1

2

3

4

5

MSE

Gaussian-Sparsity Ratio 1.0

Ternary-A = 5c

Ternary-A = 10c

Ternary-A = 20c

Ternary-A = 30c

α

μ

μ

overhead, privacy, and accuracy. 1 This is mainly realized by accounting for privacy amplification by sparsification. At a high level, when fewer coordinates are shared (which corresponds to a larger privacy amplification and a larger MSE), the ternary stochastic compressor introduces less ambiguity to each coordinate (which corresponds to worse privacy protection and a smaller MSE) such that both the privacy guarantee and the MSE remain the same. Since we use different differential privacy measures from [13] (i.e., µ -GDP in this work and ϵ -DP in [13]), we focus on the comparison between the proposed ternary stochastic compressor and the Gaussian mechanism (which is order-optimal in most parameter regimes, see [30]) in the following discussion and present the detailed comparison with SQKR in the experiments in Section 7.

Let AB = c 2 + σ 2 , it can be observed that the f -DP guarantee of the ternary compressor approaches that of the Gaussian mechanism as d increases, and the corresponding variance is given by V ar ( BZ i,j ) = σ 2 + c 2 -x 2 i,j . When A = B , i.e., no sparsification is applied, we have V ar ( BZ i,j ) -V ar ( Z Gauss i,j ) = c 2 -x 2 i,j . Specifically, when x i,j ∈ {-c, c } , ∀ 1 ≤ j ≤ d , the ternary compressor demonstrates the same f -DP privacy guarantee and variance as that for the Gaussian mechanism, i.e., the improvement in communication efficiency is obtained for free (in the large d regime) . When B > A , we have V ar ( BZ i,j ) -V ar ( Z Gauss i,j ) = (1 -B A ) σ 2 + c 2 -B A x 2 i,j , and there exists some B such that the ternary compressor outperforms the Gaussian mechanism in terms of both variance and communication efficiency. It is worth mentioning that the privacy guarantee of the Gaussian mechanism is derived by utilizing the post-processing property. We believe that sparsification brings improvement in privacy for the Gaussian mechanism as well, which is, however, beyond the scope of this paper.

Optimality: It has been shown that, for k -bit unbiased compression mechanisms, there is a lower bound of Ω( C 2 d/k ) in MSE [40]. For the proposed ternary compressor, the MSE and the communication cost are given by O ( ABd ) and A (log( d )+1) d/B bits, respectively. Let k = A (log( d )+1) d/B , it achieves an MSE of O ( A 2 d 2 (log( d ) + 1) /k ) . Since A > c = C/ √ d , the MSE of the ternary compressor is given by O ( C 2 d (log( d )+1) /k ) , which implies that it is order-optimal up to a factor of log( d ) . Note that the factor of log( d ) is used to represent the indices of coordinates that are non-zero, which can be eliminated by allowing for shared randomness between the users and the server.

7 Experiments

In this section, we examine the performance of the proposed ternary compressor in the case of distributed mean estimation. We follow the set-up of [9] and generate N = 1000 user vectors with dimension d = 250 , i.e., x 1 , ..., x N ∈ R 250 . Each local vector has bounded l 2 and l ∞ norms, i.e., || x i || 2 ≤ C = 1 and || x i || ∞ ≤ c = 1 √ d .

Fig. 4 compares the proposed ternary stochastic compressor with SQKR and the Gaussian mechanism. More specifically, the left figure in Fig. 4 compares the privacy guarantees (in terms of the tradeoff between type I and type II error rates) of the ternary stochastic compressor and SQKR given the

same communication overhead and MSE. It can be observed that the proposed ternary stochastic compressor outperforms SQKR in terms of privacy preservation, i.e., given the same type I error rate α , the type II error rate β of the ternary stochastic compressor is significantly larger than that of SQKR, which implies better privacy protection. For example, for SQKR with ϵ = 2 , given type I error rate α = 0 . 5 , the type II error rate of the attacker is around f SQKR ( α ) = 0 . 068 , while the ternary compressor attains f ternary ( α ) = 0.484. Given the same MSE and communication cost as that of SQKR with ϵ SQKR = { 1 , 2 , 5 } , if we translate the privacy guarantees of the ternary compressor from f -DP to ϵ -DP via Lemma 1 (we numerically test different ϵ 's such that f ternary ( α ) ≥ max { 0 , 1 -δ -e ϵ α, e -ϵ (1 -δ -α ) } holds for δ = 0 ), we have ϵ ternary = { 0 . 05 , 0 . 2 , 3 . 9 } for the ternary compressor, which demonstrates its effectiveness. The middle and right figures in Fig. 4 show the tradeoff between MSE and DP guarantees for the Gaussian mechanism and the proposed ternary compressor. Particularly, in the middle figure, the tradeoff curves for the ternary compressor with all the examined sparsity ratios overlap with that of the Gaussian mechanism with A/B = 1 since they essentially have the same privacy guarantees, and the difference in MSE is negligible. For the Gaussian mechanism with A B < 1 , the MSE is larger due to sparsification, which validates our discussion in Section 6. In the right figure, we examine the MSEs of the proposed ternary compressor with various A 's and B 's. It can be observed that the corresponding tradeoff between MSE and privacy guarantee matches that of the Gaussian mechanism well, which validates that the improvement in communication efficiency for the proposed ternary compressor is obtained for free.

8 Limitation

The main results derived in this paper are for the scalar case, which are extended to the vector case by invoking the central limit theorem. In this case, the privacy guarantees derived in Theorem 5 are tight only in the large d regime. Fortunately, in applications like distributed learning, d corresponds to the model size (usually in the orders of millions for modern neural networks). Moreover, despite that the privacy-accuracy tradeoff of the proposed ternary compressor matches that of the Gaussian mechanism which is order-optimal in ( ϵ, δ ) -DP, the optimality of the proposed ternary compressor in the f -DP regime needs to be further established.

9 Conclusion

In this paper, we derived the privacy guarantees of discrete-valued mechanisms with finite output space in the lens of f -differential privacy, which covered various differentially private mechanisms and compression mechanisms as special cases. Through leveraging the privacy amplification by sparsification, a ternary compressor that achieves better accuracy-privacy-communication tradeoff than existing methods is proposed. It is expected that the proposed methods can find broader applications in the design of communication efficient and differentially private federated data analysis techniques.

Acknowledgments and Disclosure of Funding

Richeng Jin was supported in part by the National Natural Science Foundation of China under Grant No. 62301487, in part by the Zhejiang Provincial Natural Science Foundation of China under Grant No. LQ23F010021, and in part by the Ng Teng Fong Charitable Foundation in the form of ZJUSUTD IDEA Grant No. 188170-11102. Zhonggen Su was supported by the Fundamental Research Funds for the Central Universities Grants. Zhaoyang Zhang was supported in part by the National Natural Science Foundation of China under Grant No. U20A20158, in part by the National Key R&D Program of China under Grant No. 2020YFB1807101, and in part by the Zhejiang Provincial Key R&D Program under Grant No. 2023C01021. Huaiyu Dai was supported by the US National Science Foundation under Grant No. ECCS-2203214. The views expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

D. Wang, S. Shi, Y. Zhu, and Z. Han, 'Federated analytics: Opportunities and challenges,' IEEE Network , vol. 36, no. 1, pp. 151-158, 2021.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, 'Communication-efficient learning of deep networks from decentralized data,' in Artificial Intelligence and Statistics . PMLR, 2017, pp. 1273-1282.

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al. , 'Advances and open problems in federated learning,' Foundations and Trends in Machine Learning , vol. 14, no. 1, 2021.

C. Dwork, F. McSherry, K. Nissim, and A. Smith, 'Calibrating noise to sensitivity in private data analysis,' in Theory of cryptography conference . Springer, 2006, pp. 265-284.

I. Mironov, 'On significance of the least significant bits for differential privacy,' in Proceedings of the 2012 ACM conference on Computer and communications security , 2012, pp. 650-661.

N. Agarwal, A. T. Suresh, F. X. X. Yu, S. Kumar, and B. McMahan, 'cpSGD: Communicationefficient and differentially-private distributed SGD,' in Advances in Neural Information Processing Systems , 2018, pp. 7564-7575.

P. Kairouz, Z. Liu, and T. Steinke, 'The distributed discrete gaussian mechanism for federated learning with secure aggregation,' in International Conference on Machine Learning . PMLR, 2021, pp. 5201-5212.

N. Agarwal, P. Kairouz, and Z. Liu, 'The skellam mechanism for differentially private federated learning,' Advances in Neural Information Processing Systems , vol. 34, pp. 5052-5064, 2021.

W.-N. Chen, A. Ozgur, and P. Kairouz, 'The poisson binomial mechanism for unbiased federated learning with secure aggregation,' in International Conference on Machine Learning . PMLR, 2022, pp. 3490-3506.

T. T. Nguyˆ en, X. Xiao, Y . Yang, S. C. Hui, H. Shin, and J. Shin, 'Collecting and analyzing data from smart device users with local differential privacy,' arXiv preprint arXiv:1606.05053 , 2016.

T. Wang, J. Zhao, X. Yang, and X. Ren, 'Locally differentially private data collection and analysis,' arXiv preprint arXiv:1906.01777 , 2019.

V. Gandikota, D. Kane, R. K. Maity, and A. Mazumdar, 'vqsgd: Vector quantized stochastic gradient descent,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2021, pp. 2197-2205.

W.-N. Chen, P. Kairouz, and A. Ozgur, 'Breaking the communication-privacy-accuracy trilemma,' Advances in Neural Information Processing Systems , vol. 33, pp. 3312-3324, 2020.

A. T. Suresh, X. Y. Felix, S. Kumar, and H. B. McMahan, 'Distributed mean estimation with limited communication,' in International conference on machine learning . PMLR, 2017, pp. 3329-3337.

J. Dong, A. Roth, and W. Su, 'Gaussian differential privacy,' Journal of the Royal Statistical Society , 2021.

I. Mironov, 'R´ enyi differential privacy,' in IEEE Computer Security Foundations Symposium (CSF) . IEEE, 2017, pp. 263-275.

A. El Ouadrhiri and A. Abdelhadi, 'Differential privacy for deep and federated learning: A survey,' IEEE Access , vol. 10, pp. 22 359-22 380, 2022.

C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, 'Our data, ourselves: Privacy via distributed noise generation,' in Annual international conference on the theory and applications of cryptographic techniques . Springer, 2006, pp. 486-503.

S. P. Kasiviswanathan, H. K. Lee, K. Nissim, S. Raskhodnikova, and A. Smith, 'What can we learn privately?' SIAM Journal on Computing , vol. 40, no. 3, pp. 793-826, 2011.

C. L. Canonne, G. Kamath, and T. Steinke, 'The discrete gaussian for differential privacy,' Advances in Neural Information Processing Systems , vol. 33, pp. 15 676-15 688, 2020.

G. Cormode and I. L. Markov, 'Bit-efficient numerical aggregation and stronger privacy for trust in federated analytics,' arXiv preprint arXiv:2108.01521 , 2021.

P. Kairouz, S. Oh, and P. Viswanath, 'Extremal mechanisms for local differential privacy,' The Journal of Machine Learning Research , vol. 17, no. 1, pp. 492-542, 2016.

A. Girgis, D. Data, S. Diggavi, P. Kairouz, and A. T. Suresh, 'Shuffled model of differential privacy in federated learning,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2021, pp. 2521-2529.

A. M. Girgis, D. Data, S. Diggavi, P. Kairouz, and A. T. Suresh, 'Shuffled model of federated learning: Privacy, accuracy and communication trade-offs,' IEEE journal on selected areas in information theory , vol. 2, no. 1, pp. 464-478, 2021.

V. Feldman and K. Talwar, 'Lossless compression of efficient private local randomizers,' in International Conference on Machine Learning . PMLR, 2021, pp. 3208-3219.

A. Shah, W.-N. Chen, J. Balle, P. Kairouz, and L. Theis, 'Optimal compression of locally differentially private mechanisms,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2022, pp. 7680-7723.

K. Chaudhuri, C. Guo, and M. Rabbat, 'Privacy-aware compression for federated data analysis,' in The 38th Conference on Uncertainty in Artificial Intelligence , 2022.

A. Koskela, J. J¨ alk¨ o, L. Prediger, and A. Honkela, 'Tight differential privacy for discrete-valued mechanisms and for the subsampled gaussian mechanism using fft,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2021, pp. 3358-3366.

A. Koskela, J. J¨ alk¨ o, and A. Honkela, 'Computing tight differential privacy guarantees using fft,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2020, pp. 2560-2569.

W.-N. Chen, D. Song, A. Ozgur, and P. Kairouz, 'Privacy amplification via compression: Achieving the optimal privacy-accuracy-communication trade-off in distributed mean estimation,' arXiv preprint arXiv:2304.01541 , 2023.

C. Dwork and G. N. Rothblum, 'Concentrated differential privacy,' arXiv preprint arXiv:1603.01887 , 2016.

M. Bun, C. Dwork, G. N. Rothblum, and T. Steinke, 'Composable and versatile privacy via truncated cdp,' in Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , 2018, pp. 74-86.

Q. Zheng, J. Dong, Q. Long, and W. Su, 'Sharp composition bounds for gaussian differential privacy via edgeworth expansion,' in International Conference on Machine Learning . PMLR, 2020, pp. 11 420-11 435.

Z. Bu, J. Dong, Q. Long, and W. J. Su, 'Deep learning with gaussian differential privacy,' Harvard data science review , vol. 2020, no. 23, 2020.

Q. Zheng, S. Chen, Q. Long, and W. Su, 'Federated f-differential privacy,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2021, pp. 2251-2259.

R. Jin, Y. Huang, X. He, H. Dai, and T. Wu, 'Stochastic-Sign SGD for federated learning with theoretical guarantees,' arXiv preprint arXiv:2002.10940 , 2020.

W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, 'TernGrad: Ternary gradients to reduce communication in distributed deep learning,' in Advances in Neural Information Processing Systems , 2017, pp. 1509-1519.

M. Safaryan, E. Shulgin, and P. Richt´ arik, 'Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor,' arXiv preprint arXiv:2002.08958 , 2020.

Y. Lyubarskii and R. Vershynin, 'Uncertainty principles and vector quantization,' IEEE Transactions on Information Theory , vol. 56, no. 7, pp. 3491-3501, 2010.

W.-N. Chen, C. A. C. Choo, P. Kairouz, and A. T. Suresh, 'The fundamental price of secure aggregation in differentially private federated learning,' in International Conference on Machine Learning . PMLR, 2022, pp. 3056-3089.

E. L. Lehmann, J. P. Romano, and G. Casella, Testing statistical hypotheses . Springer, 2005, vol. 3.

J. Lee, M. Kim, S. W. Kwak, and S. Jung, 'Differentially private multivariate statistics with an application to contingency table analysis,' arXiv preprint arXiv:2211.15019 , 2022.

Y. Liu, K. Sun, L. Kong, and B. Jiang, 'Identification, amplification and measurement: A bridge to gaussian differential privacy,' arXiv preprint arXiv:2210.09269 , 2022.

Breaking the Communication-Privacy-Accuracy Tradeoff with f -Differential Privacy: Supplementary Material

A Tradeoff Functions for a Generic Discrete-Valued Mechanism

We consider a general randomization protocol M ( · ) with discrete and finite output space. In this case, we can always find a one-to-one mapping between the range of M ( · ) and a subset of Z . With such consideration, we assume that the output of the randomization protocol is an integer, i.e., M ( S ) ∈ Z M ⊂ Z , ∀ S , without loss of generality. Given the randomization protocol and the hypothesis testing problem in (2), we derive its tradeoff function as a function of the type I error rate in the following lemma.

Lemma 2. For two neighboring datasets S and S ′ , suppose that the range of the randomized mechanism R ( M ( S )) ∪ R ( M ( S ′ )) = Z U M = [ Z U L , . . . , Z U R ] ⊂ Z and R ( M ( S )) ∩ R ( M ( S ′ )) = Z I M = [ Z I L , . . . , Z I R ] ⊂ Z . Let X = M ( S ) and Y = M ( S ′ ) . Then,

Case (1) If M ( S ) ∈ [ Z I L , Z I L + 1 , . . . , Z U R ] , M ( S ′ ) ∈ [ Z U L , Z U L + 1 , . . . , Z I R ] , and P ( Y = k ) P ( X = k ) is a decreasing function of k for k ∈ Z I M , the tradeoff function in Definition 2 is given by

Case (2) If M ( S ) ∈ [ Z U L , Z U L +1 , · · · , Z I R ] , M ( S ′ ) ∈ [ Z I L , Z I L +1 , · · · , Z U R ] , and P ( Y = k ) P ( X = k ) is an increasing function of k for k ∈ Z I M , the tradeoff function in Definition 2 is given by

Remark 6. It is assumed in Lemma 2 that P ( Y = k ) P ( X = k ) is a decreasing function (for part (1)) or an increasing function (for part (2)) of k ∈ Z I M , without loss of generality. In practice, thanks to the post-processing property of DP [15], one can relabel the output of the mechanism to ensure that this condition holds and Lemma 2 can be adapted accordingly.

Remark 7. We note that in Lemma 2, both X and Y depend on both the randomized mechanism M ( · ) and the neighboring datasets S and S ′ . Therefore, the infimums of the tradeoff functions in (16) and (17) are mechanism-specific, which should be analyzed individually. After identifying the neighboring datasets S and S ′ that minimize β + ϕ ( α ) and β -ϕ ( α ) for a mechanism M ( · ) (which is highly non-trivial), we can obtain the distributions of X and Y in (16) and (17) and derive the corresponding f -DP guarantees.

Remark 8. Since β + ϕ ( α ) is a piecewise function with decreasing slopes w.r.t k (see, e.g., Fig. 1), it can be readily shown that β + ϕ ( α ) ≥ max { P ( Y ≥ k ) + P ( Y = k ) P ( X = k ) P ( X < k ) -P ( Y = k ) P ( X = k ) α, 0 } , ∀ k ∈ Z I M . As a result, utilizing Lemma 1, we may obtain different pairs of ( ϵ, δ ) given different k 's.

Remark 9. Although we assume a finite output space, a similar method can be applied to the mechanisms with an infinite range. Taking the discrete Gaussian noise [20] as an example, M ( x ) = x + V with P ( V = v ) = e -v 2 / 2 σ 2 ∑ v ∈ Z e -v 2 / 2 σ 2 . One may easily verify that P ( M ( x i )= k ) P ( M ( x ′ i )= k ) is a decreasing function of k if x ′ i > x i (and increasing otherwise). Then we can find some threshold v for the rejection rule ϕ such that α ϕ = P ( M ( x i ) ≤ v ) = α , and the corresponding β ϕ ( α ) = 1 -P ( M ( x ′ i ) ≤ v ) .

The key to proving Lemma 2 is finding the rejection rule ϕ such that β ϕ ( α ) is minimized for a pre-determined α ∈ [0 , 1] . To this end, we utilize the Neyman-Pearson Lemma [41], which states that for a given α , the most powerful rejection rule is threshold-based, i.e., if the likelihood ratio P ( Y = k ) P ( X = k ) is larger than/equal to/smaller than a threshold h , H 0 is rejected with probability 1/ γ /0. More specifically, since X and Y may have different ranges, we divide the discussion into two cases (i.e., Case (1) and Case (2) in Lemma 2). The Neyman-Pearson Lemma [41] is given as follows.

Lemma 3. (Neyman-Pearson Lemma [41]) Let P and Q be probability distributions on Ω with densities p and q , respectively. For the hypothesis testing problem H 0 : P vs H 1 : Q , a test ϕ : Ω → [0 , 1] is the most powerful test at level α if and only if there are two constants h ∈ [0 , + ∞ ] and γ ∈ [0 , 1] such that ϕ has the form

and E P [ ϕ ] = α . The rejection rule suggests that H 0 is rejected with a probability of ϕ ( x ) given the observation x .

Given Lemma 3, the problem is then reduced to finding the corresponding h and γ such that the type I error rate α ϕ = α . For part (1) (the results for part (2) can be shown similarly), we divide the range of α (i.e., [0 , 1] ) into multiple segments, as shown in Fig. 5. To achieve α = 0 , we set h = ∞ and γ = 1 , which suggests that the hypothesis H 0 is always rejected when k < Z I L and accepted otherwise. To achieve α ∈ ( P ( X < k ) , P ( X ≤ k )] , for k ∈ [ Z I L , Z I R ] , we set h = P ( Y = k ) P ( X = k ) and γ = α -P ( X<k ) P ( X = k ) . In this case, it can be shown that α ϕ = α ∈ ( P ( X < k ) , P ( X ≤ k )] . To achieve α ∈ ( P ( X < Z I R +1) , 1] , we set h = 0 , and γ = α -P ( X< Z I R +1) P ( X> Z I R ) . In this case, it can be shown that α ϕ = α ∈ ( P ( X < Z I R +1) , 1] . The corresponding β ϕ can be derived accordingly, which is given by (16). The complete proof is given below.

Proof. Given Lemma 3, the problem is reduced to finding the parameters h and γ in (18) such that E P [ ϕ ] = α , which can be proved as follows.

Case (1) We divide α ∈ [0 , 1] into Z U R -Z I L +1 segments: [ P ( X < Z U L ) , P ( X < Z I L )] ∪ ( P ( X < Z I L ) , P ( X ≤ Z I L )] ∪··· ∪ ( P ( X < k ) , P ( X ≤ k )] ∪··· ∪ ( P ( X < Z U R ) , P ( X ≤ Z U R )] , as shown in Fig. 5.

When α = P ( X < Z U L ) = P ( X < Z I L ) = 0 , we set h = + ∞ . In this case, noticing that P ( Y = k ) P ( X = k ) = h for k < Z I L , and P ( Y = k ) P ( X = k ) < h otherwise, we have

The infimum is attained when γ = 1 , which yields β + ϕ (0) = P ( Y ≥ Z I L ) .

We adjust γ such that E P [ ϕ ] = α , which yields

and and

When α ∈ ( P ( X < k ) , P ( X ≤ k )] for k ∈ ( Z I R , Z U R ] , we set h = 0 . In this case, P ( Y = k ′ ) P ( X = k ′ ) = h for k ′ > Z I R , and P ( Y = k ′ ) P ( X = k ′ ) > h for k ′ ≤ Z I R . As a result,

Similarly, we can prove the second part of Lemma 2 as follows.

Case (2) We also divide α ∈ [0 , 1] into Z U R -Z I L +1 segments: [ P ( X > Z U L ) , P ( X ≥ Z U L )] ∪ · · · ∪ ( P ( X > k ) , P ( X ≥ k )] ∪ · · · ∪ ( P ( X > Z I R ) , P ( X ≥ Z I R )] , as shown in Fig. 6.

and and

and

When

α

∈

(

P

(

X > k

)

, P

Setting E P [ ϕ ] = α yields

and

(

X

k

)]

for

k

∈

[

Z

I

L

,

Z

I

R

]

, we set

h

=

P

P

(

(

Y

X

=

k

=

)

k

)

. In this case,

in which

The infimum is attained when γ = 1 , which yields β -ϕ (0) = P ( Y ≤ Z I R ) .

B Proofs of Theoretical Results

B.1 Proof of Theorem 1

Theorem 1. Let ˜ Z = Binom ( M,p ) , the binomial noise mechanism in Algorithm 1 is f bn ( α ) -differentially private with

Given that P ( ˜ Z = k ) = ( M k ) p k (1 -p ) M -k , it can be readily shown that when p = 0 . 5 , both β + ϕ, inf ( α ) and β -ϕ, inf ( α ) are maximized, and f ( α ) = β + ϕ, inf ( α ) = β -ϕ, inf ( α ) .

Before proving Theorem 1, we first show the following lemma.

Proof of Lemma 4. When x i > x ′ i , it can be easily verified that P ( X = k ) > 0 only for k ∈ [ x i , x i + 1 , · · · , x i + M ] , P ( Y = k ) > 0 only for k ∈ [ x ′ i , x ′ i +1 , · · · , x ′ i + M ] . For k ∈ [ x i , · · · , x ′ i + M ] , we have

It can be observed that P ( Y = k ) P ( X = k ) is a decreasing function of k .

When x i < x ′ i , it can be easily verified that P ( X = k ) > 0 only for k ∈ [ x i , x i +1 , · · · , x i + M ] , P ( Y = k ) > 0 only for k ∈ [ x ′ i , x ′ i +1 , · · · , x ′ i + M ] . For k ∈ [ x ′ i , · · · , x i + M ] , we have

It can be observed that P ( Y = k ) P ( X = k ) is an increasing function of k , and invoking Lemma 2 completes the proof.

Given Lemma 4, we are ready to prove Theorem 1.

Proof of Theorem 1. Let ˜ Z = Binom ( M,p ) , X = x i + ˜ Z and Y = x ′ i + ˜ Z . Two cases are considered:

Case 1: x i > x ′ i .

In this case, according to Lemma 4, we have

In the following, we show the infimum of β + ϕ ( α ) . For the ease of presentation, let ˜ k = k -x i and x i -x ′ i = ∆ . Then, we have

(40) can be rewritten as

Since α ∈ [ P ( ˜ Z < ˜ k ) , P ( ˜ Z ≤ ˜ k )] , we have P ( ˜ Z < ˜ k ) -α ∈ [ -P ( ˜ Z = ˜ k ) , 0] . If P ( ˜ Z = ˜ k + ∆ + 1) -P ( ˜ Z = ˜ k + ∆) > 0 , J (∆ + 1 , ˜ k ) -J (∆ , ˜ k ) < -P ( ˜ Z = ˜ k + ∆) < 0 . If P ( ˜ Z = ˜ k +∆+1) -P ( ˜ Z = ˜ k +∆) < 0 , J (∆ + 1 , ˜ k ) -J (∆ , ˜ k ) < -P ( ˜ Z = ˜ k +∆+1) < 0 . As a result, the infimum of β + ϕ ( α ) is attained when ∆ = l , i.e., x i = l and x ′ i = 0 , which yields

Case 2: x i < x ′ i .

In this case, according to Lemma 4, we have

In the following, we show the infimum of β ( α ) . For the ease of presentation, let ˜ k = k -x i and x ′ i -x i = ∆ . Then, we have

(45) can be rewritten as

Since α ∈ [ P ( ˜ Z > ˜ k ) , P ( ˜ Z ≥ ˜ k )] , we have P ( ˜ Z > ˜ k ) -α ∈ [ -P ( ˜ Z = ˜ k ) , 0] . If P ( ˜ Z = ˜ k -∆ -1) -P ( ˜ Z = ˜ k -∆) > 0 , then J (∆ + 1 , ˜ k ) -J (∆ , ˜ k ) < -P ( ˜ Z = ˜ k -∆) < 0 . If P ( ˜ Z = ˜ k -∆ -1) -P ( ˜ Z = ˜ k -∆) < 0 , then J (∆+1 , ˜ k ) -J (∆ , ˜ k ) < -P ( ˜ Z = ˜ k -∆ -1) < 0 . As a result, the infimum of β -ϕ ( α ) is attained when ∆ = l , i.e., x i = 0 and x ′ i = l , which yields

Combining (44) and (49) completes the first part of the proof. When p = 0 . 5 , it can be found that both β + ϕ, inf ( α ) and β -ϕ, inf ( α ) are maximized, and f ( α ) = β + ϕ, inf ( α ) = β -ϕ, inf ( α ) .

B.2 Proof of Theorem 2

Theorem 2. The binomial mechanism in Algorithm 2 is f bm ( α ) -differentially private with f bm ( α ) = min { β + ϕ, inf ( α ) , β -ϕ, inf ( α ) } , in which

Proof. Observing that the output space of the binomial mechanism remains the same for different data x i , i.e., Z I L = Z U L = 0 and Z I R = Z U R = M in Lemma 2. Moreover, let X = Binom ( M,p ) and Y = Binom ( M,q ) , we have P ( Y = k ) P ( X = k ) = ( M k ) q k (1 -q ) M -k ( M k ) p k (1 -p ) M -k = ( 1 -q 1 -p ) M ( q (1 -p ) p (1 -q ) ) k . Similarly, we consider the following two cases.

Case 1: q < p .

In this case, we can find that P ( Y = k ) P ( X = k ) is a decreasing function of k . Therefore, according to Lemma 2, we have

In the following, we show that the infimum is attained when p = p max and q = p min . For Binomial distribution Y , we have ∂P ( Y <k ) ≤ 0 and ∂P ( Y ≤ k ) ≤ 0 , ∀ k .

Therefore, the infimum is attained when q = p min .

Suppose X = Binom ( M,p ) and ˆ X = Binom ( M, ˆ p ) . Without loss of generality, assume p > ˆ p . Suppose that α ∈ [ P ( X < k ) , P ( X ≤ k )] and α ∈ [ P ( ˆ X < ˆ k ) , P ( ˆ X ≤ ˆ k )] for some k and ˆ k are satisfied simultaneously, it can be readily shown that k ≥ ˆ k . In addition, α ∈ [max { P ( X <

and

Obviously, P ( Y ≥ k ) -P ( Y ≥ ˆ k ) ≤ 0 and P ( Y > k ) -P ( Y > ˆ k ) ≤ 0 for k ≥ ˆ k . Observing that β + ϕ,p ( α ) -β + ϕ, ˆ p ( α ) is a linear function of α ∈ [max { P ( X < k ) , P ( ˆ X < ˆ k ) } , min { P ( X ≤ k ) , P ( ˆ X ≤ ˆ k ) } ] given Y , X , ˆ X , k and ˆ k , we consider the following four possible cases:

P ( X < k ) ≤ P ( ˆ X < ˆ k ) and α = P ( ˆ X < ˆ k ) : In this case, P ( Y = k )[ P ( X<k ) -α ] P ( X = k ) = P ( Y = k )[ P ( X<k ) -P ( ˆ X< ˆ k )] P ( X = k ) ≤ 0 . As a result, β + ϕ,p ( α ) -β + ϕ, ˆ p ( α ) ≤ 0 .

When k = ˆ k , since p > ˆ p , we have P ( ˆ X < ˆ k ) -P ( X < ˆ k ) > 0 , which violates the condition that P ( X < k ) > P ( ˆ X < ˆ k ) .

As a result, β + ϕ,p ( α ) -β + ϕ, ˆ p ( α ) ≤ P ( Y > k ) -P ( Y > ˆ k ) ≤ 0 .

4) P ( X ≤ k ) > P ( ˆ X ≤ ˆ k ) and α = P ( ˆ X ≤ ˆ k ) : In this case, when k = ˆ k , P ( X ≤ ˆ k ) -P ( ˆ X ≤ ˆ k ) > 0 , which violates the condition that P ( X ≤ k ) > P ( ˆ X ≤ ˆ k ) .

Since k > ˆ k , P ( Y ≥ k ) -P ( Y > ˆ k ) ≤ 0 . In addition, P ( X < k ) -P ( ˆ X ≤ ˆ k ) ≤ 0 since α ∈ [max { P ( X < k ) , P ( ˆ X < ˆ k ) } , P ( ˆ X ≤ ˆ k )] . As a result, β + ϕ,p ( α ) -β + ϕ, ˆ p ( α ) ≤ P ( Y > k ) -P ( Y > ˆ k ) ≤ 0 .

Now that β + ϕ,p ( α ) -β + ϕ, ˆ p ( α ) is a linear function of α ∈ [max { P ( X < k ) , P ( ˆ X < ˆ k ) } , min { P ( X ≤ k ) , P ( ˆ X ≤ ˆ k ) } ] , which is non-positive in the extreme points (i.e., the boundaries), we can conclude that β + ϕ,p ( α ) -β + ϕ, ˆ p ( α ) ≤ 0 for any α ∈ [max { P ( X < k ) , P ( ˆ X < ˆ k ) } , min { P ( X ≤ k ) , P ( ˆ X ≤ ˆ k ) } ] . Therefore, the infimum of β + ϕ ( α ) is attained when p = p max .

Case 2: q > p .

In this case, we can find that P ( Y = k ) P ( X = k ) is an increasing function of k . As a result, according to Lemma 2, we have

Similarly, it can be shown that the infimum is attained when q = p and p = p max min As a result, we have

.

B.3 Proof of Theorem 3

Theorem 3. The ternary stochastic compressor is f ternary ( α ) -differentially private with

We provide the f -DP analysis for a generic ternary stochastic compressor defined as follows.

Definition 8 ( Generic Ternary Stochastic Compressor ) . For any given x ∈ [ -c, c ] , the generic compressor ternary outputs ternary ( x, p 1 , p 0 , p -1 ) , which is given by

where p 0 is the design parameter that controls the level of sparsity and p 1 ( x ) , p -1 ( x ) ∈ [ p min , p max ] . It can be readily verified that p 1 = A + x 2 B , p 0 = 1 -A B , p -1 = A -x 2 B (and therefore p min = A -c 2 B and p max = A + c 2 B ) for the ternary stochastic compressor in Definition 6.

In the following, we show the f -DP of the generic ternary stochastic compressor, and the corresponding f -DP guarantee for the compressor in Definition 6 can be obtained with p min = A -c 2 B , p max = A + c 2 B , and p 0 = 1 -A B .

Lemma 5. Suppose that p 0 is independent of x , p max + p min = 1 -p 0 , and p 1 ( x ) > p 1 ( y ) , ∀ x > y . The ternary compressor is f ternary ( α ) -differentially private with

Proof. Similar to the binomial mechanism, the output space of the ternary mechanism remains the same for different inputs. Let Y = ternary ( x ′ i , p 1 , p 0 , p -1 ) and X = ternary ( x i , p 1 , p 0 , p -1 ) , we have

When x i > x ′ i , it can be observed that P ( Y = k ) P ( X = k ) is a decreasing function of k . According to Lemma 2, we have

When x i < x ′ i , it can be observed that P ( Y = k ) P ( X = k ) is an increasing function of k . According to Lemma 2, we have

The infimum of β + ϕ ( α ) is attained when p -1 ( x ′ i ) = p max and p -1 ( x i ) = p min , while the infimum of β -ϕ ( α ) is attained when p 1 ( x ′ i ) = p max and p 1 ( x i ) = p min . As a result, we have

which completes the proof.

B.4 Proof of Theorem 4

Theorem 4. Given a vector x i = [ x i, 1 , x i, 2 , · · · , x i,d ] with | x i,j | ≤ c, ∀ j . Applying the ternary compressor to the j -th coordinate of x i independently yields µ -GDP with µ = -2Φ -1 ( 1 1+( A + c A -c ) d ) .

Before proving Theorem 4, we first introduce the following lemma.

Lemma 6. [42, 43] Any ( ϵ, 0) -DP algorithm is also µ -GDP for µ = -2Φ -1 ( 1 1+ e ϵ ) , in which Φ( · ) is the cumulative density function of normal distribution.

Proof. According to Theorem 3, in the scalar case, the ternary stochastic compressor is f ternary ( α ) -differentially private with

It can be easily verified that f ternary ( α ) ≥ max { 0 , 1 -( A + c A -c ) α, ( A -c A + c )(1 -α ) } . Invoking Lemma 1 suggests that it is (log( A + c A -c ) , 0) -DP. Extending it to the d -dimensional case yields ( d log( A + c A -c ) M , 0) -DP. As a result, according to Lemma 6, it is -2Φ -1 ( 1 1+( A + c A -c ) d ) -GDP.

B.5 Proof of Theorem 5

Theorem 5. For a vector x i = [ x i, 1 , x i, 2 , · · · , x i,d ] with | x i,j | ≤ c, ∀ j , the ternary compressor with B ≥ A > c is f ternary ( α ) -DP with

in which

Before proving Theorem 5, we first define the following functions as in [15],

The central limit theorem for f -DP is formally introduced as follows.

Lemma 7 ([15]) . Let f 1 , ..., f n be symmetric trade-off functions such that κ 3 ( f i ) < ∞ for all 1 ≤ i ≤ d . Denote

Given Lemma 7, we are ready to prove Theorem 5.

Proof. Given f i ( α ) in (62), we have

The corresponding µ and γ are given as follows

which completes the proof.

C f -DP of the Poisson Binomial Mechanism

The Poisson binomial mechanism [9] is presented in Algorithm 3. In the following, we show the

Algorithm 3 Poisson Binomial Mechanism

f -DP guarantee of the Poisson binomial mechanism with M = 1 . The extension to the proof for M > 1 is straightforward by following a similar technique.

Theorem 6. The Poisson binomial mechanism with M = 1 in Algorithm 3 is f pb ( α ) -differentially private with

Proof. For Poisson Binomial, let

in which PB stands for Poisson Binomial. In this case,

In addition,

P

(

Y

=

k

+1)

P

(

X

=

k

)

-

P

(

Y

=

k

)

P

(

X

=

k

+1)

Since P ( Z = k +1) P ( Z = k -1) -( P ( Z = k )) 2 < 0 for Poisson Binomial distribution, we have

That being said, P ( Y = k ) P ( X = k ) is an increasing function of k if p i < p ′ i and a decreasing function of k if p i > p ′ i . Following the same analysis as that in the proof of Theorem 2, for p i > p ′ i , we have for α ∈ [ P ( X < k

In the following, we show that the infimum of β + ϕ ( α ) is attained when p i = p max and p ′ i = p min .

Case 1: k = 0 . In this case,

Plugging (89) into (88) yields

It is obvious that the infimum is attained when p i = p max and p ′ i = p min .

Case 2: k > 0 . In this case,

Plugging (91) into (88) yields

The p ′ i related term is given by

Observing that (93) is a linear function of α , we only need to examine α ∈ { P ( X < k ) , P ( X ≤ k ) } . More specifically, when α = P ( X ≤ k ) , it is reduced to P ( Z = k ) p ′ i ; when α = P ( X < k ) , it is reduced to P ( Z = k -1) p ′ i . In both cases, the infimum is attained when p ′ i = p min .

Given that p ′ i = p min , the same technique as in the proof of Theorem 2 can be applied to show that the infimum is attained when p = p max .

Since P ( Y = k ) P ( X = k ) is a decreasing function of k when p i > p ′ i , we have

Given that β + ϕ ( α ) is a decreasing function of α with β + ϕ (0) = 1 and β + ϕ (1) = 0 , we can readily conclude that β + ϕ ( α ) ≥ max { 0 , 1 -1 -p min 1 -p max α } and β + ϕ ( α ) ≥ p min p max (1 -α ) . That being said, β + ϕ ( α ) ≥ max { 0 , 1 -1 -p min 1 -p max α, p min p max (1 -α ) } .

Similarly, for p i < p ′ i , we have

for α ∈ [ P ( X > k ) , P ( X ≥ k )] and k ∈ { 0 , 1 , 2 , · · · , N } . The infimum is attained when p i = p min , p ′ i = p max .

Since P ( Y = k ) P ( X = k ) is an increasing function of k when p i < p ′ i , we have

Given that β -ϕ ( α ) is an increasing function of α with β -ϕ (0) = 1 and β -ϕ (1) = 0 , we can easily conclude that β -ϕ ( α ) ≥ max { 0 , 1 -p max p min α } and β -ϕ ( α ) ≥ 1 -p max 1 -p min (1 -α ) . That being said, β -ϕ ( α ) ≥ max { 0 , 1 -p max p min α, 1 -p max 1 -p min (1 -α ) } .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies distributed mean estimation under privacy and communication constraints. This paper focuses on characterizing the recently defined notion of $f$-DP for communication-efficient mechanisms, where $f$-DP can be converted to the standard $(\\epsilon,\\delta)$-DP. The paper analyzed the $f$-DP of different known mechanisms in the literature. Furthermore, a new algorithm named ternary compression is proposed.\n\nSTRENGTHS:\nThe distributed mean estimation is an important topic and has lots of applications in federated learning under privacy and communication constraints. The presentation is well written.\n\nWEAKNESSES:\nAlthough the topic is interesting, my main concerns are about the contributions of this paper. \n\n * The first part of the paper is to analyze the $f$-DP of well-known discrete mechanisms. While analyzing the $f$-DP is useful, it is not hard to compute it. I don't know what are the major challenges in this analysis and if this analysis provides any new insights into analyzing the $f$-DP. Hence, in my opinion, this part isn't novel enough as a main contribution of this paper.\n* The expressions of the $f$-DP are not in a closed form; the expressions are functions of a set of probabilities that might be computationally expensive to compute for large system parameters.\n* The proposed mechanism of ternary compression is not novel. There are lots of similar schemes that don't send some coordinates with some probability from sampling the coordinates for example [13] and [23]. Actually, the Ternary mechanism can be seen as a combination of CLDP for a set of coordinates that are chosen i.i.d. with probability $B/A$. \n* In general, this ternary mechanism has higher privacy because it has lower accuracy, since with some non-zero probability (when $B>A$), the client will not send anything. \n\n*  The optimal trade-offs between privacy, communication, and accuracy have already been characterized for LDP in different works in the literature, and hence, I don't understand what is the major difference in this paper in comparison with the existing work.\n\n* Typos: please define the abbreviation of GDP (Gaussian differential privacy) for using it. In Algorithm 1, $x_i\\in\\lbrace 0,1,\\ldots,l \\rbrace$ instead of square brackets. Figure 4 is small and the lines are close enough so that it is hard to distinguish the different mechanisms.\n\nQUESTIONS:\n* It is mentioned in the abstract and also in the introduction that *it remains an open problem whether such discrete-valued mechanisms provide any privacy protection.* \nI don't know which open problem the authors referring to. If it is the distributed mean estimation under jointly local differential privacy and communication constraints, then this problem is already solved in the literature.\n\n* In line 51, *SQKR doesn't account for the privacy introduced during sparsification.* I don't understand this part. AFAIK, SQKR is order optimal, and hence, it doesn't lose privacy analysis. \n\n* In lines 259-261, what do these numbers of $\\epsilon$ refer to, and from where it comes?\n\n* In lines 331-333, *we essentially remove the dependency of accuracy on the communication overhead.* Could you please explain more? In general, it is supposed that there is a trade-off between communication and accuracy, so the accuracy depends on the communication budget.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the f-DP guarantee of several discrete-valued mechanisms in the local-DP model. In particular, closed-form expressions for binomial noise mechanism and Binomial mechanics are derived. \n\nThen, the paper considers the popular problem of aggregating d-dimensional vectors from local users subject to privacy communication constraints. Under the f-DP framework, this paper presents a new \"ternary\" mechanism and analyzes its f-DP privacy guarantee. Roughly, the ternary mechanism allows each local user to sparsify their vectors by keeping each coordinate with only a small probability and adding noises to the alive coordinates. The hope is that by sparsification and randomization, one can optimize communication (i.e., minimizing the messages from each user to the server) and privacy cost (e.g., each user has its signal \"somewhat hidden\" in the noises).\n\nThe paper's main claim is that, by working with f-DP and the ternary mechanism, we can benefit in **both** privacy and communication by the sparsification process in the ternary mechanism. Intuitively, if the sparsifying threshold is high (i.e., we only keep very few coordinates), we can hope to add less noise to achieve privacy, as the sparsification has already introduced some uncertainty in the data. The paper claims that this is something that prior works cannot offer. The reviewer, unfortunately, did not have a chance to verify the claim.\n\nSTRENGTHS:\n* Working with f-DP **and** in the local model seems like a new approach.\n* The proposed algorithm is natural and simple to implement, it is backed by the theory and performs favorably in the experiments.\n\nWEAKNESSES:\n* It's a bit hard to interpret/digest the closed-form f-DP guarantee of all these mechanisms. Have some plot/converting to Renyi-DP or (eps,delta)-DP might help.\n\nQUESTIONS:\n* Your algorithm sends **in expection** O(d * A/B) bits. Is there a way to improve this to a worst-case guarantee? How about, say, randomly selecting (A/B * d) coordinates? Is it possible to work out the f-DP (or any other reasonable DP notion) property of this variant?\n\nLIMITATIONS:\nSome future directions are mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper analyses the privacy that is provided by stochastic rounding methods when doing distributed mean estimation with local differential privacy. It finds that they can contribute to the privacy guarantee thus achieving a better tradeoff.

It then "breaks" the privacy communication utility trade-off in the sense of pointing out that for the high privacy regime extra communication won't buy you any more accuracy.

STRENGTHS:
Previous naive means of implementing DP aggregation with randomised rounding have failed to take advantage of the privacy provided by stochastic compression. This should be utilised if possible which they attempt to do.

WEAKNESSES:
The presentation fails to show a decent comparison showing hwo much is really gained by taking advantage of this randomness. It isn't clear the gain isn't negligible. There is a graph comparing ternary and binary quantisation but it isn't clear whether the parameters actually correspond to low communication, which would require sufficient sparsity.

The supposed break of the three way trade off is really just the observation that in the high privacy regime there is no gain to lots of communication. This is not what break means, the title is overselling.

What is more this observation is very far from new. The idea that O(epsilon) communication suffices goes back a long way for various problems. This was proven with complete generality in https://arxiv.org/pdf/2102.12099.pdf.

QUESTIONS:
Fixing two of privacy/communication/utility how much does taking advantage of the randomness in the compression actually help the other one?

Am I missing something with the breaking of the trade-off?

LIMITATIONS:
They don't really talk about the limitations, indeed the limitations aren't really clear from the presentation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn a federated setting where a server coordinates the collaborative analysis of multiple users with local data, communication efficiency and data privacy are two major issues of consideration. Classical DP mechanisms such as Laplace or Gaussian mechanisms add noises as real numbers -- at the same time, to save communication cost it is preferred to have discrete noises with limited range (resolution). This paper considers all three aspects: communication efficiency, privacy, and accuracy. Specifically, the authors would like to study privacy guarantee of data compression schemes, which intuitively introduces information loss and thus is good for protecting privacy.\n \nThe authors use f-DP, and considers binomial mechanisms. Binomial mechanisms have been considered in [6] and [9], this paper provides tighter bounds. In addition, the authors also consider a compression scheme that optimizes for communication efficiency. This is a combination of a sign based quantization and sparsification.\n\nSTRENGTHS:\nI like the motivation of the paper and the consideration of all three objectives, communication efficiency, privacy and accuracy together. The tighter analysis and mechanisms for communication efficiency is of use in practice. \n\nAlthough the work builds on prior work the improvement by considering compression for privacy amplification is nice. The paper is generally well written.\n\nWEAKNESSES:\nA few minor issues, see below.\n\nQUESTIONS:\nAre there any lower bounds on the tradeoff of the three considerations? Any discussion here would be useful.\n\nWriting: it would be good to summarize somewhere in the paper the best recommendation for practitioners, what parameters to choose etc.\n\nIs there general improvement (amplification of privacy) using compression/sparsification? Can the authors elaborate/discuss this?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}