Title: Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributed mean estimation

privacy

compression

communication

federated analytics.
Abstract: We study the mean estimation problem under communication and local differential privacy constraints. While previous work has proposed order-optimal algorithms for the same problem (i.e., asymptotically optimal as we spend more bits), exact optimality (in the non-asymptotic setting) still has not been achieved. In this work, we take a step towards characterizing the exact-optimal approach in the presence of shared randomness (a random variable shared between the server and the user) and identify several conditions for exact optimality. We prove that one of the conditions is to utilize a rotationally symmetric shared random codebook. Based on this, we propose a randomization mechanism where the codebook is a randomly rotated simplex -- satisfying the properties of the exact-optimal codebook. The proposed mechanism is based on a $k$-closest encoding which we prove to be exact-optimal for the randomly rotated simplex codebook.

Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation

Abstract

We study the mean estimation problem under communication and local differential privacy constraints. While previous work has proposed order -optimal algorithms for the same problem (i.e., asymptotically optimal as we spend more bits), exact optimality (in the non-asymptotic setting) still has not been achieved. In this work, we take a step towards characterizing the exact -optimal approach in the presence of shared randomness (a random variable shared between the server and the user) and identify several conditions for exact optimality. We prove that one of the conditions is to utilize a rotationally symmetric shared random codebook. Based on this, we propose a randomization mechanism where the codebook is a randomly rotated simplex - satisfying the properties of the exact -optimal codebook. The proposed mechanism is based on a k -closest encoding which we prove to be exact -optimal for the randomly rotated simplex codebook.

1 Introduction

The distributed mean estimation problem has attracted attention from the machine learning community as it is a canonical statistical formulation for many stochastic optimization problems such as distributed SGD [1, 3, 31, 32] and federated learning [33, 34]. As these tasks require data collection from the users, the mean estimation problem has often been studied under privacy constraints to protect users' sensitive information. More specifically, several works [2, 4, 7, 8, 9, 29, 35] have analyzed and improved the tradeoff between the utility and ε -local differential privacy ( ε -LDP) - the predominant paradigm in privacy mechanisms, which guarantees that an adversary cannot distinguish the user data from the outcome of the privacy mechanism [10, 24]. Among them, [4, 8, 9] developed algorithms that are asymptotically optimal, achieving an optimal mean squared error (MSE) proportional to Θ ( d n min( ε,ε 2 ) ) , where n is the number of users, and d is the input dimension. Later, [7] proved the corresponding lower bounds that hold for all privacy regimes. However, only PrivUnit [4] enjoys exact optimality among a large family of mechanisms, as proved by [2], while others provide only order optimality and their performance in practice depends heavily on the constant factors.

Another important consideration in the applications of mean estimation is the communication cost during user data collection. For instance, in federated learning, clients need to send overparameterized machine learning models at every round, which becomes a significant bottleneck due to limited resources and bandwidth available to the clients [22, 25, 28]. This has motivated extensive research on mean estimation [31, 34, 38] and distributed SGD [1, 3, 13, 26, 37] under communication constraints; and communication-efficient federated learning [17, 18, 21, 33].

Wei-Ning Chen

Albert No

In addition to the lines of work that studied these constraints (either privacy or communication) separately, recently, there has also been advancement in the joint problem of mean estimation under both privacy and communication constraints. [6] introduced an order -optimal mechanism SQKR requiring only O ( ε ) bits by using shared randomness - a random variable shared between the server and the user (see Section 2 for the formal definition). Later, [30] demonstrated better MSE with another order -optimal algorithm, MMRC , by simulating PrivUnit using an importance sampling technique [5, 15] - again with shared randomness. In the absence of shared randomness, the order -optimal mechanisms proposed by [6] do not achieve the best-known accuracy under this setting and are outperformed by the lossless compression approach in [12] that compresses PrivUnit using a pseudorandom generator (PRG). Due to not using shared randomness, these mechanisms require more bits than others [6, 30] that use shared randomness in the scenarios where it is actually available.

1.1 Contributions

To our knowledge, no existing mechanism achieves exact optimality under both privacy and communication constraints with shared randomness 1 . In this work, we address this gap by treating the joint problem as a lossy compression problem under ε -LDP constraints.

Our first contribution is to demonstrate that the exact optimal scheme with shared randomness can be represented as random coding with a codebook-generating distribution. Specifically, under b bits of communication constraint, the server and the user generate a codebook consisting of M = 2 b vectors (codewords) using shared randomness. The user then selects an index of a vector under a distribution that satisfies ε -LDP constraints, and the server claims the corresponding vector upon receiving the index. We term this approach as 'random coding with a codebook' and demonstrate that this (random codebook generation) is the optimal way to use shared randomness.

Next, we prove that the exact -optimal codebook-generating distribution must be rotationally symmetric. In other words, for any codebook-generating distribution, the distribution remains the same after random rotation. Based on this insight, we propose Random Rotating Simplex Coding ( RRSC ), where the codebook-generating distribution is a uniformly randomly rotating simplex. This choice of codebook distribution is reasonable as it maximizes the separation between codewords, which efficiently covers the sphere. The corresponding encoding scheme is the k -closest encoding, where the topk closest codewords to the input obtain high sampling probability, and the remaining ones are assigned low probabilities. We show that this scheme is exact -optimal for the random rotating simplex codebook.

The proposed codebook generation is valid only when M < d (or b ≤ log d where b is the communication budget) due to the simplex structure of the codebook. Note that as shown in [6], b ≤ log d bits of communication budget is sufficient to achieve orderwise optimal MSEs under an ε -LDP constraint for any ε ≤ O (log d ) , which is usually a common scenario in practical applications such as federated learning where d can range from millions to billions. In addition, we can also extend the scheme for cases when M ≥ d , by using a codebook consisting of (nearly) maximally separated M vectors on the sphere. As the number of bits b used for communication increases, we demonstrate that the proposed scheme approaches PrivUnit , which is the exact -optimal scheme without communication constraints.

Finally, through empirical comparisons, we demonstrate that RRSC outperforms the existing order -optimal methods such as SQKR [6] and MMRC [30]. We also observe that the performance of RRSC is remarkably close to that of PrivUnit when the number of bits is set to b = ϵ .

1.2 Related Work

The ℓ 2 mean estimation problem is a canonical statistical formulation for many distributed stochastic optimization methods, such as communication (memory)-efficient SGD [31, 34] or private SGD [24]. For instance, as shown in [14], as long as the final estimator of the mean is unbiased, the ℓ 2 estimation error (i.e., the variance) determines the convergence rate of the distributed SGD. As a result, there is a long thread of works that study the mean estimation problem under communication constraints [3, 8, 13, 31, 34, 39], privacy constraints [2, 16], or a joint of both [1, 6, 12, 30].

Among them, [6] shows that Θ( ε ) bits are sufficient to achieve the order -optimal MSE Θ ( d n min( ε,ε 2 ) ) and proposes SQKR , an order -optimal mean estimation scheme under both privacy and communication constraints. Notice that the MSE of SQKR is orderwise optimal up to a constant factor. Later on, in [12], it is shown that the pre-constant factor in SQKR is indeed suboptimal, resulting in an unignorable gap in the MSE compared to PrivUnit - an optimal ℓ 2 mean estimation scheme under ε -LDP. In the original PrivUnit , the output space is a d -dimensional sphere S d -1 and hence requires O ( d ) bits of communication, which is far from the optimal O ( ε ) communication bound. However, [12] shows that one can (almost) losslessly compress PrivUnit via a pseudo-random generator (PRG). Under the assumption of an existing exponentially strong PRG, [12] proves that one can compress the output of PrivUnit using polylog ( d ) bits with negligible performance loss. Similarly, [30] shows that with the help of shared randomness, PrivUnit can be (nearly) losslessly compressed to Θ( ε ) bits via a channel simulation technique, called MMRC . We remark that although the privacy-utility trade-offs in [12] and [30] are (nearly) exactly optimal, the communication efficiency is only order -optimal. That is, under an exact b -bit communication constraint, the MSEs of [12] (denoted as FT21 ) and MMRC [30] may be suboptimal. In this work, we aim to achieve the exact -optimal MSE under both communication and privacy constraints.

Furthermore, we show that SQKR , FT21 , and MMRC can be viewed as special cases in our framework i.e., (random) coding with their own codebook design. We elaborate on this in Section 5 and provide more details on prior work in Appendix I.

2 Problem Setting and Preliminaries

In this section, we formally define LDP (with shared randomness) and describe our problem setting.

Local Differential Privacy (LDP) A randomized algorithm Q : X → Y is ε -LDP if

LDP with Shared Randomness In this work, we assume that the encoder and the decoder have access to a shared source of randomness U ∈ U , where the random encoder (randomizer) Q privatizes x with additional randomness U . Then, the corresponding ε -LDP constraint is

for P U -almost all u .

Notation We let S d -1 = { u ∈ R d : || u || 2 = 1 } denote the unit sphere, e i ∈ R d the standard basis vectors for i = 1 , . . . , d , ⌊ k ⌋ the greatest integer less than or equal to k , and [ M ] = { 1 , . . . , M } .

Problem Setting We consider the ℓ 2 mean estimation problem with n users where each user i has a private unit vector v i ∈ S d -1 for 1 ≤ i ≤ n . The server wants to recover the mean 1 n ∑ n i =1 v i after each user sends a message using up to b -bits under an ε -LDP constraint. We allow shared randomness between each user and the server. More concretely, the i -th user and the server both have access to a random variable U i ∈ R t (which is independent of the private local vector v i ) for some t ≥ 1 and the i -th user has a random encoder (randomizer) f i : S d -1 × R t → [ M ] , where M = 2 b . We denote by Q f i ( m i | v i , u i ) the transition probability induced by the random encoder f i , i.e., the probability that f i outputs m i given the source v i and the shared randomness u i is

We require that the random encoder f i satisfies ε -LDP, i.e.,

for all v i , v ′ i ∈ S d -1 , m i ∈ [ M ] and P U i -almost all u i ∈ R t .

The server receives m i = f i ( v i , U i ) from all users and generates unbiased estimate of the mean A ( m 1 , . . . , m n , U 1 , . . . , U n ) that satisfies

Then, the goal is to minimize the worst-case error

where f denotes the collection of all encoders ( f 1 , . . . , f n ) . We note that the error is also a function of the distribution of shared randomness, which was not the case for PrivUnit [2, 4].

3 Main Results

3.1 Canonical Protocols

Similar to Asi et al. [2], we first define the canonical protocol when both communication and privacy constraints exist. The canonical protocols are where the server recovers each user's vector and estimates the mean by averaging them. In other words, the server has a decoder g i : [ M ] × R t → S d -1 for 1 ≤ i ≤ M which is dedicated to the i -th user's encoder f i , where the mean estimation is a simple additive aggregation, i.e.,

Our first result is that the exact -optimal mean estimation scheme should follow the above canonical protocol.

Lemma 3.1. For any n -user mean estimation protocol ( f, A , P U n ) that satisfies unbiasedness and ε -LDP, there exists an unbiased canonical protocol with decoders g = ( g 1 , . . . , g n ) that satisfies ε -LDP and achieves lower MSE, i.e.,

where Err 1 ( f, g, P U ) is the worst-case error for a single user with a decoder g .

The main proof techniques are similar to [2], where we define the marginalizing decoder:

̸

The expectation in (10) is with respect to the uniform distribution of v j 's. We defer the full proof to Appendix A.

Since the exact -optimal n -user mean estimation scheme is simply additively aggregating user-wise exact -optimal scheme, throughout the paper, we will focus on the single-user case and drop the index i when it is clear from the context. In this simpler formulation, we want the server to have an unbiased estimate ˆ v = g ( m,U ) , i.e.,

for all v ∈ S d -1 . We assume that the decoder g : [ M ] × R t → R d is deterministic, since the randomized decoder does not improve the performance. Then, the corresponding error becomes

Finally, we want to minimize the following worst-case error over all ( f, g ) pairs that satisfy the unbiasedness condition in (12)

3.2 Exact Optimality of the Codebook

Wepropose a special way of leveraging shared randomness, which we term as random codebook . First, we define a codebook U M = ( U 1 , . . . , U M ) ∈ ( R d ) M , consisting of M number of d -dimensional random vectors generated via shared randomness (i.e., both the server and the user know these random vectors). We then define the corresponding simple selecting decoder g + : [ M ] × ( R d ) M → R d , which simply picks the m -th vector of the codebook upon receiving the message m from the user:

Our first theorem shows that there exists a scheme with a random codebook and a simple selecting decoder that achieves the exact -optimal error. More precisely, instead of considering the general class of shared randomness (with general dimension t ) and the decoder, it is enough to consider the random codebook U M ∈ ( R d ) M as the shared randomness and the simple selector g + as the decoder.

Lemma 3.2. For any f, g, P U with U ∈ R t that are unbiased and that satisfy ε -LDP, there exists a shared randomness ˜ U M ∈ ( R d ) M and random encoder f 0 : S d -1 × ( R d ) M → [ M ] such that

for all v ∈ S d -1 , where f 0 , g + , P ˜ u M also satisfy unbiasedness and ε -LDP.

The main step of the proof is to set an implicit random codebook with codewords ˜ U m = g ( m,U ) for m = 1 , ..., 2 b and show that we can obtain an essentially equivalent scheme with a different form of shared randomness ˜ U M , which is an explicit random codebook. The detailed proof is given in Appendix B. Thus, without loss of generality, we can assume t = M × d and the random codebook U M is the new shared randomness, where the decoder is a simple selector. Since we fix the decoder, we drop g to simplify our notation. We say the random encoder f satisfies unbiasedness condition if

and the worst-case error is

Thus, the goal is now to find the exact -optimum codebook generating distribution P U M , and the random encoder f (or the probability assignment Q f ( ·| v, U ) ). We then argue that the exact -optimal codebook should be rotationally symmetric.

Definition 3.3. A random codebook U M ∈ ( R d ) M is rotationally symmetric if ( U 1 , . . . , U M ) ( d ) = ( A 0 U 1 , . . . , A 0 U M ) for any d × d orthonormal matrix A 0 .

The next lemma shows that the exact -optimal P U M is rotationally symmetric.

Lemma 3.4. Let P U M be a codebook generating distribution, and suppose random encoder f satisfies unbiasedness and ε -LDP. Then, there exists a random encoder f 1 and rotationally symmetric random codebook ¯ U M such that

which also satisfies unbiasedness and ε -LDP.

This is mainly because the goal is to minimize the worst-case error, and the codebook-generating distribution should be symmetric in all directions. The proof is provided in Appendix C. The next lemma shows that the exact -optimal scheme has constant error for all v ∈ S d -1 .

Lemma 3.5. For any rotationally symmetric codebook generating distribution P U M and an unbiased randomized encoder f that satisfies ε -LDP, there exists a random encoder f 2 such that

for all v, v ′ ∈ S d -1 .

The formal proof is given in Appendix D. Since the codebook is symmetric (Lemma 3.4), the exact -optimal encoding strategy remains the same for any input v . Thus, without loss of generality, we can assume that the input is a standard unit vector v = e 1 = (1 , 0 , . . . , 0) .

3.3 Rotationally Symmetric Simplex Codebook

Now, we focus on a particular rotationally symmetric codebook. Notice that the codebook U M has a similar role to the codebook in lossy compression, in the sense that we prefer the codeword U m close to the input vector v . Thus, it is natural to consider the maximally separated codebook so that the M vectors U 1 , . . . , U M cover the source space effectively. For M < d , the maximally separated M vectors on the unit sphere S d -1 is a simplex. More precisely, let s 1 , . . . , s M ∈ R d form a simplex:

̸

Then, we can define the rotationally symmetric simplex codebook U M

where A is uniformly drawn orthogonal matrix and r > 0 is a normalizing constant. We then need to find the corresponding encoder f that minimizes the error. Recall that the error is

and it is natural to assign high probabilities to the message m with low distortion ∥ U m -v ∥ 2 as long as ε -LDP constraint allows. More precisely, we call the following probability assignment ' k -closest' encoding:

where we allow non-integer k . The choice of r = r k is described in Section 3.4. We call this approach Randomly Rotating Simplex Coding ( RRSC ) and provide the pseudocode in Algorithm 1. We note that the codewords U m 's with smallest ∥ v -U m ∥ 2 and codewords U m 's with largest ⟨ v, U m ⟩ coincide for a codebook with fixed-norm codewords U m 's, which is the case for the rotationally symmetric simplex codebook. Our main theorem is that the k -closest encoding is exact -optimum if the codebook generating distribution is rotationally symmetric simplex.

Theorem 3.6. For a rotationally symmetric simplex codebook, there exists a k such that the ' k -closest' encoding is the exact-optimum unbiased scheme that satisfies ε -LDP constraint.

The main step of the proof is to show that all the probabilities should be either the maximum or the minimum in order to minimize the error, and the proof is given in Appendix E.

3.4 k -closest Encoding for General Rotationally Symmetric Codebook

In this section, we demonstrate that the k -closest encoding consistently yields an unbiased scheme for any rotationally symmetric codebook. To be more specific, for any given spherically symmetric

Algorithm 1 Randomly Rotating Simplex Coding RRSC ( k ).

Inputs: v ∈ S d - 1 , k , r k , codebook size M = 2 b .

Codebook Generation:

Generate the simplex s , . . . , s ∈ R d

Sample orthogonal matrix A ∈ R uniformly using the shared random

1 M in (23). d × d SEED . M

Encoding:

for m ∈ [ M ] do

end if

end for

Sample codeword index m ∗ ← Q f ( ·| v, U M ) .

Output: m ∗ , encoded in b = log M bits.

codebook U M , there exists a scalar r k that ensures that the k -closest encoding with r k U M = ( r k U 1 , . . . , r k U M ) is unbiased. Let T k ( v, U M ) = { m : U m is one of the k -closest } , and without loss of generality, let us assume v = e 1 . Then,

where E [ ∑ U m ] = 0 due to rotationally symmetric codebook and we assume an integer k for the sake of simplicity. Since the codebook is rotationally symmetric and we pick k -closest vectors toward v = e 1 , each codeword U m ∈ T k ( e 1 , U M ) is symmetric in all directions other than v = e 1 . Thus, in expectation, the decoded vector is aligned with e 1 , and there exists r k such that

For a rotationally symmetric simplex codebook, where U m = As m for a uniform random orthogonal matrix A , we have an (almost) analytic formula.

Lemma 3.7. Normalization constant r k for RRSC ( k ) is

where C k 2 is an expected sum of topk coordinates of uniform random vector a ∈ S d -1 .

The key idea in the proof is to show that encoding e 1 with As M is equivalent to encoding uniform random vector a ∈ S d -1 with s M . The formal proof is provided in Appendix F.

The following lemma controls the asymptotic behavior of C k :

Lemma 3.8. Let C k be defined as in Lemma 3.7. Then, it holds that

In addition, there exist absolute constants C 1 , C 2 > 0 such that as long as ⌊ M/k ⌋ > C 1 and k > C 2 ,

As a corollary, Lemma 3.8 implies the order-wise optimality of RRSC :

By picking k = max(1 , Me -ε ) , the above error is O ( d min( ε 2 ,ε,b ) ) . We provide the proof of Lemma 3.8 in Appendix G.

3.5 Convergence to PrivUnit

As the communication constraint b increases, the exact -optimal scheme with communication constraint should coincide with the exact -optimal scheme without communication constraint, which is PrivUnit . Note that the rotationally symmetric simplex can be defined only when M = 2 b < d , due to its simplex structure. However, we have a natural extension where the codebook is a collection of M (nearly) maximally separated vectors on the sphere of radius r , where we can assume that M codewords are uniformly distributed on the sphere of radius r k if M is large enough. Consider the case where q = k M is fixed and M = 2 b is large. Since the k -closest encoding yields an unbiased scheme with error Err( f, P U M ) = r 2 k -1 , where r k is normalizing constant, for uniformly distributed M codewords on the sphere, the constant r k should satisfy

where U m, 1 is the first coordinate of uniformly drawn U m from the unit sphere S d -1 . Then, as M increases, U m, 1 being one of the topk becomes equivalent to U m, 1 > γ , where γ is the threshold such that Pr[ U m, 1 > γ ] = q . Hence, assigning higher probabilities to the topk closest codewords becomes equivalent to assigning high probabilities to the codewords with ⟨ U m , e 1 ⟩ > γ where v = e 1 . This is essentially how PrivUnit operates.

3.6 Complexity of RRSC

Each user has d × d orthonormal matrix A and needs to find k smallest ⟨ v, As m ⟩ for 1 ≤ m ≤ M . Since ⟨ v, As m ⟩ = ⟨ A ⊺ v, s m ⟩ , it requires O ( d 2 ) to compute A ⊺ v and additional O ( Md ) to compute all inner products for 1 ≤ m ≤ M . However, if M ≪ d , we have a simpler equivalent scheme using

where a ⊺ m is the m -th row of the matrix A . Then, it only requires storing the first M rows of the matrix and O ( Md ) to obtain all inner products in (34) by avoiding O ( d 2 ) to construct A ⊺ v .

On the other hand, the server computes As m upon receiving a message m . The corresponding time complexity is O ( Md ) (per user) since s m has M non-zero values. We note that both MMRC [30] and FT21 [12] require the same encoding complexity O ( Md ) as RRSC , where they choose M = O (exp( ε )) .

4 Experiments

We empirically demonstrate the communication-privacy-utility tradeoffs of RRSC and compare it with order -optimal schemes under privacy and communication constraints, namely SQKR [6] and MMRC [30]. We also show that RRSC performs comparably with PrivUnit [4], which offers the exact -optimal privacy-utility tradeoffs without communication constraints [2]. In our simulations, we use

1

the 'optimized' PrivUnit mechanism, called PrivUnitG , introduced in [2], which performs better than PrivUnit in practice since it provides an easy-to-analyze approximation of PrivUnit but with analytically better-optimized hyperparameters. Similar to [6, 30], we generate data independently but non-identically to capture the distribution-free setting with µ = 0 . More precisely, for the first half of the users, we set

v

, . . . , v n/

2

∼

N

(1

,

1)

⊗

d

̸

; and for the second half of the users, we set v n/ 2+1 , . . . , v n i.i.d ∼ N (10 , 1) ⊗ d . We further normalize each v i to ensure that they lie on S d -1 . We report the average ℓ 2 error over 10 rounds together with the confidence intervals. To find the optimal values for k and r k , we compute the optimal r k using the formula in (33) for k = 1 , . . . , M and pick the k that gives the smallest r k (which corresponds to the bias). To estimate the expectation C k in (33), we run a Monte Carlo simulation with 1 M trials. We report the k we use for each experiment in the captions. Additional experimental results are provided in Appendix H.

In Figure 1-(left, middle), we report ℓ 2 error for ε = 1 , . . . , 8 , where for each method (except PrivUnitG ), the number of bits is equal to b = ε . In Figure 1-(right), we report ℓ 2 error by fixing ϵ = 6 and sweeping the bitrate from b = 1 to b = 8 for RRSC and MMRC . For SQKR , we only sweep for b ≤ ε as it leads to poor performance for b > ε . In each figure, RRSC performs comparably to PrivUnitG even for small b and outperforms both SKQR and MMRC by large margins.

5

0.200

0.200

RRSC(ours)

4

0.175

0.175

SQKR

0.150

0.150

MMRC

error

0.125

0.125

PrivUnitG

0.100

0.100

2

0.075

0.075

1

0.050

0.050

0.025

0.025

0

0.000

0.000

1

2

3

4

5

6

7

8

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

# bits (b)

The codebase for this work is open-sourced at https://github.com/BerivanIsik/rrsc .

5 Discussion & Conclusion

We proved that using a rotationally symmetric codebook is a necessary condition for the exact optimality of mean estimation mechanisms with privacy and communication constraints. We then proposed Random Rotating Simplex Coding ( RRSC ) based on a k -closest encoding mechanism and proved that RRSC is exact -optimal for the random rotating simplex codebook. We now discuss some important features of RRSC and provide conjectures for future work.

Unified Framework It turns out that SQKR [6], FT21 [12] and MMRC [30] can be viewed as special cases in our framework. Specifically, SQKR [6] uses Kashin's representation of v = ∑ N j =1 a j u j , where { a j } N j =1 ∈ [ -c/ √ d, c/ √ d ] for some (1 + µ ) d with µ > 0 and c > 0 . Then the SQKR encoder quantizes each a j into a 1-bit message q i , and draws k samples with the help of shared randomness. This can be viewed as random coding with a codebook-generating distribution. More concretely, the corresponding codebook U M consists of k non-zero values of ± c/ √ d where the randomness is from selecting k indices using shared randomness. On the other hand, since MMRC [30] is simulating the channel corresponding to a privacy mechanism, it can be viewed as pre-generating random codewords U M according to the reference distribution, where the importance sampling is also a way of assigning probabilities to each codeword. As elaborated in Section 3.5, it is observed that with an increase in the communication constraint b , the suggested k -closest encoding gradually transforms into a threshold-based encoding, analogous to that of MMRC . The codebook associated with FT21 [12]

i.i.d

depends on the PRG it uses. Let PRG : { 0 , 1 } b →{ 0 , 1 } Θ( d ) be a PRG that takes a b -bit seed and maps it into Θ( d ) bits, where b ≪ d , and let g : { 0 , 1 } Θ( d ) → R d . For example, if we represent each coordinate of x ∈ R d as a 32-bit float, then g ( · ) maps the float representation of x (a 32 -bit string) to x . With a PRG, FT21 mimics PrivUnit by first generating a b -bit seed m , computing g ( PRG ( m )) , and then performing rejection sampling on the seed space. The above procedure can be treated as a special case in our framework, where the deterministic codebook consists of 2 b points on R d : C FT21 := { g ( PRG ( m )) : m ∈ { 0 , 1 } b } . The probabilities assigned to each codeword according to the rejection sampling are equivalent to a threshold-based assignment.

Shared randomness When M ≤ d +1 , additional randomization is required during codebook generation to achieve an unbiased scheme, as discussed in [13]. Furthermore, both the encoder and decoder must possess this randomization information. In the proposed RRSC scheme, this randomization is achieved through the random rotation of the simplex code using shared randomness. However, it is possible to circumvent the need for shared randomness by having the server generate random rotation matrices using its private coin and communicate them to the users. This approach replaces shared randomness with downlink communication, which is typically more affordable than uplink communication. It should be noted that directly transmitting the rotation matrices would require O ( d 2 ) bits. Nonetheless, the server can generate them using a predetermined pseudo-random generator (PRG) and transmit only the seeds of it to the users. Drawing from a similar argument as in [12], assuming the existence of exponentially strong PRGs, seeds with polylog ( d ) bits are sufficient.

Future Work We showed the exact -optimality of k -closest encoding for the rotating simplex codebook. In general, it also achieves unbiasedness and the following error formulation E P U M [ ∑ M m =1 Q f ( m | v, U M ) ∥ v -U m ∥ 2 ] implies the exact -optimality of k -closest encoding for any rotationally symmetric codebook, which leads us to the following conjecture.

Conjecture 5.1. The proposed k -closest encoding is exact-optimal for any rotationally symmetric codebook.

It also remains unclear whether k can depend on the realization of the codebook U M in general, which we leave to future work. We also proved that the exact -optimal codebook must be rotationally symmetric. We conjecture that the maximally separated codebook (simplex codebook) is exact -optimal as it provides the most effective coverage of the space S d -1 . This, too, is left as a topic for future work.

Conjecture 5.2. The rotationally symmetric simplex codebook is the exact-optimal codebook.

Limitations and Broader Impact While we take an important step towards exact optimality by proving several necessary conditions and by providing a mechanism that is exact -optimal for a family of codebooks, we still have the above conjectures left to be proven in future work.

References

N. Agarwal, A. T. Suresh, F. X. X. Yu, S. Kumar, and B. McMahan. cpsgd: Communicationefficient and differentially-private distributed sgd. Advances in Neural Information Processing Systems , 31, 2018.

H. Asi, V. Feldman, and K. Talwar. Optimal algorithms for mean estimation under local differential privacy. In International Conference on Machine Learning , pages 1046-1056. PMLR, 2022.

L. P. Barnes, H. A. Inan, B. Isik, and A. Özgür. rtop-k: A statistical estimation approach to distributed sgd. IEEE Journal on Selected Areas in Information Theory , 1(3):897-907, 2020.

A. Bhowmick, J. Duchi, J. Freudiger, G. Kapoor, and R. Rogers. Protection against reconstruction and its applications in private federated learning. arXiv preprint arXiv:1812.00984 , 2018.

S. Chatterjee and P. Diaconis. The sample size required in importance sampling. The Annals of Applied Probability , 28(2):1099-1135, 2018.

W.-N. Chen, P. Kairouz, and A. Ozgur. Breaking the communication-privacy-accuracy trilemma. Advances in Neural Information Processing Systems , 33:3312-3324, 2020.

J. Duchi and R. Rogers. Lower bounds for locally private estimation via communication complexity. In Conference on Learning Theory , pages 1161-1191. PMLR, 2019.

J. C. Duchi, M. I. Jordan, and M. J. Wainwright. Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science , pages 429-438. IEEE, 2013.

J. C. Duchi, M. I. Jordan, and M. J. Wainwright. Minimax optimal procedures for locally private estimation. Journal of the American Statistical Association , 113(521):182-201, 2018.

C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

V. Feldman, J. Nelson, H. Nguyen, and K. Talwar. Private frequency estimation via projective geometry. In International Conference on Machine Learning , pages 6418-6433. PMLR, 2022.

V. Feldman and K. Talwar. Lossless compression of efficient private local randomizers. In International Conference on Machine Learning , pages 3208-3219. PMLR, 2021.

V. Gandikota, D. Kane, R. K. Maity, and A. Mazumdar. vqsgd: Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics , pages 2197-2205. PMLR, 2021.

S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

M. Havasi, R. Peharz, and J. M. Hernández-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations , 2019.

Z. Huang, Y. Liang, and K. Yi. Instance-optimal mean estimation under differential privacy. Advances in Neural Information Processing Systems , 34:25993-26004, 2021.

B. Isik, F. Pase, D. Gunduz, S. Koyejo, T. Weissman, and M. Zorzi. Communication-efficient federated learning through importance sampling. arXiv preprint arXiv:2306.12625 , 2023.

B. Isik, F. Pase, D. Gunduz, T. Weissman, and Z. Michele. Sparse random networks for communication-efficient federated learning. In The Eleventh International Conference on Learning Representations , 2023.

B. Isik and T. Weissman. Learning under storage and privacy constraints. In 2022 IEEE International Symposium on Information Theory (ISIT) , pages 1844-1849. IEEE, 2022.

B. Isik and T. Weissman. Lossy compression of noisy data for private and data-efficient learning. IEEE Journal on Selected Areas in Information Theory , 2023.

B. Isik, T. Weissman, and A. No. An information-theoretic justification for model pruning. In International Conference on Artificial Intelligence and Statistics , pages 3821-3846. PMLR, 2022.

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

B. Kashin. Section of some finite-dimensional sets and classes of smooth functions (in russian) izv. Acad. Nauk. SSSR , 41:334-351, 1977.

S. P. Kasiviswanathan, H. K. Lee, K. Nissim, S. Raskhodnikova, and A. Smith. What can we learn privately? SIAM Journal on Computing , 40(3):793-826, 2011.

J. Koneˇ cn` y, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 , 2016.

Y. Lin, S. Han, H. Mao, Y. Wang, and B. Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations , 2018.

Y. Lyubarskii and R. Vershynin. Uncertainty principles and vector quantization. IEEE Transactions on Information Theory , 56(7):3491-3501, 2010.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

T. T. Nguyên, X. Xiao, Y. Yang, S. C. Hui, H. Shin, and J. Shin. Collecting and analyzing data from smart device users with local differential privacy. arXiv preprint arXiv:1606.05053 , 2016.

A. Shah, W.-N. Chen, J. Balle, P. Kairouz, and L. Theis. Optimal compression of locally differentially private mechanisms. In International Conference on Artificial Intelligence and Statistics , pages 7680-7723. PMLR, 2022.

A. T. Suresh, X. Y. Felix, S. Kumar, and H. B. McMahan. Distributed mean estimation with limited communication. In International conference on machine learning , pages 3329-3337. PMLR, 2017.

A. T. Suresh, Z. Sun, J. Ro, and F. Yu. Correlated quantization for distributed mean estimation and optimization. In International Conference on Machine Learning , pages 20856-20876. PMLR, 2022.

S. Vargaftik, R. B. Basat, A. Portnoy, G. Mendelson, Y. B. Itzhak, and M. Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning. In International Conference on Machine Learning , pages 21984-22014. PMLR, 2022.

S. Vargaftik, R. Ben-Basat, A. Portnoy, G. Mendelson, Y. Ben-Itzhak, and M. Mitzenmacher. Drive: One-bit distributed mean estimation. Advances in Neural Information Processing Systems , 34:362-377, 2021.

T. Wang, J. Zhao, X. Yang, and X. Ren. Locally differentially private data collection and analysis. arXiv preprint arXiv:1906.01777 , 2019.

S. L. Warner. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association , 60(309):63-69, 1965.

W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. Advances in neural information processing systems , 30, 2017.

Y. Zhang, J. Duchi, M. I. Jordan, and M. J. Wainwright. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. Advances in Neural Information Processing Systems , 26, 2013.

Y. Zhang, M. J. Wainwright, and J. C. Duchi. Communication-efficient algorithms for statistical optimization. Advances in neural information processing systems , 25, 2012.

A Proof of Lemma 3.1

Proof. For n -user mean estimation protocol ( f, A , P U M ) , following the notation and steps from [2, Proof of Lemma 3.1], we define the marginalized output

̸

Then, we define the user-specific decoder by averaging g i ( m i , U i ; v n ) with respect to i.i.d. uniform P unif :

where v n \ i indicates the v n vector except v i . Due to the symmetry of P unif, it is clear that g i is unbiased. We also define

Consider an average error where v 1 , . . . , v n are drawn i.i.d. uniformly on the sphere S d -1 .

Then, we need to show the same inequality for the worst-case error.

where the last equality (46) is from Lemma 3.2, Lemma 3.4, and Lemma 3.5. Thus, the user-specific decoder achieves lower MSE:

Since we keep random encoder f i the same, the canonical protocol with g i also satisfies ε -LDP constraint. This concludes the proof.

B Proof of Lemma 3.2

Proof. Let ˜ U m = g ( m,U ) for all 1 ≤ m ≤ M . Without loss of generality g ( · , U ) is one-to-one, i.e., { u : ˜ u m = g ( m,u ) for all m } has at most one element (with probability 1), and u = g -1 (˜ u M ) is well-defined. Then, we define a randomizer f 0 ( v, ˜ U M ) that satisfies

It is clear that f 0 satisfies ε -LDP constraint. Then,

We also need to show that the composition of the new randomizer f 0 and selector g + is unbiased.

Finally, Q f 0 ( m | v, ˜ u M ) is a valid transition probability, since

for all ˜ u M . This concludes the proof.

C Proof of Lemma 3.4

Proof. Let A be a uniformly random orthogonal matrix and ¯ U M = A ⊺ U M . We further let f 1 be a randomized encoder that satisfies

Then, Q f 1 is a valid probability since

Also, we have

Finally, we need to check unbiasedness.

The key step is that the original encoder f is unbiased, which implies

for all A .

Now, we are ready to prove the main inequality.

for all v . This concludes the proof.

D Proof of Lemma 3.5

Proof. For v, v ′ ∈ S d -1 , let A 0 be an orthonormal matrix such that v ′ = A 0 v . Let f 2 be a randomized encoder such that

for uniform random orthonormal matrix. Then,

Similar to the previous proofs, Q f 2 is a well-defined probability distribution, and f 2 is unbiased as well as ε -LDP. Since P U M is rotationally symmetric and f 2 is also randomized via the uniform random orthogonal matrix, we have

Compared to a given randomizer f , we have

for all v ∈ S d -1 . This concludes the proof.

E Proof of Theorem 3.6

Proof. The rotationally symmetric simplex codebook with normalization constant r is ( rAs 1 , . . . , rAs M ) . Let f be the unbiased encoder satisfying ε -LDP. Let Q max = max Q f ( m | v, rAs M ) and Q min = min Q f ( m | v, rAs M ) , our objective is to demonstrate that Q max is less than or equal to e ε Q min . We will employ a proof by contradiction to establish this. Suppose Q f ( m 1 | v 1 , rA 1 s M ) > e ε Q f ( m 2 | v 2 , rA 2 s M ) for some m 1 , v 1 , A 1 , m 2 , v 2 , and A 2 . Let ˜ A be the row switching matrix where r ˜ AA 1 s m 1 = rA 1 s m 2 and r ˜ AA 1 s m 2 = rA 1 s m 1 , then we have

We further let A ′ be an orthogonal matrix such that A ′ ˜ AA 1 = A 2 , then

If we let v ′ 1 = A ′ ˜ Av 1 , then

which contradicts the ε -LDP constraint.

For an unbiased encoder, the error is

Thus, we need to find r that minimizes the error.

On the other hand, the encoder needs to satisfy unbiasedness. Without loss of generality, we assume v = e 1 , then we need

where the expectation is with respect to the random orthonormal matrix A . If we focus on the first index of the vector, then

where a ⊺ = ( a 1 , . . . , a d ) is the first row of A and has uniform distribution on the sphere S d -1 . Thus, it is clear that assigning higher probability (close to Q max ) to the larger a ⊺ s m .

If Q max is strictly smaller than e ε Q min , then we can always scale up the larger probabilities and scale down the lower probabilities to keep the probability sum to one (while decreasing the error). Hence, we can assume that Q min = q 0 and Q max = e ε q 0 for some 1 > q 0 > 0 .

Now, let k be such that

where q i is an intermediate value such that q i ∈ [ q 0 , e ε q 0 ] . Then, the optimal strategy is clear: (i) assign e ε q 0 to ⌊ k ⌋ -th closest codewords s m 's, (ii) assign q i to the ( ⌊ k ⌋ +1) -th closest codeword, and (iii) assign q 0 to the remaining codewords. This implies that the k -closest coding is optimal.

F Proof of Lemma 3.7

Proof. Following (28) with U m = As m and v = e 1 , we have

By focusing on the first coordinate of the above equation and observing that ⟨ e 1 , As M ⟩ = ⟨ a, s m ⟩ where a is the first row of the rotation matrix A , we must have

Note that since A is a random orthogonal matrix drawn from the Haar measure on SO ( d ) , a is distributed uniformly over the unit sphere S d -1 .

Next, observe that by definition,

where 1 M = (1 , 1 , ..., 1 ︸ ︷︷ ︸ M entries , 0 , ..., 0) ∈ { 0 , 1 } d (that is, ( 1 M ) m = ✶ { m ≤ M } ). Therefore,

and hence plugging in (96) yields

where (1) a ( i | M ) denotes the i -th largest entry of the first M coordinates of a and (2) the last equality holds since a is uniformly distributed over S d -1 .

G Proof of Lemma 3.8

Proof. First of all, observe that

where (a) holds due to the spherical symmetry of a . Next, we bound (i) and (ii) separately.

Claim G.1 (Bounding (i)) . For any d ≥ M > 2 , it holds that

Proof of Claim G.1. Observe that when a is distributed uniformly over S d -1 , it holds that

where A d = B denotes A and B have the same distribution, and Z 1 , ..., Z d i.i.d. ∼ N (0 , 1) . As a result, we must have

By Jensen's inequality, it holds that

where (a) holds since x ↦→ √ 1 / (1 + x ) is a convex mapping for x > 0 , and (b) holds due to the fact that ∑ i Z 2 i follows from a χ 2 distribution and that the ratio of two independent χ 2 random variables follows an F -distribution.

On the other hand, it also holds that

where (a) holds since √ · is concave, (b) holds since x ↦→ 1 1+ x is convex, and (c) again is due to the fact that the ratio of two independent χ 2 random variables follows an F -distribution.

Claim G.2 (Bounding (ii)) . As long as

k ≥ 400 · log 10 ,

it holds that

Proof of Claim G.2. We start by re-writing a ′ :

This yields that

and hence

Upper bound. To upper bound the above, observe that

Let E 1 := { ( Z 1 , ..., Z M ) | ∑ M i =1 Z 2 i ≤ M (1 -γ ) } where γ > 0 will be optimized later. Then it holds that

On the other hand, the Borell-TIS inequality ensures

where Z i ∼ N (0 , σ 2 ) (in our case, σ = 1 ). Since E [ Z (1) ] ≤ √ 2 log M , it holds that

Therefore, define E 2 := { Z (1) ≥ √ 2 log M + ξ } and we obtain

where the last inequality holds by picking γ = 0 . 1 and ξ = √ log M .

Lower bound. The analysis of the lower bound is more sophisticated. To begin with, let

denote the good event such that the denominator of our target is well-controlled, where γ > 0 again will be optimized later. By the concentration of χ 2 random variables, it holds that

Next, to lower bound ∑ k i =1 Z ( i ) , we partition ( Z 1 , Z 2 , ..., Z M ) into k blocks B 1 , B 2 , ..., B k where each block contains at least N = ⌊ M/k ⌋ samples: B j := [( j -1) · N +1 : j · N ] for j ∈ [ k -1] and B k = [ M ] \ ( ⋃ k -1 j =1 B j ) . Define ˜ Z ( j ) (1) be the maximum samples in the j -th block: ˜ Z ( j ) (1) := max i ∈ B j Z i . Then, it is obvious that

To this end, we define E 1 to be the good event that 90% of ˜ Z ( j ) (1) 's are large enough (i.e., concentrated to the expectation):

Note that by the Borell-TIS inequality, for any j ∈ [ k ] ,

so setting ξ = log 100 implies Pr { ˜ Z ( j ) (1) ≥ √ log N √ π log 2 -ξ } ≥ 0 . 98 . Since blocks are independent with each other, applying Hoeffding's bound yields

when k ≥ 400 · log 10 ≥ log 10 / 0 . 08 2 .

Next, we define a 'not-too-bad' event where ∑ k j =1 ˜ Z ( j ) (1) is not catastrophically small:

for some ξ > 0 to be optimized later. Observe that E 2 holds with high probability:

where (a) holds since the each of the topk values must be greater than k times the average, and (b) holds due to the Hoeffding's bound on the sum of i.i.d. Gaussian variables.

Lastly, a trivial bound implies that

Now, we are ready to bound E Z 1 ,...,Z M i.i.d. ∼N (0 , 1) [ 1 √ ∑ i M =1 Z 2 i ∑ k i =1 Z ( i ) ] . We begin by decomposing it into three parts:

We bound these three terms separately. To bound the first one, observe that condition on E 1 ∩ E 2 , ∑ k i =1 Z ( i ) ≥ ˜ Z ( j ) (1) ≥ 0 . 9 k √ log N π log 2 -k √ M γ . As a result,

To bound the second term, observe that under E 2 ,

so we have

For the third term, it holds that

Combining (102), (103), and (104) together, we arrive at

Finally, setting γ = O ( 1 √ M ) and ξ = O (1) yields the desired lower bound

H Additional Experimental Results

In Figure 2, we provide additional empirical results by sweeping the number of users n from 2 , 000 to 10 , 000 on the left and sweeping the dimension d from 200 to 1 , 000 on the right.

I Additional Details on Prior LDP Schemes

For completeness, we provide additional details on prior LDP mean estimation schemes in this section, including PrivUnit [4], SQKR [6], FT21 [12], and MMRC [30]. We skip prior work analyzing compression-privacy-utility tradeoffs that do not specifically focus on the distributed mean estimation problem [19, 20] or others that study frequency estimation [6, 11, 30].

0.200

0.200

0.175

0.175

0.150

rror

0.150

0.125

RRSC (ours)

0.125

SQKR

0.100

0.100

MMRC

0.075

0.075

PrivUnitG

0.050

0.050

0.025

0.025

0.000

0.000

2000

3000

4000

5000

6000

7000

8000

900010000

200

300

400

500

600

700

800

900

1000

n

d

I.1 PrivUnit [4]

[2] considered the mean estimation problem under DP constraint (without communication constraint) when X = S d -1 = { v ∈ R d : ∥ v ∥ 1 = 1 } . Since there is no communication constraint, they assumed canonical protocol where the random encoder is f : S d -1 → R d and the decoder is a simple additive aggregator

The authors showed that PrivUnit is an exact optimal among the family of unbiased locally private procedures.

Recall that given an input vector v ∈ S d -1 , the local randomized PrivUnit ( p, q ) has the following distribution up to normalization:

where Z has a uniform distribution on S d -1 . Let S γ be the surface area of hypersphere cap { z ∈ S d -1 |⟨ z, v ⟩ ≥ γ } , with S -1 representing the surface area of the d dimensional hypersphere. We denoted q = Pr[ Z 1 ≤ γ ] = ( S -1 -S γ ) /S -1 (convention from [4, 2]). The normalization factor is required to obtain unbiasedness.

[2] also introduced PrivUnitG , which is a Gaussian approximation of PrivUnit . In this approach, Z is sampled from an i.i.d. N (0 , 1 /d ) distribution. This simplifies the process of determining more accurate parameters p, q , and γ . Consequently, in practical applications, PrivUnitG surpasses PrivUnit in performance owing to superior parameter optimization.

I.2 SQKR [6]

Next, we outline the encoder and decoder of SQKR in this section. The encoding function mainly consists of three steps: (1) computing Kashin's representation, (2) quantization, and (3) sampling and privatization.

Compute Kashin's representation A tight frame is a set of vectors { u j } N j =1 ∈ R d that satisfy Parseval's identity, i.e. ∥ v ∥ 2 2 = ∑ N j =1 ⟨ u j , v ⟩ 2 for all v ∈ R d . We say that the expansion v = ∑ N j =1 a j u j is a Kashin's representation of x at level K if max j | a j | ≤ K √ N ∥ v ∥ 2 [23]. [27] shows that if N > (1 + µ ) d for some µ > 0 , then there exists a tight frame { u j } N j =1 such that for any x ∈ R d , one can find a Kashin's representation at level K = Θ(1) . This implies that we can represent the local vector v with coefficients { a j } N j =1 ∈ [ -c/ √ d, c/ √ d ] N for some constants c and N = Θ( d ) .

Quantization In the quantization step, each client quantizes each a j into a 1 -bit message q j ∈ { -c/ √ d, c/ √ d } with E [ q j ] = a j . This yields an unbiased estimator of { a j } N j =1 , which can be described in N = Θ( d ) bits. Moreover, due to the small range of each a j , the variance of q j is bounded by O (1 /d ) .

Sampling and privatization To further reduce { q j } to k = min( ⌈ ε ⌉ , b ) bits, client i draws k independent samples from { q j } N j =1 with the help of shared randomness, and privatizes its k bits message via 2 k -RR mechanism[36], yielding the final privatized report of k bits, which it sends to the server.

Upon receiving the report from client i , the server can construct unbiased estimators ˆ a j for each { a j } N j =1 , and hence reconstruct ˆ v = ∑ N j =1 ˆ a j u j , which yields an unbiased estimator of v . In [6], it is shown that the variance of ˆ v can be controlled by O ( d/ min ( ε 2 , ε, b )) .

I.3 FT21 [12] and MMRC [30]

Both FT21 and MMRC aim to simulate a given ε -LDP scheme. More concretely, consider an ε -LDP mechanism q ( ·| v ) that we wish to compress, which in our case, PrivUnit . A number of candidates u 1 , · · · , u N are drawn from a fixed reference distribution p ( u ) (known to both the client and the server), which in our case, uniform distribution on the sphere S d -1 . Under FT21 [12], these candidates are generated from an (exponentially strong) PRG, with seed length ℓ = polylog ( d ) . The client then performs rejection sampling and sends the seed of the sampled candidates to the server. See Algorithm 2 for an illustration.

On the other hand, under MRC [30] the LDP mechanism is simulated via a minimal random coding technique [15]. Specifically, the candidates are generated via shared randomness, and the client performs an importance sampling and sends the index of the sampled one to the server, as illustrated in Algorithm 3. It can be shown that when the target mechanism is ε -LDP, the communication costs of both strategies are Θ( ε ) bits. It is also worth noting that both strategies will incur some bias (though the bias can be made exponentially small as one increases the communication cost), and [30] provides a way to correct the bias when the target mechanism is PrivUnit (or general cap-based mechanisms).

Algorithm 3 Simulating LDP mechanisms via importance sampling [30]

Inputs:

ε -LDP mechanism q ( ·| v ) , ref. distribution p ( · ) , # of candidates M

Draw samples u 1 , · · · , u M from p ( u ) using the shared source of randomness.

w

(

k

)

q

(

u

k

|

v

)

/p

(

u

k

)

.

end for

Output:

k ∗

Peer reviews:
comments: SUMMARY:
The authors consider the problem of high dimensional mean estimation with communication and privacy constraints. For federated learning or distributed SGD, model updates must be communicated to central server, but as models become much larger this can be a bottleneck within the computation. As a result, previous works has considered the setting in which there is a restriction on the number of bits that can be communicated, which the authors also follow. Furthermore, the authors consider the differentially private setting adding another constraint. This setting was also considered in previous work, some of which achieved optimality up to constant factors. The authors improve upon that work and achieve exact optimality and this theoretical improvement is backed by their empirical experiments.

STRENGTHS:
Improves upon previous work for a reasonably well-studied problem and achieve optimal tradeoff between communication-privacy-utility and further show how the previous work are special cases in their method.

WEAKNESSES:
Minor gripe that some of the notation could have been expanded upon more clearly (for example: P_U-almost) but can understand that page limits can add difficulty.

QUESTIONS:
None

LIMITATIONS:
The authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the distributed mean estimation (DME) problem with communication & privacy constraints.  The goal is to construct an unbiased estimate of a unit vector $v$ using $b$-bits that minimizes the mean squared error and provides $\\epsilon$-LDP. It is well known that any scheme achieving the above communication will have to quantize the unit sphere using $M=2^b$ points. Further, to achieve $\\epsilon$-LDP, a particular quantization point is chosen and returned according to an appropriate distribution. \n\nThe main contribution of this work is characterizing DME schemes that achieve optimal error in the presence of a communication budget and privacy constraints. The authors show that a random set of points generated using a rotationally symmetric distribution will achieve the optimal error. The intuition is that such a point set will be maximally separated and will most efficiently cover the sphere. $\\epsilon$-LDP is achieved using a randomized response mechanism.\n\nSTRENGTHS:\n- Presents a DME scheme under exact communication constraints instead of asymptotic error bounds that are optimal. \n- The idea of treating the k-nearest codewords equally instead of just the closest seems to provide improvements over the prior works. This clean trick can be of independent interest.\n\nWEAKNESSES:\n- The work provides sufficient conditions for a DME scheme to be optimal, i.e., an optimal scheme has a particular canonical setup. \nThese conditions are not necessary, and not all schemes satisfying the canonical setup achieve optimal error. \n\n- Either some proofs have (probably fixable) errors, or I have misunderstood. So at least rephrasing or an elaborate explanation is required. Details are provided in the next section (Questions)\n\nQUESTIONS:\n1) As mentioned in the paper, the prior works of SQKR, MMRC, and FT21 can be seen as specific instantiations of random codebooks. Could it be possible that their schemes are exact-optimal as well and satisfy the conditions mentioned in this work? While a discussion is provided, can you comment on what condition is not met by each of these that confirms that they are not exact-optimal?\n\n2) Line 217-220 leading to Eq 29 needs further justification. Intuitively, this holds if the k points are equidistant, but I am not sure why the k-nearest neighbors will give an unbiased estimate. In the current manuscript, this is just a statement and not a formal proof of the fact that \"k-closest encoding consistently yields an unbiased scheme for any rotationally symmetric codebook.\" Further, it would be good to mention that for a rotationally symmetric codebook, the $\\sum_m U_m = 0$. \n\n3) Proof of Lemma 3.7 - What is top-k set? I am assuming it is the set of k-nearest neighbors to random a in s^M\nShould the summation have only $s_m$ and not $a^Ts_m $? \nTypo in eq 94 - the second summation should be over s_i.  \nEq 96 shows that you get an unbiased estimate of $e_1$ and not $a$. Also, other coordinates cannot be ignored. \n$r_k$ value is roughly $M$, so the error of the RRSC scheme will be $\\sim M^2$. So for $M = d$, there is an error of $d^2$ which is higher than the order optimal ones $(O(d/\\log d))$ for the same amount of allowed communication. A brief clarification on this would be great. \n\n4) For the shared randomness setup, will the server/nodes have to regenerate the codebook for each invocation of the algorithm? \n\n5) Some suggestions to improve the readability in my opinion would be: \n\n- Fix errors in proofs or provide better justifications\n- In Equation 26 the probability of choosing the k+1-th closest codeword is zero if k is an integer. This seems to be inconsistent with Algorithm 1, and the analysis that follows in Section 3.4\n- Clarify what the `=' in Def 3.3 means. It can be confusing to think that the two codebooks are equal (permutations of each other) rather than their distributions being identical.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies the mean estimation problem under communication and local differential privacy constraints in the non-asymptotic (exact optimal) setting, and proposed a randomization mechanism that satisfies the identified necessary property of the exact optimality.

STRENGTHS:
1. The authors proved a necessary condition that the codebook-generating distribution needs to be rotationally symmetric.

2. The authors further proposed the first exact optimality algorithm Random Rotating Simplex Coding (RRSC) that matches the necessary condition.

3. Empirical results in the paper showed that the proposed RRSC outperforms the state-of-art benchmarks (order-optimal) for the task.

4. The authors proposed interesting conjectures and clear future directions based on the design and properties of the algorithm.

WEAKNESSES:
1. line 177, "an random" to "a random"

2. The design of k-closest encoding and Theorem 3.6 (along with its proof in the Appendix) lacks the intuition on k, which seems to be valuable to explore both theoretically and empirically (as stated in the conjecture).

QUESTIONS:
For the choice of $k$, in the experiments are to minimize $C_k$ based on different $k$'s. How can such choice of $k$ guarantees the requirement in Theorem 3.6 (and its proof)?

LIMITATIONS:
Lack of intuition on k as described in weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper focuses on the problem of distributed mean estimation under local differential privacy (DP) and communication constraints, with shared randomness between users and the server. Previous works either achieve exact optimal mean squared error (MSE) using $O(d)$ bits or achieve order-optimal MSE with a large constant using $O(\varepsilon)$ bits. In this paper, the authors aim to tackle the same problem under shared randomness, aiming for both exact optimal MSE and communication efficiency. The proposed solution approaches the problem as a lossy compression problem. The authors demonstrate that the optimal scheme can be represented by a codebook through random coding. Additionally, they establish that the exact-optimal codebook-generating distribution must be rotationally symmetric. Empirically, the authors demonstrate that the proposed methods outperform existing approaches.

STRENGTHS:
1. This paper is very technical. The paper is clearly written and lays out its contributions succinctly. 

2. The proposed framework achieves exact optimality in terms of both MSE and communication efficiency.

WEAKNESSES:
It would be better if the authors could discuss why shared randomness is necessary to achieve exact optimality.

QUESTIONS:
See weaknesses

LIMITATIONS:
The limitations are discussed. This paper does not have a negative societal impact
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the problem of distributed mean estimation and aims to obtain \"exact optimal\" estimators under communication, (local) differential privacy and utility constraints. Exact optimality here means that instead of focusing on the order of complexity, the focus is also on the constants as well. Prior work either focused on \"order optimality\" or were exactly optimal for privacy and utility but only order optimal [Feldman-Talwar'11], [Shah-Chen-Balle-Kairouz-Theis'22]. This work achieves exact optimality through a k-closest encoding and a randomly generated codebook shared between the server and users.\n\nSTRENGTHS:\nThey obtain exact optimality for privacy-utility-communication tradeoffs of mean estimation under l_2 norm error. This improves upon prior work that could only obtain order optimality or exact optimality but only for some parameters. \n\nTheir work unifies the framework that existed in prior work [Feldman-Talwar'11], [Shah-Chen-Balle-Kairouz-Theis'22].\n\nTheir experiments show significant improvement in communication budget required compared to the previous work, especially in the setting where the number of bits is small.\n\nThe theorem regarding optimality of rotationally symmetric shared random codebooks could be of independent interest.\n\nWEAKNESSES:\nThere is no \"main theorem\" that sums up the results in the main results section for mean estimation, and I think it's necessary to include such a theorem.\n\nI think exact optimality compared to order optimality is a more niche setting. That being said, that's not necessarily a weakness, and it could be impactful in practice.\n\nQUESTIONS:\nI think part of the paragraph **unified framework** in the discussion section could be mentioned in the related work section as well.\n\nAs mentioned in the previous section a main theorem that clearly states the trade offs this work obtain for mean estimation should be included. The score currently given is assuming that such a theorem will be included.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}