Title: Correlation Aware Sparsified Mean Estimation Using Random Projection
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributed vector mean estimation

communication efficiency

cross-client correlation
Abstract: We study the problem of communication-efficient distributed vector mean estimation, which is a commonly used subroutine in distributed optimization and Federated Learning (FL). Rand-$k$ sparsification is a commonly used technique to reduce communication cost, where each client sends $k < d$ of its coordinates to the server. However, Rand-$k$ is agnostic to any correlations, that might exist between clients in practical scenarios. The recently proposed Rand-$k$-Spatial estimator leverages the cross-client correlation information at the server to improve Rand-$k$'s performance. Yet, the performance of Rand-$k$-Spatial is suboptimal, and improving mean estimation is key to a faster convergence in distributed optimization. We propose the Rand-Proj-Spatial estimator with a more flexible encoding-decoding procedure, which generalizes the encoding of Rand-$k$ by projecting the client vectors to a random $k$-dimensional subspace. We utilize Subsampled Randomized Hadamard Transform (SRHT) as the projection matrix, and show that Rand-Proj-Spatial with SRHT outperforms Rand-$k$-Spatial, using the correlation information more efficiently. Furthermore, we propose an approach to incorporate varying degrees of correlation, and suggest a practical variant of Rand-Proj-Spatial when the correlation information is not available to the server. Finally, experiments on real-world distributed optimization tasks showcase the superior performance of Rand-Proj-Spatial compared to Rand-$k$-Spatial and other more sophisticated sparsification techniques.

Correlation Aware Sparsified Mean Estimation Using Random Projection

Abstract

We study the problem of communication-efficient distributed vector mean estimation, a commonly used subroutine in distributed optimization and Federated Learning (FL). Randk sparsification is a commonly used technique to reduce communication cost, where each client sends k < d of its coordinates to the server. However, Randk is agnostic to any correlations, that might exist between clients in practical scenarios. The recently proposed Randk -Spatial estimator leverages the cross-client correlation information at the server to improve Randk 's performance. Yet, the performance of Randk -Spatial is suboptimal. We propose the Rand-Proj-Spatial estimator with a more flexible encoding-decoding procedure, which generalizes the encoding of Randk by projecting the client vectors to a random k -dimensional subspace. We utilize Subsampled Randomized Hadamard Transform (SRHT) as the projection matrix and show that Rand-Proj-Spatial with SRHT outperforms Randk -Spatial, using the correlation information more efficiently. Furthermore, we propose an approach to incorporate varying degrees of correlation and suggest a practical variant of Rand-Proj-Spatial when the correlation information is not available to the server. Experiments on real-world distributed optimization tasks showcase the superior performance of Rand-ProjSpatial compared to Randk -Spatial and other more sophisticated sparsification techniques.

1 Introduction

In modern machine learning applications, data is naturally distributed across a large number of edge devices or clients. The underlying learning task in such settings is modeled by distributed optimization or the recent paradigm of Federated Learning (FL) [1, 2, 3, 4]. A crucial subtask in distributed learning is for the server to compute the mean of the vectors sent by the clients. In FL, for example, clients run training steps on their local data and once-in-a-while send their local models (or local gradients) to the server, which averages them to compute the new global model. However, with the ever-increasing size of machine learning models [5, 6], and the limited battery life of the edge clients, communication cost is often the major constraint for the clients. This motivates the problem of (empirical) distributed mean estimation (DME) under communication constraints, as illustrated in Figure 1. Each of the n clients holds a vector x i ∈ R d , on which there are no distributional assumptions. Given a communication budget, each client sends a compressed version ̂ x i of its vector to the server, which utilizes these to compute an estimate of the mean vector 1 n ∑ n i =1 x i .

Quantization and sparsification are two major techniques for reducing the communication costs of DME. Quantization [7, 8, 9, 10] involves compressing each coordinate of the client vector to a given precision and aims to reduce the number of bits to represent each coordinate, achieving a constant reduction in the communication cost. However, the communication cost still remains Θ( d ) .

ECE

Sparsification, on the other hand, aims to reduce the number of coordinates each clinet sends and compresses each client vector to only k ≪ d of its coordinates (e.g. Randk [11]). As a result, sparsification reduces communication costs more aggressively compared to quantization, achieving better communication efficiency at a cost of only O ( k ) . While in practice, one can use a combination of quantization and sparsification techniques for communication cost reduction, in this work, we focus on the more aggressive sparsification techniques. We call k , the dimension of the vector each client sends to the server, the per-client communication budget.

Most existing works on sparsification ignore the potential correlation (or similarity) among the client vectors, which often exists in practice. For example, the data of a specific client in federated learning can be similar to that of multiple clients. Hence, it is reasonable to expect their models (or gradients) to be similar as well. To the best of our knowledge, [12] is the first work to account for spatial correlation across individual client vectors. They propose the Randk -Spatial family of unbiased estimators, which generalizes Randk and achieves a better estimation error in the presence of crossclient correlation. However, their ap-

Client 1

Client 2

Client n

X1 E Rd

X2 E IRd

Xn E IRd

Encoder

Encoder

Encoder

000

x1 E Rk

&2

EIRk

xn E IRk

x1

x2

Server

X = Decode(x1, x2, .., Xn)

proach is focused only on the server-side decoding procedure, while the clients do simple Randk encoding.

In this work, we consider a more general encoding scheme that directly compresses a vector from R d to R k using a (random) linear map. The encoded vector consists of k linear combinations of the original coordinates. Intuitively, this has a higher chance of capturing the large-magnitude coordinates ('heavy hitters') of the vector than randomly sampling k out of the d coordinates (Randk ), which is crucial for the estimator to recover the true mean vector. For example, consider a vector where only a few coordinates are heavy hitters. For small k , Randk has a decent chance of missing all the heavy hitters. But with a linear-maps-based general encoding procedure, the large coordinates are more likely to be encoded in the linear measurements, resulting in a more accurate estimator of the mean vector. Guided by this intuition, we ask:

Can we design an improved joint encoding-decoding scheme that utilizes the correlation information and achieves an improved estimation error?

One naïve solution is to apply the same random rotation matrix G ∈ R d × d to each client vector, before applying Randk or Randk -Spatial encoding. Indeed, such preprocessing is applied to improve the estimator using quantization techniques on heterogeneous vectors [13, 10]. However, as we see in Appendix A.1, for sparsification, we can show that this leads to no improvement. But what happens if every client uses a different random matrix, or applies a random k × d -dimensional linear map? How to design the corresponding decoding procedure to leverage cross-client correlation? As there is no way for one to directly apply the decoding procedure of Randk -Spatial in such cases. To answer these questions, we propose the Rand-Proj-Spatial family estimator. We propose a flexible encoding procedure in which each client applies its own random linear map to encode the vector. Further, our novel decoding procedure can better leverage cross-client correlation. The resulting mean estimator generalizes and improves over the Randk -Spatial family estimator.

Next, we discuss some reasonable restrictions we expect our mean estimator to obey. 1) Unbiased . An unbiased mean estimator is theoretically more convenient compared to a biased one [14]. 2) Non-adaptive . We focus on an encoding procedure that does not depend on the actual client data, as opposed to the adaptive ones, e.g. Randk with vector-based sampling probability [11, 15]. Designing a data-adaptive encoding procedure is computationally expensive as this might require using an iterative procedure to find out the sampling probabilities [11]. In practice, however, clients often have limited computational power compared to the server. Further, as discussed earlier, mean estimation is often a subroutine in more complicated tasks. For applications with streaming data [16],

the additional computational overhead of adaptive schemes is challenging to maintain. Note that both Randk and Randk -Spatial family estimator [12] are unbiased and non-adaptive .

In this paper, we focus on the severely communication-constrained case nk ≤ d , when the server receives very limited information about any single client vector. If nk ≫ d , we see in Appendix A.2 that the cross-client information has no additional advantage in terms of improving the mean estimate under both Randk -Spatial or Rand-Proj-Spatial, with different choices of random linear maps. Furthermore, when nk ≫ d , the performance of both the estimators converges to that of Randk . Intuitively, this means when the server receives sufficient information regarding the client vectors, it does not need to leverage cross-client correlation to improve the mean estimator.

Our contributions can be summarized as follows:

We propose the Rand-Proj-Spatial family estimator with a more flexible encoding-decoding procedure, which can better leverage the cross-client correlation information to achieve a more general and improved mean estimator compared to existing ones.

We show the benefit of using Subsampled Randomized Hadamard Transform (SRHT) as the random linear maps in Rand-Proj-Spatial in terms of better mean estimation error (MSE). We theoretically analyze the case when the correlation information is known at the server (see Theorems 4.3, 4.4 and Section 4.3). Further, we propose a practical configuration called Rand-Proj-Spatial(Avg) when the correlation is unknown.

We conduct experiments on common distributed optimization tasks, and demonstrate the superior performance of Rand-Proj-Spatial compared to existing sparsification techniques.

2 Related Work

Quantization and Sparsification. Commonly used techniques to achieve communication efficiency are quantization, sparsification, or more generic compression schemes, which generalize the former two [17]. Quantization involves either representing each coordinate of the vector by a small number of bits [8, 9, 10, 18, 19, 20], or more involved vector quantization techniques [21, 22]. Sparsification [15, 23, 24, 25, 26], on the other hand, involves communicating a small number k < d of coordinates, to the server. Common protocols include Randk [11], sending k uniformly randomly selected coordinates; Topk [27], sending the k largest magnitude coordinates; and a combination of the two [28]. Some recent works, with a focus on distributed learning, further refine these communicationsaving mechanisms [29] by incorporating temporal correlation or error feedback [14, 25].

Distributed Mean Estimation (DME). DME has wide applications in distributed optimization and FL. Most of the existing literature on DME either considers statistical mean estimation [30, 31], assuming that the data across clients is generated i.i.d. according to the same distribution, or empirical mean estimation [10, 32, 33, 12, 11, 34, 35], without making any distributional assumptions on the data. A recent line of work on empirical DME considers applying additional information available to the server, to further improve the mean estimate. This side information includes cross-client correlation [12, 13], or the memory of the past updates sent by the clients [36].

Subsampled Randomized Hadamard Transformation (SRHT). SRHT was introduced for random dimensionality reduction using sketching [37, 38, 39]. Common applications of SRHT include faster computation of matrix problems, such as low-rank approximation [40, 41], and machine learning tasks, such as ridge regression [42], and least square problems [43, 44, 45]. SRHT has also been applied to improve communication efficiency in distributed optimization [46] and FL [47, 48].

3 Preliminaries

Notation. We use bold lowercase (uppercase) letters, e.g. x ( G ) to denote vectors (matrices). e j ∈ R d , for j ∈ [ d ] , denotes the j -th canonical basis vector. ∥· ∥ 2 denotes the Euclidean norm. For a vector x , x ( j ) denotes its j -th coordinate. Given integer m , we denote by [ m ] the set { 1 , 2 , . . . , m } .

Problem Setup. Consider n geographically separated clients coordinated by a central server. Each client i ∈ [ n ] holds a vector x i ∈ R d , while the server wants to estimate the mean vector ¯ x ≜ 1 n ∑ n i =1 x i . Given a per-client communication budget of k ∈ [ d ] , each client i computes ̂ x i and sends it to the central server. ̂ x i is an approximation of x i that belongs to a random k -dimensional

subspace. Each client also sends a random seed to the server, which conveys the subspace information, and can usually be communicated using a negligible amount of bits. Having received the encoded vectors { ̂ x i } n i =1 , the server then computes ̂ x ∈ R d , an estimator of ¯ x . We consider the severely communication-constrained setting where nk ≤ d , when only a limited amount of information about the client vectors is seen by the server.

Error Metric. We measure the quality of the decoded vector ̂ x using the Mean Squared Error (MSE) E [ ∥ ̂ x -¯ x ∥ 2 2 ] , where the expectation is with respect to all the randomness in the encoding-decoding scheme. Our goal is to design an encoding-decoding algorithm to achieve an unbiased estimate ̂ x (i.e. E [ ̂ x ] = ¯ x ) that minimizes the MSE, given the per-client communication budget k . To consider an example, in randk sparsification, each client sends randomly selected k out of its d coordinates to the server. The server then computes the mean estimate as ̂ x ( Randk ) = 1 n d k ∑ n i =1 ̂ x i . By [12, Lemma 1], the MSE of Randk sparsification is given by

The Randk -Spatial Family Estimator. For large values of d k , the Randk MSE in Eq. 1 can be prohibitive. [12] proposed the Randk -Spatial family estimator, which achieves an improved MSE, by leveraging the knowledge of the correlation between client vectors at the server. The encoded vectors { ̂ x i } are the same as in Randk . However, the j -th coordinate of the decoded vector is given as

Here, T : R → R is a pre-defined transformation function of M j , the number of clients which sent their j -th coordinate, and ¯ β is a normalization constant to ensure ̂ x is an unbiased estimator of x . The resulting MSE is given by

̸

̸

where c 1 , c 2 are constants dependent on n, d, k and T , but independent of client vectors { x i } n i =1 . When the client vectors are orthogonal, i.e., ⟨ x i , x l ⟩ = 0 , for all i = l , [12] show that with appropriately chosen T , the MSE in Eq. 3 reduces to Eq. 1. However, if there exists a positive correlation between the vectors, the MSE in Eq. 3 is strictly smaller than that for Randk Eq. 1.

4 The Rand-Proj-Spatial Family Estimator

While the Randk -Spatial family estimator proposed in [12] focuses only on improving the decoding at the server, we consider a more general encoding-decoding scheme. Rather than simply communicating k out of the d coordinates of its vector x i to the server, client i applies a (random) linear map G i ∈ R k × d to x i and sends ̂ x i = G i x i ∈ R k to the server. The decoding process on the server first projects the encoded vectors { G i x i } n i =1 back to the d -dimensional space and then forms an estimate ̂ x . We motivate our new decoding procedure with the following regression problem:

To understand the motivation behind Eq. 4, first consider the special case where G i = I d for all i ∈ [ n ] , that is, the clients communicate their vectors without compressing. The server can then exactly compute the mean ¯ x = 1 n ∑ n i =1 x i . Equivalently, ¯ x is the solution of arg min x ∑ n i =1 ∥ x -x i ∥ 2 2 . In the more general setting, we require that the mean estimate ̂ x when encoded using the map G i , should be 'close' to the encoded vector G i x i originally sent by client i , for all clients i ∈ [ n ] .

We note the above intuition can also be translated into different regression problems to motivate the design of the new decoding procedure. We discuss in Appendix B.2 intuitive alternatives which,

unfortunately, either do not enable the usage of cross-client correlation information, or do not use such information effectively. We choose the formulation in Eq. 4 due to its analytical tractability and its direct relevance to our target error metric MSE. We note that it is possible to consider the problem in Eq. 4 in the other norms, such as the sum of ℓ 2 norms (without the squares) or the ℓ ∞ norm. We leave this as a future direction to explore.

The solution to Eq. 4 is given by ̂ x ( Rand-Proj ) = ( ∑ n i =1 G T i G i ) † ∑ n i =1 G T i G i x i , where † denotes the Moore-Penrose pseudo inverse [49]. However, while ̂ x ( Rand-Proj ) minimizes the error of the regression problem, our goal is to design an unbiased estimator that also improves the MSE. Therefore, we make the following two modifications to ̂ x ( Rand-Proj ) : First, to ensure that the mean estimate is unbiased, we scale the solution by a normalization factor ¯ β 1 . Second, to incorporate varying degrees of correlation among the clients, we propose to apply a scalar transformation function T : R → R to each of the eigenvalues of ∑ n i =1 G T i G i . The resulting Rand-Proj-Spatial family estimator is given by

Though applying the transformation function T in Rand-Proj-Spatial requires computing the eigendecomposition of ∑ n i =1 G T i G i . However, this happens only at the server, which has more computational power than the clients. Next, we observe that for appropriate choice of { G i } n i =1 , the Rand-Proj-Spatial family estimator reduces to the Randk -Spatial family estimator [12].

Lemma 4.1 (Recovering Randk -Spatial) . Suppose client i generates a subsampling matrix E i = [ e i 1 , . . . , e i k ] ⊤ , where { e j } d j =1 are the canonical basis vectors, and { i 1 , . . . , i k } are sampled from { 1 , . . . , d } without replacement. The encoded vectors are given as ̂ x i = E i x i . Given a function T , ̂ x computed as in Eq. 5 recovers the Randk -Spatial estimator.

The proof details are in Appendix C.5. We discuss the choice of T and how it compares to Randk -Spatial in detail in Section 4.3.

Remark 4.2. In the simple case when G i 's are subsampling matrices (as in Randk -Spatial [12]), the j -th diagonal entry of ∑ n i =1 G T i G i , M j conveys the number of clients which sent the j -th coordinate. Randk -Spatial incorporates correlation among client vectors by applying a function T to M j . Intuitively, it means scaling different coordinates differently. This is in contrast to Randk , which scales all the coordinates by d/k . In our more general case, we apply a function T to the eigenvalues of ∑ n i =1 G T i G i to similarly incorporate correlation in Rand-Proj-Spatial.

To showcase the utility of the Rand-Proj-Spatial family estimator, we propose to set the random linear maps G i to be scaled Subsampled Randomized Hadamard Transform (SRHT, e.g. [38]). Assuming d to be a power of 2 , the linear map G i is given as

where E i ∈ R k × d is the subsampling matrix, H ∈ R d × d is the (deterministic) Hadamard matrix and D i ∈ R d × d is a diagonal matrix with independent Rademacher random variables as its diagonal entries. We choose SRHT due to its superior performance compared to other random matrices. Other possible choices of random matrices for Rand-Proj-Spatial estimator include sketching matrices commonly used for dimensionality reduction, such as Gaussian [50, 51], row-normalized Gaussian, and Count Sketch [52], as well as error-correction coding matrices, such as Low-Density Parity Check (LDPC) [53] and Fountain Codes [54]. However, in the absence of correlation between client vectors, all these matrices suffer a higher MSE.

In the following, we first compare the MSE of Rand-Proj-Spatial with SRHT against Randk and Randk -Spatial in two extreme cases: when all the client vectors are identical, and when all the client vectors are orthogonal to each other. In both cases, we highlight the transformation function T used in Rand-Proj-Spatial (Eq. 5) to incorporate the knowledge of cross-client correlation. We define

̸

to measure the correlation between the client vectors. Note that R∈ [ -1 , n -1] . R = 0 implies all client vectors are orthogonal, while R = n -1 implies identical client vectors.

4.1 Case I: Identical Client Vectors ( R = n -1 )

When all the client vectors are identical ( x i ≡ x ), [12] showed that setting the transformation T to identity, i.e., T ( m ) = m , for all m , leads to the minimum MSE in the Randk -Spatial family of estimators. The resulting estimator is called Randk -Spatial (Max). Under the same setting, using the same transformation T in Rand-Proj-Spatial with SRHT, the decoded vector in Eq. 5 simplifies to

n

where S := ∑ i =1 G T i G i . By construction, rank ( S ) ≤ nk , and we focus on the case nk ≤ d . Limitation of Subsampling matrices. As mentioned above, with G i = E i , ∀ i ∈ [ n ] , we recover the Randk -Spatial family of estimators. In this case, S is a diagonal matrix, where each diagonal entry S jj = M j , j ∈ [ d ] . M j is the number of clients which sent their j -th coordinate to the server. To ensure rank ( S ) = nk , we need S jj ≤ 1 , ∀ j , i.e., each of the d coordinates is sent by at most one client. If all the clients sample their matrices { E i } n i =1 independently, this happens with probability

Therefore, to guarantee that S is full-rank, each client would need the subsampling information of all the other clients. This not only requires additional communication but also has serious privacy implications. Essentially, the limitation with subsampling matrices E i is that the eigenvectors of S are restricted to be canonical basis vectors { e j } d j =1 . Generalizing G i 's to general rank k matrices relaxes this constraint and hence we can ensure that S is full-rank with high probability. In the next result, we show the benefit of choosing G i as SRHT matrices. We call the resulting estimator Rand-Proj-Spatial(Max).

Theorem 4.3 (MSE under Full Correlation) . Consider n clients, each holding the same vector x ∈ R d . Suppose we set T ( λ ) = λ , ¯ β = d k in Eq. 5, and the random linear map G i at each client to be an SRHT matrix. Let δ be the probability that S = ∑ n i =1 G T i G i does not have full rank. Then, for nk ≤ d ,

The proof details are in Appendix C.1. To compare the performance of Rand-Proj-Spatial(Max) against Randk , we show in Appendix C.2 that for n ≥ 2 , as long as δ ≤ 2 3 , the MSE of Rand-Proj-Spatial(Max) is less than that of Randk . Furthermore, in Appendix C.3 we empirically demonstrate that with d ∈ { 32 , 64 , 128 , . . . , 1024 } and different values of nk ≤ d , the rank of S is full with high probability, i.e., δ ≈ 0 . This implies E [ ∥ ̂ x ( Rand-Proj-Spatial(Max) ) -¯ x ∥ 2 2 ] ≈ ( d nk -1) ∥ x ∥ 2 2 .

Futhermore, since setting G i as SRHT significantly increases the probability of recovering nk coordinates of x , the MSE of Rand-Proj-Spatial with SRHT (Eq. 4.3) is strictly less than that of Randk -Spatial (Eq. 3). We also compare the MSEs of the three estimators in Figure 2 in the following setting: ∥ x ∥ 2 = 1 , d = 1024 , n ∈ { 10 , 20 , 50 , 100 } and small k values such that nk < d .

20

40

60

80

100

k

0

1

2

3

4

5

MSE

n = 10, d = 1024

Rand-k

Rand-k-Spatial(Max)

Rand-Proj-Spatial(Max)

10

20

30

40

50

k

0

1

2

3

4

5

MSE

n = 20, d = 1024

Rand-k

Rand-k-Spatial(Max)

Rand-Proj-Spatial(Max)

4

8

12

16

20

k

0

1

2

3

4

5

MSE

n = 50, d = 1024

Rand-k

Rand-k-Spatial(Max)

Rand-Proj-Spatial(Max)

2

4

6

8

10

k

0

1

2

3

4

5

MSE

n = 100, d = 1024

Rand-k

Rand-k-Spatial(Max)

Rand-Proj-Spatial(Max)

4.2 Case II: Orthogonal Client Vectors ( R = 0 )

When all the client vectors are orthogonal to each other, [12] showed that Randk has the lowest MSE among the Randk -Spatial family of decoders. We show in the next result that if we set the

random linear maps G i at client i to be SRHT, and choose the fixed transformation T ≡ 1 as in [12], Rand-Proj-Spatial achieves the same MSE as that of Randk .

Theorem 4.4 (MSE under No Correlation) . Consider n clients, each holding a vector x i ∈ R d , ∀ i ∈ [ n ] . Suppose we set T ≡ 1 , ¯ β = d 2 k in Eq. 5, and the random linear map G i at each client to be an SRHT matrix. Then, for nk ≤ d ,

The proof details are in Appendix C.4. Theorem 4.4 above shows that with zero correlation among client vectors, Rand-Proj-Spatial achieves the same MSE as that of Randk .

4.3 Incorporating Varying Degrees of Correlation

In practice, it unlikely that all the client vectors are either identical or orthogonal to each other. In general, there is some 'imperfect' correlation among the client vectors, i.e., R∈ (0 , n -1) . Given correlation level R , [12] shows that the estimator from the Randk -Spatial family that minimizes the MSE is given by the following transformation.

Recall from Section 4.1 (Section 4.2) that setting T ( m ) = 1 ( T ( m ) = m ) leads to the estimator among the Randk -Spatial family that minimizes MSE when there is zero (maximum) correlation among the client vectors. We observe the function T defined in Eq. 11 essentially interpolates between the two extreme cases, using the normalized degree of correlation R n -1 ∈ [ -1 n -1 , 1] as the weight. This motivates us to apply the same function T defined in Eq. 11 on the eigenvalues of S = ∑ n i =1 G T i G i in Rand-Proj-Spatial. As we shall see in our results, the resulting Rand-Proj-Spatial family estimator improves over the MSE of both Randk and Randk -Spatial family estimator.

We note that deriving a closed-form expression of MSE for Rand-Proj-Spatial with SRHT in the general case with the transformation function T (Eq. 11) is hard (we elaborate on this in Appendix B.3), as this requires a closed form expression for the non-asymptotic distributions of eigenvalues and eigenvectors of the random matrix S . To the best of our knowledge, previous analyses of SRHT, for example in [37, 38, 39, 45, 55], rely on the asymptotic properties of SRHT, such as the limiting eigen spectrum, or concentration bounds on the singular values, to derive asymptotic or approximate guarantees. However, to analyze the MSE of Rand-Proj-Spatial, we need an exact, non-asymptotic analysis of the eigenvalues and eigenvectors distribution of SRHT. Given the apparent intractability of the theoretical analysis, we compare the MSE of Rand-Proj-Spatial, Randk -Spatial, and Randk via simulations.

Simulations. In each experiment, we first simulate ¯ β in Eq. 5, which ensures our estimator is unbiased, based on 1000 random runs. Given the degree of correlation R , we then compute the squared error, i.e. ∥ ̂ x ( Rand-Proj-Spatial ) -¯ x ∥ 2 2 , where Rand-Proj-Spatial has G i as SRHT matrix (Eq. 6) and T as in Eq. 11. We plot the average over 1000 random runs as an approximation to MSE. Each client holds a d -dimensional base vector e j for some j ∈ [ d ] , and so two clients either hold the same or orthogonal vectors. We control the degree of correlation R by changing the number of clients which hold the same vector. We consider d = 1024 , n ∈ { 21 , 51 } . We consider positive correlation values, where R is chosen to be linearly spaced within [0 , n -1] . Hence, for n = 21 , we use R ∈ { 4 , 8 , 12 , 16 } and for n = 51 , we use R ∈ { 10 , 20 , 30 , 40 } . All results are presented in Figure 3. As expected, given R , Rand-Proj-Spatial consistently achieves a lower MSE than the lowest possible MSE from the Randk -Spatial family decoder. Additional results with different values of n, d, k , including the setting nk ≪ d , can be found in Appendix B.4.

A Practical Configuration. In reality, it is hard to know the correlation information R among the client vectors. [12] uses the transformation function which interpolates to the middle point between the full correlation and no correlation cases, such that T ( m ) = 1 + n 2 m -1 n -1 . Randk -Spatial with such T is called Randk -Spatial(Avg). Following this approach, we evaluate Rand-Proj-Spatial with SRHT using this T , and call it Rand-Proj-Spatial(Avg) in practical settings (see Figure 4).

4

8

12

16

4.2

4.3

4.4

4.5

4.6

4.7

4.8

MSE

n = 21, k = 10

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

4

8

12

16

1.8

1.9

2.0

2.1

2.2

2.3

2.4

n = 21, k = 20

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

4

8

12

16

1.1

1.2

1.3

1.4

1.5

1.6

n = 21, k = 30

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

4

8

12

16

0.7

0.8

0.9

1.0

1.1

n = 21, k = 40

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

10

20

30

40

4.4

4.5

4.6

4.7

4.8

4.9

5.0

MSE

n = 51, k = 4

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

10

20

30

40

1.9

2.0

2.1

2.2

2.3

2.4

2.5

n = 51, k = 8

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

10

20

30

40

1.1

1.2

1.3

1.4

1.5

1.6

n = 51, k = 12

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

10

20

30

40

0.7

0.8

0.9

1.0

1.1

1.2

n = 51, k = 16

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

5 Experiments

We consider three practical distributed optimization tasks for evaluation: distributed power iteration, distributed k -means and distributed linear regression. We compare Rand-Proj-Spatial(Avg) against Randk , Randk -Spatial(Avg), and two more sophisticated but widely used sparsification schemes: non-uniform coordinate-wise gradient sparsification [15] (we call it Randk (Wangni)) and the Induced compressor with Randk + Topk [14]. The results are presented in Figure 4.

Dataset. For both distributed power iteration and distributed k -means, we use the test set of the Fashion-MNIST dataset [56] consisting of 10000 samples. The original images from Fashion-MNIST are 28 × 28 in size. We preprocess and resize each image to be 32 × 32 . Resizing images to have their dimension as a power of 2 is a common technique used in computer vision to accelerate the convolution operation. We use the UJIndoor dataset 2 for distributed linear regression. We subsample 10000 data points, and use the first 512 out of the total 520 features on signals of phone calls. The task is to predict the longitude of the location of a phone call. In all the experiments in Figure 4, the datasets are split IID across the clients via random shuffling. In Appendix D.1, we have additional results for non-IID data split across the clients.

Setup and Metric. Recall that n denotes the number of clients, k the per-client communication budget, and d the vector dimension. For Rand-Proj-Spatial, we use the first 50 iterations to estimate ¯ β (see Eq. 5). Note that ¯ β only depends on n, k, d , and T (the transformation function in Eq. 5), but is independent of the dataset. We repeat the experiments across 10 independent runs, and report the mean MSE (solid lines) and one standard deviation (shaded regions) for each estimator. For each task, we plot the squared error of the mean estimator ̂ x , i.e., ∥ ̂ x -¯ x ∥ 2 2 , and the values of the task-specific loss function, detailed below.

Tasks and Settings:

1. Distributed power iteration. We estimate the principle eigenvector of the covariance matrix, with the dataset ( Fashion-MNIST ) distributed across the n clients. In each iteration, each client computes a local principle eigenvector estimate based on a single power iteration and sends an encoded version to the server. The server then computes a global estimate and sends it back to the clients. The task-specific loss here is ∥ v t -v top ∥ 2 , where v t is the global estimate of the principal eigenvector at iteration t , and v top is the true principle eigenvector.

2. Distributed k -means. We perform k -means clustering [57] with the data distributed across n clients ( Fashion-MNIST , 10 classes) using Lloyd's algorithm. At each iteration, each client performs a single iteration of k -means to find its local centroids and sends the encoded version to the server. The server then computes an estimate of the global centroids and sends them back to the clients. We report the average squared mean estimation error across 10 clusters, and the k -means loss, i.e., the sum of the squared distances of the data points to the centroids.

For both distributed power iterations and distributed k -means, we run the experiments for 30 iterations and consider two different settings: n = 10 , k = 102 and n = 50 , k = 20 .

Distributed Power Iteration

0

10

20

30

Iteration No.

2

3

4

5

6

Squared Error

1e8

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.6

0.7

0.8

0.9

Loss

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.0

1.5

2.0

2.5

3.0

Squared Error

1e7

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.6

0.7

0.8

0.9

Loss

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

50

100

150

200

250

Squared Error

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.0

1.2

1.4

1.6

1.8

Loss

1e6

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

50

100

150

200

250

300

350

Squared Error

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.0

1.2

1.4

1.6

1.8

2.0

Loss

1e6

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means

10

20

30

40

50

Iteration No.

1

2

3

4

Squared Error

1e5

n=10, k=50, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.0

2.2

2.4

2.6

Loss

1e4

n=10, k=50, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2

4

6

Squared Error

1e5

n=50, k=10, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.0

2.2

2.4

2.6

Loss

1e4

n=50, k=10, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Linear Regression

3. Distributed linear regression. We perform linear regression on the UJIndoor dataset distributed across n clients using SGD. At each iteration, each client computes a local gradient and sends an encoded version to the server. The server computes a global estimate of the gradient, performs an SGD step, and sends the updated parameter to the clients. We run the experiments for 50 iterations with learning rate 0 . 001 . The task-specific loss is the linear regression loss, i.e. empirical mean squared error. To have a proper scale that better showcases the difference in performance of different estimators, we plot the results starting from the 10th iteration.

Results. It is evident from Figure 4 that Rand-Proj-Spatial(Avg), our estimator with the practical configuration T (see Section 4.3) that does not require the knowledge of the actual degree of correlation among clients, consistently outperforms the other estimators in all three tasks. Additional experiments for the three tasks are included in Appendix D.1. Furthermore, we present the wall-clock time to encode and decode client vectors using different sparsification schemes in Figure 5. Though Rand-Proj-Spatial(Avg) has the longest decoding time, the encoding time of Rand-Proj-Spatial(Avg) is less than that of the adaptive Randk (Wangni) sparsifier. In practice, the server has more compu-

Distributed Power Iteration

0

10

20

30

Iteration No.

2

4

6

8

Encoding Time

1e

3 n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Decoding Time

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

2

4

6

8

Encoding Time

1e

3

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Decoding Time

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.4

0.6

0.8

1.0

Encoding Time

n=10, k=102, d=1024

0

10

20

30

Iteration No.

0

5

10

15

20

25

30

Decoding Time

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1

2

3

Encoding Time

n=50, k=20, d=1024

0

10

20

30

Iteration No.

0

5

10

15

20

25

30

Decoding Time

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means

0

10

20

30

40

50

Iteration No.

0.1

0.2

0.3

0.4

Encoding Time

1e

1

n=10, k=50, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

40

50

Iteration No.

0

1

2

3

4

Decoding Time

1e

1

n=10, k=50, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

40

50

Iteration No.

0.0

0.5

1.0

1.5

2.0

Encoding Time

1e

1

n=50, k=10, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

40

50

Iteration No.

0

1

2

3

4

Decoding Time

1e

1

n=50, k=10, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Linear Regression

tational power than the clients and hence can afford a longer decoding time. Therefore, it is more important to have efficient encoding procedures.

6 Limitations

We note two practical limitations of the proposed Rand-Proj-Spatial.

Computation Time of Rand-Proj-Spatial. The encoding time of Rand-Proj-Spatial is O ( kd ) , while the decoding time is O ( d 2 · nk ) . The computation bottleneck in decoding is computing the eigendecomposition of the d × d matrix ∑ n i =1 G T i G i of rank at most nk . Improving the computation time for both the encoding and decoding schemes is an important direction for future work.

Perfect Shared Randomness. It is common to assume perfect shared randomness between the server and the clients in distributed settings [58]. However, to perfectly simulate randomness using Pseudo Random Number Generator (PRNG), at least log 2 d bits of the seed need to be exchanged in practice. We acknowledge this gap between theory and practice.

7 Conclusion

In this paper, we propose the Rand-Proj-Spatial estimator, a novel encoding-decoding scheme, for communication-efficient distributed mean estimation. The proposed client-side encoding generalizes and improves the commonly used Randk sparsification, by utilizing projections onto general k -dimensional subspaces. On the server side, cross-client correlation is leveraged to improve the approximation error. Compared to existing methods, the proposed scheme consistently achieves better mean estimation error across a variety of tasks. Potential future directions include improving the computation time of Rand-Proj-Spatial and exploring whether the proposed Rand-Proj-Spatial achieves the optimal estimation error among the class of non-adaptive estimators, given correlation information. Furthermore, combining sparsification and quantization techniques and deriving such algorithms with the optimal communication cost-estimation error trade-offs would be interesting.

Acknowledgments

We would like to thank the anonymous reviewer for providing valuable feedback on the title of this work, interesting open problems, alternative motivating regression problems and practical limitations of shared randomness. This work was supported in part by NSF grants CCF 2045694, CCF 2107085, CNS-2112471, and ONR N00014-23-1- 2149.

References

Jakub Koneˇ cn` y, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 , 2016.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics , pages 1273-1282. PMLR, 2017.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan AlShedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 , 2021.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

John A Gubner. Distributed estimation and quantization. IEEE Transactions on Information Theory , 39(4):1456-1459, 1993.

Peter Davies, Vijaykrishna Gurunathan, Niusha Moshrefi, Saleh Ashkboos, and Dan Alistarh. New bounds for distributed mean estimation and variance reduction, 2021.

Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning, 2022.

Ananda Theertha Suresh, X Yu Felix, Sanjiv Kumar, and H Brendan McMahan. Distributed mean estimation with limited communication. In International conference on machine learning , pages 3329-3337. PMLR, 2017.

Jakub Koneˇ cn` y and Peter Richtárik. Randomized distributed mean estimation: Accuracy vs. communication. Frontiers in Applied Mathematics and Statistics , 4:62, 2018.

Divyansh Jhunjhunwala, Ankur Mallick, Advait Harshal Gadhikar, Swanand Kadhe, and Gauri Joshi. Leveraging spatial and temporal correlations in sparsified mean estimation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Ananda Theertha Suresh, Ziteng Sun, Jae Ro, and Felix Yu. Correlated quantization for distributed mean estimation and optimization. In International Conference on Machine Learning , pages 20856-20876. PMLR, 2022.

Samuel Horváth and Peter Richtarik. A better alternative to error feedback for communicationefficient distributed learning. In International Conference on Learning Representations , 2021.

Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication-efficient distributed optimization. Advances in Neural Information Processing Systems , 31, 2018.

Matthew Nokleby and Waheed U Bajwa. Stochastic optimization from distributed streaming data in rate-limited networks. IEEE transactions on signal and information processing over networks , 5(1):152-167, 2018.

Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-sgd: Distributed sgd with quantization, sparsification and local computations. Advances in Neural Information Processing Systems , 32, 2019.

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. Advances in neural information processing systems , 30, 2017.

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning , pages 560-569. PMLR, 2018.

Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics , pages 20212031. PMLR, 2020.

Nir Shlezinger, Mingzhe Chen, Yonina C Eldar, H Vincent Poor, and Shuguang Cui. Uveqfed: Universal vector quantization for federated learning. IEEE Transactions on Signal Processing , 69:500-514, 2020.

Venkata Gandikota, Daniel Kane, Raj Kumar Maity, and Arya Mazumdar. vqsgd: Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics , pages 2197-2205. PMLR, 2021.

Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. Advances in Neural Information Processing Systems , 31, 2018.

Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances in Neural Information Processing Systems , 31, 2018.

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning , pages 3252-3261. PMLR, 2019.

Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems , 31(9):3400-3413, 2019.

Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, and Simon See. Understanding top-k sparsification in distributed deep learning. arXiv preprint arXiv:1911.08772 , 2019.

Leighton Pate Barnes, Huseyin A Inan, Berivan Isik, and Ayfer Özgür. rtop-k: A statistical estimation approach to distributed sgd. IEEE Journal on Selected Areas in Information Theory , 1(3):897-907, 2020.

Emre Ozfatura, Kerem Ozfatura, and Deniz Gündüz. Time-correlated sparsification for communication-efficient federated learning. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 461-466. IEEE, 2021.

Yuchen Zhang, John Duchi, Michael I Jordan, and Martin J Wainwright. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. Advances in Neural Information Processing Systems , 26, 2013.

Ankit Garg, Tengyu Ma, and Huy Nguyen. On communication cost of distributed statistical estimation and dimensionality. Advances in Neural Information Processing Systems , 27, 2014.

Wei-Ning Chen, Peter Kairouz, and Ayfer Ozgur. Breaking the communication-privacy-accuracy trilemma. Advances in Neural Information Processing Systems , 33:3312-3324, 2020.

Prathamesh Mayekar, Ananda Theertha Suresh, and Himanshu Tyagi. Wyner-ziv estimators: Efficient distributed mean estimation with side-information. In International Conference on Artificial Intelligence and Statistics , pages 3502-3510. PMLR, 2021.

Shay Vargaftik, Ran Ben-Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzenmacher. Drive: One-bit distributed mean estimation. Advances in Neural Information Processing Systems , 34:362-377, 2021.

Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben Itzhak, and Michael Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning. In International Conference on Machine Learning , pages 21984-22014. PMLR, 2022.

Kai Liang and Youlong Wu. Improved communication efficiency for distributed mean estimation with side information. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 3185-3190. IEEE, 2021.

Nir Ailon and Bernard Chazelle. Approximate nearest neighbors and the fast johnsonlindenstrauss transform. In Proceedings of the Thirty-Eighth Annual ACM Symposium on Theory of Computing , STOC '06, page 557-563, New York, NY, USA, 2006. Association for Computing Machinery.

Joel A. Tropp. Improved analysis of the subsampled randomized hadamard transform, 2011.

Jonathan Lacotte, Sifan Liu, Edgar Dobriban, and Mert Pilanci. Optimal iterative sketching with the subsampled randomized hadamard transform, 2020.

Oleg Balabanov, Matthias Beaupère, Laura Grigori, and Victor Lederer. Block subsampled randomized Hadamard transform for low-rank approximation on distributed architectures. working paper or preprint, October 2022.

Christos Boutsidis and Alex Gittens. Improved matrix algorithms via the subsampled randomized hadamard transform, 2013.

Yichao Lu, Paramveer Dhillon, Dean P Foster, and Lyle Ungar. Faster ridge regression via the subsampled randomized hadamard transform. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Mostafa Haghir Chehreghani. Subsampled randomized hadamard transform for regression of dynamic graphs. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , CIKM '20, page 2045-2048, New York, NY, USA, 2020. Association for Computing Machinery.

Dan Teng, Xiaowei Zhang, Li Cheng, and Delin Chu. Least squares approximation via sparse subsampled randomized hadamard transform. IEEE Transactions on Big Data , 8(2):446-457, 2022.

Jonathan Lacotte and Mert Pilanci. Optimal randomized first-order methods for least-squares problems, 2020.

Nikita Ivkin, Daniel Rothchild, Enayat Ullah, Ion Stoica, Raman Arora, et al. Communicationefficient distributed sgd with sketching. Advances in Neural Information Processing Systems , 32, 2019.

Farzin Haddadpour, Belhal Karimi, Ping Li, and Xiaoyun Li. Fedsketch: Communicationefficient and private federated learning via sketching. arXiv preprint arXiv:2008.04975 , 2020.

Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-efficient federated learning with sketching. In International Conference on Machine Learning , pages 8253-8265. PMLR, 2020.

Gene H Golub and Charles F Van Loan. Matrix computations . JHU press, 2013.

Kilian Q Weinberger, Fei Sha, and Lawrence K Saul. Learning a kernel matrix for nonlinear dimensionality reduction. In Proceedings of the twenty-first international conference on Machine learning , page 106, 2004.

Rohit Tripathy, Ilias Bilionis, and Marcial Gonzalez. Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation. Journal of Computational Physics , 321:191-223, 2016.

Gregory T. Minton and Eric Price. Improved concentration bounds for count-sketch, 2013.

R. Gallager. Low-density parity-check codes. IRE Transactions on Information Theory , 8(1):2128, 1962.

Amin Shokrollahi. Fountain codes. Iee Proceedings-communications - IEE PROC-COMMUN , 152, 01 2005.

Zijian Lei and Liang Lan. Improved subsampled randomized hadamard transform for linear svm. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4519-4526, 2020.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Maria-Florina F Balcan, Steven Ehrlich, and Yingyu Liang. Distributed k -means and k -median clustering on general topologies. Advances in neural information processing systems , 26, 2013.

Mingxun Zhou, Tianhao Wang, T-H. Hubert Chan, Giulia Fanti, and Elaine Shi. Locally differentially private sparse vector aggregation. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 422-439, 2022.

Gene H. Golub. Some modified matrix eigenvalue problems. SIAM Review , 15(2):318-334, 1973.

Ming Gu and Stanley C. Eisenstat. A stable and efficient algorithm for the rank-one modification of the symmetric eigenproblem. SIAM Journal on Matrix Analysis and Applications , 15(4):12661276, 1994.

Peter Arbenz, Walter Gander, and Gene H. Golub. Restricted rank modification of the symmetric eigenvalue problem: Theoretical considerations. Linear Algebra and its Applications , 104:7595, 1988.

H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data, 2023.

Appendices

A Additional Details on Motivation in Introduction

A.1 Preprocssing all client vectors by the same random matrix does not improve performance

Consider n clients. Suppose client i holds a vector x i ∈ R d . We want to apply Randk or Randk -Spatial, while also making the encoding process more flexible than just randomly choosing k out of d coordinates. One naïve way of doing this is for each client to pre-process its vector by applying an orthogonal matrix G ∈ R d × d that is the same across all clients. Such a technique might be helpful in improving the performance of quantization because the MSE due to quantization often depends on how uniform the coordinates of x i 's are, i.e. whether the coordinates of x i have values close to each other. G is designed to be the random matrix (e.g. SRHT) that rotates x i and makes its coordinates uniform.

Each client sends the server ̂ x i = E i G x i , where E i ∈ R k × d is the subsamaping matrix. If we use Randk , the server can decode each client vector by first applying the decoding procedure of Randk and then rotating it back to the original space, i.e., ̂ x (Naïve) i = d k G T E T i E i G x i . Note that

Hence, ̂ x (Naïve) i is unbiased. The MSE of ̂ x (Naïve) = 1 n ∑ n i =1 ̂ x (Naïve) i is given as

̸

Next, we bound the first term in Eq. 12.

The second term in Eq. 12 can also be simplified as follows.

And the MSE is now

Plugging Eq. 13 and Eq. 14 into Eq. 12, we get the MSE is

which has exactly the same MSE as that of Randk . The problem is that if each client applies the same rotational matrix G , simply rotating the vectors will not change the ℓ 2 norm of the decoded vector, and hence the MSE. Similarly, if one applies Randk -Spatial, one ends up having exactly the same MSE as that of Randk -Spatial as well. Hence, we need to design a new decoding procedure when the encoding procedure at the clients are more flexible.

A.2 nk ≫ d is not interesting

One can rewrite ∑ n i =1 G T i G i in the Rand-Proj-Spatial estimator (Eq. 5) as ∑ n i =1 G T i G i = ∑ nk j =1 g i g T i , where g j ∈ R d and g ik , g ik +1 , . . . , g ( i +1) k are the rows of G i . Since when nk ≫ d , ∑ nk j =1 g j g T j → E [ ∑ n j =1 g j g T j ] due to Law of Large Numbers, one way to see the limiting MSE of Rand-Proj-Spatial when nk is large is to approximate ∑ n i =1 ∑ nk j =1 g i g T i by its expectation.

By Lemma 4.1, when G i = E i , Rand-Proj-Spatial recovers Randk -Spatial. We now discuss the limiting behavior of Randk -Spatial when nk ≫ d by leveraging our proposed Rand-Proj-Spatial. In this case, each g j can be viewed as a random based vector e w for w randomly chosen in [ d ] . ∑ nk i =1 g j g T j → E [ ∑ nk i =1 g j g T j ] = ∑ nk i =1 1 d I d = nk d I d . And so the scalar ¯ β in Eq. 5 to ensure an unbiased estimator is computed as

which is exactly the same MSE as Randk . This implies when nk is large, the MSE of Randk -Spatial does not get improved compared to Randk with correlation information. Intuitively, this implies when nk ≫ d , the server gets enough amount of information from the client, and does not need correlation to improve its estimator. Hence, we focus on the more interesting case when nk < d -that is, when the server does not have enough information from the clients, and thus wants to use additional information, i.e. cross-client correlation, to improve its estimator.

B Additional Details on the Rand-Proj-Spatial Family Estimator

B.1 ¯ β is a scalar

From Eq. 20 in the proof of Theorem 4.3 and Eq. 25 in the proof of Theorem 4.4, it is evident that the unbiasedness of the mean estimator ̂ x Rand-Proj-Spatial is ensured collectively by

The random sampling matrices { E i } .

The orthogonality of scaled Hadamard matrices H T H = d I d = HH T .

The rademacher diagonal matrices, with the property ( D i ) 2 = I d .

B.2 Alternative motivating regression problems

Alternative motivating regression problem 1.

Let G i ∈ R k × d and W i ∈ R d × k be the encoding and decoding matrix for client i . One possible alternative estimator that translates the intuition that the decoded vector should be close to the client's original vector, for all clients, is by solving the following regression problem,

where W = ( W 1 , W 2 , . . . , W n ) and the constraint enforces unbiasedness of the estimator. The estimator is then the solution of the above problem. However, we note that optimizing a decoding matrix W i for each client leads to performing individual decoding of each client's compressed vector instead of a joint decoding process that considers all clients' compressed vectors. Only a joint decoding process can achieve the goal of leveraging cross-client information to reduce the estimation error. Indeed, we show as follows that solving the above optimization problem in Eq. 15 recovers the MSE of our baseline Randk . Note

̸

̸

By the constraint of unbiasedness, i.e., ¯ x = 1 n ∑ n i =1 x i = 1 n ∑ n i =1 E [ W i G i x i ] , there is

We now show that a sufficient and necessary condition to satisfy the above unbiasedness constraint is that for all i ∈ [ n ] , E [ W i G i ] = I d .

Sufficiency. It is obvious that if for all i ∈ [ n ] , E [ W i G i ] = I d , then we have 1 n E [( I d -W i G i ) x i ] = 0 .

Necessity. Consider the special case that for some i ∈ [ n ] and λ ∈ [ d ] , x i = n e λ , where e λ is the λ -th canonical basis vector, and x j = 0 , and for all j ∈ [ n ] \ { i } . Then,

where [ · ] λ denotes the λ -th column of matrix E [ W i G i ] .

Since our approach is agnostic to the choice of vectors, we need this choice of decoder matrices, by varying λ over [ d ] , we see that we need E [ W i G i ] = I d . And by varying i over [ n ] , we see that we need E [ W j G j ] = I d for all j ∈ [ n ] .

Therefore, ¯ x = 1 n ∑ n i =1 E [ W i G i x i ] ⇔∀ i ∈ [ n ] , E [ W i G i ] = I d .

This implies the second term of f ( W ) in Eq. 16 is 0, that is,

̸

Hence, we only need to solve

Since each W i appears in f 2 ( W ) separately, each W i can be optimized separately, via solving

One natural solution is to take W i = d k G † i , ∀ i ∈ [ n ] . For i ∈ [ n ] , let G i = V i Λ i U T i be its SVD, where V i ∈ R k × d and U i ∈ R d × d are orthogonal matrices. Then,

where Σ is a diagonal matrix with 0s and 1s on the diagonal.

For simplicity, we assume the random matrix U i follows a continuous distribution. U i being discrete follows a similar analysis. Let µ ( U i ) be the measure of U i .

which means the estimator 1 n ∑ n i =1 k d G † i G i satisfies unbiasedness. The MSE is now

Again, let G i = V i Λ i U T i be its SVD and consider W i G i = d k U i Σ i U T i , where Σ i is a diagonal matrix with 0s and 1s. Then,

Since G i has rank k , Σ i is a diagonal matrix with k out of d entries being 1 and the rest being 0. Let µ ( U i ) be the measure of U i . Hence, for i ∈ [ n ] ,

Therefore, the MSE of the estimator, which is the solution of the optimization problem in Eq. 15, is

which is the same MSE as that of Randk .

Alternative motivating regression problem 2.

Another motivating regression problem based on which we can design our estimator is

Note that G i ∈ R k × d , ∀ i ∈ [ n ] , and so the solution to the above problem is

and to ensure unbiasedness of the estimator, we can set ¯ β ∈ R and have the estimator as

It is not hard to see this estimator does not lead to an MSE as low as Rand-Proj-Spatial does. Consider the full correlation case, i.e., x i = x , ∀ i ∈ [ n ] , for example, the estimator is now

Note that rank ( 1 n ∑ n i =1 G i ) is at most k , since G i ∈ R k × d , ∀ i ∈ [ k ] . This limits the amount of information of x the server can recover.

While recall that in this case, the Rand-Proj-Spatial estimator is

where S can have rank at most nk .

B.3 Why deriving the MSE of Rand-Proj-Spatial with SRHT is hard

To analyze Eq. 11, one needs to compute the distribution of eigendecomposition of S = ∑ n i =1 G T i G i , i.e. the sum of the covariance of SRHT. To the best of our knowledge, there is no non-trivial closed form expression of the distribution of eigen-decomposition of even a single G T i G i , when G i is SRHT, or other commonly used random matrices, e.g. Gaussian. When G i is SRHT, since G T i G i = D i HE T i E i HD i and the eigenvalues of E T i E i are just diagonal entries, one might attempt to analyze HD i . While the hardmard matrix H 's eigenvalues and eigenvectors are known 3 , the result can hardly be applied to analyze the distribution of singular values or singular vectors of HD i .

Even if one knows the eigen-decomposition of a single G T i G i , it is still hard to get the eigendecomposition of S . The eigenvalues of a matrix A can be viewed as a non-linear function in the A , and hence it is in general hard to derive closed form expressions for the eigenvalues of A + B , given the eigenvalues of A and that of B . One exception is when A and B have the same eigenvector and the eigenvalues of A + B becomes a sum of the eigenvalues of A and B . Recall when G i = E i , Rand-Proj-Spatial recovers Randk -Spatial. Since E T i E i 's all have the same eigenvectors (i.e. same as I d ), the eigenvalues of S = ∑ n i =1 E T i E i are just the sum of diagonal entries of E T i E i 's. Hence, deriving the MSE for Randk -Spatial is not hard compared to the more general case when G T i G i 's can have different eigenvectors.

Since one can also view S = ∑ nk i =1 g i g T i , i.e. the sum of nk rank-one matrices, one might attempt to recursively analyze the eigen-decomposition of ∑ n ′ i =1 g i g T i + g n ′ +1 g T n ′ +1 for n ′ ≤ n . One related problem is eigen-decomposition of a low-rank updated matrix in perturbation analysis: Given the eigen-decomposition of a matrix A , what is the eigen-decomposition of A + V V T , where V is low-rank matrix (or more commonly rank-one)? To compute the eigenvalues of A + V V T directly from that of A , the most effective and widely applied solution is to solve the so-called secular equation, e.g. [59, 60, 61]. While this can be done computationally efficiently, it is hard to get a closed form expression for the eigenvalues of A + V V T from the secular equation.

The previous analysis of SRHT in e.g. [37, 38, 39, 45, 55] is based on asymptotic properties of SRHT, such as the limiting eigen-spectrum, or concentration bounds that bounds the singular values. To analyze the MSE of Rand-Proj-Spatial, however, we need an exact, non-asymptotic analysis of the distribution of SRHT. Concentration bounds does not apply, since computing the pseudo-inverse in Eq. 5 naturally bounds the eigenvalues, and applying concentration bounds will only lead to a loose upper bound on MSE.

B.4 More simulation results on incorporating various degrees of correlation

1

2

3

4

3.2

3.4

3.6

3.8

MSE

n = 5, k = 50

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

1

2

3

4

1.0

1.2

1.4

1.6

1.8

n = 5, k = 100

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

1

2

3

4

0.4

0.6

0.8

1.0

1.2

n = 5, k = 150

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

1

2

3

4

0.0

0.2

0.4

0.6

0.8

n = 5, k = 200

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

2

4

6

8

2.5

2.6

2.7

2.8

2.9

3.0

MSE

n = 11, k = 30

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

2

4

6

8

1.3

1.4

1.5

1.6

1.7

n = 11, k = 50

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

2

4

6

8

0.8

0.9

1.0

1.1

1.2

n = 11, k = 70

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

2

4

6

8

0.5

0.6

0.7

0.8

0.9

n = 11, k = 90

Rand-k

Rand-k-Spatial(Opt)

Rand-Proj-Spatial

C All Proof Details

C.1 Proof of Theorem 4.3

Theorem 4.3 (MSE under Full Correlation) . Consider n clients, each holding the same vector x ∈ R d . Suppose we set T ( λ ) = λ , ¯ β = d k in Eq. 5, and the random linear map G i at each client to be an SRHT matrix. Let δ be the probability that S = ∑ n i =1 G T i G i does not have full rank. Then, for nk ≤ d ,

Proof. All clients have the same vector x 1 = x 2 = · · · = x n = x ∈ R d . Hence, ¯ x = 1 n ∑ n i =1 x i = x , and the decoding scheme is

where S = ∑ n i =1 G T i G i . Let S = U Λ U T be its eigendecomposition. Since S is a real symmetric matrix, U is orthogonal, i.e., U T U = I d = UU T . Also, S † = U Λ † U T , where Λ † is a diagonal matrix, such that

̸

Let δ c be the probability that S has rank c , for c ∈ { k, k +1 , . . . , nk -1 } . Note that δ = ∑ nk -1 c = k δ c . For vector m ∈ R d , we use diag ( m ) ∈ R d × d to denote the matrix whose diagonal entries correspond to the coordinates of m and the rest of the entries are zeros.

Computing ¯ β . First, we compute ¯ β . To ensure that our estimator ̂ x ( Rand-Proj-Spatial(Max) ) is unbiased, we need ¯ β E [ S † S x ] = x . Consequently,

where in ( a ) , m ∈ R d such that

Also, by construction of S , rank ( diag ( m )) ≤ nk . Further, ( b ) follows by symmetry across the d dimensions.

Since δk ≤ ∑ nk -1 c = k δ c c ≤ δ ( nk -1) , there is

Computing the MSE. Next, we use the value of ¯ β in Eq. 20 to compute MSE.

Substituting Eq. 23 in Eq. 22, we get

where the inequality is by Eq 21.

C.2 Comparing against Randk

Next, we compare the MSE of Rand-Proj-Spatial(Max) with the MSE of the baseline Randk analytically in the full-correlation case. Recall that in this case,

Since nk ≤ d , for n ≥ 2 , the above implies when

the MSE of Rand-Proj-Spatial(Max) is always less than that of Randk .

C.3 S has full rank with high probability

We empirically verify that δ ≈ 0 . With d ∈ { 32 , 64 , 128 , . . . , 1024 } and 4 different nk value such that nk ≤ d for each d , we compute rank ( S ) for 10 5 trials for each pair of ( nk, d ) values, and plot the results for all trials. All results are presented in Figure 7. As one can observe from the plots, rank ( S ) = nk with high probability, suggesting δ ≈ 0 .

This implies the MSE of Rand-Proj-Spatial(Max) is

in the full correlation case.

C.4 Proof of Theorem 4.4

Theorem 4.4 (MSE under No Correlation) . Consider n clients, each holding a vector x i ∈ R d , ∀ i ∈ [ n ] . Suppose we set T ≡ 1 , ¯ β = d 2 k in Eq. 5, and the random linear map G i at each client to be

We have

0

20000

40000

60000

80000

100000

Trials

10

20

30

Rank(S)

d = 32

nk=8

nk=16

nk=32

0

20000

40000

60000

80000

100000

Trials

20

40

60

Rank(S)

d = 64

nk=8

nk=16

nk=32

nk=64

0

20000

40000

60000

80000

100000

Trials

25

50

75

100

125

Rank(S)

d = 128

nk=16

nk=32

nk=64

nk=128

0

20000

40000

60000

80000

100000

Trials

50

100

150

200

250

Rank(S)

d = 256

nk=16

nk=32

nk=128

nk=256

0

20000

40000

60000

80000

100000

Trials

0

100

200

300

400

500

Rank(S)

d = 512

nk=16

nk=128

nk=256

nk=512

0

20000

40000

60000

80000

100000

Trials

200

400

600

800

1000

Rank(S)

d = 1024

nk=64

nk=128

nk=512

nk=1024

an SRHT matrix. Then, for nk ≤ d ,

Proof. When the client vectors are all orthogonal to each other, we define the transformation function on the eigenvalue to be T ( λ ) = 1 , ∀ λ ≥ 0 . We show that by considering the above constant T , SRHT becomes the same as rand k . Recall S = ∑ n i =1 G T i G i and let G T G = U Λ U T be its eigendecompostion. Then,

Hence, ( T ( S )) † = I d . And the decoded vector for client i becomes

D i is a diagonal matrix. Also, E T i E i ∈ R d × d is a diagonal matrix, where the i -th entry is 0 or 1. Computing ¯ β . To ensure that ̂ x is an unbiased estimator, from Eq. 24

Computing the MSE.

Note that in Eq. 26

̸

since E [ E T i E i ] = ( k/d ) I d , H T H = d I d and for i = l

which is exactly the same as the MSE of rand k .

C.5 Rand-Proj-Spatial recovers Randk -Spatial (Proof of Lemma 4.1)

Lemma 4.1 (Recovering Randk -Spatial) . Suppose client i generates a subsampling matrix E i = [ e i 1 , . . . , e i k ] ⊤ , where { e j } d j =1 are the canonical basis vectors, and { i 1 , . . . , i k } are sampled from { 1 , . . . , d } without replacement. The encoded vectors are given as ̂ x i = E i x i . Given a function T , ̂ x computed as in Eq. 5 recovers the Randk -Spatial estimator.

Proof. If client i applies E i ∈ R k × d as the random matrix to encode x i in Rand-Proj-Spatial, by Eq. 5, client i 's encoded vector is now

Notice E T i E i is a diagonal matrix, where the j -th diagonal entry is 1 if coordinate j of x i is chosen. Hence, E T i E i x i can be viewed as choosing k coordinates of x i without replacement, which is exactly the same as Randk -Spatial's (and Randk 's) encoding procedure.

Notice ∑ n i =1 E T i E i is also a diagonal matrix, where the j -th diagonal entry is exactly M j , i.e. the number of clients who selects the j -th coordinate as in Randk -Spatial [12]. Furthermore, notice

( T ( ∑ n i =1 E T i E i ) ) † is also a diagonal matrix, where the j -th diagonal entry is 1 T ( M j ) , which recovers the scaling factor used in Randk -Spatial's decoding procedure.

( T ( ∑ n i =1 E T i E i ) ) † and E T i E i x i recover the scaling factor and the encoding procedure of Randk -Spatial, and ¯ β is computed in exactly the same way as Randk -Spatial does, ¯ β will be exactly the same as in Randk -Spatial.

Therefore, ˆ x ( Rand-Proj-Spatial ) i in Eq. 29 with E i as the random matrix at client i recovers ˆ x ( Randk -Spatial ) i . This implies Rand-Proj-Spatial recovers Randk -Spatial in this case.

D Additional Experiment Details and Results

Implementation. All experiments are conducted in a cluster of 20 machines, each of which has 40 cores. The implementation is in Python , mainly based on numpy and scipy . All code used for the experiments can be found at https://github.com/11hifish/Rand-Proj-Spatial .

Data Split. For the non-IID dataset split across the clients, we follow [62] to split Fashion-MNIST , which is used in distributed power iteration and distributed k -means. Specifically, the data is first sorted by labels and then divided into 2 n shards with each shard corresponding to the data of a particular label. Each client is then assigned 2 shards (i.e., data from 2 classes). However, this approach only works for datasets with discrete labels (i.e. datasets used in classification tasks). For the other dataset UJIndoor , which is used in distributed linear regression, we first sort the dataset by the ground truth prediction and then divides the sorted dataset across the clients.

D.1 Additional experimental results

For each one of the three tasks, distributed power iteration, distributed k -means, and distributed linear regression, we provide additional results when the data split is IID across the clients for smaller n, k values in Section D.1.1, and when the data split is Non-IID across the clients in Section D.1.2. For the Non-IID case, we use the same settings (i.e. n, k, d values) as in the IID case.

Discussion. For smaller n, k values compared to the data dimension d , there is less information or less correlation from the client vectors. Hence, both Randk -Spatial and Rand-Proj-Spatial perform better as nk increases. When n, k is small, one might notice Rand-Proj-Spatial performs worse than Randk -Wangni in some settings. However, Randk -Wangni is an adaptive estimator, which optimizes the sampling weights for choosing the client vector coordinates through an iterative process. That means Randk -Wangni requires more computation from the clients, while in practice, the clients often have limited computational power. In contrast, our Rand-Proj-Spatial estimator is non-adaptive and the server does more computation instead of the clients. This is more practical since the central server usually has more computational power than the clients in applications like FL. See the introduction for more discussion.

In most settings, we observe the proposed Rand-Proj-Spatial has a better performance compared to Randk -Spatial. Furthermore, as one would expect, both Randk -Spatial and Rand-Proj-Spatial perform better when the data split is IID across the clients since there is more correlation among the client vectors in the IID case than in the Non-IID case.

D.1.1 More results in the IID case

Distributed Power Iteration and Distribued K -Means. We use the Fashion-MNIST dataset for both distributed power iteration and distributed k -means, which has a dimension of d = 1024 . We consider more settings for distributed power iteration and distributed k -means here: n = 10 , k ∈ { 5 , 25 , 51 } , and n = 50 , k ∈ { 5 , 10 } .

Distributed Power Iteration

0

10

20

30

Iteration No.

0.0

0.5

1.0

1.5

2.0

Squared Error

1e10

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

1.2

1.3

Loss

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Squared Error

1e9

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

1.2

Loss

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.4

0.6

0.8

1.0

1.2

1.4

Squared Error

1e9

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

Loss

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Power Iteration

0

10

20

30

Iteration No.

0.5

1.0

1.5

Squared Error

1e8

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

1.2

Loss

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

2

3

4

5

6

Squared Error

1e7

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

Loss

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Power Iteration

Distributed K Means

0

10

20

30

Iteration No.

0

2000

4000

6000

8000

Squared Error

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.5

1.0

1.5

2.0

Loss

1e7

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0

250

500

750

1000

1250

Squared Error

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

2.0

2.5

3.0

3.5

4.0

4.5

Loss

1e6

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0

100

200

300

400

500

600

Squared Error

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.50

1.75

2.00

2.25

2.50

Loss

1e6

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means

0

10

20

30

Iteration No.

0

500

1000

1500

Squared Error

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

2

3

4

5

Loss

1e6

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0

200

400

600

Squared Error

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.50

1.75

2.00

2.25

2.50

2.75

Loss

1e6

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means

Distributed Linear Regression. We use the UJIndoor dataset distributed linear regression, which has a dimension of d = 512 . We consider more settings here: n = 10 , k ∈ { 5 , 25 } and n = 50 , k ∈ { 1 , 5 } .

Distributed Linear Regression

10

20

30

40

50

Iteration No.

0.0

0.5

1.0

1.5

2.0

Squared Error

1e7

n=10, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

3

4

5

6

7

8

9

Loss

1e4

n=10, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

0.2

0.4

0.6

0.8

1.0

Squared Error

1e6

n=10, k=25, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.0

2.2

2.4

2.6

2.8

3.0

Loss

1e4

n=10, k=25, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2

4

6

8

Squared Error

1e6

n=50, k=1, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

3.0

3.5

4.0

4.5

Loss

1e4

n=50, k=1, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

0.5

1.0

1.5

2.0

Squared Error

1e6

n=50, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.00

2.25

2.50

2.75

3.00

3.25

Loss

1e4

n=50, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Linear Regression

D.1.2 Additional results in the Non-IID case

In this section, we report results when the dataset split across the clients are Non-IID, using the same datasets as in the IID case. We choose exactly the same set of n, k values as in the IID case.

Distributed Power Iteration and Distributed K -Means. Again, both distributed power iteration and distributed k -means use the Fashion-MNIST dataset, with a dimension d = 1024 . We consider the following settings for both tasks: n = 10 , k ∈ { 5 , 25 , 51 , 102 } and n = 50 , k ∈ { 5 , 10 , 20 } .

Distributed Power Iteration (Non-IID)

0

10

20

30

Iteration No.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Squared Error

1e10

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

1.2

1.3

Loss

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1

2

3

4

Squared Error

1e9

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

1.2

Loss

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.50

0.75

1.00

1.25

1.50

1.75

Squared Error

1e9

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

Loss

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

4

6

8

Squared Error

1e8

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.65

0.70

0.75

0.80

0.85

0.90

0.95

Loss

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Power Iteration (Non-IID)

0

10

20

30

Iteration No.

0.5

1.0

1.5

2.0

Squared Error

1e8

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

1.2

Loss

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

2

4

6

8

Squared Error

1e7

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.8

0.9

1.0

1.1

Loss

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Power Iteration (Non-IID)

0

10

20

30

Iteration No.

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Squared Error

1e7

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.7

0.8

0.9

1.0

Loss

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Power Iteration (Non-IID)

Distributed Linear Regression. Again, we use the UJIndoor dataset for distributed linear regression, which has a dimension d = 512 . We consider the following settings: n = 10 , k ∈ { 5 , 25 , 50 } and n = 50 , k ∈ { 1 , 5 , 50 } .

0

10

20

30

Iteration No.

0

2000

4000

6000

8000

Squared Error

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0.5

1.0

1.5

2.0

Loss

1e7

n=10, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

0

200

400

600

800

1000

1200

Squared Error

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

2.00

2.25

2.50

2.75

3.00

3.25

Loss

1e6

n=10, k=25, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means (Non-IID)

0

10

20

30

Iteration No.

0

100

200

300

400

500

600

Squared Error

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.5

2.0

2.5

3.0

Loss

1e6

n=10, k=51, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

50

100

150

200

250

Squared Error

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.0

1.2

1.4

1.6

1.8

Loss

1e6

n=10, k=102, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means (Non-IID)

0

10

20

30

Iteration No.

0

250

500

750

1000

1250

Squared Error

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

2

3

4

5

Loss

1e6

n=50, k=5, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

100

200

300

400

500

Squared Error

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.50

1.75

2.00

2.25

2.50

2.75

Loss

1e6

n=50, k=10, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means (Non-IID)

0

10

20

30

Iteration No.

50

100

150

200

250

Squared Error

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

0

10

20

30

Iteration No.

1.2

1.4

1.6

1.8

2.0

Loss

1e6

n=50, k=20, d=1024

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed K Means (Non-IID)

10

20

30

40

50

Iteration No.

0

1

2

3

4

Squared Error

1e8

n=10, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

3.0

3.5

4.0

4.5

5.0

5.5

Loss

1e4

n=10, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

1

2

3

4

5

Squared Error

1e7

n=10, k=25, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.0

2.5

3.0

3.5

Loss

1e4

n=10, k=25, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Linear Regression (Non-IID)

10

20

30

40

50

Iteration No.

0.50

0.75

1.00

1.25

1.50

Squared Error

1e7

n=10, k=50, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.0

2.2

2.4

2.6

Loss

1e4

n=10, k=50, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Linear Regression (Non-IID)

10

20

30

40

50

Iteration No.

0

1

2

3

Squared Error

1e8

n=50, k=1, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

3.0

3.5

4.0

4.5

5.0

Loss

1e4

n=50, k=1, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2

4

6

8

Squared Error

1e7

n=50, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.00

2.25

2.50

2.75

3.00

3.25

3.50

Loss

1e4

n=50, k=5, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Linear Regression (Non-IID)

10

20

30

40

50

Iteration No.

1.0

1.5

2.0

2.5

Squared Error

1e7

n=50, k=10, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

10

20

30

40

50

Iteration No.

2.0

2.2

2.4

2.6

2.8

Loss

1e4

n=50, k=10, d=512

Rand-k

Rand-k-Spatial(Avg)

Rand-k(Wangni)

Induced Compressor

Rand-Proj-Spatial(Avg)

Distributed Linear Regression (Non-IID)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a compression algorithm that optimizes the accuracy in a setting where each client can send $k+O(1)\\ll d$ values to the server. The algorithm leverages random projections and proposes a way to make the resulting estimate unbiased, which is valuable for DME.\n\nSTRENGTHS:\n+ DME is an important and well-studied problem, so advances are welcomed.\n\n+ Leveraging correlations between clients' vectors leads to significant improvement in accuracy in some cases.\n\n+ The setting of $n\\cdot k \\ll d$ is challenging and interesting.\n\nWEAKNESSES:\n- The decoding time at the server is not presented, may be prohibitively high in some cases, and must be discussed upfront.\n\n- The authors compare only with Rand-$k$ and Rand-$k$-Spatial. There are other solutions that leverage correlations between clients' vectors (e.g., New Bounds For Distributed Mean Estimation and Variance Reduction, ICLR 2021) and it is unclear to me whether the Rand-$*$ approaches are better or what are the tradeoffs. The authors do not cite the paper or compare with such approaches even qualitatively.\n\n- It is unclear to me that these approaches, in which each client send $k$ floats are better than quantization methods that allow sending, e.g., $32k$ values with one bit each. One example that can use as a comparison point is [33].\n\n- The authors discuss a case where $n\\cdot k \\ll d$; however, probably due to time complexity, even running it for $d \\gg 1$ seems unlikely. The evaluation only shows $d\\le 1024$ and no runtime numbers are provided.\n\nQUESTIONS:\n* Do you view Rand-Proj-Spatial as being orthogonal to quantization techniques? For example, what would be the impact of quantizing the $k$ values sent by each worker?\n\n* Am I correct to understand that the server's decoding requires $\\Theta(d^2\\cdot k\\cdot n)$ time? If not, what is it?\n\n* You suggest that each client $i$ would use a different projection $G_i$. This may be important for the error, but also requires the server to compute the pinverse of each client's message separately. Instead, consider using a single projection $G$ that all clients use, and thus the server could compute a single pinverse after summing the messages. How would the error bounds change in such an implementation?\n\nLIMITATIONS:\nn/a.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the distributed mean estimation (DME) problem. In particular, the paper proposes a new DME technique called Rand-Proj-Spatial. In  Rand-Proj-Spatial, each client uses SRHT for dimensionality reduction and sends the transformed lower-dimensional vector to the server. The server then recovers the mean estimate by computing a formula derived from an optimization problem designed to minimize the MSE given the client transforms and considering possible correlations among client vectors. Rand-Proj-Spatial is also unbiased, which is desired in the context of DME.\n\nThe main contribution of Rand-Proj-Spatial is that it both improves with increased correlation among client vectors and utilizes SRHT for dimensionality reduction instead of subsampling (like rand-k), achieving lower MSE than previous rand-k-based DME techniques.\n\nSTRENGTHS:\n1.\tThe new DME technique improves upon previous rand-k-based DME techniques.\n2.\tThe reconstruction technique is interesting and not symmetric, putting the computational burden on the server.\n3.\tRand-Proj-Spatial leverages possible correlations among client vectors which can be expected and useful in some distributed scenarios.\n\nWEAKNESSES:\n1.\tThe worst-case MSE of Rand-Proj-Spatial appears to be $O(d/n)$. Namely, unless $k = \\Theta(d)$, the MSE of Rand-Proj-Spatial grows linearly with the dimension of the problem and may cause an asymptotic increase in the number of required optimization rounds in many (S)GD-based scenarios (e.g., neural network training). \n\n2.\tNo comparison to strong DME baselines, e.g., [1][2][3]: In the paper’s evaluation, the authors consider scenarios with d=1024,  21-51 clients, and k=4-40. These regimes can be readily compared to, e.g., [1][2][3] with 1-2 bits per coordinate. Moreover,  some of these support sub-bit regimes as well.\n    \n3.\tInsufficient evaluation: the evaluation considers low-dimensional (mostly convex) scenarios. It would be more convincing to demonstrate the advantage of Rand-Proj-Spatial over, e.g., neural networks with sufficient dimension and demonstrate that Rand-Proj-Spatial results in better performance than existing DME techniques considering all aspects of the algorithm (i.e., error-to-bandwidth tradeoff and computational overhead). \n\n4.\tThe encoding and decoding time of Rand-Proj-Spatial are not evaluated and compared to previous DME techniques. These times are of major importance in practical scenarios.\n\n[1] Suresh, Ananda Theertha, et al. \"Distributed mean estimation with limited communication.\" International conference on machine learning. PMLR, 2017.\n\n[2] Davies, Peter, et al. \"New Bounds For Distributed Mean Estimation and Variance Reduction.\" International Conference on Learning Representations, 2021.\n\n[3] Vargaftik, Shay, et al. \"Eden: Communication-efficient and robust distributed mean estimation for federated learning.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\nCan the authors address the concerns pointed out in weaknesses (1)-(4)?\n\nLIMITATIONS:\nThe authors point out a possible direction for future work. I would suggest adding disclaimers about the computational overhead and the asymptotic error-bandwidth tradeoffs compared to existing DME techniques.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the problem of distributed mean estimation, wherein each node in a set of distributed nodes contains a vector, and the goal of the parameter server is to estimate the mean of those vectors. Unlike some other works, no distributional assumption is assumed over the vectors, and the error metric is the mean squared error between the true mean and the estimated mean. Each node sends a compressed version of its vector to the parameter server, and this is the sole source of estimation error. Any randomness in the estimation error arises from the randomness of the compression algorithm used at the nodes. This setup is motivated from distributed optimization / federated learning setups where the mean of the gradients is estimated at the parameter server at every iteration of the optimization algorithm. Since modern machine learning tasks deal with very high dimensional models and gradients, it becomes necessary to compress them before sending it to the parameter server, so as not to overwhelm the communication requirements between nodes and parameter server (which is often a bottleneck).\n\nThis particular work focuses on the effect of correlation between the vectors at different nodes, and considers exploiting this correlation information to design better compressors that achieve a lower mean squared error than correlation-agnostic compressors. Prior work (Jhunjhunwala et. al. (2021)) proposed the **rand-$k$ spatial family** of mean estimators that used correlation information. They did so by first compressing the vector at each node using a rand-$k$ compressor. Rand-$k$ essentially subsamples $k$ coordinates out of a $d$-dimensional vector to get a compressed $k$-dimensional vector, zeroing out the remaining coordinated and yielding a compression factor of $\\frac{d}{k}$. These compressed vectors are transmitted to the parameter server, which then decodes the mean by appropriately scaling the received vectors that ensures an unbiased estimate of the mean. When correlation information is available, the server can decode the received vectors with a different scaling mechanism, yielding a smaller estimation error.\n\nThe primary contribution of this work is a modification of the encoding strategy to incorporate random projections in the encoding step at the clients. Instead of directly applying rand-$k$ to the vector, the vector is first projected on a random subspace (more specifically, the range space of a subsampled randomized Hadamard matrix) and then subsequently rand-$k$ compressor is applied. \n\nFollowing this idea, the authors in this work propose a correlation-aware encoding and decoding strategy for distributed mean estimation under communication constraints, analytically derive upper bounds on the estimation error of their proposed scheme, and finally, numerically evaluate the performance of their scheme relative to existing benchmarks. Please let me know if I have missed anything in my understanding of the contributions of the paper. I would be more than happy to rectify any misunderstandings on my end.\n\nSTRENGTHS:\nThe following are the primary contributions of the paper:\n\n1. The authors propose the *rand-proj-spatial* family of mean estimators as a generalization of the previously proposed *rand-$k$-spatial* estimators, that uses subsampled randomized Hadamard transform to first project the vectors onto a random subspace and subsequently apply a rand-$k$ compressor with appropriate decoding at the server.\n\n2. Several numerical experiments on distributed power iteration, distributed $k$-means, and distributed linear regression have been performed to demonstrate the superiority of the proposed strategy over existing correlation-aware and correlation agnostic strategies.\n\nOne of the interesting observations made by the authors is Appendix A.1, which states that applying random projection + rand-$k$ directly to the vectors may not yield any benefit. In other words, if we apply the $\\frac{d}{k}$ scaling individually for each vector with the hope of obtaining an unbiased estimate for each nodes' vector at the server, the expected estimation error is the same as that obtained by rand-$k$ compression without the projection. However, lower MSE can be obtained if the vectors received from each of the nodes are jointly decoded as is done in the *rand-proj-spatial* estimator as proposed in eq. (5) of the paper. At a first glance, this seems a little counter-intuitive since (as the authors point out in the introduction), the motivation to take random projections stems from the idea that random projections equalize the coordinate magnitudes. \n\nHowever, a careful study reveals that when the error metric is the $\\ell_2$-estimation error, random projections do not really help, because the $\\ell_2$-norm of the vectors approximately remains preserved when an orthonormal projection is applied. However, I do believe the benefit will be apparent if other error metrics are considered, such as the $\\ell_{\\infty}$ estimation error (more remarks on why this might be the case in the *limitations* section).\n\nWEAKNESSES:\nI have one critical concern which is the fact that the authors have titled their paper *Correlation Aware Distributed Vector Mean Estimation* as a very generic and broad approach, whereas the contributions of the paper are focused on exploiting correlation for communication compression for a very specific class of rand-$k$ compression strategies, and it is not immediately obvious how this correlation-aware compression strategy can be extended to other classes of compressor such as top-$k$ (instead of rand-$k$), which is a biased compressor -- but is very popular and has been extensively used for distributed optimization / federated learning applications. For example, the paper by Suresh et. al. (2022) (reference [10]) also exploits correlation for quantization purposes. Please note that this is not a drawback or a comment on the technical contributions of the paper. However, the title can be slightly misleading in a sense since the paper does not focus on the benefits of correlation beyond rand-$k$ compression. For instance, correlation can be exploited for client selection, or for designing correlation-aware private mean estimation algorithms. It is my personal opinion and would be highly appreciated if the authors chose a more descriptive title.\n\nIn addition to this, the *rand-proj-spatial* family of estimators proposed in this paper is also a very specific design, and there is no indication if it is the best strategy. In other words, given that our underlying task is communication compression, is *rand-proj-spatial* the best way to exploit correlation? I understand this is not an easy question to answer since lower bounds on the estimation error have not been derived in this paper. However, when authors introduce the optimization problem in eq. (4), it seemed to me that this optimization problem was formulated with a solution in mind -- a solution which is analogous (and a generalization) to the prior work on *rand-$k$ spatial* family of estimators. A natural question to ask here is why the problem was not initially formulated more generally as follows:\n\n$\\hat{\\mathbf{x}} = argmin_{\\mathbf{W}, \\mathbf{x}} \\mathbb{E}\\left\\lVert \\overline{\\mathbf{x}} - \\sum_{i=1}^{n}\\mathbf{W}_i\\mathbf{G}_i\\mathbf{x}_i \\right\\rVert_2^2$\n\nwhere $\\overline{\\mathbf{x}} = \\frac{1}{n}\\sum_{i=1}^{n}\\mathbf{x}_i$, and $\\mathbf{W} = (\\mathbf{W}_1, \\ldots, \\mathbf{W}_n)$ are the individual decoding matrices over which optimization is done as well. Furthermore, one can solve this optimization problem subject to the unbiasedness constraint and potentially come up with a better decoding strategy than the one proposed in this paper? It would be appreciated if the authors discussed how (or under what assumptions) the optimization problem of eq. (4) relates to the this general formulation above, and what are the challenges that prevent one from solving the above general optimization problem directly. Moreover, in the above formulation, the encoding strategy is still the same, i.e., rand-proj + rand-$k$ -- one could also replace the encoding function by a more general class of (potentially biased) contractive compressors (such as top-$k$) and come up with a more very holistic approach to the problem of *correlation aware compression for distributed mean estimation* (which would justify this title).\n\nPlease note once again that this is not a weakness on the technical contributions of the paper as such -- just a suggestion that I genuinely believe would help highlight the contributions of the paper in a better context.\n\nQUESTIONS:\nI have a few questions and suggestions for the authors, and would highly appreciate if they would take them into consideration. I would be more than happy to re-evaluate my review contingent on their response. The following are some of my concerns:\n\nThe idea of random projection + compression like rand-$k$ is not novel by itself (although it's implications in the presence of correlation as explored in this paper are, to the best of my knowledge). Accordingly, there are some very relevant references missing from the related works section and it would be highly appreciated if the authors included those:\n\n1. R. Saha, M. Pilanci and A. J. Goldsmith, \"Efficient Randomized Subspace Embeddings for Distributed Optimization Under a Communication Budget,\" in IEEE Journal on Selected Areas in Information Theory, vol. 3, no. 2, pp. 183-196, June 2022, doi: 10.1109/JSAIT.2022.3198412 (This work adopts a unified approach for studying the effects of random transforms on compressors -- it poses randomized Hadamard transform as a computationally efficient relaxation of Democratic (Kashin) representations, and also studies rand-$k$ compression (among others) with both kashin + near-kashin -- eventually showing that they attain information-theoretically optimal convergence rates for optimization).\n\n2. Mher Safaryan and others, Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor, Information and Inference: A Journal of the IMA, Volume 11, Issue 2, June 2022, Pages 557–580, https://doi.org/10.1093/imaiai/iaab006 (This work studies a general class of contractive compressors which includes random-projections based compressors -- they focus on Kashin compression and study different variants of accelerated distributed optimization algorithms).\n\nThese properties are not specific to subsampled Randomized Hadamard transforms, -- but extend broadly to the class of randomized matrices that belong to Johnson-Lindenstrauss embeddings and/or follow Restricted Isometry Property (RIP). These results might perhaps be helpful to the authors in deriving bounds on MSE for *rand-proj-spatial* for other classes of matrices (such as random Haar orthonormal). Once again, this is not a critical drawback, but a suggestion that will help place the work better in context with respect to existing literature.\n\nLIMITATIONS:\nI have some other minor concerns and would be happy if the authors addressed / discussed them:\n\n1. The authors mention on line $138$ *\"Each client also sends a random seed to the server, which conveys the subspace information, and can usually be communicated using a negligible amount of bits.\"* -- Is this really negligible? Can $\\mathbf{G}$ be conveyed between the nodes and server with less than $d$ bits of information? Since each node shares an independently sampled random matrix, can correlation help improve this shared randomness generation?\n\n2. The statement of Theorem 4.3 is a high probability result but the equation (9) holds in expectation -- this is slightly weird. It would be better if the authors stated the result purely as a high probability statement or purely in terms of expectation (the latter might be simpler since the failure probability is $o(1)$). If this is not possible, they should atleast explicitly specify in the statement of the theorem which random variables is the expectation computed over, and which random variables is the high probability bound over? Moreover, $o(1)$ is not explicitly defined -- is it with respect to dimension / number of nodes -- please specify?\n\nIt would also be highly if the authors had a separate limitations sections that discuss these and other concerns raised in *weaknesses* and *questions*, which are my primary concerns. I would be more than happy to re-evaluate my assessment if these concerns are addressed satisfactorily.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
In distributed learning, computing the mean of the vectors sent by the clients is an important subtask. Motivated by this, the distributed mean estimation problem is studied in this paper. The two main techniques commonly used for these problems are Quantization and sparsification. Rand-K was one prominent sparsification method used earlier. In [9], this was generalized to Rand-k-Spatial to exploit the spatial correlations of the data using the server-side decoding procedure. A more general encoding scheme, Rand-Proj-Spatial is proposed in this paper which utilizes the cross-client correlation information. This uses Subsampled Randomized Hadamard Transform as random linear maps. They prove that this improves mean estimation under some assumptions and some experiments supporting their claim.

STRENGTHS:
The paper is well presented. The problem generalizes the existing works and the results look interesting.

WEAKNESSES:
Their techniques are not adaptive to the client vectors. More motivation on why studying why correlation information of clients is practical will be interesting.

QUESTIONS:
Is Rand-Proj-Spatial optimal or close to optimal, given the correlation information? Some intuition will be good.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the problem of communication-efficient distributed vector mean estimation and proposes a new estimator called Rand-Proj-Spatial that improves upon existing techniques. The paper highlights the challenges of distributed optimization and Federated Learning, where communication cost can be a bottleneck, and explains how sparsification techniques can reduce this cost. The authors then introduce the Rand-k sparsification technique and show how it can be used to reduce communication cost in distributed vector mean estimation. They also propose the Rand-Proj-Spatial estimator, which combines Rand-k sparsification with a Subsampled Randomized Hadamard Transform to improve upon the performance of existing techniques. The paper concludes with experimental results that demonstrate the effectiveness of the Rand-Proj-Spatial estimator in various scenarios. Overall, the contributions of this paper include a new estimator for distributed vector mean estimation that is more communication-efficient than existing techniques, as well as insights into the challenges of communication-efficient distributed optimization and Federated Learning.\n\nSTRENGTHS:\nIn terms of originality, the paper proposes a new estimator called Rand-Proj-Spatial that combines existing techniques in a novel way to improve upon the performance of distributed vector mean estimation. The use of Subsampled Randomized Hadamard Transform in the encoding-decoding procedure is a creative combination of existing ideas that has not been explored in this context before. \n\nIn terms of quality, the paper is well-written and clearly explains the challenges of communication-efficient distributed vector mean estimation and the proposed solutions. The authors provide rigorous theoretical analysis and experimental results to support their claims, which adds to the credibility of their contributions.\n\nIn terms of clarity, the paper is well-organized and easy to follow. The authors provide clear explanations of the technical concepts and use visual aids to help readers understand the proposed techniques. \n\nIn terms of significance, the paper addresses an important problem in the field of distributed optimization and Federated Learning. The proposed Rand-Proj-Spatial estimator has the potential to significantly reduce communication cost in these settings, which can have practical implications for real-world applications.\n\nWEAKNESSES:\n1. While the paper compare the proposed Rand-Proj-Spatial estimator with Rand-k-Spatial and other sparsification techniques, it would be beneficial to compare the proposed estimator with other state-of-the-art techniques for distributed vector mean estimation.\n\n2. While the paper briefly mentions that the Subsampled Randomized Hadamard Transform can be computed efficiently, it would be helpful to provide a more detailed analysis of the computational complexity of the proposed estimator.\n\nQUESTIONS:\n1. What is the key difficulty/novelty in the design and analysis?\n\n2. Can you provide more detailed analysis of the trade-off between communication cost and estimation accuracy for the proposed estimator and other sparsification techniques?\n\nLIMITATIONS:\nNot.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}