Title: One-Pass Distribution Sketch for Measuring Data Heterogeneity in Federated Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Distribution sketch

federated learning
Abstract: Federated learning (FL) is a machine learning paradigm where multiple client devices train models collaboratively without data exchange. Data heterogeneity problem is naturally inherited in FL since data in different clients follow diverse distributions.  To mitigate the negative influence of data heterogeneity, we need to start by measuring it across clients. However, the efficient measurement between distributions is a challenging problem, especially in high dimensionality. In this paper, we propose a one-pass distribution sketch to represent the client data distribution. Our sketching algorithm only requires a single pass of the client data, which is efficient in terms of time and memory. Moreover, we show in both theory and practice that the distance between two distribution sketches represents the divergence between their corresponding distributions. Furthermore, we demonstrate with extensive experiments that our distribution sketch improves the client selection in the FL training. We also showcase that our distribution sketch is an efficient solution to the cold start problem in FL for new clients with unlabeled data.

One-Pass Distribution Sketch for Measuring Data Heterogeneity in Federated Learning

Abstract

Federated learning (FL) is a machine learning paradigm where multiple client devices train models collaboratively without data exchange. Data heterogeneity problem is naturally inherited in FL since data in different clients follow diverse distributions. To mitigate the negative influence of data heterogeneity, we need to start by measuring it across clients. However, the efficient measurement between distributions is a challenging problem, especially in high dimensionality. In this paper, we propose a one-pass distribution sketch to represent the client data distribution. Our sketching algorithm only requires a single pass of the client data, which is efficient in terms of time and memory. Moreover, we show in both theory and practice that the distance between two distribution sketches represents the divergence between their corresponding distributions. Furthermore, we demonstrate with extensive experiments that our distribution sketch improves the client selection in the FL training. We also showcase that our distribution sketch is an efficient solution to the cold start problem in FL for new clients with unlabeled data.

1 Introduction

Federated learning (FL) has recently become the go-to solution for challenging on-device applications such as typing prediction [1], machine translation [2, 3] and fraud detection [4, 5]. Initially proposed by McMahan et. al. [6], the FL setting defines a learning problem where multiple clients collaborate to construct a global model from local client data. Unlike in centralized learning, where clients can freely exchange data, FL requires that private data never leave the client device. The FL framework is uniquely positioned to comply with recent data privacy legislation [7, 8, 9], and it has correspondingly seen a surge of interest in machine learning (ML) research and deployment [10, 11, 12].

Data Heterogeneity in FL. Clients' local datasets may be drawn from significantly different statistical distributions [6, 13, 14, 12, 15, 16, 17, 18, 19, 20, 21, 22]. For instance, in the typing prediction task, each client is a cellphone user. Differences in user interest, language, and geography will produce extremely diverse word distributions. Data heterogeneity is a severe problem for FL because the models trained on each client may have many different behaviors. It is challenging to aggregate such different client models into a coherent and performant global model.

Zhaozhuo Xu ∗

Anshumali Shrivastava

Benefits of Measuring Data Heterogeneity. We currently lack good methods to measure data heterogeneity - but we expect that such methods could substantially improve the efficiency and quality of FL [23, 24, 25]. We will use the term global distribution to denote the combination of client data distributions. If a client's data distribution is close to the global distribution, we can obtain a better global model by focusing training resources on this client (improving efficiency). Data heterogeneity measures can also help implement an essential personalization procedure of FL [26, 27, 28]. In personalized FL, each client increases local predictive performance by finetuning the global model on their own data. This results in a personalized model for each client. Now, suppose that a new client arrives with unlabeled data. We can search for a client who has data that is similar to the new client and use its personalized model. To summarize, data heterogeneity measurement allows us to quantify the differences between clients in FL to improve FL algorithms and systems.

Challenges in Measuring Data Heterogeneity in FL. The data heterogeneity measurement can be formulated as a statistical problem where we wish to quantify the degree to which probability distributions differ. However, this divergence estimation can be difficult in high-dimensional space. Existing approaches, including earth mover distance [29, 30] or Fréchet distance [31, 32] are computationally expensive. Moreover, many classical algorithms fail for complex high-dimensional input distributions; learning is much more effective [32]. However, learning introduces extra computational overheads. Since the clients may be mobile devices with limited hardware resources in the FL setting, expensive computation will introduce a significant efficiency issue in practical deployment. Further, to keep with the privacy advantage of FL, data heterogeneity measurements must to private.

1.1 Our Proposal: One-Pass Distribution Sketch with Hashing

In this paper, we propose an efficient approach to represent client data distributions in FL. Motivated by the recent success of locality sensitive hashing in kernel density estimation, we develop a one-pass distribution sketch for measuring data heterogeneity on FL clients. We make the following specific contributions.

We propose an effective and efficient distribution sketching method that requires only one pass over the client data. We represent the data distribution with a matrix of counts, and we introduce a data heterogeneity measure which is the Euclidean distance between clients' sketch matrices.

We develop an efficient sketch-based sampling strategy for federated optimization. Given sketches for each FL client, we find a global distribution sketch by merging all client sketches. We propose a form of client importance sampling where, in each training round, we preferentially select clients that have a low distance to the global sketch. We observe a significant acceleration in FL, with faster convergence and better generalization than uniform sampling.

We develop an efficient cold start approach for personalized FL where a new client (with unlabeled data) selects the most appropriate model from the personalized finetuned models of other clients. To do this, we compare the distribution sketch of the new client with each existing client. We perform a nearest neighbor search over clients under the sketch distance to identify the client that has similar data distribution. Our cold-start approach provides a simple yet effective way to find a suitable model for new clients.

2 Preliminary

Locality Sensitive Hashing (LSH): LSH functions serve as a key ingredient in our distribution sketch. LSH is a randomized function family [33, 34, 35, 36, 37]. Each function in this family maps an input vector into a hash value, usually binary code or integer. If two vectors are close to each other, with a high probability they will share the same hash value. Formally, we define the LSH function as:

Definition 2.1 ( ( D,cD,p 1 , p 2 ) -sensitive hash family). We define a function family H to be ( D,cD,p 1 , p 2 ) -sensitive with respect to distance function d : R d × R d → R if for any two vector x, y ∈ R d , any h ∈ H chosen uniformly at random satisfies:

If d ( x, y ) ≤ D then Pr [ h ( x ) = h ( y )] ≥ p 1

If d ( x, y ) ≥ cD then Pr [ h ( x ) = h ( y )] ≤ p 2

Here D ∈ R and c > 1 . We denote Pr [ h ( x ) = h ( y )] as the collision probability of vector x and y . Here the term 'collision' means x and y have the same hash value. In practice, we will use LSH functions where the collision probability between x and y is a monotonically increasing function with respect to a distance measure d : R d × R d → R . Formally, we denote this relationship as:

We introduce the realization of the LSH function in Section A of the Appendix.

Repeated Array of Count Estimators (RACE): RACE is an efficient Kernel Density Estimator (KDE) with applications in outlier detection [38, 39], graph compression [39], genetic profiling [40] and continue learning [41]. RACE builds a two-dimensional matrix A ∈ R R × B using R LSH functions (see Definition 2.1), each LSH function h i maps a vector in R d to an integer in [ B ] . Given a dataset D ⊂ R d , for every x ∈ D , RACE maps it to R different hash values, denoted as { h i ( x ) | i ∈ [ R ] } . Then we increment entry A i,h i ( x ) with 1 for every i ∈ [ R ] . Following this manner, RACE maintains a matrix where each column can be regarded as a histogram of the dataset D . As a result, we can use matrix A for KDE and its downstream tasks.

3 One-Pass Distribution Sketch

In this section, we introduce the one-pass distribution sketch for measuring data heterogeneity in FL.

3.1 One-Pass Sketch Construction

Intuition: We observe that the RACE matrix can be used as a representation for a data distribution. Given a dataset D ⊂ R d , if we build a RACE matrix S by inserting every element in D , we build a histogram of data samples with hashing-based projection. Next, if we normalize the counts in the RACE matrix S with the dataset size |D| , we will get a representation of D .

Following this intuition, we introduce our distribution sketch algorithm in Algorithm 1. Our distribution sketch S is a two-dimensional matrix with R rows and B columns. Given a dataset D , we first initialize a zero matrix S ∈ R R × B and R LSH functions { h 1 , · · · , h R } . Here, each hash function h i generates an integer hash value within [ B ] . We suggest R to be O ( √ |D| ) . The choice of B depends on the LSH functions we use. Next, for every x ∈ D , we compute R hash values { h 1 ( x ) , · · · , h R ( x ) } . Finally, we increment S i,h i ( x ) for every i ∈ [ R ] . In practice, for vision tasks with dense feature vectors, we use SRP hash (See Definition A.1) as our LSH function. For language modeling tasks, we tokenize every sentence in the dataset into a bag of

Algorithm 1 One-Pass Distribution Sketch

words or characters [42]. Next, a sentence can be formed as a binary vector with a nonzero index representing a token. In this case, we can use MinHash (see Definition A.2) as an LSH function.

Complexity Analysis: Given a dataset with n vectors, the running time of Algorithm 1 is O ( nRd + RB ) . Since the number of LSH functions R and hash range B are usually smaller than the dataset size n , Algorithm 1 is a linear passing over the dataset. Moreover, the space complexity of Algorithm 1 is O ( RB ) , which is a much smaller than the memory cost of storing the dataset.

3.2 Theoretical Analysis

In this section, we describe how to measure the data heterogeneity using the one-pass distribution sketch in Algorithm 1. We will estimate the total variation distance (TVD), which roughly measures the largest pointwise difference between two distributions over their support.

Definition 3.1. Given two probability measures P and Q on a space (Ω , F ) ,

If P and Q have density functions p and q , this expression is equivalent to the following ℓ 1 norm.

Our goal is to describe a method for consistent estimation of the TVD that is efficient to implement in the distributed and private FL setting. At a high level, our strategy is as follows.

Given a dataset D drawn from a distribution with probability density function p ( x ) , construct a sketch-based consistent estimator S D ( x ) for the kernel density estimate (KDE) over D .

Argue that S D is a consistent estimator of p , because the KDE is a consistent estimator of p and S D is a consistent estimator of the KDE.

Use S D as a plug-in estimator for p in the TVD integral (Equation 3).

Assumptions: We consider probability measures on R d that have a distribution function. We assume that the probability densities p and q are continuous almost everywhere, so that the KDE → p in step 2 [43]. We also assume that the support of P and Q is a bounded subset of R d , so that our kernel estimator has the properties described in Section 3.2.1. Finally, our results are for convergence in probability rather than the (stronger) almost sure uniform convergence.

Notation: In a slight abuse of notation, we will use O ( · ) to denote both stochastic bounds and the more typical (deterministic) asymptotic bounds. More concretely, X N = O ( f ( N )) means that for any ϵ > 0 , we can find constants c and N 0 such that for all N > N 0 , Pr[ | X | ≥ cf ( N )] ≤ ϵ . In the deterministic case, this still holds with ϵ = 0 . We will assume two probability measures P, Q having densities p, q and two datasets D p ∼ p and D q ∼ q . We denote the sketch of D p with S p and D q with S q . We write the distribution sketch S generated by Algorithm 1 as a functional S ( x ) . We explicitly define this functional in Algorithm 4, Section D.

3.2.1 Consistent Estimation with Sketches

This section shows that our sketches provide a consistent estimator of the data-generating distribution. We introduce the distribution sketch's query algorithm in Algorithm 4 (See Section D in the appendix). For a sketch constructed via Algorithm 1, given a query q ∈ R d , we use the same LSH functions in construction sketch and get R hash values. The i th hash value corresponds to a location in the i th row. Next, we take the median over the average counts in the hashed locations. We show that the median-of-means count S ( q ) converges in probability to the kernel density of the LSH kernel. We begin with the definition of an LSH kernel.

Definition 3.2 (LSH Kernel) . We define a kernel k ( x, y ) to be an LSH kernel if for every x, y , k ( x, y ) represents the LSH collision probability (See Eq. (1) ) of x and y , which is monotonically decreasing with distance d ( x, y ) ≥ 0 . An LSH kernel k ( x, y ) is a positive semi-definite kernel.

Next, we have the following statement.

Theorem 3.3 (Informal version of Theorem D.1) . Let P ( x ) denote a probability density function. Let D ∼ iid P ( x ) denote a dataset. Let k ( x, y ) be an LSH kernel (see Definition 3.2). Let S define the function implemented by Algorithm 4. We show that

with convergence rate O ( √ log R/R ) .

Theorem 3.3 shows the consistent estimation of the KDE as R → ∞ , which is itself a consistent estimator of the distribution under some assumptions about k ( x, y ) and P . The convergence of the KDE to the distribution as N → ∞ is a well-studied problem under various conditions [44]. However, for the weak form of convergence which we consider here, it is sufficient for the kernel to integrate to 1, for the kernel bandwidth to decay with N , and for the bandwidth to be bounded above by N 1 /d [45].

We argue that these conditions hold for the LSH kernels described by [46]. The bandwidth requirements are easy to satisfy because we can explicitly set the LSH kernel bandwidth by adjusting the

hash function parameters - for example, by setting the number of bits in an SRP hash. The other requirement can be tricky to satisfy for LSH kernels, such as the p -stable LSH probability, which is not absolutely integrable over R d . For this reason, some LSH kernels cannot be normalized to have a unit integral. However, this is not a problem because we assume that P and Q have bounded support in R d . Note that this type of input domain truncation is a standard component of LSH design and analysis [47]. Under these conditions, Theorem 3.3 implies consistent estimation of P ( x ) by S ( x ) provided that R is allowed to weakly grow with N (i.e. R = ω (1) ).

Private Sketches: When the sketch is required to preserve ϵ -differential privacy, we must add independent Laplace noise to the sketch so that each counter is private [48]. This constrains our choice of R because the noise is distributed Z iid ∼ Laplace( Rϵ -1 ) .

Theorem 3.4 (Informal version of Theorem D.2) . Let S be an ϵ -differentially private distribution sketch of a dataset D = { x 1 , ...x N } with D ∼ iid P ( x ) and let k ( x, y ) be an LSH kernel (see

Definition 3.2). Let S define the function implemented by Algorithm 4. Then

with convergence rate O ( √ log R/R + √ R log R/ ( Nϵ ) when R = ω (1) , e.g. R = log N .

In exchange for privacy, we pay a price in the convergence rate, which now has an additional ϵ -1 term. While this dependence can be improved to ϵ -1 / 2 (by optimizing R for the fastest convergence), this requires linear memory, which is prohibitive in distributed settings.

3.2.2 Divergence Estimation

Under the conditions assumed by our analysis (continuous density on ❘ d ), the TVD between KDEs is a consistent estimator of the TVD [49, p1579]. Therefore, we can construct a consistent estimate of the TVD by simply plugging our estimators into Equation 3. The integral itself can be calculated using standard numerical integration techniques (e.g., Monte Carlo estimation or sparse grids), giving us a consistent sketch-based estimator for the TVD.

This method is implementable and comes with error bounds, but it requires a costly numerical integration whose complexity scales poorly with dimension ( O ( d ) per sample). Therefore, we justify a simpler method of computing the distance, albeit without formal guarantees.

As a collection of histograms, our sketch is a piecewise-constant approximation to the KDE over a partition π of events. This permits a simpler expression for Equation 3, where we compute the value in each partition, weighed by the partition size. Here, we suppose that the partition π consists of cells { A 1 , ... } . We let x ∈ A be any point inside A and use λ ( A ) to denote the Lebesgue measure of A .

To further simplify this expression, we make the following observations. Each cell of the partition, which we denote as A ∈ π , is defined by the intersected buckets of the hash function. In other words, each cell contains the points that have the same hash mapping under all R partitions.

While the expression for max π λ ( A ) is not generally known for most LSH functions, it is clear that λ ( A ) is non-increasing with R because each cell of the partition π is formed by intersecting R sub-partitions (one from each hash function). This operation almost surely decreases the size of A with each additional row in the sketch, since drawing two identical LSH functions is a zero-probability event. Also, E [ λ ( A )] is the same for all the cells in the partition by symmetry arguments.

Therefore, it is reasonable to consider an approximation where we compute entry-wise differences between sketches. Let ∆ r,b be the difference between row r and column (or bucket) b of the matrices S p and S q . We can construct a TVD estimator as a sum over all possible intersections of hash buckets.

There are B buckets in each repetition and, therefore, B R total 'intersection paths' through the sketch. To circumvent the O ( B R ) complexity of computing each path, we consider the sum of absolute deviations | ∆ 1 ,b 1 | + ... + | ∆ R,b R | instead of | ∆ 1 ,b 1 + ... +∆ R,b R | . This gives an upper bound on the TVD that can be computed in O ( BR ) time because we can now distribute the sums. In particular, we can compute the upper bound by taking the Frobenius norm (Euclidean distance) of S p -S q . We denote ∥ S p -S q ∥ 2 as sketch distance . Our empirical results show that this heuristic is an excellent indicator of distribution similarity.

4 One-Pass Distribution Sketch for FL

In this section, we introduce how to use our one-pass distribution sketch for effective and efficient FL. We will start with the client selection task with our sketch. Next, we will introduce how to perform an efficient cold start with our sketch in personalized FL shown in Section 6.

4.1 Distribution-Aware Client Selection

In our work, we study the setting where there are n clients checking into the server. In every round of iterative federated optimization, L clients are activated and ready for training. We assume that L clients are activated uniformly random out of n clients. Our client selection algorithm is presented in Algorithm 3 in the supplementary material. For each one of the n clients, we compute its sketch with Algorithm 1 and send it to the server. As introduced in Theorem 3.4, our sketch can be ϵ -differentially private with respect to the client data. As a result, we are able to communicate with the sketch between the server and clients. Next, the server takes the average of all the client sketches towards a global sketch S g . In each round of iterative federated optimization, the server looks up the sketch for every activated client. Next, we compute the Euclidean distance between S g and every activated client sketch S j . Next, we use the reciprocal of the distance as sample probability p j . Next, we normalize the p j across the L clients with Softmax function [50]. Next, we sample K clients out of the L activated clients. Each client is sampled with normalized probability p j . Finally, we perform local training on the K selected clients and aggregate their model updates.

Algorithm 2 Cold Start with Distribution Sketch Input: Clients C = { c 1 , · · · , c n } , New Client C q , LSH function family H , Hash range B , Rows R , Random seed s . Output: Personalized model w . for i ∈ [ n ] do Get client data D i from client c i . Get client model w i from client c i . Compute sketch S i using Algorithm 1 with parameter D i , H , B , R and s . Send S i to server end for Get client data D q from client c q . Compute sketch S q using Algorithm 1 with parameter D q , H , B , R and s . Send S q to the server. d min = ∞ id = null for i ∈ [ n ] do d i = ∥ S q -S i ∥ 2 if d i < d min then d min ← d i id ← i end if end for return w id

Our algorithm has three features: (1) We propose a way of estimating global data distribution across clients without data exchange. (2) We can select clients with data distribution closer to the global distribution for a better global model. (3) Our algorithm is efficient. We only need one-shot, onepass computation to generate a one-pass distribution sketch of client data. Moreover, the sketch is mergeable so a linear combination will give us a sketch for global data distribution. Furthermore, we can efficiently estimate the divergence of the client data distribution to the global distribution by taking the Euclidean distance of their sketches.

4.2 Nearest Neighbor Retrieval for Cold Start Client in Personalized FL

In this paper, we study the cold start problem in personalized FL. In this problem, many clients are connected to the server. Moreover, they collaborate together and produce a global model through iterative federated optimization [12]. Next, each client builds a personalized model based on the global model that achieves decent predictive performance on its local data (See Section 6). Next, a new client connects to the server. The new client has local but unlabeled data. In this setting, it

remains unknown which model, either global or local, should be applied to the new client as a cold starter.

We propose Algorithm 2 for this cold start problem. For the clients that already connect to the server, we compute its sketch with Algorithm 1 and send it to the server. Next, given a new client, we perform the sketching via Algorithm 1 with the same parameters for existing clients. Next, we compute the Euclidean distance between the new client sketch with the existing clients' sketches and retrieve the nearest neighbor client with minimum distance. Finally, the server retrieves the personalized model on the nearest neighbor client and sends it to the new client.

We want to highlight that Algorithm 2 formulates the cold start problem in personalized FL as a nearest neighbor search problem. By measuring data heterogeneity via Euclidean distance between distribution sketches, we provide an intuitive way for selecting a personalized client model.

5 Experiment

In this section, we conduct empirical evaluations to answer three questions: (1) Does the one-pass sketch distance reflect the differences between distributions? (2) Does the sketch distance help convergence in FL, and (3) Does the sketch distance retrieve the best-personalized models? To answer these three questions, we conducted three sets of experiments.

5.1 Dataset, Model and Implementation

Dataset: We evaluate Algorithm 3 and Algorithm 2 on both vision and language datasets. For visual classification, we use the MNIST dataset [51] and FEMNIST [52]. For MNIST, we introduce a federated dataset where each client is generated by random sampling from MNIST under Dirichlet distribution with α = 0 . 5 . We also introduce MNIST Uniform + Dirichlet , a dataset with half of the client data generated from random sampling from MNIST under Dirichlet distribution with α = 0 . 5 and the other half of the client generated from uniform sampling from MNIST. We also use the Shakespeare next-character prediction dataset [6] for language-based FL. The clients in both FEMNIST and Shakespeare datasets are fixed.

Model: For the MNIST dataset, we use a fully-connected network with a single 512-dimension hidden layer. For the FEMNIST dataset, we use LeNet-5 convolutional network [53, 51]. For the Shakespeare dataset, we build a model with one embedding layer, one long short-term memory (LSTM) [54] block, and an output layer. The input and output vocabulary size is 80, and the hidden dimension of the model is 256.

Implementation: Our FL codebase, including FL workflow, LSH functions, and proposed algorithms, is implemented on PyTorch [55]. We test Algorithm 3 and Algorithm 2 on a server with 8 Nvidia Tesla V100 GPU and a 48-core/96-thread processor (Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz).

5.2 One-Pass Sketch Distance

In this section, we investigate the correlation between one-pass sketch distance and distribution differences.

Setting: We generate two groups of subsets from MNIST, Group A and Group B. Datasets from Group A are drawn randomly from MNIST under Dirichlet distribution with α = 0 . 5 . Datasets from Group B are drawn randomly from MNIST under the uniform distribution. For every dataset, we construct a one-pass sketch. Then, we calculate the pairwise one-pass sketch distances between every dataset.

Result: We present the result in Figure 1. We annotate the one-pass sketch distance in each cell. First, we observe symmetry along the diagonal, which satisfies the basic principle of distance, d ( x, y ) = d ( y, x ) . Second, we observe group separation. The pairwise distances within group B are smaller, which is expected as they were drawn uniformly. The pairwise distances within Group A are larger and more diverse, which is expected as each is drawn randomly from the Dirichlet distribution. Lastly, for any dataset from Group A, its distance from any dataset from Group B is similar. Our results answer the first question: the distance between the distribution sketches constructed by Algorithm 1 can reflect their divergence in the distribution.

A_0

A_1

A_2

A_3

A_4

B_0

B_1

B_2

B_3

B_4

A_0

A_1

A_2

A_3

A_4

B_0

B_1

B_2

B_3

B_4

0

0.09

0.16

0.12

0.11

0.073

0.07

0.07

0.071

0.073

0.09

0

0.11

0.11

0.11

0.05

0.05

0.05

0.051

0.051

0.16

0.11

0

0.14

0.18

0.1

0.1

0.11

0.1

0.1

0.12

0.11

0.14

0

0.14

0.075

0.076

0.077

0.075

0.075

0.11

0.11

0.18

0.14

0

0.1

0.1

0.098

0.1

0.1

0.073

0.05

0.1

0.075

0.1

0

0.018

0.017

0.017

0.016

0.07

0.05

0.1

0.076

0.1

0.018

0

0.017

0.016

0.016

0.07

0.05

0.11

0.077

0.098

0.017

0.017

0

0.017

0.017

0.071

0.051

0.1

0.075

0.1

0.017

0.016

0.017

0

0.015

0.073

0.051

0.1

0.075

0.1

0.016

0.016

0.017

0.015

0

5.3 Distribution-Aware Client Sampling

In this section, we apply the one-pass distribution sketch (see Algorithm 1) in the client selection setting (See Section 6) and investigate whether data heterogeneity measurement helps with FL towards faster convergence. Setting: As shown in Section 4.2, in every round of iterative federated

0

5

10

15

20

25

30

35

40

45

Communication Round

0.75

0.80

0.85

0.90

0.95

Test accuracy

FedAvg

FedAvg + Sketch Distance

(a) MNIST

0

1

2

3

4

5

6

7

8

9

Communication Round

0.75

0.80

0.85

0.90

0.95

Test accuracy

FedAvg

FedAvg + Sketch Distance

(b)

MNIST

Uniform

+

Dirichlet

0

10

20

30

40

50

60

70

Communication Round

0.1

0.2

0.3

0.4

0.5

0.6

Test accuracy

FedAvg

FedAvg + Sketch Distance

(c) FEMNIST

0

3

6

9

12

15

18

21

Communication Round

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Test accuracy

FedAvg

FedAvg + Sketch Distance

optimization, there would be L activated clients. The standard federated optimization approach would uniformly randomly sample K clients as participants [6]. We denote this approach as Fedavg . However, due to the data heterogeneity, we often observe model divergence during aggregation. As a result, we propose Algorithm 3 to select the clients with distribution closer to the global distribution. In this experiment, we compare Algorithm 3 with Fedavg to showcase the importance of data heterogeneity in client selection. We set L = 3 K . For the MNIST dataset (both MNIST and MNIST Uniform + Direchlet), both Algorithm 3 and Fedavg are trained by 200 rounds. In each round, K = 3 clients are selected from L active clients. Next, each client is trained for 20 epochs with batch size 32 and learning rate η = 0 . 0001 . The FEMNIST uses the same training rounds, epochs, batch size, and learning rate as MNIST, except K = 10 . For the Shakespeare dataset, both Algorithm 3 and Fedavg are trained by 20 rounds. Each round, K = 3 clients are selected from L active clients. Next, each client is trained for 20 epochs with batch size 10 and learning rate η = 0 . 01 . We introduce early stopping to prevent overfitting.

Result: We present the test accuracy plot in Figure 2. As shown in the figure, on four FL datasets/settings, Algorithm 3 converges faster than Fedavg. In almost every round of optimization,

Algorithm 3 achieves better accuracy than Fedavg. This experiment's results answer the second question. The distances between distribution sketches can be used to sample helpful clients towards faster convergence.

5.4 Cold Start in Personalized FL

In this section, we would like to evaluate the performance of Algorithm 2 in the cold start problem of personalized FL (see Section 4.2).

Setting: For every dataset, we uniformly randomly divided them into warm clients and cold clients. The warm clients will participate in global training using Fedavg. Once we have trained the global model. Each client locally fine-tunes the global model. As a result, we have a global model on the server, and each client has its personalized model. For cold clients, we only use its label for evaluation. We use MNIST, FEMNIST, and Shakespeare datasets as introduced in Section 5.1. We generate MNIST clients following the by-class format shown in [13]. There are 100 clients in total. We set half of them as cold clients and the rest of them as warm clients. For MNIST, to get the global model, we train 200 rounds, and each round uniformly selects 10 clients. Each client is trained for 10 epochs on its local data with batch size

1.0

T

Y

0.8

0.6

0.4

Test

0.2

0.0

Global

Random

Neighbor

Best

32 and learning rate 0 . 0001 . Next, we finetune each client for 10 epochs with the same batch size and learning rate global model training for a local model. For FEMNIST, we set half of the clients as cold clients and the rest as warm clients. We use the same parameters as MNIST, except each round uses 50 clients. For Shakespeare, we use 20 clients as cold start clients, and the rest of them are warm clients. To get the global model, we train 5 rounds, and each round uniformly selects 3 clients. Each client is trained for 20 epochs on its local data with batch size 10 and learning rate 0 . 01 . Next, we finetune each client for 20 epochs for a local model. We set the batch size and learning rate as the same as global model training.

Result: We present the result in Table 5.4. For warm clients, we observe that the local models generally have better test accuracy due to local finetuning. For cold clients, using the model from its nearest neighbor always achieves high accuracy compared to the global model. Moreover, we provide an ablation study on MNIST. We obtain the test accuracy on each cold client with the global model, the model retrieved by uniformly random sampling, and the model retrieved by Algorithm 2. We also run a brutal force search on the existing personalized models and find the best model with the highest test accuracy. Note that this search is impractical since we use client data label information. But it is an upper bound for Algorithm 2. As shown in Figure 3, we observe that Algorithm 2 outperforms both the global model and random sampling with higher test accuracy on clients. Moreover, Algorithm 2 approximates the upper bound well. The experiment in this section answers the third question: Algorithm 2 can retrieve suitable personalized models for new clients with unlabeled data.

6 Related Work

Client Selection in FL. Iterative federated optimization is the de-facto standard for FL. In this optimization strategy, a server aggregates the model updates from each device to create a better

global model. Client selection [56, 57, 58, 59] serves as an essential component of FL in the common situation where many clients are connected to the server. In each round of iterative federated optimization, the server identifies a subset of clients as participants [60], asks participants to train on client data, and finally aggregates the updates. Client selection is necessary for two reasons. First, large-cohort training can cause optimization algorithms to fail. If we use all the clients, local objective misalignment can lead to catastrophic training failures [61]. This observation is backed up by theoretical analysis, which suggests that proper client selection can stabilize convergence [13, 62, 63]. Second, there are systems challenges. The communication required to coordinate a large number of clients imposes a heavy burden on network bandwidth [64, 65], reducing the efficiency of the FL system. Recent work on client selection strategies in FL focuses on the model's performance on the client data. For instance, [66] and [67] use the training loss of the client model on its data as a utility score for client selection. [68] and [69] use client model updates and build importance scores to select clients. In this work, we focus on using data heterogeneity for client selection without model information. As a result, we can avoid the computation and communication on unselected clients.

Personalized FL. Personalization is a natural way of improving local predictive performance on a client [70, 71, 27, 72]. Given the global model trained across diverse clients, it is a common practice to fine-tune the global model on every client's data to obtain a better local model [14, 73]. In this paper, we focus on a new challenging setting from practice: the cold start problem in personalized FL. In this setting, there are plenty of clients with personalized models on their devices. Next, a new client comes with unlabeled data. We want to cold start this client with a suitable local model. In this way, we can still provide decent performance on this client without training. Our work focuses on retrieving a local model from a client with a similar data distribution to the new client. We aim to achieve this strategy with an effective data heterogeneity measurement.

7 Conclusion

We study the problem of data heterogeneity in federated learning (FL), where different clients contain data generated from different distributions. We argue that data heterogeneity should be effectively measured for better global model training and FL personalization. However, there is no explicit and efficient way of measuring data heterogeneity. To address this challenge, we propose a one-pass distribution sketch that represents the data distribution while preserving ϵ -differential privacy. We show that the sketch distance can approximate the total variation distance between clients and introduce this technique to the client selection and cold start personalization tasks in FL. Our experiments showcase our sketch's practical impact in effective and efficient FL.

References

Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604 , 2018.

Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. Fednlp: A research platform for federated learning in natural language processing. arXiv preprint arXiv:2104.08815 , 2021.

Peyman Passban, Tanya Roosta, Rahul Gupta, Ankit Chadha, and Clement Chung. Training mixed-domain translation models via federated learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2576-2586, 2022.

Wensi Yang, Yuhang Zhang, Kejiang Ye, Li Li, and Cheng-Zhong Xu. Ffd: A federated learning based method for credit card fraud detection. In International conference on big data , pages 18-32. Springer, 2019.

Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open banking. In Federated learning , pages 240-254. Springer, 2020.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Colin Tankard. What the gdpr means for businesses. Network Security , 2016(6):5-8, 2016.

Jacquelyn K O'herrin, Norman Fost, and Kenneth A Kudsk. Health insurance portability accountability act (hipaa) regulations: effect on medical record research. Annals of surgery , 239(6):772, 2004.

Nguyen Truong, Kai Sun, Siyao Wang, Florian Guitton, and YiKe Guo. Privacy preservation in federated learning: An insightful survey from the gdpr perspective. Computers & Security , 110:102402, 2021.

Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518 , 2020.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan AlShedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 , 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429-450, 2020.

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 , 2018.

Shenglai Zeng, Zonghang Li, Hongfang Yu, Yihong He, Zenglin Xu, Dusit Niyato, and Han Yu. Heterogeneous federated learning via grouped sequential-to-parallel training. In International Conference on Database Systems for Advanced Applications , pages 455-471. Springer, 2022.

Riccardo Zaccone, Andrea Rizzardi, Debora Caldarola, Marco Ciccone, and Barbara Caputo. Speeding up heterogeneous federated learning with sequentially trained superclients. In 2022 26th International Conference on Pattern Recognition (ICPR) , pages 3376-3382. IEEE, 2022.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning , pages 5132-5143. PMLR, 2020.

Alp Emre Durmus, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. Federated learning based on dynamic regularization. In International Conference on Learning Representations , 2021.

Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn` y, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations , 2020.

Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning , pages 18250-18280. PMLR, 2022.

Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In European Conference on Computer Vision , pages 654-672. Springer, 2022.

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-efficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643 , 2019.

Zheng Chai, Hannan Fayyaz, Zeshan Fayyaz, Ali Anwar, Yi Zhou, Nathalie Baracaldo, Heiko Ludwig, and Yue Cheng. Towards taming the resource and data heterogeneity in federated learning. In OpML , pages 19-21, 2019.

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine , 37(3):50-60, 2020.

Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Based Systems , 216:106775, 2021.

Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems , 33:21394-21405, 2020.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning , pages 6357-6368. PMLR, 2021.

Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Alexandr Andoni, Piotr Indyk, and Robert Krauthgamer. Earth mover distance over highdimensional spaces. In SODA , volume 8, pages 343-352, 2008.

Wenzhe Yang, Sheng Wang, Yuan Sun, and Zhiyong Peng. Fast dataset search with earth mover's distance. Proceedings of the VLDB Endowment , 15(11):2517-2529, 2022.

DC Dowson and BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis , 12(3):450-455, 1982.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposium on the Theory of Computing (STOC) , pages 604-613, Dallas, TX, 1998.

Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S. Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the 20th ACM Symposium on Computational Geometry (SoCG) , pages 253-262, Brooklyn, NY, 2004.

Alexandr Andoni, Piotr Indyk, Huy L Nguyen, and Ilya Razenshteyn. Beyond locality-sensitive hashing. In Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms , pages 1018-1028. SIAM, 2014.

Alexandr Andoni and Ilya Razenshteyn. Optimal data-dependent hashing for approximate near neighbors. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing (STOC) , pages 793-801, 2015.

Alexandr Andoni, Thijs Laarhoven, Ilya Razenshteyn, and Erik Waingarten. Optimal hashingbased time-space trade-offs for approximate near neighbors. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 47-66. SIAM, 2017.

Chen Luo and Anshumali Shrivastava. Arrays of (locality-sensitive) count estimators (ace) anomaly detection on the edge. In Proceedings of the 2018 World Wide Web Conference , pages 1439-1448, 2018.

Benjamin Coleman, Richard Baraniuk, and Anshumali Shrivastava. Sub-linear memory sketches for near neighbor search on streaming data. In International Conference on Machine Learning , pages 2089-2099. PMLR, 2020.

Benjamin Coleman, Benito Geordie, Li Chou, RA Leo Elworth, Todd Treangen, and Anshumali Shrivastava. One-pass diversified sampling with application to terabyte-scale genomic sequence streams. In International Conference on Machine Learning , pages 4202-4218. PMLR, 2022.

Zichang Liu, Benjamin Coleman, Tianyi Zhang, and Anshumali Shrivastava. Retaining knowledge for learning with dynamic definition. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 14944-14958. Curran Associates, Inc., 2022.

Zellig S Harris. Distributional structure. Word , 10(2-3):146-162, 1954.

Luc P Devroye and TJ Wagner. The l _ 1 convergence of kernel density estimates. The Annals of Statistics , 7(5):1136-1139, 1979.

Dominik Wied and Rafael Weißbach. Consistency of the kernel density estimator: a survey. Statistical Papers , 53(1):1-21, 2012.

Luc Devroye. The equivalence of weak, strong and complete convergence in l1 for kernel density estimates. The Annals of Statistics , pages 896-904, 1983.

Benjamin Coleman and Anshumali Shrivastava. Sub-linear race sketches for approximate kernel density estimation on streaming data. In Proceedings of The Web Conference 2020 , pages 1739-1749, 2020.

Indradyumna Roy, Rishi Agarwal, Soumen Chakrabarti, Anirban Dasgupta, and Abir De. Locality sensitive hashing in fourier frequency domain for soft set containment search. Advances in Neural Information Processing Systems , 2023.

Benjamin Coleman and Anshumali Shrivastava. A one-pass distributed and private sketch for kernel sums with applications to machine learning at scale. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , CCS '21, page 3252-3265, New York, NY, USA, 2021. Association for Computing Machinery.

Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. On the empirical estimation of integral probability metrics. 2012.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT press, 2016.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Koneˇ cn` y, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 , 2018.

Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551, 1989.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics , pages 10351-10375. PMLR, 2022.

Wenkai Yang, Yankai Lin, Guangxiang Zhao, Peng Li, Jie Zhou, and Xu Sun. When to trust aggregated gradients: Addressing negative client sampling in federated learning. arXiv preprint arXiv:2301.10400 , 2023.

Carl Smestad and Jingyue Li. A systematic literature review on client selection in federated learning. arXiv preprint arXiv:2306.04862 , 2023.

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243 , 2020.

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Koneˇ cn` y, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems , 1:374-388, 2019.

Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large-cohort training for federated learning. Advances in neural information processing systems , 34:20461-20475, 2021.

Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. Journal of Machine Learning Research , 22, 2021.

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. A general theory for client sampling in federated learning. In IJCAI 2022-31st International joint conférence on artificial intellignce , 2022.

Jie Xu and Heqiang Wang. Client selection and bandwidth allocation in wireless federated learning networks: A long-term perspective. IEEE Transactions on Wireless Communications , 20(2):1188-1200, 2020.

Osama Shahid, Seyedamin Pouriyeh, Reza M Parizi, Quan Z Sheng, Gautam Srivastava, and Liang Zhao. Communication efficiency in federated learning: Achievements and challenges. arXiv preprint arXiv:2107.10996 , 2021.

Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learning via guided participant selection. In 15th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 21) , pages 19-35, 2021.

Chenning Li, Xiao Zeng, Mi Zhang, and Zhichao Cao. Pyramidfl: A fine-grained client selection framework for efficient federated learning. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking , pages 158-171, 2022.

Georgios Kollias, Theodoros Salonidis, and Shiqiang Wang. Sketch to skip and select: Communication efficient federated learning using locality sensitive hashing. In International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2022 , 2022.

Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. In ICC 2019-2019 IEEE international conference on communications (ICC) , pages 1-7. IEEE, 2019.

Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems , 30, 2017.

Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems , 33:1958619597, 2020.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems , 33:3557-3568, 2020.

Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758 , 2020.

Moses S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings on 34th Annual ACM Symposium on Theory of Computing (STOC) , pages 380-388, Montreal, Canada, 2002.

Ping Li. Sign-full random projections. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI) , pages 4205-4212, Honolulu, HI, 2019.

Xiaoyun Li and Ping Li. Random projections with asymmetric quantization. Advances in Neural Information Processing Systems , 32, 2019.

Xiaoyun Li and Ping Li. Quantization algorithms for random fourier features. In International Conference on Machine Learning , pages 6369-6380. PMLR, 2021.

Allan H Murphy. The finley affair: A signal event in the history of forecast verification. Weather and forecasting , 11(1):3-20, 1996.

Joshua Engels, Benjamin Coleman, and Anshumali Shrivastava. Practical near neighbor search via group testing. Advances in Neural Information Processing Systems , 34:9950-9962, 2021.

Xiaoyun Li and Ping Li. Rejection sampling for weighted jaccard similarity revisited. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Joshua Engels, Benjamin Coleman, Vihan Lakshman, and Anshumali Shrivastava. Dessert: An efficient algorithm for vector set search with vector set queries. arXiv preprint arXiv:2210.15748 , 2022.

Ping Li, Anshumali Shrivastava, Joshua Moore, and Arnd König. Hashing algorithms for large-scale learning. Advances in neural information processing systems , 24, 2011.

Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. Distributed hierarchical gpu parameter server for massive scale deep learning ads systems. In Proceedings of the 3rd Conference on Third Conference on Machine Learning and Systems (MLSys) , Huston, TX, 2020.

Andrei Z. Broder, Moses Charikar, Alan M. Frieze, and Michael Mitzenmacher. Min-wise independent permutations. In Proceedings of the Thirtieth Annual ACM Symposium on the Theory of Computing (STOC) , pages 327-336, Dallas, TX, 1998.

Ping Li, Art Owen, and Cun-Hui Zhang. One permutation hashing. Advances in Neural Information Processing Systems , 25, 2012.

Ping Li, Xiaoyun Li, and Cun-Hui Zhang. Re-randomized densification for one permutation hashing and bin-wise consistent weighted sampling. Advances in Neural Information Processing Systems , 32, 2019.

Xiaoyun Li and Ping Li. C-MinHash: Improving minwise hashing with circulant permutation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 12857-12887. PMLR, 17-23 Jul 2022.

Ping Li, Gennady Samorodnitsky, and John E. Hopcroft. Sign cauchy projections and chi-square kernel. In Advances in Neural Information Processing Systems (NIPS) , pages 2571-2579, Lake Tahoe, NV, 2013.

Ping Li and Cun-Hui Zhang. Theory of the GMM kernel. In Proceedings of the 26th International Conference on World Wide Web (WWW) , pages 1053-1062, Perth, Australia, 2017.

Ping Li. Several tunable GMM kernels. arXiv preprint arXiv:1805.02830 , 2018.

Amir Zandieh, Navid Nouri, Ameya Velingker, Michael Kapralov, and Ilya Razenshteyn. Scaling up kernel ridge regression via locality sensitive hashing. In International Conference on Artificial Intelligence and Statistics , pages 4088-4097. PMLR, 2020.

SS Barsov and Vladimir V Ul'yanov. Estimates of the proximity of gaussian measures. In Sov. Math., Dokl , volume 34, pages 462-466, 1987.

A Design of LSH Functions:

In practice, we could realize d ( x, y ) with cosine similarity for dense vectors. In this case, the LSH function family H should have the following form.

Definition A.1 (Sign Random Projection (SRP) Hash [74]) . We define a function family H as follow: Given a vectors x ∈ R d , any h : R d →{ 0 , 1 } that is in the family H , we have

where A ∈ R d × 1 is a random matrix that every entry of A is sampled from normal distribution N (0 , 1) , sign is the sign function that set positive value to 1 and others to 0 . Moreover, for two vectors x, y ∈ R d we have

As shown in Definition A.1, the SRP hash is an LSH function built upon random Gaussian projections, which is a fundamental technique in ML [75, 76, 77] If two vectors are close in terms of cosine similarity, their collision probability would also be high.

The SRP hash is usually designed for dense vectors. If both x ∈ { 0 , 1 } d and y ∈ { 0 , 1 } d are high dimensional binary vectors with large d . Their Jaccard similarity [78] is also an important measure for search [79, 80, 81] and learning tasks [82, 83]. There also exists a family of LSH functions for Jacacrd similarity. We define this LSH function as:

Definition A.2 (MinHash [84]) . A function family H is a MinHash family if for any h ∈ H , given a vectors x ∈ { 0 , 1 } d , we have

where Π is a permutation function on the binary vector x . The arg min operation takes the index of the first non-zero value in Π( x ) . Moreover, given two binary vectors x, y ∈ { 0 , 1 } d , we have

where the right term represents the Jaccard similarity of binary vectors x and y .

Following Definition A.2, MinHash serves as a powerful tool for Jaccard similarity estimation [85, 86, 87]. We will use both SRP hash and MinHash in the following section to build a sketch for data distribution.

In this paper, we take a kernel view of the collision probability of LSH (see Definition 3.2). This view aligns with a series of research in efficient kernel decomposition [88, 89, 90], kernel density estimation [46] and kernel learning [91].

B More Algorithms

In this section, we introduce the client selection algorithm with our one-pass sketch.

Appendix

Algorithm 3 Client Selection with Distribution Sketch

C More Experiments

In this section, we provide two-fold of more experiments. We start with a simulation of our sketch (see Algorithm 1) for total variation distance (TVD) estimation. Next, we perform client selection (see Algorithm 3) on CIFAR-10 datasets to observe the convergence improvements of our approach over FedAvg.

C.1 Simulation for Total Variation Distance Estimation

We computed the total variation distance (TVD) for two Gaussians with the same covariance and different means, one of the few TVDs between high-dimensional distributions for which we have an exact formula. This formula is TVD = 2Φ( z ) -1 where z = √ µ 1 -µ 2 ) ⊤ C -1 ( µ 1 -µ 2 ) (see [34, 92] for reference). To get the value of λ ( A ) (see Section 3.2.2), we integrated the highdimensional collision probability over the d-ball. We drew 1000 samples from two 16-dimensional Gaussians with random means, computed their sketch and TVD distances, and compared the results.

C.2 CIFAR-10 Experiments

We conducted CIFAR-10 experiments using 100 clients, where L=3K and K=2. The client dataset follows a non-iid distribution, as specified in the split provided at https://github.com/yjlee22/ FedShare . We visualized the iteration-wise convergence of FedAvg with and without the proposed client selection (see Figure 5). The results demonstrate that our distribution sketch improves the test accuracy convergence rate, leading to faster iterations.

1.6

1.4

TVD

1.2

Approx

1.0

0.8

0.6

0.4

0.5

0.6

0.7

0.8

0.9

1.0

TVD

0

2

4

6

8

10

12

Communication Round

39

40

41

42

43

44

Test accuracy

FedAvg

FedAvg + Sketch Distance

D Proofs

To formally prove the Theorems in the paper. We start with introducing a query algorithm to the one-pass distribution sketch.

Next, we introduce the formal statements in the paper as below.

Theorem D.1 (Formal version of Theorem 3.3) . Let p ( x ) denote a probability density function. Let D ∼ iid p ( x ) denote a dataset. Let k ( x, y ) be an LSH kernel (see Definition 3.2). Let S define the

function implemented by Algorithm 4. We show that

with convergence rate O ( √ log R/ √ R ) .

Proof. Let κ ( x ) = ∑ x i ∈D √ k ( x, x i ) be the (non-normalized) kernel density estimate. Theorem 3.4 of [48] provides the following inequality for any δ , where ˜ κ ( x ) = ∑ x i ∈D √ k ( x, x i ) :

This is equivalent to the following inequality, which we can obtain by dividing both sides of the inequality inside the probability by N .

We want to show that the error ∣ ∣ S ( x ) -1 N κ ( x ) ∣ ∣ converges in probability to zero, because this directly proves the main claim of the theorem. To do this, we must show that for any ∆ > 0 ,

This can be done by setting δ = 1 R , which yields the following inequality:

Because ˜ κ < N , the following (simpler, but somewhat looser) inequality also holds:

This implies that S ( x ) → i . p . 1 N κ ( x ) by considering R large enough that √ log R/R < ∆ .

Theorem D.2 (Formal version of Theorem 3.4) . Let S be an ϵ -differentially private distribution sketch of a dataset D = { x 1 , ...x N } with D ∼ iid p ( x ) and let k ( x, y ) be an LSH kernel (see Definition 3.2). Let S define the function implemented by Algorithm 4. Then

with convergence rate O ( √ log R/R + √ R log R/ ( Nϵ ) when R = ω (1) (e.g. R = log N ).

Proof. As before, let κ ( x ) = ∑ x i ∈D √ k ( x, x i ) be the (non-normalized) kernel density estimate and let ˜ κ ( x ) = ∑ x i ∈D √ k ( x, x i ) . For the ϵ -differentially private version of the algorithm, Theorem 3.4 of [48] provides the following inequality for any δ > 0 .

Again, we divide both sides of the inner inequality by N , and we also loosen (and simplify) the inequality with the observation that ˜ κ ( x ) /N < 1 .

We want to show that the error ∣ ∣ S ( x ) -1 N κ ( x ) ∣ ∣ converges in probability to zero, because this directly proves the main claim of the theorem. To do this, we must show that for any ∆ > 0 ,

As before, we choose δ = 1 R , which yields the following inequality:

This is the same as:

Here, we will loosen the inequality again (also for the sake of presentation). Because √ a + b ≤ √ a + √ b , the following inequality is also satisfied:

Here, the convergence rate is O ( √ log R/R + √ R log R/ ( Nϵ ) . For us to have the error converge in probability to zero, we need for

as N → ∞ . A simple way to achieve this is for R to be weakly dependent on N (i.e. choose R = ω (1) ). For example, choosing R = log N satisfies the conditions.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use a locally sensitive hashing based sketch to measure the similarity between two datasets in a privacy-preserving manner. This sketch can be computed efficiently and differential privacy can be guaranteed via the Laplace mechanism. The paper concludes with presenting two potential applications of their similarity measure: selecting participating clients in federated learning based on the similarity of local datasets, and selecting an initial model for a new client in personalized federated learning.\n\nSTRENGTHS:\n- The idea is clear and sound, a privacy-preserving dataset similartiy measure is useful, in particular in federated and collaborative learning scenarios.\n- Theoretically sound definition of the sketch from which an estimator of the total variation distance can be derived.\n- The paper presents two applications of the similarity measure for federated learning.\n\nWEAKNESSES:\n- The distribution-aware client sampling is interesting, but not well-motivated. Sampling clients with differing local data from the average distribution can be helpful in some settings, while being detrimental to the learning process in others. While the empricial results look promising, the analysis is far too shallow to permit conclusions. \n--- after rebuttal ---\nThe authors have addressed this weakness in the rebuttal.\n\nQUESTIONS:\n- Would the definition 3.1 of TVD not be more appropriately placed at the beginning of Sec 3.2.2?\n- Would it be possible to quantify the difference of the upper bound (using the Euklidean distance between sketches) to the true TVD, based on properties of the sketches? E.g., assuming sufficiently smooth data distributions.\n- Could you add synthetic experiments where data is drawn from known distributions, so that the accuracy of the TVD estimation can be more clearly assessed?\n\nLIMITATIONS:\nThe paper is clear about the assumptions and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe presence of **data heterogeneity** poses a critical challenge in **federated learning** (FL), as it leads to slower and noisier collaborative learning. To address this issue and improve the learning process with faster convergence, it is here proposed a more efficient client selection strategy based on distribution estimation. By utilizing an **efficient distribution sketch**, the server gains access to estimated distributions of the clients' local data, enabling the construction of a sketch representing the underlying global distribution while maintaining privacy constraints. With this information at hand, clients with data distributions closer to the global one can be more actively involved in the training process, thereby mitigating the negative effects of data heterogeneity and facilitating the development of a better global model. In addition, this approach offers the advantage of providing new clients - potentially holding unlabeled data - with initial models that are better aligned with their local distribution, thus enhancing their local training. To be more specific, new clients can receive the model trained by other users with similar local distributions, thus benefiting from their previously acquired knowledge.\n\nSTRENGTHS:\n1. The paper addresses the issue of data heterogeneity in FL, relevant to this field of research.\n2. A new method for estimating local and global data distributions is introduced. This method is both efficient and privacy preserving. It can also be adapted to be $\\epsilon$-differentially private. The distribution sketch enables convergence speed up by allowing a smarter client selection, thus reducing the communication rounds between clients and server, significant for the evaluation of FL algorithms.\n3. Both theoretical and complexity analyses are presented for the proposed method. Claims are supported by proofs.\n4. Empirical validation is presented on both vision and NLP datasets.\n5.The paper is well written, easy to follow and understand.\n6. Limitations and potential negative impacts are discussed. Computational requirements are identified.\n\nWEAKNESSES:\n1. Some relevant related works are missing. \n    * Client selection: [1,2]\n    * Personalized FL: [3]\n    * I believe discussing other works proposing different data distribution estimation techniques is relevant to this paper, but is currently missing. Examples are [4,5,6,7].\n    * Algorithms addressing statistical heterogeneity in FL. Examples are unlimited: the most well-known baselines are [10-15]. Recently works such as [16,17] are gaining visibility, as well as [18].\n    * Lastly, since the paper highlights the positive effect of having new clients starting from pretrained models, current literature addressing the impact of pretraining in FL might be of interest [8,9].\n2. The efficacy of the one-pass distribution sketch in efficiently addressing the cold start of new *unlabeled* clients is emphasized. However, it is important to note that these clients still require labeling in order to effectively use the model and perform local training. As no alternative solution is proposed in this context, the necessity for labeled data on the client side remains unchanged. Consequently, the ability for new clients to initially possess unlabeled data does not yield a substantial difference in practical real-world scenarios.\n3. In realistic cross-device [19] federated scenarios, billions of clients with very small datasets are involved in the training. This implies $|\\cup_{k\\in[K]} \\mathcal{D}_k| \\gg |\\mathcal{D}_k|$, with $K$ being the number of clients with local datasets $\\mathcal{D}_k, k\\in[K]$. As a consequence, it is very unlikely that distribution of single local datasets may resemble the global one. Referring to the notation introduced in Algorithm 3, in the real-world scenario, $p_i - p_j \\approx 0 \\forall i,j\\in[L]$. It is not clear how this method would be effective in this case.\n4. To my understanding, in order to perform the warm start on new clients, the server has to keep memory of the last model trained by each client, and store it, together with the distribution sketch. This becomes unfeasible with the increasing number of clients in realistic settings. If the server retrieves the personalized model from the nearest neighbor client instead, this introduces additional communication, and such cost increases with the constant arrival of new clients. This limitation is nowhere addressed.\n5. By favoring clients having local distributions similar to the global one, the resulting learned model might potentially be unfair towards out-of-distribution users. I believe this behavior arises issues in terms of **fairness** towards less represented distributions.\n6. The method is only compared with FedAvg, the first algorithm proposed for FL. Missing comparison with methods i) using different techniques for estimating the local distributions, ii) performing smarter client selection, iii) addressing statistical heterogeneity in FL, iv) using pretrained models in FL.\n\n**Minor weaknesses** (did not affect my rating)\n1. Missing hyperparameters ranges used for tuning the baselines and some details on the federated settings. Please refer to Questions.\n\n**References**\n\n[1] Cho, Yae Jee, Jianyu Wang, and Gauri Joshi. \"Towards understanding biased client selection in federated learning.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\n[2] Yang, Wenkai, et al. \"When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning.\" TMLR (2023).\n\n[3] Fallah, Alireza, Aryan Mokhtari, and Asuman Ozdaglar. \"Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach.\" Advances in Neural Information Processing Systems 33 (2020): 3557-3568.\n\n[4] Zeng, Shenglai, et al. \"Heterogeneous federated learning via grouped sequential-to-parallel training.\" International Conference on Database Systems for Advanced Applications. Cham: Springer International Publishing, 2022.\n\n[5] Zaccone, Riccardo, et al. \"Speeding up heterogeneous federated learning with sequentially trained superclients.\" 2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2022.\n\n[6] Lu, You-Ru, Xiaoqian Wang, and Dengfeng Sun. \"Tackling Imbalanced Class in Federated Learning via Class Distribution Estimation.\" (2022).\n\n[7] Chen, Dawei, et al. \"Digital twin for federated analytics using a Bayesian approach.\" IEEE Internet of Things Journal 8.22 (2021): 16301-16312.\n\n[8] Nguyen, John, et al. \"Where to begin? Exploring the impact of pre-training and initialization in federated learning.\" NeurIPSW (2022).\n\n[9] Chen, Hong-You, et al. \"On the importance and applicability of pre-training for federated learning.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[10]  Li, Tian, et al. \"Federated optimization in heterogeneous networks.\" Proceedings of Machine learning and systems 2 (2020): 429-450.\n\n[11] Karimireddy, Sai Praneeth, et al. \"Scaffold: Stochastic controlled averaging for federated learning.\" International conference on machine learning. PMLR, 2020.\n\n[12] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. International Conference on Learning Representations, 2021. \n\n[13] Reddi, Sashank, et al. \"Adaptive federated optimization.\" ICLR (2021).\n\n[14] Li, Qinbin, Bingsheng He, and Dawn Song. \"Model-contrastive federated learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[15] Karimireddy, Sai Praneeth, et al. \"Mime: Mimicking centralized stochastic algorithms in federated learning.\" Advances in Neural Information Processing Systems, 2021 .\n\n[16] Qu, Zhe, et al. \"Generalized federated learning via sharpness aware minimization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[17] Caldarola, Debora, Barbara Caputo, and Marco Ciccone. \"Improving generalization in federated learning by seeking flat minima.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[18] Wang, Jianyu, et al. \"Slowmo: Improving communication-efficient distributed sgd with slow momentum.\" ICLR (2020).\n\n[19] Kairouz, Peter, et al. \"Advances and open problems in federated learning.\" Foundations and Trends® in Machine Learning 14.1–2 (2021): 1-210.\n\nQUESTIONS:\n**Questions**\n1. Regarding the client selection strategy,\n    * How does the model behave on out-of-distribution (OOD) clients, *i.e.*, the clients having maximum distance from the global distribution sketch?\n    * How often are clients having larger distances from the global distribution sketch selected for training with respect to clients having more similar distributions?\n    * Referring to 3) in Weaknesses, what happens when all local distributions are very distant from the global one, i.e. if $p_i - p_j \\approx 0 \\forall i,j\\in[L]$? (referring to the notation in Algorithm 3)\n    * How does this approach compare with other smart client selection strategies? Please see 1) in Weaknesses for examples on related works.\n2. Regarding experiments in Figure 2:\n    * The proposed method seems to be particularly effective on the Shakespeare dataset rather than on MNIST/FEMNIST. Is this behavior more related to the dataset  itself, or to the task? Is the one-pass distribution sketch more effective for NLP? Experiments on an additional vision or NLP dataset could help clarify this behavior. Examples are federated CIFAR10/100, or StackOverflow.\n    * The text mentions the experiments were run for 200 rounds (line 315), but in the plots the X axes show a different number of rounds for each experiment. Which is the correct information?\n3. How are the values of $B$ and $R$ (Alg. 1) selected? Are they hyperparameters? Which values were chosen? How sensitive is the method to such a choice?\n4. Is there a risk of privacy leakage from the distribution sketch if no Laplace noise is added? \n\n**Minor questions** (did not affect my rating)\n1. How many clients were used in the experiments? \n2. What is the average number of samples per client?\n\n**Suggestions**\n1. I would suggest revising the abstract (lines 10-14) to obtain a more effective result. For instance, it is not clear how the distribution sketch helps in the client selection, or how this method helps training on new unlabeled clients. A few additional details might send the message better.\n\n**Minor suggestions** (did not affect my rating)\n* Typos: \n    * line 45: *is* computational ... --> \"are\"\n    * In Sec. 3.2, Algorithm **3** from Sec. D appears referenced as Algorithm **0**.\n    * lines 332-334 probably needs rephrasing (specifically referring to the sentence \"Once we have the global model\")\n* Def 2.1 introduces $p_1$, $p_2$ and $D$, whose set of belonging is not formally defined - *e.g.*, $\\mathbb{R}$.\n\nLIMITATIONS:\nMain limitations were addressed by the authors. \n\nAdditional limitation: potential lack of fairness towards users belonging to less represented distributions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to use sketching for measuring heterogeneity in user data distributions in federated learning settings. The sketching is based on locality sensitive hashing and the authors have shown that in theory the measurement from the count sketch converges to the true kernel density estimation. The authors showed that the heterogeneity measurement and the count sketch can be used to 1) improve convergence by sampling users smartly according to their density in the distribution and 2) improve personalization by warming up the new client with a known nearest client’s model. The effectiveness of the approaches have been empirically tested on simple benchmark FL datasets.\n\nSTRENGTHS:\n* Measuring data heterogeneity has been an important problem to solve and the authors proposed a simple, intuitive yet effective idea. The implementation is easy to extend to existing FL frameworks. \n* The experiments demonstrated not only the effectiveness of heterogeneity measurement but also how to apply the measurement for improving federated learning convergence and personalization, showing the general applicability of the proposed methods.\n\nWEAKNESSES:\n* For cross-device setting, some algorithms (e.g. with differential privacy) might incur large communication costs depending on the size of the sketch matrix. \n* The datasets and models used in the experiments are very small scale and some do not reflect the true heterogeneity in the wild.\n* Figure 1 can be explained better with the label distribution for users in group A. E.g. Do users with higher dissimilarity have dissimilar digits in MNIST and vice versa?\n* Minor points about writing:\n1) RACE introduced in section 2 has matrix $A \\in {R}^{B\\times R}$ while the sketch matrix in the following paragraph has transposed the axes. There would be less friction of reading if they are consistent.\n2) Line 140, algorithm 0 should be algorithm 4.\n\nQUESTIONS:\n* What are the values for B and R chosen in this paper?\n* How should one choose these values for different datasets?\n\nLIMITATIONS:\nThere is no or minor potential negative societal impact of this paper. The authors have acknowledged limitations in the appendix of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a private strategy for measuring data heterogeneity using a RACE matrix. The RACE matrix essentially measures the normalized frequency of data points being assigned to particular LSH codes. These LSH count matrices are sent up to a server where it is applied into two different regimes:\\\\\n1. For distribution-aware client selection, the server samples clients with probability directly associated to closeness between the local RACE and global RACE matrix. \n2. For cold-start learning, a new client receives a personalized client model whose distribution matrix is closest to the new client's distribution matrix. \n\nThe approach is assessed over federated training on several datasets. Local accuracies are compared by using the global or locally closest model across both warm and cold clients.\n\nSTRENGTHS:\n-The RACE matrix is theoretically a good estimator of DTV between distributions. \n-Scheme for developing client selection probabilities is simple. \n-There are apparently not many existing algorithms for addressing cold start learning. \n-Tested over multiple datasets with promising local accuracies.\n\nWEAKNESSES:\n1. The RACE matrix is apparently already well-known as an excellent kernel density estimator. The contribution seems to be application of it to the federated setting, which has limited novelty. \n2. Under the suggested client scheme, the global model is favoring the largest set of homogeneous clients (popularity bias). Clients with atypical data distributions will have a low chance of being selected for global averaging, so there is no benefit for them to participate. If a client with an atypical distribution were to be randomly selected, their model would be washed out with a global model which is, on average, favoring distributions unlike theirs. \n3. The RACE estimation could be heavily affected by quantity skew (which will accompany label-distribution skew in practice). Many entries will be under-filled if a local dataset simply isn't large enough and the output space of the hash function larger than 2. \n4. In the cold start setting, the central server needs to be storing every possible client model. This would not be scalable. Such approach might better fit decentralized learning. \n5. MNIST and FEMNIST are too easy to classify. CIFAR-10 training, at the minimum would have been nice to see, which suffers far more from label-distribution heterogeneity, see: \"Federated Learning on Non-IID Data Silos: An Experimental Study,\" Li et al. (2021).\n\nQUESTIONS:\n1. I can't seem to easily find the settings for the LSH setup. How many LSH functions were used per sketch? Were they refreshed after every averaging round? Does the server distribute the hash functions? \n2. Were there any measurement regarding how the global model accuracy is affected by personalized model selection in the cold client setting? I imagine there should be a classical tradeoff between global and personalized accuracy.\n\nLIMITATIONS:\nNo limitations discussed. Please consider including drawbacks.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}