Find Your Friends: Personalized Federated Learning with the Right Collaborators
Authors: Yi Sui
Junfeng Wen
Yenson Lau
Brendan Leigh Ross
Jesse C Cresswell
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated learning
Personalized federated learning
Decentralized federated learning

Abstract: In the traditional federated learning setting, a central server coordinates a network of clients to train one global model. However, the global model may serve many clients poorly due to data heterogeneity. Moreover, there may not exist a trusted central party that can coordinate the clients to ensure that each of them can benefit from others. To address these concerns, we present a novel decentralized framework, FedeRiCo, where each client can learn as much or as little from other clients as is optimal for its local data distribution. Based on expectation-maximization, FedeRiCo estimates the utilities of other participants’ models on each client’s data so that everyone can select the right collaborators for learning. As a result, our algorithm outperforms other federated, personalized, and/or decentralized approaches on several benchmark datasets, being the only approach that consistently performs better than training with local data only.

Under review as a conference paper at ICLR 2023
FIND YOUR FRIENDS: PERSONALIZED FEDERATED LEARNING WITH THE RIGHT COLLABORATORS
Anonymous authors Paper under double-bling review
ABSTRACT
In the traditional federated learning setting, a central server coordinates a net- work of clients to train one global model. However, the global model may serve many clients poorly due to data heterogeneity. Moreover, there may not exist a trusted central party that can coordinate the clients ensure that each of them can benefit from others. To address these concerns, present novel decentral- ized framework, FedeRiCo, where each client can learn as much or as little from other clients is optimal for its local data distribution. Based expectation- maximization, FedeRiCo estimates the utilities of other participants' models on each client's data that everyone select the right collaborators for learning. As result, our algorithm outperforms other federated, personalized, and/or de- centralized approaches several benchmark datasets, being the only approach that consistently performs better than training with local data only.
INTRODUCTION
Federated learning (FL) MMMMahan et al. 2017) offers a framework in which a single server-side model collaboratively trained across decentralized datasets held by clients. has been success- fully deployed in practice for developing machine learning models without direct access to user data, which essential highly regulated industries such as banking and healthcare (Long et al. 2020) Sadilek et al. (021). For example, several hospitals that each collect patient data may want merge their datasets for increased diversity and dataset size but are prohibited due to privacy regulations.
8.5
8.5
1.0
9.5
8.8
0.0
0.9
0.0
Client
Client
Client
8..5
8.5
6.5
0.0
8.8
6.6
client cLiont
cliant
Client
Client
Client
Noisy data points generated for each client along a sine curve (solid magenta line) where the r-axis and y-axis correspond to input and output respectively. The corresponding model learned by FedAvg (dotted line) fails to adapt to the local data seen by each client, in contrast to the models learned by each client using our FedeRiCo (dashed lines). Right: The weights used by FedeRiCo to average participant outputs for each client. As the client index increases, the data is generated from successive intervals of the sine curve, and collaborator weights change accordingly.
Figure 1: Left:
Traditional FL methods like Federated Averaging (FedAvg) (McMahan et al. 2017) can achieve noticeable improvement over local training when the participating clients' data are homogeneous. However, each client's data is likely to have different distribution from others in practice (Zhao ct al.) 2018; Adnan et al., (022). Such differences make much more challenging to learn a global model that works well for all participants. As illustrative example, consider simple scenario

---

Under review as a conference paper at ICLR 2023
where each client seeks to fit linear model to limited data, on interval of the sine curve as shown in Fig. [ This is analogous to the FL setting where several participating clients would like to collaborate, but each client only has access to data from its own data distribution. It is clear that no single linear model can be adequate to describe the entire joint dataset, so a global model learned by FedAvg can perform poorly, as shown by the dotted line. Ideally, each client should benefit from collaboration by increasing the effective size and diversity of data, but in practice, forcing everyone use the same global model without proper personalization can hurt performance on their own data distribution (Kulkarni et al., 2020; Tan et al. 2022).
To address this, we propose Federating with the Right Collaborators (FedeRiCo), novel framework suitable for every client to find other participants with similar data distributions to collaborate with. Back to our illustration in Fig. [] FedeRiCo enables each client to choose the right collaborators as shown on the plots on the right-hand side: each client is able to correctly leverage information from the neighboring clients when it is beneficial to do so. The final personalized models can serve the local distributions well, as demonstrated in the left plot.
More specifically, our FedeRiCo assumes that each client has an underlying data distribution, and exploits the hidden relationship among the clients' data. By selecting the most relevant clients, each client can collaborate as much or as little as they need, and learn a personalized mixture model to fit the local data. Additionally, FedeRiCo achieves this in fully decentralized manner that is not beholden to any central authority (Li et al.
2021a Huang et al. 2021 Kalra et al. 2021).
Our contributions We propose FedeRiCo, novel decentralized and personalized FL frame- (EM). Within this framework, we propose a
work derived based on pnpectation-maximizann communication-effiiennn protocol suitable for fully-decentralized learning. Through extensive ex- periments on several benchmark datasets, we demonstrate that our approach finds good client col- laboration and outperforms other methods in the non-i.i.d. data distributions setting.
Paper outline The rest of the paper is organized as follows. In Section 2 we discuss related approaches towards decentralized federated learning and personalization. Section 3 describes our algorithm formulation and its relationship to pnpectatioommaiiizzatioo and an efficient protocol for updating clients. We provide experimental results in Section 4, and conclude in Section 5.
2 RELATED WORK FOR PERSONALIZED FL
Meta-learning Federated learning can be interpreted as meta-learning problem, where the goal is to extract a global meta-model based on data from several clients. This meta-model can be learned using, for instance, the well-known Federated Averaging (FedAvg) algorithm (McMahan et al. 2017) and personalization can then be achieved by locally fine-tuning the meta-model (Jiang al., 2019.. Later studies explored methods to learn improved meta-models. Khodak et al. (2019) pro- posed ARUBA, meta- meta-learning algorithm based on online convex optimization, and demonstrates that it can improve upon FedAvg's performance. Per-FedAvg (Fallah et al.) 2020) uses the Model Agnostic Meta-Learning (MAML) framework to build the initial meta-model. However, MAML. requires computing or approximating the Hessian term and can therefore be computationally pro- hibitive. Acar et al. (2021) adopted gradient correction methods explicitly de-bias the meta-model from the statistical heterogeneity of client data and achieved sample-efficient customization of the meta-model.
Model regularization/ interpolation Several works improve personalization performance by reg- ularizing the divergence between the global and local models (Hanzely & Richtárik 2020 Li et al., 20216, Huang et al.) (202I). Similarly, PFedMe (T Dinh et al. 2020) formulate lateu personalization as a proximal regularization problem using Moreau envelopes. FMI (Shen et al.) (020) adopts knowl- edge distillation to regularize the predictions between local and global models and handle model heterogeneity. In recent work, SFL (Chen et al. 2022) also formulates the personalization as a bi-level optimization problem with an additional regularization term on the distance between local models and its neighbor models according to a connection graph. Specifically, SFL adopts GCN to represent the connection graph and learns the graph part of the optimization to encourage useful client collaborations. Introduced by Mansour et al. (2020) as one of the three methods for achiev- ing personalization in FL., model interpolation involves mixing client's local model with jointly
2

---

Under review as a conference paper at ICLR 2023
trained global model to build personalized models for each client. Deng al. (2020) further derive generalization bounds for mixtures of local and global models.
Multi-task learning Personalized FL naturally fits into the multi-task learning (MTL) framework. MOCHA (Smith et al., 2017) utilizes MTL to address both systematic and statistical heterogeneity but is restricted to simple convex models. VIRTUAL (Corinzia et al.) 2019) is federated MTL framework for non-convex models based on a hierarchical Bayesian network formed by the central server and the clients, and inference is performed using variational methods. SPO (Cui et al. 2021) applies Specific Pareto Optimization to identify the optimal collaborator sets and learn hypernet- work for all clients. While also aiming to identify necessary collaborators, SPO adopts a centralized FL setting with clients jointly training the hypernetwork. In contrast, our work focuses on decen- tralized FL. where clients aggregate updates from collaborators, and jointly make predictions.
In a similar spirit to our work. Marfoq et al. (2021) assume that the data distribution of each client mixture of several underlying distrbbutions/componentss Federated MTL is then formulated as problem of modeling the underlying distributions using Federated Expectation Catatioon-Mxiiizaaiionn (Fe- dEM), Clients jointly update set of several component models, and each maintains a customized set of weights, corresponding to the mixing coefficients of the underlying distributions, for predic- tions. One shortcoming of FedEM is that it uses an instance-level weight assignment in training time but client-level weight assignment in inference time. As a concrete example, consider client consisting of a 20%/80% data mixture from distributions A and B. FedEM will learn two models, one for each distribution. Given new data point at inference time, the client will always predict 0.2 ..e-pred_ + 0.8 -.edd..rggg regardless of whether it came from distribution A or B. This is caused by the mismatched behaviour between training and inference time. On the contrary, FedeRiCo naturally considers a client-level weight assignment for both training and inference in a decentralized setting.
Other approaches Clustering- baaser approaches are also popular for personalized FL (Sattler al., 2020 Ghosh et al. 2020, Mansour et al., (2200). Such personalization lacks flexibility since each client can only collaborate with other clients within the same cluster. FedFomo (Zhang et al.) 202T) interpolates the model updates of each client with those of other clients to improve local perfor- mance. FedPer (Arivazhagan et al. 2019) divides the neural network model into base and personal- ization layers. Base layers are trained jointly, whereas personalization layers are trained locally.
3 FEDERATED LEARNING WITH THE RIGHT COLLABORATORS
3.1 PROBLEM FORMULATION
We consider a federated learning (FL. scenario with K clients. Let [K] 2 K} de- note the set of positive integers up until K. Each client E [K] consists of a local dataset
D. (xxx) (i) (6) ) where n; is the number of examples for client i, and the input x, E X Ms
and output y, E y' are drawn from joint distribution D; over the space X' x y.
The goal of personalized FL is to find prediction model h; : x' ) that can perform well on the local distribution D. for each client. One of the main challenges in personalized FL is that we do not know if two clients i and j share the same underlying data distribution. If their data distributions are vastly different, forcing them to collaborate is likely to result in worse performance compared to local training without collaboration. Our method, Federating with the Right Collaborators (Fed- eRiCo), is designed to address this problem so that each client can choose to collaborate or not, depending on their data distributions. FedeRiCo is a decentralized framework (i.e. without a central server). For better exposition, Section 3.2 first demonstrates how our algorithm works in hypo- thetical all-to-all communication setting, an assumption that is then removed in Section 3.3 which presents several practical considerations for FedeRiCo to work with limited communication.
3.2 FEDERICO WITH ALLL COMMUNICATION
++
Z
Note that every local distribution D; can always be repre- sented as mixture of (D,1 with some client weights π, [may...., [ak] E AK where AK is the (K 1)-dimensional
K Clients
Figure 2: Graphical model
3

---

Under review as a conference paper at ICLR 2023
simplex [1 Let be the latent assignment variable of client i, and II :....... Tk] be the prior 11 = Pr(2; = j). Suppose that the conditional probability p. p. ())x) satisfies - log p: (1 x) = (ho; (x), v) + e for some parameters of E Rd loss function é :: ) + R+ and normalization
constant c. By using the stacked notation d = . Φ R R Fig. 2 shows the graphical model of how the local dataset is generated. Our goal is to learn the parameters θ (Φ, II) by maximizing the log-likelihod::
1 K f()) == log (Dg((D:)) = Σ log logg(D;; 1 K log (D, θ)
1
(I)
where := U,D, Dj and := E; n4. One standard approach to optimization with latent variables is expectation maximization (EM) (Dempster al. 1977). The corresponding variational lower bound is given by (all detailed derivations of this section can be found in Appendix
C(g. +) n Σ Boccs)loor(( 21 0]] +C
where C' is constant not depending on O. To obtain concrete objective functions suitable for optimization, we further assume that (x(x) = p(z). Vi E [K]. Similar Marfoq et al. (2021). this assumption is required due to technical reasons and can be relaxed if needed. With this assumption, we perform the following updates at each iteration l:
E-step: For each client, find the best q, which is the posterior p(z: = ji ((t-1())))) given the current parameters ((--1);
w = (0 = j) x u exp Σ l
(x y!!)
(3)
M-step: Given the posterior (1) from the E-sepp, maximize L w.r.t. O = (0) II):
K Σ (w,
n w and
h E argmin
(4)
+
R2
where
(..( =
(n,, (x( y/
(5)
Bear in mind that each client can only see its local data D, in the federated setting. The E-step
easy to compute once the models from other clients Φj.j j o are available. m is also easy to
(t) obtain as the posterior wij is stored locally. However, (+)) is trickier to compute since each client can potentially update & towards different directions due to data heterogeneity amongst the clients.
To stabilize optimization and avoid overfitting from client updates, we rely on small gradient steps in lieu of full optimization in each round. To compute ((t) algorithmically, each client i:
1. Fixes w(() and computes the local gradient V (et,s(((())))
local D;.
2. Broadcasts V VÉ...((('* (((+-1)) to and receives V Eno, (t) from other clients j # i. The models are updated based on the aggregated gradient with step size > 0:
K
(()) ((1-1) 17
(
(6)
Each client uses Tx((x)= = E, with 20 (x) for prediction after convergence.
Remark The posterior (!(() (or equivalently the prior in the next iteration m!!)) reflects the importance of model Φ; on the data D,. When (n(() one-hot with a one in the ith position, client
can perform learning by itself without collaborating with others. When (0 is more diverse, client
One-hot n, always feasihle, hut other mixing cnefficients may exist.
4

---

Under review as a conference paper at ICLR 2023
can find the right collaborators with useful models Φ, Such flexibility enables each client to make its own decision on whether or not to collaborate with others, hence the name of our algorithm.
Remark 2 Unlike prior work (Mansour et al. 2020 Marfog et al. 2021), our assignment variable and probability II are on the client level. If we assume that all clients share the same prior (i.c., there is only a vector instead of matrix IT), the algorithm would be similar to HypCluster (Mansour ee al. 2020). Marfoq et al. (202T) used a similar formulation as ours but their assignment variable is on the instance level: every data point (instead of client) comes from mixture of distributions. Such an approach can cause several issues at inference time, as the assignment for novel data point and Section 4 for further comparison.
unknown. We refer the interested readers Section 2
Theoretical Convergence Under some regularity assumptions, our algorithm converges as follows:
E.6 when the clients use SGD with learning dg_ and after sufficient rounds T, the iterates of our algorithu satisfy rate r) VT
Theorem 3.1. |Convergencel Under Assumptions E.
1 T
V
S O VT (
7 -
T Anf(
Ans(
IT') S
1
(7)
T3/4
where the expectation is over the random batch samples and Anf( 11') = f(2",119) 11',11 22220 0.
Due to space limitations, further details and the complete proof are deferred to Appendix E The above theorem shows that the gradient w.r.t. the model parameters & and the improvement over the mixing coefficients II becomes small we increase the round T, thus converging to a stationary point of the log- ligeliiel objective f.
3.3 COMMUNICATION EFFICIENT PROTOCOL
So far, we have discussed how FedeRiCo works in the all-to-all communication setting. In prac- tice, FedeRiCo does not require excessive model transmission, and this subsection discusses several practical considerations to ensure communication efficiency. Specifically, we tackle the bottlenecks in both the E-step (3) and the M-step (4) since they require joint information of all models 4. Due to space constraint, the pseudocode is provided in Algorithm]] Appendix B
E-step For client i, the key missing quantity to compute the loss ((1)) likelihood p(D, = j; ((t-1)), of other clients' models Φy.j Since the models & are being updated slowly, one can expect that ((1)) will not be significantly different from the loss {()) (-2)) of the previous iteration. Therefore, each client can maintain a list of losses for all the clients, sample a subset of clients in each round using a sampling scheme S (e.g.. e-greedy sampling as discussed later), and only update the losses of the chosen clients.
(3 without all-to-all communication is
M-step To clearly see how + is updated in the M-step, let's focus on the update to specific client's model Φi. According to (5) and (6). the update to Φi is given by
12
(x w1
Note that the aggregation is based on wi instead of w Intuitively, this suggests d, should be updated based on how the model is being used by other clients rather than how client i itself uses it. If Φ; does not appear be useful to all clients, i.e. w (t) o, Vj. it does not get updated. Therefore, whenever client is sampled by another client j using the sampling scheme S, it will send Φb to and receives the gradient update gij := w.; = Σn Vo,e (xx,,x (x)),, (i) from client j. One
(t)
issue here is that giy is governed by wis which could be arbitrarily small, leading to no effective update to Φ; We will show how this can be addressed by using an e-greedy sampling scheme.
Sampling scheme S We deploy e-greedy scheme where, in each round, each client uniformly samples clients with probability e E [0, 1] and samples the client(s) with the highest posterior(s) oth- erwise. This allows a trade off between emphasizing gradient updates from high-performing clients
5

---

Under review as a conference paper at ICLR 2023
(small e), versus receiving updates from clients uniformly to find potential collaborators (large e). The number M of sampled clients (neighbors) per round and e can be tuned based on the specific problem instance. We will show the effect of varying the hyperparameters in the experiments.
Tracking the losses for the posterior The final practical consideration is the computation of
(! the posterior wi}- From the E-step (3) and the M-step 44) one can see that wij is the softmax
transformation of the negative accumulative loss [ 21=1 (-1 p(T) "i over rounds (see Appendix A for derivation). However, the accumulative loss can be sensitive to noise and initialization. If one of the models, say 6 performs slightly better than other models for client i at the beginning of training, then client is likely to sample Φ more frequently, thus enforcing the use of Φj even when other better models exist. To address this, we instead keep track of the exponential moving average of the loss with a momentum parameter 3 E [0. 1), D = (1 -1--pt -) + B1 and compute u
using D.p This encourages clients to seek new collaborators rather than focusing on existing ones.
4 EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
We conduct a range of experiments to evaluate the performance of our proposed FedeRiCo with multiple datasets. Additional experiment details and results can be found in Appendix D
Datasets We compare different methods on several real-world datasets. We evaluate on cass-claasifiiattin tasks with the CIFAR-10, CIFAR- 100 (Krizhevsky et al., 2009). and Office- Homd (Venkateswara et al. 2017) datasets. Particularly, we consider a non-III data partition among clients by first splitting data by labels into several groups with disjoint label sets. Each group is considered a distribution, and each client samples from one distribution to form its local data. For each client, we randomly divide the local data into 80% training data and 20% test data.
Baseline methods We compare our FedeRiCo to several federated learning baselines. Fe- 2017) trains single global model for every client. We also compare to other
dAvg (McMahan personalized FL approaches including FedAvg with local tuning (FedAvg+) (Jiang et al. 2019), Clustered FL (Sattler al. 2020). FedEM (Marfoq al., 2021 FedFomo (Zhang et al. 2021 , as well as local training baseline. All accuracy results are reported in mean and standard deviation across different random data split and random training seeds. Unless specified otherwise, we use neighbors with 0.3 and momentum B = 0.6 as the default hyperparamters for FedeRiCo in all experiments. For FedEM, we use components, which provides sufficient capacity to accommo- date different numbers of label groups (or data distributions). For FedFomo, we hold out 20% of
the training data for client weight calculations. For FedAvg+, we follow Marfoq et al. 2021 and update the local model with epoch of local training.
Training settings For all models, we use the Adam optimizer with learning rate 0.01. CIFAR experiments 150 rounds of training, while Office-Home experiments use 400 rounds. CIFAR- 10 results are reported across 5 different data splits and 3 different training seeds for each data split. CIFAR-I00 and Office-Home results are reported across 3 different data splits with a different training seed for each split.
4.2 PERFORMANCE COMPARISON
The performance of each FL method is shown in Table [ Following the settings introduced by Marfoq et al. (2021), each client evaluated on its own local testing data and the average accuracies weighted by local dataset sizes are reported. We observe that FedeRiCo has the best performance across all datasets and number of data distributions, Here, local training can be seen as an indicator to assess if other methods benefit from client collaboration as local training has no collaboration at all. We observe that our proposed FedeRiCo the only method that consistently outperforms local training, meaning that FedeRiCo is the only method that consistently encourages effective
This dataset has been made publically available for rescarch purposes only. We use implementations from https: //tp:://iiii httphhw..o.ooooooooo////ea////ee for Clustered FL. and Fe-
dEM

---

Under review as a conference paper at ICLR 2023
Table I: Accuracy (in percentage) with different number of data distributions. Best results in bold.
CIFAR-10 10 # of distributions
CIFAR-100 distributions
Office-Home # of distributions
Method
2
3
4
4
2
3
FodAvg
11.44 1:
11.73
13.93
2:304
21.28
17:41
18.36
66.58
1
53.36
4-21
51.25 4:37
FodAvg*
12.45 2:15
29.86 a: 17st
45.65
29.95 1:
35.33 am
36.17
2:10
80.21
81.88 100
84.50
11
Local Training
40.09
55.27
69.03
16.60
25.99
31.05
76.76
83.30
88.05
Clustered FL
11.50 3.5
15.24 8:5
16.43
20.93
155
23.15
TM
15.15 c.0
66.58 LM
53.36 +-22
51.25 4:7
FedEM
41.21
55.08 +:7
63.61
26.25
24.11 =
19.23
22.59
28.72
22.46 339
FodFomo
42.24
59.45
71.05
1/30
12.15
1:130
20.49
2:300
24.53
78.61 1:17
82.57
1:04
87.86 2.0
FedeRiCa
56.61 3.5
69.76
78.22
1:00
30.95
39.19
41.41
1000
83.56
1:3
90.28
2:075
93.76
2012
client collaborations. Notably, both FedEM and FedFomo performs comparably well to FedeRiCo on CIFAR- 10 but worse when the dataset becomes more complex like CIFAR-100. This indicates that building the right collaborations among clients becomes a harder problem for more complex datasets. Moreover, FedEM can become worse as the number of distributions increases, even worse than local training, showing that is increasingly hard for clients to participate effectively under the FedEM framework for complex problems with more data distributions.
In addition, Clustered FL has similar performance FedAvg, indicating that is hard for Clustered FL to split into the right clusters. In Clustered FL (Sattler et al. (020), every client starts in the same cluster and cluster split only happens when the FL objective close a stationary point, i.c. the norm of averaged gradient update from all clients inside the cluster is small. Therefore, in non- i.i.d setting like ours, the averaged gradient update might always be noisy and large, as clients with different distributions pushing diverse updates to the clustered model. As a result, the cluster splitting rarely happens which makes clustered FL more like FedAvg.
4.3 CLIENT COLLABORATION
Client 0 (cistribution B)
Client
Icistribution
Client 2 Idistribution 3)
Client ldistribution 1)
9.5
9.3
Allatt
Clteee Maighe
9.6
clipst
cllatt
Clipst
cliant
0.1
Cliest
<llas
9.8
9.8
Bouses
Rounds
Raands
Roardr
Cliont (cistribution 1)
Client ldistribution
Client [distribution 3)
Client Isistribution
1.8
0.7
0.3
0.2
0.5
(Uiert N0.3115
0.3
C.Sort Moogats
0.0
0.3
0.1
0.1
0.3
0.4
Figure 3: Client weights over time of FedeRiCo with CIFAR 100 data and four different client dis- tributions. Clients are color coded by their private data's distribution.
In this section, we investigate client collaboration by plotting the personalized client weights wif) of FedeRiCo over training. With different client data distributions, we show that FedeRiCo can assign more weight to clients from the same distribution. As shown in Fig. 3 we observe that clients with similar distributions collaborate make the final predictions. For example, clients 3, 4 and 7 use mixture of predictions from each other (in light blue) whereas client 0 only uses itself for prediction since it is the only client coming from distribution (in dark blue) in this particular random split.
On the contrary, as shown in Fig. even with 4 components, FedEM fails to use all of them for predictions for the 4 different data distributions. In fact, clients 2, 3, 4, 6 and 7 coming from two different distributions are using only the model of component for prediction, whereas component O is never used by any client. Based this, we find FedeRiCo better encourages the clients to
7

---

Under review as a conference paper at ICLR 2023
Client (distribution B)
Client Idistribution
Client ldistribution 3)
Client ldistribution
(lieet WO10HT
0.2
0.3
ClSort WOLggnT
0.2
0.3
0.5
0.5
compase1t
comporeet
0.2
0.2
0.2
0.2
4.3
0.4
Idistribution 1)
Idistribution 31
Clsent Ieistribution
9.3
9.4
9.2
Cllent Vc ghn
C'lert NC Chn
9.4
0.4
0.2
9.2
30
L00
120
102
120
30
120
120
0
200
159
Figure 4: Component weights over training for FedEM with 4 components, on CIFAR100 data with 4 different client distributions. Clients are color coded by their private data's distribution.
collaborate with other similar clients and less with different clients. Each client can collaborate much little they need. Additionally, since all the non-similar clients have weight of (almost) O, each client only needs a few models from their collaborators for prediction.
4.4
EFFECT OF USING EXPONENTIAL MOVING AVERAGE LOSS
client
client
Client
client 1
leights
1.0
1.0
C'Luutt Weiight Beigghs
Heilghts
el 2llent
r1 lent
9.2
0.5
Client
Client
Client
0.5
0.1
Q.1
0.0
0.0
0.0
0.0
30
109
150
0
50
100
150
50
200
130
5B
360
150
Rounds
Rounds
Rounds
Rounds
(a) Client weights with accumulative loss.
(b) Client weights with exponential moving average.
Figure 5: Effect client weights with different implementations. The client weights CIFAR-10
with 2 different client distributions are reported.
Here, we visualize the effect of using the exponential moving average loss by plotting client weights with both accumulative loss and exponential moving average loss in Fig. We observe that with the accumulative loss in Fig. [a| the client weights quickly converge to one-hot, while with the exponential moving average loss in Fig. 5bl the client weights are more distributed to similar clients. This corresponds to expectation stated in Section 3.3 the clients using exponential moving average loss are expected to seek for more collaboration compared to using accumulative loss.
4.5 HYPERPARAMETER SENSITIVITY
In this section, we explore the effect of hyperparameters of our proposed FedeRiCo.
Effect of c-greedy sampling Here we show the effect of different values. Recall that each client deploys an e-greedy selection strategy. The smaller the value of e, the more greedy the client is in selecting the most relevant collaborators with high weights, leading to less exploration. Fig. ba shows the accuracy along with training rounds with different e values on the Office-Home dataset. One can see that there is trade-off between exploration and exploitation. If is too high (e.g., e 1, uniform sampling), then estimates of the likelihoods/osses are more accurate. However, some gradient updates will vanish because the client weight is close to zero (see Section 3.3), resulting
We used uniform sampling for Fig. 5a (e- 1) as most of the client weights after few rounds.
8

---

Under review as a conference paper at ICLR 2023
0.60
3.8
6.85
8-88
0.
R-3
He3
0.3
0.E
0.9
150
206
24
005
140
170
0:10
180
150
700
750
75a
Roandd
Thuanda
(a) for sampling.
(b) # of sampled neighbors.
(c) Momentum 8.
Figure 6: Test accuracy with different hyperparameters.
in slow convergence. On the other hand, if e is too small, the client may miss some important collaborators due to lack of exploration. As a result, we use a moderate e = 0.3 in all experiments.
Effect of number of sampled neighbors We plot accuracy with number of neighbors M
{0, 1,3, 5, } on CIFARI00 with 4 different client distributions, where M 0 is similar to Local Training as no collaboration happens. As shown in Fig. 6b) when the number neighbors increases, FedeRiCo converges more slowly as each client is receiving more updates on other client's 's models. While a smaller number of neighbors seems to have lower final aceuracy, we notice that even with
M 1, we still observe significant improvement compared to no collaboration. Therefore, we use
M = 3 neighbors in our experiments as it has reasonable performance and communication cost.
E103
100
p+26
1+02
Figure 7: Client weights (client 0) with different momentum values 3 on the client weight update. Effect of client weight momentum We plot the overall test accuracy of client on the Office-Home dataset with 4 different data distributions over 3 E {0.1, 0.3. 0.6, 0.9) in Fig. 6c and similarly for the client weights in Fig. 7 With smaller 3, as shown in Fig. 7 we observe a smoother update on the client weights, which is expected as the old tracking loss dominates the new one. Although various values produce similar final client weights, bigger 3 can lead to more drastic changes in early training. However. one shouldn't 't pick a very small just because can produce smoother weights. As shown in Fig, 66c the algorithm may converge more slowly with smaller 3. Therefore, we use 3 0.6 as encourages smoother updates and also maintains good convergence speed.
CONCLUSION AND DISCUSSION
In this paper, we proposed FedeRiCo, novel framework for decentralized and personalized FL. derived from EM for non-i.i.d client data. We evaluated FedeRiCo across different datasets and demonstrated that FedeRiCo outperforms multiple existing personalized FL baselines and encour- ages clients to collaborate with similar clients, i.e., the right collaborators.
Decentralized FL provides alternative architecture in the absence of a commonly trusted server. For example, CB-DEM (fforero008cconsennss)) studies the distributed EM algorithm for classifica- tion in wireless networks with local bridge sensors to ensure all sensors reach consensus. Compared to our decentralized communication-eeficco. protocol, CB-DEM requires global information every round for the consensus synchronization. While decentralized FL removes the risk of single point failure compared to centralized FL by using peer-to-peer communication, it also raises concerns about security risks with the absence of mutually trusted central server. Therefore, a promising direction is to incorporate trust mechanisms long with decentralization (Kairouz al.) 2019), such as blockchain frameworks (Qin et al. 2022). Additionally, FL schemes do not provide an explicit guarantee that private information will not be leaked. However, most FL frameworks, including ours, are compatible with privacy-preserving training techniques such as differential privacy, which is another promising and interesting research direction (Wei al. 2020 Truex et al. 2020).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ruizhao Zhu, Ramon Matas, Matthew Mattina, Paul What- mough, and Venkatesh Saligrama. Debiasing model updates for improving personalized federated training. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2021. UtL.htttp:://prreeeeiini
ml mlr.press/
Mohammed Adnan, Shivam Kalra, Jesse C. Cresswell, Graham W. Taylor, and Hamid R. Tizhoosh. Federated learning and differential privacy for medical image analysis. Scientific reports, 12(1): 1-10, 2022.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary, Fed- erated learning with personalization layers. arXiv preprint arXiv: 191200991,.00818, 2019.
Fengwen Chen, Guodong Long, Zonghan Wu, Tianyi Zhou, and Jing Jiang. Personalized federated learning with graph. arXiv preprint arXiv:2203.0000 2022.
Luca Corinzia, Ami Beuret, and Joachim M Buhmann. Variational federated multi-task learning. arXiv preprint
2019.
Sen Cui, Jian Liang, Weishen Pan, Kun Chen, Changshui Zhang, and Fei Wang. Collaboration org/ abs 210..01929
equilibrium in federated learning, 2021. URL
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, 2020. URL htt.::tttt:///aa
org/
13461
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learn- ing with theoretical guarantees: A model-agnostic meta-learning approach. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3557-3568. Curran Associates, Inc., 2020. URL
neuriissss...nnnssso...........s cc/ppper/222///11e pdf
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F Balcan, and H. Lin (eds.), Advances in Neural Information P Processing Systems, volume 33, pp. 19586-19597. Cur- ran Associates, Inc., 2020, URL/attpa://prrceeiiingg newripe cc/ tctppppp/p//.c///////cp/
f File://2cc0000001113333322222222
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models, 2020.
URL
http://wwwr..aa///......5 05516
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangehuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35-9, pp. 7865-7873, 2021.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning per- sonalization via model agnostic meta learning. arXiv preprint arXiv: 1909. /2488, 2019.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb. David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgüi,, Rasmus Pagh, Mariana Raykova, Hang Qi. Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich. Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang. Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learnin,, 2019.
10

---

Under review as a conference paper at ICLR 2023
Shivam Kalra, Junfeng Wen, Jesse C. Cresswell, Maksims Volkovs, and Hamid R. Tizhoosh. Proxyfl: Decentralized federated learning through proxy model sharing. arXiv preprint arXiv:2111.11343 2021.
Mikhail Khodak, Maria-Florine F Balcan, and Ameet S Talwalkar. Adaptive gradient-based meta- learning methods. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-uuc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL http::://ppoocedaagggeeoo// newriss.erpppp... http://pppeeeeaaoooooooc./////// EBaalodc66221055ee222001111eeee
pdf
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Viraj Kulkarni, Milind Kulkarni, and Aniruddha Pant. Survey of personalization techniques for federated learning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4,, pp. 794-797. IEEE, 2020.
Chengxi Li, Gang Li, and Pramod K. Varshney. Decentralized federated learning via mutual knowl- edge transfer. IEEE Internet of Things Journal, 2021a.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR, 2021b.
Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open banking. In Federated learning, pp. 240-254. Springer, 2020.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002. 2020.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems, 34, 2021.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiiee learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Zhen Qin, Shuiguang Deng. Xueqiang Yan, Schahram Dustdar, and Albert Y Zomaya. Secure and efficient decentralized federated learning with data representation protection. arXiv preprint 2rX.v:2205.10568. 2022.
Adam Sadilek. Luyang Liu, Dung Nguyen, Methun Kamruzzaman, Stylianos Serghiou, Benjamin Rader, Alex Ingerman, Stefan Mellem, Peter Kairouz, Elaine O. Nsoesie, Jamie MacFarlane, Anil Vullikanti, Madhav Marathe, Paul Eastham, John S. Brownstein, Blaise Aguera y. Arcas, Michael D. Howell, and John Hernandez. Privacy-first health research with federated learning. upj Digital Medicine, 4(1), September 2021. doi: 10. 10100//441746-00-004882222 URL https: 7/doi. 10. 1 1038/ /4//8/4440 641746-111 021 - 00489-2
Felix Sattler, Klaus-Rooe Robert Müller, and Wojeiech Samek. Clustered federated learning: Model-
agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neu- ral networks and learning systems, 32(3):370-3322, 2020.
Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Kun Kuang, Fei Wu, and Chao Wu. Federated mutual learning. arXiv preprint arXiv:2006. 16765, 2020.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning, Advances in neural information processing systems, 30, 2017.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en- velopes. Advances in Neural Information Processing Systems, 09:21394--1403 2020.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning.
IEEE Transactions Neural Networks and Learning Systems, 2022.
11

---

Under review as a conference paper at ICLR 2023
Stacey Truex, Ling Liu, Ka-Ho Chow, Mehmet Emre Gursoy, and Wenqi Wei. Ldp-fed: Federated learning with local differential privacy. In Proceedings of the Third ACM International Workshop Edge Systems, Analytics and Networking, pp. 61-66, 2020.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan, Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018-5027, 2017.
Kang Wei, Jun Li, Ming Ding. Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Trunsactions on Information Forensics and Security, 15:3454-3469, 2020.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized fed- erated learning with first order model optimization. In International Conference Learning Representations, 2021. URL /ttpttss///ppeeeeiiiiiieee net http://wwwetttoouuoomo/oooeooooe
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda. Damon Civin, and Vikas Chandra. Federated Learning with Non-IID Data, 2018. URL /tLLL//ppe//aarttt.rrrr///////00
12


Reviews:
[{"review_id": "YNJMMn9QdI", "reviewer": "Reviewer_Knpq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on personalized server free federated learning. A decentralized algorithm is proposed based on EM algorithm in which each device learns from the devices which has similar data distributions. Theoretical results are provided along with empirical justifications.", "summary_of_the_review": "Overall the paper is written well, and the idea is interesting, but it is not clear how much is the contribution in terms of algorithmic development as compared to existing decentralized EM algorithms in the literature. ", "strength_and_weaknesses": "*Strengths*\n- The paper is well written and easy to follow. \n- The problem is interesting and proposed solution is discussed theoretically as well as experimentally. \n\n*Weakness*\n\n- I am not sure if decentralized setting is well motivated for federated learning. This is specially be because the FL is specifically for larger number of devices, and implementing a decentralized algorithm is almost impossible in those settings. The authors should provide some specific use cases for the motivation.\n\n- The proposed framework seems to solve the issue of FedAvg, but it brings is new issue of requirement of agents communicating with each other which can be impractical to consider. Please provide some examples. \n\n- How to select the neighbors, what kind of communication graph is required for the algorithm to work? \n\n- What are the additional challenges in the theoretical analysis for the decentralized settings? For instance, what are the additional challenges need to be addressed as compared to decentralized EM algorithm in the following paper, and it's related papers. \n\nP. A. Forero, A. Cano and G. B. Giannakis, \"Consensus-based distributed expectation-maximization algorithm for density estimation and classification using wireless sensor networks,\" 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, 2008, pp. 1989-1992, doi: 10.1109/ICASSP.2008.4518028.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "SeUzmqazDZU", "reviewer": "Reviewer_Tffe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem that the global model may perform poorly for most clients, e.g. the data at different clients is very diverse. It addresses this problem by learning how data distributions of different clients mix together as the distribution at client i. It utilizes EM algorithm to keep updating the mixture weights and in the end each client makes predictions by a mixture of all clients' local models. Further, it proposes a version with efficient computation. In the experiments, the efficacy of proposed algorithms is justified.", "summary_of_the_review": "The problem and the proposed algorithm are well motivated. However, experiments at one reasonable setting are missed.", "strength_and_weaknesses": "**Strength**\n1. The problem is well motivated.\n2. This paper proposed a novel algorithm (to my best knowledge, though I am not very familiar with the literature) with the guarantee of convergence.\n\n**Weaknesses**\n1. According the description of dataset, is it possible that when clients are in the sample group, some data will simultaneously appear in the training data (of some client) and test data (of other client)?\n2. The paper seems to miss the experiment that different distributions have some overlap instead of disjoint support, e.g. client 1 with 80% label 0 and 20% label 1 and client 2 with 20% label 0 and 80% label 1? When this happens, for sanity check, would the mixtures of distributions by learned weights are the distributions of each client?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QMFisNwqI5", "reviewer": "Reviewer_xqTG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes FedeRiCo, a decentralized federated learning framework that allows clients to collaborate much or little with other participating clients to enhance the performance of the federated learned models. FedeRiCo leverages an EM-style optimization procedure to solve the global model weights and clients' collaboration weights concurrently. Both theoretical analysis and empirical results are shown to justify the effectiveness of the proposed FedeRiCo method.", "summary_of_the_review": "The paper seeks to tackle an interesting and important problem in federated learning to enhance collaboration among participating clients. The proposed method makes sense but is somewhat known. I am thus concerned about the novelty of the paper. Moreover, the privacy guarantees of the proposed method are not clear, which is a clear weakness.", "strength_and_weaknesses": "Strength:\n- the paper is well-written and the targetted problem is well motivated.  \n- both theoretical and empirical results are shown to demonstrate the effectiveness of the proposed method.  \n\nWeaknesses:\n- the proposed EM algorithm seems to be somewhat known, which limits the novelty of the proposed FedeRiCo method.  \n- the privacy guarantees for the decentralized federated learning framework are not clear. It seems the individual client's gradient can be seen by the other clients (in the small gradient steps in lieu of full optimization in each round). The authors are expected to discuss the privacy guarantees more explicitly.\n- the scale of the experiment is small, how does FedeRiCo perform on the ImageNet scale datasets?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JnURWkP6tl7", "reviewer": "Reviewer_r1RL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the data heterogeneity issue in Federated Learning (FL) by letting local clients leverage model parameters from nodes with similar data distributions. Each client approximates the data similarity of its neighbors through a weighing parameter $w_{ij}$ which is learned by EM optimization. In order to achieve this learning, each client needs to broadcast its learned model gradients to (selected) neighbors. The weighted ensemble of neighboring models is used for inference as the final model once upon training convergence.\n", "summary_of_the_review": "This paper proposed an FL approach that ensemble local models with similar data distributions to tackle data heterogeneity. This approach is technically sound but could raise potential concerns regarding communication efficiency and scalability.", "strength_and_weaknesses": "\nPros:\n+ Technical sound approach towards addressing data heterogeneity.\n+ Well-designed experiments that validate each component in the proposed approach.\n+ Comprehensive related work.\n\nCons:\n\nLowe efficiency in 1) communication during learning and 2) model inference after learning convergence. Even with the modifications that relax the all-to-all communication pre-requisite, each client still needs to download gradients from multiple neighbors. While FedAvg style algorithm is more communication efficient. The final model inference still requires a forward pass through multiple models. In general, I feel that this design is not practical when the model size is scaled up or when the number of users is high.\n\nMinor:\n\nAn algorithm box should be provided to summarize the proposed approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject