Personalized Federated Learning with First Order Model Optimization
Authors: Michael Zhang
Karan Sapra
Sanja Fidler
Serena Yeung
Jose M. Alvarez
Venue: ICLR 2021 Poster (2021)
Keywords: Federated learning
personalized learning

Abstract: While federated learning traditionally aims to train a single global model across decentralized local datasets, one model may not always be ideal for all participating clients. Here we propose an alternative, where each client only federates with other relevant clients to obtain a stronger model per client-specific objectives. To achieve this personalization, rather than computing a single model average with constant weights for the entire federation as in traditional FL, we efficiently calculate optimal weighted model combinations for each client, based on figuring out how much a client can benefit from another's model. We do not assume knowledge of any underlying data distributions or client similarities, and allow each client to optimize for arbitrary target distributions of interest, enabling greater flexibility for personalization. We evaluate and characterize our method on a variety of federated settings, datasets, and degrees of local data heterogeneity. Our method outperforms existing alternatives, while also enabling new features for personalized FL such as transfer outside of local data distributions.

Published as a conference paper at ICLR 2021
PERSONALIZED FEDERATED LEARNING WITH FIRST ORDER MODEL OPTIMIZATION
Michael Zhang'
Karan Sapra NVIDIA
Sanja Fidler
Stanford University
NVIDIA sfi sficlerennidiaa.cmm
mzhang@cs stanford.
ksapra@nnvidaa.cm
Serena Yeung
Jose M. Alvarez NVIDIA josea@nvidia.com
Stanford University
syyeung@stanford.edd
ABSTRACT
While federated learning traditionally aims to train a single global model across decentralized local datasets, one model may always be ideal for all participat- ing clients. Here we propose an alternative, where each client only federates with other relevant clients to obtain a stronger model per client-specifie objectives. To achieve this personalization, rather than computing single model average with constant weights for the entire federation as traditional FL, we efficiently calcu- late optimal weighted model combinations for each client, based on figuring out how much client can benefit from another's model. We do not assume knowl- edge of any underlying data distributions client similarities, and allow each client to optimize for arbitrary target distributions of interest, enabling greater flexibility for personalization. We evaluate and characterize method on riety of federated settings, datasets, and degrees local data heterogeneity. Our method outperforms existing alternatives, while also enabling new features for personalized FI. such as transfer outside of local data distributions.
INTRODUCTION
Federated learning (FL) has shown great promise in recent years for training single global model over decentralized data. While seminally motivated by effective inference on a general test set sim- ilar in distribution the decentralized data in aggregate (McMahan et al. 2016. Bonawitz al.) 2019), here we focus on federated learning from a client-centric or personalized perspective. We aim to enable stronger performance on personalized target distributions for each participating client. Such settings can be motivated by cross-silo FL, where clients are autonomous data vendors (e.g. hospitals managing patient data, or corporations carrying customer information) that wish collab- orate without sharing private data (Kairouz et al., 2019). Instead of merely being source of data and model training for the global server, clients can then take on more active role: their federated participation may be contingent on satisfying client-specific target tasks and distributions. A strong framework in practice would then flexibly accommodate these objectives, allowing clients to optimize for arbitrary distributions simultancously in single federation.
In this setting, FL's realistic lack of an independent and identically distributed (IID) data assumption across clients may be both burden and a blessing. Learning single global model across non-IID data batches can pose challenges such as non goaraneaddte convergence and model parameter diver- gence (Hsich al. 2019 Zhao et al. 2018 Li et al. 2020). Furthermore, trying fine-tune these global models may result in poor adaptation to local client test sets (Jiang et al.) (019).. However, the non-IID nature of each client's local data also provide useful signal for distinguishing their underlying local data distributions, without sharing any data. We leverage this signal to propose new framework for personalized FL. Instead of giving all clients the same global model average weighted by local training size as in prior work (McMahan et al. (016)), for each client we compute
"Corresponding author; work done while interning at NVIDIA

---

Published as a conference paper at ICLR 2021
weighted combination of the available models to best align with that client's interests, modeled by evaluation on a personalized target test distribution.
Key here is that after each federating round, we maintain the client-uploaded parameters individ- ually, allowing clients in the next round to download these copies independently each other. Each federated update is then two-step process: given local objective, clients (1) evaluate how well their received models perform on their target task and (2) use these respective performances to weight each model's parameters in personalized update. We show that this intuitive process can be thought of as particularly coarse version of popular iterative optimization algorithms such as SGD, where instead of directly accessing other clients' data points and iteratively training our model with the granularity of gradient decent, we limit ourselves to working with their uploaded models. We hence propose an efficient method to calculate these optimal combinations for each client, call- ing it FedFomo, as (1) each client's 's federated update is calculated with simple first-order model optimization approximating a personalized gradient step, and (2) draws inspiration from the "fear of missing out", every client no longer necessarily factoring in contributions from all active clients during each federation round. In other words, curiosity can kill the cat. Each model's personalized performance can be saved however by restricting unhelpful models from each federated update.
We evaluate our method on federated image classification and show that it outperforms other meth- ods in various non-IID scenarios. Furthermore, show that because compute federated updates directly with respect to client-specified local objectives, our framework can also optimize for out- of of-dittribution performance, where client's s target distributions are different from their local training ones. In contrast, prior work that personalized based on similarity to a client's 's own model parameters Mansour et al. 2020 Sattler al. 2020) restricts this optimization the local data distribution. We thus enable new Teatures in personalized FL, and empirically demonstrate up to 70% improvement in some settings, with larger gains as the number of clients or level of non-IIDness increases.
Our contributions
1. We propose flexible federated learning framework that allows clients to personalize to specific target data distributions irrespective of their available local training data.
2. Within this framework, we introduce method to efficiently calculate the optimal weighted combination of uploaded models as a personalized federated update
3. Our method strongly outperforms other methods in non-lID federated learning settings.
2 RELATED WORK
Federated Learning with Non-IID Data While fine-tuning a global model on a client's local data a natural strategy to personalize (Mansour et al. 2020; Wang al. 2019), prior work has shown that non-lID decentralized data can introduce challenges such as parameter divergence (Zhao et al., 2018], data distribution biases (Hsieh et al., 2019). and unguaranteed convergence Li al.] (2020). Several recent methods then try to improve the robustness of global models under heavily non non-III datasets. FedProx (Li et al.] 2020) adds proximal term the local training objective to keep updated parameter close to the original downloaded model. This serves to reduce potential weight divergence defined in IZhao et al. (2018), who instead allow clients to share small subsets of their data among each other. This effectively makes each client's 's local training set closer in distribution to the global test set. More recently, Hsu et al. (2019) propose to add momentum to the global model update in F FedAvgM to reduce the possibly harmful oscillations associated with averaging local models after several rounds of stochastic gradient descent for non-identically distributed data.
While these advances may make global model more robust across non-IlI local data, they do not directly address local-level data distribution performance relevant to individual clients. Diang eet al. (2019) argue this latter task may be more important in non-III FL settings, as local training data differences may suggest that only subset of all potential features are relevant to each client. Their target distributions may be fairly different from the global aggregate in highly personalized scenarios, with the resulting dataset shift difficult to handle with single model.
Personalized Federated Learning Given the challenges above, other approaches train multiple models or personalizing components to tackle multiple target distributions. Smmth et al. (2017) pro- pose multi-task learning for FL with MOCHA, a distributed MTL framework that frames clients as tasks and learns model per client. Mixture methods (Deng et al. 2020 (Hanzlyy & Richtárik)
2

---

Published as a conference paper at ICLR 2021
2020 Mansour et al., 2020) compute personalized combinations of model parameters from training both local models and the global model, while Peterson et al. (2019) ensure that this is done with local privacy guarantees. Liang et al. (2020) apply this mixing across network layers, with lower layers acting as local encoders that map a client's observed data to input for a globally shared clas- sifier. Rather than only mix with a shared global model, work allows for greater control and distinct mixing parameters with multiple local models. Fallah et al. (2020) instead optimize the global model for fast personalization through meta-learning, while T Dinh et al. (2020) train global and local models under regularization with Moreau envelopes. Alternatively, Clustered FL (Sattler al. 2020; Ghosh et al.) 2020, Briggs et al., 2020; Mansour et al.] (020) assumes that inherent partitions or data distributions exist behind clients' local data, and aim to cluster these partitions to federate within each cluster. Our work does not restrict which models are computed together, allow- ing clients to download suitable models independently. We also compute client-specific weighted averages for greater personalization. Finally, unlike prior work, we allow clients to receive person alized updates for target distributions different from their local training data.
FEDERATED FIRST ORDER MODEL OPTIMIZATION
We now present FedF omo, personalized FL framework to efficiently compute client-optimizii federated updates. We adopt the general structure of most FL methods, where we iteratively cycle between downloading model parameters from server to client, training the models locally on each client's 's data, and sending back the updated models for future rounds. However, as we do not compute a single global model, each federated download introduces two new steps: (1) figuring out which models to send to which clients, and (2) computing their personalized weighted combinations. We define our problem and describe how we accomplish (1) and (2) in the following sections.
Problem Definition and Notation Our work most naturally applies heterogeneous federated settings where participating clients are critically not restricted to single local training or target test distribution, and apriori we do not know anything about these distributions. To model this, let C be a population with [C] = K total clients, where each client G E C carries local data D; sampled from
some distribution D and local model parameters g(()) during any round t. Each c₁ also maintains some personalized objective or task Ti motivating their participation in the federation. We focus on supervised classification as universal task setting. Each client and task are then associated with
a test dataset D!stt ~ D We define each T, := min L(e D where [(0; D) O + R is the loss function associated with dataset D, and @ denotes the space of models possible with our presumed network architecture. We assume no knowledge regarding clients and their data distribu- tions, nor that test and local data belong to the same distribution. We aim to obtain the optimal set of model parameters (0)
00x (0"....,
arg min Eie[K] LT, (0:).
3.1 COMPUTING FEDERATED UPDATES WITH FOMO
Unlike previous work in federated learning, FedFome learns optimal combinations of the available server models for each participating client. To do so, we leverage information from clients in two different ways. First, we aim to directly optimize for each client's target objective. We assume that clients can distinguish between good and bad models on their target tasks, through the use of a labeled validation data split D D, in the client's local data. D should be similar in distribution to the target test dataset Disst The client can then evaluate any arbitrary model f2 on this validation set, and quantify the performance through the computed loss, denoted by L. (0,). Second, we directly leverage the potential heterogeneity among client models. (hhao et al. (2018) explore this phenomenon as a failure mode for traditional single model FL, where they show that diverging model weights come directly from local data heterogeneity. However, instead of combining these parameters into a single global model, we maintain the uploaded models individually as a means to preserve a model's potential contribution to another client. Critically, these two ideas together not only allow us to compute more personal model updates within non-lll) local data distributions, but also enable clients to optimize for data distributions different from their own local data's.
Federated learning an iterative local model update The central premise of our work stems from viewing each federated model download and subsequent changing of local model parameters-an optimization step towards some objective. In traditional FL, this objective in- volves performing well on the global population distribution, similar in representation to the union of all local datasets. Assuming N federating clients, we compute each global model G time
3

---

Published as a conference paper at ICLR 2021
(G()) -En=1 Wn, 0,2 where w,
D Eラ= NV
D If client es downloads this model,
200)) we can view this change to their local model as an update: ol(t+1) - -,,0 wn "0%" -6 since Σn Win = 1. This then updates client's current local model parameters in directions speci- fied by the weights wo and models [0,] } in the federation. A natural choice to optimize for the global target distribution sets Wi, above and in McMahan al. (2017). e.g. as an unbiased estimate of global model parameters. However, in our personalized scenario, we are more interested in comput- ing the update uniquely with respect to each client's 's target task. We then wish find the optimal weights w (W1,.... WN) that optimize for the client's 's objective, minimizing (6,))
Efficient personalization with FedFomo Intuitively, we wish to find models ( m E [M]] such that moving towards their parameters leads to better performance accordingly weight these 0 higher in model average. If a client carries a satisfying number of local data points associated with their target objective Li, then they could obtain reasonable model through local training alone, e.g. directly updating their model parameters through SGD:
target distribution. and
B + (((0)) (o
However, without this data, clients are more motivated federate. In doing they obtain useful updates, albeit in the more restricted form of fixed model parameters {0x E N). Then for personalized or non-IID target distributions, we can iteratively solve for the optimal combination of client models w* = arg min (f) by computing:
0 + gf(0) - Vv
where size-N vector of ones. Unfortunately, as the larger federated learning algorithm is already an iterative process with many rounds of communication, computing w" through Eq. 2 may be cumbersome. Worse, if the model averages are only computed server-side as in traditional FL, Eq. becomes prohibitively expensive in communication rounds (McMahan et al.) 2017).
Following this line of reasoning however, we thus derive an approximation of w" for any client:
Given previous local model parameters (((1-1), set of fellow federating models available to down- load {0t(() } and local client objective captured by Lis we propose weights of the form:
(,(0((t-1) (69())
Wn
110 (11
(3)
+ EnEINy Wn (0% (((t))((()11)) directly optimizes for client cs's objective up to a first-order approxxmatin the optimal wt. We default to the original parameters of !) if wn < 0 above, i.e. wn, = max max(wn, 0), and among positive w,, normalize to max(u,, get final weights = Σ.. max(w.,.0) to maintain w* E [0, 1] and En=1 W (0.
where the resulting federated update
(((()
We derive Eq. 3 as a first-order approximation w" in Appendix A.I Here we note that our formulation captures the intuition of federating with client models that perform better than our own model, e.g. have a smaller loss on L.. Moreso, we weigh models more heavily as this positive loss delta increases, or the distance between our current parameters and theirs decreases, in essence most heavily weighing the models that most efficiently improve our performance. We use local parameters (-1 directly compute how much we should factor in current parameters of(() which also helps prevent overfitting as (tt - ()( < causes "carly-stopping" at 0 (t-1))
Communication and bandwidth overhead Because the server can send multiple requested mod- els in one download to any client, we still maintain round of communication for model down- loads and one round for uploads in between E local training epochs. Furthermore, because w in Eq. 3 is simple to calculate, the actual model update can also happen client-side, keeping the total number of communications with T total training epochs at 27'
E J. as in FedAvg.
However FedFomo also needs to consider the additional bandwidth from downloading multiple models. While quantization and distillation (Chen et al., 2017 Hinton et al., 2013) Xu et al. 2018) can alleviate this, also avoid worst case N overhead with respect to the number of active clients
4

---

Published as a conference paper at ICLR 2021
N by restricting the number of models downloaded M. Whether we can achieve good personaliza- tion here involves figuring out which models benefit which clients, and our goal is then to send as many helpful models possible given limited bandwidth.
To do so, we invoke a sampling scheme where the likelihood of sending model f to client & re- lies on how well 8g performed regarding client e; target objective in previous rounds. Accord- ingly, we maintain an affinity matrix P composed of vectors p. = (pt.,.....PP,.A PP.,)) where pi.j measures the likelihood of sending 02 client C₁, and each round send the available uploaded models corresponding to the top M values according to each participating client's p. Initially we set P p= diag(1.... P 1), i.e. each model has an equal chance of being downloaded. Then during each federated update, we update f + w from Eq. [3 where w can now be negative. If N & K, we may benefit from additional exploration, and employ an e-greedy sampling strategy where instead of picking strictly in order of p. we have & chance to send a random model to the client. We investigate the robustness of Fed? omo to these parameters through ablations of and M in the next section.
4 EXPERIMENTS
Experimental Setup We consider two different scenarios for simulating non-identical data distri- butions across federating clients. First we evaluate with the pathological non-III setup in McManan et al. (2016), where each client is randomly assigned 2 classes among 10 total classes. We also use latent distribution non-IID setup, where we first partition our datasets based on feature and semantic similarity, and then sample from them to setup different local client data distributions. We use num- ber of distributions E {2, 3, 4, 5, 10} and report the average Earth Mover's 's Distance (EMD) between local client data and the total dataset across all clients to quantify non -on-IIDness. We evenly allocate clients among distributions and include further details in Appendix A.5] We evaluate under both setups with two FL scenarios: 15 and 100 clients with 100% and 10% participation respectively, re- porting final accuracy after training with E = 5 local epochs per round for 20 communication rounds in the former and 100 rounds in the latter. Based on prior work (McMahan et al. 2016 Liang al., 2020). we compare methods with the MNIST (LeCun et al., 1998), CIFAR-I0 (Krizhevsky et al. 2009 and CIFAR- 100 datasets. We use the same CNN architecture as in McMahan et al. (2016).
Federated Learning Baselines We compare FedF omo against methods broadly falling under two categories: they (1) propose modifications to train a single global model more robust to non-llD local datasets, or (2) aim to train more than one model or model component to personalize performance directly to client test sets. For (1), we consider F FedAvg, FedP and the 5% data-sharing strat- egy with F FedAvg, while in (2) we compare our method to MOCHA, LG-FedAvg, Per-FedAvg, pFedMe, Clustered Federated Learning (CFL), and a local training baseline. All accuracies are re- ported with mean and standard deviation over three runs, with local training epochs E = 5, the same number of communication rounds (20 for 15 clients, 100% participation; 100 for 100 clients, 10% participation) and learning rate 0.01 for MNIST, 0.1 for CIFAR-10). We implemented all results]]
Pathological Non-IID We follow precedent and report accuracy after assigning two classes out of the ten to each client for the pathological setting in Table I Across datasets and client setups, our proposed FedF omo strongly outperforms alternative methods in settings with larger number clients, and achieves competitive accuracy in the 15 client scenario. In the larger 100 client scenario, each individual client participates less frequently but also carries less local training data. Such settings motivate higher demand for efficient federated updates, as there are less training rounds for each client overall. Meanwhile, methods that try to train a single robust model perform with mixed success over the FedAvg bascline, and notably do not perform better than local training alone. Despite the competitive performance, we note that this pathological setting is not the most natural scenario to apply FedF ome. In particular when there are less clients, each client's 's target distribution carries only 2 random classes, there is no guarantee that any two clients share the same objective such that they can clearly benefit each other. With more clients however, we can also expect higher frequencies of target distribution overlap, and accordingly find that we outperform all other methods.
Latent Distribution Non-IID We next report how each FL method performs in the latent distri- bution setting in T with additional results in Fig. I Here we study the relative performance of
"LG-FeAvvg and MOCEA were implemented with code from git.uhbbccmm//lingg2999 and Per-F FedAdg were implemented with code from ggthhb..omm/Chaarieeiii/plllllll CFI, was implemend with code from hatthh/.ommmiiiissaisssseeeddaed
pFedMe
5

---

Published as a conference paper at ICLR 2021
MNIST
CIFAR-10
15 clients
100 clients
15 clients
100 clients
Local Training
99.62 + 0.21
94.25 4.28
92.73 0.87
85.42 + 4.06
FedAvg McMahan
2016
91.97 + 2.17
78.83 9.68
57.12 1.14
53.08 + 7.40
FedAvg
Data (Zhao el al. 2018
91.99 + 2.14
78.85 9.66
4 1.67
56.62 : 8.92
FedProx Li
al.. 2020
90.94 + 1.85
79.06 10.88
54.60 = 3.26
52.92 + 5.56
LG-FedAvg (Lrang et al. 2020
99.79 + 0.07
81.17
92.36 1.00
84.17 + 4.45
MOCHA (Smith et al. 2017)
94.74 2.27
5.85
93.85 + 2.04
76.09 + 8.49
Clustered FL (CFL) (Sattler et al. 2020
95.00 + 3.61
92.26
85.07 = 8.16
77.75 + 1.78
Per-FedAvg (Fallah ei al., 2020)
92.39 + 4.72
85.32 + 12.93
81.96 8.12
72.40 + 4.06
PFedMe (T Dinh el al. 2020)
97.70 # 1.26
88.40 + 10.86
83.85 = 5.11
71.75 + 6.78
Ours clients downloaded)
99.62 + 2.91
98.81 1.26
93.01 0.96
92.10 + 5.20
Ours (10 clients downloaded)
99.63 + 0.07
98.71 2.86
92.73 0.96
92.67 + 4.21
Table 1: Personalized FL accuracy with pathological non-IID splits, Best results in bold. FedF omo outperforms or is competitive with prior work across settings, especially with larger populations.
FedFomo across various levels of statistical heterogeneity, and again show that our method strongly outperforms others highly non-IlD settings. The performance gap widens as local datasets become more non-IID, where global FL methods may suffer more from combining increasingly divergent weights while also experiencing high target data distribution shift (quantified with higher EMD) due to local data heterogeneity. Sharing small amount of data among clients uniformly helps, does actively trying to reduce this divergence through FedP rox, but higher performance most convinc- ingly come from methods that do not rely single model. The opposite trend occurs with local training, as more distributions using the same 10 or 100 classes leads to smaller within-distributioon variance. Critically, FedF omo is competitive with local training in the most extreme non-IID case while strongly outperforming FedAvg, and outperforms both in moderately non-lID settings (EMD [1, 2)), suggesting that we can selectively leverage model updates that best fit client objec- tives to justify federating. When data is more IID, any individual client model may benefit another, and it becomes harder for selective update to beat general model average. FedF omo also out- performs sessonalziingcoompnees and multi-model approaches (MOCHA and LG-FedAvg), where regarding data heterogeneity we see similar but weaker and more stochastic trends performance.
CIFAR-10 Number of Latent Distributions (EMD)
Method
2 (1.05)
3 (1.4))
4 (1.28)
(2.80)
10 (2.70)
Local Training
60.03 + 9.22
66.61 + 9.90
69.12 + 12.07
76.52 + 11.46
92.64 + 7.32
FedAvg
38.92 1 11.88
21.56 1 9.14
22.34 1: 12.36
32.13 1 1.95
10.10 1: 3.65
FedAvg Data
53.43 + 2.89
33.87 2.53
65.73 1 1.07
63.32 0.49
41.61 0.92
FedProx
66.42 1: 1.79
31.38 1 2.54
50.61 1 1.53
48.20 0.14
13.41 1 3.39
LG-FedAvg
70.87 1.12
67.25 1.97
63.64 2.52
94.42 - 1.25
MOCHA
83.79 1.54
73.68 + 2.80
71.23 4.08
69.02 + 2.93
94.28 + 0.81
CFL
72.58 = 10.30
75.69 1.11
78.31 12.90
70.04 13.56
85.22 - 6.70
Per-FedAvg
63.85 5.11
7
76.61 + 6.65
93.97 2.34
PFedMe
49.87 - 3.16
66.95 + 10.65
69.00 1.97
78.66 + 3.72
94.57 + 1.95
Ours (n=5)
77.823 2.24
82.38 + 0.66
84.45 0.21
85.050 0.13
95.55 + 0.26
Ours (n=10)
79.59 + 0.34
83.66 0.72
84.35 + 0.38
85.534 + 0.53
95.55 + 0.06
Table 2: In-distribution federated accuracy with 15 clients, 100% participation, across heterogeneity levels (measured by EMD). FedF omo performs better than or competitively with existing methods.
Personalized model weighting We next investigate FedFomo's personalization by learning op- timal client to client weights overtime, visualizing P during training in Fig. 2 We depict clients with the same local data distributions next to each other (e.g. clients 0, 1, belong to distribution 0). Given the initial diagonal P depicting equal weighting for all other clients, we hope FedF omo increases the weights of clients that belong to the same distribution, discovering the underlying par- titions without knowledge of client datasets. In Fig 2 we show this for the 15 client 5 non-IID latent distribution setting on CIFAR-10 10 with 5 clients downloaded and Æ = 0.3 (lighter = higher weight). These default parameters adjust well to settings with more total clients (Fig 2p), and when we change the number of latent distributions (and IID-ness) in the federation (Fig 2+).

---

Published as a conference paper at ICLR 2021
CEAR10. 100
CFFR100 100 cients
100
fccureyy
60
Accuras
40
10 231
(20.04
(1,455)
(1.01) 5 (2.50)
10 (2.71)
10.59
211.44)
(1.00)
(2.02)
(2.19)
10 12.48)
Number Distributiors (END)
Number Distributions (EMD)
Local FedAvg
FedAvg + shared
LG-FedAvg
Ours (n=5)
FedProx
MOCHA
Ours (n=10)
Figure I: Classification accuracy of FL frameworks with 100 clients over latent distributions.
boud
Round
botal charts
discovery over
(b)
(c) Support
Figure 2: FedF omo client-to-client weights over time and across different FL settings, We reliably upweight clients with the same training and target distributions.
Exploration with and number of models downloaded M To further understand FedFomo behavior and convergence in non-IID personalized settings with respect limited download band- width capability, we conduct an ablation over e and M, reporting results on the 15 client CIFAR-10 5-distribution setting in Fig- loer 100 training epochs. We did not find consistent correlation be- tween and model performance, although this is tied to M inherently (expecting reduced variance with higher M). With fixed e, greater M led to higher performance, as we can evaluate more models and identify the "correct" model-client assignments earlier on.
E 0.1
6 0.3
0.5
0.9
336-
CAB
385
M=12
Mell
N:9
030
6055
1013
641
H+C
Hw3
5000
06
01
11
14
600
600
41EE
31
Figure 3: Ablations over e-greedy exploration and number of models downloaded on CIFAR-10.
nua--f-oocllddittibbttio personalization We now consider the non-lID federated setting where each client optimizes for target distributions not the same as their local data distribution. Here, although client may sufficiently train an adequate model for one domain, it has another target data distribution of interest with hard to access relevant data. For example, in a self-driving scenario, client may not have enough data for certain classes due to geographical constraints, motivating the need to leverage info from others. To simulate this scenario, after organizing data into latent distributions, randomly shuffle (Dval, Diett) pair among clients. We test the CIFAR-10 10 and CIFAR-10 datasets with clients, full participation, and 5 latent distributions, repeating the shuffling five times, and report mean accuracy over all clients.
7

---

Published as a conference paper at ICLR 2021
CIFAR30, 15 diants
150
80
CIFAR-10
CIFAR-100
Local Training
20.39 3.36
7.40 + 1.31
FedAvg
23.11 2.51
13.06 + 1.48
(2.24)
4 (2.36)
: 12-781
18 (2.0)
FedAvg Data
42.15 + 2.16
21.98 + 4.98
Number stribaaions (END)
FedProx
39.79 = 8.57
11.39 2.85
Fodkig shared
LG-FedAvg
38.95 = 1.85
18.50 1.10
MOCHA
(n=10)
Clenn welggtt
Cleent 33 selghts
Clent weights
MOCHA
30.80 2.60
13.73 2.83
Clustered FL
29.73
19.75 + 1.58
00:
Per-FedAvg
39.8 = 5.38
21.30 + 1.35
PFedMe
43.7 + 7.27
25.41 2.33
-10
-20
Ours (n=S) Ours (n=10)
64.06 1 2.80
34.43 + 1.48
20 30 15
10 20 2 46
20 25 20 10
63.98 = 1.81
40.94 - 1.62
Figure 4: Top Personalization on target distri- bution # that of local training data. Bottom FedF omo upweights other clients with local data ~ target distribution (5 latent non-IlI dist.)
Table 3: Out-of-client distribution evaluation
with 5 latent distributions and 15 clients. FedFomo outperforms all alternatives in various datasets.
As shown in Fig. 4 and Table 3, our method consistently strongly outperforms alternatives in both non-IID CIFAR-IO and CIFAR-100 federated settings. We compare methods using the same train and test splits randomly shuffled between clients, such that through shuffling we encounter poten- tially large amounts of data variation between client's training data and its test set. This then supports the validity of the validation split and downloaded model evaluation components in our method to uniquely optimize for arbitrary data distributions different from a client's local training data. All methods other than ours are unable to convincingly handle optimizing for a target distribu- tion that is different from the client's initially assigned local training data. Sharing data expectedly stands out among other methods that do not directly optimize for a client's 's objective, as each client then increases the label representation overlap between its train and test sets. We note that in the 2- 2-distribution setting. where each client's 's training data consists of 5 classes on average, the higher performance of other methods may likely be a result of our simulation, where with only two distri- butions to shuffle between it is more likely that more clients end up with the same test distribution.
To shed further light on FedF 's performance, we visualize how client weights evolve over time in this setting (Fig. | bottom), where to effectively personalize for one client, FedF omo should specifically increase the weights for the other clients belonging to the original client's target distri- bution. Furthermore, in the optimal scenario we should upweight all clients with this distribution while downweighting the rest. Here we show that this indeed seems to be the case, denoting local training distributions with color. We depict clients 12, 13, and 14, which all carry the same local data distribution, but 13 and 14 optimize for out-of-local distributions. In all cases, F FddF omo up- weights clients specifically carrying the same data distribution, such that while with shuffling we do not know apriori 13 and 14's target distributions, Fed7 omo discovers these and who should federate with whom in this setting as well. We include similar plots for all clients in Appendix A.2 (Fig.
Locally Private FedFomo While we can implement FedF omo such that downloaded model pa- rameters are inaccessible and any identifying connections between clients and their uploaded mod- els are removed to subsequently preserve anonymity, unique real world privacy concerns may rise when sharing individual model parameters. Accordingly, we now address training redF'omo under (Dor. (e, -)ddifferettial privacy (DP). Dwork et al. (2014) present further details, but briefly DP ensures
that given two near identical datasets, the probability that querying one produces a result is nearly the same as querying the other (under control by and 8). Particularly useful here are DP's compos- ability and robustness to post-processing which ensure that if we train model parameters 0 to satisfy DP, then any function on 0 is also DP. We then perform local training with DP-SGD (Abadi et al.) 2016) for a DP variant of FedF omo, which adds a tunable amount of Gaussian noise to each gradi- ent and reduces the connection between model update and individual samples in the local training
8

---

Published as a conference paper at ICLR 2021
CIFAR-10
CIFAR-100
Method
Accuracy
Accuracy
FedAvg
1 10-
0
19.37 1.42
5.09 1 0.38
FedAvg
x 10-3
10.26 + 0.21
17.60 1.64
8.20 + 0.69
5.05 0.31
FedAvg
x 10-5
2
3.57 + 0.08
16.19 1.62
2.33 + 0.21
4.33 + 0.27
Ours
1 10-5
0
71.56 + 1.20
26.76 + 1.20
Ours
1 10
6.89
0.13
71.28 + 1.06
8.20 + 0.69
26.14 + 1.05
Ours
1 10-5
2
1.70 0.04
65.97 + 0.95
1.71 + 0.15
15.95 + 0.94
Table 4: In-distribution classification with differentially private federated learning. With DP-SGD, FedFomo maintains high personalization accuracy with reasonable privacy guarantees.
Pederat Round
Federation Bound
Fecteration
0 La A
12 a
Cedioro 9-33
Feaforno 202
radiay 9:00
Figure 5: Left: Even with privacy-preserving updates, FedFomo still uncovers the underlying data
distributions at large. Right We gain privacy benefits without substantial drop in performance.
data. More noise makes models more private at the cost of performance, and here we investigate if FedFome retains its performance with increased privacy under noisy local updates.
We consider the in-distribution personalization task with 5 latent non-IID distributions from the CIFAR-10 and CIFAR-100 100 datasets, with 15 clients and full participation at each round, and com- pare FedF omo against FedAvg with varying levels of Gaussian noise, specified by a. With other parameters fixed, higher should enable more noisy updates and greater privacy (lower E), the potential cost of performance. At fixed 8, we wish to obtain high classification accuracy and low s. We the Opacus Pytorch library[for DP-SGD, and as baselines the library's provided SGD optimizer with a = 0. For DP runs, we set 5 = x 10-5 < 3 x 10-4, the inverse of the average number of local data points of each client, to maintain reasonable privacy.
FedF omo and FedAvg with
In Table FedF omo able to retain a sizeable improvement over FedAvg, even against the non- DP FedAvg, and does with minimal e. As expected, greater leads to improved privacy (lower s) at the cost of decreased performance. Additionally, in Fig. we show that even with noisy gradients protect individual data point privacy, FedF omo maintains its ability discover the larger latent distributions among local data (albeit with more noise initially). Most importantly, despite adding noise that could potentially derail our federated update, we are able to substantially reduce privacy violation risks under (e, difffefrrntt privacy while maintaining strong performance.
5 CONCLUSION
We present FedFomo, a flexible personalized FL framework that achieves strong performance across various non-lID settings, and uniquely enables clients to also optimize for target distribu- tions distinct from their local training data. To do so, we capture the intuition that clients should download personalized weighted combinations of other models based how suitable they are to- wards the client's own target objective, and propose a method to efficiently calculate such optimal combinations by downloading individual models in lieu of previously used model averages. Beyond outperforming alternative personalized FL methods, we empirically show that FedFomo able to discover the underlying local client data distributions, and for each client specifically upweights the other models trained on data most aligned to the client's target objective. We finally explore how our method behaves with additional privacy guarantees, and show that we can still preserve the core functionality of FedF omo and maintain strong personalization in federated settings.
ggithhh.ommpyyrocchhpppuuus

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308 -18--18, 2016.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning at scale: System design. arXiv preprint arXiv:/902.0///
2019.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. arXiv preprint arXiv:2004. /791, 2020.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances Neural Information Process- ing Systems, pp- 742-751, 2017.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003. 13461, 2020.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):21 (5344)211-407, 1-007, 2014.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint #rXi::2002.07948. 2020.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ranchandran. An efficient framework for clustered federated learning. arXiv preprint arXiv:2006. 040888 2020.
Filip Hanzely and Peter Richtárik. Federated learning of mixture of global and local models. arXiv preprint arXiv:2002.05516 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
Thomas Hofmann. Latent semantic models for collaborative filtering, ACM Transactions Infor- mation Systems (TOIS), 22(1):89-115, 2004.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B Gibbons. The non-iid data quagmire of decentralized machine learning. arXiv preprint arXiv:1910.00133 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.063: 2019.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning per- sonalization via model agnostic meta learning, arXiv preprint arXiv: 12488, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977. 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2324 1998.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429 450, 2020.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis- Philipp Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01522; 2020.
10

---

Published as a conference paper at ICLR 2021
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effcceent learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282, PMLR, 2017.
H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas. Federated learning of deep networks using model averaging. CoRR, abs/1602.05629, 2016. URLhttt://arxiv,
brg/abs/1602.. 05629
Daniel Peterson, Pallika Kanani, and Virendra Marathe. Private federated learning with domain adaptation. arXiv preprint 1rXiv:1912.06733. 2019.
Felix Sattler, Klaus-Robert Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neu- ral Networks and Learning Systems, 2020.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. CoRR, abs/1705. 10467, 2017. http:///a//xx wrg. aro/////////.. 10467
Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33, 2020.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint 102222, 2019.
Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong. Deep neural network compression with single and multiple level quantization. arXiv preprint
2018.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. CoRR, abs/1 805800582 2018. URL 1806. 00582
org/abs)

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 1
DERIVING THE FOMO UPDATE
Recall that we can view each federated model download can viewed as an iterative update,
N Wn
(0
(4)
72
where given client's s current parameters (t) the weights (W1,.... wx) in conjunction with the model deltas " -8((t) determine how much each client should move its local model parameters optimize for some objective. Unlike more common methods in machine learning such gradient descent, the paths we can take to get to this objective are restricted by the fixed model parameters (0% available to time t. While traditional FL methods presume this objective to be global test set performance, from client-centric perspective we should able to set this objective with respect any dataset target distribution of interest.
We then view this problem as a constrained optimization problem where ΣnEIN] Wi = 1. As a small discrepancy, if E [N], then to also calculate w; or how much client should weigh its own model in the federated update directly, we reparameterize Eq. as an update from a version of the local model prior to its current state, e.g.
g((+1)) 00 (((1-1)
+
Wn (0
0g 12((-11)
(5)
and again we have current budget of to allocate to all weights w. Additionally, to go along with Eq. [ we deviate a bit from the optimal f + term in Eq. and set
( = - + 0000 al (o
(6)
There is then parallel structure between Eq. 5 and Eq. 6 and we proceed by trying to find optimal w that would let our update Fllooeey approximate optimal update taking the gradient V V.- We accordingly note the equivalence from Eq. 5 and Eq. where for desired Wn.
Wn (0%)) 0((--1)) --1 T V (,
or in matrix form:
(0 0((e-1))
C.(0
Jur
(8)
WN
(0
Twn Li(0 (11-))))
Then for each weight w., we solve for its optimal value by equating the left and right-hand corre- sponding vector components. We do so by deriving first order approximation of à First, for each Wn, we define the function:
Dwn. L. (( 1).
(n((w = wn, gc(t) + +(1+ ww) -
(9)
as an alternate parameterization of the 0's as functions of weights. We can see that for all n E [N],
(0) =00t 0.(e--1)
a (((0((1-1))
a
Own (0()
Own
12

---

Published as a conference paper at ICLR 2021
Then using first-order Taylor series approximation, we also note that
Ci(pn(w')) x2 (i())((0)) + Own (0)) -0)
(10)
such that our initial point (((1-1), we can approximate the derivative when w' as:
a Ben
L, (4pn (0))
① (0)) (1(Pn(1)) (o(()(())
Own
(11)
Own (.(0ttt11) = L.(0 - L.(0,
following from Eq. 9 Then for each vector element in Eq. indexed by n [NV] we can plug in the corresponding partial derivative from Eq. MT and solve for the corresponding wn to get
(0%%)) (0((1-11)
Wn = a
|10 -0((11111
(12)
as the individual weight for client G to weight model 0n in its federated update.
We arrive at Eq. by distributing the negative a to capture the right direction in each update, but also note that the constant cancels out because we normalize ensure our weights sum to such that the weights w,,, that we actually use in practice are given by:
max Wnn 0)
W
(13)
m= max Wn 0)
A.2 ADDITIONAL LATENT DISTRIBUTION NON-IID EXPERIMENTS
CIFAR-100 Here show results on the latent non-IID in-distribution personalization setup for the CIFAR-100 dataset. As in the CIFAR-10 10 setting, we compare FedF omo against various recent alternative methods when personalizing to a target distribution that is the same as the client's local training data, and report accuracy as an average over all client runs. We also show results partitioning the CIFAR- 100 dataset into increasing number of data distributions for clients total, and report the increasing EMD in parentheses. In Table 5, F FedF omo consistently outperforms all alternatives with more non-IID data across different clients. We note similar patterns to that of the CIFAR- dataset, where method is more competitive when client data is more similar (lower EMD, number of distributions), but handily outperforms others as we increase this statistical label heterogeneity.
CIFAR 100 Number of Latent Distributions (EMD)
Method
(1.58)
(1.96)
4 (2.21)
5 (2.41)
(2.71)
Local Training
23.36 + 0.33
23.89 2.03
28.44 1.97
23.11 9.44
41.26 1.31
1.31
FedAvg
28.01 + 0.74
18.95 0.22
25.69 0.41
21.26 0.89
18.19 + 0.987
FedAvg + Data
28.12 + 0.63
19.01 0.27
2
25.18 + 0.39
18.23 + 0.835
FedProx
28.21 +: 0.79
$7.00000000
25.79 0.05
24.93 + 0.38
18.18 0.82
LG-FedAvg
26.97 1: 7.52
24.69 4.29
24.79 4.50
25.62 5.70
27.53 9.12
MOCHA
33.66 4.14
33.61
2.344
32.34 7.09
34.72 1 7.80
Clustered FL
41.50 6.66
8.30
36.78 - 12.05
34.43 - 10.14
Per-FedAvg
6.90
7.37
3..8
36.58 - 6.72
38.41 + 7.41
PFedMe
31.53 - 3.83
32.39 4.36
30.86 3.75
30.86 - 3.80
37.70 + 2.13
Ours (n=5)
35.44 + 1.91
36.21 + 4.92
38.41 + 2.58
42.96 - 1.24
44 29 = 1.22
Ours (n=10)
37.09 + 1.95
37.09 3.84
39.94 + 0.74
43.06 - 0.42
43.75 75 + 1.74
Table 5: In-distribution personalized federated classification on the CIFAR-100 dataset
13

---

Published as a conference paper at ICLR 2021
A.3 CLIENT WEIGHTING WITH PERSONALIZATION
In-local vs out-of-local distribution personalization Following the visualizations for client weights in the out-of-local distribution personalization setting (Fig. we include additional visual- izations for the remaining clients (Fig. 6). For comparison, we also include the same visualizations for the 15 client 5 non-IlI latent distribution setup on CIFAR-10, but when clients optimize for target distribution the same as their local training data's 's (Fig. 7 In both, we use color to denote the client's local training data distribution, such that if FedF omo able to identify the right clients to federated with that client, we should see the weights for those colors increase or remain steady over federation rounds, while all other client weights drop.
As seen in both Fig. and Fig. 7 FedF omo quickly downweights clients with unhelpful data dis- tributions. For the in-distribution personalization, it is able to increase and maintain higher weights for the clients from the same distribution, and consistently does for the other two clients that belong to its distribution. In the out-of-local distribution personalization setting, due to our shuffling procedure have instances where certain clients have in-distribution targets, while others have out-of-distributie targets. We see that F FedF omo is able to accommodate both simultaneously, and learns to separate all clients belonging to the target distributions of each client from the rest.
Client weiggts
Ellet weights
Client 2 weights
Flant 3 seights
Glent wrights
weights
1.5
-15
2.0
233
15 Cleni songrit
33
10 22
50 Client
20
30
50 Clent welgght
20
32
40 Clent
11
20
Rounds
Federaiion
10 30 45
Delt
Chiri wanghts
weights
12.5
20.0
-05
13
3.5
3.
-10
0 23
25
05
-03
Podercciaon Tourds
Poderalian RAmrS
Pedinsaiin Rounch
on Hlarsh
Pederction Nlunncs
Figure 6: Client-to-client weights over time when personalizing for non-local target distributions. FedFont omo quickly downweights non-relevant clients while upweighting those that are helpful.
weights
0.0
-24
-16
-0.6
Okrt.
Cart
-0
O
Fesertiin Reunds
Federation Roundd Olenn: weights
Fosembon Raundd Clent 8 welghtt
Fode Fode-ston Rounds
Federation Rounds
Fece-9:00 RAINES weights
9 weights
Clenn: weights
10 welghts
-21
0.0
URK 20
-27
-62
-2.4
-03
Cart
-20
-25
-1.4
-24
-34
36
-0M
-64
Ckrt 34
45.
Foserntion Rounds
Fodo/stin do
Fedelstion Rounds
Fode 9100 Rounds
Foderttion RoundS
Fedemmton Round:
Figure/ 7: Client-to-client weights over time when personalizing for local target distributions. omo downweights non-relevant clients while upweighting or keeping steady helpful ones.
14

---

Published as a conference paper at ICLR 2021
CIFAR-10
CIFAR-100
Method
5
Accuracy
Accuracy
FedAvg
x 10
0
19.37 1.42
OC
5.09 1 0.38
FedAvg
x 10
10.26 + 0.21
17.60 1.64
8.20 = 0.69
5.05 0.31
FodAvg
x 10
2
3.57 + 0.08
16.19 + 1.62
2.33 + 0.21
4.33 + 0.27
Ours
x 10-8
0
71.56 + 1.20
0C 8.20 + 0.69
26.76 + 1.20
Ours
x 10
6.89
0.13
71.28 1.06
26.14 = 1.05
Ours
1 10
2
1.70 + 0.04
65.97 + 0.95
1.71 + 0.15
15.95 + 0.94
Ours (MA)
x 10
0
47.90 + 2.79
9 10.32 + 0.89
12.02 + 1.34
Ours (MA)
x 10 5
9.26
0.19
46.33 4.04
11.60 + 0.65
Ours (MA)
x 10
2
3.20 + 0.07
43.76 + 3.08
3.22 = 0.28
9.52 + 0.72
Table 6: In-distribution classification with differentially private federated learning, with the addition of FedFon omo with model average baseline (Ours (MA)).
A.4 ADDITIONAL PRIVACY EXPERIMENTS
As follow-up on the privacy experiments Section we also consider multiple model variant of FedFome where instead of a client downloading single model 6,, and evaluating against its own previous model 6.- the client downloads the simple average of all the uploaded models except On (i.c. N E)E(NT 0n) and compares this against the simple average of all uploaded models. This tackles an orthogonal notion of privacy compared to the previous solution of introducing noise to local model gradients via DPDSGD, now individual data point membership is harder to distill from shared parameters that come from the average of multiple local models. To calculate weights, note a sign change with respect Eq. 33 and the baseline model, now w,, should be positive if the model average without O₂ 's contribution results in larger target objective loss than the model average with On. Given client e; considering model On. this leads to FedF omo weights:
x L, N -
Σ 0, ( Σ 8,) JE
(14)
JEIN]
We evaluate this variant with the same comparison over (e, d)-differential privacy parameters on the 15 client latent-distribution scenarios our previous privacy analysis. We set 8 = 1 x 10-5 to setup practical privacy guarantees with respect to the number of datapoints in each client's local training set, and consider Gaussian noise {0, 1, 2. for baseline and (s, 8) -differentially private performances. At fixed 6, we wish to obtain high classification accuracy with low privacy loss (E).
In Table we include results for this model average baseline variant (Ours (MA)) on the CIFAR-10 and CIFAR-100 100 datasets, along with the differentially private federated classification results in Ta- ble using DP-SGD during local training for additional context. For both datasets, we still handily outperform non-private FedAvg, although performance drops considerably with respect to the sin- gle model download F FedFon omo variant. We currently hypothesize that this may be due to a more noisy calculation of another model'ss potential contribution to the client's current model, as now consider the effects of many more models in our loss comparisons as well. Figuring out a balance between the presented weighting schemas to attain high personalization and high privacy by downloading model averages then remains interesting future work.
15

---

Published as a conference paper at ICLR 2021
A.5 LATENT DISTRIBUTION NID-IID MOTIVATION AND SETUP
In this subsection, we discuss our latent distribution non-IID setting in more detail. We believe the pathological setup though useful might represent more realistic frequent occurring setups. As an example, world-wide dataset of road landscapes may vary greatly across different data points, but variance in their feature representations can commonly be explained by their location. In another scenario, we can imagine that certain combinations of songs, genres of music altogether are more likely to be liked by the same person than others. fact, the very basis and success of popular recommender system algorithms such as collaborative filtering and latent factor models rely on this scenario (Hofmann] 2004). Accordingly, in this sense statistical heterogeneity and client local data non-liDnes is more likely to happen in groups.
We thus propose and utilize latent distribution method to evaluate FedF omo against other more recent proposed FL work. To this setting, first compute image representations by training a VGG-11 convolutional neural network to at least 85% classification accuracy on a corresponding dataset. We then inference every data point, and treat the 4096-dimensional vector produced the second fully-connecteo layer as a semantic embedding for each individual image. After further reduction 256 dimensions through PCA, we use K-Means clustering partition our dataset into D disjoint distributions. Given K total clients, we then evenly assign each client to a distribution D. For each client finally obtain its local data by sampling randomly from D without replacement. For datasets with pre-defined train and test splits, we cluster embeddings from both at the same time such that similar images across splits are assigned the same K K-means cluster, and respect these original splits such that all Distt images come from the original test split. (Fig. g
In-
8
VGG-11
imae espenssive reprosstations
regresentative vectors dinensooraltt
K-Mears clustering to porttion proupe orique closs
Figure 8: Visual overview for generating latent distributions using image classification datasets.
A.6 MODEL. IMPLEMENTATION DETAILS
We train with SGD, 0.1 learning rate, momentum, 1e-4 weight decay, and 0.99 learning rate decay for CIFAR-10/100. and do the same except with 0.01 learning rate for MNIST. For FedFomo use 5 and n = 10 downloads per client, = 0.3 with 0.05 decay each round, and separate Drain and val with an 80-20 split.
A.7 ADDITIONAL DESIGN ABLATIONS
In this section present additional work key hyperparameters aspects of FedF omo to give further insight into our method's functionality and robustness to parameters. We consider key design choices related to the size of each client's validation split.
Size of the validation split To better organize federated uploaded models into personalized fed- erated updates, our method requires local validation split val that reflects the client's objective or target test distribution. Here, given pre-defined amount of locally available data, we ask the natural question how a client should best go about dividing its data points between those to train own local model and those to evaluate others with respect to computing a more informed personalized update through FedF omo. We use the 15 client 100% participation setup with 5 latent distributions organized over the CIFAR-10 dataset, and consider both the evaluation curve and final test accuracy over allocating a fraction E {0.01, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 0.9) of all clients' local data to D and track evaluation over 20 communication rounds with epochs of local training per round. On
16

---

Published as a conference paper at ICLR 2021
average, each client has 3333 local data points. over five runs in Fig 9
We denote final accuracy and standard deviation
CIFAR-10, 15 clients
Final Accuracy Comparison
90
90
80
85
Accuracy
80
70
75
70
50
65
40
0
10
15
20
0.01
0.05
0.10
0.20
0.40
0.60
0.80
0.90
Communication rounds
Validation
Val Split Ratio: 0.01
Val Split Ratio: 0.10
Val Split Ratio: 0.40
Val Split Ratio: 0.80
Val Split Ratio: 0.05
Val Spiit Ratio: 0.20
Val Split Ratio: 0.60
Val Split Ratio: 0.90
Figure 9: In-distribution accuracy over validation split ratio.
As reported in Fig. 9 we observe faster convergence to higher accuracy when allocating under half of all local data points to the validation split, with a notable drop-off using more data points. This is most likely result of reducing the amount of data available for each client train their model locally. Eventually this stagnates, and observe a slight decrease in performance between validation split fraction 0.05 and 0.1.
17


Reviews:
[{"review_id": "PleULbFRUwD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new FL method that computes in every communication round for each client a personalized model as starting point for the next round of federation. The paper defines the client-specific objective as some loss function of the weighted combination of all (or subset) models on a client-specific validation set. This personalized weighted combination of the models especially fits situations where not all clients have congruent objectives such as in non-IID settings. The paper evaluates the proposed FL algorithm on standard datasets for image classification by comparing to alternative FL methods. \n\nPros:\n- The paper targets a relevant topic\n- The proposed approach also works in out-of-client-distribution settings\n\nCons:\n- The authors mention related clustered federated learning approaches (Sattler et al., 2020; Ghosh et al., 2020; Briggs et al., 2020; Mansour et al., 2020), but do not compare against any of these methods.\n\n- The main advantage of the proposed method is that it also works in out-of-client-distribution settings (otherwise the performance is comparable, see Table 1), i.e., when the target distribution which is not the same as their local training data distribution. However, this section is very short and the evaluation not very insightful. I recommend to more carefully investigate this property of the proposed method.\n\n- I would like to see some more theoretical analysis and insights. For instance, Sattler et al. 2020 proved that their clustered FL approach is able correctly identify the clustered (assuming that data was generated from K different distributions) if the empirical risk computed on the loss approaches the true risk. Can you give similar guarantees?\n\n- You should also report the communication overhead of your method and compare it with related approaches. Downloading multiple models definitely increases overhead. Methods such as the clustered FL (Sattler et al. 2020) allow to compute specialised models for free, whereas you need to download multiple copies of the model at every communication round. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "HJ_8BFEdka", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed FOMO, a personalized FL framework. Comparing to existing methods that conduct local adaptation starting from a single global model, FOMO keeps track of all models, and personalize by computing the weighted version among all models.  The experimental results show improvement comparing to existing methods.\n\nThe following points from this paper are novel and interesting:\n\n(1) personalization via weighting multiple models is intuitively more stable, comparing to client-side gradient descent. Especially for non-iid settings, personalize from a set of models, offers more flexibility and robustness to personalize, than personalize from a single model. Moreover, assuming having a validation dataset, the client with multiple models can have an easy way to choose an existing model.\n\n(2) The method to approximate optimal w, is clean and shows interesting improvement. Although the experiments are not done on FL golden standards such as EMNIST/Shakespear, the experiments and reasonings are well presented and well conveyed.\n\nHowever, there are a few points that need to be taken care of:\n\n(1) FOMO assumes that the server keeps track of all clients' models,  this changes the classical assumption of FL: all model updates should be anonymous. This leads to privacy issues. The author addresses the privacy concern in section A.3, yet I am not satisfied. Here is the reason: (a) when any model is downloaded to the client, there is no scheme to avoid the client to analyze the model since all models share the same structure, only weights are different. With the other's model, a malicious client can do some harm, which needs differential privacy (DP) guarantees. (b) The DP approach adds noise to updated models, which could lead to a significant personalization performance drop, which hasn't been discussed in the paper, and DP is critical to ensure data privacy. In my opinion, FOMO introduces a powerful idea of downloading multiple models to improve personalization, yet introduces the problem of privacy.\n\n(2) Comparing to keep track of models of all clients (I am not satisfied with the claim in A.3), further research, should consider keeping track of multiple models, with anonymous FL updates. The clients can use multiple models to update, while since the models are updated via anonymous gradients, then no privacy issue will arise.\n\nI feel this is an interesting direction for FL personalization research, while the privacy issue is not properly addressed. I am expecting the authors to address the privacy part with some updated experiment. Given the above concerns, I set my score to 5. If the privacy issue is properly addressed I am glad to increase the score.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ZZYF2UHYww", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "EDIT: Based on the author's modifications to the manuscript, I have increased my score from 5 to 6.\n\nThis paper introduces an approach to personalized federated learning. After each communication round, nodes receive full models from other nodes. The nodes construct a weighted average of the received models so that the mix performs well on their local *validation* data. The first interesting design choice is an approximation to the node-local optimal weighting problem. The second contribution is a scheme that decides which local models to share among which nodes, avoiding excessive $O(K^2)$ communication of full models. The proposed model is experimentally demonstrated to be competitive or better than previous work on an extensive array of tasks. \n\nThe writing is generally clear. I did have the feeling that the abstract, introduction and related work sections were slightly repetitive and could be made more concise.\n\nThe setup is partially motivated by data privacy. It does not require any data to be shared between users. It does, however, requiring many full models to be exchanged. While there might definitely be scenarios in which this is reasonable, the large amount of communication could be prohibitive in many practical scenarios, and exchanging local models might still raise privacy concerns.\n\nAlthough I think the framework presented in this paper is interesting and potentially relevant, I do see shortcomings in the current manuscript:\n- The two contributions (the approximation to the optimal weighting problem + model exchange coordiation scheme to save bandwidth) could be better motivated. For the weight computation, it is not completely clear to me why this is called 'first-order'. What is this a first order approximation of? What is the effect of this approximation compared to the 'real' solution or other choices you may have considered? For the model exchange coordination scheme, I am missing some details in the description. I am not confident I could implement this scheme from the paper. I would also expect to see some form of ablation study here: what is the effect of varying the number of other models shared with nodes. What is the effect of the parameter $\\epsilon$, etc.'? From Table 1, it seems that sharing more models with workers (n=10) is not necessarily better than (n=5). \n- There seems to be no explicit scheme to keep the personal models of the nodes close together. I would like to see some experimental or theoretical analysis or insight into why the models don't drift apart until they cannot benefit from each other anymore. How do the weights w_n evolve over time?\n\n\nA few minor remarks:\n- In Eqn. 5, I wonder if the denominator is correct. Is it dividing by a vector?\n- Is division by zero never a problem in the proposed weighting normalization scheme?\n- I couldn't follow the sentence \"where clients (1) given a local objective clients their received models and\".\n- In \"from some distribution $D_j$ and local\" (notation section), I believe $D_j$ should be $D_i$.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0gNFO1T2-9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper Summary:** The paper addresses an important topic in federated learning which is personalization. The authors propose a two steps process to achieve the personalization: 1. Figuring out which models to send to which clients; 2. Computing their personalized weighted combinations for each client. To determine the weights, the authors use first order approximation. \n\n**Pros**: \n1. Personalized federated learning is an important problem, and I see a lot of value in practical applications of federated learning\n2. The proposed method is simple and intuitive\n**Cons**: \n1. I see strong claims in the paper such as 'we are the first to our knowledge to enable transfer to new specific data domains of interest through personalized FL. ' \n2. Some references are missing in the related work sections such as (Dinh et al. NeurIPS 2020), (Fallah et al. ArXiv 2020), and (Peterson et al. ArXiv 2019).\n\n**Areas to Improve**: \n1. I like to see more experiments to be added in regards to comparison with some recent works especially with (Dinh et al. NeurIPS 2020).\n2. Would be helpful to provide a convergence analysis of the proposed algorithm.\n\n**Missing References:**\n1. Personalized Federated Learning with Moreau Envelopes (Dinh et al. NeurIPS 2020)\n2. Private Federated Learning with Domain Adaptation (Peterson et al. ArXiv 2019)\n3. Personalized Federated Learning: A Meta-Learning Approach (Fallah et al. ArXiv 2020) \n\n**Minor Concerns:**\n1. Grammatical error here 'given a local objective clients their received models and'\n2. Eq. 6 some errors in the superscript of \"\\theta\" \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a personalized federated learning method, which personalizes by computing a weighted combination of neighboring compatible models. Reviewers uniformly liked the quality of writing and level of novelty, and agree on the relevance of the problem and solution. The solution was deemed creative and particularly impactful in the important case of heterogeneous data on each node, and experiments showed convincing improvements. The discussion between reviewers and authors was constructive and has lead to further improvements of the paper. Slight concerns remained on privacy with all models stored on the server, and breath of personalized FL benchmarks used, but reviewers agreed the contributions overall are still significant enough. Future work remains on the theory of the proposed model.
Decision: Accept (Poster)