Improving Federated Learning Personalization via Model Agnostic Meta Learning
Authors: Yihan Jiang
Jakub Konečný
Keith Rush
Sreeram Kannan
Venue:  (2020)
Keywords: Federated Learning
Model Agnostic Meta Learning
Personalization

Abstract: Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL. We present FL as a natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm. 2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization result. 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These results raise new questions for FL, MAML, and broader ML research.

Under review as a conference paper at ICLR 2020
IMPROVING FEDERATED LEARNING PERSONALIZA- TION VIA MODEL AGNOSTIC META LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for fast, gradient-based, few-shot adaptation to heterogeneous distri- bution tasks, has number of similarities with the objective of personalization for FL. We present FL natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging (McMahan al.) 2017), can be interpreted a meta learning algorithm. 2) Careful fine-tuning can yield global model with higher accuracy, which the same time casier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization result. 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These results raise new questions for FL. MAML, and broader ML research.
INTRODUCTION
In recent years, the growth of machine learning applications was driven by aggregation of large amounts of data in datacenter, where a model can be trained using large scale distributed sys- tem (Dean et al. 2012, LeCun al. 2015). Both the research community and general public are becoming increasingly aware that there is variety of scenarios where this kind of data collection comes with significant risks, mainly related to notions of privacy and trust.
In the presence of user generated data, such as activity on mobile phones, Federated Learning (FL) (McMahan & Ramage 2017] proposes an alternative approach for training a high quality global model without ever sending raw data to the cloud. The FL system proposed by Google (Bonawitz ctt al. 2019) seleets a sample of available devices and sends them a model to be trained. The de- vices compute update the model based on optimization procedure with locally available data, and the central system aggregates the updates from different devices. Such iteration re- peated many times until the model has converged. The users' training data does leave their devices. The basic FL algorithm, Federated Averaging (FedAvg) (McMahan et al. 2017). has been used in production applications, for instance for next word prediction in mobile keyboard (Hard) et al. 2018), which shows that Federated Learning can outperform the best model trained in dat- acenter. Successful algorithmic extensions to the central idea include training a differential private model (McMahan et al. 2018), compression (Konečný et al. 2016b) Caldas et al.) 2018a). secure aggregation (Bonawitz al. 2017), and a smaller number of always-participatinn nodes (Yang et al.) 2019).
FL applications generally face non-i.i.d and unbalanced data available to devices, which makes challenging to ensure good performance across different devices with FL-trained global model. Theoretical guarantees are only available under restrictive assumptions and for convex objectives, cf. Li et al. (2019b). In this work, we interested in personalization methods that adapt the model for data available on each device, individually. We refer to a trained global model as the initial model, and the locally adapted model as the personalized model. Existing FL. personalization work

---

Under review as a conference paper at ICLR 2020
directly takes a converged initial model and conducts personalization evaluation via gradient descent (Beaufays et al. 2019). However, in this approach, the training and personalization procedures are completely disconnected, which results in potentially suboptimal personalized models.
Meta Learning optimizes the performance after adaptation given few-shot adaptation examples on heterogeneous tasks, and has increasing applications in the context of Supervised Learning and Reinforcement Learning. Model Agnostic Meta Learning (MAML) introduced byninn et al. (2017) solely gradient-based Meta Learning algorithm, which runs in two connected stages; meta- training and meta-testing. Meta-training learns a sensitive initial model which can conduct fast adaptation on range of tasks, and meta-testing adapts the initial model for particular task.
Both tasks for MAML, and clients for FL, are heterogeneous. For each task in MAML and client in FL, existing algorithms use a variant of gradient descent locally, and send an overall update to coordinator to update the global model. If we present the FL training process as meta-training in the MAML language, and the FL personalization via gradient descent meta-testing, we show in Section 2 that FedAvg (McMahan et al.] 2017) and Reptile (Nichol et al. 2018). two popular FL and MAML algorithms, are very similar to each other; see also Khodak et al. (2019).
In order to make FL personalization useful in practice, we propose that the following objectives must all be addressed, simultaneously.
Improved Personalized Model for large majority of the clients.
() Solid Initial Model - some clients have limited or even no data for personalization. Fast Convergence reach a high quality model in small number of training rounds.
Typically, the MAML algorithms only focus objective (1); that was the original motivation in Finn et al. (2017). Existing FL. works usually focus on objectives (2) and (3), and take the personalized performance secondary. This is largely due to the fact that it was not obvious that getting solid initial model is feasible or practical if devices are available occasionally and with limited resources.
In this work, study these three objectives jointly, and our main contributions are:
We point out the connection between two widely used FL and MAML algorithms, and interpret existing FI. algorithm in the light of existing MAMI algorithms.
We propose a novel modification of FedAvg, with two stages of training and fine-tuning, for optimizing the three above objectives.
We empirically demonstrate that FedAvg is already a meta learning algorithm, optimizing for personalized performance, as opposed to quality of the global model. Furthermore, we show that the fine tuning stage enables better and more stable personalized performance.
We observe that different global models with the same accuracy, can exhibit very different
capacity for personalization.
We highlight that these results challenge the existing objectives in the FL literature, and motivate new problems for the broader Machine Learning research community.
INTERPRETING FEDAVG AS A META LEARNING ALGORITHM
In this section, we highlight the similarities between the FL and MAML algorithms and interpret FedAvg as linear combination of a naive baseline and a collection of existing MAML methods.
Algorithm I presents conceptual algorithm with nested structure (left column), of which the MAML meta-training algorithm, Reptile (middle column), and FL-training algorithm, FedAvg (right column), are particular instances. We assume that is loss function common to all of the following arguments. In each iteration, a MAML algorithm trains across a random batch of tasks {T For each task T;, conducts an inner-loop update, and aggregates gradients from each sampled task with an luter-loop update. In each training round, FI. uses random selection of clients {T.). For each client T; and its weight w,, it runs an optimization procedure for a number of epochs over the local data, and sends the update to the server, which aggregates the updates to form new global model. If we simplify the setting and assume all clients have the same amount of data, causing the weights w; to be identical, Reptile and FedAvg in fact become the same algorithms. Several other MAML algorithms (Finn et al. 2017 Antoniou et al., 2018), or other non-M MAML/FL methods (Zhang et al. 2019), can also be viewed as instances of the conceptual method in left column of Algorithm]
2

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Connects FL and MAML (left), Reptile Batch Version(middle), and FedAvg (right).
oupervr learning rale a
Require: Reptile Step K.
Require:
FedAvg Local Epoch E.
InnerL.oop/CTien learning Initial model parameterrs &
fanction Faner Loop(0. Tu.
function "lientUppdaae(. T, Split local dataset into batches
Sample K-shot data D,,k from T;-
while not done do Sample batch of tasks/clients {2i) Sampled taskfclient 2;; do if FT. then
for each local step from to K. do
for each local epoch from 1 to E do B
6, V (.,e)
02 9 AV I.(,,
gt. w. = ClientU (adate((,220 Tv, 8) else MAMI. then
end for end for Return Ss 6
end for Return 6,
9: Inner T..D)
end if end for
end function
eud function
Require: Meta Batch Size M.
Require: Clients trinnigg round M. function Servert peeteerrrppaaeett
function Outer Loop(0, (9:). ).
pdate(0, (gi, w;)
B SeruerUpdate((, (at mt)).
else if MAML then
0 Ei= 1 9r
0= 0+ Eim wage// 3d
ws
0 Loop((, (a)).))
Return /
Return 0
end if while
end function
end function
In the following, we rearrange the summands comprising the update formula of FedAvg/Reptile algorithm to reveal the connection with other existing methods linear combination of the Feder- ated SGD (FedSGD) (McMahan et al., 2017) and First Order MAMI. (FOMAML)) algorithms (Finn) al. 2017) with different number of steps. For clarity, we assume identical weights wy in FedAvg.
Consider T participating clients and let 0 parameters of the relevant model. For each client i, define its local loss function as L; (0), and let 9j be the gradient computed in jth iteration during local grrallgradeet--eee based optimization process.
FedSGD was proposed as naive bascline against which to compare FL algorithms. For each client, simply takes single gradient step based on the local data, which is sent back to the server. is sensible baseline because it is variant of what traditional optimization method would do if we were to collect all the data in a central location, albeit inefficient in the FL setting. That means that FedSGD optimizes the performance of the initial model, is the usual objective in datacenter training. The local update produced by FedSGD, @FedSGD can be written as
- T
OL, 00
T
DFedSGD
Σ
oi
211
Next, we derive the update of FOMAML in similar terms. Assuming client learning rate 3, the personalized model of client i, obtained after K-step gradient update is 0% (k) 0 (1,(1,) Differentiating the client update formula, we get
1) j= E
86
0 _ (0) I I-
LL 002
Directly optimizing the current model for the personalized performance after locally adapting K gradient steps, results in the general MAML update proposed by Finn et al. (2017).
9MAML = iLMAML
T
( 00
T Σ T
K - Σ 1=1
(L,) 002
(3)
00
T
1=1
MAML requires to compute 2nd-order gradients, which can be computationally expensive and creates potentially infeasible memory requirements. avoid computing the 2nd-order term, FO- MAML simply ignores it, resulting in a first-order approximation of the objective (Finn et al, 2017). FOMAML(K) then uses the (K + 1)th gradient as the local update, after K gradient steps.
T
T
1 T
(K) = T Σ
(0)) 1 1((0*) = T
Σ 1=1
Σ gk i=1
(4)
T
3

---

Under review as a conference paper at ICLR 2020
Now have derived the building blocks of the FedAvg. As presented in Algorithm | the update of FedAvg, 9FedAvg. is the average of client updates, which are the sums of local gradient updates. Rearranging the terms presents its interpretation
linear combination of the above ideas.
K
K-1 Σ j=1
gFodAng
T
93 -
-
Σ
oi
94+1 9FedSGD + gFOMAME(i) (5)
Note that interpolating to special case, 9FedSGD can be seen as (POMAML(O) optimizing the per- formance after 0 local updates, i.e., the current model. This sheds light onto the existing Federated Averaging algorithm, as the linear combination of algorithms optimizing personalized performance after a range of local updates. Note, however, this does not mean that FedAvg optimizes for the linear combination of the objectives of the respective algorithms. Nevertheless, we show in the following section that using = results in a model hard to personalize, and increasing K signifi- cantly improves the personalization performance, up until certain point where the performance of initial model becomes unstable.
3 PERSONALIZED FEDAVG
In this section, we present Personalized FedAvg algorithm, which is the result of experimental adap- tation of the FedAvg algorithm to improve the three objectives proposed in the introduction.
We denote Fed.Aug(E) E') the Federated Averaging method from Algorithm right, run for E local epochs, weighting the updates proportionally to the amount of data available locally. We denote Replile(K K) the method from Algorithm [ middle, run in the FI. setting for K local steps, irre- spective of the amount of data available locally. Based on variety of experiments we explored, propose Personalized FedAvg in Algorithm
Algorithm Personalized FedAvg
1: Run Fed.Avg( (`) with momentum SGD as server optimizer and a relatively larger E.
2: Switch to Reptile(K) with Adam as server optimizer to fine-tune the initial model.
3: Conduct personalization with the same client optimizer used during training.
In general, FedAvg training with several local epochs ensures reasonably fast convergence in terms of number of communication rounds. Due to complexity of production system, this measure was studied as the proxy for convergence speed of FL. algorithms. We find that this method with mo- mentum SGD as the server optimizer already optimizes for the personalized model objective (I) form the introduction while the initial model - objective (2) is relatively unstable. Based on prior work, the recommendation to address this problem would be to decrease E or the local learning rate, stabilizing initial model at the cost of slowing down convergence (McMahan et al. 2017 Wang & Joshi 2018) objective (3).
We propose fine-tuning stage using Reptile () with small K and Adam the server optimizer, to improve the initial model, while preserving and stabilizing the personalized model. We observed that Adam yields better results than other optimizers, see Table in Appendix [A.]] and makes the best personalized performance achievable with broader set of hyperparameters, see Figure 2 The subsequent deployment and personalization is conducted using the same client optimizer as used for training. as we observe that this choice yields the best results for FedAAg/Reptile httpr/peiil--rrrinnedd models.
Experimental setup. We use the EMNIST-62 dataset as our primary benchmark (Caldas et al., 2018b). It is the original source of the MNIST dataset, which comes with author id, and noticable variations in style, pen width, size, etc., making a suitable source for simulated FL experiments. The dataset contains 3400 users, each with train/test data split, with a total of 671, 585 train and 77, 483 test images. We choose the first 2, 500 users as the initial training clients, leaving the remaining 900 clients for evaluation of personalization; these clients are not touched during training. The evaluation metrics are the initial and personalized accuracy, uniformly averaged among all of the FL-personalization clients. This is preferred to weighted average, in a production system we care about the future performance on each device, regardless of the amount of data available for personalization. Unless specified otherwise, we use the baseline convolutional model available
4

---

Under review as a conference paper at ICLR 2020
in TensorFlow Federated (Ingerman & Ostrowski 2019 using SGD with learning rate 0.02 and batch size of 20 as the client optimizer, and SGD with momentum of 0.9 and learning rate 1.0 as the server optimizer. Each experiment mean and standard deviation of initial and personalized accuracies are reported. We also use the shakespeare dataset for next-character prediction, split in similar manner with first 500 clients used for training and remaining 215 for evaluation of personalization.
repeated 9 times with random initialization, and the
3.1 CONVERGENCE OF FEDAVG
In Figure 1 left, present the convergence of both initial and personalized model during training using the Federated Averaging algorithm. The results correspond to training with E' being 2 and 10, with visualization of the empirical mean and variance observed in the 9 replicas of the experiment. Detailed values about performance after 500 rounds of training, and the number of rounds to reach 80% accuracy, are provided in Table 1
These results provide number of valuable insights.
ENNIST Fuskig Denvergence Speed Fescanized kRbh3 local
Cervez Speed of Shacospzers Perscarlaod with local
E-2pem
Boil
8.50
1112
138
837
1.42
dlgg intaes
1.90
rackigg knd pary
612
hopill Bid
330
300
406
540
100
200
300
404
Figure 1: Training Convergence on EMNIST-62 (left) and Shakespeare (right).
First, the personalized accuracy converges significantly higher than the initial accuracy. This clearly validates the EMNIST-62 as an interesting simulated dataset to use to study Federated Learning, with significantly non-i.i.d. data available to each client.
Second, it provides empirical support to the claim made in Section 2 that Federated Averaging is already Meta Learning algorithm. The personalized accuracy not only converges faster and higher, but the results are also of much smaller variance than those of the initial accuracy.
Third, the personalized accuracy after training with E 10 is significantly higher than the person- alized accuracy after training with E = 2. This is despite the fact that the gap in the initial accuracy between these two variants somewhat smaller. Moreover, the variance of personalized accuracy is nearly 3-times smaller for training with E = 10, compared to E = 2, despite the variance of the initial accuracy being smaller for the F case. This supports the insight from Equation 5 that Federated Averaging with more gradient steps locally should emphasize the personalized accuracy more, potentially at the cost of initial accuracy.
Finally, this challenges the objectives in the existing literature focusing on the setting of Federated Learning. FedAvg was presented as optimizing the performance of a shared, global model, while in fact it might achieve this only performance. We argue that in the presence of non-i.i.d. data available to different clients, the objective of Federated Learning should also be the personalized performance. Consequently, the recommendations that in order to stabilize the convergence, one might decrease the number of local epochs or the local learning rate (MeMahan et al. [2017] Wang & Joshi 2018), are in some scenarios misguided. In this experiment, even though the initial accuracy is very noisy and roughly constant suboptimal level, the personalized accuracy keeps increasing.
necessary partial step towards optimizing the personalized
To evaluate the convergence speed more closely, Table I measures the accuracies after 500 rounds of training and the average number of communication rounds where the initial and personalized accuracy first time reaches 80%. While Figure [hooss results using 5 clients per round, the table below also shows the same experiment with 20 clients per round, which general provides even better and more stable personalized accuracy. The common pattern is that increasing E initially helps, until a certain threshold. From this experiment, in the range of 10 seems to be the best.
We conduct a similar experiment with the Shakespeare data. The result is in Figure [ right, but does not provide any interesting insights, as the personalized performance shows only small positive
Presented experiments were implemented in TFF, and will make the code publicly available.
5

---

Under review as a conference paper at ICLR 2020
EMNIST-62 5 clients
Initial Ace 0.7473(0.0260)
Personalized Ace
Epochs to 0.8 (init/pers)
FedAvg E=2
0.8292 (0.0061)
310.0/63.6 111.1/33.9
FedAvg E=5
0.8028 (0.0512)
0.8712 (0.0049)
FedAvg E=10
0.7879(0.0316)
0.8820 (0.0023)
137.5/30.0
FedAvg E=20
0.7430(0.0309)
0.8782 (0.0021)
152.5/32.2
EMNIST-62 20 clients
FedAvg E-2
0.8403 (0.0173)
0.8957 (0.0011)
82.5/50.0
FedAvg E=5
0.8471 (0.0084)
0.9057 (0.0017)
65.6/31.25
FedAvg E=10
0.8480 (0.0036)
0.9032 (0.0017)
68.7/25.0
FedAvg E=20
0.8391 (0.0081)
0.8953 (0.0022)
82.1/46.4
Table 1: EMNIST-62 performance for and 20 clients per communication round.
improvement. We conjecture that this is due to the nature of the objective even though the data is non-i.i.d., the next-character prediction is mostly focused on a local structure of the language in general, and is similar across all users 2 We thus do not study this problem further in this paper. is likely that for a next-word prediction task, personalization would make more significant difference.
3.2 BEHAVIOR OF PERSONAI.IZATION
In this section, we study the ability of models to personalize more closely. In particular, we look at the personalization performance as function of the number of local epochs spent personalizing. and the effect of both local personalization optimizer and the optimizer used to train the initial model.
In Figure 2 left, we study the performance of three different models, personalized using different local optimizers. The models we test here are: one randomly chosen from the models trained with E = 10 in the previous section, with initial accuracy of 74.41%. The other two models are the results of fine-tuning that specific model with Replile(1) ) and Replile(10) and Adam as the server optimizer for further 200 communication rounds, again using clients per round. For all three models, we show the results of personalization using Adam with default parameters and using SGD with learning rate of 0.02 and batch size of 100.
Persgnalisationn T721ST-62 with Rept Finetuniig
Personplization EMNST-62 centralized trained modilss
0.97
0.87
Frelvoi-tt
0.84
Alee and mooe Misspc Sop
0.33
Fivntane S40
Reppitn Fivetate Adm
Adan indil (500poci Sap 5.05
031
Repille neture. 560 2.00
0.1
Aster model (10spoct Sap 6.2
Updatic Epacts
Usdute Esecto
Figure 2: Personalized accuracy as a function of local update epochs. Federated initial models (left), initial models trained by centralizing data and using default Adam optimizer (right).
In all three cases, Adam produces reasonable personalized result, but inferior to the result of SGD. We tried SGD with a range of other learning rates, and in all cases we observed this value to work the best. Note that this is the same local optimizer that was used during training and fine tuning, which is similar to the MAML works, where the same algorithm is user for meta-training and meta-testing.
The effect of fine-tuning is very encouraging. Using Reptile(10), we get a model with better initial accuracy, which also gets slightly improved personalized accuracy. Most importantly, it gets to roughly the same performance for a wide range of local personalization epochs. Such property is of immense value for pracical deployment, as only limited tools for model quality validation can be available on the devices where personalization happens. Using Replile(1) significantly improves the initial accuracy, but the personalized accuracy actually drops! Even though the Equation 5 sug- gests that this algorithm should not take into account the personalized performance, it is not clear that such result should be intuitively expected- does not mean it actively supresses personaliza- tion, either, and it is only fine tuning of model already trained for the personalized performance.
However, we did not try replace the default model suggested in TensorFlow Federated.

---

Under review as a conference paper at ICLR 2020
We note that this can be seen analogous observation to those of Nichol et al. (2018) where Reptile(1) yields a model with clearly inferior personalized performance.
Motivated by this somewhat surprising result, ask the following question: If the training data were available in a central location, and an initial model was trained using standard optimization algorithms, how would the personalization change? We train such "centralized initial model" using Adam with default settings, and evaluate the personalization performance of a model snapshot after 10 and 50 training epochs. These two models have similar initial accuracies as the two models fine tuned with Reptile(10) and Replile(1), respectively. The results are in Figure 2 right.
The main message is that it is significantly harder to personalize these centralized initial models. Using SGD with learning rate of 0.02 is not good a significantly smaller learning rate is needed to prevent the models from diverging, and then the personalization improvement is relatively small. In this case, using Adam does provide better result, but still below the fine tuning performance in Figure left. It is worth noting that the personalized performance of this converged model is similar to that of the model we get after fine tuning with Reptile(1), although using different personalization optimizer. At the moment, we are unable to suggest sound explanation for this similarity.
At the start of the Section 3 we recommended using Adam as the server optimizer for fine tuning, and here we only presented such results. We did try different optimizers, and found them to yield worse results with higher variance, especially in terms of the initial accuracy. See Appendix A..] for more details, where we see that all of the optimizers can deliver higher initial accuracy at the cost of slightly lower personalized accuracy.
To strengthen the above observations, we look at the distribution of the initial and personalized accuracies of fine tuned models over multiple experiments. In Figure we look the same three models as in Figure 2 It is clear that the initial model has a large variance in the initial accuracy. the results are in the range of 12%. but the personalized accuracies are only within the range of El 1%. We chose one model, as indicated by the arrows to be fine tuned with Reptile 10) and Reptile(1). In both cases, fine tuning results in more consistent results in both the initial and personalized accuracy. Moreover, the best personalized accuracy with Reptile(1) is worse than the worst personalized accuracy with Reptile(10).
Finetune Ferformance on EMRIST-2
1.0000
Pooleeg E-10 without Fentuue
2.0015
Fieetune with Fediico
3.0058
Feedaan ileppiietO0
3.8635
3.0000
3.5755
3.5755
3.725
38708
Intal Acc
Figure 3: Distribution of initial and personalized accuracies of fine tuned models.
In Appendix A.2
we look at a similar visualization on a per-client basis for a given model. Studying this distribution is of great importance, in practical deployment, user's 's experience might incur disproportionate cost, relative to the benefit of comparable improve- ment in the model quality. We do not study this question deeper in this work, though.
small degradation in
Initial Ace
Personalized Acc
Reptile(1) Finetuned test clients Reptile(1) 1) Finetuned train clients
0.8320 (0.0133)
0.8764 (0.0017)
0.8577 (0.0019)
0.8927 (0.0015)
Reptile(10) Finetuned test clients
0.8116 (0.0148)
0.8858 (0.0014)
Reptile(10) 10) Finetuned train clients
0.8612 (0.0020)
0.9028(0.0009)
Table 2: Test performance on clients seen and unseen during FL-training
Finally, look at the performance of fine tuned models discussed above both train and test clients. Table 2
shows that if we only looked at the initial accuracy, basic MI. principles would
"The same model was used as the starting point for fine tuning in Figure

---

Under review as a conference paper at ICLR 2020
suggest the (10) models over-fitting, due to larger gap between the train and test clients. However, the personalized accuracy tells a different story the gap is roughly the same for both model types, and for both train and test clients, Reptile(10) provides significantly better personal- ized accuracy, suggesting we need a novel way to predict the generalization of personalized models.
4 DISCUSSION AND FUTURE WORK
It this work, we argue that in the context of Federated Learning, the accuracy of the global model after personalization should be of much greater interest than it has been. Investigation of the topic re- veals close similarities between the fields of Federated Learning and Model Agnostic Meta Learning, and raises new questions for these areas, as well as for the broader Machine Learning community.
Challenges for Federated Learning. Framing papers in the area of Federated Learning (McMa- han et al. 2017: Konečný et al., 2016a; Li et al. 2019a), formulate the objective as training of a shared global model, based on a decentralized data storage where each node / client has access to non-i.i.d sample from the overall distribution. The objective is identical to one the broader MI. community would optimize for, had all the data been available in a centralized location.
We argue that in this setting, the primary objective should be the adaptation to the statistical hetero- geneity present at different data nodes, and demonstrate that the popular FL algorithm, Federated Averaging, does in fact optimize the personalized performance, and while doing so, also improves the performance of the global model. Experiments we perform demonstrate that the algorithm used its capacity to personalize. Moreover, solely optimizing
to train the model has major influence the accuracy of the global model tends to have negative impact on its capacity to personalize, which further questions the correctness of the commonly presented objectives of Federated Learning.
Challenges for Model Agnostic Meta Learning. The objectives in the Model Agnostic Meta Learning literature are usually only the model performance after adaptation to given task (Finn) et al. 2017). In this work, we present the setting of Federated Learning as a good source of practical applications for MAML algorithms. However, to have impact in FI., these methods need to also consider the performance of the initial model[aa as in practice there will be many clients without data available for personalization. In addition, the connectivity constraints in production deployment emphasize the importance of fast convergence in terms of number of communication rounds. We suggest these objectives become the subject of MAML works, in addition to the performance after adaptation, and to consider the datasets with a natural user/client structure being established for Federated Learning (Caldas et al. 2018b) as the source of experiments for supervised learning.
Challenges for broader Machine Learning. The empirical evaluation in this work raises a num- ber of questions of relevance to Machine Learning research in general. particular, Pelearly shows that models with similar initial accuracy can have very different capacity to personalize to task of the same type as was trained on. This observation raises obvious questions for which we currently cannot provide answer. How does the training algorithm impact personalization ability of the trained model? Is there something we can measure that will predict the adaptability of the model? Is it something we can directly optimize for, potentially leading to novel optimization meth- ods? These questions can relate to a gap highlighted in Table While the common measures could suggest the global model is overfitting the training data, this not true of the personalized model.
Transfer Learning is another technique for which our result could inspire a novel solution. It is very common for machine learning practitioners to take trained model from the research community, replace the final layer with a different output class of interest, and retrain for the new task (Qquab et al. 2014). We conjecture that the algorithms proposed in the FL and MAML communities, could yield base models for which this kind of domain adaptation would yield better results.
Finally, we believe that a systematic analysis of optimization algorithms of the inner-outer structure presented in Algorithm [ could provide novel insights into the connections between optimization and generalization. Apart from the FL and MAML algorithms. Zhhng et al.) (2019) recently proposed method that can be interpreted outer optimizer in the general algorithm, which improves the stability of a variety of existing optimization methods used as the inner optimizer.
WWe recognize that some MAMI. datasets Lake et al. 2015) do not admit good notion of initial accuracy.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint arXiv:1810.09500, 2018.
Francoise Beaufays, Khe Chai Sim, and Petr Zadrazil. An investigation into on-device personaliza- tion of end-to-end automatic speech recognition models. In Interspeech, 2019.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191. ACM, 2017.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. 2019.
Sebastian Caldas, Jakub Konečný, H Brendan McMahan, and Ameet Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv: 1812.02200, 2018a.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint arXiv: 1r12v11812.01097, 2018b.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223-1231, 2012.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126 1135. JMLR. org, 2017.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eich- ner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 1et::8111000000 1.03604, 2018.
Alex
Ingerman and
Krzys
Ostrowski. URL
Introducing tensorflow
fed-
erated, 2019.
Satps://medes um. wwm/tensorrloow
Mikhail
Khodak, Maria Florina-Bale Balcan, and Ameet Talwalkar. Adaptive gradient-based meta- learning methods. arXiv preprini 17Xiv:190..02717, 2019.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv: 1610.0227 201
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint 1XVv:1610.00592 2016b.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning 2015.
through probabilistic program induction. Science,
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436, 2015.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. arXiv preprint arXiv:I908.077 2019a.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg non-iid data. arXiv preprint arXiv: 0007.000722000 2019b.

---

Under review as a conference paper at ICLR 2020
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiee learning deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282, 2017.
H Brendan McMahan and Daniel Ramage. Federated learning: Collaborative machine learn- ing without centralized training data, 2017. https://aat Federated-
http://i.gooollellooo.oooo/////1 /tt.e///ooooo...///////ooooooooo www.oooollllieeeoeeooooaoooaaeee html
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private
recurrent language models. International Conference on Learning Representations, 2018. URL
htttps://openrevill
net / /oo//oorro forum P d=
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint /rXivv:8033. /803.00999 2018.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level im- age representations using convolutional neural networks. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1717-1724, 2014.
Jianyu Wang and Gauri Joshi. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update sgd. arXiv preprint arXiv: 181.0038100003//3, 2018
Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):12, 2019.
Michael R Zhang, James Lucas, Geoffrey Hinton, and Jimmy Ba. Lookahead optimizer: k steps forward, step back. arXiv preprint arXiv: 1xXvv:907.086100 2019.
A APPENDIX
This Appendix contains further details referenced from the main body of the paper.
A.1 FINE TUNING OPTIMIZERS
Table 3 summarizes the attempts fine tuning the model user main body with different server optimizers. We see that comparing the same client optimizers, Adam consistently provides better and more stable results in terms initial accuracy.
EMNIST-62 Model
Initial Acc
Pers Acc
FedAvg E=10 w/o finetuning Momentum(r==.00, 0.9)
0.7441
0,8842
Finetune with FedSGD (Reptile(T)) Adam
0.8320 (0.0133)
0.8764 (0.0017)
Finetune with Reptile(S) Adam
0.8265 (0.0160)
0.8807 (0.0020)
Finetune with Reptile(10) Adam
0,8116 (0.0148)
0.8858 (0.0014)
Finetune with Reptile(I) SGD(I=001)
0.8260 (0.0161)
0.8758 (0.0020)
Finetune with Reptile(1) Momentum(1=00001.000
0.9)
0.8279 (0.0119)
0.8745 (0.0014)
Finetune with Reptile(5) SGD(Ir=0.1)
0.8148 (0.0280)
0.8829 (0.0012)
Finetune with Reptile(5) Momentum(1=00.01, 0.9)
0.8005 (0.0329)
0.8828 (0.0015)
Finetune with Reptile(10) SGD(Ir=0.1)
0.8074 (0.0294)
0.8855 (0.0012)
Finetune with Reptile(10) Momentum(11-0001.c 0.9)
0.8140 (0.0246)
0.8860 (0.0009)
Table Fine-tuning Result of EMNIST-62 with different server optimizers.
A.2 PER-CLIENT PERSONALIZATION RESULTS
Figure 4 visualizes the distribution of initial and personalized accuracies on a per-client basis. Each dot represents random sample of the test clients used for personalization experiments. Studying this distribution is of great importance, as in practical deployment, degrading a user's experience might incur disproportionate cost, compared to the benefit of comparable improvement. Designing methods that robustly identify the clients below the diagonal line and at least revert to the initial model is worth of future investigation.
10

---

Under review as a conference paper at ICLR 2020
personalaatton
Shakesssars gersendizatio
0.0
-0
3.00
-25
-05
Figure 4: Performance for sample of test clients for EMNIST-62 (left) and Shakespeare (right)


Reviews:
[{"review_id": "r1xmrgpU9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I thank authors for the rebuttal. I agree that direction of exploring personalization in FL is interesting. With a stronger methodological contribution, this could become a good paper.\n\n----------------------------------------------------------------------------------------------------------------\nThe main contribution of this paper is to notice the connection between Federated Averaging (FedAvg) and Model Agnostic Meta Learning algorithms (MAML). Authors also consider an algorithm that first trains with FedAvg and then continues training using Reptile.\n\nPros:\n\nInterpretation of FedAvg as a meta-learning algorithm is interesting.\n\nCons:\n\nVery limited methodological contribution. Proposed algorithm is essentially two existing algorithms applied one after another.\n\nExperiments are not conducted rigorously enough. There are many arbitrary hyperparameter choices which may bias the conclusions made in the paper. Statement \"We tried SGD with a range of other learning rates, and in all cases we observed this value to work the best.\" is alarming suggesting that authors tried a variation of settings observing test data performance and reported a few selected runs. Although \"each experiment was repeated 9 times with random initialization\", the train/test split of the clients was fixed. Randomizing over client train/test split could help to improve the reliability of the results.\n\nEMNIST-62 is the only dataset analyzed in some detail. This dataset has drastically varying P(y|x) across clients, i.e. some people write zeros as some others write 'o's. This suggests that it is very hard to train a good global model and personalization is necessary. However this doesn't mean that Shakespeare dataset \"does not provide any interesting insights\". Perhaps, it is indeed more interesting and challenging, demanding more advanced methodology.\n\nIn Figure 1, number of communication rounds may be impractical for FL (considering also addition 200 Reptile rounds). On Shakespeare, FedAvg paper reports 54% accuracy achieved in under 50 communication rounds in one of the settings. There are also recent works on improving communication efficiency that were not discussed or studied for personalization quality, e.g. FedProx from \"Federated Optimization in Heterogeneous Networks\" and PFNM from \"Bayesian Nonparametric Federated Learning of Neural Networks\".\n\nQuestions about Figure 2 experiments:\n1. Fine-tuning requires 200 extra epochs over the initially trained model. What's the initial model accuracy when FedAvg is further trained with Adam optimizer for 200 extra communication rounds?\n2. The personalized test accuracy with FedAvg and Reptile fine-tuning reaches the same value in 10 update epochs, even when Reptile fine-tuning gets 200 extra initial training epochs. Does Reptile fine-tuning provide additional benefits to the initial model as compared to running FedAvg for more number of epochs?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxMzGThcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the application of techniques from meta-learning (a method\nto train a single model which can then be easily adjusted to perform well on\nmultiple tasks) to federated learning (the task of distributed training of\nmodels on distributed datasets).  The paper notes that standard meta-learning\nalgorithms are similar to standard federated learning algorithms, and uses\nthis perspective to produce a merged method and evaluate it empirically.\n\nPros.\n+ The motivation of the paper is clear and indeed these methods seems similar,\n and meta-learning can help with federated learning.\n\nCons.\n- The resulting method appears somewhat underdeveloped; it is simply to run\n some amount of federated learning and then some amount of meta-learning,\n whereas the first parts of the paper led me to believe that a single\n simultaneous merge of the methods is the way to go.  The paper does not\n report any fine-grained evaluation of various such choices, thus I don't know\n why they did that they did, and thus do not find their choices compelling.\n- The Reptile method is already presented in the original paper with\n a distributed counterpart, so why not just run that?  I am not convinced that\n some more minor modification of Reptile could not already do well on this\n paper.\n- The empirical evaluation is not very extensive, so I am also not convinced\n there, and in particular I need convincing of this type to believe that\n regular reptile is beaten by FedAvg+reptile.\n\nMinor comments.\nPage 1, second paragraph, the word \"outperform\".  I'm not sure what the\nperformance measure is; in federated learning, we care about many things, for\ninstance privacy, keeping the work on the distributed clients low, etc.\nPage 2, the \"three objectives\".  I feel meta-learning is doing all three too.\nPage 3, Algorithm 1.  I realize space is a concern, but this was hard to read.\nPage 4, Algorithm 2.  \"relatively larger\" is vague.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Byxju1vRFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers personalization federated learning problem in which the goal is to personalize the global model on a given client/device based on available data on that device/client.  The paper claims not only their proposed method can lead to fast convergence time but also provide a solid initial model per device/client and results in a better-personalized model. To evaluate the performance of their method, EMNIST-62 and Shakespeare data are used.\n\nEven though personalization in federated learning is very interesting and challenging, I am not sure about the contribution of this paper and what is exactly proposed in this paper: \n\n1)  Section 2: this paper shows the relationship between FedAvg and MAML. In my view, the connection is very straight forward and can be shown in a couple of sentences. I might be missing something here, but it is not obvious to me what this paper adds to the connection between MAML and FedAvg.  \n\n2)  Personalized FedAvg Section: The same is about section 3. In my view, Algorithm 2 doesn't say anything new rather than to use Adam in local machine and SGD on global models and to optimize for \"E\" steps. But what if we use other datasets rather than EMNIST-62 and Shakespeare? will these recommendations still hold, i.e. using SGD on server and Adam on the devices? Per section 3 of this paper, Algorithm 2 indeed is the result of the experimental adaptation of the FedAvg algorithm so generalization to other datasets won't be obvious and it is a big question to me.\n \n3) Also, the paper mentioned that this method can work even if there is no local data available on some of the devices/clients. I wasn't able to understand how personalization possible if there is no data to personalize. Wouldn't a device/client just use the global model?\n\nIn summary, I find the contribution and novelty of this paper limited and the empirical findings of this paper can't be always applicable to other datasets and scenarios. Plus, I am not convinced this paper shows anything different than FedAvg rather than some recommendations about local and global optimizer selections.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have reached consensus that while the paper is interesting, it could use more time.  We urge the authors to continue their investigations.
Decision: Reject