Federated learning using mixture of experts
Authors: Edvin Listo Zec
John Martinsson
Olof Mogren
Leon René Sütfeld
Daniel Gillblad
Venue:  (2021)
Keywords: federated learning
mixture of experts

Abstract: Federated learning has received attention for its efficiency and privacy benefits,in settings where data is distributed among devices.   Although federated learning shows significant promise as a key approach when data cannot be shared or centralized, current incarnations show limited privacy properties and have short-comings  when  applied  to  common  real-world  scenarios.   One  such  scenario is heterogeneous data among devices, where data may come from different generating distributions. In this paper, we propose a federated learning framework using a mixture of experts to balance the specialist nature of a locally trained model with the generalist knowledge of a global model in a federated learning setting.  Our results show that the mixture of experts model is better suited as a personalized model for devices when data is heterogeneous, outperforming both global and lo-cal models.  Furthermore, our framework gives strict privacy guarantees, which allows clients to select parts of their data that may be excluded from the federation.   The evaluation shows that the proposed solution is robust to the setting where some users require a strict privacy setting and do not disclose their models to a central server at all, opting out from the federation partially or entirely.  The proposed framework is general enough to include any kind of machine learning models, and can even use combinations of different kinds.

Under review as a conference paper at ICLR 2021
FEDERATED LEARNING USING A MIXTURE OF EXPERTS
Anonymous authors
Paper under double-blind review
ABSTRACT
Federated learning has received attention for its efficiency and privacy benefits, in settings where data is distributed among devices. Although federated learn- ing shows significant promise key approach when data cannot be shared centralized, current incarnations show limited privacy properties and have short- comings when applied common real-world scenarios. One such scenario is heterogeneous data among devices, where data may come from different generat- ing distributions. In this paper, we propose a federated learning framework using a mixture of experts to balance the specialist nature of a locally trained model with the generalist knowledge of a global model in a federated learning setting. Our results show that the mixture of experts model is better suited as personalized model for devices when data is heterogeneous, outperforming both global and lo- cal models. Furthermore, our framework gives strict privacy guarantees, which allows clients to select parts of their data that may be excluded from the feder- ation. The evaluation shows that the proposed solution is robust the setting where some users require strict privacy setting and do not disclose their models central server all, opting out from the federation partially entirely. The proposed framework is general enough to include any kinds of machine learning models, and can even use combinations of different kinds.
INTRODUCTION
client 1
server
client 4 (opt-out)
client 2 cupt-out)
h-
client 3
client k
h?
h+
Figure 1: Overview: Federated mixtures of experts using local gating functions. Some clients opt- out from federation, not contributing to the global model and keeping their data completely private.
In many real-world scenarios, data is distributed over large number of devices, due privacy concerns or communication limitations. Federated learning is framework that can leverage this data in a distributed learning setup. This allows for exploiting both the compute power of all par- ticipating clients, and to benefit from large joint training data set. Furthermore, this is beneficial for privacy and data security. For example, keyboard prediction for smartphones, thousands or even millions of users produce keyboard input that can be leveraged can ensue directly on the devices, doing away with the need for costly data transfer, storage, and immense compute on central server (Hard et al., 2018). The medical field is another example area where data is extremely sensitive and may have to stay on premise, and a setting where anal- ysis may require distributed and proteccin approaches. In settings with such firm privacy
training data. The training

---

Under review as a conference paper at ICLR 2021
requirements, standard federated learning approaches may not be enough guarantee the needed
privacy.
The optimization problem that solve in federated learning setting is
n
min [(w) min WER*
p( [<x(w; 4, v)]
wERd
where lg is the loss for client k and (z, v) samples from the kth client's data distribution Pk- A central server coordinating training between the K local clients. The most prevalent algorithm for solving this optimization is the federated averaging (FEDAvG) algorithm (McMahan et al.) (017). In this solution, each client has its own client model, parameterized by w which is trained on a local dataset for E local epochs. When all clients have completed the training, their weights are the central server where they are aggregated into a global model, parameterized by w In FEDAVG, the & client models are combined via layer-wise averaging of parameters, weighted by the size of their respective local datasets:
to
Tik Σ n
W
wi+1
where ny the size of the dataset of client & and n = E ng. Finally, the new global model is sent out to each client, where it constitutes the starting point for the next round of (local) training. This process is repeated for a defined number of global communication rounds.
The averaging of local models in parameter space generally works but requires some care to be taken in order to ensure convergence. McMahan et al. (2017) showed that all local models need to be initialized with the same random seed for FEDAVG to work. Extended phases of local training between communication rounds can similarly break training, indicating that the individual client models will over time diverge towards different local minima in the loss landscape. Similarly, dif- ferent distributions between client datasets will also lead to divergence of client models (MeMahan al. 2017
Depending on the use case, however, the existence of local datasets and the option to train models locally can be advantageous: specialized local models, optimized for the data distribution at hand may yield higher performance in the local context than single global model. Keyboard prediction, for example. based on a global model may represent a good approximation of the population aver- age, but could provide a better experience at the hands of a user when biased towards their individual writing style and word choices. A natural question arises: when a global Fl.-trained model better than specialized local model? A specialist would be expected to perform better than global gen- cralist in a pathological non-iid setting, whereas the global generalist would be expected to perform better in an iid setting.
To address the issue of specialized local models within the federated learning setting, we propose a general framework based on mixtures of experts of local and global model on each client. Local expert models on each client are trained in parallel to the global model, followed by training local gating functions ht (ar) that aggregate the models' output depending on the input. We show advantages of this approach over fine-tuning the global model on local data in a variety of settings, and analyze the effect that different levels of variation between the local data distributions have on performance.
While standard federated learning already shows some privacy enhancing properties, it has been shown that in some settings, properties of the client and of the training data may be reconstructed from the weights communicated to the server (Wang et al. 2019). To this end, in this paper we will work with stronger notion of privacy. While existing solutions may be private enough for some settings, we will assume that client that require privacy for some of its data, needs this data to not influence the training of the global model at all. Instead, our framework allows for complete opting out from the federation with all or some of the data at any given client. Clients with such preferences will still benefit from the global model and retain a high level of performance on their own, skewed data distribution. This is important when local datasets are particularly sensitive, as may be the case in medical applications. Our experimental evaluation demonstrate the robustness of our learning framework with different levels of skewness in the data, and under varying fractions of opt-out clients.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Distributed machine learning has been studied as a strategy to allow for training data to remain with the clients, giving some aspects of privacy, while leveraging the power of learning from bigger data and compute (Konečný et al. 2016; Shokri & Shmatikov, 2015 McMahan et al. 2017 Vanhaese- brouck et al., 2016 Bellet et al. 2018). The federated averaging technique (McMahan et al. 2017) has been influential and demonstrated that layer-wise averaging of the weights in neural network models trained separately at the clients successful in many settings, producing a federated model that demonstrates some ability to generalize from limited subsets of data at the clients. However, it has been shown that federated averaging struggles when data not independent and identically dis- tributed among the clients (the non-iid setting), which shows that there is need for personalization within federated learning (Kairouz et al., 2019).
In general, addressing class imbalance with deep learning is still relatively understudied problem (Johnson& & Khoshgoftaar] (2019). A common approach for personalization is to first train a general- ist model and then fine-tune using more specific data. This approach is used in meta-learning (Finn) et al. 2017). domain adaptation (Mansour et al., 2009), and transfer learning (Oquab et al. 2014). This approach was proposed for the distributed setting by WWang et al. (2119) who used federated averaging to obtain a generalist model which was later fine-tuned locally on each client, using its specific training data. Some work has been inspired by the meta-learning paradigm to learn models that are specialized at the clients (Jiang et al. 2019 Fallah et al. 2020). AArvazhagan et al. (2019) combined this strategy and ideas from transfer learning with deep neural networks and presented a solution where shallow layers are frozen, and the deeper layers are retrained at every client.
Zhao et al. (018)) propose a strategy to improve training on non-iid client data by creating a subset of data which globally shared between all clients. Recent strategies have also explored knowledge distillation techniques for federated learning (Jeong et al., 2018 He et al. 2020: Lin et al. 2020),
which show promising results in non-iid settings.
Hanzely& & Richtárik (020) proposed solution that provides an explicit trade-off between global and local models by the introduction of alternative learning scheme that does not take the full federation step at every round, but instead takes a step in the direction towards the fed- erated average. Deng al.] (2020) proposed to combine a global model w trained using fed- erated averaging, with a local model v with a weight a;, To find optimal a; they optimize of = arg min... fi (a,v + (1 a;) w) every communication round. While this weighting scheme will balance the two models. it has no way of adapting to the strengths of the different members of the mix.
Mixture of experts (Jacobs et al. 1991> is the combination of several competing neural networks trained together with gating network to solve a common task. It was presented as an ensemble method which can be trained end to end using gradient descent. In the current work, we will apply the mixture to leverage the specific strengths of a global model trained with federated averaging, and a local model trained locally each client.
3 FEDERATED LEARNING USING A MIXTURE OF EXPERTS
In this work, we present framework for federated learning that builds on federated averaging and mixtures of experts. Our framework includes a personalized model for each client, which included in mixture together with a globally trained model using federated learning. The local models never leave the clients, which gives strong privacy properties, while the global model is trained using federated averaging and leverages larger compute and data. In our framework seen in Figure [] some clients can choose to opt-out from the federation, meaning that no information of their data is leaving the client, ensuring privacy of those clients.
Let fg be the global model with parameters Wg. We denote the index of clients by A and the local models by f/ with parameters w/- The gating function is denoted by h** parameterized with w Training in the proposed framework is divided into three main parts. First, a global model f, is trained using federated averaging using opt-in data (see Section 3.1). Second, a local model f is trained using all available data client. Third, fg and J are further trained together with gating
3

---

Under review as a conference paper at ICLR 2021
model h each client locally, using all available data the client. Steps one and two may be performed in parallel allowed by the available resources.
3.1 PRIVACY GUARANTEES
The proposed framework allows for strict form of privacy guarantee. Each client may choose arbitrary part of their data which they consider being too sensitive to use for federated learning, and information from this data will ever leave the client. The system will still leverage learning from this data by using it to train the local model ff and the gating model I This is a very flexible and useful property. For example, it allows for user of the private local models, while transforming it using some privatization mechanism and use the censored version to train the federated model.
the sensitive data in training
In general, each client dataset D is split into two non-overlapping datasets, Di and Dr, one of which has to be non-empty. The local model f and the gating model Ae is trained using the whole dataset D = D U UDD while the global model fg is trained with FEDAVG using only the opt-in dataset DJ This is visualized in Figure
3.2 OPTIMIZATION
Step 1: Federated averaging. We train the global model using FEDAVG. In other words, globally we optimize
min Celobal my EIR
min Eldd
1
Σ REDS
(en))DD [Ca(wgi x, Dy)l
D
for the opt-in dataset D4. Here lg the loss for the global model wy on client k for the prediction [x) = 0g. and D4 the kth clients opt-in data distribution.
Step 2: Train local models. The local models ff are trained only locally, sharing no information between clients, minimizing the the local loss over w Rd.
min ((wf) = min E ((..))> wfERd
[E 9.)] Vk I,...,, n.
(4)
unf ER
Here l is the loss for the prediction ÿr J (w/; 2) from the local model the input xr and D is the kith clients dataset.
Step 3: Train local mixtures. The local mixture of experts are trained using the gating models ht with the prediction error given by weighing the trained models fy and ft::
Sn- 10(2() (x) (1 (x)) fx) vk I,.... In other words, at the end of communication round, given f and fg. we optimize the mixture equation 5
(5)
min C(w wf, wh) 2o sunh
min B(x.g))D [Ca of wht x. U. 9n) wy. .0
(6)
locally for every client k- = 1,...,, input x with the model from equation over the data distribution Dk of client k. A summary of the method is described in Algorithm
Here lg is the loss from predicting új for the label y given the
Algorithm
1: input: Models participating in FEDAVG Wi,..., local expert models w/, local gate w. learning rate Ir. decay rates 82
2: Randomly initialize wi...., Wy with the same seed.
3: Randomly initialize wf and w..
4: Wy <- FEDAVG(W,......
// Train for E. local epochs and G communication rounds
5: for client do
6: we +- Addamuw Ir, B1, B2) // Train local model on client
7: Wgg wtt w Adam(ug, w², w/, lr, Br, 82) 1/ Train mixture of experts client k
8: end for
4

---

Under review as a conference paper at ICLR 2021
3.3 EXPERIMENTAL SETUP
Datasets. Our experiments are carried out on the datasets CIFAR-10, 10, CIFAR-100 (Krizhevsky ct al. 2009) and Fashion-MNIST (Xiao et al., 2. In order to simulate heterogeneous client data without labely overlapping between clients, we partition the data into 5 clients for CIFAR-10 and Fashion-MNise and 50 clients for CIFAR-100, For CIFAR-IO0 we used a client sampling strategy where in each communication round in the federation randomly sampled fraction of 0.1 of clients to participate.
Sampling non-iid. The datasets sampled in such way that each client's datasets contains two majority classes which together form p% of the client data and the remaining classes form (1 p)% of the client data. We perform experiments where vary p to see what effect the degree of heterogeneity has on performance. In the extreme case p = 1.0, each client only has two labels in total. Note that this is an extreme partitioning, where there is no overlap of majority class labels between clients, i.e. the local data distributions P; n Pj = 0 for all pairs of clients i. Also note that, we perform experiments with very little data in our experimental setup. summary of the experimental set-up can be seen in Table !
Opt-out factor. Some users might want to opt-out from participating to a global model, due to privacy reasons. These users will still receive the global model. To simulate this scenario in the experimental evaluation, we introduce an opt-out factor denoted by q. This is fraction deciding the number of clients participating in the FEDAVG optimization. The clients that participate in the federated learning optimization have all their data DÝ. while the rest of the clients that opt- out have all their data in D. q = means all clients are opt-in and participating. We perform experiments varying q, to how robust our algorithm is to different levels of client participation. In Figure | we visualize how the opt-out factor can used.
Models. In our setup, both the local model fi and the global model fg are CNNs with the same architecture. However, they are not constrained to be the same model and could be implemented any two differentiable models. The CNN has two convolutional layers with kernel size of 5, and two fully-connected layers. All layers have ReLU activations. The gating function hg has the same architecture as fg and fi, but with sigmoid activation in the last layer. We use Adam (Kingma & Ba 2014) to optimize all models, with a learning rate of 0.0001 all experiments.
Baselines. We three different models baselines. First, the locally trained model f for each client. Second, FEDAVG. Third, the final model output from FEDAVG fine-tuned for each client on its own local data. We train f/+, the fine-tuned model and the mixture using early stopping for 200 epochs, monitoring validation loss on each client.
Evaluation. For evaluation we have a held-out validation set for each client. For both CIFAR-10 and CIFAR-100 we have 400 data points for evaluation per client, sampled with the same majority class fraction p. We report an average accuracy over all clients.
Experimental set-up
Datasets
CIFAR- 10, CIFAR-100, Fashion- MNIST
Models (fg. ff )
CNNs: 2 conv. layers (5x5 kernels) and two FC layers
Activation functions
ReLU
Communication rounds (FedAvg)
1500
Local epochs (FedAvg)
3
Local batch size (FedAvg)
10
Epochs (mixture)
200 with early stopping
Batch size (mixture)
10
Optimizer
Adam
Learning rate
0.0001
Decay rates (31, (2)
(0.9, 0.999)
Training data size per client
(5, 10, 20, 38, 50, 62, 75, 100, {0.2, 0.3, 0.4, 0.5, 0.6, 0.8, 0.9, 1.0] 400
Majority class fraction
Validation data size per client
Table 1: Summary of experiments, including hyper-parameters.
5

---

Under review as a conference paper at ICLR 2021
4 RESULTS
For the sake of reproducibility, code will be made available.
In Table 2 we report accuracies and standard deviations CIFAR-10 10 for all models when data highly non-iid, i.. for p {0.8, 0.9,
1.0). In Figure we report accuracies over all majority
fractions p. By comparing Figures 3a 3 and 3c we see that FEDAVG performs substantially worse when the opt-out fraction increases, i.e. when more clients opt-out from the federation. As consequence of a weakened global model, the fine-tuned baseline decreases much in performance as well. However, our proposed mixture leverages both the global and local models and does not degrade much in performance. In Figures 2 and 4 we that
that the mixture performs just as well, or better, than the fine-tuned baseline for CIFAR- 100 and Mushion-MNIIT, respectively.
A common client sampling strategy to mitigate scalability issues in federated learning is to sample a fraction of clients to participate in each communication round FEDAVG, In Figure we sec experiments performed with different sampling fractions CIFAR- 100. In Figure 2a client fraction of 1.0 was used, and Figure 2b] client fraction of 0.1 was used. It can be seen that the difference in validation accuracy low between these two, showing that proposed method is robust to client sampling.
Experiments were also carried see what effect training data size has on performance. This is summarized as heatmap in Figure 5a for CIFAR-10 and in Figure [5b for the difference validation accuracy between the mixture and the fine-tuned baseline is shown for different majority class fractions and different client train set sizes. We see here that the mixture model outperforms the baseline most of the time, a trend which gets stronger when the training set size increases.
where
p
FEDAVG
Local
Fine-tuned
Mixture
1.0
27.35 = 3.27
84.05 + 1.84
85.30 + 3.30
86.45 5.28
0.9
35.05 = 3.47
75.05 + 3.69
75.21 + 2.88
75.35 + 2.62
0.8
39.14
65.66 + 1.37
65.61 1.52
67.17 1.96
Table 2: Accuracy unbalanced validation set for highly non-iid majority class fractions p (no class overlap between clients) on CIFAR-10, Accuracies and standard deviations reported are over four runs. Number of clients is 5, opt-out fraction q 0.0 and number of data points per clients is 500.
QFAR
QFAR 100
80
Local
Nature
Local
Mature
Pioeuncd
Fivetundd
Feeffug
Frittigg
30
20
0
0
Magrnty tractoon
Magoriyy Iraction
(a) Client fraction sampled: 1.0
(b) Client fraction sampled: 0.1
Figure 2: Accuracy on unbalanced local validation data for CIFAR-100 for different majority class fractions p. Opt-out factor q 0.0, number of clients is 50 and number of data points per client 100. Reported values are means over four runs. (a) we sample a fraction 1.0 of all clients to participate each communication round of FEDAVG, and in (b) sample fraction 0.1 of clients.

---

Under review as a conference paper at ICLR 2021
Nature Local
Nattre
Local Finslandd Fedllvg
Fintuaned
Fedl-gg
80
40
Majo
Fractiio
fraction
(a) Opt-out factor
(b) Opt-out factor g 0.5
80
10
0.2
0.3
0.4
0.5
0.6
67
0.8 0.9
1.8
(c) Opt-out factor g
Figure 3: Accuracy unbalanced local validation data for CIFAR-10 for different majority class fractions p. Number of clients 5 and train set size per client is 500. (a) opt-out factor 0.0 (b) opt-out factor 0.5 and (c) opt-out factor 0.9. Reported values are means over four runs.
Reshion
100
Halure
Lccal
Fintuued
Pedhig
70
60
50
60
02
0.3
0.4
0.5
0.6
0.7
0.8 0.9
1.0
02
03
3.4
0.5
3/5
0.0
55
10
Mayor dy
Siepeiyy dlesa
(a) Opt-out factor q =
(b) Opt-out factor g = 0.9
Figure 4: Accuracy unbalanced local validation data for Fashion- for different majority class fractions p. Number of clients is 5 and number of data points per client is 600. (a) opt-out factor 0.0 (b) opt-out factor 4 0.9. Reported values means over four runs.
7

---

Under review as a conference paper at ICLR 2021
Offerrnce datten betwear tretured CFAR-30. Cefference in anddaaiio eccurey betioen mieture and Bostuned baceline Fachlo ValeT.
Rujerty Freedor
000
(a) CIFAR-I0
(b) Fashion-MNIST
Figure 5: Heatmap showing the difference in validation accuracy on CIFAR-IO (a) and Fashion-
MNIST (b) for different majority class fractions and client train set sizes between the mixture of experts (ours) and the fine-tuned baseline. Number of clients and opt-out factor = 0.0. Reported values are means over four runs.
DISCUSSION
To address the problems of learning personalized model in a federated setting when the client data heterogeneous, we have proposed novel framework for federated mixtures experts where global model is combined with local specialist models. We find that with skewed non-iid data on the clients, our approach outperforms all other baselines, including FEDAVG, a locally trained model, and models trained first with FEDAVG and then fine-tuned on each local client in most settings. The experimental evaluation for CIFAR-10 10 shows that our approach outperforms all other methods, including the strong fine-tuning baseline (see Figure 3}. In Figure we that our proposed model outperforms the fine-tuned baseline in most settings for CIFARIO and Fashion-MNIST, especially when the number of data points per client increase. For CIFAR- 100, the proposed framework out- performs all other methods, regardless of the level of skewness (see Figure 2). In this setting, large part of the training data for each client comes from a very limited set of the available classes: two out of 100. As such, very few training examples will be available from the minority classes. This is a crucial result: the proposed framework is very robust extremely skewed training data.
The framework also gives strong privacy guarantees, where clients are able to opt-out from federa- tion, keeping their data private. The experiments show that our proposed solution robust high opt-out fraction of users, seen in Figures anddd whereas the fine-tuned baseline is not.
CONCLUSIONS
In this work, we have presented a framework for federated learning that builds on mixtures of ex- perts. This framework allows to learn model that balances the generalist nature of the global federated model and the specialist nature of the local models.
Our approach is not only an intuitive approach for the generalist vs specialist balance, but also allows for varying participation of the different clients in the federation. Clients may either opt-in and participate in the federation, or opt-out entirely by training only a local model with all its local data but still receive global model from the opt-in participants. This gives flexible solution for strong privacy guarantees in real-world settings. Our experiments show that the setting where many clients opt-out from federation, the fine-tuned baseline degrades in performance whereas proposed mixture model does not.
The proposed framework is compatible with any gradient-based based machine learning models, and can incorporate combinations of these, strengthening the potential of this direction of research, and leveraging the beneficial properties of ensembles of various machine learning models.
8

---

Under review as a conference paper at ICLR 2021
The experimental evaluation conducted in this work showed the proposed solution to achieve state- of of-the-art results in three different benchmark datasets when data highly skewed, and when parts of the clients in the federation opts out from the training.
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint 191.0008/, 2019.
Aurélien Bellet, Rachid Guerraoui, Mahsa Taziki, and Marc Tommasi. Personalized and private peer-to-peer machine learning. In International Conference Artificial Intelligence and Statis- fics, pp. 473-481, 2018.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint WXXi::2003.13461 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint #rXiv:2002.07945 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Filip Hanzely and Peter Richtárik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05531 2020.
Andrew Hard, Chloé M Kiddon, Daniel Ramage, Francoise Beaufays, Hubert Eichner, Kanishka Rao, Rajiv Mathews, and Sean Augenstein. Federated learning for mobile keyboard prediction, 2018. hnttps:/aarxiv.org//// hat////ss/1111... 03604
Chaoyang He, Salman Avestimehr, and Murali Annavaram. Group knowledge transfer: Collabora- tive training of large cnns on the edge. arXiv preprint arXiv:2007.14513. 2020.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyur Kim. Communication-efficeeet on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint 1rXiv: 1449, 2018
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning per-
sonalization via model agnostic meta learning. arXiv preprint 190. 2019.
Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data 6(1):27, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. arXiv preprint arXiv: 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 6980, 2014.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint wwXiv:1610.05499 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33, 2020.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.34450 2009.

---

Under review as a conference paper at ICLR 2021
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication learning deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp.
PMLR, 2017.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level im- age representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724. 2014.
Reza Shokri and Vitaly Shmatikov. Privacy-pree Privacy-preeerving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pp. 1310-1321, 2015.
Paul Vanhaesebrouck, Aurélien Bellet, and Marc Tommasi. Decentralized collaborative learning of personalized models over networks. arXiv preprint arXiv 2016.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10255 2019.
Z. Wang, M. Song, Z. Zhang, Y. Song. Q. Wang, and H. Qi. Beyond inferring class representatives: User-level privacy leakage from federated learning. In IEEE INFOCOM 2019 IEEE Conference Computer Communications, pp. 2512-2520, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnise a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv: 00582, 2018.
10


Reviews:
[{"review_id": "xa8Val7q1lu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a federated learning framework using a mixture of experts to trade-off the local model and the global model in a federated learning setting. A three-step pipeline is designed to train personalized FL with a mixture of global model and local model. \n\nPros:\n\n1. The proposed setting is a new scenario that considers both opt-in and opt-out devices.\n\n2. The mixture of the global and local model is a reasonable solution to solve the personalization problem in FL. \n\n\nCons:\n\n1. The paper lacks an overall loss function for the whole procedure. In particular, the three steps have three different loss functions, and the updating of the shared global model will cause inconsistent in minimizing multiple loss functions defined by equations 4, 5, and 6. \n\n2. In Section 4, there is a big blank space before the section head and Table 2. The content layout and arrangement could be improved. \n\n3. In Table 1 dataset CIFAR-10, the number of clients is 5 that is very small number for federated setting.  Moreover, the CIFAR-100 data only divided into 50 clients that are also a small number for FL.\n\n4. In Table 2, p = 1 indicates that each client has two classes only. However, the results of FedAvg are 17.13% that is much less than the reported results (85%) of the FedAvg paper by Google. \n\n5. Authors claim the proposed method can protect user privacy since a client can select which data need to be excluded from the federation. However, no corresponding experiments support this claim.\n\n6. Convergence of the gating function is not discussed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NFiZ7nO5oAV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Strengths\n\nThe authors target an important problem in Federated Learning: how to personalize the model to mitigate the nonIIDness.\n\n2. Weakness\n\nThe proposed method is not novel. The third step which fine-tunes in the local and global models using a gate network is essentially fusing the global and local models. It is surprising to me that this method works better than fine-tuned after FedAvg. Most importantly, such an empirical method lacks analysis or convincing experimental results.\n\nHyper-parameters are not well-discussed. The author mention that all experiments use the same learning rate 0.0001. This is definitely misleading. We have to adequately tune the hyper-parameters for each baseline and then make a fair comparison. Using the same learning rate for all baselines are wrong experimental settings. I believe fine-tuning after FedAvg can even get comparable performance if fully tune the hyper-parameters (learning rate, decay, batch size, epochs, rounds, etc).\n\nThe authors claim that “client and global models are not constrained to be the same model and could be implemented any two differentiable models.” However, the authors do not provide the experimental result for this argument. I guess when the model architecture is different, the difficulty of hyper-parameter optimization will increase, which weaken the application of the proposed method.\n\nThe proposed method has a severe efficiency problem. It requires holding three DNNs at the edge. This is impractical in federated learning where the edge devices are mainly resource-constrained (low memory, low computational ability)\n\nThe training time is not mentioned.\n\nThe proposed method does not use a client sampling strategy, a common practice in cross-device FL, to mitigate the scalability issue. What the performance if we want to learn 10 thousand sensors? Please check the original FedAvg for details.\n\nThe dataset CIFAR10 and CIFAR100 are not difficult enough to demonstrate the concept of the proposed algorithms. Does it still work in a high-resolution setting like ImageNet (224*224). I believe training three DNNs will lead to serious efficiency issues.\n\nThe opt-in and opt-out strategy is totally empirical without any intuition about why it works. That the author connects this strategy with a privacy guarantee is somewhat misleading to readers. Please provide an analysis in revision and properly describe the benefit.\n\nIn the Introduction section, the following argument is a lack of evidence. Please cite related works to make the argument more convincing. \n“Extended phases of local training between communication rounds can similarly break training, indicating that the individual client models will over time diverge towards different local minima in the loss landscape. Similarly, different distributions between client datasets will also lead to divergence of client models”.\n\nThe overall writing does not affect my understanding but can be improved. \n\nRelated works are not fully discussed. In some knowledge distillation-based method, the personalization is also their benefit. For example, FedGKT [1] also has a personal client model and a global server model, which is just a single step training method without the need of multiple steps training.\n\n[1] Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge. https://arxiv.org/abs/2007.14513\n\n3. Overall Score\n\nGiven the above concerns, I recommend reject this paper in the current stage. \n\n4. Questions\n\nMay I have the comparison results between \"the naive fine-tuning after FedAvg\" and the proposed method? Please fully tune hyper-parameters for each baseline.\n\n5. Suggestions\n\nI encourage the authors to do a deeper analysis and a better experimental design. If all the above concerns are addressed, I am happy to increase the score.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DJsb25ue4Vu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed method is a federated method allowing to have a certain amount of data shared between all the learners and some data specific to each learner. The targeted field of application is classification for problems where strong privacy is crucial. The method consists in learning a global classifier (with the shared data) as well as local classifiers (one per learner, using the local data). The inference, for each learner, is done with a local expert (another neural network) trained to combine inferences from the local and global models.\n\nThe first objection I will make to this paper is the confusion it causes when talking about privacy. Federated learning methods, in a context of privacy protection, aim at building a global model, using private data, without revealing anything about the private data. Here, private data are not used to build the global model, and the local model obtained is not intended to be given, which makes talking about privacy seem irrelevant to me. \n\nFor the reason explained above, I wonder about the motivation of the paper. Isn't the best solution, in this case, simply to learn a standard classifier for each local problem, using both local and global data and using a loss function that weights the examples according to the class distributions? It could be argued that if there are many local problems it is more expensive than learning a global classifier and combining it with a local classifier.  In this case efficiency is not privacy is the motivation and the gain should be evaluated in terms of efficiency. This is true, but in this case it becomes a domain adaptation problem: a model in which the distributions of p(y_i) are different from those of the local problem is given and should be used to improve the local problem. In this case, the domain adaptation methods apply and should have been investigated by the authors.\n\nMy last comment is on equation (8). I do not understand the relationship between equation (7) and (8). In (7) only the weights of the expert are learned. In (8) all weights are learned. If equation (8) is applied, this means that local data is used to learn the global model through w_g, so there is a leak of local information which contradicts the strict respect of privacy.  \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BDkcsuk_F1j", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a novel personalized federated learning method using a mixture of global and local models. In particular, a gating function is proposed to leverage the trade-off of two models on the device, and this solution is inspired by a classic work – Mixture of experts (Jacobs, 1991).\n\nPros:\n\n1. The paper proposes a new federated setting by considering opt-in and opt-out. \n\n2. In step 3, the training of the local mixture-of-experts method doesn’t need to upload data and gradients to the server-side. \n\n3. The proposed personalized federated learning framework is a practical solution. \n\nCons:\n\n1. The proposed framework requires each client to choose which part of the data is sensitive. This is a very strong assumption in real-world applications.\n\n2. The proposed framework includes three steps. Step 1 & 2 are FedAvg and local supervised learning that are existing methods. Step 3 is to train a personalized local model by mixing local and global models. The mixed-use of global and local models (equation 6) is not a novel way of federated learning. The only novelty part of the method is to apply the mixture-of-experts (Jacobs et al., 1991) method to combine the local and global models. \n\n3. The paper’s writing could be improved. The paper is an integration of the mixture-of-experts method with existing personalized federated learning. The paper's contribution is incremental.\n\n4. The authors should add a discussion to clarify how the old method (mixture-of-experts) can fit into the new environment: federated learning setting and deep learning. \n\n5. To be a paper with self-contained contents, the paper should give a clear definition of the gating model/function h^k, and how to solve the optimization problem described in equation 8.\n\n6. An overall workflow or architecture figure is recommended. \n\n7. An algorithm description is also required. \n\n8. The experiment part is too weak to validate the effectiveness of the proposed method. For example, more datasets and baselines are required.\n\n9. A typo: “We denote the number of clients by k…” It should be “denote the index of clients by k … ”\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a personalized federated learning approach using a mixture of global and local models. Four reviewers evaluated this paper; one of the reviewers is luke-warm (6) while the rest of the reviewers pretty negative to this work (3, 3, 3). The reviewers pointed out many weaknesses, especially about novelty, motivation, contribution, presentation, etc. Most importantly, although the idea of a "mixture of experts" makes sense, it is not clear what the real technical contribution of this paper is in terms of federated learning.

Considering all the comments by the reviewers, I believe that this paper is not ready yet for publication. The authors need to improve the novelty and technical soundness of the proposed direction to convince the readers including reviewers. 
Decision: Reject